[AMDGPU] Fix for branch offset hardware workaround

[rtaylor]

Summary

This fixes a hardware bug that makes a branch offset of 0x3f unsafe.
This replaces the 32 bit branch with offset 0x3f to a 64 bit
instruction that includes the same 32 bit branch and the encoding
for a s_nop 0 to follow. The relaxer than modifies the offsets
accordingly.

Change-Id: I10b7aed99d651f8159401b01bb421f105fa6288e

[arsenm]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

56–57

I would much rather avoid the proliferation of junk opcodes for this. Can you use a bundle or another way to add an independent nop instruction?

test/MC/AMDGPU/offsetbug.s

18–20

Can you use trivial instructions for sizes? I don't trust the size of most instructions to stay the same

82–83

Formatting

[arsenm]

lib/Target/AMDGPU/SOPInstructions.td

1011–1018

If the opcodes end up unavoidable, the class should be fixed to generate the branch and the dummy at the same time, rather than requiring repeating each definition with the opcode value

[rtaylor]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

56–57

I'm not sure that's possible, relaxInstruction is return the MCInst Res but I can't say for sure as I've never worked with bundles and I'm not sure how the compiler treats them. I agree though that this would be more ideal than having new 64 bit instructions.

lib/Target/AMDGPU/SOPInstructions.td

1011–1018

I can have a look at doing this.

test/MC/AMDGPU/offsetbug.s

18–20

Yes, I can. I can make them nops.

82–83

Ah, missed this, thanks.

[rtaylor]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

56–57

This would require a bundle to have a specific opcode, which from what I can see they do not.

[arsenm]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

56–57

I'm not sure what you mean. There's one BUNDLE opcode, and then instructions following I are marked as in the bundle

[rampitec]

Can we have a disasm test? I want to see that we do not mess with decoding.

[rtaylor]

In D63494#1548719, @rampitec wrote:

Can we have a disasm test? I want to see that we do not mess with decoding.

Something other than the -disassemble checks?

[rampitec]

In D63494#1548974, @rtaylor wrote:

In D63494#1548719, @rampitec wrote:

Can we have a disasm test? I want to see that we do not mess with decoding.

Something other than the -disassemble checks?

Ah, missed it. Thanks, this is sufficient.

[rtaylor]

In D63494#1548986, @rampitec wrote:

In D63494#1548974, @rtaylor wrote:

In D63494#1548719, @rampitec wrote:

Can we have a disasm test? I want to see that we do not mess with decoding.

Something other than the -disassemble checks?

Ah, missed it. Thanks, this is sufficient.

Np. Ok, thanks.

[arsenm]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

64

These names aren't good. I would prefer a sufficient like _pad_s_nop or something. 64 could mean many different things

[rampitec]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

113

I'd suggest using a multiclass to define and InstrMapping instead of switches.

[rtaylor]

Changes per suggestions

[rtaylor]

lib/Target/AMDGPU/SOPInstructions.td

1011–1018

I don't think _Real is the best naming since there isn't a corresponding Pseudo but I had no clue what to call this multiclass so if anyone has a suggestion, I'm all ears.

[rampitec]

lib/Target/AMDGPU/SOPInstructions.td

960

SOPP_With_Relaxation maybe?

962

Can yo use InstrMapping mapping here to avoid switches?

[arsenm]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

61

I would lowercase the suffix. The general naming convention its HARDWARE_NAME_compiler_defined_variant

[arsenm]

test/MC/AMDGPU/offsetbug.s

2–8

Can you split the test into separate files? I think there should be one function that needs 1 relaxation, another where the relaxation of one triggers the relaxation of the other, and another that triggers this twice

[rtaylor]

lib/Target/AMDGPU/SOPInstructions.td

962

Do you feel that this is worth it in this case? Seems overly complicated for a very specific use case but if it's the standard to go in this direction than ok.

[rampitec]

lib/Target/AMDGPU/SOPInstructions.td

962

You just get rid of one switch, so it is not that big deal now.

1011–1018

_With_Relaxation?

[rtaylor]

lib/Target/AMDGPU/SOPInstructions.td

962

That's ok, I've pretty much done it.

[rtaylor]

Suggested changes: Added InstrMapping. Three test files. Lower-cased suffixes for MI defs. Changed multiclass suffix.

[rampitec]

lib/Target/AMDGPU/SIInstrInfo.td

2316 ↗

(On Diff #205887)

I must be missing something, but how does it work? You are passing the same asm string into the both versions. What do you map to what then?

test/MC/AMDGPU/offsetbug_once.s

9 ↗

(On Diff #205887)

Indent.

test/MC/AMDGPU/offsetbug_one_and_one.s

78 ↗

(On Diff #205887)

Indent.

test/MC/AMDGPU/offsetbug_twice.s

112 ↗

(On Diff #205887)

And indent ;)

[rtaylor]

Fixed idents.

[rtaylor]

lib/Target/AMDGPU/SIInstrInfo.td

2316 ↗

(On Diff #205887)

The asm string is the same for both, the s_nop is added in at encoding. I'm using the Size field, 4 maps to 8.

This is quite similar to how VOP is doing in during getVOPe32 and get VOPe64, the difference is that they also use a VOP3 flag but it's not needed here. Here, if you notice I am just adding the s_nop in class SOPP64e.

[rampitec]

LGTM, except one indent.

lib/Target/AMDGPU/SIInstrInfo.td

2316 ↗

(On Diff #205887)

Got it, thanks!

test/MC/AMDGPU/offsetbug_once.s

9 ↗

(On Diff #205911)

Another indent.

[arsenm]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

10

This is a layering violation

72–75

return test();

[arsenm]

lib/Target/AMDGPU/SOPInstructions.td

927–929

I don't understand how this gets you a nop?

932

SOPP_w_nop or something? The current naming sounds like an actually different encoding (same with SOPPe64)

[rampitec]

lib/Target/AMDGPU/SOPInstructions.td

927–929

I guess s_nop opcode is 0, simm is 0 and SOPP encoding is 0x17f.
Maybe it would be more readable to write something like:

let Inst{63-55} = S_NOP.Inst{31-23}; // encoding
let Inst{54-48} = S_NOP.Inst{22-16}; // opcode

[rtaylor]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

10

Where would you like the InstrMapping to go? The others are here.

[rtaylor]

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

10

I suppose I could put it into Utils/AMDGPUBaseInfo.h which is already included in AMDGPUAsmBackend.cpp.... though that seems at first glance to also be a layering violation. Maybe it would be better if this was a mapping table, it would fit into AMDGPUBaseInfo.h but I think InstrMapping is more fitting.

72–75

Not sure what you mean here though the else is useless, so I'll remove that and just return false.

lib/Target/AMDGPU/SOPInstructions.td

927–929

Yes, the opcode for s_nop is 0. The value I'm using is 0 and the encoding I had is the same as everywhere else but I can see how this makes it more understandable... maybe a simple comment here would help? I fixed this but it just makes tablegen less readable honestly, since now the S_NOP def has to go after the SOPP class but before the SOPP_w_nop encoding, so it's all alone among classes. Am I missing something? Prototyping would work but again, make everything less readable.

932

Sure, that's fine.

test/MC/AMDGPU/offsetbug_once.s

9 ↗

(On Diff #205911)

Thanks.

[rtaylor]

Fixed suggestions

[arsenm]

LGTM

lib/Target/AMDGPU/AMDGPU.td

 >;
 
 def FeatureFlatSegmentOffsetBug : SubtargetFeature<"flat-segment-offset-bug",
   "HasFlatSegmentOffsetBug",
   "true",
   "GFX10 bug, inst_offset ignored in flat segment"
 >;
 
+def FeatureOffset3fBug : SubtargetFeature<"offset-3f-bug",
+  "HasOffset3fBug",
+  "true",
+  "Branch offset of 3f hardware bug"
+>;
+
 class SubtargetFeatureLDSBankCount <int Value> : SubtargetFeature <
   "ldsbankcount"#Value,
   "LDSBankCount",
   !cast<string>(Value),
   "The number of LDS banks per compute unit."
 >;
 
 def FeatureLDSBankCount16 : SubtargetFeatureLDSBankCount<16>;
def FeatureGroup {
   list<SubtargetFeature> GFX10_1_Bugs = [
     FeatureVcmpxPermlaneHazard,
     FeatureVMEMtoScalarWriteHazard,
     FeatureSMEMtoVectorWriteHazard,
     FeatureInstFwdPrefetchBug,
     FeatureVcmpxExecWARHazard,
     FeatureLdsBranchVmemWARHazard,
     FeatureNSAtoVMEMBug,
+    FeatureOffset3fBug,
     FeatureFlatSegmentOffsetBug
    ];
 }
 
 def FeatureISAVersion10_1_0 : FeatureSet<
   !listconcat(FeatureGroup.GFX10_1_Bugs,
     [FeatureGFX10,
      FeatureLDSBankCount32,
def HasDot2Insts : Predicate<"Subtarget->hasDot2Insts()">,
   AssemblerPredicate<"FeatureDot2Insts">;
 
 def HasDot5Insts : Predicate<"Subtarget->hasDot5Insts()">,
   AssemblerPredicate<"FeatureDot5Insts">;
 
 def HasDot6Insts : Predicate<"Subtarget->hasDot6Insts()">,
   AssemblerPredicate<"FeatureDot6Insts">;
 
+def HasOffset3fBug : Predicate<"!Subtarget->hasOffset3fBug()">,
+  AssemblerPredicate<"FeatureOffset3fBug">;
+
 def EnableLateCFGStructurize : Predicate<
   "EnableLateStructurizeCFG">;
 
 // Include AMDGPU TD files
 include "SISchedule.td"
 include "GCNProcessors.td"
 include "AMDGPUInstrInfo.td"
 include "AMDGPURegisterInfo.td"
 include "AMDGPURegisterBanks.td"
 include "AMDGPUInstructions.td"
 include "SIInstrInfo.td"
 include "AMDGPUCallingConv.td"
 include "AMDGPUSearchableTables.td"

lib/Target/AMDGPU/AMDGPUSubtarget.h

protected:
 
   bool HasVcmpxPermlaneHazard;
   bool HasVMEMtoScalarWriteHazard;
   bool HasSMEMtoVectorWriteHazard;
   bool HasInstFwdPrefetchBug;
   bool HasVcmpxExecWARHazard;
   bool HasLdsBranchVmemWARHazard;
   bool HasNSAtoVMEMBug;
+  bool HasOffset3fBug;
   bool HasFlatSegmentOffsetBug;
 
   // Dummy feature to use for assembler in tablegen.
   bool FeatureDisable;
 
   SelectionDAGTargetInfo TSInfo;
 private:
   SIInstrInfo InstrInfo;
public:
   bool hasDPP8() const {
     return HasDPP8;
   }
 
   bool hasR128A16() const {
     return HasR128A16;
   }
 
+  bool hasOffset3fBug() const {
+    return HasOffset3fBug;
+  }
+
   bool hasNSAEncoding() const {
     return HasNSAEncoding;
   }
 
   bool hasMadF16() const;
 
   bool enableSIScheduler() const {
     return EnableSIScheduler;

lib/Target/AMDGPU/AMDGPUSubtarget.cpp

GCNSubtarget::GCNSubtarget(const Triple &TT, StringRef GPU, StringRef FS,
 
     HasVcmpxPermlaneHazard(false),
     HasVMEMtoScalarWriteHazard(false),
     HasSMEMtoVectorWriteHazard(false),
     HasInstFwdPrefetchBug(false),
     HasVcmpxExecWARHazard(false),
     HasLdsBranchVmemWARHazard(false),
     HasNSAtoVMEMBug(false),
+    HasOffset3fBug(false),
     HasFlatSegmentOffsetBug(false),
 
     FeatureDisable(false),
     InstrInfo(initializeSubtargetDependencies(TT, GPU, FS)),
     TLInfo(TM, *this),
     FrameLowering(TargetFrameLowering::StackGrowsUp, getStackAlignment(), 0) {
   CallLoweringInfo.reset(new AMDGPUCallLowering(*getTargetLowering()));
   Legalizer.reset(new AMDGPULegalizerInfo(*this, TM));

lib/Target/AMDGPU/MCTargetDesc/AMDGPUAsmBackend.cpp

 //===-- AMDGPUAsmBackend.cpp - AMDGPU Assembler Backend -------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 /// \file
 //===----------------------------------------------------------------------===//
 
 #include "MCTargetDesc/AMDGPUFixupKinds.h"
 #include "MCTargetDesc/AMDGPUMCTargetDesc.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/MC/MCAsmBackend.h"
 #include "llvm/MC/MCAssembler.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCFixupKindInfo.h"
 #include "llvm/MC/MCObjectWriter.h"
public:
   unsigned getNumFixupKinds() const override { return AMDGPU::NumTargetFixupKinds; };
 
   void applyFixup(const MCAssembler &Asm, const MCFixup &Fixup,
                   const MCValue &Target, MutableArrayRef<char> Data,
                   uint64_t Value, bool IsResolved,
                   const MCSubtargetInfo *STI) const override;
   bool fixupNeedsRelaxation(const MCFixup &Fixup, uint64_t Value,
                             const MCRelaxableFragment *DF,
-                            const MCAsmLayout &Layout) const override {
-    return false;
-  }
+                            const MCAsmLayout &Layout) const override;
+
   void relaxInstruction(const MCInst &Inst, const MCSubtargetInfo &STI,
-                        MCInst &Res) const override {
-    llvm_unreachable("Not implemented");
-  }
+                        MCInst &Res) const override;
+
   bool mayNeedRelaxation(const MCInst &Inst,
-                         const MCSubtargetInfo &STI) const override {
-    return false;
-  }
+                         const MCSubtargetInfo &STI) const override;
 
   unsigned getMinimumNopSize() const override;
   bool writeNopData(raw_ostream &OS, uint64_t Count) const override;
 
   const MCFixupKindInfo &getFixupKindInfo(MCFixupKind Kind) const override;
 };
 
 } //End anonymous namespace
 
+static unsigned getRelaxedOpcode(const MCInst &Inst) {
+  unsigned Op = Inst.getOpcode();
+  switch (Op) {
+  default:
+    return Op;
+  case AMDGPU::S_BRANCH:
+    return AMDGPU::S_BRANCH_64;
+  case AMDGPU::S_CBRANCH_SCC0:
+    return AMDGPU::S_CBRANCH_SCC0_64;
+  case AMDGPU::S_CBRANCH_SCC1:
+    return AMDGPU::S_CBRANCH_SCC1_64;
+  case AMDGPU::S_CBRANCH_VCCZ:
+    return AMDGPU::S_CBRANCH_VCCZ_64;
+  case AMDGPU::S_CBRANCH_VCCNZ:
+    return AMDGPU::S_CBRANCH_VCCNZ_64;
+  case AMDGPU::S_CBRANCH_EXECZ:
+    return AMDGPU::S_CBRANCH_EXECZ_64;
+  case AMDGPU::S_CBRANCH_EXECNZ:
+    return AMDGPU::S_CBRANCH_EXECNZ_64;
+  case AMDGPU::S_CBRANCH_CDBGSYS:
+    return AMDGPU::S_CBRANCH_CDBGSYS_64;
+  case AMDGPU::S_CBRANCH_CDBGSYS_AND_USER:
+    return AMDGPU::S_CBRANCH_CDBGSYS_AND_USER_64;
+  case AMDGPU::S_CBRANCH_CDBGSYS_OR_USER:
+    return AMDGPU::S_CBRANCH_CDBGSYS_OR_USER_64;
+  case AMDGPU::S_CBRANCH_CDBGUSER:
+    return AMDGPU::S_CBRANCH_CDBGUSER_64;
+  } // end of switch
+}
+
+void AMDGPUAsmBackend::relaxInstruction(const MCInst &Inst,
+                                        const MCSubtargetInfo &STI,
+                                        MCInst &Res) const {
+  unsigned RelaxedOpcode = getRelaxedOpcode(Inst);
+  Res.setOpcode(RelaxedOpcode);
+  Res.addOperand(Inst.getOperand(0));
+  return;
+}
+
+bool AMDGPUAsmBackend::fixupNeedsRelaxation(const MCFixup &Fixup,
+                                            uint64_t Value,
+                                            const MCRelaxableFragment *DF,
+                                            const MCAsmLayout &Layout) const {
+  // if the branch target has an offset of x3f this needs to be relaxed to
+  // add a s_nop 0 immediately after branch to effectively increment offset
+  // for hardware workaround in gfx1010
+  if (((int64_t(Value)/4)-1) == 0x3f)
+    return true;
+  else
+    return false;
+}
+
+bool AMDGPUAsmBackend::mayNeedRelaxation(const MCInst &Inst,
+                       const MCSubtargetInfo &STI) const {
+  if (!STI.getFeatureBits()[AMDGPU::FeatureOffset3fBug])
+    return false;
+
+  switch (Inst.getOpcode()) {
+  case AMDGPU::S_BRANCH:
+  case AMDGPU::S_CBRANCH_SCC0:
+  case AMDGPU::S_CBRANCH_SCC1:
+  case AMDGPU::S_CBRANCH_VCCZ:
+  case AMDGPU::S_CBRANCH_VCCNZ:
+  case AMDGPU::S_CBRANCH_EXECZ:
+  case AMDGPU::S_CBRANCH_EXECNZ:
+  case AMDGPU::S_CBRANCH_CDBGSYS:
+  case AMDGPU::S_CBRANCH_CDBGSYS_AND_USER:
+  case AMDGPU::S_CBRANCH_CDBGSYS_OR_USER:
+  case AMDGPU::S_CBRANCH_CDBGUSER:
+    return true;
+  } // end of switch
+
+  return false;
+}
+
 static unsigned getFixupKindNumBytes(unsigned Kind) {
   switch (Kind) {
   case AMDGPU::fixup_si_sopp_br:
     return 2;
   case FK_SecRel_1:
   case FK_Data_1:
     return 1;
   case FK_SecRel_2:

lib/Target/AMDGPU/SOPInstructions.td

 class SOPPe <bits<7> op> : Enc32 {
   bits <16> simm16;
 
   let Inst{15-0} = simm16;
   let Inst{22-16} = op;
   let Inst{31-23} = 0x17f; // encoding
 }
 
+class SOPPe64 <bits<7> op> : Enc64 {
+  bits <16> simm16;
+
+  let Inst{15-0} = simm16;
+  let Inst{22-16} = op;
+  let Inst{31-23} = 0x17f; // encoding
+  let Inst{47-32} = 0x0000;
+  let Inst{54-48} = 0x00;
+  let Inst{63-55} = 0x17f; //encoding
+}
+
+class SOPP64 <bits<7> op, dag ins, string asm, list<dag> pattern = []> :
+  InstSI <(outs), ins, asm, pattern >, SOPPe64 <op> {
+
+  let mayLoad = 0;
+  let mayStore = 0;
+  let hasSideEffects = 0;
+  let SALU = 1;
+  let SOPP = 1;
+  let Size = 8;
+  let SchedRW = [WriteSALU];
+
+  let UseNamedOperandTable = 1;
+}
+
 class SOPP <bits<7> op, dag ins, string asm, list<dag> pattern = []> :
   InstSI <(outs), ins, asm, pattern >, SOPPe <op> {
 
   let mayLoad = 0;
   let mayStore = 0;
   let hasSideEffects = 0;
   let SALU = 1;
   let SOPP = 1;
   let Size = 4;
   let SchedRW = [WriteSALU];
 
   let UseNamedOperandTable = 1;
 }
 
-
 def S_NOP : SOPP <0x00000000, (ins i16imm:$simm16), "s_nop $simm16">;
 
 let isTerminator = 1 in {
 
 def S_ENDPGM : SOPP <0x00000001, (ins EndpgmImm:$simm16), "s_endpgm$simm16"> {
   let isBarrier = 1;
   let isReturn = 1;
 }
 
 def S_ENDPGM_SAVED : SOPP <0x0000001B, (ins), "s_endpgm_saved"> {
   let SubtargetPredicate = isGFX8Plus;

 } // End SubtargetPredicate = isGFX10Plus
 
 let isBranch = 1, SchedRW = [WriteBranch] in {
 def S_BRANCH : SOPP <
   0x00000002, (ins sopp_brtarget:$simm16), "s_branch $simm16",
   [(br bb:$simm16)]> {
   let isBarrier = 1;
 }
+def S_BRANCH_64 :SOPP64 <
+  0x0000002, (ins sopp_brtarget:$simm16), "s_branch $simm16",
+  [(br bb:$simm16)]> {
+  let isBarrier = 1;
+}
 
 let Uses = [SCC] in {
 def S_CBRANCH_SCC0 : SOPP <
   0x00000004, (ins sopp_brtarget:$simm16),
   "s_cbranch_scc0 $simm16"
 >;
 def S_CBRANCH_SCC1 : SOPP <
   0x00000005, (ins sopp_brtarget:$simm16),
   "s_cbranch_scc1 $simm16"
 >;
+def S_CBRANCH_SCC0_64 : SOPP64 <
+  0x00000004, (ins sopp_brtarget:$simm16),
+  "s_cbranch_scc0 $simm16"
+>;
+def S_CBRANCH_SCC1_64 : SOPP64 <
+  0x00000005, (ins sopp_brtarget:$simm16),
+  "s_cbranch_scc1 $simm16"
+>;
 } // End Uses = [SCC]
 
 let Uses = [VCC] in {
 def S_CBRANCH_VCCZ : SOPP <
   0x00000006, (ins sopp_brtarget:$simm16),
   "s_cbranch_vccz $simm16"
 >;
 def S_CBRANCH_VCCNZ : SOPP <
   0x00000007, (ins sopp_brtarget:$simm16),
   "s_cbranch_vccnz $simm16"
 >;
+def S_CBRANCH_VCCZ_64 : SOPP64 <
+  0x00000006, (ins sopp_brtarget:$simm16),
+  "s_cbranch_vccz $simm16"
+>;
+def S_CBRANCH_VCCNZ_64 : SOPP64 <
+  0x00000007, (ins sopp_brtarget:$simm16),
+  "s_cbranch_vccnz $simm16"
+>;
 } // End Uses = [VCC]
 
 let Uses = [EXEC] in {
 def S_CBRANCH_EXECZ : SOPP <
   0x00000008, (ins sopp_brtarget:$simm16),
   "s_cbranch_execz $simm16"
 >;
 def S_CBRANCH_EXECNZ : SOPP <
   0x00000009, (ins sopp_brtarget:$simm16),
   "s_cbranch_execnz $simm16"
 >;
+def S_CBRANCH_EXECZ_64 : SOPP64 <
+  0x00000008, (ins sopp_brtarget:$simm16),
+  "s_cbranch_execz $simm16"
+>;
+def S_CBRANCH_EXECNZ_64 : SOPP64 <
+  0x00000009, (ins sopp_brtarget:$simm16),
+  "s_cbranch_execnz $simm16"
+>;
 } // End Uses = [EXEC]
 
 def S_CBRANCH_CDBGSYS : SOPP <
   0x00000017, (ins sopp_brtarget:$simm16),
   "s_cbranch_cdbgsys $simm16"
 >;
+def S_CBRANCH_CDBGSYS_64 : SOPP64 <
+  0x00000017, (ins sopp_brtarget:$simm16),
+  "s_cbranch_cdbgsys $simm16"
+>;
 
 def S_CBRANCH_CDBGSYS_AND_USER : SOPP <
   0x0000001A, (ins sopp_brtarget:$simm16),
   "s_cbranch_cdbgsys_and_user $simm16"
 >;
+def S_CBRANCH_CDBGSYS_AND_USER_64 : SOPP64 <
+  0x0000001A, (ins sopp_brtarget:$simm16),
+  "s_cbranch_cdbgsys_and_user $simm16"
+>;
 
 def S_CBRANCH_CDBGSYS_OR_USER : SOPP <
   0x00000019, (ins sopp_brtarget:$simm16),
   "s_cbranch_cdbgsys_or_user $simm16"
 >;
+def S_CBRANCH_CDBGSYS_OR_USER_64 : SOPP64 <
+  0x00000019, (ins sopp_brtarget:$simm16),
+  "s_cbranch_cdbgsys_or_user $simm16"
+>;
 
 def S_CBRANCH_CDBGUSER : SOPP <
   0x00000018, (ins sopp_brtarget:$simm16),
   "s_cbranch_cdbguser $simm16"
 >;
+def S_CBRANCH_CDBGUSER_64 : SOPP64 <
+  0x00000018, (ins sopp_brtarget:$simm16),
+  "s_cbranch_cdbguser $simm16"
+>;
 
 } // End isBranch = 1
 } // End isTerminator = 1
 
 let hasSideEffects = 1 in {
 def S_BARRIER : SOPP <0x0000000a, (ins), "s_barrier",
   [(int_amdgcn_s_barrier)]> {
   let SchedRW = [WriteBarrier];

test/MC/AMDGPU/offsetbug.s

+// RUN: llvm-mc -arch=amdgcn -mcpu=gfx1010 -show-encoding %s | FileCheck %s --check-prefix=GFX10
+// RUN: llvm-mc -arch=amdgcn -mcpu=gfx1010 -filetype=obj %s | llvm-objdump -disassemble -mcpu=gfx1010 - | FileCheck %s --check-prefix=BIN
+  s_getpc_b64 s[0:1]
+  v_add_nc_u32_e32 v4, s6, v0
+  s_mov_b64 s[16:17], s[0:1]
+  s_mov_b64 s[18:19], s[0:1]
+  s_mov_b64 s[24:25], s[0:1]
+  s_mov_b32 s0, s5
+  s_mov_b32 s18, s4
+  s_mov_b32 s16, s3
+  s_mov_b32 s24, s2
+  s_load_dwordx4 s[8:11], s[0:1], 0x10
+  s_load_dwordx4 s[12:15], s[0:1], 0x0
+  s_load_dwordx4 s[4:7], s[18:19], 0x0
+  s_load_dwordx4 s[20:23], s[16:17], 0x0
+  s_load_dwordx4 s[0:3], s[24:25], 0x0
+  s_waitcnt lgkmcnt(0)
+  tbuffer_load_format_x v0, v4, s[8:11],  format:22, 0 idxen offset:4
+  tbuffer_load_format_xyzw v[9:12], v4, s[8:11],  format:56, 0 idxen offset:8
+  tbuffer_load_format_xyzw v[13:16], v4, s[8:11],  format:56, 0 idxen offset:12
+  s_waitcnt vmcnt(1)
+  s_cbranch_vccnz BB0_2
+// GFX10: s_cbranch_vccnz BB0_2           ; encoding: [A,A,0x87,0xbf]
+// GFX10-NEXT: ;   fixup A - offset: 0, value: BB0_2, kind: fixup_si_sopp_br
+// BIN: s_cbranch_vccnz BB0_2 // 00000000006C: BF870060
+  tbuffer_load_format_xyzw v[8:11], v4, s[8:11],  format:56, 0 idxen offset:16
+  tbuffer_load_format_x v1, v4, s[8:11],  format:22, 0 idxen offset:20
+  tbuffer_load_format_x v2, v4, s[8:11],  format:22, 0 idxen offset:24
+  tbuffer_load_format_x v3, v4, s[8:11],  format:22, 0 idxen
+  tbuffer_load_format_xyzw v[4:7], v4, s[12:15],  format:74, 0 idxen
+  s_buffer_load_dword s62, s[4:7], 0x0
+  v_nop
+  s_buffer_load_dwordx8 s[12:19], s[20:23], 0x0
+  s_buffer_load_dwordx4 s[8:11], s[20:23], 0x20
+  s_waitcnt lgkmcnt(0)
+  s_and_b64 vcc, exec, s[28:29]
+  s_cbranch_vccnz BB0_1
+// GFX10: s_cbranch_vccnz BB0_1           ; encoding: [A,A,0x87,0xbf]
+// GFX10-NEXT: ;   fixup A - offset: 0, value: BB0_1, kind: fixup_si_sopp_br
+// BIN: s_cbranch_vccnz BB0_1 // 0000000000BC: BF870041
+  s_nop 0
+  s_cbranch_execz BB0_3
+// GFX10: s_cbranch_execz BB0_3           ; encoding: [A,A,0x88,0xbf]
+// GFX10-NEXT: ;   fixup A - offset: 0, value: BB0_3, kind: fixup_si_sopp_br
+// BIN: s_cbranch_execz BB0_3 // 0000000000C8: BF880040
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+        s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_nop 0
+  s_buffer_load_dword s26, s[0:3], 0x48
+  s_waitcnt lgkmcnt(0)
+  s_and_b64 vcc, exec, s[28:29]
+BB0_1:
+  s_buffer_load_dword s28, s[4:7], 0x10
+BB0_3:
+  s_waitcnt lgkmcnt(0)
+  exp param0 v3, v0, v1, v2
+  exp param1 v4, v4, v4, off
+  s_cbranch_vccnz BB0_2
+// GFX10: s_cbranch_vccnz BB0_2           ; encoding: [A,A,0x87,0xbf]
+// GFX10-NEXT: ;   fixup A - offset: 0, value: BB0_2, kind: fixup_si_sopp_br
+// BIN: s_cbranch_vccnz BB0_2 // 0000000001E0: BF870003
+        s_nop 0
+        s_nop 0
+        s_nop 0
+BB0_2:
+        s_nop 0
+        s_nop 0
+        s_endpgm