[AMDGPU][llvm-mc] Fixes to support buffer atomics.

[artem.tamazov]

Summary

Fixes for MUBUF_Atomic instructions to make operand list valid:

• For RTN insns, make a copy of $vdata_in operand as $vdata.
• Do not add operand for GLC, it is hardcoded and comes as a token.

Workaround to avoid adding multiple optional default operands; more info in comments.
Tests added.

[artem.tamazov]

Ping. BTW I am going to add some tests for buffer atomics.

[artem.tamazov]

lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

1485–1487 ↗

(On Diff #57249)

Previously added operand shall be examined here. Going to fix and update the patch soon.

[artem.tamazov]

Don't add operand for GLC (add new cvtMubufAtomic() for that).
Fix w/a in parseIntWithPrefix().
Add ~1/3 of planned tests (more tomorrow).

[artem.tamazov]

Tests added, all done.

[SamWot]

lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

90–100 ↗

(On Diff #57583)

I think there is no need in this define. This workaround is always enabled and I don't see when it could be disabled. When we will be able to get rid of this we will just remove this type of operands.

670–671 ↗

(On Diff #57583)

Move this to private section of AMDGPUAsmParser where all private function reside. This really harms readability of code.

2142 ↗

(On Diff #57583)

Do we need this check at all?
At least you can replace it with simple assert

[SamWot]

lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

2141 ↗

(On Diff #57583)

You can create converter just for atomics with return value instead of converter for all atomics. In that case there would be no need in this check. This would be more stable because in change I'm currently working glc would be always parsed as immediate (not token) and this check will fail.

[artem.tamazov]

lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

90–100 ↗

(On Diff #57583)

While there is no programmatic purpose of this macro (in this specific case), there is another duties for it.
It replaces comments like "// workaround for..." which I would insert otherwise.
It clearly denotes lines of code, so it is extremely easy to remove the workaround.
It provides programmatic link to the textual description of the workaround.
Will remove, if you insist ))

[artem.tamazov]

lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

2123 ↗

(On Diff #57599)

Good idea, thanks!

[SamWot]

LGTM

[artem.tamazov]

I am going to commit this in 24hrs if no objections/comments arrive.

llvm/trunk/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

 };
 
 } // namespace SendMsg
 } // namespace AMDGPU
 } // namespace llvm
 
 using namespace llvm;
 
+// In some cases (e.g. buffer atomic instructions) MatchOperandParserImpl()
+// may invoke tryCustomParseOperand() multiple times with the same MCK value.
+// That leads to adding of the same "default" operand multiple times in a row,
+// which is wrong. The workaround adds only the 1st default operand, while for
+// the rest the "dummy" operands being added. The reason for dummies is that if
+// we just skip adding an operand, then parser would get stuck in endless loop.
+// Dummies shall be removed prior matching & emitting MCInsts.
+//
+// Comment out this macro to disable the workaround.
+#define WORKAROUND_USE_DUMMY_OPERANDS_INSTEAD_MUTIPLE_DEFAULT_OPERANDS
+
 namespace {
 
 struct OptionalOperand;
 
 enum RegisterKind { IS_UNKNOWN, IS_VGPR, IS_SGPR, IS_TTMP, IS_SPECIAL };
 
 class AMDGPUOperand : public MCParsedAsmOperand {
   enum KindTy {
     Token,
     Immediate,
     Register,
     Expression
+#ifdef WORKAROUND_USE_DUMMY_OPERANDS_INSTEAD_MUTIPLE_DEFAULT_OPERANDS
+    ,Dummy
+#endif
   } Kind;
 
   SMLoc StartLoc, EndLoc;
 
 public:
   AMDGPUOperand(enum KindTy K) : MCParsedAsmOperand(), Kind(K) {}
 
   MCContext *Ctx;
void addSoppBrTargetOperands(MCInst &Inst, unsigned N) const {
     if (isImm())
       addImmOperands(Inst, N);
     else {
       assert(isExpr());
       Inst.addOperand(MCOperand::createExpr(Expr));
     }
   }
 
+#ifdef WORKAROUND_USE_DUMMY_OPERANDS_INSTEAD_MUTIPLE_DEFAULT_OPERANDS
+  bool isDummy() const {
+    return Kind == Dummy;
+  }
+#endif
+
   bool isToken() const override {
     return Kind == Token;
   }
 
   bool isImm() const override {
     return Kind == Immediate;
   }
 
case Immediate:
       OS << " mods: " << Imm.Modifiers << '>';
       break;
     case Token:
       OS << '\'' << getToken() << '\'';
       break;
     case Expression:
       OS << "<expr " << *Expr << '>';
       break;
+#ifdef WORKAROUND_USE_DUMMY_OPERANDS_INSTEAD_MUTIPLE_DEFAULT_OPERANDS
+    case Dummy:
+      OS << "<dummy>";
+      break;
+#endif
     }
   }
 
   static AMDGPUOperand::Ptr CreateImm(int64_t Val, SMLoc Loc,
                                       enum ImmTy Type = ImmTyNone,
                                       bool IsFPImm = false) {
     auto Op = llvm::make_unique<AMDGPUOperand>(Immediate);
     Op->Imm.Val = Val;
#endif
   static AMDGPUOperand::Ptr CreateExpr(const class MCExpr *Expr, SMLoc S) {
     auto Op = llvm::make_unique<AMDGPUOperand>(Expression);
     Op->Expr = Expr;
     Op->StartLoc = S;
     Op->EndLoc = S;
     return Op;
   }
 
+#ifdef WORKAROUND_USE_DUMMY_OPERANDS_INSTEAD_MUTIPLE_DEFAULT_OPERANDS
+  static AMDGPUOperand::Ptr CreateDummy(SMLoc S) {
+    auto Op = llvm::make_unique<AMDGPUOperand>(Dummy);
+    Op->StartLoc = S;
+    Op->EndLoc = S;
+    return Op;
+  }
+#endif
+
   bool isSWaitCnt() const;
   bool isHwreg() const;
   bool isSendMsg() const;
   bool isMubufOffset() const;
   bool isSMRDOffset() const;
   bool isSMRDLiteralOffset() const;
   bool isDPPCtrl() const;
 };
private:
   bool ParseDirectiveAMDGPUHsaKernel();
   bool ParseDirectiveAMDGPUHsaModuleGlobal();
   bool ParseDirectiveAMDGPUHsaProgramGlobal();
   bool ParseSectionDirectiveHSADataGlobalAgent();
   bool ParseSectionDirectiveHSADataGlobalProgram();
   bool ParseSectionDirectiveHSARodataReadonlyAgent();
   bool AddNextRegisterToList(unsigned& Reg, unsigned& RegWidth, RegisterKind RegKind, unsigned Reg1, unsigned RegNum);
   bool ParseAMDGPURegister(RegisterKind& RegKind, unsigned& Reg, unsigned& RegNum, unsigned& RegWidth);
+  void cvtMubufImpl(MCInst &Inst, const OperandVector &Operands, bool IsAtomic, bool IsAtomicReturn);
 
 public:
   enum AMDGPUMatchResultTy {
     Match_PreferE32 = FIRST_TARGET_MATCH_RESULT_TY
   };
 
   AMDGPUAsmParser(const MCSubtargetInfo &STI, MCAsmParser &_Parser,
                const MCInstrInfo &MII,
struct OperandInfoTy {
     OperandInfoTy(int64_t Id_) : Id(Id_), IsSymbolic(false) { }
   };
   bool parseSendMsg(OperandInfoTy &Msg, OperandInfoTy &Operation, int64_t &StreamId);
 public:
   OperandMatchResultTy parseSendMsgOp(OperandVector &Operands);
   OperandMatchResultTy parseSOppBrTarget(OperandVector &Operands);
   AMDGPUOperand::Ptr defaultHwreg() const;
 
-
-  void cvtMubuf(MCInst &Inst, const OperandVector &Operands);
+  void cvtMubuf(MCInst &Inst, const OperandVector &Operands) { cvtMubufImpl(Inst, Operands, false, false); }
+  void cvtMubufAtomic(MCInst &Inst, const OperandVector &Operands) { cvtMubufImpl(Inst, Operands, true, false); }
+  void cvtMubufAtomicReturn(MCInst &Inst, const OperandVector &Operands) { cvtMubufImpl(Inst, Operands, true, true); }
   AMDGPUOperand::Ptr defaultMubufOffset() const;
   AMDGPUOperand::Ptr defaultGLC() const;
   AMDGPUOperand::Ptr defaultSLC() const;
   AMDGPUOperand::Ptr defaultTFE() const;
 
   OperandMatchResultTy parseOModSI(OperandVector &Operands) { return parseAMDGPUOperand(Operands, "omod"); }
   OperandMatchResultTy parseClampSI(OperandVector &Operands) { return parseAMDGPUOperand(Operands, "clamp"); }
   OperandMatchResultTy parseSMRDOffset(OperandVector &Operands) { return parseAMDGPUOperand(Operands, "smrd_offset"); }

 
 bool AMDGPUAsmParser::MatchAndEmitInstruction(SMLoc IDLoc, unsigned &Opcode,
                                               OperandVector &Operands,
                                               MCStreamer &Out,
                                               uint64_t &ErrorInfo,
                                               bool MatchingInlineAsm) {
   MCInst Inst;
 
+#ifdef WORKAROUND_USE_DUMMY_OPERANDS_INSTEAD_MUTIPLE_DEFAULT_OPERANDS
+  // Remove dummies prior matching. Iterate backwards becase vector::erase()
+  // invalidates all iterators which refer after erase point.
+  for (auto I = Operands.rbegin(), E = Operands.rend(); I != E; ) {
+    auto X = I++;
+    if (static_cast<AMDGPUOperand*>(X->get())->isDummy()) {
+      Operands.erase(X.base() -1);
+    }
+  }
+#endif
+
   switch (MatchInstructionImpl(Operands, Inst, ErrorInfo, MatchingInlineAsm)) {
     default: break;
     case Match_Success:
       Inst.setLoc(IDLoc);
       Out.EmitInstruction(Inst, getSTI());
       return false;
     case Match_MissingFeature:
       return Error(IDLoc, "instruction not supported on this GPU");
AMDGPUAsmParser::parseIntWithPrefix(const char *Prefix, OperandVector &Operands,
   if (Res != MatchOperand_Success)
     return Res;
 
   if (ConvertResult && !ConvertResult(Value)) {
     return MatchOperand_ParseFail;
   }
 
   Operands.push_back(AMDGPUOperand::CreateImm(Value, S, ImmTy));
+
+#ifdef WORKAROUND_USE_DUMMY_OPERANDS_INSTEAD_MUTIPLE_DEFAULT_OPERANDS
+  if (Value == Default && AddDefault) {
+    // Reverse lookup in previously added operands (skip just added one)
+    // for the first non-dummy operand. If it is of the same type,
+    // then replace just added default operand with dummy.
+    for (auto I = Operands.rbegin(), E = Operands.rend(); I != E; ++I) {
+      if (I == Operands.rbegin())
+        continue;
+      if (static_cast<AMDGPUOperand*>(I->get())->isDummy())
+        continue;
+      if (static_cast<AMDGPUOperand*>(I->get())->isImmTy(ImmTy)) {
+        Operands.pop_back();
+        Operands.push_back(AMDGPUOperand::CreateDummy(S)); // invalidates iterators
+        break;
+      }
+    }
+  }
+#endif
   return MatchOperand_Success;
 }
 
 AMDGPUAsmParser::OperandMatchResultTy
 AMDGPUAsmParser::parseNamedBit(const char *Name, OperandVector &Operands,
                                enum AMDGPUOperand::ImmTy ImmTy,
                                      bool AddDefault) {
   int64_t Bit = 0;

 AMDGPUOperand::Ptr AMDGPUAsmParser::defaultSLC() const {
   return AMDGPUOperand::CreateImm(0, SMLoc(), AMDGPUOperand::ImmTySLC);
 }
 
 AMDGPUOperand::Ptr AMDGPUAsmParser::defaultTFE() const {
   return AMDGPUOperand::CreateImm(0, SMLoc(), AMDGPUOperand::ImmTyTFE);
 }
 
-void AMDGPUAsmParser::cvtMubuf(MCInst &Inst,
-                               const OperandVector &Operands) {
+void AMDGPUAsmParser::cvtMubufImpl(MCInst &Inst,
+                               const OperandVector &Operands,
+                               bool IsAtomic, bool IsAtomicReturn) {
   OptionalImmIndexMap OptionalIdx;
+  assert(IsAtomicReturn ? IsAtomic : true);
 
   for (unsigned i = 1, e = Operands.size(); i != e; ++i) {
     AMDGPUOperand &Op = ((AMDGPUOperand &)*Operands[i]);
 
     // Add the register arguments
     if (Op.isReg()) {
       Op.addRegOperands(Inst, 1);
       continue;
if (Op.isToken()) {
       continue;
     }
     assert(Op.isImm());
 
     // Handle optional arguments
     OptionalIdx[Op.getImmTy()] = i;
   }
 
+  // Copy $vdata_in operand and insert as $vdata for MUBUF_Atomic RTN insns.
+  if (IsAtomicReturn) {
+    MCInst::iterator I = Inst.begin(); // $vdata_in is always at the beginning.
+    Inst.insert(I, *I);
+  }
+
   addOptionalImmOperand(Inst, Operands, OptionalIdx, AMDGPUOperand::ImmTyOffset);
+  if (!IsAtomic) { // glc is hard-coded.
     addOptionalImmOperand(Inst, Operands, OptionalIdx, AMDGPUOperand::ImmTyGLC);
+  }
   addOptionalImmOperand(Inst, Operands, OptionalIdx, AMDGPUOperand::ImmTySLC);
   addOptionalImmOperand(Inst, Operands, OptionalIdx, AMDGPUOperand::ImmTyTFE);
 }
 
 //===----------------------------------------------------------------------===//
 // mimg
 //===----------------------------------------------------------------------===//
 

llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.td

 }
 
 multiclass MUBUF_Atomic <mubuf op, string name, RegisterClass rc,
                          ValueType vt, SDPatternOperator atomic> {
 
   let mayStore = 1, mayLoad = 1, hasPostISelHook = 1, hasSideEffects = 1 in {
 
     // No return variants
-    let glc = 0 in {
+    let glc = 0, AsmMatchConverter = "cvtMubufAtomic" in {
 
       defm _ADDR64 : MUBUFAtomicAddr64_m <
         op, name#"_addr64", (outs),
         (ins rc:$vdata, VReg_64:$vaddr, SReg_128:$srsrc,
              SCSrc_32:$soffset, offset:$offset, slc:$slc),
         name#" $vdata, $vaddr, $srsrc, $soffset addr64$offset$slc", [], 0
       >;
 
let glc = 0, AsmMatchConverter = "cvtMubufAtomic" in {
           name#" $vdata, $vaddr, $srsrc, $soffset idxen offen$offset$slc",
           [], 0
         >;
       }
     } // glc = 0
 
     // Variant that return values
     let glc = 1, Constraints = "$vdata = $vdata_in",
+        AsmMatchConverter = "cvtMubufAtomicReturn",
         DisableEncoding = "$vdata_in"  in {
 
       defm _RTN_ADDR64 : MUBUFAtomicAddr64_m <
         op, name#"_rtn_addr64", (outs rc:$vdata),
         (ins rc:$vdata_in, VReg_64:$vaddr, SReg_128:$srsrc,
              SCSrc_32:$soffset, offset:$offset, slc:$slc),
-        name#" $vdata, $vaddr, $srsrc, $soffset addr64"#"$offset"#" glc"#"$slc",
+        name#" $vdata, $vaddr, $srsrc, $soffset addr64$offset glc$slc",
         [(set vt:$vdata,
          (atomic (MUBUFAddr64Atomic v4i32:$srsrc, i64:$vaddr, i32:$soffset,
                               i16:$offset, i1:$slc), vt:$vdata_in))], 1
       >;
 
       defm _RTN_OFFSET : MUBUFAtomicOffset_m <
         op, name#"_rtn_offset", (outs rc:$vdata),
         (ins rc:$vdata_in, SReg_128:$srsrc, SCSrc_32:$soffset,
              offset:$offset, slc:$slc),
-        name#" $vdata, off, $srsrc, $soffset $offset glc$slc",
+        name#" $vdata, off, $srsrc, $soffset$offset glc$slc",
         [(set vt:$vdata,
          (atomic (MUBUFOffsetAtomic v4i32:$srsrc, i32:$soffset, i16:$offset,
                                     i1:$slc), vt:$vdata_in))], 1
       >;
 
       let offen = 1, idxen = 0 in {
         defm _RTN_OFFEN : MUBUFAtomicOther_m <
           op, name#"_rtn_offen", (outs rc:$vdata),
           (ins rc:$vdata_in, VGPR_32:$vaddr, SReg_128:$srsrc, SCSrc_32:$soffset,
                 offset:$offset, slc:$slc),
-          name#" $vdata, $vaddr, $srsrc, $soffset offen"#"$offset"#" glc"#"$slc",
+          name#" $vdata, $vaddr, $srsrc, $soffset offen$offset glc$slc",
           [], 1
         >;
       }
 
       let offen = 0, idxen = 1 in {
         defm _RTN_IDXEN : MUBUFAtomicOther_m <
           op, name#"_rtn_idxen", (outs rc:$vdata),
           (ins rc:$vdata_in, VGPR_32:$vaddr, SReg_128:$srsrc, SCSrc_32:$soffset,
                 offset:$offset, slc:$slc),
-          name#" $vdata, $vaddr, $srsrc, $soffset idxen"#"$offset"#" glc"#"$slc",
+          name#" $vdata, $vaddr, $srsrc, $soffset idxen$offset glc$slc",
           [], 1
         >;
       }
 
       let offen = 1, idxen = 1 in {
         defm _RTN_BOTHEN : MUBUFAtomicOther_m <
           op, name#"_rtn_bothen", (outs rc:$vdata),
           (ins rc:$vdata_in, VReg_64:$vaddr, SReg_128:$srsrc, SCSrc_32:$soffset,
                 offset:$offset, slc:$slc),
-          name#" $vdata, $vaddr, $srsrc, $soffset idxen offen"#"$offset"#" glc"#"$slc",
+          name#" $vdata, $vaddr, $srsrc, $soffset idxen offen$offset glc$slc",
           [], 1
         >;
       }
     } // glc = 1
 
   } // mayStore = 1, mayLoad = 1, hasPostISelHook = 1
 }
 

llvm/trunk/test/MC/AMDGPU/mubuf.s

 // NOCI: error: instruction not supported on this GPU
 // NOVI: error: instruction not supported on this GPU
 
 buffer_wbinvl1_vol
 // CI: buffer_wbinvl1_vol ; encoding: [0x00,0x00,0xc0,0xe1,0x00,0x00,0x00,0x00]
 // VI: buffer_wbinvl1_vol ; encoding: [0x00,0x00,0xfc,0xe0,0x00,0x00,0x00,0x00]
 // NOSI: error: instruction not supported on this GPU
 
-// TODO: Atomics
+//===----------------------------------------------------------------------===//
+// Atomics
+//===----------------------------------------------------------------------===//
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 ; encoding: [0x00,0x80,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, v[2:3], s[8:11], s4 addr64
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], s4 addr64 ; encoding: [0x00,0x80,0xf0,0xe0,0x02,0x01,0x02,0x04]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 slc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 slc ; encoding: [0x00,0x80,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 offset:4
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 offset:4 ; encoding: [0x04,0x80,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 offset:4 slc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 offset:4 slc ; encoding: [0x04,0x80,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, off, s[8:11], 56
+// SICI: buffer_atomic_inc v1, off, s[8:11], 56 ; encoding: [0x00,0x00,0xf0,0xe0,0x00,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, off, s[8:11], 56 ; encoding: [0x00,0x00,0x2c,0xe1,0x00,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, off, s[8:11], 56 slc
+// SICI: buffer_atomic_inc v1, off, s[8:11], 56 slc ; encoding: [0x00,0x00,0xf0,0xe0,0x00,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, off, s[8:11], 56 slc ; encoding: [0x00,0x00,0x2e,0xe1,0x00,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, off, s[8:11], s4 slc
+// SICI: buffer_atomic_inc v1, off, s[8:11], s4 slc ; encoding: [0x00,0x00,0xf0,0xe0,0x00,0x01,0x42,0x04]
+// VI:   buffer_atomic_inc v1, off, s[8:11], s4 slc ; encoding: [0x00,0x00,0x2e,0xe1,0x00,0x01,0x02,0x04]
+
+buffer_atomic_inc v1, off, s[8:11], 56 offset:4
+// SICI: buffer_atomic_inc v1, off, s[8:11], 56 offset:4 ; encoding: [0x04,0x00,0xf0,0xe0,0x00,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, off, s[8:11], 56 offset:4 ; encoding: [0x04,0x00,0x2c,0xe1,0x00,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, off, s[8:11], 56 offset:4 slc
+// SICI: buffer_atomic_inc v1, off, s[8:11], 56 offset:4 slc ; encoding: [0x04,0x00,0xf0,0xe0,0x00,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, off, s[8:11], 56 offset:4 slc ; encoding: [0x04,0x00,0x2e,0xe1,0x00,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 offen
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 offen ; encoding: [0x00,0x10,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen ; encoding: [0x00,0x10,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 offen slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 offen slc ; encoding: [0x00,0x10,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen slc ; encoding: [0x00,0x10,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 ; encoding: [0x04,0x10,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 ; encoding: [0x04,0x10,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], s4 offen offset:4
+// SICI: buffer_atomic_inc v1, v2, s[8:11], s4 offen offset:4 ; encoding: [0x04,0x10,0xf0,0xe0,0x02,0x01,0x02,0x04]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], s4 offen offset:4 ; encoding: [0x04,0x10,0x2c,0xe1,0x02,0x01,0x02,0x04]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 slc ; encoding: [0x04,0x10,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 slc ; encoding: [0x04,0x10,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 idxen
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 idxen ; encoding: [0x00,0x20,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen ; encoding: [0x00,0x20,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 idxen slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 idxen slc ; encoding: [0x00,0x20,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen slc ; encoding: [0x00,0x20,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 ; encoding: [0x04,0x20,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 ; encoding: [0x04,0x20,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 slc ; encoding: [0x04,0x20,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 slc ; encoding: [0x04,0x20,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc ; encoding: [0x04,0x20,0xf0,0xe0,0x02,0x01,0x42,0x04]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc ; encoding: [0x04,0x20,0x2e,0xe1,0x02,0x01,0x02,0x04]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen ; encoding: [0x00,0x30,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen ; encoding: [0x00,0x30,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], s4 idxen offen
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], s4 idxen offen ; encoding: [0x00,0x30,0xf0,0xe0,0x02,0x01,0x02,0x04]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], s4 idxen offen ; encoding: [0x00,0x30,0x2c,0xe1,0x02,0x01,0x02,0x04]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen slc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen slc ; encoding: [0x00,0x30,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen slc ; encoding: [0x00,0x30,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 ; encoding: [0x04,0x30,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 ; encoding: [0x04,0x30,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 slc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 slc ; encoding: [0x04,0x30,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 slc ; encoding: [0x04,0x30,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 glc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 glc ; encoding: [0x00,0xc0,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, v[2:3], s[8:11], s4 addr64 glc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], s4 addr64 glc ; encoding: [0x00,0xc0,0xf0,0xe0,0x02,0x01,0x02,0x04]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 glc slc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 glc slc ; encoding: [0x00,0xc0,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 offset:4 glc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 offset:4 glc ; encoding: [0x04,0xc0,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 offset:4 glc slc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 addr64 offset:4 glc slc ; encoding: [0x04,0xc0,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// NOVI: error: instruction not supported on this GPU
+
+buffer_atomic_inc v1, off, s[8:11], 56 glc
+// SICI: buffer_atomic_inc v1, off, s[8:11], 56 glc ; encoding: [0x00,0x40,0xf0,0xe0,0x00,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, off, s[8:11], 56 glc ; encoding: [0x00,0x40,0x2c,0xe1,0x00,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, off, s[8:11], 56 glc slc
+// SICI: buffer_atomic_inc v1, off, s[8:11], 56 glc slc ; encoding: [0x00,0x40,0xf0,0xe0,0x00,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, off, s[8:11], 56 glc slc ; encoding: [0x00,0x40,0x2e,0xe1,0x00,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, off, s[8:11], s4 glc slc
+// SICI: buffer_atomic_inc v1, off, s[8:11], s4 glc slc ; encoding: [0x00,0x40,0xf0,0xe0,0x00,0x01,0x42,0x04]
+// VI:   buffer_atomic_inc v1, off, s[8:11], s4 glc slc ; encoding: [0x00,0x40,0x2e,0xe1,0x00,0x01,0x02,0x04]
+
+buffer_atomic_inc v1, off, s[8:11], 56 offset:4 glc
+// SICI: buffer_atomic_inc v1, off, s[8:11], 56 offset:4 glc ; encoding: [0x04,0x40,0xf0,0xe0,0x00,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, off, s[8:11], 56 offset:4 glc ; encoding: [0x04,0x40,0x2c,0xe1,0x00,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, off, s[8:11], 56 offset:4 glc slc
+// SICI: buffer_atomic_inc v1, off, s[8:11], 56 offset:4 glc slc ; encoding: [0x04,0x40,0xf0,0xe0,0x00,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, off, s[8:11], 56 offset:4 glc slc ; encoding: [0x04,0x40,0x2e,0xe1,0x00,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 offen glc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 offen glc ; encoding: [0x00,0x50,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen glc ; encoding: [0x00,0x50,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 offen glc slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 offen glc slc ; encoding: [0x00,0x50,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen glc slc ; encoding: [0x00,0x50,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 glc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 glc ; encoding: [0x04,0x50,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 glc ; encoding: [0x04,0x50,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], s4 offen offset:4 glc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], s4 offen offset:4 glc ; encoding: [0x04,0x50,0xf0,0xe0,0x02,0x01,0x02,0x04]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], s4 offen offset:4 glc ; encoding: [0x04,0x50,0x2c,0xe1,0x02,0x01,0x02,0x04]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 glc slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 glc slc ; encoding: [0x04,0x50,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 glc slc ; encoding: [0x04,0x50,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 idxen glc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 idxen glc ; encoding: [0x00,0x60,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen glc ; encoding: [0x00,0x60,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 idxen glc slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 idxen glc slc ; encoding: [0x00,0x60,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen glc slc ; encoding: [0x00,0x60,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 glc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 glc ; encoding: [0x04,0x60,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 glc ; encoding: [0x04,0x60,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 glc slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 glc slc ; encoding: [0x04,0x60,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 glc slc ; encoding: [0x04,0x60,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 glc slc
+// SICI: buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 glc slc ; encoding: [0x04,0x60,0xf0,0xe0,0x02,0x01,0x42,0x04]
+// VI:   buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 glc slc ; encoding: [0x04,0x60,0x2e,0xe1,0x02,0x01,0x02,0x04]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen glc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen glc ; encoding: [0x00,0x70,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen glc ; encoding: [0x00,0x70,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], s4 idxen offen glc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], s4 idxen offen glc ; encoding: [0x00,0x70,0xf0,0xe0,0x02,0x01,0x02,0x04]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], s4 idxen offen glc ; encoding: [0x00,0x70,0x2c,0xe1,0x02,0x01,0x02,0x04]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen glc slc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen glc slc ; encoding: [0x00,0x70,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen glc slc ; encoding: [0x00,0x70,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 glc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 glc ; encoding: [0x04,0x70,0xf0,0xe0,0x02,0x01,0x02,0xb8]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 glc ; encoding: [0x04,0x70,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+
+buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 glc slc
+// SICI: buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 glc slc ; encoding: [0x04,0x70,0xf0,0xe0,0x02,0x01,0x42,0xb8]
+// VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 glc slc ; encoding: [0x04,0x70,0x2e,0xe1,0x02,0x01,0x02,0xb8]

llvm/trunk/test/MC/AMDGPU/trap.s

 
 s_setpc_b64   [ttmp2,ttmp3]
 // SICI: s_setpc_b64 ttmp[2:3]           ; encoding: [0x72,0x20,0x80,0xbe]
 // VI:   s_setpc_b64 ttmp[2:3]           ; encoding: [0x72,0x1d,0x80,0xbe]
 
 v_readfirstlane_b32  ttmp8, v1
 // SICI: v_readfirstlane_b32 ttmp8, v1   ; encoding: [0x01,0x05,0xf0,0x7e]
 // VI:   v_readfirstlane_b32 ttmp8, v1   ; encoding: [0x01,0x05,0xf0,0x7e]
+
+buffer_atomic_inc  v1, off, ttmp[8:11], 56 glc
+// SICI: buffer_atomic_inc v1, off, ttmp[8:11], 56  glc ; encoding: [0x00,0x40,0xf0,0xe0,0x00,0x01,0x1e,0xb8]
+// VI:   buffer_atomic_inc v1, off, ttmp[8:11], 56  glc ; encoding: [0x00,0x40,0x2c,0xe1,0x00,0x01,0x1e,0xb8]

llvm/trunk/test/MC/Disassembler/AMDGPU/mubuf_vi.txt

 0x00 0x00 0x7c 0xe0 0x00 0x01 0x01 0x01
 
 # VI:   buffer_wbinvl1   ; encoding: [0x00,0x00,0xf8,0xe0,0x00,0x00,0x00,0x00]
 0x00 0x00 0xf8 0xe0 0x00 0x00 0x00 0x00
 
 # VI: buffer_wbinvl1_vol ; encoding: [0x00,0x00,0xfc,0xe0,0x00,0x00,0x00,0x00]
 0x00 0x00 0xfc 0xe0 0x00 0x00 0x00 0x00
 
+# VI:   buffer_atomic_inc v1, off, s[8:11], 56 ; encoding: [0x00,0x00,0x2c,0xe1,0x00,0x01,0x02,0xb8]
+0x00 0x00 0x2c 0xe1 0x00 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], 56 slc ; encoding: [0x00,0x00,0x2e,0xe1,0x00,0x01,0x02,0xb8]
+0x00 0x00 0x2e 0xe1 0x00 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], s4 slc ; encoding: [0x00,0x00,0x2e,0xe1,0x00,0x01,0x02,0x04]
+0x00 0x00 0x2e 0xe1 0x00 0x01 0x02 0x04
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], 56 offset:4 ; encoding: [0x04,0x00,0x2c,0xe1,0x00,0x01,0x02,0xb8]
+0x04 0x00 0x2c 0xe1 0x00 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], 56 offset:4 slc ; encoding: [0x04,0x00,0x2e,0xe1,0x00,0x01,0x02,0xb8]
+0x04 0x00 0x2e 0xe1 0x00 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen ; encoding: [0x00,0x10,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x10 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen slc ; encoding: [0x00,0x10,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x10 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 ; encoding: [0x04,0x10,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x10 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], s4 offen offset:4 ; encoding: [0x04,0x10,0x2c,0xe1,0x02,0x01,0x02,0x04]
+0x04 0x10 0x2c 0xe1 0x02 0x01 0x02 0x04
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 slc ; encoding: [0x04,0x10,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x10 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen ; encoding: [0x00,0x20,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x20 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen slc ; encoding: [0x00,0x20,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x20 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 ; encoding: [0x04,0x20,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x20 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 slc ; encoding: [0x04,0x20,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x20 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc ; encoding: [0x04,0x20,0x2e,0xe1,0x02,0x01,0x02,0x04]
+0x04 0x20 0x2e 0xe1 0x02 0x01 0x02 0x04
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen ; encoding: [0x00,0x30,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x30 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], s4 idxen offen ; encoding: [0x00,0x30,0x2c,0xe1,0x02,0x01,0x02,0x04]
+0x00 0x30 0x2c 0xe1 0x02 0x01 0x02 0x04
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen slc ; encoding: [0x00,0x30,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x30 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 ; encoding: [0x04,0x30,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x30 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 slc ; encoding: [0x04,0x30,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x30 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], 56 glc ; encoding: [0x00,0x40,0x2c,0xe1,0x00,0x01,0x02,0xb8]
+0x00 0x40 0x2c 0xe1 0x00 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], 56 glc slc ; encoding: [0x00,0x40,0x2e,0xe1,0x00,0x01,0x02,0xb8]
+0x00 0x40 0x2e 0xe1 0x00 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], s4 glc slc ; encoding: [0x00,0x40,0x2e,0xe1,0x00,0x01,0x02,0x04]
+0x00 0x40 0x2e 0xe1 0x00 0x01 0x02 0x04
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], 56 offset:4 glc ; encoding: [0x04,0x40,0x2c,0xe1,0x00,0x01,0x02,0xb8]
+0x04 0x40 0x2c 0xe1 0x00 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, off, s[8:11], 56 offset:4 glc slc ; encoding: [0x04,0x40,0x2e,0xe1,0x00,0x01,0x02,0xb8]
+0x04 0x40 0x2e 0xe1 0x00 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen glc ; encoding: [0x00,0x50,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x50 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen glc slc ; encoding: [0x00,0x50,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x50 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 glc ; encoding: [0x04,0x50,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x50 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], s4 offen offset:4 glc ; encoding: [0x04,0x50,0x2c,0xe1,0x02,0x01,0x02,0x04]
+0x04 0x50 0x2c 0xe1 0x02 0x01 0x02 0x04
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 offen offset:4 glc slc ; encoding: [0x04,0x50,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x50 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen glc ; encoding: [0x00,0x60,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x60 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen glc slc ; encoding: [0x00,0x60,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x60 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 glc ; encoding: [0x04,0x60,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x60 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], 56 idxen offset:4 glc slc ; encoding: [0x04,0x60,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x60 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 glc slc ; encoding: [0x04,0x60,0x2e,0xe1,0x02,0x01,0x02,0x04]
+0x04 0x60 0x2e 0xe1 0x02 0x01 0x02 0x04
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen glc ; encoding: [0x00,0x70,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x70 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], s4 idxen offen glc ; encoding: [0x00,0x70,0x2c,0xe1,0x02,0x01,0x02,0x04]
+0x00 0x70 0x2c 0xe1 0x02 0x01 0x02 0x04
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen glc slc ; encoding: [0x00,0x70,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x00 0x70 0x2e 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 glc ; encoding: [0x04,0x70,0x2c,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x70 0x2c 0xe1 0x02 0x01 0x02 0xb8
+
+# VI:   buffer_atomic_inc v1, v[2:3], s[8:11], 56 idxen offen offset:4 glc slc ; encoding: [0x04,0x70,0x2e,0xe1,0x02,0x01,0x02,0xb8]
+0x04 0x70 0x2e 0xe1 0x02 0x01 0x02 0xb8