[RISCV] Implement codegen for cmpxchg on RV32IA

[asb]

Summary

Utilise a similar lowering strategy to D47882.

All cmpxchg are lowered as 'strong' currently and failure ordering is ignored. I believe this is conservative but correct.

[efriedma]

You can introduce a target-specific SelectionDAG node without adding a corresponding IR intrinsic. See ISD::FIRST_TARGET_MEMORY_OPCODE.

[t.p.northover]

I obviously haven't been following RISC-V enough to give final reviews, but this mostly looked sensible to me. Just one question:

lib/Target/RISCV/RISCVISelLowering.cpp

1744 ↗

(On Diff #151176)

Is this an intentional choice not to support 64-bit pointers? Even if so, an assert might be good.

[asb]

lib/Target/RISCV/RISCVISelLowering.cpp

1744 ↗

(On Diff #151176)

RV64I isn't upstreamed yet. Most other parts of the atomics handling have asserts, and I should definitely add one here. I could try and have it do the right thing, but I think asserting is better than committing code that hasn't been tested. Good catch, thanks.

[t.p.northover]

lib/Target/RISCV/RISCVISelLowering.cpp

1744 ↗

(On Diff #151176)

Sounds good to me. A quick grep gave a handful of riscv64 uses so I wasn't entirely sure.

[asb]

Updated so common code for generating masks etc is reused through AtomicExpandPass.

[asb]

In D48131#1131431, @efriedma wrote:

You can introduce a target-specific SelectionDAG node without adding a corresponding IR intrinsic. See ISD::FIRST_TARGET_MEMORY_OPCODE.

(Sorry, just noticed I had an unsubmitted comment here).

I think in this case I do need an intrinsic, as I want mask calculation and so on to be produced as early as possible (i.e. in IR with the help of AtomicExpandPass), and then want to pass this through to my target pseudo-instruction. The only way of doing so is via an intrinsic. If the mask generation were taking place when creating or transforming the SelectionDAG it seems ISD::FIRST_TARGET_MEMORY_OPCODE would be the best approach.

[t.p.northover]

lib/Target/RISCV/RISCVExpandPseudoInsts.cpp

485–489 ↗

(On Diff #165974)

Might be better to keep the loophead/looptail labels? I was a little confused when I saw that what looks at first glance like a linear sequence of 4 instructions went into two different basic blocks.

[asb]

Update to consistently use loophead/looptail labels in comments.

[rogfer01]

lib/Target/RISCV/RISCVISelLowering.cpp

1705 ↗

(On Diff #166415)

@asb, I have a minor question here:

At some point we will want to implement this for rv64a. I presume that for rv64a we still want to have setMinCmpXchgSizeInBits(32) so the i32 and the i64 cases are handled in RISCVExpandPseudoInsts.cpp as non-masked lr/sc loops. Currently AtomicExpand.cpp will choose to use the smaller size supported (i32) and the intrinsic here will be using i32 values. This will require some form of legalisation in 64-bit, won't it? Will we end custom legalising this intrinsic call for the 64-bit case to a riscv_masked_cmpxchg_i64? Or perhaps there is a neater way to achieve this?

Thanks!

[asb]

lib/Target/RISCV/RISCVISelLowering.cpp

1705 ↗

(On Diff #166415)

Hi Roger. If i64 is the only legal type, I don't see a way around having an i64 form of the intrinsic. You then need to promote the operands to i64.

So you want to do something like

unsigned XLen = Subtarget.getXLen();
Value *Ordering = Builder.getIntN(XLen, static_cast<uint64_t>(Ord));

and use Builder.CreateSExt on the other operands.

[rogfer01]

lib/Target/RISCV/RISCVISelLowering.cpp

1705 ↗

(On Diff #166415)

Hi Alex,

If we have an i64 intrinsic, do we expect it to return i64 too? I'd say yes but then AtomicExpand.cpp (line 964), which has been working on the minimum supported CAS type of i32 will attempt to build a shr mixing an i64 and i32 and the builder complains.

Maybe I have give up the idea of having i32 as the minimum cas type in rv64a and set it to i64, and then devise a way to use a non-masked LL/SC loop later.

Anyways I don't want to bother you too much with rv64 at this point.

Thanks a lot for the prompt answer.

[asb]

Rebased patch. No changes, but I had to re-generate the new atomic-cmpxchg.ll lines as regalloc is slightly different after the recent enableMultipleCopyHints change.

[asb]

I've re-checked this patch still applies cleanly against current HEAD.

Ping? I'd really like to get this landed to complete this atomics lowering work.

[lewis-revill]

lib/Target/RISCV/RISCVInstrInfoA.td

263 ↗

(On Diff #169192)

Is there a reason why this isn't first defined as a class independent of width, like PseudoCmpXchg is (and the other pseudos in this file for that matter)? I'm just thinking of what might be used in the RV64IA implementation.

[asb]

lib/Target/RISCV/RISCVInstrInfoA.td

263 ↗

(On Diff #169192)

The masked cmpxchg and atomicrmw are only useful for 1 and 2-byte accesses, so there's no need to ever produce a 64-bit masked cmpxchg or atomicrmw.

[lewis-revill]

lib/Target/RISCV/RISCVInstrInfoA.td

263 ↗

(On Diff #169192)

Understood, thanks!

[asb]

Ping?

[asb]

This patch still applies with minimal fuzz and all tests still pass.

I'd really appreciate a review. Ping?

[jyknight]

Looks good, ship it.

llvm/trunk/include/llvm/IR/IntrinsicsRISCV.td

 def int_riscv_masked_atomicrmw_add_i32  : MaskedAtomicRMW32Intrinsic;
 def int_riscv_masked_atomicrmw_sub_i32  : MaskedAtomicRMW32Intrinsic;
 def int_riscv_masked_atomicrmw_nand_i32 : MaskedAtomicRMW32Intrinsic;
 def int_riscv_masked_atomicrmw_max_i32  : MaskedAtomicRMW32WithSextIntrinsic;
 def int_riscv_masked_atomicrmw_min_i32  : MaskedAtomicRMW32WithSextIntrinsic;
 def int_riscv_masked_atomicrmw_umax_i32 : MaskedAtomicRMW32Intrinsic;
 def int_riscv_masked_atomicrmw_umin_i32 : MaskedAtomicRMW32Intrinsic;
 
+def int_riscv_masked_cmpxchg_i32
+    : Intrinsic<[llvm_i32_ty], [llvm_anyptr_ty, llvm_i32_ty, llvm_i32_ty,
+                                llvm_i32_ty, llvm_i32_ty],
+                [IntrArgMemOnly, NoCapture<0>]>;
+
 } // TargetPrefix = "riscv"

llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp

void expandAtomicOpToLLSC(
         Instruction *I, Type *ResultTy, Value *Addr, AtomicOrdering MemOpOrder,
         function_ref<Value *(IRBuilder<> &, Value *)> PerformOp);
     void expandPartwordAtomicRMW(
         AtomicRMWInst *I,
         TargetLoweringBase::AtomicExpansionKind ExpansionKind);
     AtomicRMWInst *widenPartwordAtomicRMW(AtomicRMWInst *AI);
     void expandPartwordCmpXchg(AtomicCmpXchgInst *I);
     void expandAtomicRMWToMaskedIntrinsic(AtomicRMWInst *AI);
+    void expandAtomicCmpXchgToMaskedIntrinsic(AtomicCmpXchgInst *CI);
 
     AtomicCmpXchgInst *convertCmpXchgToIntegerType(AtomicCmpXchgInst *CI);
     static Value *insertRMWCmpXchgLoop(
         IRBuilder<> &Builder, Type *ResultType, Value *Addr,
         AtomicOrdering MemOpOrder,
         function_ref<Value *(IRBuilder<> &, Value *)> PerformOp,
         CreateCmpXchgInstFun CreateCmpXchg);
     bool tryExpandAtomicCmpXchg(AtomicCmpXchgInst *CI);
Value *OldResult = TLI->emitMaskedAtomicRMWIntrinsic(
       Builder, AI, PMV.AlignedAddr, ValOperand_Shifted, PMV.Mask, PMV.ShiftAmt,
       AI->getOrdering());
   Value *FinalOldResult = Builder.CreateTrunc(
       Builder.CreateLShr(OldResult, PMV.ShiftAmt), PMV.ValueType);
   AI->replaceAllUsesWith(FinalOldResult);
   AI->eraseFromParent();
 }
 
+void AtomicExpand::expandAtomicCmpXchgToMaskedIntrinsic(AtomicCmpXchgInst *CI) {
+  IRBuilder<> Builder(CI);
+
+  PartwordMaskValues PMV = createMaskInstrs(
+      Builder, CI, CI->getCompareOperand()->getType(), CI->getPointerOperand(),
+      TLI->getMinCmpXchgSizeInBits() / 8);
+
+  Value *CmpVal_Shifted = Builder.CreateShl(
+      Builder.CreateZExt(CI->getCompareOperand(), PMV.WordType), PMV.ShiftAmt,
+      "CmpVal_Shifted");
+  Value *NewVal_Shifted = Builder.CreateShl(
+      Builder.CreateZExt(CI->getNewValOperand(), PMV.WordType), PMV.ShiftAmt,
+      "NewVal_Shifted");
+  Value *OldVal = TLI->emitMaskedAtomicCmpXchgIntrinsic(
+      Builder, CI, PMV.AlignedAddr, CmpVal_Shifted, NewVal_Shifted, PMV.Mask,
+      CI->getSuccessOrdering());
+  Value *FinalOldVal = Builder.CreateTrunc(
+      Builder.CreateLShr(OldVal, PMV.ShiftAmt), PMV.ValueType);
+
+  Value *Res = UndefValue::get(CI->getType());
+  Res = Builder.CreateInsertValue(Res, FinalOldVal, 0);
+  Value *Success = Builder.CreateICmpEQ(
+      CmpVal_Shifted, Builder.CreateAnd(OldVal, PMV.Mask), "Success");
+  Res = Builder.CreateInsertValue(Res, Success, 1);
+
+  CI->replaceAllUsesWith(Res);
+  CI->eraseFromParent();
+}
+
 Value *AtomicExpand::insertRMWLLSCLoop(
     IRBuilder<> &Builder, Type *ResultTy, Value *Addr,
     AtomicOrdering MemOpOrder,
     function_ref<Value *(IRBuilder<> &, Value *)> PerformOp) {
   LLVMContext &Ctx = Builder.getContext();
   BasicBlock *BB = Builder.GetInsertBlock();
   Function *F = BB->getParent();
 
if (ValueSize < MinCASSize)
       expandPartwordCmpXchg(CI);
     return false;
   case TargetLoweringBase::AtomicExpansionKind::LLSC: {
     assert(ValueSize >= MinCASSize &&
            "MinCmpXchgSizeInBits not yet supported for LL/SC expansions.");
     return expandAtomicCmpXchg(CI);
   }
   case TargetLoweringBase::AtomicExpansionKind::MaskedIntrinsic:
-    llvm_unreachable(
-        "MaskedIntrinsic expansion of cmpxhg not yet implemented");
+    expandAtomicCmpXchgToMaskedIntrinsic(CI);
+    return true;
   }
 }
 
 // Note: This function is exposed externally by AtomicExpandUtils.h
 bool llvm::expandAtomicRMWToCmpXchg(AtomicRMWInst *AI,
                                     CreateCmpXchgInstFun CreateCmpXchg) {
   IRBuilder<> Builder(AI);
   Value *Loaded = AtomicExpand::insertRMWCmpXchgLoop(

llvm/trunk/lib/Target/RISCV/RISCVExpandPseudoInsts.cpp

private:
   bool expandAtomicBinOp(MachineBasicBlock &MBB,
                          MachineBasicBlock::iterator MBBI, AtomicRMWInst::BinOp,
                          bool IsMasked, int Width,
                          MachineBasicBlock::iterator &NextMBBI);
   bool expandAtomicMinMaxOp(MachineBasicBlock &MBB,
                             MachineBasicBlock::iterator MBBI,
                             AtomicRMWInst::BinOp, bool IsMasked, int Width,
                             MachineBasicBlock::iterator &NextMBBI);
+  bool expandAtomicCmpXchg(MachineBasicBlock &MBB,
+                           MachineBasicBlock::iterator MBBI, bool IsMasked,
+                           int Width, MachineBasicBlock::iterator &NextMBBI);
 };
 
 char RISCVExpandPseudo::ID = 0;
 
 bool RISCVExpandPseudo::runOnMachineFunction(MachineFunction &MF) {
   TII = static_cast<const RISCVInstrInfo *>(MF.getSubtarget().getInstrInfo());
   bool Modified = false;
   for (auto &MBB : MF)
case RISCV::PseudoMaskedAtomicLoadMin32:
     return expandAtomicMinMaxOp(MBB, MBBI, AtomicRMWInst::Min, true, 32,
                                 NextMBBI);
   case RISCV::PseudoMaskedAtomicLoadUMax32:
     return expandAtomicMinMaxOp(MBB, MBBI, AtomicRMWInst::UMax, true, 32,
                                 NextMBBI);
   case RISCV::PseudoMaskedAtomicLoadUMin32:
     return expandAtomicMinMaxOp(MBB, MBBI, AtomicRMWInst::UMin, true, 32,
                                 NextMBBI);
+  case RISCV::PseudoCmpXchg32:
+    return expandAtomicCmpXchg(MBB, MBBI, false, 32, NextMBBI);
+  case RISCV::PseudoMaskedCmpXchg32:
+    return expandAtomicCmpXchg(MBB, MBBI, true, 32, NextMBBI);
   }
 
   return false;
 }
 
 static unsigned getLRForRMW32(AtomicOrdering Ordering) {
   switch (Ordering) {
   default:
bool RISCVExpandPseudo::expandAtomicMinMaxOp(
   computeAndAddLiveIns(LiveRegs, *LoopHeadMBB);
   computeAndAddLiveIns(LiveRegs, *LoopIfBodyMBB);
   computeAndAddLiveIns(LiveRegs, *LoopTailMBB);
   computeAndAddLiveIns(LiveRegs, *DoneMBB);
 
   return true;
 }
 
+bool RISCVExpandPseudo::expandAtomicCmpXchg(
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, bool IsMasked,
+    int Width, MachineBasicBlock::iterator &NextMBBI) {
+  assert(Width == 32 && "RV64 atomic expansion currently unsupported");
+  MachineInstr &MI = *MBBI;
+  DebugLoc DL = MI.getDebugLoc();
+  MachineFunction *MF = MBB.getParent();
+  auto LoopHeadMBB = MF->CreateMachineBasicBlock(MBB.getBasicBlock());
+  auto LoopTailMBB = MF->CreateMachineBasicBlock(MBB.getBasicBlock());
+  auto DoneMBB = MF->CreateMachineBasicBlock(MBB.getBasicBlock());
+
+  // Insert new MBBs.
+  MF->insert(++MBB.getIterator(), LoopHeadMBB);
+  MF->insert(++LoopHeadMBB->getIterator(), LoopTailMBB);
+  MF->insert(++LoopTailMBB->getIterator(), DoneMBB);
+
+  // Set up successors and transfer remaining instructions to DoneMBB.
+  LoopHeadMBB->addSuccessor(LoopTailMBB);
+  LoopHeadMBB->addSuccessor(DoneMBB);
+  LoopTailMBB->addSuccessor(DoneMBB);
+  LoopTailMBB->addSuccessor(LoopHeadMBB);
+  DoneMBB->splice(DoneMBB->end(), &MBB, MI, MBB.end());
+  DoneMBB->transferSuccessors(&MBB);
+  MBB.addSuccessor(LoopHeadMBB);
+
+  unsigned DestReg = MI.getOperand(0).getReg();
+  unsigned ScratchReg = MI.getOperand(1).getReg();
+  unsigned AddrReg = MI.getOperand(2).getReg();
+  unsigned CmpValReg = MI.getOperand(3).getReg();
+  unsigned NewValReg = MI.getOperand(4).getReg();
+  AtomicOrdering Ordering =
+      static_cast<AtomicOrdering>(MI.getOperand(IsMasked ? 6 : 5).getImm());
+
+  if (!IsMasked) {
+    // .loophead:
+    //   lr.w dest, (addr)
+    //   bne dest, cmpval, done
+    BuildMI(LoopHeadMBB, DL, TII->get(getLRForRMW32(Ordering)), DestReg)
+        .addReg(AddrReg);
+    BuildMI(LoopHeadMBB, DL, TII->get(RISCV::BNE))
+        .addReg(DestReg)
+        .addReg(CmpValReg)
+        .addMBB(DoneMBB);
+    // .looptail:
+    //   sc.w scratch, newval, (addr)
+    //   bnez scratch, loophead
+    BuildMI(LoopTailMBB, DL, TII->get(getSCForRMW32(Ordering)), ScratchReg)
+        .addReg(AddrReg)
+        .addReg(NewValReg);
+    BuildMI(LoopTailMBB, DL, TII->get(RISCV::BNE))
+        .addReg(ScratchReg)
+        .addReg(RISCV::X0)
+        .addMBB(LoopHeadMBB);
+  } else {
+    // .loophead:
+    //   lr.w dest, (addr)
+    //   and scratch, dest, mask
+    //   bne scratch, cmpval, done
+    unsigned MaskReg = MI.getOperand(5).getReg();
+    BuildMI(LoopHeadMBB, DL, TII->get(getLRForRMW32(Ordering)), DestReg)
+        .addReg(AddrReg);
+    BuildMI(LoopHeadMBB, DL, TII->get(RISCV::AND), ScratchReg)
+        .addReg(DestReg)
+        .addReg(MaskReg);
+    BuildMI(LoopHeadMBB, DL, TII->get(RISCV::BNE))
+        .addReg(ScratchReg)
+        .addReg(CmpValReg)
+        .addMBB(DoneMBB);
+
+    // .looptail:
+    //   xor scratch, dest, newval
+    //   and scratch, scratch, mask
+    //   xor scratch, dest, scratch
+    //   sc.w scratch, scratch, (adrr)
+    //   bnez scratch, loophead
+    insertMaskedMerge(TII, DL, LoopTailMBB, ScratchReg, DestReg, NewValReg,
+                      MaskReg, ScratchReg);
+    BuildMI(LoopTailMBB, DL, TII->get(getSCForRMW32(Ordering)), ScratchReg)
+        .addReg(AddrReg)
+        .addReg(ScratchReg);
+    BuildMI(LoopTailMBB, DL, TII->get(RISCV::BNE))
+        .addReg(ScratchReg)
+        .addReg(RISCV::X0)
+        .addMBB(LoopHeadMBB);
+  }
+
+  NextMBBI = MBB.end();
+  MI.eraseFromParent();
+
+  LivePhysRegs LiveRegs;
+  computeAndAddLiveIns(LiveRegs, *LoopHeadMBB);
+  computeAndAddLiveIns(LiveRegs, *LoopTailMBB);
+  computeAndAddLiveIns(LiveRegs, *DoneMBB);
+
+  return true;
+}
+
 } // end of anonymous namespace
 
 INITIALIZE_PASS(RISCVExpandPseudo, "riscv-expand-pseudo",
                 RISCV_EXPAND_PSEUDO_NAME, false, false)
 namespace llvm {
 
 FunctionPass *createRISCVExpandPseudoPass() { return new RISCVExpandPseudo(); }
 
 } // end of namespace llvm

llvm/trunk/lib/Target/RISCV/RISCVISelLowering.h

bool IsEligibleForTailCallOptimization(CCState &CCInfo,
     CallLoweringInfo &CLI, MachineFunction &MF,
     const SmallVector<CCValAssign, 16> &ArgLocs) const;
 
   TargetLowering::AtomicExpansionKind
   shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
   virtual Value *emitMaskedAtomicRMWIntrinsic(
       IRBuilder<> &Builder, AtomicRMWInst *AI, Value *AlignedAddr, Value *Incr,
       Value *Mask, Value *ShiftAmt, AtomicOrdering Ord) const override;
+  TargetLowering::AtomicExpansionKind
+  shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *CI) const override;
+  virtual Value *
+  emitMaskedAtomicCmpXchgIntrinsic(IRBuilder<> &Builder, AtomicCmpXchgInst *CI,
+                                   Value *AlignedAddr, Value *CmpVal,
+                                   Value *NewVal, Value *Mask,
+                                   AtomicOrdering Ord) const override;
 };
 }
 
 #endif

llvm/trunk/lib/Target/RISCV/RISCVISelLowering.cpp

bool RISCVTargetLowering::getTgtMemIntrinsic(IntrinsicInfo &Info,
   case Intrinsic::riscv_masked_atomicrmw_xchg_i32:
   case Intrinsic::riscv_masked_atomicrmw_add_i32:
   case Intrinsic::riscv_masked_atomicrmw_sub_i32:
   case Intrinsic::riscv_masked_atomicrmw_nand_i32:
   case Intrinsic::riscv_masked_atomicrmw_max_i32:
   case Intrinsic::riscv_masked_atomicrmw_min_i32:
   case Intrinsic::riscv_masked_atomicrmw_umax_i32:
   case Intrinsic::riscv_masked_atomicrmw_umin_i32:
+  case Intrinsic::riscv_masked_cmpxchg_i32:
     PointerType *PtrTy = cast<PointerType>(I.getArgOperand(0)->getType());
     Info.opc = ISD::INTRINSIC_W_CHAIN;
     Info.memVT = MVT::getVT(PtrTy->getElementType());
     Info.ptrVal = I.getArgOperand(0);
     Info.offset = 0;
     Info.align = 4;
     Info.flags = MachineMemOperand::MOLoad | MachineMemOperand::MOStore |
                  MachineMemOperand::MOVolatile;
if (AI->getOperation() == AtomicRMWInst::Min ||
     Value *SextShamt = Builder.CreateSub(
         Builder.getInt32(Subtarget.getXLen() - ValWidth), ShiftAmt);
     return Builder.CreateCall(LrwOpScwLoop,
                               {AlignedAddr, Incr, Mask, SextShamt, Ordering});
   }
 
   return Builder.CreateCall(LrwOpScwLoop, {AlignedAddr, Incr, Mask, Ordering});
 }
+
+TargetLowering::AtomicExpansionKind
+RISCVTargetLowering::shouldExpandAtomicCmpXchgInIR(
+    AtomicCmpXchgInst *CI) const {
+  unsigned Size = CI->getCompareOperand()->getType()->getPrimitiveSizeInBits();
+  if (Size == 8 || Size == 16)
+    return AtomicExpansionKind::MaskedIntrinsic;
+  return AtomicExpansionKind::None;
+}
+
+Value *RISCVTargetLowering::emitMaskedAtomicCmpXchgIntrinsic(
+    IRBuilder<> &Builder, AtomicCmpXchgInst *CI, Value *AlignedAddr,
+    Value *CmpVal, Value *NewVal, Value *Mask, AtomicOrdering Ord) const {
+  Value *Ordering = Builder.getInt32(static_cast<uint32_t>(Ord));
+  Type *Tys[] = {AlignedAddr->getType()};
+  Function *MaskedCmpXchg = Intrinsic::getDeclaration(
+      CI->getModule(), Intrinsic::riscv_masked_cmpxchg_i32, Tys);
+  return Builder.CreateCall(MaskedCmpXchg,
+                            {AlignedAddr, CmpVal, NewVal, Mask, Ordering});
+}

llvm/trunk/lib/Target/RISCV/RISCVInstrInfoA.td

class PseudoAMO : Pseudo<(outs GPR:$res, GPR:$scratch),
                          (ins GPR:$addr, GPR:$incr, ixlenimm:$ordering), []> {
   let Constraints = "@earlyclobber $res,@earlyclobber $scratch";
   let mayLoad = 1;
   let mayStore = 1;
   let hasSideEffects = 0;
 }
 
 def PseudoAtomicLoadNand32 : PseudoAMO;
-// Ordering constants must be kept in sync with the AtomicOrdering enum in 
+// Ordering constants must be kept in sync with the AtomicOrdering enum in
 // AtomicOrdering.h.
 def : Pat<(atomic_load_nand_32_monotonic GPR:$addr, GPR:$incr),
           (PseudoAtomicLoadNand32 GPR:$addr, GPR:$incr, 2)>;
 def : Pat<(atomic_load_nand_32_acquire GPR:$addr, GPR:$incr),
           (PseudoAtomicLoadNand32 GPR:$addr, GPR:$incr, 4)>;
 def : Pat<(atomic_load_nand_32_release GPR:$addr, GPR:$incr),
           (PseudoAtomicLoadNand32 GPR:$addr, GPR:$incr, 5)>;
 def : Pat<(atomic_load_nand_32_acq_rel GPR:$addr, GPR:$incr),

 def : PseudoMaskedAMOMinMaxPat<int_riscv_masked_atomicrmw_min_i32,
                                PseudoMaskedAtomicLoadMin32>;
 def PseudoMaskedAtomicLoadUMax32 : PseudoMaskedAMOUMinUMax;
 def : PseudoMaskedAMOPat<int_riscv_masked_atomicrmw_umax_i32,
                          PseudoMaskedAtomicLoadUMax32>;
 def PseudoMaskedAtomicLoadUMin32 : PseudoMaskedAMOUMinUMax;
 def : PseudoMaskedAMOPat<int_riscv_masked_atomicrmw_umin_i32,
                          PseudoMaskedAtomicLoadUMin32>;
+
+/// Compare and exchange
+
+class PseudoCmpXchg
+    : Pseudo<(outs GPR:$res, GPR:$scratch),
+             (ins GPR:$addr, GPR:$cmpval, GPR:$newval, i32imm:$ordering), []> {
+  let Constraints = "@earlyclobber $res,@earlyclobber $scratch";
+  let mayLoad = 1;
+  let mayStore = 1;
+  let hasSideEffects = 0;
+}
+
+// Ordering constants must be kept in sync with the AtomicOrdering enum in
+// AtomicOrdering.h.
+multiclass PseudoCmpXchgPat<string Op, Pseudo CmpXchgInst> {
+  def : Pat<(!cast<PatFrag>(Op#"_monotonic") GPR:$addr, GPR:$cmp, GPR:$new),
+            (CmpXchgInst GPR:$addr, GPR:$cmp, GPR:$new, 2)>;
+  def : Pat<(!cast<PatFrag>(Op#"_acquire") GPR:$addr, GPR:$cmp, GPR:$new),
+            (CmpXchgInst GPR:$addr, GPR:$cmp, GPR:$new, 4)>;
+  def : Pat<(!cast<PatFrag>(Op#"_release") GPR:$addr, GPR:$cmp, GPR:$new),
+            (CmpXchgInst GPR:$addr, GPR:$cmp, GPR:$new, 5)>;
+  def : Pat<(!cast<PatFrag>(Op#"_acq_rel") GPR:$addr, GPR:$cmp, GPR:$new),
+            (CmpXchgInst GPR:$addr, GPR:$cmp, GPR:$new, 6)>;
+  def : Pat<(!cast<PatFrag>(Op#"_seq_cst") GPR:$addr, GPR:$cmp, GPR:$new),
+            (CmpXchgInst GPR:$addr, GPR:$cmp, GPR:$new, 7)>;
+}
+
+def PseudoCmpXchg32 : PseudoCmpXchg;
+defm : PseudoCmpXchgPat<"atomic_cmp_swap_32", PseudoCmpXchg32>;
+
+def PseudoMaskedCmpXchg32
+    : Pseudo<(outs GPR:$res, GPR:$scratch),
+             (ins GPR:$addr, GPR:$cmpval, GPR:$newval, GPR:$mask,
+              i32imm:$ordering), []> {
+  let Constraints = "@earlyclobber $res,@earlyclobber $scratch";
+  let mayLoad = 1;
+  let mayStore = 1;
+  let hasSideEffects = 0;
+}
+
+def : Pat<(int_riscv_masked_cmpxchg_i32
+            GPR:$addr, GPR:$cmpval, GPR:$newval, GPR:$mask, imm:$ordering),
+          (PseudoMaskedCmpXchg32
+            GPR:$addr, GPR:$cmpval, GPR:$newval, GPR:$mask, imm:$ordering)>;
+
 } // Predicates = [HasStdExtA]

llvm/trunk/test/CodeGen/RISCV/atomic-cmpxchg.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -verify-machineinstrs < %s \
 ; RUN:   | FileCheck -check-prefix=RV32I %s
+; RUN: llc -mtriple=riscv32 -mattr=+a -verify-machineinstrs < %s \
+; RUN:   | FileCheck -check-prefix=RV32IA %s
 
 define void @cmpxchg_i8_monotonic_monotonic(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_monotonic_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    mv a3, zero
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_monotonic_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB0_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB0_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB0_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB0_1
+; RV32IA-NEXT:  .LBB0_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val monotonic monotonic
   ret void
 }
 
 define void @cmpxchg_i8_acquire_monotonic(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_acquire_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 2
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_acquire_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB1_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB1_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB1_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB1_1
+; RV32IA-NEXT:  .LBB1_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val acquire monotonic
   ret void
 }
 
 define void @cmpxchg_i8_acquire_acquire(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_acquire_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 2
 ; RV32I-NEXT:    mv a4, a3
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_acquire_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB2_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB2_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB2_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB2_1
+; RV32IA-NEXT:  .LBB2_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val acquire acquire
   ret void
 }
 
 define void @cmpxchg_i8_release_monotonic(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_release_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 3
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_release_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB3_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB3_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB3_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w.rl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB3_1
+; RV32IA-NEXT:  .LBB3_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val release monotonic
   ret void
 }
 
 define void @cmpxchg_i8_release_acquire(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_release_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 3
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_release_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB4_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB4_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB4_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w.rl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB4_1
+; RV32IA-NEXT:  .LBB4_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val release acquire
   ret void
 }
 
 define void @cmpxchg_i8_acq_rel_monotonic(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_acq_rel_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 4
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_acq_rel_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB5_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB5_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB5_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w.rl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB5_1
+; RV32IA-NEXT:  .LBB5_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val acq_rel monotonic
   ret void
 }
 
 define void @cmpxchg_i8_acq_rel_acquire(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_acq_rel_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 4
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_acq_rel_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB6_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB6_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB6_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w.rl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB6_1
+; RV32IA-NEXT:  .LBB6_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val acq_rel acquire
   ret void
 }
 
 define void @cmpxchg_i8_seq_cst_monotonic(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_seq_cst_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_seq_cst_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB7_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB7_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB7_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w.aqrl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB7_1
+; RV32IA-NEXT:  .LBB7_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val seq_cst monotonic
   ret void
 }
 
 define void @cmpxchg_i8_seq_cst_acquire(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_seq_cst_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_seq_cst_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB8_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB8_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB8_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w.aqrl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB8_1
+; RV32IA-NEXT:  .LBB8_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val seq_cst acquire
   ret void
 }
 
 define void @cmpxchg_i8_seq_cst_seq_cst(i8* %ptr, i8 %cmp, i8 %val) {
 ; RV32I-LABEL: cmpxchg_i8_seq_cst_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sb a1, 11(sp)
 ; RV32I-NEXT:    addi a1, sp, 11
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    mv a4, a3
 ; RV32I-NEXT:    call __atomic_compare_exchange_1
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i8_seq_cst_seq_cst:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    slli a3, a0, 3
+; RV32IA-NEXT:    andi a3, a3, 24
+; RV32IA-NEXT:    addi a4, zero, 255
+; RV32IA-NEXT:    sll a4, a4, a3
+; RV32IA-NEXT:    andi a2, a2, 255
+; RV32IA-NEXT:    sll a2, a2, a3
+; RV32IA-NEXT:    andi a1, a1, 255
+; RV32IA-NEXT:    sll a1, a1, a3
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB9_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a3, (a0)
+; RV32IA-NEXT:    and a5, a3, a4
+; RV32IA-NEXT:    bne a5, a1, .LBB9_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB9_1 Depth=1
+; RV32IA-NEXT:    xor a5, a3, a2
+; RV32IA-NEXT:    and a5, a5, a4
+; RV32IA-NEXT:    xor a5, a3, a5
+; RV32IA-NEXT:    sc.w.aqrl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB9_1
+; RV32IA-NEXT:  .LBB9_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i8* %ptr, i8 %cmp, i8 %val seq_cst seq_cst
   ret void
 }
 
 define void @cmpxchg_i16_monotonic_monotonic(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_monotonic_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    mv a3, zero
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_monotonic_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB10_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB10_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB10_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB10_1
+; RV32IA-NEXT:  .LBB10_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val monotonic monotonic
   ret void
 }
 
 define void @cmpxchg_i16_acquire_monotonic(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_acquire_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 2
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_acquire_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB11_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB11_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB11_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB11_1
+; RV32IA-NEXT:  .LBB11_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val acquire monotonic
   ret void
 }
 
 define void @cmpxchg_i16_acquire_acquire(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_acquire_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 2
 ; RV32I-NEXT:    mv a4, a3
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_acquire_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB12_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB12_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB12_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB12_1
+; RV32IA-NEXT:  .LBB12_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val acquire acquire
   ret void
 }
 
 define void @cmpxchg_i16_release_monotonic(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_release_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 3
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_release_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB13_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB13_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB13_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w.rl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB13_1
+; RV32IA-NEXT:  .LBB13_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val release monotonic
   ret void
 }
 
 define void @cmpxchg_i16_release_acquire(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_release_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 3
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_release_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB14_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB14_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB14_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w.rl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB14_1
+; RV32IA-NEXT:  .LBB14_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val release acquire
   ret void
 }
 
 define void @cmpxchg_i16_acq_rel_monotonic(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_acq_rel_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 4
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_acq_rel_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB15_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB15_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w.rl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB15_1
+; RV32IA-NEXT:  .LBB15_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val acq_rel monotonic
   ret void
 }
 
 define void @cmpxchg_i16_acq_rel_acquire(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_acq_rel_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 4
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_acq_rel_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB16_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB16_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB16_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w.rl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB16_1
+; RV32IA-NEXT:  .LBB16_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val acq_rel acquire
   ret void
 }
 
 define void @cmpxchg_i16_seq_cst_monotonic(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_seq_cst_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_seq_cst_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB17_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB17_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB17_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w.aqrl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB17_1
+; RV32IA-NEXT:  .LBB17_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val seq_cst monotonic
   ret void
 }
 
 define void @cmpxchg_i16_seq_cst_acquire(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_seq_cst_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_seq_cst_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB18_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB18_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB18_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w.aqrl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB18_1
+; RV32IA-NEXT:  .LBB18_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val seq_cst acquire
   ret void
 }
 
 define void @cmpxchg_i16_seq_cst_seq_cst(i16* %ptr, i16 %cmp, i16 %val) {
 ; RV32I-LABEL: cmpxchg_i16_seq_cst_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sh a1, 10(sp)
 ; RV32I-NEXT:    addi a1, sp, 10
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    mv a4, a3
 ; RV32I-NEXT:    call __atomic_compare_exchange_2
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i16_seq_cst_seq_cst:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    lui a3, 16
+; RV32IA-NEXT:    addi a3, a3, -1
+; RV32IA-NEXT:    and a1, a1, a3
+; RV32IA-NEXT:    and a2, a2, a3
+; RV32IA-NEXT:    slli a4, a0, 3
+; RV32IA-NEXT:    andi a4, a4, 24
+; RV32IA-NEXT:    sll a3, a3, a4
+; RV32IA-NEXT:    sll a2, a2, a4
+; RV32IA-NEXT:    sll a1, a1, a4
+; RV32IA-NEXT:    andi a0, a0, -4
+; RV32IA-NEXT:  .LBB19_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a4, (a0)
+; RV32IA-NEXT:    and a5, a4, a3
+; RV32IA-NEXT:    bne a5, a1, .LBB19_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB19_1 Depth=1
+; RV32IA-NEXT:    xor a5, a4, a2
+; RV32IA-NEXT:    and a5, a5, a3
+; RV32IA-NEXT:    xor a5, a4, a5
+; RV32IA-NEXT:    sc.w.aqrl a5, a5, (a0)
+; RV32IA-NEXT:    bnez a5, .LBB19_1
+; RV32IA-NEXT:  .LBB19_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i16* %ptr, i16 %cmp, i16 %val seq_cst seq_cst
   ret void
 }
 
 define void @cmpxchg_i32_monotonic_monotonic(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_monotonic_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    mv a3, zero
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_monotonic_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB20_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB20_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB20_1 Depth=1
+; RV32IA-NEXT:    sc.w a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB20_1
+; RV32IA-NEXT:  .LBB20_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val monotonic monotonic
   ret void
 }
 
 define void @cmpxchg_i32_acquire_monotonic(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_acquire_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 2
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_acquire_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB21_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB21_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB21_1 Depth=1
+; RV32IA-NEXT:    sc.w a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB21_1
+; RV32IA-NEXT:  .LBB21_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val acquire monotonic
   ret void
 }
 
 define void @cmpxchg_i32_acquire_acquire(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_acquire_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 2
 ; RV32I-NEXT:    mv a4, a3
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_acquire_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB22_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB22_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB22_1 Depth=1
+; RV32IA-NEXT:    sc.w a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB22_1
+; RV32IA-NEXT:  .LBB22_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val acquire acquire
   ret void
 }
 
 define void @cmpxchg_i32_release_monotonic(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_release_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 3
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_release_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB23_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB23_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB23_1 Depth=1
+; RV32IA-NEXT:    sc.w.rl a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB23_1
+; RV32IA-NEXT:  .LBB23_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val release monotonic
   ret void
 }
 
 define void @cmpxchg_i32_release_acquire(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_release_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 3
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_release_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB24_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB24_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB24_1 Depth=1
+; RV32IA-NEXT:    sc.w.rl a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB24_1
+; RV32IA-NEXT:  .LBB24_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val release acquire
   ret void
 }
 
 define void @cmpxchg_i32_acq_rel_monotonic(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_acq_rel_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 4
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_acq_rel_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB25_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB25_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB25_1 Depth=1
+; RV32IA-NEXT:    sc.w.rl a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB25_1
+; RV32IA-NEXT:  .LBB25_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val acq_rel monotonic
   ret void
 }
 
 define void @cmpxchg_i32_acq_rel_acquire(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_acq_rel_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 4
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_acq_rel_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB26_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aq a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB26_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB26_1 Depth=1
+; RV32IA-NEXT:    sc.w.rl a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB26_1
+; RV32IA-NEXT:  .LBB26_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val acq_rel acquire
   ret void
 }
 
 define void @cmpxchg_i32_seq_cst_monotonic(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_seq_cst_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_seq_cst_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB27_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB27_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB27_1 Depth=1
+; RV32IA-NEXT:    sc.w.aqrl a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB27_1
+; RV32IA-NEXT:  .LBB27_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val seq_cst monotonic
   ret void
 }
 
 define void @cmpxchg_i32_seq_cst_acquire(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_seq_cst_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_seq_cst_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB28_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB28_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB28_1 Depth=1
+; RV32IA-NEXT:    sc.w.aqrl a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB28_1
+; RV32IA-NEXT:  .LBB28_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val seq_cst acquire
   ret void
 }
 
 define void @cmpxchg_i32_seq_cst_seq_cst(i32* %ptr, i32 %cmp, i32 %val) {
 ; RV32I-LABEL: cmpxchg_i32_seq_cst_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a1, 8(sp)
 ; RV32I-NEXT:    addi a1, sp, 8
 ; RV32I-NEXT:    addi a3, zero, 5
 ; RV32I-NEXT:    mv a4, a3
 ; RV32I-NEXT:    call __atomic_compare_exchange_4
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i32_seq_cst_seq_cst:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:  .LBB29_1: # =>This Inner Loop Header: Depth=1
+; RV32IA-NEXT:    lr.w.aqrl a3, (a0)
+; RV32IA-NEXT:    bne a3, a1, .LBB29_3
+; RV32IA-NEXT:  # %bb.2: # in Loop: Header=BB29_1 Depth=1
+; RV32IA-NEXT:    sc.w.aqrl a4, a2, (a0)
+; RV32IA-NEXT:    bnez a4, .LBB29_1
+; RV32IA-NEXT:  .LBB29_3:
+; RV32IA-NEXT:    ret
   %res = cmpxchg i32* %ptr, i32 %cmp, i32 %val seq_cst seq_cst
   ret void
 }
 
 define void @cmpxchg_i64_monotonic_monotonic(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_monotonic_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a4
 ; RV32I-NEXT:    mv a4, zero
 ; RV32I-NEXT:    mv a5, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_monotonic_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a4
+; RV32IA-NEXT:    mv a4, zero
+; RV32IA-NEXT:    mv a5, zero
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val monotonic monotonic
   ret void
 }
 
 define void @cmpxchg_i64_acquire_monotonic(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_acquire_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    mv a5, a4
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a4, zero, 2
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a5
 ; RV32I-NEXT:    mv a5, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_acquire_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    mv a5, a4
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a4, zero, 2
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a5
+; RV32IA-NEXT:    mv a5, zero
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val acquire monotonic
   ret void
 }
 
 define void @cmpxchg_i64_acquire_acquire(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_acquire_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a5, zero, 2
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a4
 ; RV32I-NEXT:    mv a4, a5
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_acquire_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a5, zero, 2
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a4
+; RV32IA-NEXT:    mv a4, a5
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val acquire acquire
   ret void
 }
 
 define void @cmpxchg_i64_release_monotonic(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_release_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    mv a5, a4
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a4, zero, 3
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a5
 ; RV32I-NEXT:    mv a5, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_release_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    mv a5, a4
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a4, zero, 3
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a5
+; RV32IA-NEXT:    mv a5, zero
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val release monotonic
   ret void
 }
 
 define void @cmpxchg_i64_release_acquire(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_release_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    mv a6, a4
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a4, zero, 3
 ; RV32I-NEXT:    addi a5, zero, 2
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a6
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_release_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    mv a6, a4
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a4, zero, 3
+; RV32IA-NEXT:    addi a5, zero, 2
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a6
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val release acquire
   ret void
 }
 
 define void @cmpxchg_i64_acq_rel_monotonic(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_acq_rel_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    mv a5, a4
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a4, zero, 4
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a5
 ; RV32I-NEXT:    mv a5, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_acq_rel_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    mv a5, a4
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a4, zero, 4
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a5
+; RV32IA-NEXT:    mv a5, zero
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val acq_rel monotonic
   ret void
 }
 
 define void @cmpxchg_i64_acq_rel_acquire(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_acq_rel_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    mv a6, a4
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a4, zero, 4
 ; RV32I-NEXT:    addi a5, zero, 2
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a6
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_acq_rel_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    mv a6, a4
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a4, zero, 4
+; RV32IA-NEXT:    addi a5, zero, 2
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a6
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val acq_rel acquire
   ret void
 }
 
 define void @cmpxchg_i64_seq_cst_monotonic(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_seq_cst_monotonic:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    mv a5, a4
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a4, zero, 5
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a5
 ; RV32I-NEXT:    mv a5, zero
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_seq_cst_monotonic:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    mv a5, a4
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a4, zero, 5
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a5
+; RV32IA-NEXT:    mv a5, zero
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val seq_cst monotonic
   ret void
 }
 
 define void @cmpxchg_i64_seq_cst_acquire(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_seq_cst_acquire:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    mv a6, a4
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a4, zero, 5
 ; RV32I-NEXT:    addi a5, zero, 2
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a6
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_seq_cst_acquire:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    mv a6, a4
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a4, zero, 5
+; RV32IA-NEXT:    addi a5, zero, 2
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a6
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val seq_cst acquire
   ret void
 }
 
 define void @cmpxchg_i64_seq_cst_seq_cst(i64* %ptr, i64 %cmp, i64 %val) {
 ; RV32I-LABEL: cmpxchg_i64_seq_cst_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp)
 ; RV32I-NEXT:    sw a2, 4(sp)
 ; RV32I-NEXT:    sw a1, 0(sp)
 ; RV32I-NEXT:    mv a1, sp
 ; RV32I-NEXT:    addi a5, zero, 5
 ; RV32I-NEXT:    mv a2, a3
 ; RV32I-NEXT:    mv a3, a4
 ; RV32I-NEXT:    mv a4, a5
 ; RV32I-NEXT:    call __atomic_compare_exchange_8
 ; RV32I-NEXT:    lw ra, 12(sp)
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
+;
+; RV32IA-LABEL: cmpxchg_i64_seq_cst_seq_cst:
+; RV32IA:       # %bb.0:
+; RV32IA-NEXT:    addi sp, sp, -16
+; RV32IA-NEXT:    sw ra, 12(sp)
+; RV32IA-NEXT:    sw a2, 4(sp)
+; RV32IA-NEXT:    sw a1, 0(sp)
+; RV32IA-NEXT:    mv a1, sp
+; RV32IA-NEXT:    addi a5, zero, 5
+; RV32IA-NEXT:    mv a2, a3
+; RV32IA-NEXT:    mv a3, a4
+; RV32IA-NEXT:    mv a4, a5
+; RV32IA-NEXT:    call __atomic_compare_exchange_8
+; RV32IA-NEXT:    lw ra, 12(sp)
+; RV32IA-NEXT:    addi sp, sp, 16
+; RV32IA-NEXT:    ret
   %res = cmpxchg i64* %ptr, i64 %cmp, i64 %val seq_cst seq_cst
   ret void
 }