Allow pointer types for atomicrmw xchg

[tkf]

Summary

This adds support for pointer types for atomic xchg and let us write instructions such as atomicrmw xchg i64** %0, i64* %1 seq_cst. This is similar to the patch for allowing atomicrmw xchg on floating point types https://reviews.llvm.org/D52416.

Motivation: In Julia, we keep GC-managed objects as pointer in LLVM IR so that our compiler pass can insert GC-related instructions to track these objects. Thus, we cannot easily use bitcast (like clang does) to use atomicrmw xchg for swapping a pointer value (https://github.com/JuliaLang/julia/issues/44932). Adding direct support for xchg of pointer in LLVM would let us use native instructions in a wider class of programs. I think it may be beneficial to other languages with GC using approaches similar to Julia.

[efriedma]

llvm/lib/AsmParser/LLParser.cpp

7464

You can change this to get the size from the datalayout, instead of using getPrimitiveSizeInBits(). (Not that I really expect anyone to declare a non-power-of-two pointer type, but I don't think we actually forbid it.)

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

Can we try to preserve the pointer type where possible? cmpxchg can take a pointer.

[efriedma]

llvm/lib/IR/Verifier.cpp

3929

Is the verifier missing a check for the power-of-two constraint that the IR parser enforces?

[tkf]

Use datalayout for detecting pointer size

[tkf]

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

emitLoadLinked [1] and emitStoreConditional [2] for AArch64TargetLowering use CreateBitCast to/from integer always. So, if I understand correctly, we would need casting to integer in the TLI or somewhere. Looking at https://reviews.llvm.org/D103232 that added this for floating point types, doing this early and unconditionally seems to be intentional. Or is the situation different in pointer types compared to floating point types?

[1]: https://github.com/llvm/llvm-project/blob/eeccdd318d25f68745cdf8199f7e8c251d0cf65e/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp#L19710
[2]: https://github.com/llvm/llvm-project/blob/eeccdd318d25f68745cdf8199f7e8c251d0cf65e/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp#L19747

llvm/lib/IR/Verifier.cpp

3929

It looks like visitAtomicRMWInst calls checkAtomicMemAccessSize which checks the power-of-two size constraint.

[efriedma]

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

Unlike floats, pointers are allowed as operands to cmpxchg. So if we end up in insertRMWCmpXchgLoop, we shouldn't need to ptrtoint/inttoptr to force the type of the cmpxchg to an integer type.

Wasn't thinking about the other codepaths for lowering atomicrmw. I guess some of them require integers; maybe some could be modified in the future.

Hopefully it's not too hard to move this code specifically to the places that need it? If that's tricky for some reason, we can skip it for now, I guess.

[tkf]

Add TLI->shouldConvertAtomicRMWToIntegerType so that each target can opt-out convertAtomicXchgToIntegerType.

[tkf]

Document shouldConvertAtomicRMWToIntegerType

[tkf]

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

I added TLI->shouldConvertAtomicRMWToIntegerType(RMWI) to keep the pointer type in x86 (and resolve the TODO comment just below in the source code). Is this a valid solution?

[asb]

One thing that's not immediately clear to me is what the the appropriate handling of pointers with a non-default address space should be? I think specifically, atomicrmw of non-integral pointer types may have to be disallowed?

[efriedma]

One thing that's not immediately clear to me is what the the appropriate handling of pointers with a non-default address space should be? I think specifically, atomicrmw of non-integral pointer types may have to be disallowed?

I don't see a particular problem here? I mean, an atomic xchg is perfectly sensible for non-integral pointers; semantically, it doesn't even require examining the pointer bits. I guess it's possible some targets can't implement an atomic pointer xchg, but it's not like we're forcing frontends to use it...

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

I think we want to move the convertAtomicXchgToIntegerType() call into tryExpandAtomicRMW()/widenPartwordAtomicRMW(); at that point, we can tell whether we need to force a conversion to an integer type, without explicitly asking the target, based on the IR instructions we're going to generate.

We don't really want to make targets implement hooks to compute things we already know.

[asb]

In D124728#3507010, @efriedma wrote:

One thing that's not immediately clear to me is what the the appropriate handling of pointers with a non-default address space should be? I think specifically, atomicrmw of non-integral pointer types may have to be disallowed?

I don't see a particular problem here? I mean, an atomic xchg is perfectly sensible for non-integral pointers; semantically, it doesn't even require examining the pointer bits. I guess it's possible some targets can't implement an atomic pointer xchg, but it's not like we're forcing frontends to use it...

My concern was cases where there's no available atomic operation that can handle the pointer (e.g. atomic operations are only available on GPRs and it can't be stored in a GPR). Though as you say, frontends can just not generate it.

[tkf]

Move convertAtomicXchgToIntegerType to tryExpandAtomicRMW

[tkf]

In D124728#3506193, @asb wrote:

I think specifically, atomicrmw of non-integral pointer types may have to be disallowed?

In Julia, we use non-integral pointer types for GC-tracked objects. Using xchg for such pointers is the primary motivation for this patch. We would like to keep non-integral pointer types in our lowering stages while maintaining xchg instructions.

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

I tried to make something you suggested. It calls convertAtomicXchgToIntegerType in tryExpandAtomicRMW. It avoids convertAtomicXchgToIntegerType on AtomicExpansionKind::CmpXChg if the value operand is pointer.

I initially tried it only for LL/SC but it breaks floating point AArch64 tests on LegalizeDAG. I wonder if that's why https://reviews.llvm.org/D103232 introduced the conversion that is triggered in many cases?

move the convertAtomicXchgToIntegerType() call into tryExpandAtomicRMW()/widenPartwordAtomicRMW()

widenPartwordAtomicRMW is called only when the op is or, and, or xor which are not supported on pointers and floats. So, I don't think we need to call it before widenPartwordAtomicRMW?

[efriedma]

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

but it breaks floating point AArch64 tests

I assume what happens here is that we don't expand the operation, the SelectionDAG lowers it to a floating-point atomic operation, then it explodes because we don't have legalization support for that? I guess the current patch doesn't make that situation any worse.

widenPartwordAtomicRMW is called only when the op is or, and, or xor

Wasn't really thinking about that part; sure, I guess we don't need to worry about that.

548

Do you also want to check for AtomicExpansionKind::None here?

[tkf]

llvm/lib/CodeGen/AtomicExpandPass.cpp

548

I saw the failure from LegalizeDAG in an AArch64 test that does atomicrmw xchg on floating point. It occurs with AtomicExpansionKind::None. This is the failure in LegalizeDAG I mentioned above. Since presumably the legalization does not support xchg of floats, we need to convert it on xchg of ints here.

I think making this part simple requires adding supports to the legalization. But I'm not familiar with that part of LLVM.

[efriedma]

llvm/lib/CodeGen/AtomicExpandPass.cpp

548

No, I meant if we have pointer, and the expansion kind is None. I wasn't suggesting you change what we do for floating-point ops.

[tkf]

Don't convert pointers to integers for AtomicExpansionKind::None

[tkf]

llvm/lib/CodeGen/AtomicExpandPass.cpp

548

Thanks for the clarification. I exclude the AtomicExpansionKind::None case when the value operand type is a pointer.

[efriedma]

llvm/lib/CodeGen/AtomicExpandPass.cpp

523

Is this change necessary?

[jyknight]

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

Sorry to chime in so late -- but I think the _original_ version you had, doing the coercion to integer here, unconditionally, was better.

While we support cmpxchg of pointer args emitted by users, we coerce all cmpxchg to integer types right below. And that's required -- none of the non-trivial lowerings of cmpxchg do not support non-integer values.

So, I really don't think there's value to trying to preserve non-integer types for cmpxchg as emitted by a atomicrmw xchg lowering: coerce to integer unconditionally, and be done with it.

[tkf]

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

I'm happy to roll back to the original patch.

@efriedma What do you think?

[efriedma]

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

The idea here was is to give more flexibility to backends that use non-integral pointer types... but I missed that the code below forces cmpxchg to use an integral type. Sure, we can revert it, and let whoever needs that look into it as a followup.

[tkf]

Rollback to the version with unconditional casting to integer, as discussed in https://reviews.llvm.org/D124728#inline-1208265

Remove unneeded check of pointer type, as pointed out by https://reviews.llvm.org/D124728#inline-1208221

[tkf]

llvm/lib/CodeGen/AtomicExpandPass.cpp

288

I reverted the patch to the initial version plus some other improvements that are applicable to the initial version.

[jyknight]

Thanks!

[tkf]

Can this patch land? Is there anything else that needs to be done?

[jyknight]

In D124728#3534101, @tkf wrote:

Can this patch land? Is there anything else that needs to be done?

Yes. If you don't have permission to push it, I can do so for you. Just let me know, thanks!

[tkf]

Yes, please merge this. I don't have permission to push it.

[tkf]

Resolve conflicts with https://reviews.llvm.org/D125652

llvm/docs/LangRef.rst

 -  umax
 -  umin
 -  fadd
 -  fsub
 
 For most of these operations, the type of '<value>' must be an integer
 type whose bit width is a power of two greater than or equal to eight
 and less than or equal to a target-specific size limit. For xchg, this
-may also be a floating point type with the same size constraints as
-integers.  For fadd/fsub, this must be a floating point type.  The
+may also be a floating point or a pointer type with the same size constraints
+as integers.  For fadd/fsub, this must be a floating point type.  The
 type of the '``<pointer>``' operand must be a pointer to that type. If
 the ``atomicrmw`` is marked as ``volatile``, then the optimizer is not
 allowed to modify the number or order of execution of this
 ``atomicrmw`` with other :ref:`volatile operations <volatile>`.
 
 The instruction can take an optional ``align`` attribute.
 The alignment must be a power of two greater or equal to the size of the
 `<value>` type. If unspecified, the alignment is assumed to be equal to the

llvm/lib/AsmParser/LLParser.cpp

int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
   if (!Ptr->getType()->isPointerTy())
     return error(PtrLoc, "atomicrmw operand must be a pointer");
   if (!cast<PointerType>(Ptr->getType())
            ->isOpaqueOrPointeeTypeMatches(Val->getType()))
     return error(ValLoc, "atomicrmw value and pointer type do not match");
 
   if (Operation == AtomicRMWInst::Xchg) {
     if (!Val->getType()->isIntegerTy() &&
-        !Val->getType()->isFloatingPointTy()) {
-      return error(ValLoc,
-                   "atomicrmw " + AtomicRMWInst::getOperationName(Operation) +
-                       " operand must be an integer or floating point type");
+        !Val->getType()->isFloatingPointTy() &&
+        !Val->getType()->isPointerTy()) {
+      return error(
+          ValLoc,
+              " operand must be an integer, floating point, or pointer type");
     }
   } else if (IsFP) {
     if (!Val->getType()->isFloatingPointTy()) {
       return error(ValLoc, "atomicrmw " +
                                AtomicRMWInst::getOperationName(Operation) +
                                " operand must be a floating point type");
     }
   } else {
     if (!Val->getType()->isIntegerTy()) {
       return error(ValLoc, "atomicrmw " +
                                AtomicRMWInst::getOperationName(Operation) +
                                " operand must be an integer");
     }
   }
 
-  unsigned Size = Val->getType()->getPrimitiveSizeInBits();
+  unsigned Size =
+      PFS.getFunction().getParent()->getDataLayout().getTypeStoreSizeInBits(
+          Val->getType());
   if (Size < 8 || (Size & (Size - 1)))
     return error(ValLoc, "atomicrmw operand must be power-of-two byte-sized"
                          " integer");
   const Align DefaultAlignment(
       PFS.getFunction().getParent()->getDataLayout().getTypeStoreSize(
           Val->getType()));
   AtomicRMWInst *RMWI =
       new AtomicRMWInst(Operation, Ptr, Val,
                         Alignment.getValueOr(DefaultAlignment), Ordering, SSID);
   RMWI->setVolatile(isVolatile);
   Inst = RMWI;
   return AteExtraComma ? InstExtraComma : InstNormal;

llvm/lib/CodeGen/AtomicExpandPass.cpp

if (LI) {
       // - into a load if it is idempotent
       // - into a Cmpxchg/LL-SC loop otherwise
       // we try them in that order.
 
       if (isIdempotentRMW(RMWI) && simplifyIdempotentRMW(RMWI)) {
         MadeChange = true;
       } else {
         AtomicRMWInst::BinOp Op = RMWI->getOperation();
-        if (Op == AtomicRMWInst::Xchg &&
-            RMWI->getValOperand()->getType()->isFloatingPointTy()) {
-          // TODO: add a TLI hook to control this so that each target can
-          // convert to lowering the original type one at a time.
-          RMWI = convertAtomicXchgToIntegerType(RMWI);
-          assert(RMWI->getValOperand()->getType()->isIntegerTy() &&
-                 "invariant broken");
-          MadeChange = true;
-        }
         unsigned MinCASSize = TLI->getMinCmpXchgSizeInBits() / 8;
         unsigned ValueSize = getAtomicOpSize(RMWI);
         if (ValueSize < MinCASSize &&
             (Op == AtomicRMWInst::Or || Op == AtomicRMWInst::Xor ||
              Op == AtomicRMWInst::And)) {
           RMWI = widenPartwordAtomicRMW(RMWI);
           MadeChange = true;
         }
 
Type *NewTy =
       getCorrespondingIntegerType(RMWI->getType(), M->getDataLayout());
 
   IRBuilder<> Builder(RMWI);
 
   Value *Addr = RMWI->getPointerOperand();
   Value *Val = RMWI->getValOperand();
   Type *PT = PointerType::get(NewTy, RMWI->getPointerAddressSpace());
   Value *NewAddr = Builder.CreateBitCast(Addr, PT);
-  Value *NewVal = Builder.CreateBitCast(Val, NewTy);
+  Value *NewVal = Val->getType()->isPointerTy()
+                      ? Builder.CreatePtrToInt(Val, NewTy)
+                      : Builder.CreateBitCast(Val, NewTy);
 
   auto *NewRMWI =
       Builder.CreateAtomicRMW(AtomicRMWInst::Xchg, NewAddr, NewVal,
                               RMWI->getAlign(), RMWI->getOrdering());
   NewRMWI->setVolatile(RMWI->isVolatile());
   LLVM_DEBUG(dbgs() << "Replaced " << *RMWI << " with " << *NewRMWI << "\n");
 
-  Value *NewRVal = Builder.CreateBitCast(NewRMWI, RMWI->getType());
+  Value *NewRVal = RMWI->getType()->isPointerTy()
+                       ? Builder.CreateIntToPtr(NewRMWI, RMWI->getType())
+                       : Builder.CreateBitCast(NewRMWI, RMWI->getType());
   RMWI->replaceAllUsesWith(NewRVal);
   RMWI->eraseFromParent();
   return NewRMWI;
 }
 
 bool AtomicExpand::tryExpandAtomicLoad(LoadInst *LI) {
   switch (TLI->shouldExpandAtomicLoadInIR(LI)) {
   case TargetLoweringBase::AtomicExpansionKind::None:

 
 static void createCmpXchgInstFun(IRBuilder<> &Builder, Value *Addr,
                                  Value *Loaded, Value *NewVal, Align AddrAlign,
                                  AtomicOrdering MemOpOrder, SyncScope::ID SSID,
                                  Value *&Success, Value *&NewLoaded) {
   Type *OrigTy = NewVal->getType();
 
   // This code can go away when cmpxchg supports FP types.
-  bool NeedBitcast = OrigTy->isFloatingPointTy();
+  bool NeedBitcast = OrigTy->isFloatingPointTy() || OrigTy->isPointerTy();
   if (NeedBitcast) {
     IntegerType *IntTy = Builder.getIntNTy(OrigTy->getPrimitiveSizeInBits());
     unsigned AS = Addr->getType()->getPointerAddressSpace();
     Addr = Builder.CreateBitCast(Addr, IntTy->getPointerTo(AS));
     NewVal = Builder.CreateBitCast(NewVal, IntTy);
     Loaded = Builder.CreateBitCast(Loaded, IntTy);
   }
 
   Value *Pair = Builder.CreateAtomicCmpXchg(
       Addr, Loaded, NewVal, AddrAlign, MemOpOrder,
       AtomicCmpXchgInst::getStrongestFailureOrdering(MemOpOrder), SSID);
   Success = Builder.CreateExtractValue(Pair, 1, "success");
   NewLoaded = Builder.CreateExtractValue(Pair, 0, "newloaded");
 
   if (NeedBitcast)
     NewLoaded = Builder.CreateBitCast(NewLoaded, OrigTy);
 }
 
 bool AtomicExpand::tryExpandAtomicRMW(AtomicRMWInst *AI) {
+  bool MadeChange = false;
   LLVMContext &Ctx = AI->getModule()->getContext();
   TargetLowering::AtomicExpansionKind Kind = TLI->shouldExpandAtomicRMWInIR(AI);
+  if (AI->getOperation() == AtomicRMWInst::Xchg &&
+      (AI->getValOperand()->getType()->isFloatingPointTy() ||
+       (Kind != TargetLoweringBase::AtomicExpansionKind::CmpXChg &&
+        AI->getValOperand()->getType()->isPointerTy()))) {
+    AI = convertAtomicXchgToIntegerType(AI);
+    assert(AI->getValOperand()->getType()->isIntegerTy() && "invariant broken");
+    MadeChange = true;
+  }
   switch (Kind) {
   case TargetLoweringBase::AtomicExpansionKind::None:
-    return false;
+    return MadeChange;
   case TargetLoweringBase::AtomicExpansionKind::LLSC: {
     unsigned MinCASSize = TLI->getMinCmpXchgSizeInBits() / 8;
     unsigned ValueSize = getAtomicOpSize(AI);
     if (ValueSize < MinCASSize) {
       expandPartwordAtomicRMW(AI,
                               TargetLoweringBase::AtomicExpansionKind::LLSC);
     } else {
       auto PerformOp = [&](IRBuilder<> &Builder, Value *Loaded) {
         return buildAtomicRMWValue(AI->getOperation(), Builder, Loaded,
                                    AI->getValOperand());
       };
       expandAtomicOpToLLSC(AI, AI->getType(), AI->getPointerOperand(),
                            AI->getAlign(), AI->getOrdering(), PerformOp);
     }
     return true;
   }
   case TargetLoweringBase::AtomicExpansionKind::CmpXChg: {
     unsigned MinCASSize = TLI->getMinCmpXchgSizeInBits() / 8;
     unsigned ValueSize = getAtomicOpSize(AI);
     if (ValueSize < MinCASSize) {
       // TODO: Handle atomicrmw fadd/fsub
       if (AI->getType()->isFloatingPointTy())
-        return false;
+        return MadeChange;
 
       expandPartwordAtomicRMW(AI,
                               TargetLoweringBase::AtomicExpansionKind::CmpXChg);
     } else {
       SmallVector<StringRef> SSNs;
       Ctx.getSyncScopeNames(SSNs);
       auto MemScope = SSNs[AI->getSyncScopeID()].empty()
                           ? "system"

llvm/lib/IR/Verifier.cpp

 }
 
 void Verifier::visitAtomicRMWInst(AtomicRMWInst &RMWI) {
   Check(RMWI.getOrdering() != AtomicOrdering::Unordered,
         "atomicrmw instructions cannot be unordered.", &RMWI);
   auto Op = RMWI.getOperation();
   Type *ElTy = RMWI.getOperand(1)->getType();
   if (Op == AtomicRMWInst::Xchg) {
-    Check(ElTy->isIntegerTy() || ElTy->isFloatingPointTy(),
+    Check(ElTy->isIntegerTy() || ElTy->isFloatingPointTy() ||
+              ElTy->isPointerTy(),
           "atomicrmw " + AtomicRMWInst::getOperationName(Op) +
               " operand must have integer or floating point type!",
           &RMWI, ElTy);
   } else if (AtomicRMWInst::isFPOperation(Op)) {
     Check(ElTy->isFloatingPointTy(),
           "atomicrmw " + AtomicRMWInst::getOperationName(Op) +
               " operand must have floating point type!",
           &RMWI, ElTy);

llvm/test/Assembler/invalid-atomicrmw-xchg-must-be-integer-fp-or-pointer-type.ll

+; RUN: not llvm-as -disable-output %s 2>&1 | FileCheck %s
+
+; CHECK: error: atomicrmw xchg operand must be an integer, floating point, or pointer type
+define void @f(<3 x i1>* %ptr) {
+  atomicrmw xchg <3 x i1>* %ptr, <3 x i1> <i1 1, i1 2, i1 3> seq_cst
+  ret void
+}

llvm/test/Assembler/invalid-atomicrmw-xchg-must-be-integer-or-fp-type.ll

-; RUN: not llvm-as -disable-output %s 2>&1 | FileCheck %s
-
-; CHECK: error: atomicrmw xchg operand must be an integer or floating point type
-define void @f(i32** %ptr) {
-  atomicrmw xchg i32** %ptr, i32* null seq_cst
-  ret void
-}

llvm/test/Bitcode/compatibility.ll

; CHECK: %atomicrmw.fadd = atomicrmw fadd float* %word, float 1.000000e+00 monotonic
   %atomicrmw.fadd = atomicrmw fadd float* %word, float 1.0 monotonic
 
 ; CHECK: %atomicrmw.fsub = atomicrmw fsub float* %word, float 1.000000e+00 monotonic
   %atomicrmw.fsub = atomicrmw fsub float* %word, float 1.0 monotonic
 
   ret void
 }
 
+define void @pointer_atomics(i8** %word) {
+; CHECK: %atomicrmw.xchg = atomicrmw xchg i8** %word, i8* null monotonic
+  %atomicrmw.xchg = atomicrmw xchg i8** %word, i8* null monotonic
+  ret void
+}
+
 ;; Fast Math Flags
 define void @fastmathflags_unop(float %op1) {
   %f.nnan = fneg nnan float %op1
   ; CHECK: %f.nnan = fneg nnan float %op1
   %f.ninf = fneg ninf float %op1
   ; CHECK: %f.ninf = fneg ninf float %op1
   %f.nsz = fneg nsz float %op1
   ; CHECK: %f.nsz = fneg nsz float %op1

llvm/test/CodeGen/AArch64/atomicrmw-xchg-pointer.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --force-update
+; RUN: llc -verify-machineinstrs -mtriple=aarch64-- -O1 -fast-isel=0 -global-isel=false %s -o - | FileCheck %s -check-prefix=NOLSE
+; RUN: llc -verify-machineinstrs -mtriple=aarch64-- -mattr=+lse -O1 -fast-isel=0 -global-isel=false %s -o - | FileCheck %s -check-prefix=LSE
+
+define i8* @test_rmw_xchg_pointer(i8** %dst, i8* %new) {
+; NOLSE-LABEL: test_rmw_xchg_pointer:
+; NOLSE:       // %bb.0:
+; NOLSE-NEXT:    mov x8, x0
+; NOLSE-NEXT:  .LBB0_1: // %atomicrmw.start
+; NOLSE-NEXT:    // =>This Inner Loop Header: Depth=1
+; NOLSE-NEXT:    ldaxr x0, [x8]
+; NOLSE-NEXT:    stlxr w9, x1, [x8]
+; NOLSE-NEXT:    cbnz w9, .LBB0_1
+; NOLSE-NEXT:  // %bb.2: // %atomicrmw.end
+; NOLSE-NEXT:    ret
+;
+; LSE-LABEL: test_rmw_xchg_pointer:
+; LSE:       // %bb.0:
+; LSE-NEXT:    swpal x1, x0, [x0]
+; LSE-NEXT:    ret
+  %res = atomicrmw xchg i8** %dst, i8* %new seq_cst
+  ret i8* %res
+}

llvm/test/CodeGen/AMDGPU/flat_atomics_i64.ll

 ; GCN: flat_atomic_swap_x2 v[{{[0-9]+:[0-9]+}}], v{{\[[0-9]+:[0-9]+\]$}}
 define amdgpu_kernel void @atomic_xchg_f64_offset(double* %out, double %in) {
 entry:
   %gep = getelementptr double, double* %out, i64 4
   %tmp0 = atomicrmw volatile xchg double* %gep, double %in seq_cst
   ret void
 }
 
+; GCN-LABEL: {{^}}atomic_xchg_pointer_offset:
+; GCN: flat_atomic_swap_x2 v[{{[0-9]+:[0-9]+}}], v{{\[[0-9]+:[0-9]+\]$}}
+define amdgpu_kernel void @atomic_xchg_pointer_offset(i8** %out, i8* %in) {
+entry:
+  %gep = getelementptr i8*, i8** %out, i32 4
+  %val = atomicrmw volatile xchg i8** %gep, i8* %in seq_cst
+  ret void
+}
+
 ; GCN-LABEL: {{^}}atomic_xchg_i64_ret_offset:
 ; GCN: flat_atomic_swap_x2 [[RET:v\[[0-9]+:[0-9]+\]]], v[{{[0-9]+:[0-9]+}}], v{{\[[0-9]+:[0-9]+\]}} glc{{$}}
 ; GCN: flat_store_dwordx2 v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
 define amdgpu_kernel void @atomic_xchg_i64_ret_offset(i64* %out, i64* %out2, i64 %in) {
 entry:
   %gep = getelementptr i64, i64* %out, i64 4
   %tmp0 = atomicrmw volatile xchg i64* %gep, i64 %in seq_cst
   store i64 %tmp0, i64* %out2

llvm/test/CodeGen/AMDGPU/global_atomics_i64.ll

 ; GFX9: global_atomic_swap_x2 v{{[0-9]+}}, v[{{[0-9]+:[0-9]+}}], s{{\[[0-9]+:[0-9]+\]}} offset:32{{$}}
 define amdgpu_kernel void @atomic_xchg_f64_offset(double addrspace(1)* %out, double %in) {
 entry:
   %gep = getelementptr double, double addrspace(1)* %out, i64 4
   %tmp0 = atomicrmw volatile xchg double addrspace(1)* %gep, double %in seq_cst
   ret void
 }
 
+; GCN-LABEL: {{^}}atomic_xchg_pointer_offset:
+; CIVI: buffer_atomic_swap_x2 v{{\[[0-9]+:[0-9]+\]}}, off, s[{{[0-9]+}}:{{[0-9]+}}], 0 offset:32{{$}}
+
+; GFX9: global_atomic_swap_x2 v{{[0-9]+}}, v[{{[0-9]+:[0-9]+}}], s{{\[[0-9]+:[0-9]+\]}} offset:32{{$}}
+define amdgpu_kernel void @atomic_xchg_pointer_offset(i8* addrspace(1)* %out, i8* %in) {
+entry:
+  %gep = getelementptr i8*, i8* addrspace(1)* %out, i64 4
+  %tmp0 = atomicrmw volatile xchg i8* addrspace(1)* %gep, i8* %in seq_cst
+  ret void
+}
+
 ; GCN-LABEL: {{^}}atomic_xchg_i64_ret_offset:
 ; CIVI: buffer_atomic_swap_x2 [[RET:v\[[0-9]+:[0-9]+\]]], off, s[{{[0-9]+}}:{{[0-9]+}}], 0 offset:32 glc{{$}}
 ; CIVI: buffer_store_dwordx2 [[RET]]
 
 ; GFX9: global_atomic_swap_x2 [[RET:v\[[0-9]+:[0-9]+\]]], v{{[0-9]+}}, v{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}} offset:32 glc{{$}}
 define amdgpu_kernel void @atomic_xchg_i64_ret_offset(i64 addrspace(1)* %out, i64 addrspace(1)* %out2, i64 %in) {
 entry:
   %gep = getelementptr i64, i64 addrspace(1)* %out, i64 4

llvm/test/CodeGen/AMDGPU/local-atomics64.ll

 ; GCN: s_endpgm
 define amdgpu_kernel void @lds_atomic_xchg_ret_f64_offset(double addrspace(1)* %out, double addrspace(3)* %ptr) nounwind {
   %gep = getelementptr double, double addrspace(3)* %ptr, i32 4
   %result = atomicrmw xchg double addrspace(3)* %gep, double 4.0 seq_cst
   store double %result, double addrspace(1)* %out, align 8
   ret void
 }
 
+; GCN-LABEL: {{^}}lds_atomic_xchg_ret_pointer_offset:
+; SICIVI: s_mov_b32 m0
+; GFX9-NOT: m0
+
+; GCN: ds_wrxchg_rtn_b64 {{.*}} offset:32
+; GCN: s_endpgm
+define amdgpu_kernel void @lds_atomic_xchg_ret_pointer_offset(i8* addrspace(1)* %out, i8* addrspace(3)* %ptr) nounwind {
+  %gep = getelementptr i8*, i8* addrspace(3)* %ptr, i32 4
+  %result = atomicrmw xchg i8* addrspace(3)* %gep, i8* null seq_cst
+  store i8* %result, i8* addrspace(1)* %out, align 8
+  ret void
+}
+
 ; GCN-LABEL: {{^}}lds_atomic_add_ret_i64:
 ; SICIVI: s_mov_b32 m0
 ; GFX9-NOT: m0
 
 ; GCN: ds_add_rtn_u64
 ; GCN: s_endpgm
 define amdgpu_kernel void @lds_atomic_add_ret_i64(i64 addrspace(1)* %out, i64 addrspace(3)* %ptr) nounwind {
   %result = atomicrmw add i64 addrspace(3)* %ptr, i64 4 seq_cst

llvm/test/CodeGen/X86/atomic64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -O0 -mtriple=x86_64-- -mcpu=corei7 -verify-machineinstrs | FileCheck %s --check-prefix X64
 ; RUN: llc < %s -O0 -mtriple=i386-- -mcpu=i486 -verify-machineinstrs | FileCheck %s --check-prefix I486
 
 @sc64 = external dso_local global i64
 @fsc64 = external dso_local global double
+@psc64 = external dso_local global i8*
 
 define void @atomic_fetch_add64() nounwind {
 ; X64-LABEL: atomic_fetch_add64:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    lock incq sc64(%rip)
 ; X64-NEXT:    lock addq $3, sc64(%rip)
 ; X64-NEXT:    movl $5, %eax
 ; X64-NEXT:    lock xaddq %rax, sc64(%rip)

 ; I486-NEXT:    movl $fsc64, (%eax)
 ; I486-NEXT:    calll __atomic_exchange_8@PLT
 ; I486-NEXT:    movl %ebp, %esp
 ; I486-NEXT:    popl %ebp
 ; I486-NEXT:    retl
   %t1 = atomicrmw xchg double* @fsc64, double %x acquire
   ret void
 }
+
+define void @atomic_fetch_swapptr(i8* %x) nounwind {
+; X64-LABEL: atomic_fetch_swapptr:
+; X64:       # %bb.0:
+; X64-NEXT:    xchgq %rdi, psc64(%rip)
+; X64-NEXT:    retq
+;
+; I486-LABEL: atomic_fetch_swapptr:
+; I486:       # %bb.0:
+; I486-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; I486-NEXT:    xchgl %eax, psc64
+; I486-NEXT:    retl
+  %t1 = atomicrmw xchg i8** @psc64, i8* %x acquire
+  ret void
+}

llvm/test/Transforms/AtomicExpand/AArch64/expand-atomicrmw-xchg-pointer.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -O1 -S -mtriple=aarch64-- -atomic-expand %s | FileCheck %s
+; RUN: opt -O1 -S -mtriple=aarch64-- -mattr=+outline-atomics -atomic-expand %s | FileCheck %s --check-prefix=OUTLINE-ATOMICS
+
+define void @atomic_swap_pointer(i8** %ptr, i8* %val) nounwind {
+; CHECK-LABEL: @atomic_swap_pointer(
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i8** [[PTR:%.*]] to i64*
+; CHECK-NEXT:    [[TMP2:%.*]] = ptrtoint i8* [[VAL:%.*]] to i64
+; CHECK-NEXT:    br label [[ATOMICRMW_START:%.*]]
+; CHECK:       atomicrmw.start:
+; CHECK-NEXT:    [[TMP3:%.*]] = call i64 @llvm.aarch64.ldaxr.p0i64(i64* elementtype(i64) [[TMP1]])
+; CHECK-NEXT:    [[TMP4:%.*]] = call i32 @llvm.aarch64.stxr.p0i64(i64 [[TMP2]], i64* elementtype(i64) [[TMP1]])
+; CHECK-NEXT:    [[TRYAGAIN:%.*]] = icmp ne i32 [[TMP4]], 0
+; CHECK-NEXT:    br i1 [[TRYAGAIN]], label [[ATOMICRMW_START]], label [[ATOMICRMW_END:%.*]]
+; CHECK:       atomicrmw.end:
+; CHECK-NEXT:    [[TMP5:%.*]] = inttoptr i64 [[TMP3]] to i8*
+; CHECK-NEXT:    ret void
+;
+; OUTLINE-ATOMICS-LABEL: @atomic_swap_pointer(
+; OUTLINE-ATOMICS-NEXT:    [[TMP1:%.*]] = bitcast i8** [[PTR:%.*]] to i64*
+; OUTLINE-ATOMICS-NEXT:    [[TMP2:%.*]] = ptrtoint i8* [[VAL:%.*]] to i64
+; OUTLINE-ATOMICS-NEXT:    [[TMP3:%.*]] = atomicrmw xchg i64* [[TMP1]], i64 [[TMP2]] acquire, align 8
+; OUTLINE-ATOMICS-NEXT:    [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to i8*
+; OUTLINE-ATOMICS-NEXT:    ret void
+;
+  %t1 = atomicrmw xchg i8** %ptr, i8* %val acquire
+  ret void
+}

llvm/test/Transforms/AtomicExpand/X86/expand-atomic-xchg-pointer.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -mtriple=i686-linux-gnu -atomic-expand %s | FileCheck %s
+
+define i8* @atomic_xchg_pointer(i8** %ptr) nounwind {
+; CHECK-LABEL: @atomic_xchg_pointer(
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i8** [[PTR:%.*]] to i64*
+; CHECK-NEXT:    [[TMP2:%.*]] = atomicrmw xchg i64* [[TMP1]], i64 0 seq_cst, align 8
+; CHECK-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to i8*
+; CHECK-NEXT:    ret i8* [[TMP3]]
+;
+  %result = atomicrmw xchg i8** %ptr, i8* null seq_cst
+  ret i8* %result
+}
+
+define i8* @atomic_xchg_pointer_as1(i8* addrspace(1)* %ptr) nounwind {
+; CHECK-LABEL: @atomic_xchg_pointer_as1(
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i8* addrspace(1)* [[PTR:%.*]] to i64 addrspace(1)*
+; CHECK-NEXT:    [[TMP2:%.*]] = atomicrmw xchg i64 addrspace(1)* [[TMP1]], i64 0 seq_cst, align 8
+; CHECK-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to i8*
+; CHECK-NEXT:    ret i8* [[TMP3]]
+;
+  %result = atomicrmw xchg i8* addrspace(1)* %ptr, i8* null seq_cst
+  ret i8* %result
+}