[NFC][LLVM] Merge shouldExpandAtomic*InIR into shouldExpandAtomicInstInIR

[tianshilei1992]

Summary

This patch merges the four virtual functions, shouldExpandAtomic{Load,Store,CmpX,RMW}InIR
into one function shouldExpandAtomicInstInIR such that we don't have to encode
the instruction information in the function name.

[jyknight]

IMO, this change hurts the readability of the code, not improves it.

[tianshilei1992]

In D126079#3528179, @jyknight wrote:

IMO, this change hurts the readability of the code, not improves it.

We already have the instruction information in the argument, and it is the only argument, is it necessary to encode it in the function name as well? One of the advantages having overloaded function is that we don't need to write a series of functions with different names whose only difference is just the argument type.

[MaskRay]

Overloaded virtual functions seem more difficult to read and are more error-prone. Having distinct function names can be useful as people searching code can find the relevant one easily.

[tra]

My original suggestion which resulted in this patch was to generalize the API to just one function accepting Instruction* and let the target deal with the details.
For many targets there's a fair amount of duplicated code across the per-instruction variant implementation of shouldExpandX and using single call would allow to consolidate the atomic expansion logic in fewer places.

Using overloads, indeed, only makes things more opaque without reducing the API surface.

[tianshilei1992]

In D126079#3528232, @tra wrote:

My original suggestion which resulted in this patch was to generalize the API to just one function accepting Instruction* and let the target deal with the details.
For many targets there's a fair amount of duplicated code across the per-instruction variant implementation of shouldExpandX and using single call would allow to consolidate the atomic expansion logic in fewer places.

Using overloads, indeed, only makes things more opaque without reducing the API surface.

I did have one patch locally that only keeps the function shouldExpandAtomicInstInIR(Instruction *) and in the body it has something like if (auto *SI = dyn_cast<StoreInst>(I)). In terms of readability, IMO it is same as overload. However, of course it can reduce number of APIs. I can use that method, but it looks like the controversy here is readability.

[jyknight]

In D126079#3528275, @tianshilei1992 wrote:

I did have one patch locally that only keeps the function shouldExpandAtomicInstInIR(Instruction *) and in the body it has something like if (auto *SI = dyn_cast<StoreInst>(I)). In terms of readability, IMO it is same as overload. However, of course it can reduce number of APIs. I can use that method, but it looks like the controversy here is readability.

Without seeing the result of that refactoring I can't say for sure, but I can imagine that could be an improvement over the status quo.

[asb]

In D126079#3528290, @jyknight wrote:

In D126079#3528275, @tianshilei1992 wrote:

I did have one patch locally that only keeps the function shouldExpandAtomicInstInIR(Instruction *) and in the body it has something like if (auto *SI = dyn_cast<StoreInst>(I)). In terms of readability, IMO it is same as overload. However, of course it can reduce number of APIs. I can use that method, but it looks like the controversy here is readability.

Without seeing the result of that refactoring I can't say for sure, but I can imagine that could be an improvement over the status quo.

FWIW, I think that would be better too.

[tianshilei1992]

use one function

[tra]

Looks like it's a wash -- public API is smaller, but the implementations didn't really benefit much. If anything, they got a bit less readable.

I'm fine with dropping this change.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

19702

For targets that do have distinct per-instruction logic it may make sense to keep it in separate functions and just add a top-level dispatch function calling the right variant.

llvm/docs/Atomics.rst

 ``atomicrmw`` can be represented using a loop with LL/SC-style instructions
 which take some sort of exclusive lock on a cache line (``LDREX`` and ``STREX``
 on ARM, etc.).
 
 It is often easiest for backends to use AtomicExpandPass to lower some of the
 atomic constructs. Here are some lowerings it can do:
 
 * cmpxchg -> loop with load-linked/store-conditional
-  by overriding ``shouldExpandAtomicCmpXchgInIR()``, ``emitLoadLinked()``,
-  ``emitStoreConditional()``
-* large loads/stores -> ll-sc/cmpxchg
-  by overriding ``shouldExpandAtomicStoreInIR()``/``shouldExpandAtomicLoadInIR()``
+  by overriding ``shouldExpandAtomicInstInIR()`` for ``AtomicCmpXchgInst``,
+  ``emitLoadLinked()``, and ``emitStoreConditional()``
+* large loads/stores -> ll-sc/cmpxchg by overriding
+  ``shouldExpandAtomicInstInIR()`` for ``LoadInst``/``StoreInst``
 * strong atomic accesses -> monotonic accesses + fences by overriding
   ``shouldInsertFencesForAtomic()``, ``emitLeadingFence()``, and
   ``emitTrailingFence()``
 * atomic rmw -> loop with cmpxchg or load-linked/store-conditional
   by overriding ``expandAtomicRMWInIR()``
 * expansion to __atomic_* libcalls for unsupported sizes.
 * part-word atomicrmw/cmpxchg -> target-specific intrinsic by overriding
-  ``shouldExpandAtomicRMWInIR``, ``emitMaskedAtomicRMWIntrinsic``,
-  ``shouldExpandAtomicCmpXchgInIR``, and ``emitMaskedAtomicCmpXchgIntrinsic``.
+  ``shouldExpandAtomicInstInIR`` for ``AtomicRMWInst``/``AtomicCmpXchgInst``,
+  ``emitMaskedAtomicRMWIntrinsic``, and ``emitMaskedAtomicCmpXchgIntrinsic``.
 
 For an example of these look at the ARM (first five lowerings) or RISC-V (last
 lowering) backend.
 
 AtomicExpandPass supports two strategies for lowering atomicrmw/cmpxchg to
 load-linked/store-conditional (LL/SC) loops. The first expands the LL/SC loop
 in IR, calling target lowering hooks to emit intrinsics for the LL and SC
 operations. However, many architectures have strict requirements for LL/SC

llvm/include/llvm/CodeGen/TargetLowering.h

virtual bool shouldSignExtendTypeInLibCall(EVT Type, bool IsSigned) const {
     return IsSigned;
   }
 
   /// Returns true if arguments should be extended in lib calls.
   virtual bool shouldExtendTypeInLibCall(EVT Type) const {
     return true;
   }
 
-  /// Returns how the given (atomic) load should be expanded by the
-  /// IR-level AtomicExpand pass.
-  virtual AtomicExpansionKind shouldExpandAtomicLoadInIR(LoadInst *LI) const {
+  /// Returns how the given (atomic) instruction should be expanded by the
+  /// IR-level AtomicExpand pass. This is a combination of previous four API
+  /// functions.
+  virtual AtomicExpansionKind shouldExpandAtomicInstInIR(Instruction *I) const {
+    if (auto *RMW = dyn_cast<AtomicRMWInst>(I))
+      return RMW->isFloatingPointOperation() ? AtomicExpansionKind::CmpXChg
+                                             : AtomicExpansionKind::None;
+
     return AtomicExpansionKind::None;
   }
 
   /// Returns how the given (atomic) load should be cast by the IR-level
   /// AtomicExpand pass.
   virtual AtomicExpansionKind shouldCastAtomicLoadInIR(LoadInst *LI) const {
     if (LI->getType()->isFloatingPointTy())
       return AtomicExpansionKind::CastToInteger;
     return AtomicExpansionKind::None;
   }
 
-  /// Returns how the given (atomic) store should be expanded by the IR-level
-  /// AtomicExpand pass into. For instance AtomicExpansionKind::Expand will try
-  /// to use an atomicrmw xchg.
-  virtual AtomicExpansionKind shouldExpandAtomicStoreInIR(StoreInst *SI) const {
-    return AtomicExpansionKind::None;
-  }
-
   /// Returns how the given (atomic) store should be cast by the IR-level
   /// AtomicExpand pass into. For instance AtomicExpansionKind::CastToInteger
   /// will try to cast the operands to integer values.
   virtual AtomicExpansionKind shouldCastAtomicStoreInIR(StoreInst *SI) const {
     if (SI->getValueOperand()->getType()->isFloatingPointTy())
       return AtomicExpansionKind::CastToInteger;
     return AtomicExpansionKind::None;
   }
 
-  /// Returns how the given atomic cmpxchg should be expanded by the IR-level
-  /// AtomicExpand pass.
-  virtual AtomicExpansionKind
-  shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const {
-    return AtomicExpansionKind::None;
-  }
-
-  /// Returns how the IR-level AtomicExpand pass should expand the given
-  /// AtomicRMW, if at all. Default is to never expand.
-  virtual AtomicExpansionKind shouldExpandAtomicRMWInIR(AtomicRMWInst *RMW) const {
-    return RMW->isFloatingPointOperation() ?
-      AtomicExpansionKind::CmpXChg : AtomicExpansionKind::None;
-  }
-
   /// Returns how the given atomic atomicrmw should be cast by the IR-level
   /// AtomicExpand pass.
   virtual AtomicExpansionKind
   shouldCastAtomicRMWIInIR(AtomicRMWInst *RMWI) const {
     if (RMWI->getOperation() == AtomicRMWInst::Xchg &&
         RMWI->getValOperand()->getType()->isFloatingPointTy())
       return AtomicExpansionKind::CastToInteger;
 
     return AtomicExpansionKind::None;
   }
 
   /// On some platforms, an AtomicRMW that never actually modifies the value
   /// (such as fetch_add of 0) can be turned into a fence followed by an
   /// atomic load. This may sound useless, but it makes it possible for the
   /// processor to keep the cacheline shared, dramatically improving
   /// performance. And such idempotent RMWs are useful for implementing some
   /// kinds of locks, see for example (justification + benchmarks):
   /// http://www.hpl.hp.com/techreports/2012/HPL-2012-68.pdf
   /// This method tries doing that transformation, returning the atomic load if
   /// it succeeds, and nullptr otherwise.
-  /// If shouldExpandAtomicLoadInIR returns true on that load, it will undergo
+  /// If shouldExpandAtomicInstInIR returns true on that load, it will undergo
   /// another round of expansion.
   virtual LoadInst *
   lowerIdempotentRMWIntoFencedLoad(AtomicRMWInst *RMWI) const {
     return nullptr;
   }
 
   /// Returns how the platform's atomic operations are extended (ZERO_EXTEND,
   /// SIGN_EXTEND, or ANY_EXTEND).

llvm/lib/CodeGen/AtomicExpandPass.cpp

if (TLI->shouldInsertFencesForAtomic(I)) {
       } else if (SI && isReleaseOrStronger(SI->getOrdering())) {
         FenceOrdering = SI->getOrdering();
         SI->setOrdering(AtomicOrdering::Monotonic);
       } else if (RMWI && (isReleaseOrStronger(RMWI->getOrdering()) ||
                           isAcquireOrStronger(RMWI->getOrdering()))) {
         FenceOrdering = RMWI->getOrdering();
         RMWI->setOrdering(AtomicOrdering::Monotonic);
       } else if (CASI &&
-                 TLI->shouldExpandAtomicCmpXchgInIR(CASI) ==
+                 TLI->shouldExpandAtomicInstInIR(CASI) ==
                      TargetLoweringBase::AtomicExpansionKind::None &&
                  (isReleaseOrStronger(CASI->getSuccessOrdering()) ||
                   isAcquireOrStronger(CASI->getSuccessOrdering()) ||
                   isAcquireOrStronger(CASI->getFailureOrdering()))) {
         // If a compare and swap is lowered to LL/SC, we can do smarter fence
         // insertion, with a stronger one on the success path than on the
         // failure path. As a result, fence insertion is directly done by
         // expandAtomicCmpXchg in that case.
AtomicExpand::convertAtomicXchgToIntegerType(AtomicRMWInst *RMWI) {
 
   Value *NewRVal = Builder.CreateBitCast(NewRMWI, RMWI->getType());
   RMWI->replaceAllUsesWith(NewRVal);
   RMWI->eraseFromParent();
   return NewRMWI;
 }
 
 bool AtomicExpand::tryExpandAtomicLoad(LoadInst *LI) {
-  switch (TLI->shouldExpandAtomicLoadInIR(LI)) {
+  switch (TLI->shouldExpandAtomicInstInIR(LI)) {
   case TargetLoweringBase::AtomicExpansionKind::None:
     return false;
   case TargetLoweringBase::AtomicExpansionKind::LLSC:
     expandAtomicOpToLLSC(
         LI, LI->getType(), LI->getPointerOperand(), LI->getAlign(),
         LI->getOrdering(),
         [](IRBuilder<> &Builder, Value *Loaded) { return Loaded; });
     return true;
   case TargetLoweringBase::AtomicExpansionKind::LLOnly:
     return expandAtomicLoadToLL(LI);
   case TargetLoweringBase::AtomicExpansionKind::CmpXChg:
     return expandAtomicLoadToCmpXchg(LI);
   case TargetLoweringBase::AtomicExpansionKind::NotAtomic:
     LI->setAtomic(AtomicOrdering::NotAtomic);
     return true;
   default:
     llvm_unreachable("Unhandled case in tryExpandAtomicLoad");
   }
 }
 
 bool AtomicExpand::tryExpandAtomicStore(StoreInst *SI) {
-  switch (TLI->shouldExpandAtomicStoreInIR(SI)) {
+  switch (TLI->shouldExpandAtomicInstInIR(SI)) {
   case TargetLoweringBase::AtomicExpansionKind::None:
     return false;
   case TargetLoweringBase::AtomicExpansionKind::Expand:
     expandAtomicStore(SI);
     return true;
   case TargetLoweringBase::AtomicExpansionKind::NotAtomic:
     SI->setAtomic(AtomicOrdering::NotAtomic);
     return true;

 }
 
 void AtomicExpand::expandAtomicStore(StoreInst *SI) {
   // This function is only called on atomic stores that are too large to be
   // atomic if implemented as a native store. So we replace them by an
   // atomic swap, that can be implemented for example as a ldrex/strex on ARM
   // or lock cmpxchg8/16b on X86, as these are atomic for larger sizes.
   // It is the responsibility of the target to only signal expansion via
-  // shouldExpandAtomicRMW in cases where this is required and possible.
+  // shouldExpandAtomicInstInIR in cases where this is required and possible.
   IRBuilder<> Builder(SI);
   AtomicRMWInst *AI = Builder.CreateAtomicRMW(
       AtomicRMWInst::Xchg, SI->getPointerOperand(), SI->getValueOperand(),
       SI->getAlign(), SI->getOrdering());
   SI->eraseFromParent();
 
   // Now we have an appropriate swap instruction, lower it as usual.
   tryExpandAtomicRMW(AI);
static void createCmpXchgInstFun(IRBuilder<> &Builder, Value *Addr,
   NewLoaded = Builder.CreateExtractValue(Pair, 0, "newloaded");
 
   if (NeedBitcast)
     NewLoaded = Builder.CreateBitCast(NewLoaded, OrigTy);
 }
 
 bool AtomicExpand::tryExpandAtomicRMW(AtomicRMWInst *AI) {
   LLVMContext &Ctx = AI->getModule()->getContext();
-  TargetLowering::AtomicExpansionKind Kind = TLI->shouldExpandAtomicRMWInIR(AI);
+  TargetLowering::AtomicExpansionKind Kind =
+      TLI->shouldExpandAtomicInstInIR(AI);
   switch (Kind) {
   case TargetLoweringBase::AtomicExpansionKind::None:
     return false;
   case TargetLoweringBase::AtomicExpansionKind::LLSC: {
     unsigned MinCASSize = TLI->getMinCmpXchgSizeInBits() / 8;
     unsigned ValueSize = getAtomicOpSize(AI);
     if (ValueSize < MinCASSize) {
       expandPartwordAtomicRMW(AI,
Value *AtomicExpand::insertRMWCmpXchgLoop(
   Builder.SetInsertPoint(ExitBB, ExitBB->begin());
   return NewLoaded;
 }
 
 bool AtomicExpand::tryExpandAtomicCmpXchg(AtomicCmpXchgInst *CI) {
   unsigned MinCASSize = TLI->getMinCmpXchgSizeInBits() / 8;
   unsigned ValueSize = getAtomicOpSize(CI);
 
-  switch (TLI->shouldExpandAtomicCmpXchgInIR(CI)) {
+  switch (TLI->shouldExpandAtomicInstInIR(CI)) {
   default:
     llvm_unreachable("Unhandled case in tryExpandAtomicCmpXchg");
   case TargetLoweringBase::AtomicExpansionKind::None:
     if (ValueSize < MinCASSize)
       return expandPartwordCmpXchg(CI);
     return false;
   case TargetLoweringBase::AtomicExpansionKind::LLSC: {
     return expandAtomicCmpXchg(CI);

llvm/lib/Target/AArch64/AArch64ISelLowering.h

Value *emitStoreConditional(IRBuilderBase &Builder, Value *Val, Value *Addr,
                               AtomicOrdering Ord) const override;
 
   void emitAtomicCmpXchgNoStoreLLBalance(IRBuilderBase &Builder) const override;
 
   bool isOpSuitableForLDPSTP(const Instruction *I) const;
   bool shouldInsertFencesForAtomic(const Instruction *I) const override;
 
   TargetLoweringBase::AtomicExpansionKind
-  shouldExpandAtomicLoadInIR(LoadInst *LI) const override;
-  TargetLoweringBase::AtomicExpansionKind
-  shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
-  TargetLoweringBase::AtomicExpansionKind
-  shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
-
-  TargetLoweringBase::AtomicExpansionKind
-  shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const override;
+  shouldExpandAtomicInstInIR(Instruction *I) const override;
 
   bool useLoadStackGuardNode() const override;
   TargetLoweringBase::LegalizeTypeAction
   getPreferredVectorAction(MVT VT) const override;
 
   /// If the target has a standard location for the stack protector cookie,
   /// returns the address of that location. Otherwise, returns nullptr.
   Value *getIRStackGuard(IRBuilderBase &IRB) const override;

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

bool AArch64TargetLowering::isOpSuitableForLDPSTP(const Instruction *I) const {
   return false;
 }
 
 bool AArch64TargetLowering::shouldInsertFencesForAtomic(
     const Instruction *I) const {
   return isOpSuitableForLDPSTP(I);
 }
 
+TargetLoweringBase::AtomicExpansionKind
+AArch64TargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *SI = dyn_cast<StoreInst>(I)) {
     // Loads and stores less than 128-bits are already atomic; ones above that
     // are doomed anyway, so defer to the default libcall and blame the OS when
     // things go wrong.
-TargetLoweringBase::AtomicExpansionKind
-AArch64TargetLowering::shouldExpandAtomicStoreInIR(StoreInst *SI) const {
     unsigned Size = SI->getValueOperand()->getType()->getPrimitiveSizeInBits();
     if (Size != 128 || isOpSuitableForLDPSTP(SI))
       return AtomicExpansionKind::None;
     return AtomicExpansionKind::Expand;
   }
 
+  if (auto *LI = dyn_cast<LoadInst>(I)) {
     // Loads and stores less than 128-bits are already atomic; ones above that
     // are doomed anyway, so defer to the default libcall and blame the OS when
     // things go wrong.
-TargetLowering::AtomicExpansionKind
-AArch64TargetLowering::shouldExpandAtomicLoadInIR(LoadInst *LI) const {
     unsigned Size = LI->getType()->getPrimitiveSizeInBits();
 
     if (Size != 128 || isOpSuitableForLDPSTP(LI))
       return AtomicExpansionKind::None;
 
     // At -O0, fast-regalloc cannot cope with the live vregs necessary to
-  // implement atomicrmw without spilling. If the target address is also on the
-  // stack and close enough to the spill slot, this can lead to a situation
-  // where the monitor always gets cleared and the atomic operation can never
-  // succeed. So at -O0 lower this operation to a CAS loop.
+    // implement atomicrmw without spilling. If the target address is also on
+    // the stack and close enough to the spill slot, this can lead to a
+    // situation where the monitor always gets cleared and the atomic operation
+    // can never succeed. So at -O0 lower this operation to a CAS loop.
     if (getTargetMachine().getOptLevel() == CodeGenOpt::None)
       return AtomicExpansionKind::CmpXChg;
 
     return AtomicExpansionKind::LLSC;
   }
 
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
     // For the real atomic operations, we have ldxr/stxr up to 128 bits,
-TargetLowering::AtomicExpansionKind
-AArch64TargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
     if (AI->isFloatingPointOperation())
       return AtomicExpansionKind::CmpXChg;
 
     unsigned Size = AI->getType()->getPrimitiveSizeInBits();
-  if (Size > 128) return AtomicExpansionKind::None;
+    if (Size > 128)
+      return AtomicExpansionKind::None;
 
     // Nand is not supported in LSE.
     // Leave 128 bits to LLSC or CmpXChg.
     if (AI->getOperation() != AtomicRMWInst::Nand && Size < 128) {
       if (Subtarget->hasLSE())
         return AtomicExpansionKind::None;
       if (Subtarget->outlineAtomics()) {
         // [U]Min/[U]Max RWM atomics are used in __sync_fetch_ libcalls so far.
         // Don't outline them unless
         // (1) high level <atomic> support approved:
         //   http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p0493r1.pdf
         // (2) low level libgcc and compiler-rt support implemented by:
         //   min/max outline atomics helpers
         if (AI->getOperation() != AtomicRMWInst::Min &&
             AI->getOperation() != AtomicRMWInst::Max &&
             AI->getOperation() != AtomicRMWInst::UMin &&
             AI->getOperation() != AtomicRMWInst::UMax) {
           return AtomicExpansionKind::None;
         }
       }
     }
 
     // At -O0, fast-regalloc cannot cope with the live vregs necessary to
-  // implement atomicrmw without spilling. If the target address is also on the
-  // stack and close enough to the spill slot, this can lead to a situation
-  // where the monitor always gets cleared and the atomic operation can never
-  // succeed. So at -O0 lower this operation to a CAS loop.
+    // implement atomicrmw without spilling. If the target address is also on
+    // the stack and close enough to the spill slot, this can lead to a
+    // situation where the monitor always gets cleared and the atomic operation
+    // can never succeed. So at -O0 lower this operation to a CAS loop.
     if (getTargetMachine().getOptLevel() == CodeGenOpt::None)
       return AtomicExpansionKind::CmpXChg;
 
     return AtomicExpansionKind::LLSC;
   }
 
-TargetLowering::AtomicExpansionKind
-AArch64TargetLowering::shouldExpandAtomicCmpXchgInIR(
-    AtomicCmpXchgInst *AI) const {
+  if (auto *AI = dyn_cast<AtomicCmpXchgInst>(I)) {
     // If subtarget has LSE, leave cmpxchg intact for codegen.
     if (Subtarget->hasLSE() || Subtarget->outlineAtomics())
       return AtomicExpansionKind::None;
     // At -O0, fast-regalloc cannot cope with the live vregs necessary to
-  // implement cmpxchg without spilling. If the address being exchanged is also
-  // on the stack and close enough to the spill slot, this can lead to a
+    // implement cmpxchg without spilling. If the address being exchanged is
+    // also on the stack and close enough to the spill slot, this can lead to a
     // situation where the monitor always gets cleared and the atomic operation
     // can never succeed. So at -O0 we need a late-expanded pseudo-inst instead.
     if (getTargetMachine().getOptLevel() == CodeGenOpt::None)
       return AtomicExpansionKind::None;
 
     // 128-bit atomic cmpxchg is weird; AtomicExpand doesn't know how to expand
     // it.
-  unsigned Size = AI->getCompareOperand()->getType()->getPrimitiveSizeInBits();
+    unsigned Size =
+        AI->getCompareOperand()->getType()->getPrimitiveSizeInBits();
     if (Size > 64)
       return AtomicExpansionKind::None;
 
     return AtomicExpansionKind::LLSC;
   }
 
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 Value *AArch64TargetLowering::emitLoadLinked(IRBuilderBase &Builder,
                                              Type *ValueTy, Value *Addr,
                                              AtomicOrdering Ord) const {
   Module *M = Builder.GetInsertBlock()->getParent()->getParent();
   bool IsAcquire = isAcquireOrStronger(Ord);
 
   // Since i128 isn't legal and intrinsics don't get type-lowered, the ldrexd
   // intrinsic must return {i64, i64} and we have to recombine them into a

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h

public:
   /// type of implicit parameter.
   uint32_t getImplicitParameterOffset(const MachineFunction &MF,
                                       const ImplicitParameter Param) const;
 
   MVT getFenceOperandTy(const DataLayout &DL) const override {
     return MVT::i32;
   }
 
-  AtomicExpansionKind shouldExpandAtomicRMWInIR(AtomicRMWInst *) const override;
+  TargetLoweringBase::AtomicExpansionKind
+  shouldExpandAtomicInstInIR(Instruction *I) const override;
 
   bool isConstantUnsignedBitfieldExtractLegal(unsigned Opc, LLT Ty1,
                                               LLT Ty2) const override;
 };
 
 namespace AMDGPUISD {
 
 enum NodeType : unsigned {

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

default:
       return false;
     }
   }
   default:
     return false;
   }
 }
 
-TargetLowering::AtomicExpansionKind
-AMDGPUTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *RMW) const {
+TargetLoweringBase::AtomicExpansionKind
+AMDGPUTargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *RMW = dyn_cast<AtomicRMWInst>(I)) {
     switch (RMW->getOperation()) {
     case AtomicRMWInst::Nand:
     case AtomicRMWInst::FAdd:
     case AtomicRMWInst::FSub:
       return AtomicExpansionKind::CmpXChg;
     default:
       return AtomicExpansionKind::None;
     }
   }
 
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 bool AMDGPUTargetLowering::isConstantUnsignedBitfieldExtractLegal(
     unsigned Opc, LLT Ty1, LLT Ty2) const {
   return (Ty1 == LLT::scalar(32) || Ty1 == LLT::scalar(64)) &&
          Ty2 == LLT::scalar(32);
 }

llvm/lib/Target/AMDGPU/SIISelLowering.h

bool isCanonicalized(Register Reg, MachineFunction &MF,
                        unsigned MaxDepth = 5) const;
   bool denormalsEnabledForType(const SelectionDAG &DAG, EVT VT) const;
   bool denormalsEnabledForType(LLT Ty, MachineFunction &MF) const;
 
   bool isKnownNeverNaNForTargetNode(SDValue Op,
                                     const SelectionDAG &DAG,
                                     bool SNaN = false,
                                     unsigned Depth = 0) const override;
-  AtomicExpansionKind shouldExpandAtomicRMWInIR(AtomicRMWInst *) const override;
-  AtomicExpansionKind shouldExpandAtomicLoadInIR(LoadInst *LI) const override;
-  AtomicExpansionKind shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
-  AtomicExpansionKind
-  shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const override;
+
+  TargetLoweringBase::AtomicExpansionKind
+  shouldExpandAtomicInstInIR(Instruction *I) const override;
 
   virtual const TargetRegisterClass *
   getRegClassFor(MVT VT, bool isDivergent) const override;
   virtual bool requiresUniformRegister(MachineFunction &MF,
                                        const Value *V) const override;
   Align getPrefLoopAlignment(MachineLoop *ML) const override;
 
   void allocateHSAUserSGPRs(CCState &CCInfo,

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

 static bool fpModeMatchesGlobalFPAtomicMode(const AtomicRMWInst *RMW) {
   const fltSemantics &Flt = RMW->getType()->getScalarType()->getFltSemantics();
   auto DenormMode = RMW->getParent()->getParent()->getDenormalMode(Flt);
   if (&Flt == &APFloat::IEEEsingle())
     return DenormMode == DenormalMode::getPreserveSign();
   return DenormMode == DenormalMode::getIEEE();
 }
 
-TargetLowering::AtomicExpansionKind
-SITargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *RMW) const {
+TargetLoweringBase::AtomicExpansionKind
+SITargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *SI = dyn_cast<StoreInst>(I))
+    return SI->getPointerAddressSpace() == AMDGPUAS::PRIVATE_ADDRESS
+               ? AtomicExpansionKind::NotAtomic
+               : AtomicExpansionKind::None;
+
+  if (auto *LI = dyn_cast<LoadInst>(I))
+    return LI->getPointerAddressSpace() == AMDGPUAS::PRIVATE_ADDRESS
+               ? AtomicExpansionKind::NotAtomic
+               : AtomicExpansionKind::None;
+
+  if (auto *RMW = dyn_cast<AtomicRMWInst>(I)) {
     unsigned AS = RMW->getPointerAddressSpace();
     if (AS == AMDGPUAS::PRIVATE_ADDRESS)
       return AtomicExpansionKind::NotAtomic;
 
     auto ReportUnsafeHWInst = [&](TargetLowering::AtomicExpansionKind Kind) {
       OptimizationRemarkEmitter ORE(RMW->getFunction());
       LLVMContext &Ctx = RMW->getFunction()->getContext();
       SmallVector<StringRef> SSNs;
       Ctx.getSyncScopeNames(SSNs);
       auto MemScope = SSNs[RMW->getSyncScopeID()].empty()
                           ? "system"
                           : SSNs[RMW->getSyncScopeID()];
       ORE.emit([&]() {
         return OptimizationRemark(DEBUG_TYPE, "Passed", RMW)
                << "Hardware instruction generated for atomic "
                << RMW->getOperationName(RMW->getOperation())
                << " operation at memory scope " << MemScope
                << " due to an unsafe request.";
       });
       return Kind;
     };
 
     switch (RMW->getOperation()) {
     case AtomicRMWInst::FAdd: {
       Type *Ty = RMW->getType();
 
       // We don't have a way to support 16-bit atomics now, so just leave them
       // as-is.
       if (Ty->isHalfTy())
         return AtomicExpansionKind::None;
 
-    if (!Ty->isFloatTy() && (!Subtarget->hasGFX90AInsts() || !Ty->isDoubleTy()))
+      if (!Ty->isFloatTy() &&
+          (!Subtarget->hasGFX90AInsts() || !Ty->isDoubleTy()))
         return AtomicExpansionKind::CmpXChg;
 
       if ((AS == AMDGPUAS::GLOBAL_ADDRESS || AS == AMDGPUAS::FLAT_ADDRESS) &&
           Subtarget->hasAtomicFaddInsts()) {
         if (Subtarget->hasGFX940Insts())
           return AtomicExpansionKind::None;
 
         // The amdgpu-unsafe-fp-atomics attribute enables generation of unsafe
         // floating point atomic instructions. May generate more efficient code,
-      // but may not respect rounding and denormal modes, and may give incorrect
-      // results for certain memory destinations.
+        // but may not respect rounding and denormal modes, and may give
+        // incorrect results for certain memory destinations.
         if (RMW->getFunction()
                 ->getFnAttribute("amdgpu-unsafe-fp-atomics")
                 .getValueAsString() != "true")
           return AtomicExpansionKind::CmpXChg;
 
         if (Subtarget->hasGFX90AInsts()) {
           if (Ty->isFloatTy() && AS == AMDGPUAS::FLAT_ADDRESS)
             return AtomicExpansionKind::CmpXChg;
 
           auto SSID = RMW->getSyncScopeID();
           if (SSID == SyncScope::System ||
               SSID == RMW->getContext().getOrInsertSyncScopeID("one-as"))
             return AtomicExpansionKind::CmpXChg;
 
           return ReportUnsafeHWInst(AtomicExpansionKind::None);
         }
 
         if (AS == AMDGPUAS::FLAT_ADDRESS)
           return AtomicExpansionKind::CmpXChg;
 
         return RMW->use_empty() ? ReportUnsafeHWInst(AtomicExpansionKind::None)
                                 : AtomicExpansionKind::CmpXChg;
       }
 
       // DS FP atomics do respect the denormal mode, but the rounding mode is
       // fixed to round-to-nearest-even.
-    // The only exception is DS_ADD_F64 which never flushes regardless of mode.
+      // The only exception is DS_ADD_F64 which never flushes regardless of
+      // mode.
       if (AS == AMDGPUAS::LOCAL_ADDRESS && Subtarget->hasLDSFPAtomicAdd()) {
         if (!Ty->isDoubleTy())
           return AtomicExpansionKind::None;
 
         if (fpModeMatchesGlobalFPAtomicMode(RMW))
           return AtomicExpansionKind::None;
 
         return RMW->getFunction()
                            ->getFnAttribute("amdgpu-unsafe-fp-atomics")
                            .getValueAsString() == "true"
                    ? ReportUnsafeHWInst(AtomicExpansionKind::None)
                    : AtomicExpansionKind::CmpXChg;
       }
 
       return AtomicExpansionKind::CmpXChg;
     }
     default:
       break;
     }
 
-  return AMDGPUTargetLowering::shouldExpandAtomicRMWInIR(RMW);
+    return AMDGPUTargetLowering::shouldExpandAtomicInstInIR(RMW);
   }
 
-TargetLowering::AtomicExpansionKind
-SITargetLowering::shouldExpandAtomicLoadInIR(LoadInst *LI) const {
-  return LI->getPointerAddressSpace() == AMDGPUAS::PRIVATE_ADDRESS
-             ? AtomicExpansionKind::NotAtomic
-             : AtomicExpansionKind::None;
-}
-
-TargetLowering::AtomicExpansionKind
-SITargetLowering::shouldExpandAtomicStoreInIR(StoreInst *SI) const {
-  return SI->getPointerAddressSpace() == AMDGPUAS::PRIVATE_ADDRESS
-             ? AtomicExpansionKind::NotAtomic
-             : AtomicExpansionKind::None;
-}
-
-TargetLowering::AtomicExpansionKind
-SITargetLowering::shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *CmpX) const {
+  if (auto *CmpX = dyn_cast<AtomicCmpXchgInst>(I))
     return CmpX->getPointerAddressSpace() == AMDGPUAS::PRIVATE_ADDRESS
                ? AtomicExpansionKind::NotAtomic
                : AtomicExpansionKind::None;
+
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
 }
 
 const TargetRegisterClass *
 SITargetLowering::getRegClassFor(MVT VT, bool isDivergent) const {
   const TargetRegisterClass *RC = TargetLoweringBase::getRegClassFor(VT, false);
   const SIRegisterInfo *TRI = Subtarget->getRegisterInfo();
   if (RC == &AMDGPU::VReg_1RegClass && !isDivergent)
     return Subtarget->getWavefrontSize() == 64 ? &AMDGPU::SReg_64RegClass

llvm/lib/Target/ARM/ARMISelLowering.h

public:
     bool lowerInterleavedLoad(LoadInst *LI,
                               ArrayRef<ShuffleVectorInst *> Shuffles,
                               ArrayRef<unsigned> Indices,
                               unsigned Factor) const override;
     bool lowerInterleavedStore(StoreInst *SI, ShuffleVectorInst *SVI,
                                unsigned Factor) const override;
 
     bool shouldInsertFencesForAtomic(const Instruction *I) const override;
+
     TargetLoweringBase::AtomicExpansionKind
-    shouldExpandAtomicLoadInIR(LoadInst *LI) const override;
-    TargetLoweringBase::AtomicExpansionKind
-    shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
-    TargetLoweringBase::AtomicExpansionKind
-    shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
-    TargetLoweringBase::AtomicExpansionKind
-    shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const override;
+    shouldExpandAtomicInstInIR(Instruction *I) const override;
 
     bool useLoadStackGuardNode() const override;
 
     void insertSSPDeclarations(Module &M) const override;
     Value *getSDagStackGuard(const Module &M) const override;
     Function *getSSPStackGuardCheck(const Module &M) const override;
 
     bool canCombineStoreAndExtract(Type *VectorTy, Value *Idx,

llvm/lib/Target/ARM/ARMISelLowering.cpp

Instruction *ARMTargetLowering::emitTrailingFence(IRBuilderBase &Builder,
   case AtomicOrdering::Acquire:
   case AtomicOrdering::AcquireRelease:
   case AtomicOrdering::SequentiallyConsistent:
     return makeDMB(Builder, ARM_MB::ISH);
   }
   llvm_unreachable("Unknown fence ordering in emitTrailingFence");
 }
 
+TargetLoweringBase::AtomicExpansionKind
+ARMTargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *SI = dyn_cast<StoreInst>(I)) {
     // Loads and stores less than 64-bits are already atomic; ones above that
     // are doomed anyway, so defer to the default libcall and blame the OS when
-// things go wrong. Cortex M doesn't have ldrexd/strexd though, so don't emit
-// anything for those.
-TargetLoweringBase::AtomicExpansionKind
-ARMTargetLowering::shouldExpandAtomicStoreInIR(StoreInst *SI) const {
+    // things go wrong. Cortex M doesn't have ldrexd/strexd though, so don't
+    // emit anything for those.
     bool has64BitAtomicStore;
     if (Subtarget->isMClass())
       has64BitAtomicStore = false;
     else if (Subtarget->isThumb())
       has64BitAtomicStore = Subtarget->hasV7Ops();
     else
       has64BitAtomicStore = Subtarget->hasV6Ops();
 
     unsigned Size = SI->getValueOperand()->getType()->getPrimitiveSizeInBits();
     return Size == 64 && has64BitAtomicStore ? AtomicExpansionKind::Expand
                                              : AtomicExpansionKind::None;
   }
 
+  if (auto *LI = dyn_cast<LoadInst>(I)) {
     // Loads and stores less than 64-bits are already atomic; ones above that
     // are doomed anyway, so defer to the default libcall and blame the OS when
-// things go wrong. Cortex M doesn't have ldrexd/strexd though, so don't emit
-// anything for those.
+    // things go wrong. Cortex M doesn't have ldrexd/strexd though, so don't
+    // emit anything for those.
     // FIXME: ldrd and strd are atomic if the CPU has LPAE (e.g. A15 has that
     // guarantee, see DDI0406C ARM architecture reference manual,
     // sections A8.8.72-74 LDRD)
-TargetLowering::AtomicExpansionKind
-ARMTargetLowering::shouldExpandAtomicLoadInIR(LoadInst *LI) const {
     bool has64BitAtomicLoad;
     if (Subtarget->isMClass())
       has64BitAtomicLoad = false;
     else if (Subtarget->isThumb())
       has64BitAtomicLoad = Subtarget->hasV7Ops();
     else
       has64BitAtomicLoad = Subtarget->hasV6Ops();
 
     unsigned Size = LI->getType()->getPrimitiveSizeInBits();
     return (Size == 64 && has64BitAtomicLoad) ? AtomicExpansionKind::LLOnly
                                               : AtomicExpansionKind::None;
   }
 
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
     // For the real atomic operations, we have ldrex/strex up to 32 bits,
     // and up to 64 bits on the non-M profiles
-TargetLowering::AtomicExpansionKind
-ARMTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
     if (AI->isFloatingPointOperation())
       return AtomicExpansionKind::CmpXChg;
 
     unsigned Size = AI->getType()->getPrimitiveSizeInBits();
     bool hasAtomicRMW;
     if (Subtarget->isMClass())
       hasAtomicRMW = Subtarget->hasV8MBaselineOps();
     else if (Subtarget->isThumb())
       hasAtomicRMW = Subtarget->hasV7Ops();
     else
       hasAtomicRMW = Subtarget->hasV6Ops();
     if (Size <= (Subtarget->isMClass() ? 32U : 64U) && hasAtomicRMW) {
       // At -O0, fast-regalloc cannot cope with the live vregs necessary to
       // implement atomicrmw without spilling. If the target address is also on
       // the stack and close enough to the spill slot, this can lead to a
-    // situation where the monitor always gets cleared and the atomic operation
-    // can never succeed. So at -O0 lower this operation to a CAS loop.
+      // situation where the monitor always gets cleared and the atomic
+      // operation can never succeed. So at -O0 lower this operation to a CAS
+      // loop.
       if (getTargetMachine().getOptLevel() == CodeGenOpt::None)
         return AtomicExpansionKind::CmpXChg;
       return AtomicExpansionKind::LLSC;
     }
     return AtomicExpansionKind::None;
   }
 
-// Similar to shouldExpandAtomicRMWInIR, ldrex/strex can be used  up to 32
-// bits, and up to 64 bits on the non-M profiles.
-TargetLowering::AtomicExpansionKind
-ARMTargetLowering::shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const {
-  // At -O0, fast-regalloc cannot cope with the live vregs necessary to
-  // implement cmpxchg without spilling. If the address being exchanged is also
-  // on the stack and close enough to the spill slot, this can lead to a
+  if (auto *AI = dyn_cast<AtomicCmpXchgInst>(I)) {
+    // ldrex/strex can be used  up to 32 bits, and up to 64 bits on the non-M
+    // profiles. At -O0, fast-regalloc cannot cope with the live vregs necessary
+    // to implement cmpxchg without spilling. If the address being exchanged is
+    // also on the stack and close enough to the spill slot, this can lead to a
     // situation where the monitor always gets cleared and the atomic operation
     // can never succeed. So at -O0 we need a late-expanded pseudo-inst instead.
     unsigned Size = AI->getOperand(1)->getType()->getPrimitiveSizeInBits();
     bool HasAtomicCmpXchg;
     if (Subtarget->isMClass())
       HasAtomicCmpXchg = Subtarget->hasV8MBaselineOps();
     else if (Subtarget->isThumb())
       HasAtomicCmpXchg = Subtarget->hasV7Ops();
     else
       HasAtomicCmpXchg = Subtarget->hasV6Ops();
     if (getTargetMachine().getOptLevel() != 0 && HasAtomicCmpXchg &&
         Size <= (Subtarget->isMClass() ? 32U : 64U))
       return AtomicExpansionKind::LLSC;
     return AtomicExpansionKind::None;
   }
 
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 bool ARMTargetLowering::shouldInsertFencesForAtomic(
     const Instruction *I) const {
   return InsertFencesForAtomic;
 }
 
 bool ARMTargetLowering::useLoadStackGuardNode() const { return true; }
 
 void ARMTargetLowering::insertSSPDeclarations(Module &M) const {

llvm/lib/Target/Hexagon/HexagonISelLowering.h

public:
   bool shouldReduceLoadWidth(SDNode *Load, ISD::LoadExtType ExtTy,
                              EVT NewVT) const override;
 
   // Handling of atomic RMW instructions.
   Value *emitLoadLinked(IRBuilderBase &Builder, Type *ValueTy, Value *Addr,
                         AtomicOrdering Ord) const override;
   Value *emitStoreConditional(IRBuilderBase &Builder, Value *Val, Value *Addr,
                               AtomicOrdering Ord) const override;
-  AtomicExpansionKind shouldExpandAtomicLoadInIR(LoadInst *LI) const override;
-  AtomicExpansionKind shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
-  AtomicExpansionKind
-  shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const override;
-
-  AtomicExpansionKind
-  shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override {
-    return AtomicExpansionKind::LLSC;
-  }
+  TargetLoweringBase::AtomicExpansionKind
+  shouldExpandAtomicInstInIR(Instruction *I) const override;
 
 private:
   void initializeHVXLowering();
   unsigned getPreferredHvxVectorAction(MVT VecTy) const;
 
   bool validateConstPtrAlignment(SDValue Ptr, Align NeedAlign, const SDLoc &dl,
                                  SelectionDAG &DAG) const;
   SDValue replaceMemWithUndef(SDValue Op, SelectionDAG &DAG) const;

llvm/lib/Target/Hexagon/HexagonISelLowering.cpp

Value *HexagonTargetLowering::emitStoreConditional(IRBuilderBase &Builder,
   Val = Builder.CreateBitCast(Val, CastTy);
 
   Value *Call = Builder.CreateCall(Fn, {Addr, Val}, "stcx");
   Value *Cmp = Builder.CreateICmpEQ(Call, Builder.getInt32(0), "");
   Value *Ext = Builder.CreateZExt(Cmp, Type::getInt32Ty(M->getContext()));
   return Ext;
 }
 
-TargetLowering::AtomicExpansionKind
-HexagonTargetLowering::shouldExpandAtomicLoadInIR(LoadInst *LI) const {
+TargetLoweringBase::AtomicExpansionKind
+HexagonTargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *SI = dyn_cast<StoreInst>(I)) {
     // Do not expand loads and stores that don't exceed 64 bits.
-  return LI->getType()->getPrimitiveSizeInBits() > 64
-             ? AtomicExpansionKind::LLOnly
+    return SI->getValueOperand()->getType()->getPrimitiveSizeInBits() > 64
+               ? AtomicExpansionKind::Expand
                : AtomicExpansionKind::None;
   }
 
-TargetLowering::AtomicExpansionKind
-HexagonTargetLowering::shouldExpandAtomicStoreInIR(StoreInst *SI) const {
+  if (auto *LI = dyn_cast<LoadInst>(I)) {
     // Do not expand loads and stores that don't exceed 64 bits.
-  return SI->getValueOperand()->getType()->getPrimitiveSizeInBits() > 64
-             ? AtomicExpansionKind::Expand
+    return LI->getType()->getPrimitiveSizeInBits() > 64
+               ? AtomicExpansionKind::LLOnly
                : AtomicExpansionKind::None;
   }
 
-TargetLowering::AtomicExpansionKind
-HexagonTargetLowering::shouldExpandAtomicCmpXchgInIR(
-    AtomicCmpXchgInst *AI) const {
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I))
     return AtomicExpansionKind::LLSC;
+
+  if (auto *AI = dyn_cast<AtomicCmpXchgInst>(I))
+    return AtomicExpansionKind::LLSC;
+
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
 }

llvm/lib/Target/NVPTX/NVPTXISelLowering.h

public:
   AtomicExpansionKind shouldCastAtomicLoadInIR(LoadInst *LI) const override {
     return AtomicExpansionKind::None;
   }
 
   AtomicExpansionKind shouldCastAtomicStoreInIR(StoreInst *SI) const override {
     return AtomicExpansionKind::None;
   }
 
-  AtomicExpansionKind
-  shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
+  TargetLoweringBase::AtomicExpansionKind
+  shouldExpandAtomicInstInIR(Instruction *I) const override;
 
 private:
   const NVPTXSubtarget &STI; // cache the subtarget here
   SDValue getParamSymbol(SelectionDAG &DAG, int idx, EVT) const;
 
   SDValue LowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerCONCAT_VECTORS(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerEXTRACT_VECTOR_ELT(SDValue Op, SelectionDAG &DAG) const;

llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp

case ISD::LOAD:
     ReplaceLoadVector(N, DAG, Results);
     return;
   case ISD::INTRINSIC_W_CHAIN:
     ReplaceINTRINSIC_W_CHAIN(N, DAG, Results);
     return;
   }
 }
 
-NVPTXTargetLowering::AtomicExpansionKind
-NVPTXTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
+TargetLoweringBase::AtomicExpansionKind
+NVPTXTargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
     Type *Ty = AI->getValOperand()->getType();
 
     if (AI->isFloatingPointOperation()) {
       if (AI->getOperation() == AtomicRMWInst::BinOp::FAdd) {
         if (Ty->isFloatTy())
           return AtomicExpansionKind::None;
         if (Ty->isDoubleTy() && STI.hasAtomAddF64())
           return AtomicExpansionKind::None;
       }
       return AtomicExpansionKind::CmpXChg;
     }
 
     assert(Ty->isIntegerTy() && "Ty should be integer at this point");
     auto ITy = cast<llvm::IntegerType>(Ty);
 
     switch (AI->getOperation()) {
     default:
       return AtomicExpansionKind::CmpXChg;
     case AtomicRMWInst::BinOp::And:
     case AtomicRMWInst::BinOp::Or:
     case AtomicRMWInst::BinOp::Xor:
     case AtomicRMWInst::BinOp::Xchg:
       switch (ITy->getBitWidth()) {
       case 8:
       case 16:
         return AtomicExpansionKind::CmpXChg;
       case 32:
         return AtomicExpansionKind::None;
       case 64:
         if (STI.hasAtomBitwise64())
           return AtomicExpansionKind::None;
         return AtomicExpansionKind::CmpXChg;
       default:
         llvm_unreachable("unsupported width encountered");
       }
     case AtomicRMWInst::BinOp::Add:
     case AtomicRMWInst::BinOp::Sub:
     case AtomicRMWInst::BinOp::Max:
     case AtomicRMWInst::BinOp::Min:
     case AtomicRMWInst::BinOp::UMax:
     case AtomicRMWInst::BinOp::UMin:
       switch (ITy->getBitWidth()) {
       case 8:
       case 16:
         return AtomicExpansionKind::CmpXChg;
       case 32:
         return AtomicExpansionKind::None;
       case 64:
         if (STI.hasAtomMinMax64())
           return AtomicExpansionKind::None;
         return AtomicExpansionKind::CmpXChg;
       default:
         llvm_unreachable("unsupported width encountered");
       }
     }
 
     return AtomicExpansionKind::CmpXChg;
   }
 
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 // Pin NVPTXTargetObjectFile's vtables to this file.
 NVPTXTargetObjectFile::~NVPTXTargetObjectFile() = default;
 
 MCSection *NVPTXTargetObjectFile::SelectSectionForGlobal(
     const GlobalObject *GO, SectionKind Kind, const TargetMachine &TM) const {
   return getDataSection();
 }

llvm/lib/Target/PowerPC/PPCISelLowering.h

public:
 
     Instruction *emitLeadingFence(IRBuilderBase &Builder, Instruction *Inst,
                                   AtomicOrdering Ord) const override;
     Instruction *emitTrailingFence(IRBuilderBase &Builder, Instruction *Inst,
                                    AtomicOrdering Ord) const override;
 
     bool shouldInlineQuadwordAtomics() const;
 
-    TargetLowering::AtomicExpansionKind
-    shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
-
-    TargetLowering::AtomicExpansionKind
-    shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const override;
+    TargetLoweringBase::AtomicExpansionKind
+    shouldExpandAtomicInstInIR(Instruction *I) const override;
 
     Value *emitMaskedAtomicRMWIntrinsic(IRBuilderBase &Builder,
                                         AtomicRMWInst *AI, Value *AlignedAddr,
                                         Value *Incr, Value *Mask,
                                         Value *ShiftAmt,
                                         AtomicOrdering Ord) const override;
     Value *emitMaskedAtomicCmpXchgIntrinsic(IRBuilderBase &Builder,
                                             AtomicCmpXchgInst *CI,

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

if (Flags.isByVal()) {
           AddPtr = DAG.getNode(ISD::ADD, dl, PtrVT, PtrOff, Const);
         }
         Chain = CallSeqStart = createMemcpyOutsideCallSeq(Arg, AddPtr,
                                                           CallSeqStart,
                                                           Flags, DAG, dl);
         ArgOffset += PtrByteSize;
         continue;
       }
-      // Copy the object to parameter save area if it can not be entirely passed 
+      // Copy the object to parameter save area if it can not be entirely passed
       // by registers.
       // FIXME: we only need to copy the parts which need to be passed in
       // parameter save area. For the parts passed by registers, we don't need
       // to copy them to the stack although we need to allocate space for them
       // in parameter save area.
       if ((NumGPRs - GPR_idx) * PtrByteSize < Size)
         Chain = CallSeqStart = createMemcpyOutsideCallSeq(Arg, PtrOff,
                                                           CallSeqStart,
static unsigned mapArgRegToOffsetAIX(unsigned Reg, const PPCFrameLowering *FL) {
 
   llvm_unreachable("Only general purpose registers expected.");
 }
 
 //   AIX ABI Stack Frame Layout:
 //
 //   Low Memory +--------------------------------------------+
 //   SP   +---> | Back chain                                 | ---+
-//        |     +--------------------------------------------+    |   
+//        |     +--------------------------------------------+    |
 //        |     | Saved Condition Register                   |    |
 //        |     +--------------------------------------------+    |
 //        |     | Saved Linkage Register                     |    |
 //        |     +--------------------------------------------+    | Linkage Area
 //        |     | Reserved for compilers                     |    |
 //        |     +--------------------------------------------+    |
 //        |     | Reserved for binders                       |    |
 //        |     +--------------------------------------------+    |
SDValue PPCTargetLowering::LowerTRUNCATEVector(SDValue Op,
   unsigned TrgNumElts = TrgVT.getVectorNumElements();
   EVT EltVT = TrgVT.getVectorElementType();
   if (!isOperationCustom(Op.getOpcode(), TrgVT) ||
       TrgVT.getSizeInBits() > 128 || !isPowerOf2_32(TrgNumElts) ||
       !isPowerOf2_32(EltVT.getSizeInBits()))
     return SDValue();
 
   SDValue N1 = Op.getOperand(0);
-  EVT SrcVT = N1.getValueType();  
+  EVT SrcVT = N1.getValueType();
   unsigned SrcSize = SrcVT.getSizeInBits();
   if (SrcSize > 256 ||
       !isPowerOf2_32(SrcVT.getVectorNumElements()) ||
       !isPowerOf2_32(SrcVT.getVectorElementType().getSizeInBits()))
     return SDValue();
   if (SrcSize == 256 && SrcVT.getVectorNumElements() < 2)
     return SDValue();
 

 bool PPCTargetLowering::shouldInlineQuadwordAtomics() const {
   // TODO: 16-byte atomic type support for AIX is in progress; we should be able
   // to inline 16-byte atomic ops on AIX too in the future.
   return Subtarget.isPPC64() &&
          (EnableQuadwordAtomics || !Subtarget.getTargetTriple().isOSAIX()) &&
          Subtarget.hasQuadwordAtomics();
 }
 
-TargetLowering::AtomicExpansionKind
-PPCTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
+TargetLoweringBase::AtomicExpansionKind
+PPCTargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
     unsigned Size = AI->getType()->getPrimitiveSizeInBits();
     if (shouldInlineQuadwordAtomics() && Size == 128)
       return AtomicExpansionKind::MaskedIntrinsic;
-  return TargetLowering::shouldExpandAtomicRMWInIR(AI);
+    return TargetLowering::shouldExpandAtomicInstInIR(AI);
   }
 
-TargetLowering::AtomicExpansionKind
-PPCTargetLowering::shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const {
+  if (auto *AI = dyn_cast<AtomicCmpXchgInst>(I)) {
     unsigned Size = AI->getNewValOperand()->getType()->getPrimitiveSizeInBits();
     if (shouldInlineQuadwordAtomics() && Size == 128)
       return AtomicExpansionKind::MaskedIntrinsic;
-  return TargetLowering::shouldExpandAtomicCmpXchgInIR(AI);
+    return TargetLowering::shouldExpandAtomicInstInIR(AI);
+  }
+
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
 }
 
 static Intrinsic::ID
 getIntrinsicForAtomicRMWBinOp128(AtomicRMWInst::BinOp BinOp) {
   switch (BinOp) {
   default:
     llvm_unreachable("Unexpected AtomicRMW BinOp");
   case AtomicRMWInst::Xchg:

llvm/lib/Target/RISCV/RISCVISelLowering.h

public:
   bool shouldConsiderGEPOffsetSplit() const override { return true; }
 
   bool decomposeMulByConstant(LLVMContext &Context, EVT VT,
                               SDValue C) const override;
 
   bool isMulAddWithConstProfitable(SDValue AddNode,
                                    SDValue ConstNode) const override;
 
-  TargetLowering::AtomicExpansionKind
-  shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
+  TargetLoweringBase::AtomicExpansionKind
+  shouldExpandAtomicInstInIR(Instruction *I) const override;
   Value *emitMaskedAtomicRMWIntrinsic(IRBuilderBase &Builder, AtomicRMWInst *AI,
                                       Value *AlignedAddr, Value *Incr,
                                       Value *Mask, Value *ShiftAmt,
                                       AtomicOrdering Ord) const override;
-  TargetLowering::AtomicExpansionKind
-  shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *CI) const override;
   Value *emitMaskedAtomicCmpXchgIntrinsic(IRBuilderBase &Builder,
                                           AtomicCmpXchgInst *CI,
                                           Value *AlignedAddr, Value *CmpVal,
                                           Value *NewVal, Value *Mask,
                                           AtomicOrdering Ord) const override;
 
   /// Returns true if the target allows unaligned memory accesses of the
   /// specified type.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

 Instruction *RISCVTargetLowering::emitTrailingFence(IRBuilderBase &Builder,
                                                     Instruction *Inst,
                                                     AtomicOrdering Ord) const {
   if (isa<LoadInst>(Inst) && isAcquireOrStronger(Ord))
     return Builder.CreateFence(AtomicOrdering::Acquire);
   return nullptr;
 }
 
-TargetLowering::AtomicExpansionKind
-RISCVTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
-  // atomicrmw {fadd,fsub} must be expanded to use compare-exchange, as floating
-  // point operations can't be used in an lr/sc sequence without breaking the
-  // forward-progress guarantee.
+TargetLoweringBase::AtomicExpansionKind
+RISCVTargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
+    // atomicrmw {fadd,fsub} must be expanded to use compare-exchange, as
+    // floating point operations can't be used in an lr/sc sequence without
+    // breaking the forward-progress guarantee.
     if (AI->isFloatingPointOperation())
       return AtomicExpansionKind::CmpXChg;
 
     unsigned Size = AI->getType()->getPrimitiveSizeInBits();
     if (Size == 8 || Size == 16)
       return AtomicExpansionKind::MaskedIntrinsic;
     return AtomicExpansionKind::None;
   }
 
+  if (auto *CI = dyn_cast<AtomicCmpXchgInst>(I)) {
+    unsigned Size =
+        CI->getCompareOperand()->getType()->getPrimitiveSizeInBits();
+    if (Size == 8 || Size == 16)
+      return AtomicExpansionKind::MaskedIntrinsic;
+    return AtomicExpansionKind::None;
+  }
+
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 static Intrinsic::ID
 getIntrinsicForMaskedAtomicRMWBinOp(unsigned XLen, AtomicRMWInst::BinOp BinOp) {
   if (XLen == 32) {
     switch (BinOp) {
     default:
       llvm_unreachable("Unexpected AtomicRMW BinOp");
     case AtomicRMWInst::Xchg:
       return Intrinsic::riscv_masked_atomicrmw_xchg_i32;
Result =
         Builder.CreateCall(LrwOpScwLoop, {AlignedAddr, Incr, Mask, Ordering});
   }
 
   if (XLen == 64)
     Result = Builder.CreateTrunc(Result, Builder.getInt32Ty());
   return Result;
 }
 
-TargetLowering::AtomicExpansionKind
-RISCVTargetLowering::shouldExpandAtomicCmpXchgInIR(
-    AtomicCmpXchgInst *CI) const {
-  unsigned Size = CI->getCompareOperand()->getType()->getPrimitiveSizeInBits();
-  if (Size == 8 || Size == 16)
-    return AtomicExpansionKind::MaskedIntrinsic;
-  return AtomicExpansionKind::None;
-}
-
 Value *RISCVTargetLowering::emitMaskedAtomicCmpXchgIntrinsic(
     IRBuilderBase &Builder, AtomicCmpXchgInst *CI, Value *AlignedAddr,
     Value *CmpVal, Value *NewVal, Value *Mask, AtomicOrdering Ord) const {
   unsigned XLen = Subtarget.getXLen();
   Value *Ordering = Builder.getIntN(XLen, static_cast<uint64_t>(Ord));
   Intrinsic::ID CmpXchgIntrID = Intrinsic::riscv_masked_cmpxchg_i32;
   if (XLen == 64) {
     CmpVal = Builder.CreateSExt(CmpVal, Builder.getInt64Ty());

llvm/lib/Target/Sparc/SparcISelLowering.h

public:
 
     bool shouldInsertFencesForAtomic(const Instruction *I) const override {
       // FIXME: We insert fences for each atomics and generate
       // sub-optimal code for PSO/TSO. (Approximately nobody uses any
       // mode but TSO, which makes this even more silly)
       return true;
     }
 
-    AtomicExpansionKind shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
+    TargetLoweringBase::AtomicExpansionKind
+    shouldExpandAtomicInstInIR(Instruction *I) const override;
 
     void ReplaceNodeResults(SDNode *N,
                             SmallVectorImpl<SDValue>& Results,
                             SelectionDAG &DAG) const override;
 
     MachineBasicBlock *expandSelectCC(MachineInstr &MI, MachineBasicBlock *BB,
                                       unsigned BROpcode) const;
   };
 } // end namespace llvm
 
 #endif    // SPARC_ISELLOWERING_H

llvm/lib/Target/Sparc/SparcISelLowering.cpp

SparcTargetLowering::LowerCall_64(TargetLowering::CallLoweringInfo &CLI,
 
   return Chain;
 }
 
 //===----------------------------------------------------------------------===//
 // TargetLowering Implementation
 //===----------------------------------------------------------------------===//
 
-TargetLowering::AtomicExpansionKind SparcTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
+TargetLoweringBase::AtomicExpansionKind
+SparcTargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
     if (AI->getOperation() == AtomicRMWInst::Xchg &&
         AI->getType()->getPrimitiveSizeInBits() == 32)
       return AtomicExpansionKind::None; // Uses xchg instruction
 
     return AtomicExpansionKind::CmpXChg;
   }
 
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 /// IntCondCCodeToICC - Convert a DAG integer condition code to a SPARC ICC
 /// condition.
 static SPCC::CondCodes IntCondCCodeToICC(ISD::CondCode CC) {
   switch (CC) {
   default: llvm_unreachable("Unknown integer condition code!");
   case ISD::SETEQ:  return SPCC::ICC_E;
   case ISD::SETNE:  return SPCC::ICC_NE;
   case ISD::SETLT:  return SPCC::ICC_L;

llvm/lib/Target/VE/VEISelLowering.h

bool shouldInsertFencesForAtomic(const Instruction *I) const override {
     // VE uses release consistency, so need fence for each atomics.
     return true;
   }
   Instruction *emitLeadingFence(IRBuilderBase &Builder, Instruction *Inst,
                                 AtomicOrdering Ord) const override;
   Instruction *emitTrailingFence(IRBuilderBase &Builder, Instruction *Inst,
                                  AtomicOrdering Ord) const override;
   TargetLoweringBase::AtomicExpansionKind
-  shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
+  shouldExpandAtomicInstInIR(Instruction *I) const override;
   ISD::NodeType getExtendForAtomicOps() const override {
     return ISD::ANY_EXTEND;
   }
 
   /// Custom Lower {
   TargetLoweringBase::LegalizeAction
   getCustomOperationAction(SDNode &) const override;
 

llvm/lib/Target/VE/VEISelLowering.cpp

case AtomicOrdering::SequentiallyConsistent:
                      0);
     }
   }
 
   // MEMBARRIER is a compiler barrier; it codegens to a no-op.
   return DAG.getNode(VEISD::MEMBARRIER, DL, MVT::Other, Op.getOperand(0));
 }
 
-TargetLowering::AtomicExpansionKind
-VETargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
+TargetLoweringBase::AtomicExpansionKind
+VETargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
     // We have TS1AM implementation for i8/i16/i32/i64, so use it.
     if (AI->getOperation() == AtomicRMWInst::Xchg) {
       return AtomicExpansionKind::None;
     }
     // FIXME: Support "ATMAM" instruction for LOAD_ADD/SUB/AND/OR.
 
     // Otherwise, expand it using compare and exchange instruction to not call
     // __sync_fetch_and_* functions.
     return AtomicExpansionKind::CmpXChg;
   }
 
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 static SDValue prepareTS1AM(SDValue Op, SelectionDAG &DAG, SDValue &Flag,
                             SDValue &Bits) {
   SDLoc DL(Op);
   AtomicSDNode *N = cast<AtomicSDNode>(Op);
   SDValue Ptr = N->getOperand(1);
   SDValue Val = N->getOperand(2);
   EVT PtrVT = Ptr.getValueType();
   bool Byte = N->getMemoryVT() == MVT::i8;

llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.h

public:
   MVT getPointerTy(const DataLayout &DL, uint32_t AS = 0) const override;
   MVT getPointerMemTy(const DataLayout &DL, uint32_t AS = 0) const override;
 
 private:
   /// Keep a pointer to the WebAssemblySubtarget around so that we can make the
   /// right decision when generating code for different targets.
   const WebAssemblySubtarget *Subtarget;
 
-  AtomicExpansionKind shouldExpandAtomicRMWInIR(AtomicRMWInst *) const override;
+  TargetLoweringBase::AtomicExpansionKind
+  shouldExpandAtomicInstInIR(Instruction *I) const override;
   bool shouldScalarizeBinop(SDValue VecOp) const override;
   FastISel *createFastISel(FunctionLoweringInfo &FuncInfo,
                            const TargetLibraryInfo *LibInfo) const override;
   MVT getScalarShiftAmountTy(const DataLayout &DL, EVT) const override;
   MachineBasicBlock *
   EmitInstrWithCustomInserter(MachineInstr &MI,
                               MachineBasicBlock *MBB) const override;
   const char *getTargetNodeName(unsigned Opcode) const override;

llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp

MVT WebAssemblyTargetLowering::getPointerMemTy(const DataLayout &DL,
                                                uint32_t AS) const {
   if (AS == WebAssembly::WasmAddressSpace::WASM_ADDRESS_SPACE_EXTERNREF)
     return MVT::externref;
   if (AS == WebAssembly::WasmAddressSpace::WASM_ADDRESS_SPACE_FUNCREF)
     return MVT::funcref;
   return TargetLowering::getPointerMemTy(DL, AS);
 }
 
-TargetLowering::AtomicExpansionKind
-WebAssemblyTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
+TargetLoweringBase::AtomicExpansionKind
+WebAssemblyTargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
     // We have wasm instructions for these
     switch (AI->getOperation()) {
     case AtomicRMWInst::Add:
     case AtomicRMWInst::Sub:
     case AtomicRMWInst::And:
     case AtomicRMWInst::Or:
     case AtomicRMWInst::Xor:
     case AtomicRMWInst::Xchg:
       return AtomicExpansionKind::None;
     default:
       break;
     }
     return AtomicExpansionKind::CmpXChg;
   }
 
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 bool WebAssemblyTargetLowering::shouldScalarizeBinop(SDValue VecOp) const {
   // Implementation copied from X86TargetLowering.
   unsigned Opc = VecOp.getOpcode();
 
   // Assume target opcodes can't be scalarized.
   // TODO - do we have any exceptions?
   if (Opc >= ISD::BUILTIN_OP_END)
     return false;

llvm/lib/Target/X86/X86ISelLowering.h

private:
     bool CanLowerReturn(CallingConv::ID CallConv, MachineFunction &MF,
                         bool isVarArg,
                         const SmallVectorImpl<ISD::OutputArg> &Outs,
                         LLVMContext &Context) const override;
 
     const MCPhysReg *getScratchRegisters(CallingConv::ID CC) const override;
 
     TargetLoweringBase::AtomicExpansionKind
-    shouldExpandAtomicLoadInIR(LoadInst *LI) const override;
-    TargetLoweringBase::AtomicExpansionKind
-    shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
-    TargetLoweringBase::AtomicExpansionKind
-    shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
+    shouldExpandAtomicInstInIR(Instruction *I) const override;
+
     TargetLoweringBase::AtomicExpansionKind
     shouldExpandLogicAtomicRMWInIR(AtomicRMWInst *AI) const;
     void emitBitTestAtomicRMWIntrinsic(AtomicRMWInst *AI) const override;
 
     LoadInst *
     lowerIdempotentRMWIntoFencedLoad(AtomicRMWInst *AI) const override;
 
     bool lowerAtomicStoreAsStoreSDNode(const StoreInst &SI) const override;

llvm/lib/Target/X86/X86ISelLowering.cpp

if (OpWidth == 64)
     return Subtarget.canUseCMPXCHG8B() && !Subtarget.is64Bit();
   if (OpWidth == 128)
     return Subtarget.canUseCMPXCHG16B();
 
   return false;
 }
 
 TargetLoweringBase::AtomicExpansionKind
-X86TargetLowering::shouldExpandAtomicStoreInIR(StoreInst *SI) const {
+X86TargetLowering::shouldExpandAtomicInstInIR(Instruction *I) const {
+  if (auto *SI = dyn_cast<StoreInst>(I)) {
     Type *MemType = SI->getValueOperand()->getType();
 
     bool NoImplicitFloatOps =
         SI->getFunction()->hasFnAttribute(Attribute::NoImplicitFloat);
     if (MemType->getPrimitiveSizeInBits() == 64 && !Subtarget.is64Bit() &&
         !Subtarget.useSoftFloat() && !NoImplicitFloatOps &&
         (Subtarget.hasSSE1() || Subtarget.hasX87()))
       return AtomicExpansionKind::None;
 
     return needsCmpXchgNb(MemType) ? AtomicExpansionKind::Expand
                                    : AtomicExpansionKind::None;
   }
 
+  if (auto *LI = dyn_cast<LoadInst>(I)) {
     // Note: this turns large loads into lock cmpxchg8b/16b.
     // TODO: In 32-bit mode, use MOVLPS when SSE1 is available?
-TargetLowering::AtomicExpansionKind
-X86TargetLowering::shouldExpandAtomicLoadInIR(LoadInst *LI) const {
     Type *MemType = LI->getType();
 
     // If this a 64 bit atomic load on a 32-bit target and SSE2 is enabled, we
     // can use movq to do the load. If we have X87 we can load into an 80-bit
     // X87 register and store it to a stack temporary.
     bool NoImplicitFloatOps =
         LI->getFunction()->hasFnAttribute(Attribute::NoImplicitFloat);
     if (MemType->getPrimitiveSizeInBits() == 64 && !Subtarget.is64Bit() &&
         !Subtarget.useSoftFloat() && !NoImplicitFloatOps &&
         (Subtarget.hasSSE1() || Subtarget.hasX87()))
       return AtomicExpansionKind::None;
 
     return needsCmpXchgNb(MemType) ? AtomicExpansionKind::CmpXChg
                                    : AtomicExpansionKind::None;
   }
 
+  if (auto *AI = dyn_cast<AtomicRMWInst>(I)) {
+    unsigned NativeWidth = Subtarget.is64Bit() ? 64 : 32;
+    Type *MemType = AI->getType();
+
+    // If the operand is too big, we must see if cmpxchg8/16b is available
+    // and default to library calls otherwise.
+    if (MemType->getPrimitiveSizeInBits() > NativeWidth) {
+      return needsCmpXchgNb(MemType) ? AtomicExpansionKind::CmpXChg
+                                     : AtomicExpansionKind::None;
+    }
+
+    AtomicRMWInst::BinOp Op = AI->getOperation();
+    switch (Op) {
+    default:
+      llvm_unreachable("Unknown atomic operation");
+    case AtomicRMWInst::Xchg:
+    case AtomicRMWInst::Add:
+    case AtomicRMWInst::Sub:
+      // It's better to use xadd, xsub or xchg for these in all cases.
+      return AtomicExpansionKind::None;
+    case AtomicRMWInst::Or:
+    case AtomicRMWInst::And:
+    case AtomicRMWInst::Xor:
+      return shouldExpandLogicAtomicRMWInIR(AI);
+    case AtomicRMWInst::Nand:
+    case AtomicRMWInst::Max:
+    case AtomicRMWInst::Min:
+    case AtomicRMWInst::UMax:
+    case AtomicRMWInst::UMin:
+    case AtomicRMWInst::FAdd:
+    case AtomicRMWInst::FSub:
+      // These always require a non-trivial set of data operations on x86. We
+      // must use a cmpxchg loop.
+      return AtomicExpansionKind::CmpXChg;
+    }
+  }
+
+  return TargetLoweringBase::shouldExpandAtomicInstInIR(I);
+}
+
 TargetLowering::AtomicExpansionKind
 X86TargetLowering::shouldExpandLogicAtomicRMWInIR(AtomicRMWInst *AI) const {
   // If the atomicrmw's result isn't actually used, we can just add a "lock"
   // prefix to a normal instruction for these operations.
   if (AI->use_empty())
     return AtomicExpansionKind::None;
 
   // If the atomicrmw's result is used by a single bit AND, we may use
void X86TargetLowering::emitBitTestAtomicRMWIntrinsic(AtomicRMWInst *AI) const {
   Value *Addr = Builder.CreatePointerCast(AI->getPointerOperand(),
                                           Type::getInt8PtrTy(Ctx));
   Value *Result = Builder.CreateCall(BitTest, {Addr, Builder.getInt8(Imm)});
   I->replaceAllUsesWith(Result);
   I->eraseFromParent();
   AI->eraseFromParent();
 }
 
-TargetLowering::AtomicExpansionKind
-X86TargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
-  unsigned NativeWidth = Subtarget.is64Bit() ? 64 : 32;
-  Type *MemType = AI->getType();
-
-  // If the operand is too big, we must see if cmpxchg8/16b is available
-  // and default to library calls otherwise.
-  if (MemType->getPrimitiveSizeInBits() > NativeWidth) {
-    return needsCmpXchgNb(MemType) ? AtomicExpansionKind::CmpXChg
-                                   : AtomicExpansionKind::None;
-  }
-
-  AtomicRMWInst::BinOp Op = AI->getOperation();
-  switch (Op) {
-  default:
-    llvm_unreachable("Unknown atomic operation");
-  case AtomicRMWInst::Xchg:
-  case AtomicRMWInst::Add:
-  case AtomicRMWInst::Sub:
-    // It's better to use xadd, xsub or xchg for these in all cases.
-    return AtomicExpansionKind::None;
-  case AtomicRMWInst::Or:
-  case AtomicRMWInst::And:
-  case AtomicRMWInst::Xor:
-    return shouldExpandLogicAtomicRMWInIR(AI);
-  case AtomicRMWInst::Nand:
-  case AtomicRMWInst::Max:
-  case AtomicRMWInst::Min:
-  case AtomicRMWInst::UMax:
-  case AtomicRMWInst::UMin:
-  case AtomicRMWInst::FAdd:
-  case AtomicRMWInst::FSub:
-    // These always require a non-trivial set of data operations on x86. We must
-    // use a cmpxchg loop.
-    return AtomicExpansionKind::CmpXChg;
-  }
-}
-
 LoadInst *
 X86TargetLowering::lowerIdempotentRMWIntoFencedLoad(AtomicRMWInst *AI) const {
   unsigned NativeWidth = Subtarget.is64Bit() ? 64 : 32;
   Type *MemType = AI->getType();
   // Accesses larger than the native width are turned into cmpxchg/libcalls, so
   // there is no benefit in turning such RMWs into loads, and it is actually
   // harmful as it introduces a mfence.
   if (MemType->getPrimitiveSizeInBits() > NativeWidth)
if (N->getOpcode() == ISD::VSELECT && Cond.hasOneUse() &&
     SDValue CondNew = DAG.getNOT(DL, Cond, CondVT);
     // Vselect cond, op1, op2 = Vselect not(cond), op2, op1
     return DAG.getSelect(DL, VT, CondNew, RHS, LHS);
   }
 
   // Attempt to convert a (vXi1 bitcast(iX Cond)) selection mask before it might
   // get split by legalization.
   if (N->getOpcode() == ISD::VSELECT && Cond.getOpcode() == ISD::BITCAST &&
-      CondVT.getVectorElementType() == MVT::i1 && Cond.hasOneUse() && 
+      CondVT.getVectorElementType() == MVT::i1 && Cond.hasOneUse() &&
       TLI.isTypeLegal(VT.getScalarType())) {
     EVT ExtCondVT = VT.changeVectorElementTypeToInteger();
     if (SDValue ExtCond = combineToExtendBoolVectorInReg(
             ISD::SIGN_EXTEND, DL, ExtCondVT, Cond, DAG, DCI, Subtarget)) {
       ExtCond = DAG.getNode(ISD::TRUNCATE, DL, CondVT, ExtCond);
       return DAG.getSelect(DL, VT, ExtCond, LHS, RHS);
     }
   }