Add AtomicExpandPass::bracketInstWithFences, and use it whenever getInsertFencesForAtomic would trigger in SelectionDAGBuilder

[morisset]

Summary

The goal is to eventually remove all the code related to getInsertFencesForAtomic
in SelectionDAGBuilder as it is wrong (designed for ARM, not really portable, works
mostly by accident because the backends are overly conservative), and repeats the
same logic that goes in emitLeading/TrailingFence.

In this patch, I make AtomicExpandPass insert the fences as it knows better
where to put them. Because this requires getting the fences and not just
passing an IRBuilder around, I had to change the return type of
emitLeading/TrailingFence.
This code only triggers on ARM for now. Because it is earlier in the pipeline
than SelectionDAGBuilder, it triggers and lowers atomic accesses to atomic so
SelectionDAGBuilder does not add barriers anymore on ARM.

If this patch is accepted I plan to implement emitLeading/TrailingFence for all
backends that setInsertFencesForAtomic(true), which will allow both making them
less conservative and simplifying SelectionDAGBuilder once they are all using
this interface.

Depends on D5090 (very minor dependency, both just happen to modify
AtomicExpandPass::runOnFunction).

[morisset]

Cleanup of some now redundant code that I had missed.

I would love getting some reviews on this patch, other patches depend on it
(D5180 being one that is otherwise ready to land). Thanks !

[jfb]

include/llvm/Target/TargetLowering.h

963

What should the return value be? Are there cases where a backend emits more than one instruction?

lib/CodeGen/AtomicExpandPass.cpp

177

Should this still be here, since the code wants to handle more than just ARM?

[morisset]

Thanks for the review, I will fix the comments as described below.

include/llvm/Target/TargetLowering.h

963

The return value is either nullptr (if the backend did not insert anything) or a pointer to the inserted (IR-level) instruction. There is no reason for a backend to insert several (IR-level) instructions, the only thing that makes sense for a backend to insert is an intrinsic corresponding to its hardware fence(s). Even if it requires several machine instructions for a fence, it can still be just one IR-level instruction.
I will add a comment to that effect.

lib/CodeGen/AtomicExpandPass.cpp

177

Probably not exactly as worded. I will reword it by saying that on some architectures load-linked is atomic for bigger sizes, and that for example on ARM...

[jfb]

Probably worth putting in the commit message: no tests changed because this shouldn't cause any functional changes?

[morisset]

Updated comments.

[jfb]

lib/CodeGen/AtomicExpandPass.cpp

179

That reminds me: ARM guarantees that ldrd and strd are atomic if the CPU has LPAE (e.g. A15 has that guarantee, see DDI0406C ARM architecture reference manual, sections A8.8.72-74 LDRD). Could you add this to your list of things to implement (emit ldrd/strd when appropriate)?

lib/Target/ARM/ARMISelLowering.cpp

11020

Fold this into the default case for the switch.

11040

Same.

[rnk]

lib/Target/ARM/ARMISelLowering.cpp

11020

We can't do this because MSVC will warn about falling off the end of the function.

[morisset]

lib/CodeGen/AtomicExpandPass.cpp

179

I just commited a FIXME to that effect in r218326.

[jfb]

Looks good.

lib/CodeGen/AtomicExpandPass.cpp

179

Thanks!

lib/Target/ARM/ARMISelLowering.cpp

11020

Grumble grumble MSVC.

[morisset]

Closed by commit rL218329 (authored by @morisset).

include/llvm/Target/TargetLowering.h

virtual Value *emitStoreConditional(IRBuilder<> &Builder, Value *Val,
                                       Value *Addr, AtomicOrdering Ord) const {
     llvm_unreachable("Store conditional unimplemented on this target");
   }
 
   /// Inserts in the IR a target-specific intrinsic specifying a fence.
   /// It is called by AtomicExpandPass before expanding an
   ///   AtomicRMW/AtomicCmpXchg/AtomicStore/AtomicLoad.
   /// RMW and CmpXchg set both IsStore and IsLoad to true.
   /// Backends with !getInsertFencesForAtomic() should keep a no-op here.
-  virtual void emitLeadingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
+  virtual Instruction* emitLeadingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
           bool IsStore, bool IsLoad) const {
     assert(!getInsertFencesForAtomic());
+    return nullptr;
   }
 
   /// Inserts in the IR a target-specific intrinsic specifying a fence.
   /// It is called by AtomicExpandPass after expanding an
   ///   AtomicRMW/AtomicCmpXchg/AtomicStore/AtomicLoad.
   /// RMW and CmpXchg set both IsStore and IsLoad to true.
   /// Backends with !getInsertFencesForAtomic() should keep a no-op here.
-  virtual void emitTrailingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
+  virtual Instruction* emitTrailingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
           bool IsStore, bool IsLoad) const {
     assert(!getInsertFencesForAtomic());
+    return nullptr;
   }
 
   /// Returns true if the given (atomic) store should be expanded by the
   /// IR-level AtomicExpand pass into an "atomic xchg" which ignores its input.
   virtual bool shouldExpandAtomicStoreInIR(StoreInst *SI) const {
     return false;
   }
 

lib/CodeGen/AtomicExpandPass.cpp

public:
     explicit AtomicExpand(const TargetMachine *TM = nullptr)
       : FunctionPass(ID), TM(TM) {
       initializeAtomicExpandPass(*PassRegistry::getPassRegistry());
     }
 
     bool runOnFunction(Function &F) override;
 
   private:
+    bool bracketInstWithFences(Instruction *I, AtomicOrdering Order,
+                               bool IsStore, bool IsLoad);
     bool expandAtomicLoad(LoadInst *LI);
     bool expandAtomicStore(StoreInst *SI);
     bool expandAtomicRMW(AtomicRMWInst *AI);
     bool expandAtomicRMWToLLSC(AtomicRMWInst *AI);
     bool expandAtomicRMWToCmpXchg(AtomicRMWInst *AI);
     bool expandAtomicCmpXchg(AtomicCmpXchgInst *CI);
   };
 }
bool AtomicExpand::runOnFunction(Function &F) {
   }
 
   bool MadeChange = false;
   for (auto I : AtomicInsts) {
     auto LI = dyn_cast<LoadInst>(I);
     auto SI = dyn_cast<StoreInst>(I);
     auto RMWI = dyn_cast<AtomicRMWInst>(I);
     auto CASI = dyn_cast<AtomicCmpXchgInst>(I);
-
     assert((LI || SI || RMWI || CASI || isa<FenceInst>(I)) &&
            "Unknown atomic instruction");
 
+    auto FenceOrdering = Monotonic;
+    bool IsStore, IsLoad;
+    if (TargetLowering->getInsertFencesForAtomic()) {
+      if (LI && isAtLeastAcquire(LI->getOrdering())) {
+        FenceOrdering = LI->getOrdering();
+        LI->setOrdering(Monotonic);
+        IsStore = false;
+        IsLoad = true;
+      } else if (SI && isAtLeastRelease(SI->getOrdering())) {
+        FenceOrdering = SI->getOrdering();
+        SI->setOrdering(Monotonic);
+        IsStore = true;
+        IsLoad = false;
+      } else if (RMWI && !TargetLowering->shouldExpandAtomicRMWInIR(RMWI)) {
+        // We do not bother calling bracketInstWithFences if the RMW is already
+        // set for expansion, as the expansion already inserts whatever fence is
+        // required.
+        FenceOrdering = RMWI->getOrdering();
+        RMWI->setOrdering(Monotonic);
+        IsStore = IsLoad = true;
+      } else if (CASI && !TargetLowering->hasLoadLinkedStoreConditional()) {
+        // Same as above.
+        FenceOrdering = CASI->getSuccessOrdering();
+        CASI->setSuccessOrdering(Monotonic);
+        CASI->setFailureOrdering(Monotonic);
+        IsStore = IsLoad = true;
+      }
+
+      if (FenceOrdering != Monotonic) {
+        MadeChange |= bracketInstWithFences(I, FenceOrdering, IsStore, IsLoad);
+      }
+    }
+
     if (LI && TargetLowering->shouldExpandAtomicLoadInIR(LI)) {
       MadeChange |= expandAtomicLoad(LI);
     } else if (SI && TargetLowering->shouldExpandAtomicStoreInIR(SI)) {
       MadeChange |= expandAtomicStore(SI);
     } else if (RMWI && TargetLowering->shouldExpandAtomicRMWInIR(RMWI)) {
       MadeChange |= expandAtomicRMW(RMWI);
     } else if (CASI && TargetLowering->hasLoadLinkedStoreConditional()) {
       MadeChange |= expandAtomicCmpXchg(CASI);
     }
   }
   return MadeChange;
 }
 
+bool AtomicExpand::bracketInstWithFences(Instruction *I, AtomicOrdering Order,
+                                         bool IsStore, bool IsLoad) {
+  IRBuilder<> Builder(I);
+
+  auto LeadingFence =
+      TM->getSubtargetImpl()->getTargetLowering()->emitLeadingFence(
+      Builder, Order, IsStore, IsLoad);
+
+  auto TrailingFence =
+      TM->getSubtargetImpl()->getTargetLowering()->emitTrailingFence(
+      Builder, Order, IsStore, IsLoad);
+  // The trailing fence is emitted before the instruction instead of after
+  // because there is no easy way of setting Builder insertion point after
+  // an instruction. So we must erase it from the BB, and insert it back
+  // in the right place.
+  // We have a guard here because not every atomic operation generates a
+  // trailing fence.
+  if (TrailingFence) {
+    TrailingFence->removeFromParent();
+    TrailingFence->insertAfter(I);
+  }
+
+  return (LeadingFence || TrailingFence);
+}
+
 bool AtomicExpand::expandAtomicLoad(LoadInst *LI) {
   auto TLI = TM->getSubtargetImpl()->getTargetLowering();
   // If getInsertFencesForAtomic() returns true, then the target does not want
   // to deal with memory orders, and emitLeading/TrailingFence should take care
   // of everything. Otherwise, emitLeading/TrailingFence are no-op and we
   // should preserve the ordering.
   AtomicOrdering MemOpOrder =
       TLI->getInsertFencesForAtomic() ? Monotonic : LI->getOrdering();
   IRBuilder<> Builder(LI);
 
   // Note that although no fence is required before atomic load on ARM, it is
   // required before SequentiallyConsistent loads for the recommended Power
   // mapping (see http://www.cl.cam.ac.uk/~pes20/cpp/cpp0xmappings.html).
   // So we let the target choose what to emit.
   TLI->emitLeadingFence(Builder, LI->getOrdering(),
                         /*IsStore=*/false, /*IsLoad=*/true);
 
   // The only 64-bit load guaranteed to be single-copy atomic by ARM is
   // an ldrexd (A3.5.3).
   Value *Val =
       TLI->emitLoadLinked(Builder, LI->getPointerOperand(), MemOpOrder);
 
   TLI->emitTrailingFence(Builder, LI->getOrdering(),
                          /*IsStore=*/false, /*IsLoad=*/true);
 
   LI->replaceAllUsesWith(Val);
   LI->eraseFromParent();
 
   return true;

lib/Target/ARM/ARMISelLowering.h

bool functionArgumentNeedsConsecutiveRegisters(
         Type *Ty, CallingConv::ID CallConv, bool isVarArg) const override;
 
     bool hasLoadLinkedStoreConditional() const override;
     Value *emitLoadLinked(IRBuilder<> &Builder, Value *Addr,
                           AtomicOrdering Ord) const override;
     Value *emitStoreConditional(IRBuilder<> &Builder, Value *Val,
                                 Value *Addr, AtomicOrdering Ord) const override;
 
-    void emitLeadingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
+    Instruction* emitLeadingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
                           bool IsStore, bool IsLoad) const override;
-    void emitTrailingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
+    Instruction* emitTrailingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
                            bool IsStore, bool IsLoad) const override;
 
     bool shouldExpandAtomicLoadInIR(LoadInst *LI) const override;
     bool shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
     bool shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
 
     bool useLoadStackGuardNode() const override;
 

lib/Target/ARM/ARMISelLowering.cpp

bool ARMTargetLowering::shouldConvertConstantLoadToIntImm(const APInt &Imm,
   unsigned Bits = Ty->getPrimitiveSizeInBits();
   if (Bits == 0 || Bits > 32)
     return false;
   return true;
 }
 
 bool ARMTargetLowering::hasLoadLinkedStoreConditional() const { return true; }
 
-static void makeDMB(IRBuilder<> &Builder, ARM_MB::MemBOpt Domain) {
+static Instruction* makeDMB(IRBuilder<> &Builder, ARM_MB::MemBOpt Domain) {
   Module *M = Builder.GetInsertBlock()->getParent()->getParent();
   Function *DMB = llvm::Intrinsic::getDeclaration(M, Intrinsic::arm_dmb);
   Constant *CDomain = Builder.getInt32(Domain);
-  Builder.CreateCall(DMB, CDomain);
+  return Builder.CreateCall(DMB, CDomain);
 }
 
 // Based on http://www.cl.cam.ac.uk/~pes20/cpp/cpp0xmappings.html
-void ARMTargetLowering::emitLeadingFence(IRBuilder<> &Builder,
+Instruction* ARMTargetLowering::emitLeadingFence(IRBuilder<> &Builder,
                                          AtomicOrdering Ord, bool IsStore,
                                          bool IsLoad) const {
   if (!getInsertFencesForAtomic())
-    return;
+    return nullptr;
 
   switch (Ord) {
   case NotAtomic:
   case Unordered:
     llvm_unreachable("Invalid fence: unordered/non-atomic");
   case Monotonic:
   case Acquire:
-    return; // Nothing to do
+    return nullptr; // Nothing to do
   case SequentiallyConsistent:
     if (!IsStore)
-      return; // Nothing to do
-              /*FALLTHROUGH*/
+      return nullptr; // Nothing to do
   case Release:
   case AcquireRelease:
     if (Subtarget->isSwift())
-      makeDMB(Builder, ARM_MB::ISHST);
+      return makeDMB(Builder, ARM_MB::ISHST);
     // FIXME: add a comment with a link to documentation justifying this.
     else
-      makeDMB(Builder, ARM_MB::ISH);
-    return;
+      return makeDMB(Builder, ARM_MB::ISH);
   }
 }
 
-void ARMTargetLowering::emitTrailingFence(IRBuilder<> &Builder,
+Instruction* ARMTargetLowering::emitTrailingFence(IRBuilder<> &Builder,
                                           AtomicOrdering Ord, bool IsStore,
                                           bool IsLoad) const {
   if (!getInsertFencesForAtomic())
-    return;
+    return nullptr;
 
   switch (Ord) {
   case NotAtomic:
   case Unordered:
     llvm_unreachable("Invalid fence: unordered/not-atomic");
   case Monotonic:
   case Release:
-    return; // Nothing to do
+    return nullptr; // Nothing to do
   case Acquire:
   case AcquireRelease:
-    case SequentiallyConsistent:
-    makeDMB(Builder, ARM_MB::ISH);
-    return;
+    return makeDMB(Builder, ARM_MB::ISH);
   }
 }
 
 // Loads and stores less than 64-bits are already atomic; ones above that
 // are doomed anyway, so defer to the default libcall and blame the OS when
 // things go wrong. Cortex M doesn't have ldrexd/strexd though, so don't emit
 // anything for those.
 bool ARMTargetLowering::shouldExpandAtomicStoreInIR(StoreInst *SI) const {
   unsigned Size = SI->getValueOperand()->getType()->getPrimitiveSizeInBits();
   return (Size == 64) && !Subtarget->isMClass();