[MachineSink][AArch64] Sink instruction copies when they can replace copy into hard register or folded into addressing mode

[chill]

Summary

This patch adds a new code transformation to the MachineSink pass,
that tries to sink copies of an instruction, when the copies can be folded
into the addressing modes of load/store instructions, or
replace another instruction (currently, copies into a hard register).

The criteria for performing the transformation is that:

• the register pressure at the sink destination block must not exceed the register pressure limits
• the latency and throughput of the load/store or the copy must not deteriorate
• the original instruction must be deleted

[chill]

Bug fixes.

[chill]

Update:

• added a command line option --aarch64-enable-sink-fold=[true|false]
• fixed a test

[labrinea]

A few nitpicks.

llvm/include/llvm/CodeGen/TargetInstrInfo.h

1449

typo

1463

typo

llvm/lib/CodeGen/MachineSink.cpp

483

pressure

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

2770

indentation

3098

indentation

[dmgreen]

High level comment - I like having the ability to do this, is there a reason it is a sink, as opposed to a peephole? I think a lot of the time it may just always be profitable to combine into address operands or add+lsl. Is it due to the register pressure?

[chill]

Update:

• added folding of 32-bit zero-/sign-extends into load/store addressing mode
• removed an arbitrary restriction of not folding instructions in the same basic block
• implemented a slightly more efficient cleanup of dead COPYs
• removed references to nonexistent virtual registers from debug instructions

[chill]

.. and now with the correct patch ... :x

[chill]

In D152828#4528166, @dmgreen wrote:

High level comment - I like having the ability to do this, is there a reason it is a sink, as opposed to a peephole? I think a lot of the time it may just always be profitable to combine into address operands or add+lsl. Is it due to the register pressure?

Yes, I think we should take register pressure into account, as we can replace use of one register with a use of two registers (e.g. ldr Xd, [Xa] -> ldr Xd, [Xn, Xm]).

[chill]

Fixed a miscompilation.

[dmgreen]

There is a lot of code here, but it looks like a good idea. I was looking at folding shifts into and/or recently too, and it looked like there were cases where it is better to be careful about where it is combined based on the uses. The same is likely true for add/sub and addressing modes too.

llvm/lib/CodeGen/MachineSink.cpp

511

Doesn't need a ; to end the inside of the LLVM_DEBUG.

535

-> addressing mode memory instruction?

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

2795

addressing

2942

I think it's worth clarifying how these should work with LSLFast and addressing operands. I had an explanation of how I thought LSLFast should work in https://reviews.llvm.org/D155470#4527270. Let me know if you think doesn't sound right. There is a patch to split LSLFast into multiple parts as a first step in https://reviews.llvm.org/D157982.

3343

Would constrainRegClass work too, to avoid the COPY?

3362

instruction

9307

shift -> Shift

llvm/test/CodeGen/AArch64/arm64-xaluo.ll

105 ↗

(On Diff #548667)

Can you regenerate these tests in the parent, to remove the unrelated changes from here?

llvm/test/CodeGen/AArch64/sink-and-fold.ll

305

Are these tests new? Can this be removed in the parent?

[chill]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

3343

We come here with virtual register in AM.BaseReg in the GPR64 register class, but the load/store instruction needs a register in the GPR64sp, which is not a subclass of GPR64 (it's less constrained, a superclass).

IIUC, constrainRegClass would work in the opposite direction of what we need, from GPR64sp to GPR64.

Alternatively, that might be fixed in the verifier, to not complain if an operand is not in the exact register class, but is nevertheless in a more constrained class (a subclass).

[chill]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

3343

All right, having said that, I just tried using constrainRegClass and it worked. I'll look some more into it.

[chill]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

3343

I see what happens, neither one of GPR64 or GPR64sp is a superclass/subclass of the other as I thought (one has XZR where the other has SP). So constrainRegClass yields GPR64common and the machine verifier already works exactly suggested above.

[chill]

llvm/test/CodeGen/AArch64/sink-and-fold.ll

305

These tests are supposed to be in the new patch as well.

[dmgreen]

I've been wondering how to stage this, whilst trying to move towards https://reviews.llvm.org/D155470#4527270. I think this implements something closer to "Ext23Fast" from the 4 options there, but checks for LSLFast.

It might be best to go with this, and then we can adjust the Target features and clean up the uses in another patch. Otherwise we are trying to do too many things at once, and there is already quite a bit of code here. I have some questions inline about the folding of add's, but otherwise from what I can tell this looks good.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

2939

I believe it is shifts of 1 or 4 that would be more expensive for OoO cores, but the other shift types are also cheap. AddrLSLFast means any addressing mode with a LSL with shift <= 3 are cheap. ALULSLFast means adds/subs with LSL<=4 are fast.

I think the logic should be similar to that in DAGCombine (ignoring register pressure for a moment). If we are optimizing for size or there are no other uses the fold should be beneficial. Otherwise we treat it as cheap if we have AddrLSLFast and the shift is <= 3. An ADDXrs could take 2 cycles anyway so could be more aggressive?

Does this take into account the number of uses, and should it? Should it fold more under Optsize?

[chill]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

2939

Yeah, that makes sense. We already perform the transformation only if the original instruction is removed (i.e. can be folded into all of its users), if we are also optimizing for size, then we can ignore here the (minor) increase on the cycle count, if that would increase the chances for the transformation happening.

2942

I guess we can be more precise by distinguishing between cores where shifts 0, 1, 2, and 3 are fast
and cores where just shifts 0, 2, and 3 are fast. Not sure if it's worth adding an extra feature. Lacking such a feature, for now the code considers shift 1 to be slow by default, but it could just as well consider shift 1 to be fast by default.

[dmgreen]

I have been wondering what to do with this, whether to nitpick the costmodel in ways that might not be useful, but it is probably better to get it in and work out any adjustments to the target features as needed. I think this is closer to what we are aiming for in terms of heuristics.

Is it worth setting EnableSinkAndFold to false, we can commit this then have another patch (doesn't need review) to enable it? Just in case there are problems it can help ease the commit-revert cycles.

Otherwise I was looking through the sinking code and had an extra question.

llvm/lib/CodeGen/MachineSink.cpp

467–468

Is this assuming that the UseInst is a copy or that canFoldIntoAddrMode instructions always have an Operand(1) which is a reg? Is UseInst.getOperand(1) always MO in for AArch64?

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

2942

Arm OoO cores can treat any extend as cheap, so long as the shift is 2 or 3. For in order cores it appears to be any shift. My understanding of LSLFast was the the UXTW/SXTW would still not be considered cheap. What you have is probably good though, and matches my understanding of what many cores implement.

[chill]

llvm/lib/CodeGen/MachineSink.cpp

467–468

The comment is obsolete and the expression is incorrect.

We have here the original instruction

DefReg = opc UsedRegA, UsedRegB

and a chain of copies (maybe empty)

DefReg ->  ... -> Reg

and the instruction we're trying to fold into

... = opc Op0, Op1, ..., Reg, ...

We are replacing Reg with some expression, which involves UsedRegA and UsedRegB , so potentially where we had one register now we would have two, so register pressure may increase.
However, if Reg is the in the same register class as UsedRegA or UsedRegB, then register pressure for that register class does not increase, as we are simply replacing one virtual reg with another virtual reg.

The expression should be just const TargetRegisterClass *RCS = MRI->getRegClass(Reg);

[dmgreen]

LGTM, thanks

llvm/include/llvm/CodeGen/TargetInstrInfo.h

struct RegImmPair {
   Register Reg;
   int64_t Imm;
 
   RegImmPair(Register Reg, int64_t Imm) : Reg(Reg), Imm(Imm) {}
 };
 
 /// Used to describe addressing mode similar to ExtAddrMode in CodeGenPrepare.
 /// It holds the register values, the scale value and the displacement.
+/// It also holds a descriptor for the expression used to calculate the address
+/// from the operands.
 struct ExtAddrMode {
+  enum class Formula {
+    Basic = 0,         // BaseReg + ScaledReg * Scale + Displacement
+    SExtScaledReg = 1, // BaseReg + sext(ScaledReg) * Scale + Displacement
+    ZExtScaledReg = 2  // BaseReg + zext(ScaledReg) * Scale + Displacement
+  };
+
   Register BaseReg;
   Register ScaledReg;
-  int64_t Scale;
-  int64_t Displacement;
+  int64_t Scale = 0;
+  int64_t Displacement = 0;
+  Formula Form = Formula::Basic;
+  ExtAddrMode() = default;
 };
 
 //---------------------------------------------------------------------------
 ///
 /// TargetInstrInfo - Interface to description of machine instruction set
 ///
 class TargetInstrInfo : public MCInstrInfo {
 public:
public:
   /// struct ExtAddrMode which contains all relevant information to make up the
   /// address.
   virtual std::optional<ExtAddrMode>
   getAddrModeFromMemoryOp(const MachineInstr &MemI,
                           const TargetRegisterInfo *TRI) const {
     return std::nullopt;
   }
 
+  /// Check if it's possible and beneficial to fold the addressing computation
+  /// `AddrI` into the addressing mode of the load/store instruction `MemI`. The
+  /// memory instruction is a user of the virtual register `Reg`, which in turn
+  /// is the ultimate destination of zero or more COPY instructions from the
+  /// output register of `AddrI`.
+  /// Return the adddressing mode after folding in `AM`.
+  virtual bool canFoldIntoAddrMode(const MachineInstr &MemI, Register Reg,
+                                   const MachineInstr &AddrI,
+                                   ExtAddrMode &AM) const {
+    return false;
+  }
+
+  /// Emit a load/store instruction with the same value register as `MemI`, but
+  /// using the address from `AM`. The addressing mode must have been obtained
+  /// from `canFoldIntoAddr` for the same memory instruction.
+  virtual MachineInstr *emitLdStWithAddr(MachineInstr &MemI,
+                                         const ExtAddrMode &AM) const {
+    llvm_unreachable("target did not implement emitLdStWithAddr()");
+  }
+
   /// Returns true if MI's Def is NullValueReg, and the MI
   /// does not change the Zero value. i.e. cases such as rax = shr rax, X where
   /// NullValueReg = rax. Note that if the NullValueReg is non-zero, this
   /// function can return true even if becomes zero. Specifically cases such as
   /// NullValueReg = shl NullValueReg, 63.
   virtual bool preservesZeroValueInReg(const MachineInstr *MI,
                                        const Register NullValueReg,
                                        const TargetRegisterInfo *TRI) const {

llvm/include/llvm/CodeGen/TargetPassConfig.h

protected:
 
   // Target Pass Options
   // Targets provide a default setting, user flags override.
   bool DisableVerify = false;
 
   /// Default setting for -enable-tail-merge on this target.
   bool EnableTailMerge = true;
 
+  /// Enable sinking of instructions in MachineSink where a computation can be
+  /// folded into the addressing mode of a memory load/store instruction or
+  /// replace a copy.
+  bool EnableSinkAndFold = false;
+
   /// Require processing of functions such that callees are generated before
   /// callers.
   bool RequireCodeGenSCCOrder = false;
 
   /// Add the actual instruction selection passes. This does not include
   /// preparation passes on IR.
   bool addCoreISelPasses();
 
public:
   static std::string
   getLimitedCodeGenPipelineReason(const char *Separator = "/");
 
   void setDisableVerify(bool Disable) { setOpt(DisableVerify, Disable); }
 
   bool getEnableTailMerge() const { return EnableTailMerge; }
   void setEnableTailMerge(bool Enable) { setOpt(EnableTailMerge, Enable); }
 
+  bool getEnableSinkAndFold() const { return EnableSinkAndFold; }
+  void setEnableSinkAndFold(bool Enable) { setOpt(EnableSinkAndFold, Enable); }
+
   bool requiresCodeGenSCCOrder() const { return RequireCodeGenSCCOrder; }
   void setRequiresCodeGenSCCOrder(bool Enable = true) {
     setOpt(RequireCodeGenSCCOrder, Enable);
   }
 
   /// Allow the target to override a specific pass without overriding the pass
   /// pipeline. When passes are added to the standard pipeline at the
   /// point where StandardID is expected, add TargetID in its place.

llvm/lib/CodeGen/ImplicitNullChecks.cpp

ImplicitNullChecks::isSuitableMemoryOp(const MachineInstr &MI,
   // Implementation restriction for faulting_op insertion
   // TODO: This could be relaxed if we find a test case which warrants it.
   if (MI.getDesc().getNumDefs() > 1)
    return SR_Unsuitable;
 
   if (!MI.mayLoadOrStore() || MI.isPredicable())
     return SR_Unsuitable;
   auto AM = TII->getAddrModeFromMemoryOp(MI, TRI);
-  if (!AM)
+  if (!AM || AM->Form != ExtAddrMode::Formula::Basic)
     return SR_Unsuitable;
   auto AddrMode = *AM;
   const Register BaseReg = AddrMode.BaseReg, ScaledReg = AddrMode.ScaledReg;
   int64_t Displacement = AddrMode.Displacement;
 
   // We need the base of the memory instruction to be same as the register
   // where the null check is performed (i.e. PointerReg).
   if (BaseReg != PointerReg && ScaledReg != PointerReg)

llvm/lib/CodeGen/MachineSink.cpp

 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachinePostDominators.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/RegisterClassInfo.h"
 #include "llvm/CodeGen/RegisterPressure.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
+#include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/Pass.h"

 STATISTIC(NumCycleSunk,  "Number of machine instructions sunk into a cycle");
 STATISTIC(NumSplit,     "Number of critical edges split");
 STATISTIC(NumCoalesces, "Number of copies coalesced");
 STATISTIC(NumPostRACopySink, "Number of copies sunk after RA");
 
 namespace {
 
   class MachineSinking : public MachineFunctionPass {
+    const TargetSubtargetInfo *STI = nullptr;
     const TargetInstrInfo *TII = nullptr;
     const TargetRegisterInfo *TRI = nullptr;
     MachineRegisterInfo *MRI = nullptr;      // Machine register information
     MachineDominatorTree *DT = nullptr;      // Machine dominator tree
     MachinePostDominatorTree *PDT = nullptr; // Machine post dominator tree
     MachineCycleInfo *CI = nullptr;
     MachineBlockFrequencyInfo *MBFI = nullptr;
     const MachineBranchProbabilityInfo *MBPI = nullptr;
class MachineSinking : public MachineFunctionPass {
 
     std::map<std::pair<MachineBasicBlock *, MachineBasicBlock *>, bool>
         HasStoreCache;
     std::map<std::pair<MachineBasicBlock *, MachineBasicBlock *>,
              std::vector<MachineInstr *>>
         StoreInstrCache;
 
     /// Cached BB's register pressure.
-    std::map<MachineBasicBlock *, std::vector<unsigned>> CachedRegisterPressure;
+    std::map<const MachineBasicBlock *, std::vector<unsigned>>
+        CachedRegisterPressure;
+
+    bool EnableSinkAndFold;
 
   public:
     static char ID; // Pass identification
 
     MachineSinking() : MachineFunctionPass(ID) {
       initializeMachineSinkingPass(*PassRegistry::getPassRegistry());
     }
 
     bool runOnMachineFunction(MachineFunction &MF) override;
 
     void getAnalysisUsage(AnalysisUsage &AU) const override {
       MachineFunctionPass::getAnalysisUsage(AU);
       AU.addRequired<AAResultsWrapperPass>();
       AU.addRequired<MachineDominatorTree>();
       AU.addRequired<MachinePostDominatorTree>();
       AU.addRequired<MachineCycleInfoWrapperPass>();
       AU.addRequired<MachineBranchProbabilityInfo>();
       AU.addPreserved<MachineCycleInfoWrapperPass>();
       AU.addPreserved<MachineLoopInfo>();
       if (UseBlockFreqInfo)
         AU.addRequired<MachineBlockFrequencyInfo>();
+      AU.addRequired<TargetPassConfig>();
     }
 
     void releaseMemory() override {
       CEBCandidates.clear();
     }
 
   private:
     bool ProcessBlock(MachineBasicBlock &MBB);
private:
     bool isProfitableToSinkTo(Register Reg, MachineInstr &MI,
                               MachineBasicBlock *MBB,
                               MachineBasicBlock *SuccToSinkTo,
                               AllSuccsCache &AllSuccessors);
 
     bool PerformTrivialForwardCoalescing(MachineInstr &MI,
                                          MachineBasicBlock *MBB);
 
+    bool PerformSinkAndFold(MachineInstr &MI, MachineBasicBlock *MBB);
+
     SmallVector<MachineBasicBlock *, 4> &
     GetAllSortedSuccessors(MachineInstr &MI, MachineBasicBlock *MBB,
                            AllSuccsCache &AllSuccessors) const;
 
-    std::vector<unsigned> &getBBRegisterPressure(MachineBasicBlock &MBB);
+    std::vector<unsigned> &getBBRegisterPressure(const MachineBasicBlock &MBB);
+
+    bool registerPressureSetExceedsLimit(unsigned NRegs,
+                                         const TargetRegisterClass *RC,
+                                         const MachineBasicBlock &MBB);
   };
 
 } // end anonymous namespace
 
 char MachineSinking::ID = 0;
 
 char &llvm::MachineSinkingID = MachineSinking::ID;
 
bool MachineSinking::PerformTrivialForwardCoalescing(MachineInstr &MI,
   // Conservatively, clear any kill flags, since it's possible that they are no
   // longer correct.
   MRI->clearKillFlags(SrcReg);
 
   ++NumCoalesces;
   return true;
 }
 
+bool MachineSinking::PerformSinkAndFold(MachineInstr &MI,
+                                        MachineBasicBlock *MBB) {
+  if (MI.isCopy() || MI.mayLoadOrStore() ||
+      MI.getOpcode() == TargetOpcode::REG_SEQUENCE)
+    return false;
+
+  // Don't sink instructions that the target prefers not to sink.
+  if (!TII->shouldSink(MI))
+    return false;
+
+  // Check if it's safe to move the instruction.
+  bool SawStore = true;
+  if (!MI.isSafeToMove(AA, SawStore))
+    return false;
+
+  // Convergent operations may not be made control-dependent on additional
+  // values.
+  if (MI.isConvergent())
+    return false;
+
+  // Don't sink defs/uses of hard registers or if the instruction defines more
+  // than one register.
+  // Don't sink more than two register uses - it'll cover most of the cases and
+  // greatly simplifies the register pressure checks.
+  Register DefReg;
+  Register UsedRegA, UsedRegB;
+  for (const MachineOperand &MO : MI.operands()) {
+    if (MO.isImm() || MO.isRegMask() || MO.isRegLiveOut() || MO.isMetadata() ||
+        MO.isMCSymbol() || MO.isDbgInstrRef() || MO.isCFIIndex() ||
+        MO.isIntrinsicID() || MO.isPredicate() || MO.isShuffleMask())
+      continue;
+    if (!MO.isReg())
+      return false;
+
+    Register Reg = MO.getReg();
+    if (Reg == 0)
+      continue;
+
+    if (Reg.isVirtual()) {
+      if (MO.isDef()) {
+        if (DefReg)
+          return false;
+        DefReg = Reg;
+        continue;
+      }
+
+      if (UsedRegA == 0)
+        UsedRegA = Reg;
+      else if (UsedRegB == 0)
+        UsedRegB = Reg;
+      else
+        return false;
+      continue;
+    }
+
+    if (Reg.isPhysical() &&
+        (MRI->isConstantPhysReg(Reg) || TII->isIgnorableUse(MO)))
+      continue;
+
+    return false;
+  }
+
+  // Scan uses of the destination register. Every use, except the last, must be
+  // a copy, with a chain of copies terminating with either a copy into a hard
+  // register, or a load/store instruction where the use is part of the
+  // address (*not* the stored value).
+  using SinkInfo = std::pair<MachineInstr *, ExtAddrMode>;
+  SmallVector<SinkInfo> SinkInto;
+  SmallVector<Register> Worklist;
+
+  const TargetRegisterClass *RC = MRI->getRegClass(DefReg);
+  const TargetRegisterClass *RCA =
+      UsedRegA == 0 ? nullptr : MRI->getRegClass(UsedRegA);
+  const TargetRegisterClass *RCB =
+      UsedRegB == 0 ? nullptr : MRI->getRegClass(UsedRegB);
+
+  Worklist.push_back(DefReg);
+  while (!Worklist.empty()) {
+    Register Reg = Worklist.pop_back_val();
+
+    for (MachineOperand &MO : MRI->use_nodbg_operands(Reg)) {
+      ExtAddrMode MaybeAM;
+      MachineInstr &UseInst = *MO.getParent();
+      if (UseInst.isCopy()) {
+        Register DstReg;
+        if (const MachineOperand &O = UseInst.getOperand(0); O.isReg())
+          DstReg = O.getReg();
+        if (DstReg == 0)
+          return false;
+        if (DstReg.isVirtual()) {
+          Worklist.push_back(DstReg);
+          continue;
+        }
+        // If we are going to replace a copy, the original instruction must be
+        // as cheap as a copy.
+        if (!TII->isAsCheapAsAMove(MI))
+          return false;
+        // The hard register must be in the register class of the original
+        // instruction's destination register.
+        if (!RC->contains(DstReg))
+          return false;
+      } else if (UseInst.mayLoadOrStore()) {
+        ExtAddrMode AM;
+        if (!TII->canFoldIntoAddrMode(UseInst, Reg, MI, AM))
+          return false;
+        MaybeAM = AM;
+      } else {
+        return false;
+      }
+
+      if (UseInst.getParent() != MI.getParent()) {
+        // If the register class of the register we are replacingis a superset
+        // of any of the register classes of the operands of the materialized
+        // instruction don't consider that live range extended.
+        const TargetRegisterClass *RCS = MRI->getRegClass(Reg);
+        if (RCA && RCA->hasSuperClassEq(RCS))
+          RCA = nullptr;
+        else if (RCB && RCB->hasSuperClassEq(RCS))
+          RCB = nullptr;
+        if (RCA || RCB) {
+          if (RCA == nullptr) {
+            RCA = RCB;
+            RCB = nullptr;
+          }
+
+          unsigned NRegs = !!RCA + !!RCB;
+          if (RCA == RCB)
+            RCB = nullptr;
+
+          // Check we don't exceed register pressure at the destination.
+          const MachineBasicBlock &MBB = *UseInst.getParent();
+          if (RCB == nullptr) {
+            if (registerPressureSetExceedsLimit(NRegs, RCA, MBB))
+              return false;
+          } else if (registerPressureSetExceedsLimit(1, RCA, MBB) ||
+                     registerPressureSetExceedsLimit(1, RCB, MBB)) {
+            return false;
+          }
+        }
+      }
+
+      SinkInto.emplace_back(&UseInst, MaybeAM);
+    }
+  }
+
+  if (SinkInto.empty())
+    return false;
+
+  // Now we know we can fold the instruction in all its users.
+  if (UsedRegA)
+    MRI->clearKillFlags(UsedRegA);
+  if (UsedRegB)
+    MRI->clearKillFlags(UsedRegB);
+
+  for (auto &[SinkDst, MaybeAM] : SinkInto) {
+    MachineInstr *New = nullptr;
+    LLVM_DEBUG(dbgs() << "Sinking copy of"; MI.dump(); dbgs() << "into";
+               SinkDst->dump());
+    if (SinkDst->isCopy()) {
+      // Sink a copy of the instruction, replacing a COPY instruction.
+      MachineBasicBlock::iterator InsertPt = SinkDst->getIterator();
+      Register DstReg = SinkDst->getOperand(0).getReg();
+      TII->reMaterialize(*SinkDst->getParent(), InsertPt, DstReg, 0, MI, *TRI);
+      // If the original instruction did not have source location, reuse a one
+      // from the COPY.
+      New = &*std::prev(InsertPt);
+      if (const DebugLoc &NewLoc = New->getDebugLoc(); !NewLoc)
+        New->setDebugLoc(SinkDst->getDebugLoc());
+      // Sink DBG_VALUEs, which refer to the original instruction's destination
+      // (DefReg).
+      MachineBasicBlock &SinkMBB = *SinkDst->getParent();
+      auto &DbgUsers = SeenDbgUsers[DefReg];
+      for (auto &U : DbgUsers) {
+        MachineInstr *DbgMI = U.getPointer();
+        if (U.getInt())
+          continue;
+        MachineInstr *NewDbgMI = SinkDst->getMF()->CloneMachineInstr(DbgMI);
+        NewDbgMI->getOperand(0).setReg(DstReg);
+        SinkMBB.insertAfter(InsertPt, NewDbgMI);
+      }
+    } else {
+      // Fold instruction into the addressing mode of a memory instruction.
+      New = TII->emitLdStWithAddr(*SinkDst, MaybeAM);
+    }
+    LLVM_DEBUG(dbgs() << "yielding"; New->dump());
+    SinkDst->eraseFromParent();
+  }
+
+  MI.eraseFromParent();
+
+  // Collect instructions that need to be deleted (COPYs). We cannot delete them
+  // while traversing register uses.
+  SmallVector<MachineInstr *> CleanupInstrs;
+  Worklist.push_back(DefReg);
+  while (!Worklist.empty()) {
+    Register Reg = Worklist.pop_back_val();
+
+    for (MachineOperand &MO : MRI->use_operands(Reg)) {
+      MachineInstr *U = MO.getParent();
+      assert((U->isCopy() || U->isDebugInstr()) &&
+             "Only debug uses and copies must remain");
+      if (U->isCopy()) {
+        Worklist.push_back(U->getOperand(0).getReg());
+        CleanupInstrs.push_back(U);
+      } else {
+        MO.setReg(0);
+        MO.setSubReg(0);
+      }
+    }
+  }
+
+  // Delete the dead COPYs.
+  for (MachineInstr *Del : CleanupInstrs)
+    Del->eraseFromParent();
+
+  return true;
+}
+
 /// AllUsesDominatedByBlock - Return true if all uses of the specified register
 /// occur in blocks dominated by the specified block. If any use is in the
 /// definition block, then return false since it is never legal to move def
 /// after uses.
 bool MachineSinking::AllUsesDominatedByBlock(Register Reg,
                                              MachineBasicBlock *MBB,
                                              MachineBasicBlock *DefMBB,
                                              bool &BreakPHIEdge,

 }
 
 bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   LLVM_DEBUG(dbgs() << "******** Machine Sinking ********\n");
 
-  TII = MF.getSubtarget().getInstrInfo();
-  TRI = MF.getSubtarget().getRegisterInfo();
+  STI = &MF.getSubtarget();
+  TII = STI->getInstrInfo();
+  TRI = STI->getRegisterInfo();
   MRI = &MF.getRegInfo();
   DT = &getAnalysis<MachineDominatorTree>();
   PDT = &getAnalysis<MachinePostDominatorTree>();
   CI = &getAnalysis<MachineCycleInfoWrapperPass>().getCycleInfo();
   MBFI = UseBlockFreqInfo ? &getAnalysis<MachineBlockFrequencyInfo>() : nullptr;
   MBPI = &getAnalysis<MachineBranchProbabilityInfo>();
   AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
   RegClassInfo.runOnMachineFunction(MF);
+  TargetPassConfig *PassConfig = &getAnalysis<TargetPassConfig>();
+  EnableSinkAndFold = PassConfig->getEnableSinkAndFold();
 
   bool EverMadeChange = false;
 
   while (true) {
     bool MadeChange = false;
 
     // Process all basic blocks.
     CEBCandidates.clear();
bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
   for (auto I : RegsToClearKillFlags)
     MRI->clearKillFlags(I);
   RegsToClearKillFlags.clear();
 
   return EverMadeChange;
 }
 
 bool MachineSinking::ProcessBlock(MachineBasicBlock &MBB) {
-  // Can't sink anything out of a block that has less than two successors.
-  if (MBB.succ_size() <= 1 || MBB.empty()) return false;
+  if ((!EnableSinkAndFold && MBB.succ_size() <= 1) || MBB.empty())
+    return false;
 
   // Don't bother sinking code out of unreachable blocks. In addition to being
   // unprofitable, it can also lead to infinite looping, because in an
   // unreachable cycle there may be nowhere to stop.
   if (!DT->isReachableFromEntry(&MBB)) return false;
 
   bool MadeChange = false;
 
if (!ProcessedBegin)
       --I;
 
     if (MI.isDebugOrPseudoInstr()) {
       if (MI.isDebugValue())
         ProcessDbgInst(MI);
       continue;
     }
 
-    bool Joined = PerformTrivialForwardCoalescing(MI, &MBB);
-    if (Joined) {
+    if (EnableSinkAndFold && PerformSinkAndFold(MI, &MBB)) {
+      MadeChange = true;
+      continue;
+    }
+
+    // Can't sink anything out of a block that has less than two successors.
+    if (MBB.succ_size() <= 1)
+      continue;
+
+    if (PerformTrivialForwardCoalescing(MI, &MBB)) {
       MadeChange = true;
       continue;
     }
 
     if (SinkInstruction(MI, SawStore, AllSuccessors)) {
       ++NumSunk;
       MadeChange = true;
     }
 
     // If we just processed the first instruction in the block, we're done.
   } while (!ProcessedBegin);
 
   SeenDbgUsers.clear();
   SeenDbgVars.clear();
   // recalculate the bb register pressure after sinking one BB.
   CachedRegisterPressure.clear();
-
   return MadeChange;
 }
 
 void MachineSinking::ProcessDbgInst(MachineInstr &MI) {
   // When we see DBG_VALUEs for registers, record any vreg it reads, so that
   // we know what to sink if the vreg def sinks.
   assert(MI.isDebugValue() && "Expected DBG_VALUE for processing");
 
bool MachineSinking::PostponeSplitCriticalEdge(MachineInstr &MI,
   }
 
   ToSplit.insert(std::make_pair(FromBB, ToBB));
 
   return true;
 }
 
 std::vector<unsigned> &
-MachineSinking::getBBRegisterPressure(MachineBasicBlock &MBB) {
+MachineSinking::getBBRegisterPressure(const MachineBasicBlock &MBB) {
   // Currently to save compiling time, MBB's register pressure will not change
   // in one ProcessBlock iteration because of CachedRegisterPressure. but MBB's
   // register pressure is changed after sinking any instructions into it.
   // FIXME: need a accurate and cheap register pressure estiminate model here.
   auto RP = CachedRegisterPressure.find(&MBB);
   if (RP != CachedRegisterPressure.end())
     return RP->second;
 
   RegionPressure Pressure;
   RegPressureTracker RPTracker(Pressure);
 
   // Initialize the register pressure tracker.
   RPTracker.init(MBB.getParent(), &RegClassInfo, nullptr, &MBB, MBB.end(),
                  /*TrackLaneMasks*/ false, /*TrackUntiedDefs=*/true);
 
-  for (MachineBasicBlock::iterator MII = MBB.instr_end(),
+  for (MachineBasicBlock::const_iterator MII = MBB.instr_end(),
                                          MIE = MBB.instr_begin();
        MII != MIE; --MII) {
-    MachineInstr &MI = *std::prev(MII);
+    const MachineInstr &MI = *std::prev(MII);
     if (MI.isDebugInstr() || MI.isPseudoProbe())
       continue;
     RegisterOperands RegOpers;
     RegOpers.collect(MI, *TRI, *MRI, false, false);
     RPTracker.recedeSkipDebugValues();
     assert(&*RPTracker.getPos() == &MI && "RPTracker sync error!");
     RPTracker.recede(RegOpers);
   }
 
   RPTracker.closeRegion();
   auto It = CachedRegisterPressure.insert(
       std::make_pair(&MBB, RPTracker.getPressure().MaxSetPressure));
   return It.first->second;
 }
 
+bool MachineSinking::registerPressureSetExceedsLimit(
+    unsigned NRegs, const TargetRegisterClass *RC,
+    const MachineBasicBlock &MBB) {
+  unsigned Weight = NRegs * TRI->getRegClassWeight(RC).RegWeight;
+  const int *PS = TRI->getRegClassPressureSets(RC);
+  std::vector<unsigned> BBRegisterPressure = getBBRegisterPressure(MBB);
+  for (; *PS != -1; PS++)
+    if (Weight + BBRegisterPressure[*PS] >=
+        TRI->getRegPressureSetLimit(*MBB.getParent(), *PS))
+      return true;
+  return false;
+}
+
 /// isProfitableToSinkTo - Return true if it is profitable to sink MI.
 bool MachineSinking::isProfitableToSinkTo(Register Reg, MachineInstr &MI,
                                           MachineBasicBlock *MBB,
                                           MachineBasicBlock *SuccToSinkTo,
                                           AllSuccsCache &AllSuccessors) {
   assert (SuccToSinkTo && "Invalid SinkTo Candidate BB");
 
   if (MBB == SuccToSinkTo)
bool MachineSinking::isProfitableToSinkTo(Register Reg, MachineInstr &MI,
 
   MachineCycle *MCycle = CI->getCycle(MBB);
 
   // If the instruction is not inside a cycle, it is not profitable to sink MI to
   // a post dominate block SuccToSinkTo.
   if (!MCycle)
     return false;
 
-  auto isRegisterPressureSetExceedLimit = [&](const TargetRegisterClass *RC) {
-    unsigned Weight = TRI->getRegClassWeight(RC).RegWeight;
-    const int *PS = TRI->getRegClassPressureSets(RC);
-    // Get register pressure for block SuccToSinkTo.
-    std::vector<unsigned> BBRegisterPressure =
-        getBBRegisterPressure(*SuccToSinkTo);
-    for (; *PS != -1; PS++)
-      // check if any register pressure set exceeds limit in block SuccToSinkTo
-      // after sinking.
-      if (Weight + BBRegisterPressure[*PS] >=
-          TRI->getRegPressureSetLimit(*MBB->getParent(), *PS))
-        return true;
-    return false;
-  };
-
   // If this instruction is inside a Cycle and sinking this instruction can make
   // more registers live range shorten, it is still prifitable.
   for (const MachineOperand &MO : MI.operands()) {
     // Ignore non-register operands.
     if (!MO.isReg())
       continue;
     Register Reg = MO.getReg();
     if (Reg == 0)
if (MO.isDef()) {
       // 1: defination is outside of cycle.
       // 2: defination is in this cycle, but it is a PHI in the cycle header.
       if (Cycle != MCycle || (DefMI->isPHI() && Cycle && Cycle->isReducible() &&
                               Cycle->getHeader() == DefMI->getParent()))
         continue;
       // The DefMI is defined inside the cycle.
       // If sinking this operand makes some register pressure set exceed limit,
       // it is not profitable.
-      if (isRegisterPressureSetExceedLimit(MRI->getRegClass(Reg))) {
+      if (registerPressureSetExceedsLimit(1, MRI->getRegClass(Reg),
+                                          *SuccToSinkTo)) {
         LLVM_DEBUG(dbgs() << "register pressure exceed limit, not profitable.");
         return false;
       }
     }
   }
 
   // If MI is in cycle and all its operands are alive across the whole cycle or
   // if no operand sinking make register pressure set exceed limit, it is

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

bool AArch64TargetLowering::isLegalAddressingMode(const DataLayout &DL,
   uint64_t NumBytes = 0;
   if (Ty->isSized()) {
     uint64_t NumBits = DL.getTypeSizeInBits(Ty);
     NumBytes = NumBits / 8;
     if (!isPowerOf2_64(NumBits))
       NumBytes = 0;
   }
 
-  if (!AM.Scale) {
-    int64_t Offset = AM.BaseOffs;
-
-    // 9-bit signed offset
-    if (isInt<9>(Offset))
-      return true;
-
-    // 12-bit unsigned offset
-    unsigned shift = Log2_64(NumBytes);
-    if (NumBytes && Offset > 0 && (Offset / NumBytes) <= (1LL << 12) - 1 &&
-        // Must be a multiple of NumBytes (NumBytes is a power of 2)
-        (Offset >> shift) << shift == Offset)
-      return true;
-    return false;
-  }
-
-  // Check reg1 + SIZE_IN_BYTES * reg2 and reg1 + reg2
-
-  return AM.Scale == 1 || (AM.Scale > 0 && (uint64_t)AM.Scale == NumBytes);
+  return Subtarget->getInstrInfo()->isLegalAddressingMode(NumBytes, AM.BaseOffs,
+                                                          AM.Scale);
 }
 
 bool AArch64TargetLowering::shouldConsiderGEPOffsetSplit() const {
   // Consider splitting large offset of struct or array.
   return true;
 }
 
 bool AArch64TargetLowering::isFMAFasterThanFMulAndFAdd(

llvm/lib/Target/AArch64/AArch64InstrInfo.h

public:
 
   /// Hint that pairing the given load or store is unprofitable.
   static void suppressLdStPair(MachineInstr &MI);
 
   std::optional<ExtAddrMode>
   getAddrModeFromMemoryOp(const MachineInstr &MemI,
                           const TargetRegisterInfo *TRI) const override;
 
+  bool canFoldIntoAddrMode(const MachineInstr &MemI, Register Reg,
+                           const MachineInstr &AddrI,
+                           ExtAddrMode &AM) const override;
+
+  MachineInstr *emitLdStWithAddr(MachineInstr &MemI,
+                                 const ExtAddrMode &AM) const override;
+
   bool getMemOperandsWithOffsetWidth(
       const MachineInstr &MI, SmallVectorImpl<const MachineOperand *> &BaseOps,
       int64_t &Offset, bool &OffsetIsScalable, unsigned &Width,
       const TargetRegisterInfo *TRI) const override;
 
   /// If \p OffsetIsScalable is set to 'true', the offset is scaled by `vscale`.
   /// This is true for some SVE instructions like ldr/str that have a
   /// 'reg + imm' addressing mode where the immediate is an index to the
static void decomposeStackOffsetForFrameOffsets(const StackOffset &Offset,
                                                   int64_t &NumBytes,
                                                   int64_t &NumPredicateVectors,
                                                   int64_t &NumDataVectors);
   static void decomposeStackOffsetForDwarfOffsets(const StackOffset &Offset,
                                                   int64_t &ByteSized,
                                                   int64_t &VGSized);
 
   bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const override;
+
+  // Return true if address of the form BaseReg + Scale * ScaledReg + Offset can
+  // be used for a load/store of NumBytes. BaseReg is always present and
+  // implicit.
+  bool isLegalAddressingMode(unsigned NumBytes, int64_t Offset,
+                             unsigned Scale) const;
+
 #define GET_INSTRINFO_HELPER_DECLS
 #include "AArch64GenInstrInfo.inc"
 
 protected:
   /// If the specific machine instruction is an instruction that moves/copies
   /// value from one register to another register return destination and source
   /// registers as machine operands.
   std::optional<DestSourcePair>

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

AArch64InstrInfo::getAddrModeFromMemoryOp(const MachineInstr &MemI,
   ExtAddrMode AM;
   AM.BaseReg = Base->getReg();
   AM.Displacement = Offset;
   AM.ScaledReg = 0;
   AM.Scale = 0;
   return AM;
 }
 
+bool AArch64InstrInfo::canFoldIntoAddrMode(const MachineInstr &MemI,
+                                           Register Reg,
+                                           const MachineInstr &AddrI,
+                                           ExtAddrMode &AM) const {
+  // Filter out instructions into which we cannot fold.
+  unsigned NumBytes;
+  int64_t OffsetScale = 1;
+  switch (MemI.getOpcode()) {
+  default:
+    return false;
+
+  case AArch64::LDURQi:
+  case AArch64::STURQi:
+    NumBytes = 16;
+    break;
+
+  case AArch64::LDURDi:
+  case AArch64::STURDi:
+  case AArch64::LDURXi:
+  case AArch64::STURXi:
+    NumBytes = 8;
+    break;
+
+  case AArch64::LDURWi:
+  case AArch64::LDURSWi:
+  case AArch64::STURWi:
+    NumBytes = 4;
+    break;
+
+  case AArch64::LDURHi:
+  case AArch64::STURHi:
+  case AArch64::LDURHHi:
+  case AArch64::STURHHi:
+  case AArch64::LDURSHXi:
+  case AArch64::LDURSHWi:
+    NumBytes = 2;
+    break;
+
+  case AArch64::LDRBroX:
+  case AArch64::LDRBBroX:
+  case AArch64::LDRSBXroX:
+  case AArch64::LDRSBWroX:
+  case AArch64::STRBroX:
+  case AArch64::STRBBroX:
+  case AArch64::LDURBi:
+  case AArch64::LDURBBi:
+  case AArch64::LDURSBXi:
+  case AArch64::LDURSBWi:
+  case AArch64::STURBi:
+  case AArch64::STURBBi:
+  case AArch64::LDRBui:
+  case AArch64::LDRBBui:
+  case AArch64::LDRSBXui:
+  case AArch64::LDRSBWui:
+  case AArch64::STRBui:
+  case AArch64::STRBBui:
+    NumBytes = 1;
+    break;
+
+  case AArch64::LDRQroX:
+  case AArch64::STRQroX:
+  case AArch64::LDRQui:
+  case AArch64::STRQui:
+    NumBytes = 16;
+    OffsetScale = 16;
+    break;
+
+  case AArch64::LDRDroX:
+  case AArch64::STRDroX:
+  case AArch64::LDRXroX:
+  case AArch64::STRXroX:
+  case AArch64::LDRDui:
+  case AArch64::STRDui:
+  case AArch64::LDRXui:
+  case AArch64::STRXui:
+    NumBytes = 8;
+    OffsetScale = 8;
+    break;
+
+  case AArch64::LDRWroX:
+  case AArch64::LDRSWroX:
+  case AArch64::STRWroX:
+  case AArch64::LDRWui:
+  case AArch64::LDRSWui:
+  case AArch64::STRWui:
+    NumBytes = 4;
+    OffsetScale = 4;
+    break;
+
+  case AArch64::LDRHroX:
+  case AArch64::STRHroX:
+  case AArch64::LDRHHroX:
+  case AArch64::STRHHroX:
+  case AArch64::LDRSHXroX:
+  case AArch64::LDRSHWroX:
+  case AArch64::LDRHui:
+  case AArch64::STRHui:
+  case AArch64::LDRHHui:
+  case AArch64::STRHHui:
+  case AArch64::LDRSHXui:
+  case AArch64::LDRSHWui:
+    NumBytes = 2;
+    OffsetScale = 2;
+    break;
+  }
+
+  // Check the fold operand is not the loaded/stored value.
+  const MachineOperand &BaseRegOp = MemI.getOperand(0);
+  if (BaseRegOp.isReg() && BaseRegOp.getReg() == Reg)
+    return false;
+
+  // Handle memory instructions with a [Reg, Reg] addressing mode.
+  if (MemI.getOperand(2).isReg()) {
+    // Bail if the addressing mode already includes extension of the offset
+    // register.
+    if (MemI.getOperand(3).getImm())
+      return false;
+
+    // Check if we actually have a scaled offset.
+    if (MemI.getOperand(4).getImm() == 0)
+      OffsetScale = 1;
+
+    // If the address instructions is folded into the base register, then the
+    // addressing mode must not have a scale. Then we can swap the base and the
+    // scaled registers.
+    if (MemI.getOperand(1).getReg() == Reg && OffsetScale != 1)
+      return false;
+
+    switch (AddrI.getOpcode()) {
+    default:
+      return false;
+
+    case AArch64::SBFMXri:
+      // sxtw Xa, Wm
+      // ldr Xd, [Xn, Xa, lsl #N]
+      // ->
+      // ldr Xd, [Xn, Wm, sxtw #N]
+      if (AddrI.getOperand(2).getImm() != 0 ||
+          AddrI.getOperand(3).getImm() != 31)
+        return false;
+
+      AM.BaseReg = MemI.getOperand(1).getReg();
+      if (AM.BaseReg == Reg)
+        AM.BaseReg = MemI.getOperand(2).getReg();
+      AM.ScaledReg = AddrI.getOperand(1).getReg();
+      AM.Scale = OffsetScale;
+      AM.Displacement = 0;
+      AM.Form = ExtAddrMode::Formula::SExtScaledReg;
+      return true;
+
+    case TargetOpcode::SUBREG_TO_REG: {
+      // mov Wa, Wm
+      // ldr Xd, [Xn, Xa, lsl #N]
+      // ->
+      // ldr Xd, [Xn, Wm, uxtw #N]
+
+      // Zero-extension looks like an ORRWrs followed by a SUBREG_TO_REG.
+      if (AddrI.getOperand(1).getImm() != 0 ||
+          AddrI.getOperand(3).getImm() != AArch64::sub_32)
+        return false;
+
+      const MachineRegisterInfo &MRI = AddrI.getMF()->getRegInfo();
+      Register OffsetReg = AddrI.getOperand(2).getReg();
+      if (!OffsetReg.isVirtual() || !MRI.hasOneNonDBGUse(OffsetReg))
+        return false;
+
+      const MachineInstr &DefMI = *MRI.getVRegDef(OffsetReg);
+      if (DefMI.getOpcode() != AArch64::ORRWrs ||
+          DefMI.getOperand(1).getReg() != AArch64::WZR ||
+          DefMI.getOperand(3).getImm() != 0)
+        return false;
+
+      AM.BaseReg = MemI.getOperand(1).getReg();
+      if (AM.BaseReg == Reg)
+        AM.BaseReg = MemI.getOperand(2).getReg();
+      AM.ScaledReg = DefMI.getOperand(2).getReg();
+      AM.Scale = OffsetScale;
+      AM.Displacement = 0;
+      AM.Form = ExtAddrMode::Formula::ZExtScaledReg;
+      return true;
+    }
+    }
+  }
+
+  // Handle memory instructions with a [Reg, #Imm] addressing mode.
+  auto canFoldAddSubImmIntoAddrMode = [&](int64_t Offset) -> bool {
+    Offset += MemI.getOperand(2).getImm() * OffsetScale;
+    if (!isLegalAddressingMode(NumBytes, Offset, /* Scale */ 0))
+      return false;
+    AM.BaseReg = AddrI.getOperand(1).getReg();
+    AM.ScaledReg = 0;
+    AM.Scale = 0;
+    AM.Displacement = Offset;
+    AM.Form = ExtAddrMode::Formula::Basic;
+    return true;
+  };
+
+  auto canFoldAddRegIntoAddrMode =
+      [&](int64_t Scale,
+          ExtAddrMode::Formula Form = ExtAddrMode::Formula::Basic) -> bool {
+    if (MemI.getOperand(2).getImm() != 0)
+      return false;
+    if (!isLegalAddressingMode(NumBytes, /* Offset */ 0, Scale))
+      return false;
+    AM.BaseReg = AddrI.getOperand(1).getReg();
+    AM.ScaledReg = AddrI.getOperand(2).getReg();
+    AM.Scale = Scale;
+    AM.Displacement = 0;
+    AM.Form = Form;
+    return true;
+  };
+
+  auto avoidSlowSTRQ = [&](const MachineInstr &MemI) {
+    unsigned Opcode = MemI.getOpcode();
+    return (Opcode == AArch64::STURQi || Opcode == AArch64::STRQui) &&
+           Subtarget.isSTRQroSlow();
+  };
+
+  int64_t Offset = 0;
+  const bool OptSize = MemI.getMF()->getFunction().hasOptSize();
+  switch (AddrI.getOpcode()) {
+  default:
+    return false;
+
+  case AArch64::ADDXri:
+    // add Xa, Xn, #N
+    // ldr Xd, [Xa, #M]
+    // ->
+    // ldr Xd, [Xn, #N'+M]
+    Offset = AddrI.getOperand(2).getImm() << AddrI.getOperand(3).getImm();
+    return canFoldAddSubImmIntoAddrMode(Offset);
+
+  case AArch64::SUBXri:
+    // sub Xa, Xn, #N
+    // ldr Xd, [Xa, #M]
+    // ->
+    // ldr Xd, [Xn, #N'+M]
+    Offset = AddrI.getOperand(2).getImm() << AddrI.getOperand(3).getImm();
+    return canFoldAddSubImmIntoAddrMode(-Offset);
+
+  case AArch64::ADDXrs: {
+    // add Xa, Xn, Xm, lsl #N
+    // ldr Xd, [Xa]
+    // ->
+    // ldr Xd, [Xn, Xm, lsl #N]
+
+    // Don't fold the add if the result would be slower, unless optimising for
+    // size.
+    int64_t Shift = AddrI.getOperand(3).getImm();
+    if (!OptSize) {
+      if ((Shift != 2 && Shift != 3) || !Subtarget.hasAddrLSLFast())
+        return false;
+      if (avoidSlowSTRQ(MemI))
+        return false;
+    }
+    return canFoldAddRegIntoAddrMode(1 << Shift);
+  }
+
+  case AArch64::ADDXrr:
+    // add Xa, Xn, Xm
+    // ldr Xd, [Xa]
+    // ->
+    // ldr Xd, [Xn, Xm, lsl #0]
+
+    // Don't fold the add if the result would be slower, unless optimising for
+    // size.
+    if (!OptSize && avoidSlowSTRQ(MemI))
+      return false;
+    return canFoldAddRegIntoAddrMode(1);
+
+  case AArch64::ADDXrx:
+    // add Xa, Xn, Wm, {s,u}xtw #N
+    // ldr Xd, [Xa]
+    // ->
+    // ldr Xd, [Xn, Wm, {s,u}xtw #N]
+
+    // Don't fold the add if the result would be slower, unless optimising for
+    // size.
+    if (!OptSize && avoidSlowSTRQ(MemI))
+      return false;
+
+    // Can fold only sign-/zero-extend of a word.
+    unsigned Imm = static_cast<unsigned>(AddrI.getOperand(3).getImm());
+    AArch64_AM::ShiftExtendType Extend = AArch64_AM::getArithExtendType(Imm);
+    if (Extend != AArch64_AM::UXTW && Extend != AArch64_AM::SXTW)
+      return false;
+
+    return canFoldAddRegIntoAddrMode(1 << AArch64_AM::getArithShiftValue(Imm),
+                                     (Extend == AArch64_AM::SXTW)
+                                         ? ExtAddrMode::Formula::SExtScaledReg
+                                         : ExtAddrMode::Formula::ZExtScaledReg);
+  }
+}
+
+// Given an opcode for an instruction with a [Reg, #Imm] addressing mode,
+// return the opcode of an instruction performing the same operation, but using
+// the [Reg, Reg] addressing mode.
+static unsigned regOffsetOpcode(unsigned Opcode) {
+  switch (Opcode) {
+  default:
+    llvm_unreachable("Address folding not implemented for instruction");
+
+  case AArch64::LDURQi:
+  case AArch64::LDRQui:
+    return AArch64::LDRQroX;
+  case AArch64::STURQi:
+  case AArch64::STRQui:
+    return AArch64::STRQroX;
+  case AArch64::LDURDi:
+  case AArch64::LDRDui:
+    return AArch64::LDRDroX;
+  case AArch64::STURDi:
+  case AArch64::STRDui:
+    return AArch64::STRDroX;
+  case AArch64::LDURXi:
+  case AArch64::LDRXui:
+    return AArch64::LDRXroX;
+  case AArch64::STURXi:
+  case AArch64::STRXui:
+    return AArch64::STRXroX;
+  case AArch64::LDURWi:
+  case AArch64::LDRWui:
+    return AArch64::LDRWroX;
+  case AArch64::LDURSWi:
+  case AArch64::LDRSWui:
+    return AArch64::LDRSWroX;
+  case AArch64::STURWi:
+  case AArch64::STRWui:
+    return AArch64::STRWroX;
+  case AArch64::LDURHi:
+  case AArch64::LDRHui:
+    return AArch64::LDRHroX;
+  case AArch64::STURHi:
+  case AArch64::STRHui:
+    return AArch64::STRHroX;
+  case AArch64::LDURHHi:
+  case AArch64::LDRHHui:
+    return AArch64::LDRHHroX;
+  case AArch64::STURHHi:
+  case AArch64::STRHHui:
+    return AArch64::STRHHroX;
+  case AArch64::LDURSHXi:
+  case AArch64::LDRSHXui:
+    return AArch64::LDRSHXroX;
+  case AArch64::LDURSHWi:
+  case AArch64::LDRSHWui:
+    return AArch64::LDRSHWroX;
+  case AArch64::LDURBi:
+  case AArch64::LDRBui:
+    return AArch64::LDRBroX;
+  case AArch64::LDURBBi:
+  case AArch64::LDRBBui:
+    return AArch64::LDRBBroX;
+  case AArch64::LDURSBXi:
+  case AArch64::LDRSBXui:
+    return AArch64::LDRSBXroX;
+  case AArch64::LDURSBWi:
+  case AArch64::LDRSBWui:
+    return AArch64::LDRSBWroX;
+  case AArch64::STURBi:
+  case AArch64::STRBui:
+    return AArch64::STRBroX;
+  case AArch64::STURBBi:
+  case AArch64::STRBBui:
+    return AArch64::STRBBroX;
+  }
+}
+
+// Given an opcode for an instruction with a [Reg, #Imm] addressing mode, return
+// the opcode of an instruction performing the same operation, but using the
+// [Reg, #Imm] addressing mode with scaled offset.
+unsigned scaledOffsetOpcode(unsigned Opcode, unsigned &Scale) {
+  switch (Opcode) {
+  default:
+    llvm_unreachable("Address folding not implemented for instruction");
+
+  case AArch64::LDURQi:
+    Scale = 16;
+    return AArch64::LDRQui;
+  case AArch64::STURQi:
+    Scale = 16;
+    return AArch64::STRQui;
+  case AArch64::LDURDi:
+    Scale = 8;
+    return AArch64::LDRDui;
+  case AArch64::STURDi:
+    Scale = 8;
+    return AArch64::STRDui;
+  case AArch64::LDURXi:
+    Scale = 8;
+    return AArch64::LDRXui;
+  case AArch64::STURXi:
+    Scale = 8;
+    return AArch64::STRXui;
+  case AArch64::LDURWi:
+    Scale = 4;
+    return AArch64::LDRWui;
+  case AArch64::LDURSWi:
+    Scale = 4;
+    return AArch64::LDRSWui;
+  case AArch64::STURWi:
+    Scale = 4;
+    return AArch64::STRWui;
+  case AArch64::LDURHi:
+    Scale = 2;
+    return AArch64::LDRHui;
+  case AArch64::STURHi:
+    Scale = 2;
+    return AArch64::STRHui;
+  case AArch64::LDURHHi:
+    Scale = 2;
+    return AArch64::LDRHHui;
+  case AArch64::STURHHi:
+    Scale = 2;
+    return AArch64::STRHHui;
+  case AArch64::LDURSHXi:
+    Scale = 2;
+    return AArch64::LDRSHXui;
+  case AArch64::LDURSHWi:
+    Scale = 2;
+    return AArch64::LDRSHWui;
+  case AArch64::LDURBi:
+    Scale = 1;
+    return AArch64::LDRBui;
+  case AArch64::LDURBBi:
+    Scale = 1;
+    return AArch64::LDRBBui;
+  case AArch64::LDURSBXi:
+    Scale = 1;
+    return AArch64::LDRSBXui;
+  case AArch64::LDURSBWi:
+    Scale = 1;
+    return AArch64::LDRSBWui;
+  case AArch64::STURBi:
+    Scale = 1;
+    return AArch64::STRBui;
+  case AArch64::STURBBi:
+    Scale = 1;
+    return AArch64::STRBBui;
+  case AArch64::LDRQui:
+  case AArch64::STRQui:
+    Scale = 16;
+    return Opcode;
+  case AArch64::LDRDui:
+  case AArch64::STRDui:
+  case AArch64::LDRXui:
+  case AArch64::STRXui:
+    Scale = 8;
+    return Opcode;
+  case AArch64::LDRWui:
+  case AArch64::LDRSWui:
+  case AArch64::STRWui:
+    Scale = 4;
+    return Opcode;
+  case AArch64::LDRHui:
+  case AArch64::STRHui:
+  case AArch64::LDRHHui:
+  case AArch64::STRHHui:
+  case AArch64::LDRSHXui:
+  case AArch64::LDRSHWui:
+    Scale = 2;
+    return Opcode;
+  case AArch64::LDRBui:
+  case AArch64::LDRBBui:
+  case AArch64::LDRSBXui:
+  case AArch64::LDRSBWui:
+  case AArch64::STRBui:
+  case AArch64::STRBBui:
+    Scale = 1;
+    return Opcode;
+  }
+}
+
+// Given an opcode for an instruction with a [Reg, #Imm] addressing mode, return
+// the opcode of an instruction performing the same operation, but using the
+// [Reg, #Imm] addressing mode with unscaled offset.
+unsigned unscaledOffsetOpcode(unsigned Opcode) {
+  switch (Opcode) {
+  default:
+    llvm_unreachable("Address folding not implemented for instruction");
+
+  case AArch64::LDURQi:
+  case AArch64::STURQi:
+  case AArch64::LDURDi:
+  case AArch64::STURDi:
+  case AArch64::LDURXi:
+  case AArch64::STURXi:
+  case AArch64::LDURWi:
+  case AArch64::LDURSWi:
+  case AArch64::STURWi:
+  case AArch64::LDURHi:
+  case AArch64::STURHi:
+  case AArch64::LDURHHi:
+  case AArch64::STURHHi:
+  case AArch64::LDURSHXi:
+  case AArch64::LDURSHWi:
+  case AArch64::LDURBi:
+  case AArch64::STURBi:
+  case AArch64::LDURBBi:
+  case AArch64::STURBBi:
+  case AArch64::LDURSBWi:
+  case AArch64::LDURSBXi:
+    return Opcode;
+  case AArch64::LDRQui:
+    return AArch64::LDURQi;
+  case AArch64::STRQui:
+    return AArch64::STURQi;
+  case AArch64::LDRDui:
+    return AArch64::LDURDi;
+  case AArch64::STRDui:
+    return AArch64::STURDi;
+  case AArch64::LDRXui:
+    return AArch64::LDURXi;
+  case AArch64::STRXui:
+    return AArch64::STURXi;
+  case AArch64::LDRWui:
+    return AArch64::LDURWi;
+  case AArch64::LDRSWui:
+    return AArch64::LDURSWi;
+  case AArch64::STRWui:
+    return AArch64::STURWi;
+  case AArch64::LDRHui:
+    return AArch64::LDURHi;
+  case AArch64::STRHui:
+    return AArch64::STURHi;
+  case AArch64::LDRHHui:
+    return AArch64::LDURHHi;
+  case AArch64::STRHHui:
+    return AArch64::STURHHi;
+  case AArch64::LDRSHXui:
+    return AArch64::LDURSHXi;
+  case AArch64::LDRSHWui:
+    return AArch64::LDURSHWi;
+  case AArch64::LDRBBui:
+    return AArch64::LDURBBi;
+  case AArch64::LDRBui:
+    return AArch64::LDURBi;
+  case AArch64::STRBBui:
+    return AArch64::STURBBi;
+  case AArch64::STRBui:
+    return AArch64::STURBi;
+  case AArch64::LDRSBWui:
+    return AArch64::LDURSBWi;
+  case AArch64::LDRSBXui:
+    return AArch64::LDURSBXi;
+  }
+}
+
+// Given the opcode of a memory load/store instruction, return the opcode of an
+// instruction performing the same operation, but using
+// the [Reg, Reg, {s,u}xtw #N] addressing mode with sign-/zero-extend of the
+// offset register.
+static unsigned offsetExtendOpcode(unsigned Opcode) {
+  switch (Opcode) {
+  default:
+    llvm_unreachable("Address folding not implemented for instruction");
+
+  case AArch64::LDRQroX:
+  case AArch64::LDURQi:
+  case AArch64::LDRQui:
+    return AArch64::LDRQroW;
+  case AArch64::STRQroX:
+  case AArch64::STURQi:
+  case AArch64::STRQui:
+    return AArch64::STRQroW;
+  case AArch64::LDRDroX:
+  case AArch64::LDURDi:
+  case AArch64::LDRDui:
+    return AArch64::LDRDroW;
+  case AArch64::STRDroX:
+  case AArch64::STURDi:
+  case AArch64::STRDui:
+    return AArch64::STRDroW;
+  case AArch64::LDRXroX:
+  case AArch64::LDURXi:
+  case AArch64::LDRXui:
+    return AArch64::LDRXroW;
+  case AArch64::STRXroX:
+  case AArch64::STURXi:
+  case AArch64::STRXui:
+    return AArch64::STRXroW;
+  case AArch64::LDRWroX:
+  case AArch64::LDURWi:
+  case AArch64::LDRWui:
+    return AArch64::LDRWroW;
+  case AArch64::LDRSWroX:
+  case AArch64::LDURSWi:
+  case AArch64::LDRSWui:
+    return AArch64::LDRSWroW;
+  case AArch64::STRWroX:
+  case AArch64::STURWi:
+  case AArch64::STRWui:
+    return AArch64::STRWroW;
+  case AArch64::LDRHroX:
+  case AArch64::LDURHi:
+  case AArch64::LDRHui:
+    return AArch64::LDRHroW;
+  case AArch64::STRHroX:
+  case AArch64::STURHi:
+  case AArch64::STRHui:
+    return AArch64::STRHroW;
+  case AArch64::LDRHHroX:
+  case AArch64::LDURHHi:
+  case AArch64::LDRHHui:
+    return AArch64::LDRHHroW;
+  case AArch64::STRHHroX:
+  case AArch64::STURHHi:
+  case AArch64::STRHHui:
+    return AArch64::STRHHroW;
+  case AArch64::LDRSHXroX:
+  case AArch64::LDURSHXi:
+  case AArch64::LDRSHXui:
+    return AArch64::LDRSHXroW;
+  case AArch64::LDRSHWroX:
+  case AArch64::LDURSHWi:
+  case AArch64::LDRSHWui:
+    return AArch64::LDRSHWroW;
+  case AArch64::LDRBroX:
+  case AArch64::LDURBi:
+  case AArch64::LDRBui:
+    return AArch64::LDRBroW;
+  case AArch64::LDRBBroX:
+  case AArch64::LDURBBi:
+  case AArch64::LDRBBui:
+    return AArch64::LDRBBroW;
+  case AArch64::LDRSBXroX:
+  case AArch64::LDURSBXi:
+  case AArch64::LDRSBXui:
+    return AArch64::LDRSBXroW;
+  case AArch64::LDRSBWroX:
+  case AArch64::LDURSBWi:
+  case AArch64::LDRSBWui:
+    return AArch64::LDRSBWroW;
+  case AArch64::STRBroX:
+  case AArch64::STURBi:
+  case AArch64::STRBui:
+    return AArch64::STRBroW;
+  case AArch64::STRBBroX:
+  case AArch64::STURBBi:
+  case AArch64::STRBBui:
+    return AArch64::STRBBroW;
+  }
+}
+
+MachineInstr *AArch64InstrInfo::emitLdStWithAddr(MachineInstr &MemI,
+                                                 const ExtAddrMode &AM) const {
+
+  const DebugLoc &DL = MemI.getDebugLoc();
+  MachineBasicBlock &MBB = *MemI.getParent();
+  MachineRegisterInfo &MRI = MemI.getMF()->getRegInfo();
+
+  if (AM.Form == ExtAddrMode::Formula::Basic) {
+    if (AM.ScaledReg) {
+      // The new instruction will be in the form `ldr Rt, [Xn, Xm, lsl #imm]`.
+      unsigned Opcode = regOffsetOpcode(MemI.getOpcode());
+      MRI.constrainRegClass(AM.BaseReg, &AArch64::GPR64spRegClass);
+      auto B = BuildMI(MBB, MemI, DL, get(Opcode))
+                   .addReg(MemI.getOperand(0).getReg(),
+                           MemI.mayLoad() ? RegState::Define : 0)
+                   .addReg(AM.BaseReg)
+                   .addReg(AM.ScaledReg)
+                   .addImm(0)
+                   .addImm(AM.Scale > 1)
+                   .setMemRefs(MemI.memoperands())
+                   .setMIFlags(MemI.getFlags());
+      return B.getInstr();
+    }
+
+    assert(AM.ScaledReg == 0 && AM.Scale == 0 &&
+           "Addressing mode not supported for folding");
+
+    // The new instruction will be in the form `ld[u]r Rt, [Xn, #imm]`.
+    unsigned Scale = 1;
+    unsigned Opcode = MemI.getOpcode();
+    if (isInt<9>(AM.Displacement))
+      Opcode = unscaledOffsetOpcode(Opcode);
+    else
+      Opcode = scaledOffsetOpcode(Opcode, Scale);
+
+    auto B = BuildMI(MBB, MemI, DL, get(Opcode))
+                 .addReg(MemI.getOperand(0).getReg(),
+                         MemI.mayLoad() ? RegState::Define : 0)
+                 .addReg(AM.BaseReg)
+                 .addImm(AM.Displacement / Scale)
+                 .setMemRefs(MemI.memoperands())
+                 .setMIFlags(MemI.getFlags());
+    return B.getInstr();
+  }
+
+  if (AM.Form == ExtAddrMode::Formula::SExtScaledReg ||
+      AM.Form == ExtAddrMode::Formula::ZExtScaledReg) {
+    // The new instruction will be in the form `ldr Rt, [Xn, Wm, {s,u}xtw #N]`.
+    assert(AM.ScaledReg && !AM.Displacement &&
+           "Address offset can be a register or an immediate, but not both");
+    unsigned Opcode = offsetExtendOpcode(MemI.getOpcode());
+    MRI.constrainRegClass(AM.BaseReg, &AArch64::GPR64spRegClass);
+    // Make sure the offset register is in the correct register class.
+    Register OffsetReg = AM.ScaledReg;
+    const TargetRegisterClass *RC = MRI.getRegClass(OffsetReg);
+    if (RC->hasSuperClassEq(&AArch64::GPR64RegClass)) {
+      OffsetReg = MRI.createVirtualRegister(&AArch64::GPR32RegClass);
+      BuildMI(MBB, MemI, DL, get(TargetOpcode::COPY), OffsetReg)
+          .addReg(AM.ScaledReg, 0, AArch64::sub_32);
+    }
+    auto B = BuildMI(MBB, MemI, DL, get(Opcode))
+                 .addReg(MemI.getOperand(0).getReg(),
+                         MemI.mayLoad() ? RegState::Define : 0)
+                 .addReg(AM.BaseReg)
+                 .addReg(OffsetReg)
+                 .addImm(AM.Form == ExtAddrMode::Formula::SExtScaledReg)
+                 .addImm(AM.Scale != 1)
+                 .setMemRefs(MemI.memoperands())
+                 .setMIFlags(MemI.getFlags());
+
+    return B.getInstr();
+  }
+
+  llvm_unreachable(
+      "Function must not be called with an addressing mode it can't handle");
+}
+
 bool AArch64InstrInfo::getMemOperandWithOffsetWidth(
     const MachineInstr &LdSt, const MachineOperand *&BaseOp, int64_t &Offset,
     bool &OffsetIsScalable, unsigned &Width,
     const TargetRegisterInfo *TRI) const {
   assert(LdSt.mayLoadOrStore() && "Expected a memory operation.");
   // Handle only loads/stores with base register followed by immediate offset.
   if (LdSt.getNumExplicitOperands() == 3) {
     // Non-paired instruction (e.g., ldr x1, [x0, #8]).
bool AArch64InstrInfo::isWhileOpcode(unsigned Opc) const {
   return get(Opc).TSFlags & AArch64::InstrFlagIsWhile;
 }
 
 unsigned int
 AArch64InstrInfo::getTailDuplicateSize(CodeGenOptLevel OptLevel) const {
   return OptLevel >= CodeGenOptLevel::Aggressive ? 6 : 2;
 }
 
+bool AArch64InstrInfo::isLegalAddressingMode(unsigned NumBytes, int64_t Offset,
+                                             unsigned Scale) const {
+  if (Offset && Scale)
+    return false;
+
+  // Check Reg + Imm
+  if (!Scale) {
+    // 9-bit signed offset
+    if (isInt<9>(Offset))
+      return true;
+
+    // 12-bit unsigned offset
+    unsigned Shift = Log2_64(NumBytes);
+    if (NumBytes && Offset > 0 && (Offset / NumBytes) <= (1LL << 12) - 1 &&
+        // Must be a multiple of NumBytes (NumBytes is a power of 2)
+        (Offset >> Shift) << Shift == Offset)
+      return true;
+    return false;
+  }
+
+  // Check reg1 + SIZE_IN_BYTES * reg2 and reg1 + reg2
+  return Scale == 1 || (Scale > 0 && Scale == NumBytes);
+}
+
 unsigned llvm::getBLRCallOpcode(const MachineFunction &MF) {
   if (MF.getSubtarget<AArch64Subtarget>().hardenSlsBlr())
     return AArch64::BLRNoIP;
   else
     return AArch64::BLR;
 }
 
 bool AArch64InstrInfo::isReallyTriviallyReMaterializable(

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

static cl::opt<bool> EnableGISelLoadStoreOptPreLegal(
     cl::desc("Enable GlobalISel's pre-legalizer load/store optimization pass"),
     cl::init(true), cl::Hidden);
 
 static cl::opt<bool> EnableGISelLoadStoreOptPostLegal(
     "aarch64-enable-gisel-ldst-postlegal",
     cl::desc("Enable GlobalISel's post-legalizer load/store optimization pass"),
     cl::init(false), cl::Hidden);
 
+static cl::opt<bool>
+    EnableSinkFold("aarch64-enable-sink-fold",
+                   cl::desc("Enable sinking and folding of instruction copies"),
+                   cl::init(false), cl::Hidden);
+
 extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAArch64Target() {
   // Register the target.
   RegisterTargetMachine<AArch64leTargetMachine> X(getTheAArch64leTarget());
   RegisterTargetMachine<AArch64beTargetMachine> Y(getTheAArch64beTarget());
   RegisterTargetMachine<AArch64leTargetMachine> Z(getTheARM64Target());
   RegisterTargetMachine<AArch64leTargetMachine> W(getTheARM64_32Target());
   RegisterTargetMachine<AArch64leTargetMachine> V(getTheAArch64_32Target());
   auto PR = PassRegistry::getPassRegistry();

 
 /// AArch64 Code Generator Pass Configuration Options.
 class AArch64PassConfig : public TargetPassConfig {
 public:
   AArch64PassConfig(AArch64TargetMachine &TM, PassManagerBase &PM)
       : TargetPassConfig(TM, PM) {
     if (TM.getOptLevel() != CodeGenOptLevel::None)
       substitutePass(&PostRASchedulerID, &PostMachineSchedulerID);
+    setEnableSinkAndFold(EnableSinkFold);
   }
 
   AArch64TargetMachine &getAArch64TargetMachine() const {
     return getTM<AArch64TargetMachine>();
   }
 
   ScheduleDAGInstrs *
   createMachineScheduler(MachineSchedContext *C) const override {

llvm/lib/Target/X86/X86InstrInfo.cpp

 
 bool X86InstrInfo::verifyInstruction(const MachineInstr &MI,
                                      StringRef &ErrInfo) const {
   std::optional<ExtAddrMode> AMOrNone = getAddrModeFromMemoryOp(MI, nullptr);
   if (!AMOrNone)
     return true;
 
   ExtAddrMode AM = *AMOrNone;
-
+  assert(AM.Form == ExtAddrMode::Formula::Basic);
   if (AM.ScaledReg != X86::NoRegister) {
     switch (AM.Scale) {
     case 1:
     case 2:
     case 4:
     case 8:
       break;
     default:

llvm/test/CodeGen/AArch64/addsub-shifted-reg-cheap-as-move.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
-; RUN: llc                  < %s -o - | FileCheck %s
-; RUN: llc -mattr=+alu-lsl-fast < %s -o - | FileCheck %s -check-prefix=LSLFAST
+; RUN: llc                                                       < %s -o - | FileCheck %s
+; RUN: llc -mattr=+alu-lsl-fast --aarch64-enable-sink-fold=false < %s -o - | FileCheck %s -check-prefix=LSLFAST
 target triple = "aarch64-linux"
 
 declare void @g(...)
 
 ; Check that ADDWrs/ADDXrs with shift > 4 is considered relatively
 ; slow, thus CSE-d.
 define void @f0(i1 %c0, i1 %c1, ptr %a, i64 %i) {
 ; CHECK-LABEL: f0:

llvm/test/CodeGen/AArch64/align-down.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu --aarch64-enable-sink-fold=true < %s | FileCheck %s
 
 ; Fold
 ;   ptr - (ptr & (alignment-1))
 ; To
 ;   ptr & (0 - alignment)
 ;
 ; This needs to be a backend-level fold because only by now pointers
 ; are just registers; in middle-end IR this can only be done via @llvm.ptrmask()

 }
 
 ; Extra use tests
 
 define i32 @t3_extrause0(i32 %ptr, i32 %alignment, i32* %mask_storage) nounwind {
 ; CHECK-LABEL: t3_extrause0:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    neg w8, w1
+; CHECK-NEXT:    sub w9, w1, #1
 ; CHECK-NEXT:    and w0, w0, w8
-; CHECK-NEXT:    sub w8, w1, #1
-; CHECK-NEXT:    str w8, [x2]
+; CHECK-NEXT:    str w9, [x2]
 ; CHECK-NEXT:    ret
   %mask = add i32 %alignment, -1
   store i32 %mask, i32* %mask_storage
   %bias = and i32 %ptr, %mask
   %r = sub i32 %ptr, %bias
   ret i32 %r
 }
 define i32 @n4_extrause1(i32 %ptr, i32 %alignment, i32* %bias_storage) nounwind {

llvm/test/CodeGen/AArch64/and-mask-removal.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-apple-darwin -aarch64-enable-collect-loh=false < %s  | FileCheck %s --check-prefixes=CHECK,CHECK-SD
-; RUN: llc -mtriple=aarch64-apple-darwin -aarch64-enable-collect-loh=false -global-isel < %s  | FileCheck %s --check-prefixes=CHECK,CHECK-GI
+; RUN: llc -mtriple=aarch64-apple-darwin -aarch64-enable-collect-loh=false -aarch64-enable-sink-fold=true < %s  | FileCheck %s --check-prefixes=CHECK,CHECK-SD
+; RUN: llc -mtriple=aarch64-apple-darwin -aarch64-enable-collect-loh=false -aarch64-enable-sink-fold=true -global-isel < %s  | FileCheck %s --check-prefixes=CHECK,CHECK-GI
 
 @board = common global [400 x i8] zeroinitializer, align 1
 @next_string = common global i32 0, align 4
 @string_number = common global [400 x i32] zeroinitializer, align 4
 
 ; Function Attrs: nounwind ssp
 define void @new_position(i32 %pos) {
 ; CHECK-SD-LABEL: new_position:
 ; CHECK-SD:       ; %bb.0: ; %entry
-; CHECK-SD-NEXT:    adrp x9, _board@GOTPAGE
+; CHECK-SD-NEXT:    adrp x8, _board@GOTPAGE
 ; CHECK-SD-NEXT:    ; kill: def $w0 killed $w0 def $x0
-; CHECK-SD-NEXT:    sxtw x8, w0
-; CHECK-SD-NEXT:    ldr x9, [x9, _board@GOTPAGEOFF]
-; CHECK-SD-NEXT:    ldrb w9, [x9, x8]
-; CHECK-SD-NEXT:    sub w9, w9, #1
-; CHECK-SD-NEXT:    cmp w9, #1
+; CHECK-SD-NEXT:    ldr x8, [x8, _board@GOTPAGEOFF]
+; CHECK-SD-NEXT:    ldrb w8, [x8, w0, sxtw]
+; CHECK-SD-NEXT:    sub w8, w8, #1
+; CHECK-SD-NEXT:    cmp w8, #1
 ; CHECK-SD-NEXT:    b.hi LBB0_2
 ; CHECK-SD-NEXT:  ; %bb.1: ; %if.then
-; CHECK-SD-NEXT:    adrp x9, _next_string@GOTPAGE
-; CHECK-SD-NEXT:    adrp x10, _string_number@GOTPAGE
-; CHECK-SD-NEXT:    ldr x9, [x9, _next_string@GOTPAGEOFF]
-; CHECK-SD-NEXT:    ldr x10, [x10, _string_number@GOTPAGEOFF]
-; CHECK-SD-NEXT:    ldr w9, [x9]
-; CHECK-SD-NEXT:    str w9, [x10, x8, lsl #2]
+; CHECK-SD-NEXT:    adrp x8, _next_string@GOTPAGE
+; CHECK-SD-NEXT:    adrp x9, _string_number@GOTPAGE
+; CHECK-SD-NEXT:    ldr x8, [x8, _next_string@GOTPAGEOFF]
+; CHECK-SD-NEXT:    ldr x9, [x9, _string_number@GOTPAGEOFF]
+; CHECK-SD-NEXT:    ldr w8, [x8]
+; CHECK-SD-NEXT:    str w8, [x9, w0, sxtw #2]
 ; CHECK-SD-NEXT:  LBB0_2: ; %if.end
 ; CHECK-SD-NEXT:    ret
 ;
 ; CHECK-GI-LABEL: new_position:
 ; CHECK-GI:       ; %bb.0: ; %entry
 ; CHECK-GI-NEXT:    adrp x8, _board@GOTPAGE
 ; CHECK-GI-NEXT:    ldr x8, [x8, _board@GOTPAGEOFF]
 ; CHECK-GI-NEXT:    ldrb w8, [x8, w0, sxtw]

llvm/test/CodeGen/AArch64/arm64-abi-varargs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -aarch64-load-store-renaming=true < %s -mtriple=arm64-apple-ios7.0.0 -mcpu=cyclone -enable-misched=false | FileCheck %s
+; RUN: llc -aarch64-load-store-renaming=true -aarch64-enable-sink-fold=true < %s -mtriple=arm64-apple-ios7.0.0 -mcpu=cyclone -enable-misched=false | FileCheck %s
 
 ; rdar://13625505
 ; Here we have 9 fixed integer arguments the 9th argument in on stack, the
 ; varargs start right after at 8-byte alignment.
 define void @fn9(ptr %a1, i32 %a2, i32 %a3, i32 %a4, i32 %a5, i32 %a6, i32 %a7, i32 %a8, i32 %a9, ...) nounwind noinline ssp {
 ; CHECK-LABEL: fn9:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #64

llvm/test/CodeGen/AArch64/arm64-extract-insert-varidx.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 3
-; RUN: llc < %s -verify-machineinstrs -mtriple=aarch64-none-linux-gnu -mattr=+neon -fp-contract=fast | FileCheck %s --check-prefix=CHECK-SDAG
-; RUN: llc < %s -global-isel -global-isel-abort=2 -verify-machineinstrs -mtriple=aarch64-none-linux-gnu -mattr=+neon -fp-contract=fast | FileCheck %s --check-prefix=CHECK-GISEL
+; RUN: llc < %s -verify-machineinstrs -mtriple=aarch64-none-linux-gnu -mattr=+neon -fp-contract=fast -aarch64-enable-sink-fold=true | FileCheck %s --check-prefix=CHECK-SDAG
+; RUN: llc < %s -global-isel -global-isel-abort=2 -verify-machineinstrs -mtriple=aarch64-none-linux-gnu -mattr=+neon -fp-contract=fast -aarch64-enable-sink-fold=true | FileCheck %s --check-prefix=CHECK-GISEL
 
 define <4 x i8> @test_varidx_extract_v8s8(<8 x i8> %x, i32 %idx) {
 ; CHECK-SDAG-LABEL: test_varidx_extract_v8s8:
 ; CHECK-SDAG:       // %bb.0:
 ; CHECK-SDAG-NEXT:    sub sp, sp, #16
 ; CHECK-SDAG-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-SDAG-NEXT:    add x8, sp, #8
 ; CHECK-SDAG-NEXT:    // kill: def $w0 killed $w0 def $x0

 ; CHECK-SDAG-NEXT:    add sp, sp, #16
 ; CHECK-SDAG-NEXT:    ret
 ;
 ; CHECK-GISEL-LABEL: test_varidx_extract_v8s8:
 ; CHECK-GISEL:       // %bb.0:
 ; CHECK-GISEL-NEXT:    sub sp, sp, #16
 ; CHECK-GISEL-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-GISEL-NEXT:    mov w9, w0
-; CHECK-GISEL-NEXT:    add x8, sp, #8
 ; CHECK-GISEL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-GISEL-NEXT:    add x8, sp, #8
 ; CHECK-GISEL-NEXT:    str d0, [sp, #8]
 ; CHECK-GISEL-NEXT:    and x9, x9, #0x7
 ; CHECK-GISEL-NEXT:    mov b2, v0.b[1]
 ; CHECK-GISEL-NEXT:    mov b3, v0.b[2]
 ; CHECK-GISEL-NEXT:    lsl x10, x9, #1
 ; CHECK-GISEL-NEXT:    mov b0, v0.b[3]
 ; CHECK-GISEL-NEXT:    sub x9, x10, x9
-; CHECK-GISEL-NEXT:    add x8, x8, x9
-; CHECK-GISEL-NEXT:    ldrb w8, [x8]
+; CHECK-GISEL-NEXT:    ldrb w8, [x8, x9]
 ; CHECK-GISEL-NEXT:    fmov s1, w8
 ; CHECK-GISEL-NEXT:    mov v1.h[1], v2.h[0]
 ; CHECK-GISEL-NEXT:    mov v1.h[2], v3.h[0]
 ; CHECK-GISEL-NEXT:    mov v1.h[3], v0.h[0]
 ; CHECK-GISEL-NEXT:    fmov d0, d1
 ; CHECK-GISEL-NEXT:    add sp, sp, #16
 ; CHECK-GISEL-NEXT:    ret
   %tmp = extractelement <8 x i8> %x, i32 %idx

llvm/test/CodeGen/AArch64/arm64-long-shift.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=arm64-eabi -mcpu=cyclone | FileCheck %s
+; RUN: llc < %s -mtriple=arm64-eabi -mcpu=cyclone -aarch64-enable-sink-fold=true | FileCheck %s
 
 define i128 @shl(i128 %r, i128 %s) nounwind readnone {
 ; CHECK-LABEL: shl:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    lsl x8, x1, x2
 ; CHECK-NEXT:    mvn w9, w2
 ; CHECK-NEXT:    lsr x10, x0, #1
 ; CHECK-NEXT:    lsr x9, x10, x9

 define i128 @shl_mask(i128 %r, i128 %s) nounwind readnone {
 ; CHECK-LABEL: shl_mask:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    lsl x8, x1, x2
 ; CHECK-NEXT:    lsr x9, x0, #1
 ; CHECK-NEXT:    and x10, x2, #0x3f
 ; CHECK-NEXT:    eor x10, x10, #0x3f
 ; CHECK-NEXT:    lsr x9, x9, x10
-; CHECK-NEXT:    orr x1, x8, x9
 ; CHECK-NEXT:    lsl x0, x0, x2
+; CHECK-NEXT:    orr x1, x8, x9
 ; CHECK-NEXT:    ret
   %mask = and i128 %s, 63
   %shl = shl i128 %r, %mask
   ret i128 %shl
 }
 
 define i128 @ashr(i128 %r, i128 %s) nounwind readnone {
 ; CHECK-LABEL: ashr:

 define i128 @ashr_mask(i128 %r, i128 %s) nounwind readnone {
 ; CHECK-LABEL: ashr_mask:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    lsr x8, x0, x2
 ; CHECK-NEXT:    lsl x9, x1, #1
 ; CHECK-NEXT:    and x10, x2, #0x3f
 ; CHECK-NEXT:    eor x10, x10, #0x3f
 ; CHECK-NEXT:    lsl x9, x9, x10
-; CHECK-NEXT:    orr x0, x9, x8
 ; CHECK-NEXT:    asr x1, x1, x2
+; CHECK-NEXT:    orr x0, x9, x8
 ; CHECK-NEXT:    ret
   %mask = and i128 %s, 63
   %shr = ashr i128 %r, %mask
   ret i128 %shr
 }
 
 define i128 @lshr(i128 %r, i128 %s) nounwind readnone {
 ; CHECK-LABEL: lshr:

 define i128 @lshr_mask(i128 %r, i128 %s) nounwind readnone {
 ; CHECK-LABEL: lshr_mask:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    lsr x8, x0, x2
 ; CHECK-NEXT:    lsl x9, x1, #1
 ; CHECK-NEXT:    and x10, x2, #0x3f
 ; CHECK-NEXT:    eor x10, x10, #0x3f
 ; CHECK-NEXT:    lsl x9, x9, x10
-; CHECK-NEXT:    orr x0, x9, x8
 ; CHECK-NEXT:    lsr x1, x1, x2
+; CHECK-NEXT:    orr x0, x9, x8
 ; CHECK-NEXT:    ret
   %mask = and i128 %s, 63
   %shr = lshr i128 %r, %mask
   ret i128 %shr
 }

llvm/test/CodeGen/AArch64/arm64-stp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
-; RUN: llc < %s -mtriple=arm64-eabi -aarch64-enable-stp-suppress=false -verify-machineinstrs -mcpu=cyclone | FileCheck %s
+; RUN: llc < %s -mtriple=arm64-eabi -aarch64-enable-stp-suppress=false -verify-machineinstrs -mcpu=cyclone -aarch64-enable-sink-fold=true | FileCheck %s
 
 define void @stp_int(i32 %a, i32 %b, ptr nocapture %p) nounwind {
 ; CHECK-LABEL: stp_int:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    stp w0, w1, [x2]
 ; CHECK-NEXT:    ret
   store i32 %a, ptr %p, align 4
   %add.ptr = getelementptr inbounds i32, ptr %p, i64 1
entry:
   ret void
 }
 
 ; Read of %b to compute %tmp2 shouldn't prevent formation of stp
 define i32 @stp_int_rar_hazard(i32 %a, i32 %b, ptr nocapture %p) nounwind {
 ; CHECK-LABEL: stp_int_rar_hazard:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr w8, [x2, #8]
-; CHECK-NEXT:    add w8, w8, w1
 ; CHECK-NEXT:    stp w0, w1, [x2]
-; CHECK-NEXT:    mov x0, x8
+; CHECK-NEXT:    add w0, w8, w1
 ; CHECK-NEXT:    ret
   store i32 %a, ptr %p, align 4
   %ld.ptr = getelementptr inbounds i32, ptr %p, i64 2
   %tmp = load i32, ptr %ld.ptr, align 4
   %tmp2 = add i32 %tmp, %b
   %add.ptr = getelementptr inbounds i32, ptr %p, i64 1
   store i32 %b, ptr %add.ptr, align 4
   ret i32 %tmp2
 }
 
 ; Read of %b to compute %tmp2 shouldn't prevent formation of stp
 define i32 @stp_int_rar_hazard_after(i32 %w0, i32 %a, i32 %b, ptr nocapture %p) nounwind {
 ; CHECK-LABEL: stp_int_rar_hazard_after:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr w8, [x3, #4]
-; CHECK-NEXT:    add w0, w8, w2
 ; CHECK-NEXT:    stp w1, w2, [x3]
+; CHECK-NEXT:    add w0, w8, w2
 ; CHECK-NEXT:    ret
   store i32 %a, ptr %p, align 4
   %ld.ptr = getelementptr inbounds i32, ptr %p, i64 1
   %tmp = load i32, ptr %ld.ptr, align 4
   %tmp2 = add i32 %tmp, %b
   %add.ptr = getelementptr inbounds i32, ptr %p, i64 1
   store i32 %b, ptr %add.ptr, align 4
   ret i32 %tmp2
 }

llvm/test/CodeGen/AArch64/arm64_32-addrs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=arm64_32-apple-ios %s -o - | FileCheck %s
+; RUN: llc -mtriple=arm64_32-apple-ios %s -aarch64-enable-sink-fold=true -o - | FileCheck %s
 
 ; If %base < 96 then the sum will not wrap (in an unsigned sense), but "ldr w0,
 ; [x0, #-96]" would.
 define i32 @test_valid_wrap(i32 %base) {
 ; CHECK-LABEL: test_valid_wrap:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    sub w8, w0, #96
 ; CHECK-NEXT:    ldr w0, [x8]
; CHECK-NEXT:    ret
   %val = load i8, ptr %newaddr
   ret i8 %val
 }
 
 ;
 define i8 @test_valid_wrap_optimizable2(ptr %base, i32 %offset) {
 ; CHECK-LABEL: test_valid_wrap_optimizable2:
 ; CHECK:       ; %bb.0:
+; CHECK-NEXT:    mov w8, #-100 ; =0xffffff9c
 ; CHECK-NEXT:    ; kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    sxtw x8, w1
-; CHECK-NEXT:    mov w9, #-100 ; =0xffffff9c
-; CHECK-NEXT:    ldrb w0, [x8, x9]
+; CHECK-NEXT:    ldrb w0, [x8, w1, sxtw]
 ; CHECK-NEXT:    ret
 
   %newaddr = getelementptr inbounds i8, ptr inttoptr(i32 -100 to ptr), i32 %offset
   %val = load i8, ptr %newaddr
   ret i8 %val
 }

llvm/test/CodeGen/AArch64/atomic-ops-lse.ll

-; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+lse < %s | FileCheck %s
-; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+lse -mattr=+outline-atomics < %s | FileCheck %s
-; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+outline-atomics < %s | FileCheck %s --check-prefix=OUTLINE-ATOMICS
-; RUN: llc -mtriple=aarch64_be-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+lse < %s | FileCheck %s
-; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+lse < %s | FileCheck %s --check-prefix=CHECK-REG
+; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+lse -aarch64-enable-sink-fold=true < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+lse -mattr=+outline-atomics -aarch64-enable-sink-fold=true < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+outline-atomics -aarch64-enable-sink-fold=true < %s | FileCheck %s --check-prefix=OUTLINE-ATOMICS
+; RUN: llc -mtriple=aarch64_be-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+lse -aarch64-enable-sink-fold=true < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+lse -aarch64-enable-sink-fold=true < %s | FileCheck %s --check-prefix=CHECK-REG
 
 ; Point of CHECK-REG is to make sure UNPREDICTABLE instructions aren't created
 ; (i.e. reusing a register for status & data in store exclusive).
 ; CHECK-REG-NOT: stlxrb w[[NEW:[0-9]+]], w[[NEW]], [x{{[0-9]+}}]
 ; CHECK-REG-NOT: stlxrb w[[NEW:[0-9]+]], x[[NEW]], [x{{[0-9]+}}]
 
 @var8 = dso_local global i8 0
 @var16 = dso_local global i16 0
; CHECK-NOT: dmb
   ret i64 %old
 }
 
 define dso_local i8 @test_atomic_load_and_i8(i8 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i8:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i8:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var8
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var8
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr1_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var8, i8 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var8
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var8
 
 ; CHECK: ldclralb w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i8 %old
 }
 
 define dso_local i16 @test_atomic_load_and_i16(i16 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i16:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i16:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var16
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var16
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr2_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var16, i16 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var16
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var16
 
 ; CHECK: ldclralh w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i16 %old
 }
 
 define dso_local i32 @test_atomic_load_and_i32(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var32, i32 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclral w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i32 %old
 }
 
 define dso_local i64 @test_atomic_load_and_i64(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var64, i64 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
; CHECK-NOT: dmb
   ret i64 %old
 }
 
 define dso_local void @test_atomic_load_and_i32_noret(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_noret:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_noret:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var32, i32 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclral w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local void @test_atomic_load_and_i64_noret(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_noret:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_noret:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var64, i64 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local i8 @test_atomic_load_and_i8_acq_rel(i8 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i8_acq_rel:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i8_acq_rel:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var8
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var8
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr1_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var8, i8 %offset acq_rel
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var8
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var8
 
 ; CHECK: ldclralb w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i8 %old
 }
 
 define dso_local i16 @test_atomic_load_and_i16_acq_rel(i16 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i16_acq_rel:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i16_acq_rel:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var16
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var16
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr2_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var16, i16 %offset acq_rel
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var16
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var16
 
 ; CHECK: ldclralh w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i16 %old
 }
 
 define dso_local i32 @test_atomic_load_and_i32_acq_rel(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_acq_rel:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_acq_rel:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var32, i32 %offset acq_rel
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclral w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i32 %old
 }
 
 define dso_local i64 @test_atomic_load_and_i64_acq_rel(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_acq_rel:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_acq_rel:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var64, i64 %offset acq_rel
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclral x[[NOT]], x[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i64 %old
 }
 
 define dso_local void @test_atomic_load_and_i32_noret_acq_rel(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_noret_acq_rel:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_noret_acq_rel:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var32, i32 %offset acq_rel
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclral w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local void @test_atomic_load_and_i64_noret_acq_rel(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_noret_acq_rel:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_noret_acq_rel:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var64, i64 %offset acq_rel
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclral x[[NOT]], x[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local i8 @test_atomic_load_and_i8_acquire(i8 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i8_acquire:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i8_acquire:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var8
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var8
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr1_acq
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var8, i8 %offset acquire
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var8
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var8
 
 ; CHECK: ldclrab w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i8 %old
 }
 
 define dso_local i16 @test_atomic_load_and_i16_acquire(i16 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i16_acquire:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i16_acquire:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var16
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var16
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr2_acq
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var16, i16 %offset acquire
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var16
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var16
 
 ; CHECK: ldclrah w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i16 %old
 }
 
 define dso_local i32 @test_atomic_load_and_i32_acquire(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_acquire:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_acquire:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_acq
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var32, i32 %offset acquire
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclra w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i32 %old
 }
 
 define dso_local i64 @test_atomic_load_and_i64_acquire(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_acquire:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_acquire:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_acq
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var64, i64 %offset acquire
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclra x[[NOT]], x[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i64 %old
 }
 
 define dso_local void @test_atomic_load_and_i32_noret_acquire(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_noret_acquire:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_noret_acquire:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_acq
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var32, i32 %offset acquire
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclra w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local void @test_atomic_load_and_i64_noret_acquire(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_noret_acquire:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_noret_acquire:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_acq
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var64, i64 %offset acquire
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclra x[[NOT]], x[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local i8 @test_atomic_load_and_i8_monotonic(i8 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i8_monotonic:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i8_monotonic:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var8
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var8
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr1_relax
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var8, i8 %offset monotonic
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var8
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var8
 
 ; CHECK: ldclrb w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i8 %old
 }
 
 define dso_local i16 @test_atomic_load_and_i16_monotonic(i16 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i16_monotonic:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i16_monotonic:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var16
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var16
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr2_relax
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var16, i16 %offset monotonic
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var16
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var16
 
 ; CHECK: ldclrh w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i16 %old
 }
 
 define dso_local i32 @test_atomic_load_and_i32_monotonic(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_monotonic:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_monotonic:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_relax
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var32, i32 %offset monotonic
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclr w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i32 %old
 }
 
 define dso_local i64 @test_atomic_load_and_i64_monotonic(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_monotonic:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_monotonic:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_relax
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var64, i64 %offset monotonic
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclr x[[NOT]], x[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i64 %old
 }
 
 define dso_local void @test_atomic_load_and_i32_noret_monotonic(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_noret_monotonic:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_noret_monotonic:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_relax
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var32, i32 %offset monotonic
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclr w{{[0-9]+}}, w[[NEW:[1-9][0-9]*]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local void @test_atomic_load_and_i64_noret_monotonic(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_noret_monotonic:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_noret_monotonic:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_relax
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var64, i64 %offset monotonic
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclr x{{[0-9]+}}, x[[NEW:[1-9][0-9]*]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local i8 @test_atomic_load_and_i8_release(i8 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i8_release:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i8_release:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var8
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var8
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr1_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var8, i8 %offset release
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var8
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var8
 
 ; CHECK: ldclrlb w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i8 %old
 }
 
 define dso_local i16 @test_atomic_load_and_i16_release(i16 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i16_release:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i16_release:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var16
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var16
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr2_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var16, i16 %offset release
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var16
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var16
 
 ; CHECK: ldclrlh w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i16 %old
 }
 
 define dso_local i32 @test_atomic_load_and_i32_release(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_release:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_release:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var32, i32 %offset release
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclrl w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i32 %old
 }
 
 define dso_local i64 @test_atomic_load_and_i64_release(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_release:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_release:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var64, i64 %offset release
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclrl x[[NOT]], x[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i64 %old
 }
 
 define dso_local void @test_atomic_load_and_i32_noret_release(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_noret_release:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_noret_release:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var32, i32 %offset release
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclrl w{{[0-9]*}}, w[[NEW:[1-9][0-9]*]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local void @test_atomic_load_and_i64_noret_release(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_noret_release:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_noret_release:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var64, i64 %offset release
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclrl x{{[0-9]*}}, x[[NEW:[1-9][0-9]*]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local i8 @test_atomic_load_and_i8_seq_cst(i8 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i8_seq_cst:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i8_seq_cst:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var8
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var8
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr1_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var8, i8 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var8
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var8
 
 ; CHECK: ldclralb w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i8 %old
 }
 
 define dso_local i16 @test_atomic_load_and_i16_seq_cst(i16 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i16_seq_cst:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i16_seq_cst:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var16
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var16
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr2_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var16, i16 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var16
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var16
 
 ; CHECK: ldclralh w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i16 %old
 }
 
 define dso_local i32 @test_atomic_load_and_i32_seq_cst(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_seq_cst:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_seq_cst:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var32, i32 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclral w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i32 %old
 }
 
 define dso_local i64 @test_atomic_load_and_i64_seq_cst(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_seq_cst:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_seq_cst:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   %old = atomicrmw and ptr @var64, i64 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64
 
 ; CHECK: ldclral x[[NOT]], x[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret i64 %old
 }
 
 define dso_local void @test_atomic_load_and_i32_noret_seq_cst(i32 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i32_noret_seq_cst:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i32_noret_seq_cst:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE-ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr4_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var32, i32 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn w[[NOT:[0-9]+]], w[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var32
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var32
 
 ; CHECK: ldclral w[[NOT]], w[[NEW:[0-9]+]], [x[[ADDR]]]
 ; CHECK-NOT: dmb
   ret void
 }
 
 define dso_local void @test_atomic_load_and_i64_noret_seq_cst(i64 %offset) nounwind {
 ; CHECK-LABEL: test_atomic_load_and_i64_noret_seq_cst:
 ; OUTLINE-ATOMICS-LABEL: test_atomic_load_and_i64_noret_seq_cst:
 ; OUTLINE-ATOMICS:       // %bb.0:
 ; OUTLINE-ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE-ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE-ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE-ATOMICS-NEXT:    bl __aarch64_ldclr8_acq_rel
 ; OUTLINE-ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ret
   atomicrmw and ptr @var64, i64 %offset seq_cst
 ; CHECK-NOT: dmb
 ; CHECK: mvn x[[NOT:[0-9]+]], x[[OLD:[0-9]+]]
 ; CHECK: adrp [[TMPADDR:x[0-9]+]], var64
 ; CHECK: add x[[ADDR:[0-9]+]], [[TMPADDR]], {{#?}}:lo12:var64

llvm/test/CodeGen/AArch64/atomic-ops.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,INLINE_ATOMICS
-; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+outline-atomics < %s | FileCheck %s --check-prefixes=CHECK,OUTLINE_ATOMICS
+; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -aarch64-enable-sink-fold=true < %s | FileCheck %s --check-prefixes=CHECK,INLINE_ATOMICS
+; RUN: llc -mtriple=aarch64-none-linux-gnu -disable-post-ra -verify-machineinstrs -mattr=+outline-atomics -aarch64-enable-sink-fold=true < %s | FileCheck %s --check-prefixes=CHECK,OUTLINE_ATOMICS
 
 @var8 = dso_local global i8 0
 @var16 = dso_local global i16 0
 @var32 = dso_local global i32 0
 @var64 = dso_local global i64 0
 
 define dso_local i8 @test_atomic_load_add_i8(i8 %offset) nounwind {
 ; INLINE_ATOMICS-LABEL: test_atomic_load_add_i8:

 ; INLINE_ATOMICS-NEXT:    cbnz w11, .LBB8_1
 ; INLINE_ATOMICS-NEXT:  // %bb.2: // %atomicrmw.end
 ; INLINE_ATOMICS-NEXT:    mov w0, w8
 ; INLINE_ATOMICS-NEXT:    ret
 ;
 ; OUTLINE_ATOMICS-LABEL: test_atomic_load_and_i8:
 ; OUTLINE_ATOMICS:       // %bb.0:
 ; OUTLINE_ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE_ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE_ATOMICS-NEXT:    adrp x1, var8
 ; OUTLINE_ATOMICS-NEXT:    add x1, x1, :lo12:var8
+; OUTLINE_ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE_ATOMICS-NEXT:    bl __aarch64_ldclr1_rel
 ; OUTLINE_ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE_ATOMICS-NEXT:    ret
    %old = atomicrmw and ptr @var8, i8 %offset release
    ret i8 %old
 }
 
 define dso_local i16 @test_atomic_load_and_i16(i16 %offset) nounwind {

 ; INLINE_ATOMICS-NEXT:    cbnz w11, .LBB9_1
 ; INLINE_ATOMICS-NEXT:  // %bb.2: // %atomicrmw.end
 ; INLINE_ATOMICS-NEXT:    mov w0, w8
 ; INLINE_ATOMICS-NEXT:    ret
 ;
 ; OUTLINE_ATOMICS-LABEL: test_atomic_load_and_i16:
 ; OUTLINE_ATOMICS:       // %bb.0:
 ; OUTLINE_ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE_ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE_ATOMICS-NEXT:    adrp x1, var16
 ; OUTLINE_ATOMICS-NEXT:    add x1, x1, :lo12:var16
+; OUTLINE_ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE_ATOMICS-NEXT:    bl __aarch64_ldclr2_relax
 ; OUTLINE_ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE_ATOMICS-NEXT:    ret
    %old = atomicrmw and ptr @var16, i16 %offset monotonic
    ret i16 %old
 }
 
 define dso_local i32 @test_atomic_load_and_i32(i32 %offset) nounwind {

 ; INLINE_ATOMICS-NEXT:    cbnz w11, .LBB10_1
 ; INLINE_ATOMICS-NEXT:  // %bb.2: // %atomicrmw.end
 ; INLINE_ATOMICS-NEXT:    mov w0, w8
 ; INLINE_ATOMICS-NEXT:    ret
 ;
 ; OUTLINE_ATOMICS-LABEL: test_atomic_load_and_i32:
 ; OUTLINE_ATOMICS:       // %bb.0:
 ; OUTLINE_ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE_ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE_ATOMICS-NEXT:    adrp x1, var32
 ; OUTLINE_ATOMICS-NEXT:    add x1, x1, :lo12:var32
+; OUTLINE_ATOMICS-NEXT:    mvn w0, w0
 ; OUTLINE_ATOMICS-NEXT:    bl __aarch64_ldclr4_acq_rel
 ; OUTLINE_ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE_ATOMICS-NEXT:    ret
    %old = atomicrmw and ptr @var32, i32 %offset seq_cst
    ret i32 %old
 }
 
 define dso_local i64 @test_atomic_load_and_i64(i64 %offset) nounwind {

 ; INLINE_ATOMICS-NEXT:    cbnz w11, .LBB11_1
 ; INLINE_ATOMICS-NEXT:  // %bb.2: // %atomicrmw.end
 ; INLINE_ATOMICS-NEXT:    mov x0, x8
 ; INLINE_ATOMICS-NEXT:    ret
 ;
 ; OUTLINE_ATOMICS-LABEL: test_atomic_load_and_i64:
 ; OUTLINE_ATOMICS:       // %bb.0:
 ; OUTLINE_ATOMICS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; OUTLINE_ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE_ATOMICS-NEXT:    adrp x1, var64
 ; OUTLINE_ATOMICS-NEXT:    add x1, x1, :lo12:var64
+; OUTLINE_ATOMICS-NEXT:    mvn x0, x0
 ; OUTLINE_ATOMICS-NEXT:    bl __aarch64_ldclr8_acq
 ; OUTLINE_ATOMICS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; OUTLINE_ATOMICS-NEXT:    ret
    %old = atomicrmw and ptr @var64, i64 %offset acquire
    ret i64 %old
 }
 
 define dso_local i8 @test_atomic_load_or_i8(i8 %offset) nounwind {

llvm/test/CodeGen/AArch64/cmp-select-sign.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=arm64-apple-iphoneos -o - %s | FileCheck %s
+; RUN: llc -mtriple=arm64-apple-iphoneos -aarch64-enable-sink-fold=true -o - %s | FileCheck %s
 
 define i3 @sign_i3(i3 %a) {
 ; CHECK-LABEL: sign_i3:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sbfx w8, w0, #2, #1
 ; CHECK-NEXT:    orr w0, w8, #0x1
 ; CHECK-NEXT:    ret
   %c = icmp sgt i3 %a, -1
; CHECK-NEXT:    ret
   ret <4 x i32> %res
 }
 
 ; i65 is not legal.
 define <4 x i65> @sign_4xi65(<4 x i65> %a) {
 ; CHECK-LABEL: sign_4xi65:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sbfx x8, x1, #0, #1
-; CHECK-NEXT:    sbfx x9, x3, #0, #1
-; CHECK-NEXT:    sbfx x10, x7, #0, #1
+; CHECK-NEXT:    sbfx x9, x5, #0, #1
+; CHECK-NEXT:    sbfx x10, x3, #0, #1
 ; CHECK-NEXT:    lsr x1, x8, #63
 ; CHECK-NEXT:    orr x8, x8, #0x1
-; CHECK-NEXT:    lsr x3, x9, #63
+; CHECK-NEXT:    lsr x3, x10, #63
 ; CHECK-NEXT:    fmov d0, x8
-; CHECK-NEXT:    sbfx x8, x5, #0, #1
-; CHECK-NEXT:    lsr x7, x10, #63
-; CHECK-NEXT:    orr x2, x9, #0x1
-; CHECK-NEXT:    orr x6, x10, #0x1
-; CHECK-NEXT:    lsr x5, x8, #63
-; CHECK-NEXT:    orr x4, x8, #0x1
+; CHECK-NEXT:    sbfx x8, x7, #0, #1
+; CHECK-NEXT:    lsr x5, x9, #63
+; CHECK-NEXT:    orr x2, x10, #0x1
+; CHECK-NEXT:    orr x4, x9, #0x1
+; CHECK-NEXT:    lsr x7, x8, #63
+; CHECK-NEXT:    orr x6, x8, #0x1
 ; CHECK-NEXT:    mov v0.d[1], x1
 ; CHECK-NEXT:    fmov x0, d0
 ; CHECK-NEXT:    ret
   %c = icmp sgt <4 x i65> %a, <i65 -1, i65 -1, i65 -1, i65 -1>
   %res = select <4 x i1> %c, <4 x i65> <i65 1, i65 1, i65 1, i65 1>, <4 x i65 > <i65 -1, i65 -1, i65 -1, i65 -1>
   ret <4 x i65> %res
 }
 
 declare void @use_4xi1(<4 x i1>)

llvm/test/CodeGen/AArch64/cmpxchg-idioms.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-apple-ios7.0 -o - %s | FileCheck %s
-; RUN: llc -mtriple=aarch64-apple-ios7.0 -mattr=+outline-atomics -o - %s | FileCheck %s --check-prefix=OUTLINE-ATOMICS
+; RUN: llc -mtriple=aarch64-apple-ios7.0 -aarch64-enable-sink-fold=true -o - %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-apple-ios7.0 -mattr=+outline-atomics -aarch64-enable-sink-fold=true -o - %s | FileCheck %s --check-prefix=OUTLINE-ATOMICS
 
 define i32 @test_return(ptr %p, i32 %oldval, i32 %newval) {
 ; CHECK-LABEL: test_return:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:  LBB0_1: ; %cmpxchg.start
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldaxr w8, [x0]
 ; CHECK-NEXT:    cmp w8, w1

 ; CHECK-NEXT:    ; in Loop: Header=BB3_6 Depth=1
 ; CHECK-NEXT:    sub w22, w22, #1
 ; CHECK-NEXT:    orr w9, w21, w20
 ; CHECK-NEXT:    ldr w10, [x19, w22, sxtw #2]
 ; CHECK-NEXT:    cmp w9, w10
 ; CHECK-NEXT:    b.eq LBB3_6
 ; CHECK-NEXT:  ; %bb.8: ; %if.then
 ; CHECK-NEXT:    ; in Loop: Header=BB3_6 Depth=1
-; CHECK-NEXT:    sxtw x8, w22
-; CHECK-NEXT:    str w9, [x19, x8, lsl #2]
+; CHECK-NEXT:    str w9, [x19, w22, sxtw #2]
 ; CHECK-NEXT:    bl _foo
 ; CHECK-NEXT:    mov w8, wzr
 ; CHECK-NEXT:    b LBB3_6
 ; CHECK-NEXT:  LBB3_9: ; %for.cond.cleanup
 ; CHECK-NEXT:    ldp x29, x30, [sp, #32] ; 16-byte Folded Reload
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ldp x20, x19, [sp, #16] ; 16-byte Folded Reload
 ; CHECK-NEXT:    ldp x22, x21, [sp], #48 ; 16-byte Folded Reload

 ; OUTLINE-ATOMICS-NEXT:    ; in Loop: Header=BB3_1 Depth=1
 ; OUTLINE-ATOMICS-NEXT:    sub w22, w22, #1
 ; OUTLINE-ATOMICS-NEXT:    orr w9, w21, w20
 ; OUTLINE-ATOMICS-NEXT:    ldr w10, [x19, w22, sxtw #2]
 ; OUTLINE-ATOMICS-NEXT:    cmp w9, w10
 ; OUTLINE-ATOMICS-NEXT:    b.eq LBB3_1
 ; OUTLINE-ATOMICS-NEXT:  ; %bb.3: ; %if.then
 ; OUTLINE-ATOMICS-NEXT:    ; in Loop: Header=BB3_1 Depth=1
-; OUTLINE-ATOMICS-NEXT:    sxtw x8, w22
-; OUTLINE-ATOMICS-NEXT:    str w9, [x19, x8, lsl #2]
+; OUTLINE-ATOMICS-NEXT:    str w9, [x19, w22, sxtw #2]
 ; OUTLINE-ATOMICS-NEXT:    bl _foo
 ; OUTLINE-ATOMICS-NEXT:    mov w8, wzr
 ; OUTLINE-ATOMICS-NEXT:    b LBB3_1
 ; OUTLINE-ATOMICS-NEXT:  LBB3_4: ; %for.cond.cleanup
 ; OUTLINE-ATOMICS-NEXT:    ldp x29, x30, [sp, #32] ; 16-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    and w0, w8, #0x1
 ; OUTLINE-ATOMICS-NEXT:    ldp x20, x19, [sp, #16] ; 16-byte Folded Reload
 ; OUTLINE-ATOMICS-NEXT:    ldp x22, x21, [sp], #48 ; 16-byte Folded Reload

llvm/test/CodeGen/AArch64/loop-sink.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-# RUN: llc -mtriple aarch64 -run-pass=machine-sink  -sink-insts-to-avoid-spills %s -o - 2>&1 | FileCheck %s
+# RUN: llc -mtriple aarch64 -run-pass=machine-sink  -sink-insts-to-avoid-spills -aarch64-enable-sink-fold=true %s -o - 2>&1 | FileCheck %s
 --- |
   target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64"
 
   %struct.A = type { i32, i32, i32, i32, i32, i32 }
 
   @A = external dso_local global [100 x i32], align 4
 

 body:             |
   ; CHECK-LABEL: name: cant_sink_adds_call_in_block
   ; CHECK: bb.0.bb:
   ; CHECK-NEXT:   successors: %bb.1(0x80000000)
   ; CHECK-NEXT:   liveins: $x0, $x1
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:gpr64common = COPY $x1
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:gpr64common = COPY $x0
-  ; CHECK-NEXT:   [[ADDXri:%[0-9]+]]:gpr64sp = nuw ADDXri [[COPY]], 4, 0
+  ; CHECK-NEXT:   [[ADDXri:%[0-9]+]]:gpr64sp = ADDXri [[COPY1]], 1, 0
   ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:gpr64all = COPY [[ADDXri]]
-  ; CHECK-NEXT:   [[ADDXri1:%[0-9]+]]:gpr64sp = nuw ADDXri [[COPY]], 8, 0
-  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gpr64all = COPY [[ADDXri1]]
-  ; CHECK-NEXT:   [[ADDXri2:%[0-9]+]]:gpr64sp = nuw ADDXri [[COPY]], 12, 0
-  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gpr64all = COPY [[ADDXri2]]
-  ; CHECK-NEXT:   [[ADDXri3:%[0-9]+]]:gpr64sp = nuw ADDXri [[COPY]], 16, 0
-  ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:gpr64all = COPY [[ADDXri3]]
-  ; CHECK-NEXT:   [[ADDXri4:%[0-9]+]]:gpr64sp = nuw ADDXri [[COPY]], 20, 0
-  ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:gpr64all = COPY [[ADDXri4]]
-  ; CHECK-NEXT:   [[ADDXri5:%[0-9]+]]:gpr64sp = ADDXri [[COPY1]], 1, 0
-  ; CHECK-NEXT:   [[COPY7:%[0-9]+]]:gpr64all = COPY [[ADDXri5]]
   ; CHECK-NEXT:   [[MOVaddrJT:%[0-9]+]]:gpr64common = MOVaddrJT target-flags(aarch64-page) %jump-table.0, target-flags(aarch64-pageoff, aarch64-nc) %jump-table.0
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.1..backedge:
   ; CHECK-NEXT:   successors: %bb.9(0x09249249), %bb.2(0x76db6db7)
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[PHI:%[0-9]+]]:gpr64sp = PHI [[COPY7]], %bb.0, %7, %bb.9
+  ; CHECK-NEXT:   [[PHI:%[0-9]+]]:gpr64sp = PHI [[COPY2]], %bb.0, %7, %bb.9
   ; CHECK-NEXT:   [[LDRBBui:%[0-9]+]]:gpr32 = LDRBBui [[PHI]], 0 :: (load (s8) from %ir.lsr.iv)
   ; CHECK-NEXT:   [[SUBREG_TO_REG:%[0-9]+]]:gpr64 = SUBREG_TO_REG 0, killed [[LDRBBui]], %subreg.sub_32
-  ; CHECK-NEXT:   [[COPY8:%[0-9]+]]:gpr32sp = COPY [[SUBREG_TO_REG]].sub_32
-  ; CHECK-NEXT:   [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri killed [[COPY8]], 50, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gpr32sp = COPY [[SUBREG_TO_REG]].sub_32
+  ; CHECK-NEXT:   [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri killed [[COPY3]], 50, 0, implicit-def $nzcv
   ; CHECK-NEXT:   Bcc 8, %bb.9, implicit $nzcv
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2..backedge:
   ; CHECK-NEXT:   successors: %bb.3(0x13b13b14), %bb.9(0x09d89d8a), %bb.4(0x13b13b14), %bb.5(0x13b13b14), %bb.6(0x13b13b14), %bb.7(0x13b13b14), %bb.8(0x13b13b14)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   early-clobber %21:gpr64, early-clobber %22:gpr64sp = JumpTableDest32 [[MOVaddrJT]], [[SUBREG_TO_REG]], %jump-table.0
   ; CHECK-NEXT:   BR killed %21
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.3.bb7:
   ; CHECK-NEXT:   successors: %bb.9(0x80000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
   ; CHECK-NEXT:   $x0 = COPY [[COPY]]
   ; CHECK-NEXT:   BL @_Z6assignPj, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit $x0, implicit-def $sp
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
   ; CHECK-NEXT:   B %bb.9
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.4.bb9:
   ; CHECK-NEXT:   successors: %bb.9(0x80000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
-  ; CHECK-NEXT:   $x0 = COPY [[COPY2]]
+  ; CHECK-NEXT:   $x0 = nuw ADDXri [[COPY]], 4, 0
   ; CHECK-NEXT:   BL @_Z6assignPj, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit $x0, implicit-def $sp
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
   ; CHECK-NEXT:   B %bb.9
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.5.bb10:
   ; CHECK-NEXT:   successors: %bb.9(0x80000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
-  ; CHECK-NEXT:   $x0 = COPY [[COPY3]]
+  ; CHECK-NEXT:   $x0 = nuw ADDXri [[COPY]], 8, 0
   ; CHECK-NEXT:   BL @_Z6assignPj, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit $x0, implicit-def $sp
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
   ; CHECK-NEXT:   B %bb.9
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.6.bb11:
   ; CHECK-NEXT:   successors: %bb.9(0x80000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
-  ; CHECK-NEXT:   $x0 = COPY [[COPY4]]
+  ; CHECK-NEXT:   $x0 = nuw ADDXri [[COPY]], 12, 0
   ; CHECK-NEXT:   BL @_Z6assignPj, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit $x0, implicit-def $sp
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
   ; CHECK-NEXT:   B %bb.9
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.7.bb12:
   ; CHECK-NEXT:   successors: %bb.9(0x80000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
-  ; CHECK-NEXT:   $x0 = COPY [[COPY5]]
+  ; CHECK-NEXT:   $x0 = nuw ADDXri [[COPY]], 16, 0
   ; CHECK-NEXT:   BL @_Z6assignPj, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit $x0, implicit-def $sp
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
   ; CHECK-NEXT:   B %bb.9
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.8.bb13:
   ; CHECK-NEXT:   successors: %bb.9(0x80000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
-  ; CHECK-NEXT:   $x0 = COPY [[COPY6]]
+  ; CHECK-NEXT:   $x0 = nuw ADDXri [[COPY]], 20, 0
   ; CHECK-NEXT:   BL @_Z6assignPj, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit $x0, implicit-def $sp
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.9..backedge.backedge:
   ; CHECK-NEXT:   successors: %bb.1(0x80000000)
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[ADDXri6:%[0-9]+]]:gpr64sp = ADDXri [[PHI]], 1, 0
-  ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:gpr64all = COPY [[ADDXri6]]
+  ; CHECK-NEXT:   [[ADDXri1:%[0-9]+]]:gpr64sp = ADDXri [[PHI]], 1, 0
+  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gpr64all = COPY [[ADDXri1]]
   ; CHECK-NEXT:   B %bb.1
   bb.0 (%ir-block.bb):
     successors: %bb.1(0x80000000)
     liveins: $x0, $x1
 
     %9:gpr64common = COPY $x1
     %8:gpr64common = COPY $x0
     %10:gpr64sp = nuw ADDXri %9, 4, 0

llvm/test/CodeGen/AArch64/nontemporal-load.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc --mattr=+sve < %s -mtriple aarch64-apple-darwin | FileCheck %s
-; RUN: llc --mattr=+sve < %s -mtriple aarch64_be-unknown-unknown | FileCheck --check-prefix CHECK-BE %s
+; RUN: llc --mattr=+sve -aarch64-enable-sink-fold=true < %s -mtriple aarch64-apple-darwin | FileCheck %s
+; RUN: llc --mattr=+sve -aarch64-enable-sink-fold=true < %s -mtriple aarch64_be-unknown-unknown | FileCheck --check-prefix CHECK-BE %s
 
 define <4 x double> @test_ldnp_v4f64(ptr %A) {
 ; CHECK-LABEL: test_ldnp_v4f64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    ldnp q0, q1, [x0]
 ; CHECK-NEXT:    ret
 ;
 ; CHECK-BE-LABEL: test_ldnp_v4f64:

 ; CHECK-BE-NEXT:    ret
   %lv = load <4 x i65>, ptr %A, align 8, !nontemporal !0
   ret <4 x i65> %lv
 }
 
 define <4 x i63> @test_ldnp_v4i63(ptr %A) {
 ; CHECK-LABEL: test_ldnp_v4i63:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    ldp x8, x9, [x0]
-; CHECK-NEXT:    ldp x10, x11, [x0, #16]
-; CHECK-NEXT:    extr x12, x9, x8, #63
-; CHECK-NEXT:    and x0, x8, #0x7fffffffffffffff
-; CHECK-NEXT:    extr x9, x10, x9, #62
-; CHECK-NEXT:    extr x3, x11, x10, #61
-; CHECK-NEXT:    and x1, x12, #0x7fffffffffffffff
-; CHECK-NEXT:    and x2, x9, #0x7fffffffffffffff
+; CHECK-NEXT:    ldp x8, x9, [x0, #16]
+; CHECK-NEXT:    ldp x10, x11, [x0]
+; CHECK-NEXT:    extr x3, x9, x8, #61
+; CHECK-NEXT:    extr x9, x11, x10, #63
+; CHECK-NEXT:    extr x8, x8, x11, #62
+; CHECK-NEXT:    and x0, x10, #0x7fffffffffffffff
+; CHECK-NEXT:    and x1, x9, #0x7fffffffffffffff
+; CHECK-NEXT:    and x2, x8, #0x7fffffffffffffff
 ; CHECK-NEXT:    ret
 ;
 ; CHECK-BE-LABEL: test_ldnp_v4i63:
 ; CHECK-BE:       // %bb.0:
 ; CHECK-BE-NEXT:    ldp x9, x8, [x0, #8]
 ; CHECK-BE-NEXT:    ldr x11, [x0, #24]
 ; CHECK-BE-NEXT:    ldr x10, [x0]
 ; CHECK-BE-NEXT:    and x3, x11, #0x7fffffffffffffff

llvm/test/CodeGen/AArch64/optimize-imm.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -o - %s -mtriple=aarch64-- | FileCheck %s
+; RUN: llc -o - %s -mtriple=aarch64-- -aarch64-enable-sink-fold=true | FileCheck %s
 
 define void @and1(i32 %a, ptr nocapture %p) {
 ; CHECK-LABEL: and1:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    and w8, w0, #0xfffffffd
 ; CHECK-NEXT:    strb w8, [x1]
 ; CHECK-NEXT:    ret
 entry:

 ; Check that, when (and %t1, 129) is transformed to (and %t0, 0),
 ; (xor %arg, 129) doesn't get transformed to (xor %arg, 0).
 define i64 @PR33100(i64 %arg) {
 ; CHECK-LABEL: PR33100:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    sub sp, sp, #16
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    mov w8, #129 // =0x81
+; CHECK-NEXT:    mov w9, #8 // =0x8
 ; CHECK-NEXT:    eor x0, x0, x8
-; CHECK-NEXT:    mov w8, #8 // =0x8
-; CHECK-NEXT:    str x8, [sp, #8]
+; CHECK-NEXT:    str x9, [sp, #8]
 ; CHECK-NEXT:    add sp, sp, #16
 ; CHECK-NEXT:    ret
 entry:
   %alloca0 = alloca i64
   store i64 8, ptr %alloca0, align 4
   %t0 = load i64, ptr %alloca0, align 4
   %t1 = shl i64 %arg, %t0
   %and0 = and i64 %t1, 129
   %xor0 = xor i64 %arg, 129
   %t2 = add i64 %and0, %xor0
   ret i64 %t2
 }

llvm/test/CodeGen/AArch64/overeager_mla_fusing.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc %s --mtriple aarch64 -verify-machineinstrs -o - | FileCheck %s
+; RUN: llc %s --mtriple aarch64 -verify-machineinstrs -aarch64-enable-sink-fold=true -o - | FileCheck %s
 
 define dso_local void @jsimd_idct_ifast_neon_intrinsic(ptr nocapture readonly %dct_table, ptr nocapture readonly %coef_block, ptr nocapture readonly %output_buf, i32 %output_col) local_unnamed_addr #0 {
 ; CHECK-LABEL: jsimd_idct_ifast_neon_intrinsic:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ldr q0, [x1, #32]
 ; CHECK-NEXT:    ldr q1, [x1, #96]
-; CHECK-NEXT:    mov w9, w3
 ; CHECK-NEXT:    ldr q2, [x0, #32]
 ; CHECK-NEXT:    ldr q3, [x0, #96]
 ; CHECK-NEXT:    ldr x8, [x2, #48]
 ; CHECK-NEXT:    mul v0.8h, v2.8h, v0.8h
 ; CHECK-NEXT:    mul v1.8h, v3.8h, v1.8h
 ; CHECK-NEXT:    add v2.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    sub v0.8h, v0.8h, v1.8h
-; CHECK-NEXT:    str q2, [x8, x9]
+; CHECK-NEXT:    str q2, [x8, w3, uxtw]
 ; CHECK-NEXT:    ldr x8, [x2, #56]
-; CHECK-NEXT:    str q0, [x8, x9]
+; CHECK-NEXT:    str q0, [x8, w3, uxtw]
 ; CHECK-NEXT:    ret
 entry:
   %add.ptr5 = getelementptr inbounds i16, ptr %coef_block, i64 16
   %0 = load <8 x i16>, ptr %add.ptr5, align 16
 
   %add.ptr17 = getelementptr inbounds i16, ptr %coef_block, i64 48
   %1 = load <8 x i16>, ptr %add.ptr17, align 16
 

llvm/test/CodeGen/AArch64/rand.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64 -mattr=+v8.5a,+rand %s -o - | FileCheck %s
+; RUN: llc -mtriple=aarch64 -mattr=+v8.5a,+rand -aarch64-enable-sink-fold=true %s -o - | FileCheck %s
 
 define  i32 @rndr(ptr %__addr) {
 ; CHECK-LABEL: rndr:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mrs x10, RNDR
-; CHECK-NEXT:    mov x9, x0
-; CHECK-NEXT:    cset w8, eq
-; CHECK-NEXT:    str x10, [x9]
-; CHECK-NEXT:    and w8, w8, #0x1
-; CHECK-NEXT:    mov w0, w8
+; CHECK-NEXT:    mrs x9, RNDR
+; CHECK-NEXT:    mov x8, x0
+; CHECK-NEXT:    cset w10, eq
+; CHECK-NEXT:    str x9, [x8]
+; CHECK-NEXT:    and w0, w10, #0x1
 ; CHECK-NEXT:    ret
   %1 = tail call { i64, i1 } @llvm.aarch64.rndr()
   %2 = extractvalue { i64, i1 } %1, 0
   %3 = extractvalue { i64, i1 } %1, 1
   store i64 %2, ptr %__addr, align 8
   %4 = zext i1 %3 to i32
   ret i32 %4
 }
 
 
 define  i32 @rndrrs(ptr  %__addr) {
 ; CHECK-LABEL: rndrrs:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mrs x10, RNDRRS
-; CHECK-NEXT:    mov x9, x0
-; CHECK-NEXT:    cset w8, eq
-; CHECK-NEXT:    str x10, [x9]
-; CHECK-NEXT:    and w8, w8, #0x1
-; CHECK-NEXT:    mov w0, w8
+; CHECK-NEXT:    mrs x9, RNDRRS
+; CHECK-NEXT:    mov x8, x0
+; CHECK-NEXT:    cset w10, eq
+; CHECK-NEXT:    str x9, [x8]
+; CHECK-NEXT:    and w0, w10, #0x1
 ; CHECK-NEXT:    ret
   %1 = tail call { i64, i1 } @llvm.aarch64.rndrrs()
   %2 = extractvalue { i64, i1 } %1, 0
   %3 = extractvalue { i64, i1 } %1, 1
   store i64 %2, ptr %__addr, align 8
   %4 = zext i1 %3 to i32
   ret i32 %4
 }
 
 declare { i64, i1 } @llvm.aarch64.rndr()
 declare { i64, i1 } @llvm.aarch64.rndrrs()

llvm/test/CodeGen/AArch64/shrink-constant-multiple-users.ll

-; RUN: llc -mtriple arm64-ios- %s -o - | FileCheck %s
+; RUN: llc -mtriple arm64-ios- -aarch64-enable-sink-fold=true  %s -o - | FileCheck %s
 
 ; Check the -8 constant is shrunk if there are multiple users of the AND instruction.
 
 ; CHECK-LABEL:  _test:
-; CHECK:          and x0, x0, #0xfffffff8
-; CHECK-NEXT:     add x19, x0, #10
+; CHECK:          and x19, x0, #0xfffffff8
+; CHECK-NEXT:     mov x0, x19
 ; CHECK-NEXT:     bl  _user
+; CHECK:          add x0, x19, #10
 
 define i64 @test(i32 %a) {
   %ext = zext i32 %a to i64
   %v1 = and i64 %ext, -8
   %v2 = add i64 %v1, 10
   call void @user(i64 %v1)
   ret i64 %v2
 }
 
 declare void @user(i64)

llvm/test/CodeGen/AArch64/sink-and-fold.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s | FileCheck %s
+; RUN: llc -aarch64-enable-sink-fold=true < %s | FileCheck %s
 target triple = "aarch64-linux"
 
 declare i32 @use(...)
 
 define i32 @f0(i1 %c1, ptr %p) nounwind {
 ; CHECK-LABEL: f0:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    mov w8, w0
-; CHECK-NEXT:    add x0, x1, #8
-; CHECK-NEXT:    tbz w8, #0, .LBB0_2
+; CHECK-NEXT:    tbz w0, #0, .LBB0_2
 ; CHECK-NEXT:  // %bb.1: // %if.then
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    add x0, x1, #8
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB0_2: // %if.else
-; CHECK-NEXT:    ldr w0, [x0]
+; CHECK-NEXT:    ldur w0, [x1, #8]
 ; CHECK-NEXT:    ret
 entry:
   %a = getelementptr i32, ptr %p, i32 2
   br i1 %c1, label %if.then, label %if.else
 
 if.then:
   %v0 = call i32 @use(ptr %a)
   br label %exit
 
 if.else:
   %v1 = load i32, ptr %a
   br label %exit
 
 exit:
   %v = phi i32 [%v0, %if.then], [%v1, %if.else]
   ret i32 %v
 }
 
 define i32 @f1(i1 %c1, ptr %p, i64 %i) nounwind {
 ; CHECK-LABEL: f1:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    mov w8, w0
-; CHECK-NEXT:    add x0, x1, x2
-; CHECK-NEXT:    tbz w8, #0, .LBB1_2
+; CHECK-NEXT:    tbz w0, #0, .LBB1_2
 ; CHECK-NEXT:  // %bb.1: // %if.then
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    add x0, x1, x2
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB1_2: // %if.else
-; CHECK-NEXT:    ldr w0, [x0]
+; CHECK-NEXT:    ldr w0, [x1, x2]
 ; CHECK-NEXT:    ret
 entry:
   %a = getelementptr i8, ptr %p, i64 %i
   br i1 %c1, label %if.then, label %if.else
 
 if.then:
   %v0 = call i32 @use(ptr %a)
   br label %exit
if.else:
   br label %exit
 
 exit:
   %v = phi i32 [%v0, %if.then], [%v1, %if.else]
   ret i32 %v
 }
 
 ; Address calculation cheap enough on some cores.
-define i32 @f3(i1 %c1, ptr %p, i64 %i) nounwind  "target-features"="+alu-lsl-fast" {
+define i32 @f3(i1 %c1, ptr %p, i64 %i) nounwind  "target-features"="+alu-lsl-fast,+addr-lsl-fast" {
 ; CHECK-LABEL: f3:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    mov w8, w0
-; CHECK-NEXT:    add x0, x1, x2, lsl #2
-; CHECK-NEXT:    tbz w8, #0, .LBB3_2
+; CHECK-NEXT:    tbz w0, #0, .LBB3_2
 ; CHECK-NEXT:  // %bb.1: // %if.then
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    add x0, x1, x2, lsl #2
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB3_2: // %if.else
-; CHECK-NEXT:    ldr w0, [x0]
+; CHECK-NEXT:    ldr w0, [x1, x2, lsl #2]
 ; CHECK-NEXT:    ret
 entry:
   %a = getelementptr i32, ptr %p, i64 %i
   br i1 %c1, label %if.then, label %if.else
 
 if.then:
   %v0 = call i32 @use(ptr %a)
   br label %exit
 
 if.else:
   %v1 = load i32, ptr %a
   br label %exit
 
 exit:
   %v = phi i32 [%v0, %if.then], [%v1, %if.else]
   ret i32 %v
 }
 
 define void @f4(ptr %a, i64 %n) nounwind "target-features"="+alu-lsl-fast,+addr-lsl-fast" {
 ; CHECK-LABEL: f4:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    cmp x1, #1
 ; CHECK-NEXT:    b.lt .LBB4_9
 ; CHECK-NEXT:  // %bb.1: // %LI.preheader
-; CHECK-NEXT:    str x30, [sp, #-64]! // 8-byte Folded Spill
-; CHECK-NEXT:    stp x24, x23, [sp, #16] // 16-byte Folded Spill
-; CHECK-NEXT:    mov x23, xzr
-; CHECK-NEXT:    stp x20, x19, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x23, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x22, x21, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    mov x22, xzr
+; CHECK-NEXT:    stp x20, x19, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov x19, x1
 ; CHECK-NEXT:    mov x20, x0
-; CHECK-NEXT:    stp x22, x21, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    b .LBB4_3
 ; CHECK-NEXT:  .LBB4_2: // %LI.latch
 ; CHECK-NEXT:    // in Loop: Header=BB4_3 Depth=1
-; CHECK-NEXT:    cmp x23, x19
-; CHECK-NEXT:    mov x23, x24
+; CHECK-NEXT:    cmp x22, x19
+; CHECK-NEXT:    mov x22, x23
 ; CHECK-NEXT:    b.ge .LBB4_8
 ; CHECK-NEXT:  .LBB4_3: // %LI
 ; CHECK-NEXT:    // =>This Loop Header: Depth=1
 ; CHECK-NEXT:    // Child Loop BB4_6 Depth 2
-; CHECK-NEXT:    add x22, x20, x23, lsl #2
 ; CHECK-NEXT:    mov x21, xzr
-; CHECK-NEXT:    add x24, x23, #1
+; CHECK-NEXT:    add x23, x22, #1
 ; CHECK-NEXT:    b .LBB4_6
 ; CHECK-NEXT:  .LBB4_4: // %if.else
 ; CHECK-NEXT:    // in Loop: Header=BB4_6 Depth=2
-; CHECK-NEXT:    ldr w0, [x22]
+; CHECK-NEXT:    ldr w0, [x20, x22, lsl #2]
 ; CHECK-NEXT:  .LBB4_5: // %LJ.latch
 ; CHECK-NEXT:    // in Loop: Header=BB4_6 Depth=2
 ; CHECK-NEXT:    add x8, x21, #1
 ; CHECK-NEXT:    str w0, [x20, x21, lsl #2]
 ; CHECK-NEXT:    sub x9, x8, #1
 ; CHECK-NEXT:    mov x21, x8
 ; CHECK-NEXT:    cmp x9, x19
 ; CHECK-NEXT:    b.ge .LBB4_2
 ; CHECK-NEXT:  .LBB4_6: // %LJ
 ; CHECK-NEXT:    // Parent Loop BB4_3 Depth=1
 ; CHECK-NEXT:    // => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    ldr w8, [x20, x21, lsl #2]
 ; CHECK-NEXT:    tbz w8, #31, .LBB4_4
 ; CHECK-NEXT:  // %bb.7: // %if.then
 ; CHECK-NEXT:    // in Loop: Header=BB4_6 Depth=2
-; CHECK-NEXT:    mov x0, x22
+; CHECK-NEXT:    add x0, x20, x22, lsl #2
 ; CHECK-NEXT:    mov x1, x21
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    b .LBB4_5
 ; CHECK-NEXT:  .LBB4_8:
-; CHECK-NEXT:    ldp x20, x19, [sp, #48] // 16-byte Folded Reload
-; CHECK-NEXT:    ldp x22, x21, [sp, #32] // 16-byte Folded Reload
-; CHECK-NEXT:    ldp x24, x23, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x30, [sp], #64 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x22, x21, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x23, [sp], #48 // 16-byte Folded Reload
 ; CHECK-NEXT:  .LBB4_9: // %exit
 ; CHECK-NEXT:    ret
 entry:
     %c0 = icmp slt i64 %n, 1
     br i1 %c0, label %exit, label %LI
 
 LI:
     %i = phi i64 [0, %entry], [%i.next, %LI.latch]

 define void @f5(ptr %a, i32 %n, i32 %k) nounwind {
 ; CHECK-LABEL: f5:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    cmp w1, #1
 ; CHECK-NEXT:    b.lt .LBB5_7
 ; CHECK-NEXT:  // %bb.1: // %L.preheader
 ; CHECK-NEXT:    str x30, [sp, #-48]! // 8-byte Folded Spill
 ; CHECK-NEXT:    mov w8, #12 // =0xc
-; CHECK-NEXT:    stp x22, x21, [sp, #16] // 16-byte Folded Spill
-; CHECK-NEXT:    add x21, x0, #8
-; CHECK-NEXT:    smaddl x8, w2, w8, x0
 ; CHECK-NEXT:    stp x20, x19, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
+; CHECK-NEXT:    smaddl x20, w2, w8, x0
+; CHECK-NEXT:    stp x22, x21, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    add x21, x0, #8
 ; CHECK-NEXT:    mov w22, #-1 // =0xffffffff
-; CHECK-NEXT:    add x20, x8, #4
 ; CHECK-NEXT:    b .LBB5_4
 ; CHECK-NEXT:  .LBB5_2: // %if.else
 ; CHECK-NEXT:    // in Loop: Header=BB5_4 Depth=1
-; CHECK-NEXT:    ldr w0, [x20]
+; CHECK-NEXT:    ldur w0, [x20, #4]
 ; CHECK-NEXT:  .LBB5_3: // %L.latch
 ; CHECK-NEXT:    // in Loop: Header=BB5_4 Depth=1
 ; CHECK-NEXT:    add w22, w22, #1
 ; CHECK-NEXT:    str w0, [x21], #12
 ; CHECK-NEXT:    cmp w22, w19
 ; CHECK-NEXT:    b.ge .LBB5_6
 ; CHECK-NEXT:  .LBB5_4: // %L
 ; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldr w8, [x21]
 ; CHECK-NEXT:    tbz w8, #31, .LBB5_2
 ; CHECK-NEXT:  // %bb.5: // %if.then
 ; CHECK-NEXT:    // in Loop: Header=BB5_4 Depth=1
+; CHECK-NEXT:    add x0, x20, #4
 ; CHECK-NEXT:    add w1, w22, #1
-; CHECK-NEXT:    mov x0, x20
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    b .LBB5_3
 ; CHECK-NEXT:  .LBB5_6:
 ; CHECK-NEXT:    ldp x20, x19, [sp, #32] // 16-byte Folded Reload
 ; CHECK-NEXT:    ldp x22, x21, [sp, #16] // 16-byte Folded Reload
 ; CHECK-NEXT:    ldr x30, [sp], #48 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB5_7: // %exit
 ; CHECK-NEXT:    ret
L.latch:
     store i32 %u, ptr %ai.ptr
     %c2 = icmp slt i32 %i, %n
     br i1 %c2, label %L, label %exit
 
 exit:
     ret void
 }
 
 define i32 @f6(i1 %c, ptr %a, i32 %i) {
 ; CHECK-LABEL: f6:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $w2 killed $w2 def $x2
-; CHECK-NEXT:    sxtw x8, w2
 ; CHECK-NEXT:    tbz w0, #0, .LBB6_2
 ; CHECK-NEXT:  // %bb.1: // %if.then
 ; CHECK-NEXT:    mov w0, wzr
-; CHECK-NEXT:    str wzr, [x1, x8, lsl #2]
+; CHECK-NEXT:    str wzr, [x1, w2, sxtw #2]
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB6_2: // %if.else
-; CHECK-NEXT:    ldr w0, [x1, x8, lsl #2]
+; CHECK-NEXT:    ldr w0, [x1, w2, sxtw #2]
 ; CHECK-NEXT:    ret
 entry:
     %j = sext i32 %i to i64
     br i1 %c, label %if.then, label %if.else
 
 if.then:
     %p0 = getelementptr i32, ptr %a, i64 %j
     store i32 0, ptr %p0
     br label %exit
 
 if.else:
     %p1 = getelementptr i32, ptr %a, i64 %j
     %v0 = load i32, ptr %p1
     br label %exit
 
 exit:
     %v = phi i32 [0, %if.then], [%v0, %if.else]
     ret i32 %v
 }
 
 define i8 @f7(i1 %c, ptr %a, i32 %i) {
 ; CHECK-LABEL: f7:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    mov w8, w2
 ; CHECK-NEXT:    tbz w0, #0, .LBB7_2
 ; CHECK-NEXT:  // %bb.1: // %if.then
 ; CHECK-NEXT:    mov w0, wzr
-; CHECK-NEXT:    strb wzr, [x1, x8]
+; CHECK-NEXT:    strb wzr, [x1, w2, uxtw]
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB7_2: // %if.else
-; CHECK-NEXT:    ldrb w0, [x1, x8]
+; CHECK-NEXT:    ldrb w0, [x1, w2, uxtw]
 ; CHECK-NEXT:    ret
 entry:
     %j = zext i32 %i to i64
     br i1 %c, label %if.then, label %if.else
 
 if.then:
     %p0 = getelementptr i8, ptr %a, i64 %j
     store i8 0, ptr %p0
     br label %exit
 
 if.else:
     %p1 = getelementptr i8, ptr %a, i64 %j
     %v0 = load i8, ptr %p1
     br label %exit
 
 exit:
     %v = phi i8 [0, %if.then], [%v0, %if.else]
     ret i8 %v
 }
 
 define i32 @f8(i1 %c, ptr %a, i32 %i) {
 ; CHECK-LABEL: f8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    add x8, x1, w2, sxtw #2
 ; CHECK-NEXT:    tbz w0, #0, .LBB8_2
 ; CHECK-NEXT:  // %bb.1: // %if.then
 ; CHECK-NEXT:    mov w0, wzr
-; CHECK-NEXT:    str wzr, [x8]
+; CHECK-NEXT:    str wzr, [x1, w2, sxtw #2]
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB8_2: // %if.else
-; CHECK-NEXT:    ldr w0, [x8]
+; CHECK-NEXT:    ldr w0, [x1, w2, sxtw #2]
 ; CHECK-NEXT:    ret
 entry:
     %p = getelementptr i32, ptr %a, i32 %i
     br i1 %c, label %if.then, label %if.else
 
 if.then:
     store i32 0, ptr %p
     br label %exit
 
 if.else:
     %v0 = load i32, ptr %p
     br label %exit
 
 exit:
     %v = phi i32 [0, %if.then], [%v0, %if.else]
     ret i32 %v
 }
 
 define i64 @f9(i1 %c, ptr %a, i32 %i) {
 ; CHECK-LABEL: f9:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    mov w8, w2
 ; CHECK-NEXT:    tbz w0, #0, .LBB9_2
 ; CHECK-NEXT:  // %bb.1: // %if.then
 ; CHECK-NEXT:    mov x0, xzr
-; CHECK-NEXT:    str xzr, [x1, x8, lsl #3]
+; CHECK-NEXT:    str xzr, [x1, w2, uxtw #3]
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB9_2: // %if.else
-; CHECK-NEXT:    ldr x0, [x1, x8, lsl #3]
+; CHECK-NEXT:    ldr x0, [x1, w2, uxtw #3]
 ; CHECK-NEXT:    ret
 entry:
     %j = zext i32 %i to i64
     %p = getelementptr i64, ptr %a, i64 %j
     br i1 %c, label %if.then, label %if.else
 
 if.then:
     store i64 0, ptr %p

llvm/test/CodeGen/AArch64/swift-async-win.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
-; RUN: llc -mtriple aarch64-unknown-windows -swift-async-fp=never -filetype asm -o - %s | FileCheck %s
+; RUN: llc -mtriple aarch64-unknown-windows -swift-async-fp=never -aarch64-enable-sink-fold=true -filetype asm -o - %s | FileCheck %s
 
 ; ModuleID = '_Concurrency.ll'
 source_filename = "_Concurrency.ll"
 target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-windows-msvc19.32.31302"
 
 %swift.context = type { ptr, ptr }
 
 ; Function Attrs: argmemonly nofree nosync nounwind willreturn
 declare void @llvm.lifetime.end.p0(i64 immarg, ptr nocapture) #0
 
 ; NOTE: we do not see the canonical windows frame setup due to the `nounwind`
 ; attribtue on the function.
 
 ; Function Attrs: nounwind
 define hidden swifttailcc void @"$ss23withCheckedContinuation8function_xSS_yScCyxs5NeverOGXEtYalFTQ0_"(ptr nocapture readonly %0) #1 {
 ; CHECK-LABEL: $ss23withCheckedContinuation8function_xSS_yScCyxs5NeverOGXEtYalFTQ0_:
 ; CHECK:       // %bb.0: // %entryresume.0
 ; CHECK-NEXT:    sub sp, sp, #48
 ; CHECK-NEXT:    stp x30, x29, [sp, #24] // 16-byte Folded Spill
 ; CHECK-NEXT:    add x29, sp, #24
 ; CHECK-NEXT:    str x19, [sp, #40] // 8-byte Folded Spill
-; CHECK-NEXT:    sub x8, x29, #8
 ; CHECK-NEXT:    adrp x19, __imp_swift_task_dealloc
 ; CHECK-NEXT:    str xzr, [sp, #16]
-; CHECK-NEXT:    ldr x9, [x0]
-; CHECK-NEXT:    str x9, [x8]
+; CHECK-NEXT:    ldr x8, [x0]
+; CHECK-NEXT:    stur x8, [x29, #-8]
 ; CHECK-NEXT:    ldr x20, [x0]
-; CHECK-NEXT:    ldp x22, x0, [x9, #16]
-; CHECK-NEXT:    str x20, [x8]
+; CHECK-NEXT:    ldp x22, x0, [x8, #16]
+; CHECK-NEXT:    stur x20, [x29, #-8]
 ; CHECK-NEXT:    ldr x19, [x19, :lo12:__imp_swift_task_dealloc]
 ; CHECK-NEXT:    blr x19
 ; CHECK-NEXT:    mov x0, x22
 ; CHECK-NEXT:    blr x19
 ; CHECK-NEXT:    ldp x30, x29, [sp, #24] // 16-byte Folded Reload
 ; CHECK-NEXT:    mov x0, x20
 ; CHECK-NEXT:    ldr x1, [x20, #8]
 ; CHECK-NEXT:    ldr x19, [sp, #40] // 8-byte Folded Reload

llvm/test/CodeGen/AArch64/swift-async.ll

-; RUN: llc -mtriple=arm64-apple-ios15 %s -o - | FileCheck %s --check-prefixes=CHECK-NOAUTH,CHECK
-; RUN: llc -mtriple=arm64-apple-ios15 -mcpu=apple-a13 %s -o - | FileCheck %s --check-prefixes=CHECK-NOAUTH,CHECK
-; RUN: llc -mtriple=arm64e-apple-ios15 %s -o - | FileCheck %s --check-prefixes=CHECK-AUTH,CHECK
+; RUN: llc -mtriple=arm64-apple-ios15 -aarch64-enable-sink-fold=true %s -o - | FileCheck %s --check-prefixes=CHECK-NOAUTH,CHECK
+; RUN: llc -mtriple=arm64-apple-ios15 -aarch64-enable-sink-fold=true -mcpu=apple-a13 %s -o - | FileCheck %s --check-prefixes=CHECK-NOAUTH,CHECK
+; RUN: llc -mtriple=arm64e-apple-ios15 -aarch64-enable-sink-fold=true %s -o - | FileCheck %s --check-prefixes=CHECK-AUTH,CHECK
 
 ; Important details in prologue:
 ;   * x22 is stored just below x29
 ;   * Enough stack space is allocated for everything
 define swifttailcc void @simple(ptr swiftasync %ctx) "frame-pointer"="all" {
 ; CHECK-LABEL: simple:
 ; CHECK: orr x29, x29, #0x100000000000000
 ; CHECK: sub sp, sp, #32

 ; CHECK-AUTH: str x17, [sp, #8]
 
   %ptr = call ptr @llvm.swift.async.context.addr()
   ret ptr %ptr
 }
 
 define swifttailcc void @write_frame_context(ptr swiftasync %ctx, ptr %newctx) "frame-pointer"="non-leaf" {
 ; CHECK-LABEL: write_frame_context:
-; CHECK: sub x[[ADDR:[0-9]+]], x29, #8
-; CHECK: str x0, [x[[ADDR]]]
+; CHECK: stur x0, [x29, #-8]
   %ptr = call ptr @llvm.swift.async.context.addr()
   store ptr %newctx, ptr %ptr
   ret void
 }
 
 define swifttailcc void @simple_fp_elim(ptr swiftasync %ctx) "frame-pointer"="non-leaf" {
 ; CHECK-LABEL: simple_fp_elim:
 ; CHECK-NOT: orr x29, x29, #0x100000000000000