[ARM] Simplify address calculation for NEON load/store

[asavonic]

Summary

The patch attempts to optimize a sequence of SIMD loads from the same
base pointer:

%0 = gep float*, float* base, i32 4
%1 = bitcast float* %0 to <4 x float>*
%2 = load <4 x float>, <4 x float>* %1
...
%n1 = gep float*, float* base, i32 N
%n2 = bitcast float* %n1 to <4 x float>*
%n3 = load <4 x float>, <4 x float>* %n2

For AArch64 the compiler generates a sequence of LDR Qt, [Xn, #16].
However, 32-bit NEON VLD1/VST1 lack the [Wn, #imm] addressing mode, so
the address is computed before every ld/st instruction:

add r2, r0, #32
add r0, r0, #16
vld1.32 {d18, d19}, [r2]
vld1.32 {d22, d23}, [r0]

This can be improved by computing address for the first load, and then
using a post-indexed form of VLD1/VST1 to load the rest:

add r0, r0, #16
vld1.32 {d18, d19}, [r0]!
vld1.32 {d22, d23}, [r0]

In order to do that, the patch adds more patterns to DAGCombine:

• (load (add ptr inc1)) and (add ptr inc2) are now folded if inc1 and inc2 are constants.
• (or ptr inc) is now recognized as a pointer increment if ptr is sufficiently aligned.

In addition to that, we now search for all possible base updates and
then pick the best one.

[dmgreen]

My first thought was why can't this be handled by LSR, but I can see how that might not work very well trying to precisely match VLD offsets. And the tests you added have no loops :)

(I also had some thoughts about whether this was useful in general, or if a sufficiently powerful cpu would break these into microops in either case, leading to the same performance in the end. But the code does look cleaner now, I can see how it would improve things)

The way we handled this in MVE was to "distribute" the increments in the ARMLoadStoreOptimizer pass. The instructions in MVE are different, and that does involve checking through Machine Instructions for Adds that can be better distributed into postinc instructions. LSR got it mostly right, DAG Combine did an OKish job most of the time, and we fixed up what went wrong later in the pipeline.

It seems to have worked out OK as far as I can tell, is there a reason we can't do the same thing here? Adding the new pass seems fine if we need it, but I'm less sanguine about having to disable a lot of Add folds in DAGCombiner.

llvm/test/CodeGen/ARM/arm-post-indexing-opt.ll

1–2

It is best not to mix llc and opt test files. They are easier kept as separate tests, with autogenerated check lines in each.
It's also good to show before and after in the review by pre-committing the tests, which makes the review easier by making it obvious what has changed.

[asavonic]

In D108988#2979176, @dmgreen wrote:

My first thought was why can't this be handled by LSR, but I can see how that might not work very well trying to precisely match VLD offsets. And the tests you added have no loops :)

The patch is focused on sequential loads that have the same base
pointer and constant offsets, but it can also work if such sequence is
in a loop body:

void test(float *a, float *b, int n) {
  for (int i = 0; i < n; ++i) {
    v4f32 A1 = vld1q_f32(a + 16 * i);
    v4f32 A2 = vld1q_f32(a + 16 * i + 4);
    v4f32 A3 = vld1q_f32(a + 16 * i + 8);
    v4f32 A4 = vld1q_f32(a + 16 * i + 12);
    vst1q_f32(b + 4 * i, A1);
    vst1q_f32(b + 4 * i, A2);
    vst1q_f32(b + 4 * i, A3);
    vst1q_f32(b + 4 * i, A4);
  }

LSR seems to only handle values that are loop IV, so these constant
offsets are not optimized. The loop body is compiled to:

add     lr, r0, r3
subs    r2, r2, #1
mov     r4, lr
vld1.32 {d16, d17}, [r4], r12
vld1.32 {d18, d19}, [r4]
add     r4, lr, #32              ; <-- extra address computation
vld1.32 {d20, d21}, [r4]
add     r4, lr, #16              ; <--
vld1.32 {d22, d23}, [r4]
add     r4, r1, r3
add     r5, r4, #16              ; <--
add     r3, r3, #64
mov     lr, r4
add     r4, r4, #32              ; <--
vst1.32 {d16, d17}, [lr], r12
vst1.32 {d22, d23}, [r5]
vst1.32 {d20, d21}, [r4]
vst1.32 {d18, d19}, [lr]
bne     .LBB0_2

In the first revision of this patch ARMPostIndexingOpt was confused by
GEP patterns produced by LSR. This is now fixed and the sequence is
optimized to:

add     r3, r0, r12
subs    r2, r2, #1
vld1.32 {d16, d17}, [r3]!
vld1.32 {d18, d19}, [r3]!
vld1.32 {d20, d21}, [r3]!
vld1.32 {d22, d23}, [r3]
add     r3, r1, r12
add     r12, r12, #64
vst1.32 {d16, d17}, [r3]!
vst1.32 {d18, d19}, [r3]!
vst1.32 {d20, d21}, [r3]!
vst1.32 {d22, d23}, [r3]
bne     .LBB0_2

(I also had some thoughts about whether this was useful in general, or if a sufficiently powerful cpu would break these into microops in either case, leading to the same performance in the end. But the code does look cleaner now, I can see how it would improve things)

I've measured execution time of the loop above, and it is ~7% faster
on Cortex-A72. It may be different on other hardware though.

The way we handled this in MVE was to "distribute" the increments in the ARMLoadStoreOptimizer pass. The instructions in MVE are different, and that does involve checking through Machine Instructions for Adds that can be better distributed into postinc instructions. LSR got it mostly right, DAG Combine did an OKish job most of the time, and we fixed up what went wrong later in the pipeline.

It seems to have worked out OK as far as I can tell, is there a reason we can't do the same thing here?

I think the approach is still the same, the new pass just works for
cases that LSR does not handle.

Adding the new pass seems fine if we need it, but I'm less sanguine about having to disable a lot of Add folds in DAGCombiner.

Agree, this is potentially the most problematic change. It is limited
to just (load/store (add)) and works only before legalization, so this
/hopefully/ reduces its impact to just the patterns we need.

[asavonic]

• Added handling for GEP patterns generated by LSR.
• Split llc and opt LIT tests.
• Pre-committed llc LIT test.

[dmgreen]

The way we handled this in MVE was to "distribute" the increments in the ARMLoadStoreOptimizer pass. The instructions in MVE are different, and that does involve checking through Machine Instructions for Adds that can be better distributed into postinc instructions. LSR got it mostly right, DAG Combine did an OKish job most of the time, and we fixed up what went wrong later in the pipeline.

It seems to have worked out OK as far as I can tell, is there a reason we can't do the same thing here?

I think the approach is still the same, the new pass just works for
cases that LSR does not handle.

The MVE method works quite differently I feel. It fixes up problems later in the pipeline at the mir level, not trying to get them perfectly correct before ISel. It can be awkward dealing with mir though, and difficult to look through all the instructions that can be generated looking at ways to distribute postincs more evenly if the results are not already close enough. I don't think I would recommend it for this problem, it would probably be simpler to fix it up in the DAG than trying to do it later in MIR.

Adding the new pass seems fine if we need it, but I'm less sanguine about having to disable a lot of Add folds in DAGCombiner.

Agree, this is potentially the most problematic change. It is limited
to just (load/store (add)) and works only before legalization, so this
/hopefully/ reduces its impact to just the patterns we need.

Unfortunately I'm not sure we can say this in general, even if it is quite rare. I feel like having this reassociationCanBreakPostIndexingPattern method in the generic DAG combine code disabling so many of the ADD folds is not a good sign. It feels like it's working around the fact that this is a quite fragile way of trying to get postinc working. The main issues I found I think could be fixed with one use checks in reassociationCanBreakPostIndexingPattern, but is there a way to make this work without disabling the generic folds? Perhaps by adding new folds for fixing postinc patterns, and getting "add-like-ors" to behave like adds in more places?

[asavonic]

is there a way to make this work without disabling the generic folds? Perhaps by adding new folds for fixing postinc patterns, and getting "add-like-ors" to behave like adds in more places?

Done. Everything is moved to DAGCombine now, so the new pass is not required.
This allows to catch more cases, and not require any changes around
ADD-to-OR tranformation.

Code from CombineBaseUpdate is moved to TryCombineBaseUpdate without
major changes, but now we look for more candidates (pointer updates)
and try to pick the best one.

[dmgreen]

Nice work. I'm glad this works this way.

llvm/lib/Target/ARM/ARMISelLowering.cpp

15207

How useful are these error messages do you think, in the long run?

The number of combines tried can be quite high, and these may just end up adding noise for people not looking at CombineBaseUpdate combines. It seems to be fairly uncommon to add debug messages for DAG combines.

But if you think they are useful, feel free to keep them around.

15522

Can this use haveNoCommonBitsSet, similar to ARMDAGToDAGISel::SelectAddLikeOr?

15540

Do we need to check both operands? The Add/Or should be canonicalized to have the constant on the RHS.

llvm/test/CodeGen/ARM/alloc-no-stack-realign.ll

19 ↗

(On Diff #378619)

How come this test is changing?

[asavonic]

• Used DAG.haveNoCommonBitsSet to check for ADD-like OR.
• Removed handling for non-canonical DAG.
• Removed extra debug messages.

[asavonic]

llvm/lib/Target/ARM/ARMISelLowering.cpp

15207

Let's remove them.
They were useful for debugging, but standard messages from DAG combiner are good enough.

15522

Done.

15540

Thanks. I was not sure that we can expect that.
Removed the extra check.

llvm/test/CodeGen/ARM/alloc-no-stack-realign.ll

19 ↗

(On Diff #378619)

Oh, this one is tricky.
The original test expected to see an OR instruction when the stack is aligned, and an ADD instruction when it is not. After the patch both ADD and OR are folded with loads/stores, so the two sequences (test1 and test2) are completely identical.

I changed the IR slightly, so that OR is not folded.

[dmgreen]

Thanks. LGTM

llvm/lib/Target/ARM/ARMISelLowering.cpp

15404

LLVM tends to leave the brackets off single statement if blocks.

[asavonic]

llvm/lib/Target/ARM/ARMISelLowering.cpp

15404

Thank you. Fixed that before landing.

llvm/include/llvm/CodeGen/TargetLowering.h

public:
   ///
   /// If the address space cannot be determined, it will be -1.
   ///
   /// TODO: Remove default argument
   virtual bool isLegalAddressingMode(const DataLayout &DL, const AddrMode &AM,
                                      Type *Ty, unsigned AddrSpace,
                                      Instruction *I = nullptr) const;
 
+  /// Return true if it is beneficial to retain post-indexing-friendly patterns
+  /// while performing optimizations.
+  virtual bool shouldRetainImmediatePostIncrement(const DataLayout &DL,
+                                                  Type *Ty, CombineLevel Level,
+                                                  unsigned AddrSpace,
+                                                  int64_t Increment) const {
+    return false;
+  }
+
   /// Return the cost of the scaling factor used in the addressing mode
   /// represented by AM for this target, for a load/store of the specified type.
   ///
   /// If the AM is supported, the return value must be >= 0.
   /// If the AM is not supported, it returns a negative value.
   /// TODO: Handle pre/postinc as well.
   /// TODO: Remove default argument
   virtual InstructionCost getScalingFactorCost(const DataLayout &DL,

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

private:
     SDValue visitFADDForFMACombine(SDNode *N);
     SDValue visitFSUBForFMACombine(SDNode *N);
     SDValue visitFMULForFMADistributiveCombine(SDNode *N);
 
     SDValue XformToShuffleWithZero(SDNode *N);
     bool reassociationCanBreakAddressingModePattern(unsigned Opc,
                                                     const SDLoc &DL, SDValue N0,
                                                     SDValue N1);
+    bool reassociationCanBreakPostIndexingPattern(SDNode *N);
     SDValue reassociateOpsCommutative(unsigned Opc, const SDLoc &DL, SDValue N0,
                                       SDValue N1);
     SDValue reassociateOps(unsigned Opc, const SDLoc &DL, SDValue N0,
                            SDValue N1, SDNodeFlags Flags);
 
     SDValue visitShiftByConstant(SDNode *N);
 
     SDValue foldSelectOfConstants(SDNode *N);
if (LoadStore) {
       if (!TLI.isLegalAddressingMode(DAG.getDataLayout(), AM, AccessTy, AS))
         return true;
     }
   }
 
   return false;
 }
 
+bool DAGCombiner::reassociationCanBreakPostIndexingPattern(SDNode *N) {
+  const DataLayout &DL = DAG.getDataLayout();
+  if (N->getOpcode() != ISD::ADD)
+    return false;
+
+  auto Const = dyn_cast<ConstantSDNode>(N->getOperand(1));
+  if (!Const)
+    return false;
+
+  const APInt &APIntVal = Const->getAPIntValue();
+  if (APIntVal.getBitWidth() > 64)
+    return false;
+  const int64_t ConstValue = APIntVal.getSExtValue();
+
+  // Check for (load/store (add x, const))
+
+  for (SDNode *Node : N->getOperand(0)->uses()) {
+    auto LoadStore = dyn_cast<MemSDNode>(Node);
+    if (!LoadStore)
+      continue;
+
+    EVT VT = LoadStore->getMemoryVT();
+    Type *AccessTy = VT.getTypeForEVT(*DAG.getContext());
+    unsigned AS = LoadStore->getAddressSpace();
+    if (TLI.shouldRetainImmediatePostIncrement(DL, AccessTy, Level, AS,
+                                               ConstValue))
+      return true;
+  }
+  return false;
+}
+
 // Helper for DAGCombiner::reassociateOps. Try to reassociate an expression
 // such as (Opc N0, N1), if \p N0 is the same kind of operation as \p Opc.
 SDValue DAGCombiner::reassociateOpsCommutative(unsigned Opc, const SDLoc &DL,
                                                SDValue N0, SDValue N1) {
   EVT VT = N0.getValueType();
 
   if (N0.getOpcode() != Opc)
     return SDValue();

 }
 
 SDValue DAGCombiner::visitADD(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   EVT VT = N0.getValueType();
   SDLoc DL(N);
 
+  // Prevent ADD reassociation as well as converting ADD -> OR
+  if (reassociationCanBreakPostIndexingPattern(N) ||
+      reassociationCanBreakPostIndexingPattern(N0.getNode()) ||
+      reassociationCanBreakPostIndexingPattern(N1.getNode())) {
+    return SDValue();
+  }
+
   if (SDValue Combined = visitADDLike(N))
     return Combined;
 
   if (SDValue V = foldAddSubBoolOfMaskedVal(N, DAG))
     return V;
 
   if (SDValue V = foldAddSubOfSignBit(N, DAG))
     return V;

llvm/lib/Target/ARM/ARM.h

 Pass *createMVETailPredicationPass();
 FunctionPass *createARMLowOverheadLoopsPass();
 FunctionPass *createARMBlockPlacementPass();
 Pass *createARMParallelDSPPass();
 FunctionPass *createARMISelDag(ARMBaseTargetMachine &TM,
                                CodeGenOpt::Level OptLevel);
 FunctionPass *createA15SDOptimizerPass();
 FunctionPass *createARMLoadStoreOptimizationPass(bool PreAlloc = false);
+FunctionPass *createARMPostIndexingOptimizationPass();
 FunctionPass *createARMExpandPseudoPass();
 FunctionPass *createARMConstantIslandPass();
 FunctionPass *createMLxExpansionPass();
 FunctionPass *createThumb2ITBlockPass();
 FunctionPass *createMVEVPTBlockPass();
 FunctionPass *createMVETPAndVPTOptimisationsPass();
 FunctionPass *createARMOptimizeBarriersPass();
 FunctionPass *createThumb2SizeReductionPass(
     std::function<bool(const Function &)> Ftor = nullptr);
 InstructionSelector *
 createARMInstructionSelector(const ARMBaseTargetMachine &TM, const ARMSubtarget &STI,
                              const ARMRegisterBankInfo &RBI);
 Pass *createMVEGatherScatterLoweringPass();
 FunctionPass *createARMSLSHardeningPass();
 FunctionPass *createARMIndirectThunks();
 Pass *createMVELaneInterleavingPass();
 
 void LowerARMMachineInstrToMCInst(const MachineInstr *MI, MCInst &OutMI,
                                   ARMAsmPrinter &AP);
 
 void initializeARMParallelDSPPass(PassRegistry &);
 void initializeARMLoadStoreOptPass(PassRegistry &);
+void initializeARMPostIndexingOptPass(PassRegistry &);
 void initializeARMPreAllocLoadStoreOptPass(PassRegistry &);
 void initializeARMConstantIslandsPass(PassRegistry &);
 void initializeARMExpandPseudoPass(PassRegistry &);
 void initializeThumb2SizeReducePass(PassRegistry &);
 void initializeThumb2ITBlockPass(PassRegistry &);
 void initializeMVEVPTBlockPass(PassRegistry &);
 void initializeMVETPAndVPTOptimisationsPass(PassRegistry &);
 void initializeARMLowOverheadLoopsPass(PassRegistry &);

llvm/lib/Target/ARM/ARMISelLowering.h

public:
 
 
     /// isLegalAddressingMode - Return true if the addressing mode represented
     /// by AM is legal for this target, for a load/store of the specified type.
     bool isLegalAddressingMode(const DataLayout &DL, const AddrMode &AM,
                                Type *Ty, unsigned AS,
                                Instruction *I = nullptr) const override;
 
+    bool shouldRetainImmediatePostIncrement(const DataLayout &DL, Type *Ty,
+                                            CombineLevel Level,
+                                            unsigned AddrSpace,
+                                            int64_t Increment) const override;
+
     /// getScalingFactorCost - Return the cost of the scaling used in
     /// addressing mode represented by AM.
     /// If the AM is supported, the return value must be >= 0.
     /// If the AM is not supported, the return value must be negative.
     InstructionCost getScalingFactorCost(const DataLayout &DL,
                                          const AddrMode &AM, Type *Ty,
                                          unsigned AS) const override;
 

llvm/lib/Target/ARM/ARMISelLowering.cpp

static SDValue CombineBaseUpdate(SDNode *N,
   const unsigned AddrOpIdx = ((isIntrinsic || isStore) ? 2 : 1);
   SDValue Addr = N->getOperand(AddrOpIdx);
   MemSDNode *MemN = cast<MemSDNode>(N);
   SDLoc dl(N);
 
   // Search for a use of the address operand that is an increment.
   for (SDNode::use_iterator UI = Addr.getNode()->use_begin(),
          UE = Addr.getNode()->use_end(); UI != UE; ++UI) {
     SDNode *User = *UI;
     if (User->getOpcode() != ISD::ADD ||
         UI.getUse().getResNo() != Addr.getResNo())
       continue;
 
     // Check that the add is independent of the load/store.  Otherwise, folding
     // it would create a cycle. We can avoid searching through Addr as it's a
     // predecessor to both.
     SmallPtrSet<const SDNode *, 32> Visited;
for (SDNode::use_iterator UI = Addr.getNode()->use_begin(),
     Ops.push_back(DAG.getConstant(Alignment, dl, MVT::i32));
 
     // If this is a non-standard-aligned STORE, the penultimate operand is the
     // stored value.  Bitcast it to the aligned type.
     if (AlignedVecTy != VecTy && N->getOpcode() == ISD::STORE) {
       SDValue &StVal = Ops[Ops.size()-2];
       StVal = DAG.getNode(ISD::BITCAST, dl, AlignedVecTy, StVal);
     }
 
     EVT LoadVT = isLaneOp ? VecTy.getVectorElementType() : AlignedVecTy;
     SDValue UpdN = DAG.getMemIntrinsicNode(NewOpc, dl, SDTys, Ops, LoadVT,
                                            MemN->getMemOperand());
 
     // Update the uses.
     SmallVector<SDValue, 5> NewResults;
     for (unsigned i = 0; i < NumResultVecs; ++i)
       NewResults.push_back(SDValue(UpdN.getNode(), i));
for (SDNode::use_iterator UI = Addr.getNode()->use_begin(),
     unsigned NumBytes = NumVecs * VecTy.getSizeInBits() / 8;
 
     // If the increment is a constant, it must match the memory ref size.
     SDValue Inc = User->getOperand(User->getOperand(0) == Addr ? 1 : 0);
     ConstantSDNode *CInc = dyn_cast<ConstantSDNode>(Inc.getNode());
     if (!CInc || CInc->getZExtValue() != NumBytes)
       continue;
 
     // Create the new updating load/store node.
     // First, create an SDVTList for the new updating node's results.
     EVT Tys[6];
     unsigned NumResultVecs = (isLoadOp ? NumVecs : 0);
     unsigned n;
     for (n = 0; n < NumResultVecs; ++n)
       Tys[n] = VecTy;
     Tys[n++] = MVT::i32;
     Tys[n] = MVT::Other;
     SDVTList SDTys = DAG.getVTList(makeArrayRef(Tys, NumResultVecs + 2));
 
     // Then, gather the new node's operands.
     SmallVector<SDValue, 8> Ops;
     Ops.push_back(N->getOperand(0)); // incoming chain
     Ops.push_back(N->getOperand(2)); // ptr
     Ops.push_back(Inc);
 
     for (unsigned i = 3; i < N->getNumOperands(); ++i)
       Ops.push_back(N->getOperand(i));
 
     SDValue UpdN = DAG.getMemIntrinsicNode(NewOpc, dl, SDTys, Ops, VecTy,
                                            MemN->getMemOperand());
 
     // Update the uses.
     SmallVector<SDValue, 5> NewResults;
     for (unsigned i = 0; i < NumResultVecs; ++i)
       NewResults.push_back(SDValue(UpdN.getNode(), i));
case MVT::isVoid:
       // Allow r << imm, but the imm has to be a multiple of two.
       if (Scale & 1) return false;
       return isPowerOf2_32(Scale);
     }
   }
   return true;
 }
 
+bool ARMTargetLowering::shouldRetainImmediatePostIncrement(
+    const DataLayout &DL, Type *Ty, CombineLevel Level, unsigned AddrSpace,
+    int64_t Increment) const {
+  // NEON has rather restricted address calculation for vector load / store
+  // instructions compared to MVE or AArch64 ASIMD.
+  if (Subtarget->hasMVEIntegerOps())
+    return false;
+
+  // If the first DAG optimization pass did not consume this increment,
+  // try combining as usual during subsequent optimization passes.
+  if (Level != CombineLevel::BeforeLegalizeTypes)
+    return false;
+
+  if (!Ty->isVectorTy())
+    return false;
+
+  unsigned BitSize = DL.getTypeSizeInBits(Ty);
+
+  return BitSize > 64 && isPowerOf2_32(BitSize);
+}
+
 /// isLegalICmpImmediate - Return true if the specified immediate is legal
 /// icmp immediate, that is the target has icmp instructions which can compare
 /// a register against the immediate without having to materialize the
 /// immediate into a register.
 bool ARMTargetLowering::isLegalICmpImmediate(int64_t Imm) const {
   // Thumb2 and ARM modes can use cmn for negative immediates.
   if (!Subtarget->isThumb())
     return ARM_AM::getSOImmVal((uint32_t)Imm) != -1 ||

llvm/lib/Target/ARM/ARMPostIndexingOptimizer.cpp

+//===- ARMPostIndexingOptimizer.cpp - Prepare ld/st for post-indexing -----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file ARMPostIndexingOpt pass transforms address operands of load/store
+/// instructions to allow them to be emitted as NEON load/store with post-index
+/// addressing mode. For example:
+///
+///   %first = gep %base, %offset1
+///   load %first
+///   %second = gep %base, %offset2
+///   load %second
+///
+/// this sequence may be tranformed into:
+///
+///   %first = gep %base, %offset1
+///   load %first
+///   %second = gep %first, (%offset2 - %offset1)
+///   load %second
+///
+/// The transformation is done only if:
+///  1) GEPs are constant
+///  2) Difference between offsets is compatible with either "[Rn]!"
+///     or "[Rn], Rm" addressing modes.
+//
+//===----------------------------------------------------------------------===//
+
+#include "ARM.h"
+#include "ARMSubtarget.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
+#include "llvm/CodeGen/TargetPassConfig.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicsARM.h"
+#include "llvm/Pass.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Target/TargetMachine.h"
+#include "llvm/Transforms/Utils/Local.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "arm-post-indexing-opt"
+#define PASS_DESC "ARM post-indexed access optimizer"
+
+namespace {
+
+// LdStInfo the base address of a load or store, and an immediate offset of a
+// memory access obtained by a best-effort heuristic as well as other useful
+// instruction properties.
+struct LdStInfo {
+  LdStInfo(const DataLayout &DL, Instruction *LdSt, unsigned BaseOperandIndex,
+           int AccessSize);
+  // A memory access in question
+  Instruction *LdSt;
+  // An actual operand of LdSt can be updated throughout this pass execution,
+  // so store an index instead
+  unsigned BaseOperandIndex;
+  // A guessed base address
+  Value *IndirectBase;
+  // An immediate offset to add to IndirectBase
+  int32_t Offset;
+  // An access size that can be used for post-indexed addressing mode
+  int AccessSize;
+
+  // Returns current *direct* base operand
+  Value *getBaseOperand() { return LdSt->getOperand(BaseOperandIndex); }
+};
+
+struct ARMPostIndexingOpt : public FunctionPass {
+  static char ID;
+
+  ARMPostIndexingOpt() : FunctionPass(ID) {}
+
+  StringRef getPassName() const override { return PASS_DESC; }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<TargetLibraryInfoWrapperPass>();
+    AU.addRequired<TargetPassConfig>();
+    AU.setPreservesCFG();
+  }
+
+  bool runOnFunction(Function &F) override;
+};
+
+} // end anonymous namespace
+
+char ARMPostIndexingOpt::ID = 0;
+INITIALIZE_PASS(ARMPostIndexingOpt, DEBUG_TYPE, PASS_DESC, false, false)
+
+// Returns (nullptr, 0) for instructions not handled by this pass
+static std::pair<Type *, unsigned> getDataTypeAndBaseIndex(Instruction *I) {
+  if (LoadInst *Load = dyn_cast<LoadInst>(I))
+    return std::make_pair(Load->getType(), 0);
+  if (StoreInst *Store = dyn_cast<StoreInst>(I))
+    return std::make_pair(Store->getValueOperand()->getType(), 1);
+
+  if (IntrinsicInst *Intrinsic = dyn_cast<IntrinsicInst>(I)) {
+    switch (Intrinsic->getIntrinsicID()) {
+    case Intrinsic::arm_neon_vld1:
+    case Intrinsic::arm_neon_vld2:
+    case Intrinsic::arm_neon_vld3:
+    case Intrinsic::arm_neon_vld4:
+      return std::make_pair(Intrinsic->getType(), 0);
+    case Intrinsic::arm_neon_vst1:
+    case Intrinsic::arm_neon_vst2:
+    case Intrinsic::arm_neon_vst3:
+    case Intrinsic::arm_neon_vst4:
+      return std::make_pair(Intrinsic->getOperand(1)->getType(), 0);
+    default:
+      break;
+    }
+  }
+  return std::make_pair(nullptr, 0);
+}
+
+LdStInfo::LdStInfo(const DataLayout &DL, Instruction *LdSt,
+                   unsigned BaseOperandIndex, int AccessSize)
+    : LdSt(LdSt), BaseOperandIndex(BaseOperandIndex), Offset(0),
+      AccessSize(AccessSize) {
+  IndirectBase = LdSt->getOperand(BaseOperandIndex);
+  for (;;) {
+    IndirectBase = IndirectBase->stripPointerCasts();
+    // Match GetElementPtrInst as well as corresponding ContantExpr
+    if (auto *GEP = dyn_cast<GEPOperator>(IndirectBase)) {
+      APInt APOffset(32, 0, /* isSigned = */ true);
+      if (GEP->accumulateConstantOffset(DL, APOffset)) {
+        IndirectBase = GEP->getPointerOperand();
+        Offset += APOffset.getSExtValue();
+        continue;
+      }
+    }
+    return;
+  }
+}
+
+// Guess a common stride (aside from post-incrementing by access size) that
+// is suitable for "[Rn], Rm" addressing mode, if any
+static int32_t guessCustomAccessStride(ArrayRef<LdStInfo> Instructions) {
+  int32_t Stride = 0; // not decided
+  assert(!Instructions.empty());
+  for (auto I = Instructions.begin(), End = Instructions.end();
+       std::next(I) != End; ++I) {
+    int32_t ThisStride = std::next(I)->Offset - I->Offset;
+    // Check if "[Rn]!" addressing mode can be used
+    if (ThisStride == I->AccessSize)
+      continue;
+    // If this is the first instruction requiring a register operand,
+    // request this stride value
+    if (Stride == 0)
+      Stride = ThisStride;
+    // If multiple different stride values have to be used,
+    // conservatively refrain from using "[Rn], Rm" addressing mode
+    if (Stride != ThisStride)
+      return 0;
+  }
+  return Stride;
+}
+
+// Rewrite a memory address used by Second to use address of First incremented
+// by a constant value
+static bool rewriteAddressCalculation(LdStInfo &First, LdStInfo &Second,
+                                      int32_t RegStride,
+                                      const TargetLibraryInfo &TLI) {
+  LLVM_DEBUG(dbgs() << "Rewriting load/store for post-indexing: ";
+             Second.LdSt->dump());
+
+  IRBuilder<> IRB(Second.LdSt);
+  const DataLayout &DL = Second.LdSt->getModule()->getDataLayout();
+
+  int32_t Stride = Second.Offset - First.Offset;
+  if (Stride != First.AccessSize && Stride != RegStride)
+    return false;
+
+  // In case GEPOperand matched ContantExpr, replace it by instruction to
+  // prevent folding
+  if (auto Const = dyn_cast<ConstantExpr>(First.getBaseOperand())) {
+    auto Inst = Const->getAsInstruction();
+    Inst->insertBefore(First.LdSt);
+    First.LdSt->replaceUsesOfWith(Const, Inst);
+  }
+
+  Value *FirstBase = First.getBaseOperand();
+  Value *OldSecondBase = Second.getBaseOperand();
+  PointerType *FirstBaseTy = cast<PointerType>(FirstBase->getType());
+  PointerType *SecondBaseTy = cast<PointerType>(OldSecondBase->getType());
+  assert(FirstBaseTy->getAddressSpace() == 0 && "Unexpected address space");
+  assert(SecondBaseTy->getAddressSpace() == 0 && "Unexpected address space");
+
+  int32_t FirstElementSize =
+      DL.getTypeSizeInBits(FirstBaseTy->getElementType()) / 8;
+
+  Value *NewSecondBase;
+  if (FirstBaseTy == SecondBaseTy && Stride % FirstElementSize == 0) {
+    int32_t ElementStride = Stride / FirstElementSize;
+    Type *EltTy = FirstBaseTy->getPointerElementType();
+    NewSecondBase =
+        IRB.CreateConstGEP1_32(EltTy, FirstBase, ElementStride, "postinc");
+  } else {
+    Value *FirstBaseBytePtr =
+        IRB.CreateBitCast(FirstBase, IRB.getInt8PtrTy(), "oldbase.byteptr");
+    Value *NewSecondBaseBytePtr = IRB.CreateConstGEP1_32(
+        IRB.getInt8Ty(), FirstBaseBytePtr, Stride, "postinc.byteptr");
+    NewSecondBase =
+        IRB.CreateBitCast(NewSecondBaseBytePtr, SecondBaseTy, "postinc");
+  }
+  Second.LdSt->replaceUsesOfWith(OldSecondBase, NewSecondBase);
+  LLVM_DEBUG(dbgs() << "New load/store: "; Second.LdSt->dump());
+  RecursivelyDeleteTriviallyDeadInstructions(OldSecondBase, &TLI, nullptr);
+  return true;
+}
+
+static bool isProfitable(const SmallVectorImpl<LdStInfo> &Instructions,
+                         int32_t RegStride) {
+  if (Instructions.size() < 2)
+    return false;
+
+  unsigned Matches = 1; // start from 1 since we look for pairs
+  for (auto I = Instructions.begin(), End = Instructions.end();
+       std::next(I) != End; ++I) {
+    int32_t Stride = std::next(I)->Offset - I->Offset;
+    if (Stride == I->AccessSize || Stride == RegStride)
+      Matches++;
+  }
+
+  if (Matches < 4 && Instructions.size() > Matches) {
+    // Bail out if there are other users of the base pointer and not a
+    // lot of consecutive accesses.
+    return false;
+  }
+  return true;
+}
+
+static bool runOnBasicBlock(BasicBlock &BB, const TargetLibraryInfo &TLI) {
+  DenseMap<Value *, SmallVector<LdStInfo, 16>> LdStMap;
+  const DataLayout &DL = BB.getModule()->getDataLayout();
+
+  // Collect relevant load/store instructions, grouped by guessed base address
+  for (auto &I : BB) {
+    Type *ValueTy;
+    unsigned BaseOperandIndex;
+    std::tie(ValueTy, BaseOperandIndex) = getDataTypeAndBaseIndex(&I);
+    if (!ValueTy || !ValueTy->isVectorTy())
+      continue;
+
+    unsigned AccessSize = DL.getTypeSizeInBits(ValueTy) / 8;
+    if (isPowerOf2_32(AccessSize)) {
+      LdStInfo LSI(DL, &I, BaseOperandIndex, AccessSize);
+      LdStMap[LSI.IndirectBase].push_back(std::move(LSI));
+      LLVM_DEBUG(dbgs() << "found load/store, base: " << LSI.IndirectBase
+                        << ", offset: " << LSI.Offset << '\n'
+                        << I << '\n');
+    }
+  }
+
+  // For each group, form a chain of address increments
+  bool Modified = false;
+  for (auto BaseAndWorklist : LdStMap) {
+    auto Worklist = BaseAndWorklist.second;
+    assert(!Worklist.empty());
+    int32_t RegStride = guessCustomAccessStride(Worklist);
+    if (!isProfitable(Worklist, RegStride)) {
+      LLVM_DEBUG(dbgs() << "not profitable to transform this ld/st group: "
+                        << BaseAndWorklist.first << '\n');
+      continue;
+    }
+    for (auto I = Worklist.begin(), E = Worklist.end(); std::next(I) != E;
+         ++I) {
+      Modified |= rewriteAddressCalculation(*I, *std::next(I), RegStride, TLI);
+    }
+  }
+  return Modified;
+}
+
+bool ARMPostIndexingOpt::runOnFunction(Function &F) {
+  if (skipFunction(F))
+    return false;
+  // If MVE is available, skip this function.
+  const auto &TPC = getAnalysis<TargetPassConfig>();
+  const auto &TM = TPC.getTM<TargetMachine>();
+  const auto &STI = TM.getSubtarget<ARMSubtarget>(F);
+  if (STI.hasMVEIntegerOps())
+    return false;
+
+  const auto &TLI = getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
+
+  bool Modified = false;
+  for (auto &BB : F)
+    Modified |= runOnBasicBlock(BB, TLI);
+  return Modified;
+}
+
+FunctionPass *llvm::createARMPostIndexingOptimizationPass() {
+  return new ARMPostIndexingOpt();
+}

llvm/lib/Target/ARM/ARMTargetMachine.cpp

extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeARMTarget() {
   RegisterTargetMachine<ARMLETargetMachine> X(getTheARMLETarget());
   RegisterTargetMachine<ARMLETargetMachine> A(getTheThumbLETarget());
   RegisterTargetMachine<ARMBETargetMachine> Y(getTheARMBETarget());
   RegisterTargetMachine<ARMBETargetMachine> B(getTheThumbBETarget());
 
   PassRegistry &Registry = *PassRegistry::getPassRegistry();
   initializeGlobalISel(Registry);
   initializeARMLoadStoreOptPass(Registry);
+  initializeARMPostIndexingOptPass(Registry);
   initializeARMPreAllocLoadStoreOptPass(Registry);
   initializeARMParallelDSPPass(Registry);
   initializeARMConstantIslandsPass(Registry);
   initializeARMExecutionDomainFixPass(Registry);
   initializeARMExpandPseudoPass(Registry);
   initializeThumb2SizeReducePass(Registry);
   initializeMVEVPTBlockPass(Registry);
   initializeMVETPAndVPTOptimisationsPass(Registry);
if (TM->getOptLevel() != CodeGenOpt::None) {
     // FIXME: IR passes can delete address-taken basic blocks, deleting
     // corresponding blockaddresses. ARMConstantPoolConstant holds references to
     // address-taken basic blocks which can be invalidated if the function
     // containing the blockaddress has already been codegen'd and the basic
     // block is removed. Work around this by forcing all IR passes to run before
     // any ISel takes place. We should have a more principled way of handling
     // this. See D99707 for more details.
     addPass(createBarrierNoopPass());
+    addPass(createARMPostIndexingOptimizationPass());
   }
 
   return false;
 }
 
 bool ARMPassConfig::addInstSelector() {
   addPass(createARMISelDag(getARMTargetMachine(), getOptLevel()));
   return false;

llvm/lib/Target/ARM/CMakeLists.txt

add_llvm_target(ARMCodeGen
   ARMLoadStoreOptimizer.cpp
   ARMLowOverheadLoops.cpp
   ARMBlockPlacement.cpp
   ARMMCInstLower.cpp
   ARMMachineFunctionInfo.cpp
   ARMMacroFusion.cpp
   ARMRegisterInfo.cpp
   ARMOptimizeBarriersPass.cpp
+  ARMPostIndexingOptimizer.cpp
   ARMRegisterBankInfo.cpp
   ARMSelectionDAGInfo.cpp
   ARMSLSHardening.cpp
   ARMSubtarget.cpp
   ARMTargetMachine.cpp
   ARMTargetObjectFile.cpp
   ARMTargetTransformInfo.cpp
   MLxExpansionPass.cpp

llvm/test/CodeGen/ARM/O3-pipeline.ll

 ; CHECK-NEXT:      Lazy Block Frequency Analysis
 ; CHECK-NEXT:      Optimization Remark Emitter
 ; CHECK-NEXT:      Hardware Loop Insertion
 ; CHECK-NEXT:      Scalar Evolution Analysis
 ; CHECK-NEXT:      Loop Pass Manager
 ; CHECK-NEXT:        Transform predicated vector loops to use MVE tail predication
 ; CHECK-NEXT:      A No-Op Barrier Pass
 ; CHECK-NEXT:      FunctionPass Manager
+; CHECK-NEXT:      ARM post-indexed access optimizer
 ; CHECK-NEXT:      Safe Stack instrumentation pass
 ; CHECK-NEXT:      Insert stack protectors
 ; CHECK-NEXT:      Module Verifier
 ; CHECK-NEXT:      Dominator Tree Construction
 ; CHECK-NEXT:      Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:      Function Alias Analysis Results
 ; CHECK-NEXT:      Natural Loop Information
 ; CHECK-NEXT:      Post-Dominator Tree Construction

llvm/test/CodeGen/ARM/arm-post-indexing-opt.ll

+; RUN: llc -o - < %s | FileCheck --check-prefix=ASM %s
+; RUN: opt --arm-post-indexing-opt -S -o - < %s | FileCheck --check-prefix=IR %s
+
+target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
+target triple = "armv8-unknown-linux-gnueabihf"
+
+define <4 x float> @test(float* %A) {
+  %X.ptr = bitcast float* %A to <4 x float>*
+  %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+  %Y.ptr.elt = getelementptr inbounds float, float* %A, i32 4
+  %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+  %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+  %Z.ptr.elt = getelementptr inbounds float, float* %A, i32 8
+  %Z.ptr = bitcast float* %Z.ptr.elt to <4 x float>*
+  %Z = load <4 x float>, <4 x float>* %Z.ptr, align 4
+  %tmp.sum = fadd <4 x float> %X, %Y
+  %sum = fadd <4 x float> %tmp.sum, %Z
+  ret <4 x float> %sum
+}
+; IR-LABEL: define <4 x float> @test(float* %A) {
+; IR-NEXT:    %X.ptr = bitcast float* %A to <4 x float>*
+; IR-NEXT:    %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+; IR-NEXT:    %postinc = getelementptr <4 x float>, <4 x float>* %X.ptr, i32 1
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %postinc, align 4
+; IR-NEXT:    %postinc1 = getelementptr <4 x float>, <4 x float>* %postinc, i32 1
+; IR-NEXT:    %Z = load <4 x float>, <4 x float>* %postinc1, align 4
+; IR-NEXT:    %tmp.sum = fadd <4 x float> %X, %Y
+; IR-NEXT:    %sum = fadd <4 x float> %tmp.sum, %Z
+; IR-NEXT:    ret <4 x float> %sum
+; IR-NEXT:  }
+
+;ASM-LABEL: test:
+;ASM-NOT: add
+;ASM-NOT: sub
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE:[0-9]+]]]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+
+define <4 x float> @test_stride(float* %A) {
+  %X.ptr = bitcast float* %A to <4 x float>*
+  %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+  %Y.ptr.elt = getelementptr inbounds float, float* %A, i32 6
+  %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+  %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+  %Z.ptr.elt = getelementptr inbounds float, float* %A, i32 12
+  %Z.ptr = bitcast float* %Z.ptr.elt to <4 x float>*
+  %Z = load <4 x float>, <4 x float>* %Z.ptr, align 4
+  %tmp.sum = fadd <4 x float> %X, %Y
+  %sum = fadd <4 x float> %tmp.sum, %Z
+  ret <4 x float> %sum
+}
+
+; IR-LABEL: define <4 x float> @test_stride(float* %A) {
+; IR-NEXT:    %X.ptr = bitcast float* %A to <4 x float>*
+; IR-NEXT:    %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+; IR-NEXT:    %oldbase.byteptr = bitcast <4 x float>* %X.ptr to i8*
+; IR-NEXT:    %postinc.byteptr = getelementptr i8, i8* %oldbase.byteptr, i32 24
+; IR-NEXT:    %postinc = bitcast i8* %postinc.byteptr to <4 x float>*
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %postinc, align 4
+; IR-NEXT:    %oldbase.byteptr1 = bitcast <4 x float>* %postinc to i8*
+; IR-NEXT:    %postinc.byteptr2 = getelementptr i8, i8* %oldbase.byteptr1, i32 24
+; IR-NEXT:    %postinc3 = bitcast i8* %postinc.byteptr2 to <4 x float>*
+; IR-NEXT:    %Z = load <4 x float>, <4 x float>* %postinc3, align 4
+; IR-NEXT:    %tmp.sum = fadd <4 x float> %X, %Y
+; IR-NEXT:    %sum = fadd <4 x float> %tmp.sum, %Z
+; IR-NEXT:    ret <4 x float> %sum
+; IR-NEXT:  }
+
+; ASM-LABEL: test_stride:
+; ASM: mov r[[STRIDE:[0-9]+]], #24
+; ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE:[0-9]+]]], r[[STRIDE]]
+; ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]], r[[STRIDE]]
+; ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+
+define <4 x float> @test_stride_mixed(float* %A) {
+  %X.ptr = bitcast float* %A to <4 x float>*
+  %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+  %Y.ptr.elt = getelementptr inbounds float, float* %A, i32 6
+  %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+  %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+  %Z.ptr.elt = getelementptr inbounds float, float* %A, i32 10
+  %Z.ptr = bitcast float* %Z.ptr.elt to <4 x float>*
+  %Z = load <4 x float>, <4 x float>* %Z.ptr, align 4
+  %tmp.sum = fadd <4 x float> %X, %Y
+  %sum = fadd <4 x float> %tmp.sum, %Z
+  ret <4 x float> %sum
+}
+
+; IR-LABEL: define <4 x float> @test_stride_mixed(float* %A) {
+; IR-NEXT:    %X.ptr = bitcast float* %A to <4 x float>*
+; IR-NEXT:    %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+; IR-NEXT:    %oldbase.byteptr = bitcast <4 x float>* %X.ptr to i8*
+; IR-NEXT:    %postinc.byteptr = getelementptr i8, i8* %oldbase.byteptr, i32 24
+; IR-NEXT:    %postinc = bitcast i8* %postinc.byteptr to <4 x float>*
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %postinc, align 4
+; IR-NEXT:    %postinc1 = getelementptr <4 x float>, <4 x float>* %postinc, i32 1
+; IR-NEXT:    %Z = load <4 x float>, <4 x float>* %postinc1, align 4
+; IR-NEXT:    %tmp.sum = fadd <4 x float> %X, %Y
+; IR-NEXT:    %sum = fadd <4 x float> %tmp.sum, %Z
+; IR-NEXT:    ret <4 x float> %sum
+; IR-NEXT:  }
+
+; ASM-LABEL: test_stride_mixed:
+; ASM: mov r[[STRIDE:[0-9]+]], #24
+; ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE:[0-9]+]]], r[[STRIDE]]
+; ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+; ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+
+; Refrain from using multiple stride registers
+define <4 x float> @test_stride_noop(float* %A) {
+  %X.ptr = bitcast float* %A to <4 x float>*
+  %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+  %Y.ptr.elt = getelementptr inbounds float, float* %A, i32 6
+  %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+  %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+  %Z.ptr.elt = getelementptr inbounds float, float* %A, i32 14
+  %Z.ptr = bitcast float* %Z.ptr.elt to <4 x float>*
+  %Z = load <4 x float>, <4 x float>* %Z.ptr, align 4
+  %tmp.sum = fadd <4 x float> %X, %Y
+  %sum = fadd <4 x float> %tmp.sum, %Z
+  ret <4 x float> %sum
+}
+
+; IR-LABEL: define <4 x float> @test_stride_noop(float* %A) {
+; IR-NEXT:    %X.ptr = bitcast float* %A to <4 x float>*
+; IR-NEXT:    %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+; IR-NEXT:    %Y.ptr.elt = getelementptr inbounds float, float* %A, i32 6
+; IR-NEXT:    %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+; IR-NEXT:    %Z.ptr.elt = getelementptr inbounds float, float* %A, i32 14
+; IR-NEXT:    %Z.ptr = bitcast float* %Z.ptr.elt to <4 x float>*
+; IR-NEXT:    %Z = load <4 x float>, <4 x float>* %Z.ptr, align 4
+; IR-NEXT:    %tmp.sum = fadd <4 x float> %X, %Y
+; IR-NEXT:    %sum = fadd <4 x float> %tmp.sum, %Z
+; IR-NEXT:    ret <4 x float> %sum
+; IR-NEXT:  }
+
+define <4 x float> @test_positive_initial_offset(float* %A) {
+  %X.ptr.elt = getelementptr inbounds float, float* %A, i32 8
+  %X.ptr = bitcast float* %X.ptr.elt to <4 x float>*
+  %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+  %Y.ptr.elt = getelementptr inbounds float, float* %A, i32 12
+  %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+  %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+  %Z.ptr.elt = getelementptr inbounds float, float* %A, i32 16
+  %Z.ptr = bitcast float* %Z.ptr.elt to <4 x float>*
+  %Z = load <4 x float>, <4 x float>* %Z.ptr, align 4
+  %tmp.sum = fadd <4 x float> %X, %Y
+  %sum = fadd <4 x float> %tmp.sum, %Z
+  ret <4 x float> %sum
+}
+
+; IR-LABEL: define <4 x float> @test_positive_initial_offset(float* %A) {
+; IR-NEXT:    %X.ptr.elt = getelementptr inbounds float, float* %A, i32 8
+; IR-NEXT:    %X.ptr = bitcast float* %X.ptr.elt to <4 x float>*
+; IR-NEXT:    %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+; IR-NEXT:    %postinc = getelementptr <4 x float>, <4 x float>* %X.ptr, i32 1
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %postinc, align 4
+; IR-NEXT:    %postinc1 = getelementptr <4 x float>, <4 x float>* %postinc, i32 1
+; IR-NEXT:    %Z = load <4 x float>, <4 x float>* %postinc1, align 4
+; IR-NEXT:    %tmp.sum = fadd <4 x float> %X, %Y
+; IR-NEXT:    %sum = fadd <4 x float> %tmp.sum, %Z
+; IR-NEXT:    ret <4 x float> %sum
+; IR-NEXT:  }
+
+;ASM-LABEL: test_positive_initial_offset:
+;ASM: add r[[BASE:[0-9]+]], r[[BASE]], #32
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+
+define <4 x float> @test_negative_initial_offset(float* %A) {
+  %X.ptr.elt = getelementptr inbounds float, float* %A, i32 -16
+  %X.ptr = bitcast float* %X.ptr.elt to <4 x float>*
+  %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+  %Y.ptr.elt = getelementptr inbounds float, float* %A, i32 -12
+  %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+  %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+  %Z.ptr.elt = getelementptr inbounds float, float* %A, i32 -8
+  %Z.ptr = bitcast float* %Z.ptr.elt to <4 x float>*
+  %Z = load <4 x float>, <4 x float>* %Z.ptr, align 4
+  %tmp.sum = fadd <4 x float> %X, %Y
+  %sum = fadd <4 x float> %tmp.sum, %Z
+  ret <4 x float> %sum
+}
+
+; IR-LABEL: define <4 x float> @test_negative_initial_offset(float* %A) {
+; IR-NEXT:    %X.ptr.elt = getelementptr inbounds float, float* %A, i32 -16
+; IR-NEXT:    %X.ptr = bitcast float* %X.ptr.elt to <4 x float>*
+; IR-NEXT:    %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+; IR-NEXT:    %postinc = getelementptr <4 x float>, <4 x float>* %X.ptr, i32 1
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %postinc, align 4
+; IR-NEXT:    %postinc1 = getelementptr <4 x float>, <4 x float>* %postinc, i32 1
+; IR-NEXT:    %Z = load <4 x float>, <4 x float>* %postinc1, align 4
+; IR-NEXT:    %tmp.sum = fadd <4 x float> %X, %Y
+; IR-NEXT:    %sum = fadd <4 x float> %tmp.sum, %Z
+; IR-NEXT:    ret <4 x float> %sum
+; IR-NEXT:  }
+
+;ASM-LABEL: test_negative_initial_offset:
+;ASM: sub r[[BASE:[0-9]+]], r[[BASE]], #64
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+
+@global_float_array = external global [128 x float], align 4
+define <4 x float> @test_global() {
+  %X = load <4 x float>, <4 x float>* bitcast (float* getelementptr inbounds ([128 x float], [128 x float]* @global_float_array, i32 0, i32 8) to <4 x float>*), align 4
+  %Y = load <4 x float>, <4 x float>* bitcast (float* getelementptr inbounds ([128 x float], [128 x float]* @global_float_array, i32 0, i32 12) to <4 x float>*), align 4
+  %Z = load <4 x float>, <4 x float>* bitcast (float* getelementptr inbounds ([128 x float], [128 x float]* @global_float_array, i32 0, i32 16) to <4 x float>*), align 4
+  %tmp.sum = fadd <4 x float> %X, %Y
+  %sum = fadd <4 x float> %tmp.sum, %Z
+  ret <4 x float> %sum
+}
+
+; IR-LABEL: define <4 x float> @test_global() {
+; IR-NEXT:    %1 = bitcast float* getelementptr inbounds ([128 x float], [128 x float]* @global_float_array, i32 0, i32 8) to <4 x float>*
+; IR-NEXT:    %X = load <4 x float>, <4 x float>* %1, align 4
+; IR-NEXT:    %postinc = getelementptr <4 x float>, <4 x float>* %1, i32 1
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %postinc, align 4
+; IR-NEXT:    %postinc1 = getelementptr <4 x float>, <4 x float>* %postinc, i32 1
+; IR-NEXT:    %Z = load <4 x float>, <4 x float>* %postinc1, align 4
+; IR-NEXT:    %tmp.sum = fadd <4 x float> %X, %Y
+; IR-NEXT:    %sum = fadd <4 x float> %tmp.sum, %Z
+; IR-NEXT:    ret <4 x float> %sum
+; IR-NEXT:  }
+
+;ASM-LABEL: test_global:
+;ASM: add r[[BASE:[0-9]+]], r[[BASE]], #32
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+
+define <4 x float> @test_stack() {
+; Use huge alignment to test that ADD would not be converted to OR
+  %array = alloca [32 x float], align 128
+  %arraydecay = getelementptr inbounds [32 x float], [32 x float]* %array, i32 0, i32 0
+  call void @external_function(float* %arraydecay)
+  %X.ptr = bitcast [32 x float]* %array to <4 x float>*
+  %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+  %Y.ptr.elt = getelementptr inbounds [32 x float], [32 x float]* %array, i32 0, i32 4
+  %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+  %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+  %Z.ptr.elt = getelementptr inbounds [32 x float], [32 x float]* %array, i32 0, i32 8
+  %Z.ptr = bitcast float* %Z.ptr.elt to <4 x float>*
+  %Z = load <4 x float>, <4 x float>* %Z.ptr, align 4
+  %tmp.sum = fadd <4 x float> %X, %Y
+  %sum = fadd <4 x float> %tmp.sum, %Z
+  ret <4 x float> %sum
+}
+
+; IR-LABEL: define <4 x float> @test_stack() {
+; IR-NEXT:    %array = alloca [32 x float], align 128
+; IR-NEXT:    %arraydecay = getelementptr inbounds [32 x float], [32 x float]* %array, i32 0, i32 0
+; IR-NEXT:    call void @external_function(float* %arraydecay)
+; IR-NEXT:    %X.ptr = bitcast [32 x float]* %array to <4 x float>*
+; IR-NEXT:    %X = load <4 x float>, <4 x float>* %X.ptr, align 4
+; IR-NEXT:    %postinc = getelementptr <4 x float>, <4 x float>* %X.ptr, i32 1
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %postinc, align 4
+; IR-NEXT:    %postinc1 = getelementptr <4 x float>, <4 x float>* %postinc, i32 1
+; IR-NEXT:    %Z = load <4 x float>, <4 x float>* %postinc1, align 4
+; IR-NEXT:    %tmp.sum = fadd <4 x float> %X, %Y
+; IR-NEXT:    %sum = fadd <4 x float> %tmp.sum, %Z
+; IR-NEXT:    ret <4 x float> %sum
+; IR-NEXT:  }
+
+;ASM-LABEL: test_stack:
+;ASM: bl external_function
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE:[0-9]+]]:128]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]:128]!
+;ASM: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+
+define <2 x double> @test_double(double* %A) {
+  %X.ptr.elt = getelementptr inbounds double, double* %A, i32 8
+  %X.ptr = bitcast double* %X.ptr.elt to <2 x double>*
+  %X = load <2 x double>, <2 x double>* %X.ptr, align 8
+  %Y.ptr.elt = getelementptr inbounds double, double* %A, i32 10
+  %Y.ptr = bitcast double* %Y.ptr.elt to <2 x double>*
+  %Y = load <2 x double>, <2 x double>* %Y.ptr, align 8
+  %Z.ptr.elt = getelementptr inbounds double, double* %A, i32 12
+  %Z.ptr = bitcast double* %Z.ptr.elt to <2 x double>*
+  %Z = load <2 x double>, <2 x double>* %Z.ptr, align 8
+  %tmp.sum = fadd <2 x double> %X, %Y
+  %sum = fadd <2 x double> %tmp.sum, %Z
+  ret <2 x double> %sum
+}
+
+; IR-LABEL: define <2 x double> @test_double(double* %A) {
+; IR-NEXT:    %X.ptr.elt = getelementptr inbounds double, double* %A, i32 8
+; IR-NEXT:    %X.ptr = bitcast double* %X.ptr.elt to <2 x double>*
+; IR-NEXT:    %X = load <2 x double>, <2 x double>* %X.ptr, align 8
+; IR-NEXT:    %postinc = getelementptr <2 x double>, <2 x double>* %X.ptr, i32 1
+; IR-NEXT:    %Y = load <2 x double>, <2 x double>* %postinc, align 8
+; IR-NEXT:    %postinc1 = getelementptr <2 x double>, <2 x double>* %postinc, i32 1
+; IR-NEXT:    %Z = load <2 x double>, <2 x double>* %postinc1, align 8
+; IR-NEXT:    %tmp.sum = fadd <2 x double> %X, %Y
+; IR-NEXT:    %sum = fadd <2 x double> %tmp.sum, %Z
+; IR-NEXT:    ret <2 x double> %sum
+; IR-NEXT:  }
+
+;ASM-LABEL: test_double:
+;ASM: add r[[BASE:[0-9]+]], r[[BASE]], #64
+;ASM: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+;ASM: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+
+define void @test_various_instructions(float* %A) {
+  %X.ptr = bitcast float* %A to i8*
+  %X = call <4 x float> @llvm.arm.neon.vld1.v4f32.p0i8(i8* %X.ptr, i32 1)
+  %Y.ptr.elt = getelementptr inbounds float, float* %A, i32 4
+  %Y.ptr = bitcast float* %Y.ptr.elt to <4 x float>*
+  %Y = load <4 x float>, <4 x float>* %Y.ptr, align 4
+  %Z.ptr.elt = getelementptr inbounds float, float* %A, i32 8
+  %Z.ptr = bitcast float* %Z.ptr.elt to i8*
+  %Z = fadd <4 x float> %X, %Y
+  tail call void @llvm.arm.neon.vst1.p0i8.v4f32(i8* nonnull %Z.ptr, <4 x float> %Z, i32 4)
+  ret void
+}
+
+; IR-LABEL: define void @test_various_instructions(float* %A) {
+; IR-NEXT:    %X.ptr = bitcast float* %A to i8*
+; IR-NEXT:    %X = call <4 x float> @llvm.arm.neon.vld1.v4f32.p0i8(i8* %X.ptr, i32 1)
+; IR-NEXT:    %postinc.byteptr = getelementptr i8, i8* %X.ptr, i32 16
+; IR-NEXT:    %postinc = bitcast i8* %postinc.byteptr to <4 x float>*
+; IR-NEXT:    %Y = load <4 x float>, <4 x float>* %postinc, align 4
+; IR-NEXT:    %Z = fadd <4 x float> %X, %Y
+; IR-NEXT:    %oldbase.byteptr = bitcast <4 x float>* %postinc to i8*
+; IR-NEXT:    %postinc.byteptr1 = getelementptr i8, i8* %oldbase.byteptr, i32 16
+; IR-NEXT:    tail call void @llvm.arm.neon.vst1.p0i8.v4f32(i8* nonnull %postinc.byteptr1, <4 x float> %Z, i32 4)
+; IR-NEXT:    ret void
+; IR-NEXT:  }
+
+; ASM-LABEL: test_various_instructions:
+; ASM:       @ %bb.0:
+; ASM-NEXT:  vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE:[0-9]+]]]!
+; ASM-NEXT:  vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]!
+; ASM-NEXT:  vadd.f32
+; ASM-NEXT:  vst1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[BASE]]]
+; ASM-NEXT:  bx lr
+
+declare void @external_function(float*)
+declare <4 x float> @llvm.arm.neon.vld1.v4f32.p0i8(i8*, i32) nounwind readonly
+declare void @llvm.arm.neon.vst1.p0i8.v4f32(i8*, <4 x float>, i32) nounwind argmemonly

llvm/test/CodeGen/ARM/misched-fusion-aes.ll

 
 ; CHECK-LABEL: aesea:
 ; CHECK: aese.8 [[QA:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesmc.8 {{q[0-9][0-9]?}}, [[QA]]
 
 ; CHECK: aese.8 [[QB:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesmc.8 {{q[0-9][0-9]?}}, [[QB]]
 
-; CHECK: aese.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 ; CHECK: aese.8 [[QC:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesmc.8 {{q[0-9][0-9]?}}, [[QC]]
+; CHECK: aese.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 
 ; CHECK: aese.8 [[QD:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesmc.8 {{q[0-9][0-9]?}}, [[QD]]
 
 ; CHECK: aese.8 [[QE:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesmc.8 {{q[0-9][0-9]?}}, [[QE]]
 
+; CHECK: aese.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 ; CHECK: aese.8 [[QF:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesmc.8 {{q[0-9][0-9]?}}, [[QF]]
 
 ; CHECK: aese.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 ; CHECK: aese.8 [[QG:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesmc.8 {{q[0-9][0-9]?}}, [[QG]]
 
-; CHECK: aese.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
-
 ; CHECK: aese.8 [[QH:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesmc.8 {{q[0-9][0-9]?}}, [[QH]]
 }
 
 define void @aesda(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   %d0 = load <16 x i8>, <16 x i8>* %a0
   %a1 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 1
   %d1 = load <16 x i8>, <16 x i8>* %a1

 
 ; CHECK-LABEL: aesda:
 ; CHECK: aesd.8 [[QA:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesimc.8 {{q[0-9][0-9]?}}, [[QA]]
 
 ; CHECK: aesd.8 [[QB:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesimc.8 {{q[0-9][0-9]?}}, [[QB]]
 
-; CHECK: aesd.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 ; CHECK: aesd.8 [[QC:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesimc.8 {{q[0-9][0-9]?}}, [[QC]]
+; CHECK: aesd.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 
 ; CHECK: aesd.8 [[QD:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesimc.8 {{q[0-9][0-9]?}}, [[QD]]
 
 ; CHECK: aesd.8 [[QE:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesimc.8 {{q[0-9][0-9]?}}, [[QE]]
 
+; CHECK: aesd.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 ; CHECK: aesd.8 [[QF:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesimc.8 {{q[0-9][0-9]?}}, [[QF]]
 
 ; CHECK: aesd.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 ; CHECK: aesd.8 [[QG:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesimc.8 {{q[0-9][0-9]?}}, [[QG]]
 
-; CHECK: aesd.8 {{q[0-9][0-9]?}}, {{q[0-9][0-9]?}}
 ; CHECK: aesd.8 [[QH:q[0-9][0-9]?]], {{q[0-9][0-9]?}}
 ; CHECK-NEXT: aesimc.8 {{q[0-9][0-9]?}}, [[QH]]
 }
 
 define void @aes_load_store(<16 x i8> *%p1, <16 x i8> *%p2 , <16 x i8> *%p3) {
 entry:
   %x1 = alloca <16 x i8>, align 16
   %x2 = alloca <16 x i8>, align 16

llvm/test/Transforms/LoopStrengthReduce/ARM/ivchain-ARM.ll

for.body:                                         ; preds = %for.body, %entry
   br i1 %exitcond.3, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body
   ret void
 }
 
 ; @testNeon is an important example of the nead for ivchains.
 ;
-; Currently we have two extra add.w's that keep the store address
-; live past the next increment because ISEL is unfortunately undoing
-; the store chain. ISEL also fails to convert all but one of the stores to
-; post-increment addressing. However, the loads should use
-; post-increment addressing, no add's or add.w's beyond the three
-; mentioned. Most importantly, there should be no spills or reloads!
+; Loads and stores should use post-increment addressing, no add's or add.w's.
+; Most importantly, there should be no spills or reloads!
 ;
 ; A9: testNeon:
 ; A9: %.lr.ph
-; A9: add.w r
 ; A9-NOT: lsl.w
 ; A9-NOT: {{ldr|str|adds|add r}}
-; A9: vst1.8 {{.*}} [r{{[0-9]+}}], r{{[0-9]+}}
-; A9: add.w r
-; A9-NOT: {{ldr|str|adds|add r}}
 ; A9-NOT: add.w r
 ; A9: bne
 define hidden void @testNeon(i8* %ref_data, i32 %ref_stride, i32 %limit, <16 x i8>* nocapture %data) nounwind optsize {
   %1 = icmp sgt i32 %limit, 0
   br i1 %1, label %.lr.ph, label %45
 
 .lr.ph:                                           ; preds = %0
   %2 = shl nsw i32 %ref_stride, 1