[AArch64] Improve load/store optimizer to handle LDUR + LDR.

[ab]

Summary

This patch allows the mixing of scaled and unscaled load/stores to form
load/store pairs.

PR24465

[mcrosier]

I haven't done any testing beyond basic lit tests. Hopefully, I'll have correctness/perf testing in flight by tomorrow. Regardless, I think this is in good enough shape to review.

[kristof.beyls]

test/CodeGen/AArch64/ldp-stp-scaled-unscaled-pairs.ll

2

My preference is to use -march=aarch64 instead of -march=arm64 as much as possible, as aarch64 is the official name for the architecture.
Is there a specific reason -mcpu=cortex-a57 needs to be added to the test line? I don't have a strong opinion on whether it should be there or not, just wondering.

82

Looking at this test case, I see that before this patch, the following code is produced:

ldur    x8, [x0, #-8]
ldr      x9, [x0]

. If I'm not mistaken, ldur x8, [x0, #-8] has the same functionality as ldr x8, [x0, #-1]? If so, wouldn't it be better to make sure
we produce ldr instead of ldur in the first place?
If we would do that, and there still is a good reason to have special code to convert LDR + LDUR into LDP, I guess none of the above test cases really show that (although I haven't investigated every single test case in detail)?

[kristof.beyls]

test/CodeGen/AArch64/ldp-stp-scaled-unscaled-pairs.ll

82

D'oh! The ARMARM clearly states that the scaled immediate offsets in ldr x8, [x0, #imm] can only be positive/unsigned. Please ignore my comment above!

[mcrosier]

Addressed Kristof's comments. Included full diff.

[mcrosier]

Fix test7 which I modified/broke, while doing additional testing.

[mzolotukhin]

Hi Chad,

I checked on a small testcase, and with this patch we do merge STUR and STR. There is one potential issue though: in some cases we intentionally split STP into two STUR/STR, as there is a big performance penalty if STP crosses a cache line (see e.g. performSTORECombine in AArch64ISelLowering.cpp and getMemoryOpCost in AArch64TargetTransformInfo.cpp). So, I think we might want to perform this combining only for non-temporal or known-to-be-well-aligned memory accesses. What do you think, does it make sense?

Thanks,
Michael

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

395

This looks strange. Is it expected that scaled and unscaled offset differ by MemSize^2? (in one case you multiply by MemSize, in the other - divide)

656

Same here.

[mcrosier]

Hi Michael,
So, you're saying that performSTORECombine splits 16B stores that cross line boundaries for performance reasons. AFAICT, this combine is only enabled with Cyclone. Later the AArch64LoadStore optimizer runs and combines these split stores undoing the optimization performed during ISelLowering.

If I understand things correctly, I think this makes sense. However, I'd probably implement this in a separate patch and test it on other subtargets (e.g., A57). Sound reasonable?

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

395

My assumption is that a scaled offset is always the unscaled offset divide by the MemSize and conversely the unscaled offset is always the scaled offset divided by the MemSize; I believe this is valid.

I.e.,
Unscaled Offset = Scaled Offset * MemSize;
Scaled Offset = Unscaled Offset / MemSize;

What did I miss? I'm trying to simply the logic so that both offsets are either scaled or unscaled, but not a mix of the two. I'm guessing you're concerned my conversion is incorrect.

[mzolotukhin]

Hi Chad,

As far as I understand, currently ISelLowering splits the unaligned stores, from which we happen to get STUR and STR, which we can't combine to STP without this patch. With this patch, we'll be able to merge them back, so we'll undo that optimization.

I think that being able to combine STUR/STR to STP is good by itself, but from the patch alone we'll probably get regressions (AFAIR, the biggest one was matmul from LNT testsuite). That's my only concern, but I don't mind landing the patch and addressing this issue later if we actually observe it.

Also, I can't provide a high-quality review of this code, as I'm not very familiar with it, so I'd appreciate other people's feedback on the patch.

Thanks!

Michael

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

395

Yep, I was concerned that it was incorrect, but I just read the code incorrectly. Sorry for that!
(I thought that we do PairedOffset = UnscaledOffset*MemSize in one case and PairedOffset = UnscaleOffset/MemSize in the other).

[ab]

This LGTM (modulo nits), but let's see what the testsuite says first.

In D12116#231366, @mzolotukhin wrote:

As far as I understand, currently ISelLowering splits the unaligned stores, from which we happen to get STUR and STR, which we can't combine to STP without this patch. With this patch, we'll be able to merge them back, so we'll undo that optimization.

But this issue isn't specific to this patch, right? If the unaligned store was split to STR+STR we would have generated an STP even before this change. I agree we'll need to do something about this though, but separately, and for both mixed and non-mixed STR/STUR pairs.

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

390

MI -> Paired, FirstMI -> I ?

557–560

What about removing CanMergeOpc? It's assigned to several times with different values, and all could be folded into ifs/returns.

569–570

Same: fold the condition into the if, remove CanMergeOpc?

577–579

I see this is the only use of the get*From* helpers. What about:

getMatchingPairOpcode(Opc) == getMatchingPairOpcode(PairOpc)

That'll let us get rid of the added functions.

580–583

Same: fold the condition into the return, remove CanMergeOpc?

677

To make sure I understand: the previous code was "incorrect" in checking MI rather than FirstMI, but it didn't make a difference because we didn't allow mismatched opcodes, right?

test/CodeGen/AArch64/ldp-stp-scaled-unscaled-pairs.ll

9

Kill the "entry:"s ?

[mcrosier]

Thanks for the feedback, Ahmed.

In D12116#236954, @ab wrote:

This LGTM (modulo nits), but let's see what the testsuite says first.

Unfortunately, I don't have a setup that can test with the testsuite at the moment. That is in the works...

This transformation is very narrow and did not hit anything in Spec2000. Therefore, I'm going to move onto other work with a higher ROI. However, feel free to push this one along.

In D12116#231366, @mzolotukhin wrote:

As far as I understand, currently ISelLowering splits the unaligned stores, from which we happen to get STUR and STR, which we can't combine to STP without this patch. With this patch, we'll be able to merge them back, so we'll undo that optimization.

But this issue isn't specific to this patch, right? If the unaligned store was split to STR+STR we would have generated an STP even before this change. I agree we'll need to do something about this though, but separately, and for both mixed and non-mixed STR/STUR pairs.

That is absolutely correct! This patch only applies to the very narrow case of STR/STUR. The common case is when we're pairing STUR/STUR, which has already been committed.

[mcrosier]

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

677

I'm not sure the checking was "incorrect", but your latter comment is correct; because we didn't allow mismatches opcodes it didn't make a difference. However, because we're converting MI to be either Scaled or Unscaled to match FirstMI, we need to use IsUscaled in the context of this patch.

[mzolotukhin]

Hi Chad, Ahmed,

I ran some testing for this patch and found a bug there. The issue is that when we scale offset, we need to make sure that the original value is divisible by the scale factor. Here is a test to illustrate the issue:

define <2 x i64> @test4(i64* %p) nounwind {
  %a1 = bitcast i64* %p to <2 x i64>*
  %tmp1 = load <2 x i64>, < 2 x i64>* %a1, align 8           # Load <p[0], p[1]>

  %add.ptr2 = getelementptr inbounds i64, i64* %p, i64 3
  %a2 = bitcast i64* %add.ptr2 to <2 x i64>*
  %tmp2 = load <2 x i64>, <2 x i64>* %a2, align 8            # Load <p[3], p[4]>
  %add = add nsw <2 x i64> %tmp1, %tmp2
  ret <2 x i64> %add
}

The current patch will combine these two loads, which is incorrect.

Michael

[ab]

Thanks Michael. Chad, I'll fix that and resubmit, thanks for the patch!

[ab]

And this should catch the unscaled offset not being a multiple of the scale (with abundant asserts).

[mcrosier]

Thanks, Ahmed.

[mzolotukhin]

Hi,

I just finished LNT+externals run, and saw no failures. Thanks for the fix!

Michael

[mcrosier]

Thanks for seeing this one through, Ahmed. LGTM.

[mcrosier]

Committed r246769.

test/CodeGen/AArch64/ldp-stp-scaled-unscaled-pairs.ll

1

BTW, I added -aarch64-neon-syntax=apple to the final commit, so that this will pass on all platforms.

[mcrosier]

I speculatively reverted r246769 in r246782 as the compiler looks to be ICEing on Multisource/Benchmarks/tramp3d-v4.

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

AArch64LoadStoreOpt::mergePairedInsns(MachineBasicBlock::iterator I,
   // Insert our new paired instruction after whichever of the paired
   // instructions MergeForward indicates.
   MachineBasicBlock::iterator InsertionPoint = MergeForward ? Paired : I;
   // Also based on MergeForward is from where we copy the base register operand
   // so we get the flags compatible with the input code.
   const MachineOperand &BaseRegOp =
       MergeForward ? getLdStBaseOp(Paired) : getLdStBaseOp(I);
 
+  int Offset = getLdStOffsetOp(I).getImm();
+  int PairedOffset = getLdStOffsetOp(Paired).getImm();
+  bool PairedIsUnscaled = isUnscaledLdSt(Paired->getOpcode());
+
+  // We're trying to pair instructions that differ in how they are scaled.
+  // If I is scaled then scale the offset of Paired accordingly.
+  // Otherwise, do the opposite (i.e., make Paired's offset unscaled).
+  if (IsUnscaled != PairedIsUnscaled) {
+    int MemSize = getMemSize(Paired);
+    assert(!(PairedOffset % getMemSize(Paired)) &&
+           "Offset should be a multiple of the stride!");
+    PairedOffset =
+        PairedIsUnscaled ? PairedOffset / MemSize : PairedOffset * MemSize;
+  }
+
   // Which register is Rt and which is Rt2 depends on the offset order.
   MachineInstr *RtMI, *Rt2MI;
-  if (getLdStOffsetOp(I).getImm() ==
-      getLdStOffsetOp(Paired).getImm() + OffsetStride) {
+  if (Offset == PairedOffset + OffsetStride) {
     RtMI = Paired;
     Rt2MI = I;
     // Here we swapped the assumption made for SExtIdx.
     // I.e., we turn ldp I, Paired into ldp Paired, I.
     // Update the index accordingly.
     if (SExtIdx != -1)
       SExtIdx = (SExtIdx + 1) % 2;
   } else {
     RtMI = I;
     Rt2MI = Paired;
   }
-  // Handle Unscaled
+  // Scale the immediate offset, if necessary.
   int OffsetImm = getLdStOffsetOp(RtMI).getImm();
-  if (IsUnscaled && EnableAArch64UnscaledMemOp)
-    OffsetImm /= OffsetStride;
+  if (isUnscaledLdSt(RtMI->getOpcode()) && EnableAArch64UnscaledMemOp) {
+    assert(!(OffsetImm % getMemSize(RtMI)) &&
+           "Offset should be a multiple of the stride!");
+    OffsetImm /= getMemSize(RtMI);
+  }
 
   // Construct the new instruction.
   MachineInstrBuilder MIB = BuildMI(*I->getParent(), InsertionPoint,
                                     I->getDebugLoc(), TII->get(NewOpc))
                                 .addOperand(getLdStRegOp(RtMI))
                                 .addOperand(getLdStRegOp(Rt2MI))
                                 .addOperand(BaseRegOp)
                                 .addImm(OffsetImm);
if (MO.isDef()) {
         UsedRegs.set(*AI);
     }
   }
 }
 
 static bool inBoundsForPair(bool IsUnscaled, int Offset, int OffsetStride) {
   // Convert the byte-offset used by unscaled into an "element" offset used
   // by the scaled pair load/store instructions.
-  if (IsUnscaled)
+  if (IsUnscaled) {
+    // If the byte-offset isn't a multiple of the stride, there's no point
+    // trying to match it.
+    if (Offset % OffsetStride)
+      return false;
     Offset /= OffsetStride;
+  }
 
   return Offset <= 63 && Offset >= -64;
 }
 
 // Do alignment, specialized to power of 2 and for signed ints,
 // avoiding having to do a C-style cast from uint_64t to int when
 // using RoundUpToAlignment from include/llvm/Support/MathExtras.h.
 // FIXME: Move this function to include/MathExtras.h?
static bool mayAlias(MachineInstr *MIa,
                      const AArch64InstrInfo *TII) {
   for (auto &MIb : MemInsns)
     if (mayAlias(MIa, MIb, TII))
       return true;
 
   return false;
 }
 
+static bool canMergeOpc(unsigned Opc, unsigned PairOpc, LdStPairFlags &Flags) {
+  // Opcodes match: nothing more to check.
+  if (Opc == PairOpc)
+    return true;
+
+  // Try to match a sign-extended load/store with a zero-extended load/store.
+  Flags.setSExtIdx(-1);
+  bool IsValidLdStrOpc, PairIsValidLdStrOpc;
+  unsigned NonSExtOpc = getMatchingNonSExtOpcode(Opc, &IsValidLdStrOpc);
+  assert(IsValidLdStrOpc &&
+         "Given Opc should be a Load or Store with an immediate");
+  // Opc will be the first instruction in the pair.
+  if (NonSExtOpc == getMatchingNonSExtOpcode(PairOpc, &PairIsValidLdStrOpc)) {
+    Flags.setSExtIdx(NonSExtOpc == (unsigned)Opc ? 1 : 0);
+    return true;
+  }
+
+  // FIXME: Can we also match a mixed sext/zext unscaled/scaled pair?
+
+  // If the second instruction isn't even a load/store, bail out.
+  if (!PairIsValidLdStrOpc)
+    return false;
+
+  // Try to match an unscaled load/store with a scaled load/store.
+  return isUnscaledLdSt(Opc) != isUnscaledLdSt(PairOpc) &&
+         getMatchingPairOpcode(Opc) == getMatchingPairOpcode(PairOpc);
+}
+
 /// findMatchingInsn - Scan the instructions looking for a load/store that can
 /// be combined with the current instruction into a load/store pair.
 MachineBasicBlock::iterator
 AArch64LoadStoreOpt::findMatchingInsn(MachineBasicBlock::iterator I,
                                       LdStPairFlags &Flags,
                                       unsigned Limit) {
   MachineBasicBlock::iterator E = I->getParent()->end();
   MachineBasicBlock::iterator MBBI = I;
for (unsigned Count = 0; MBBI != E && Count < Limit; ++MBBI) {
     // Skip DBG_VALUE instructions. Otherwise debug info can affect the
     // optimization by changing how far we scan.
     if (MI->isDebugValue())
       continue;
 
     // Now that we know this is a real instruction, count it.
     ++Count;
 
-    bool CanMergeOpc = Opc == MI->getOpcode();
-    Flags.setSExtIdx(-1);
-    if (!CanMergeOpc) {
-      bool IsValidLdStrOpc;
-      unsigned NonSExtOpc = getMatchingNonSExtOpcode(Opc, &IsValidLdStrOpc);
-      assert(IsValidLdStrOpc &&
-             "Given Opc should be a Load or Store with an immediate");
-      // Opc will be the first instruction in the pair.
-      Flags.setSExtIdx(NonSExtOpc == (unsigned)Opc ? 1 : 0);
-      CanMergeOpc = NonSExtOpc == getMatchingNonSExtOpcode(MI->getOpcode());
-    }
-
-    if (CanMergeOpc && getLdStOffsetOp(MI).isImm()) {
+    if (canMergeOpc(Opc, MI->getOpcode(), Flags) &&
+        getLdStOffsetOp(MI).isImm()) {
       assert(MI->mayLoadOrStore() && "Expected memory operation.");
       // If we've found another instruction with the same opcode, check to see
       // if the base and offset are compatible with our starting instruction.
       // These instructions all have scaled immediate operands, so we just
       // check for +1/-1. Make sure to check the new instruction offset is
       // actually an immediate and not a symbolic reference destined for
       // a relocation.
       //
       // Pairwise instructions have a 7-bit signed offset field. Single insns
       // have a 12-bit unsigned offset field. To be a valid combine, the
       // final offset must be in range.
       unsigned MIBaseReg = getLdStBaseOp(MI).getReg();
       int MIOffset = getLdStOffsetOp(MI).getImm();
+
+      // We're trying to pair instructions that differ in how they are scaled.
+      // If FirstMI is scaled then scale the offset of MI accordingly.
+      // Otherwise, do the opposite (i.e., make MI's offset unscaled).
+      bool MIIsUnscaled = isUnscaledLdSt(MI);
+      if (IsUnscaled != MIIsUnscaled) {
+        int MemSize = getMemSize(MI);
+        if (MIIsUnscaled) {
+          // If the unscaled offset isn't a multiple of the MemSize, we can't
+          // pair the operations together: bail and keep looking.
+          if (MIOffset % MemSize)
+            continue;
+          MIOffset /= MemSize;
+        } else {
+          MIOffset *= MemSize;
+        }
+      }
+
       if (BaseReg == MIBaseReg && ((Offset == MIOffset + OffsetStride) ||
                                    (Offset + OffsetStride == MIOffset))) {
         int MinOffset = Offset < MIOffset ? Offset : MIOffset;
         // If this is a volatile load/store that otherwise matched, stop looking
         // as something is going on that we don't have enough information to
         // safely transform. Similarly, stop if we see a hint to avoid pairs.
         if (MI->hasOrderedMemoryRef() || TII->isLdStPairSuppressed(MI))
           return E;
         // If the resultant immediate offset of merging these instructions
         // is out of range for a pairwise instruction, bail and keep looking.
-        bool MIIsUnscaled = isUnscaledLdSt(MI);
-        if (!inBoundsForPair(MIIsUnscaled, MinOffset, OffsetStride)) {
+        if (!inBoundsForPair(IsUnscaled, MinOffset, OffsetStride)) {
           trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
           MemInsns.push_back(MI);
           continue;
         }
         // If the alignment requirements of the paired (scaled) instruction
         // can't express the offset of the unscaled input, bail and keep
         // looking.
         if (IsUnscaled && EnableAArch64UnscaledMemOp &&

test/CodeGen/AArch64/ldp-stp-scaled-unscaled-pairs.ll

+; RUN: llc < %s -march=aarch64 -aarch64-stp-suppress=false -verify-machineinstrs -asm-verbose=false | FileCheck %s
+
+; CHECK-LABEL: test_strd_sturd:
+; CHECK-NEXT: stp d0, d1, [x0, #-8]
+; CHECK-NEXT: ret
+define void @test_strd_sturd(float* %ptr, <2 x float> %v1, <2 x float> %v2) #0 {
+  %tmp1 = bitcast float* %ptr to <2 x float>*
+  store <2 x float> %v2, <2 x float>* %tmp1, align 16
+  %add.ptr = getelementptr inbounds float, float* %ptr, i64 -2
+  %tmp = bitcast float* %add.ptr to <2 x float>*
+  store <2 x float> %v1, <2 x float>* %tmp, align 16
+  ret void
+}
+
+; CHECK-LABEL: test_sturd_strd:
+; CHECK-NEXT: stp d0, d1, [x0, #-8]
+; CHECK-NEXT: ret
+define void @test_sturd_strd(float* %ptr, <2 x float> %v1, <2 x float> %v2) #0 {
+  %add.ptr = getelementptr inbounds float, float* %ptr, i64 -2
+  %tmp = bitcast float* %add.ptr to <2 x float>*
+  store <2 x float> %v1, <2 x float>* %tmp, align 16
+  %tmp1 = bitcast float* %ptr to <2 x float>*
+  store <2 x float> %v2, <2 x float>* %tmp1, align 16
+  ret void
+}
+
+; CHECK-LABEL: test_strq_sturq:
+; CHECK-NEXT: stp q0, q1, [x0, #-16]
+; CHECK-NEXT: ret
+define void @test_strq_sturq(double* %ptr, <2 x double> %v1, <2 x double> %v2) #0 {
+  %tmp1 = bitcast double* %ptr to <2 x double>*
+  store <2 x double> %v2, <2 x double>* %tmp1, align 16
+  %add.ptr = getelementptr inbounds double, double* %ptr, i64 -2
+  %tmp = bitcast double* %add.ptr to <2 x double>*
+  store <2 x double> %v1, <2 x double>* %tmp, align 16
+  ret void
+}
+
+; CHECK-LABEL: test_sturq_strq:
+; CHECK-NEXT: stp q0, q1, [x0, #-16]
+; CHECK-NEXT: ret
+define void @test_sturq_strq(double* %ptr, <2 x double> %v1, <2 x double> %v2) #0 {
+  %add.ptr = getelementptr inbounds double, double* %ptr, i64 -2
+  %tmp = bitcast double* %add.ptr to <2 x double>*
+  store <2 x double> %v1, <2 x double>* %tmp, align 16
+  %tmp1 = bitcast double* %ptr to <2 x double>*
+  store <2 x double> %v2, <2 x double>* %tmp1, align 16
+  ret void
+}
+
+; CHECK-LABEL: test_ldrx_ldurx:
+; CHECK-NEXT: ldp [[V0:x[0-9]+]], [[V1:x[0-9]+]], [x0, #-8]
+; CHECK-NEXT: add x0, [[V0]], [[V1]]
+; CHECK-NEXT: ret
+define i64 @test_ldrx_ldurx(i64* %p) #0 {
+  %tmp = load i64, i64* %p, align 4
+  %add.ptr = getelementptr inbounds i64, i64* %p, i64 -1
+  %tmp1 = load i64, i64* %add.ptr, align 4
+  %add = add nsw i64 %tmp1, %tmp
+  ret i64 %add
+}
+
+; CHECK-LABEL: test_ldurx_ldrx:
+; CHECK-NEXT: ldp [[V0:x[0-9]+]], [[V1:x[0-9]+]], [x0, #-8]
+; CHECK-NEXT: add x0, [[V0]], [[V1]]
+; CHECK-NEXT: ret
+define i64 @test_ldurx_ldrx(i64* %p) #0 {
+  %add.ptr = getelementptr inbounds i64, i64* %p, i64 -1
+  %tmp1 = load i64, i64* %add.ptr, align 4
+  %tmp = load i64, i64* %p, align 4
+  %add = add nsw i64 %tmp1, %tmp
+  ret i64 %add
+}
+
+; CHECK-LABEL: test_ldrsw_ldursw:
+; CHECK-NEXT: ldpsw [[V0:x[0-9]+]], [[V1:x[0-9]+]], [x0, #-4]
+; CHECK-NEXT: add x0, [[V0]], [[V1]]
+; CHECK-NEXT: ret
+define i64 @test_ldrsw_ldursw(i32* %p) #0 {
+  %tmp = load i32, i32* %p, align 4
+  %add.ptr = getelementptr inbounds i32, i32* %p, i64 -1
+  %tmp1 = load i32, i32* %add.ptr, align 4
+  %sexttmp = sext i32 %tmp to i64
+  %sexttmp1 = sext i32 %tmp1 to i64
+  %add = add nsw i64 %sexttmp1, %sexttmp
+  ret i64 %add
+}
+
+; Also make sure we only match valid offsets.
+; CHECK-LABEL: test_ldrq_ldruq_invalidoffset:
+; CHECK-NEXT: ldr q[[V0:[0-9]+]], [x0]
+; CHECK-NEXT: ldur q[[V1:[0-9]+]], [x0, #24]
+; CHECK-NEXT: add.2d v0, v[[V0]], v[[V1]]
+; CHECK-NEXT: ret
+define <2 x i64> @test_ldrq_ldruq_invalidoffset(i64* %p) nounwind {
+  %a1 = bitcast i64* %p to <2 x i64>*
+  %tmp1 = load <2 x i64>, < 2 x i64>* %a1, align 8           
+  %add.ptr2 = getelementptr inbounds i64, i64* %p, i64 3
+  %a2 = bitcast i64* %add.ptr2 to <2 x i64>*
+  %tmp2 = load <2 x i64>, <2 x i64>* %a2, align 8            
+  %add = add nsw <2 x i64> %tmp1, %tmp2
+  ret <2 x i64> %add
+}
+
+attributes #0 = { nounwind }