[ARM] Adjust strd/ldrd codegen alignment requirements

[momo5502]

Summary

In change https://reviews.llvm.org/D152790, it was discovered that the alignment requirement calculation for LDRD/STRD codegen was suboptimal and the calculation for volatile loads and stores was adjusted.

This change here adopts the calculation for the remaining non-volatile occurances.

[efriedma]

LGTM

I suspect we're actually missing a check somewhere for LDRD formation post-RA. But this looks like an improvement.

(I'll commit for you.)

[momo5502]

In D153800#4450112, @efriedma wrote:

LGTM

I suspect we're actually missing a check somewhere for LDRD formation post-RA. But this looks like an improvement.

(I'll commit for you.)

Thank you very much. Maybe I have time to check for other occurances tomorrow.

[DavidSpickett]

I've reverted this due to a failure on all of Linaro's 2 stage 32 bit Arm bots, both v7 and v8 (though both run on v8 hardware).

For example:
https://lab.llvm.org/buildbot/#/builders/178/builds/5128
https://lab.llvm.org/buildbot/#/builders/187/builds/11994
https://lab.llvm.org/buildbot/#/builders/182/builds/6724

            7: error: Simplified template DW_AT_name could not be reconstituted: 
check:10'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            8:  original: f3<unsigned char, (unsigned char)'\x00'> 
check:10'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            9:  reconstituted: f3<unsigned char, (unsigned char)'\x7f'>

I haven't dug deeper yet but if that is one byte misplaced, it could make some sense.

Do you have access to a machine where you can reproduce this? A 32 bit container running on 64 bit hardware will also work.

If you do not, I can do the investigation.

[momo5502]

In D153800#4468445, @DavidSpickett wrote:

I've reverted this due to a failure on all of Linaro's 2 stage 32 bit Arm bots, both v7 and v8 (though both run on v8 hardware).

For example:
https://lab.llvm.org/buildbot/#/builders/178/builds/5128
https://lab.llvm.org/buildbot/#/builders/187/builds/11994
https://lab.llvm.org/buildbot/#/builders/182/builds/6724

            7: error: Simplified template DW_AT_name could not be reconstituted: 
check:10'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            8:  original: f3<unsigned char, (unsigned char)'\x00'> 
check:10'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            9:  reconstituted: f3<unsigned char, (unsigned char)'\x7f'>

I haven't dug deeper yet but if that is one byte misplaced, it could make some sense.

Do you have access to a machine where you can reproduce this? A 32 bit container running on 64 bit hardware will also work.

If you do not, I can do the investigation.

I think I do have access to a machine where I can reproduce this, however I am currently very busy and won't have time to investigate probably until end of this week.
If you have time to look at it, I would be very thankful. Otherwise I can investigate this as soon as I have time.

[DavidSpickett]

Good to know, I'll look into it this week then. Let you know later what the status is.

[DavidSpickett]

So far I have narrowed it down to something in DWARFDebugFrame.cpp.o in libLLVMDebugInfoDWARF.a. You can confirm this by using ar x to get the .o from a working build, then patching it into the broken build with ar r (interestingly, this object seems to have fewer stdrd/ldrd after this change).

From there I am not sure how to narrow it down. If I could set an strd/ldrd alignment per function that would help but as far as I can see strict alignment applies per object. One could split the file into two and gradually move functions from one to the other I suppose.

[DavidSpickett]

And I should say that I don't think the alignment here is wrong. If it were we would have an exception not a test failure. More likely that we are now emitting ldrd/strd in a new situation and there is a bug elsewhere in handling that situation.

[efriedma]

One could split the file into two and gradually move functions from one to the other I suppose.

opt-bisect-limit?

[momo5502]

@DavidSpickett do you have instructions for me on how to setup everything to reproduce the problem.
So I can support you with the investigations.

[DavidSpickett]

I recommend following what the armv8-a builder does: https://lab.llvm.org/buildbot/#/builders/178/builds/5155. If you click on the "cmake stage..." steps it will show you the commands used. You can cut out some of the enabled projects and/or targets if you want to save some time.

We are running on a 64 bit Arm machine with a 32 bit docker container.

In case the hardware is a factor (I hope not) it is a Ampere Mt. Jade with 160x Neoverse-N1 (this information can also be found in the "worker" tab of the buildbot web UI).

$ arch
armv8l
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

If you really want the exact container you can use https://git.linaro.org/ci/dockerfiles.git/tree/focal-armhf-tcwg-base/focal-armhf-tcwg-llvmbot/Dockerfile but if you do, change the entrypoint. As by default it's looking to launch a buildbot agent.

You're probably fine using the official Ubuntu Focal container and installing things as needed.

At the time of the build the host compiler was clang 15.0.0, using the release https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.0/clang+llvm-15.0.0-armv7a-linux-gnueabihf.tar.xz. For unrelated reasons, I'm about to update the bots to use 16.0.4, but given that this is a stage 2 failure it shouldn't make a difference.

It was useful for me to build 2 copies, one with this change and one without. That was how I was able to narrow it to llvm-dwarfdump because using the dwarfdump from the working copy passed the test.

To run only the failing test:

./bin/llvm-lit ../llvm-project/llvm/test/tools/llvm-dwarfdump/X86/simplified-template-names.s -a

[momo5502]

I managed to reproduce this and isolate the function responsible for the triggering the bug. It's llvm::format_object<long long>::snprint(char*, unsigned int) const. I haven't dug deeper yet, but that's at least some progress :D

[DavidSpickett]

Great. @luporl is on bots for us this week, so they can help you if you need more info.

[momo5502]

Here is the IR code of that specific function. The first load has an align of 4, which used to be excluded by the load optimization before this commit.

; Function Attrs: mustprogress nounwind uwtable(sync)
define linkonce_odr hidden noundef i32 @_ZNK4llvm13format_objectIJxEE7snprintEPcj(ptr noundef nonnull align 8 dereferenceable(16) %this, ptr noundef %Buffer, i32 noundef %BufferSize) unnamed_addr #6 comdat align 2 {
entry:
  %Fmt.i = getelementptr inbounds %"class.llvm::format_object_base", ptr %this, i32 0, i32 1
  %0 = load ptr, ptr %Fmt.i, align 4, !tbaa !63
  %Vals.i = getelementptr inbounds %"class.llvm::format_object.75", ptr %this, i32 0, i32 1
  %1 = load i64, ptr %Vals.i, align 8, !tbaa !44
  %call2.i = tail call i32 (ptr, i32, ptr, ...) @snprintf(ptr noundef %Buffer, i32 noundef %BufferSize, ptr noundef %0, i64 noundef %1) #26
  ret i32 %call2.i
}

The diff of lowering that load with the optimization (on the left) and without the optimization from this commit (on the right) is attached.

[efriedma]

That code diff doesn't look like it should have any effect. My best guess is that there's an uninitialized variable somewhere else, and the push/pop makes it have a different value.

[momo5502]

I think I figured it out:

There are 4 arguments to snprintf here: Buffer, BufferSize, Format, Value. Value being the long long to be formatted as unsigned char.
It seems that the calling convention snprintf expects is all values in registers, so r0, r1, r2, r3. Meaning it expects the Value to be in r3.

However, the code that is generated in this function prepares the arguments like this: r0, r1, stack, stack, so the last two arguments are on the stack. I assume this is what we generate if the function is a vararg.
That seems to be an ABI incompatibility. I'm not sure what the correct calling convention is, but it seems like we generate what is mentioned here: https://github.com/NationalSecurityAgency/ghidra/issues/3927

A bit of context as to why it used to work:
The value to be formatted is passed to the function is the value of [r0+#12]. It happens to be 0.
The correct code should move that into r3 for the call to snprintf, however, it is instead pushed onto the stack. By chance, r3 was overwritten with a different 0 value in the old case, so it was still 'correct'. With my change active, it is now being overwriten by the third argument passed to our function, r2, which happens to be 0x7F.

I'm not an ARM expert and I'm again not sure what the right calling convention for varargs is, but that is for sure the cause of this issue.

[efriedma]

It seems that the calling convention snprintf expects is all values in registers, so r0, r1, r2, r3. Meaning it expects the Value to be in r3.

A "long long" can't fit into r3.

The normal calling convention rules (https://github.com/ARM-software/abi-aa/blob/main/aapcs32/aapcs32.rst#parameter-passing) require skipping r3 and storing the "long long" value on the stack. So snprintf should ignore the value in r3 unless we're passing in a bad format string. (Which is possible, I guess... it looks like the "llvm::format" function isn't marked with a format attribute, so the format string isn't checked anywhere.)

[momo5502]

In D153800#4490477, @efriedma wrote:

It seems that the calling convention snprintf expects is all values in registers, so r0, r1, r2, r3. Meaning it expects the Value to be in r3.

A "long long" can't fit into r3.

The normal calling convention rules (https://github.com/ARM-software/abi-aa/blob/main/aapcs32/aapcs32.rst#parameter-passing) require skipping r3 and storing the "long long" value on the stack. So snprintf should ignore the value in r3 unless we're passing in a bad format string. (Which is possible, I guess... it looks like the "llvm::format" function isn't marked with a format attribute, so the format string isn't checked anywhere.)

Oh you are absolutely right, I overlooked that :D

The first 3 arguments are passed in r0, r1, r2 and only the value is passed on the stack as two 4 byte blocks.
snprintf seems to expect them in r3 and r4 then.

As the format string truncates the value, only the lower byte is relevant, which is probably the reason I overlooked it.

The way you describe it sounds like a bug in snprintf then, doesn't it?
It seems that gcc also puts them on the stack: https://godbolt.org/z/h1oY1o3vc

I used the Dockerfile @DavidSpickett linked. Does it contain a custom variant of the C-runtime? So could it be an issue only with that specific version of snprintf?

[efriedma]

snprintf(buf, bufsize, "%d", 1LL) has undefined behavior.

[momo5502]

In D153800#4490748, @efriedma wrote:

snprintf(buf, bufsize, "%d", 1LL) has undefined behavior.

Jup. that's it.

All these formats should be something along the lines of %llx not %x: https://github.com/llvm/llvm-project/blob/89fab98e884f05076bbd420d95b5de3596f5452c/llvm/lib/DebugInfo/DWARF/DWARFTypePrinter.cpp#L427
Changing that fixes the issue.

Thanks for your help :D I will check if there are more occurances and prepare a proper fix tomorrow.

[momo5502]

I thought about static_cast'ing Val instead of changing the format string. In case the underlying type of Val changes in the future, one would otherwise have to adapt the format string as well.
Or is there a way to emit a compiler warning if the format string mismatches? You mentioned marking llvm::format with a format attribute, @eli.friedman? Does that trigger a warning?

[efriedma]

llvm::format currently doesn't have a format attribute, but we might be able to add it; see D112579.

Or you can switch the code to llvm::formatv, which doesn't have this kind of issue.

[momo5502]

I created a fix: https://reviews.llvm.org/D155093

llvm/lib/Target/ARM/ARMLoadStoreOptimizer.cpp

bool ARMPreAllocLoadStoreOpt::CanFormLdStDWord(
   // At the moment, we ignore the memoryoperand's value.
   // If we want to use AliasAnalysis, we should check it accordingly.
   if (!Op0->hasOneMemOperand() ||
       (*Op0->memoperands_begin())->isVolatile() ||
       (*Op0->memoperands_begin())->isAtomic())
     return false;
 
   Align Alignment = (*Op0->memoperands_begin())->getAlign();
-  const Function &Func = MF->getFunction();
-  Align ReqAlign =
-      STI->hasV6Ops() ? TD->getABITypeAlign(Type::getInt64Ty(Func.getContext()))
-                      : Align(8); // Pre-v6 need 8-byte align
+  Align ReqAlign = STI->getDualLoadStoreAlignment();
   if (Alignment < ReqAlign)
     return false;
 
   // Then make sure the immediate offset fits.
   int OffImm = getMemoryOpOffset(*Op0);
   if (isT2) {
     int Limit = (1 << 8) * Scale;
     if (OffImm >= Limit || (OffImm <= -Limit) || (OffImm & (Scale-1)))

llvm/test/CodeGen/ARM/copy-by-struct-i32.ll

 ; ASSEMBLY-NEXT:    sub sp, sp, #136
 ; ASSEMBLY-NEXT:    ldrd r4, r5, [r0]
 ; ASSEMBLY-NEXT:    add lr, sp, #56
 ; ASSEMBLY-NEXT:    ldm r1, {r0, r12}
 ; ASSEMBLY-NEXT:    subs r4, r4, #1
 ; ASSEMBLY-NEXT:    sbc r5, r5, #0
 ; ASSEMBLY-NEXT:    ldr r2, [r1, #8]
 ; ASSEMBLY-NEXT:    ldr r3, [r1, #12]
-; ASSEMBLY-NEXT:    str r5, [sp, #132]
+; ASSEMBLY-NEXT:    strd r4, r5, [sp, #128]
 ; ASSEMBLY-NEXT:    add r5, r1, #16
-; ASSEMBLY-NEXT:    str r4, [sp, #128]
 ; ASSEMBLY-NEXT:    mov r4, sp
 ; ASSEMBLY-NEXT:    vld1.32 {d16}, [r5]!
 ; ASSEMBLY-NEXT:    vst1.32 {d16}, [r4]!
 ; ASSEMBLY-NEXT:    vld1.32 {d16}, [r5]!
 ; ASSEMBLY-NEXT:    vst1.32 {d16}, [r4]!
 ; ASSEMBLY-NEXT:    vld1.32 {d16}, [r5]!
 ; ASSEMBLY-NEXT:    vst1.32 {d16}, [r4]!
 ; ASSEMBLY-NEXT:    vld1.32 {d16}, [r5]!

llvm/test/CodeGen/ARM/ha-alignstack-call.ll

 ; pass in memory, no split
 declare dso_local float @f1_1(double, double, double, double, double, double, double, float, %struct.S1 alignstack(8)) local_unnamed_addr #0
 define dso_local float @f1_1_call() local_unnamed_addr #0 {
 entry:
   %call = tail call float @f1_1(double 0.000000e+00, double 1.000000e-01, double 2.000000e-01, double 3.000000e-01, double 4.000000e-01, double 5.000000e-01, double 6.000000e-01, float 0x3FE6666660000000, %struct.S1 alignstack(8) { [2 x float] [float 0x3FE99999A0000000, float 0.000000e+00] }) #0
   ret float %call
 }
 ; CHECK-LABEL: f1_1_call:
-; CHECK:       movw r1, #52429
-; CHECK:       mov  r0, #0
-; CHECK:       movt r1, #16204
-; CHECK-DAG:   str  r1, [sp]
-; CHECK-DAG:   str  r0, [sp, #4]
+; CHECK:       movw r0, #52429
+; CHECK:       mov  r1, #0
+; CHECK:       movt r0, #16204
+; CHECK-DAG:   strd r0, r1, [sp]
 ; CHECK:       bl   f1_1
 
 ; pass in memory, alignment 8
 declare dso_local float @f1_2(double, double, double, double, double, double, double, double, float, %struct.S1 alignstack(8)) local_unnamed_addr #0
 define dso_local float @f1_2_call() local_unnamed_addr #0 {
 entry:
   %call = tail call float @f1_2(double 0.000000e+00, double 1.000000e-01, double 2.000000e-01, double 3.000000e-01, double 4.000000e-01, double 5.000000e-01, double 6.000000e-01, double 0x3FE6666666666666, float 0x3FE99999A0000000, %struct.S1 alignstack(8) { [2 x float] [float 0x3FECCCCCC0000000, float 0.000000e+00] }) #0
   ret float %call
 }
 ; CHECK-LABEL: f1_2_call:
-; CHECK-DAG:   mov   r0, #0
-; CHECK-DAG:   movw  r1, #26214
-; CHECK:       str   r0, [sp, #12]
+; CHECK-DAG:   movw  r0, #26214
+; CHECK-DAG:   mov   r1, #0
+; CHECK:       movt  r0, #16230
+; CHECK:       strd  r0, r1, [sp, #8]
 ; CHECK:       movw  r0, #52429
-; CHECK:       movt  r1, #16230
 ; CHECK:       movt  r0, #16204
-; CHECK-DAG:   str   r1, [sp, #8]
 ; CHECK-DAG:   str   r0, [sp]
 ; CHECK:       bl    f1_2
 
 
 ; pass in registers
 declare dso_local float @f2_0(double, double, double, double, double, double, %struct.S2 alignstack(8)) local_unnamed_addr #0
 define dso_local float @f2_0_call() local_unnamed_addr #0 {
 entry:

llvm/test/CodeGen/ARM/indexed-mem.ll

 ; CHECK-V8M:       @ %bb.0:
 ; CHECK-V8M-NEXT:    ldrd r2, r3, [r0], #8
 ; CHECK-V8M-NEXT:    add r2, r3
 ; CHECK-V8M-NEXT:    str r2, [r1]
 ; CHECK-V8M-NEXT:    bx lr
 ;
 ; CHECK-V8A-LABEL: post_inc_ldrd:
 ; CHECK-V8A:       @ %bb.0:
-; CHECK-V8A-NEXT:    ldm r0!, {r2, r3}
+; CHECK-V8A-NEXT:    ldrd r2, r3, [r0]
+; CHECK-V8A-NEXT:    add r0, r0, #8
 ; CHECK-V8A-NEXT:    add r2, r2, r3
 ; CHECK-V8A-NEXT:    str r2, [r1]
 ; CHECK-V8A-NEXT:    bx lr
   %ld = load i32, ptr %base
   %addr.1 = getelementptr i32, ptr %base, i32 1
   %ld.1 = load i32, ptr %addr.1
   %addr.2 = getelementptr i32, ptr %base, i32 2
   %res = add i32 %ld, %ld.1
   store i32 %res, ptr %addr.3
   ret ptr %addr.2
 }
 
 define ptr @pre_inc_str_multi(ptr %base) {
 ; CHECK-V8M-LABEL: pre_inc_str_multi:
 ; CHECK-V8M:       @ %bb.0:
 ; CHECK-V8M-NEXT:    ldrd r1, r2, [r0]
 ; CHECK-V8M-NEXT:    add r1, r2
 ; CHECK-V8M-NEXT:    str r1, [r0, #8]!
 ; CHECK-V8M-NEXT:    bx lr
 ;
 ; CHECK-V8A-LABEL: pre_inc_str_multi:
 ; CHECK-V8A:       @ %bb.0:
-; CHECK-V8A-NEXT:    ldm r0, {r1, r2}
-; CHECK-V8A-NEXT:    add r1, r1, r2
+; CHECK-V8A-NEXT:    ldrd r2, r3, [r0]
+; CHECK-V8A-NEXT:    add r1, r2, r3
 ; CHECK-V8A-NEXT:    str r1, [r0, #8]!
 ; CHECK-V8A-NEXT:    bx lr
   %addr.1 = getelementptr i32, ptr %base, i32 1
   %ld = load i32, ptr %base
   %ld.1 = load i32, ptr %addr.1
   %res = add i32 %ld, %ld.1
   %addr.2 = getelementptr i32, ptr %base, i32 2
   store i32 %res, ptr %addr.2
   ret ptr %addr.2
 }
 
 define ptr @pre_dec_str_multi(ptr %base) {
 ; CHECK-V8M-LABEL: pre_dec_str_multi:
 ; CHECK-V8M:       @ %bb.0:
 ; CHECK-V8M-NEXT:    ldrd r1, r2, [r0]
 ; CHECK-V8M-NEXT:    add r1, r2
 ; CHECK-V8M-NEXT:    str r1, [r0, #-4]!
 ; CHECK-V8M-NEXT:    bx lr
 ;
 ; CHECK-V8A-LABEL: pre_dec_str_multi:
 ; CHECK-V8A:       @ %bb.0:
-; CHECK-V8A-NEXT:    ldm r0, {r1, r2}
-; CHECK-V8A-NEXT:    add r1, r1, r2
+; CHECK-V8A-NEXT:    ldrd r2, r3, [r0]
+; CHECK-V8A-NEXT:    add r1, r2, r3
 ; CHECK-V8A-NEXT:    str r1, [r0, #-4]!
 ; CHECK-V8A-NEXT:    bx lr
   %addr.1 = getelementptr i32, ptr %base, i32 1
   %ld = load i32, ptr %base
   %ld.1 = load i32, ptr %addr.1
   %res = add i32 %ld, %ld.1
   %addr.2 = getelementptr i32, ptr %base, i32 -1
   store i32 %res, ptr %addr.2

llvm/test/CodeGen/ARM/prera-ldst-aliasing.mir

bb.0.entry:
     liveins: $r0, $r1
 
     %1 : gpr = COPY $r1
     %0 : gpr = COPY $r0
     %2 : gpr = t2LDRi12 %1, 0, 14, $noreg :: (load (s32) from %ir.y)
     t2STRi12 killed %2, %0, 0, 14, $noreg :: (store (s32) into %ir.x)
     %3 : gpr = t2LDRi12 %1, 4, 14, $noreg :: (load (s32) from %ir.arrayidx2)
     t2STRi12 killed %3, %0, 4, 14, $noreg :: (store (s32) into %ir.arrayidx3)
-    ; CHECK: t2LDRi12
-    ; CHECK-NEXT: t2LDRi12
-    ; CHECK-NEXT: t2STRi12
-    ; CHECK-NEXT: t2STRi12
+    ; CHECK: t2LDRDi8
+    ; CHECK-NEXT: t2STRDi8
     tBX_RET 14, $noreg
 
 ...
 

llvm/test/CodeGen/ARM/prera-ldst-insertpt.mir

bb.0.entry:
     %6 : rgpr, %7 : rgpr  = t2UMULL killed %3, %5, 14, $noreg
     %8 : rgpr, %9 : rgpr  = t2UMULL killed %4, %5, 14, $noreg
     t2STRi12 %1, %0, 0, 14, $noreg :: (store (s32))
     %10 : rgpr = t2LSLri %2, 1, 14, $noreg, $noreg
     t2STRi12 killed %10, %0, 4, 14, $noreg :: (store (s32))
 
     ; Make sure we move the paired stores next to each other, and
     ; insert them in an appropriate location.
-    ; CHECK: t2STRi12 %1,
-    ; CHECK-NEXT: t2STRi12 killed %10,
+    ; CHECK: t2STRDi8 %1, %10,
     ; CHECK-NEXT: t2MOVi
     ; CHECK-NEXT: t2ADDrs
 
     %11 : rgpr = t2MOVi 55, 14, $noreg, $noreg
     %12 : gprnopc = t2ADDrs %11, killed %7, 19, 14, $noreg, $noreg
     t2STRi12 killed %12, %0, 16, 14, $noreg :: (store (s32))
     %13 : gprnopc = t2ADDrs %11, killed %9, 19, 14, $noreg, $noreg
     t2STRi12 killed %13, %0, 20, 14, $noreg :: (store (s32))
 
     ; Make sure we move the paired stores next to each other.
-    ; CHECK: t2STRi12 killed %12,
-    ; CHECK-NEXT: t2STRi12 killed %13,
+    ; CHECK: t2STRDi8 %12, %13,
 
     tBX_RET 14, $noreg
 ---
 # CHECK-LABEL: name: b
 name:            b
 alignment:       2
 tracksRegLiveness: true
 liveins:
bb.0.entry:
     ; insert them in an appropriate location.
     ; CHECK:      t2STRi12 {{.*}}, 0
     ; CHECK-NEXT: t2STRi12 {{.*}}, 4
     ; CHECK-NEXT: t2STRi12 {{.*}}, 8
     ; CHECK-NEXT: t2MUL
     ; CHECK-NEXT: t2MOVi32imm
 
     ; CHECK-LIMIT-LABEL: name: b
-    ; CHECK-LIMIT:      t2STRi12 {{.*}}, 0
-    ; CHECK-LIMIT-NEXT: t2STRi12 {{.*}}, 4
+    ; CHECK-LIMIT:      t2STRDi8 {{.*}}, {{.*}}, {{.*}}, 0
     ; CHECK-LIMIT-NEXT: t2MUL
     ; CHECK-LIMIT-NEXT: t2STRi12 {{.*}}, 8
 
     %4 : rgpr = t2MUL %1, %1, 14, $noreg
     %5 : rgpr = t2MOVi32imm -858993459
     %6 : rgpr, %7 : rgpr  = t2UMULL killed %3, %5, 14, $noreg
     %8 : rgpr, %9 : rgpr  = t2UMULL killed %4, %5, 14, $noreg
     %10 : rgpr = t2LSLri %2, 1, 14, $noreg, $noreg
     %11 : rgpr = t2MOVi 55, 14, $noreg, $noreg
     %12 : gprnopc = t2ADDrs %11, killed %7, 19, 14, $noreg, $noreg
     t2STRi12 killed %12, %0, 16, 14, $noreg :: (store (s32))
     %13 : gprnopc = t2ADDrs %11, killed %9, 19, 14, $noreg, $noreg
     t2STRi12 killed %13, %0, 20, 14, $noreg :: (store (s32))
 
     ; Make sure we move the paired stores next to each other.
-    ; CHECK: t2STRi12 {{.*}}, 16
-    ; CHECK-NEXT: t2STRi12 {{.*}}, 20
+    ; CHECK: t2STRDi8 %12, %13, %0, 16
 
     tBX_RET 14, $noreg
 
 ...

llvm/test/CodeGen/Thumb2/LowOverheadLoops/while-loops.ll

 ; CHECK-NEXT:    .setfp r7, sp, #12
 ; CHECK-NEXT:    add r7, sp, #12
 ; CHECK-NEXT:    .save {r8, r9, r10, r11}
 ; CHECK-NEXT:    push.w {r8, r9, r10, r11}
 ; CHECK-NEXT:    .pad #16
 ; CHECK-NEXT:    sub sp, #16
 ; CHECK-NEXT:    wls lr, r1, .LBB2_3
 ; CHECK-NEXT:  @ %bb.1: @ %while.body.preheader
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    add.w r10, r3, #4
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    mvn r9, #1
-; CHECK-NEXT:    @ implicit-def: $r8
+; CHECK-NEXT:    adds r6, r3, #4
+; CHECK-NEXT:    adds r1, r0, #4
+; CHECK-NEXT:    mvn r8, #1
+; CHECK-NEXT:    @ implicit-def: $r9
 ; CHECK-NEXT:    @ implicit-def: $r4
 ; CHECK-NEXT:    str r2, [sp] @ 4-byte Spill
 ; CHECK-NEXT:  .LBB2_2: @ %while.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr r2, [r0]
-; CHECK-NEXT:    asrs r5, r4, #31
-; CHECK-NEXT:    str r0, [sp, #12] @ 4-byte Spill
-; CHECK-NEXT:    muls r2, r3, r2
-; CHECK-NEXT:    adds r4, r4, r2
-; CHECK-NEXT:    adc.w r2, r5, r2, asr #31
-; CHECK-NEXT:    ldr.w r5, [r9, #4]
-; CHECK-NEXT:    adds.w r4, r4, #-2147483648
-; CHECK-NEXT:    adc r1, r2, #0
-; CHECK-NEXT:    str r1, [sp, #8] @ 4-byte Spill
-; CHECK-NEXT:    smull r5, r6, r5, r8
-; CHECK-NEXT:    ldr.w r2, [r9]
-; CHECK-NEXT:    asrs r4, r1, #31
+; CHECK-NEXT:    str r1, [sp, #12] @ 4-byte Spill
+; CHECK-NEXT:    asrs r2, r4, #31
+; CHECK-NEXT:    ldr r1, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT:    ldr r1, [r1]
+; CHECK-NEXT:    muls r1, r3, r1
+; CHECK-NEXT:    adds r4, r4, r1
+; CHECK-NEXT:    adc.w r1, r2, r1, asr #31
+; CHECK-NEXT:    adds.w r2, r4, #-2147483648
+; CHECK-NEXT:    ldrd r2, r4, [r8]
+; CHECK-NEXT:    adc r5, r1, #0
 ; CHECK-NEXT:    str r2, [sp, #4] @ 4-byte Spill
-; CHECK-NEXT:    subs r5, r1, r5
-; CHECK-NEXT:    sbcs r4, r6
-; CHECK-NEXT:    adds.w r6, r5, #-2147483648
-; CHECK-NEXT:    adc r5, r4, #0
-; CHECK-NEXT:    ldr r4, [r0, #-4]
+; CHECK-NEXT:    smull r4, r2, r4, r9
+; CHECK-NEXT:    asrs r1, r5, #31
+; CHECK-NEXT:    str r5, [sp, #8] @ 4-byte Spill
+; CHECK-NEXT:    subs r4, r5, r4
+; CHECK-NEXT:    sbcs r1, r2
+; CHECK-NEXT:    ldr r2, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT:    adds.w r10, r4, #-2147483648
+; CHECK-NEXT:    adc r1, r1, #0
+; CHECK-NEXT:    ldr r4, [r2, #-4]
 ; CHECK-NEXT:    muls r4, r3, r4
 ; CHECK-NEXT:    adds r3, #4
-; CHECK-NEXT:    adds.w r0, r4, #-2147483648
-; CHECK-NEXT:    asr.w r1, r4, #31
-; CHECK-NEXT:    ldr.w r4, [r10]
-; CHECK-NEXT:    adc r1, r1, #0
-; CHECK-NEXT:    mul r2, r4, r12
-; CHECK-NEXT:    add.w r12, r12, #4
+; CHECK-NEXT:    adds.w r12, r4, #-2147483648
+; CHECK-NEXT:    asr.w r5, r4, #31
+; CHECK-NEXT:    ldr r4, [r6]
+; CHECK-NEXT:    adc r5, r5, #0
+; CHECK-NEXT:    mul r2, r4, r0
+; CHECK-NEXT:    adds r0, #4
 ; CHECK-NEXT:    add.w r2, r2, #-2147483648
-; CHECK-NEXT:    asrl r0, r1, r2
+; CHECK-NEXT:    asrl r12, r5, r2
+; CHECK-NEXT:    smull r2, r5, r4, r12
+; CHECK-NEXT:    lsll r2, r5, #30
+; CHECK-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
+; CHECK-NEXT:    asr.w r11, r5, #31
+; CHECK-NEXT:    mov r12, r5
+; CHECK-NEXT:    lsll r12, r11, r4
+; CHECK-NEXT:    mul r2, r2, r9
+; CHECK-NEXT:    lsrl r12, r11, #2
+; CHECK-NEXT:    adds r2, #2
+; CHECK-NEXT:    lsll r12, r11, r2
 ; CHECK-NEXT:    ldr r2, [sp] @ 4-byte Reload
-; CHECK-NEXT:    smull r0, r1, r4, r0
-; CHECK-NEXT:    lsll r0, r1, #30
-; CHECK-NEXT:    asr.w r11, r1, #31
-; CHECK-NEXT:    mov r0, r1
-; CHECK-NEXT:    ldr r1, [sp, #4] @ 4-byte Reload
-; CHECK-NEXT:    lsll r0, r11, r4
-; CHECK-NEXT:    lsrl r0, r11, #2
-; CHECK-NEXT:    mul r1, r1, r8
-; CHECK-NEXT:    adds r1, #2
-; CHECK-NEXT:    lsll r0, r11, r1
-; CHECK-NEXT:    ldr r1, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT:    add.w r0, r0, #-2147483648
-; CHECK-NEXT:    asrl r6, r5, r0
-; CHECK-NEXT:    movs r0, #2
-; CHECK-NEXT:    lsrl r6, r5, #2
-; CHECK-NEXT:    str r6, [r0]
-; CHECK-NEXT:    mov r8, r6
-; CHECK-NEXT:    ldr r0, [r9], #-4
-; CHECK-NEXT:    mls r0, r0, r4, r1
-; CHECK-NEXT:    adds.w r4, r0, #-2147483648
-; CHECK-NEXT:    asr.w r1, r0, #31
+; CHECK-NEXT:    add.w r5, r12, #-2147483648
+; CHECK-NEXT:    asrl r10, r1, r5
+; CHECK-NEXT:    ldr r5, [sp, #8] @ 4-byte Reload
+; CHECK-NEXT:    lsrl r10, r1, #2
+; CHECK-NEXT:    movs r1, #2
+; CHECK-NEXT:    mov r9, r10
+; CHECK-NEXT:    str.w r10, [r1]
+; CHECK-NEXT:    ldr r1, [r8], #-4
+; CHECK-NEXT:    mls r5, r1, r4, r5
+; CHECK-NEXT:    adds.w r4, r5, #-2147483648
+; CHECK-NEXT:    asr.w r1, r5, #31
 ; CHECK-NEXT:    adc r1, r1, #0
 ; CHECK-NEXT:    lsrl r4, r1, #2
-; CHECK-NEXT:    rsbs r0, r4, #0
-; CHECK-NEXT:    str r0, [r2]
-; CHECK-NEXT:    str r0, [r10, #-4]
-; CHECK-NEXT:    add.w r10, r10, #4
-; CHECK-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    rsbs r1, r4, #0
+; CHECK-NEXT:    str r1, [r2]
+; CHECK-NEXT:    str r1, [r6, #-4]
+; CHECK-NEXT:    adds r6, #4
+; CHECK-NEXT:    ldr r1, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT:    adds r1, #4
 ; CHECK-NEXT:    le lr, .LBB2_2
 ; CHECK-NEXT:  .LBB2_3: @ %while.end
 ; CHECK-NEXT:    add sp, #16
 ; CHECK-NEXT:    pop.w {r8, r9, r10, r11}
 ; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
 entry:
   %0 = inttoptr i32 %e to ptr
   %tobool.not70 = icmp eq i32 %d, 0

llvm/test/CodeGen/Thumb2/mve-float16regloops.ll

 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11}
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
-; CHECK-NEXT:    ldrd r12, r6, [r0, #4]
+; CHECK-NEXT:    ldrd r6, r12, [r0, #4]
 ; CHECK-NEXT:    ldrb.w r9, [r0]
 ; CHECK-NEXT:    vldr.16 s0, .LCPI17_0
 ; CHECK-NEXT:    lsr.w r8, r3, #1
 ; CHECK-NEXT:    b .LBB17_3
 ; CHECK-NEXT:  .LBB17_1: @ %if.else
 ; CHECK-NEXT:    @ in Loop: Header=BB17_3 Depth=1
 ; CHECK-NEXT:    vmovx.f16 s5, s4
-; CHECK-NEXT:    vstr.16 s4, [r12]
+; CHECK-NEXT:    vstr.16 s4, [r6]
 ; CHECK-NEXT:  .LBB17_2: @ %if.end
 ; CHECK-NEXT:    @ in Loop: Header=BB17_3 Depth=1
-; CHECK-NEXT:    vstr.16 s5, [r12, #2]
-; CHECK-NEXT:    adds r6, #10
+; CHECK-NEXT:    vstr.16 s5, [r6, #2]
+; CHECK-NEXT:    add.w r12, r12, #10
 ; CHECK-NEXT:    subs.w r9, r9, #1
-; CHECK-NEXT:    add.w r12, r12, #4
+; CHECK-NEXT:    add.w r6, r6, #4
 ; CHECK-NEXT:    mov r1, r2
 ; CHECK-NEXT:    beq .LBB17_8
 ; CHECK-NEXT:  .LBB17_3: @ %do.body
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB17_5 Depth 2
-; CHECK-NEXT:    vldrh.u16 q2, [r6]
+; CHECK-NEXT:    vldrh.u16 q2, [r12]
 ; CHECK-NEXT:    movs r5, #0
 ; CHECK-NEXT:    vmov q4, q2
 ; CHECK-NEXT:    vshlc q4, r5, #16
-; CHECK-NEXT:    vldrh.u16 q3, [r6, #4]
+; CHECK-NEXT:    vldrh.u16 q3, [r12, #4]
 ; CHECK-NEXT:    vmov q5, q3
 ; CHECK-NEXT:    vshlc q5, r5, #16
-; CHECK-NEXT:    vldrh.u16 q1, [r12]
+; CHECK-NEXT:    vldrh.u16 q1, [r6]
 ; CHECK-NEXT:    vmov.f32 s5, s1
 ; CHECK-NEXT:    mov r5, r2
 ; CHECK-NEXT:    wls lr, r8, .LBB17_6
 ; CHECK-NEXT:  @ %bb.4: @ %while.body.preheader
 ; CHECK-NEXT:    @ in Loop: Header=BB17_3 Depth=1
 ; CHECK-NEXT:    mov r5, r2
 ; CHECK-NEXT:  .LBB17_5: @ %while.body
 ; CHECK-NEXT:    @ Parent Loop BB17_3 Depth=1

 ; CHECK-NEXT:  @ %bb.7: @ %if.then
 ; CHECK-NEXT:    @ in Loop: Header=BB17_3 Depth=1
 ; CHECK-NEXT:    ldrh r0, [r1]
 ; CHECK-NEXT:    vfma.f16 q1, q2, r0
 ; CHECK-NEXT:    vmov.u16 r0, q1[0]
 ; CHECK-NEXT:    vfma.f16 q1, q3, r0
 ; CHECK-NEXT:    strh r0, [r5]
 ; CHECK-NEXT:    vmovx.f16 s2, s4
-; CHECK-NEXT:    vstr.16 s2, [r12]
+; CHECK-NEXT:    vstr.16 s2, [r6]
 ; CHECK-NEXT:    b .LBB17_2
 ; CHECK-NEXT:  .LBB17_8: @ %do.end
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.9:
 ; CHECK-NEXT:  .LCPI17_0:

llvm/test/CodeGen/Thumb2/mve-float32regloops.ll

 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11}
 ; CHECK-NEXT:    .pad #24
 ; CHECK-NEXT:    sub sp, #24
 ; CHECK-NEXT:    mov r8, r3
 ; CHECK-NEXT:    ldrb.w r12, [r0]
-; CHECK-NEXT:    ldrd r3, r0, [r0, #4]
+; CHECK-NEXT:    ldrd r0, r3, [r0, #4]
 ; CHECK-NEXT:    movs r4, #0
 ; CHECK-NEXT:    cmp.w r8, #0
 ; CHECK-NEXT:    strd r4, r4, [sp, #16]
 ; CHECK-NEXT:    beq .LBB17_5
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:    movs r5, #2
 ; CHECK-NEXT:    viwdup.u32 q0, r4, r5, #1
 ; CHECK-NEXT:    mov r4, sp
 ; CHECK-NEXT:  .LBB17_2: @ %bb29
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB17_3 Depth 2
-; CHECK-NEXT:    ldrd r5, r7, [r0]
-; CHECK-NEXT:    vldrw.u32 q1, [r3]
-; CHECK-NEXT:    ldr r6, [r0, #12]
-; CHECK-NEXT:    vldr s8, [r0, #8]
+; CHECK-NEXT:    ldrd r5, r7, [r3]
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    ldr r6, [r3, #12]
+; CHECK-NEXT:    vldr s8, [r3, #8]
 ; CHECK-NEXT:    vstrw.32 q1, [r4]
 ; CHECK-NEXT:    vdup.32 q1, r7
-; CHECK-NEXT:    vldr s12, [r0, #16]
+; CHECK-NEXT:    vldr s12, [r3, #16]
 ; CHECK-NEXT:    vmov.f32 s6, s8
 ; CHECK-NEXT:    dls lr, r8
 ; CHECK-NEXT:    vmov.f32 s7, s8
 ; CHECK-NEXT:    vdup.32 q2, r6
 ; CHECK-NEXT:    vmov.f32 s10, s12
 ; CHECK-NEXT:    mov r7, r2
 ; CHECK-NEXT:    vmov.f32 s11, s12
 ; CHECK-NEXT:  .LBB17_3: @ %bb55
 ; CHECK-NEXT:    @ Parent Loop BB17_2 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    vldrw.u32 q4, [r1, q0, uxtw #2]
 ; CHECK-NEXT:    vldrw.u32 q5, [r4, q0, uxtw #2]
 ; CHECK-NEXT:    vldrw.u32 q3, [sp, #8]
 ; CHECK-NEXT:    adds r1, #8
 ; CHECK-NEXT:    vfma.f32 q5, q4, r5
 ; CHECK-NEXT:    vfma.f32 q3, q5, q2
 ; CHECK-NEXT:    vstmia r7!, {s20, s21}
 ; CHECK-NEXT:    vfma.f32 q3, q4, q1
 ; CHECK-NEXT:    vstrw.32 q3, [r4]
 ; CHECK-NEXT:    le lr, .LBB17_3
 ; CHECK-NEXT:  @ %bb.4: @ %bb75
 ; CHECK-NEXT:    @ in Loop: Header=BB17_2 Depth=1
 ; CHECK-NEXT:    subs.w r12, r12, #1
-; CHECK-NEXT:    add.w r0, r0, #20
-; CHECK-NEXT:    vstrb.8 q3, [r3], #16
+; CHECK-NEXT:    add.w r3, r3, #20
+; CHECK-NEXT:    vstrb.8 q3, [r0], #16
 ; CHECK-NEXT:    mov r1, r2
 ; CHECK-NEXT:    bne .LBB17_2
 ; CHECK-NEXT:    b .LBB17_7
 ; CHECK-NEXT:  .LBB17_5: @ %bb21.preheader
 ; CHECK-NEXT:    dls lr, r12
-; CHECK-NEXT:    mov r0, sp
+; CHECK-NEXT:    mov r1, sp
 ; CHECK-NEXT:  .LBB17_6: @ %bb21
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q0, [r3], #16
-; CHECK-NEXT:    vstrw.32 q0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
+; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    le lr, .LBB17_6
 ; CHECK-NEXT:  .LBB17_7: @ %bb80
 ; CHECK-NEXT:    add sp, #24
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, pc}
 bb:
   %i = alloca [6 x float], align 4
   %i4 = getelementptr inbounds %struct.arm_biquad_cascade_stereo_df2T_instance_f32, ptr %arg, i32 0, i32 1

 %struct.arm_biquad_cascade_df2T_instance_f32 = type { i8, ptr, ptr }
 define void @arm_biquad_cascade_df2T_f32(ptr nocapture readonly %S, ptr nocapture readonly %pSrc, ptr nocapture %pDst, i32 %blockSize) {
 ; CHECK-LABEL: arm_biquad_cascade_df2T_f32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13}
-; CHECK-NEXT:    ldrd r12, r6, [r0, #4]
+; CHECK-NEXT:    ldrd r6, r12, [r0, #4]
 ; CHECK-NEXT:    lsr.w r8, r3, #1
 ; CHECK-NEXT:    ldrb r0, [r0]
 ; CHECK-NEXT:    vldr s0, .LCPI20_0
 ; CHECK-NEXT:    b .LBB20_3
 ; CHECK-NEXT:  .LBB20_1: @ %if.else
 ; CHECK-NEXT:    @ in Loop: Header=BB20_3 Depth=1
 ; CHECK-NEXT:    vmov.f32 s6, s5
-; CHECK-NEXT:    vstr s4, [r12]
+; CHECK-NEXT:    vstr s4, [r6]
 ; CHECK-NEXT:  .LBB20_2: @ %if.end
 ; CHECK-NEXT:    @ in Loop: Header=BB20_3 Depth=1
-; CHECK-NEXT:    vstr s6, [r12, #4]
-; CHECK-NEXT:    adds r6, #20
+; CHECK-NEXT:    vstr s6, [r6, #4]
+; CHECK-NEXT:    add.w r12, r12, #20
 ; CHECK-NEXT:    subs r0, #1
-; CHECK-NEXT:    add.w r12, r12, #8
+; CHECK-NEXT:    add.w r6, r6, #8
 ; CHECK-NEXT:    mov r1, r2
 ; CHECK-NEXT:    beq .LBB20_8
 ; CHECK-NEXT:  .LBB20_3: @ %do.body
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB20_5 Depth 2
-; CHECK-NEXT:    vldrw.u32 q3, [r6]
+; CHECK-NEXT:    vldrw.u32 q3, [r12]
 ; CHECK-NEXT:    movs r5, #0
 ; CHECK-NEXT:    vmov q4, q3
 ; CHECK-NEXT:    vshlc q4, r5, #32
-; CHECK-NEXT:    vldrw.u32 q2, [r6, #8]
+; CHECK-NEXT:    vldrw.u32 q2, [r12, #8]
 ; CHECK-NEXT:    vmov q5, q2
 ; CHECK-NEXT:    vshlc q5, r5, #32
-; CHECK-NEXT:    vldrw.u32 q1, [r12]
+; CHECK-NEXT:    vldrw.u32 q1, [r6]
 ; CHECK-NEXT:    vmov.f32 s6, s0
 ; CHECK-NEXT:    mov r5, r2
 ; CHECK-NEXT:    vmov.f32 s7, s0
 ; CHECK-NEXT:    wls lr, r8, .LBB20_6
 ; CHECK-NEXT:  @ %bb.4: @ %while.body.preheader
 ; CHECK-NEXT:    @ in Loop: Header=BB20_3 Depth=1
 ; CHECK-NEXT:    vmov q6, q1
 ; CHECK-NEXT:    mov r5, r2

 ; CHECK-NEXT:    beq .LBB20_1
 ; CHECK-NEXT:  @ %bb.7: @ %if.then
 ; CHECK-NEXT:    @ in Loop: Header=BB20_3 Depth=1
 ; CHECK-NEXT:    ldr r1, [r1]
 ; CHECK-NEXT:    vfma.f32 q1, q3, r1
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vstr s4, [r5]
 ; CHECK-NEXT:    vfma.f32 q1, q2, r1
-; CHECK-NEXT:    vstr s5, [r12]
+; CHECK-NEXT:    vstr s5, [r6]
 ; CHECK-NEXT:    b .LBB20_2
 ; CHECK-NEXT:  .LBB20_8: @ %do.end
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13}
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, pc}
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.9:
 ; CHECK-NEXT:  .LCPI20_0:
 ; CHECK-NEXT:    .long 0x00000000 @ float 0

llvm/test/CodeGen/Thumb2/mve-gather-ptrs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp -enable-arm-maskedldst %s -o - | FileCheck %s
 
 ; i32
 
 define arm_aapcs_vfpcc <2 x i32> @ptr_v2i32(ptr %offptr) {
 ; CHECK-LABEL: ptr_v2i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ldrd r1, r0, [r0]
-; CHECK-NEXT:    ldr r0, [r0]
+; CHECK-NEXT:    ldrd r0, r1, [r0]
 ; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov q0[2], q0[0], r1, r0
+; CHECK-NEXT:    ldr r0, [r0]
+; CHECK-NEXT:    vmov q0[2], q0[0], r0, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <2 x ptr>, ptr %offptr, align 4
   %gather = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> %offs, i32 4, <2 x i1> <i1 true, i1 true>, <2 x i32> undef)
   ret <2 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @ptr_v4i32(ptr %offptr) {
entry:
   ret <16 x i32> %gather
 }
 
 ; f32
 
 define arm_aapcs_vfpcc <2 x float> @ptr_v2f32(ptr %offptr) {
 ; CHECK-LABEL: ptr_v2f32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ldrd r1, r0, [r0]
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vldr s0, [r1]
+; CHECK-NEXT:    ldrd r0, r1, [r0]
+; CHECK-NEXT:    vldr s1, [r1]
+; CHECK-NEXT:    vldr s0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <2 x ptr>, ptr %offptr, align 4
   %gather = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> %offs, i32 4, <2 x i1> <i1 true, i1 true>, <2 x float> undef)
   ret <2 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @ptr_v4f32(ptr %offptr) {
entry:
   %offs = load <8 x ptr>, ptr %offptr, align 4
   %gather = call <8 x i16> @llvm.masked.gather.v8i16.v8p0(<8 x ptr> %offs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> undef)
   ret <8 x i16> %gather
 }
 
 define arm_aapcs_vfpcc <2 x i32> @ptr_v2i16_sext(ptr %offptr) {
 ; CHECK-LABEL: ptr_v2i16_sext:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ldrd r1, r0, [r0]
-; CHECK-NEXT:    ldrsh.w r0, [r0]
+; CHECK-NEXT:    ldrd r0, r1, [r0]
 ; CHECK-NEXT:    ldrsh.w r1, [r1]
-; CHECK-NEXT:    vmov q0[2], q0[0], r1, r0
-; CHECK-NEXT:    asrs r0, r0, #31
+; CHECK-NEXT:    ldrsh.w r0, [r0]
+; CHECK-NEXT:    vmov q0[2], q0[0], r0, r1
 ; CHECK-NEXT:    asrs r1, r1, #31
-; CHECK-NEXT:    vmov q0[3], q0[1], r1, r0
+; CHECK-NEXT:    asrs r0, r0, #31
+; CHECK-NEXT:    vmov q0[3], q0[1], r0, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <2 x ptr>, ptr %offptr, align 4
   %gather = call <2 x i16> @llvm.masked.gather.v2i16.v2p0(<2 x ptr> %offs, i32 2, <2 x i1> <i1 true, i1 true>, <2 x i16> undef)
   %ext = sext <2 x i16> %gather to <2 x i32>
   ret <2 x i32> %ext
 }
 
 define arm_aapcs_vfpcc <2 x i32> @ptr_v2i16_zext(ptr %offptr) {
 ; CHECK-LABEL: ptr_v2i16_zext:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ldrd r1, r0, [r0]
+; CHECK-NEXT:    ldrd r0, r1, [r0]
 ; CHECK-NEXT:    vmov.i64 q0, #0xffff
-; CHECK-NEXT:    ldrh r0, [r0]
 ; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov q1[2], q1[0], r1, r0
+; CHECK-NEXT:    ldrh r0, [r0]
+; CHECK-NEXT:    vmov q1[2], q1[0], r0, r1
 ; CHECK-NEXT:    vand q0, q1, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <2 x ptr>, ptr %offptr, align 4
   %gather = call <2 x i16> @llvm.masked.gather.v2i16.v2p0(<2 x ptr> %offs, i32 2, <2 x i1> <i1 true, i1 true>, <2 x i16> undef)
   %ext = zext <2 x i16> %gather to <2 x i32>
   ret <2 x i32> %ext
 }

llvm/test/CodeGen/Thumb2/mve-masked-ldst.ll

 define void @foo_sext_v2i64_v2i32(ptr %dest, ptr %mask, ptr %src) {
 ; CHECK-LE-LABEL: foo_sext_v2i64_v2i32:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    .save {r4, r5, r7, lr}
 ; CHECK-LE-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-LE-NEXT:    .pad #4
 ; CHECK-LE-NEXT:    sub sp, #4
 ; CHECK-LE-NEXT:    ldrd r12, lr, [r1]
-; CHECK-LE-NEXT:    movs r3, #0
+; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    @ implicit-def: $q1
-; CHECK-LE-NEXT:    rsbs.w r1, r12, #0
+; CHECK-LE-NEXT:    rsbs.w r3, r12, #0
 ; CHECK-LE-NEXT:    vmov q0[2], q0[0], r12, lr
-; CHECK-LE-NEXT:    sbcs.w r1, r3, r12, asr #31
-; CHECK-LE-NEXT:    csetm r1, lt
+; CHECK-LE-NEXT:    sbcs.w r3, r1, r12, asr #31
+; CHECK-LE-NEXT:    csetm r3, lt
 ; CHECK-LE-NEXT:    rsbs.w r4, lr, #0
-; CHECK-LE-NEXT:    sbcs.w r4, r3, lr, asr #31
-; CHECK-LE-NEXT:    bfi r3, r1, #0, #1
-; CHECK-LE-NEXT:    csetm r1, lt
-; CHECK-LE-NEXT:    bfi r3, r1, #1, #1
-; CHECK-LE-NEXT:    lsls r1, r3, #31
+; CHECK-LE-NEXT:    sbcs.w r4, r1, lr, asr #31
+; CHECK-LE-NEXT:    bfi r1, r3, #0, #1
+; CHECK-LE-NEXT:    csetm r3, lt
+; CHECK-LE-NEXT:    bfi r1, r3, #1, #1
+; CHECK-LE-NEXT:    lsls r3, r1, #31
 ; CHECK-LE-NEXT:    itt ne
-; CHECK-LE-NEXT:    ldrne r1, [r2]
-; CHECK-LE-NEXT:    vmovne.32 q1[0], r1
-; CHECK-LE-NEXT:    lsls r1, r3, #30
+; CHECK-LE-NEXT:    ldrne r3, [r2]
+; CHECK-LE-NEXT:    vmovne.32 q1[0], r3
+; CHECK-LE-NEXT:    lsls r1, r1, #30
 ; CHECK-LE-NEXT:    itt mi
 ; CHECK-LE-NEXT:    ldrmi r1, [r2, #4]
 ; CHECK-LE-NEXT:    vmovmi.32 q1[2], r1
 ; CHECK-LE-NEXT:    vmov r2, s6
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    vmov r3, s0
 ; CHECK-LE-NEXT:    vmov r4, s4
 ; CHECK-LE-NEXT:    vmov q1[2], q1[0], r4, r2

 define void @foo_sext_v2i64_v2i32_unaligned(ptr %dest, ptr %mask, ptr %src) {
 ; CHECK-LE-LABEL: foo_sext_v2i64_v2i32_unaligned:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    .save {r4, r5, r7, lr}
 ; CHECK-LE-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-LE-NEXT:    .pad #4
 ; CHECK-LE-NEXT:    sub sp, #4
 ; CHECK-LE-NEXT:    ldrd r12, lr, [r1]
-; CHECK-LE-NEXT:    movs r3, #0
+; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    @ implicit-def: $q0
-; CHECK-LE-NEXT:    rsbs.w r1, r12, #0
+; CHECK-LE-NEXT:    rsbs.w r3, r12, #0
 ; CHECK-LE-NEXT:    vmov q1[2], q1[0], r12, lr
-; CHECK-LE-NEXT:    sbcs.w r1, r3, r12, asr #31
-; CHECK-LE-NEXT:    csetm r1, lt
+; CHECK-LE-NEXT:    sbcs.w r3, r1, r12, asr #31
+; CHECK-LE-NEXT:    csetm r3, lt
 ; CHECK-LE-NEXT:    rsbs.w r4, lr, #0
-; CHECK-LE-NEXT:    sbcs.w r4, r3, lr, asr #31
-; CHECK-LE-NEXT:    bfi r3, r1, #0, #1
-; CHECK-LE-NEXT:    csetm r1, lt
-; CHECK-LE-NEXT:    bfi r3, r1, #1, #1
-; CHECK-LE-NEXT:    lsls r1, r3, #31
+; CHECK-LE-NEXT:    sbcs.w r4, r1, lr, asr #31
+; CHECK-LE-NEXT:    bfi r1, r3, #0, #1
+; CHECK-LE-NEXT:    csetm r3, lt
+; CHECK-LE-NEXT:    bfi r1, r3, #1, #1
+; CHECK-LE-NEXT:    lsls r3, r1, #31
 ; CHECK-LE-NEXT:    itt ne
-; CHECK-LE-NEXT:    ldrne r1, [r2]
-; CHECK-LE-NEXT:    vmovne.32 q0[0], r1
-; CHECK-LE-NEXT:    lsls r1, r3, #30
+; CHECK-LE-NEXT:    ldrne r3, [r2]
+; CHECK-LE-NEXT:    vmovne.32 q0[0], r3
+; CHECK-LE-NEXT:    lsls r1, r1, #30
 ; CHECK-LE-NEXT:    itt mi
 ; CHECK-LE-NEXT:    ldrmi r1, [r2, #4]
 ; CHECK-LE-NEXT:    vmovmi.32 q0[2], r1
 ; CHECK-LE-NEXT:    vmov r2, s2
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    vmov r3, s4
 ; CHECK-LE-NEXT:    vmov r4, s0
 ; CHECK-LE-NEXT:    vmov q0[2], q0[0], r4, r2

 define void @foo_zext_v2i64_v2i32(ptr %dest, ptr %mask, ptr %src) {
 ; CHECK-LE-LABEL: foo_zext_v2i64_v2i32:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    .save {r4, lr}
 ; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    .pad #4
 ; CHECK-LE-NEXT:    sub sp, #4
 ; CHECK-LE-NEXT:    ldrd r12, lr, [r1]
-; CHECK-LE-NEXT:    movs r3, #0
+; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    @ implicit-def: $q0
 ; CHECK-LE-NEXT:    vmov.i64 q2, #0xffffffff
-; CHECK-LE-NEXT:    rsbs.w r1, r12, #0
+; CHECK-LE-NEXT:    rsbs.w r3, r12, #0
 ; CHECK-LE-NEXT:    vmov q1[2], q1[0], r12, lr
-; CHECK-LE-NEXT:    sbcs.w r1, r3, r12, asr #31
-; CHECK-LE-NEXT:    csetm r1, lt
+; CHECK-LE-NEXT:    sbcs.w r3, r1, r12, asr #31
+; CHECK-LE-NEXT:    csetm r3, lt
 ; CHECK-LE-NEXT:    rsbs.w r4, lr, #0
-; CHECK-LE-NEXT:    sbcs.w r4, r3, lr, asr #31
-; CHECK-LE-NEXT:    bfi r3, r1, #0, #1
-; CHECK-LE-NEXT:    csetm r1, lt
-; CHECK-LE-NEXT:    bfi r3, r1, #1, #1
-; CHECK-LE-NEXT:    lsls r1, r3, #31
+; CHECK-LE-NEXT:    sbcs.w r4, r1, lr, asr #31
+; CHECK-LE-NEXT:    bfi r1, r3, #0, #1
+; CHECK-LE-NEXT:    csetm r3, lt
+; CHECK-LE-NEXT:    bfi r1, r3, #1, #1
+; CHECK-LE-NEXT:    lsls r3, r1, #31
 ; CHECK-LE-NEXT:    itt ne
-; CHECK-LE-NEXT:    ldrne r1, [r2]
-; CHECK-LE-NEXT:    vmovne.32 q0[0], r1
-; CHECK-LE-NEXT:    lsls r1, r3, #30
+; CHECK-LE-NEXT:    ldrne r3, [r2]
+; CHECK-LE-NEXT:    vmovne.32 q0[0], r3
+; CHECK-LE-NEXT:    lsls r1, r1, #30
 ; CHECK-LE-NEXT:    itt mi
 ; CHECK-LE-NEXT:    ldrmi r1, [r2, #4]
 ; CHECK-LE-NEXT:    vmovmi.32 q0[2], r1
 ; CHECK-LE-NEXT:    vmov r2, s4
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    vand q0, q0, q2
 ; CHECK-LE-NEXT:    rsbs r3, r2, #0
 ; CHECK-LE-NEXT:    vmov r3, s6

 define void @foo_zext_v2i64_v2i32_unaligned(ptr %dest, ptr %mask, ptr %src) {
 ; CHECK-LE-LABEL: foo_zext_v2i64_v2i32_unaligned:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    .save {r4, lr}
 ; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    .pad #4
 ; CHECK-LE-NEXT:    sub sp, #4
 ; CHECK-LE-NEXT:    ldrd r12, lr, [r1]
-; CHECK-LE-NEXT:    movs r3, #0
+; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    @ implicit-def: $q0
 ; CHECK-LE-NEXT:    vmov.i64 q2, #0xffffffff
-; CHECK-LE-NEXT:    rsbs.w r1, r12, #0
+; CHECK-LE-NEXT:    rsbs.w r3, r12, #0
 ; CHECK-LE-NEXT:    vmov q1[2], q1[0], r12, lr
-; CHECK-LE-NEXT:    sbcs.w r1, r3, r12, asr #31
-; CHECK-LE-NEXT:    csetm r1, lt
+; CHECK-LE-NEXT:    sbcs.w r3, r1, r12, asr #31
+; CHECK-LE-NEXT:    csetm r3, lt
 ; CHECK-LE-NEXT:    rsbs.w r4, lr, #0
-; CHECK-LE-NEXT:    sbcs.w r4, r3, lr, asr #31
-; CHECK-LE-NEXT:    bfi r3, r1, #0, #1
-; CHECK-LE-NEXT:    csetm r1, lt
-; CHECK-LE-NEXT:    bfi r3, r1, #1, #1
-; CHECK-LE-NEXT:    lsls r1, r3, #31
+; CHECK-LE-NEXT:    sbcs.w r4, r1, lr, asr #31
+; CHECK-LE-NEXT:    bfi r1, r3, #0, #1
+; CHECK-LE-NEXT:    csetm r3, lt
+; CHECK-LE-NEXT:    bfi r1, r3, #1, #1
+; CHECK-LE-NEXT:    lsls r3, r1, #31
 ; CHECK-LE-NEXT:    itt ne
-; CHECK-LE-NEXT:    ldrne r1, [r2]
-; CHECK-LE-NEXT:    vmovne.32 q0[0], r1
-; CHECK-LE-NEXT:    lsls r1, r3, #30
+; CHECK-LE-NEXT:    ldrne r3, [r2]
+; CHECK-LE-NEXT:    vmovne.32 q0[0], r3
+; CHECK-LE-NEXT:    lsls r1, r1, #30
 ; CHECK-LE-NEXT:    itt mi
 ; CHECK-LE-NEXT:    ldrmi r1, [r2, #4]
 ; CHECK-LE-NEXT:    vmovmi.32 q0[2], r1
 ; CHECK-LE-NEXT:    vmov r2, s4
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    vand q0, q0, q2
 ; CHECK-LE-NEXT:    rsbs r3, r2, #0
 ; CHECK-LE-NEXT:    vmov r3, s6

llvm/test/CodeGen/Thumb2/mve-satmul-loops.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s
 
 define arm_aapcs_vfpcc void @ssatmul_s_q31(ptr nocapture readonly %pSrcA, ptr nocapture readonly %pSrcB, ptr noalias nocapture %pDst, i32 %N) {
 ; CHECK-LABEL: ssatmul_s_q31:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
-; CHECK-NEXT:    .pad #12
-; CHECK-NEXT:    sub sp, #12
+; CHECK-NEXT:    .pad #8
+; CHECK-NEXT:    sub sp, #8
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    beq.w .LBB0_8
 ; CHECK-NEXT:  @ %bb.1: @ %entry
+; CHECK-NEXT:    mov r11, r2
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    bne .LBB0_3
 ; CHECK-NEXT:  @ %bb.2:
-; CHECK-NEXT:    movs r7, #0
+; CHECK-NEXT:    movs r2, #0
 ; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    mov r6, r1
-; CHECK-NEXT:    mov r10, r2
+; CHECK-NEXT:    mov r8, r1
+; CHECK-NEXT:    mov r10, r11
 ; CHECK-NEXT:    b .LBB0_6
 ; CHECK-NEXT:  .LBB0_3: @ %vector.ph
-; CHECK-NEXT:    bic r5, r3, #1
-; CHECK-NEXT:    str r3, [sp, #4] @ 4-byte Spill
-; CHECK-NEXT:    subs r7, r5, #2
-; CHECK-NEXT:    movs r6, #1
-; CHECK-NEXT:    add.w r3, r1, r5, lsl #2
+; CHECK-NEXT:    bic r2, r3, #1
 ; CHECK-NEXT:    adr r4, .LCPI0_0
+; CHECK-NEXT:    subs r7, r2, #2
+; CHECK-NEXT:    movs r6, #1
+; CHECK-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-NEXT:    add.w r10, r11, r2, lsl #2
 ; CHECK-NEXT:    add.w lr, r6, r7, lsr #1
-; CHECK-NEXT:    str r3, [sp, #8] @ 4-byte Spill
-; CHECK-NEXT:    str r5, [sp] @ 4-byte Spill
-; CHECK-NEXT:    add.w r10, r2, r5, lsl #2
-; CHECK-NEXT:    add.w r12, r0, r5, lsl #2
+; CHECK-NEXT:    str r2, [sp] @ 4-byte Spill
+; CHECK-NEXT:    add.w r8, r1, r2, lsl #2
+; CHECK-NEXT:    add.w r12, r0, r2, lsl #2
 ; CHECK-NEXT:    vldrw.u32 q0, [r4]
 ; CHECK-NEXT:    vmvn.i32 q1, #0x80000000
 ; CHECK-NEXT:  .LBB0_4: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldrd r4, r6, [r0], #8
+; CHECK-NEXT:    ldrd r4, r2, [r0], #8
 ; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    ldrd r7, r8, [r1], #8
-; CHECK-NEXT:    smull r4, r11, r7, r4
-; CHECK-NEXT:    asrl r4, r11, #31
+; CHECK-NEXT:    ldrd r7, r6, [r1], #8
+; CHECK-NEXT:    smull r4, r7, r7, r4
+; CHECK-NEXT:    asrl r4, r7, #31
 ; CHECK-NEXT:    rsbs.w r9, r4, #-2147483648
 ; CHECK-NEXT:    mov.w r9, #-1
-; CHECK-NEXT:    sbcs.w r3, r9, r11
+; CHECK-NEXT:    sbcs.w r3, r9, r7
 ; CHECK-NEXT:    csetm r3, lt
 ; CHECK-NEXT:    bfi r5, r3, #0, #8
-; CHECK-NEXT:    smull r6, r3, r8, r6
-; CHECK-NEXT:    asrl r6, r3, #31
-; CHECK-NEXT:    rsbs.w r7, r6, #-2147483648
-; CHECK-NEXT:    vmov q2[2], q2[0], r4, r6
-; CHECK-NEXT:    sbcs.w r7, r9, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r11, r3
-; CHECK-NEXT:    csetm r7, lt
-; CHECK-NEXT:    mvn r6, #-2147483648
-; CHECK-NEXT:    bfi r5, r7, #8, #8
+; CHECK-NEXT:    smull r2, r3, r6, r2
+; CHECK-NEXT:    asrl r2, r3, #31
+; CHECK-NEXT:    rsbs.w r6, r2, #-2147483648
+; CHECK-NEXT:    vmov q2[2], q2[0], r4, r2
+; CHECK-NEXT:    sbcs.w r6, r9, r3
+; CHECK-NEXT:    vmov q2[3], q2[1], r7, r3
+; CHECK-NEXT:    csetm r6, lt
+; CHECK-NEXT:    bfi r5, r6, #8, #8
 ; CHECK-NEXT:    vmsr p0, r5
+; CHECK-NEXT:    mvn r5, #-2147483648
 ; CHECK-NEXT:    vpsel q2, q2, q0
-; CHECK-NEXT:    vmov r3, r4, d4
-; CHECK-NEXT:    subs r3, r3, r6
-; CHECK-NEXT:    sbcs r3, r4, #0
-; CHECK-NEXT:    mov.w r4, #0
-; CHECK-NEXT:    csetm r3, lt
-; CHECK-NEXT:    bfi r4, r3, #0, #8
-; CHECK-NEXT:    vmov r3, r5, d5
-; CHECK-NEXT:    subs r3, r3, r6
-; CHECK-NEXT:    sbcs r3, r5, #0
-; CHECK-NEXT:    csetm r3, lt
-; CHECK-NEXT:    bfi r4, r3, #8, #8
-; CHECK-NEXT:    vmsr p0, r4
+; CHECK-NEXT:    vmov r2, r3, d4
+; CHECK-NEXT:    subs r2, r2, r5
+; CHECK-NEXT:    sbcs r2, r3, #0
+; CHECK-NEXT:    mov.w r3, #0
+; CHECK-NEXT:    csetm r2, lt
+; CHECK-NEXT:    bfi r3, r2, #0, #8
+; CHECK-NEXT:    vmov r2, r4, d5
+; CHECK-NEXT:    subs r2, r2, r5
+; CHECK-NEXT:    sbcs r2, r4, #0
+; CHECK-NEXT:    csetm r2, lt
+; CHECK-NEXT:    bfi r3, r2, #8, #8
+; CHECK-NEXT:    vmsr p0, r3
 ; CHECK-NEXT:    vpsel q2, q2, q1
-; CHECK-NEXT:    vmov r3, s10
-; CHECK-NEXT:    vmov r4, s8
-; CHECK-NEXT:    strd r4, r3, [r2], #8
+; CHECK-NEXT:    vmov r2, s10
+; CHECK-NEXT:    vmov r3, s8
+; CHECK-NEXT:    strd r3, r2, [r11], #8
 ; CHECK-NEXT:    le lr, .LBB0_4
 ; CHECK-NEXT:  @ %bb.5: @ %middle.block
-; CHECK-NEXT:    ldrd r7, r3, [sp] @ 8-byte Folded Reload
-; CHECK-NEXT:    ldr r6, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT:    cmp r7, r3
+; CHECK-NEXT:    ldrd r2, r3, [sp] @ 8-byte Folded Reload
+; CHECK-NEXT:    cmp r2, r3
 ; CHECK-NEXT:    beq .LBB0_8
 ; CHECK-NEXT:  .LBB0_6: @ %for.body.preheader
-; CHECK-NEXT:    sub.w lr, r3, r7
+; CHECK-NEXT:    sub.w lr, r3, r2
 ; CHECK-NEXT:    mov.w r0, #-1
 ; CHECK-NEXT:    mov.w r1, #-2147483648
-; CHECK-NEXT:    mvn r2, #-2147483648
+; CHECK-NEXT:    mvn r3, #-2147483648
 ; CHECK-NEXT:  .LBB0_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr r3, [r12], #4
-; CHECK-NEXT:    ldr r4, [r6], #4
-; CHECK-NEXT:    smull r4, r3, r4, r3
-; CHECK-NEXT:    asrl r4, r3, #31
-; CHECK-NEXT:    subs r5, r1, r4
-; CHECK-NEXT:    sbcs.w r5, r0, r3
-; CHECK-NEXT:    cset r5, lt
-; CHECK-NEXT:    cmp r5, #0
-; CHECK-NEXT:    csel r4, r4, r1, ne
-; CHECK-NEXT:    csel r3, r3, r0, ne
-; CHECK-NEXT:    subs r5, r4, r2
-; CHECK-NEXT:    sbcs r3, r3, #0
-; CHECK-NEXT:    csel r3, r4, r2, lt
-; CHECK-NEXT:    str r3, [r10], #4
+; CHECK-NEXT:    ldr r2, [r12], #4
+; CHECK-NEXT:    ldr r4, [r8], #4
+; CHECK-NEXT:    smull r2, r5, r4, r2
+; CHECK-NEXT:    asrl r2, r5, #31
+; CHECK-NEXT:    subs r4, r1, r2
+; CHECK-NEXT:    sbcs.w r4, r0, r5
+; CHECK-NEXT:    cset r4, lt
+; CHECK-NEXT:    cmp r4, #0
+; CHECK-NEXT:    csel r2, r2, r1, ne
+; CHECK-NEXT:    csel r4, r5, r0, ne
+; CHECK-NEXT:    subs r5, r2, r3
+; CHECK-NEXT:    sbcs r4, r4, #0
+; CHECK-NEXT:    csel r2, r2, r3, lt
+; CHECK-NEXT:    str r2, [r10], #4
 ; CHECK-NEXT:    le lr, .LBB0_7
 ; CHECK-NEXT:  .LBB0_8: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #12
+; CHECK-NEXT:    add sp, #8
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.9:
 ; CHECK-NEXT:  .LCPI0_0:
 ; CHECK-NEXT:    .long 2147483648 @ 0x80000000
 ; CHECK-NEXT:    .long 4294967295 @ 0xffffffff
 ; CHECK-NEXT:    .long 2147483648 @ 0x80000000
 ; CHECK-NEXT:    .long 4294967295 @ 0xffffffff

 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    beq .LBB3_8
 ; CHECK-NEXT:  @ %bb.1: @ %entry
+; CHECK-NEXT:    mov r8, r2
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    bne .LBB3_3
 ; CHECK-NEXT:  @ %bb.2:
 ; CHECK-NEXT:    movs r7, #0
 ; CHECK-NEXT:    mov r12, r0
 ; CHECK-NEXT:    mov r11, r1
-; CHECK-NEXT:    mov r8, r2
+; CHECK-NEXT:    mov r2, r8
 ; CHECK-NEXT:    b .LBB3_6
 ; CHECK-NEXT:  .LBB3_3: @ %vector.ph
 ; CHECK-NEXT:    bic r5, r3, #1
 ; CHECK-NEXT:    movs r6, #1
 ; CHECK-NEXT:    subs r7, r5, #2
 ; CHECK-NEXT:    str r5, [sp] @ 4-byte Spill
-; CHECK-NEXT:    add.w r8, r2, r5, lsl #2
+; CHECK-NEXT:    add.w r2, r8, r5, lsl #2
 ; CHECK-NEXT:    add.w r11, r1, r5, lsl #2
 ; CHECK-NEXT:    add.w lr, r6, r7, lsr #1
 ; CHECK-NEXT:    add.w r12, r0, r5, lsl #2
 ; CHECK-NEXT:    vmov.i8 q0, #0xff
 ; CHECK-NEXT:  .LBB3_4: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrd r4, r9, [r0], #8
 ; CHECK-NEXT:    ldrd r5, r10, [r1], #8

 ; CHECK-NEXT:    vmov q1[2], q1[0], r4, r10
 ; CHECK-NEXT:    sbcs r5, r5, #0
 ; CHECK-NEXT:    csetm r5, lo
 ; CHECK-NEXT:    bfi r6, r5, #8, #8
 ; CHECK-NEXT:    vmsr p0, r6
 ; CHECK-NEXT:    vpsel q1, q1, q0
 ; CHECK-NEXT:    vmov r4, s6
 ; CHECK-NEXT:    vmov r5, s4
-; CHECK-NEXT:    strd r5, r4, [r2], #8
+; CHECK-NEXT:    strd r5, r4, [r8], #8
 ; CHECK-NEXT:    le lr, .LBB3_4
 ; CHECK-NEXT:  @ %bb.5: @ %middle.block
 ; CHECK-NEXT:    ldr r7, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    cmp r7, r3
 ; CHECK-NEXT:    beq .LBB3_8
 ; CHECK-NEXT:  .LBB3_6: @ %for.body.preheader
 ; CHECK-NEXT:    sub.w lr, r3, r7
 ; CHECK-NEXT:  .LBB3_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldr r0, [r12], #4
 ; CHECK-NEXT:    ldr r1, [r11], #4
 ; CHECK-NEXT:    umull r0, r1, r1, r0
 ; CHECK-NEXT:    lsrl r0, r1, #31
-; CHECK-NEXT:    subs.w r2, r0, #-1
+; CHECK-NEXT:    subs.w r3, r0, #-1
 ; CHECK-NEXT:    sbcs r1, r1, #0
 ; CHECK-NEXT:    it hs
 ; CHECK-NEXT:    movhs.w r0, #-1
-; CHECK-NEXT:    str r0, [r8], #4
+; CHECK-NEXT:    str r0, [r2], #4
 ; CHECK-NEXT:    le lr, .LBB3_7
 ; CHECK-NEXT:  .LBB3_8: @ %for.cond.cleanup
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 entry:
   switch i32 %N, label %vector.ph [
     i32 0, label %for.cond.cleanup
     i32 1, label %for.body.preheader

llvm/test/CodeGen/Thumb2/mve-scatter-ptrs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp -enable-arm-maskedldst %s -o - | FileCheck %s
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp -enable-arm-maskedldst -opaque-pointers %s -o - | FileCheck %s
 
 ; i32
 
 ; Expand
 define arm_aapcs_vfpcc void @ptr_v2i32(<2 x i32> %v, ptr %offptr) {
 ; CHECK-LABEL: ptr_v2i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrd r1, r0, [r0]
-; CHECK-NEXT:    str r2, [r1]
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    str r1, [r0]
+; CHECK-NEXT:    ldrd r0, r1, [r0]
+; CHECK-NEXT:    str r2, [r0]
+; CHECK-NEXT:    vmov r0, s2
+; CHECK-NEXT:    str r0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <2 x ptr>, ptr %offptr, align 4
   call void @llvm.masked.scatter.v2i32.v2p0(<2 x i32> %v, <2 x ptr> %offs, i32 4, <2 x i1> <i1 true, i1 true>)
   ret void
 }
 
 ; VSTRW.32 Qd, [offs, 0]

 }
 
 ; f32
 
 ; Expand
 define arm_aapcs_vfpcc void @ptr_v2f32(<2 x float> %v, ptr %offptr) {
 ; CHECK-LABEL: ptr_v2f32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ldrd r1, r0, [r0]
-; CHECK-NEXT:    vstr s0, [r1]
-; CHECK-NEXT:    vstr s1, [r0]
+; CHECK-NEXT:    ldrd r0, r1, [r0]
+; CHECK-NEXT:    vstr s0, [r0]
+; CHECK-NEXT:    vstr s1, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <2 x ptr>, ptr %offptr, align 4
   call void @llvm.masked.scatter.v2f32.v2p0(<2 x float> %v, <2 x ptr> %offs, i32 4, <2 x i1> <i1 true, i1 true>)
   ret void
 }
 
 ; VSTRW.32 Qd, [offs, 0]
entry:
   ret void
 }
 
 ; Expand
 define arm_aapcs_vfpcc void @ptr_v2i16_trunc(<2 x i32> %v, ptr %offptr) {
 ; CHECK-LABEL: ptr_v2i16_trunc:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrd r1, r0, [r0]
-; CHECK-NEXT:    strh r2, [r1]
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    strh r1, [r0]
+; CHECK-NEXT:    ldrd r0, r1, [r0]
+; CHECK-NEXT:    strh r2, [r0]
+; CHECK-NEXT:    vmov r0, s2
+; CHECK-NEXT:    strh r0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <2 x ptr>, ptr %offptr, align 4
   %ext = trunc <2 x i32> %v to <2 x i16>
   call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> %ext, <2 x ptr> %offs, i32 2, <2 x i1> <i1 true, i1 true>)
   ret void
 }
 

llvm/test/CodeGen/Thumb2/mve-vecreduce-slp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s
 
 ; Various reductions generated fro SLP vectorizing unrolled loops. Generated
 ; from https://godbolt.org/z/ebxdPh1Kz with some less interesting cases removed.
 
 define i32 @addv2i32i32(ptr %x) {
 ; CHECK-LABEL: addv2i32i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ldrd r1, r0, [r0]
+; CHECK-NEXT:    ldrd r0, r1, [r0]
 ; CHECK-NEXT:    add r0, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = load i32, ptr %x, align 4
   %arrayidx.1 = getelementptr inbounds i32, ptr %x, i32 1
   %1 = load i32, ptr %arrayidx.1, align 4
   %add.1 = add nsw i32 %1, %0
   ret i32 %add.1
entry:
   ret i8 %21
 }
 
 
 
 define i32 @mlav2i32i32(ptr %x, ptr %y) {
 ; CHECK-LABEL: mlav2i32i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ldrd r2, r0, [r0]
-; CHECK-NEXT:    ldrd r3, r1, [r1]
-; CHECK-NEXT:    muls r2, r3, r2
-; CHECK-NEXT:    mla r0, r1, r0, r2
+; CHECK-NEXT:    ldrd r0, r2, [r0]
+; CHECK-NEXT:    ldrd r1, r3, [r1]
+; CHECK-NEXT:    muls r0, r1, r0
+; CHECK-NEXT:    mla r0, r3, r2, r0
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = load i32, ptr %x, align 4
   %1 = load i32, ptr %y, align 4
   %mul = mul nsw i32 %1, %0
   %arrayidx.1 = getelementptr inbounds i32, ptr %x, i32 1
   %2 = load i32, ptr %arrayidx.1, align 4
   %arrayidx1.1 = getelementptr inbounds i32, ptr %y, i32 1

llvm/test/CodeGen/Thumb2/mve-vld3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp,+fp64 -verify-machineinstrs %s -o - | FileCheck %s
 
 ; i32
 
 define void @vld3_v2i32(ptr %src, ptr %dst) {
 ; CHECK-LABEL: vld3_v2i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    ldrd r2, r0, [r0, #16]
+; CHECK-NEXT:    ldrd r0, r2, [r0, #16]
 ; CHECK-NEXT:    vmov.f32 s6, s3
 ; CHECK-NEXT:    vmov r12, lr, d0
 ; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    add r2, r3
+; CHECK-NEXT:    add r0, r3
 ; CHECK-NEXT:    add.w r3, r12, lr
 ; CHECK-NEXT:    add r0, r2
 ; CHECK-NEXT:    vmov r2, s2
 ; CHECK-NEXT:    add r2, r3
 ; CHECK-NEXT:    strd r2, r0, [r1]
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %l1 = load <6 x i32>, ptr %src, align 4

 
 ; i8
 
 define void @vld3_v2i8(ptr %src, ptr %dst) {
 ; CHECK-LABEL: vld3_v2i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .pad #8
 ; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldrd r2, r0, [r0]
-; CHECK-NEXT:    strd r2, r0, [sp]
+; CHECK-NEXT:    ldrd r0, r2, [r0]
+; CHECK-NEXT:    strd r0, r2, [sp]
 ; CHECK-NEXT:    mov r0, sp
 ; CHECK-NEXT:    vldrb.u16 q0, [r0]
 ; CHECK-NEXT:    vmov.u16 r0, q0[4]
 ; CHECK-NEXT:    vmov.u16 r2, q0[3]
 ; CHECK-NEXT:    add r0, r2
 ; CHECK-NEXT:    vmov.u16 r2, q0[5]
 ; CHECK-NEXT:    add r0, r2
 ; CHECK-NEXT:    strb r0, [r1, #1]

llvm/test/CodeGen/Thumb2/mve-vst2.ll

entry:
   ret void
 }
 
 ; f16
 
 define void @vst2_v2f16(ptr %src, ptr %dst) {
 ; CHECK-LABEL: vst2_v2f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ldrd r2, r0, [r0]
-; CHECK-NEXT:    vmov.32 q1[0], r2
-; CHECK-NEXT:    vmov.32 q0[0], r0
+; CHECK-NEXT:    ldrd r0, r2, [r0]
+; CHECK-NEXT:    vmov.32 q1[0], r0
+; CHECK-NEXT:    vmov.32 q0[0], r2
 ; CHECK-NEXT:    vmovx.f16 s5, s4
 ; CHECK-NEXT:    vins.f16 s4, s0
 ; CHECK-NEXT:    vmovx.f16 s0, s0
 ; CHECK-NEXT:    vins.f16 s5, s0
 ; CHECK-NEXT:    vmov r0, r2, d2
 ; CHECK-NEXT:    str r2, [r1, #4]
 ; CHECK-NEXT:    str r0, [r1]
 ; CHECK-NEXT:    bx lr

llvm/test/CodeGen/Thumb2/postinc-distribute.mir

 body:             |
   bb.0:
     liveins: $r0
 
     ; CHECK-LABEL: name: t2LDRi12_posoff
     ; CHECK: liveins: $r0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:gprnopc = COPY $r0
-    ; CHECK-NEXT: [[t2LDRi12_:%[0-9]+]]:rgpr = t2LDRi12 [[COPY]], 0, 14 /* CC::al */, $noreg :: (load (s32))
-    ; CHECK-NEXT: [[t2LDRi12_1:%[0-9]+]]:rgpr = t2LDRi12 [[COPY]], 4, 14 /* CC::al */, $noreg :: (load (s32))
+    ; CHECK-NEXT: [[t2LDRDi8_:%[0-9]+]]:rgpr, [[t2LDRDi8_1:%[0-9]+]]:rgpr = t2LDRDi8 [[COPY]], 0, 14 /* CC::al */, $noreg :: (load (s32))
     ; CHECK-NEXT: [[t2ADDri:%[0-9]+]]:rgpr = nuw t2ADDri [[COPY]], 32, 14 /* CC::al */, $noreg, $noreg
     ; CHECK-NEXT: [[t2LDRi8_:%[0-9]+]]:rgpr = t2LDRi8 [[COPY]], -8, 14 /* CC::al */, $noreg :: (load (s32))
     ; CHECK-NEXT: $r0 = COPY [[t2ADDri]]
     ; CHECK-NEXT: tBX_RET 14 /* CC::al */, $noreg, implicit $r0
     %0:gprnopc = COPY $r0
     %1:rgpr = t2LDRi12 %0, 0, 14, $noreg :: (load (s32), align 4)
     %2:rgpr = nuw t2ADDri %0, 32, 14, $noreg, $noreg
     %3:rgpr = t2LDRi12 %0, 4, 14, $noreg :: (load (s32), align 4)

 body:             |
   bb.0:
     liveins: $r0
 
     ; CHECK-LABEL: name: t2LDRi12_negoff
     ; CHECK: liveins: $r0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:gprnopc = COPY $r0
-    ; CHECK-NEXT: [[t2LDRi12_:%[0-9]+]]:rgpr = t2LDRi12 [[COPY]], 0, 14 /* CC::al */, $noreg :: (load (s32))
-    ; CHECK-NEXT: [[t2LDRi12_1:%[0-9]+]]:rgpr = t2LDRi12 [[COPY]], 4, 14 /* CC::al */, $noreg :: (load (s32))
+    ; CHECK-NEXT: [[t2LDRDi8_:%[0-9]+]]:rgpr, [[t2LDRDi8_1:%[0-9]+]]:rgpr = t2LDRDi8 [[COPY]], 0, 14 /* CC::al */, $noreg :: (load (s32))
     ; CHECK-NEXT: [[t2SUBri:%[0-9]+]]:rgpr = nuw t2SUBri [[COPY]], 32, 14 /* CC::al */, $noreg, $noreg
     ; CHECK-NEXT: [[t2LDRi8_:%[0-9]+]]:rgpr = t2LDRi8 [[COPY]], -8, 14 /* CC::al */, $noreg :: (load (s32))
     ; CHECK-NEXT: $r0 = COPY [[t2SUBri]]
     ; CHECK-NEXT: tBX_RET 14 /* CC::al */, $noreg, implicit $r0
     %0:gprnopc = COPY $r0
     %1:rgpr = t2LDRi12 %0, 0, 14, $noreg :: (load (s32), align 4)
     %2:rgpr = nuw t2SUBri %0, 32, 14, $noreg, $noreg
     %3:rgpr = t2LDRi12 %0, 4, 14, $noreg :: (load (s32), align 4)