[ARM] Fix so immediates and pc relative checks

[simonwallis2]

Summary

ARM Constant Islands would not detect values that
could not be represented by SO immediates,
breaking only during the emission of object
files.

[simonwallis2]

LGTM.

[dmgreen]

llvm/test/CodeGen/Thumb2/LowOverheadLoops/fast-fp-loops.ll

326

I've not looked into any of the details yet, but what's happening with these changes? I wouldn't expect any of them to be hitting the limit on a adr.

[dmgreen]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

1792

I guess, should this be true?

[dnsampaio]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

1792

Indeed, we are testing t1 coding here.

llvm/test/CodeGen/Thumb2/LowOverheadLoops/fast-fp-loops.ll

326

Were consequence of not performing the reduction, due the error below.

[dnsampaio]

Fixed true -> false for isSOImm when checking for reducing to t1 instructions

[simonwallis2]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

1170

IsSoImm and NegOk arguments are in the wrong order.

[dnsampaio]

Feel free to move on this patch, I'm in vacations.

[simonwallis2]

Fix order of arguments in 1 call to isCPEntryInRange()

Supply the optional final argument in another instance of call to isCPEntryInRange(),
for readability.

[dmgreen]

Thanks. That does fix the problems I was seeing in thumb2 codesize test cases.

I am still unsure about the PCOffset change. Is that for correctness or an optimisation? Or correctness specific to SoImm due to it's non-uniform nature? I would be fairly surprised if we were always getting the limit of a t2ADR wrong and it had not caused problems up to this point. Either way, can we add some tests for it? And if it is just an independent optimization, consider moving it to a separate patch.

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

1011–1014

long int -> int64_t

[simonwallis2]

After review feedback, I removed the change to pc offset estimates from this patch.
This patch is now just about fixing the implementation of SO immediates.

Rewrite long int as int64_t.

[dmgreen]

Constant island pass is notorious for subtle bugs and being difficult to test.

I created this test case that breaks, by iterating over all values of SPACE.

I think this is why Diogo would have added the PCOffset handling. The pass is usually conservative about the ends of the offset ranges. Now there is a point where we need to be exact where the SOImm range changes from being 2byte addressable to 4byte addressable.

Dave

# RUN: llc --filetype=obj -start-before=arm-cp-islands -o - %s | \
# RUN: llvm-objdump --arch=armv8a --disassemble - | FileCheck %s

--- |
  target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
  target triple = "armv8.2a-arm-none-eabi"

  define dso_local i32 @main() #0 {
    ret i32 0
  }

  attributes #0 = { "frame-pointer"="all" }
  !4 = !{i32 210}

...
---
name:            main
alignment:       4
tracksRegLiveness: true
constants:
  - id:              0
    value:           half 0xH5440
    alignment:       2
machineFunctionInfo: {}
body:             |
  bb.0 (%ir-block.0):
    liveins: $lr

    $sp = frame-setup STMDB_UPD $sp, 14, $noreg, killed $r11, killed $lr
    frame-setup CFI_INSTRUCTION def_cfa_offset 8
    frame-setup CFI_INSTRUCTION offset $lr, -4
    frame-setup CFI_INSTRUCTION offset $r11, -8
    $r11 = frame-setup MOVr killed $sp, 14, $noreg, $noreg
    frame-setup CFI_INSTRUCTION def_cfa_register $r11
    $sp = frame-setup SUBri killed $sp, 80, 14, $noreg, $noreg

    renamable $r1 = LEApcrel %const.0, 14, $noreg
    renamable $r1 = LDRH killed renamable $r1, $noreg, 0, 14, $noreg :: (load 2 from constant-pool)
    INLINEASM &nop, 1, !4
    renamable $r0 = SPACE 1024, undef renamable $r0

    $sp = frame-destroy MOVr $r11, 14, $noreg, $noreg
    $sp = frame-destroy LDMIA_RET $sp, 14, $noreg, def $r11, def $pc, implicit killed $r0
...

[simonwallis2]

Reinstated the change to pc offset.
This is necessary for ARM-state SO immediates.

Added a test case which tries to place a constant pool after a gap of a specified count of bytes,
or when that is not a valid offset it falls back to inserting the constant pool locally.
This test is invoked several times at significant offsets to demonstrate that the limits are correct.

[dmgreen]

Thanks. Nice test.

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

286

We should also include ARM::LEApcrelJT, as they can use SoImm's too.

287

Why 4 and not 8? I think it's meant to be "2 instructions widths"

288

You needn't add t2LEApcrel here. There are lots of opcodes that could be pc-relative (as in initializeFunctionInfo). We only need to include the ones we are being careful about the adjustment - the SoImm codes.

[simonwallis2]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

287

Value of 4 was determined empirically.

[simonwallis2]

getPCOffset: added handling of LEApcrelJT

[dmgreen]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

287

You fill me with confidence. And would 8 empirically have problems too?

[simonwallis2]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

287

Values of 0 or 8 cause: error: out of range pc-relative fixup value
A value of 4 did not cause failures.

Why 4 and not 8?
It depends on where you start measuring. I have not pinpointed where,
but I believe that the base address from which the offset was calculated has already been adjusted by 4 bytes,
to take into account the width of the load instruction,
that is to say that it points to the address of the instruction after the load, rather than the load itself.

[efriedma]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

287

From the ARM reference manual:

• When executing an ARM instruction, PC reads as the address of the current instruction plus 8.
• When executing a Thumb instruction, PC reads as the address of the current instruction plus 4.

So it sort of makes sense that we'd need to distinguish between ARM-mode and Thumb-mode... but it's suspicious that this is only getting applied to LEApcrel, and not all ARM-mode instructions.

1016

I'm a little concerned that using getSOImmVal is going to throw off the algorithm. The algorithm is based around there being some range of legal addresses, and moving constants into that range. The current code doesn't generate discontinuities like this, and I'm worried adding them will cause problems for convergence.

Can we split this change off from whatever changes are necessary for correctness? It should be conservatively correct to treat an soimm offset as "a multiple of 4 between -1020 and 1020".

[simonwallis2]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

1016

I'm a little concerned that using getSOImmVal is going to throw off the algorithm. The algorithm is based around there being some range of legal addresses, and moving constants into that range. The current code doesn't generate discontinuities like this, and I'm worried adding them will cause problems for convergence.

Can we split this change off from whatever changes are necessary for correctness?

I take on board your observations about a range of legal addresses.
Although I have not seen evidence of any problems for convergence with non-contiguous offsets,
they are not necessary for this bugfix.

1016

It should be conservatively correct to treat an soimm offset as "a multiple of 4 between -1020 and 1020".

The current treatment of an SoImm offset as "a multiple of 4 between -1020 and 1020" is part of the cause of this bug.

I will split this patch into smaller parts and resubmit.

[dnsampaio]

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

287

Just as insight, I only applied to the one instruction there was a problem for the moment.
Following the debug information, you can see that the offset values that were computed in this pass were 4 bytes out from the code generated. I did not investigate if it also ocured in arm mode.

llvm/lib/Target/ARM/ARMConstantIslandPass.cpp

 //===----------------------------------------------------------------------===//
 
 #include "ARM.h"
 #include "ARMBaseInstrInfo.h"
 #include "ARMBasicBlockInfo.h"
 #include "ARMMachineFunctionInfo.h"
 #include "ARMSubtarget.h"
 #include "MCTargetDesc/ARMBaseInfo.h"
+#include "MCTargetDesc/ARMMCTargetDesc.h"
 #include "Thumb2InstrInfo.h"
 #include "Utils/ARMBaseInfo.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringRef.h"
bool findAvailableWater(CPUser&U, unsigned UserOffset,
                             water_iterator &WaterIter, bool CloserWater);
     void createNewWater(unsigned CPUserIndex, unsigned UserOffset,
                         MachineBasicBlock *&NewMBB);
     bool handleConstantPoolUser(unsigned CPUserIndex, bool CloserWater);
     void removeDeadCPEMI(MachineInstr *CPEMI);
     bool removeUnusedCPEntries();
     bool isCPEntryInRange(MachineInstr *MI, unsigned UserOffset,
                           MachineInstr *CPEMI, unsigned Disp, bool NegOk,
-                          bool DoDump = false);
+                          const bool IsSoImm, bool DoDump = false);
     bool isWaterInRange(unsigned UserOffset, MachineBasicBlock *Water,
                         CPUser &U, unsigned &Growth);
     bool fixupImmediateBr(ImmBranch &Br);
     bool fixupConditionalBr(ImmBranch &Br);
     bool fixupUnconditionalBr(ImmBranch &Br);
     bool optimizeThumb2Instructions();
     bool optimizeThumb2Branches();
     bool reorderThumb2JumpTables();
     bool preserveBaseRegister(MachineInstr *JumpMI, MachineInstr *LEAMI,
                               unsigned &DeadSize, bool &CanDeleteLEA,
                               bool &BaseRegKill);
     bool optimizeThumb2JumpTables();
     MachineBasicBlock *adjustJTTargetBlockForward(MachineBasicBlock *BB,
                                                   MachineBasicBlock *JTBB);
 
     unsigned getUserOffset(CPUser&) const;
     void dumpBBs();
     void verify();
 
+    unsigned getPCOffset(unsigned Opcode) {
+      switch (Opcode) {
+      case ARM::LEApcrel:
+        return 8;
+      case ARM::t2LEApcrel:
+        return 4;
+      default:
+        return 0;
+      }
+      llvm_unreachable("Should not be here");
+    }
+
     bool isOffsetInRange(unsigned UserOffset, unsigned TrialOffset,
-                         unsigned Disp, bool NegativeOK, bool IsSoImm = false);
+                         unsigned PCOffset, unsigned Disp, bool NegativeOK,
+                         bool IsSoImm);
     bool isOffsetInRange(unsigned UserOffset, unsigned TrialOffset,
                          const CPUser &U) {
       return isOffsetInRange(UserOffset, TrialOffset,
-                             U.getMaxDisp(), U.NegOk, U.IsSoImm);
+                             getPCOffset(U.MI->getOpcode()), U.getMaxDisp(),
+                             U.NegOk, U.IsSoImm);
     }
   };
 
 } // end anonymous namespace
 
 char ARMConstantIslands::ID = 0;
 
 /// verify - check BBOffsets, BBSizes, alignment of islands
 void ARMConstantIslands::verify() {
 #ifndef NDEBUG
   BBInfoVector &BBInfo = BBUtils->getBBInfo();
   assert(std::is_sorted(MF->begin(), MF->end(),
                         [&BBInfo](const MachineBasicBlock &LHS,
                                   const MachineBasicBlock &RHS) {
                           return BBInfo[LHS.getNumber()].postOffset() <
                                  BBInfo[RHS.getNumber()].postOffset();
                         }));
   LLVM_DEBUG(dbgs() << "Verifying " << CPUsers.size() << " CP users.\n");
   for (unsigned i = 0, e = CPUsers.size(); i != e; ++i) {
     CPUser &U = CPUsers[i];
     unsigned UserOffset = getUserOffset(U);
     // Verify offset using the real max displacement without the safety
     // adjustment.
-    if (isCPEntryInRange(U.MI, UserOffset, U.CPEMI, U.getMaxDisp()+2, U.NegOk,
-                         /* DoDump = */ true)) {
+    if (isCPEntryInRange(U.MI, UserOffset, U.CPEMI, U.getMaxDisp() + 2, U.NegOk,
+                         U.IsSoImm, /* DoDump = */ true)) {
       LLVM_DEBUG(dbgs() << "OK\n");
       continue;
     }
     LLVM_DEBUG(dbgs() << "Out of range.\n");
     dumpBBs();
     LLVM_DEBUG(MF->dump());
     llvm_unreachable("Constant pool entry out of range!");
   }

 
 /// isOffsetInRange - Checks whether UserOffset (the location of a constant pool
 /// reference) is within MaxDisp of TrialOffset (a proposed location of a
 /// constant pool entry).
 /// UserOffset is computed by getUserOffset above to include PC adjustments. If
 /// the mod 4 alignment of UserOffset is not known, the uncertainty must be
 /// subtracted from MaxDisp instead. CPUser::getMaxDisp() does that.
 bool ARMConstantIslands::isOffsetInRange(unsigned UserOffset,
-                                         unsigned TrialOffset, unsigned MaxDisp,
+                                         unsigned TrialOffset,
+                                         unsigned PCOffset, unsigned MaxDisp,
                                          bool NegativeOK, bool IsSoImm) {
-  if (UserOffset <= TrialOffset) {
+  long int Delta =
+      (long int)TrialOffset - ((long int)PCOffset + (long int)UserOffset);
+  if (Delta >= 0) {
     // User before the Trial.
-    if (TrialOffset - UserOffset <= MaxDisp)
+    if (IsSoImm)
+      return (ARM_AM::getSOImmVal(Delta) != -1);
+    if (Delta <= MaxDisp)
       return true;
-    // FIXME: Make use full range of soimm values.
   } else if (NegativeOK) {
-    if (UserOffset - TrialOffset <= MaxDisp)
+    Delta = -Delta;
+    if (IsSoImm)
+      return (ARM_AM::getSOImmVal(Delta) != -1);
+    if (Delta <= MaxDisp)
       return true;
-    // FIXME: Make use full range of soimm values.
   }
   return false;
 }
 
 /// isWaterInRange - Returns true if a CPE placed after the specified
 /// Water (a basic block) will be in range for the specific MI.
 ///
 /// Compute how much the function will grow by inserting a CPE after Water.
if (CPEEnd > NextBlockOffset) {
     Growth = 0;
 
   return isOffsetInRange(UserOffset, CPEOffset, U);
 }
 
 /// isCPEntryInRange - Returns true if the distance between specific MI and
 /// specific ConstPool entry instruction can fit in MI's displacement field.
 bool ARMConstantIslands::isCPEntryInRange(MachineInstr *MI, unsigned UserOffset,
                                           MachineInstr *CPEMI, unsigned MaxDisp,
-                                      bool NegOk, bool DoDump) {
+                                          bool NegOk, const bool IsSoImm,
+                                          bool DoDump) {
   unsigned CPEOffset = BBUtils->getOffsetOf(CPEMI);
 
+  unsigned PCOffset = getPCOffset(MI->getOpcode());
   if (DoDump) {
     LLVM_DEBUG({
-        BBInfoVector &BBInfo = BBUtils->getBBInfo();
       unsigned Block = MI->getParent()->getNumber();
       const BasicBlockInfo &BBI = BBInfo[Block];
       dbgs() << "User of CPE#" << CPEMI->getOperand(0).getImm()
              << " max delta=" << MaxDisp
              << format(" insn address=%#x", UserOffset) << " in "
              << printMBBReference(*MI->getParent()) << ": "
              << format("%#x-%x\t", BBI.Offset, BBI.postOffset()) << *MI
              << format("CPE address=%#x offset=%+d: ", CPEOffset,
-                       int(CPEOffset - UserOffset));
+                       int(CPEOffset - (UserOffset + PCOffset)));
     });
   }
 
-  return isOffsetInRange(UserOffset, CPEOffset, MaxDisp, NegOk);
+  return isOffsetInRange(UserOffset, CPEOffset, PCOffset, MaxDisp, NegOk,
+                         IsSoImm);
 }
 
 #ifndef NDEBUG
 /// BBIsJumpedOver - Return true of the specified basic block's only predecessor
 /// unconditionally branches to its only successor.
 static bool BBIsJumpedOver(MachineBasicBlock *MBB) {
   if (MBB->pred_size() != 1 || MBB->succ_size() != 1)
     return false;

 /// 1 = entry found, and there were no code insertions or deletions
 /// 2 = entry found, and there were code insertions or deletions
 int ARMConstantIslands::findInRangeCPEntry(CPUser& U, unsigned UserOffset) {
   MachineInstr *UserMI = U.MI;
   MachineInstr *CPEMI  = U.CPEMI;
 
   // Check to see if the CPE is already in-range.
   if (isCPEntryInRange(UserMI, UserOffset, CPEMI, U.getMaxDisp(), U.NegOk,
-                       true)) {
+                       U.IsSoImm, true)) {
     LLVM_DEBUG(dbgs() << "In range\n");
     return 1;
   }
 
   // No.  Look for previously created clones of the CPE that are in range.
   unsigned CPI = getCombinedIndex(CPEMI);
   std::vector<CPEntry> &CPEs = CPEntries[CPI];
   for (unsigned i = 0, e = CPEs.size(); i != e; ++i) {
     // We already tried this one
     if (CPEs[i].CPEMI == CPEMI)
       continue;
     // Removing CPEs can leave empty entries, skip
     if (CPEs[i].CPEMI == nullptr)
       continue;
     if (isCPEntryInRange(UserMI, UserOffset, CPEs[i].CPEMI, U.getMaxDisp(),
-                     U.NegOk)) {
+                         U.IsSoImm, U.NegOk)) {
       LLVM_DEBUG(dbgs() << "Replacing CPE#" << CPI << " with CPE#"
                         << CPEs[i].CPI << "\n");
       // Point the CPUser node to the replacement
       U.CPEMI = CPEs[i].CPEMI;
       // Change the CPI in the instruction operand to refer to the clone.
       for (unsigned j = 0, e = UserMI->getNumOperands(); j != e; ++j)
         if (UserMI->getOperand(j).isCPI()) {
           UserMI->getOperand(j).setIndex(CPEs[i].CPI);
for (unsigned i = 0, e = CPUsers.size(); i != e; ++i) {
     unsigned UserOffset = getUserOffset(U);
     unsigned MaxOffs = ((1 << Bits) - 1) * Scale;
 
     // Be conservative with inline asm.
     if (!U.KnownAlignment)
       MaxOffs -= 2;
 
     // FIXME: Check if offset is multiple of scale if scale is not 4.
-    if (isCPEntryInRange(U.MI, UserOffset, U.CPEMI, MaxOffs, false, true)) {
+    if (isCPEntryInRange(U.MI, UserOffset, U.CPEMI, MaxOffs, false, true,
+                         true)) {
       LLVM_DEBUG(dbgs() << "Shrink: " << *U.MI);
       U.MI->setDesc(TII->get(NewOpc));
       MachineBasicBlock *MBB = U.MI->getParent();
       BBUtils->adjustBBSize(MBB, -2);
       BBUtils->adjustBBOffsetsAfter(MBB);
       ++NumT2CPShrunk;
       MadeChange = true;
     }

llvm/test/CodeGen/ARM/constant-island-SOImm.mir

+# RUN: llc --filetype=obj -start-before=arm-cp-islands -o - %s | llvm-objdump --arch=armv8a --disassemble - | FileCheck %s
+# CHECK: add     r{{[1-9]}}, pc, #252
+# CHECK: add     r{{[1-9]}}, pc, #260
+
+--- |
+  target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
+  target triple = "armv8.2a-arm-none-eabi"
+
+  define dso_local void @a(half %0, half %1) #0 {
+    %3 = alloca half, align 2
+    %4 = alloca half, align 2
+    store half %0, half* %3, align 2
+    store half %1, half* %4, align 2
+    ret void
+  }
+
+  define dso_local i32 @main() #0 {
+    %1 = alloca i32, align 4
+    %2 = alloca <16 x i8>, align 8
+    %3 = alloca <16 x i8>, align 8
+    %4 = alloca [16 x i8], align 1
+    %5 = alloca <16 x i8>, align 8
+    %6 = alloca i32, align 4
+    store i32 0, i32* %1, align 4
+    call void @a(half 0xH5440, half 0xH61B0)
+    %7 = getelementptr inbounds [16 x i8], [16 x i8]* %4, i32 0, i32 0
+    store i8 1, i8* %7, align 1
+    %8 = getelementptr inbounds i8, i8* %7, i32 1
+    store i8 9, i8* %8, align 1
+    %9 = getelementptr inbounds i8, i8* %8, i32 1
+    store i8 31, i8* %9, align 1
+    %10 = getelementptr inbounds i8, i8* %9, i32 1
+    store i8 30, i8* %10, align 1
+    %11 = getelementptr inbounds i8, i8* %10, i32 1
+    store i8 2, i8* %11, align 1
+    %12 = getelementptr inbounds i8, i8* %11, i32 1
+    store i8 5, i8* %12, align 1
+    %13 = getelementptr inbounds i8, i8* %12, i32 1
+    store i8 4, i8* %13, align 1
+    %14 = getelementptr inbounds i8, i8* %13, i32 1
+    store i8 7, i8* %14, align 1
+    %15 = getelementptr inbounds i8, i8* %14, i32 1
+    store i8 35, i8* %15, align 1
+    %16 = getelementptr inbounds i8, i8* %15, i32 1
+    store i8 29, i8* %16, align 1
+    %17 = getelementptr inbounds i8, i8* %16, i32 1
+    store i8 3, i8* %17, align 1
+    %18 = getelementptr inbounds i8, i8* %17, i32 1
+    store i8 85, i8* %18, align 1
+    %19 = getelementptr inbounds i8, i8* %18, i32 1
+    store i8 60, i8* %19, align 1
+    %20 = getelementptr inbounds i8, i8* %19, i32 1
+    store i8 50, i8* %20, align 1
+    %21 = getelementptr inbounds i8, i8* %20, i32 1
+    store i8 8, i8* %21, align 1
+    %22 = getelementptr inbounds i8, i8* %21, i32 1
+    store i8 14, i8* %22, align 1
+    %23 = getelementptr inbounds [16 x i8], [16 x i8]* %4, i32 0, i32 0
+    %24 = call <16 x i8> @llvm.arm.neon.vld1.v16i8.p0i8(i8* %23, i32 1)
+    store <16 x i8> %24, <16 x i8>* %3, align 8
+    %25 = load <16 x i8>, <16 x i8>* %3, align 8
+    store <16 x i8> %25, <16 x i8>* %5, align 8
+    %26 = load <16 x i8>, <16 x i8>* %5, align 8
+    store <16 x i8> %26, <16 x i8>* %2, align 8
+    call void asm sideeffect "nop", ""() #2, !srcloc !3
+    call void asm sideeffect "nop", ""() #2, !srcloc !4
+    store i32 0, i32* %6, align 4
+    br label %27
+
+  27:                                               ; preds = %31, %0
+    %28 = load i32, i32* %6, align 4
+    %29 = icmp slt i32 %28, 1051
+    br i1 %29, label %30, label %34
+
+  30:                                               ; preds = %27
+    br label %31
+
+  31:                                               ; preds = %30
+    %32 = load i32, i32* %6, align 4
+    %33 = add nsw i32 %32, 1
+    store i32 %33, i32* %6, align 4
+    br label %27
+
+  34:                                               ; preds = %27
+    ret i32 0
+  }
+  declare <16 x i8> @llvm.arm.neon.vld1.v16i8.p0i8(i8*, i32) #1
+  declare void @llvm.stackprotector(i8*, i8**) #2
+  attributes #0 = { noinline nounwind optnone "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="all" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+armv8.2-a,+crc,+crypto,+d32,+dsp,+fp-armv8,+fp-armv8d16,+fp-armv8d16sp,+fp-armv8sp,+fp16,+fp64,+fullfp16,+hwdiv,+hwdiv-arm,+neon,+ras,+strict-align,+vfp2,+vfp2sp,+vfp3,+vfp3d16,+vfp3d16sp,+vfp3sp,+vfp4,+vfp4d16,+vfp4d16sp,+vfp4sp,-thumb-mode" "unsafe-fp-math"="false" "use-soft-float"="false" }
+  attributes #1 = { argmemonly nounwind readonly }
+  attributes #2 = { nounwind }
+  !llvm.module.flags = !{!0, !1}
+  !0 = !{i32 1, !"wchar_size", i32 4}
+  !1 = !{i32 1, !"min_enum_size", i32 4}
+  !3 = !{i32 194}
+  !4 = !{i32 210}
+
+...
+---
+name:            a
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$r0' }
+  - { reg: '$r1' }
+frameInfo:
+  stackSize:       4
+  maxAlignment:    2
+  maxCallFrameSize: 0
+  localFrameSize:  4
+stack:
+  - { id: 0, offset: -2, size: 2, alignment: 2, local-offset: -2 }
+  - { id: 1, offset: -4, size: 2, alignment: 2, local-offset: -4 }
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.2):
+    liveins: $r0, $r1
+
+    $sp = frame-setup SUBri killed $sp, 4, 14, $noreg, $noreg
+    frame-setup CFI_INSTRUCTION def_cfa_offset 4
+    renamable $s0 = VMOVHR killed renamable $r1, 14, $noreg
+    renamable $s2 = VMOVHR killed renamable $r0, 14, $noreg
+    VSTRH killed renamable $s2, $sp, 1, 14, $noreg :: (store 2 into %ir.3)
+    VSTRH killed renamable $s0, $sp, 0, 14, $noreg :: (store 2 into %ir.4)
+    $sp = frame-destroy ADDri killed $sp, 4, 14, $noreg, $noreg
+    BX_RET 14, $noreg
+
+...
+---
+name:            main
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  stackSize:       88
+  offsetAdjustment: -80
+  maxAlignment:    8
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+  localFrameSize:  76
+stack:
+  - { id: 0, offset: -12, size: 4, alignment: 4, local-offset: -4 }
+  - { id: 1, offset: -32, size: 16, alignment: 8, local-offset: -24 }
+  - { id: 2, offset: -48, size: 16, alignment: 8, local-offset: -40 }
+  - { id: 3, offset: -64, size: 16, alignment: 1, local-offset: -56 }
+  - { id: 4, offset: -80, size: 16, alignment: 8, local-offset: -72 }
+  - { id: 5, offset: -84, size: 4, alignment: 4, local-offset: -76 }
+  - { id: 6, type: spill-slot, offset: -88, size: 4, alignment: 4 }
+  - { id: 7, type: spill-slot, offset: -4, size: 4, alignment: 4, callee-saved-register: '$lr', 
+      callee-saved-restored: false }
+  - { id: 8, type: spill-slot, offset: -8, size: 4, alignment: 4, callee-saved-register: '$r11' }
+constants:
+  - id:              0
+    value:           half 0xH5440
+    alignment:       2
+  - id:              1
+    value:           half 0xH61B0
+    alignment:       2
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $lr
+
+    $sp = frame-setup STMDB_UPD $sp, 14, $noreg, killed $r11, killed $lr
+    frame-setup CFI_INSTRUCTION def_cfa_offset 8
+    frame-setup CFI_INSTRUCTION offset $lr, -4
+    frame-setup CFI_INSTRUCTION offset $r11, -8
+    $r11 = frame-setup MOVr killed $sp, 14, $noreg, $noreg
+    frame-setup CFI_INSTRUCTION def_cfa_register $r11
+    $sp = frame-setup SUBri killed $sp, 80, 14, $noreg, $noreg
+    renamable $r0 = MOVi 0, 14, $noreg, $noreg
+    STRi12 renamable $r0, $r11, -4, 14, $noreg :: (store 4 into %ir.1)
+    renamable $r1 = LEApcrel %const.0, 14, $noreg
+    renamable $r1 = LDRH killed renamable $r1, $noreg, 0, 14, $noreg :: (load 2 from constant-pool)
+    renamable $r2 = LEApcrel %const.1, 14, $noreg
+    renamable $r2 = LDRH killed renamable $r2, $noreg, 0, 14, $noreg :: (load 2 from constant-pool)
+    STRi12 killed $r0, $sp, 0, 14, $noreg :: (store 4 into %stack.6)
+    $r0 = MOVr killed $r1, 14, $noreg, $noreg
+    $r1 = MOVr killed $r2, 14, $noreg, $noreg
+    BL_pred @a, 14, $noreg, csr_aapcs, implicit-def dead $lr, implicit $sp, implicit killed $r0, implicit killed $r1, implicit-def $sp
+    renamable $r0 = MOVi 1, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 24, 14, $noreg :: (store 1 into %ir.7)
+    renamable $r0 = MOVi 9, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 25, 14, $noreg :: (store 1 into %ir.8)
+    renamable $r0 = MOVi 31, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 26, 14, $noreg :: (store 1 into %ir.9)
+    renamable $r0 = MOVi 30, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 27, 14, $noreg :: (store 1 into %ir.10)
+    renamable $r0 = MOVi 2, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 28, 14, $noreg :: (store 1 into %ir.11)
+    renamable $r0 = MOVi 5, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 29, 14, $noreg :: (store 1 into %ir.12)
+    renamable $r0 = MOVi 4, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 30, 14, $noreg :: (store 1 into %ir.13)
+    renamable $r0 = MOVi 7, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 31, 14, $noreg :: (store 1 into %ir.14)
+    renamable $r0 = MOVi 35, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 32, 14, $noreg :: (store 1 into %ir.15)
+    renamable $r0 = MOVi 29, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 33, 14, $noreg :: (store 1 into %ir.16)
+    renamable $r0 = MOVi 3, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 34, 14, $noreg :: (store 1 into %ir.17)
+    renamable $r0 = MOVi 85, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 35, 14, $noreg :: (store 1 into %ir.18)
+    renamable $r0 = MOVi 60, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 36, 14, $noreg :: (store 1 into %ir.19)
+    renamable $r0 = MOVi 50, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 37, 14, $noreg :: (store 1 into %ir.20)
+    renamable $r0 = MOVi 8, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 38, 14, $noreg :: (store 1 into %ir.21)
+    renamable $r0 = MOVi 14, 14, $noreg, $noreg
+    STRBi12 killed renamable $r0, $sp, 39, 14, $noreg :: (store 1 into %ir.22)
+    renamable $r0 = ADDri $sp, 24, 14, $noreg, $noreg
+    $q8 = VLD1q8 killed renamable $r0, 0, 14, $noreg :: (load 16 from %ir.23, align 1)
+    renamable $r0 = ADDri $sp, 40, 14, $noreg, $noreg
+    VST1q64 $r0, 0, killed $q8, 14, $noreg :: (store 16 into %ir.3, align 8)
+    $q8 = VLD1q64 killed renamable $r0, 0, 14, $noreg :: (dereferenceable load 16 from %ir.3, align 8)
+    renamable $r0 = ADDri $sp, 8, 14, $noreg, $noreg
+    VST1q64 $r0, 0, killed $q8, 14, $noreg :: (store 16 into %ir.5, align 8)
+    $q8 = VLD1q64 killed renamable $r0, 0, 14, $noreg :: (dereferenceable load 16 from %ir.5, align 8)
+    renamable $r0 = SUBri $r11, 24, 14, $noreg, $noreg
+    VST1q64 killed $r0, 0, killed $q8, 14, $noreg :: (store 16 into %ir.2, align 8)
+    INLINEASM &nop, 1, !3
+    INLINEASM &nop, 1, !4
+    $r0 = LDRi12 $sp, 0, 14, $noreg :: (load 4 from %stack.6)
+    STRi12 renamable $r0, $sp, 4, 14, $noreg :: (store 4 into %ir.6)
+    B %bb.1
+
+  bb.1 (%ir-block.27):
+    successors: %bb.2, %bb.4
+
+    renamable $r0 = LDRi12 $sp, 4, 14, $noreg :: (dereferenceable load 4 from %ir.6)
+    renamable $r1 = MOVi16 1050, 14, $noreg
+    CMPrr killed renamable $r0, killed renamable $r1, 14, $noreg, implicit-def $cpsr
+    Bcc %bb.4, 12, $cpsr
+    B %bb.2
+
+  bb.2 (%ir-block.30):
+    B %bb.3
+
+  bb.3 (%ir-block.31):
+    renamable $r0 = LDRi12 $sp, 4, 14, $noreg :: (dereferenceable load 4 from %ir.6)
+    renamable $r0 = nsw ADDri killed renamable $r0, 1, 14, $noreg, $noreg
+    STRi12 killed renamable $r0, $sp, 4, 14, $noreg :: (store 4 into %ir.6)
+    B %bb.1
+
+  bb.4 (%ir-block.34):
+    renamable $r0 = MOVi 0, 14, $noreg, $noreg
+    $sp = frame-destroy MOVr $r11, 14, $noreg, $noreg
+    $sp = frame-destroy LDMIA_RET $sp, 14, $noreg, def $r11, def $pc, implicit killed $r0
+
+...

llvm/test/CodeGen/Thumb2/LowOverheadLoops/fast-fp-loops.ll

 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    vmov.i32 q5, #0x0
 ; CHECK-NEXT:    bic r3, r3, #3
 ; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
 ; CHECK-NEXT:    sub.w r12, r2, #1
-; CHECK-NEXT:    adr r2, .LCPI2_1
+; CHECK-NEXT:    adr.w r2, .LCPI2_1
 ; CHECK-NEXT:    movs r3, #0
 ; CHECK-NEXT:    vldrw.u32 q0, [r2]
 ; CHECK-NEXT:    vdup.32 q1, r12
 ; CHECK-NEXT:    vdup.32 q2, r12
 ; CHECK-NEXT:    vstrw.32 q0, [sp] @ 16-byte Spill
 ; CHECK-NEXT:    b .LBB2_4
 ; CHECK-NEXT:  .LBB2_2: @ %cond.load25
 ; CHECK-NEXT:    @ in Loop: Header=BB2_4 Depth=1

llvm/test/CodeGen/Thumb2/mve-gather-increment.ll

 ; CHECK-NEXT:    sub sp, #104
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    str r1, [sp, #60] @ 4-byte Spill
 ; CHECK-NEXT:    blt.w .LBB12_5
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph.preheader
 ; CHECK-NEXT:    bic r1, r2, #7
 ; CHECK-NEXT:    movs r7, #1
 ; CHECK-NEXT:    sub.w r3, r1, #8
-; CHECK-NEXT:    adr r6, .LCPI12_2
+; CHECK-NEXT:    adr.w r6, .LCPI12_2
 ; CHECK-NEXT:    vmov.i16 q3, #0x18
 ; CHECK-NEXT:    str r1, [sp, #56] @ 4-byte Spill
 ; CHECK-NEXT:    add.w r8, r7, r3, lsr #3
 ; CHECK-NEXT:    adr r7, .LCPI12_1
 ; CHECK-NEXT:    vldrw.u32 q0, [r7]
 ; CHECK-NEXT:    adr r3, .LCPI12_0
 ; CHECK-NEXT:    vstrw.32 q3, [sp, #64] @ 16-byte Spill
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #32] @ 16-byte Spill

 ; CHECK-NEXT:    str r2, [sp, #124] @ 4-byte Spill
 ; CHECK-NEXT:    blt.w .LBB13_5
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph.preheader
 ; CHECK-NEXT:    ldr r1, [sp, #124] @ 4-byte Reload
 ; CHECK-NEXT:    adr.w r6, .LCPI13_8
 ; CHECK-NEXT:    adr.w r7, .LCPI13_7
 ; CHECK-NEXT:    adr.w r3, .LCPI13_6
 ; CHECK-NEXT:    bic r11, r1, #7
-; CHECK-NEXT:    adr r1, .LCPI13_0
+; CHECK-NEXT:    adr.w r1, .LCPI13_0
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    adr r1, .LCPI13_1
+; CHECK-NEXT:    adr.w r1, .LCPI13_1
 ; CHECK-NEXT:    vmov.i32 q5, #0x30
 ; CHECK-NEXT:    str.w r11, [sp, #116] @ 4-byte Spill
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #96] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    adr r1, .LCPI13_5
+; CHECK-NEXT:    adr.w r1, .LCPI13_5
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #80] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r6]
 ; CHECK-NEXT:    adr.w r6, .LCPI13_9
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #64] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r7]
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #48] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r6]
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #32] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #16] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
 ; CHECK-NEXT:    vstrw.32 q0, [sp] @ 16-byte Spill
 ; CHECK-NEXT:  .LBB13_2: @ %vector.ph
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB13_3 Depth 2
 ; CHECK-NEXT:    vldrw.u32 q2, [sp, #32] @ 16-byte Reload
-; CHECK-NEXT:    adr r1, .LCPI13_3
+; CHECK-NEXT:    adr.w r1, .LCPI13_3
 ; CHECK-NEXT:    vldrw.u32 q1, [r1]
-; CHECK-NEXT:    adr r1, .LCPI13_4
+; CHECK-NEXT:    adr.w r1, .LCPI13_4
 ; CHECK-NEXT:    vstrw.32 q2, [sp, #288] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q2, [sp, #48] @ 16-byte Reload
 ; CHECK-NEXT:    vldrw.u32 q3, [r1]
-; CHECK-NEXT:    adr r1, .LCPI13_2
+; CHECK-NEXT:    adr.w r1, .LCPI13_2
 ; CHECK-NEXT:    vstrw.32 q2, [sp, #224] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q2, [sp, #80] @ 16-byte Reload
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    adr r1, .LCPI13_10
+; CHECK-NEXT:    adr.w r1, .LCPI13_10
 ; CHECK-NEXT:    vstrw.32 q2, [sp, #272] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q2, [sp, #64] @ 16-byte Reload
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #304] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    adr r1, .LCPI13_11
+; CHECK-NEXT:    adr.w r1, .LCPI13_11
 ; CHECK-NEXT:    ldr.w r9, [sp, #120] @ 4-byte Reload
 ; CHECK-NEXT:    vstrw.32 q2, [sp, #208] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q2, [sp, #96] @ 16-byte Reload
 ; CHECK-NEXT:    vldrw.u32 q6, [r1]
 ; CHECK-NEXT:    vldrw.u32 q7, [sp] @ 16-byte Reload
 ; CHECK-NEXT:    vldrw.u32 q4, [sp, #16] @ 16-byte Reload
 ; CHECK-NEXT:    vstrw.32 q2, [sp, #192] @ 16-byte Spill
 ; CHECK-NEXT:  .LBB13_3: @ %vector.body

 ; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    .pad #72
 ; CHECK-NEXT:    sub sp, #72
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    blt.w .LBB14_5
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph.preheader
-; CHECK-NEXT:    adr r5, .LCPI14_3
-; CHECK-NEXT:    adr r7, .LCPI14_1
+; CHECK-NEXT:    adr.w r5, .LCPI14_3
+; CHECK-NEXT:    adr.w r7, .LCPI14_1
 ; CHECK-NEXT:    vldrw.u32 q0, [r5]
-; CHECK-NEXT:    adr r6, .LCPI14_2
-; CHECK-NEXT:    adr r3, .LCPI14_0
+; CHECK-NEXT:    adr.w r6, .LCPI14_2
+; CHECK-NEXT:    adr.w r3, .LCPI14_0
 ; CHECK-NEXT:    bic r12, r2, #7
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #48] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r7]
 ; CHECK-NEXT:    vmov.i32 q4, #0x10
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #32] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r6]
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #16] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]

llvm/test/CodeGen/Thumb2/mve-satmul-loops.ll

 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    .pad #24
 ; CHECK-NEXT:    sub sp, #24
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    beq.w .LBB9_3
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph
 ; CHECK-NEXT:    add.w r12, r3, #7
-; CHECK-NEXT:    adr r4, .LCPI9_0
+; CHECK-NEXT:    adr.w r4, .LCPI9_0
 ; CHECK-NEXT:    bic r12, r12, #7
 ; CHECK-NEXT:    vldrw.u32 q0, [r4]
 ; CHECK-NEXT:    sub.w r12, r12, #8
 ; CHECK-NEXT:    mov.w lr, #1
-; CHECK-NEXT:    adr r4, .LCPI9_1
+; CHECK-NEXT:    adr.w r4, .LCPI9_1
 ; CHECK-NEXT:    vmov.i8 q2, #0x0
 ; CHECK-NEXT:    add.w lr, lr, r12, lsr #3
 ; CHECK-NEXT:    sub.w r12, r3, #1
 ; CHECK-NEXT:    vldrw.u32 q4, [r4]
 ; CHECK-NEXT:    movs r3, #0
 ; CHECK-NEXT:    vdup.32 q1, r12
 ; CHECK-NEXT:    vmov.i8 q3, #0xff
 ; CHECK-NEXT:    dls lr, lr

 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    .pad #56
 ; CHECK-NEXT:    sub sp, #56
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    beq.w .LBB18_3
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph
 ; CHECK-NEXT:    add.w r12, r3, #15
-; CHECK-NEXT:    adr r4, .LCPI18_0
+; CHECK-NEXT:    adr.w r4, .LCPI18_0
 ; CHECK-NEXT:    bic r12, r12, #15
 ; CHECK-NEXT:    vldrw.u32 q0, [r4]
 ; CHECK-NEXT:    sub.w r12, r12, #16
 ; CHECK-NEXT:    mov.w lr, #1
-; CHECK-NEXT:    adr r4, .LCPI18_1
+; CHECK-NEXT:    adr.w r4, .LCPI18_1
 ; CHECK-NEXT:    vmov.i8 q2, #0x0
 ; CHECK-NEXT:    add.w lr, lr, r12, lsr #4
 ; CHECK-NEXT:    sub.w r12, r3, #1
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #32] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r4]
-; CHECK-NEXT:    adr r4, .LCPI18_2
+; CHECK-NEXT:    adr.w r4, .LCPI18_2
 ; CHECK-NEXT:    movs r3, #0
 ; CHECK-NEXT:    vdup.32 q1, r12
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #16] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r4]
 ; CHECK-NEXT:    adr r4, .LCPI18_3
 ; CHECK-NEXT:    vmov.i8 q3, #0xff
 ; CHECK-NEXT:    vldrw.u32 q6, [r4]
 ; CHECK-NEXT:    vstrw.32 q0, [sp] @ 16-byte Spill

 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    .pad #56
 ; CHECK-NEXT:    sub sp, #56
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    beq.w .LBB19_3
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph
 ; CHECK-NEXT:    add.w r12, r3, #15
-; CHECK-NEXT:    adr r4, .LCPI19_0
+; CHECK-NEXT:    adr.w r4, .LCPI19_0
 ; CHECK-NEXT:    bic r12, r12, #15
 ; CHECK-NEXT:    vldrw.u32 q0, [r4]
 ; CHECK-NEXT:    sub.w r12, r12, #16
 ; CHECK-NEXT:    mov.w lr, #1
-; CHECK-NEXT:    adr r4, .LCPI19_1
+; CHECK-NEXT:    adr.w r4, .LCPI19_1
 ; CHECK-NEXT:    vmov.i8 q2, #0x0
 ; CHECK-NEXT:    add.w lr, lr, r12, lsr #4
 ; CHECK-NEXT:    sub.w r12, r3, #1
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #32] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r4]
-; CHECK-NEXT:    adr r4, .LCPI19_2
+; CHECK-NEXT:    adr.w r4, .LCPI19_2
 ; CHECK-NEXT:    movs r3, #0
 ; CHECK-NEXT:    vdup.32 q1, r12
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #16] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q0, [r4]
 ; CHECK-NEXT:    adr r4, .LCPI19_3
 ; CHECK-NEXT:    vmov.i8 q3, #0xff
 ; CHECK-NEXT:    vldrw.u32 q6, [r4]
 ; CHECK-NEXT:    vstrw.32 q0, [sp] @ 16-byte Spill