[AArch64]Extend merging narrow loads into a wider load

[junbuml]

Summary

This change extends r251438 to handle more narrow load promotions
including byte type, unscaled, and signed. For example, this change will
convert :

ldursh w1, [x0, #-2]
ldurh  w2, [x0, #-4]

into

ldur  w2, [x0, #-4]
asr   w1, w2, #16
and   w2, w2, #0xffff

[mcrosier]

Jun,
You might consider breaking this up into two separate patches:

1. Add the enableNarrowLdPromotion() functionality to address James' concern about this change not necessarily being profitable on all targets.
2. Add the functional change for merging additional narrow loads into wider loads.
   
   Chad

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

87

I don't think you need a member variable. See my comment below.

1251

Please add a period.

1276

Why not have enableNarrowLdPromotion(MBB->getParent()) return a bool and call it here directly, rather than store in a member variable? AFAICT, this is the only use.

[junbuml]

As Chad suggested, I posted a separate patch (D14244) which address James' concern about enabling this conversion only in profitable targets.

[junbuml]

Minor update in comments.

[mcrosier]

Jun,
Does this hit any cases in Spec2000, Spec2006 or the test-suite? If so, do you have any performance numbers?

Chad

[junbuml]

This change is applied in many spec2000/2006 benchmarks, but I wasn't able to see clear performance differences from this patch.

[junbuml]

Rebased this change and updated unit-tests.

[junbuml]

Just removed whitespace

[jmolloy]

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

209

Let's use "isNarrowLoad" here. No need for truncations when it adds nothing.

588

Wouldn't it be more obvious to just use:

assert(OffsetImm & 1 == 0)?

616

These magic numbers look very easy to get wrong. Is there no obvious programmatic way of deriving them?

1231

Why have you done this rename in the same patch? This clutters the patch.

1473

This stuff should not be in the same patch as a functional change. Please separate out the patches.

[junbuml]

Thanks James for your review. I will address your comments and update it soon.

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

209

As you point out below, I will keep the same function name in this patch and will post a separate patch only for renaming.

588

I will update this. Thanks!

616

I agree with your comment. Let me update this in more programmatic way!

1231

Let me keep the same function name in this patch. I will add a separate patch only for renaming.

1473

This was a minor refactoring as D14534 introduced Subtarget as a member variable. I understand this is not perfectly related with this change. I will do this in a separate refactoring patch.

[junbuml]

Addressed James' comments.

[junbuml]

Ping? Please let me know any comment !

[mcrosier]

LGTM with a minor nit. Please make sure the refactoring of isSmallTypeLdMerge->isNarrowLoad, etc lands after this..

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

616

In general, variable names should begin with a capital letter and are written in camel case.

width -> Width
lsbLow -> LSBLow or LsbLow
lsbHigh -> LSBHigh or LsbHigh
immsLow -> ImmsLow
immsHigh -> ImmsHigh

[junbuml]

Thanks James and Chad for the review!

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

616

Oh. Yes. I will fix it !

[junbuml]

Landed in r253577.

lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

 struct AArch64LoadStoreOpt : public MachineFunctionPass {
   static char ID;
   AArch64LoadStoreOpt() : MachineFunctionPass(ID) {
     initializeAArch64LoadStoreOptPass(*PassRegistry::getPassRegistry());
   }
 
   const AArch64InstrInfo *TII;
   const TargetRegisterInfo *TRI;
   const AArch64Subtarget *Subtarget;
 
   // Scan the instructions looking for a load/store that can be combined
   // with the current instruction into a load/store pair.
   // Return the matching instruction if one is found, else MBB->end().
   MachineBasicBlock::iterator findMatchingInsn(MachineBasicBlock::iterator I,
                                                LdStPairFlags &Flags,
                                                unsigned Limit);
   // Merge the two instructions indicated into a single pair-wise instruction.
static bool isUnscaledLdSt(unsigned Opc) {
   case AArch64::STURXi:
   case AArch64::LDURSi:
   case AArch64::LDURDi:
   case AArch64::LDURQi:
   case AArch64::LDURWi:
   case AArch64::LDURXi:
   case AArch64::LDURSWi:
   case AArch64::LDURHHi:
+  case AArch64::LDURBBi:
+  case AArch64::LDURSBWi:
+  case AArch64::LDURSHWi:
     return true;
   }
 }
 
 static bool isUnscaledLdSt(MachineInstr *MI) {
   return isUnscaledLdSt(MI->getOpcode());
 }
 
+static unsigned getBitExtrOpcode(MachineInstr *MI) {
+  switch (MI->getOpcode()) {
+  default:
+    llvm_unreachable("Unexpected opcode.");
+  case AArch64::LDRBBui:
+  case AArch64::LDURBBi:
+  case AArch64::LDRHHui:
+  case AArch64::LDURHHi:
+    return AArch64::UBFMWri;
+  case AArch64::LDRSBWui:
+  case AArch64::LDURSBWi:
+  case AArch64::LDRSHWui:
+  case AArch64::LDURSHWi:
+    return AArch64::SBFMWri;
+  }
+}
+
 static bool isSmallTypeLdMerge(unsigned Opc) {
   switch (Opc) {
   default:
     return false;
   case AArch64::LDRHHui:
   case AArch64::LDURHHi:
+  case AArch64::LDRBBui:
+  case AArch64::LDURBBi:
+  case AArch64::LDRSHWui:
+  case AArch64::LDURSHWi:
+  case AArch64::LDRSBWui:
+  case AArch64::LDURSBWi:
     return true;
-    // FIXME: Add other instructions (e.g, LDRBBui, LDURSHWi, LDRSHWui, etc.).
   }
 }
+
 static bool isSmallTypeLdMerge(MachineInstr *MI) {
   return isSmallTypeLdMerge(MI->getOpcode());
 }
 
 // Scaling factor for unscaled load or store.
 static int getMemScale(MachineInstr *MI) {
   switch (MI->getOpcode()) {
   default:
     llvm_unreachable("Opcode has unknown scale!");
   case AArch64::LDRBBui:
+  case AArch64::LDURBBi:
+  case AArch64::LDRSBWui:
+  case AArch64::LDURSBWi:
   case AArch64::STRBBui:
     return 1;
   case AArch64::LDRHHui:
   case AArch64::LDURHHi:
+  case AArch64::LDRSHWui:
+  case AArch64::LDURSHWi:
   case AArch64::STRHHui:
     return 2;
   case AArch64::LDRSui:
   case AArch64::LDURSi:
   case AArch64::LDRSWui:
   case AArch64::LDURSWi:
   case AArch64::LDRWui:
   case AArch64::LDURWi:
static unsigned getMatchingNonSExtOpcode(unsigned Opc,
   case AArch64::LDRXui:
   case AArch64::LDURXi:
   case AArch64::STRSui:
   case AArch64::STURSi:
   case AArch64::LDRSui:
   case AArch64::LDURSi:
   case AArch64::LDRHHui:
   case AArch64::LDURHHi:
+  case AArch64::LDRBBui:
+  case AArch64::LDURBBi:
     return Opc;
   case AArch64::LDRSWui:
     return AArch64::LDRWui;
   case AArch64::LDURSWi:
     return AArch64::LDURWi;
+  case AArch64::LDRSBWui:
+    return AArch64::LDRBBui;
+  case AArch64::LDRSHWui:
+    return AArch64::LDRHHui;
+  case AArch64::LDURSBWi:
+    return AArch64::LDURBBi;
+  case AArch64::LDURSHWi:
+    return AArch64::LDURHHi;
   }
 }
 
 static unsigned getMatchingPairOpcode(unsigned Opc) {
   switch (Opc) {
   default:
     llvm_unreachable("Opcode has no pairwise equivalent!");
   case AArch64::STRSui:
case AArch64::LDURWi:
     return AArch64::LDPWi;
   case AArch64::LDRXui:
   case AArch64::LDURXi:
     return AArch64::LDPXi;
   case AArch64::LDRSWui:
   case AArch64::LDURSWi:
     return AArch64::LDPSWi;
   case AArch64::LDRHHui:
+  case AArch64::LDRSHWui:
     return AArch64::LDRWui;
   case AArch64::LDURHHi:
+  case AArch64::LDURSHWi:
     return AArch64::LDURWi;
+  case AArch64::LDRBBui:
+  case AArch64::LDRSBWui:
+    return AArch64::LDRHHui;
+  case AArch64::LDURBBi:
+  case AArch64::LDURSBWi:
+    return AArch64::LDURHHi;
   }
 }
 
 static unsigned getPreIndexedOpcode(unsigned Opc) {
   switch (Opc) {
   default:
     llvm_unreachable("Opcode has no pre-indexed equivalent!");
   case AArch64::STRSui:
if (getLdStOffsetOp(I).getImm() ==
     RtMI = I;
     Rt2MI = Paired;
   }
 
   int OffsetImm = getLdStOffsetOp(RtMI).getImm();
 
   if (isSmallTypeLdMerge(Opc)) {
     // Change the scaled offset from small to large type.
-    if (!IsUnscaled)
+    if (!IsUnscaled) {
+      assert(((OffsetImm & 1) == 0) && "Unexpected offset to merge");
       OffsetImm /= 2;
+    }
     MachineInstr *RtNewDest = MergeForward ? I : Paired;
     // When merging small (< 32 bit) loads for big-endian targets, the order of
     // the component parts gets swapped.
     if (!Subtarget->isLittleEndian())
       std::swap(RtMI, Rt2MI);
     // Construct the new load instruction.
-    // FIXME: currently we support only halfword unsigned load. We need to
-    // handle byte type, signed, and store instructions as well.
     MachineInstr *NewMemMI, *BitExtMI1, *BitExtMI2;
     NewMemMI = BuildMI(*I->getParent(), InsertionPoint, I->getDebugLoc(),
                        TII->get(NewOpc))
                    .addOperand(getLdStRegOp(RtNewDest))
                    .addOperand(BaseRegOp)
                    .addImm(OffsetImm);
 
     // Copy MachineMemOperands from the original loads.
     concatenateMemOperands(NewMemMI, I, Paired);
 
     DEBUG(
         dbgs()
         << "Creating the new load and extract. Replacing instructions:\n    ");
     DEBUG(I->print(dbgs()));
     DEBUG(dbgs() << "    ");
     DEBUG(Paired->print(dbgs()));
     DEBUG(dbgs() << "  with instructions:\n    ");
     DEBUG((NewMemMI)->print(dbgs()));
 
+    int width = getMemScale(I) == 1 ? 8 : 16;
+    int lsbLow = 0;
+    int lsbHigh = width;
+    int immsLow = lsbLow + width - 1;
+    int immsHigh = lsbHigh + width - 1;
     MachineInstr *ExtDestMI = MergeForward ? Paired : I;
     if ((ExtDestMI == Rt2MI) == Subtarget->isLittleEndian()) {
-      // Create the bitfield extract for high half.
+      // Create the bitfield extract for high bits.
       BitExtMI1 = BuildMI(*I->getParent(), InsertionPoint, I->getDebugLoc(),
-                          TII->get(AArch64::UBFMWri))
+                          TII->get(getBitExtrOpcode(Rt2MI)))
                       .addOperand(getLdStRegOp(Rt2MI))
                       .addReg(getLdStRegOp(RtNewDest).getReg())
-                      .addImm(16)
-                      .addImm(31);
-      // Create the bitfield extract for low half.
+                      .addImm(lsbHigh)
+                      .addImm(immsHigh);
+      // Create the bitfield extract for low bits.
+      if (RtMI->getOpcode() == getMatchingNonSExtOpcode(RtMI->getOpcode())) {
+        // For unsigned, prefer to use AND for low bits.
         BitExtMI2 = BuildMI(*I->getParent(), InsertionPoint, I->getDebugLoc(),
                             TII->get(AArch64::ANDWri))
                         .addOperand(getLdStRegOp(RtMI))
                         .addReg(getLdStRegOp(RtNewDest).getReg())
-                      .addImm(15);
+                        .addImm(immsLow);
       } else {
-      // Create the bitfield extract for low half.
+        BitExtMI2 = BuildMI(*I->getParent(), InsertionPoint, I->getDebugLoc(),
+                            TII->get(getBitExtrOpcode(RtMI)))
+                        .addOperand(getLdStRegOp(RtMI))
+                        .addReg(getLdStRegOp(RtNewDest).getReg())
+                        .addImm(lsbLow)
+                        .addImm(immsLow);
+      }
+    } else {
+      // Create the bitfield extract for low bits.
+      if (RtMI->getOpcode() == getMatchingNonSExtOpcode(RtMI->getOpcode())) {
+        // For unsigned, prefer to use AND for low bits.
         BitExtMI1 = BuildMI(*I->getParent(), InsertionPoint, I->getDebugLoc(),
                             TII->get(AArch64::ANDWri))
                         .addOperand(getLdStRegOp(RtMI))
                         .addReg(getLdStRegOp(RtNewDest).getReg())
-                      .addImm(15);
-      // Create the bitfield extract for high half.
+                        .addImm(immsLow);
+      } else {
+        BitExtMI1 = BuildMI(*I->getParent(), InsertionPoint, I->getDebugLoc(),
+                            TII->get(getBitExtrOpcode(RtMI)))
+                        .addOperand(getLdStRegOp(RtMI))
+                        .addReg(getLdStRegOp(RtNewDest).getReg())
+                        .addImm(lsbLow)
+                        .addImm(immsLow);
+      }
+
+      // Create the bitfield extract for high bits.
       BitExtMI2 = BuildMI(*I->getParent(), InsertionPoint, I->getDebugLoc(),
-                          TII->get(AArch64::UBFMWri))
+                          TII->get(getBitExtrOpcode(Rt2MI)))
                       .addOperand(getLdStRegOp(Rt2MI))
                       .addReg(getLdStRegOp(RtNewDest).getReg())
-                      .addImm(16)
-                      .addImm(31);
+                      .addImm(lsbHigh)
+                      .addImm(immsHigh);
     }
     DEBUG(dbgs() << "    ");
     DEBUG((BitExtMI1)->print(dbgs()));
     DEBUG(dbgs() << "    ");
     DEBUG((BitExtMI2)->print(dbgs()));
     DEBUG(dbgs() << "\n");
 
     // Erase the old instructions.
bool AArch64LoadStoreOpt::tryToMergeLdStInst(
   if (TII->isLdStPairSuppressed(MI))
     return false;
 
   // Look ahead up to ScanLimit instructions for a pairable instruction.
   LdStPairFlags Flags;
   MachineBasicBlock::iterator Paired = findMatchingInsn(MBBI, Flags, ScanLimit);
   if (Paired != E) {
     if (isSmallTypeLdMerge(MI)) {
       ++NumSmallTypeMerged;
     } else {
       ++NumPairCreated;
       if (isUnscaledLdSt(MI))
         ++NumUnscaledPairCreated;
     }
 
     // Merge the loads into a pair. Keeping the iterator straight is a
     // pain, so we let the merge routine tell us what the next instruction
     // is after it's done mucking about.
     MBBI = mergePairedInsns(MBBI, Paired, Flags);
     return true;
   }
   return false;
 }
 
 bool AArch64LoadStoreOpt::optimizeBlock(MachineBasicBlock &MBB,
                                         bool enableNarrowLdOpt) {
   bool Modified = false;
   // Three tranformations to do here:
-  // 1) Find halfword loads that can be merged into a single 32-bit word load
+  // 1) Find narrow loads that can be converted into a single wider load
   //    with bitfield extract instructions.
   //      e.g.,
   //        ldrh w0, [x2]
   //        ldrh w1, [x2, #2]
   //        ; becomes
   //        ldr w0, [x2]
   //        ubfx w1, w0, #16, #16
   //        and w0, w0, #ffff
   // 2) Find loads and stores that can be merged into a single load or store
   //    pair instruction.
   //      e.g.,
   //        ldr x0, [x2]
   //        ldr x1, [x2, #8]
   //        ; becomes
   //        ldp x0, x1, [x2]
   // 3) Find base register updates that can be merged into the load or store
   //    as a base-reg writeback.
   //      e.g.,
   //        ldr x0, [x2]
   //        add x2, x2, #4
   //        ; becomes
   //        ldr x0, [x2], #4
 
   for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
        enableNarrowLdOpt && MBBI != E;) {
     MachineInstr *MI = MBBI;
     switch (MI->getOpcode()) {
     default:
       // Just move on to the next instruction.
       ++MBBI;
       break;
     // Scaled instructions.
+    case AArch64::LDRBBui:
     case AArch64::LDRHHui:
+    case AArch64::LDRSBWui:
+    case AArch64::LDRSHWui:
     // Unscaled instructions.
-    case AArch64::LDURHHi: {
+    case AArch64::LDURBBi:
+    case AArch64::LDURHHi:
+    case AArch64::LDURSBWi:
+    case AArch64::LDURSHWi: {
       if (tryToMergeLdStInst(MBBI)) {
         Modified = true;
         break;
       }
       ++MBBI;
       break;
     }
       // FIXME: Do the other instructions.

 bool AArch64LoadStoreOpt::enableNarrowLdMerge(MachineFunction &Fn) {
   const AArch64Subtarget *SubTarget =
       &static_cast<const AArch64Subtarget &>(Fn.getSubtarget());
   bool ProfitableArch = SubTarget->isCortexA57();
   // FIXME: The benefit from converting narrow loads into a wider load could be
   // microarchitectural as it assumes that a single load with two bitfield
   // extracts is cheaper than two narrow loads. Currently, this conversion is
   // enabled only in cortex-a57 on which performance benefits were verified.
   return ProfitableArch & (!SubTarget->requiresStrictAlign());
 }
 
 bool AArch64LoadStoreOpt::runOnMachineFunction(MachineFunction &Fn) {
   Subtarget = &static_cast<const AArch64Subtarget &>(Fn.getSubtarget());
   TII = static_cast<const AArch64InstrInfo *>(Subtarget->getInstrInfo());
   TRI = Subtarget->getRegisterInfo();
 
   bool Modified = false;

test/CodeGen/AArch64/arm64-ldr-merge.ll

define i16 @Ldrh_4_merge(i16* nocapture readonly %P) {
   %l2 = load i16, i16* %arrayidx7
   %arrayidx12 = getelementptr inbounds i16, i16* %P, i64 3
   %l3 = load i16, i16* %arrayidx12
   %add4 = sub nuw nsw i16 %l1, %l0
   %add9 = udiv i16 %add4, %l2
   %add14 = sub nuw nsw i16 %add9, %l3
   ret i16 %add14
 }
+
+; CHECK-LABEL: Ldrsh_merge
+; CHECK: ldr [[NEW_DEST:w[0-9]+]]
+; CHECK-DAG: asr [[LO_PART:w[0-9]+]], [[NEW_DEST]], #16
+; CHECK-DAG: sxth [[HI_PART:w[0-9]+]], [[NEW_DEST]]
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+
+define i32 @Ldrsh_merge(i16* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i16, i16* %p, i64 4
+  %tmp = load i16, i16* %add.ptr0
+  %add.ptr = getelementptr inbounds i16, i16* %p, i64 5
+  %tmp1 = load i16, i16* %add.ptr
+  %sexttmp = sext i16 %tmp to i32
+  %sexttmp1 = sext i16 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp1, %sexttmp
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldrsh_zsext_merge
+; CHECK: ldr [[NEW_DEST:w[0-9]+]]
+; LE-DAG: and [[LO_PART:w[0-9]+]], [[NEW_DEST]], #0xffff
+; LE-DAG: asr [[HI_PART:w[0-9]+]], [[NEW_DEST]], #16
+; BE-DAG: sxth [[LO_PART:w[0-9]+]], [[NEW_DEST]]
+; BE-DAG: lsr [[HI_PART:w[0-9]+]], [[NEW_DEST]], #16
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldrsh_zsext_merge(i16* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i16, i16* %p, i64 4
+  %tmp = load i16, i16* %add.ptr0
+  %add.ptr = getelementptr inbounds i16, i16* %p, i64 5
+  %tmp1 = load i16, i16* %add.ptr
+  %sexttmp = zext i16 %tmp to i32
+  %sexttmp1 = sext i16 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldrsh_szext_merge
+; CHECK: ldr [[NEW_DEST:w[0-9]+]]
+; LE-DAG: sxth [[LO_PART:w[0-9]+]], [[NEW_DEST]]
+; LE-DAG: lsr [[HI_PART:w[0-9]+]], [[NEW_DEST]], #16
+; BE-DAG: and [[LO_PART:w[0-9]+]], [[NEW_DEST]], #0xffff
+; BE-DAG: asr [[HI_PART:w[0-9]+]], [[NEW_DEST]], #16
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldrsh_szext_merge(i16* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i16, i16* %p, i64 4
+  %tmp = load i16, i16* %add.ptr0
+  %add.ptr = getelementptr inbounds i16, i16* %p, i64 5
+  %tmp1 = load i16, i16* %add.ptr
+  %sexttmp = sext i16 %tmp to i32
+  %sexttmp1 = zext i16 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldrb_merge
+; CHECK: ldrh [[NEW_DEST:w[0-9]+]]
+; CHECK-DAG: and [[LO_PART:w[0-9]+]], [[NEW_DEST]], #0xff
+; CHECK-DAG: ubfx [[HI_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldrb_merge(i8* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i8, i8* %p, i64 2
+  %tmp = load i8, i8* %add.ptr0
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 3
+  %tmp1 = load i8, i8* %add.ptr
+  %sexttmp = zext i8 %tmp to i32
+  %sexttmp1 = zext i8 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldrsb_merge
+; CHECK: ldrh [[NEW_DEST:w[0-9]+]]
+; CHECK-DAG: sxtb [[LO_PART:w[0-9]+]], [[NEW_DEST]]
+; CHECK-DAG: sbfx [[HI_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldrsb_merge(i8* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i8, i8* %p, i64 2
+  %tmp = load i8, i8* %add.ptr0
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 3
+  %tmp1 = load i8, i8* %add.ptr
+  %sexttmp = sext i8 %tmp to i32
+  %sexttmp1 = sext i8 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldrsb_zsext_merge
+; CHECK: ldrh [[NEW_DEST:w[0-9]+]]
+; LE-DAG: and [[LO_PART:w[0-9]+]], [[NEW_DEST]], #0xff
+; LE-DAG: sbfx [[HI_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; BE-DAG: sxtb [[LO_PART:w[0-9]+]], [[NEW_DEST]]
+; BE-DAG: ubfx [[HI_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldrsb_zsext_merge(i8* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i8, i8* %p, i64 2
+  %tmp = load i8, i8* %add.ptr0
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 3
+  %tmp1 = load i8, i8* %add.ptr
+  %sexttmp = zext i8 %tmp to i32
+  %sexttmp1 = sext i8 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldrsb_szext_merge
+; CHECK: ldrh [[NEW_DEST:w[0-9]+]]
+; LE-DAG: sxtb [[LO_PART:w[0-9]+]], [[NEW_DEST]]
+; LE-DAG: ubfx [[HI_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; BE-DAG: and [[LO_PART:w[0-9]+]], [[NEW_DEST]], #0xff
+; BE-DAG: sbfx [[HI_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldrsb_szext_merge(i8* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i8, i8* %p, i64 2
+  %tmp = load i8, i8* %add.ptr0
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 3
+  %tmp1 = load i8, i8* %add.ptr
+  %sexttmp = sext i8 %tmp to i32
+  %sexttmp1 = zext i8 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldursh_merge
+; CHECK: ldur [[NEW_DEST:w[0-9]+]]
+; CHECK-DAG: asr  [[LO_PART:w[0-9]+]], [[NEW_DEST]], #16
+; CHECK-DAG: sxth [[HI_PART:w[0-9]+]], [[NEW_DEST]]
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldursh_merge(i16* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i16, i16* %p, i64 -1
+  %tmp = load i16, i16* %add.ptr0
+  %add.ptr = getelementptr inbounds i16, i16* %p, i64 -2
+  %tmp1 = load i16, i16* %add.ptr
+  %sexttmp = sext i16 %tmp to i32
+  %sexttmp1 = sext i16 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldursh_zsext_merge
+; CHECK: ldur [[NEW_DEST:w[0-9]+]]
+; LE-DAG: lsr  [[LO_PART:w[0-9]+]], [[NEW_DEST]], #16
+; LE-DAG: sxth [[HI_PART:w[0-9]+]], [[NEW_DEST]]
+; BE-DAG: asr  [[LO_PART:w[0-9]+]], [[NEW_DEST]], #16
+; BE-DAG: and [[HI_PART:w[0-9]+]], [[NEW_DEST]], #0xffff
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldursh_zsext_merge(i16* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i16, i16* %p, i64 -1
+  %tmp = load i16, i16* %add.ptr0
+  %add.ptr = getelementptr inbounds i16, i16* %p, i64 -2
+  %tmp1 = load i16, i16* %add.ptr
+  %sexttmp = zext i16 %tmp to i32
+  %sexttmp1 = sext i16 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldursh_szext_merge
+; CHECK: ldur [[NEW_DEST:w[0-9]+]]
+; LE-DAG: asr  [[LO_PART:w[0-9]+]], [[NEW_DEST]], #16
+; LE-DAG: and [[HI_PART:w[0-9]+]], [[NEW_DEST]], #0xffff
+; BE-DAG: lsr  [[LO_PART:w[0-9]+]], [[NEW_DEST]], #16
+; BE-DAG: sxth [[HI_PART:w[0-9]+]], [[NEW_DEST]]
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldursh_szext_merge(i16* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i16, i16* %p, i64 -1
+  %tmp = load i16, i16* %add.ptr0
+  %add.ptr = getelementptr inbounds i16, i16* %p, i64 -2
+  %tmp1 = load i16, i16* %add.ptr
+  %sexttmp = sext i16 %tmp to i32
+  %sexttmp1 = zext i16 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldurb_merge
+; CHECK: ldurh [[NEW_DEST:w[0-9]+]]
+; CHECK-DAG: ubfx  [[LO_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; CHECK-DAG: and [[HI_PART:w[0-9]+]], [[NEW_DEST]], #0xff
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldurb_merge(i8* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i8, i8* %p, i64 -1
+  %tmp = load i8, i8* %add.ptr0
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 -2
+  %tmp1 = load i8, i8* %add.ptr
+  %sexttmp = zext i8 %tmp to i32
+  %sexttmp1 = zext i8 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldursb_merge
+; CHECK: ldurh [[NEW_DEST:w[0-9]+]]
+; CHECK-DAG: sbfx [[LO_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; CHECK-DAG: sxtb [[HI_PART:w[0-9]+]], [[NEW_DEST]]
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldursb_merge(i8* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i8, i8* %p, i64 -1
+  %tmp = load i8, i8* %add.ptr0
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 -2
+  %tmp1 = load i8, i8* %add.ptr
+  %sexttmp = sext i8 %tmp to i32
+  %sexttmp1 = sext i8 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldursb_zsext_merge
+; CHECK: ldurh [[NEW_DEST:w[0-9]+]]
+; LE-DAG: ubfx [[LO_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; LE-DAG: sxtb [[HI_PART:w[0-9]+]], [[NEW_DEST]]
+; BE-DAG: sbfx [[LO_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; BE-DAG: and [[HI_PART:w[0-9]+]], [[NEW_DEST]], #0xff
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldursb_zsext_merge(i8* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i8, i8* %p, i64 -1
+  %tmp = load i8, i8* %add.ptr0
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 -2
+  %tmp1 = load i8, i8* %add.ptr
+  %sexttmp = zext i8 %tmp to i32
+  %sexttmp1 = sext i8 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+
+; CHECK-LABEL: Ldursb_szext_merge
+; CHECK: ldurh [[NEW_DEST:w[0-9]+]]
+; LE-DAG: sbfx [[LO_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; LE-DAG: and [[HI_PART:w[0-9]+]], [[NEW_DEST]], #0xff
+; BE-DAG: ubfx [[LO_PART:w[0-9]+]], [[NEW_DEST]], #8, #8
+; BE-DAG: sxtb [[HI_PART:w[0-9]+]], [[NEW_DEST]]
+; LE: sub {{w[0-9]+}}, [[LO_PART]], [[HI_PART]]
+; BE: sub {{w[0-9]+}}, [[HI_PART]], [[LO_PART]]
+define i32 @Ldursb_szext_merge(i8* %p) nounwind {
+  %add.ptr0 = getelementptr inbounds i8, i8* %p, i64 -1
+  %tmp = load i8, i8* %add.ptr0
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 -2
+  %tmp1 = load i8, i8* %add.ptr
+  %sexttmp = sext i8 %tmp to i32
+  %sexttmp1 = zext i8 %tmp1 to i32
+  %add = sub nsw i32 %sexttmp, %sexttmp1
+  ret i32 %add
+}
+