[DAGCombiner] Reduce load widths of shifted masks

[samparker]

Summary

During combining, ReduceLoadWdith is used to combine AND nodes that mask loads into narrow loads. This patch allows the mask to be a shifted constant. This results in a narrow load which is then left shifted to compensate for the new offset.

[RKSimon]

test/CodeGen/ARM/and-load-combine.ll

44

I'd recommend reverting these changes from diff - you can avoid the problem by moving all the arguments to the same line

[samparker]

Moved arguments to occupy a single line.

[dnsampaio]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9083

ActiveBits can be zero, this might throw an error no?
and (load i32, 0x000FF000)
This is a shifted mask. You could use APInt.countPopulation as to get the number of 1s.

9083

Ignore. u do lsrh.

[john.brawn]

test/CodeGen/ARM/and-load-combine.ll

7

It's not clear to me what the purpose of these test changes are (or the previous version). If I revert these changes then the test still pases.

test/CodeGen/X86/fp128-i128.ll

48

r338821 made changes to this test which means the patch fails on this file.

[samparker]

Rebased and updated changes to the x86 codegen tests.

test/CodeGen/ARM/and-load-combine.ll

7

It's just because I've used the update_llc script and the format was different. Moving the arguments onto a single line removed the unnecessary diffs produced by the script.

[john.brawn]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9232

On big-endian targets ShAmt has been adjusted by the time we get here, in which case the shifts we do here are wrong, e.g. many of the tests you've added are checking that in big-endian the load is eliminated which is not what should be happening.

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9232

Ah! Thanks, I'm sure BE always trips me up here.

[samparker]

Re-adjusted ShAmt for big endian targets.

[john.brawn]

LGTM.

[fpichet]

I recently resynced an out of tree backend and I got a miscompile because of this commit.
My target is Big Endian.
The C code is :

void swap(unsigned *ptr) {
  *ptr = (*ptr & 0x0000ff00 ) << 8;		
}

The IR is

%0 = load i32, i32* %ptr, align 4
%and = and i32 %0, 65280
%shl = shl i32 %and, 8
store i32 %shl, i32* %ptr, align 4

The resulting DAG->dump() after the function ReduceLoadWidth returned will be:

t5: i32,ch = load<(load 4 from %ir.ptr)> t0, t2, undef:i32, main.c:8:4
t6: i32 = Constant<65280>
  t12: i32 = add nuw t2, Constant:i32<2>, main.c:8:4
t13: i32,ch = load<(load 1 from %ir.ptr + 2, align 2), zext from i8> t0, t12, undef:i32, main.c:8:4
    t7: i32 = and t13, Constant:i32<255>, main.c:8:4
  t9: i32 = shl t7, Constant:i32<8>, main.c:8:4
t10: ch = store<(store 4 into %ir.ptr)> t13:1, t9, t2, undef:i32, main.c:8:4

There is a missing shl by 8 missing. (yes there should be 2 shl to be combined later)

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9252 ↗

(On Diff #161681)

I think this line should be:
SDValue Shifted = DAG.getNode(ISD::SHL, DL, VT, N->getOperand(0),

[samparker]

Hi,

Could you please describe more accurately what you are expecting to see? I'm failing to see the issue.

[fpichet]

For example:

given test.ll:
define dso_local void @swap(i32* %ptr) #0 {
entry:

%0 = load i32, i32* %ptr, align 4
%and = and i32 %0, 65280
%shl = shl i32 %and, 8
store i32 %shl, i32* %ptr, align 4
ret void

}

llc -mtriple=armv7-linux-gnu < test.ll
will give:

ldrb    r1, [r0, #1]
lsl     r1, r1, #8   // 8
str     r1, [r0]
bx      lr

it should be:

ldrb    r1, [r0, #1]
lsl     r1, r1, #16  // 16
str     r1, [r0]
bx      lr

[samparker]

Thanks. Yes, I think this is to do with the way I try to insert the new shl node, the DAG is just returning the existing node hence we only have one shl node. Should have a patch up tomorrow.

[samparker]

Changed the method of creating the final shl.

[fpichet]

I tried your patch and the bug is gone. Thanks.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

   SDValue N0 = N->getOperand(0);
   EVT VT = N->getValueType(0);
   EVT ExtVT = VT;
 
   // This transformation isn't valid for vector loads.
   if (VT.isVector())
     return SDValue();
 
+  unsigned ShAmt = 0;
+  bool HasShiftedOffset = false;
   // Special case: SIGN_EXTEND_INREG is basically truncating to ExtVT then
   // extended to VT.
   if (Opc == ISD::SIGN_EXTEND_INREG) {
     ExtType = ISD::SEXTLOAD;
     ExtVT = cast<VTSDNode>(N->getOperand(1))->getVT();
   } else if (Opc == ISD::SRL) {
     // Another special-case: SRL is basically zero-extending a narrower value,
     // or it maybe shifting a higher subword, half or byte into the lowest
     // bits.
     ExtType = ISD::ZEXTLOAD;
     N0 = SDValue(N, 0);
 
     auto *LN0 = dyn_cast<LoadSDNode>(N0.getOperand(0));
     auto *N01 = dyn_cast<ConstantSDNode>(N0.getOperand(1));
     if (!N01 || !LN0)
       return SDValue();
 
     uint64_t ShiftAmt = N01->getZExtValue();
     uint64_t MemoryWidth = LN0->getMemoryVT().getSizeInBits();
     if (LN0->getExtensionType() != ISD::SEXTLOAD && MemoryWidth > ShiftAmt)
       ExtVT = EVT::getIntegerVT(*DAG.getContext(), MemoryWidth - ShiftAmt);
     else
       ExtVT = EVT::getIntegerVT(*DAG.getContext(),
                                 VT.getSizeInBits() - ShiftAmt);
   } else if (Opc == ISD::AND) {
     // An AND with a constant mask is the same as a truncate + zero-extend.
     auto AndC = dyn_cast<ConstantSDNode>(N->getOperand(1));
-    if (!AndC || !AndC->getAPIntValue().isMask())
+    if (!AndC)
+      return SDValue();
+
+    const APInt &Mask = AndC->getAPIntValue();
+    unsigned ActiveBits = 0;
+    if (Mask.isMask()) {
+      ActiveBits = Mask.countTrailingOnes();
+    } else if (Mask.isShiftedMask()) {
+      ShAmt = Mask.countTrailingZeros();
+      APInt ShiftedMask = Mask.lshr(ShAmt);
+      ActiveBits = ShiftedMask.countTrailingOnes();
+      HasShiftedOffset = true;
+    } else
       return SDValue();
 
-    unsigned ActiveBits = AndC->getAPIntValue().countTrailingOnes();
     ExtType = ISD::ZEXTLOAD;
     ExtVT = EVT::getIntegerVT(*DAG.getContext(), ActiveBits);
   }
 
-  unsigned ShAmt = 0;
   if (N0.getOpcode() == ISD::SRL && N0.hasOneUse()) {
     SDValue SRL = N0;
     if (auto *ConstShift = dyn_cast<ConstantSDNode>(SRL.getOperand(1))) {
       ShAmt = ConstShift->getZExtValue();
       unsigned EVTBits = ExtVT.getSizeInBits();
       // Is the shift amount a multiple of size of VT?
       if ((ShAmt & (EVTBits-1)) == 0) {
         N0 = N0.getOperand(0);

     SDLoc DL(N0);
     if (ShLeftAmt >= VT.getSizeInBits())
       Result = DAG.getConstant(0, DL, VT);
     else
       Result = DAG.getNode(ISD::SHL, DL, VT,
                           Result, DAG.getConstant(ShLeftAmt, DL, ShImmTy));
   }
 
+  if (HasShiftedOffset) {
+    // We're using a shifted mask, so the load now has an offset. This means we
+    // now need to shift right the mask to match the new load and then shift
+    // right the result of the AND.
+    const APInt &Mask = cast<ConstantSDNode>(N->getOperand(1))->getAPIntValue();
+    APInt ShiftedMask = Mask.lshr(ShAmt);
+    DAG.UpdateNodeOperands(N, Result, DAG.getConstant(ShiftedMask, DL, VT));
+    SDValue ShiftC = DAG.getConstant(ShAmt, DL, VT);
+    SDValue Shifted = DAG.getNode(ISD::SHL, DL, VT, SDValue(N, 0),
+                                  ShiftC);
+    DAG.ReplaceAllUsesOfValueWith(SDValue(N, 0), Shifted);
+    DAG.UpdateNodeOperands(Shifted.getNode(), SDValue(N, 0), ShiftC);
+  }
   // Return the new loaded value.
   return Result;
 }
 
 SDValue DAGCombiner::visitSIGN_EXTEND_INREG(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   EVT VT = N->getValueType(0);

test/CodeGen/ARM/and-load-combine.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=armv7 %s -o - | FileCheck %s --check-prefix=ARM
 ; RUN: llc -mtriple=armv7eb %s -o - | FileCheck %s --check-prefix=ARMEB
 ; RUN: llc -mtriple=armv6m %s -o - | FileCheck %s --check-prefix=THUMB1
 ; RUN: llc -mtriple=thumbv8m.main %s -o - | FileCheck %s --check-prefix=THUMB2
 
-define arm_aapcscc zeroext i1 @cmp_xor8_short_short(i16* nocapture readonly %a,
-                                                    i16* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_xor8_short_short(i16* nocapture readonly %a, i16* nocapture readonly %b) {
 ; ARM-LABEL: cmp_xor8_short_short:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    eor r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

 ; THUMB2-LABEL: cmp_xor8_short_short:
 ; THUMB2:       @ %bb.0: @ %entry
 ; THUMB2-NEXT:    ldrb r0, [r0]
 ; THUMB2-NEXT:    ldrb r1, [r1]
 ; THUMB2-NEXT:    eors r0, r1
 ; THUMB2-NEXT:    clz r0, r0
 ; THUMB2-NEXT:    lsrs r0, r0, #5
 ; THUMB2-NEXT:    bx lr
 entry:
   %0 = load i16, i16* %a, align 2
   %1 = load i16, i16* %b, align 2
   %xor2 = xor i16 %1, %0
   %2 = and i16 %xor2, 255
   %cmp = icmp eq i16 %2, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_xor8_short_int(i16* nocapture readonly %a,
-                                                  i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_xor8_short_int(i16* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_xor8_short_int:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    eor r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %conv = zext i16 %0 to i32
   %1 = load i32, i32* %b, align 4
   %xor = xor i32 %1, %conv
   %and = and i32 %xor, 255
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_xor8_int_int(i32* nocapture readonly %a,
-                                                i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_xor8_int_int(i32* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_xor8_int_int:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    eor r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %xor = xor i32 %1, %0
   %and = and i32 %xor, 255
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_xor16(i32* nocapture readonly %a,
-                                         i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_xor16(i32* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_xor16:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrh r0, [r0]
 ; ARM-NEXT:    ldrh r1, [r1]
 ; ARM-NEXT:    eor r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %xor = xor i32 %1, %0
   %and = and i32 %xor, 65535
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_or8_short_short(i16* nocapture readonly %a,
-                                                   i16* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_or8_short_short(i16* nocapture readonly %a, i16* nocapture readonly %b) {
 ; ARM-LABEL: cmp_or8_short_short:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    orr r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %0 = load i16, i16* %a, align 2
   %1 = load i16, i16* %b, align 2
   %or2 = or i16 %1, %0
   %2 = and i16 %or2, 255
   %cmp = icmp eq i16 %2, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_or8_short_int(i16* nocapture readonly %a,
-                                                 i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_or8_short_int(i16* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_or8_short_int:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    orr r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %conv = zext i16 %0 to i32
   %1 = load i32, i32* %b, align 4
   %or = or i32 %1, %conv
   %and = and i32 %or, 255
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_or8_int_int(i32* nocapture readonly %a,
-                                               i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_or8_int_int(i32* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_or8_int_int:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    orr r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %or = or i32 %1, %0
   %and = and i32 %or, 255
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_or16(i32* nocapture readonly %a,
-                                        i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_or16(i32* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_or16:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrh r0, [r0]
 ; ARM-NEXT:    ldrh r1, [r1]
 ; ARM-NEXT:    orr r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %or = or i32 %1, %0
   %and = and i32 %or, 65535
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_and8_short_short(i16* nocapture readonly %a,
-                                                    i16* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_and8_short_short(i16* nocapture readonly %a, i16* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and8_short_short:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    and r0, r0, r1
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %0 = load i16, i16* %a, align 2
   %1 = load i16, i16* %b, align 2
   %and3 = and i16 %0, 255
   %2 = and i16 %and3, %1
   %cmp = icmp eq i16 %2, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_and8_short_int(i16* nocapture readonly %a,
-                                                  i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_and8_short_int(i16* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and8_short_int:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    and r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %1 = load i32, i32* %b, align 4
   %2 = and i16 %0, 255
   %and = zext i16 %2 to i32
   %and1 = and i32 %1, %and
   %cmp = icmp eq i32 %and1, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_and8_int_int(i32* nocapture readonly %a,
-                                                i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_and8_int_int(i32* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and8_int_int:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    ldrb r0, [r0]
 ; ARM-NEXT:    and r0, r0, r1
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %and = and i32 %0, 255
   %and1 = and i32 %and, %1
   %cmp = icmp eq i32 %and1, 0
   ret i1 %cmp
 }
 
-define arm_aapcscc zeroext i1 @cmp_and16(i32* nocapture readonly %a,
-                                         i32* nocapture readonly %b) {
+define arm_aapcscc zeroext i1 @cmp_and16(i32* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and16:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrh r1, [r1]
 ; ARM-NEXT:    ldrh r0, [r0]
 ; ARM-NEXT:    and r0, r0, r1
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr

 entry:
   %0 = load i32, i32* %p, align 4
   %neg = xor i32 %0, -1
   %and = and i32 %neg, 255
   store i32 %and, i32* %p, align 4
   ret void
 }
 
+define arm_aapcscc void @test10(i32* nocapture %p) {
 ; ARM-LABEL: test10:
 ; ARM:       @ %bb.0: @ %entry
 ; ARM-NEXT:    ldrb r1, [r0]
 ; ARM-NEXT:    eor r1, r1, #255
 ; ARM-NEXT:    str r1, [r0]
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: test10:

 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: test10:
 ; THUMB2:       @ %bb.0: @ %entry
 ; THUMB2-NEXT:    ldrb r1, [r0]
 ; THUMB2-NEXT:    eor r1, r1, #255
 ; THUMB2-NEXT:    str r1, [r0]
 ; THUMB2-NEXT:    bx lr
-define arm_aapcscc void @test10(i32* nocapture %p) {
 entry:
   %0 = load i32, i32* %p, align 4
   %neg = and i32 %0, 255
   %and = xor i32 %neg, 255
   store i32 %and, i32* %p, align 4
   ret void
 }
 
+define arm_aapcscc i32 @test11(i32* nocapture %p) {
+; ARM-LABEL: test11:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #1]
+; ARM-NEXT:    lsl r0, r0, #8
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test11:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test11:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #1]
+; THUMB1-NEXT:    lsls r0, r0, #8
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test11:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #1]
+; THUMB2-NEXT:    lsls r0, r0, #8
+; THUMB2-NEXT:    bx lr
+  %1 = load i32, i32* %p, align 4
+  %and = and i32 %1, 65280
+  ret i32 %and
+}
+
+define arm_aapcscc i32 @test12(i32* nocapture %p) {
+; ARM-LABEL: test12:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #2]
+; ARM-NEXT:    lsl r0, r0, #16
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test12:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test12:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #2]
+; THUMB1-NEXT:    lsls r0, r0, #16
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test12:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #2]
+; THUMB2-NEXT:    lsls r0, r0, #16
+; THUMB2-NEXT:    bx lr
+  %1 = load i32, i32* %p, align 4
+  %and = and i32 %1, 16711680
+  ret i32 %and
+}
+
+define arm_aapcscc i32 @test13(i32* nocapture %p) {
+; ARM-LABEL: test13:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #3]
+; ARM-NEXT:    lsl r0, r0, #24
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test13:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test13:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #3]
+; THUMB1-NEXT:    lsls r0, r0, #24
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test13:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #3]
+; THUMB2-NEXT:    lsls r0, r0, #24
+; THUMB2-NEXT:    bx lr
+  %1 = load i32, i32* %p, align 4
+  %and = and i32 %1, 4278190080
+  ret i32 %and
+}
+
+define arm_aapcscc i32 @test14(i32* nocapture %p) {
+; ARM-LABEL: test14:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrh r0, [r0, #1]
+; ARM-NEXT:    lsl r0, r0, #8
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test14:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    ldrh r0, [r0, #1]
+; ARMEB-NEXT:    lsl r0, r0, #8
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test14:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldr r1, [r0]
+; THUMB1-NEXT:    ldr r0, .LCPI26_0
+; THUMB1-NEXT:    ands r0, r1
+; THUMB1-NEXT:    bx lr
+; THUMB1-NEXT:    .p2align 2
+; THUMB1-NEXT:  @ %bb.1:
+; THUMB1-NEXT:  .LCPI26_0:
+; THUMB1-NEXT:    .long 16776960 @ 0xffff00
+;
+; THUMB2-LABEL: test14:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrh.w r0, [r0, #1]
+; THUMB2-NEXT:    lsls r0, r0, #8
+; THUMB2-NEXT:    bx lr
+  %1 = load i32, i32* %p, align 4
+  %and = and i32 %1, 16776960
+  ret i32 %and
+}
+
+define arm_aapcscc i32 @test15(i32* nocapture %p) {
+; ARM-LABEL: test15:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrh r0, [r0, #2]
+; ARM-NEXT:    lsl r0, r0, #16
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test15:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test15:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrh r0, [r0, #2]
+; THUMB1-NEXT:    lsls r0, r0, #16
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test15:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrh r0, [r0, #2]
+; THUMB2-NEXT:    lsls r0, r0, #16
+; THUMB2-NEXT:    bx lr
+  %1 = load i32, i32* %p, align 4
+  %and = and i32 %1, 4294901760
+  ret i32 %and
+}
+
+define arm_aapcscc i32 @test16(i64* nocapture %p) {
+; ARM-LABEL: test16:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #1]
+; ARM-NEXT:    lsl r0, r0, #8
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test16:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test16:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #1]
+; THUMB1-NEXT:    lsls r0, r0, #8
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test16:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #1]
+; THUMB2-NEXT:    lsls r0, r0, #8
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 65280
+  %trunc = trunc i64 %and to i32
+  ret i32 %trunc
+}
+
+define arm_aapcscc i32 @test17(i64* nocapture %p) {
+; ARM-LABEL: test17:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #2]
+; ARM-NEXT:    lsl r0, r0, #16
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test17:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test17:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #2]
+; THUMB1-NEXT:    lsls r0, r0, #16
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test17:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #2]
+; THUMB2-NEXT:    lsls r0, r0, #16
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 16711680
+  %trunc = trunc i64 %and to i32
+  ret i32 %trunc
+}
+
+define arm_aapcscc i32 @test18(i64* nocapture %p) {
+; ARM-LABEL: test18:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #3]
+; ARM-NEXT:    lsl r0, r0, #24
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test18:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test18:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #3]
+; THUMB1-NEXT:    lsls r0, r0, #24
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test18:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #3]
+; THUMB2-NEXT:    lsls r0, r0, #24
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 4278190080
+  %trunc = trunc i64 %and to i32
+  ret i32 %trunc
+}
+
+define arm_aapcscc i64 @test19(i64* nocapture %p) {
+; ARM-LABEL: test19:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r1, [r0, #4]
+; ARM-NEXT:    mov r0, #0
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test19:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    mov r1, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test19:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r1, [r0, #4]
+; THUMB1-NEXT:    movs r0, #0
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test19:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r1, [r0, #4]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 1095216660480
+  ret i64 %and
+}
+
+define arm_aapcscc i64 @test20(i64* nocapture %p) {
+; ARM-LABEL: test20:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #5]
+; ARM-NEXT:    lsl r1, r0, #8
+; ARM-NEXT:    mov r0, #0
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test20:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    mov r1, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test20:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #5]
+; THUMB1-NEXT:    lsls r1, r0, #8
+; THUMB1-NEXT:    movs r0, #0
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test20:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #5]
+; THUMB2-NEXT:    lsls r1, r0, #8
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 280375465082880
+  ret i64 %and
+}
+
+define arm_aapcscc i64 @test21(i64* nocapture %p) {
+; ARM-LABEL: test21:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #6]
+; ARM-NEXT:    lsl r1, r0, #16
+; ARM-NEXT:    mov r0, #0
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test21:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    mov r1, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test21:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #6]
+; THUMB1-NEXT:    lsls r1, r0, #16
+; THUMB1-NEXT:    movs r0, #0
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test21:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #6]
+; THUMB2-NEXT:    lsls r1, r0, #16
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 71776119061217280
+  ret i64 %and
+}
+
+define arm_aapcscc i64 @test22(i64* nocapture %p) {
+; ARM-LABEL: test22:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrb r0, [r0, #7]
+; ARM-NEXT:    lsl r1, r0, #24
+; ARM-NEXT:    mov r0, #0
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test22:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    mov r1, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test22:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r0, [r0, #7]
+; THUMB1-NEXT:    lsls r1, r0, #24
+; THUMB1-NEXT:    movs r0, #0
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test22:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrb r0, [r0, #7]
+; THUMB2-NEXT:    lsls r1, r0, #24
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, -72057594037927936
+  ret i64 %and
+}
+
+define arm_aapcscc i64 @test23(i64* nocapture %p) {
+; ARM-LABEL: test23:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrh r1, [r0, #3]
+; ARM-NEXT:    lsl r0, r1, #24
+; ARM-NEXT:    lsr r1, r1, #8
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test23:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    ldrh r1, [r0, #3]
+; ARMEB-NEXT:    lsr r0, r1, #8
+; ARMEB-NEXT:    lsl r1, r1, #24
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test23:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r1, [r0, #3]
+; THUMB1-NEXT:    ldrb r0, [r0, #4]
+; THUMB1-NEXT:    lsls r0, r0, #8
+; THUMB1-NEXT:    adds r1, r0, r1
+; THUMB1-NEXT:    lsls r0, r1, #24
+; THUMB1-NEXT:    lsrs r1, r1, #8
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test23:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrh.w r1, [r0, #3]
+; THUMB2-NEXT:    lsls r0, r1, #24
+; THUMB2-NEXT:    lsrs r1, r1, #8
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 1099494850560
+  ret i64 %and
+}
+
+define arm_aapcscc i64 @test24(i64* nocapture %p) {
+; ARM-LABEL: test24:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrh r1, [r0, #4]
+; ARM-NEXT:    mov r0, #0
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test24:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    mov r1, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test24:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrh r1, [r0, #4]
+; THUMB1-NEXT:    movs r0, #0
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test24:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrh r1, [r0, #4]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 281470681743360
+  ret i64 %and
+}
+
+define arm_aapcscc i64 @test25(i64* nocapture %p) {
+; ARM-LABEL: test25:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrh r0, [r0, #5]
+; ARM-NEXT:    lsl r1, r0, #8
+; ARM-NEXT:    mov r0, #0
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test25:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    mov r1, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test25:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrb r1, [r0, #5]
+; THUMB1-NEXT:    ldrb r0, [r0, #6]
+; THUMB1-NEXT:    lsls r0, r0, #8
+; THUMB1-NEXT:    adds r0, r0, r1
+; THUMB1-NEXT:    lsls r1, r0, #8
+; THUMB1-NEXT:    movs r0, #0
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test25:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrh.w r0, [r0, #5]
+; THUMB2-NEXT:    lsls r1, r0, #8
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, 72056494526300160
+  ret i64 %and
+}
+
+define arm_aapcscc i64 @test26(i64* nocapture %p) {
+; ARM-LABEL: test26:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    ldrh r0, [r0, #6]
+; ARM-NEXT:    lsl r1, r0, #16
+; ARM-NEXT:    mov r0, #0
+; ARM-NEXT:    bx lr
+;
+; ARMEB-LABEL: test26:
+; ARMEB:       @ %bb.0:
+; ARMEB-NEXT:    mov r0, #0
+; ARMEB-NEXT:    mov r1, #0
+; ARMEB-NEXT:    bx lr
+;
+; THUMB1-LABEL: test26:
+; THUMB1:       @ %bb.0:
+; THUMB1-NEXT:    ldrh r0, [r0, #6]
+; THUMB1-NEXT:    lsls r1, r0, #16
+; THUMB1-NEXT:    movs r0, #0
+; THUMB1-NEXT:    bx lr
+;
+; THUMB2-LABEL: test26:
+; THUMB2:       @ %bb.0:
+; THUMB2-NEXT:    ldrh r0, [r0, #6]
+; THUMB2-NEXT:    lsls r1, r0, #16
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    bx lr
+  %1 = load i64, i64* %p, align 8
+  %and = and i64 %1, -281474976710656
+  ret i64 %and
+}

test/CodeGen/X86/fp128-i128.ll

 ; void TestUnionLD1(__float128 s, unsigned long n) {
 ;      union IEEEl2bits u;
 ;      __float128 w;
 ;      u.e = s;
 ;      u.bits.manh = n;
 ;      w = u.e;
 ;      foo(w);
 ; }
 define void @TestUnionLD1(fp128 %s, i64 %n) #0 {
 ; SSE-LABEL: TestUnionLD1:
 ; SSE:       # %bb.0: # %entry
 ; SSE-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; SSE-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; SSE-NEXT:    movabsq $281474976710655, %rcx # imm = 0xFFFFFFFFFFFF
-; SSE-NEXT:    andq %rdi, %rcx
-; SSE-NEXT:    movabsq $-281474976710656, %rdx # imm = 0xFFFF000000000000
-; SSE-NEXT:    andq -{{[0-9]+}}(%rsp), %rdx
-; SSE-NEXT:    orq %rcx, %rdx
-; SSE-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; SSE-NEXT:    movzwl -{{[0-9]+}}(%rsp), %eax
+; SSE-NEXT:    shlq $48, %rax
+; SSE-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
+; SSE-NEXT:    movabsq $281474976710655, %rdx # imm = 0xFFFFFFFFFFFF
+; SSE-NEXT:    andq %rdi, %rdx
+; SSE-NEXT:    orq %rax, %rdx
+; SSE-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; SSE-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
 ; SSE-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm0
 ; SSE-NEXT:    jmp foo # TAILCALL
 ;
 ; AVX-LABEL: TestUnionLD1:
 ; AVX:       # %bb.0: # %entry
 ; AVX-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; AVX-NEXT:    movabsq $281474976710655, %rcx # imm = 0xFFFFFFFFFFFF
-; AVX-NEXT:    andq %rdi, %rcx
-; AVX-NEXT:    movabsq $-281474976710656, %rdx # imm = 0xFFFF000000000000
-; AVX-NEXT:    andq -{{[0-9]+}}(%rsp), %rdx
-; AVX-NEXT:    orq %rcx, %rdx
-; AVX-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; AVX-NEXT:    movzwl -{{[0-9]+}}(%rsp), %eax
+; AVX-NEXT:    shlq $48, %rax
+; AVX-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
+; AVX-NEXT:    movabsq $281474976710655, %rdx # imm = 0xFFFFFFFFFFFF
+; AVX-NEXT:    andq %rdi, %rdx
+; AVX-NEXT:    orq %rax, %rdx
+; AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; AVX-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
 ; AVX-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
 ; AVX-NEXT:    jmp foo # TAILCALL
 entry:
   %0 = bitcast fp128 %s to i128
   %1 = zext i64 %n to i128
   %bf.value = shl nuw i128 %1, 64
   %bf.shl = and i128 %bf.value, 5192296858534809181786422619668480