[SelectionDAG]Reduce masked data movement chains and memory access widths

[dnsampaio]

Summary

Final goal:
detect that, in an i16 data type example:
i16 M[x] = (M[x] & 0xFF00) | (M[x] >> 8 & 0xFF)
can be reduced to an i8 data movement:
i8 M[x] = i8 M[x+1]
The existing code does not handle shifted mask, nor truncating stores and sext loads.

Step 1: Width reduction masked loads by shifted masks
the operation
and i16 (load i16, [M]), 0xFF00
can be replaced by
shl i16 (load i8, [M+1], zext i16), 8

[labrinea]

Please update your diff file in order to contain all the file content. Thanks.

[dnsampaio]

Update diff to contain all the file contents.

[samparker]

Hi Diogo,

This looks like two patches to me so please separate them. I would also expect to see more tests, this is quite a large change so try to cover some of the corner cases.

Also, one styling comment is that we use capital case for variables.

cheers,
sam

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

4016

N is an AND, right? This would be easier to read if just N is passed, which you can then add an assert on the opcode, and then you can make N0 and N1 local variables.

4028

unsigned is fine, otherwise try to avoid using C-style casting. It would also be more readable if it was called something like 'ShiftOpcode'.

4029

Don't rely on the ordering of these values. How about using the switch below to default to returning instead of an unreachable?

4035

Pull the cast out of the loop, the nullptr check is also unnecessary.

4054

Is SRA really legal here?

4064

Best to just have a TODO comment and remove these opcodes from the switch.

4081

You've already created a local var for 'shiftAmount', it would be easier to follow if you continued to use it.

4082

Don't think you need to add these as they're not new.

8661

AndC could be a nullptr here.

13001

This looks like two bits of new functionality, so please separate them into two patches.

test/CodeGen/ARM/stld-width-reduction1.ll

20 ↗

(On Diff #146310)

Please remove the attributes and metadata.

[dnsampaio]

Attended all review requests, including spitting into 2 patches.
Fixed the cases where the SRA can be accepted.
Added more unit-tests.

[rogfer01]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

451

Typo here in Redudant?

[dnsampaio]

Fixed typo.

[samparker]

Hi Diogo,

You seem to have forgot to re-add your tests now that you've split the patch.

[dnsampaio]

Re-inserted tests.

[samparker]

Hi Diogo,

Thanks for adding the tests again, but they're still not exercising your change properly - I don't see how your testing the SHL, ROTL or ROTR paths. Also by not checking the input and output registers, using regex or otherwise, you're not testing that the data flow is correct.
This patch should also be split again: into the shift folding and the load reduction, dag combining can be a pain so its best to break into down as much as possible.

cheers,
sam

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

4095

remove

6397

Is this needed for the rest of the patch to work? I would have expected that once the above transformation has been performed, that the transform you've added below would later be performed by the existing code in ReduceLoadWidth.

[dnsampaio]

Spitted the patch into 3 parts, and added further tests.

[samparker]

Hi Diogo,

This is looking good, I have just a couple of inline queries. Also, please add a couple of negative tests that use a mask that isn't a multiple of 8 bits.

cheers,
sam

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

8664–8685

remove or uncomment.

8674

OR instead of AND?

8678

Could you explain what you're checking here?

8714

Could you comment on why both of these are being set please?

test/CodeGen/ARM/2018-ShiftedAndMask.ll

54 ↗

(On Diff #151670)

Is it now possible to do a ldrh with an offset of 1 by using a mask of 0xff0?

[samparker]

test/CodeGen/ARM/2018-ShiftedAndMask.ll

54 ↗

(On Diff #151670)

i meant a mask of 0xffff00...

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

private:
     SDValue MatchBSwapHWord(SDNode *N, SDValue N0, SDValue N1);
     SDNode *MatchRotatePosNeg(SDValue Shifted, SDValue Pos, SDValue Neg,
                               SDValue InnerPos, SDValue InnerNeg,
                               unsigned PosOpcode, unsigned NegOpcode,
                               const SDLoc &DL);
     SDNode *MatchRotate(SDValue LHS, SDValue RHS, const SDLoc &DL);
     SDValue MatchLoadCombine(SDNode *N);
     SDValue ReduceLoadWidth(SDNode *N);
+    SDValue foldRedudantShiftedMasks(SDNode *N);
     SDValue ReduceLoadOpStoreWidth(SDNode *N);
     SDValue splitMergedValStore(StoreSDNode *ST);
     SDValue TransformFPLoadStorePair(SDNode *N);
     SDValue convertBuildVecZextToZext(SDNode *N);
     SDValue reduceBuildVecExtToExtBuildVec(SDNode *N);
     SDValue reduceBuildVecConvertToConvertBuildVec(SDNode *N);
     SDValue reduceBuildVecToShuffle(SDNode *N);
     SDValue createBuildVecShuffle(const SDLoc &DL, SDNode *N,
for (auto *Load : Loads) {
       CombineTo(Load, NewLoad, NewLoad.getValue(1));
     }
     DAG.ReplaceAllUsesWith(N, N->getOperand(0).getNode());
     return true;
   }
   return false;
 }
 
+// fold expressions x1 and x2 alike:
+// x1 = ( and, x, 0x00FF )
+// x2 = (( shl x, 8 ) and 0xFF00 )
+// into
+// x2 = shl x1, 8 ; reuse the computation of x1
+SDValue DAGCombiner::foldRedudantShiftedMasks(SDNode *AND) {
+  const SDValue &SHIFT = AND->getOperand(0);
+  if ((SHIFT.getNumOperands() != 2) || (!SHIFT.hasOneUse()))
+    return SDValue();
+
+  const ConstantSDNode *ShiftAmount =
+      dyn_cast<ConstantSDNode>(SHIFT.getOperand(1));
+  if (!ShiftAmount)
+    return SDValue();
+
+  const ConstantSDNode *Mask = dyn_cast<ConstantSDNode>(AND->getOperand(1));
+  if (!Mask)
+    return SDValue();
+
+  SDValue MASKED = SHIFT.getOperand(0);
+  const auto &MaskedValue = dyn_cast<SDNode>(MASKED);
+  unsigned N0Opcode = SHIFT.getOpcode();
+  for (SDNode *OtherUser : MaskedValue->uses()) {
+    if ((&(*OtherUser) == ShiftAmount) || (OtherUser->getOpcode() != ISD::AND))
+      continue;
+
+    ConstantSDNode *OtherMask =
+        dyn_cast<ConstantSDNode>(OtherUser->getOperand(1));
+
+    if (!OtherMask)
+      continue;
+
+    bool CanReduce = false;
+
+    const APInt &MaskValue = Mask->getAPIntValue();
+    const APInt &ShiftValue = ShiftAmount->getAPIntValue();
+    const APInt &OtherMaskValue = OtherMask->getAPIntValue();
+
+    KnownBits MaskedValueBits;
+    DAG.computeKnownBits(MASKED, MaskedValueBits);
+    KnownBits ShiftedValueBits;
+    DAG.computeKnownBits(SHIFT, ShiftedValueBits);
+
+    const APInt EffectiveOtherMask = OtherMaskValue & ~MaskedValueBits.Zero;
+    const APInt EffectiveMask = MaskValue & ~ShiftedValueBits.Zero;
+
+    LLVM_DEBUG(
+        dbgs() << "\tMasked value: "; MASKED.dump();
+        dbgs() << "\t\tMasked value zero bits: 0x"
+               << MaskedValueBits.Zero.toString(16, false)
+               << "\n\n\t\tApplied mask: 0x"
+               << OtherMaskValue.toString(16, false) << " : ";
+        OtherUser->dump();
+        dbgs() << "\t\tEffective mask: 0x"
+               << EffectiveOtherMask.toString(16, false)
+               << "\n\n\tShifted by: " << ShiftValue.getZExtValue() << " : ";
+        SHIFT.dump(); dbgs() << "\t\tAnd masked by: 0x"
+                             << MaskValue.toString(16, false) << " : ";
+        AND->dump(); dbgs() << "\t\tEffective mask to shifted value: 0x"
+                            << EffectiveMask.toString(16, false) << '\n';);
+
+    switch (N0Opcode) {
+    case ISD::SHL:
+      CanReduce = (EffectiveOtherMask.shl(EffectiveMask) == EffectiveMask) ||
+                  (EffectiveMask.lshr(ShiftValue) == EffectiveOtherMask);
+      break;
+    case ISD::SRA:
+      if (!MaskedValueBits.Zero.isSignBitSet()) {
+        CanReduce = (EffectiveOtherMask.ashr(ShiftValue) == EffectiveMask);
+        break;
+      } else // Same as SRL
+        N0Opcode = ISD::SRL;
+      /* fall-thru */
+    case ISD::SRL:
+      CanReduce = (EffectiveOtherMask.lshr(ShiftValue) == EffectiveMask) ||
+                  (EffectiveMask.shl(ShiftValue) == EffectiveOtherMask);
+      break;
+    case ISD::ROTL:
+      CanReduce = (EffectiveOtherMask.rotl(ShiftValue) == EffectiveMask);
+      break;
+    case ISD::ROTR:
+      CanReduce = (EffectiveOtherMask.rotr(ShiftValue) == EffectiveMask);
+      break;
+      // TODO:
+      //    case ISD::SHL_PARTS:
+      //    case ISD::SRA_PARTS:
+      //    case ISD::SRL_PARTS:
+      break;
+    default:
+      return SDValue();
+    }
+    if (CanReduce) {
+      LLVM_DEBUG(dbgs() << "\tCan replace it\n");
+
+      SDValue ShiftTheAND(OtherUser, 0);
+      const SDLoc DL(SHIFT);
+      EVT VT = AND->getValueType(0);
+      SDValue NewShift =
+          DAG.getNode(N0Opcode, DL, VT, ShiftTheAND, SHIFT.getOperand(1));
+      AddToWorklist(OtherUser);
+      return NewShift;
+    }
+  }
+  return SDValue();
+}
+
 SDValue DAGCombiner::visitAND(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   EVT VT = N1.getValueType();
 
   // x & x --> x
   if (N0 == N1)
     return N0;
SDValue DAGCombiner::visitAND(SDNode *N) {
 
   // fold (and (load x), 255) -> (zextload x, i8)
   // fold (and (extload x, i16), 255) -> (zextload x, i8)
   // fold (and (any_ext (extload x, i16)), 255) -> (zextload x, i8)
   if (!VT.isVector() && N1C && (N0.getOpcode() == ISD::LOAD ||
                                 (N0.getOpcode() == ISD::ANY_EXTEND &&
                                  N0.getOperand(0).getOpcode() == ISD::LOAD))) {
     if (SDValue Res = ReduceLoadWidth(N)) {
+      if (Res.getOpcode() == ISD::SHL)
+        return Res;
+
       LoadSDNode *LN0 = N0->getOpcode() == ISD::ANY_EXTEND
         ? cast<LoadSDNode>(N0.getOperand(0)) : cast<LoadSDNode>(N0);
 
       AddToWorklist(N);
       CombineTo(LN0, Res, Res.getValue(1));
       return SDValue(N, 0);
     }
   }
 
+  if (SDValue r = foldRedudantShiftedMasks(N))
+    return r;
+
   if (Level >= AfterLegalizeTypes) {
     // Attempt to propagate the AND back up to the leaves which, if they're
     // loads, can be combined to narrow loads and the AND node can be removed.
     // Perform after legalization so that extend nodes will already be
     // combined into the loads.
     if (BackwardsPropagateMask(N, DAG)) {
       return SDValue(N, 0);
     }
if (auto N001C = isConstOrConstSplat(N0.getOperand(0).getOperand(1))) {
                                        N0.getOperand(0).getOperand(0),
                                        DAG.getConstant(c1 + c2, DL,
                                                        ShiftCountVT)));
       }
     }
   }
 
   // fold (srl (shl x, c), c) -> (and x, cst2)
-  if (N0.getOpcode() == ISD::SHL && N0.getOperand(1) == N1 &&
-      isConstantOrConstantVector(N1, /* NoOpaques */ true)) {
+  if ((N0.getOpcode() == ISD::SHL) &&
+      (isConstantOrConstantVector(N1, /* NoOpaques */ true))) {
+    bool CanFold = N0.getOperand(1) == N1;
+    if (!CanFold) {
+      const ConstantSDNode *CN0N1 = dyn_cast<ConstantSDNode>(N0.getOperand(1));
+      if (CN0N1 && N1C)
+        CanFold = CN0N1->getZExtValue() == N1C->getZExtValue();
+    }
+
+    if (CanFold) {
+      // fold (srl (shl X, c), c) -> (c) if the X upper bits of c are known to
+      // be 0
+      // TODO: Add more instructions that produce known upper bits zero masks,
+      // other than zext loads
+      if (N1C) {
+        if (LoadSDNode *X = dyn_cast<LoadSDNode>(N0.getOperand(0))) {
+          const unsigned XSize = X->getValueSizeInBits(0);
+          const unsigned XMemSize = X->getMemOperand()->getSize() * 8;
+          if ((XSize > XMemSize) &&
+              ((XSize - XMemSize) >= N1C->getZExtValue()) &&
+              (X->getExtensionType() == ISD::LoadExtType::ZEXTLOAD))
+            return N0.getOperand(0);
+        }
+      }
       SDLoc DL(N);
       SDValue Mask =
           DAG.getNode(ISD::SRL, DL, VT, DAG.getAllOnesConstant(DL, VT), N1);
       AddToWorklist(Mask.getNode());
       return DAG.getNode(ISD::AND, DL, VT, N0.getOperand(0), Mask);
     }
+  }
 
   // fold (srl (anyextend x), c) -> (and (anyextend (srl x, c)), mask)
   if (N1C && N0.getOpcode() == ISD::ANY_EXTEND) {
     // Shifting in all undef bits?
     EVT SmallVT = N0.getOperand(0).getValueType();
     unsigned BitSize = SmallVT.getScalarSizeInBits();
     if (N1C->getZExtValue() >= BitSize)
       return DAG.getUNDEF(VT);
SDValue DAGCombiner::ReduceLoadWidth(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   EVT VT = N->getValueType(0);
   EVT ExtVT = VT;
 
   // This transformation isn't valid for vector loads.
   if (VT.isVector())
     return SDValue();
 
+  unsigned ShAmt = 0;
+  unsigned ShLeftAmt = 0;
+
   // Special case: SIGN_EXTEND_INREG is basically truncating to ExtVT then
   // extended to VT.
   if (Opc == ISD::SIGN_EXTEND_INREG) {
     ExtType = ISD::SEXTLOAD;
     ExtVT = cast<VTSDNode>(N->getOperand(1))->getVT();
   } else if (Opc == ISD::SRL) {
     // Another special-case: SRL is basically zero-extending a narrower value,
     // or it maybe shifting a higher subword, half or byte into the lowest
if (Opc == ISD::SIGN_EXTEND_INREG) {
     if (LN0->getExtensionType() != ISD::SEXTLOAD && MemoryWidth > ShiftAmt)
       ExtVT = EVT::getIntegerVT(*DAG.getContext(), MemoryWidth - ShiftAmt);
     else
       ExtVT = EVT::getIntegerVT(*DAG.getContext(),
                                 VT.getSizeInBits() - ShiftAmt);
   } else if (Opc == ISD::AND) {
     // An AND with a constant mask is the same as a truncate + zero-extend.
     auto AndC = dyn_cast<ConstantSDNode>(N->getOperand(1));
-    if (!AndC || !AndC->getAPIntValue().isMask())
+    if (!AndC)
       return SDValue();
 
-    unsigned ActiveBits = AndC->getAPIntValue().countTrailingOnes();
+    // TODO: Not only [shifted] masks should be accepted.
+    //(and ld.16 [M], 0x00AB) can be replaced by (and ld.8.zext16 [M], 0x00AB).
+    const APInt &MaskAPInt = AndC->getAPIntValue();
+    if (!(MaskAPInt.isMask() || MaskAPInt.isShiftedMask()))
+      return SDValue();
+
+    unsigned MaxBit = MaskAPInt.getBitWidth() - MaskAPInt.countLeadingZeros();
+    const unsigned MinBit = MaskAPInt.countTrailingZeros();
+    // Only accepts multiples of 8 bits, and power of 2 sizes
+    if (!MaxBit && (0 != (MaxBit | MinBit) % 8))
+      return SDValue();
+
+    unsigned ActiveBits = MaxBit - MinBit;
+    if (ActiveBits & (ActiveBits - 1))
+      return SDValue();
+
+    LLVM_DEBUG(dbgs() << "\tMask: 0x" << MaskAPInt.toString(16, false) << " : ";
+               AndC->dump();
+               dbgs() << "\t\tmaxActiveBit: " << MaxBit - 1
+                      << "\n\t\tminActiveBit: " << MinBit << '\n');
+
+    LoadSDNode *LN0 = dyn_cast<LoadSDNode>(N0);
     ExtType = ISD::ZEXTLOAD;
+    if (MinBit != 0) {
+      // How to treat if it was not a load?
+      if (LN0 == nullptr)
+        return SDValue();
+
+      const auto &mvt = LN0->getMemoryVT();
+      if (MinBit >= mvt.getSizeInBits()) {
+        // The (and) is filtering what was extended, not the actual data
+        // value...
+        if (ISD::LoadExtType::ZEXTLOAD == LN0->getExtensionType()) {
+          // We only read the zero values
+          return DAG.getConstant(0, SDLoc(N), AndC->getValueType(0));
+        }
+        // We access the sign extension, not known here
+        return SDValue();
+      }
+      if (MaxBit > mvt.getSizeInBits())
+        ExtType = LN0->getExtensionType();
+    }
+    // TODO: Accept SEXT if the architecture accepts doing a LD?SH (load + shl)
+    // An (and (ld.32bit.sext.from16 [M]), 0x00FFFF00) can be replaced by
+    //   (and (shl (ld.32bit.sext.from8 [M+1]), 8), 0x00FFFF00)
+    if (ExtType != ISD::ZEXTLOAD)
+      return SDValue();
+
     ExtVT = EVT::getIntegerVT(*DAG.getContext(), ActiveBits);
+    ShAmt = MinBit;
+    ShLeftAmt = MinBit;
+    LLVM_DEBUG(dbgs() << "\tCan replace load: "; LN0->dump();
+               dbgs() << "\tBy a load of width " << ActiveBits
+                      << " and with offset of " << ShAmt / 8 << '\n');
   }
 
-  unsigned ShAmt = 0;
   if (N0.getOpcode() == ISD::SRL && N0.hasOneUse()) {
     SDValue SRL = N0;
     if (auto *ConstShift = dyn_cast<ConstantSDNode>(SRL.getOperand(1))) {
       ShAmt = ConstShift->getZExtValue();
       unsigned EVTBits = ExtVT.getSizeInBits();
       // Is the shift amount a multiple of size of VT?
       if ((ShAmt & (EVTBits-1)) == 0) {
         N0 = N0.getOperand(0);
if (auto *ConstShift = dyn_cast<ConstantSDNode>(SRL.getOperand(1))) {
             ExtVT = MaskedVT;
         }
       }
     }
   }
 
   // If the load is shifted left (and the result isn't shifted back right),
   // we can fold the truncate through the shift.
-  unsigned ShLeftAmt = 0;
   if (ShAmt == 0 && N0.getOpcode() == ISD::SHL && N0.hasOneUse() &&
       ExtVT == VT && TLI.isNarrowingProfitable(N0.getValueType(), VT)) {
     if (ConstantSDNode *N01 = dyn_cast<ConstantSDNode>(N0.getOperand(1))) {
       ShLeftAmt = N01->getZExtValue();
       N0 = N0.getOperand(0);
     }
   }
 
ShrinkLoadReplaceStoreWithStore(const std::pair<unsigned, unsigned> &MaskInfo,
   IVal = DAG.getNode(ISD::TRUNCATE, SDLoc(IVal), VT, IVal);
 
   ++OpsNarrowed;
   return DAG
       .getStore(St->getChain(), SDLoc(St), IVal, Ptr,
                 St->getPointerInfo().getWithOffset(StOffset), NewAlign)
       .getNode();
 }
 
 /// Look for sequence of load / op / store where op is one of 'or', 'xor', and
 /// 'and' of immediates. If 'op' is only touching some of the loaded bits, try
 /// narrowing the load and store if it would end up being a win for performance
 /// or code size.
 SDValue DAGCombiner::ReduceLoadOpStoreWidth(SDNode *N) {
   StoreSDNode *ST  = cast<StoreSDNode>(N);
   if (ST->isVolatile())
     return SDValue();

test/CodeGen/ARM/2018_05_29_FoldRedundantMask.ll

+; RUN: llc -O3 -march=arm %s -o - | FileCheck %s
+
+define i16 @LSR8(i16* %a){
+entry:
+  %data16 = getelementptr inbounds i16, i16* %a, i64 0
+  %0 = load i16, i16* %data16, align 2
+  %and = and i16 %0, 65280
+  %1 = lshr i16 %0, 8
+  %and3 = and i16 %1, 255
+  %or = or i16 %and3, %and
+  ret i16 %or
+}
+
+; CHECK-LABEL: LSR8
+; CHECK: ldrb{{.*}}
+; CHECK-NEXT: orr{{.*}}lsl #8
+; CHECK-NEXT: mov{{.*}}
+
+define i16 @LSR12A(i16* %a){
+entry:
+  %data16 = getelementptr inbounds i16, i16* %a, i64 0
+  %0 = load i16, i16* %data16, align 2
+  %and = and i16 %0, 61440
+  %1 = lshr i16 %0, 12
+  %and3 = and i16 %1, 15
+  %or = or i16 %and3, %and
+  ret i16 %or
+}
+; CHECK-LABEL: LSR12A
+; CHECK: ldr{{.*}}
+; CHECK-NEXT: and{{.*}}#61440
+; CHECK-NEXT: orr{{.*}}lsr #12
+; CHECK-NEXT: mov{{.*}}
+
+define i16 @LSR12B(i16* %a){
+entry:
+  %data16 = getelementptr inbounds i16, i16* %a, i64 0
+  %0 = load i16, i16* %data16, align 2
+  %and = and i16 %0, 61440
+  %1 = lshr i16 %0, 12
+  %and3 = and i16 %1, 255
+  %or = or i16 %and3, %and
+  ret i16 %or
+}
+; CHECK-LABEL: LSR12B
+; CHECK: ldrh{{.*}}
+; CHECK-NEXT: and{{.*}}#61440
+; CHECK-NEXT: orr{{.*}}lsr #12
+; CHECK-NEXT: mov{{.*}}
+
+define i16 @LSR12C(i16* %a){
+entry:
+  %data16 = getelementptr inbounds i16, i16* %a, i64 0
+  %0 = load i16, i16* %data16, align 2
+  %and = and i16 %0, 65280
+  %1 = lshr i16 %0, 12
+  %and3 = and i16 %1, 15
+  %or = or i16 %and3, %and
+  ret i16 %or
+}
+; CHECK-LABEL: LSR12C
+; CHECK: ldrh{{.*}}
+; CHECK-NEXT: and{{.*}}
+; CHECK-NEXT: orr{{.*}}lsr #12
+; CHECK-NEXT: mov{{.*}}
+
+define i32 @ASR(i16* %a){
+entry:
+  %data16 = getelementptr inbounds i16, i16* %a, i64 0
+  %l = load i16, i16* %data16, align 2
+  %0 = zext i16 %l to i32
+  %and = and i32 %0, 64512
+  %1 = ashr i32 %0, 8
+  %and3 = and i32 %1, 65532
+  %or = or i32 %and3, %and
+  ret i32 %or
+}
+; CHECK-LABEL: ASR
+; CHECK: ldrh{{.*}}
+; CHECK-NEXT: and{{.*}} #64512
+; CHECK-NEXT: orr{{.*}}lsr #8
+; CHECK-NEXT: mov pc, lr
+; ModuleID = 'test2.cpp'
+
+define i32 @main(i32 %ar, i8** nocapture readonly %v){
+entry:
+  %0 = bitcast i8** %v to i16*
+  %idxprom = sext i32 %ar to i64
+  %arrayidx = getelementptr inbounds i16, i16* %0, i64 %idxprom
+  %1 = load i16, i16* %arrayidx, align 2
+  %conv = sext i16 %1 to i32
+  %and = and i32 %conv, 65280
+  %2 = lshr i32 %conv, 8
+  %and4 = and i32 %2, 255
+  %or = or i32 %and4, %and
+  ret i32 %or
+}
+; CHECK-LABEL: main
+; CHECK: add     r0, r1, r0, lsl #1
+; CHECK-NEXT: ldrb    r0, [r0, #1]
+; CHECK-NEXT: orr     r0, r0, r0, lsl #8
+; CHECK-NEXT: mov     pc, lr

test/CodeGen/X86/fp128-i128.ll

 ;      __float128 w;
 ;      u.e = s;
 ;      u.bits.manh = n;
 ;      w = u.e;
 ;      foo(w);
 ; }
 define void @TestUnionLD1(fp128 %s, i64 %n) #0 {
 ; CHECK-LABEL: TestUnionLD1:
-; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; CHECK-NEXT:    movabsq $281474976710655, %rcx # imm = 0xFFFFFFFFFFFF
-; CHECK-NEXT:    andq %rdi, %rcx
-; CHECK-NEXT:    movabsq $-281474976710656, %rdx # imm = 0xFFFF000000000000
-; CHECK-NEXT:    andq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-NEXT:    orq %rcx, %rdx
-; CHECK-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm0
-; CHECK-NEXT:    jmp foo # TAILCALL
+; CHECK:       # %bb.0:                                # %entry
+; CHECK-NEXT:   movaps  %xmm0, -24(%rsp)
+; CHECK-NEXT:   movzwl  -10(%rsp), %eax
+; CHECK-NEXT:   shlq  $48, %rax
+; CHECK-NEXT:   movq  -24(%rsp), %rcx
+; CHECK-NEXT:   movabsq $281474976710655, %rdx  # imm = 0xFFFFFFFFFFFF
+; CHECK-NEXT:   andq  %rdi, %rdx
+; CHECK-NEXT:   orq %rax, %rdx
+; CHECK-NEXT:   movq  %rcx, -40(%rsp)
+; CHECK-NEXT:   movq  %rdx, -32(%rsp)
+; CHECK-NEXT:   movaps  -40(%rsp), %xmm0
+; CHECK-NEXT:   jmp foo                     # TAILCALL
 entry:
   %0 = bitcast fp128 %s to i128
   %1 = zext i64 %n to i128
   %bf.value = shl nuw i128 %1, 64
   %bf.shl = and i128 %bf.value, 5192296858534809181786422619668480
   %bf.clear = and i128 %0, -5192296858534809181786422619668481
   %bf.set = or i128 %bf.shl, %bf.clear
   %2 = bitcast i128 %bf.set to fp128

test/CodeGen/X86/pr32329.ll

 @var_24 = external local_unnamed_addr constant i64, align 8
 @var_310 = external local_unnamed_addr global i64, align 8
 @var_50 = external local_unnamed_addr global i64, align 8
 @var_205 = external local_unnamed_addr global i8, align 1
 @var_218 = external local_unnamed_addr global i8, align 1
 
 define void @foo() local_unnamed_addr {
 ; X86-LABEL: foo:
-; X86:       # %bb.0: # %entry
-; X86-NEXT:    pushl %ebp
-; X86-NEXT:    .cfi_def_cfa_offset 8
-; X86-NEXT:    pushl %ebx
-; X86-NEXT:    .cfi_def_cfa_offset 12
-; X86-NEXT:    pushl %edi
-; X86-NEXT:    .cfi_def_cfa_offset 16
-; X86-NEXT:    pushl %esi
-; X86-NEXT:    .cfi_def_cfa_offset 20
-; X86-NEXT:    .cfi_offset %esi, -20
-; X86-NEXT:    .cfi_offset %edi, -16
-; X86-NEXT:    .cfi_offset %ebx, -12
-; X86-NEXT:    .cfi_offset %ebp, -8
-; X86-NEXT:    movl obj, %edx
-; X86-NEXT:    movsbl var_27, %eax
-; X86-NEXT:    movzwl var_2, %esi
-; X86-NEXT:    movl var_310, %ecx
-; X86-NEXT:    imull %eax, %ecx
-; X86-NEXT:    addl var_24, %ecx
-; X86-NEXT:    andl $4194303, %edx # imm = 0x3FFFFF
-; X86-NEXT:    leal (%edx,%edx), %ebx
-; X86-NEXT:    subl %eax, %ebx
-; X86-NEXT:    movl %ebx, %edi
-; X86-NEXT:    subl %esi, %edi
-; X86-NEXT:    imull %edi, %ecx
-; X86-NEXT:    addl $-1437483407, %ecx # imm = 0xAA51BE71
-; X86-NEXT:    movl $9, %esi
-; X86-NEXT:    xorl %ebp, %ebp
-; X86-NEXT:    shldl %cl, %esi, %ebp
-; X86-NEXT:    shll %cl, %esi
-; X86-NEXT:    testb $32, %cl
-; X86-NEXT:    cmovnel %esi, %ebp
-; X86-NEXT:    movl $0, %ecx
-; X86-NEXT:    cmovnel %ecx, %esi
-; X86-NEXT:    cmpl %edx, %edi
-; X86-NEXT:    movl %ebp, var_50+4
-; X86-NEXT:    movl %esi, var_50
-; X86-NEXT:    setge var_205
-; X86-NEXT:    imull %eax, %ebx
-; X86-NEXT:    movb %bl, var_218
-; X86-NEXT:    popl %esi
-; X86-NEXT:    .cfi_def_cfa_offset 16
-; X86-NEXT:    popl %edi
-; X86-NEXT:    .cfi_def_cfa_offset 12
-; X86-NEXT:    popl %ebx
-; X86-NEXT:    .cfi_def_cfa_offset 8
-; X86-NEXT:    popl %ebp
-; X86-NEXT:    .cfi_def_cfa_offset 4
-; X86-NEXT:    retl
+; X86:       # %bb.0:                                # %entry
+; X86-NEXT:   pushl %ebp
+; X86-NEXT:   .cfi_def_cfa_offset 8
+; X86-NEXT:   pushl %ebx
+; X86-NEXT:   .cfi_def_cfa_offset 12
+; X86-NEXT:   pushl %edi
+; X86-NEXT:   .cfi_def_cfa_offset 16
+; X86-NEXT:   pushl %esi
+; X86-NEXT:   .cfi_def_cfa_offset 20
+; X86-NEXT:   .cfi_offset %esi, -20
+; X86-NEXT:   .cfi_offset %edi, -16
+; X86-NEXT:   .cfi_offset %ebx, -12
+; X86-NEXT:   .cfi_offset %ebp, -8
+; X86-NEXT:   movsbl  var_27, %eax
+; X86-NEXT:   movzwl  var_2, %esi
+; X86-NEXT:   movl  var_310, %ecx
+; X86-NEXT:   imull %eax, %ecx
+; X86-NEXT:   addl  var_24, %ecx
+; X86-NEXT:   movl  $4194303, %edi          # imm = 0x3FFFFF
+; X86-NEXT:   andl  obj, %edi
+; X86-NEXT:   leal  (%edi,%edi), %edx
+; X86-NEXT:   subl  %eax, %edx
+; X86-NEXT:   movl  %edx, %ebx
+; X86-NEXT:   subl  %esi, %ebx
+; X86-NEXT:   imull %ebx, %ecx
+; X86-NEXT:   addl  $-1437483407, %ecx      # imm = 0xAA51BE71
+; X86-NEXT:   movl  $9, %esi
+; X86-NEXT:   xorl  %ebp, %ebp
+; X86-NEXT:   shldl %cl, %esi, %ebp
+; X86-NEXT:   shll  %cl, %esi
+; X86-NEXT:   testb $32, %cl
+; X86-NEXT:   cmovnel %esi, %ebp
+; X86-NEXT:   movl  $0, %ecx
+; X86-NEXT:   cmovnel %ecx, %esi
+; X86-NEXT:   cmpl  %edi, %ebx
+; X86-NEXT:   movl  %ebp, var_50+4
+; X86-NEXT:   movl  %esi, var_50
+; X86-NEXT:   setge var_205
+; X86-NEXT:   imull %eax, %edx
+; X86-NEXT:   movb  %dl, var_218
+; X86-NEXT:   popl  %esi
+; X86-NEXT:   .cfi_def_cfa_offset 16
+; X86-NEXT:   popl  %edi
+; X86-NEXT:   .cfi_def_cfa_offset 12
+; X86-NEXT:   popl  %ebx
+; X86-NEXT:   .cfi_def_cfa_offset 8
+; X86-NEXT:   popl  %ebp
+; X86-NEXT:   .cfi_def_cfa_offset 4
+; X86-NEXT:   retl
 ;
 ; X64-LABEL: foo:
-; X64:       # %bb.0: # %entry
-; X64-NEXT:    movl {{.*}}(%rip), %eax
-; X64-NEXT:    movsbl {{.*}}(%rip), %r9d
-; X64-NEXT:    movzwl {{.*}}(%rip), %r8d
-; X64-NEXT:    movl {{.*}}(%rip), %ecx
-; X64-NEXT:    imull %r9d, %ecx
-; X64-NEXT:    addl {{.*}}(%rip), %ecx
-; X64-NEXT:    andl $4194303, %eax # imm = 0x3FFFFF
-; X64-NEXT:    leal (%rax,%rax), %edi
-; X64-NEXT:    subl %r9d, %edi
-; X64-NEXT:    movl %edi, %esi
-; X64-NEXT:    subl %r8d, %esi
-; X64-NEXT:    imull %esi, %ecx
-; X64-NEXT:    addl $-1437483407, %ecx # imm = 0xAA51BE71
-; X64-NEXT:    movl $9, %edx
-; X64-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NEXT:    shlq %cl, %rdx
-; X64-NEXT:    movq %rdx, {{.*}}(%rip)
-; X64-NEXT:    cmpl %eax, %esi
-; X64-NEXT:    setge {{.*}}(%rip)
-; X64-NEXT:    imull %r9d, %edi
-; X64-NEXT:    movb %dil, {{.*}}(%rip)
-; X64-NEXT:    retq
+; X64:       # %bb.0:                                # %entry
+; X64-NEXT:   movsbl  var_27(%rip), %r9d
+; X64-NEXT:   movzwl  var_2(%rip), %r8d
+; X64-NEXT:   movl  var_310(%rip), %ecx
+; X64-NEXT:   imull %r9d, %ecx
+; X64-NEXT:   addl  var_24(%rip), %ecx
+; X64-NEXT:   movl  $4194303, %esi          # imm = 0x3FFFFF
+; X64-NEXT:   andl  obj(%rip), %esi
+; X64-NEXT:   leal  (%rsi,%rsi), %edi
+; X64-NEXT:   subl  %r9d, %edi
+; X64-NEXT:   movl  %edi, %edx
+; X64-NEXT:   subl  %r8d, %edx
+; X64-NEXT:   imull %edx, %ecx
+; X64-NEXT:   addl  $-1437483407, %ecx      # imm = 0xAA51BE71
+; X64-NEXT:   movl  $9, %eax
+; X64-NEXT:                                          # kill: def $cl killed $cl killed $ecx
+; X64-NEXT:   shlq  %cl, %rax
+; X64-NEXT:   movq  %rax, var_50(%rip)
+; X64-NEXT:   cmpl  %esi, %edx
+; X64-NEXT:   setge var_205(%rip)
+; X64-NEXT:   imull %r9d, %edi
+; X64-NEXT:   movb  %dil, var_218(%rip)
+; X64-NEXT:   retq
   entry:
   %bf.load = load i32, i32* bitcast (%struct.AA* @obj to i32*), align 8
   %bf.clear = shl i32 %bf.load, 1
   %add = and i32 %bf.clear, 8388606
   %0 = load i8, i8* @var_27, align 1
   %conv5 = sext i8 %0 to i32
   %sub = sub nsw i32 %add, %conv5
   %1 = load i16, i16* @var_2, align 2

test/CodeGen/X86/pr32588.ll

 ; RUN: llc < %s -mcpu=generic -mtriple=x86_64-linux | FileCheck %s
 
 @c = external local_unnamed_addr global i32, align 4
 @b = external local_unnamed_addr global i32, align 4
 @d = external local_unnamed_addr global i32, align 4
 
-; CHECK: cmpl    $1, c(%rip)
-; CHECK-NEXT: sbbl    %eax, %eax
-; CHECK-NEXT: andl    $1, %eax
-; CHECK-NEXT: movl    %eax, d(%rip)
-; CHECK-NEXT: retq
+; CHECK:      xorl  %eax, %eax
+; CHECK-NEXT: cmpl  $0, c(%rip)
+; CHECK-NEXT: sete  %al
+; CHECK-NEXT: movl  %eax, d(%rip)
+; CHECK-NEXT: retq
 
 define void @fn1() {
 entry:
   %0 = load i32, i32* @c, align 4
   %tobool1 = icmp eq i32 %0, 0
   %xor = zext i1 %tobool1 to i32
   %1 = load i32, i32* @b, align 4
   %tobool2 = icmp ne i32 %1, 0
   %tobool4 = icmp ne i32 undef, 0
   %2 = and i1 %tobool4, %tobool2
   %sub = sext i1 %2 to i32
   %div = sdiv i32 %sub, 2
   %add = add nsw i32 %div, %xor
   store i32 %add, i32* @d, align 4
   ret void
 }