[DAGCombine] Move AND nodes to multiple load leaves

[samparker]

Summary

Search from AND nodes to find whether they can be propagated back to loads, so that the AND and load can be combined into a narrow load. We search through OR, XOR and other AND nodes and all the leaves are required to be loads or constants.

[samparker]

Ping.

[RKSimon]

Some minor style comments

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

505

Add comment descriptions for both of these

3741

for (unsigned i = 0, e = N->getNumOperands(); i < e; ++i) {

3742

You should be able to use this:

SDValue Op = N->getOperand(i);

and most "Op->" can be replaced with "Op."

[samparker]

Thanks Simon, I've made adjustments according to your comments.

[niravd]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3739

IIUC if you see a non-handled case (AND, OR, XOR, LOAD) you will give up because we cannot fully remove the AND.

What about partial success? Consider: AND (OR (LOAD x) (ADD y z)) MASK -> OR (ZEXTLOAD x) (AND (ADD y z) MASK). As long as we only generate 1 or fewer ANDs its a net win*. I also suspect that narrowing loads regardless would be a net-positive for SimplifyDemandedBits, though other reviews would know more about it than I do.

3758

if (isAndLoadExtLoad(Mask, Load, Load->getValueType(0), ExtVT, NarrowLoad) && NarrowLoad) {

3773

I believe you need a return here or something like (AND (AND (ADD x y) MASK1) (LOAD z) MASK2) will result in the ADD missing MASK2.

3809

The call to CombineTo after this should update the users so you should do an update. IIUC the only reason for this is to commute the operands in some cases. I suspect this isn't necessary or should be done by applying the check to both operands.

test/CodeGen/ARM/and-load-combine.ll

1

Please commit the test cases separately so we can see the differences more easily in phabricator.

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3739

Ok, thanks, I'll look into it.

3809

The issue here was that I need to replace one of Users operands with an AND that masks part of one of User's original operands. I originally tried to create the AND and then use ReplaceAllUsesWith on the AND, in the hope that it would update User. But then because the AND is using the value that I tried to replace, a cyclic dependency is introduced. So the only way I could see is by creating a completely new node. I then call ReplaceAllUsesWith so that I can delete the original User, thus allowing the load to be combined with the new AND node.

test/CodeGen/ARM/and-load-combine.ll

1

Sorry, I'm confused as to why that would make it easier?

[niravd]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3809

Ah! Take a look at UpdateNodeOperands. It should let you undo the construction of a cyclic dependency.

test/CodeGen/ARM/and-load-combine.ll

1

It's because the CHECK-NOT: lsl lines are not intuitively obvious as to why they are there (I'm still not sure why those were selected, but presumably it's important). IIUC all that matters is that originally the larger loads are being used. That's very easy to see with with a prepatch test case checking for the original code generated. I suppose additional CHECK-NOTs would work as well, but I'm not sure if that confuses the CHECK-NOT: lsl

[spatel]

test/CodeGen/ARM/and-load-combine.ll

1

Just a drive-by comment about codegen checks: there's a script to auto-generate the FileCheck lines at llvm/utils/update_llc_test_checks.py (and it has support for ARM triples, but not all of the ones in this test file - any help in getting more triples to work with the script is greatly appreciated!).

If you use that, then it becomes very simple to add tests with complete checking *before* you make a functional change that will change the asm. This makes it easier to review exactly what changes. It also avoids the inherent fragility of 'CHECK-NOT' (imagine that someone has an 'lsl' string in the path to this test file and FileCheck sees that path string).

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3809

Great, thanks.

test/CodeGen/ARM/and-load-combine.ll

1

Ok, thank you both. I'll make the changes and try the script, hopefully with a patch for it as well.

[samparker]

Now allowing a single node in the tree that will still require the top bits to be masked off. We're also now allowing extending nodes if the original type was small enough. The tests were committed before so hopefully the diff is easier to read.

[niravd]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3755

If you pull out the the checking and recording of NodeToMask to after the switch and return immediately on sucessful cases, you can out the default case and allows us to succeed with an "and of load" if one of the loads is not a valid extLoad.

3805

You should be able to replace most of this by using ReplaceAllUsesWith and then using UpdateNodeOperands to fix up the operands of the And Node. i.e.:

DAG.ReplaceAllUsesWith(N, And);
DAG.UpdateNodeOperands(And, SDValue(N,0), Mask));

It's probably worth inlining this function as well.

3834

IMO, this fast fail check isn't worth while as it's shaving off a smallish constant amount of wotk.

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3755

I'm not sure that works here, as I want to check all the operands and an early exit on the successful case could miss other valid nodes.

3805

But as 'And' uses 'N', won't this cause the cyclic dependency that I'm trying to avoid? Or is there a nice way of initialising a dummy 'And' node that doesn't use N?

[niravd]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3755

Sorry, I miswrote. You should "continue" to the next iteration of the loop on success.

3805

It will but it won't loop forever and you'll immediately fix it. See the implementation of SelectionDAG::makeEquivalentMemoryOrdering.

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3755

Ah, ok, I'll give that a go.

3805

Great, thanks.

[samparker]

I've removed the default case, which is now handled on exit from the switch. I've also removed the helper function to create the new mask and instead perform the masking inline.

[niravd]

Modulo the extra NodeToMask lines in zero extend case, this LGTM. Thanks!

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3779

Looks like you missed removing this case.

[samparker]

Oh yeah, I'll fix it before committing. Thanks for all your help.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

                                      SDValue &ConstNode);
 
     /// This is a helper function for visitAND and visitZERO_EXTEND.  Returns
     /// true if the (and (load x) c) pattern matches an extload.  ExtVT returns
     /// the type of the loaded value to be extended.
     bool isAndLoadExtLoad(ConstantSDNode *AndC, LoadSDNode *LoadN,
                           EVT LoadResultTy, EVT &ExtVT);
 
+    /// Used by BackwardsPropagateMask to find suitable loads.
+    bool SearchForAndLoads(SDNode *N, SmallPtrSetImpl<LoadSDNode*> &Loads,
+                           ConstantSDNode *Mask, SDNode *&UncombinedNode);
+    /// Attempt to propagate a given AND node back to load leaves so that they
+    /// can be combined into narrow loads.
+    bool BackwardsPropagateMask(SDNode *N, SelectionDAG &DAG);
+
     /// Helper function for MergeConsecutiveStores which merges the
     /// component store chains.
     SDValue getMergeStoreChains(SmallVectorImpl<MemOpLink> &StoreNodes,
                                 unsigned NumStores);
 
     /// This is a helper function for MergeConsecutiveStores. When the
     /// source elements of the consecutive stores are all constants or
     /// all extracted vector elements, try to merge them into one

     return false;
 
   if (!TLI.shouldReduceLoadWidth(LoadN, ISD::ZEXTLOAD, ExtVT))
     return false;
 
   return true;
 }
 
+bool DAGCombiner::SearchForAndLoads(SDNode *N,
+                                    SmallPtrSetImpl<LoadSDNode*> &Loads,
+                                    ConstantSDNode *Mask,
+                                    SDNode *&NodeToMask) {
+  // Recursively search for the operands, looking for loads which can be
+  // narrowed.
+  for (unsigned i = 0, e = N->getNumOperands(); i < e; ++i) {
+    SDValue Op = N->getOperand(i);
+
+    // Constants should already be fixed up...
+    if (isa<ConstantSDNode>(Op))
+      continue;
+
+    if (!Op.hasOneUse() || Op.getValueType().isVector())
+      return false;
+
+    switch(Op.getOpcode()) {
+    default:
+      // Allow one node which will masked along with any loads found.
+      if (NodeToMask)
+        return false;
+      NodeToMask = Op.getNode();
+      break;
+    case ISD::LOAD: {
+      auto *Load = cast<LoadSDNode>(Op);
+      EVT ExtVT;
+      if (isAndLoadExtLoad(Mask, Load, Load->getValueType(0), ExtVT)) {
+        Loads.insert(Load);
+        break;
+      }
+      return false;
+    }
+    case ISD::ZERO_EXTEND:
+    case ISD::ANY_EXTEND:
+    case ISD::AssertZext: {
+      unsigned ActiveBits = Mask->getAPIntValue().countTrailingOnes();
+      EVT ExtVT = EVT::getIntegerVT(*DAG.getContext(), ActiveBits);
+      EVT VT = Op.getOpcode() == ISD::AssertZext ?
+        cast<VTSDNode>(Op.getOperand(1))->getVT() :
+        Op.getOperand(0).getValueType();
+
+      // We can accept extending nodes if the mask is wider or an equal
+      // width to the original type.
+      if (ExtVT.bitsGE(VT))
+        break;
+
+      // If the type is too large, we'll still have to perfrom a masking
+      // operation later.
+      if (NodeToMask)
+        return false;
+
+      NodeToMask = Op.getNode();
+      break;
+    }
+    case ISD::OR:
+    case ISD::XOR:
+    case ISD::AND:
+      if (!SearchForAndLoads(Op.getNode(), Loads, Mask, NodeToMask))
+        return false;
+      break;
+    }
+  }
+  return true;
+}
+
+static SDValue InsertAndMask(SDNode *N, SDValue Mask, SelectionDAG &DAG) {
+  // Insert an and node to mask off the higher bits.
+  SDValue And = DAG.getNode(ISD::AND, SDLoc(N), N->getValueType(0),
+                            SDValue(N, 0), Mask);
+  SDValue N0 = SDValue(N, 0);
+  for (auto *User : N->uses()) {
+    // Skip the And that we just created.
+    if (User == And.getNode())
+      continue;
+
+    // Update the node to use the newly created mask.
+    SmallVector<SDValue, 4> Ops;
+    for (unsigned i = 0, e = User->getNumOperands(); i != e; ++i) {
+      SDValue Op = User->getOperand(i);
+      if (Op == N0)
+        Ops.push_back(And);
+      else
+        Ops.push_back(Op);
+    }
+    DAG.UpdateNodeOperands(User, Ops);
+  }
+  return And;
+}
+
+bool DAGCombiner::BackwardsPropagateMask(SDNode *N, SelectionDAG &DAG) {
+  auto *Mask = dyn_cast<ConstantSDNode>(N->getOperand(1));
+  if (!Mask)
+    return false;
+
+  if (!Mask->getAPIntValue().isMask())
+    return false;
+
+  // No need to do anything if the and directly uses a load.
+  if (isa<LoadSDNode>(N->getOperand(0)))
+    return false;
+
+  SmallPtrSet<LoadSDNode*, 8> Loads;
+  SDNode *UncombinedNode = nullptr;
+  if (SearchForAndLoads(N, Loads, Mask, UncombinedNode)) {
+    if (Loads.size() == 0)
+      return false;
+
+    if (UncombinedNode)
+      InsertAndMask(UncombinedNode, N->getOperand(1), DAG);
+
+    for (auto *Load : Loads) {
+      SDValue And = InsertAndMask(Load, N->getOperand(1), DAG);
+      SDValue NewLoad = ReduceLoadWidth(And.getNode());
+      assert(NewLoad &&
+             "Shouldn't be masking the load if it can't be narrowed");
+      CombineTo(Load, NewLoad, NewLoad.getValue(1));
+    }
+    DAG.ReplaceAllUsesWith(N, N->getOperand(0).getNode());
+    return true;
+  }
+  return false;
+}
+
 SDValue DAGCombiner::visitAND(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   EVT VT = N1.getValueType();
 
   // x & x --> x
   if (N0 == N1)
     return N0;

         ? cast<LoadSDNode>(N0.getOperand(0)) : cast<LoadSDNode>(N0);
 
       AddToWorklist(N);
       CombineTo(LN0, Res, Res.getValue(1));
       return SDValue(N, 0);
     }
   }
 
+  if (Level >= AfterLegalizeTypes) {
+    // Attempt to propagate the AND back up to the leaves which, if they're
+    // loads, can be combined to narrow loads and the AND node can be removed.
+    // Perform after legalization so that extend nodes will already be
+    // combined into the loads.
+    if (BackwardsPropagateMask(N, DAG)) {
+      return SDValue(N, 0);
+    }
+  }
+
   if (SDValue Combined = visitANDLike(N0, N1, N))
     return Combined;
 
   // Simplify: (and (op x...), (op y...))  -> (op (and x, y))
   if (N0.getOpcode() == N1.getOpcode())
     if (SDValue Tmp = SimplifyBinOpWithSameOpcodeHands(N))
       return Tmp;
 

test/CodeGen/ARM/and-load-combine.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=armv7 %s -o - | FileCheck %s --check-prefix=ARM
 ; RUN: llc -mtriple=armv7eb %s -o - | FileCheck %s --check-prefix=ARMEB
 ; RUN: llc -mtriple=armv6m %s -o - | FileCheck %s --check-prefix=THUMB1
 ; RUN: llc -mtriple=thumbv8m.main %s -o - | FileCheck %s --check-prefix=THUMB2
 
 define arm_aapcscc zeroext i1 @cmp_xor8_short_short(i16* nocapture readonly %a,
+                                                    i16* nocapture readonly %b) {
 ; ARM-LABEL: cmp_xor8_short_short:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldrh r0, [r0]
-; ARM-NEXT:    ldrh r1, [r1]
-; ARM-NEXT:    eor r1, r1, r0
+; ARM-NEXT:    ldrb r2, [r0]
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    teq r1, r2
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_xor8_short_short:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldrh r0, [r0]
-; ARMEB-NEXT:    ldrh r1, [r1]
-; ARMEB-NEXT:    eor r1, r1, r0
+; ARMEB-NEXT:    ldrb r2, [r0, #1]
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
+; ARMEB-NEXT:    ldrb r1, [r1, #1]
+; ARMEB-NEXT:    teq r1, r2
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_xor8_short_short:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldrh r0, [r0]
-; THUMB1-NEXT:    ldrh r2, [r1]
+; THUMB1-NEXT:    ldrb r0, [r0]
+; THUMB1-NEXT:    ldrb r2, [r1]
 ; THUMB1-NEXT:    eors r2, r0
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    cmp r2, #0
 ; THUMB1-NEXT:    beq .LBB0_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB0_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_xor8_short_short:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldrh r0, [r0]
-; THUMB2-NEXT:    ldrh r1, [r1]
-; THUMB2-NEXT:    eors r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
-; THUMB2-NEXT:    mov.w r0, #0
+; THUMB2-NEXT:    ldrb r2, [r0]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    ldrb r1, [r1]
+; THUMB2-NEXT:    teq.w r1, r2
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                                    i16* nocapture readonly %b) {
 entry:
   %0 = load i16, i16* %a, align 2
   %1 = load i16, i16* %b, align 2
   %xor2 = xor i16 %1, %0
   %2 = and i16 %xor2, 255
   %cmp = icmp eq i16 %2, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_xor8_short_int(i16* nocapture readonly %a,
+                                                  i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_xor8_short_int:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldrh r0, [r0]
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    eor r1, r1, r0
+; ARM-NEXT:    ldrb r2, [r0]
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    teq r1, r2
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_xor8_short_int:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldrh r0, [r0]
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    eor r1, r1, r0
+; ARMEB-NEXT:    ldrb r2, [r0, #1]
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
+; ARMEB-NEXT:    ldrb r1, [r1, #3]
+; ARMEB-NEXT:    teq r1, r2
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_xor8_short_int:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldrh r0, [r0]
-; THUMB1-NEXT:    ldr r2, [r1]
+; THUMB1-NEXT:    ldrb r0, [r0]
+; THUMB1-NEXT:    ldrb r2, [r1]
 ; THUMB1-NEXT:    eors r2, r0
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    cmp r2, #0
 ; THUMB1-NEXT:    beq .LBB1_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB1_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_xor8_short_int:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldrh r0, [r0]
-; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    eors r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
-; THUMB2-NEXT:    mov.w r0, #0
+; THUMB2-NEXT:    ldrb r2, [r0]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    ldrb r1, [r1]
+; THUMB2-NEXT:    teq.w r1, r2
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                                  i32* nocapture readonly %b) {
 entry:
   %0 = load i16, i16* %a, align 2
   %conv = zext i16 %0 to i32
   %1 = load i32, i32* %b, align 4
   %xor = xor i32 %1, %conv
   %and = and i32 %xor, 255
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_xor8_int_int(i32* nocapture readonly %a,
+                                                i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_xor8_int_int:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldr r0, [r0]
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    eor r1, r1, r0
+; ARM-NEXT:    ldrb r2, [r0]
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    teq r1, r2
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_xor8_int_int:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldr r0, [r0]
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    eor r1, r1, r0
+; ARMEB-NEXT:    ldrb r2, [r0, #3]
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
+; ARMEB-NEXT:    ldrb r1, [r1, #3]
+; ARMEB-NEXT:    teq r1, r2
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_xor8_int_int:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldr r0, [r0]
-; THUMB1-NEXT:    ldr r2, [r1]
+; THUMB1-NEXT:    ldrb r0, [r0]
+; THUMB1-NEXT:    ldrb r2, [r1]
 ; THUMB1-NEXT:    eors r2, r0
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    cmp r2, #0
 ; THUMB1-NEXT:    beq .LBB2_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB2_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_xor8_int_int:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldr r0, [r0]
-; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    eors r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
-; THUMB2-NEXT:    mov.w r0, #0
+; THUMB2-NEXT:    ldrb r2, [r0]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    ldrb r1, [r1]
+; THUMB2-NEXT:    teq.w r1, r2
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                                i32* nocapture readonly %b) {
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %xor = xor i32 %1, %0
   %and = and i32 %xor, 255
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_xor16(i32* nocapture readonly %a,
+                                         i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_xor16:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldr r0, [r0]
-; ARM-NEXT:    movw r2, #65535
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    eor r1, r1, r0
+; ARM-NEXT:    ldrh r2, [r0]
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, r2
+; ARM-NEXT:    ldrh r1, [r1]
+; ARM-NEXT:    teq r1, r2
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_xor16:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldr r0, [r0]
-; ARMEB-NEXT:    movw r2, #65535
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    eor r1, r1, r0
+; ARMEB-NEXT:    ldrh r2, [r0, #2]
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, r2
+; ARMEB-NEXT:    ldrh r1, [r1, #2]
+; ARMEB-NEXT:    teq r1, r2
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_xor16:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldr r0, [r0]
-; THUMB1-NEXT:    ldr r2, [r1]
+; THUMB1-NEXT:    ldrh r0, [r0]
+; THUMB1-NEXT:    ldrh r2, [r1]
 ; THUMB1-NEXT:    eors r2, r0
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #16
+; THUMB1-NEXT:    cmp r2, #0
 ; THUMB1-NEXT:    beq .LBB3_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB3_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_xor16:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldr r0, [r0]
-; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    eors r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #16
-; THUMB2-NEXT:    mov.w r0, #0
+; THUMB2-NEXT:    ldrh r2, [r0]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    ldrh r1, [r1]
+; THUMB2-NEXT:    teq.w r1, r2
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                         i32* nocapture readonly %b) {
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %xor = xor i32 %1, %0
   %and = and i32 %xor, 65535
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_or8_short_short(i16* nocapture readonly %a,
+                                                   i16* nocapture readonly %b) {
 ; ARM-LABEL: cmp_or8_short_short:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldrh r0, [r0]
-; ARM-NEXT:    ldrh r1, [r1]
-; ARM-NEXT:    orr r1, r1, r0
+; ARM-NEXT:    ldrb r0, [r0]
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    orrs r0, r1, r0
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_or8_short_short:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldrh r0, [r0]
-; ARMEB-NEXT:    ldrh r1, [r1]
-; ARMEB-NEXT:    orr r1, r1, r0
+; ARMEB-NEXT:    ldrb r0, [r0, #1]
+; ARMEB-NEXT:    ldrb r1, [r1, #1]
+; ARMEB-NEXT:    orrs r0, r1, r0
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_or8_short_short:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldrh r0, [r0]
-; THUMB1-NEXT:    ldrh r2, [r1]
+; THUMB1-NEXT:    ldrb r0, [r0]
+; THUMB1-NEXT:    ldrb r2, [r1]
 ; THUMB1-NEXT:    orrs r2, r0
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    cmp r2, #0
 ; THUMB1-NEXT:    beq .LBB4_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB4_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_or8_short_short:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldrh r0, [r0]
-; THUMB2-NEXT:    ldrh r1, [r1]
+; THUMB2-NEXT:    ldrb r0, [r0]
+; THUMB2-NEXT:    ldrb r1, [r1]
 ; THUMB2-NEXT:    orrs r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
 ; THUMB2-NEXT:    mov.w r0, #0
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                                   i16* nocapture readonly %b) {
 entry:
   %0 = load i16, i16* %a, align 2
   %1 = load i16, i16* %b, align 2
   %or2 = or i16 %1, %0
   %2 = and i16 %or2, 255
   %cmp = icmp eq i16 %2, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_or8_short_int(i16* nocapture readonly %a,
+                                                 i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_or8_short_int:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldrh r0, [r0]
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    orr r1, r1, r0
+; ARM-NEXT:    ldrb r0, [r0]
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    orrs r0, r1, r0
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_or8_short_int:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldrh r0, [r0]
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    orr r1, r1, r0
+; ARMEB-NEXT:    ldrb r0, [r0, #1]
+; ARMEB-NEXT:    ldrb r1, [r1, #3]
+; ARMEB-NEXT:    orrs r0, r1, r0
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_or8_short_int:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldrh r0, [r0]
-; THUMB1-NEXT:    ldr r2, [r1]
+; THUMB1-NEXT:    ldrb r0, [r0]
+; THUMB1-NEXT:    ldrb r2, [r1]
 ; THUMB1-NEXT:    orrs r2, r0
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    cmp r2, #0
 ; THUMB1-NEXT:    beq .LBB5_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB5_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_or8_short_int:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldrh r0, [r0]
-; THUMB2-NEXT:    ldr r1, [r1]
+; THUMB2-NEXT:    ldrb r0, [r0]
+; THUMB2-NEXT:    ldrb r1, [r1]
 ; THUMB2-NEXT:    orrs r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
 ; THUMB2-NEXT:    mov.w r0, #0
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                                 i32* nocapture readonly %b) {
 entry:
   %0 = load i16, i16* %a, align 2
   %conv = zext i16 %0 to i32
   %1 = load i32, i32* %b, align 4
   %or = or i32 %1, %conv
   %and = and i32 %or, 255
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_or8_int_int(i32* nocapture readonly %a,
+                                               i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_or8_int_int:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldr r0, [r0]
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    orr r1, r1, r0
+; ARM-NEXT:    ldrb r0, [r0]
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    orrs r0, r1, r0
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_or8_int_int:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldr r0, [r0]
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    orr r1, r1, r0
+; ARMEB-NEXT:    ldrb r0, [r0, #3]
+; ARMEB-NEXT:    ldrb r1, [r1, #3]
+; ARMEB-NEXT:    orrs r0, r1, r0
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_or8_int_int:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldr r0, [r0]
-; THUMB1-NEXT:    ldr r2, [r1]
+; THUMB1-NEXT:    ldrb r0, [r0]
+; THUMB1-NEXT:    ldrb r2, [r1]
 ; THUMB1-NEXT:    orrs r2, r0
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    cmp r2, #0
 ; THUMB1-NEXT:    beq .LBB6_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB6_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_or8_int_int:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldr r0, [r0]
-; THUMB2-NEXT:    ldr r1, [r1]
+; THUMB2-NEXT:    ldrb r0, [r0]
+; THUMB2-NEXT:    ldrb r1, [r1]
 ; THUMB2-NEXT:    orrs r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
 ; THUMB2-NEXT:    mov.w r0, #0
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                               i32* nocapture readonly %b) {
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %or = or i32 %1, %0
   %and = and i32 %or, 255
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_or16(i32* nocapture readonly %a,
+                                        i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_or16:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldr r0, [r0]
-; ARM-NEXT:    movw r2, #65535
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    orr r1, r1, r0
+; ARM-NEXT:    ldrh r0, [r0]
+; ARM-NEXT:    ldrh r1, [r1]
+; ARM-NEXT:    orrs r0, r1, r0
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, r2
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_or16:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldr r0, [r0]
-; ARMEB-NEXT:    movw r2, #65535
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    orr r1, r1, r0
+; ARMEB-NEXT:    ldrh r0, [r0, #2]
+; ARMEB-NEXT:    ldrh r1, [r1, #2]
+; ARMEB-NEXT:    orrs r0, r1, r0
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, r2
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_or16:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldr r0, [r0]
-; THUMB1-NEXT:    ldr r2, [r1]
+; THUMB1-NEXT:    ldrh r0, [r0]
+; THUMB1-NEXT:    ldrh r2, [r1]
 ; THUMB1-NEXT:    orrs r2, r0
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #16
+; THUMB1-NEXT:    cmp r2, #0
 ; THUMB1-NEXT:    beq .LBB7_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB7_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_or16:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldr r0, [r0]
-; THUMB2-NEXT:    ldr r1, [r1]
+; THUMB2-NEXT:    ldrh r0, [r0]
+; THUMB2-NEXT:    ldrh r1, [r1]
 ; THUMB2-NEXT:    orrs r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #16
 ; THUMB2-NEXT:    mov.w r0, #0
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                        i32* nocapture readonly %b) {
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %or = or i32 %1, %0
   %and = and i32 %or, 65535
   %cmp = icmp eq i32 %and, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_and8_short_short(i16* nocapture readonly %a,
+                                                    i16* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and8_short_short:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldrh r1, [r1]
-; ARM-NEXT:    ldrh r0, [r0]
-; ARM-NEXT:    and r1, r0, r1
+; ARM-NEXT:    ldrb r2, [r0]
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    tst r2, r1
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_and8_short_short:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldrh r1, [r1]
-; ARMEB-NEXT:    ldrh r0, [r0]
-; ARMEB-NEXT:    and r1, r0, r1
+; ARMEB-NEXT:    ldrb r2, [r0, #1]
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
+; ARMEB-NEXT:    ldrb r1, [r1, #1]
+; ARMEB-NEXT:    tst r2, r1
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_and8_short_short:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldrh r1, [r1]
-; THUMB1-NEXT:    ldrh r2, [r0]
-; THUMB1-NEXT:    ands r2, r1
+; THUMB1-NEXT:    ldrb r2, [r1]
+; THUMB1-NEXT:    ldrb r3, [r0]
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    tst r3, r2
 ; THUMB1-NEXT:    beq .LBB8_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB8_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_and8_short_short:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldrh r1, [r1]
-; THUMB2-NEXT:    ldrh r0, [r0]
-; THUMB2-NEXT:    ands r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
-; THUMB2-NEXT:    mov.w r0, #0
+; THUMB2-NEXT:    ldrb r2, [r0]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    ldrb r1, [r1]
+; THUMB2-NEXT:    tst r2, r1
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                                    i16* nocapture readonly %b) {
 entry:
   %0 = load i16, i16* %a, align 2
   %1 = load i16, i16* %b, align 2
   %and3 = and i16 %0, 255
   %2 = and i16 %and3, %1
   %cmp = icmp eq i16 %2, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_and8_short_int(i16* nocapture readonly %a,
+                                                  i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and8_short_int:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldrh r0, [r0]
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    and r1, r1, r0
+; ARM-NEXT:    ldrb r2, [r0]
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    tst r1, r2
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_and8_short_int:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldrh r0, [r0]
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    and r1, r1, r0
+; ARMEB-NEXT:    ldrb r2, [r0, #1]
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
+; ARMEB-NEXT:    ldrb r1, [r1, #3]
+; ARMEB-NEXT:    tst r1, r2
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_and8_short_int:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldrh r0, [r0]
-; THUMB1-NEXT:    ldr r2, [r1]
-; THUMB1-NEXT:    ands r2, r0
+; THUMB1-NEXT:    ldrb r2, [r0]
+; THUMB1-NEXT:    ldrb r3, [r1]
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    tst r3, r2
 ; THUMB1-NEXT:    beq .LBB9_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB9_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_and8_short_int:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldrh r0, [r0]
-; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    ands r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
-; THUMB2-NEXT:    mov.w r0, #0
+; THUMB2-NEXT:    ldrb r2, [r0]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    ldrb r1, [r1]
+; THUMB2-NEXT:    tst r1, r2
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                                  i32* nocapture readonly %b) {
 entry:
   %0 = load i16, i16* %a, align 2
   %1 = load i32, i32* %b, align 4
   %2 = and i16 %0, 255
   %and = zext i16 %2 to i32
   %and1 = and i32 %1, %and
   %cmp = icmp eq i32 %and1, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_and8_int_int(i32* nocapture readonly %a,
+                                                i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and8_int_int:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    ldr r0, [r0]
-; ARM-NEXT:    and r1, r0, r1
+; ARM-NEXT:    ldrb r2, [r0]
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, #255
+; ARM-NEXT:    ldrb r1, [r1]
+; ARM-NEXT:    tst r2, r1
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_and8_int_int:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    ldr r0, [r0]
-; ARMEB-NEXT:    and r1, r0, r1
+; ARMEB-NEXT:    ldrb r2, [r0, #3]
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, #255
+; ARMEB-NEXT:    ldrb r1, [r1, #3]
+; ARMEB-NEXT:    tst r2, r1
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_and8_int_int:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldr r1, [r1]
-; THUMB1-NEXT:    ldr r2, [r0]
-; THUMB1-NEXT:    ands r2, r1
+; THUMB1-NEXT:    ldrb r2, [r1]
+; THUMB1-NEXT:    ldrb r3, [r0]
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #24
+; THUMB1-NEXT:    tst r3, r2
 ; THUMB1-NEXT:    beq .LBB10_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB10_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_and8_int_int:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    ldr r0, [r0]
-; THUMB2-NEXT:    ands r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #24
-; THUMB2-NEXT:    mov.w r0, #0
+; THUMB2-NEXT:    ldrb r2, [r0]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    ldrb r1, [r1]
+; THUMB2-NEXT:    tst r2, r1
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                                i32* nocapture readonly %b) {
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %and = and i32 %0, 255
   %and1 = and i32 %and, %1
   %cmp = icmp eq i32 %and1, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_and16(i32* nocapture readonly %a,
+                                         i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and16:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    movw r2, #65535
-; ARM-NEXT:    ldr r0, [r0]
-; ARM-NEXT:    and r1, r0, r1
+; ARM-NEXT:    ldrh r2, [r0]
 ; ARM-NEXT:    mov r0, #0
-; ARM-NEXT:    tst r1, r2
+; ARM-NEXT:    ldrh r1, [r1]
+; ARM-NEXT:    tst r2, r1
 ; ARM-NEXT:    movweq r0, #1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_and16:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    movw r2, #65535
-; ARMEB-NEXT:    ldr r0, [r0]
-; ARMEB-NEXT:    and r1, r0, r1
+; ARMEB-NEXT:    ldrh r2, [r0, #2]
 ; ARMEB-NEXT:    mov r0, #0
-; ARMEB-NEXT:    tst r1, r2
+; ARMEB-NEXT:    ldrh r1, [r1, #2]
+; ARMEB-NEXT:    tst r2, r1
 ; ARMEB-NEXT:    movweq r0, #1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_and16:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldr r1, [r1]
-; THUMB1-NEXT:    ldr r2, [r0]
-; THUMB1-NEXT:    ands r2, r1
+; THUMB1-NEXT:    ldrh r2, [r1]
+; THUMB1-NEXT:    ldrh r3, [r0]
 ; THUMB1-NEXT:    movs r0, #1
 ; THUMB1-NEXT:    movs r1, #0
-; THUMB1-NEXT:    lsls r2, r2, #16
+; THUMB1-NEXT:    tst r3, r2
 ; THUMB1-NEXT:    beq .LBB11_2
 ; THUMB1-NEXT:  @ BB#1: @ %entry
 ; THUMB1-NEXT:    mov r0, r1
 ; THUMB1-NEXT:  .LBB11_2: @ %entry
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_and16:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    ldr r0, [r0]
-; THUMB2-NEXT:    ands r0, r1
-; THUMB2-NEXT:    lsls r0, r0, #16
-; THUMB2-NEXT:    mov.w r0, #0
+; THUMB2-NEXT:    ldrh r2, [r0]
+; THUMB2-NEXT:    movs r0, #0
+; THUMB2-NEXT:    ldrh r1, [r1]
+; THUMB2-NEXT:    tst r2, r1
 ; THUMB2-NEXT:    it eq
 ; THUMB2-NEXT:    moveq r0, #1
 ; THUMB2-NEXT:    bx lr
-                                         i32* nocapture readonly %b) {
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %and = and i32 %0, 65535
   %and1 = and i32 %and, %1
   %cmp = icmp eq i32 %and1, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc i32 @add_and16(i32* nocapture readonly %a, i32 %y, i32 %z) {
 ; ARM-LABEL: add_and16:
 ; ARM:       @ BB#0: @ %entry
-; ARM-NEXT:    ldr r0, [r0]
 ; ARM-NEXT:    add r1, r1, r2
+; ARM-NEXT:    ldrh r0, [r0]
+; ARM-NEXT:    uxth r1, r1
 ; ARM-NEXT:    orr r0, r0, r1
-; ARM-NEXT:    uxth r0, r0
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: add_and16:
 ; ARMEB:       @ BB#0: @ %entry
-; ARMEB-NEXT:    ldr r0, [r0]
 ; ARMEB-NEXT:    add r1, r1, r2
+; ARMEB-NEXT:    ldrh r0, [r0, #2]
+; ARMEB-NEXT:    uxth r1, r1
 ; ARMEB-NEXT:    orr r0, r0, r1
-; ARMEB-NEXT:    uxth r0, r0
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: add_and16:
 ; THUMB1:       @ BB#0: @ %entry
 ; THUMB1-NEXT:    adds r1, r1, r2
-; THUMB1-NEXT:    ldr r0, [r0]
+; THUMB1-NEXT:    uxth r1, r1
+; THUMB1-NEXT:    ldrh r0, [r0]
 ; THUMB1-NEXT:    orrs r0, r1
-; THUMB1-NEXT:    uxth r0, r0
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: add_and16:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    ldr r0, [r0]
 ; THUMB2-NEXT:    add r1, r2
+; THUMB2-NEXT:    ldrh r0, [r0]
+; THUMB2-NEXT:    uxth r1, r1
 ; THUMB2-NEXT:    orrs r0, r1
-; THUMB2-NEXT:    uxth r0, r0
 ; THUMB2-NEXT:    bx lr
 entry:
   %x = load i32, i32* %a, align 4
   %add = add i32 %y, %z
   %or = or i32 %x, %add
   %and = and i32 %or, 65535
   ret i32 %and
 }
 
 define arm_aapcscc i32 @test1(i32* %a, i32* %b, i32 %x, i32 %y) {
 ; ARM-LABEL: test1:
 ; ARM:       @ BB#0: @ %entry
 ; ARM-NEXT:    mul r2, r2, r3
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    ldr r0, [r0]
+; ARM-NEXT:    ldrh r1, [r1]
+; ARM-NEXT:    ldrh r0, [r0]
 ; ARM-NEXT:    eor r0, r0, r1
-; ARM-NEXT:    orr r0, r0, r2
-; ARM-NEXT:    uxth r0, r0
+; ARM-NEXT:    uxth r1, r2
+; ARM-NEXT:    orr r0, r0, r1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: test1:
 ; ARMEB:       @ BB#0: @ %entry
 ; ARMEB-NEXT:    mul r2, r2, r3
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    ldr r0, [r0]
+; ARMEB-NEXT:    ldrh r1, [r1, #2]
+; ARMEB-NEXT:    ldrh r0, [r0, #2]
 ; ARMEB-NEXT:    eor r0, r0, r1
-; ARMEB-NEXT:    orr r0, r0, r2
-; ARMEB-NEXT:    uxth r0, r0
+; ARMEB-NEXT:    uxth r1, r2
+; ARMEB-NEXT:    orr r0, r0, r1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: test1:
 ; THUMB1:       @ BB#0: @ %entry
+; THUMB1:         ldrh r1, [r1]
+; THUMB1-NEXT:    ldrh r4, [r0]
+; THUMB1-NEXT:    eors r4, r1
 ; THUMB1-NEXT:    muls r2, r3, r2
-; THUMB1-NEXT:    ldr r1, [r1]
-; THUMB1-NEXT:    ldr r0, [r0]
-; THUMB1-NEXT:    eors r0, r1
-; THUMB1-NEXT:    orrs r0, r2
-; THUMB1-NEXT:    uxth r0, r0
-; THUMB1-NEXT:    bx lr
+; THUMB1-NEXT:    uxth r0, r2
+; THUMB1-NEXT:    orrs r0, r4
+; THUMB1-NEXT:    pop
 ;
 ; THUMB2-LABEL: test1:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    muls r2, r3, r2
-; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    ldr r0, [r0]
+; THUMB2-NEXT:    ldrh r1, [r1]
+; THUMB2-NEXT:    ldrh r0, [r0]
 ; THUMB2-NEXT:    eors r0, r1
-; THUMB2-NEXT:    orrs r0, r2
-; THUMB2-NEXT:    uxth r0, r0
+; THUMB2-NEXT:    mul r1, r2, r3
+; THUMB2-NEXT:    uxth r1, r1
+; THUMB2-NEXT:    orrs r0, r1
 ; THUMB2-NEXT:    bx lr
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %mul = mul i32 %x, %y
   %xor = xor i32 %0, %1
   %or = or i32 %xor, %mul
   %and = and i32 %or, 65535
entry:
   %and = and i32 %or, 65535
   ret i32 %and
 }
 
 define arm_aapcscc i32 @test4(i32* %a, i32* %b, i32 %x, i32 %y) {
 ; ARM-LABEL: test4:
 ; ARM:       @ BB#0: @ %entry
 ; ARM-NEXT:    mul r2, r2, r3
-; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    ldr r0, [r0]
+; ARM-NEXT:    ldrh r1, [r1]
+; ARM-NEXT:    ldrh r0, [r0]
 ; ARM-NEXT:    eor r0, r0, r1
-; ARM-NEXT:    orr r0, r0, r2
-; ARM-NEXT:    uxth r0, r0
+; ARM-NEXT:    uxth r1, r2
+; ARM-NEXT:    orr r0, r0, r1
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: test4:
 ; ARMEB:       @ BB#0: @ %entry
 ; ARMEB-NEXT:    mul r2, r2, r3
-; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    ldr r0, [r0]
+; ARMEB-NEXT:    ldrh r1, [r1, #2]
+; ARMEB-NEXT:    ldrh r0, [r0, #2]
 ; ARMEB-NEXT:    eor r0, r0, r1
-; ARMEB-NEXT:    orr r0, r0, r2
-; ARMEB-NEXT:    uxth r0, r0
+; ARMEB-NEXT:    uxth r1, r2
+; ARMEB-NEXT:    orr r0, r0, r1
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: test4:
 ; THUMB1:       @ BB#0: @ %entry
+; THUMB1:         ldrh r1, [r1]
+; THUMB1-NEXT:    ldrh r4, [r0]
+; THUMB1-NEXT:    eors r4, r1
 ; THUMB1-NEXT:    muls r2, r3, r2
-; THUMB1-NEXT:    ldr r1, [r1]
-; THUMB1-NEXT:    ldr r0, [r0]
-; THUMB1-NEXT:    eors r0, r1
-; THUMB1-NEXT:    orrs r0, r2
-; THUMB1-NEXT:    uxth r0, r0
-; THUMB1-NEXT:    bx lr
+; THUMB1-NEXT:    uxth r0, r2
+; THUMB1-NEXT:    orrs r0, r4
+; THUMB1-NEXT:    pop
 ;
 ; THUMB2-LABEL: test4:
 ; THUMB2:       @ BB#0: @ %entry
-; THUMB2-NEXT:    muls r2, r3, r2
-; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    ldr r0, [r0]
+; THUMB2-NEXT:    ldrh r1, [r1]
+; THUMB2-NEXT:    ldrh r0, [r0]
 ; THUMB2-NEXT:    eors r0, r1
-; THUMB2-NEXT:    orrs r0, r2
-; THUMB2-NEXT:    uxth r0, r0
+; THUMB2-NEXT:    mul r1, r2, r3
+; THUMB2-NEXT:    uxth r1, r1
+; THUMB2-NEXT:    orrs r0, r1
 ; THUMB2-NEXT:    bx lr
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %mul = mul i32 %x, %y
   %xor = xor i32 %0, %1
   %or = or i32 %xor, %mul
   %and = and i32 %or, 65535
   ret i32 %and
 }
 
 define arm_aapcscc i32 @test5(i32* %a, i32* %b, i32 %x, i16 zeroext %y) {
 ; ARM-LABEL: test5:
 ; ARM:       @ BB#0: @ %entry
 ; ARM-NEXT:    ldr r1, [r1]
-; ARM-NEXT:    ldr r0, [r0]
+; ARM-NEXT:    ldrh r0, [r0]
 ; ARM-NEXT:    mul r1, r2, r1
 ; ARM-NEXT:    eor r0, r0, r3
+; ARM-NEXT:    uxth r1, r1
 ; ARM-NEXT:    orr r0, r0, r1
-; ARM-NEXT:    uxth r0, r0
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: test5:
 ; ARMEB:       @ BB#0: @ %entry
 ; ARMEB-NEXT:    ldr r1, [r1]
-; ARMEB-NEXT:    ldr r0, [r0]
+; ARMEB-NEXT:    ldrh r0, [r0, #2]
 ; ARMEB-NEXT:    mul r1, r2, r1
 ; ARMEB-NEXT:    eor r0, r0, r3
+; ARMEB-NEXT:    uxth r1, r1
 ; ARMEB-NEXT:    orr r0, r0, r1
-; ARMEB-NEXT:    uxth r0, r0
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: test5:
 ; THUMB1:       @ BB#0: @ %entry
-; THUMB1-NEXT:    ldr r1, [r1]
-; THUMB1-NEXT:    muls r1, r2, r1
-; THUMB1-NEXT:    ldr r0, [r0]
-; THUMB1-NEXT:    eors r0, r3
-; THUMB1-NEXT:    orrs r0, r1
+; THUMB1:         ldrh r4, [r0]
+; THUMB1-NEXT:    eors r4, r3
+; THUMB1-NEXT:    ldr r0, [r1]
+; THUMB1-NEXT:    muls r0, r2, r0
 ; THUMB1-NEXT:    uxth r0, r0
-; THUMB1-NEXT:    bx lr
+; THUMB1-NEXT:    orrs r0, r4
+; THUMB1-NEXT:    pop
 ;
 ; THUMB2-LABEL: test5:
 ; THUMB2:       @ BB#0: @ %entry
 ; THUMB2-NEXT:    ldr r1, [r1]
-; THUMB2-NEXT:    ldr r0, [r0]
+; THUMB2-NEXT:    ldrh r0, [r0]
 ; THUMB2-NEXT:    muls r1, r2, r1
 ; THUMB2-NEXT:    eors r0, r3
+; THUMB2-NEXT:    uxth r1, r1
 ; THUMB2-NEXT:    orrs r0, r1
-; THUMB2-NEXT:    uxth r0, r0
 ; THUMB2-NEXT:    bx lr
 entry:
   %0 = load i32, i32* %a, align 4
   %1 = load i32, i32* %b, align 4
   %mul = mul i32 %x, %1
   %ext = zext i16 %y to i32
   %xor = xor i32 %0, %ext
   %or = or i32 %xor, %mul
   %and = and i32 %or, 65535
   ret i32 %and
 }
+