[ARM] bottom-top mul support in ARMParallelDSP

[samparker]

Summary

On failing to find sequences that can be converted into dual macs, try to find sequential 16-bit loads that are used by muls which we can then use smultb, smulbt, smultt with a wide load.

[samparker]

Added a negative test for loading i8s as well as a positive test for 64-bit macs.

[SjoerdMeijer]

lib/Target/ARM/ARMParallelDSP.cpp

662 ↗

(On Diff #165086)

I think the alias checks are the same as in MatchSMLAD, so it looks like we can share that, and avoid the duplication here. I could definitely live with that being done in a follow up patch.

test/CodeGen/ARM/paralleldsp-top-bottom.ll

28 ↗

(On Diff #165086)

These tests are not ideal, in the sense we are not directly checking for the smultb, smulbt, smultt patterns here. Instead, we rely on these ashr + mul patterns that we now generate, to be picked up and turned in to smlab variants later, like it is tested in test/CodeGen/ARM/smul.ll. And thus, since breaking these shr + mul rewrite patterns would give problems in smul.ll, I think these tests in their current form are acceptable as I understood that turning these tests in to llc tests would complicate things a bit.

[samparker]

lib/Target/ARM/ARMParallelDSP.cpp

662 ↗

(On Diff #165086)

Will do.

[samparker]

Commit was reverted in rL342260.

[samparker]

The patch caused an assert on some vectorized code because I missed a check that the muls are plain integer types. parallel-dsp-top-bottom-neg-vec.ll has been added which was the reproducer provided.

[SjoerdMeijer]

lib/Target/ARM/ARMParallelDSP.cpp

650 ↗

(On Diff #166256)

Diff'ing this was a bit difficult, but these are the 2 new lines, which matches with your last comment. Looks like a straightforward fix to me.

[samparker]

Committed in rL342870.

[samparker]

This patch was reverted again in rL343082.

[samparker]

Two breaking assumptions were that:

• the base load would be before the offset load.
• each load would only have one user - this is true but I also really meant and assumed that the sext of the load has one user.

So, I've rewritten the sext user replacement code as the previous version was horrible. I've also added some code to reorder the load and address calculations. The way I've figured out the ordering looks bad, but I couldn't find a simple way in the APIs to find this kind of info out.

I've also created a ParallelDSP subdirectory for the tests and moved the smlad stuff into it as well.

[SjoerdMeijer]

lib/Target/ARM/ARMParallelDSP.cpp

719 ↗

(On Diff #167745)

Nit: this loop is doing the same things as the previous loop, can we make and call a function?

730 ↗

(On Diff #167745)

Nit: looks like this is workaround for not having comparison operators on these iterators. Perhaps we can put this in little helper function too, sweep it under the carpet a little bit, but with a TODO/FIXME and some comments added to it. If a helper function is too much (this function MatchTopBottomMuls is becoming rather big), than at lease some comments here.

[samparker]

Cheers Sjoerd, I've added two helper functions to try to clean this up a bit.

[SjoerdMeijer]

cheers, LGTM

llvm/trunk/lib/Target/ARM/ARMParallelDSP.cpp

namespace {
   struct OpChain;
   struct BinOpChain;
   struct Reduction;
 
   using OpChainList     = SmallVector<std::unique_ptr<OpChain>, 8>;
   using ReductionList   = SmallVector<Reduction, 8>;
   using ValueList       = SmallVector<Value*, 8>;
   using MemInstList     = SmallVector<Instruction*, 8>;
+  using LoadInstList    = SmallVector<LoadInst*, 8>;
   using PMACPair        = std::pair<BinOpChain*,BinOpChain*>;
   using PMACPairList    = SmallVector<PMACPair, 8>;
   using Instructions    = SmallVector<Instruction*,16>;
   using MemLocList      = SmallVector<MemoryLocation, 4>;
 
   struct OpChain {
     Instruction   *Root;
     ValueList     AllValues;
-    MemInstList   VecLd;    // List of all load instructions.
+    MemInstList   VecLd;    // List of all sequential load instructions.
+    LoadInstList  Loads;    // List of all load instructions.
     MemLocList    MemLocs;  // All memory locations read by this tree.
     bool          ReadOnly = true;
 
     OpChain(Instruction *I, ValueList &vl) : Root(I), AllValues(vl) { }
     virtual ~OpChain() = default;
 
     void SetMemoryLocations() {
       const auto Size = LocationSize::unknown();
       for (auto *V : AllValues) {
         if (auto *I = dyn_cast<Instruction>(V)) {
           if (I->mayWriteToMemory())
             ReadOnly = false;
-          if (auto *Ld = dyn_cast<LoadInst>(V))
+          if (auto *Ld = dyn_cast<LoadInst>(V)) {
             MemLocs.push_back(MemoryLocation(Ld->getPointerOperand(), Size));
+            Loads.push_back(Ld);
+          }
         }
       }
     }
 
     unsigned size() const { return AllValues.size(); }
   };
 
   // 'BinOpChain' and 'Reduction' are just some bookkeeping data structures.
Instruction *CreateSMLADCall(LoadInst *VecLd0, LoadInst *VecLd1,
                                  Instruction *InsertAfter);
 
     /// Try to match and generate: SMLAD, SMLADX - Signed Multiply Accumulate
     /// Dual performs two signed 16x16-bit multiplications. It adds the
     /// products to a 32-bit accumulate operand. Optionally, the instruction can
     /// exchange the halfwords of the second operand before performing the
     /// arithmetic.
     bool MatchSMLAD(Function &F);
+    bool MatchTopBottomMuls(BasicBlock *LoopBody);
 
   public:
     static char ID;
 
     ARMParallelDSP() : LoopPass(ID) { }
 
     void getAnalysisUsage(AnalysisUsage &AU) const override {
       LoopPass::getAnalysisUsage(AU);
bool runOnLoop(Loop *TheLoop, LPPassManager &) override {
       }
 
       LoopAccessInfo LAI(L, SE, TLI, AA, DT, LI);
       bool Changes = false;
 
       LLVM_DEBUG(dbgs() << "\n== Parallel DSP pass ==\n");
       LLVM_DEBUG(dbgs() << " - " << F.getName() << "\n\n");
       Changes = MatchSMLAD(F);
+      if (!Changes)
+        Changes = MatchTopBottomMuls(Header);
       return Changes;
     }
   };
 }
 
 // MaxBitwidth: the maximum supported bitwidth of the elements in the DSP
 // instructions, which is set to 16. So here we should collect all i8 and i16
 // narrow operations.
LLVM_DEBUG(
     dbgs() << "\nAccumulating integer additions (reductions) found:\n";
     for (auto &R : Reductions) {
       dbgs() << "-  "; R.Phi->dump();
       dbgs() << "-> "; R.AccIntAdd->dump();
     }
   );
 }
 
-static void AddMACCandidate(OpChainList &Candidates,
+static void AddMulCandidate(OpChainList &Candidates,
                             Instruction *Mul,
                             Value *MulOp0, Value *MulOp1) {
-  LLVM_DEBUG(dbgs() << "OK, found acc mul:\t"; Mul->dump());
+  LLVM_DEBUG(dbgs() << "OK, found mul:\t"; Mul->dump());
   assert(Mul->getOpcode() == Instruction::Mul &&
          "expected mul instruction");
   ValueList LHS;
   ValueList RHS;
   if (IsNarrowSequence<16>(MulOp0, LHS) &&
       IsNarrowSequence<16>(MulOp1, RHS)) {
     LLVM_DEBUG(dbgs() << "OK, found narrow mul: "; Mul->dump());
     Candidates.push_back(make_unique<BinOpChain>(Mul, LHS, RHS));
std::function<bool(Value*)> Match =
 
     switch (I->getOpcode()) {
     case Instruction::Add:
       if (Match(I->getOperand(0)) || (Match(I->getOperand(1))))
         return true;
       break;
     case Instruction::Mul:
       if (match (I, (m_Mul(m_Value(MulOp0), m_Value(MulOp1))))) {
-        AddMACCandidate(Candidates, I, MulOp0, MulOp1);
+        AddMulCandidate(Candidates, I, MulOp0, MulOp1);
         return false;
       }
       break;
     case Instruction::SExt:
       if (match (I, (m_SExt(m_Mul(m_Value(MulOp0), m_Value(MulOp1)))))) {
         Instruction *Mul = cast<Instruction>(I->getOperand(0));
-        AddMACCandidate(Candidates, Mul, MulOp0, MulOp1);
+        AddMulCandidate(Candidates, Mul, MulOp0, MulOp1);
         return false;
       }
       break;
     }
     return false;
   };
 
   while (Match (Acc));
if (I.mayWriteToMemory())
       Writes.push_back(&I);
   }
 }
 
 // Check whether statements in the basic block that write to memory alias with
 // the memory locations accessed by the MAC-chains.
 // TODO: we need the read statements when we accept more complicated chains.
 static bool AreAliased(AliasAnalysis *AA, Instructions &Reads,
-                       Instructions &Writes, OpChainList &MACCandidates) {
+                       Instructions &Writes, OpChainList &Candidates) {
   LLVM_DEBUG(dbgs() << "Alias checks:\n");
-  for (auto &MAC : MACCandidates) {
-    LLVM_DEBUG(dbgs() << "mul: "; MAC->Root->dump());
+  for (auto &Candidate : Candidates) {
+    LLVM_DEBUG(dbgs() << "mul: "; Candidate->Root->dump());
+    Candidate->SetMemoryLocations();
 
     // At the moment, we allow only simple chains that only consist of reads,
     // accumulate their result with an integer add, and thus that don't write
     // memory, and simply bail if they do.
-    if (!MAC->ReadOnly)
+    if (!Candidate->ReadOnly)
       return true;
 
     // Now for all writes in the basic block, check that they don't alias with
     // the memory locations accessed by our MAC-chain:
     for (auto *I : Writes) {
       LLVM_DEBUG(dbgs() << "- "; I->dump());
-      assert(MAC->MemLocs.size() >= 2 && "expecting at least 2 memlocs");
-      for (auto &MemLoc : MAC->MemLocs) {
+      assert(Candidate->MemLocs.size() >= 2 && "expecting at least 2 memlocs");
+      for (auto &MemLoc : Candidate->MemLocs) {
         if (isModOrRefSet(intersectModRef(AA->getModRefInfo(I, MemLoc),
                                           ModRefInfo::ModRef))) {
           LLVM_DEBUG(dbgs() << "Yes, aliases found\n");
           return true;
         }
       }
     }
   }
 
   LLVM_DEBUG(dbgs() << "OK: no aliases found!\n");
   return false;
 }
 
-static bool CheckMACMemory(OpChainList &Candidates) {
+static bool CheckMulMemory(OpChainList &Candidates) {
   for (auto &C : Candidates) {
     // A mul has 2 operands, and a narrow op consist of sext and a load; thus
     // we expect at least 4 items in this operand value list.
     if (C->size() < 4) {
       LLVM_DEBUG(dbgs() << "Operand list too short.\n");
       return false;
     }
-    C->SetMemoryLocations();
     ValueList &LHS = static_cast<BinOpChain*>(C.get())->LHS;
     ValueList &RHS = static_cast<BinOpChain*>(C.get())->RHS;
 
     // Use +=2 to skip over the expected extend instructions.
     for (unsigned i = 0, e = LHS.size(); i < e; i += 2) {
       if (!isa<LoadInst>(LHS[i]) || !isa<LoadInst>(RHS[i]))
         return false;
     }
   }
   return true;
 }
 
+static LoadInst *CreateLoadIns(IRBuilder<NoFolder> &IRB, LoadInst *BaseLoad,
+                               const Type *LoadTy) {
+  const unsigned AddrSpace = BaseLoad->getPointerAddressSpace();
+ 
+  Value *VecPtr = IRB.CreateBitCast(BaseLoad->getPointerOperand(),
+                                     LoadTy->getPointerTo(AddrSpace));
+  return IRB.CreateAlignedLoad(VecPtr, BaseLoad->getAlignment());
+}
+
+/// Given two instructions, return the one that comes first in the basic block.
+/// A work around for not being able to do > or < on bb iterators.
+static Instruction* GetFirst(Instruction *A, Instruction *B) {
+  BasicBlock::iterator First(A);
+  BasicBlock::iterator Second(B);
+
+  BasicBlock *BB = A->getParent();
+  assert(BB == B->getParent() &&
+         "Can't compare instructions in different blocks");
+  BasicBlock::iterator Last = BB->end();
+
+  // Iterate through the block, if the 'First' iterator is found, then return
+  // Second.
+  while (Second != Last) {
+    if (Second == First)
+      return B;
+    ++Second;
+  }
+  return A;
+}
+
+/// Attempt to widen loads and use smulbb, smulbt, smultb and smultt muls.
+// TODO: This, like smlad generation, expects the leave operands to be loads
+// that are sign extended. We should be able to handle scalar values as well
+// performing these muls on word x half types to generate smulwb and smulwt.
+bool ARMParallelDSP::MatchTopBottomMuls(BasicBlock *LoopBody) {
+  LLVM_DEBUG(dbgs() << "Attempting to find BT|TB muls.\n");
+
+  OpChainList Candidates;
+  for (auto &I : *LoopBody) {
+    if (I.getOpcode() == Instruction::Mul) {
+      Type *Ty = I.getType();
+      if (Ty->isIntegerTy() &&
+          (Ty->getScalarSizeInBits() == 32 ||
+           Ty->getScalarSizeInBits() == 64))
+      AddMulCandidate(Candidates, &I, I.getOperand(0), I.getOperand(1));
+    }
+  }
+
+  if (Candidates.empty())
+    return false;
+
+  Instructions Reads;
+  Instructions Writes;
+  AliasCandidates(LoopBody, Reads, Writes);
+
+  if (AreAliased(AA, Reads, Writes, Candidates))
+    return false;
+
+  DenseMap<LoadInst*, LoadInst*> SeqLoads;
+  SmallPtrSet<LoadInst*, 8> OffsetLoads;
+
+  for (unsigned i = 0; i < Candidates.size(); ++i) {
+    for (unsigned j = 0; j < Candidates.size(); ++j) {
+      if (i == j)
+        continue;
+
+      OpChain *MulChain0 = Candidates[i].get();
+      OpChain *MulChain1 = Candidates[j].get();
+
+      for (auto *Ld0 : MulChain0->Loads) {
+        if (SeqLoads.count(Ld0) || OffsetLoads.count(Ld0))
+          continue;
+
+        for (auto *Ld1 : MulChain1->Loads) {
+          if (SeqLoads.count(Ld1) || OffsetLoads.count(Ld1))
+            continue;
+
+          MemInstList VecMem;
+          if (AreSequentialLoads(Ld0, Ld1, VecMem)) {
+            SeqLoads[Ld0] = Ld1;
+            OffsetLoads.insert(Ld1);
+          }
+        }
+      }
+    }
+  }
+
+  if (SeqLoads.empty())
+    return false;
+
+  IRBuilder<NoFolder> IRB(LoopBody);
+  const Type *Ty = IntegerType::get(M->getContext(), 32);
+
+  auto IsUserMul = [](Use &U) {
+    auto *Mul = cast<Instruction>(U.getUser());
+    return Mul->getOpcode() == Instruction::Mul;
+  };
+
+  LLVM_DEBUG(dbgs() << "Found some sequential loads, now widening:\n");
+  for (auto &Pair : SeqLoads) {
+    LoadInst *BaseLd = Pair.first;
+    LoadInst *OffsetLd = Pair.second;
+
+    // Check that all the base users are muls.
+    auto *BaseSExt = cast<Instruction>(BaseLd->user_back());
+    for (Use &U : BaseSExt->uses()) {
+      if (!IsUserMul(U))
+        return false;
+    }
+
+    // Check that all the offset users are muls.
+    // TODO We exit early on finding a sext user which isn't a mul, but many
+    // arm instructions would be able to perform the necessary shift too.
+    auto *OffsetSExt = cast<Instruction>(OffsetLd->user_back());
+    for (Use &U : OffsetSExt->uses()) {
+      if (!IsUserMul(U))
+        return false;
+    }
+
+    LLVM_DEBUG(dbgs() << " - with base load: " << *BaseLd << "\n");
+    LLVM_DEBUG(dbgs() << " - with offset load: " << *OffsetLd << "\n");
+    Instruction *InsertPt = GetFirst(BaseLd, OffsetLd);
+    IRB.SetInsertPoint(InsertPt);
+    LoadInst *WideLd = CreateLoadIns(IRB, BaseLd, Ty);
+    LLVM_DEBUG(dbgs() << " - created wide load: " << *WideLd << "\n");
+
+    // Move the pointer operands before their users.
+    std::function<void(Instruction*, Instruction*)> MoveBefore =
+      [&MoveBefore](Instruction *Source, Instruction *Sink) -> void {
+      Source->moveBefore(Sink);
+      for (Use &U : Source->operands()) {
+        Value *Op = U.get();
+        if (auto *I = dyn_cast<Instruction>(Op)) {
+          if (isa<PHINode>(I) || I->getParent() != Source->getParent())
+            continue;
+          MoveBefore(I, Source);
+        }
+      }
+    };
+
+    // If we're inserting the load before BaseLd, we probably need to move the
+    // the pointer operand too. This operand is cast to an i32* in
+    // CreateLoadIns.
+    if (InsertPt != BaseLd) {
+      if (auto *GEP = dyn_cast<GetElementPtrInst>(BaseLd->getPointerOperand()))
+        MoveBefore(GEP, cast<Instruction>(WideLd->getPointerOperand()));
+    }
+
+    // BaseUser needs to: (asr (shl WideLoad, 16), 16)
+    // OffsetUser needs to: (asr WideLoad, 16)
+    auto *Top = cast<Instruction>(IRB.CreateAShr(WideLd, 16));
+    auto *Shl = cast<Instruction>(IRB.CreateShl(WideLd, 16));
+    auto *Bottom = cast<Instruction>(IRB.CreateAShr(Shl, 16));
+
+    BaseSExt->replaceAllUsesWith(Bottom);
+    OffsetSExt->replaceAllUsesWith(Top);
+
+    BaseSExt->eraseFromParent();
+    OffsetSExt->eraseFromParent();
+    BaseLd->eraseFromParent();
+    OffsetLd->eraseFromParent();
+  }
+  LLVM_DEBUG(dbgs() << "Block after top bottom mul replacements:\n"
+             << *LoopBody << "\n");
+  return true;
+}
+
 // Loop Pass that needs to identify integer add/sub reductions of 16-bit vector
 // multiplications.
 // To use SMLAD:
 // 1) we first need to find integer add reduction PHIs,
 // 2) then from the PHI, look for this pattern:
 //
 // acc0 = phi i32 [0, %entry], [%acc1, %loop.body]
 // ld0 = load i16

 // before the loop begins.
 //
 bool ARMParallelDSP::MatchSMLAD(Function &F) {
   BasicBlock *Header = L->getHeader();
   LLVM_DEBUG(dbgs() << "= Matching SMLAD =\n";
              dbgs() << "Header block:\n"; Header->dump();
              dbgs() << "Loop info:\n\n"; L->dump());
 
-  bool Changed = false;
   ReductionList Reductions;
   MatchReductions(F, L, Header, Reductions);
+  if (Reductions.empty())
+    return false;
 
   for (auto &R : Reductions) {
     OpChainList MACCandidates;
     MatchParallelMACSequences(R, MACCandidates);
-    if (!CheckMACMemory(MACCandidates))
+    if (!CheckMulMemory(MACCandidates))
       continue;
 
     R.MACCandidates = std::move(MACCandidates);
 
     LLVM_DEBUG(dbgs() << "MAC candidates:\n";
       for (auto &M : R.MACCandidates)
         M->Root->dump();
       dbgs() << "\n";);
   }
 
   // Collect all instructions that may read or write memory. Our alias
   // analysis checks bail out if any of these instructions aliases with an
   // instruction from the MAC-chain.
   Instructions Reads, Writes;
   AliasCandidates(Header, Reads, Writes);
 
+  bool Changed = false;
   for (auto &R : Reductions) {
     if (AreAliased(AA, Reads, Writes, R.MACCandidates))
       return false;
     PMACPairList PMACPairs = CreateParallelMACPairs(R.MACCandidates);
     Changed |= InsertParallelMACs(R, PMACPairs);
   }
 
   LLVM_DEBUG(if (Changed) dbgs() << "Header block:\n"; Header->dump(););
   return Changed;
 }
 
-static LoadInst *CreateLoadIns(IRBuilder<NoFolder> &IRB, LoadInst &BaseLoad,
-                               const Type *LoadTy) {
-  const unsigned AddrSpace = BaseLoad.getPointerAddressSpace();
-
-  Value *VecPtr = IRB.CreateBitCast(BaseLoad.getPointerOperand(),
-                                    LoadTy->getPointerTo(AddrSpace));
-  return IRB.CreateAlignedLoad(VecPtr, BaseLoad.getAlignment());
-}
-
 Instruction *ARMParallelDSP::CreateSMLADCall(LoadInst *VecLd0, LoadInst *VecLd1,
                                              Instruction *Acc, bool Exchange,
                                              Instruction *InsertAfter) {
   LLVM_DEBUG(dbgs() << "Create SMLAD intrinsic using:\n"
              << "- " << *VecLd0 << "\n"
              << "- " << *VecLd1 << "\n"
              << "- " << *Acc << "\n"
              << "Exchange: " << Exchange << "\n");
 
   IRBuilder<NoFolder> Builder(InsertAfter->getParent(),
                               ++BasicBlock::iterator(InsertAfter));
 
   // Replace the reduction chain with an intrinsic call
   const Type *Ty = IntegerType::get(M->getContext(), 32);
-  LoadInst *NewLd0 = CreateLoadIns(Builder, VecLd0[0], Ty);
-  LoadInst *NewLd1 = CreateLoadIns(Builder, VecLd1[0], Ty);
+  LoadInst *NewLd0 = CreateLoadIns(Builder, &VecLd0[0], Ty);
+  LoadInst *NewLd1 = CreateLoadIns(Builder, &VecLd1[0], Ty);
   Value* Args[] = { NewLd0, NewLd1, Acc };
   Function *SMLAD = nullptr;
   if (Exchange)
     SMLAD = Acc->getType()->isIntegerTy(32) ?
       Intrinsic::getDeclaration(M, Intrinsic::arm_smladx) :
       Intrinsic::getDeclaration(M, Intrinsic::arm_smlaldx);
   else
     SMLAD = Acc->getType()->isIntegerTy(32) ?

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad0.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; The Cortex-M0 does not support unaligned accesses:
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+;
+; Check DSP extension:
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+
+define dso_local i32 @OneReduction(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+;
+; CHECK-LABEL: @OneReduction
+; CHECK:  %mac1{{\.}}026 = phi i32 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
+; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
+; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
+; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
+; CHECK:  [[V8]] = call i32 @llvm.arm.smlad(i32 [[V5]], i32 [[V7]], i32 %mac1{{\.}}026)
+; CHECK-NOT: call i32 @llvm.arm.smlad
+;
+; CHECK-UNSUPPORTED-NOT:  call i32 @llvm.arm.smlad
+;
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+; One reduction statement here:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+
+; Here the Mul is the LHS, and the Add the RHS.
+  %add11 = add i32 %mul9, %add10
+
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+
+define dso_local arm_aapcs_vfpcc i32 @TwoReductions(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+;
+; CHECK-LABEL: @TwoReductions
+;
+; CHECK:  %mac1{{\.}}058 = phi i32 [ [[V10:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  %mac2{{\.}}057 = phi i32 [ [[V17:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V10]] = call i32 @llvm.arm.smlad(i32 %{{.*}}, i32 %{{.*}}, i32 %mac1{{\.}}058)
+; CHECK:  [[V17]] = call i32 @llvm.arm.smlad(i32 %{{.*}}, i32 %{{.*}}, i32 %mac2{{\.}}057)
+; CHECK-NOT: call i32 @llvm.arm.smlad
+;
+entry:
+  %cmp55 = icmp sgt i32 %arg, 0
+  br i1 %cmp55, label %for.body.preheader, label %for.cond.cleanup
+
+for.cond.cleanup:
+  %mac2.0.lcssa = phi i32 [ 0, %entry ], [ %add28, %for.body ]
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add16, %for.body ]
+  %add30 = add nsw i32 %mac1.0.lcssa, %mac2.0.lcssa
+  ret i32 %add30
+
+for.body.preheader:
+  br label %for.body
+
+for.body:
+; And two reduction statements here:
+  %mac1.058 = phi i32 [ %add16, %for.body ], [ 0, %for.body.preheader ]
+  %mac2.057 = phi i32 [ %add28, %for.body ], [ 0, %for.body.preheader ]
+
+  %i.056 = phi i32 [ %add29, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.056
+  %0 = load i16, i16* %arrayidx, align 2
+  %add1 = or i32 %i.056, 1
+  %arrayidx2 = getelementptr inbounds i16, i16* %arg3, i32 %add1
+  %1 = load i16, i16* %arrayidx2, align 2
+  %add3 = or i32 %i.056, 2
+  %arrayidx4 = getelementptr inbounds i16, i16* %arg3, i32 %add3
+  %2 = load i16, i16* %arrayidx4, align 2
+
+  %add5 = or i32 %i.056, 3
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg3, i32 %add5
+  %3 = load i16, i16* %arrayidx6, align 2
+  %arrayidx8 = getelementptr inbounds i16, i16* %arg2, i32 %i.056
+  %4 = load i16, i16* %arrayidx8, align 2
+  %conv = sext i16 %4 to i32
+  %conv9 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv9
+  %arrayidx11 = getelementptr inbounds i16, i16* %arg2, i32 %add1
+  %5 = load i16, i16* %arrayidx11, align 2
+  %conv12 = sext i16 %5 to i32
+  %conv13 = sext i16 %1 to i32
+  %mul14 = mul nsw i32 %conv12, %conv13
+  %add15 = add i32 %mul, %mac1.058
+  %add16 = add i32 %add15, %mul14
+  %arrayidx18 = getelementptr inbounds i16, i16* %arg2, i32 %add3
+  %6 = load i16, i16* %arrayidx18, align 2
+  %conv19 = sext i16 %6 to i32
+  %conv20 = sext i16 %2 to i32
+  %mul21 = mul nsw i32 %conv19, %conv20
+  %arrayidx23 = getelementptr inbounds i16, i16* %arg2, i32 %add5
+  %7 = load i16, i16* %arrayidx23, align 2
+  %conv24 = sext i16 %7 to i32
+  %conv25 = sext i16 %3 to i32
+  %mul26 = mul nsw i32 %conv24, %conv25
+  %add27 = add i32 %mul21, %mac2.057
+  %add28 = add i32 %add27, %mul26
+  %add29 = add nuw nsw i32 %i.056, 4
+  %cmp = icmp slt i32 %add29, %arg
+  br i1 %cmp, label %for.body, label %for.cond.cleanup
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad1.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+
+; CHECK-LABEL: @test1
+; CHECK:  %mac1{{\.}}026 = phi i32 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
+; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
+; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
+; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
+; CHECK:  [[V8]] = call i32 @llvm.arm.smlad(i32 [[V5]], i32 [[V7]], i32 %mac1{{\.}}026)
+
+define dso_local i32 @test1(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+
+; And here the Add is the LHS, the Mul the RHS
+  %add11 = add i32 %add10, %mul9
+
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+
+; Here we have i8 loads, which we do want to support, but don't handle yet.
+;
+; CHECK-LABEL: @test2
+; CHECK-NOT:   call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test2(i32 %arg, i32* nocapture readnone %arg1, i8* nocapture readonly %arg2, i8* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i8, i8* %arg3, align 2
+  %.pre27 = load i8, i8* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i8, i8* %arg3, i32 %i.025
+  %0 = load i8, i8* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i8, i8* %arg3, i32 %add
+  %1 = load i8, i8* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i8, i8* %arg2, i32 %i.025
+  %2 = load i8, i8* %arrayidx3, align 2
+  %conv = sext i8 %2 to i32
+  %conv4 = sext i8 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i8, i8* %arg2, i32 %add
+  %3 = load i8, i8* %arrayidx6, align 2
+  %conv7 = sext i8 %3 to i32
+  %conv8 = sext i8 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+  %add11 = add i32 %add10, %mul9
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad10.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; Reduction statement is an i64 type: we only support i32 so check that the
+; rewrite isn't triggered.
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i64 @test(i64 %arg, i64* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i64 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
+  ret i64 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i64 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i64 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i64 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i64 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i64 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i64
+  %conv4 = sext i16 %0 to i64
+  %mul = mul nsw i64 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i64 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i64
+  %conv8 = sext i16 %1 to i64
+  %mul9 = mul nsw i64 %conv7, %conv8
+  %add10 = add i64 %mul, %mac1.026
+
+  %add11 = add i64 %mul9, %add10
+
+  %exitcond = icmp ne i64 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad11.ll

+; REQUIRES: asserts
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S -stats 2>&1 | FileCheck %s
+;
+; A more complicated chain: 4 mul operations, so we expect 2 smlad calls.
+;
+; CHECK:  %mac1{{\.}}054 = phi i32 [ [[V17:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V8:%[0-9]+]] = bitcast i16* %arrayidx8 to i32*
+; CHECK:  [[V9:%[0-9]+]] = load i32, i32* [[V8]], align 2
+; CHECK:  [[V10:%[0-9]+]] = bitcast i16* %arrayidx to i32*
+; CHECK:  [[V11:%[0-9]+]] = load i32, i32* [[V10]], align 2
+; CHECK:  [[V12:%[0-9]+]] = call i32 @llvm.arm.smlad(i32 [[V9]], i32 [[V11]], i32 %mac1{{\.}}054)
+; CHECK:  [[V13:%[0-9]+]] = bitcast i16* %arrayidx17 to i32*
+; CHECK:  [[V14:%[0-9]+]] = load i32, i32* [[V13]], align 2
+; CHECK:  [[V15:%[0-9]+]] = bitcast i16* %arrayidx4 to i32*
+; CHECK:  [[V16:%[0-9]+]] = load i32, i32* [[V15]], align 2
+; CHECK:  [[V17:%[0-9]+]] = call i32 @llvm.arm.smlad(i32 [[V14]], i32 [[V16]], i32 [[V12]])
+;
+; And we don't want to see a 3rd smlad:
+; CHECK-NOT: call i32 @llvm.arm.smlad
+;
+; CHECK:  2 arm-parallel-dsp - Number of smlad instructions generated
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp52 = icmp sgt i32 %arg, 0
+  br i1 %cmp52, label %for.body.preheader, label %for.cond.cleanup
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add28, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body.preheader:
+  br label %for.body
+
+for.body:
+  %mac1.054 = phi i32 [ %add28, %for.body ], [ 0, %for.body.preheader ]
+  %i.053 = phi i32 [ %add29, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.053
+  %0 = load i16, i16* %arrayidx, align 2
+  %add1 = or i32 %i.053, 1
+  %arrayidx2 = getelementptr inbounds i16, i16* %arg3, i32 %add1
+  %1 = load i16, i16* %arrayidx2, align 2
+  %add3 = or i32 %i.053, 2
+  %arrayidx4 = getelementptr inbounds i16, i16* %arg3, i32 %add3
+  %2 = load i16, i16* %arrayidx4, align 2
+  %add5 = or i32 %i.053, 3
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg3, i32 %add5
+  %3 = load i16, i16* %arrayidx6, align 2
+  %arrayidx8 = getelementptr inbounds i16, i16* %arg2, i32 %i.053
+  %4 = load i16, i16* %arrayidx8, align 2
+  %conv = sext i16 %4 to i32
+  %conv9 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv9
+  %arrayidx11 = getelementptr inbounds i16, i16* %arg2, i32 %add1
+  %5 = load i16, i16* %arrayidx11, align 2
+  %conv12 = sext i16 %5 to i32
+  %conv13 = sext i16 %1 to i32
+  %mul14 = mul nsw i32 %conv12, %conv13
+  %arrayidx17 = getelementptr inbounds i16, i16* %arg2, i32 %add3
+  %6 = load i16, i16* %arrayidx17, align 2
+  %conv18 = sext i16 %6 to i32
+  %conv19 = sext i16 %2 to i32
+  %mul20 = mul nsw i32 %conv18, %conv19
+  %arrayidx23 = getelementptr inbounds i16, i16* %arg2, i32 %add5
+  %7 = load i16, i16* %arrayidx23, align 2
+  %conv24 = sext i16 %7 to i32
+  %conv25 = sext i16 %3 to i32
+  %mul26 = mul nsw i32 %conv24, %conv25
+  %add15 = add i32 %mul, %mac1.054
+  %add21 = add i32 %add15, %mul14
+  %add27 = add i32 %add21, %mul20
+  %add28 = add i32 %add27, %mul26
+  %add29 = add nuw nsw i32 %i.053, 4
+  %cmp = icmp slt i32 %add29, %arg
+  br i1 %cmp, label %for.body, label %for.cond.cleanup
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad12.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; The loop header is not the loop latch.
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+; This is the loop header:
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body2 ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body2 ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+  %add11 = add i32 %mul9, %add10
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body2, label %for.cond.cleanup
+
+; And this is the loop latch:
+for.body2:
+  br label %for.body
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad2.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; Operands of both muls are not symmetrical (see also comments inlined below), check
+; that the rewrite isn't triggered.
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+
+; This zero-extends the 2nd operand of %mul:
+  %conv4 = zext i16 %0 to i32
+
+  %mul = mul nsw i32 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+
+; And here we only have sign-extensions. Thus, the operands of
+; %mul and %mul9 are not symmetrical:
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+  %add11 = add i32 %add10, %mul9
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad3.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; The loads are not consecutive: check that the rewrite isn't triggered.
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+
+; Here we add another constants offset of 2, to make sure the
+; loads to %3 and %2 are not consecutive:
+
+  %add5 = add nuw nsw i32 %i.025, 2
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add5
+  %3 = load i16, i16* %arrayidx6, align 2
+
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+  %add11 = add i32 %add10, %mul9
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad4.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; The loads are not narrow loads: check that the rewrite isn't triggered.
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+; Arg2 is now an i32, while Arg3 is still and i16:
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i32* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp22 = icmp sgt i32 %arg, 0
+  br i1 %cmp22, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add9, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %0 = phi i16 [ %1, %for.body ], [ %.pre, %for.body.preheader ]
+  %mac1.024 = phi i32 [ %add9, %for.body ], [ 0, %for.body.preheader ]
+  %i.023 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %add = add nuw nsw i32 %i.023, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %conv = sext i16 %0 to i32
+
+; This is a 'normal' i32 load to %2:
+  %arrayidx3 = getelementptr inbounds i32, i32* %arg2, i32 %i.023
+  %2 = load i32, i32* %arrayidx3, align 4
+
+; This mul has now 1 operand which is a narrow load, and the other a normal
+; i32 load:
+  %mul = mul nsw i32 %2, %conv
+
+  %add4 = add nuw nsw i32 %i.023, 2
+  %arrayidx5 = getelementptr inbounds i32, i32* %arg2, i32 %add4
+  %3 = load i32, i32* %arrayidx5, align 4
+  %conv6 = sext i16 %1 to i32
+  %mul7 = mul nsw i32 %3, %conv6
+  %add8 = add i32 %mul, %mac1.024
+  %add9 = add i32 %add8, %mul7
+  %exitcond = icmp eq i32 %add, %arg
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad5.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; The loads are volatile loads: check that the rewrite isn't triggered.
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load volatile i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load volatile i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load volatile i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load volatile i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+  %add11 = add i32 %add10, %mul9
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad6.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; Alias check: check that the rewrite isn't triggered when there's a store
+; instruction possibly aliasing any mul load operands; arguments are passed
+; without 'restrict' enabled.
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+
+; Store inserted here, aliasing with arrayidx, arrayidx1, arrayidx3
+  store i16 42, i16* %arrayidx, align 2
+
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+  %add11 = add i32 %mul9, %add10
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad7.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; Alias check: check that the rewrite isn't triggered when there's a store
+; aliasing one of the mul load operands. Arguments are now annotated with
+; 'noalias'.
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test(i32 %arg, i32* noalias %arg1, i16* noalias readonly %arg2, i16* noalias readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+
+; Store inserted here, aliasing only with loads from 'arrayidx'.
+  store i16 42, i16* %arrayidx, align 2
+
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %add10 = add i32 %mul, %mac1.026
+
+; Here the Mul is the LHS, and the Add the RHS.
+  %add11 = add i32 %mul9, %add10
+
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad8.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; Mul with operands that are not simple load and sext/zext chains: this is not
+; yet supported so the rewrite shouldn't trigger (but we do want to support this
+; soon).
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3, i16* %arg4) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  %gep0 = getelementptr inbounds i16, i16* %arg4, i32 0
+  %gep1 = getelementptr inbounds i16, i16* %arg4, i32 1
+  %.add4 = load i16, i16* %gep0, align 2
+  %.add5 = load i16, i16* %gep1, align 2
+  %.zext4 = zext i16 %.add4 to i32
+  %.zext5 = zext i16 %.add5 to i32
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %add1 = add i32 %conv, %.zext4
+
+; This mul has a more complicated pattern as an operand, %add1
+; is another add and load, which we don't support for now.
+  %mul = mul nsw i32 %add1, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %add2 = add i32 %conv7, %.zext5
+
+; Same here
+  %mul9 = mul nsw i32 %add2, %conv8
+  %add10 = add i32 %mul, %mac1.026
+
+  %add11 = add i32 %mul9, %add10
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlad9.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; Muls with operands that are constants: not yet supported, so the rewrite
+; should not trigger (but we do want to add this soon).
+;
+; CHECK-NOT:  call i32 @llvm.arm.smlad
+;
+define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
+  ret i32 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %v2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %v2 to i32
+
+; RHS operand of this mul is a constant
+  %mul = mul nsw i32 %conv, 43
+
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %v3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %v3 to i32
+
+; And this RHS operand is a constant too.
+  %mul9 = mul nsw i32 %conv7, 42
+
+  %add10 = add i32 %mul, %mac1.026
+  %add11 = add i32 %mul9, %add10
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smladx-1.ll

+; RUN: opt -mtriple=thumbv8m.main -mcpu=cortex-m33 -arm-parallel-dsp %s -S -o - | FileCheck %s
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+
+define i32 @smladx(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
+
+; CHECK-LABEL: smladx
+; CHECK: = phi i32 [ 0, %for.body.preheader.new ],
+; CHECK: [[ACC0:%[^ ]+]] = phi i32 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
+; CHECK: [[PIN23:%[^ ]+]] = bitcast i16* %pIn2.3 to i32*
+; CHECK: [[IN23:%[^ ]+]] = load i32, i32* [[PIN23]], align 2
+; CHECK: [[PIN12:%[^ ]+]] = bitcast i16* %pIn1.2 to i32*
+; CHECK: [[IN12:%[^ ]+]] = load i32, i32* [[PIN12]], align 2
+; CHECK: [[ACC1:%[^ ]+]] = call i32 @llvm.arm.smladx(i32 [[IN23]], i32 [[IN12]], i32 [[ACC0]])
+; CHECK: [[PIN21:%[^ ]+]] = bitcast i16* %pIn2.1 to i32*
+; CHECK: [[IN21:%[^ ]+]] = load i32, i32* [[PIN21]], align 2
+; CHECK: [[PIN10:%[^ ]+]] = bitcast i16* %pIn1.0 to i32*
+; CHECK: [[IN10:%[^ ]+]] = load i32, i32* [[PIN10]], align 2
+; CHECK: [[ACC2]] = call i32 @llvm.arm.smladx(i32 [[IN21]], i32 [[IN10]], i32 [[ACC1]])
+; CHECK-NOT: call i32 @llvm.arm.smlad
+; CHECK-UNSUPPORTED-NOT:  call i32 @llvm.arm.smlad
+
+entry:
+  %cmp9 = icmp eq i32 %limit, 0
+  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
+
+for.body.preheader:
+  %0 = add i32 %limit, -1
+  %xtraiter = and i32 %limit, 3
+  %1 = icmp ult i32 %0, 3
+  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
+
+for.body.preheader.new:
+  %unroll_iter = sub i32 %limit, %xtraiter
+  br label %for.body
+
+for.cond.cleanup.loopexit.unr-lcssa:
+  %add.lcssa.ph = phi i32 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
+  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
+  %sum.010.unr = phi i32 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
+  %lcmp.mod = icmp eq i32 %xtraiter, 0
+  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil
+
+for.body.epil:
+  %i.011.epil = phi i32 [ %inc.epil, %for.body.epil ], [ %i.011.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %sum.010.epil = phi i32 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %epil.iter = phi i32 [ %epil.iter.sub, %for.body.epil ], [ %xtraiter, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %sub.epil = sub i32 %j, %i.011.epil
+  %arrayidx.epil = getelementptr inbounds i16, i16* %pIn2, i32 %sub.epil
+  %2 = load i16, i16* %arrayidx.epil, align 2
+  %conv.epil = sext i16 %2 to i32
+  %arrayidx1.epil = getelementptr inbounds i16, i16* %pIn1, i32 %i.011.epil
+  %3 = load i16, i16* %arrayidx1.epil, align 2
+  %conv2.epil = sext i16 %3 to i32
+  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
+  %add.epil = add nsw i32 %mul.epil, %sum.010.epil
+  %inc.epil = add nuw i32 %i.011.epil, 1
+  %epil.iter.sub = add i32 %epil.iter, -1
+  %epil.iter.cmp = icmp eq i32 %epil.iter.sub, 0
+  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
+
+for.cond.cleanup:
+  %sum.0.lcssa = phi i32 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
+  ret i32 %sum.0.lcssa
+
+for.body:
+  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
+  %sum.010 = phi i32 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
+  %niter = phi i32 [ %unroll_iter, %for.body.preheader.new ], [ %niter.nsub.3, %for.body ]
+  %pIn2Base = phi i16* [ %pIn2, %for.body.preheader.new ], [ %pIn2.4, %for.body ]
+  %pIn2.0 = getelementptr inbounds i16, i16* %pIn2Base, i32 0
+  %In2 = load i16, i16* %pIn2.0, align 2
+  %pIn1.0 = getelementptr inbounds i16, i16* %pIn1, i32 %i.011
+  %In1 = load i16, i16* %pIn1.0, align 2
+  %inc = or i32 %i.011, 1
+  %pIn2.1 = getelementptr inbounds i16, i16* %pIn2Base, i32 -1
+  %In2.1 = load i16, i16* %pIn2.1, align 2
+  %pIn1.1 = getelementptr inbounds i16, i16* %pIn1, i32 %inc
+  %In1.1 = load i16, i16* %pIn1.1, align 2
+  %inc.1 = or i32 %i.011, 2
+  %pIn2.2 = getelementptr inbounds i16, i16* %pIn2Base, i32 -2
+  %In2.2 = load i16, i16* %pIn2.2, align 2
+  %pIn1.2 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.1
+  %In1.2 = load i16, i16* %pIn1.2, align 2
+  %inc.2 = or i32 %i.011, 3
+  %pIn2.3 = getelementptr inbounds i16, i16* %pIn2Base, i32 -3
+  %In2.3 = load i16, i16* %pIn2.3, align 2
+  %pIn1.3 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.2
+  %In1.3 = load i16, i16* %pIn1.3, align 2
+  %sextIn1 = sext i16 %In1 to i32
+  %sextIn1.1 = sext i16 %In1.1 to i32
+  %sextIn1.2 = sext i16 %In1.2 to i32
+  %sextIn1.3 = sext i16 %In1.3 to i32
+  %sextIn2 = sext i16 %In2 to i32
+  %sextIn2.1 = sext i16 %In2.1 to i32
+  %sextIn2.2 = sext i16 %In2.2 to i32
+  %sextIn2.3 = sext i16 %In2.3 to i32
+  %mul = mul nsw i32 %sextIn1, %sextIn2
+  %mul.1 = mul nsw i32 %sextIn1.1, %sextIn2.1
+  %mul.2 = mul nsw i32 %sextIn1.2, %sextIn2.2
+  %mul.3 = mul nsw i32 %sextIn1.3, %sextIn2.3
+  %add = add nsw i32 %mul, %sum.010
+  %add.1 = add nsw i32 %mul.1, %add
+  %add.2 = add nsw i32 %mul.2, %add.1
+  %add.3 = add nsw i32 %mul.3, %add.2
+  %inc.3 = add i32 %i.011, 4
+  %pIn2.4 = getelementptr inbounds i16, i16* %pIn2Base, i32 -4
+  %niter.nsub.3 = add i32 %niter, -4
+  %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
+  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
+}
+
+define i32 @smladx_swap(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
+
+; CHECK-LABEL: smladx_swap
+; CHECK: for.body.preheader.new:
+; CHECK: [[PIN1Base:[^ ]+]] = getelementptr i16, i16* %pIn1
+; CHECK: [[PIN2Base:[^ ]+]] = getelementptr i16, i16* %pIn2
+
+; CHECK: for.body:
+; CHECK: [[PIN2:%[^ ]+]] = phi i16* [ [[PIN2_NEXT:%[^ ]+]], %for.body ], [ [[PIN2Base]], %for.body.preheader.new ]
+; CHECK: [[PIN1:%[^ ]+]] = phi i16* [ [[PIN1_NEXT:%[^ ]+]], %for.body ], [ [[PIN1Base]], %for.body.preheader.new ]
+; CHECK: [[IV:%[^ ]+]] = phi i32
+; CHECK: [[ACC0:%[^ ]+]] = phi i32 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
+
+; CHECK: [[PIN1_2:%[^ ]+]] = getelementptr i16, i16* [[PIN1]], i32 -2
+; CHECK: [[PIN2_2:%[^ ]+]] = getelementptr i16, i16* [[PIN2]], i32 -2
+
+
+; CHECK: [[PIN2_2_CAST:%[^ ]+]] = bitcast i16* [[PIN2_2]] to i32*
+; CHECK: [[IN2_2:%[^ ]+]] = load i32, i32* [[PIN2_2_CAST]], align 2
+; CHECK: [[PIN1_CAST:%[^ ]+]] = bitcast i16* [[PIN1]] to i32*
+; CHECK: [[IN1:%[^ ]+]] = load i32, i32* [[PIN1_CAST]], align 2
+; CHECK: [[ACC1:%[^ ]+]] = call i32 @llvm.arm.smladx(i32 [[IN2_2]], i32 [[IN1]], i32 [[ACC0]])
+
+; CHECK: [[PIN2_CAST:%[^ ]+]] = bitcast i16* [[PIN2]] to i32*
+; CHECK: [[IN2:%[^ ]+]] = load i32, i32* [[PIN2_CAST]], align 2
+; CHECK: [[PIN1_2_CAST:%[^ ]+]] = bitcast i16* [[PIN1_2]] to i32*
+; CHECK: [[IN1_2:%[^ ]+]] = load i32, i32* [[PIN1_2_CAST]], align 2
+; CHECK: [[ACC2]] = call i32 @llvm.arm.smladx(i32 [[IN2]], i32 [[IN1_2]], i32 [[ACC1]])
+
+; CHECK: [[PIN1_NEXT]] = getelementptr i16, i16* [[PIN1]], i32 4
+; CHECK: [[PIN2_NEXT]] = getelementptr i16, i16* [[PIN2]], i32 -4
+
+; CHECK-NOT: call i32 @llvm.arm.smlad
+; CHECK-UNSUPPORTED-NOT:  call i32 @llvm.arm.smlad
+
+entry:
+  %cmp9 = icmp eq i32 %limit, 0
+  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
+
+for.body.preheader:
+  %0 = add i32 %limit, -1
+  %xtraiter = and i32 %limit, 3
+  %1 = icmp ult i32 %0, 3
+  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
+
+for.body.preheader.new:
+  %unroll_iter = sub i32 %limit, %xtraiter
+  %scevgep6 = getelementptr i16, i16* %pIn1, i32 2
+  %2 = add i32 %j, -1
+  %scevgep11 = getelementptr i16, i16* %pIn2, i32 %2
+  br label %for.body
+
+for.cond.cleanup.loopexit.unr-lcssa:
+  %add.lcssa.ph = phi i32 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
+  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
+  %sum.010.unr = phi i32 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
+  %lcmp.mod = icmp eq i32 %xtraiter, 0
+  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil.preheader
+
+for.body.epil.preheader:
+  %scevgep = getelementptr i16, i16* %pIn1, i32 %i.011.unr
+  %3 = sub i32 %j, %i.011.unr
+  %scevgep2 = getelementptr i16, i16* %pIn2, i32 %3
+  %4 = sub i32 0, %xtraiter
+  br label %for.body.epil
+
+for.body.epil:
+  %lsr.iv5 = phi i32 [ %4, %for.body.epil.preheader ], [ %lsr.iv.next, %for.body.epil ]
+  %lsr.iv3 = phi i16* [ %scevgep2, %for.body.epil.preheader ], [ %scevgep4, %for.body.epil ]
+  %lsr.iv = phi i16* [ %scevgep, %for.body.epil.preheader ], [ %scevgep1, %for.body.epil ]
+  %sum.010.epil = phi i32 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.body.epil.preheader ]
+  %5 = load i16, i16* %lsr.iv3, align 2
+  %conv.epil = sext i16 %5 to i32
+  %6 = load i16, i16* %lsr.iv, align 2
+  %conv2.epil = sext i16 %6 to i32
+  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
+  %add.epil = add nsw i32 %mul.epil, %sum.010.epil
+  %scevgep1 = getelementptr i16, i16* %lsr.iv, i32 1
+  %scevgep4 = getelementptr i16, i16* %lsr.iv3, i32 -1
+  %lsr.iv.next = add nsw i32 %lsr.iv5, 1
+  %epil.iter.cmp = icmp eq i32 %lsr.iv.next, 0
+  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
+
+for.cond.cleanup:
+  %sum.0.lcssa = phi i32 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
+  ret i32 %sum.0.lcssa
+
+for.body:
+  %pin2 = phi i16* [ %pin2_sub4, %for.body ], [ %scevgep11, %for.body.preheader.new ]
+  %pin1 = phi i16* [ %pin1_add4, %for.body ], [ %scevgep6, %for.body.preheader.new ]
+  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
+  %sum.010 = phi i32 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
+  %pin2_add1 = getelementptr i16, i16* %pin2, i32 1
+  %In2 = load i16, i16* %pin2_add1, align 2
+  %pin1_sub2 = getelementptr i16, i16* %pin1, i32 -2
+  %In1 = load i16, i16* %pin1_sub2, align 2
+  %In2.1 = load i16, i16* %pin2, align 2
+  %pin1_sub1 = getelementptr i16, i16* %pin1, i32 -1
+  %In1.1 = load i16, i16* %pin1_sub1, align 2
+  %pin2_sub1 = getelementptr i16, i16* %pin2, i32 -1
+  %In2.2 = load i16, i16* %pin2_sub1, align 2
+  %In1.2 = load i16, i16* %pin1, align 2
+  %pin2_sub2 = getelementptr i16, i16* %pin2, i32 -2
+  %In2.3 = load i16, i16* %pin2_sub2, align 2
+  %pin1_add1 = getelementptr i16, i16* %pin1, i32 1
+  %In1.3 = load i16, i16* %pin1_add1, align 2
+  %sextIn2 = sext i16 %In2 to i32
+  %sextIn1 = sext i16 %In1 to i32
+  %sextIn2.1 = sext i16 %In2.1 to i32
+  %sextIn1.1 = sext i16 %In1.1 to i32
+  %sextIn2.2 = sext i16 %In2.2 to i32
+  %sextIn1.2 = sext i16 %In1.2 to i32
+  %sextIn2.3 = sext i16 %In2.3 to i32
+  %sextIn1.3 = sext i16 %In1.3 to i32
+  %mul = mul nsw i32 %sextIn2, %sextIn1
+  %add = add nsw i32 %mul, %sum.010
+  %mul.1 = mul nsw i32 %sextIn2.1, %sextIn1.1
+  %add.1 = add nsw i32 %mul.1, %add
+  %mul.2 = mul nsw i32 %sextIn2.2, %sextIn1.2
+  %add.2 = add nsw i32 %mul.2, %add.1
+  %mul.3 = mul nsw i32 %sextIn2.3, %sextIn1.3
+  %add.3 = add nsw i32 %mul.3, %add.2
+  %inc.3 = add i32 %i.011, 4
+  %pin1_add4 = getelementptr i16, i16* %pin1, i32 4
+  %pin2_sub4 = getelementptr i16, i16* %pin2, i32 -4
+  %niter.ncmp.3 = icmp eq i32 %unroll_iter, %inc.3
+  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlald0.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; The Cortex-M0 does not support unaligned accesses:
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+;
+; Check DSP extension:
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+
+define dso_local i64 @OneReduction(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+;
+; CHECK-LABEL: @OneReduction
+; CHECK:  %mac1{{\.}}026 = phi i64 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
+; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
+; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
+; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
+; CHECK:  [[V8]] = call i64 @llvm.arm.smlald(i32 [[V5]], i32 [[V7]], i64 %mac1{{\.}}026)
+; CHECK-NOT: call i64 @llvm.arm.smlald
+;
+; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlald
+;
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
+  ret i64 %mac1.0.lcssa
+
+for.body:
+; One reduction statement here:
+  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i64
+  %conv4 = sext i16 %0 to i64
+  %mul = mul nsw i64 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i64
+  %conv8 = sext i16 %1 to i64
+  %mul9 = mul nsw i64 %conv7, %conv8
+  %add10 = add i64 %mul, %mac1.026
+
+; Here the Mul is the LHS, and the Add the RHS.
+  %add11 = add i64 %mul9, %add10
+
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+
+define dso_local arm_aapcs_vfpcc i64 @TwoReductions(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+;
+; CHECK-LABEL: @TwoReductions
+;
+; CHECK:  %mac1{{\.}}058 = phi i64 [ [[V10:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  %mac2{{\.}}057 = phi i64 [ [[V17:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V10]] = call i64 @llvm.arm.smlald(i32 %{{.*}}, i32 %{{.*}}, i64 %mac1{{\.}}058)
+; CHECK:  [[V17]] = call i64 @llvm.arm.smlald(i32 %{{.*}}, i32 %{{.*}}, i64 %mac2{{\.}}057)
+; CHECK-NOT: call i64 @llvm.arm.smlald
+;
+entry:
+  %cmp55 = icmp sgt i32 %arg, 0
+  br i1 %cmp55, label %for.body.preheader, label %for.cond.cleanup
+
+for.cond.cleanup:
+  %mac2.0.lcssa = phi i64 [ 0, %entry ], [ %add28, %for.body ]
+  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add16, %for.body ]
+  %add30 = add nsw i64 %mac1.0.lcssa, %mac2.0.lcssa
+  ret i64 %add30
+
+for.body.preheader:
+  br label %for.body
+
+for.body:
+; And two reduction statements here:
+  %mac1.058 = phi i64 [ %add16, %for.body ], [ 0, %for.body.preheader ]
+  %mac2.057 = phi i64 [ %add28, %for.body ], [ 0, %for.body.preheader ]
+
+  %i.056 = phi i32 [ %add29, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.056
+  %0 = load i16, i16* %arrayidx, align 2
+  %add1 = or i32 %i.056, 1
+  %arrayidx2 = getelementptr inbounds i16, i16* %arg3, i32 %add1
+  %1 = load i16, i16* %arrayidx2, align 2
+  %add3 = or i32 %i.056, 2
+  %arrayidx4 = getelementptr inbounds i16, i16* %arg3, i32 %add3
+  %2 = load i16, i16* %arrayidx4, align 2
+
+  %add5 = or i32 %i.056, 3
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg3, i32 %add5
+  %3 = load i16, i16* %arrayidx6, align 2
+  %arrayidx8 = getelementptr inbounds i16, i16* %arg2, i32 %i.056
+  %4 = load i16, i16* %arrayidx8, align 2
+  %conv = sext i16 %4 to i64
+  %conv9 = sext i16 %0 to i64
+  %mul = mul nsw i64 %conv, %conv9
+  %arrayidx11 = getelementptr inbounds i16, i16* %arg2, i32 %add1
+  %5 = load i16, i16* %arrayidx11, align 2
+  %conv12 = sext i16 %5 to i64
+  %conv13 = sext i16 %1 to i64
+  %mul14 = mul nsw i64 %conv12, %conv13
+  %add15 = add i64 %mul, %mac1.058
+  %add16 = add i64 %add15, %mul14
+  %arrayidx18 = getelementptr inbounds i16, i16* %arg2, i32 %add3
+  %6 = load i16, i16* %arrayidx18, align 2
+  %conv19 = sext i16 %6 to i64
+  %conv20 = sext i16 %2 to i64
+  %mul21 = mul nsw i64 %conv19, %conv20
+  %arrayidx23 = getelementptr inbounds i16, i16* %arg2, i32 %add5
+  %7 = load i16, i16* %arrayidx23, align 2
+  %conv24 = sext i16 %7 to i64
+  %conv25 = sext i16 %3 to i64
+  %mul26 = mul nsw i64 %conv24, %conv25
+  %add27 = add i64 %mul21, %mac2.057
+  %add28 = add i64 %add27, %mul26
+  %add29 = add nuw nsw i32 %i.056, 4
+  %cmp = icmp slt i32 %add29, %arg
+  br i1 %cmp, label %for.body, label %for.cond.cleanup
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlald1.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+
+; CHECK-LABEL: @test1
+; CHECK:  %mac1{{\.}}026 = phi i64 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
+; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
+; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
+; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
+; CHECK:  [[V8]] = call i64 @llvm.arm.smlald(i32 [[V5]], i32 [[V7]], i64 %mac1{{\.}}026)
+
+define dso_local i64 @test1(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
+  ret i64 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i64
+  %conv4 = sext i16 %0 to i64
+  %mul = mul nsw i64 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i64
+  %conv8 = sext i16 %1 to i64
+  %mul9 = mul nsw i64 %conv7, %conv8
+  %add10 = add i64 %mul, %mac1.026
+
+; And here the Add is the LHS, the Mul the RHS
+  %add11 = add i64 %add10, %mul9
+
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+
+; Here we have i8 loads, which we do want to support, but don't handle yet.
+;
+; CHECK-LABEL: @test2
+; CHECK-NOT:   call i64 @llvm.arm.smlad
+;
+define dso_local i64 @test2(i32 %arg, i32* nocapture readnone %arg1, i8* nocapture readonly %arg2, i8* nocapture readonly %arg3) {
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i8, i8* %arg3, align 2
+  %.pre27 = load i8, i8* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
+  ret i64 %mac1.0.lcssa
+
+for.body:
+  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i8, i8* %arg3, i32 %i.025
+  %0 = load i8, i8* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i8, i8* %arg3, i32 %add
+  %1 = load i8, i8* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i8, i8* %arg2, i32 %i.025
+  %2 = load i8, i8* %arrayidx3, align 2
+  %conv = sext i8 %2 to i64
+  %conv4 = sext i8 %0 to i64
+  %mul = mul nsw i64 %conv, %conv4
+  %arrayidx6 = getelementptr inbounds i8, i8* %arg2, i32 %add
+  %3 = load i8, i8* %arrayidx6, align 2
+  %conv7 = sext i8 %3 to i64
+  %conv8 = sext i8 %1 to i64
+  %mul9 = mul nsw i64 %conv7, %conv8
+  %add10 = add i64 %mul, %mac1.026
+  %add11 = add i64 %add10, %mul9
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlald2.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+;
+; The Cortex-M0 does not support unaligned accesses:
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+;
+; Check DSP extension:
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+
+define dso_local i64 @OneReduction(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+;
+; CHECK-LABEL: @OneReduction
+; CHECK:  %mac1{{\.}}026 = phi i64 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
+; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
+; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
+; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
+; CHECK:  [[V8]] = call i64 @llvm.arm.smlald(i32 [[V5]], i32 [[V7]], i64 %mac1{{\.}}026)
+; CHECK-NOT: call i64 @llvm.arm.smlald
+;
+; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlald
+;
+entry:
+  %cmp24 = icmp sgt i32 %arg, 0
+  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:
+  %.pre = load i16, i16* %arg3, align 2
+  %.pre27 = load i16, i16* %arg2, align 2
+  br label %for.body
+
+for.cond.cleanup:
+  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
+  ret i64 %mac1.0.lcssa
+
+for.body:
+; One reduction statement here:
+  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
+
+  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
+  %0 = load i16, i16* %arrayidx, align 2
+  %add = add nuw nsw i32 %i.025, 1
+  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
+  %1 = load i16, i16* %arrayidx1, align 2
+  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
+  %2 = load i16, i16* %arrayidx3, align 2
+  %conv = sext i16 %2 to i32
+  %conv4 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv4
+  %sext0 = sext i32 %mul to i64
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
+  %3 = load i16, i16* %arrayidx6, align 2
+  %conv7 = sext i16 %3 to i32
+  %conv8 = sext i16 %1 to i32
+  %mul9 = mul nsw i32 %conv7, %conv8
+  %sext1 = sext i32 %mul9 to i64
+  %add10 = add i64 %sext0, %mac1.026
+
+; Here the Mul is the LHS, and the Add the RHS.
+  %add11 = add i64 %sext1, %add10
+
+  %exitcond = icmp ne i32 %add, %arg
+  br i1 %exitcond, label %for.body, label %for.cond.cleanup
+}
+
+define dso_local arm_aapcs_vfpcc i64 @TwoReductions(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
+;
+; CHECK-LABEL: @TwoReductions
+;
+; CHECK:  %mac1{{\.}}058 = phi i64 [ [[V10:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  %mac2{{\.}}057 = phi i64 [ [[V17:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
+; CHECK:  [[V10]] = call i64 @llvm.arm.smlald(i32 %{{.*}}, i32 %{{.*}}, i64 %mac1{{\.}}058)
+; CHECK:  [[V17]] = call i64 @llvm.arm.smlald(i32 %{{.*}}, i32 %{{.*}}, i64 %mac2{{\.}}057)
+; CHECK-NOT: call i64 @llvm.arm.smlald
+;
+entry:
+  %cmp55 = icmp sgt i32 %arg, 0
+  br i1 %cmp55, label %for.body.preheader, label %for.cond.cleanup
+
+for.cond.cleanup:
+  %mac2.0.lcssa = phi i64 [ 0, %entry ], [ %add28, %for.body ]
+  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add16, %for.body ]
+  %add30 = add nsw i64 %mac1.0.lcssa, %mac2.0.lcssa
+  ret i64 %add30
+
+for.body.preheader:
+  br label %for.body
+
+for.body:
+; And two reduction statements here:
+  %mac1.058 = phi i64 [ %add16, %for.body ], [ 0, %for.body.preheader ]
+  %mac2.057 = phi i64 [ %add28, %for.body ], [ 0, %for.body.preheader ]
+
+  %i.056 = phi i32 [ %add29, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.056
+  %0 = load i16, i16* %arrayidx, align 2
+  %add1 = or i32 %i.056, 1
+  %arrayidx2 = getelementptr inbounds i16, i16* %arg3, i32 %add1
+  %1 = load i16, i16* %arrayidx2, align 2
+  %add3 = or i32 %i.056, 2
+  %arrayidx4 = getelementptr inbounds i16, i16* %arg3, i32 %add3
+  %2 = load i16, i16* %arrayidx4, align 2
+
+  %add5 = or i32 %i.056, 3
+  %arrayidx6 = getelementptr inbounds i16, i16* %arg3, i32 %add5
+  %3 = load i16, i16* %arrayidx6, align 2
+  %arrayidx8 = getelementptr inbounds i16, i16* %arg2, i32 %i.056
+  %4 = load i16, i16* %arrayidx8, align 2
+  %conv = sext i16 %4 to i32
+  %conv9 = sext i16 %0 to i32
+  %mul = mul nsw i32 %conv, %conv9
+  %sext0 = sext i32 %mul to i64
+  %arrayidx11 = getelementptr inbounds i16, i16* %arg2, i32 %add1
+  %5 = load i16, i16* %arrayidx11, align 2
+  %conv12 = sext i16 %5 to i32
+  %conv13 = sext i16 %1 to i32
+  %mul14 = mul nsw i32 %conv12, %conv13
+  %sext1 = sext i32 %mul14 to i64
+  %add15 = add i64 %sext0, %mac1.058
+  %add16 = add i64 %add15, %sext1
+  %arrayidx18 = getelementptr inbounds i16, i16* %arg2, i32 %add3
+  %6 = load i16, i16* %arrayidx18, align 2
+  %conv19 = sext i16 %6 to i32
+  %conv20 = sext i16 %2 to i32
+  %mul21 = mul nsw i32 %conv19, %conv20
+  %sext2 = sext i32 %mul21 to i64
+  %arrayidx23 = getelementptr inbounds i16, i16* %arg2, i32 %add5
+  %7 = load i16, i16* %arrayidx23, align 2
+  %conv24 = sext i16 %7 to i32
+  %conv25 = sext i16 %3 to i32
+  %mul26 = mul nsw i32 %conv24, %conv25
+  %sext3 = sext i32 %mul26 to i64
+  %add27 = add i64 %sext2, %mac2.057
+  %add28 = add i64 %add27, %sext3
+  %add29 = add nuw nsw i32 %i.056, 4
+  %cmp = icmp slt i32 %add29, %arg
+  br i1 %cmp, label %for.body, label %for.cond.cleanup
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlaldx-1.ll

+; RUN: opt -mtriple=thumbv8m.main -mcpu=cortex-m33 -arm-parallel-dsp %s -S -o - | FileCheck %s
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+
+define i64 @smlaldx(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
+
+; CHECK-LABEL: smlaldx
+; CHECK: = phi i32 [ 0, %for.body.preheader.new ],
+; CHECK: [[ACC0:%[^ ]+]] = phi i64 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
+; CHECK: [[PIN23:%[^ ]+]] = bitcast i16* %pIn2.3 to i32*
+; CHECK: [[IN23:%[^ ]+]] = load i32, i32* [[PIN23]], align 2
+; CHECK: [[PIN12:%[^ ]+]] = bitcast i16* %pIn1.2 to i32*
+; CHECK: [[IN12:%[^ ]+]] = load i32, i32* [[PIN12]], align 2
+; CHECK: [[ACC1:%[^ ]+]] = call i64 @llvm.arm.smlaldx(i32 [[IN23]], i32 [[IN12]], i64 [[ACC0]])
+; CHECK: [[PIN21:%[^ ]+]] = bitcast i16* %pIn2.1 to i32*
+; CHECK: [[IN21:%[^ ]+]] = load i32, i32* [[PIN21]], align 2
+; CHECK: [[PIN10:%[^ ]+]] = bitcast i16* %pIn1.0 to i32*
+; CHECK: [[IN10:%[^ ]+]] = load i32, i32* [[PIN10]], align 2
+; CHECK: [[ACC2]] = call i64 @llvm.arm.smlaldx(i32 [[IN21]], i32 [[IN10]], i64 [[ACC1]])
+; CHECK-NOT: call i64 @llvm.arm.smlad
+; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlad
+
+entry:
+  %cmp9 = icmp eq i32 %limit, 0
+  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
+
+for.body.preheader:
+  %0 = add i32 %limit, -1
+  %xtraiter = and i32 %limit, 3
+  %1 = icmp ult i32 %0, 3
+  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
+
+for.body.preheader.new:
+  %unroll_iter = sub i32 %limit, %xtraiter
+  br label %for.body
+
+for.cond.cleanup.loopexit.unr-lcssa:
+  %add.lcssa.ph = phi i64 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
+  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
+  %sum.010.unr = phi i64 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
+  %lcmp.mod = icmp eq i32 %xtraiter, 0
+  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil
+
+for.body.epil:
+  %i.011.epil = phi i32 [ %inc.epil, %for.body.epil ], [ %i.011.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %sum.010.epil = phi i64 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %epil.iter = phi i32 [ %epil.iter.sub, %for.body.epil ], [ %xtraiter, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %sub.epil = sub i32 %j, %i.011.epil
+  %arrayidx.epil = getelementptr inbounds i16, i16* %pIn2, i32 %sub.epil
+  %2 = load i16, i16* %arrayidx.epil, align 2
+  %conv.epil = sext i16 %2 to i32
+  %arrayidx1.epil = getelementptr inbounds i16, i16* %pIn1, i32 %i.011.epil
+  %3 = load i16, i16* %arrayidx1.epil, align 2
+  %conv2.epil = sext i16 %3 to i32
+  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
+  %sext.mul.epil = sext i32 %mul.epil to i64
+  %add.epil = add nsw i64 %sext.mul.epil, %sum.010.epil
+  %inc.epil = add nuw i32 %i.011.epil, 1
+  %epil.iter.sub = add i32 %epil.iter, -1
+  %epil.iter.cmp = icmp eq i32 %epil.iter.sub, 0
+  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
+
+for.cond.cleanup:
+  %sum.0.lcssa = phi i64 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
+  ret i64 %sum.0.lcssa
+
+for.body:
+  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
+  %sum.010 = phi i64 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
+  %niter = phi i32 [ %unroll_iter, %for.body.preheader.new ], [ %niter.nsub.3, %for.body ]
+  %pIn2Base = phi i16* [ %pIn2, %for.body.preheader.new ], [ %pIn2.4, %for.body ]
+  %pIn2.0 = getelementptr inbounds i16, i16* %pIn2Base, i32 0
+  %In2 = load i16, i16* %pIn2.0, align 2
+  %pIn1.0 = getelementptr inbounds i16, i16* %pIn1, i32 %i.011
+  %In1 = load i16, i16* %pIn1.0, align 2
+  %inc = or i32 %i.011, 1
+  %pIn2.1 = getelementptr inbounds i16, i16* %pIn2Base, i32 -1
+  %In2.1 = load i16, i16* %pIn2.1, align 2
+  %pIn1.1 = getelementptr inbounds i16, i16* %pIn1, i32 %inc
+  %In1.1 = load i16, i16* %pIn1.1, align 2
+  %inc.1 = or i32 %i.011, 2
+  %pIn2.2 = getelementptr inbounds i16, i16* %pIn2Base, i32 -2
+  %In2.2 = load i16, i16* %pIn2.2, align 2
+  %pIn1.2 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.1
+  %In1.2 = load i16, i16* %pIn1.2, align 2
+  %inc.2 = or i32 %i.011, 3
+  %pIn2.3 = getelementptr inbounds i16, i16* %pIn2Base, i32 -3
+  %In2.3 = load i16, i16* %pIn2.3, align 2
+  %pIn1.3 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.2
+  %In1.3 = load i16, i16* %pIn1.3, align 2
+  %sextIn1 = sext i16 %In1 to i32
+  %sextIn1.1 = sext i16 %In1.1 to i32
+  %sextIn1.2 = sext i16 %In1.2 to i32
+  %sextIn1.3 = sext i16 %In1.3 to i32
+  %sextIn2 = sext i16 %In2 to i32
+  %sextIn2.1 = sext i16 %In2.1 to i32
+  %sextIn2.2 = sext i16 %In2.2 to i32
+  %sextIn2.3 = sext i16 %In2.3 to i32
+  %mul = mul nsw i32 %sextIn1, %sextIn2
+  %mul.1 = mul nsw i32 %sextIn1.1, %sextIn2.1
+  %mul.2 = mul nsw i32 %sextIn1.2, %sextIn2.2
+  %mul.3 = mul nsw i32 %sextIn1.3, %sextIn2.3
+  %sext.mul = sext i32 %mul to i64
+  %sext.mul.1 = sext i32 %mul.1 to i64
+  %sext.mul.2 = sext i32 %mul.2 to i64
+  %sext.mul.3 = sext i32 %mul.3 to i64
+  %add = add nsw i64 %sext.mul, %sum.010
+  %add.1 = add nsw i64 %sext.mul.1, %add
+  %add.2 = add nsw i64 %sext.mul.2, %add.1
+  %add.3 = add nsw i64 %sext.mul.3, %add.2
+  %inc.3 = add i32 %i.011, 4
+  %pIn2.4 = getelementptr inbounds i16, i16* %pIn2Base, i32 -4
+  %niter.nsub.3 = add i32 %niter, -4
+  %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
+  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
+}
+
+define i64 @smlaldx_swap(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
+
+entry:
+  %cmp9 = icmp eq i32 %limit, 0
+  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
+
+for.body.preheader:
+  %0 = add i32 %limit, -1
+  %xtraiter = and i32 %limit, 3
+  %1 = icmp ult i32 %0, 3
+  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
+
+for.body.preheader.new:
+  %unroll_iter = sub i32 %limit, %xtraiter
+  %scevgep6 = getelementptr i16, i16* %pIn1, i32 2
+  %2 = add i32 %j, -1
+  %scevgep11 = getelementptr i16, i16* %pIn2, i32 %2
+  br label %for.body
+
+for.cond.cleanup.loopexit.unr-lcssa:
+  %add.lcssa.ph = phi i64 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
+  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
+  %sum.010.unr = phi i64 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
+  %lcmp.mod = icmp eq i32 %xtraiter, 0
+  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil.preheader
+
+for.body.epil.preheader:
+  %scevgep = getelementptr i16, i16* %pIn1, i32 %i.011.unr
+  %3 = sub i32 %j, %i.011.unr
+  %scevgep2 = getelementptr i16, i16* %pIn2, i32 %3
+  %4 = sub i32 0, %xtraiter
+  br label %for.body.epil
+
+for.body.epil:
+  %lsr.iv5 = phi i32 [ %4, %for.body.epil.preheader ], [ %lsr.iv.next, %for.body.epil ]
+  %lsr.iv3 = phi i16* [ %scevgep2, %for.body.epil.preheader ], [ %scevgep4, %for.body.epil ]
+  %lsr.iv = phi i16* [ %scevgep, %for.body.epil.preheader ], [ %scevgep1, %for.body.epil ]
+  %sum.010.epil = phi i64 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.body.epil.preheader ]
+  %5 = load i16, i16* %lsr.iv3, align 2
+  %conv.epil = sext i16 %5 to i32
+  %6 = load i16, i16* %lsr.iv, align 2
+  %conv2.epil = sext i16 %6 to i32
+  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
+  %sext.mul.epil = sext i32 %mul.epil to i64
+  %add.epil = add nsw i64 %sext.mul.epil, %sum.010.epil
+  %scevgep1 = getelementptr i16, i16* %lsr.iv, i32 1
+  %scevgep4 = getelementptr i16, i16* %lsr.iv3, i32 -1
+  %lsr.iv.next = add nsw i32 %lsr.iv5, 1
+  %epil.iter.cmp = icmp eq i32 %lsr.iv.next, 0
+  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
+
+for.cond.cleanup:
+  %sum.0.lcssa = phi i64 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
+  ret i64 %sum.0.lcssa
+
+; CHECK-LABEL: smlaldx_swap
+; CHECK: for.body.preheader.new:
+; CHECK: [[PIN1Base:[^ ]+]] = getelementptr i16, i16* %pIn1
+; CHECK: [[PIN2Base:[^ ]+]] = getelementptr i16, i16* %pIn2
+
+; CHECK: for.body:
+; CHECK: [[PIN2:%[^ ]+]] = phi i16* [ [[PIN2_NEXT:%[^ ]+]], %for.body ], [ [[PIN2Base]], %for.body.preheader.new ]
+; CHECK: [[PIN1:%[^ ]+]] = phi i16* [ [[PIN1_NEXT:%[^ ]+]], %for.body ], [ [[PIN1Base]], %for.body.preheader.new ]
+; CHECK: [[IV:%[^ ]+]] = phi i32
+; CHECK: [[ACC0:%[^ ]+]] = phi i64 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
+
+; CHECK: [[PIN1_2:%[^ ]+]] = getelementptr i16, i16* [[PIN1]], i32 -2
+; CHECK: [[PIN2_2:%[^ ]+]] = getelementptr i16, i16* [[PIN2]], i32 -2
+
+; CHECK: [[PIN2_2_CAST:%[^ ]+]] = bitcast i16* [[PIN2_2]] to i32*
+; CHECK: [[IN2_2:%[^ ]+]] = load i32, i32* [[PIN2_2_CAST]], align 2
+; CHECK: [[PIN1_CAST:%[^ ]+]] = bitcast i16* [[PIN1]] to i32*
+; CHECK: [[IN1:%[^ ]+]] = load i32, i32* [[PIN1_CAST]], align 2
+; CHECK: [[ACC1:%[^ ]+]] = call i64 @llvm.arm.smlaldx(i32 [[IN2_2]], i32 [[IN1]], i64 [[ACC0]])
+
+; CHECK: [[PIN2_CAST:%[^ ]+]] = bitcast i16* [[PIN2]] to i32*
+; CHECK: [[IN2:%[^ ]+]] = load i32, i32* [[PIN2_CAST]], align 2
+; CHECK: [[PIN1_2_CAST:%[^ ]+]] = bitcast i16* [[PIN1_2]] to i32*
+; CHECK: [[IN1_2:%[^ ]+]] = load i32, i32* [[PIN1_2_CAST]], align 2
+; CHECK: [[ACC2]] = call i64 @llvm.arm.smlaldx(i32 [[IN2]], i32 [[IN1_2]], i64 [[ACC1]])
+
+; CHECK: [[PIN1_NEXT]] = getelementptr i16, i16* [[PIN1]], i32 4
+; CHECK: [[PIN2_NEXT]] = getelementptr i16, i16* [[PIN2]], i32 -4
+
+; CHECK-NOT: call i64 @llvm.arm.smlad
+; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlad
+
+for.body:
+  %pin2 = phi i16* [ %pin2.sub4, %for.body ], [ %scevgep11, %for.body.preheader.new ]
+  %pin1 = phi i16* [ %pin1.add4, %for.body ], [ %scevgep6, %for.body.preheader.new ]
+  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
+  %sum.010 = phi i64 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
+  %pin2.add1 = getelementptr i16, i16* %pin2, i32 1
+  %In2 = load i16, i16* %pin2.add1, align 2
+  %pin1.sub2 = getelementptr i16, i16* %pin1, i32 -2
+  %In1 = load i16, i16* %pin1.sub2, align 2
+  %In2.1 = load i16, i16* %pin2, align 2
+  %pin1.sub1 = getelementptr i16, i16* %pin1, i32 -1
+  %In1.1 = load i16, i16* %pin1.sub1, align 2
+  %pin2.sub1 = getelementptr i16, i16* %pin2, i32 -1
+  %In2.2 = load i16, i16* %pin2.sub1, align 2
+  %In1.2 = load i16, i16* %pin1, align 2
+  %pin2.sub2 = getelementptr i16, i16* %pin2, i32 -2
+  %In2.3 = load i16, i16* %pin2.sub2, align 2
+  %pin1.add1 = getelementptr i16, i16* %pin1, i32 1
+  %In1.3 = load i16, i16* %pin1.add1, align 2
+  %sextIn2 = sext i16 %In2 to i32
+  %sextIn1 = sext i16 %In1 to i32
+  %sextIn2.1 = sext i16 %In2.1 to i32
+  %sextIn1.1 = sext i16 %In1.1 to i32
+  %sextIn2.2 = sext i16 %In2.2 to i32
+  %sextIn1.2 = sext i16 %In1.2 to i32
+  %sextIn2.3 = sext i16 %In2.3 to i32
+  %sextIn1.3 = sext i16 %In1.3 to i32
+  %mul = mul nsw i32 %sextIn2, %sextIn1
+  %sext.mul = sext i32 %mul to i64
+  %add = add nsw i64 %sext.mul, %sum.010
+  %mul.1 = mul nsw i32 %sextIn2.1, %sextIn1.1
+  %sext.mul.1 = sext i32 %mul.1 to i64
+  %add.1 = add nsw i64 %sext.mul.1, %add
+  %mul.2 = mul nsw i32 %sextIn2.2, %sextIn1.2
+  %sext.mul.2 = sext i32 %mul.2 to i64
+  %add.2 = add nsw i64 %sext.mul.2, %add.1
+  %mul.3 = mul nsw i32 %sextIn2.3, %sextIn1.3
+  %sext.mul.3 = sext i32 %mul.3 to i64
+  %add.3 = add nsw i64 %sext.mul.3, %add.2
+  %inc.3 = add i32 %i.011, 4
+  %pin1.add4 = getelementptr i16, i16* %pin1, i32 4
+  %pin2.sub4 = getelementptr i16, i16* %pin2, i32 -4
+  %niter.ncmp.3 = icmp eq i32 %unroll_iter, %inc.3
+  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/smlaldx-2.ll

+; RUN: opt -mtriple=thumbv8m.main -mcpu=cortex-m33 -arm-parallel-dsp %s -S -o - | FileCheck %s
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
+
+define i64 @smlaldx(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
+
+; CHECK-LABEL: smlaldx
+; CHECK: = phi i32 [ 0, %for.body.preheader.new ],
+; CHECK: [[ACC0:%[^ ]+]] = phi i64 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
+; CHECK: [[PIN23:%[^ ]+]] = bitcast i16* %pIn2.3 to i32*
+; CHECK: [[IN23:%[^ ]+]] = load i32, i32* [[PIN23]], align 2
+; CHECK: [[PIN12:%[^ ]+]] = bitcast i16* %pIn1.2 to i32*
+; CHECK: [[IN12:%[^ ]+]] = load i32, i32* [[PIN12]], align 2
+; CHECK: [[ACC1:%[^ ]+]] = call i64 @llvm.arm.smlaldx(i32 [[IN23]], i32 [[IN12]], i64 [[ACC0]])
+; CHECK: [[PIN21:%[^ ]+]] = bitcast i16* %pIn2.1 to i32*
+; CHECK: [[IN21:%[^ ]+]] = load i32, i32* [[PIN21]], align 2
+; CHECK: [[PIN10:%[^ ]+]] = bitcast i16* %pIn1.0 to i32*
+; CHECK: [[IN10:%[^ ]+]] = load i32, i32* [[PIN10]], align 2
+; CHECK: [[ACC2]] = call i64 @llvm.arm.smlaldx(i32 [[IN21]], i32 [[IN10]], i64 [[ACC1]])
+; CHECK-NOT: call i64 @llvm.arm.smlad
+; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlad
+
+entry:
+  %cmp9 = icmp eq i32 %limit, 0
+  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
+
+for.body.preheader:
+  %0 = add i32 %limit, -1
+  %xtraiter = and i32 %limit, 3
+  %1 = icmp ult i32 %0, 3
+  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
+
+for.body.preheader.new:
+  %unroll_iter = sub i32 %limit, %xtraiter
+  br label %for.body
+
+for.cond.cleanup.loopexit.unr-lcssa:
+  %add.lcssa.ph = phi i64 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
+  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
+  %sum.010.unr = phi i64 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
+  %lcmp.mod = icmp eq i32 %xtraiter, 0
+  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil
+
+for.body.epil:
+  %i.011.epil = phi i32 [ %inc.epil, %for.body.epil ], [ %i.011.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %sum.010.epil = phi i64 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %epil.iter = phi i32 [ %epil.iter.sub, %for.body.epil ], [ %xtraiter, %for.cond.cleanup.loopexit.unr-lcssa ]
+  %sub.epil = sub i32 %j, %i.011.epil
+  %arrayidx.epil = getelementptr inbounds i16, i16* %pIn2, i32 %sub.epil
+  %2 = load i16, i16* %arrayidx.epil, align 2
+  %conv.epil = sext i16 %2 to i32
+  %arrayidx1.epil = getelementptr inbounds i16, i16* %pIn1, i32 %i.011.epil
+  %3 = load i16, i16* %arrayidx1.epil, align 2
+  %conv2.epil = sext i16 %3 to i32
+  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
+  %sext.mul.epil = sext i32 %mul.epil to i64
+  %add.epil = add nsw i64 %sext.mul.epil, %sum.010.epil
+  %inc.epil = add nuw i32 %i.011.epil, 1
+  %epil.iter.sub = add i32 %epil.iter, -1
+  %epil.iter.cmp = icmp eq i32 %epil.iter.sub, 0
+  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
+
+for.cond.cleanup:
+  %sum.0.lcssa = phi i64 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
+  ret i64 %sum.0.lcssa
+
+for.body:
+  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
+  %sum.010 = phi i64 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
+  %niter = phi i32 [ %unroll_iter, %for.body.preheader.new ], [ %niter.nsub.3, %for.body ]
+  %pIn2Base = phi i16* [ %pIn2, %for.body.preheader.new ], [ %pIn2.4, %for.body ]
+  %pIn2.0 = getelementptr inbounds i16, i16* %pIn2Base, i32 0
+  %In2 = load i16, i16* %pIn2.0, align 2
+  %pIn1.0 = getelementptr inbounds i16, i16* %pIn1, i32 %i.011
+  %In1 = load i16, i16* %pIn1.0, align 2
+  %inc = or i32 %i.011, 1
+  %pIn2.1 = getelementptr inbounds i16, i16* %pIn2Base, i32 -1
+  %In2.1 = load i16, i16* %pIn2.1, align 2
+  %pIn1.1 = getelementptr inbounds i16, i16* %pIn1, i32 %inc
+  %In1.1 = load i16, i16* %pIn1.1, align 2
+  %inc.1 = or i32 %i.011, 2
+  %pIn2.2 = getelementptr inbounds i16, i16* %pIn2Base, i32 -2
+  %In2.2 = load i16, i16* %pIn2.2, align 2
+  %pIn1.2 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.1
+  %In1.2 = load i16, i16* %pIn1.2, align 2
+  %inc.2 = or i32 %i.011, 3
+  %pIn2.3 = getelementptr inbounds i16, i16* %pIn2Base, i32 -3
+  %In2.3 = load i16, i16* %pIn2.3, align 2
+  %pIn1.3 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.2
+  %In1.3 = load i16, i16* %pIn1.3, align 2
+  %sextIn1 = sext i16 %In1 to i32
+  %sextIn1.1 = sext i16 %In1.1 to i32
+  %sextIn1.2 = sext i16 %In1.2 to i32
+  %sextIn1.3 = sext i16 %In1.3 to i32
+  %sextIn2 = sext i16 %In2 to i32
+  %sextIn2.1 = sext i16 %In2.1 to i32
+  %sextIn2.2 = sext i16 %In2.2 to i32
+  %sextIn2.3 = sext i16 %In2.3 to i32
+  %mul = mul nsw i32 %sextIn1, %sextIn2
+  %mul.1 = mul nsw i32 %sextIn1.1, %sextIn2.1
+  %mul.2 = mul nsw i32 %sextIn1.2, %sextIn2.2
+  %mul.3 = mul nsw i32 %sextIn1.3, %sextIn2.3
+  %sext.mul = sext i32 %mul to i64
+  %sext.mul.1 = sext i32 %mul.1 to i64
+  %sext.mul.2 = sext i32 %mul.2 to i64
+  %sext.mul.3 = sext i32 %mul.3 to i64
+  %add = add nsw i64 %sum.010, %sext.mul
+  %add.1 = add nsw i64 %sext.mul.1, %add
+  %add.2 = add nsw i64 %add.1, %sext.mul.2
+  %add.3 = add nsw i64 %sext.mul.3, %add.2
+  %inc.3 = add i32 %i.011, 4
+  %pIn2.4 = getelementptr inbounds i16, i16* %pIn2Base, i32 -4
+  %niter.nsub.3 = add i32 %niter, -4
+  %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
+  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
+}
+
+define i64 @smlaldx_swap(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
+
+entry:
+  %cmp9 = icmp eq i32 %limit, 0
+  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
+
+for.body.preheader:
+  %0 = add i32 %limit, -1
+  %xtraiter = and i32 %limit, 3
+  %1 = icmp ult i32 %0, 3
+  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
+
+for.body.preheader.new:
+  %unroll_iter = sub i32 %limit, %xtraiter
+  %scevgep6 = getelementptr i16, i16* %pIn1, i32 2
+  %2 = add i32 %j, -1
+  %scevgep11 = getelementptr i16, i16* %pIn2, i32 %2
+  br label %for.body
+
+for.cond.cleanup.loopexit.unr-lcssa:
+  %add.lcssa.ph = phi i64 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
+  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
+  %sum.010.unr = phi i64 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
+  %lcmp.mod = icmp eq i32 %xtraiter, 0
+  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil.preheader
+
+for.body.epil.preheader:
+  %scevgep = getelementptr i16, i16* %pIn1, i32 %i.011.unr
+  %3 = sub i32 %j, %i.011.unr
+  %scevgep2 = getelementptr i16, i16* %pIn2, i32 %3
+  %4 = sub i32 0, %xtraiter
+  br label %for.body.epil
+
+for.body.epil:
+  %lsr.iv5 = phi i32 [ %4, %for.body.epil.preheader ], [ %lsr.iv.next, %for.body.epil ]
+  %lsr.iv3 = phi i16* [ %scevgep2, %for.body.epil.preheader ], [ %scevgep4, %for.body.epil ]
+  %lsr.iv = phi i16* [ %scevgep, %for.body.epil.preheader ], [ %scevgep1, %for.body.epil ]
+  %sum.010.epil = phi i64 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.body.epil.preheader ]
+  %5 = load i16, i16* %lsr.iv3, align 2
+  %conv.epil = sext i16 %5 to i32
+  %6 = load i16, i16* %lsr.iv, align 2
+  %conv2.epil = sext i16 %6 to i32
+  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
+  %sext.mul.epil = sext i32 %mul.epil to i64
+  %add.epil = add nsw i64 %sext.mul.epil, %sum.010.epil
+  %scevgep1 = getelementptr i16, i16* %lsr.iv, i32 1
+  %scevgep4 = getelementptr i16, i16* %lsr.iv3, i32 -1
+  %lsr.iv.next = add nsw i32 %lsr.iv5, 1
+  %epil.iter.cmp = icmp eq i32 %lsr.iv.next, 0
+  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
+
+for.cond.cleanup:
+  %sum.0.lcssa = phi i64 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
+  ret i64 %sum.0.lcssa
+
+; CHECK-LABEL: smlaldx_swap
+; CHECK: for.body.preheader.new:
+; CHECK: [[PIN1Base:[^ ]+]] = getelementptr i16, i16* %pIn1
+; CHECK: [[PIN2Base:[^ ]+]] = getelementptr i16, i16* %pIn2
+
+; CHECK: for.body:
+; CHECK: [[PIN2:%[^ ]+]] = phi i16* [ [[PIN2_NEXT:%[^ ]+]], %for.body ], [ [[PIN2Base]], %for.body.preheader.new ]
+; CHECK: [[PIN1:%[^ ]+]] = phi i16* [ [[PIN1_NEXT:%[^ ]+]], %for.body ], [ [[PIN1Base]], %for.body.preheader.new ]
+; CHECK: [[IV:%[^ ]+]] = phi i32
+; CHECK: [[ACC0:%[^ ]+]] = phi i64 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
+; CHECK: [[PIN1_2:%[^ ]+]] = getelementptr i16, i16* [[PIN1]], i32 -2
+; CHECK: [[PIN2_2:%[^ ]+]] = getelementptr i16, i16* [[PIN2]], i32 -2
+
+; CHECK: [[PIN2_CAST:%[^ ]+]] = bitcast i16* [[PIN2]] to i32*
+; CHECK: [[IN2:%[^ ]+]] = load i32, i32* [[PIN2_CAST]], align 2
+; CHECK: [[PIN1_2_CAST:%[^ ]+]] = bitcast i16* [[PIN1_2]] to i32*
+; CHECK: [[IN1_2:%[^ ]+]] = load i32, i32* [[PIN1_2_CAST]], align 2
+; CHECK: [[ACC1:%[^ ]+]] = call i64 @llvm.arm.smlaldx(i32 [[IN2]], i32 [[IN1_2]], i64 [[ACC0]])
+
+; CHECK: [[PIN1_CAST:%[^ ]+]] = bitcast i16* [[PIN1]] to i32*
+; CHECK: [[IN1:%[^ ]+]] = load i32, i32* [[PIN1_CAST]], align 2
+; CHECK: [[PIN2_2_CAST:%[^ ]+]] = bitcast i16* [[PIN2_2]] to i32*
+; CHECK: [[IN2_2:%[^ ]+]] = load i32, i32* [[PIN2_2_CAST]], align 2
+; CHECK: [[ACC2]] = call i64 @llvm.arm.smlaldx(i32 [[IN1]], i32 [[IN2_2]], i64 [[ACC1]])
+
+; CHECK: [[PIN1_NEXT]] = getelementptr i16, i16* [[PIN1]], i32 4
+; CHECK: [[PIN2_NEXT]] = getelementptr i16, i16* [[PIN2]], i32 -4
+
+; CHECK-NOT: call i64 @llvm.arm.smlad
+; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlad
+
+for.body:
+  %pin2 = phi i16* [ %pin2.sub4, %for.body ], [ %scevgep11, %for.body.preheader.new ]
+  %pin1 = phi i16* [ %pin1.add4, %for.body ], [ %scevgep6, %for.body.preheader.new ]
+  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
+  %sum.010 = phi i64 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
+  %pin2.add1 = getelementptr i16, i16* %pin2, i32 1
+  %In2 = load i16, i16* %pin2.add1, align 2
+  %pin1.sub2 = getelementptr i16, i16* %pin1, i32 -2
+  %In1 = load i16, i16* %pin1.sub2, align 2
+  %In2.1 = load i16, i16* %pin2, align 2
+  %pin1.sub1 = getelementptr i16, i16* %pin1, i32 -1
+  %In1.1 = load i16, i16* %pin1.sub1, align 2
+  %pin2.sub1 = getelementptr i16, i16* %pin2, i32 -1
+  %In2.2 = load i16, i16* %pin2.sub1, align 2
+  %In1.2 = load i16, i16* %pin1, align 2
+  %pin2.sub2 = getelementptr i16, i16* %pin2, i32 -2
+  %In2.3 = load i16, i16* %pin2.sub2, align 2
+  %pin1.add1 = getelementptr i16, i16* %pin1, i32 1
+  %In1.3 = load i16, i16* %pin1.add1, align 2
+  %sextIn2 = sext i16 %In2 to i32
+  %sextIn1 = sext i16 %In1 to i32
+  %sextIn2.1 = sext i16 %In2.1 to i32
+  %sextIn1.1 = sext i16 %In1.1 to i32
+  %sextIn2.2 = sext i16 %In2.2 to i32
+  %sextIn1.2 = sext i16 %In1.2 to i32
+  %sextIn2.3 = sext i16 %In2.3 to i32
+  %sextIn1.3 = sext i16 %In1.3 to i32
+  %mul = mul nsw i32 %sextIn2, %sextIn1
+  %sext.mul = sext i32 %mul to i64
+  %add = add nsw i64 %sext.mul, %sum.010
+  %mul.1 = mul nsw i32 %sextIn2.1, %sextIn1.1
+  %sext.mul.1 = sext i32 %mul.1 to i64
+  %add.1 = add nsw i64 %sext.mul.1, %add
+  %mul.2 = mul nsw i32 %sextIn2.2, %sextIn1.2
+  %sext.mul.2 = sext i32 %mul.2 to i64
+  %add.2 = add nsw i64 %add.1, %sext.mul.2
+  %mul.3 = mul nsw i32 %sextIn2.3, %sextIn1.3
+  %sext.mul.3 = sext i32 %mul.3 to i64
+  %add.3 = add nsw i64 %add.2, %sext.mul.3
+  %inc.3 = add i32 %i.011, 4
+  %pin1.add4 = getelementptr i16, i16* %pin1, i32 4
+  %pin2.sub4 = getelementptr i16, i16* %pin2, i32 -4
+  %niter.ncmp.3 = icmp eq i32 %unroll_iter, %inc.3
+  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/top-bottom-multi-use.ll

+; RUN: opt -mtriple=thumbv8m.main -mcpu=cortex-m33 -S -arm-parallel-dsp %s -o - | FileCheck %s
+; RUN: opt -mtriple=thumbv7a-linux-android -arm-parallel-dsp -S %s -o - | FileCheck %s
+
+; CHECK-LABEL: sext_multi_use_undef
+define void @sext_multi_use_undef() {
+entry:
+  br label %for.body
+
+for.body:
+  %0 = load i16, i16* undef, align 2
+  %conv3 = sext i16 %0 to i32
+  %1 = load i16, i16* undef, align 2
+  %conv7 = sext i16 %1 to i32
+  %mul8 = mul nsw i32 %conv7, %conv3
+  %x.addr.180 = getelementptr inbounds i16, i16* undef, i32 1
+  %2 = load i16, i16* %x.addr.180, align 2
+  %conv1582 = sext i16 %2 to i32
+  %mul.i7284 = mul nsw i32 %conv7, %conv1582
+  br label %for.body
+}
+
+; CHECK-LABEL: sext_multi_use
+; CHECK: [[PtrA:%[^ ]+]] = bitcast i16* %a to i32*
+; CHECK: [[DataA:%[^ ]+]] = load i32, i32* [[PtrA]], align 2
+; CHECK: [[Top:%[^ ]+]] = ashr i32 [[DataA]], 16
+; CHECK: [[Shl:%[^ ]+]] = shl i32 [[DataA]], 16
+; CHECK: [[Bottom:%[^ ]+]] = ashr i32 [[Shl]], 16
+; CHECK: [[DataB:%[^ ]+]] = load i16, i16* %b, align 2
+; CHECK: [[SextB:%[^ ]+]] = sext i16 [[DataB]] to i32
+; CHECK: [[Mul0:%[^ ]+]] = mul nsw i32 [[SextB]], [[Bottom]]
+; CHECK: [[Mul1:%[^ ]+]] = mul nsw i32 [[SextB]], [[Top]]
+define void @sext_multi_use(i16* %a, i16* %b) {
+entry:
+  br label %for.body
+
+for.body:
+  %0 = load i16, i16* %a, align 2
+  %conv3 = sext i16 %0 to i32
+  %1 = load i16, i16* %b, align 2
+  %conv7 = sext i16 %1 to i32
+  %mul8 = mul nsw i32 %conv7, %conv3
+  %x.addr.180 = getelementptr inbounds i16, i16* %a, i32 1
+  %2 = load i16, i16* %x.addr.180, align 2
+  %conv1582 = sext i16 %2 to i32
+  %mul.i7284 = mul nsw i32 %conv7, %conv1582
+  br label %for.body
+}
+
+; CHECK-LABEL: sext_multi_use_reorder
+; CHECK: [[PtrA:%[^ ]+]] = bitcast i16* %a to i32*
+; CHECK: [[DataA:%[^ ]+]] = load i32, i32* [[PtrA]], align 2
+; CHECK: [[Top:%[^ ]+]] = ashr i32 [[DataA]], 16
+; CHECK: [[Shl:%[^ ]+]] = shl i32 [[DataA]], 16
+; CHECK: [[Bottom:%[^ ]+]] = ashr i32 [[Shl]], 16
+; CHECK: [[Mul0:%[^ ]+]] = mul nsw i32 [[Top]], [[Bottom]]
+; CHECK: [[DataB:%[^ ]+]] = load i16, i16* %b, align 2
+; CHECK: [[SextB:%[^ ]+]] = sext i16 [[DataB]] to i32
+; CHECK: [[Mul1:%[^ ]+]] = mul nsw i32 [[Top]], [[SextB]]
+define void @sext_multi_use_reorder(i16* %a, i16* %b) {
+entry:
+  br label %for.body
+
+for.body:
+  %0 = load i16, i16* %a, align 2
+  %conv3 = sext i16 %0 to i32
+  %x.addr.180 = getelementptr inbounds i16, i16* %a, i32 1
+  %1 = load i16, i16* %x.addr.180, align 2
+  %conv7 = sext i16 %1 to i32
+  %mul8 = mul nsw i32 %conv7, %conv3
+  %2 = load i16, i16* %b, align 2
+  %conv1582 = sext i16 %2 to i32
+  %mul.i7284 = mul nsw i32 %conv7, %conv1582
+  br label %for.body
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/top-bottom-neg-vec.ll

+; RUN: opt -mtriple=thumbv7-unknown-linux-android -arm-parallel-dsp -S %s -o - | FileCheck %s
+
+@a = local_unnamed_addr global i32 0, align 4
+@b = local_unnamed_addr global i8* null, align 4
+@c = local_unnamed_addr global i8 0, align 1
+@d = local_unnamed_addr global i16* null, align 4
+
+; CHECK-LABEL: @convolve
+; CHECK-NOT: bitcast i16* [[ANY:%[^ ]+]] to i32*
+define void @convolve() local_unnamed_addr #0 {
+entry:
+  br label %for.cond
+
+for.cond:
+  %e.0 = phi i32 [ undef, %entry ], [ %e.1.lcssa, %for.end ]
+  %f.0 = phi i32 [ undef, %entry ], [ %f.1.lcssa, %for.end ]
+  %g.0 = phi i32 [ undef, %entry ], [ %g.1.lcssa, %for.end ]
+  %cmp13 = icmp slt i32 %g.0, 1
+  br i1 %cmp13, label %for.body.lr.ph, label %for.end
+
+for.body.lr.ph:
+  %0 = load i16*, i16** @d, align 4
+  %1 = load i8*, i8** @b, align 4
+  %2 = load i32, i32* @a, align 4
+  %3 = sub i32 1, %g.0
+  %min.iters.check = icmp ugt i32 %3, 3
+  %ident.check = icmp eq i32 %2, 1
+  %or.cond = and i1 %min.iters.check, %ident.check
+  br i1 %or.cond, label %vector.ph, label %for.body.preheader
+
+vector.ph:
+  %n.vec = and i32 %3, -4
+  %ind.end = add i32 %g.0, %n.vec
+  %4 = mul i32 %2, %n.vec
+  %ind.end20 = add i32 %f.0, %4
+  %5 = insertelement <4 x i32> <i32 undef, i32 0, i32 0, i32 0>, i32 %e.0, i32 0
+  br label %vector.body
+
+vector.body:
+  %index = phi i32 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+  %vec.phi = phi <4 x i32> [ %5, %vector.ph ], [ %14, %vector.body ]
+  %offset.idx = add i32 %g.0, %index
+  %6 = mul i32 %2, %index
+  %offset.idx21 = add i32 %f.0, %6
+  %7 = getelementptr inbounds i16, i16* %0, i32 %offset.idx
+  %8 = bitcast i16* %7 to <4 x i16>*
+  %wide.load = load <4 x i16>, <4 x i16>* %8, align 2
+  %9 = sext <4 x i16> %wide.load to <4 x i32>
+  %10 = getelementptr inbounds i8, i8* %1, i32 %offset.idx21
+  %11 = bitcast i8* %10 to <4 x i8>*
+  %wide.load25 = load <4 x i8>, <4 x i8>* %11, align 1
+  %12 = zext <4 x i8> %wide.load25 to <4 x i32>
+  %13 = mul nsw <4 x i32> %12, %9
+  %14 = add nsw <4 x i32> %13, %vec.phi
+  %index.next = add i32 %index, 4
+  %15 = icmp eq i32 %index.next, %n.vec
+  br i1 %15, label %middle.block, label %vector.body
+
+middle.block:
+  %rdx.shuf = shufflevector <4 x i32> %14, <4 x i32> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef>
+  %bin.rdx = add <4 x i32> %14, %rdx.shuf
+  %rdx.shuf26 = shufflevector <4 x i32> %bin.rdx, <4 x i32> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
+  %bin.rdx27 = add <4 x i32> %bin.rdx, %rdx.shuf26
+  %16 = extractelement <4 x i32> %bin.rdx27, i32 0
+  %cmp.n = icmp eq i32 %3, %n.vec
+  br i1 %cmp.n, label %for.end, label %for.body.preheader
+
+for.body.preheader:
+  %g.116.ph = phi i32 [ %g.0, %for.body.lr.ph ], [ %ind.end, %middle.block ]
+  %f.115.ph = phi i32 [ %f.0, %for.body.lr.ph ], [ %ind.end20, %middle.block ]
+  %e.114.ph = phi i32 [ %e.0, %for.body.lr.ph ], [ %16, %middle.block ]
+  br label %for.body
+
+for.body:
+  %g.116 = phi i32 [ %inc, %for.body ], [ %g.116.ph, %for.body.preheader ]
+  %f.115 = phi i32 [ %add4, %for.body ], [ %f.115.ph, %for.body.preheader ]
+  %e.114 = phi i32 [ %add, %for.body ], [ %e.114.ph, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i16, i16* %0, i32 %g.116
+  %17 = load i16, i16* %arrayidx, align 2
+  %conv = sext i16 %17 to i32
+  %arrayidx2 = getelementptr inbounds i8, i8* %1, i32 %f.115
+  %18 = load i8, i8* %arrayidx2, align 1
+  %conv3 = zext i8 %18 to i32
+  %mul = mul nsw i32 %conv3, %conv
+  %add = add nsw i32 %mul, %e.114
+  %inc = add nsw i32 %g.116, 1
+  %add4 = add nsw i32 %2, %f.115
+  %cmp = icmp slt i32 %g.116, 0
+  br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+  %e.1.lcssa = phi i32 [ %e.0, %for.cond ], [ %16, %middle.block ], [ %add, %for.body ]
+  %f.1.lcssa = phi i32 [ %f.0, %for.cond ], [ %ind.end20, %middle.block ], [ %add4, %for.body ]
+  %g.1.lcssa = phi i32 [ %g.0, %for.cond ], [ %ind.end, %middle.block ], [ %inc, %for.body ]
+  %conv5 = trunc i32 %e.1.lcssa to i8
+  store i8 %conv5, i8* @c, align 1
+  br label %for.cond
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/top-bottom-neg.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+; RUN: opt -mtriple=thumbv7a-linux-android -arm-parallel-dsp -S %s -o - | FileCheck %s
+
+; CHECK-LABEL: topbottom_mul_alias
+; CHECK-NOT: bitcast i16*
+define void @topbottom_mul_alias(i32 %N, i32* nocapture readnone %Out, i16* nocapture readonly %In1, i16* nocapture readonly %In2) {
+entry:
+  br label %for.body
+
+for.body:
+  %iv = phi i32 [ 0, %entry ], [ %iv.next, %for.body ]
+  %count = phi i32 [ %N, %entry ], [ %count.next, %for.body ]
+  %PIn1.0 = getelementptr inbounds i16, i16* %In1, i32 %iv
+  %In1.0 = load i16, i16* %PIn1.0, align 2
+  %SIn1.0 = sext i16 %In1.0 to i32
+  %PIn2.0 = getelementptr inbounds i16, i16* %In2, i32 %iv
+  %In2.0 = load i16, i16* %PIn2.0, align 2
+  %SIn2.0 = sext i16 %In2.0 to i32
+  %mul5.us.i.i = mul nsw i32 %SIn1.0, %SIn2.0
+  %Out.0 = getelementptr inbounds i32, i32* %Out, i32 %iv
+  store i32 %mul5.us.i.i, i32* %Out.0, align 4
+  %iv.1 = or i32 %iv, 1
+  %PIn1.1 = getelementptr inbounds i16, i16* %In1, i32 %iv.1
+  %In1.1 = load i16, i16* %PIn1.1, align 2
+  %SIn1.1 = sext i16 %In1.1 to i32
+  %PIn2.1 = getelementptr inbounds i16, i16* %In2, i32 %iv.1
+  %In2.1 = load i16, i16* %PIn2.1, align 2
+  %SIn2.1 = sext i16 %In2.1 to i32
+  %mul5.us.i.1.i = mul nsw i32 %SIn1.1, %SIn2.1
+  %Out.1 = getelementptr inbounds i32, i32* %Out, i32 %iv.1
+  store i32 %mul5.us.i.1.i, i32* %Out.1, align 4
+  %iv.2 = or i32 %iv, 2
+  %PIn1.2 = getelementptr inbounds i16, i16* %In1, i32 %iv.2
+  %In1.2 = load i16, i16* %PIn1.2, align 2
+  %SIn1.2 = sext i16 %In1.2 to i32
+  %PIn2.2 = getelementptr inbounds i16, i16* %In2, i32 %iv.2
+  %In2.2 = load i16, i16* %PIn2.2, align 2
+  %SIn2.2 = sext i16 %In2.2 to i32
+  %mul5.us.i.2.i = mul nsw i32 %SIn1.2, %SIn2.2
+  %Out.2 = getelementptr inbounds i32, i32* %Out, i32 %iv.2
+  store i32 %mul5.us.i.2.i, i32* %Out.2, align 4
+  %iv.3 = or i32 %iv, 3
+  %PIn1.3 = getelementptr inbounds i16, i16* %In1, i32 %iv.3
+  %In1.3 = load i16, i16* %PIn1.3, align 2
+  %SIn1.3 = sext i16 %In1.3 to i32
+  %PIn2.3 = getelementptr inbounds i16, i16* %In2, i32 %iv.3
+  %In2.3 = load i16, i16* %PIn2.3, align 2
+  %SIn2.3 = sext i16 %In2.3 to i32
+  %mul5.us.i.3.i = mul nsw i32 %SIn1.3, %SIn2.3
+  %Out.3 = getelementptr inbounds i32, i32* %Out, i32 %iv.3
+  store i32 %mul5.us.i.3.i, i32* %Out.3, align 4
+  %iv.next = add i32 %iv, 4
+  %count.next = add i32 %count, -4
+  %niter375.ncmp.3.i = icmp eq i32 %count.next, 0
+  br i1 %niter375.ncmp.3.i, label %exit, label %for.body
+
+exit:
+  ret void
+}
+
+; TODO: We should be able to handle this by splatting the const value.
+; CHECK-LABEL: topbottom_mul_const
+; CHECK-NOT: bitcast i16*
+define void @topbottom_mul_const(i32 %N, i32* noalias nocapture readnone %Out, i16* nocapture readonly %In, i16 signext %const) {
+entry:
+  %conv4.i.i = sext i16 %const to i32
+  br label %for.body
+
+for.body:
+  %iv = phi i32 [ 0, %entry ], [ %iv.next, %for.body ]
+  %count = phi i32 [ %N, %entry ], [ %count.next, %for.body ]
+  %PIn.0 = getelementptr inbounds i16, i16* %In, i32 %iv
+  %In.0 = load i16, i16* %PIn.0, align 2
+  %conv.us.i144.i = sext i16 %In.0 to i32
+  %mul5.us.i.i = mul nsw i32 %conv.us.i144.i, %conv4.i.i
+  %Out.0 = getelementptr inbounds i32, i32* %Out, i32 %iv
+  store i32 %mul5.us.i.i, i32* %Out.0, align 4
+  %iv.1 = or i32 %iv, 1
+  %PIn.1 = getelementptr inbounds i16, i16* %In, i32 %iv.1
+  %In.1 = load i16, i16* %PIn.1, align 2
+  %conv.us.i144.1.i = sext i16 %In.1 to i32
+  %mul5.us.i.1.i = mul nsw i32 %conv.us.i144.1.i, %conv4.i.i
+  %Out.1 = getelementptr inbounds i32, i32* %Out, i32 %iv.1
+  store i32 %mul5.us.i.1.i, i32* %Out.1, align 4
+  %iv.2 = or i32 %iv, 2
+  %PIn.2 = getelementptr inbounds i16, i16* %In, i32 %iv.2
+  %In.3 = load i16, i16* %PIn.2, align 2
+  %conv.us.i144.2.i = sext i16 %In.3 to i32
+  %mul5.us.i.2.i = mul nsw i32 %conv.us.i144.2.i, %conv4.i.i
+  %Out.2 = getelementptr inbounds i32, i32* %Out, i32 %iv.2
+  store i32 %mul5.us.i.2.i, i32* %Out.2, align 4
+  %iv.3 = or i32 %iv, 3
+  %PIn.3 = getelementptr inbounds i16, i16* %In, i32 %iv.3
+  %In.4 = load i16, i16* %PIn.3, align 2
+  %conv.us.i144.3.i = sext i16 %In.4 to i32
+  %mul5.us.i.3.i = mul nsw i32 %conv.us.i144.3.i, %conv4.i.i
+  %Out.3 = getelementptr inbounds i32, i32* %Out, i32 %iv.3
+  store i32 %mul5.us.i.3.i, i32* %Out.3, align 4
+  %iv.next = add i32 %iv, 4
+  %count.next = add i32 %count, -4
+  %niter375.ncmp.3.i = icmp eq i32 %count.next, 0
+  br i1 %niter375.ncmp.3.i, label %exit, label %for.body
+
+exit:
+  ret void
+}
+
+; TODO: We should be able to handle this and use smulwt and smulwb.
+; CHECK-LABEL: topbottom_mul_word_load_const
+; CHECK-NOT: bitcast i16*
+define void @topbottom_mul_word_load_const(i32 %N, i32* noalias nocapture readnone %Out, i16* nocapture readonly %In, i32* %C) {
+entry:
+  %const = load i32, i32* %C
+  br label %for.body
+
+for.body:
+  %iv = phi i32 [ 0, %entry ], [ %iv.next, %for.body ]
+  %count = phi i32 [ %N, %entry ], [ %count.next, %for.body ]
+  %PIn.0 = getelementptr inbounds i16, i16* %In, i32 %iv
+  %In.0 = load i16, i16* %PIn.0, align 2
+  %conv.us.i144.i = sext i16 %In.0 to i32
+  %mul5.us.i.i = mul nsw i32 %conv.us.i144.i, %const
+  %Out.0 = getelementptr inbounds i32, i32* %Out, i32 %iv
+  store i32 %mul5.us.i.i, i32* %Out.0, align 4
+  %iv.1 = or i32 %iv, 1
+  %PIn.1 = getelementptr inbounds i16, i16* %In, i32 %iv.1
+  %In.1 = load i16, i16* %PIn.1, align 2
+  %conv.us.i144.1.i = sext i16 %In.1 to i32
+  %mul5.us.i.1.i = mul nsw i32 %conv.us.i144.1.i, %const
+  %Out.1 = getelementptr inbounds i32, i32* %Out, i32 %iv.1
+  store i32 %mul5.us.i.1.i, i32* %Out.1, align 4
+  %iv.2 = or i32 %iv, 2
+  %PIn.2 = getelementptr inbounds i16, i16* %In, i32 %iv.2
+  %In.3 = load i16, i16* %PIn.2, align 2
+  %conv.us.i144.2.i = sext i16 %In.3 to i32
+  %mul5.us.i.2.i = mul nsw i32 %conv.us.i144.2.i, %const
+  %Out.2 = getelementptr inbounds i32, i32* %Out, i32 %iv.2
+  store i32 %mul5.us.i.2.i, i32* %Out.2, align 4
+  %iv.3 = or i32 %iv, 3
+  %PIn.3 = getelementptr inbounds i16, i16* %In, i32 %iv.3
+  %In.4 = load i16, i16* %PIn.3, align 2
+  %conv.us.i144.3.i = sext i16 %In.4 to i32
+  %mul5.us.i.3.i = mul nsw i32 %conv.us.i144.3.i, %const
+  %Out.3 = getelementptr inbounds i32, i32* %Out, i32 %iv.3
+  store i32 %mul5.us.i.3.i, i32* %Out.3, align 4
+  %iv.next = add i32 %iv, 4
+  %count.next = add i32 %count, -4
+  %niter375.ncmp.3.i = icmp eq i32 %count.next, 0
+  br i1 %niter375.ncmp.3.i, label %exit, label %for.body
+
+exit:
+  ret void
+}
+
+; CHECK-LABEL: topbottom_mul_8
+; CHECK-NOT: bitcast i16*
+define void @topbottom_mul_8(i32 %N, i32* noalias nocapture readnone %Out, i8* nocapture readonly %In1, i8* nocapture readonly %In2) {
+entry:
+  br label %for.body
+
+for.body:
+  %iv = phi i32 [ 0, %entry ], [ %iv.next, %for.body ]
+  %count = phi i32 [ %N, %entry ], [ %count.next, %for.body ]
+  %PIn1.0 = getelementptr inbounds i8, i8* %In1, i32 %iv
+  %In1.0 = load i8, i8* %PIn1.0, align 1
+  %SIn1.0 = sext i8 %In1.0 to i32
+  %PIn2.0 = getelementptr inbounds i8, i8* %In2, i32 %iv
+  %In2.0 = load i8, i8* %PIn2.0, align 1
+  %SIn2.0 = sext i8 %In2.0 to i32
+  %mul5.us.i.i = mul nsw i32 %SIn1.0, %SIn2.0
+  %Out.0 = getelementptr inbounds i32, i32* %Out, i32 %iv
+  store i32 %mul5.us.i.i, i32* %Out.0, align 4
+  %iv.1 = or i32 %iv, 1
+  %PIn1.1 = getelementptr inbounds i8, i8* %In1, i32 %iv.1
+  %In1.1 = load i8, i8* %PIn1.1, align 1
+  %SIn1.1 = sext i8 %In1.1 to i32
+  %PIn2.1 = getelementptr inbounds i8, i8* %In2, i32 %iv.1
+  %In2.1 = load i8, i8* %PIn2.1, align 1
+  %SIn2.1 = sext i8 %In2.1 to i32
+  %mul5.us.i.1.i = mul nsw i32 %SIn1.1, %SIn2.1
+  %Out.1 = getelementptr inbounds i32, i32* %Out, i32 %iv.1
+  store i32 %mul5.us.i.1.i, i32* %Out.1, align 4
+  %iv.2 = or i32 %iv, 2
+  %PIn1.2 = getelementptr inbounds i8, i8* %In1, i32 %iv.2
+  %In1.2 = load i8, i8* %PIn1.2, align 1
+  %SIn1.2 = sext i8 %In1.2 to i32
+  %PIn2.2 = getelementptr inbounds i8, i8* %In2, i32 %iv.2
+  %In2.2 = load i8, i8* %PIn2.2, align 1
+  %SIn2.2 = sext i8 %In2.2 to i32
+  %mul5.us.i.2.i = mul nsw i32 %SIn1.2, %SIn2.2
+  %Out.2 = getelementptr inbounds i32, i32* %Out, i32 %iv.2
+  store i32 %mul5.us.i.2.i, i32* %Out.2, align 4
+  %iv.3 = or i32 %iv, 3
+  %PIn1.3 = getelementptr inbounds i8, i8* %In1, i32 %iv.3
+  %In1.3 = load i8, i8* %PIn1.3, align 1
+  %SIn1.3 = sext i8 %In1.3 to i32
+  %PIn2.3 = getelementptr inbounds i8, i8* %In2, i32 %iv.3
+  %In2.3 = load i8, i8* %PIn2.3, align 1
+  %SIn2.3 = sext i8 %In2.3 to i32
+  %mul5.us.i.3.i = mul nsw i32 %SIn1.3, %SIn2.3
+  %Out.3 = getelementptr inbounds i32, i32* %Out, i32 %iv.3
+  store i32 %mul5.us.i.3.i, i32* %Out.3, align 4
+  %iv.next = add i32 %iv, 4
+  %count.next = add i32 %count, -4
+  %niter375.ncmp.3.i = icmp eq i32 %count.next, 0
+  br i1 %niter375.ncmp.3.i, label %exit, label %for.body
+
+exit:
+  ret void
+}

llvm/trunk/test/CodeGen/ARM/ParallelDSP/top-bottom-order.ll

+; RUN: opt -mtriple=thumbv8m.main -mcpu=cortex-m33 -arm-parallel-dsp -S %s -o - | FileCheck %s
+; RUN: opt -mtriple=thumbv7a-linux-android -arm-parallel-dsp -S %s -o - | FileCheck %s
+
+; CHECK-LABEL: reorder_gep_arguments 
+; CHECK: [[Sub:%[^ ]+]] = xor i32 %iv, -1
+; CHECK: [[IdxPtr:%[^ ]+]] = getelementptr inbounds i16, i16* %arrayidx.us, i32 [[Sub]]
+; CHECK: [[IdxPtrCast:%[^ ]+]] = bitcast i16* [[IdxPtr]] to i32*
+; CHECK: [[Idx:%[^ ]+]] = load i32, i32* [[IdxPtrCast]], align 2
+; CHECK: [[Top:%[^ ]+]] = ashr i32 [[Idx]], 16
+; CHECK: [[Shl:%[^ ]+]] = shl i32 [[Idx]], 16
+; CHECK: [[Bottom:%[^ ]+]] = ashr i32 [[Shl]], 16
+; CHECK: [[BPtr:%[^ ]+]] = getelementptr inbounds i16, i16* %B, i32 %iv
+; CHECK: [[BData:%[^ ]+]] = load i16, i16* [[BPtr]], align 2
+; CHECK: [[BSext:%[^ ]+]] = sext i16 [[BData]] to i32
+; CHECK: [[Mul0:%[^ ]+]] = mul nsw i32 [[BSext]], [[Top]]
+; CHECK: [[BPtr1:%[^ ]+]] = getelementptr inbounds i16, i16* %B, i32 %add48.us
+; CHECK: [[BData1:%[^ ]+]] = load i16, i16* [[BPtr1]], align 2
+; CHECK: [[B1Sext:%[^ ]+]] = sext i16 [[BData1]] to i32
+; CHECK: [[Mul1:%[^ ]+]] = mul nsw i32 [[B1Sext]], [[Bottom]]
+
+define i32 @reorder_gep_arguments(i16* %B, i16* %arrayidx.us, i32 %d) {
+entry:
+  br label %for.body36.us
+
+for.body36.us:
+  %iv = phi i32 [ %add53.us, %for.body36.us ], [ 5, %entry ]
+  %out32_Q12.0114.us = phi i32 [ %add52.us, %for.body36.us ], [ 0, %entry ]
+  %sub37.us = sub nsw i32 0, %iv
+  %arrayidx38.us = getelementptr inbounds i16, i16* %arrayidx.us, i32 %sub37.us
+  %0 = load i16, i16* %arrayidx38.us, align 2
+  %conv39.us = sext i16 %0 to i32
+  %arrayidx40.us = getelementptr inbounds i16, i16* %B, i32 %iv
+  %1 = load i16, i16* %arrayidx40.us, align 2
+  %conv41.us = sext i16 %1 to i32
+  %mul42.us = mul nsw i32 %conv41.us, %conv39.us
+  %add43.us = add i32 %mul42.us, %out32_Q12.0114.us
+  %sub45.us = xor i32 %iv, -1
+  %arrayidx46.us = getelementptr inbounds i16, i16* %arrayidx.us, i32 %sub45.us
+  %2 = load i16, i16* %arrayidx46.us, align 2
+  %conv47.us = sext i16 %2 to i32
+  %add48.us = or i32 %iv, 1
+  %arrayidx49.us = getelementptr inbounds i16, i16* %B, i32 %add48.us
+  %3 = load i16, i16* %arrayidx49.us, align 2
+  %conv50.us = sext i16 %3 to i32
+  %mul51.us = mul nsw i32 %conv50.us, %conv47.us
+  %add52.us = add i32 %add43.us, %mul51.us
+  %add53.us = add nuw nsw i32 %iv, 2
+  %cmp34.us = icmp slt i32 %add53.us, %d
+  br i1 %cmp34.us, label %for.body36.us, label %exit
+
+exit:
+  ret i32 %add52.us
+}
+

llvm/trunk/test/CodeGen/ARM/ParallelDSP/top-bottom.ll

+; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
+; RUN: opt -mtriple=thumbv7a-linux-android -arm-parallel-dsp -S %s -o - | FileCheck %s
+
+; CHECK-LABEL: topbottom_mul
+define void @topbottom_mul(i32 %N, i32* noalias nocapture readnone %Out, i16* nocapture readonly %In1, i16* nocapture readonly %In2) {
+entry:
+  br label %for.body
+
+; CHECK: for.body:
+; CHECK: [[Cast_PIn1_0:%[^ ]+]] = bitcast i16* %PIn1.0 to i32*
+; CHECK: [[PIn1_01:%[^ ]+]] = load i32, i32* [[Cast_PIn1_0]], align 2
+; CHECK: [[PIn1_1:%[^ ]+]] = ashr i32 [[PIn1_01]], 16
+; CHECK: [[PIn1_01_shl:%[^ ]+]] = shl i32 [[PIn1_01]], 16
+; CHECK: [[PIn1_0:%[^ ]+]] = ashr i32 [[PIn1_01_shl]], 16
+
+; CHECK: [[Cast_PIn2_0:%[^ ]+]] = bitcast i16* %PIn2.0 to i32*
+; CHECK: [[PIn2_01:%[^ ]+]] = load i32, i32* [[Cast_PIn2_0]], align 2
+; CHECK: [[PIn2_1:%[^ ]+]] = ashr i32 [[PIn2_01]], 16
+; CHECK: [[PIn2_01_shl:%[^ ]+]] = shl i32 [[PIn2_01]], 16
+; CHECK: [[PIn2_0:%[^ ]+]] = ashr i32 [[PIn2_01_shl]], 16
+
+; CHECK: mul nsw i32 [[PIn1_0]], [[PIn2_0]]
+; CHECK: mul nsw i32 [[PIn1_1]], [[PIn2_1]]
+
+; CHECK: [[Cast_PIn1_2:%[^ ]+]] = bitcast i16* %PIn1.2 to i32*
+; CHECK: [[PIn1_23:%[^ ]+]] = load i32, i32* [[Cast_PIn1_2]], align 2
+; CHECK: [[PIn1_3:%[^ ]+]] = ashr i32 [[PIn1_23]], 16
+; CHECK: [[PIn1_23_shl:%[^ ]+]] = shl i32 [[PIn1_23]], 16
+; CHECK: [[PIn1_2:%[^ ]+]] = ashr i32 [[PIn1_23_shl]], 16
+
+; CHECK: [[Cast_PIn2_2:%[^ ]+]] = bitcast i16* %PIn2.2 to i32*
+; CHECK: [[PIn2_23:%[^ ]+]] = load i32, i32* [[Cast_PIn2_2]], align 2
+; CHECK: [[PIn2_3:%[^ ]+]] = ashr i32 [[PIn2_23]], 16
+; CHECK: [[PIn2_23_shl:%[^ ]+]] = shl i32 [[PIn2_23]], 16
+; CHECK: [[PIn2_2:%[^ ]+]] = ashr i32 [[PIn2_23_shl]], 16
+
+; CHECK: mul nsw i32 [[PIn1_2]], [[PIn2_2]]
+; CHECK: mul nsw i32 [[PIn1_3]], [[PIn2_3]]
+
+for.body:
+  %iv = phi i32 [ 0, %entry ], [ %iv.next, %for.body ]
+  %count = phi i32 [ %N, %entry ], [ %count.next, %for.body ]
+  %PIn1.0 = getelementptr inbounds i16, i16* %In1, i32 %iv
+  %In1.0 = load i16, i16* %PIn1.0, align 2
+  %SIn1.0 = sext i16 %In1.0 to i32
+  %PIn2.0 = getelementptr inbounds i16, i16* %In2, i32 %iv
+  %In2.0 = load i16, i16* %PIn2.0, align 2
+  %SIn2.0 = sext i16 %In2.0 to i32
+  %mul5.us.i.i = mul nsw i32 %SIn1.0, %SIn2.0
+  %Out.0 = getelementptr inbounds i32, i32* %Out, i32 %iv
+  store i32 %mul5.us.i.i, i32* %Out.0, align 4
+  %iv.1 = or i32 %iv, 1
+  %PIn1.1 = getelementptr inbounds i16, i16* %In1, i32 %iv.1
+  %In1.1 = load i16, i16* %PIn1.1, align 2
+  %SIn1.1 = sext i16 %In1.1 to i32
+  %PIn2.1 = getelementptr inbounds i16, i16* %In2, i32 %iv.1
+  %In2.1 = load i16, i16* %PIn2.1, align 2
+  %SIn2.1 = sext i16 %In2.1 to i32
+  %mul5.us.i.1.i = mul nsw i32 %SIn1.1, %SIn2.1
+  %Out.1 = getelementptr inbounds i32, i32* %Out, i32 %iv.1
+  store i32 %mul5.us.i.1.i, i32* %Out.1, align 4
+  %iv.2 = or i32 %iv, 2
+  %PIn1.2 = getelementptr inbounds i16, i16* %In1, i32 %iv.2
+  %In1.2 = load i16, i16* %PIn1.2, align 2
+  %SIn1.2 = sext i16 %In1.2 to i32
+  %PIn2.2 = getelementptr inbounds i16, i16* %In2, i32 %iv.2
+  %In2.2 = load i16, i16* %PIn2.2, align 2
+  %SIn2.2 = sext i16 %In2.2 to i32
+  %mul5.us.i.2.i = mul nsw i32 %SIn1.2, %SIn2.2
+  %Out.2 = getelementptr inbounds i32, i32* %Out, i32 %iv.2
+  store i32 %mul5.us.i.2.i, i32* %Out.2, align 4
+  %iv.3 = or i32 %iv, 3
+  %PIn1.3 = getelementptr inbounds i16, i16* %In1, i32 %iv.3
+  %In1.3 = load i16, i16* %PIn1.3, align 2
+  %SIn1.3 = sext i16 %In1.3 to i32
+  %PIn2.3 = getelementptr inbounds i16, i16* %In2, i32 %iv.3
+  %In2.3 = load i16, i16* %PIn2.3, align 2
+  %SIn2.3 = sext i16 %In2.3 to i32
+  %mul5.us.i.3.i = mul nsw i32 %SIn1.3, %SIn2.3
+  %Out.3 = getelementptr inbounds i32, i32* %Out, i32 %iv.3
+  store i32 %mul5.us.i.3.i, i32* %Out.3, align 4
+  %iv.next = add i32 %iv, 4
+  %count.next = add i32 %count, -4
+  %niter375.ncmp.3.i = icmp eq i32 %count.next, 0
+  br i1 %niter375.ncmp.3.i, label %exit, label %for.body
+
+exit:
+  ret void
+}
+
+; CHECK-LABEL: topbottom_mul_load_const
+define void @topbottom_mul_load_const(i32 %N, i32* noalias nocapture readnone %Out, i16* nocapture readonly %In, i16* %C) {
+entry:
+  %const = load i16, i16* %C
+  %conv4.i.i = sext i16 %const to i32
+  br label %for.body
+
+; CHECK: for.body:
+; CHECK: [[Cast_PIn_0:%[^ ]+]] = bitcast i16* %PIn.0 to i32*
+; CHECK: [[PIn_01:%[^ ]+]] = load i32, i32* [[Cast_PIn_0]], align 2
+; CHECK: [[PIn_1:%[^ ]+]] = ashr i32 [[PIn_01]], 16
+; CHECK: [[PIn_01_shl:%[^ ]+]] = shl i32 [[PIn_01]], 16
+; CHECK: [[PIn_0:%[^ ]+]] = ashr i32 [[PIn_01_shl]], 16
+
+; CHECK: mul nsw i32 [[PIn_0]], %conv4.i.i
+; CHECK: mul nsw i32 [[PIn_1]], %conv4.i.i
+
+; CHECK: [[Cast_PIn_2:%[^ ]+]] = bitcast i16* %PIn.2 to i32*
+; CHECK: [[PIn_23:%[^ ]+]] = load i32, i32* [[Cast_PIn_2]], align 2
+; CHECK: [[PIn_3:%[^ ]+]] = ashr i32 [[PIn_23]], 16
+; CHECK: [[PIn_23_shl:%[^ ]+]] = shl i32 [[PIn_23]], 16
+; CHECK: [[PIn_2:%[^ ]+]] = ashr i32 [[PIn_23_shl]], 16
+
+; CHECK: mul nsw i32 [[PIn_2]], %conv4.i.i
+; CHECK: mul nsw i32 [[PIn_3]], %conv4.i.i
+
+for.body:
+  %iv = phi i32 [ 0, %entry ], [ %iv.next, %for.body ]
+  %count = phi i32 [ %N, %entry ], [ %count.next, %for.body ]
+  %PIn.0 = getelementptr inbounds i16, i16* %In, i32 %iv
+  %In.0 = load i16, i16* %PIn.0, align 2
+  %conv.us.i144.i = sext i16 %In.0 to i32
+  %mul5.us.i.i = mul nsw i32 %conv.us.i144.i, %conv4.i.i
+  %Out.0 = getelementptr inbounds i32, i32* %Out, i32 %iv
+  store i32 %mul5.us.i.i, i32* %Out.0, align 4
+  %iv.1 = or i32 %iv, 1
+  %PIn.1 = getelementptr inbounds i16, i16* %In, i32 %iv.1
+  %In.1 = load i16, i16* %PIn.1, align 2
+  %conv.us.i144.1.i = sext i16 %In.1 to i32
+  %mul5.us.i.1.i = mul nsw i32 %conv.us.i144.1.i, %conv4.i.i
+  %Out.1 = getelementptr inbounds i32, i32* %Out, i32 %iv.1
+  store i32 %mul5.us.i.1.i, i32* %Out.1, align 4
+  %iv.2 = or i32 %iv, 2
+  %PIn.2 = getelementptr inbounds i16, i16* %In, i32 %iv.2
+  %In.3 = load i16, i16* %PIn.2, align 2
+  %conv.us.i144.2.i = sext i16 %In.3 to i32
+  %mul5.us.i.2.i = mul nsw i32 %conv.us.i144.2.i, %conv4.i.i
+  %Out.2 = getelementptr inbounds i32, i32* %Out, i32 %iv.2
+  store i32 %mul5.us.i.2.i, i32* %Out.2, align 4
+  %iv.3 = or i32 %iv, 3
+  %PIn.3 = getelementptr inbounds i16, i16* %In, i32 %iv.3
+  %In.4 = load i16, i16* %PIn.3, align 2
+  %conv.us.i144.3.i = sext i16 %In.4 to i32
+  %mul5.us.i.3.i = mul nsw i32 %conv.us.i144.3.i, %conv4.i.i
+  %Out.3 = getelementptr inbounds i32, i32* %Out, i32 %iv.3
+  store i32 %mul5.us.i.3.i, i32* %Out.3, align 4
+  %iv.next = add i32 %iv, 4
+  %count.next = add i32 %count, -4
+  %niter375.ncmp.3.i = icmp eq i32 %count.next, 0
+  br i1 %niter375.ncmp.3.i, label %exit, label %for.body
+
+exit:
+  ret void
+}
+
+; CHECK-LABEL: topbottom_mul_64
+define void @topbottom_mul_64(i32 %N, i64* noalias nocapture readnone %Out, i16* nocapture readonly %In1, i16* nocapture readonly %In2) {
+entry:
+  br label %for.body
+
+; CHECK: for.body:
+; CHECK: [[Cast_PIn1_0:%[^ ]+]] = bitcast i16* %PIn1.0 to i32*
+; CHECK: [[PIn1_01:%[^ ]+]] = load i32, i32* [[Cast_PIn1_0]], align 2
+; CHECK: [[PIn1_1:%[^ ]+]] = ashr i32 [[PIn1_01]], 16
+; CHECK: [[PIn1_01_shl:%[^ ]+]] = shl i32 [[PIn1_01]], 16
+; CHECK: [[PIn1_0:%[^ ]+]] = ashr i32 [[PIn1_01_shl]], 16
+
+; CHECK: [[Cast_PIn2_0:%[^ ]+]] = bitcast i16* %PIn2.0 to i32*
+; CHECK: [[PIn2_01:%[^ ]+]] = load i32, i32* [[Cast_PIn2_0]], align 2
+; CHECK: [[PIn2_1:%[^ ]+]] = ashr i32 [[PIn2_01]], 16
+; CHECK: [[PIn2_01_shl:%[^ ]+]] = shl i32 [[PIn2_01]], 16
+; CHECK: [[PIn2_0:%[^ ]+]] = ashr i32 [[PIn2_01_shl]], 16
+
+; CHECK: [[Mul0:%[^ ]+]] = mul nsw i32 [[PIn1_0]], [[PIn2_0]]
+; CHECK: [[SMul0:%[^ ]+]] = sext i32 [[Mul0]] to i64
+; CHECK: [[Mul1:%[^ ]+]] = mul nsw i32 [[PIn1_1]], [[PIn2_1]]
+; CHECK: [[SMul1:%[^ ]+]] = sext i32 [[Mul1]] to i64
+; CHECK: add i64 [[SMul0]], [[SMul1]]
+
+; CHECK: [[Cast_PIn1_2:%[^ ]+]] = bitcast i16* %PIn1.2 to i32*
+; CHECK: [[PIn1_23:%[^ ]+]] = load i32, i32* [[Cast_PIn1_2]], align 2
+; CHECK: [[PIn1_3:%[^ ]+]] = ashr i32 [[PIn1_23]], 16
+; CHECK: [[PIn1_23_shl:%[^ ]+]] = shl i32 [[PIn1_23]], 16
+; CHECK: [[PIn1_2:%[^ ]+]] = ashr i32 [[PIn1_23_shl]], 16
+
+; CHECK: [[Cast_PIn2_2:%[^ ]+]] = bitcast i16* %PIn2.2 to i32*
+; CHECK: [[PIn2_23:%[^ ]+]] = load i32, i32* [[Cast_PIn2_2]], align 2
+; CHECK: [[PIn2_3:%[^ ]+]] = ashr i32 [[PIn2_23]], 16
+; CHECK: [[PIn2_23_shl:%[^ ]+]] = shl i32 [[PIn2_23]], 16
+; CHECK: [[PIn2_2:%[^ ]+]] = ashr i32 [[PIn2_23_shl]], 16
+
+; CHECK: [[Mul2:%[^ ]+]] = mul nsw i32 [[PIn1_2]], [[PIn2_2]]
+; CHECK: [[SMul2:%[^ ]+]] = sext i32 [[Mul2]] to i64
+; CHECK: [[Mul3:%[^ ]+]] = mul nsw i32 [[PIn1_3]], [[PIn2_3]]
+; CHECK: [[SMul3:%[^ ]+]] = sext i32 [[Mul3]] to i64
+; CHECK: add i64 [[SMul2]], [[SMul3]]
+
+for.body:
+  %iv = phi i32 [ 0, %entry ], [ %iv.next, %for.body ]
+  %iv.out = phi i32 [ 0, %entry] , [ %iv.out.next, %for.body ]
+  %count = phi i32 [ %N, %entry ], [ %count.next, %for.body ]
+  %PIn1.0 = getelementptr inbounds i16, i16* %In1, i32 %iv
+  %In1.0 = load i16, i16* %PIn1.0, align 2
+  %SIn1.0 = sext i16 %In1.0 to i32
+  %PIn2.0 = getelementptr inbounds i16, i16* %In2, i32 %iv
+  %In2.0 = load i16, i16* %PIn2.0, align 2
+  %SIn2.0 = sext i16 %In2.0 to i32
+  %mul5.us.i.i = mul nsw i32 %SIn1.0, %SIn2.0
+  %sext.0 = sext i32 %mul5.us.i.i to i64
+  %iv.1 = or i32 %iv, 1
+  %PIn1.1 = getelementptr inbounds i16, i16* %In1, i32 %iv.1
+  %In1.1 = load i16, i16* %PIn1.1, align 2
+  %SIn1.1 = sext i16 %In1.1 to i32
+  %PIn2.1 = getelementptr inbounds i16, i16* %In2, i32 %iv.1
+  %In2.1 = load i16, i16* %PIn2.1, align 2
+  %SIn2.1 = sext i16 %In2.1 to i32
+  %mul5.us.i.1.i = mul nsw i32 %SIn1.1, %SIn2.1
+  %sext.1 = sext i32 %mul5.us.i.1.i to i64
+  %mac.0 = add i64 %sext.0, %sext.1
+  %Out.0 = getelementptr inbounds i64, i64* %Out, i32 %iv.out
+  store i64 %mac.0, i64* %Out.0, align 4
+  %iv.2 = or i32 %iv, 2
+  %PIn1.2 = getelementptr inbounds i16, i16* %In1, i32 %iv.2
+  %In1.2 = load i16, i16* %PIn1.2, align 2
+  %SIn1.2 = sext i16 %In1.2 to i32
+  %PIn2.2 = getelementptr inbounds i16, i16* %In2, i32 %iv.2
+  %In2.2 = load i16, i16* %PIn2.2, align 2
+  %SIn2.2 = sext i16 %In2.2 to i32
+  %mul5.us.i.2.i = mul nsw i32 %SIn1.2, %SIn2.2
+  %sext.2 = sext i32 %mul5.us.i.2.i to i64
+  %iv.3 = or i32 %iv, 3
+  %PIn1.3 = getelementptr inbounds i16, i16* %In1, i32 %iv.3
+  %In1.3 = load i16, i16* %PIn1.3, align 2
+  %SIn1.3 = sext i16 %In1.3 to i32
+  %PIn2.3 = getelementptr inbounds i16, i16* %In2, i32 %iv.3
+  %In2.3 = load i16, i16* %PIn2.3, align 2
+  %SIn2.3 = sext i16 %In2.3 to i32
+  %mul5.us.i.3.i = mul nsw i32 %SIn1.3, %SIn2.3
+  %sext.3 = sext i32 %mul5.us.i.3.i to i64
+  %mac.1 = add i64 %sext.2, %sext.3
+  %iv.out.1 = or i32 %iv.out, 1
+  %Out.1 = getelementptr inbounds i64, i64* %Out, i32 %iv.out.1
+  store i64 %mac.1, i64* %Out.1, align 4
+  %iv.next = add i32 %iv, 4
+  %iv.out.next = add i32 %iv.out, 2
+  %count.next = add i32 %count, -4
+  %niter375.ncmp.3.i = icmp eq i32 %count.next, 0
+  br i1 %niter375.ncmp.3.i, label %exit, label %for.body
+
+exit:
+  ret void
+}

llvm/trunk/test/CodeGen/ARM/smlad0.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; The Cortex-M0 does not support unaligned accesses:
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-;
-; Check DSP extension:
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-
-define dso_local i32 @OneReduction(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-;
-; CHECK-LABEL: @OneReduction
-; CHECK:  %mac1{{\.}}026 = phi i32 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
-; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
-; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
-; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
-; CHECK:  [[V8]] = call i32 @llvm.arm.smlad(i32 [[V5]], i32 [[V7]], i32 %mac1{{\.}}026)
-; CHECK-NOT: call i32 @llvm.arm.smlad
-;
-; CHECK-UNSUPPORTED-NOT:  call i32 @llvm.arm.smlad
-;
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-; One reduction statement here:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-
-; Here the Mul is the LHS, and the Add the RHS.
-  %add11 = add i32 %mul9, %add10
-
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-
-define dso_local arm_aapcs_vfpcc i32 @TwoReductions(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-;
-; CHECK-LABEL: @TwoReductions
-;
-; CHECK:  %mac1{{\.}}058 = phi i32 [ [[V10:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  %mac2{{\.}}057 = phi i32 [ [[V17:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V10]] = call i32 @llvm.arm.smlad(i32 %{{.*}}, i32 %{{.*}}, i32 %mac1{{\.}}058)
-; CHECK:  [[V17]] = call i32 @llvm.arm.smlad(i32 %{{.*}}, i32 %{{.*}}, i32 %mac2{{\.}}057)
-; CHECK-NOT: call i32 @llvm.arm.smlad
-;
-entry:
-  %cmp55 = icmp sgt i32 %arg, 0
-  br i1 %cmp55, label %for.body.preheader, label %for.cond.cleanup
-
-for.cond.cleanup:
-  %mac2.0.lcssa = phi i32 [ 0, %entry ], [ %add28, %for.body ]
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add16, %for.body ]
-  %add30 = add nsw i32 %mac1.0.lcssa, %mac2.0.lcssa
-  ret i32 %add30
-
-for.body.preheader:
-  br label %for.body
-
-for.body:
-; And two reduction statements here:
-  %mac1.058 = phi i32 [ %add16, %for.body ], [ 0, %for.body.preheader ]
-  %mac2.057 = phi i32 [ %add28, %for.body ], [ 0, %for.body.preheader ]
-
-  %i.056 = phi i32 [ %add29, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.056
-  %0 = load i16, i16* %arrayidx, align 2
-  %add1 = or i32 %i.056, 1
-  %arrayidx2 = getelementptr inbounds i16, i16* %arg3, i32 %add1
-  %1 = load i16, i16* %arrayidx2, align 2
-  %add3 = or i32 %i.056, 2
-  %arrayidx4 = getelementptr inbounds i16, i16* %arg3, i32 %add3
-  %2 = load i16, i16* %arrayidx4, align 2
-
-  %add5 = or i32 %i.056, 3
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg3, i32 %add5
-  %3 = load i16, i16* %arrayidx6, align 2
-  %arrayidx8 = getelementptr inbounds i16, i16* %arg2, i32 %i.056
-  %4 = load i16, i16* %arrayidx8, align 2
-  %conv = sext i16 %4 to i32
-  %conv9 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv9
-  %arrayidx11 = getelementptr inbounds i16, i16* %arg2, i32 %add1
-  %5 = load i16, i16* %arrayidx11, align 2
-  %conv12 = sext i16 %5 to i32
-  %conv13 = sext i16 %1 to i32
-  %mul14 = mul nsw i32 %conv12, %conv13
-  %add15 = add i32 %mul, %mac1.058
-  %add16 = add i32 %add15, %mul14
-  %arrayidx18 = getelementptr inbounds i16, i16* %arg2, i32 %add3
-  %6 = load i16, i16* %arrayidx18, align 2
-  %conv19 = sext i16 %6 to i32
-  %conv20 = sext i16 %2 to i32
-  %mul21 = mul nsw i32 %conv19, %conv20
-  %arrayidx23 = getelementptr inbounds i16, i16* %arg2, i32 %add5
-  %7 = load i16, i16* %arrayidx23, align 2
-  %conv24 = sext i16 %7 to i32
-  %conv25 = sext i16 %3 to i32
-  %mul26 = mul nsw i32 %conv24, %conv25
-  %add27 = add i32 %mul21, %mac2.057
-  %add28 = add i32 %add27, %mul26
-  %add29 = add nuw nsw i32 %i.056, 4
-  %cmp = icmp slt i32 %add29, %arg
-  br i1 %cmp, label %for.body, label %for.cond.cleanup
-}

llvm/trunk/test/CodeGen/ARM/smlad1.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-
-; CHECK-LABEL: @test1
-; CHECK:  %mac1{{\.}}026 = phi i32 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
-; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
-; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
-; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
-; CHECK:  [[V8]] = call i32 @llvm.arm.smlad(i32 [[V5]], i32 [[V7]], i32 %mac1{{\.}}026)
-
-define dso_local i32 @test1(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-
-; And here the Add is the LHS, the Mul the RHS
-  %add11 = add i32 %add10, %mul9
-
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-
-; Here we have i8 loads, which we do want to support, but don't handle yet.
-;
-; CHECK-LABEL: @test2
-; CHECK-NOT:   call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test2(i32 %arg, i32* nocapture readnone %arg1, i8* nocapture readonly %arg2, i8* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i8, i8* %arg3, align 2
-  %.pre27 = load i8, i8* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i8, i8* %arg3, i32 %i.025
-  %0 = load i8, i8* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i8, i8* %arg3, i32 %add
-  %1 = load i8, i8* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i8, i8* %arg2, i32 %i.025
-  %2 = load i8, i8* %arrayidx3, align 2
-  %conv = sext i8 %2 to i32
-  %conv4 = sext i8 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i8, i8* %arg2, i32 %add
-  %3 = load i8, i8* %arrayidx6, align 2
-  %conv7 = sext i8 %3 to i32
-  %conv8 = sext i8 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-  %add11 = add i32 %add10, %mul9
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-
-

llvm/trunk/test/CodeGen/ARM/smlad10.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; Reduction statement is an i64 type: we only support i32 so check that the
-; rewrite isn't triggered.
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i64 @test(i64 %arg, i64* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i64 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
-  ret i64 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i64 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i64 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i64 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i64 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i64 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i64
-  %conv4 = sext i16 %0 to i64
-  %mul = mul nsw i64 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i64 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i64
-  %conv8 = sext i16 %1 to i64
-  %mul9 = mul nsw i64 %conv7, %conv8
-  %add10 = add i64 %mul, %mac1.026
-
-  %add11 = add i64 %mul9, %add10
-
-  %exitcond = icmp ne i64 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-

llvm/trunk/test/CodeGen/ARM/smlad11.ll

-; REQUIRES: asserts
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S -stats 2>&1 | FileCheck %s
-;
-; A more complicated chain: 4 mul operations, so we expect 2 smlad calls.
-;
-; CHECK:  %mac1{{\.}}054 = phi i32 [ [[V17:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V8:%[0-9]+]] = bitcast i16* %arrayidx8 to i32*
-; CHECK:  [[V9:%[0-9]+]] = load i32, i32* [[V8]], align 2
-; CHECK:  [[V10:%[0-9]+]] = bitcast i16* %arrayidx to i32*
-; CHECK:  [[V11:%[0-9]+]] = load i32, i32* [[V10]], align 2
-; CHECK:  [[V12:%[0-9]+]] = call i32 @llvm.arm.smlad(i32 [[V9]], i32 [[V11]], i32 %mac1{{\.}}054)
-; CHECK:  [[V13:%[0-9]+]] = bitcast i16* %arrayidx17 to i32*
-; CHECK:  [[V14:%[0-9]+]] = load i32, i32* [[V13]], align 2
-; CHECK:  [[V15:%[0-9]+]] = bitcast i16* %arrayidx4 to i32*
-; CHECK:  [[V16:%[0-9]+]] = load i32, i32* [[V15]], align 2
-; CHECK:  [[V17:%[0-9]+]] = call i32 @llvm.arm.smlad(i32 [[V14]], i32 [[V16]], i32 [[V12]])
-;
-; And we don't want to see a 3rd smlad:
-; CHECK-NOT: call i32 @llvm.arm.smlad
-;
-; CHECK:  2 arm-parallel-dsp - Number of smlad instructions generated
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp52 = icmp sgt i32 %arg, 0
-  br i1 %cmp52, label %for.body.preheader, label %for.cond.cleanup
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add28, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body.preheader:
-  br label %for.body
-
-for.body:
-  %mac1.054 = phi i32 [ %add28, %for.body ], [ 0, %for.body.preheader ]
-  %i.053 = phi i32 [ %add29, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.053
-  %0 = load i16, i16* %arrayidx, align 2
-  %add1 = or i32 %i.053, 1
-  %arrayidx2 = getelementptr inbounds i16, i16* %arg3, i32 %add1
-  %1 = load i16, i16* %arrayidx2, align 2
-  %add3 = or i32 %i.053, 2
-  %arrayidx4 = getelementptr inbounds i16, i16* %arg3, i32 %add3
-  %2 = load i16, i16* %arrayidx4, align 2
-  %add5 = or i32 %i.053, 3
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg3, i32 %add5
-  %3 = load i16, i16* %arrayidx6, align 2
-  %arrayidx8 = getelementptr inbounds i16, i16* %arg2, i32 %i.053
-  %4 = load i16, i16* %arrayidx8, align 2
-  %conv = sext i16 %4 to i32
-  %conv9 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv9
-  %arrayidx11 = getelementptr inbounds i16, i16* %arg2, i32 %add1
-  %5 = load i16, i16* %arrayidx11, align 2
-  %conv12 = sext i16 %5 to i32
-  %conv13 = sext i16 %1 to i32
-  %mul14 = mul nsw i32 %conv12, %conv13
-  %arrayidx17 = getelementptr inbounds i16, i16* %arg2, i32 %add3
-  %6 = load i16, i16* %arrayidx17, align 2
-  %conv18 = sext i16 %6 to i32
-  %conv19 = sext i16 %2 to i32
-  %mul20 = mul nsw i32 %conv18, %conv19
-  %arrayidx23 = getelementptr inbounds i16, i16* %arg2, i32 %add5
-  %7 = load i16, i16* %arrayidx23, align 2
-  %conv24 = sext i16 %7 to i32
-  %conv25 = sext i16 %3 to i32
-  %mul26 = mul nsw i32 %conv24, %conv25
-  %add15 = add i32 %mul, %mac1.054
-  %add21 = add i32 %add15, %mul14
-  %add27 = add i32 %add21, %mul20
-  %add28 = add i32 %add27, %mul26
-  %add29 = add nuw nsw i32 %i.053, 4
-  %cmp = icmp slt i32 %add29, %arg
-  br i1 %cmp, label %for.body, label %for.cond.cleanup
-}

llvm/trunk/test/CodeGen/ARM/smlad12.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; The loop header is not the loop latch.
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-; This is the loop header:
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body2 ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body2 ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-  %add11 = add i32 %mul9, %add10
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body2, label %for.cond.cleanup
-
-; And this is the loop latch:
-for.body2:
-  br label %for.body
-}

llvm/trunk/test/CodeGen/ARM/smlad2.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; Operands of both muls are not symmetrical (see also comments inlined below), check
-; that the rewrite isn't triggered.
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-
-; This zero-extends the 2nd operand of %mul:
-  %conv4 = zext i16 %0 to i32
-
-  %mul = mul nsw i32 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-
-; And here we only have sign-extensions. Thus, the operands of
-; %mul and %mul9 are not symmetrical:
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-  %add11 = add i32 %add10, %mul9
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}

llvm/trunk/test/CodeGen/ARM/smlad3.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; The loads are not consecutive: check that the rewrite isn't triggered.
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-
-; Here we add another constants offset of 2, to make sure the
-; loads to %3 and %2 are not consecutive:
-
-  %add5 = add nuw nsw i32 %i.025, 2
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add5
-  %3 = load i16, i16* %arrayidx6, align 2
-
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-  %add11 = add i32 %add10, %mul9
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-

llvm/trunk/test/CodeGen/ARM/smlad4.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; The loads are not narrow loads: check that the rewrite isn't triggered.
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-; Arg2 is now an i32, while Arg3 is still and i16:
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i32* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp22 = icmp sgt i32 %arg, 0
-  br i1 %cmp22, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add9, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %0 = phi i16 [ %1, %for.body ], [ %.pre, %for.body.preheader ]
-  %mac1.024 = phi i32 [ %add9, %for.body ], [ 0, %for.body.preheader ]
-  %i.023 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %add = add nuw nsw i32 %i.023, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %conv = sext i16 %0 to i32
-
-; This is a 'normal' i32 load to %2:
-  %arrayidx3 = getelementptr inbounds i32, i32* %arg2, i32 %i.023
-  %2 = load i32, i32* %arrayidx3, align 4
-
-; This mul has now 1 operand which is a narrow load, and the other a normal
-; i32 load:
-  %mul = mul nsw i32 %2, %conv
-
-  %add4 = add nuw nsw i32 %i.023, 2
-  %arrayidx5 = getelementptr inbounds i32, i32* %arg2, i32 %add4
-  %3 = load i32, i32* %arrayidx5, align 4
-  %conv6 = sext i16 %1 to i32
-  %mul7 = mul nsw i32 %3, %conv6
-  %add8 = add i32 %mul, %mac1.024
-  %add9 = add i32 %add8, %mul7
-  %exitcond = icmp eq i32 %add, %arg
-  br i1 %exitcond, label %for.cond.cleanup, label %for.body
-}

llvm/trunk/test/CodeGen/ARM/smlad5.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; The loads are volatile loads: check that the rewrite isn't triggered.
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load volatile i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load volatile i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load volatile i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load volatile i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-  %add11 = add i32 %add10, %mul9
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-

llvm/trunk/test/CodeGen/ARM/smlad6.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; Alias check: check that the rewrite isn't triggered when there's a store
-; instruction possibly aliasing any mul load operands; arguments are passed
-; without 'restrict' enabled.
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-
-; Store inserted here, aliasing with arrayidx, arrayidx1, arrayidx3
-  store i16 42, i16* %arrayidx, align 2
-
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-  %add11 = add i32 %mul9, %add10
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-

llvm/trunk/test/CodeGen/ARM/smlad7.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; Alias check: check that the rewrite isn't triggered when there's a store
-; aliasing one of the mul load operands. Arguments are now annotated with
-; 'noalias'.
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test(i32 %arg, i32* noalias %arg1, i16* noalias readonly %arg2, i16* noalias readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-
-; Store inserted here, aliasing only with loads from 'arrayidx'.
-  store i16 42, i16* %arrayidx, align 2
-
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %add10 = add i32 %mul, %mac1.026
-
-; Here the Mul is the LHS, and the Add the RHS.
-  %add11 = add i32 %mul9, %add10
-
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-

llvm/trunk/test/CodeGen/ARM/smlad8.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; Mul with operands that are not simple load and sext/zext chains: this is not
-; yet supported so the rewrite shouldn't trigger (but we do want to support this
-; soon).
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3, i16* %arg4) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  %gep0 = getelementptr inbounds i16, i16* %arg4, i32 0
-  %gep1 = getelementptr inbounds i16, i16* %arg4, i32 1
-  %.add4 = load i16, i16* %gep0, align 2
-  %.add5 = load i16, i16* %gep1, align 2
-  %.zext4 = zext i16 %.add4 to i32
-  %.zext5 = zext i16 %.add5 to i32
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %add1 = add i32 %conv, %.zext4
-
-; This mul has a more complicated pattern as an operand, %add1
-; is another add and load, which we don't support for now.
-  %mul = mul nsw i32 %add1, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %add2 = add i32 %conv7, %.zext5
-
-; Same here
-  %mul9 = mul nsw i32 %add2, %conv8
-  %add10 = add i32 %mul, %mac1.026
-
-  %add11 = add i32 %mul9, %add10
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}

llvm/trunk/test/CodeGen/ARM/smlad9.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; Muls with operands that are constants: not yet supported, so the rewrite
-; should not trigger (but we do want to add this soon).
-;
-; CHECK-NOT:  call i32 @llvm.arm.smlad
-;
-define dso_local i32 @test(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i32 [ 0, %entry ], [ %add11, %for.body ]
-  ret i32 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i32 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %v2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %v2 to i32
-
-; RHS operand of this mul is a constant
-  %mul = mul nsw i32 %conv, 43
-
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %v3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %v3 to i32
-
-; And this RHS operand is a constant too.
-  %mul9 = mul nsw i32 %conv7, 42
-
-  %add10 = add i32 %mul, %mac1.026
-  %add11 = add i32 %mul9, %add10
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-

llvm/trunk/test/CodeGen/ARM/smladx-1.ll

-; RUN: opt -mtriple=thumbv8m.main -mcpu=cortex-m33 -arm-parallel-dsp %s -S -o - | FileCheck %s
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-
-define i32 @smladx(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
-
-; CHECK-LABEL: smladx
-; CHECK: = phi i32 [ 0, %for.body.preheader.new ],
-; CHECK: [[ACC0:%[^ ]+]] = phi i32 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
-; CHECK: [[PIN23:%[^ ]+]] = bitcast i16* %pIn2.3 to i32*
-; CHECK: [[IN23:%[^ ]+]] = load i32, i32* [[PIN23]], align 2
-; CHECK: [[PIN12:%[^ ]+]] = bitcast i16* %pIn1.2 to i32*
-; CHECK: [[IN12:%[^ ]+]] = load i32, i32* [[PIN12]], align 2
-; CHECK: [[ACC1:%[^ ]+]] = call i32 @llvm.arm.smladx(i32 [[IN23]], i32 [[IN12]], i32 [[ACC0]])
-; CHECK: [[PIN21:%[^ ]+]] = bitcast i16* %pIn2.1 to i32*
-; CHECK: [[IN21:%[^ ]+]] = load i32, i32* [[PIN21]], align 2
-; CHECK: [[PIN10:%[^ ]+]] = bitcast i16* %pIn1.0 to i32*
-; CHECK: [[IN10:%[^ ]+]] = load i32, i32* [[PIN10]], align 2
-; CHECK: [[ACC2]] = call i32 @llvm.arm.smladx(i32 [[IN21]], i32 [[IN10]], i32 [[ACC1]])
-; CHECK-NOT: call i32 @llvm.arm.smlad
-; CHECK-UNSUPPORTED-NOT:  call i32 @llvm.arm.smlad
-
-entry:
-  %cmp9 = icmp eq i32 %limit, 0
-  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
-
-for.body.preheader:
-  %0 = add i32 %limit, -1
-  %xtraiter = and i32 %limit, 3
-  %1 = icmp ult i32 %0, 3
-  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
-
-for.body.preheader.new:
-  %unroll_iter = sub i32 %limit, %xtraiter
-  br label %for.body
-
-for.cond.cleanup.loopexit.unr-lcssa:
-  %add.lcssa.ph = phi i32 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
-  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
-  %sum.010.unr = phi i32 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
-  %lcmp.mod = icmp eq i32 %xtraiter, 0
-  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil
-
-for.body.epil:
-  %i.011.epil = phi i32 [ %inc.epil, %for.body.epil ], [ %i.011.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %sum.010.epil = phi i32 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %epil.iter = phi i32 [ %epil.iter.sub, %for.body.epil ], [ %xtraiter, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %sub.epil = sub i32 %j, %i.011.epil
-  %arrayidx.epil = getelementptr inbounds i16, i16* %pIn2, i32 %sub.epil
-  %2 = load i16, i16* %arrayidx.epil, align 2
-  %conv.epil = sext i16 %2 to i32
-  %arrayidx1.epil = getelementptr inbounds i16, i16* %pIn1, i32 %i.011.epil
-  %3 = load i16, i16* %arrayidx1.epil, align 2
-  %conv2.epil = sext i16 %3 to i32
-  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
-  %add.epil = add nsw i32 %mul.epil, %sum.010.epil
-  %inc.epil = add nuw i32 %i.011.epil, 1
-  %epil.iter.sub = add i32 %epil.iter, -1
-  %epil.iter.cmp = icmp eq i32 %epil.iter.sub, 0
-  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
-
-for.cond.cleanup:
-  %sum.0.lcssa = phi i32 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
-  ret i32 %sum.0.lcssa
-
-for.body:
-  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
-  %sum.010 = phi i32 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
-  %niter = phi i32 [ %unroll_iter, %for.body.preheader.new ], [ %niter.nsub.3, %for.body ]
-  %pIn2Base = phi i16* [ %pIn2, %for.body.preheader.new ], [ %pIn2.4, %for.body ]
-  %pIn2.0 = getelementptr inbounds i16, i16* %pIn2Base, i32 0
-  %In2 = load i16, i16* %pIn2.0, align 2
-  %pIn1.0 = getelementptr inbounds i16, i16* %pIn1, i32 %i.011
-  %In1 = load i16, i16* %pIn1.0, align 2
-  %inc = or i32 %i.011, 1
-  %pIn2.1 = getelementptr inbounds i16, i16* %pIn2Base, i32 -1
-  %In2.1 = load i16, i16* %pIn2.1, align 2
-  %pIn1.1 = getelementptr inbounds i16, i16* %pIn1, i32 %inc
-  %In1.1 = load i16, i16* %pIn1.1, align 2
-  %inc.1 = or i32 %i.011, 2
-  %pIn2.2 = getelementptr inbounds i16, i16* %pIn2Base, i32 -2
-  %In2.2 = load i16, i16* %pIn2.2, align 2
-  %pIn1.2 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.1
-  %In1.2 = load i16, i16* %pIn1.2, align 2
-  %inc.2 = or i32 %i.011, 3
-  %pIn2.3 = getelementptr inbounds i16, i16* %pIn2Base, i32 -3
-  %In2.3 = load i16, i16* %pIn2.3, align 2
-  %pIn1.3 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.2
-  %In1.3 = load i16, i16* %pIn1.3, align 2
-  %sextIn1 = sext i16 %In1 to i32
-  %sextIn1.1 = sext i16 %In1.1 to i32
-  %sextIn1.2 = sext i16 %In1.2 to i32
-  %sextIn1.3 = sext i16 %In1.3 to i32
-  %sextIn2 = sext i16 %In2 to i32
-  %sextIn2.1 = sext i16 %In2.1 to i32
-  %sextIn2.2 = sext i16 %In2.2 to i32
-  %sextIn2.3 = sext i16 %In2.3 to i32
-  %mul = mul nsw i32 %sextIn1, %sextIn2
-  %mul.1 = mul nsw i32 %sextIn1.1, %sextIn2.1
-  %mul.2 = mul nsw i32 %sextIn1.2, %sextIn2.2
-  %mul.3 = mul nsw i32 %sextIn1.3, %sextIn2.3
-  %add = add nsw i32 %mul, %sum.010
-  %add.1 = add nsw i32 %mul.1, %add
-  %add.2 = add nsw i32 %mul.2, %add.1
-  %add.3 = add nsw i32 %mul.3, %add.2
-  %inc.3 = add i32 %i.011, 4
-  %pIn2.4 = getelementptr inbounds i16, i16* %pIn2Base, i32 -4
-  %niter.nsub.3 = add i32 %niter, -4
-  %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
-  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
-}
-
-define i32 @smladx_swap(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
-
-; CHECK-LABEL: smladx_swap
-; CHECK: for.body.preheader.new:
-; CHECK: [[PIN1Base:[^ ]+]] = getelementptr i16, i16* %pIn1
-; CHECK: [[PIN2Base:[^ ]+]] = getelementptr i16, i16* %pIn2
-
-; CHECK: for.body:
-; CHECK: [[PIN2:%[^ ]+]] = phi i16* [ [[PIN2_NEXT:%[^ ]+]], %for.body ], [ [[PIN2Base]], %for.body.preheader.new ]
-; CHECK: [[PIN1:%[^ ]+]] = phi i16* [ [[PIN1_NEXT:%[^ ]+]], %for.body ], [ [[PIN1Base]], %for.body.preheader.new ]
-; CHECK: [[IV:%[^ ]+]] = phi i32
-; CHECK: [[ACC0:%[^ ]+]] = phi i32 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
-
-; CHECK: [[PIN1_2:%[^ ]+]] = getelementptr i16, i16* [[PIN1]], i32 -2
-; CHECK: [[PIN2_2:%[^ ]+]] = getelementptr i16, i16* [[PIN2]], i32 -2
-
-
-; CHECK: [[PIN2_2_CAST:%[^ ]+]] = bitcast i16* [[PIN2_2]] to i32*
-; CHECK: [[IN2_2:%[^ ]+]] = load i32, i32* [[PIN2_2_CAST]], align 2
-; CHECK: [[PIN1_CAST:%[^ ]+]] = bitcast i16* [[PIN1]] to i32*
-; CHECK: [[IN1:%[^ ]+]] = load i32, i32* [[PIN1_CAST]], align 2
-; CHECK: [[ACC1:%[^ ]+]] = call i32 @llvm.arm.smladx(i32 [[IN2_2]], i32 [[IN1]], i32 [[ACC0]])
-
-; CHECK: [[PIN2_CAST:%[^ ]+]] = bitcast i16* [[PIN2]] to i32*
-; CHECK: [[IN2:%[^ ]+]] = load i32, i32* [[PIN2_CAST]], align 2
-; CHECK: [[PIN1_2_CAST:%[^ ]+]] = bitcast i16* [[PIN1_2]] to i32*
-; CHECK: [[IN1_2:%[^ ]+]] = load i32, i32* [[PIN1_2_CAST]], align 2
-; CHECK: [[ACC2]] = call i32 @llvm.arm.smladx(i32 [[IN2]], i32 [[IN1_2]], i32 [[ACC1]])
-
-; CHECK: [[PIN1_NEXT]] = getelementptr i16, i16* [[PIN1]], i32 4
-; CHECK: [[PIN2_NEXT]] = getelementptr i16, i16* [[PIN2]], i32 -4
-
-; CHECK-NOT: call i32 @llvm.arm.smlad
-; CHECK-UNSUPPORTED-NOT:  call i32 @llvm.arm.smlad
-
-entry:
-  %cmp9 = icmp eq i32 %limit, 0
-  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
-
-for.body.preheader:
-  %0 = add i32 %limit, -1
-  %xtraiter = and i32 %limit, 3
-  %1 = icmp ult i32 %0, 3
-  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
-
-for.body.preheader.new:
-  %unroll_iter = sub i32 %limit, %xtraiter
-  %scevgep6 = getelementptr i16, i16* %pIn1, i32 2
-  %2 = add i32 %j, -1
-  %scevgep11 = getelementptr i16, i16* %pIn2, i32 %2
-  br label %for.body
-
-for.cond.cleanup.loopexit.unr-lcssa:
-  %add.lcssa.ph = phi i32 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
-  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
-  %sum.010.unr = phi i32 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
-  %lcmp.mod = icmp eq i32 %xtraiter, 0
-  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil.preheader
-
-for.body.epil.preheader:
-  %scevgep = getelementptr i16, i16* %pIn1, i32 %i.011.unr
-  %3 = sub i32 %j, %i.011.unr
-  %scevgep2 = getelementptr i16, i16* %pIn2, i32 %3
-  %4 = sub i32 0, %xtraiter
-  br label %for.body.epil
-
-for.body.epil:
-  %lsr.iv5 = phi i32 [ %4, %for.body.epil.preheader ], [ %lsr.iv.next, %for.body.epil ]
-  %lsr.iv3 = phi i16* [ %scevgep2, %for.body.epil.preheader ], [ %scevgep4, %for.body.epil ]
-  %lsr.iv = phi i16* [ %scevgep, %for.body.epil.preheader ], [ %scevgep1, %for.body.epil ]
-  %sum.010.epil = phi i32 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.body.epil.preheader ]
-  %5 = load i16, i16* %lsr.iv3, align 2
-  %conv.epil = sext i16 %5 to i32
-  %6 = load i16, i16* %lsr.iv, align 2
-  %conv2.epil = sext i16 %6 to i32
-  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
-  %add.epil = add nsw i32 %mul.epil, %sum.010.epil
-  %scevgep1 = getelementptr i16, i16* %lsr.iv, i32 1
-  %scevgep4 = getelementptr i16, i16* %lsr.iv3, i32 -1
-  %lsr.iv.next = add nsw i32 %lsr.iv5, 1
-  %epil.iter.cmp = icmp eq i32 %lsr.iv.next, 0
-  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
-
-for.cond.cleanup:
-  %sum.0.lcssa = phi i32 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
-  ret i32 %sum.0.lcssa
-
-for.body:
-  %pin2 = phi i16* [ %pin2_sub4, %for.body ], [ %scevgep11, %for.body.preheader.new ]
-  %pin1 = phi i16* [ %pin1_add4, %for.body ], [ %scevgep6, %for.body.preheader.new ]
-  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
-  %sum.010 = phi i32 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
-  %pin2_add1 = getelementptr i16, i16* %pin2, i32 1
-  %In2 = load i16, i16* %pin2_add1, align 2
-  %pin1_sub2 = getelementptr i16, i16* %pin1, i32 -2
-  %In1 = load i16, i16* %pin1_sub2, align 2
-  %In2.1 = load i16, i16* %pin2, align 2
-  %pin1_sub1 = getelementptr i16, i16* %pin1, i32 -1
-  %In1.1 = load i16, i16* %pin1_sub1, align 2
-  %pin2_sub1 = getelementptr i16, i16* %pin2, i32 -1
-  %In2.2 = load i16, i16* %pin2_sub1, align 2
-  %In1.2 = load i16, i16* %pin1, align 2
-  %pin2_sub2 = getelementptr i16, i16* %pin2, i32 -2
-  %In2.3 = load i16, i16* %pin2_sub2, align 2
-  %pin1_add1 = getelementptr i16, i16* %pin1, i32 1
-  %In1.3 = load i16, i16* %pin1_add1, align 2
-  %sextIn2 = sext i16 %In2 to i32
-  %sextIn1 = sext i16 %In1 to i32
-  %sextIn2.1 = sext i16 %In2.1 to i32
-  %sextIn1.1 = sext i16 %In1.1 to i32
-  %sextIn2.2 = sext i16 %In2.2 to i32
-  %sextIn1.2 = sext i16 %In1.2 to i32
-  %sextIn2.3 = sext i16 %In2.3 to i32
-  %sextIn1.3 = sext i16 %In1.3 to i32
-  %mul = mul nsw i32 %sextIn2, %sextIn1
-  %add = add nsw i32 %mul, %sum.010
-  %mul.1 = mul nsw i32 %sextIn2.1, %sextIn1.1
-  %add.1 = add nsw i32 %mul.1, %add
-  %mul.2 = mul nsw i32 %sextIn2.2, %sextIn1.2
-  %add.2 = add nsw i32 %mul.2, %add.1
-  %mul.3 = mul nsw i32 %sextIn2.3, %sextIn1.3
-  %add.3 = add nsw i32 %mul.3, %add.2
-  %inc.3 = add i32 %i.011, 4
-  %pin1_add4 = getelementptr i16, i16* %pin1, i32 4
-  %pin2_sub4 = getelementptr i16, i16* %pin2, i32 -4
-  %niter.ncmp.3 = icmp eq i32 %unroll_iter, %inc.3
-  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
-}

llvm/trunk/test/CodeGen/ARM/smlald0.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; The Cortex-M0 does not support unaligned accesses:
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-;
-; Check DSP extension:
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-
-define dso_local i64 @OneReduction(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-;
-; CHECK-LABEL: @OneReduction
-; CHECK:  %mac1{{\.}}026 = phi i64 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
-; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
-; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
-; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
-; CHECK:  [[V8]] = call i64 @llvm.arm.smlald(i32 [[V5]], i32 [[V7]], i64 %mac1{{\.}}026)
-; CHECK-NOT: call i64 @llvm.arm.smlald
-;
-; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlald
-;
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
-  ret i64 %mac1.0.lcssa
-
-for.body:
-; One reduction statement here:
-  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i64
-  %conv4 = sext i16 %0 to i64
-  %mul = mul nsw i64 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i64
-  %conv8 = sext i16 %1 to i64
-  %mul9 = mul nsw i64 %conv7, %conv8
-  %add10 = add i64 %mul, %mac1.026
-
-; Here the Mul is the LHS, and the Add the RHS.
-  %add11 = add i64 %mul9, %add10
-
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-
-define dso_local arm_aapcs_vfpcc i64 @TwoReductions(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-;
-; CHECK-LABEL: @TwoReductions
-;
-; CHECK:  %mac1{{\.}}058 = phi i64 [ [[V10:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  %mac2{{\.}}057 = phi i64 [ [[V17:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V10]] = call i64 @llvm.arm.smlald(i32 %{{.*}}, i32 %{{.*}}, i64 %mac1{{\.}}058)
-; CHECK:  [[V17]] = call i64 @llvm.arm.smlald(i32 %{{.*}}, i32 %{{.*}}, i64 %mac2{{\.}}057)
-; CHECK-NOT: call i64 @llvm.arm.smlald
-;
-entry:
-  %cmp55 = icmp sgt i32 %arg, 0
-  br i1 %cmp55, label %for.body.preheader, label %for.cond.cleanup
-
-for.cond.cleanup:
-  %mac2.0.lcssa = phi i64 [ 0, %entry ], [ %add28, %for.body ]
-  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add16, %for.body ]
-  %add30 = add nsw i64 %mac1.0.lcssa, %mac2.0.lcssa
-  ret i64 %add30
-
-for.body.preheader:
-  br label %for.body
-
-for.body:
-; And two reduction statements here:
-  %mac1.058 = phi i64 [ %add16, %for.body ], [ 0, %for.body.preheader ]
-  %mac2.057 = phi i64 [ %add28, %for.body ], [ 0, %for.body.preheader ]
-
-  %i.056 = phi i32 [ %add29, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.056
-  %0 = load i16, i16* %arrayidx, align 2
-  %add1 = or i32 %i.056, 1
-  %arrayidx2 = getelementptr inbounds i16, i16* %arg3, i32 %add1
-  %1 = load i16, i16* %arrayidx2, align 2
-  %add3 = or i32 %i.056, 2
-  %arrayidx4 = getelementptr inbounds i16, i16* %arg3, i32 %add3
-  %2 = load i16, i16* %arrayidx4, align 2
-
-  %add5 = or i32 %i.056, 3
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg3, i32 %add5
-  %3 = load i16, i16* %arrayidx6, align 2
-  %arrayidx8 = getelementptr inbounds i16, i16* %arg2, i32 %i.056
-  %4 = load i16, i16* %arrayidx8, align 2
-  %conv = sext i16 %4 to i64
-  %conv9 = sext i16 %0 to i64
-  %mul = mul nsw i64 %conv, %conv9
-  %arrayidx11 = getelementptr inbounds i16, i16* %arg2, i32 %add1
-  %5 = load i16, i16* %arrayidx11, align 2
-  %conv12 = sext i16 %5 to i64
-  %conv13 = sext i16 %1 to i64
-  %mul14 = mul nsw i64 %conv12, %conv13
-  %add15 = add i64 %mul, %mac1.058
-  %add16 = add i64 %add15, %mul14
-  %arrayidx18 = getelementptr inbounds i16, i16* %arg2, i32 %add3
-  %6 = load i16, i16* %arrayidx18, align 2
-  %conv19 = sext i16 %6 to i64
-  %conv20 = sext i16 %2 to i64
-  %mul21 = mul nsw i64 %conv19, %conv20
-  %arrayidx23 = getelementptr inbounds i16, i16* %arg2, i32 %add5
-  %7 = load i16, i16* %arrayidx23, align 2
-  %conv24 = sext i16 %7 to i64
-  %conv25 = sext i16 %3 to i64
-  %mul26 = mul nsw i64 %conv24, %conv25
-  %add27 = add i64 %mul21, %mac2.057
-  %add28 = add i64 %add27, %mul26
-  %add29 = add nuw nsw i32 %i.056, 4
-  %cmp = icmp slt i32 %add29, %arg
-  br i1 %cmp, label %for.body, label %for.cond.cleanup
-}

llvm/trunk/test/CodeGen/ARM/smlald1.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-
-; CHECK-LABEL: @test1
-; CHECK:  %mac1{{\.}}026 = phi i64 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
-; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
-; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
-; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
-; CHECK:  [[V8]] = call i64 @llvm.arm.smlald(i32 [[V5]], i32 [[V7]], i64 %mac1{{\.}}026)
-
-define dso_local i64 @test1(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
-  ret i64 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i64
-  %conv4 = sext i16 %0 to i64
-  %mul = mul nsw i64 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i64
-  %conv8 = sext i16 %1 to i64
-  %mul9 = mul nsw i64 %conv7, %conv8
-  %add10 = add i64 %mul, %mac1.026
-
-; And here the Add is the LHS, the Mul the RHS
-  %add11 = add i64 %add10, %mul9
-
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-
-; Here we have i8 loads, which we do want to support, but don't handle yet.
-;
-; CHECK-LABEL: @test2
-; CHECK-NOT:   call i64 @llvm.arm.smlad
-;
-define dso_local i64 @test2(i32 %arg, i32* nocapture readnone %arg1, i8* nocapture readonly %arg2, i8* nocapture readonly %arg3) {
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i8, i8* %arg3, align 2
-  %.pre27 = load i8, i8* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
-  ret i64 %mac1.0.lcssa
-
-for.body:
-  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i8, i8* %arg3, i32 %i.025
-  %0 = load i8, i8* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i8, i8* %arg3, i32 %add
-  %1 = load i8, i8* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i8, i8* %arg2, i32 %i.025
-  %2 = load i8, i8* %arrayidx3, align 2
-  %conv = sext i8 %2 to i64
-  %conv4 = sext i8 %0 to i64
-  %mul = mul nsw i64 %conv, %conv4
-  %arrayidx6 = getelementptr inbounds i8, i8* %arg2, i32 %add
-  %3 = load i8, i8* %arrayidx6, align 2
-  %conv7 = sext i8 %3 to i64
-  %conv8 = sext i8 %1 to i64
-  %mul9 = mul nsw i64 %conv7, %conv8
-  %add10 = add i64 %mul, %mac1.026
-  %add11 = add i64 %add10, %mul9
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-

llvm/trunk/test/CodeGen/ARM/smlald2.ll

-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 < %s -arm-parallel-dsp -S | FileCheck %s
-;
-; The Cortex-M0 does not support unaligned accesses:
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-;
-; Check DSP extension:
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-
-define dso_local i64 @OneReduction(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-;
-; CHECK-LABEL: @OneReduction
-; CHECK:  %mac1{{\.}}026 = phi i64 [ [[V8:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V4:%[0-9]+]] = bitcast i16* %arrayidx3 to i32*
-; CHECK:  [[V5:%[0-9]+]] = load i32, i32* [[V4]], align 2
-; CHECK:  [[V6:%[0-9]+]] = bitcast i16* %arrayidx to i32*
-; CHECK:  [[V7:%[0-9]+]] = load i32, i32* [[V6]], align 2
-; CHECK:  [[V8]] = call i64 @llvm.arm.smlald(i32 [[V5]], i32 [[V7]], i64 %mac1{{\.}}026)
-; CHECK-NOT: call i64 @llvm.arm.smlald
-;
-; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlald
-;
-entry:
-  %cmp24 = icmp sgt i32 %arg, 0
-  br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
-
-for.body.preheader:
-  %.pre = load i16, i16* %arg3, align 2
-  %.pre27 = load i16, i16* %arg2, align 2
-  br label %for.body
-
-for.cond.cleanup:
-  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add11, %for.body ]
-  ret i64 %mac1.0.lcssa
-
-for.body:
-; One reduction statement here:
-  %mac1.026 = phi i64 [ %add11, %for.body ], [ 0, %for.body.preheader ]
-
-  %i.025 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.025
-  %0 = load i16, i16* %arrayidx, align 2
-  %add = add nuw nsw i32 %i.025, 1
-  %arrayidx1 = getelementptr inbounds i16, i16* %arg3, i32 %add
-  %1 = load i16, i16* %arrayidx1, align 2
-  %arrayidx3 = getelementptr inbounds i16, i16* %arg2, i32 %i.025
-  %2 = load i16, i16* %arrayidx3, align 2
-  %conv = sext i16 %2 to i32
-  %conv4 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv4
-  %sext0 = sext i32 %mul to i64
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg2, i32 %add
-  %3 = load i16, i16* %arrayidx6, align 2
-  %conv7 = sext i16 %3 to i32
-  %conv8 = sext i16 %1 to i32
-  %mul9 = mul nsw i32 %conv7, %conv8
-  %sext1 = sext i32 %mul9 to i64
-  %add10 = add i64 %sext0, %mac1.026
-
-; Here the Mul is the LHS, and the Add the RHS.
-  %add11 = add i64 %sext1, %add10
-
-  %exitcond = icmp ne i32 %add, %arg
-  br i1 %exitcond, label %for.body, label %for.cond.cleanup
-}
-
-define dso_local arm_aapcs_vfpcc i64 @TwoReductions(i32 %arg, i32* nocapture readnone %arg1, i16* nocapture readonly %arg2, i16* nocapture readonly %arg3) {
-;
-; CHECK-LABEL: @TwoReductions
-;
-; CHECK:  %mac1{{\.}}058 = phi i64 [ [[V10:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  %mac2{{\.}}057 = phi i64 [ [[V17:%[0-9]+]], %for.body ], [ 0, %for.body.preheader ]
-; CHECK:  [[V10]] = call i64 @llvm.arm.smlald(i32 %{{.*}}, i32 %{{.*}}, i64 %mac1{{\.}}058)
-; CHECK:  [[V17]] = call i64 @llvm.arm.smlald(i32 %{{.*}}, i32 %{{.*}}, i64 %mac2{{\.}}057)
-; CHECK-NOT: call i64 @llvm.arm.smlald
-;
-entry:
-  %cmp55 = icmp sgt i32 %arg, 0
-  br i1 %cmp55, label %for.body.preheader, label %for.cond.cleanup
-
-for.cond.cleanup:
-  %mac2.0.lcssa = phi i64 [ 0, %entry ], [ %add28, %for.body ]
-  %mac1.0.lcssa = phi i64 [ 0, %entry ], [ %add16, %for.body ]
-  %add30 = add nsw i64 %mac1.0.lcssa, %mac2.0.lcssa
-  ret i64 %add30
-
-for.body.preheader:
-  br label %for.body
-
-for.body:
-; And two reduction statements here:
-  %mac1.058 = phi i64 [ %add16, %for.body ], [ 0, %for.body.preheader ]
-  %mac2.057 = phi i64 [ %add28, %for.body ], [ 0, %for.body.preheader ]
-
-  %i.056 = phi i32 [ %add29, %for.body ], [ 0, %for.body.preheader ]
-  %arrayidx = getelementptr inbounds i16, i16* %arg3, i32 %i.056
-  %0 = load i16, i16* %arrayidx, align 2
-  %add1 = or i32 %i.056, 1
-  %arrayidx2 = getelementptr inbounds i16, i16* %arg3, i32 %add1
-  %1 = load i16, i16* %arrayidx2, align 2
-  %add3 = or i32 %i.056, 2
-  %arrayidx4 = getelementptr inbounds i16, i16* %arg3, i32 %add3
-  %2 = load i16, i16* %arrayidx4, align 2
-
-  %add5 = or i32 %i.056, 3
-  %arrayidx6 = getelementptr inbounds i16, i16* %arg3, i32 %add5
-  %3 = load i16, i16* %arrayidx6, align 2
-  %arrayidx8 = getelementptr inbounds i16, i16* %arg2, i32 %i.056
-  %4 = load i16, i16* %arrayidx8, align 2
-  %conv = sext i16 %4 to i32
-  %conv9 = sext i16 %0 to i32
-  %mul = mul nsw i32 %conv, %conv9
-  %sext0 = sext i32 %mul to i64
-  %arrayidx11 = getelementptr inbounds i16, i16* %arg2, i32 %add1
-  %5 = load i16, i16* %arrayidx11, align 2
-  %conv12 = sext i16 %5 to i32
-  %conv13 = sext i16 %1 to i32
-  %mul14 = mul nsw i32 %conv12, %conv13
-  %sext1 = sext i32 %mul14 to i64
-  %add15 = add i64 %sext0, %mac1.058
-  %add16 = add i64 %add15, %sext1
-  %arrayidx18 = getelementptr inbounds i16, i16* %arg2, i32 %add3
-  %6 = load i16, i16* %arrayidx18, align 2
-  %conv19 = sext i16 %6 to i32
-  %conv20 = sext i16 %2 to i32
-  %mul21 = mul nsw i32 %conv19, %conv20
-  %sext2 = sext i32 %mul21 to i64
-  %arrayidx23 = getelementptr inbounds i16, i16* %arg2, i32 %add5
-  %7 = load i16, i16* %arrayidx23, align 2
-  %conv24 = sext i16 %7 to i32
-  %conv25 = sext i16 %3 to i32
-  %mul26 = mul nsw i32 %conv24, %conv25
-  %sext3 = sext i32 %mul26 to i64
-  %add27 = add i64 %sext2, %mac2.057
-  %add28 = add i64 %add27, %sext3
-  %add29 = add nuw nsw i32 %i.056, 4
-  %cmp = icmp slt i32 %add29, %arg
-  br i1 %cmp, label %for.body, label %for.cond.cleanup
-}

llvm/trunk/test/CodeGen/ARM/smlaldx-1.ll

-; RUN: opt -mtriple=thumbv8m.main -mcpu=cortex-m33 -arm-parallel-dsp %s -S -o - | FileCheck %s
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-
-define i64 @smlaldx(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
-
-; CHECK-LABEL: smlaldx
-; CHECK: = phi i32 [ 0, %for.body.preheader.new ],
-; CHECK: [[ACC0:%[^ ]+]] = phi i64 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
-; CHECK: [[PIN23:%[^ ]+]] = bitcast i16* %pIn2.3 to i32*
-; CHECK: [[IN23:%[^ ]+]] = load i32, i32* [[PIN23]], align 2
-; CHECK: [[PIN12:%[^ ]+]] = bitcast i16* %pIn1.2 to i32*
-; CHECK: [[IN12:%[^ ]+]] = load i32, i32* [[PIN12]], align 2
-; CHECK: [[ACC1:%[^ ]+]] = call i64 @llvm.arm.smlaldx(i32 [[IN23]], i32 [[IN12]], i64 [[ACC0]])
-; CHECK: [[PIN21:%[^ ]+]] = bitcast i16* %pIn2.1 to i32*
-; CHECK: [[IN21:%[^ ]+]] = load i32, i32* [[PIN21]], align 2
-; CHECK: [[PIN10:%[^ ]+]] = bitcast i16* %pIn1.0 to i32*
-; CHECK: [[IN10:%[^ ]+]] = load i32, i32* [[PIN10]], align 2
-; CHECK: [[ACC2]] = call i64 @llvm.arm.smlaldx(i32 [[IN21]], i32 [[IN10]], i64 [[ACC1]])
-; CHECK-NOT: call i64 @llvm.arm.smlad
-; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlad
-
-entry:
-  %cmp9 = icmp eq i32 %limit, 0
-  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
-
-for.body.preheader:
-  %0 = add i32 %limit, -1
-  %xtraiter = and i32 %limit, 3
-  %1 = icmp ult i32 %0, 3
-  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
-
-for.body.preheader.new:
-  %unroll_iter = sub i32 %limit, %xtraiter
-  br label %for.body
-
-for.cond.cleanup.loopexit.unr-lcssa:
-  %add.lcssa.ph = phi i64 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
-  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
-  %sum.010.unr = phi i64 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
-  %lcmp.mod = icmp eq i32 %xtraiter, 0
-  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil
-
-for.body.epil:
-  %i.011.epil = phi i32 [ %inc.epil, %for.body.epil ], [ %i.011.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %sum.010.epil = phi i64 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %epil.iter = phi i32 [ %epil.iter.sub, %for.body.epil ], [ %xtraiter, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %sub.epil = sub i32 %j, %i.011.epil
-  %arrayidx.epil = getelementptr inbounds i16, i16* %pIn2, i32 %sub.epil
-  %2 = load i16, i16* %arrayidx.epil, align 2
-  %conv.epil = sext i16 %2 to i32
-  %arrayidx1.epil = getelementptr inbounds i16, i16* %pIn1, i32 %i.011.epil
-  %3 = load i16, i16* %arrayidx1.epil, align 2
-  %conv2.epil = sext i16 %3 to i32
-  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
-  %sext.mul.epil = sext i32 %mul.epil to i64
-  %add.epil = add nsw i64 %sext.mul.epil, %sum.010.epil
-  %inc.epil = add nuw i32 %i.011.epil, 1
-  %epil.iter.sub = add i32 %epil.iter, -1
-  %epil.iter.cmp = icmp eq i32 %epil.iter.sub, 0
-  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
-
-for.cond.cleanup:
-  %sum.0.lcssa = phi i64 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
-  ret i64 %sum.0.lcssa
-
-for.body:
-  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
-  %sum.010 = phi i64 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
-  %niter = phi i32 [ %unroll_iter, %for.body.preheader.new ], [ %niter.nsub.3, %for.body ]
-  %pIn2Base = phi i16* [ %pIn2, %for.body.preheader.new ], [ %pIn2.4, %for.body ]
-  %pIn2.0 = getelementptr inbounds i16, i16* %pIn2Base, i32 0
-  %In2 = load i16, i16* %pIn2.0, align 2
-  %pIn1.0 = getelementptr inbounds i16, i16* %pIn1, i32 %i.011
-  %In1 = load i16, i16* %pIn1.0, align 2
-  %inc = or i32 %i.011, 1
-  %pIn2.1 = getelementptr inbounds i16, i16* %pIn2Base, i32 -1
-  %In2.1 = load i16, i16* %pIn2.1, align 2
-  %pIn1.1 = getelementptr inbounds i16, i16* %pIn1, i32 %inc
-  %In1.1 = load i16, i16* %pIn1.1, align 2
-  %inc.1 = or i32 %i.011, 2
-  %pIn2.2 = getelementptr inbounds i16, i16* %pIn2Base, i32 -2
-  %In2.2 = load i16, i16* %pIn2.2, align 2
-  %pIn1.2 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.1
-  %In1.2 = load i16, i16* %pIn1.2, align 2
-  %inc.2 = or i32 %i.011, 3
-  %pIn2.3 = getelementptr inbounds i16, i16* %pIn2Base, i32 -3
-  %In2.3 = load i16, i16* %pIn2.3, align 2
-  %pIn1.3 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.2
-  %In1.3 = load i16, i16* %pIn1.3, align 2
-  %sextIn1 = sext i16 %In1 to i32
-  %sextIn1.1 = sext i16 %In1.1 to i32
-  %sextIn1.2 = sext i16 %In1.2 to i32
-  %sextIn1.3 = sext i16 %In1.3 to i32
-  %sextIn2 = sext i16 %In2 to i32
-  %sextIn2.1 = sext i16 %In2.1 to i32
-  %sextIn2.2 = sext i16 %In2.2 to i32
-  %sextIn2.3 = sext i16 %In2.3 to i32
-  %mul = mul nsw i32 %sextIn1, %sextIn2
-  %mul.1 = mul nsw i32 %sextIn1.1, %sextIn2.1
-  %mul.2 = mul nsw i32 %sextIn1.2, %sextIn2.2
-  %mul.3 = mul nsw i32 %sextIn1.3, %sextIn2.3
-  %sext.mul = sext i32 %mul to i64
-  %sext.mul.1 = sext i32 %mul.1 to i64
-  %sext.mul.2 = sext i32 %mul.2 to i64
-  %sext.mul.3 = sext i32 %mul.3 to i64
-  %add = add nsw i64 %sext.mul, %sum.010
-  %add.1 = add nsw i64 %sext.mul.1, %add
-  %add.2 = add nsw i64 %sext.mul.2, %add.1
-  %add.3 = add nsw i64 %sext.mul.3, %add.2
-  %inc.3 = add i32 %i.011, 4
-  %pIn2.4 = getelementptr inbounds i16, i16* %pIn2Base, i32 -4
-  %niter.nsub.3 = add i32 %niter, -4
-  %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
-  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
-}
-
-define i64 @smlaldx_swap(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
-
-entry:
-  %cmp9 = icmp eq i32 %limit, 0
-  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
-
-for.body.preheader:
-  %0 = add i32 %limit, -1
-  %xtraiter = and i32 %limit, 3
-  %1 = icmp ult i32 %0, 3
-  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
-
-for.body.preheader.new:
-  %unroll_iter = sub i32 %limit, %xtraiter
-  %scevgep6 = getelementptr i16, i16* %pIn1, i32 2
-  %2 = add i32 %j, -1
-  %scevgep11 = getelementptr i16, i16* %pIn2, i32 %2
-  br label %for.body
-
-for.cond.cleanup.loopexit.unr-lcssa:
-  %add.lcssa.ph = phi i64 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
-  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
-  %sum.010.unr = phi i64 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
-  %lcmp.mod = icmp eq i32 %xtraiter, 0
-  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil.preheader
-
-for.body.epil.preheader:
-  %scevgep = getelementptr i16, i16* %pIn1, i32 %i.011.unr
-  %3 = sub i32 %j, %i.011.unr
-  %scevgep2 = getelementptr i16, i16* %pIn2, i32 %3
-  %4 = sub i32 0, %xtraiter
-  br label %for.body.epil
-
-for.body.epil:
-  %lsr.iv5 = phi i32 [ %4, %for.body.epil.preheader ], [ %lsr.iv.next, %for.body.epil ]
-  %lsr.iv3 = phi i16* [ %scevgep2, %for.body.epil.preheader ], [ %scevgep4, %for.body.epil ]
-  %lsr.iv = phi i16* [ %scevgep, %for.body.epil.preheader ], [ %scevgep1, %for.body.epil ]
-  %sum.010.epil = phi i64 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.body.epil.preheader ]
-  %5 = load i16, i16* %lsr.iv3, align 2
-  %conv.epil = sext i16 %5 to i32
-  %6 = load i16, i16* %lsr.iv, align 2
-  %conv2.epil = sext i16 %6 to i32
-  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
-  %sext.mul.epil = sext i32 %mul.epil to i64
-  %add.epil = add nsw i64 %sext.mul.epil, %sum.010.epil
-  %scevgep1 = getelementptr i16, i16* %lsr.iv, i32 1
-  %scevgep4 = getelementptr i16, i16* %lsr.iv3, i32 -1
-  %lsr.iv.next = add nsw i32 %lsr.iv5, 1
-  %epil.iter.cmp = icmp eq i32 %lsr.iv.next, 0
-  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
-
-for.cond.cleanup:
-  %sum.0.lcssa = phi i64 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
-  ret i64 %sum.0.lcssa
-
-; CHECK-LABEL: smlaldx_swap
-; CHECK: for.body.preheader.new:
-; CHECK: [[PIN1Base:[^ ]+]] = getelementptr i16, i16* %pIn1
-; CHECK: [[PIN2Base:[^ ]+]] = getelementptr i16, i16* %pIn2
-
-; CHECK: for.body:
-; CHECK: [[PIN2:%[^ ]+]] = phi i16* [ [[PIN2_NEXT:%[^ ]+]], %for.body ], [ [[PIN2Base]], %for.body.preheader.new ]
-; CHECK: [[PIN1:%[^ ]+]] = phi i16* [ [[PIN1_NEXT:%[^ ]+]], %for.body ], [ [[PIN1Base]], %for.body.preheader.new ]
-; CHECK: [[IV:%[^ ]+]] = phi i32
-; CHECK: [[ACC0:%[^ ]+]] = phi i64 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
-
-; CHECK: [[PIN1_2:%[^ ]+]] = getelementptr i16, i16* [[PIN1]], i32 -2
-; CHECK: [[PIN2_2:%[^ ]+]] = getelementptr i16, i16* [[PIN2]], i32 -2
-
-; CHECK: [[PIN2_2_CAST:%[^ ]+]] = bitcast i16* [[PIN2_2]] to i32*
-; CHECK: [[IN2_2:%[^ ]+]] = load i32, i32* [[PIN2_2_CAST]], align 2
-; CHECK: [[PIN1_CAST:%[^ ]+]] = bitcast i16* [[PIN1]] to i32*
-; CHECK: [[IN1:%[^ ]+]] = load i32, i32* [[PIN1_CAST]], align 2
-; CHECK: [[ACC1:%[^ ]+]] = call i64 @llvm.arm.smlaldx(i32 [[IN2_2]], i32 [[IN1]], i64 [[ACC0]])
-
-; CHECK: [[PIN2_CAST:%[^ ]+]] = bitcast i16* [[PIN2]] to i32*
-; CHECK: [[IN2:%[^ ]+]] = load i32, i32* [[PIN2_CAST]], align 2
-; CHECK: [[PIN1_2_CAST:%[^ ]+]] = bitcast i16* [[PIN1_2]] to i32*
-; CHECK: [[IN1_2:%[^ ]+]] = load i32, i32* [[PIN1_2_CAST]], align 2
-; CHECK: [[ACC2]] = call i64 @llvm.arm.smlaldx(i32 [[IN2]], i32 [[IN1_2]], i64 [[ACC1]])
-
-; CHECK: [[PIN1_NEXT]] = getelementptr i16, i16* [[PIN1]], i32 4
-; CHECK: [[PIN2_NEXT]] = getelementptr i16, i16* [[PIN2]], i32 -4
-
-; CHECK-NOT: call i64 @llvm.arm.smlad
-; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlad
-
-for.body:
-  %pin2 = phi i16* [ %pin2.sub4, %for.body ], [ %scevgep11, %for.body.preheader.new ]
-  %pin1 = phi i16* [ %pin1.add4, %for.body ], [ %scevgep6, %for.body.preheader.new ]
-  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
-  %sum.010 = phi i64 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
-  %pin2.add1 = getelementptr i16, i16* %pin2, i32 1
-  %In2 = load i16, i16* %pin2.add1, align 2
-  %pin1.sub2 = getelementptr i16, i16* %pin1, i32 -2
-  %In1 = load i16, i16* %pin1.sub2, align 2
-  %In2.1 = load i16, i16* %pin2, align 2
-  %pin1.sub1 = getelementptr i16, i16* %pin1, i32 -1
-  %In1.1 = load i16, i16* %pin1.sub1, align 2
-  %pin2.sub1 = getelementptr i16, i16* %pin2, i32 -1
-  %In2.2 = load i16, i16* %pin2.sub1, align 2
-  %In1.2 = load i16, i16* %pin1, align 2
-  %pin2.sub2 = getelementptr i16, i16* %pin2, i32 -2
-  %In2.3 = load i16, i16* %pin2.sub2, align 2
-  %pin1.add1 = getelementptr i16, i16* %pin1, i32 1
-  %In1.3 = load i16, i16* %pin1.add1, align 2
-  %sextIn2 = sext i16 %In2 to i32
-  %sextIn1 = sext i16 %In1 to i32
-  %sextIn2.1 = sext i16 %In2.1 to i32
-  %sextIn1.1 = sext i16 %In1.1 to i32
-  %sextIn2.2 = sext i16 %In2.2 to i32
-  %sextIn1.2 = sext i16 %In1.2 to i32
-  %sextIn2.3 = sext i16 %In2.3 to i32
-  %sextIn1.3 = sext i16 %In1.3 to i32
-  %mul = mul nsw i32 %sextIn2, %sextIn1
-  %sext.mul = sext i32 %mul to i64
-  %add = add nsw i64 %sext.mul, %sum.010
-  %mul.1 = mul nsw i32 %sextIn2.1, %sextIn1.1
-  %sext.mul.1 = sext i32 %mul.1 to i64
-  %add.1 = add nsw i64 %sext.mul.1, %add
-  %mul.2 = mul nsw i32 %sextIn2.2, %sextIn1.2
-  %sext.mul.2 = sext i32 %mul.2 to i64
-  %add.2 = add nsw i64 %sext.mul.2, %add.1
-  %mul.3 = mul nsw i32 %sextIn2.3, %sextIn1.3
-  %sext.mul.3 = sext i32 %mul.3 to i64
-  %add.3 = add nsw i64 %sext.mul.3, %add.2
-  %inc.3 = add i32 %i.011, 4
-  %pin1.add4 = getelementptr i16, i16* %pin1, i32 4
-  %pin2.sub4 = getelementptr i16, i16* %pin2, i32 -4
-  %niter.ncmp.3 = icmp eq i32 %unroll_iter, %inc.3
-  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
-}

llvm/trunk/test/CodeGen/ARM/smlaldx-2.ll

-; RUN: opt -mtriple=thumbv8m.main -mcpu=cortex-m33 -arm-parallel-dsp %s -S -o - | FileCheck %s
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m0 < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-; RUN: opt -mtriple=arm-arm-eabi -mcpu=cortex-m33 -mattr=-dsp < %s -arm-parallel-dsp -S | FileCheck %s --check-prefix=CHECK-UNSUPPORTED
-
-define i64 @smlaldx(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
-
-; CHECK-LABEL: smlaldx
-; CHECK: = phi i32 [ 0, %for.body.preheader.new ],
-; CHECK: [[ACC0:%[^ ]+]] = phi i64 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
-; CHECK: [[PIN23:%[^ ]+]] = bitcast i16* %pIn2.3 to i32*
-; CHECK: [[IN23:%[^ ]+]] = load i32, i32* [[PIN23]], align 2
-; CHECK: [[PIN12:%[^ ]+]] = bitcast i16* %pIn1.2 to i32*
-; CHECK: [[IN12:%[^ ]+]] = load i32, i32* [[PIN12]], align 2
-; CHECK: [[ACC1:%[^ ]+]] = call i64 @llvm.arm.smlaldx(i32 [[IN23]], i32 [[IN12]], i64 [[ACC0]])
-; CHECK: [[PIN21:%[^ ]+]] = bitcast i16* %pIn2.1 to i32*
-; CHECK: [[IN21:%[^ ]+]] = load i32, i32* [[PIN21]], align 2
-; CHECK: [[PIN10:%[^ ]+]] = bitcast i16* %pIn1.0 to i32*
-; CHECK: [[IN10:%[^ ]+]] = load i32, i32* [[PIN10]], align 2
-; CHECK: [[ACC2]] = call i64 @llvm.arm.smlaldx(i32 [[IN21]], i32 [[IN10]], i64 [[ACC1]])
-; CHECK-NOT: call i64 @llvm.arm.smlad
-; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlad
-
-entry:
-  %cmp9 = icmp eq i32 %limit, 0
-  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
-
-for.body.preheader:
-  %0 = add i32 %limit, -1
-  %xtraiter = and i32 %limit, 3
-  %1 = icmp ult i32 %0, 3
-  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
-
-for.body.preheader.new:
-  %unroll_iter = sub i32 %limit, %xtraiter
-  br label %for.body
-
-for.cond.cleanup.loopexit.unr-lcssa:
-  %add.lcssa.ph = phi i64 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
-  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
-  %sum.010.unr = phi i64 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
-  %lcmp.mod = icmp eq i32 %xtraiter, 0
-  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil
-
-for.body.epil:
-  %i.011.epil = phi i32 [ %inc.epil, %for.body.epil ], [ %i.011.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %sum.010.epil = phi i64 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %epil.iter = phi i32 [ %epil.iter.sub, %for.body.epil ], [ %xtraiter, %for.cond.cleanup.loopexit.unr-lcssa ]
-  %sub.epil = sub i32 %j, %i.011.epil
-  %arrayidx.epil = getelementptr inbounds i16, i16* %pIn2, i32 %sub.epil
-  %2 = load i16, i16* %arrayidx.epil, align 2
-  %conv.epil = sext i16 %2 to i32
-  %arrayidx1.epil = getelementptr inbounds i16, i16* %pIn1, i32 %i.011.epil
-  %3 = load i16, i16* %arrayidx1.epil, align 2
-  %conv2.epil = sext i16 %3 to i32
-  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
-  %sext.mul.epil = sext i32 %mul.epil to i64
-  %add.epil = add nsw i64 %sext.mul.epil, %sum.010.epil
-  %inc.epil = add nuw i32 %i.011.epil, 1
-  %epil.iter.sub = add i32 %epil.iter, -1
-  %epil.iter.cmp = icmp eq i32 %epil.iter.sub, 0
-  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
-
-for.cond.cleanup:
-  %sum.0.lcssa = phi i64 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
-  ret i64 %sum.0.lcssa
-
-for.body:
-  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
-  %sum.010 = phi i64 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
-  %niter = phi i32 [ %unroll_iter, %for.body.preheader.new ], [ %niter.nsub.3, %for.body ]
-  %pIn2Base = phi i16* [ %pIn2, %for.body.preheader.new ], [ %pIn2.4, %for.body ]
-  %pIn2.0 = getelementptr inbounds i16, i16* %pIn2Base, i32 0
-  %In2 = load i16, i16* %pIn2.0, align 2
-  %pIn1.0 = getelementptr inbounds i16, i16* %pIn1, i32 %i.011
-  %In1 = load i16, i16* %pIn1.0, align 2
-  %inc = or i32 %i.011, 1
-  %pIn2.1 = getelementptr inbounds i16, i16* %pIn2Base, i32 -1
-  %In2.1 = load i16, i16* %pIn2.1, align 2
-  %pIn1.1 = getelementptr inbounds i16, i16* %pIn1, i32 %inc
-  %In1.1 = load i16, i16* %pIn1.1, align 2
-  %inc.1 = or i32 %i.011, 2
-  %pIn2.2 = getelementptr inbounds i16, i16* %pIn2Base, i32 -2
-  %In2.2 = load i16, i16* %pIn2.2, align 2
-  %pIn1.2 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.1
-  %In1.2 = load i16, i16* %pIn1.2, align 2
-  %inc.2 = or i32 %i.011, 3
-  %pIn2.3 = getelementptr inbounds i16, i16* %pIn2Base, i32 -3
-  %In2.3 = load i16, i16* %pIn2.3, align 2
-  %pIn1.3 = getelementptr inbounds i16, i16* %pIn1, i32 %inc.2
-  %In1.3 = load i16, i16* %pIn1.3, align 2
-  %sextIn1 = sext i16 %In1 to i32
-  %sextIn1.1 = sext i16 %In1.1 to i32
-  %sextIn1.2 = sext i16 %In1.2 to i32
-  %sextIn1.3 = sext i16 %In1.3 to i32
-  %sextIn2 = sext i16 %In2 to i32
-  %sextIn2.1 = sext i16 %In2.1 to i32
-  %sextIn2.2 = sext i16 %In2.2 to i32
-  %sextIn2.3 = sext i16 %In2.3 to i32
-  %mul = mul nsw i32 %sextIn1, %sextIn2
-  %mul.1 = mul nsw i32 %sextIn1.1, %sextIn2.1
-  %mul.2 = mul nsw i32 %sextIn1.2, %sextIn2.2
-  %mul.3 = mul nsw i32 %sextIn1.3, %sextIn2.3
-  %sext.mul = sext i32 %mul to i64
-  %sext.mul.1 = sext i32 %mul.1 to i64
-  %sext.mul.2 = sext i32 %mul.2 to i64
-  %sext.mul.3 = sext i32 %mul.3 to i64
-  %add = add nsw i64 %sum.010, %sext.mul
-  %add.1 = add nsw i64 %sext.mul.1, %add
-  %add.2 = add nsw i64 %add.1, %sext.mul.2
-  %add.3 = add nsw i64 %sext.mul.3, %add.2
-  %inc.3 = add i32 %i.011, 4
-  %pIn2.4 = getelementptr inbounds i16, i16* %pIn2Base, i32 -4
-  %niter.nsub.3 = add i32 %niter, -4
-  %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
-  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
-}
-
-define i64 @smlaldx_swap(i16* nocapture readonly %pIn1, i16* nocapture readonly %pIn2, i32 %j, i32 %limit) {
-
-entry:
-  %cmp9 = icmp eq i32 %limit, 0
-  br i1 %cmp9, label %for.cond.cleanup, label %for.body.preheader
-
-for.body.preheader:
-  %0 = add i32 %limit, -1
-  %xtraiter = and i32 %limit, 3
-  %1 = icmp ult i32 %0, 3
-  br i1 %1, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body.preheader.new
-
-for.body.preheader.new:
-  %unroll_iter = sub i32 %limit, %xtraiter
-  %scevgep6 = getelementptr i16, i16* %pIn1, i32 2
-  %2 = add i32 %j, -1
-  %scevgep11 = getelementptr i16, i16* %pIn2, i32 %2
-  br label %for.body
-
-for.cond.cleanup.loopexit.unr-lcssa:
-  %add.lcssa.ph = phi i64 [ undef, %for.body.preheader ], [ %add.3, %for.body ]
-  %i.011.unr = phi i32 [ 0, %for.body.preheader ], [ %inc.3, %for.body ]
-  %sum.010.unr = phi i64 [ 0, %for.body.preheader ], [ %add.3, %for.body ]
-  %lcmp.mod = icmp eq i32 %xtraiter, 0
-  br i1 %lcmp.mod, label %for.cond.cleanup, label %for.body.epil.preheader
-
-for.body.epil.preheader:
-  %scevgep = getelementptr i16, i16* %pIn1, i32 %i.011.unr
-  %3 = sub i32 %j, %i.011.unr
-  %scevgep2 = getelementptr i16, i16* %pIn2, i32 %3
-  %4 = sub i32 0, %xtraiter
-  br label %for.body.epil
-
-for.body.epil:
-  %lsr.iv5 = phi i32 [ %4, %for.body.epil.preheader ], [ %lsr.iv.next, %for.body.epil ]
-  %lsr.iv3 = phi i16* [ %scevgep2, %for.body.epil.preheader ], [ %scevgep4, %for.body.epil ]
-  %lsr.iv = phi i16* [ %scevgep, %for.body.epil.preheader ], [ %scevgep1, %for.body.epil ]
-  %sum.010.epil = phi i64 [ %add.epil, %for.body.epil ], [ %sum.010.unr, %for.body.epil.preheader ]
-  %5 = load i16, i16* %lsr.iv3, align 2
-  %conv.epil = sext i16 %5 to i32
-  %6 = load i16, i16* %lsr.iv, align 2
-  %conv2.epil = sext i16 %6 to i32
-  %mul.epil = mul nsw i32 %conv2.epil, %conv.epil
-  %sext.mul.epil = sext i32 %mul.epil to i64
-  %add.epil = add nsw i64 %sext.mul.epil, %sum.010.epil
-  %scevgep1 = getelementptr i16, i16* %lsr.iv, i32 1
-  %scevgep4 = getelementptr i16, i16* %lsr.iv3, i32 -1
-  %lsr.iv.next = add nsw i32 %lsr.iv5, 1
-  %epil.iter.cmp = icmp eq i32 %lsr.iv.next, 0
-  br i1 %epil.iter.cmp, label %for.cond.cleanup, label %for.body.epil
-
-for.cond.cleanup:
-  %sum.0.lcssa = phi i64 [ 0, %entry ], [ %add.lcssa.ph, %for.cond.cleanup.loopexit.unr-lcssa ], [ %add.epil, %for.body.epil ]
-  ret i64 %sum.0.lcssa
-
-; CHECK-LABEL: smlaldx_swap
-; CHECK: for.body.preheader.new:
-; CHECK: [[PIN1Base:[^ ]+]] = getelementptr i16, i16* %pIn1
-; CHECK: [[PIN2Base:[^ ]+]] = getelementptr i16, i16* %pIn2
-
-; CHECK: for.body:
-; CHECK: [[PIN2:%[^ ]+]] = phi i16* [ [[PIN2_NEXT:%[^ ]+]], %for.body ], [ [[PIN2Base]], %for.body.preheader.new ]
-; CHECK: [[PIN1:%[^ ]+]] = phi i16* [ [[PIN1_NEXT:%[^ ]+]], %for.body ], [ [[PIN1Base]], %for.body.preheader.new ]
-; CHECK: [[IV:%[^ ]+]] = phi i32
-; CHECK: [[ACC0:%[^ ]+]] = phi i64 [ 0, %for.body.preheader.new ], [ [[ACC2:%[^ ]+]], %for.body ]
-; CHECK: [[PIN1_2:%[^ ]+]] = getelementptr i16, i16* [[PIN1]], i32 -2
-; CHECK: [[PIN2_2:%[^ ]+]] = getelementptr i16, i16* [[PIN2]], i32 -2
-
-; CHECK: [[PIN2_CAST:%[^ ]+]] = bitcast i16* [[PIN2]] to i32*
-; CHECK: [[IN2:%[^ ]+]] = load i32, i32* [[PIN2_CAST]], align 2
-; CHECK: [[PIN1_2_CAST:%[^ ]+]] = bitcast i16* [[PIN1_2]] to i32*
-; CHECK: [[IN1_2:%[^ ]+]] = load i32, i32* [[PIN1_2_CAST]], align 2
-; CHECK: [[ACC1:%[^ ]+]] = call i64 @llvm.arm.smlaldx(i32 [[IN2]], i32 [[IN1_2]], i64 [[ACC0]])
-
-; CHECK: [[PIN1_CAST:%[^ ]+]] = bitcast i16* [[PIN1]] to i32*
-; CHECK: [[IN1:%[^ ]+]] = load i32, i32* [[PIN1_CAST]], align 2
-; CHECK: [[PIN2_2_CAST:%[^ ]+]] = bitcast i16* [[PIN2_2]] to i32*
-; CHECK: [[IN2_2:%[^ ]+]] = load i32, i32* [[PIN2_2_CAST]], align 2
-; CHECK: [[ACC2]] = call i64 @llvm.arm.smlaldx(i32 [[IN1]], i32 [[IN2_2]], i64 [[ACC1]])
-
-; CHECK: [[PIN1_NEXT]] = getelementptr i16, i16* [[PIN1]], i32 4
-; CHECK: [[PIN2_NEXT]] = getelementptr i16, i16* [[PIN2]], i32 -4
-
-; CHECK-NOT: call i64 @llvm.arm.smlad
-; CHECK-UNSUPPORTED-NOT:  call i64 @llvm.arm.smlad
-
-for.body:
-  %pin2 = phi i16* [ %pin2.sub4, %for.body ], [ %scevgep11, %for.body.preheader.new ]
-  %pin1 = phi i16* [ %pin1.add4, %for.body ], [ %scevgep6, %for.body.preheader.new ]
-  %i.011 = phi i32 [ 0, %for.body.preheader.new ], [ %inc.3, %for.body ]
-  %sum.010 = phi i64 [ 0, %for.body.preheader.new ], [ %add.3, %for.body ]
-  %pin2.add1 = getelementptr i16, i16* %pin2, i32 1
-  %In2 = load i16, i16* %pin2.add1, align 2
-  %pin1.sub2 = getelementptr i16, i16* %pin1, i32 -2
-  %In1 = load i16, i16* %pin1.sub2, align 2
-  %In2.1 = load i16, i16* %pin2, align 2
-  %pin1.sub1 = getelementptr i16, i16* %pin1, i32 -1
-  %In1.1 = load i16, i16* %pin1.sub1, align 2
-  %pin2.sub1 = getelementptr i16, i16* %pin2, i32 -1
-  %In2.2 = load i16, i16* %pin2.sub1, align 2
-  %In1.2 = load i16, i16* %pin1, align 2
-  %pin2.sub2 = getelementptr i16, i16* %pin2, i32 -2
-  %In2.3 = load i16, i16* %pin2.sub2, align 2
-  %pin1.add1 = getelementptr i16, i16* %pin1, i32 1
-  %In1.3 = load i16, i16* %pin1.add1, align 2
-  %sextIn2 = sext i16 %In2 to i32
-  %sextIn1 = sext i16 %In1 to i32
-  %sextIn2.1 = sext i16 %In2.1 to i32
-  %sextIn1.1 = sext i16 %In1.1 to i32
-  %sextIn2.2 = sext i16 %In2.2 to i32
-  %sextIn1.2 = sext i16 %In1.2 to i32
-  %sextIn2.3 = sext i16 %In2.3 to i32
-  %sextIn1.3 = sext i16 %In1.3 to i32
-  %mul = mul nsw i32 %sextIn2, %sextIn1
-  %sext.mul = sext i32 %mul to i64
-  %add = add nsw i64 %sext.mul, %sum.010
-  %mul.1 = mul nsw i32 %sextIn2.1, %sextIn1.1
-  %sext.mul.1 = sext i32 %mul.1 to i64
-  %add.1 = add nsw i64 %sext.mul.1, %add
-  %mul.2 = mul nsw i32 %sextIn2.2, %sextIn1.2
-  %sext.mul.2 = sext i32 %mul.2 to i64
-  %add.2 = add nsw i64 %add.1, %sext.mul.2
-  %mul.3 = mul nsw i32 %sextIn2.3, %sextIn1.3
-  %sext.mul.3 = sext i32 %mul.3 to i64
-  %add.3 = add nsw i64 %add.2, %sext.mul.3
-  %inc.3 = add i32 %i.011, 4
-  %pin1.add4 = getelementptr i16, i16* %pin1, i32 4
-  %pin2.sub4 = getelementptr i16, i16* %pin2, i32 -4
-  %niter.ncmp.3 = icmp eq i32 %unroll_iter, %inc.3
-  br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
-}