Reducing the costs of cast instructions to enable more vectorization of smaller types in LoopVectorize

[samparker]

Summary

Hi,

So I have completely rewritten the patch so that it doesn't use getWidestType and that it searches for clamped values and instructions. I have reversed the search through the loop body so that trunc instructions can be searched up from to the zext/sext nodes, noting smaller type requirements.

Cheers,
Sam

[sbaranga]

Hi Sam,

If I understand correctly, this implementation allows for only s/zext -> op -> trunc chains (with only one operation). Is this correct (or perhaps I'm missing something)? I think in theory it should be possible to have arbitrary long chains, depending on what operations are in them.

-Silviu

lib/Transforms/Vectorize/LoopVectorize.cpp

4519

APInt::getMaxValue(ScalarWidth) should be used as (1 << ScalarWidth) - 1 might return different values depending on the host machine.

test/Transforms/LoopVectorize/AArch64/loop-vectorization-factors.ll

140

I think this would be cleaner without the metadata.

[samparker]

Hi Silviu,

Yes, that is how is it implemented at the moment and I agree longer chains should be considered. I just wasn't sure what types of nodes it wouldn't be legal for and would welcome suggestions.

Cheers,
sam

[samparker]

Now use SmallPtrSet and ValueMap in place of the standard containers for the instructions and types found by the new function.
Also compare against APInt::getMaxValue as Silviu suggested.
Removed metadata from test file.

[jmolloy]

Hi Sam,

I've given an initial review, but I'm not sure this will catch all the interesting cases.

You don't recurse anywhere, so you only check single operations. For example, you'll miss this:

%1 = zext i32 %a to i64
%2 = add %1, %1
%3 = add %2, %1
%4 = trunc i64 %3 to i32

There are more complex cases it won't catch either, but I'm less worried about that as primarily this comes up with C promotion rules, so everything gets promoted to the same size (i32) which simplifies things somewhat.

Cheers,

James

lib/Transforms/Vectorize/LoopVectorize.cpp

891

This needs doxygen comment style (///) and also a description of what a clamped type is.

895

This needs documentation. Does it assume I is a CastInst?

4379

Whitespace change.

4387

Please add a comment as to why we're reversing through the loop.

4484

There's no need for this to be split over two lines - the "bool" will fit inside the 80 char limit.

4490

"so any extends would have been found earlier." - SPAG and clarification

4491

The canonical way to do this is "if (!isa<TruncInst>(I))"

4492

This can simply be "return FreeCasts.count(I);"

4498

I know it's pedantic, but this comment implies that it was the user's fault that we were called on a non-integer operation. Actually, we just don't deal with them, so I'd suggest rewriting to something like:

"// We only care about integer operations."

4514

Canonical form has local variables as UpperCamelCase.

4516

You can merge this and the next line with:

if (ConstantInst *CI = dyn_cast<ConstantInst>(ChainInst->getOperand(i)))

[samparker]

Chains of instructions are now searched through, with the limitations that they are either an Add, And, Or, Xor, Mul or Shl.
The instructions are saved in a map with their clamped value type and so during vectorization, these types can be used instead.
This is performed by enclosing the binary operations within truncs and zext operations in the hope that these get optimised out.

[jmolloy]

Hi Sam,

Thanks for making this update. I have lots of coding style nits but primarily we need two algorithms changed from being purely recursive to being iterative. They'll still do the same thing just without overflowing the stack on large programs.

Cheers,

James

lib/Transforms/Vectorize/LoopVectorize.cpp

2639

This is recursive, and unboundedly so. This means that with a large function, we may blow the stack.

Instead of recursing, you could change this to be a simple iterative algorithm. Just iterate as you are currently doing, but call visitClampedInstr() for ALL instructions that are clamped, not just TruncInsts.

Store the newly created clamped instructions in a map<Value*,Value*>, and when you're visiting a clamped instruction check the map for each of its operands. If a value exists, we've already visited this - use it. Otherwise insert a cast.

Then, when we hit a trunc instruction, do the replacement (using OldTrunc->replaceAllUsesWith(NewTrunc)) and that'll make the entire tree live.

This will only really work if you visit all instructions in dominator order, so we visit defs before uses. Luckily the caller of this function already finds such an ordering (LoopBlocksDFS), so you can just use that.

4581

This is also unboundedly recursive. But it can be changed into an iterative algorithm.

Instead of searching bottom-up, search top-down. For every instruction (visiting defs before uses, so in LoopBlocksDFS order!), check if it is narrow. That's easy because you just need to check its immediate operands. If so, store it.

Because we visit all instructions, and we visit all defs before uses, any single isNarrowInstruction() call only needs to check its immediate operands - it doesn't need to go crawling through a tree. This removes the recursion and makes for a more elegant algorithm.

[mzolotukhin]

Hi Sam,

Some initial comments from me inline:

lib/Transforms/Vectorize/LoopVectorize.cpp

952–953

Please add a comment that the key in this map is VectorizationFactor.

2601–2602

Should it be if(Clamped != *II) ? Since Clamped originally is just an operand of I, this condition is almost always true, or am I missing something?

2612

Is it used anywhere?

4149

Please remove this unneeded change.

4478–4480

Please reword the comment so that it doesn't refer to the previous version: e.g. instead of "This has been reversed.." use something like "iterate in reverse order because..".
In the current form the comment makes sense in the context of the patch, but it would look a bit strange when you don't see the context.

4481

Maybe use auto here?

4484

This change looks strange. Is it correct at all?

4850–4851

else if should be on the same line as }.
Actually, you could just run clang-format on the patch to avoid such issues.

[samparker]

I have changed the algorithms so that they are not recursive. The first pass checks all the instructions top-down and assigns a potential (TBC) narrow type. The second pass then searches bottom up from truncs to confirm, or adjust, the narrow type. I've also added support for sub, ashr and lshr instructions, but would really appreciate my logic being confirmed.

[sbaranga]

Hi Sam,

Some comments from me inline.

Cheers,
Silviu

lib/Transforms/Vectorize/LoopVectorize.cpp

1212

I think the name "TBCNarrowInstrs" is potentially confusing. Maybe TBC -> Candidate?

5236

Why use inline here? I would normally trust the compiler to do the right thing.

5298

The check for sub could potentially be removed, but it depends on exactly what we're checking for.
It would be interesting to have an example where sub is an issue that would prevent the narrowing.

5309

The comment says "logical shifts" but the code also allows ashr? Is this correct?

5386

It would be a good idea to have a comment that explains how these were chosen, and why some of them need extra checking.

5420

Why consider here only narrowing to i8 or i16? Would narrowing from i64 to i32 also make sense?

[samparker]

Hi Silviu,

So i've added some comments for clarification and also removed the sub operands from the chains for simplicity as I didn't notice any performance difference in the tests I was looking at. I've also added support for 64-bit > 32-bit operations and updated the test file to reflect this.

cheers,

[sbaranga]

In D9822#201852, @samparker wrote:

Hi Silviu,

So i've added some comments for clarification and also removed the sub operands from the chains for simplicity as I didn't notice any performance difference in the tests I was looking at. I've also added support for 64-bit > 32-bit operations and updated the test file to reflect this.

cheers,

Thanks Sam!

Some further comments:

Subtraction is a very common operation, it would be much nicer to get it now. wrt performance numbers this would only show up in a benchmark if it would affect the hot loop _and_ the loop would end up being vectorized.

Looking at what bits we don't care about seems like the right approach to me. I think an easier way to reason about these is to have a bitmask for each value in the chain that we're looking at, indicating what bits we care about. Different operations would do different things on this bitmask, and we would compute the value of the bitmasks starting from the trunc operations.

For example:

b = add a, c means that a and c have the same bitmask as b

The same would be true for mul, left shift, all bitwise operators, and sub (since sub can be written with and add and a not).

b = shr a, 2,  and b has a bitmask of 0b00011 ( where 1 means we care, 0 don't care) means a has a bitmask of 0b01111 (technically it would be 0b01100, but this is an approximation).

This can probably be done for most operators.

Also, if you have an operation that takes a constant and you know what bits you care about it should be perfectly legal to truncate the constant and get rid of some of the bits that you don't care about.

Technically, we could set the mask values on the truncate operations and iterate using a work list until all bitmasks have converged, but that is best left for future work.

What would be your opinion on this?

Cheers,
Silviu

[samparker]

In D9822#202874, @sbaranga wrote:

In D9822#201852, @samparker wrote:

Hi Silviu,

So i've added some comments for clarification and also removed the sub operands from the chains for simplicity as I didn't notice any performance difference in the tests I was looking at. I've also added support for 64-bit > 32-bit operations and updated the test file to reflect this.

cheers,

Thanks Sam!

Some further comments:

Subtraction is a very common operation, it would be much nicer to get it now. wrt performance numbers this would only show up in a benchmark if it would affect the hot loop _and_ the loop would end up being vectorized.

Looking at what bits we don't care about seems like the right approach to me. I think an easier way to reason about these is to have a bitmask for each value in the chain that we're looking at, indicating what bits we care about. Different operations would do different things on this bitmask, and we would compute the value of the bitmasks starting from the trunc operations.

For example:

b = add a, c means that a and c have the same bitmask as b

The same would be true for mul, left shift, all bitwise operators, and sub (since sub can be written with and add and a not).

b = shr a, 2,  and b has a bitmask of 0b00011 ( where 1 means we care, 0 don't care) means a has a bitmask of 0b01111 (technically it would be 0b01100, but this is an approximation).

This can probably be done for most operators.

Also, if you have an operation that takes a constant and you know what bits you care about it should be perfectly legal to truncate the constant and get rid of some of the bits that you don't care about.

Technically, we could set the mask values on the truncate operations and iterate using a work list until all bitmasks have converged, but that is best left for future work.

What would be your opinion on this?

Cheers,
Silviu

Hey Silviu,

I believe this is the technique that James had told me about and mentioned that it had already been implemented somewhere in the code base, but without an interface. I didn't go for it originally because I didn't what the ramp up time of trying to understand, refactor and modify two separate passes and I'm still concerned about that time factor! The bitmask idea does sound like a much more useful and reusable approach, however I really can't commit the time to implement it, unless you believe it can slot into the current structure of my analysis.

Cheers,

[samparker]

Rebased and corrected some typos. Also removed sext and zext from being nodes that are searched as they will get searched from their users anyway.

[jmolloy]

Hi Sam,

I've got a bunch of typographical changes, but the biggest problem is still the unbounded recursion.

Cheers,

James

lib/Transforms/Vectorize/LoopVectorize.cpp

1170

All these functions should start with a lowercase letter.

3559

I'm confused as to why we need ->getScalarType() here - surely this can't be a vector type by definition, as it hasn't yet been vectorized?

5110

In what case can you get here and one of the types not be an integer?

5122

I think it'd be easier if you just did:

CandidateNarrowInstrs[VF][I] = Largest;

To remove a bunch of useless code.

5125

Braces around else.

5137

Again, use std::map::operator[] rather than ::erase and ::insert.

5198

You're not actually grabbing any value here.

5199

Here and elsewhere: if statements should either be one-liners or multi-liners. Please don't mix one-liners and braces.

if (a)
  b;
else
  c;  // GOOD

if (a) {
  b; c;
} else
  d; // BAD

5203

This is unboundedly recursive.

5245

Please use a switch here.

5264

Capital letter.

[samparker]

lib/Transforms/Vectorize/LoopVectorize.cpp

5203

But isn't the recursion bound by the size of CandiateNarrowInstrs, which is defined during the first pass?

[hfinkel]

lib/Transforms/Vectorize/LoopVectorize.cpp

1182

Line is too long.

5203

What bounds that size?

[samparker]

lib/Transforms/Vectorize/LoopVectorize.cpp

5203

The number of instructions in the loop body, so do I need to add a further more explicit limit?

[hfinkel]

lib/Transforms/Vectorize/LoopVectorize.cpp

5203

Yes, you'd need a more-explicit limit. We can't have a recursion bounded only by the number of instructions in a block, loop, etc.

lib/Transforms/Vectorize/LoopVectorize.cpp

   /// the factor width.
   unsigned expectedCost(unsigned VF);
 
+  // Return the vectorized type or a clamped type that may have been
+  // previously identified.
+  Type* getClampedVectorTy(Instruction *I, unsigned VF);
+
+  bool isFreeCast(Instruction *I, unsigned VF);
+
   /// Returns the execution time cost of an instruction for a given vector
   /// width. Vector width of one means scalar.
   unsigned getInstructionCost(Instruction *I, unsigned VF);

   const Function *TheFunction;
   // Loop Vectorize Hint.
   const LoopVectorizeHints *Hints;
+
+  // While searching from truncs, we store instructions which can use
+  // smaller types when vectorized.
+  std::map<Instruction*, Type*> ClampedInstrTys;
+  // Sext and Zext instructions that will disappear due to vectorization.
+  std::set<Instruction*> FreeCasts;
 };
 
 /// Utility class for getting and setting loop vectorizer hints in the form

 }
 
 unsigned LoopVectorizationCostModel::getWidestType() {
+
   unsigned MaxWidth = 8;
   const DataLayout &DL = TheFunction->getParent()->getDataLayout();
 

     BasicBlock *BB = *bb;
 
     // For each instruction in the old loop.
-    for (BasicBlock::iterator it = BB->begin(), e = BB->end(); it != e; ++it) {
+    for (BasicBlock::reverse_iterator it = BB->rbegin(), e = BB->rend();
+         it != e; ++it) {
       // Skip dbg intrinsics.
-      if (isa<DbgInfoIntrinsic>(it))
+      if (isa<DbgInfoIntrinsic>(*it))
         continue;
 
       // Ignore ephemeral values.
-      if (EphValues.count(it))
+      if (EphValues.count(&*it))
         continue;
 
-      unsigned C = getInstructionCost(it, VF);
+      unsigned C = getInstructionCost(&*it, VF);
 
       // Check if we should override the cost.
       if (ForceTargetInstructionCost.getNumOccurrences() > 0)

     Cost += BlockCost;
   }
 
+  // Clear the structures for this VF.
+  ClampedInstrTys.clear();
+  FreeCasts.clear();
+
   return Cost;
 }
 

   return false;
 }
 
+bool
+LoopVectorizationCostModel::isFreeCast(Instruction *I, unsigned VF) {
+  CastInst *CI = cast<CastInst>(I);
+  Type *DstTy = CI->getDestTy();
+  Type *DstVecTy = ToVectorTy(DstTy, VF);
+
+  // Search happens from truncs towards the possible extend instructions,
+  // so these would of been found eariler.
+  if (I->getOpcode() != Instruction::Trunc) {
+    if (FreeCasts.find(I) != FreeCasts.end())
+      return true;
+    else
+      return false;
+  }
+
+  if (!DstTy->isIntegerTy())
+    return false;
+
+  const DataLayout &DL = TheFunction->getParent()->getDataLayout();
+  unsigned ScalarWidth = DL.getTypeSizeInBits(DstTy);
+
+  // First, check whether the truncation destination size would be a legal
+  // vector type.
+  if (TTI.isTypeLegal(DstVecTy)) {
+    auto *ChainInst = cast<Instruction>(I->getOperand(0));
+    SmallVector<Instruction*, 4> PossibleFreeCasts;
+    unsigned NarrowOperands = 0;
+
+    // Then search the operands of the trunc use and look for sext/zext or
+    // constants within the size limit.
+    for (unsigned i = 0; i < ChainInst->getNumOperands(); ++i) {
+
+      if (isa<ConstantInt>(ChainInst->getOperand(i))) {
+        auto *ConstInt = cast<ConstantInt>(ChainInst->getOperand(i));
+
+        if (ConstInt->uge((1 << ScalarWidth) - 1))
+          break;
+
+          ++NarrowOperands;
+
+        } else if (isa<CastInst>(ChainInst->getOperand(i))) {
+            auto *Cast = cast<CastInst>(ChainInst->getOperand(i));
+
+            if (!Cast->isIntegerCast())
+              break;
+
+            ++NarrowOperands;
+
+            // Check whether the original size of the sext/zext is the same
+            // as the trunc destination size.
+            if (Cast->getSrcTy() == DstTy)
+              PossibleFreeCasts.push_back(Cast);
+            else
+              // If not, we still save the clamped size so later the
+              // zext/sext instruction knows what size is necessary.
+              ClampedInstrTys.insert(std::make_pair(Cast, DstVecTy));
+        }
+    }
+    // If all the operands to this instruction are narrow, make the trunc
+    // and the casts free and assign a clamped VectorTy to the ChainInst.
+    if (NarrowOperands == ChainInst->getNumOperands()) {
+      for (auto *I : PossibleFreeCasts)
+        FreeCasts.insert(I);
+      ClampedInstrTys.insert(std::make_pair(ChainInst, DstVecTy));
+      return true;
+    }
+  }
+
+  return false;
+}
+
+Type*
+LoopVectorizationCostModel::getClampedVectorTy(Instruction *I, unsigned VF) {
+  if (ClampedInstrTys.find(I) != ClampedInstrTys.end())
+    return ClampedInstrTys[I];
+  else
+    return ToVectorTy(I->getType(), VF);
+}
+
 unsigned
 LoopVectorizationCostModel::getInstructionCost(Instruction *I, unsigned VF) {
   // If we know that this instruction will remain uniform, check the cost of

     VF = 1;
 
   Type *RetTy = I->getType();
-  Type *VectorTy = ToVectorTy(RetTy, VF);
+  Type *VectorTy = getClampedVectorTy(I, VF);
 
   // TODO: We need to estimate the cost of intrinsic calls.
   switch (I->getOpcode()) {

   case Instruction::Trunc:
   case Instruction::FPTrunc:
   case Instruction::BitCast: {
+    Type *SrcTy = I->getOperand(0)->getType();
+    Type *SrcVecTy = ToVectorTy(SrcTy, VF);
+    unsigned Opcode = I->getOpcode();
     // We optimize the truncation of induction variable.
     // The cost of these is the same as the scalar operation.
-    if (I->getOpcode() == Instruction::Trunc &&
+    if (Opcode == Instruction::Trunc &&
         Legal->isInductionVariable(I->getOperand(0)))
-      return TTI.getCastInstrCost(I->getOpcode(), I->getType(),
-                                  I->getOperand(0)->getType());
+      return TTI.getCastInstrCost(Opcode, I->getType(), SrcTy);
 
-    Type *SrcVecTy = ToVectorTy(I->getOperand(0)->getType(), VF);
-    return TTI.getCastInstrCost(I->getOpcode(), VectorTy, SrcVecTy);
+    if (isFreeCast(I, VF))
+      return 0;
+    else
+      return TTI.getCastInstrCost(Opcode, VectorTy, SrcVecTy);
   }
   case Instruction::Call: {
     bool NeedToScalarize;

test/Transforms/LoopVectorize/AArch64/loop-vectorization-factors.ll

+; RUN: opt -S < %s -loop-vectorize -simplifycfg -instsimplify -instcombine  2>&1 | FileCheck %s
+
+; ModuleID = 'vector-test.c'
+target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64"
+
+; CHECK-LABEL: @add_a(
+; COST: cost of 2 for instruction: {{.*}} load <16 x i8>
+; CHECK: load <16 x i8>, <16 x i8>*
+; CHECK: load <16 x i8>, <16 x i8>*
+; COST: cost of 1 for instruction: {{.*}} add <16 x i8>
+; CHECK: add <16 x i8>
+; CHECK: add <16 x i8>
+; COST: cost of 2 for instruction: {{.*}} store <16 x i8>
+; CHECK: store <16 x i8>
+; CHECK: store <16 x i8>
+; Function Attrs: nounwind
+define void @add_a(i8* noalias nocapture readonly %p, i8* noalias nocapture %q, i32 %len) #0 {
+entry:
+  %cmp8 = icmp sgt i32 %len, 0
+  br i1 %cmp8, label %for.body, label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.body, %entry
+  ret void
+
+for.body:                                         ; preds = %entry, %for.body
+  %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 0, %entry ]
+  %arrayidx = getelementptr inbounds i8, i8* %p, i64 %indvars.iv
+  %0 = load i8, i8* %arrayidx, align 1, !tbaa !1
+  %conv = zext i8 %0 to i32
+  %add = add nuw nsw i32 %conv, 2
+  %conv1 = trunc i32 %add to i8
+  %arrayidx3 = getelementptr inbounds i8, i8* %q, i64 %indvars.iv
+  store i8 %conv1, i8* %arrayidx3, align 1, !tbaa !1
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %lftr.wideiv = trunc i64 %indvars.iv.next to i32
+  %exitcond = icmp eq i32 %lftr.wideiv, %len
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}
+
+; CHECK-LABEL: @add_b(
+; COST: cost of 2 for instruction: {{.*}} load <16 x i8>
+; CHECK: load <8 x i16>, <8 x i16>*
+; CHECK: load <8 x i16>, <8 x i16>*
+; COST: cost of 1 for instruction: {{.*}} add <16 x i8>
+; CHECK: add <8 x i16>
+; CHECK: add <8 x i16>
+; COST: cost of 2 for instruction: {{.*}} store <16 x i8>
+; CHECK: store <8 x i16>
+; CHECK: store <8 x i16>
+; Function Attrs: nounwind
+define void @add_b(i16* noalias nocapture readonly %p, i16* noalias nocapture %q, i32 %len) #0 {
+entry:
+  %cmp9 = icmp sgt i32 %len, 0
+  br i1 %cmp9, label %for.body, label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.body, %entry
+  ret void
+
+for.body:                                         ; preds = %entry, %for.body
+  %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 0, %entry ]
+  %arrayidx = getelementptr inbounds i16, i16* %p, i64 %indvars.iv
+  %0 = load i16, i16* %arrayidx, align 2, !tbaa !4
+  %conv8 = zext i16 %0 to i32
+  %add = add nuw nsw i32 %conv8, 2
+  %conv1 = trunc i32 %add to i16
+  %arrayidx3 = getelementptr inbounds i16, i16* %q, i64 %indvars.iv
+  store i16 %conv1, i16* %arrayidx3, align 2, !tbaa !4
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %lftr.wideiv = trunc i64 %indvars.iv.next to i32
+  %exitcond = icmp eq i32 %lftr.wideiv, %len
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}
+
+; CHECK-LABEL: @add_c(
+; CHECK: load <8 x i8>, <8 x i8>*
+; CHECK: add nuw nsw <8 x i16>
+; CHECK: store <8 x i16>
+; Function Attrs: nounwind
+define void @add_c(i8* noalias nocapture readonly %p, i16* noalias nocapture %q, i32 %len) #0 {
+entry:
+  %cmp8 = icmp sgt i32 %len, 0
+  br i1 %cmp8, label %for.body, label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.body, %entry
+  ret void
+
+for.body:                                         ; preds = %entry, %for.body
+  %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 0, %entry ]
+  %arrayidx = getelementptr inbounds i8, i8* %p, i64 %indvars.iv
+  %0 = load i8, i8* %arrayidx, align 1, !tbaa !1
+  %conv = zext i8 %0 to i32
+  %add = add nuw nsw i32 %conv, 2
+  %conv1 = trunc i32 %add to i16
+  %arrayidx3 = getelementptr inbounds i16, i16* %q, i64 %indvars.iv
+  store i16 %conv1, i16* %arrayidx3, align 2, !tbaa !4
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %lftr.wideiv = trunc i64 %indvars.iv.next to i32
+  %exitcond = icmp eq i32 %lftr.wideiv, %len
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}
+
+; Function Attrs: nounwind
+; CHECK-LABEL: @add_d(
+; COST: cost of 2 for instruction: {{.*}} load <4 x i16>
+; CHECK: load <4 x i16>
+; CHECK: load <4 x i16>
+; COST: cost of 1 for instruction: {{.*}} add <4 x i32>
+; CHECK: add nsw <4 x i32>
+; CHECK: add nsw <4 x i32>
+; COST: cost of 2 for instruction: {{.*}} store <4 x i32>
+; CHECK: store <4 x i32>
+; CHECK: store <4 x i32>
+define void @add_d(i16* noalias nocapture readonly %p, i32* noalias nocapture %q, i32 %len) #0 {
+entry:
+  %cmp7 = icmp sgt i32 %len, 0
+  br i1 %cmp7, label %for.body, label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.body, %entry
+  ret void
+
+for.body:                                         ; preds = %entry, %for.body
+  %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 0, %entry ]
+  %arrayidx = getelementptr inbounds i16, i16* %p, i64 %indvars.iv
+  %0 = load i16, i16* %arrayidx, align 2, !tbaa !4
+  %conv = sext i16 %0 to i32
+  %add = add nsw i32 %conv, 2
+  %arrayidx2 = getelementptr inbounds i32, i32* %q, i64 %indvars.iv
+  store i32 %add, i32* %arrayidx2, align 4, !tbaa !6
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %lftr.wideiv = trunc i64 %indvars.iv.next to i32
+  %exitcond = icmp eq i32 %lftr.wideiv, %len
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}
+
+attributes #0 = { nounwind "less-precise-fpmad"="false" "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon" "unsafe-fp-math"="false" "use-soft-float"="false" }
+
+!llvm.ident = !{!0}
+
+!0 = !{!"clang version 3.7.0 (http://llvm.org/git/clang.git 93281cbcbb9b1ea2b6788a629d6aef3284957d05) (http://llvm.org/git/llvm.git 40048e70d7a63ab64df3d0e52107f3c0e3472571)"}
+!1 = !{!2, !2, i64 0}
+!2 = !{!"omnipotent char", !3, i64 0}
+!3 = !{!"Simple C/C++ TBAA"}
+!4 = !{!5, !5, i64 0}
+!5 = !{!"short", !2, i64 0}
+!6 = !{!7, !7, i64 0}
+!7 = !{!"int", !2, i64 0}