[ARM] Enable partial and runtime unrolling

[samparker]

Summary

Enable runtime and partial loop unrolling of simple loops without calls, on Thumb2 M-Class cores. The thresholds are calculated based on the Subtarget's issue width.

[SjoerdMeijer]

Hi Sam,
Where are the tests? ;-) The ad hoc way of calculating the trip count (or the variable) in getTripCountValue() doesn't feel entirely right, is this not what SCEV should be providing?
Cheers,
Sjoerd.

[SjoerdMeijer]

Forgot to say that it would be good if you can share some numbers why this a good thing to do.

[samparker]

Bah, of course! I will get some tests together. As for the trip count.... yes, I think SCEV handles these things and I was hoping someone could point me to a nicer and more standardised way of getting the value.

cheers,
sam

[samparker]

Added some tests and changed the initial thresholds to be based purely on the issue width of the cpu, which I've added a helper function for in ARMSubtarget.

[samparker]

Hi Sjoerd,

I've been running an industry standard benchmark suite and targeting Cortex-M4 and Cortex-M7. This patch gives modest improvements to many of the benchmarks, as well as some significantly better results. Overall improvements are ~1% for Cortex-M4 and ~3% for Cortex-M7.

cheers,
sam

[efriedma]

lib/Target/ARM/ARMTargetTransformInfo.cpp

599

These heuristics seem really weird... why does the loop depth affect the performance of a loop? Why does the trip count of the parent loop affect the performance of a loop?

I mean, I can imagine these heuristics are profitable for your particular benchmarks, but it seems like a performance trap: for example, someone turns on LTO, so a function gets inlined into a loop, so we decide it's no longer profitable to unroll the inner loop, and we lose a bunch of performance.

[samparker]

lib/Target/ARM/ARMTargetTransformInfo.cpp

599

Hi Eli,

The depth check here is just for an early exit. For nested loops, I'm checking whether the trip count of the inner loop will be the same for each iteration of the outer loops, it's not the actual trip count that affects performance.

I have found that its not profitable to unroll loops with calls because it can prevent the inlining from happening, so unrolling will not happen in those cases and I hope that the inlining does occur. Some inlining could push the cost above the unrolling threshold, but that doesn't matter because we're still getting the inlining performance gains. The thresholds have been chosen to unroll small loops where the overhead of the backedge, on m-class cores, is significant.

cheers,
sam

[samparker]

Amended some comments.

[efriedma]

lib/Target/ARM/ARMTargetTransformInfo.cpp

599

The depth check here is just for an early exit. For nested loops, I'm checking whether the trip count of the inner loop will be the same for each iteration of the outer loops,

This is still problematic. You're basing the question of whether to unroll the loop based on code which is not part of the loop, which means the end result is going to be sensitive to unrelated inlining decisions.

---

Scanning the loop for calls seems fine.

test/CodeGen/ARM/loop-unrolling.ll

153

Why do we want to avoid unrolling here? At first glance, this looks like it should be profitable to unroll (more scheduler freedom, avoid branches).

[samparker]

Updated to use the freshly introduced scalar evolution and also simplified the heuristics by not checking the loop depth.

[samparker]

test/CodeGen/ARM/loop-unrolling.ll

153

Enabling runtime unrolling can generate more spill code, and for small runtime counts the unrolled version may not even be executed that frequently, so I found that unrolling inner loops whose variable trip count is computed in the parent loop can cause some very significant regressions.

[samparker]

I should also say that even though the average performance improvement is modest, some tested benchmarks see an improvement of up to 40, 50 and 80% for Cortex-M4, M7 and M33.

[efriedma]

I don't see how a trip count which varies for each iteration of the parent loop implies the trip count is small. I mean, it's possible it does for your particular benchmarks, but it seems unlikely to generalize to other code.

lib/Target/ARM/ARMTargetTransformInfo.cpp

585

There isn't any reason to check isLoopInvariant for each loop: if the count is invariant relative to the topmost loop, it will be invariant relative to every loop it contains.

[samparker]

Hi Eli,

Your comments make sense to me, so I ran an example to figure out if this heuristic was indeed nonsense. Here's the example kernel:

for (unsigned i = 0; i < max; ++i) {
  acc = 0;
  innerMax = dataSize - i;
  for (unsigned j = 0; j < innerMax; ++j) {
    acc += (input[j] * input[i+j]) >> scaleValue;
  }

The results in the graph show that often the unrolled version is faster, but the net affect across the data set is that unrolling is detrimental on performance. My other benchmark results also show that having this restriction doesn't negatively impact the performance, so I think that including the heuristic to prevent unrolling is valid.

cheers,
sam

[SjoerdMeijer]

Sam, interesting performance numbers! It looks like we see some strange bimodal behaviour: (big) improvements are canceled out by (big) regressions. Assuming that codegen for this function is the same here (just the value of dataSize is different), I am wondering if we are not just looking at some micro-architecture weirdness? So I don't think we can draw any conclusions from these numbers. Do we need more data points?

[samparker]

Yes good point. Those numbers were running on the M7, which has a branch predicator... so I will run it again on something more simple.

[efriedma]

There are two kinds of SCEV expressions which are not invariant: SCEVAddRecExpr, and SCEVUnknown. If you have a SCEVAddRecExpr, you can show the loop nest is similar to your testcase, and I guess that might change the profitability of unrolling. If you have a SCEVUnknown, I can't see how you would conclude anything useful; there's a strong possibility the trip count is actually constant, but the compiler can't prove it because of aliasing or something like that.

[samparker]

After more testing, I've found that the variant inner loop check was unnecessary so I've removed that check.

[efriedma]

lib/Target/ARM/ARMTargetTransformInfo.cpp

552

If you want this to be a no-op for other architectures, shouldn't this be "return BasicTTIImplBase::getUnrollingPreferences(L, SE, UP)"?

559

Do we really call into getUnrollingPreferences for the early unroll pass? We probably don't want to be using target-specific unroll heuristics for "createSimpleLoopUnrollPass". But I guess that's not something you need to fix in this patch.

lib/Target/ARM/ARMTargetTransformInfo.h

126

"override"?

[samparker]

lib/Target/ARM/ARMTargetTransformInfo.cpp

552

ah yes, thanks.

559

As far as I can see, the unroll pass functionality can only be controlled externally via the command line options, or by target hooks.

lib/Target/ARM/ARMTargetTransformInfo.h

126

oddly no, it's not virtual.

[samparker]

Added the default call to the base implementation.

[efriedma]

test/CodeGen/ARM/loop-unrolling.ll

2

We usually put tests for IR transforms into test/Transforms/; for unrolling in particular, that would be something like test/Transforms/LoopUnroll/ARM/ (you'll have to create a new directory, and make a lit.local.cfg so the test is only enabled if we have an ARM target).

And I'd like to see some basic test to show that this doesn't change the unroll heuristics on, for example, cortex-a57 (where BasicTTIImplBase::getUnrollingPreferences currently changes the unroll threshold).

[samparker]

I've moved the test to the suggested directory and added some more targets to it. I've also now enabled the unrolling for Thumb targets and removed the use of the issue width in the calculation.

[samparker]

Now added the test config file!

[efriedma]

test/Transforms/LoopUnroll/ARM/loop-unrolling.ll

5 ↗

(On Diff #106445)

Could you add a run to check -mcpu=cortex-a57, to verify this we're still inheriting unroller heuristics from BasicTTIImplBase?

[samparker]

Hi Eli,

I've updated the tests to target the A57 for both A32 and T32 mode.

I've actually now benchmarked the A53 and A57 with this patch and the results are favourable, but there is a bug I need to fix first before I enable it. That will be another patch.

cheers,
sam

[efriedma]

LGTM

lib/Target/ARM/ARMSubtarget.h

   bool restrictIT() const { return RestrictIT; }
 
   const std::string & getCPUString() const { return CPUString; }
 
   bool isLittle() const { return IsLittle; }
 
   unsigned getMispredictionPenalty() const;
 
+  unsigned getIssueWidth() const;
+
   /// This function returns true if the target has sincos() routine in its
   /// compiler runtime or math libraries.
   bool hasSinCos() const;
 
   /// Returns true if machine scheduler should be enabled.
   bool enableMachineScheduler() const override;
 
   /// True for some subtargets at > -O0.

lib/Target/ARM/ARMSubtarget.cpp

 
   return false;
 }
 
 unsigned ARMSubtarget::getMispredictionPenalty() const {
   return SchedModel.MispredictPenalty;
 }
 
+unsigned ARMSubtarget::getIssueWidth() const {
+  return getSchedModel().IssueWidth;
+}
+
 bool ARMSubtarget::hasSinCos() const {
   return isTargetWatchOS() ||
     (isTargetIOS() && !getTargetTriple().isOSVersionLT(7, 0));
 }
 
 bool ARMSubtarget::enableMachineScheduler() const {
   // Enable the MachineScheduler before register allocation for out-of-order
   // architectures where we do not use the PostRA scheduler anymore (for now

lib/Target/ARM/ARMTargetTransformInfo.h

 
   int getMemoryOpCost(unsigned Opcode, Type *Src, unsigned Alignment,
                       unsigned AddressSpace, const Instruction *I = nullptr);
 
   int getInterleavedMemoryOpCost(unsigned Opcode, Type *VecTy, unsigned Factor,
                                  ArrayRef<unsigned> Indices, unsigned Alignment,
                                  unsigned AddressSpace);
 
+  void getUnrollingPreferences(Loop *L, TTI::UnrollingPreferences &UP);
+
   bool shouldBuildLookupTablesForConstant(Constant *C) const {
     // In the ROPI and RWPI relocation models we can't have pointers to global
     // variables or functions in constant data, so don't convert switches to
     // lookup tables if any of the values would need relocation.
     if (ST->isROPI() || ST->isRWPI())
       return !C->needsRelocation();
 
     return true;
   }
   /// @}
 };
 
 } // end namespace llvm
 
 #endif

lib/Target/ARM/ARMTargetTransformInfo.cpp

     if (NumElts % Factor == 0 &&
         TLI->isLegalInterleavedAccessType(SubVecTy, DL))
       return Factor * TLI->getNumInterleavedAccesses(SubVecTy, DL);
   }
 
   return BaseT::getInterleavedMemoryOpCost(Opcode, VecTy, Factor, Indices,
                                            Alignment, AddressSpace);
 }
+
+static Value *getTripCountValue(Loop *L) {
+  // Canonical loops will end with a 'cmp ne I, V', where I is the incremented
+  // canonical induction variable and V is the trip count of the loop.
+  PHINode *IV = L->getCanonicalInductionVariable();
+  if (!IV)
+    return nullptr;
+
+  BasicBlock *BackedgeBlock = L->getLoopLatch();
+  Value *Inc = IV->getIncomingValueForBlock(BackedgeBlock);
+
+  if (auto *BI = dyn_cast<BranchInst>(BackedgeBlock->getTerminator())) {
+    if (BI->isConditional()) {
+      if (auto *ICI = dyn_cast<ICmpInst>(BI->getCondition())) {
+        if (ICI->getOperand(0) == Inc) {
+          return ICI->getOperand(1);
+        }
+      }
+    }
+  }
+  return nullptr;
+}
+
+void ARMTTIImpl::getUnrollingPreferences(Loop *L,
+                                         TTI::UnrollingPreferences &UP) {
+  if (!ST->isThumb2() || !ST->isMClass() || (L->getNumBlocks() != 1)
+      || !L->getExitingBlock() || !L->getUniqueExitBlock())
+    return;
+
+  BasicBlock *BB = L->getLoopLatch();
+  UP.OptSizeThreshold = 0;
+  UP.PartialOptSizeThreshold = 0;
+
+  // Scan the loop: don't unroll loops with calls.
+  for (auto &I : *BB) {
+    if (isa<CallInst>(I) || isa<InvokeInst>(I)) {
+      ImmutableCallSite CS(&I);
+      if (const Function *F = CS.getCalledFunction()) {
+        if (!isLoweredToCall(F))
+          continue;
+      }
+      return;
+    }
+  }
+
+  // Enable partial unrolling and set the initial threshold based upon
+  // the width of the microarchitecture.
+  UP.Partial = true;
+  UP.Threshold = ST->getIssueWidth() > 1 ? 300 : 150;
+  UP.PartialThreshold = ST->getIssueWidth() > 1 ? 300 : 150;
+
+  // Enable runtime unrolling for non-nested loops.
+  if (L->getLoopDepth() == 1) {
+    UP.Runtime = true;
+    return;
+  }
+
+  // For nested loops, we want the trip count to be invariant in the outer
+  // loops.
+  Value *TripCount = getTripCountValue(L);
+  if (!TripCount)
+    return;
+
+  Loop *Parent = L->getParentLoop();
+  while (Parent) {
+    if (!Parent->isLoopInvariant(TripCount))
+      return;
+    Parent = Parent->getParentLoop();
+  }
+
+  // Lower the threshold for nested runtime loops.
+  if (!isa<Constant>(TripCount))
+    UP.PartialThreshold /= 2;
+  UP.Runtime = true;
+}

test/CodeGen/ARM/loop-unrolling.ll

+; RUN: opt -mcpu=cortex-m4 -loop-unroll -S %s -o - | FileCheck %s
+
+target datalayout = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
+target triple = "thumbv7m-arm-"
+
+; CHECK-LABEL: partial
+define arm_aapcs_vfpcc void @partial(i32* nocapture %C, i32* nocapture readonly %A, i32* nocapture readonly %B) local_unnamed_addr #0 {
+entry:
+  br label %for.body
+
+for.body:
+; CHECK-LABEL: for.body
+; CHECK: [[IV0:%[a-z.0-9]+]] = phi i32 [ 0, %entry ], [ [[IV16:%[a-z.0-9]+]], %for.body ]
+; CHECK: [[IV1:%[a-z.0-9]+]] = add nuw nsw i32 [[IV0]], 1
+; CHECK: [[IV2:%[a-z.0-9]+]] = add nuw nsw i32 [[IV1]], 1
+; CHECK: [[IV3:%[a-z.0-9]+]] = add nuw nsw i32 [[IV2]], 1
+; CHECK: [[IV4:%[a-z.0-9]+]] = add nuw nsw i32 [[IV3]], 1
+; CHECK: [[IV5:%[a-z.0-9]+]] = add nuw nsw i32 [[IV4]], 1
+; CHECK: [[IV6:%[a-z.0-9]+]] = add nuw nsw i32 [[IV5]], 1
+; CHECK: [[IV7:%[a-z.0-9]+]] = add nuw nsw i32 [[IV6]], 1
+; CHECK: [[IV8:%[a-z.0-9]+]] = add nuw nsw i32 [[IV7]], 1
+; CHECK: [[IV9:%[a-z.0-9]+]] = add nuw nsw i32 [[IV8]], 1
+; CHECK: [[IV10:%[a-z.0-9]+]] = add nuw nsw i32 [[IV9]], 1
+; CHECK: [[IV11:%[a-z.0-9]+]] = add nuw nsw i32 [[IV10]], 1
+; CHECK: [[IV12:%[a-z.0-9]+]] = add nuw nsw i32 [[IV11]], 1
+; CHECK: [[IV13:%[a-z.0-9]+]] = add nuw nsw i32 [[IV12]], 1
+; CHECK: [[IV14:%[a-z.0-9]+]] = add nuw nsw i32 [[IV13]], 1
+; CHECK: [[IV15:%[a-z.0-9]+]] = add nuw nsw i32 [[IV14]], 1
+; CHECK: [[IV16:%[a-z.0-9]+]] = add nuw nsw i32 [[IV15]], 1
+; CHECK: [[CMP:%[a-z.0-9]+]] = icmp eq i32 [[IV16]], 1024
+; CHECK: br i1 [[CMP]], label [[END:%[a-z.]+]], label %for.body
+  %i.08 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i.08
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx1 = getelementptr inbounds i32, i32* %B, i32 %i.08
+  %1 = load i32, i32* %arrayidx1, align 4
+  %mul = mul nsw i32 %1, %0
+  %arrayidx2 = getelementptr inbounds i32, i32* %C, i32 %i.08
+  store i32 %mul, i32* %arrayidx2, align 4
+  %inc = add nuw nsw i32 %i.08, 1
+  %exitcond = icmp eq i32 %inc, 1024
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+
+for.cond.cleanup:
+  ret void
+}
+
+; CHECK-LABEL: runtime
+define arm_aapcs_vfpcc void @runtime(i32* nocapture %C, i32* nocapture readonly %A, i32* nocapture readonly %B, i32 %N) local_unnamed_addr #0 {
+entry:
+  %cmp8 = icmp eq i32 %N, 0
+  br i1 %cmp8, label %for.cond.cleanup, label %for.body
+
+for.body:
+; CHECK-LABEL: for.body
+; CHECK: [[IV0:%[a-z.0-9]+]] = phi i32 [ 0, [[PRE:%[a-z.0-9]+]] ], [ [[IV8:%[a-z.0-9]+]], %for.body ]
+; CHECK: [[IV1:%[a-z.0-9]+]] = add nuw nsw i32 [[IV0]], 1
+; CHECK: [[IV2:%[a-z.0-9]+]] = add nuw nsw i32 [[IV1]], 1
+; CHECK: [[IV3:%[a-z.0-9]+]] = add nuw nsw i32 [[IV2]], 1
+; CHECK: [[IV4:%[a-z.0-9]+]] = add nuw nsw i32 [[IV3]], 1
+; CHECK: [[IV5:%[a-z.0-9]+]] = add nuw nsw i32 [[IV4]], 1
+; CHECK: [[IV6:%[a-z.0-9]+]] = add nuw nsw i32 [[IV5]], 1
+; CHECK: [[IV7:%[a-z.0-9]+]] = add nuw nsw i32 [[IV6]], 1
+; CHECK: [[IV8]] = add nuw i32 [[IV7]], 1
+  %i.09 = phi i32 [ %inc, %for.body ], [ 0, %entry ]
+  %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i.09
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx1 = getelementptr inbounds i32, i32* %B, i32 %i.09
+  %1 = load i32, i32* %arrayidx1, align 4
+  %mul = mul nsw i32 %1, %0
+  %arrayidx2 = getelementptr inbounds i32, i32* %C, i32 %i.09
+  store i32 %mul, i32* %arrayidx2, align 4
+  %inc = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc, %N
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+
+for.cond.cleanup:
+  ret void
+}
+
+; CHECK-LABEL: nested_runtime
+define arm_aapcs_vfpcc void @nested_runtime(i32* nocapture %C, i16* nocapture readonly %A, i16* nocapture readonly %B, i32 %N) local_unnamed_addr #0 {
+entry:
+  %cmp25 = icmp eq i32 %N, 0
+  br i1 %cmp25, label %for.cond.cleanup, label %for.body4.lr.ph
+
+for.body4.lr.ph:                                  ; preds = %entry, %for.cond.cleanup3
+  %h.026 = phi i32 [ %inc11, %for.cond.cleanup3 ], [ 0, %entry ]
+  %mul = mul i32 %h.026, %N
+  br label %for.body4
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup3, %entry
+  ret void
+
+for.cond.cleanup3:                                ; preds = %for.body4
+  %inc11 = add nuw i32 %h.026, 1
+  %exitcond27 = icmp eq i32 %inc11, %N
+  br i1 %exitcond27, label %for.cond.cleanup, label %for.body4.lr.ph
+
+; CHECK-LABEL: for.body4
+; CHECK: [[IV0:%[a-z.0-9]+]] = phi i32 [ 0, [[PRE:%[a-z0-9.]+]] ], [ [[IV4:%[a-z.0-9]+]], %for.body4 ]
+; CHECK: [[IV1:%[a-z.0-9]+]] = add nuw nsw i32 [[IV0]], 1
+; CHECK: [[IV2:%[a-z.0-9]+]] = add nuw nsw i32 [[IV1]], 1
+; CHECK: [[IV3:%[a-z.0-9]+]] = add nuw nsw i32 [[IV2]], 1
+; CHECK: [[IV4:%[a-z.0-9]+]] = add nuw i32 [[IV3]], 1
+for.body4:                                        ; preds = %for.body4, %for.body4.lr.ph
+  %w.024 = phi i32 [ 0, %for.body4.lr.ph ], [ %inc, %for.body4 ]
+  %add = add i32 %w.024, %mul
+  %arrayidx = getelementptr inbounds i16, i16* %A, i32 %add
+  %0 = load i16, i16* %arrayidx, align 2
+  %conv = sext i16 %0 to i32
+  %arrayidx5 = getelementptr inbounds i16, i16* %B, i32 %w.024
+  %1 = load i16, i16* %arrayidx5, align 2
+  %conv6 = sext i16 %1 to i32
+  %mul7 = mul nsw i32 %conv6, %conv
+  %arrayidx8 = getelementptr inbounds i32, i32* %C, i32 %w.024
+  %2 = load i32, i32* %arrayidx8, align 4
+  %add9 = add nsw i32 %mul7, %2
+  store i32 %add9, i32* %arrayidx8, align 4
+  %inc = add nuw i32 %w.024, 1
+  %exitcond = icmp eq i32 %inc, %N
+  br i1 %exitcond, label %for.cond.cleanup3, label %for.body4
+}
+
+; CHECK-LABEL: loop_call
+define arm_aapcs_vfpcc void @loop_call(i32* nocapture %C, i32* nocapture readonly %A, i32* nocapture readonly %B) local_unnamed_addr #1 {
+entry:
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body
+  ret void
+
+; CHECK-LABEL: for.body
+; CHECK: %i.08 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+; CHECK: %inc = add nuw nsw i32 %i.08, 1
+; CHECK: %exitcond = icmp eq i32 %inc, 1024
+for.body:                                         ; preds = %for.body, %entry
+  %i.08 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i.08
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx1 = getelementptr inbounds i32, i32* %B, i32 %i.08
+  %1 = load i32, i32* %arrayidx1, align 4
+  %call = tail call arm_aapcs_vfpcc i32 @some_func(i32 %0, i32 %1) #3
+  %arrayidx2 = getelementptr inbounds i32, i32* %C, i32 %i.08
+  store i32 %call, i32* %arrayidx2, align 4
+  %inc = add nuw nsw i32 %i.08, 1
+  %exitcond = icmp eq i32 %inc, 1024
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}
+
+; CHECK-LABEL: nested_runtime_variant
+define arm_aapcs_vfpcc void @nested_runtime_variant(i32* nocapture %C, i16* nocapture readonly %A, i16* nocapture readonly %B, i32 %N) local_unnamed_addr #0 {
+entry:
+  %cmp27 = icmp eq i32 %N, 0
+  br i1 %cmp27, label %for.cond.cleanup, label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup3, %entry
+  ret void
+
+for.body:                                         ; preds = %entry, %for.cond.cleanup3
+  %indvars.iv = phi i32 [ %indvars.iv.next, %for.cond.cleanup3 ], [ %N, %entry ]
+  %h.028 = phi i32 [ %inc11, %for.cond.cleanup3 ], [ 0, %entry ]
+  %sub = sub i32 %N, %h.028
+  %cmp225 = icmp eq i32 %sub, 0
+  br i1 %cmp225, label %for.cond.cleanup3, label %for.body4.lr.ph
+
+for.body4.lr.ph:                                  ; preds = %for.body
+  %mul = mul i32 %sub, %h.028
+  br label %for.body4
+
+for.cond.cleanup3:                                ; preds = %for.body4, %for.body
+  %inc11 = add nuw i32 %h.028, 1
+  %indvars.iv.next = add i32 %indvars.iv, -1
+  %exitcond29 = icmp eq i32 %inc11, %N
+  br i1 %exitcond29, label %for.cond.cleanup, label %for.body
+
+; CHECK-LABEL: for.body4
+; CHECK: %w.026 = phi i32 [ 0, %for.body4.lr.ph ], [ %inc, %for.body4 ]
+; CHECK: %inc = add nuw i32 %w.026, 1
+; CHECK: %exitcond = icmp eq i32 %inc, %indvars.iv
+for.body4:                                        ; preds = %for.body4, %for.body4.lr.ph
+  %w.026 = phi i32 [ 0, %for.body4.lr.ph ], [ %inc, %for.body4 ]
+  %add = add i32 %w.026, %mul
+  %arrayidx = getelementptr inbounds i16, i16* %A, i32 %add
+  %0 = load i16, i16* %arrayidx, align 2
+  %conv = sext i16 %0 to i32
+  %arrayidx5 = getelementptr inbounds i16, i16* %B, i32 %w.026
+  %1 = load i16, i16* %arrayidx5, align 2
+  %conv6 = sext i16 %1 to i32
+  %mul7 = mul nsw i32 %conv6, %conv
+  %arrayidx8 = getelementptr inbounds i32, i32* %C, i32 %w.026
+  %2 = load i32, i32* %arrayidx8, align 4
+  %add9 = add nsw i32 %mul7, %2
+  store i32 %add9, i32* %arrayidx8, align 4
+  %inc = add nuw i32 %w.026, 1
+  %exitcond = icmp eq i32 %inc, %indvars.iv
+  br i1 %exitcond, label %for.cond.cleanup3, label %for.body4
+}
+
+declare arm_aapcs_vfpcc i32 @some_func(i32, i32) local_unnamed_addr #2