[TypePromotion] Search from ZExt + PHI

[avieira]

Summary

Hi all,

This is a WIP patch to teach TypePromotion to promote PHI-nodes with a single use by a ZExt or SExt, where the incoming values are either constants or loads where the target can extend the loaded value for free.

See below a motivating example:

#include <stdint.h>
int foo(uint8_t* p) {
  uint16_t index = *p;
  do {
      index = p[index];
  } while (index < 100);
  return index;
}

For aarch64 this would end up with the following IR:

define dso_local i32 @_Z3fooPh(i8* nocapture readonly %p) local_unnamed_addr #0 {
entry:
  %0 = load i8, i8* %p, align 1, !tbaa !8
  br label %do.body
do.body:                                          ; preds = %do.body, %entry
  %index.0.in = phi i8 [ %0, %entry ], [ %1, %do.body ]
  %idxprom = zext i8 %index.0.in to i64
  %arrayidx = getelementptr inbounds i8, i8* %p, i64 %idxprom
  %1 = load i8, i8* %arrayidx, align 1, !tbaa !8
  %cmp = icmp ult i8 %1, 100
  br i1 %cmp, label %do.body, label %do.end, !llvm.loop !11
do.end:                                           ; preds = %do.body
  %conv2 = zext i8 %1 to i32
  ret i32 %conv2
}

Which in turn leads to an unnecessary 'and' at the start of the loop. The goal here is to turn that PHI-node to an i64 and push the zext to the loads.

Let me know what you think of this approach and whether there might be more appropriate ways to ensure we get a better codegen for issues like this.

[dmgreen]

Do you have any tests?

llvm/lib/CodeGen/TypePromotion.cpp

905

TII -> TTI

[SjoerdMeijer]

Let me know what you think of this approach and whether there might be more appropriate ways to ensure we get a better codegen for issues like this.

Sounds like a good and nice addition to this patch, and seems to be the right place for this kind of thing.

Some first drive-by comments: some comments would be nice, some testing and perf numbers too as this pass was really tricky to get right, and like Dave said, some tests.

[avieira]

I found that other than the motivation example I showed above coming up with C-level examples, especially for cases where I don't think transforming is a good idea is difficult. I'll see if I have better luck with IR tests.

llvm/lib/CodeGen/TypePromotion.cpp

905

Ah sorry didn't add that change to the patch I posted upstream :(

[SjoerdMeijer]

In D111237#3054641, @avieira wrote:

I found that other than the motivation example I showed above coming up with C-level examples, especially for cases where I don't think transforming is a good idea is difficult. I'll see if I have better luck with IR tests.

Adding your motivating example as an IR test is a good start! :) And cases that you think should not be transformed are equally valuable, so please add them too.

[avieira]

Abandoning this in favor of D112300, I think that it would be better to try to avoid the sinking of the Exts rather than trying to undo them this late.

[avieira]

Coming back to this as I abandoned the approach in D112300, since that was deemed too early to be consulting target information.

Reworked the patch to be able to support more general cases. Only promote now if all incoming values of the PHI-node lead to free extensions.

Benchmarked SPEC 2017 intrate on aarch64 and saw no regressions and a small (0.7%) performance uplift on perlbench. Size differences were negligible.

[avieira]

I forgot to mention this address https://bugs.llvm.org/show_bug.cgi?id=51317 which is a suboptimal codegen issue encountered in snappy. Currently Snappy uses an inline asm workaround to make sure it doesn't end up with an extra 'and', this patch removes the need for that workaround.

[dmgreen]

Here's a round of review. I haven't got though everything in TryToPromotePHI yet, but it mostly seemed sensible from what I could tell. Reducing the indenting would be good, if we can.

llvm/lib/CodeGen/TypePromotion.cpp

912–943

20 is quite high, it is usually around 4. I think you can leave the amount off in most cases and it will choose a sensible default.

924

It may be possible to check for legal types, as opposed to using this single RegisterBitWidth (and ignoring vector sizes).

947

I will always be an Instruction.

948–949

It is preferred in llvm to remove indenting with early exits/continues. From what I can tell, this algorithm is in two stages now, and this code needn't change? The phis are optimized first, then sequences starting from icmps.

950

I think you may be able to turn any Transform = false; break; into return false;

959

It may be better to check for legal operations instead of the cost. Not sure though with the masked operations.

976

The user of an instruction will always be an instruction. llvm likes to use continue to reduce heavy indenting too, where it can.

1044

Is this reachable?

llvm/test/Transforms/TypePromotion/AArch64/dont-promote-phi-ext.ll

5 ↗

(On Diff #388144)

I like to remove "; Function Attrs:..", "dso_local" and "local_unnamed_addr"

[samparker]

My knee jerk reaction is that this looks like a lot of code to add little functionality... the pass can already promote phis, so, IIUC, the only functional change is to enable the use of sign extends for their operands? If so, I don't really understand why this couldn't be added more gracefully into the existing search. My handwavey suggestion would be to search from SExt nodes (in the same manner as icmp, but with a search depth limit of 2), create a dedicated function to estimate the cost of promotion and modify the IRPromoter to enable sign extending some sources. Does any of that sound reasonable?

llvm/lib/CodeGen/TypePromotion.cpp

944

Can these new loops not just be fused into the original? When I originally wrote this, I treated phis the same as icmps, but admittedly didn't find any additional improvements.

[avieira]

Thanks for the suggestion @samparker . I hadn't look too much into it after I found that the ICmp promotion leaves the code in a somewhat 'dirty' state.
For instance if you take the motivating example from Snappy, the BB before the one with the PHI + ZEXT has an icmp that feeds into the relevant PHI node and it ends up yielding:

%tag.0.in10 = phi i32 [ %2, %for.body ], [ %0, %for.body.preheader ]
%1 = trunc i32 %tag.0.in10 to i8
%tag.0 = zext i8 %1 to i64

Which ofc gets cleaned up later, removing the trunc, but it would mean I'd need to account for stuff like that when pattern matching PHI + Ext. I found it was easier to do the transformation I wanted before doing the ICmp transformations. I could alternatively try to avoid generating the trunc, but I haven't looked at why it does this yet and I don't know whether it does so by design...
I realized I didn't actually include a testcase that showcases this... I'll make sure to add one.

@dmgreen Thanks for the comments, I'm looking into those.

[samparker]

I could alternatively try to avoid generating the trunc

Yes, there's a TODO about avoiding the insertion in TypePromotion::isSink

[avieira]

In D111237#3173897, @samparker wrote:

I could alternatively try to avoid generating the trunc

Yes, there's a TODO about avoiding the insertion in TypePromotion::isSink

I see. Am I misreading this or is the TODO saying we don't need isSink? That's a rather odd one ;) I could add some code to return isSink for the user of a PHI node if that PHI node only has a single use. That would work for the ZExt case, as that's recognized in isSink.

[samparker]

Sorry for missing your last comment @avieira , I've uploaded a suggested change in D115451, though I still think something is off with isSink.

[adriantong1024]

llvm/lib/CodeGen/TypePromotion.cpp

142–145

Should we change this in a separate patch, its not the intend of this patch to change how these structs are declared.

950

I agree with @dmgreen here. Once Transform becomes false, it cant become true and we end up return from this function. So should return early.

954

can we do isa<LoadInst> here ? I also see you use getOpcode() in other places. Isn't the standard way isa cast ?

[avieira]

Hi all,

This is a complete revamp of my original patch, based on @samparker's patch in https://reviews.llvm.org/D118906/new/ with some changes. The most important ones are:

• we start searching from a ZExt and use it's result type to determine the width to promote the sequence to,
• we only promote phi node's inside a loop.

Benchmarked SPEC CPU 2017 Intrate on aarch64-none-linux. No significant changes in performance or size. Benchmarked Snappy and with this patch there is no performance regression when the inline assembly hack is removed that was put in place to prevent the generation of the superfluous 'and'.

Also ran llvm-test-suite and geomean stays the same, though I must say that there was a lot of variability from run to run even with the same binaries :(

[samparker]

I'd be tempted to split this into two and handle the NFC + InstRemove fix in one and the (zext (phi)) in another, just in case the functional change needs reverting. Completely up to you though.

And thanks for doing this!

llvm/lib/CodeGen/TypePromotion.cpp

128

Use a SmallPtrSetImpl, like we do for wrap.

181

We should also mark it as preserved: AU.addPreserved<LoopInfoWrapperPass>

937

nit: indentation looks off here.

954

We know it's a phi, so use static_cast.

[avieira]

Addressed comments by @samparker. This patch depends on D131487, D131488 and D131489.

[samparker]

LGTM, cheers

[avieira]

Had not run this for ARM and there is a testcase there that also benefits from this change moving the zext up to the loads outside of the loop.

[samparker]

llvm/test/Transforms/TypePromotion/ARM/casts.ll

53 ↗

(On Diff #451774)

Update the comment please.

[avieira]

Fixed it in a NFC.

[paulwalker-arm]

@avieira This patch looks to be breaking the AArch64/SVE buildbots. It's a compile time failure where getFixedSizeInBits() is being called on a scalable vector. Can you please take a look and perhaps revert the patch if it's not a quick fix?

See https://lab.llvm.org/buildbot/#/builders/197/builds/2467. Note 2466 also has failures but I've narrowed down the MultiSource/Applications/sgefa & SingleSource/UnitTests/Vectorizer ones to this patch.

[avieira]

In D111237#3723408, @paulwalker-arm wrote:

@avieira This patch looks to be breaking the AArch64/SVE buildbots. It's a compile time failure where getFixedSizeInBits() is being called on a scalable vector. Can you please take a look and perhaps revert the patch if it's not a quick fix?

See https://lab.llvm.org/buildbot/#/builders/197/builds/2467. Note 2466 also has failures but I've narrowed down the MultiSource/Applications/sgefa & SingleSource/UnitTests/Vectorizer ones to this patch.

Thanks for reporting this Paul. This is because I don't check the type of the ZExt is an Integer, testing a patch for that now and creating a reduced IR testcase, will post it tomorrow.

[avieira]

Fix posted in https://reviews.llvm.org/D131948

[paulwalker-arm]

In D111237#3725351, @avieira wrote:

Fix posted in https://reviews.llvm.org/D131948

Thanks for the quick fix @avieira.

llvm/lib/CodeGen/TypePromotion.cpp

 /// instructions operands where a tree, consisting of non-wrapping or safe
 /// wrapping instructions, is built, checked and promoted if possible.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/Analysis/LoopInfo.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/CodeGen/TargetLowering.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/IRBuilder.h"
class IRPromoter {
   LLVMContext &Ctx;
   unsigned PromotedWidth = 0;
   SetVector<Value *> &Visited;
   SetVector<Value *> &Sources;
   SetVector<Instruction *> &Sinks;
   SmallPtrSetImpl<Instruction *> &SafeWrap;
   IntegerType *ExtTy = nullptr;
   SmallPtrSet<Value *, 8> NewInsts;
-  SmallPtrSet<Instruction *, 4> InstsToRemove;
+  SmallPtrSet<Instruction *, 4> &InstsToRemove;
   DenseMap<Value *, SmallVector<Type *, 4>> TruncTysMap;
   SmallPtrSet<Value *, 8> Promoted;
 
   void ReplaceAllUsersOfWith(Value *From, Value *To);
   void ExtendSources();
   void ConvertTruncs();
   void PromoteTree();
   void TruncateSinks();
   void Cleanup();
 
 public:
   IRPromoter(LLVMContext &C, unsigned Width,
              SetVector<Value *> &visited, SetVector<Value *> &sources,
              SetVector<Instruction *> &sinks,
-             SmallPtrSetImpl<Instruction *> &wrap)
+             SmallPtrSetImpl<Instruction *> &wrap,
+             SmallPtrSet<Instruction *, 4> &instsToRemove)
       : Ctx(C), PromotedWidth(Width), Visited(visited),
-        Sources(sources), Sinks(sinks), SafeWrap(wrap) {
+        Sources(sources), Sinks(sinks), SafeWrap(wrap),
+        InstsToRemove(instsToRemove) {
     ExtTy = IntegerType::get(Ctx, PromotedWidth);
   }
 
   void Mutate();
 };
 
 class TypePromotion : public FunctionPass {
   unsigned TypeSize = 0;
   LLVMContext *Ctx = nullptr;
   unsigned RegisterBitWidth = 0;
   SmallPtrSet<Value *, 16> AllVisited;
   SmallPtrSet<Instruction *, 8> SafeToPromote;
   SmallPtrSet<Instruction *, 4> SafeWrap;
+  SmallPtrSet<Instruction *, 4> InstsToRemove;
 
   // Does V have the same size result type as TypeSize.
   bool EqualTypeSize(Value *V);
   // Does V have the same size, or narrower, result type as TypeSize.
   bool LessOrEqualTypeSize(Value *V);
   // Does V have a result type that is wider than TypeSize.
   bool GreaterThanTypeSize(Value *V);
   // Does V have a result type that is narrower than TypeSize.
class TypePromotion : public FunctionPass {
   bool TryToPromote(Value *V, unsigned PromotedWidth);
 
 public:
   static char ID;
 
   TypePromotion() : FunctionPass(ID) {}
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<LoopInfoWrapperPass>();
     AU.addRequired<TargetTransformInfoWrapperPass>();
     AU.addRequired<TargetPassConfig>();
     AU.setPreservesCFG();
   }
 
   StringRef getPassName() const override { return PASS_NAME; }
 
   bool runOnFunction(Function &F) override;
if (NewInsts.count(Src) && isa<TruncInst>(Src)) {
              "expected inserted trunc to be operating on i32");
       ReplaceAllUsersOfWith(ZExt, Trunc->getOperand(0));
     }
   }
 
   for (auto *I : InstsToRemove) {
     LLVM_DEBUG(dbgs() << "IR Promotion: Removing " << *I << "\n");
     I->dropAllReferences();
-    I->eraseFromParent();
   }
 }
 
 void IRPromoter::ConvertTruncs() {
   LLVM_DEBUG(dbgs() << "IR Promotion: Converting truncs..\n");
   IRBuilder<> Builder{Ctx};
 
   for (auto *V : Visited) {
for (auto *V : CurrentVisited) {
 
     if (Sinks.count(cast<Instruction>(V)))
       continue;
     ++ToPromote;
   }
 
   // DAG optimizations should be able to handle these cases better, especially
   // for function arguments.
-  if (ToPromote < 2 || (Blocks.size() == 1 && (NonFreeArgs > SafeWrap.size())))
+  if (!isa<PHINode>(V)
+      && (ToPromote < 2 || (Blocks.size() == 1 && (NonFreeArgs > SafeWrap.size()))))
     return false;
 
   IRPromoter Promoter(*Ctx, PromotedWidth, CurrentVisited, Sources, Sinks,
-                      SafeWrap);
+                      SafeWrap, InstsToRemove);
   Promoter.Mutate();
   return true;
 }
 
 bool TypePromotion::runOnFunction(Function &F) {
   if (skipFunction(F) || DisablePromotion)
     return false;
 
   LLVM_DEBUG(dbgs() << "IR Promotion: Running on " << F.getName() << "\n");
 
   auto *TPC = getAnalysisIfAvailable<TargetPassConfig>();
   if (!TPC)
     return false;
 
   AllVisited.clear();
   SafeToPromote.clear();
   SafeWrap.clear();
   bool MadeChange = false;
   const DataLayout &DL = F.getParent()->getDataLayout();
   const TargetMachine &TM = TPC->getTM<TargetMachine>();
   const TargetSubtargetInfo *SubtargetInfo = TM.getSubtargetImpl(F);
   const TargetLowering *TLI = SubtargetInfo->getTargetLowering();
   const TargetTransformInfo &TII =
       getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+  const LoopInfo &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
   RegisterBitWidth =
       TII.getRegisterBitWidth(TargetTransformInfo::RGK_Scalar).getFixedSize();
   Ctx = &F.getParent()->getContext();
 
-  // Search up from icmps to try to promote their operands.
+  // Return the preferred integer width of the instruction, or zero if we
+  // shouldn't try.
+  auto GetPromoteWidth = [&](Instruction *I) -> uint32_t {
+    if (!isa<IntegerType>(I->getType()))
+      return 0;
+
+    EVT SrcVT = TLI->getValueType(DL, I->getType());
+    if (SrcVT.isSimple() && TLI->isTypeLegal(SrcVT.getSimpleVT()))
+      return 0;
+
+    if (TLI->getTypeAction(*Ctx, SrcVT) != TargetLowering::TypePromoteInteger)
+      return 0;
+
+    EVT PromotedVT = TLI->getTypeToTransformTo(*Ctx, SrcVT);
+    if (RegisterBitWidth < PromotedVT.getFixedSizeInBits()) {
+      LLVM_DEBUG(dbgs() << "IR Promotion: Couldn't find target register "
+                        << "for promoted type\n");
+      return 0;
+    }
+
+    // TODO: Should we prefer to use RegisterBitWidth instead?
+    return PromotedVT.getFixedSizeInBits();
+  };
+
+  auto BBIsInLoop = [&](BasicBlock *BB) -> bool {
+      for (auto *L : LI)
+       if (L->contains(BB))
+         return true;
+      return false;
+  };
+
+  // Search up from icmps and phis to try to promote their operands.
   for (BasicBlock &BB : F) {
     for (Instruction &I : BB) {
       if (AllVisited.count(&I))
         continue;
 
-      if (!isa<ICmpInst>(&I))
-        continue;
-
-      auto *ICmp = cast<ICmpInst>(&I);
+      if (isa<ZExtInst>(&I) && isa<PHINode>(I.getOperand(0)) &&
+          BBIsInLoop(&BB)) {
+        LLVM_DEBUG(dbgs() << "IR Promotion: Searching from: " <<
+                   I.getOperand(0) << "\n");
+        EVT ZExtVT = TLI->getValueType(DL, I.getType());
+        Instruction *Phi = dyn_cast<Instruction>(I.getOperand(0));
+        MadeChange |= TryToPromote(Phi, ZExtVT.getFixedSizeInBits());
+      } else if (auto *ICmp = dyn_cast<ICmpInst>(&I)) {
         // Skip signed or pointer compares
-      if (ICmp->isSigned() || !isa<IntegerType>(ICmp->getOperand(0)->getType()))
+        if (ICmp->isSigned())
           continue;
 
         LLVM_DEBUG(dbgs() << "IR Promotion: Searching from: " << *ICmp << "\n");
 
         for (auto &Op : ICmp->operands()) {
-        if (auto *I = dyn_cast<Instruction>(Op)) {
-          EVT SrcVT = TLI->getValueType(DL, I->getType());
-          if (SrcVT.isSimple() && TLI->isTypeLegal(SrcVT.getSimpleVT()))
-            break;
-
-          if (TLI->getTypeAction(*Ctx, SrcVT) !=
-              TargetLowering::TypePromoteInteger)
-            break;
-          EVT PromotedVT = TLI->getTypeToTransformTo(*Ctx, SrcVT);
-          if (RegisterBitWidth < PromotedVT.getFixedSizeInBits()) {
-            LLVM_DEBUG(dbgs() << "IR Promotion: Couldn't find target register "
-                              << "for promoted type\n");
+          if (auto *OpI = dyn_cast<Instruction>(Op)) {
+            if (auto PromotedWidth = GetPromoteWidth(OpI)) {
+              MadeChange |= TryToPromote(OpI, PromotedWidth);
               break;
             }
-
-          MadeChange |= TryToPromote(I, PromotedVT.getFixedSizeInBits());
-          break;
           }
         }
       }
     }
+    if (!InstsToRemove.empty()) {
+      for (auto *I : InstsToRemove)
+        I->eraseFromParent();
+      InstsToRemove.clear();
+    }
+  }
 
   AllVisited.clear();
   SafeToPromote.clear();
   SafeWrap.clear();
 
   return MadeChange;
 }
 
 INITIALIZE_PASS_BEGIN(TypePromotion, DEBUG_TYPE, PASS_NAME, false, false)
 INITIALIZE_PASS_END(TypePromotion, DEBUG_TYPE, PASS_NAME, false, false)
 
 char TypePromotion::ID = 0;
 
 FunctionPass *llvm::createTypePromotionPass() { return new TypePromotion(); }

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       AArch64 Stack Tagging
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       Memory SSA
 ; CHECK-NEXT:       Interleaved Load Combine Pass
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Interleaved Access Pass
-; CHECK-NEXT:       Type Promotion
 ; CHECK-NEXT:       Natural Loop Information
+; CHECK-NEXT:       Type Promotion
 ; CHECK-NEXT:       CodeGen Prepare
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Exception handling preparation
 ; CHECK-NEXT:     AArch64 Promote Constant
 ; CHECK-NEXT:       FunctionPass Manager
 ; CHECK-NEXT:         Dominator Tree Construction
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Merge internal globals

llvm/test/Transforms/TypePromotion/AArch64/phi-zext-gep.ll

 ; CHECK-NEXT:    [[TAG_0_IN:%.*]] = load i8, i8* [[ARRAYIDX]], align 1
 ; CHECK-NEXT:    [[TMP1]] = zext i8 [[TAG_0_IN]] to i32
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp ult i32 [[TMP1]], 100
 ; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_END_LOOPEXIT:%.*]]
 ; CHECK:       for.end.loopexit:
 ; CHECK-NEXT:    br label [[FOR_END]]
 ; CHECK:       for.end:
 ; CHECK-NEXT:    [[TAG_0_IN_LCSSA:%.*]] = phi i32 [ [[TMP0]], [[ENTRY:%.*]] ], [ [[TMP1]], [[FOR_END_LOOPEXIT]] ]
-; CHECK-NEXT:    [[TMP2:%.*]] = trunc i32 [[TAG_0_IN_LCSSA]] to i8
 ; CHECK-NEXT:    ret i32 [[TAG_0_IN_LCSSA]]
 ;
 entry:
   %tag.0.in8 = load i8, i8* %ip, align 1
   %cmp9 = icmp ult i8 %tag.0.in8, 100
   br i1 %cmp9, label %for.body.preheader, label %for.end
 
 for.body.preheader:                               ; preds = %entry

llvm/test/Transforms/TypePromotion/AArch64/phi-zext-gep2.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=aarch64 -type-promotion -verify -S %s -o - | FileCheck %s
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+
+; Function Attrs: mustprogress nofree nosync nounwind uwtable
+define dso_local void @foo(ptr noundef %ptr0, ptr nocapture noundef readonly %ptr1, ptr nocapture noundef %dest) local_unnamed_addr {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load i8, ptr [[PTR0:%.*]], align 1
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i8 [[TMP0]] to i64
+; CHECK-NEXT:    br label [[DO_BODY:%.*]]
+; CHECK:       do.body:
+; CHECK-NEXT:    [[TO_PROMOTE:%.*]] = phi i64 [ [[TMP1]], [[ENTRY:%.*]] ], [ [[TMP4:%.*]], [[DO_BODY]] ]
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i8, ptr [[PTR1:%.*]], i64 [[TO_PROMOTE]]
+; CHECK-NEXT:    [[TMP2:%.*]] = load i8, ptr [[ARRAYIDX1]], align 2
+; CHECK-NEXT:    [[TMP3:%.*]] = zext i8 [[TMP2]] to i64
+; CHECK-NEXT:    [[COND_IN_I:%.*]] = getelementptr inbounds i8, ptr [[PTR1]], i64 [[TMP3]]
+; CHECK-NEXT:    [[COND_I:%.*]] = load i8, ptr [[COND_IN_I]], align 1
+; CHECK-NEXT:    [[TMP4]] = zext i8 [[COND_I]] to i64
+; CHECK-NEXT:    store i8 [[TMP2]], ptr [[DEST:%.*]], align 1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ult i64 [[TMP4]], 0
+; CHECK-NEXT:    br i1 [[CMP]], label [[DO_BODY]], label [[DO_END:%.*]]
+; CHECK:       do.end:
+; CHECK-NEXT:    ret void
+;
+entry:
+  %0 = load i8, ptr %ptr0, align 1
+  br label %do.body
+
+do.body:                                          ; preds = %do.body, %entry
+  %to_promote = phi i8 [ %0, %entry ], [ %cond.i, %do.body ]
+  %ext0 = zext i8 %to_promote to i64
+  %arrayidx1 = getelementptr inbounds i8, ptr %ptr1, i64 %ext0
+  %1 = load i8, ptr %arrayidx1, align 2
+  %2 = zext i8 %1 to i64
+  %cond.in.i = getelementptr inbounds i8, ptr %ptr1, i64 %2
+  %cond.i = load i8, ptr %cond.in.i, align 1
+  store i8 %1, ptr %dest, align 1
+  %cmp = icmp ult i8 %cond.i, 0
+  br i1 %cmp, label %do.body, label %do.end
+
+do.end:                                           ; preds = %do.body
+  ret void
+}