[InstCombine] convert assumes to operand bundles

[Tyker]

Summary

Instcombine will convert the nonnull and alignment assumption that use the boolean condtion
to an assumption that uses the operand bundles when knowledge retention is enabled.

[Tyker]

there is an NFC change to the test that was excluded from the diff to make it easier to read.

[jdoerfert]

Initial comments.

llvm/include/llvm/Transforms/Utils/AssumeBundleBuilder.h

45

Please add some documentation here.

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

4230

I don't get this. Doesn't it replace the operand of an assume with false if there is one with the same condition following? That seems wrong, maybe "true" is what you wanted?

4253

Comments on what is happening here and below please.

4265

Isn't this call the same as the 5 lines before?

llvm/lib/Transforms/Utils/AssumeBundleBuilder.cpp

291

It is odd that this takes an insertion point but apparently does not insert the instruction. Can we change either to confuse me less?

llvm/test/Transforms/InstCombine/assume.ll

253

Why do we have SAME checking the function twice? Is this a bug in the update script?

[Tyker]

addressed the comments
the previous patch had many stupid mistakes

llvm/test/Transforms/InstCombine/assume.ll

253

Is this a bug in the update script?

perhaps but after purging every CHECK line the issue is gone.

[jdoerfert]

Looks conceptually good but there are some oddities, potentially in the lookup.

llvm/test/Transforms/InstCombine/assume.ll

85

This doesn't seem to work properly.

@lebedev.ri found other problematic cases. One that I now know of is multiple bundles: https://godbolt.org/z/To5obz

469

Can u add a FIXME or NOTE here to indicate we could duplicate the load instead and keep a assume with ["nonnull"(%load)] around.

790

Again a fold missing CMP = true

[Tyker]

I had to merge the previous patch with other i had laying around and add some more
to make the bundle mode be at least as good as the default. so this revision got
quite a bit bigger.

I hope its still fine.

[jdoerfert]

llvm/include/llvm/Analysis/AssumeBundleQueries.h

111 ↗

(On Diff #276824)

This is not a strict order. Let's use AttrKind and WasOn as well.

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

93

Add a sentence what this controls/means.

4219

I'd add a message to the assert and a comment before the lambda.

4290

Shouldn't we build a GEP for the offset instead, if the offset is not a multiple of the alignment that is.

[Tyker]

addressed comments

[jdoerfert]

llvm/include/llvm/Analysis/AssumeBundleQueries.h

114 ↗

(On Diff #283996)

Nit: Add a message to the assert please explaining what usage is allowed.

llvm/lib/Analysis/Loads.cpp

94 ↗

(On Diff #283996)

Nit: Add a comment describe what is happening here.

I guess we should have a todo that states partial information, e.g., only AlignRK or DerefRK, should be used in other places too. The entire function should be rewritten for that so it is out of scope.

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

4290

At least a TODO would be good.

[Tyker]

Addressed comments.

[Tyker]

rebased
add cannonicalization of assume bundles

[jdoerfert]

llvm/test/Transforms/InstCombine/assume.ll

387

It is questionable if we should prefer the assume or the !nonnull metadata here.

962

Also unclear if this is better than the GEP. I guess the case with an offset of 4 is clear but this one I don't know.

[xbolva00]

Ping

Resolves https://bugs.llvm.org/show_bug.cgi?id=48224

[jdoerfert]

@Tyker are you going to upstream this? If not, can someone take over?

[Tyker]

In D82703#2489196, @jdoerfert wrote:

@Tyker are you going to upstream this? If not, can someone take over?

I can do the upstreaming but i don't know what the current state is for upstreaming this.

llvm/test/Transforms/InstCombine/assume.ll

962

this is one of the default canonicalization done when building assumes maybe we should keep both.
the idea behind it is to make the knowledge usable by all derived pointers and also minimize the number of assumes but i agree it is not clear its better.

[Tyker]

llvm/test/Transforms/InstCombine/assume.ll

387

fixed

[jdoerfert]

In D82703#2503196, @Tyker wrote:

In D82703#2489196, @jdoerfert wrote:

@Tyker are you going to upstream this? If not, can someone take over?

I can do the upstreaming but i don't know what the current state is for upstreaming this.

I think there are different things in this patch, some can be upstreamed some need minor adjustment, and some I don't understand yet. See my comments. It would be great if we can finish this line of assume optimizations though.

llvm/include/llvm/Transforms/Utils/AssumeBundleBuilder.h

65

Nit: The first sentence is hard to read.

llvm/lib/Analysis/ValueTracking.cpp

756 ↗

(On Diff #317164)

Why can we allow ephemeral here without potentially throwing away an assume that was used to argue it itself is useless? Should we not do the AllowEphemeral stuff for now?

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

4221

Nit: Typo, arguement

4356

Nit: typo: help

4363

losses what?

4371

I'm not sure why we return nullptr here and not continue going.

llvm/lib/Transforms/Utils/AssumeBundleBuilder.cpp

298

!Assume means Assume->GetModule() will crash, probably add a DL argument.

llvm/test/Analysis/ValueTracking/assume.ll

75 ↗

(On Diff #317164)

How do we get an alignment of 8 here?

[Tyker]

rebased, addressed comments. fixed issues.

llvm/lib/Analysis/ValueTracking.cpp

756 ↗

(On Diff #317164)

i fixed the issue AllowEphemeral was "solving" by other means.

llvm/test/Analysis/ValueTracking/assume.ll

75 ↗

(On Diff #317164)

this was just an addition arbitrary test. left the old test as is a added a new.

llvm/test/Transforms/InstCombine/assume.ll

962

I think we should try to avoid having any extra instructions that are only used by an assume because it is part of the issue with the boolean assume model.

With the current representation of RetainedKnowledge there is no way to represent offsets.
this restriction could be lifetd by refactoring and is not the only limitation of RetainedKnowledge that could be removed.
when forming a RetainedKnowledge of a "align"(i8* %3, i64 16, i64 4) the RetainedKnowledge will conservativly have the value 4.

since canonicalization is done on RetainedKnowledge cannonicalization can't interact with offsets and stays conservative.

[jdoerfert]

One minor comment and a nit, I assume those can be addressed before committing the patch. LGTM, thanks for finishing this up!

llvm/lib/Analysis/Loads.cpp

102 ↗

(On Diff #320289)

Nit: If you move the lambda declaration before the conditional, we get one level of indirection less.

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

4367

Nit: early continue, if (CanonRK == RK) continue

4383

Again, unsure why we return. there might be operand bundles left.

[leonardchan]

Hi. I suspect this patch might be causing this test failure we're seeing on our builders (https://luci-milo.appspot.com/p/fuchsia/builders/ci/clang-linux-x64/b8855749711840653472?):

FAIL: Clang :: CodeGen/builtin-align-assumption.c (3421 of 27242)
******************** TEST 'Clang :: CodeGen/builtin-align-assumption.c' FAILED ********************
Script:
--
: 'RUN: at line 3';   /b/s/w/ir/x/w/staging/llvm_build/bin/clang -cc1 -internal-isystem /b/s/w/ir/x/w/staging/llvm_build/lib/clang/13.0.0/include -nostdsysteminc -triple=x86_64-unknown-unknown /b/s/w/ir/x/w/llvm-project/clang/test/CodeGen/builtin-align-assumption.c -emit-llvm -O1 -o - | /b/s/w/ir/x/w/staging/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-project/clang/test/CodeGen/builtin-align-assumption.c
--
Exit Code: 1

Command Output (stderr):
--
/b/s/w/ir/x/w/llvm-project/clang/test/CodeGen/builtin-align-assumption.c:6:11: error: CHECK: expected string not found in input
// CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 64 dereferenceable(16) {{%.+}}, i8* nonnull align 1 dereferenceable(16) {{%.+}}, i64 16, i1 false)
          ^
<stdin>:7:32: note: scanning from here
define dso_local void @align_up(i8* nocapture readonly %data, i32* %ptr) local_unnamed_addr #0 {
                               ^
<stdin>:16:2: note: possible intended match here
 call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 4 dereferenceable(16) %aligned_result, i8* nonnull align 1 dereferenceable(16) %data, i64 16, i1 false)
 ^

Input file: <stdin>
Check file: /b/s/w/ir/x/w/llvm-project/clang/test/CodeGen/builtin-align-assumption.c

-dump-input=help explains the following input dump.

...

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
********************
Failed Tests (1):
  Clang :: CodeGen/builtin-align-assumption.c

Would you mind taking a look and either send out a fix or reverting? Thanks.

[justice_adams]

Hi @Tyker , I believe this patch is causing the following test to fail: clang/test/CodeGen/builtin-align-assumption.c

Here's some buildbot links for reference:
http://lab.llvm.org:8011/#/builders/123/builds/2854
http://lab.llvm.org:8011/#/builders/110/builds/888

Could you take a look?

[jdoerfert]

llvm/test/Transforms/InstCombine/assume.ll

2–3

It seems the RUN lines did not get included in this commit. I doubt this is testing much right now :(

llvm/include/llvm/Transforms/Utils/AssumeBundleBuilder.h

 // transformation may destroy or worsen information that can be derived from the
 // IR.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TRANSFORMS_UTILS_ASSUMEBUNDLEBUILDER_H
 #define LLVM_TRANSFORMS_UTILS_ASSUMEBUNDLEBUILDER_H
 
+#include "llvm/Analysis/AssumeBundleQueries.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/PassManager.h"
 
 namespace llvm {
 class IntrinsicInst;
 class AssumptionCache;
 class DominatorTree;

 /// contained in the instruction I.
 /// The AssumptionCache must be provided if it is available or the cache may
 /// become silently be invalid.
 /// The DominatorTree can optionally be provided to enable cross-block
 /// reasoning.
 void salvageKnowledge(Instruction *I, AssumptionCache *AC = nullptr,
                       DominatorTree *DT = nullptr);
 
+IntrinsicInst *buildAssumeFromKnowledge(ArrayRef<RetainedKnowledge> Knowledge,
+                                        Instruction *InsertPoint,
+                                        AssumptionCache *AC = nullptr,
+                                        DominatorTree *DT = nullptr);
+
 /// This pass attempts to minimize the number of assume without loosing any
 /// information.
 struct AssumeSimplifyPass : public PassInfoMixin<AssumeSimplifyPass> {
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 };
 
 FunctionPass *createAssumeSimplifyPass();
 
 /// This pass will try to build an llvm.assume for every instruction in the
 /// function. Its main purpose is testing.
 struct AssumeBuilderPass : public PassInfoMixin<AssumeBuilderPass> {
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 };
 
 } // namespace llvm
 
 #endif

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/InstrTypes.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Intrinsics.h"
-#include "llvm/IR/IntrinsicsX86.h"
-#include "llvm/IR/IntrinsicsARM.h"
 #include "llvm/IR/IntrinsicsAArch64.h"
+#include "llvm/IR/IntrinsicsAMDGPU.h"
+#include "llvm/IR/IntrinsicsARM.h"
 #include "llvm/IR/IntrinsicsHexagon.h"
 #include "llvm/IR/IntrinsicsNVPTX.h"
-#include "llvm/IR/IntrinsicsAMDGPU.h"
 #include "llvm/IR/IntrinsicsPowerPC.h"
+#include "llvm/IR/IntrinsicsX86.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/PatternMatch.h"
 #include "llvm/IR/Statepoint.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/User.h"
 #include "llvm/IR/Value.h"
 #include "llvm/IR/ValueHandle.h"
 #include "llvm/Support/AtomicOrdering.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/KnownBits.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/InstCombine/InstCombineWorklist.h"
+#include "llvm/Transforms/Utils/AssumeBundleBuilder.h"
 #include "llvm/Transforms/Utils/Local.h"
 #include "llvm/Transforms/Utils/SimplifyLibCalls.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <cstring>
 #include <utility>
 #include <vector>
 
 using namespace llvm;
 using namespace PatternMatch;
 
 #define DEBUG_TYPE "instcombine"
 
 STATISTIC(NumSimplified, "Number of library calls simplified");
 
 static cl::opt<unsigned> GuardWideningWindow(
     "instcombine-guard-widening-window",
     cl::init(3),
     cl::desc("How wide an instruction window to bypass looking for "
              "another guard"));
 
+extern cl::opt<bool> EnableKnowledgeRetention;
+
 /// Return the specified type promoted as it would be to pass though a va_arg
 /// area.
 static Type *getPromotedType(Type *Ty) {
   if (IntegerType* ITy = dyn_cast<IntegerType>(Ty)) {
     if (ITy->getBitWidth() < 32)
       return Type::getInt32Ty(Ty->getContext());
   }
   return Ty;
if (removeTriviallyEmptyRange(*II, *this, [](const IntrinsicInst &I) {
           return I.getIntrinsicID() == Intrinsic::lifetime_start;
         }))
       return nullptr;
     break;
   case Intrinsic::assume: {
     Value *IIOperand = II->getArgOperand(0);
     SmallVector<OperandBundleDef, 4> OpBundles;
     II->getOperandBundlesAsDefs(OpBundles);
-    bool HasOpBundles = !OpBundles.empty();
+    auto RemoveConditionFromAssume = [&](Instruction *Assume) -> Instruction * {
+      assert(isa<IntrinsicInst>(Assume));
+      if (isAssumeWithEmptyBundle(*cast<IntrinsicInst>(II)))
+        return eraseInstFromFunction(CI);
+      replaceUse(II->getOperandUse(0), ConstantInt::getFalse(II->getContext()));
+      return nullptr;
+    };
     // Remove an assume if it is followed by an identical assume.
     // TODO: Do we need this? Unless there are conflicting assumptions, the
     // computeKnownBits(IIOperand) below here eliminates redundant assumes.
     Instruction *Next = II->getNextNonDebugInstruction();
-    if (HasOpBundles &&
-        match(Next, m_Intrinsic<Intrinsic::assume>(m_Specific(IIOperand))) &&
-        !cast<IntrinsicInst>(Next)->hasOperandBundles())
-      return eraseInstFromFunction(CI);
+    if (match(Next, m_Intrinsic<Intrinsic::assume>(m_Specific(IIOperand))))
+      return RemoveConditionFromAssume(Next);
 
     // Canonicalize assume(a && b) -> assume(a); assume(b);
     // Note: New assumption intrinsics created here are registered by
     // the InstCombineIRInserter object.
     FunctionType *AssumeIntrinsicTy = II->getFunctionType();
     Value *AssumeIntrinsic = II->getCalledOperand();
     Value *A, *B;
     if (match(IIOperand, m_And(m_Value(A), m_Value(B)))) {
       Builder.CreateCall(AssumeIntrinsicTy, AssumeIntrinsic, A, OpBundles,
                          II->getName());
       Builder.CreateCall(AssumeIntrinsicTy, AssumeIntrinsic, B, II->getName());
       return eraseInstFromFunction(*II);
     }
     // assume(!(a || b)) -> assume(!a); assume(!b);
     if (match(IIOperand, m_Not(m_Or(m_Value(A), m_Value(B))))) {
       Builder.CreateCall(AssumeIntrinsicTy, AssumeIntrinsic,
                          Builder.CreateNot(A), OpBundles, II->getName());
       Builder.CreateCall(AssumeIntrinsicTy, AssumeIntrinsic,
                          Builder.CreateNot(B), II->getName());
       return eraseInstFromFunction(*II);
     }
 
+    CmpInst::Predicate Pred;
+    if (EnableKnowledgeRetention &&
+        match(IIOperand, m_Cmp(Pred, m_Value(A), m_Zero())) &&
+        Pred == CmpInst::ICMP_NE && A->getType()->isPointerTy()) {
+      if (IntrinsicInst *Replacement = buildAssumeFromKnowledge(
+              {RetainedKnowledge{Attribute::NonNull, 0, A}}, Next, &AC, &DT))
+        Replacement->insertBefore(Next);
+      if (isAssumeWithEmptyBundle(*cast<IntrinsicInst>(II)))
+        return eraseInstFromFunction(CI);
+      else
+        replaceUse(II->getOperandUse(0),
+                   ConstantInt::getFalse(II->getContext()));
+      return RemoveConditionFromAssume(II);
+    }
+
+    uint64_t AlignMask;
+    if (EnableKnowledgeRetention &&
+        match(IIOperand,
+              m_Cmp(Pred, m_And(m_Value(A), m_ConstantInt(AlignMask)),
+                    m_Zero())) &&
+        Pred == CmpInst::ICMP_EQ) {
+      if (isPowerOf2_64(AlignMask + 1)) {
+        uint64_t Offset = 0;
+        match(A, m_Add(m_Value(A), m_ConstantInt(Offset)));
+        if (match(A, m_PtrToInt(m_Value(A)))) {
+          Value *OffsetV = nullptr;
+          LLVMContext &C = A->getContext();
+          if (Offset)
+            OffsetV = ConstantInt::get(Type::getInt64Ty(C), Offset);
+          Builder.CreateAlignmentAssumption(DL, A, AlignMask + 1, OffsetV);
+          return RemoveConditionFromAssume(II);
+        }
+      }
+    }
+
     // assume( (load addr) != null ) -> add 'nonnull' metadata to load
     // (if assume is valid at the load)
-    CmpInst::Predicate Pred;
     Instruction *LHS;
     if (match(IIOperand, m_ICmp(Pred, m_Instruction(LHS), m_Zero())) &&
         Pred == ICmpInst::ICMP_NE && LHS->getOpcode() == Instruction::Load &&
         LHS->getType()->isPointerTy() &&
         isValidAssumeForContext(II, LHS, &DT)) {
       MDNode *MD = MDNode::get(II->getContext(), None);
       LHS->setMetadata(LLVMContext::MD_nonnull, MD);
-      if (!HasOpBundles)
+      if (isAssumeWithEmptyBundle(*II))
         return eraseInstFromFunction(*II);
 
       // TODO: apply nonnull return attributes to calls and invokes
       // TODO: apply range metadata for range check patterns?
     }
 
     // If there is a dominating assume with the same condition as this one,
     // then this one is redundant, and should be removed.
if (auto *PT = dyn_cast<PointerType>(II->getType())) {
       // TODO: provide a hook for this in GCStrategy.  There might be some
       // weird collector this property does not hold for.
       if (isa<ConstantPointerNull>(DerivedPtr))
         // Use null-pointer of gc_relocate's type to replace it.
         return replaceInstUsesWith(*II, ConstantPointerNull::get(PT));
 
       // isKnownNonNull -> nonnull attribute
       if (!II->hasRetAttr(Attribute::NonNull) &&
           isKnownNonZero(DerivedPtr, DL, 0, &AC, II, &DT)) {
         II->addAttribute(AttributeList::ReturnIndex, Attribute::NonNull);
         return II;
       }
     }
 
     // TODO: bitcast(relocate(p)) -> relocate(bitcast(p))
     // Canonicalize on the type from the uses to the defs
 
     // TODO: relocate((gep p, C, C2, ...)) -> gep(relocate(p), C, C2, ...)
     break;
   }
 
   case Intrinsic::experimental_guard: {
     // Is this guard followed by another guard?  We scan forward over a small
     // fixed window of instructions to handle common cases with conditions
     // computed between guards.
     Instruction *NextInst = II->getNextNonDebugInstruction();
     for (unsigned i = 0; i < GuardWideningWindow; i++) {
       // Note: Using context-free form to avoid compile time blow up
       if (!isSafeToSpeculativelyExecute(NextInst))
         break;
       NextInst = NextInst->getNextNonDebugInstruction();
     }
     Value *NextCond = nullptr;
     if (match(NextInst,
               m_Intrinsic<Intrinsic::experimental_guard>(m_Value(NextCond)))) {
       Value *CurrCond = II->getArgOperand(0);
 
       // Remove a guard that it is immediately preceded by an identical guard.
       // Otherwise canonicalize guard(a); guard(b) -> guard(a & b).
       if (CurrCond != NextCond) {
         Instruction *MoveI = II->getNextNonDebugInstruction();
         while (MoveI != NextInst) {
           auto *Temp = MoveI;
           MoveI = MoveI->getNextNonDebugInstruction();

llvm/lib/Transforms/Utils/AssumeBundleBuilder.cpp

void llvm::salvageKnowledge(Instruction *I, AssumptionCache *AC,
   Builder.addInstruction(I);
   if (IntrinsicInst *Intr = Builder.build()) {
     Intr->insertBefore(I);
     if (AC)
       AC->registerAssumption(Intr);
   }
 }
 
+IntrinsicInst *
+llvm::buildAssumeFromKnowledge(ArrayRef<RetainedKnowledge> Knowledge,
+                               Instruction *InsertPoint, AssumptionCache *AC,
+                               DominatorTree *DT) {
+  AssumeBuilderState Builder(InsertPoint->getModule(), InsertPoint, AC, DT);
+  for (const RetainedKnowledge &RK : Knowledge)
+    Builder.addKnowledge(RK);
+  return Builder.build();
+}
+
 namespace {
 
 struct AssumeSimplify {
   Function &F;
   AssumptionCache &AC;
   DominatorTree *DT;
   LLVMContext &C;
   SmallDenseSet<IntrinsicInst *> CleanupToDo;
   StringMapEntry<uint32_t> *IgnoreTag;
   SmallDenseMap<BasicBlock *, SmallVector<IntrinsicInst *, 4>, 8> BBToAssume;
   bool MadeChange = false;
 
   AssumeSimplify(Function &F, AssumptionCache &AC, DominatorTree *DT,
                  LLVMContext &C)

llvm/test/Transforms/InstCombine/assume.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -instcombine -S | FileCheck --check-prefixes=SAME,DEFAULT %s
 ; RUN: opt < %s -instcombine --enable-knowledge-retention -S | FileCheck --check-prefixes=SAME,BUNDLES %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @foo1(i32* %a) #0 {
-; CHECK-LABEL: @foo1(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A:%.*]], align 32
-; CHECK-NEXT:    [[PTRINT:%.*]] = ptrtoint i32* [[A]] to i64
-; CHECK-NEXT:    [[MASKEDPTR:%.*]] = and i64 [[PTRINT]], 31
-; CHECK-NEXT:    [[MASKCOND:%.*]] = icmp eq i64 [[MASKEDPTR]], 0
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[MASKCOND]])
-; CHECK-NEXT:    ret i32 [[TMP0]]
+; DEFAULT-LABEL: @foo1(
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A:%.*]], align 32
+; DEFAULT-NEXT:    [[PTRINT:%.*]] = ptrtoint i32* [[A]] to i64
+; DEFAULT-NEXT:    [[MASKEDPTR:%.*]] = and i64 [[PTRINT]], 31
+; DEFAULT-NEXT:    [[MASKCOND:%.*]] = icmp eq i64 [[MASKEDPTR]], 0
+; DEFAULT-NEXT:    tail call void @llvm.assume(i1 [[MASKCOND]])
+; DEFAULT-NEXT:    ret i32 [[TMP0]]
+;
+; BUNDLES-LABEL: @foo1(
+; BUNDLES-NEXT:  entry:
+; BUNDLES-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A:%.*]], align 32
+; BUNDLES-NEXT:    call void @llvm.assume(i1 true) [ "align"(i32* [[A]], i64 32) ]
+; BUNDLES-NEXT:    ret i32 [[TMP0]]
 ;
 ; SAME-LABEL: @foo1(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A:%.*]], align 32
 ; SAME-NEXT:    [[PTRINT:%.*]] = ptrtoint i32* [[A]] to i64
 ; SAME-NEXT:    [[MASKEDPTR:%.*]] = and i64 [[PTRINT]], 31
 ; SAME-NEXT:    [[MASKCOND:%.*]] = icmp eq i64 [[MASKEDPTR]], 0
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[MASKCOND]])
; been removed:
   %maskedptr = and i64 %ptrint, 31
   %maskcond = icmp eq i64 %maskedptr, 0
   tail call void @llvm.assume(i1 %maskcond)
 
   ret i32 %0
 }
 
 define i32 @foo2(i32* %a) #0 {
-; CHECK-LABEL: @foo2(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[PTRINT:%.*]] = ptrtoint i32* [[A:%.*]] to i64
-; CHECK-NEXT:    [[MASKEDPTR:%.*]] = and i64 [[PTRINT]], 31
-; CHECK-NEXT:    [[MASKCOND:%.*]] = icmp eq i64 [[MASKEDPTR]], 0
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[MASKCOND]])
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A]], align 32
-; CHECK-NEXT:    ret i32 [[TMP0]]
+; DEFAULT-LABEL: @foo2(
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[PTRINT:%.*]] = ptrtoint i32* [[A:%.*]] to i64
+; DEFAULT-NEXT:    [[MASKEDPTR:%.*]] = and i64 [[PTRINT]], 31
+; DEFAULT-NEXT:    [[MASKCOND:%.*]] = icmp eq i64 [[MASKEDPTR]], 0
+; DEFAULT-NEXT:    tail call void @llvm.assume(i1 [[MASKCOND]])
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A]], align 32
+; DEFAULT-NEXT:    ret i32 [[TMP0]]
+;
+; BUNDLES-LABEL: @foo2(
+; BUNDLES-NEXT:  entry:
+; BUNDLES-NEXT:    call void @llvm.assume(i1 true) [ "align"(i32* [[A:%.*]], i64 32) ]
+; BUNDLES-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A]], align 4
+; BUNDLES-NEXT:    ret i32 [[TMP0]]
 ;
 ; SAME-LABEL: @foo2(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[PTRINT:%.*]] = ptrtoint i32* [[A:%.*]] to i64
 ; SAME-NEXT:    [[MASKEDPTR:%.*]] = and i64 [[PTRINT]], 31
 ; SAME-NEXT:    [[MASKCOND:%.*]] = icmp eq i64 [[MASKEDPTR]], 0
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[MASKCOND]])
 ; SAME-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A]], align 32
 ; SAME-NEXT:    ret i32 [[TMP0]]
 ;
 entry:
 ; Same check as in @foo1, but make sure it works if the assume is first too.
 
   %ptrint = ptrtoint i32* %a to i64
   %maskedptr = and i64 %ptrint, 31
   %maskcond = icmp eq i64 %maskedptr, 0
   tail call void @llvm.assume(i1 %maskcond)
 
   %0 = load i32, i32* %a, align 4
   ret i32 %0
 }
 
 declare void @llvm.assume(i1) #1
 
 define i32 @simple(i32 %a) #1 {
-; CHECK-LABEL: @simple(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[A:%.*]], 4
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i32 4
+; SAME-LABEL: @simple(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[A:%.*]], 4
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    ret i32 4
 ;
 ; SAME-LABEL: @simple(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[A:%.*]], 4
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    ret i32 4
 ;
 entry:
 
 
   %cmp = icmp eq i32 %a, 4
   tail call void @llvm.assume(i1 %cmp)
   ret i32 %a
 }
 
 define i32 @can1(i1 %a, i1 %b, i1 %c) {
-; CHECK-LABEL: @can1(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    call void @llvm.assume(i1 [[A:%.*]])
-; CHECK-NEXT:    call void @llvm.assume(i1 [[B:%.*]])
-; CHECK-NEXT:    call void @llvm.assume(i1 [[C:%.*]])
-; CHECK-NEXT:    ret i32 5
+; SAME-LABEL: @can1(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    call void @llvm.assume(i1 [[A:%.*]])
+; SAME-NEXT:    call void @llvm.assume(i1 [[B:%.*]])
+; SAME-NEXT:    call void @llvm.assume(i1 [[C:%.*]])
+; SAME-NEXT:    ret i32 5
 ;
 ; SAME-LABEL: @can1(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    call void @llvm.assume(i1 [[A:%.*]])
 ; SAME-NEXT:    call void @llvm.assume(i1 [[B:%.*]])
 ; SAME-NEXT:    call void @llvm.assume(i1 [[C:%.*]])
 ; SAME-NEXT:    ret i32 5
 ;
 entry:
   %and1 = and i1 %a, %b
   %and  = and i1 %and1, %c
   tail call void @llvm.assume(i1 %and)
 
 
   ret i32 5
 }
 
 define i32 @can2(i1 %a, i1 %b, i1 %c) {
-; CHECK-LABEL: @can2(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = xor i1 [[A:%.*]], true
-; CHECK-NEXT:    call void @llvm.assume(i1 [[TMP0]])
-; CHECK-NEXT:    [[TMP1:%.*]] = xor i1 [[B:%.*]], true
-; CHECK-NEXT:    call void @llvm.assume(i1 [[TMP1]])
-; CHECK-NEXT:    ret i32 5
+; SAME-LABEL: @can2(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    [[TMP0:%.*]] = xor i1 [[A:%.*]], true
+; SAME-NEXT:    call void @llvm.assume(i1 [[TMP0]])
+; SAME-NEXT:    [[TMP1:%.*]] = xor i1 [[B:%.*]], true
+; SAME-NEXT:    call void @llvm.assume(i1 [[TMP1]])
+; SAME-NEXT:    ret i32 5
 ;
 ; SAME-LABEL: @can2(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[TMP0:%.*]] = xor i1 [[A:%.*]], true
 ; SAME-NEXT:    call void @llvm.assume(i1 [[TMP0]])
 ; SAME-NEXT:    [[TMP1:%.*]] = xor i1 [[B:%.*]], true
 ; SAME-NEXT:    call void @llvm.assume(i1 [[TMP1]])
 ; SAME-NEXT:    ret i32 5
 ;
 entry:
   %v = or i1 %a, %b
   %w = xor i1 %v, 1
   tail call void @llvm.assume(i1 %w)
 
 
   ret i32 5
 }
 
 define i32 @bar1(i32 %a) #0 {
-; CHECK-LABEL: @bar1(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i32 1
+; SAME-LABEL: @bar1(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
+; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    ret i32 1
 ;
 ; SAME-LABEL: @bar1(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
 ; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    ret i32 1
 ;
 entry:
   %and1 = and i32 %a, 3
 
 
   %and = and i32 %a, 7
   %cmp = icmp eq i32 %and, 1
   tail call void @llvm.assume(i1 %cmp)
 
   ret i32 %and1
 }
 
 define i32 @bar2(i32 %a) #0 {
-; CHECK-LABEL: @bar2(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i32 1
+; SAME-LABEL: @bar2(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
+; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    ret i32 1
 ;
 ; SAME-LABEL: @bar2(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
 ; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    ret i32 1
 ;
 entry:
 
   %and = and i32 %a, 7
   %cmp = icmp eq i32 %and, 1
   tail call void @llvm.assume(i1 %cmp)
 
   %and1 = and i32 %a, 3
   ret i32 %and1
 }
 
 define i32 @bar3(i32 %a, i1 %x, i1 %y) #0 {
-; CHECK-LABEL: @bar3(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[X:%.*]])
-; CHECK-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[Y:%.*]])
-; CHECK-NEXT:    ret i32 1
+; SAME-LABEL: @bar3(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[X:%.*]])
+; SAME-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
+; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[Y:%.*]])
+; SAME-NEXT:    ret i32 1
 ;
 ; SAME-LABEL: @bar3(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[X:%.*]])
 ; SAME-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
 ; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[Y:%.*]])
; Don't be fooled by other assumes around.
   tail call void @llvm.assume(i1 %cmp)
 
   tail call void @llvm.assume(i1 %y)
 
   ret i32 %and1
 }
 
 define i32 @bar4(i32 %a, i32 %b) {
-; CHECK-LABEL: @bar4(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
-; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    [[CMP2:%.*]] = icmp eq i32 [[A]], [[B:%.*]]
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
-; CHECK-NEXT:    ret i32 1
+; SAME-LABEL: @bar4(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
+; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    [[CMP2:%.*]] = icmp eq i32 [[A]], [[B:%.*]]
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
+; SAME-NEXT:    ret i32 1
 ;
 ; SAME-LABEL: @bar4(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[AND:%.*]] = and i32 [[A:%.*]], 7
 ; SAME-NEXT:    [[CMP:%.*]] = icmp eq i32 [[AND]], 1
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    [[CMP2:%.*]] = icmp eq i32 [[A]], [[B:%.*]]
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
 ; SAME-NEXT:    ret i32 1
 ;
 entry:
   %and1 = and i32 %b, 3
 
 
   %and = and i32 %a, 7
   %cmp = icmp eq i32 %and, 1
   tail call void @llvm.assume(i1 %cmp)
 
   %cmp2 = icmp eq i32 %a, %b
   tail call void @llvm.assume(i1 %cmp2)
 
   ret i32 %and1
 }
 
 define i32 @icmp1(i32 %a) #0 {
-; CHECK-LABEL: @icmp1(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[A:%.*]], 5
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i32 1
+; SAME-LABEL: @icmp1(
+; SAME-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[A:%.*]], 5
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    ret i32 1
 ;
 ; SAME-LABEL: @icmp1(
 ; SAME-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[A:%.*]], 5
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    ret i32 1
 ;
   %cmp = icmp sgt i32 %a, 5
   tail call void @llvm.assume(i1 %cmp)
   %conv = zext i1 %cmp to i32
   ret i32 %conv
 }
 
 define i32 @icmp2(i32 %a) #0 {
-; CHECK-LABEL: @icmp2(
-; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[A:%.*]], 5
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i32 0
+; SAME-LABEL: @icmp2(
+; SAME-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[A:%.*]], 5
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    ret i32 0
 ;
 ; SAME-LABEL: @icmp2(
 ; SAME-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[A:%.*]], 5
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    ret i32 0
 ;
   %cmp = icmp sgt i32 %a, 5
   tail call void @llvm.assume(i1 %cmp)
   %t0 = zext i1 %cmp to i32
   %lnot.ext = xor i32 %t0, 1
   ret i32 %lnot.ext
 }
 
 ; If the 'not' of a condition is known true, then the condition must be false.
 
 define i1 @assume_not(i1 %cond) {
-; CHECK-LABEL: @assume_not(
-; CHECK-NEXT:    [[NOTCOND:%.*]] = xor i1 [[COND:%.*]], true
-; CHECK-NEXT:    call void @llvm.assume(i1 [[NOTCOND]])
-; CHECK-NEXT:    ret i1 false
+; SAME-LABEL: @assume_not(
+; SAME-NEXT:    [[NOTCOND:%.*]] = xor i1 [[COND:%.*]], true
+; SAME-NEXT:    call void @llvm.assume(i1 [[NOTCOND]])
+; SAME-NEXT:    ret i1 false
 ;
 ; SAME-LABEL: @assume_not(
 ; SAME-NEXT:    [[NOTCOND:%.*]] = xor i1 [[COND:%.*]], true
 ; SAME-NEXT:    call void @llvm.assume(i1 [[NOTCOND]])
 ; SAME-NEXT:    ret i1 false
 ;
   %notcond = xor i1 %cond, true
   call void @llvm.assume(i1 %notcond)
   ret i1 %cond
 }
 
 declare void @escape(i32* %a)
 
 ; Canonicalize a nonnull assumption on a load into metadata form.
 
 define i32 @bundle1(i32* %P) {
-; CHECK-LABEL: @bundle1(
-; CHECK-NEXT:    tail call void @llvm.assume(i1 true) [ "nonnull"(i32* [[P:%.*]]) ]
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32, i32* [[P]], align 4
-; CHECK-NEXT:    ret i32 [[LOAD]]
+; SAME-LABEL: @bundle1(
+; SAME-NEXT:    tail call void @llvm.assume(i1 true) [ "nonnull"(i32* [[P:%.*]]) ]
+; SAME-NEXT:    [[LOAD:%.*]] = load i32, i32* [[P]], align 4
+; SAME-NEXT:    ret i32 [[LOAD]]
 ;
 ; SAME-LABEL: @bundle1(
 ; SAME-NEXT:    tail call void @llvm.assume(i1 true) [ "nonnull"(i32* [[P:%.*]]) ]
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32, i32* [[P]], align 4
 ; SAME-NEXT:    ret i32 [[LOAD]]
 ;
   tail call void @llvm.assume(i1 true) ["nonnull"(i32* %P)]
   %load = load i32, i32* %P
   ret i32 %load
 }
 
 define i32 @bundle2(i32* %P) {
-; CHECK-LABEL: @bundle2(
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32, i32* [[P:%.*]], align 4
-; CHECK-NEXT:    ret i32 [[LOAD]]
+; SAME-LABEL: @bundle2(
+; SAME-NEXT:    [[LOAD:%.*]] = load i32, i32* [[P:%.*]], align 4
+; SAME-NEXT:    ret i32 [[LOAD]]
 ;
 ; SAME-LABEL: @bundle2(
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32, i32* [[P:%.*]], align 4
 ; SAME-NEXT:    ret i32 [[LOAD]]
 ;
   tail call void @llvm.assume(i1 true) ["ignore"(i32* undef)]
   %load = load i32, i32* %P
   ret i32 %load
 }
 
 define i1 @nonnull1(i32** %a) {
-; CHECK-LABEL: @nonnull1(
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8, !nonnull !6
-; CHECK-NEXT:    tail call void @escape(i32* nonnull [[LOAD]])
-; CHECK-NEXT:    ret i1 false
+; DEFAULT-LABEL: @nonnull1(
+; DEFAULT-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8, !nonnull !6
+; DEFAULT-NEXT:    tail call void @escape(i32* nonnull [[LOAD]])
+; DEFAULT-NEXT:    ret i1 false
+;
+; BUNDLES-LABEL: @nonnull1(
+; BUNDLES-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; BUNDLES-NEXT:    call void @llvm.assume(i1 true) [ "nonnull"(i32* [[LOAD]]) ]
+; BUNDLES-NEXT:    tail call void @escape(i32* nonnull [[LOAD]])
+; BUNDLES-NEXT:    ret i1 false
 ;
 ; SAME-LABEL: @nonnull1(
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8, !nonnull !6
 ; SAME-NEXT:    tail call void @escape(i32* nonnull [[LOAD]])
 ; SAME-NEXT:    ret i1 false
 ;
   %load = load i32*, i32** %a
   %cmp = icmp ne i32* %load, null
   tail call void @llvm.assume(i1 %cmp)
   tail call void @escape(i32* %load)
   %rval = icmp eq i32* %load, null
   ret i1 %rval
 }
 
 ; Make sure the above canonicalization applies only
 ; to pointer types.  Doing otherwise would be illegal.
 
 define i1 @nonnull2(i32* %a) {
-; CHECK-LABEL: @nonnull2(
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32, i32* [[A:%.*]], align 4
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i32 [[LOAD]], 0
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i1 false
+; SAME-LABEL: @nonnull2(
+; SAME-NEXT:    [[LOAD:%.*]] = load i32, i32* [[A:%.*]], align 4
+; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32 [[LOAD]], 0
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    ret i1 false
 ;
 ; SAME-LABEL: @nonnull2(
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32, i32* [[A:%.*]], align 4
 ; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32 [[LOAD]], 0
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    ret i1 false
 ;
   %load = load i32, i32* %a
   %cmp = icmp ne i32 %load, 0
   tail call void @llvm.assume(i1 %cmp)
   %rval = icmp eq i32 %load, 0
   ret i1 %rval
 }
 
 ; Make sure the above canonicalization does not trigger
 ; if the assume is control dependent on something else
 
 define i1 @nonnull3(i32** %a, i1 %control) {
-; CHECK-LABEL: @nonnull3(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
-; CHECK-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
-; CHECK:       taken:
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i1 false
-; CHECK:       not_taken:
-; CHECK-NEXT:    [[RVAL_2:%.*]] = icmp sgt i32* [[LOAD]], null
-; CHECK-NEXT:    ret i1 [[RVAL_2]]
+; DEFAULT-LABEL: @nonnull3(
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; DEFAULT-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
+; DEFAULT-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
+; DEFAULT:       taken:
+; DEFAULT-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; DEFAULT-NEXT:    ret i1 false
+; DEFAULT:       not_taken:
+; DEFAULT-NEXT:    [[RVAL_2:%.*]] = icmp sgt i32* [[LOAD]], null
+; DEFAULT-NEXT:    ret i1 [[RVAL_2]]
+;
+; BUNDLES-LABEL: @nonnull3(
+; BUNDLES-NEXT:  entry:
+; BUNDLES-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
+; BUNDLES:       taken:
+; BUNDLES-NEXT:    ret i1 false
+; BUNDLES:       not_taken:
+; BUNDLES-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; BUNDLES-NEXT:    [[RVAL_2:%.*]] = icmp sgt i32* [[LOAD]], null
+; BUNDLES-NEXT:    ret i1 [[RVAL_2]]
 ;
 ; SAME-LABEL: @nonnull3(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
 ; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
 ; SAME-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
 ; SAME:       taken:
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
not_taken:
   ret i1 %rval.2
 }
 
 ; Make sure the above canonicalization does not trigger
 ; if the path from the load to the assume is potentially
 ; interrupted by an exception being thrown
 
 define i1 @nonnull4(i32** %a) {
-; CHECK-LABEL: @nonnull4(
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
-; CHECK-NEXT:    tail call void @escape(i32* [[LOAD]])
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i1 false
+; DEFAULT-LABEL: @nonnull4(
+; DEFAULT-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; DEFAULT-NEXT:    tail call void @escape(i32* [[LOAD]])
+; DEFAULT-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
+; DEFAULT-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; DEFAULT-NEXT:    ret i1 false
+;
+; BUNDLES-LABEL: @nonnull4(
+; BUNDLES-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; BUNDLES-NEXT:    tail call void @escape(i32* [[LOAD]])
+; BUNDLES-NEXT:    call void @llvm.assume(i1 true) [ "nonnull"(i32* [[LOAD]]) ]
+; BUNDLES-NEXT:    ret i1 false
 ;
 ; SAME-LABEL: @nonnull4(
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
 ; SAME-NEXT:    tail call void @escape(i32* [[LOAD]])
 ; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    ret i1 false
 ;
   %load = load i32*, i32** %a
   ;; This call may throw!
   tail call void @escape(i32* %load)
   %cmp = icmp ne i32* %load, null
   tail call void @llvm.assume(i1 %cmp)
   %rval = icmp eq i32* %load, null
   ret i1 %rval
 }
 define i1 @nonnull5(i32** %a) {
-; CHECK-LABEL: @nonnull5(
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
-; CHECK-NEXT:    tail call void @escape(i32* [[LOAD]])
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32* [[LOAD]], null
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i1 false
+; SAME-LABEL: @nonnull5(
+; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; SAME-NEXT:    tail call void @escape(i32* [[LOAD]])
+; SAME-NEXT:    [[CMP:%.*]] = icmp slt i32* [[LOAD]], null
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
+; SAME-NEXT:    ret i1 false
 ;
 ; SAME-LABEL: @nonnull5(
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
 ; SAME-NEXT:    tail call void @escape(i32* [[LOAD]])
 ; SAME-NEXT:    [[CMP:%.*]] = icmp slt i32* [[LOAD]], null
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP]])
 ; SAME-NEXT:    ret i1 false
 ;
   %load = load i32*, i32** %a
   ;; This call may throw!
   tail call void @escape(i32* %load)
   %integral = ptrtoint i32* %load to i64
   %cmp = icmp slt i64 %integral, 0
   tail call void @llvm.assume(i1 %cmp) ; %load has at least highest bit set
   %rval = icmp eq i32* %load, null
   ret i1 %rval
 }
 
 ; PR35846 - https://bugs.llvm.org/show_bug.cgi?id=35846
 
 define i32 @assumption_conflicts_with_known_bits(i32 %a, i32 %b) {
-; CHECK-LABEL: @assumption_conflicts_with_known_bits(
-; CHECK-NEXT:    [[AND1:%.*]] = and i32 [[B:%.*]], 3
-; CHECK-NEXT:    tail call void @llvm.assume(i1 false)
-; CHECK-NEXT:    [[CMP2:%.*]] = icmp eq i32 [[AND1]], 0
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
-; CHECK-NEXT:    ret i32 0
+; SAME-LABEL: @assumption_conflicts_with_known_bits(
+; SAME-NEXT:    [[AND1:%.*]] = and i32 [[B:%.*]], 3
+; SAME-NEXT:    tail call void @llvm.assume(i1 false)
+; SAME-NEXT:    [[CMP2:%.*]] = icmp eq i32 [[AND1]], 0
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
+; SAME-NEXT:    ret i32 0
 ;
 ; SAME-LABEL: @assumption_conflicts_with_known_bits(
 ; SAME-NEXT:    [[AND1:%.*]] = and i32 [[B:%.*]], 3
 ; SAME-NEXT:    tail call void @llvm.assume(i1 false)
 ; SAME-NEXT:    [[CMP2:%.*]] = icmp eq i32 [[AND1]], 0
 ; SAME-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
 ; SAME-NEXT:    ret i32 0
 ;
   %and1 = and i32 %b, 3
   %B1 = lshr i32 %and1, %and1
   %B3 = shl nuw nsw i32 %and1, %B1
   %cmp = icmp eq i32 %B3, 1
   tail call void @llvm.assume(i1 %cmp)
   %cmp2 = icmp eq i32 %B1, %B3
   tail call void @llvm.assume(i1 %cmp2)
   ret i32 %and1
 }
 
 ; PR37726 - https://bugs.llvm.org/show_bug.cgi?id=37726
 ; There's a loophole in eliminating a redundant assumption when
 ; we have conflicting assumptions. Verify that debuginfo doesn't
 ; get in the way of the fold.
 
 define void @debug_interference(i8 %x) {
-; CHECK-LABEL: @debug_interference(
-; CHECK-NEXT:    [[CMP2:%.*]] = icmp ne i8 [[X:%.*]], 0
-; CHECK-NEXT:    tail call void @llvm.assume(i1 false)
-; CHECK-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
-; CHECK-NEXT:    tail call void @llvm.assume(i1 false)
-; CHECK-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
-; CHECK-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
-; CHECK-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
-; CHECK-NEXT:    ret void
+; DEFAULT-LABEL: @debug_interference(
+; DEFAULT-NEXT:    [[CMP2:%.*]] = icmp ne i8 [[X:%.*]], 0
+; DEFAULT-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
+; DEFAULT-NEXT:    tail call void @llvm.assume(i1 false)
+; DEFAULT-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
+; DEFAULT-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
+; DEFAULT-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
+; DEFAULT-NEXT:    ret void
+;
+; BUNDLES-LABEL: @debug_interference(
+; BUNDLES-NEXT:    [[CMP2:%.*]] = icmp ne i8 [[X:%.*]], 0
+; BUNDLES-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !6, metadata !DIExpression()), !dbg !8
+; BUNDLES-NEXT:    tail call void @llvm.assume(i1 false)
+; BUNDLES-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !6, metadata !DIExpression()), !dbg !8
+; BUNDLES-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !6, metadata !DIExpression()), !dbg !8
+; BUNDLES-NEXT:    tail call void @llvm.assume(i1 [[CMP2]])
+; BUNDLES-NEXT:    ret void
 ;
 ; SAME-LABEL: @debug_interference(
 ; SAME-NEXT:    [[CMP2:%.*]] = icmp ne i8 [[X:%.*]], 0
 ; SAME-NEXT:    tail call void @llvm.assume(i1 false)
 ; SAME-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
 ; SAME-NEXT:    tail call void @llvm.assume(i1 false)
 ; SAME-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
 ; SAME-NEXT:    tail call void @llvm.dbg.value(metadata i32 5, metadata !7, metadata !DIExpression()), !dbg !9
;
   tail call void @llvm.assume(i1 %cmp2)
   ret void
 }
 
 ; This would crash.
 ; Does it ever make sense to peek through a bitcast of the icmp operand?
 
 define i32 @PR40940(<4 x i8> %x) {
-; CHECK-LABEL: @PR40940(
-; CHECK-NEXT:    [[SHUF:%.*]] = shufflevector <4 x i8> [[X:%.*]], <4 x i8> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 3>
-; CHECK-NEXT:    [[T2:%.*]] = bitcast <4 x i8> [[SHUF]] to i32
-; CHECK-NEXT:    [[T3:%.*]] = icmp ult i32 [[T2]], 65536
-; CHECK-NEXT:    call void @llvm.assume(i1 [[T3]])
-; CHECK-NEXT:    ret i32 [[T2]]
+; SAME-LABEL: @PR40940(
+; SAME-NEXT:    [[SHUF:%.*]] = shufflevector <4 x i8> [[X:%.*]], <4 x i8> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 3>
+; SAME-NEXT:    [[T2:%.*]] = bitcast <4 x i8> [[SHUF]] to i32
+; SAME-NEXT:    [[T3:%.*]] = icmp ult i32 [[T2]], 65536
+; SAME-NEXT:    call void @llvm.assume(i1 [[T3]])
+; SAME-NEXT:    ret i32 [[T2]]
 ;
 ; SAME-LABEL: @PR40940(
 ; SAME-NEXT:    [[SHUF:%.*]] = shufflevector <4 x i8> [[X:%.*]], <4 x i8> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 3>
 ; SAME-NEXT:    [[T2:%.*]] = bitcast <4 x i8> [[SHUF]] to i32
 ; SAME-NEXT:    [[T3:%.*]] = icmp ult i32 [[T2]], 65536
 ; SAME-NEXT:    call void @llvm.assume(i1 [[T3]])
 ; SAME-NEXT:    ret i32 [[T2]]
 ;
   %shuf = shufflevector <4 x i8> %x, <4 x i8> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 3>
   %t2 = bitcast <4 x i8> %shuf to i32
   %t3 = icmp ult i32 %t2, 65536
   call void @llvm.assume(i1 %t3)
   ret i32 %t2
 }
 
 define i1 @nonnull3A(i32** %a, i1 %control) {
-; CHECK-LABEL: @nonnull3A(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
-; CHECK-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
-; CHECK:       taken:
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
-; CHECK-NEXT:    call void @llvm.assume(i1 [[CMP]])
-; CHECK-NEXT:    ret i1 true
-; CHECK:       not_taken:
-; CHECK-NEXT:    [[RVAL_2:%.*]] = icmp sgt i32* [[LOAD]], null
-; CHECK-NEXT:    ret i1 [[RVAL_2]]
+; DEFAULT-LABEL: @nonnull3A(
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; DEFAULT-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
+; DEFAULT:       taken:
+; DEFAULT-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
+; DEFAULT-NEXT:    call void @llvm.assume(i1 [[CMP]])
+; DEFAULT-NEXT:    ret i1 true
+; DEFAULT:       not_taken:
+; DEFAULT-NEXT:    [[RVAL_2:%.*]] = icmp sgt i32* [[LOAD]], null
+; DEFAULT-NEXT:    ret i1 [[RVAL_2]]
+;
+; BUNDLES-LABEL: @nonnull3A(
+; BUNDLES-NEXT:  entry:
+; BUNDLES-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; BUNDLES-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
+; BUNDLES:       taken:
+; BUNDLES-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
+; BUNDLES-NEXT:    call void @llvm.assume(i1 true) [ "nonnull"(i32* [[LOAD]]) ]
+; BUNDLES-NEXT:    ret i1 [[CMP]]
+; BUNDLES:       not_taken:
+; BUNDLES-NEXT:    [[RVAL_2:%.*]] = icmp sgt i32* [[LOAD]], null
+; BUNDLES-NEXT:    ret i1 [[RVAL_2]]
 ;
 ; SAME-LABEL: @nonnull3A(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
 ; SAME-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
 ; SAME:       taken:
 ; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
 ; SAME-NEXT:    call void @llvm.assume(i1 [[CMP]])
taken:
   call void @llvm.assume(i1 %cmp)
   ret i1 %cmp
 not_taken:
   call void @llvm.assume(i1 %cmp)
   %rval.2 = icmp sgt i32* %load, null
   ret i1 %rval.2
 }
 
+define i1 @nonnull3E(i32** %a) {
+; DEFAULT-LABEL: @nonnull3E(
+; DEFAULT-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; DEFAULT-NEXT:    [[NULL:%.*]] = icmp ne i32* [[LOAD]], null
+; DEFAULT-NEXT:    call void @llvm.assume(i1 [[NULL]])
+; DEFAULT-NEXT:    ret i1 false
+;
+; BUNDLES-LABEL: @nonnull3E(
+; BUNDLES-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; BUNDLES-NEXT:    call void @llvm.assume(i1 true) [ "nonnull"(i32* [[LOAD]]) ]
+; BUNDLES-NEXT:    ret i1 false
+;
+  %load = load i32*, i32** %a
+  %null = icmp ne i32* %load, null
+  call void @llvm.assume(i1 %null)
+  ret i1 false
+}
+
+define i32* @align1(i32** %a) {
+; DEFAULT-LABEL: @align1(
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    [[TMP0:%.*]] = load i32*, i32** [[A:%.*]], align 4
+; DEFAULT-NEXT:    [[PTRINT:%.*]] = ptrtoint i32** [[A]] to i64
+; DEFAULT-NEXT:    [[OFFSETEDINT:%.*]] = add i64 [[PTRINT]], 16
+; DEFAULT-NEXT:    [[MASKEDPTR:%.*]] = and i64 [[OFFSETEDINT]], 127
+; DEFAULT-NEXT:    [[MASKCOND:%.*]] = icmp eq i64 [[MASKEDPTR]], 0
+; DEFAULT-NEXT:    tail call void @llvm.assume(i1 [[MASKCOND]])
+; DEFAULT-NEXT:    ret i32* [[TMP0]]
+;
+; BUNDLES-LABEL: @align1(
+; BUNDLES-NEXT:  entry:
+; BUNDLES-NEXT:    [[TMP0:%.*]] = load i32*, i32** [[A:%.*]], align 4
+; BUNDLES-NEXT:    call void @llvm.assume(i1 true) [ "align"(i32** [[A]], i64 128, i64 16) ]
+; BUNDLES-NEXT:    ret i32* [[TMP0]]
+;
+entry:
+  %0 = load i32*, i32** %a, align 4
+
+  %ptrint = ptrtoint i32** %a to i64
+  %offsetedint = add i64 %ptrint, 16
+  %maskedptr = and i64 %offsetedint, 127
+  %maskcond = icmp eq i64 %maskedptr, 0
+  tail call void @llvm.assume(i1 %maskcond)
+
+  ret i32* %0
+}
+
+define i32* @align2(i32** %a) {
+; SAME-LABEL: @align2(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    [[TMP0:%.*]] = load i32*, i32** [[A:%.*]], align 4
+; SAME-NEXT:    [[PTRINT:%.*]] = ptrtoint i32** [[A]] to i64
+; SAME-NEXT:    [[MASKEDPTR:%.*]] = and i64 [[PTRINT]], 85
+; SAME-NEXT:    [[MASKCOND:%.*]] = icmp eq i64 [[MASKEDPTR]], 0
+; SAME-NEXT:    tail call void @llvm.assume(i1 [[MASKCOND]])
+; SAME-NEXT:    ret i32* [[TMP0]]
+;
+entry:
+  %0 = load i32*, i32** %a, align 4
+
+  %ptrint = ptrtoint i32** %a to i64
+  %maskedptr = and i64 %ptrint, 85
+  %maskcond = icmp eq i64 %maskedptr, 0
+  tail call void @llvm.assume(i1 %maskcond)
+
+  ret i32* %0
+}
+
 define i1 @nonnull3B(i32** %a, i1 %control) {
-; CHECK-LABEL: @nonnull3B(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
-; CHECK:       taken:
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
-; CHECK-NEXT:    call void @llvm.assume(i1 [[CMP]]) [ "nonnull"(i32* [[LOAD]]), "nonnull"(i1 [[CMP]]) ]
-; CHECK-NEXT:    ret i1 true
-; CHECK:       not_taken:
-; CHECK-NEXT:    ret i1 [[CONTROL]]
+; DEFAULT-LABEL: @nonnull3B(
+; DEFAULT-NEXT:  entry:
+; DEFAULT-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
+; DEFAULT:       taken:
+; DEFAULT-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; DEFAULT-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
+; DEFAULT-NEXT:    call void @llvm.assume(i1 [[CMP]]) [ "nonnull"(i32* [[LOAD]]), "nonnull"(i1 [[CMP]]) ]
+; DEFAULT-NEXT:    ret i1 true
+; DEFAULT:       not_taken:
+; DEFAULT-NEXT:    ret i1 [[CONTROL]]
+;
+; BUNDLES-LABEL: @nonnull3B(
+; BUNDLES-NEXT:  entry:
+; BUNDLES-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
+; BUNDLES:       taken:
+; BUNDLES-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; BUNDLES-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
+; BUNDLES-NEXT:    call void @llvm.assume(i1 false) [ "nonnull"(i32* [[LOAD]]), "nonnull"(i1 [[CMP]]) ]
+; BUNDLES-NEXT:    ret i1 [[CMP]]
+; BUNDLES:       not_taken:
+; BUNDLES-NEXT:    ret i1 [[CONTROL]]
 ;
 ; SAME-LABEL: @nonnull3B(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
 ; SAME:       taken:
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
 ; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
 ; SAME-NEXT:    call void @llvm.assume(i1 [[CMP]]) [ "nonnull"(i32* [[LOAD]]), "nonnull"(i1 [[CMP]]) ]

 not_taken:
   call void @llvm.assume(i1 %cmp) ["nonnull"(i32* %load), "nonnull"(i1 %cmp)]
   ret i1 %control
 }
 
 declare i1 @tmp1(i1)
 
 define i1 @nonnull3C(i32** %a, i1 %control) {
-; CHECK-LABEL: @nonnull3C(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
-; CHECK:       taken:
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
-; CHECK-NEXT:    [[CMP2:%.*]] = call i1 @tmp1(i1 [[CMP]])
-; CHECK-NEXT:    br label [[EXIT:%.*]]
-; CHECK:       exit:
-; CHECK-NEXT:    ret i1 [[CMP2]]
-; CHECK:       not_taken:
-; CHECK-NEXT:    ret i1 [[CONTROL]]
+; SAME-LABEL: @nonnull3C(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
+; SAME:       taken:
+; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
+; SAME-NEXT:    [[CMP2:%.*]] = call i1 @tmp1(i1 [[CMP]])
+; SAME-NEXT:    br label [[EXIT:%.*]]
+; SAME:       exit:
+; SAME-NEXT:    ret i1 [[CMP2]]
+; SAME:       not_taken:
+; SAME-NEXT:    ret i1 [[CONTROL]]
 ;
 ; SAME-LABEL: @nonnull3C(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
 ; SAME:       taken:
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
 ; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
 ; SAME-NEXT:    [[CMP2:%.*]] = call i1 @tmp1(i1 [[CMP]])
exit:
   call void @llvm.assume(i1 %cmp) ["nonnull"(i32* %load), "nonnull"(i1 %cmp)]
   ret i1 %cmp2
 not_taken:
   call void @llvm.assume(i1 %cmp)
   ret i1 %control
 }
 
 define i1 @nonnull3D(i32** %a, i1 %control) {
-; CHECK-LABEL: @nonnull3D(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
-; CHECK:       taken:
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
-; CHECK-NEXT:    [[CMP2:%.*]] = call i1 @tmp1(i1 [[CMP]])
-; CHECK-NEXT:    br label [[EXIT:%.*]]
-; CHECK:       exit:
-; CHECK-NEXT:    ret i1 [[CMP2]]
-; CHECK:       not_taken:
-; CHECK-NEXT:    call void @llvm.assume(i1 true) [ "ignore"(i32* undef), "ignore"(i1 undef), "nonnull"(i1 [[CONTROL]]) ]
-; CHECK-NEXT:    ret i1 [[CONTROL]]
+; SAME-LABEL: @nonnull3D(
+; SAME-NEXT:  entry:
+; SAME-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
+; SAME:       taken:
+; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
+; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
+; SAME-NEXT:    [[CMP2:%.*]] = call i1 @tmp1(i1 [[CMP]])
+; SAME-NEXT:    br label [[EXIT:%.*]]
+; SAME:       exit:
+; SAME-NEXT:    ret i1 [[CMP2]]
+; SAME:       not_taken:
+; SAME-NEXT:    call void @llvm.assume(i1 true) [ "ignore"(i32* undef), "ignore"(i1 undef), "nonnull"(i1 [[CONTROL]]) ]
+; SAME-NEXT:    ret i1 [[CONTROL]]
 ;
 ; SAME-LABEL: @nonnull3D(
 ; SAME-NEXT:  entry:
 ; SAME-NEXT:    br i1 [[CONTROL:%.*]], label [[TAKEN:%.*]], label [[NOT_TAKEN:%.*]]
 ; SAME:       taken:
 ; SAME-NEXT:    [[LOAD:%.*]] = load i32*, i32** [[A:%.*]], align 8
 ; SAME-NEXT:    [[CMP:%.*]] = icmp ne i32* [[LOAD]], null
 ; SAME-NEXT:    [[CMP2:%.*]] = call i1 @tmp1(i1 [[CMP]])
exit:
   ret i1 %cmp2
 not_taken:
   call void @llvm.assume(i1 %cmp) ["nonnull"(i32* %load), "nonnull"(i1 %cmp), "nonnull"(i1 %control)]
   ret i1 %control
 }
 
 
 define void @always_true_assumption() {
-; CHECK-LABEL: @always_true_assumption(
-; CHECK-NEXT:    ret void
+; SAME-LABEL: @always_true_assumption(
+; SAME-NEXT:    ret void
 ;
 ; SAME-LABEL: @always_true_assumption(
 ; SAME-NEXT:    ret void
 ;
   call void @llvm.assume(i1 true)
   ret void
 
 }
 
 ; The alloca guarantees that the low bits of %a are zero because of alignment.
 ; The assume says the opposite. Make sure we don't crash.
 
 define i64 @PR31809() {
-; CHECK-LABEL: @PR31809(
-; CHECK-NEXT:    [[A:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    [[T1:%.*]] = ptrtoint i32* [[A]] to i64
-; CHECK-NEXT:    call void @llvm.assume(i1 false)
-; CHECK-NEXT:    ret i64 [[T1]]
+; SAME-LABEL: @PR31809(
+; SAME-NEXT:    [[A:%.*]] = alloca i32, align 4
+; SAME-NEXT:    [[T1:%.*]] = ptrtoint i32* [[A]] to i64
+; SAME-NEXT:    call void @llvm.assume(i1 false)
+; SAME-NEXT:    ret i64 [[T1]]
 ;
 ; SAME-LABEL: @PR31809(
 ; SAME-NEXT:    [[A:%.*]] = alloca i32, align 4
 ; SAME-NEXT:    [[T1:%.*]] = ptrtoint i32* [[A]] to i64
 ; SAME-NEXT:    call void @llvm.assume(i1 false)
 ; SAME-NEXT:    ret i64 [[T1]]
 ;
   %a = alloca i32
   %t1 = ptrtoint i32* %a to i64
   %cond = icmp eq i64 %t1, 3
   call void @llvm.assume(i1 %cond)
   ret i64 %t1
 }
 
 ; Similar to above: there's no way to know which assumption is truthful,
 ; so just don't crash.
 
 define i8 @conflicting_assumptions(i8 %x){
-; CHECK-LABEL: @conflicting_assumptions(
-; CHECK-NEXT:    call void @llvm.assume(i1 false)
-; CHECK-NEXT:    [[COND2:%.*]] = icmp eq i8 [[X:%.*]], 4
-; CHECK-NEXT:    call void @llvm.assume(i1 [[COND2]])
-; CHECK-NEXT:    ret i8 5
+; SAME-LABEL: @conflicting_assumptions(
+; SAME-NEXT:    call void @llvm.assume(i1 false)
+; SAME-NEXT:    [[COND2:%.*]] = icmp eq i8 [[X:%.*]], 4
+; SAME-NEXT:    call void @llvm.assume(i1 [[COND2]])
+; SAME-NEXT:    ret i8 5
 ;
 ; SAME-LABEL: @conflicting_assumptions(
 ; SAME-NEXT:    call void @llvm.assume(i1 false)
 ; SAME-NEXT:    [[COND2:%.*]] = icmp eq i8 [[X:%.*]], 4
 ; SAME-NEXT:    call void @llvm.assume(i1 [[COND2]])
 ; SAME-NEXT:    ret i8 5
 ;
   %add = add i8 %x, 1
   %cond1 = icmp eq i8 %x, 3
   call void @llvm.assume(i1 %cond1)
   %cond2 = icmp eq i8 %x, 4
   call void @llvm.assume(i1 %cond2)
   ret i8 %add
 }
 
 ; Another case of conflicting assumptions. This would crash because we'd
 ; try to set more known bits than existed in the known bits struct.
 
 define void @PR36270(i32 %b) {
-; CHECK-LABEL: @PR36270(
-; CHECK-NEXT:    tail call void @llvm.assume(i1 false)
-; CHECK-NEXT:    unreachable
+; SAME-LABEL: @PR36270(
+; SAME-NEXT:    tail call void @llvm.assume(i1 false)
+; SAME-NEXT:    unreachable
 ;
 ; SAME-LABEL: @PR36270(
 ; SAME-NEXT:    tail call void @llvm.assume(i1 false)
 ; SAME-NEXT:    unreachable
 ;
   %B7 = xor i32 -1, 2147483647
   %and1 = and i32 %b, 3
   %B12 = lshr i32 %B7, %and1

 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!5, !6, !7, !8}
 
 !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !3, producer: "Me", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: null, retainedTypes: null, imports: null)
 !1 = !DILocalVariable(name: "", arg: 1, scope: !2, file: null, line: 1, type: null)
 !2 = distinct !DISubprogram(name: "debug", linkageName: "debug", scope: null, file: null, line: 0, type: null, isLocal: false, isDefinition: true, scopeLine: 1, flags: DIFlagPrototyped, isOptimized: true, unit: !0)
 !3 = !DIFile(filename: "consecutive-fences.ll", directory: "")
 !5 = !{i32 2, !"Dwarf Version", i32 4}
 !6 = !{i32 2, !"Debug Info Version", i32 3}
 !7 = !{i32 1, !"wchar_size", i32 4}
 !8 = !{i32 7, !"PIC Level", i32 2}
 !9 = !DILocation(line: 0, column: 0, scope: !2)
 
 
 attributes #0 = { nounwind uwtable }
 attributes #1 = { nounwind }