RFC: Interpet {branch_weights 1, 0} as likely/unlikely

[MatzeB]

Summary

This changes BranchProbabilityInfo to interpret branch weights metadata having only 1 non-zero entry as "likely"/"unlikely" branches.
Meaning !{"branch_weights", i32 1, i32 0} being an abstract representation of a "likely" branch. The adjustments for the concrete weights of likely/unlikely are moved into BranchProbabilityInfo and are applied when computing the analysis.

[MatzeB]

And here's another straw-man proposal to get a discussion going: Because I am having trouble to say what the difference between branch_weights 1, 0 and llvm.expect even is: This change turns any branch_weights X, 0 or branch_weights 0, X into an abstract representation of likely/unlikely that BranchProbabilityInfo will interpret when computing the analysis...

[mtrofin]

llvm/include/llvm/Analysis/BranchProbabilityInfo.h

144

docstring for normalizeLikelyWeights? that's the new thing, right - the other one just moved from ProfDataUtils.h?

llvm/lib/Analysis/BranchProbabilityInfo.cpp

393

What's HadNonZeroWeight gonna do after this loop?

1369

nit: spurious

llvm/test/Transforms/SampleProfile/profile-inference.ll

215

this caught my eye - how did count = 0 get to count = 3?

[MatzeB]

llvm/test/Transforms/SampleProfile/profile-inference.ll

215

This comes from the fact that !{"branch_weights", i32 X, i32 0} gets interpreted as !{"branch_weights", i32 LikelyBranchWeight, i32 1} now. So absolutes like a branch count / block being always zero is no longer the case.

I was contemplating whether there is/should be a difference between a branch reported as 0 by profile information or a branch marked as [[unlikely]]...

I don't know myself whether this change is actually a good idea (so at the moment no plans of landing it). But I thought it would make for an interesting RFC / strawman and elicit comments from people who have worked on profiling in the past.

[wenlei]

Meaning !{"branch_weights", i32 1, i32 0} being an abstract representation of a "likely" branch.

In general, an abstract representation should not conflict with a valid concrete representation..

The adjustments for the concrete weights of likely/unlikely are moved into BranchProbabilityInfo and are applied when computing the analysis.

BPI should be an analysis that only consumes branch weights, and I don't particularly like the idea of, oh btw, it will try to canonicalize branch weights when it runs.

If we want to canonicalize likely/unlikely representation for branch weights, we should probably do it elsewhere and make sure it's created that way rather than trying to lazily fix things up on when it is used.

Proper canonicalization is probably a much bigger change, so the next question is, is it worth it?

I can see that a {X,0} and {1,0} are equivalent in terms of branch probabilities, and the use of branch metadata in BranchProbabilityInfo::calcMetadataWeights also confirms that. Is there any practical purpose of this normalization other than clean things up?

There is a subtle difference between {X,0} and {1,0}. A lot of heuristic in PGO likes to +1 to everything to avoid 0s, and {X+1,1} vs {2,1} can be hugely different. Examples are: SampleProfileLoader::generateMDProfMetadata, and BlockFrequencyInfoImplBase::addToDist (though this one should not be impacted by this change as it's on BFI layer)

The other thing is there are some existing inconsistencies as to whether we should care about absolute values of branch weights or not.

1. For branch instructions, we care about only the distribution, but not the values; for calls however, we do care about the absolute values. And for sample pgo, we annotate branch weights with actual counts, even though the values don't change BPI result. Many people working in sample PGO like to think about branch weights values as a good reference, but their absolute values may not always be meaningful. I know that may have diverged from what the llvm docs say, but that's the status quo.

2. We also don't attempt to normalize {X*Y, Z*Y} to {X, Z}. If we don't do that, do we actually care about {X,0} and {1,0}, which is special form of the former.

[MatzeB]

This is already abandoned, so will not further pursue this. But for the discussion:

In general, an abstract representation should not conflict with a valid concrete representation..

In some way this change assumed that 1,0 was not really a valid representation. Because as you point out later sample profile loaders etc. actually trying hard to avoid the ,0 case by adding +1 which feels like people maybe do not want it to be legal and then we could have re-used it :)

Normalization is also an interesting topic and FWIW we do have the BranchProbability used by BPI and MIR-representation that is more normalized (though there is also oddities, like noone forcing them to sum up to "1" in the backend...) Anyway I'll keep mulling over those in the background, as they're mostly just oddities that I cannot understand, but not necessarily a problem.

[wenlei]

Because as you point out later sample profile loaders etc. actually trying hard to avoid the ,0 case by adding +1 which feels like people maybe do not want it to be legal and then we could have re-used it :)

Well.. this is a heuristic that is not on for all patch. In the case of sample loader, when profi is on, we don't do the +1 thingy. We also don't do +1 because 0 is invalid. So it holds true that the abstract representation used here conflicts with a valid concrete representation.

Anyway I'll keep mulling over those in the background, as they're mostly just oddities that I cannot understand, but not necessarily a problem.

SG. Unfortunately, some oddities can't be cleaned up easily. We probably have to live with the fact that sometimes we care about the values of branch weights, and sometimes only its distribution..

llvm/docs/BranchWeightMetadata.rst

 .. code-block:: none
 
   !0 = !{
     !"branch_weights",
     i32 <TRUE_BRANCH_WEIGHT>,
     i32 <FALSE_BRANCH_WEIGHT>
   }
 
+Weight metadata with a zero value on one size will be interpreted as an unlikely
+outcome that will be normalized to the same probability as `llvm.expect`.
+
 ``SwitchInst``
 ^^^^^^^^^^^^^^
 
 Branch weights are assigned to every case (including the ``default`` case which
 is always case #0).
 
 .. code-block:: none
 

llvm/include/llvm/Analysis/BranchProbabilityInfo.h

public:
 
   BranchProbabilityInfo &operator=(BranchProbabilityInfo &&RHS) {
     releaseMemory();
     Probs = std::move(RHS.Probs);
     EstimatedBlockWeight = std::move(RHS.EstimatedBlockWeight);
     return *this;
   }
 
+  /// Return the weight for a "likely branch". A 1/LikelyBranchWeight ratio
+  /// corresponds to the probability of `llvm.expect` matching the expected
+  /// value.
+  /// Note: There is also `TargetTransformInfo::getPredictableBranchThreshold`
+  /// for a value matching the current target.
+  static uint32_t getLikelyBranchWeight();
+  static void normalizeLikelyWeights(SmallVectorImpl<uint32_t>& Weights);
+
   bool invalidate(Function &, const PreservedAnalyses &PA,
                   FunctionAnalysisManager::Invalidator &);
 
   void releaseMemory();
 
   void print(raw_ostream &OS) const;
 
   /// Get an edge's probability, relative to other out-edges of the Src.

llvm/include/llvm/IR/ProfDataUtils.h

 /// Retrieve the total of all weights from an instruction.
 ///
 /// \param I The instruction to extract the total weight from
 /// \param [out] TotalWeights input variable to fill with total weights
 /// \returns True on success with profile total weights filled in. False if no
 /// metadata was found.
 bool extractProfTotalWeight(const Instruction &I, uint64_t &TotalWeights);
 
-/// Return the weight for a "likely branch". A 1/LikelyBranchWeight ratio
-/// corresponds to the probability of `llvm.expect` matching the expected
-/// value.
-/// Note: There is also `TargetTransformInfo::getPredictableBranchThreshold`
-/// for a value matching the current target.
-uint32_t getLikelyBranchWeight();
-
 } // namespace llvm
 #endif

llvm/lib/Analysis/BranchProbabilityInfo.cpp

 #include <iterator>
 #include <map>
 #include <utility>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "branch-prob"
 
+// This default value is chosen to represent an extremely skewed outcome for
+// a condition, but they leave some room for interpretation by later passes.
+//
+// If the documentation for __builtin_expect() was made explicit that it should
+// only be used in extreme cases, we could make this ratio higher. As it stands,
+// programmers may be using __builtin_expect() / llvm.expect to annotate that a
+// branch is likely or unlikely to be taken.
+static cl::opt<uint32_t> LikelyBranchWeight(
+    "likely-branch-weight", cl::Hidden, cl::init(2000),
+    cl::desc("Weight of the branch likely to be taken (default = 2000)"));
+
 static cl::opt<bool> PrintBranchProb(
     "print-bpi", cl::init(false), cl::Hidden,
     cl::desc("Print the branch probability info."));
 
 cl::opt<std::string> PrintBranchProbFuncName(
     "print-bpi-func-name", cl::Hidden,
     cl::desc("The option to specify the name of the function "
              "whose branch probability info is printed."));
void BranchProbabilityInfo::getLoopExitBlocks(
   if (LB.getLoop()) {
     LB.getLoop()->getExitBlocks(Exits);
   } else {
     assert(LB.getSccNum() != -1 && "LB doesn't belong to any loop?");
     SccI->getSccExitBlocks(LB.getSccNum(), Exits);
   }
 }
 
+uint32_t BranchProbabilityInfo::getLikelyBranchWeight() {
+  return LikelyBranchWeight;
+}
+
+// Normalize branch_weights like `1:0` or `0:1:0` to use `LikelyBranchWeight`.
+void BranchProbabilityInfo::normalizeLikelyWeights(SmallVectorImpl<uint32_t>& Weights) {
+  // Check if all but one entry are zero.
+  bool HadNonZeroWeight = false;
+  for (uint32_t W : Weights) {
+    if (W != 0) {
+      if (HadNonZeroWeight)
+        return;
+      HadNonZeroWeight = true;
+    }
+  }
+
+  for (uint32_t &W : Weights) {
+    if (W != 0) {
+      W = getLikelyBranchWeight();
+    } else {
+      W = 1;
+    }
+  }
+}
+
 // Propagate existing explicit probabilities from either profile data or
 // 'expect' intrinsic processing. Examine metadata against unreachable
 // heuristic. The probability of the edge coming to unreachable block is
 // set to min of metadata and unreachable heuristic.
 bool BranchProbabilityInfo::calcMetadataWeights(const BasicBlock *BB) {
   const Instruction *TI = BB->getTerminator();
   assert(TI->getNumSuccessors() > 1 && "expected more than one successor!");
   if (!(isa<BranchInst>(TI) || isa<SwitchInst>(TI) || isa<IndirectBrInst>(TI) ||
bool BranchProbabilityInfo::calcMetadataWeights(const BasicBlock *BB) {
   // Compute the sum of all weights to later decide whether they need to
   // be scaled to fit in 32 bits.
   uint64_t WeightSum = 0;
   SmallVector<uint32_t, 2> Weights;
   SmallVector<unsigned, 2> UnreachableIdxs;
   SmallVector<unsigned, 2> ReachableIdxs;
 
   extractBranchWeights(WeightsNode, Weights);
+  normalizeLikelyWeights(Weights);
+
   for (unsigned I = 0, E = Weights.size(); I != E; ++I) {
     WeightSum += Weights[I];
     const LoopBlock SrcLoopBB = getLoopBlock(BB);
     const LoopBlock DstLoopBB = getLoopBlock(TI->getSuccessor(I));
     auto EstimatedWeight = getEstimatedEdgeWeight({SrcLoopBB, DstLoopBB});
     if (EstimatedWeight &&
         *EstimatedWeight <= static_cast<uint32_t>(BlockExecWeight::UNREACHABLE))
       UnreachableIdxs.push_back(I);

 PreservedAnalyses
 BranchProbabilityPrinterPass::run(Function &F, FunctionAnalysisManager &AM) {
   OS << "Printing analysis results of BPI for function "
      << "'" << F.getName() << "':"
      << "\n";
   AM.getResult<BranchProbabilityAnalysis>(F).print(OS);
   return PreservedAnalyses::all();
 }
+

llvm/lib/IR/ProfDataUtils.cpp

 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/Support/BranchProbability.h"
 #include "llvm/Support/CommandLine.h"
 
 using namespace llvm;
 
-// These default values are chosen to represent an extremely skewed outcome for
-// a condition, but they leave some room for interpretation by later passes.
-//
-// If the documentation for __builtin_expect() was made explicit that it should
-// only be used in extreme cases, we could make this ratio higher. As it stands,
-// programmers may be using __builtin_expect() / llvm.expect to annotate that a
-// branch is likely or unlikely to be taken.
-
-static cl::opt<uint32_t> LikelyBranchWeight(
-    "likely-branch-weight", cl::Hidden, cl::init(2000),
-    cl::desc("Weight of the branch likely to be taken (default = 2000)"));
-
 namespace {
 
 // MD_prof nodes have the following layout
 //
 // In general:
 // { String name,         Array of i32   }
 //
 // In terms of Types:
bool extractProfTotalWeight(const MDNode *ProfileData, uint64_t &TotalVal) {
   }
   return false;
 }
 
 bool extractProfTotalWeight(const Instruction &I, uint64_t &TotalVal) {
   return extractProfTotalWeight(I.getMetadata(LLVMContext::MD_prof), TotalVal);
 }
 
-uint32_t getLikelyBranchWeight() { return LikelyBranchWeight; }
-
 } // namespace llvm

llvm/lib/Transforms/Scalar/LowerExpectIntrinsic.cpp

 
 STATISTIC(ExpectIntrinsicsHandled,
           "Number of 'expect' intrinsic instructions handled");
 
 static std::tuple<uint32_t, uint32_t>
 getBranchWeight(Intrinsic::ID IntrinsicID, CallInst *CI, int BranchCount) {
   if (IntrinsicID == Intrinsic::expect) {
     // __builtin_expect
-    return std::make_tuple(getLikelyBranchWeight(), 1);
+    return std::make_tuple(1, 0);
   } else {
     // __builtin_expect_with_probability
     assert(CI->getNumOperands() >= 3 &&
            "expect with probability must have 3 arguments");
     auto *Confidence = cast<ConstantFP>(CI->getArgOperand(2));
     double TrueProb = Confidence->getValueAPF().convertToDouble();
     assert((TrueProb >= 0.0 && TrueProb <= 1.0) &&
            "probability value must be in the range [0.0, 1.0]");

llvm/lib/Transforms/Utils/MisExpect.cpp

 // compare them to the weights collected during profiling to determine if a
 // diagnostic message is warranted.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Transforms/Utils/MisExpect.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/BranchProbabilityInfo.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/ProfDataUtils.h"
 #include "llvm/Support/BranchProbability.h"
 #include "llvm/Support/CommandLine.h"
emitMisexpectDiagnostic(&I, I.getContext(), ProfiledWeight,
                             RealWeightsTotal);
 }
 
 void checkBackendInstrumentation(Instruction &I,
                                  const ArrayRef<uint32_t> RealWeights) {
   SmallVector<uint32_t> ExpectedWeights;
   if (!extractBranchWeights(I, ExpectedWeights))
     return;
+  BranchProbabilityInfo::normalizeLikelyWeights(ExpectedWeights);
   verifyMisExpect(I, RealWeights, ExpectedWeights);
 }
 
 void checkFrontendInstrumentation(Instruction &I,
                                   const ArrayRef<uint32_t> ExpectedWeights) {
   SmallVector<uint32_t> RealWeights;
   if (!extractBranchWeights(I, RealWeights))
     return;
+  BranchProbabilityInfo::normalizeLikelyWeights(RealWeights);
   verifyMisExpect(I, RealWeights, ExpectedWeights);
 }
 
 void checkExpectAnnotations(Instruction &I,
                             const ArrayRef<uint32_t> ExistingWeights,
                             bool IsFrontend) {
   if (IsFrontend) {
     checkFrontendInstrumentation(I, ExistingWeights);
   } else {
     checkBackendInstrumentation(I, ExistingWeights);
   }
 }
 
 } // namespace misexpect
 } // namespace llvm
 #undef DEBUG_TYPE

llvm/test/Transforms/CodeExtractor/PartialInlinePGOMultiRegion.ll

-; RUN: opt -S -passes=partial-inliner -min-block-execution=1 -skip-partial-inlining-cost-analysis < %s | FileCheck %s
+; RUN: opt -S -passes=partial-inliner -min-block-execution=100 -skip-partial-inlining-cost-analysis < %s | FileCheck %s
 ; Require a dummy block (if.then.b) as successor to if.then due to PI requirement
 ; of region containing more than one BB.
 define signext i32 @bar(i32 signext %value, i32 signext %ub) #0 !prof !30 {
 entry:
   %value.addr = alloca i32, align 4
   %ub.addr = alloca i32, align 4
   %sum = alloca i32, align 4
   %i = alloca i32, align 4

 !22 = !{i32 900000, i64 100, i32 1}
 !23 = !{i32 950000, i64 100, i32 1}
 !24 = !{i32 990000, i64 1, i32 4}
 !25 = !{i32 999000, i64 1, i32 4}
 !26 = !{i32 999900, i64 1, i32 4}
 !27 = !{i32 999990, i64 1, i32 4}
 !28 = !{i32 999999, i64 1, i32 4}
 !29 = !{!"clang version 6.0.0 (123456)"}
-!30 = !{!"function_entry_count", i64 2}
+!30 = !{!"function_entry_count", i64 100}
 !31 = !{!"branch_weights", i32 100, i32 1}
 !32 = !{!"branch_weights", i32 0, i32 100}

llvm/test/Transforms/CodeExtractor/PartialInlinePGORegion.ll

-; RUN: opt -S -passes=partial-inliner -min-block-execution=1 -skip-partial-inlining-cost-analysis < %s | FileCheck %s
+; RUN: opt -S -passes=partial-inliner -min-block-execution=100 -skip-partial-inlining-cost-analysis < %s | FileCheck %s
 ; Require a dummy block (if.then.b) as successor to if.then due to PI requirement
 ; of region containing more than one BB.
 define signext i32 @bar(i32 signext %value, i32 signext %ub) #0 !prof !30 {
 entry:
   %value.addr = alloca i32, align 4
   %ub.addr = alloca i32, align 4
   %sum = alloca i32, align 4
   %i = alloca i32, align 4

 !22 = !{i32 900000, i64 100, i32 1}
 !23 = !{i32 950000, i64 100, i32 1}
 !24 = !{i32 990000, i64 1, i32 4}
 !25 = !{i32 999000, i64 1, i32 4}
 !26 = !{i32 999900, i64 1, i32 4}
 !27 = !{i32 999990, i64 1, i32 4}
 !28 = !{i32 999999, i64 1, i32 4}
 !29 = !{!"clang version 6.0.0 (123456)"}
-!30 = !{!"function_entry_count", i64 2}
+!30 = !{!"function_entry_count", i64 100}
 !31 = !{!"branch_weights", i32 100, i32 1}
 !32 = !{!"branch_weights", i32 0, i32 100}

llvm/test/Transforms/LowerExpectIntrinsic/basic.ll

define i32 @test10(i64 %t6) {
   %t9 = select i1 %t8, i32 1, i32 2
 ; CHECK: select{{.*}}, !prof !1
   ret i32 %t9
 }
 
 
 declare i1 @llvm.expect.i1(i1, i1) nounwind readnone
 
-; CHECK: !0 = !{!"branch_weights", i32 2000, i32 1}
-; CHECK: !1 = !{!"branch_weights", i32 1, i32 2000}
-; CHECK: !2 = !{!"branch_weights", i32 1, i32 1, i32 2000}
-; CHECK: !3 = !{!"branch_weights", i32 2000, i32 1, i32 1}
+; CHECK: !0 = !{!"branch_weights", i32 1, i32 0}
+; CHECK: !1 = !{!"branch_weights", i32 0, i32 1}
+; CHECK: !2 = !{!"branch_weights", i32 0, i32 0, i32 1}
+; CHECK: !3 = !{!"branch_weights", i32 1, i32 0, i32 0}

llvm/test/Transforms/LowerExpectIntrinsic/expect_nonboolean.ll

 attributes #0 = { nounwind uwtable }
 attributes #1 = { nounwind readnone }
 
 !llvm.module.flags = !{!0}
 !llvm.ident = !{!1}
 
 !0 = !{i32 1, !"wchar_size", i32 4}
 !1 = !{!"clang version 5.0.0 (trunk 304373)"}
-; CHECK: [[LIKELY]] = !{!"branch_weights", i32 2000, i32 1}
-; CHECK: [[UNLIKELY]] = !{!"branch_weights", i32 1, i32 2000}
+; CHECK: [[LIKELY]] = !{!"branch_weights", i32 1, i32 0}
+; CHECK: [[UNLIKELY]] = !{!"branch_weights", i32 0, i32 1}
 

llvm/test/Transforms/LowerExpectIntrinsic/phi_merge.ll

 declare i8 @too8()
 
 ; Function Attrs: nounwind readnone
 declare i64 @llvm.expect.i64(i64, i64)
 
 !llvm.ident = !{!0}
 
 !0 = !{!"clang version 5.0.0 (trunk 302965)"}
-; CHECK: [[WEIGHT]] = !{!"branch_weights", i32 2000, i32 1}
-; CHECK: [[WEIGHT2]] = !{!"branch_weights", i32 1, i32 2000}
+; CHECK: [[WEIGHT]] = !{!"branch_weights", i32 1, i32 0}
+; CHECK: [[WEIGHT2]] = !{!"branch_weights", i32 0, i32 1}

llvm/test/Transforms/LowerExpectIntrinsic/phi_or.ll

 declare i32 @hoo()
 declare i64 @llvm.expect.i64(i64, i64)
 
 
 !llvm.ident = !{!0}
 
 
 !0 = !{!"clang version 5.0.0 (trunk 302965)"}
-; CHECK: [[WEIGHT]] = !{!"branch_weights", i32 2000, i32 1}
-; CHECK: [[WEIGHT2]] = !{!"branch_weights", i32 1, i32 2000}
+; CHECK: [[WEIGHT]] = !{!"branch_weights", i32 1, i32 0}
+; CHECK: [[WEIGHT2]] = !{!"branch_weights", i32 0, i32 1}

llvm/test/Transforms/LowerExpectIntrinsic/phi_tern.ll

 declare i64 @llvm.expect.i64(i64, i64)
 
 
 
 !llvm.ident = !{!0}
 
 !0 = !{!"clang version 5.0.0 (trunk 302965)"}
 
-; CHECK: [[WEIGHT]] = !{!"branch_weights", i32 1, i32 2000}
+; CHECK: [[WEIGHT]] = !{!"branch_weights", i32 0, i32 1}

llvm/test/Transforms/LowerExpectIntrinsic/phi_unexpect.ll

-; RUN: opt -S -passes='function(lower-expect),strip-dead-prototypes' -likely-branch-weight=2147483647 < %s | FileCheck %s
+; RUN: opt -S -passes='function(lower-expect),strip-dead-prototypes' < %s | FileCheck %s
 
 ; The C case
 ; if (__builtin_expect_with_probability(((a0 == 1) || (a1 == 1) || (a2 == 1)), 1, 0))
 ; For the above case, all 3 branches should be annotated
 ; which should be equivalent to if (__builtin_expect(((a0 == 1) || (a1 == 1) || (a2 == 1)), 0))
 
 ; The C case
 ; if (__builtin_expect_with_probability(((a0 == 1) || (a1 == 1) || (a2 == 1)), 1, 1))
 ; For the above case, we do not have enough information, so only the last branch could be annotated
 ; which should be equivalent to if (__builtin_expect(((a0 == 1) || (a1 == 1) || (a2 == 1)), 1))
 
 declare void @foo()
 
 declare i64 @llvm.expect.i64(i64, i64) nounwind readnone
 declare i64 @llvm.expect.with.probability.i64(i64, i64, double) nounwind readnone
 
 ; CHECK-LABEL: @test1_expect_1(
 ; CHECK: block0:
 ; CHECK-NOT: prof
 ; CHECK: block1:
 ; CHECK-NOT: prof
 ; CHECK: block3:
-; CHECK: br i1 %tobool, label %block4, label %block5, !prof !0
+; CHECK: br i1 %tobool, label %block4, label %block5, !prof [[PROF_TEST1:![0-9]+]]
 define void @test1_expect_1(i8 %a0, i8 %a1, i8 %a2) {
 block0:
   %c0 = icmp eq i8 %a0, 1
   br i1 %c0, label %block3, label %block1
 
 block1:
   %c1 = icmp eq i8 %a1, 1
   br i1 %c1, label %block3, label %block2

 
 ; should have exactly the same behavior as test1
 ; CHECK-LABEL: @test2_expect_with_prob_1_1(
 ; CHECK: block0:
 ; CHECK-NOT: prof
 ; CHECK: block1:
 ; CHECK-NOT: prof
 ; CHECK: block3:
-; CHECK: br i1 %tobool, label %block4, label %block5, !prof !0
+; CHECK: br i1 %tobool, label %block4, label %block5, !prof [[PROF_MAX:![0-9]+]]
 define void @test2_expect_with_prob_1_1(i8 %a0, i8 %a1, i8 %a2) {
 block0:
   %c0 = icmp eq i8 %a0, 1
   br i1 %c0, label %block3, label %block1
 
 block1:
   %c1 = icmp eq i8 %a1, 1
   br i1 %c1, label %block3, label %block2

 
 ; should have exactly the same behavior as test1
 ; CHECK-LABEL: @test3_expect_with_prob_0_0(
 ; CHECK: block0:
 ; CHECK-NOT: prof
 ; CHECK: block1:
 ; CHECK-NOT: prof
 ; CHECK: block3:
-; CHECK: br i1 %tobool, label %block4, label %block5, !prof !0
+; CHECK: br i1 %tobool, label %block4, label %block5, !prof [[PROF_MAX]]
 define void @test3_expect_with_prob_0_0(i8 %a0, i8 %a1, i8 %a2) {
 block0:
   %c0 = icmp eq i8 %a0, 1
   br i1 %c0, label %block3, label %block1
 
 block1:
   %c1 = icmp eq i8 %a1, 1
   br i1 %c1, label %block3, label %block2
block4:
   br label %block5
 
 block5:
   ret void
 }
 
 ; CHECK-LABEL: @test4_expect_0(
 ; CHECK: block0:
-; CHECK: br i1 %c0, label %block3, label %block1, !prof !1
+; CHECK: br i1 %c0, label %block3, label %block1, !prof [[PROF_TEST4:![0-9]+]]
 ; CHECK: block1:
-; CHECK: br i1 %c1, label %block3, label %block2, !prof !1
+; CHECK: br i1 %c1, label %block3, label %block2, !prof [[PROF_TEST4]]
 ; CHECK: block3:
-; CHECK: br i1 %tobool, label %block4, label %block5, !prof !1
+; CHECK: br i1 %tobool, label %block4, label %block5, !prof [[PROF_TEST4]]
 define void @test4_expect_0(i8 %a0, i8 %a1, i8 %a2) {
 block0:
   %c0 = icmp eq i8 %a0, 1
   br i1 %c0, label %block3, label %block1
 
 block1:
   %c1 = icmp eq i8 %a1, 1
   br i1 %c1, label %block3, label %block2

 
 block5:
   ret void
 }
 
 ; should have exactly the same behavior as test4
 ; CHECK-LABEL: @test5_expect_with_prob_1_0(
 ; CHECK: block0:
-; CHECK: br i1 %c0, label %block3, label %block1, !prof !1
+; CHECK: br i1 %c0, label %block3, label %block1, !prof [[PROF_MIN:![0-9]+]]
 ; CHECK: block1:
-; CHECK: br i1 %c1, label %block3, label %block2, !prof !1
+; CHECK: br i1 %c1, label %block3, label %block2, !prof [[PROF_MIN]]
 ; CHECK: block3:
-; CHECK: br i1 %tobool, label %block4, label %block5, !prof !1
+; CHECK: br i1 %tobool, label %block4, label %block5, !prof [[PROF_MIN]]
 define void @test5_expect_with_prob_1_0(i8 %a0, i8 %a1, i8 %a2) {
 block0:
   %c0 = icmp eq i8 %a0, 1
   br i1 %c0, label %block3, label %block1
 
 block1:
   %c1 = icmp eq i8 %a1, 1
   br i1 %c1, label %block3, label %block2

 
 block5:
   ret void
 }
 
 ; should have exactly the same behavior as test4
 ; CHECK-LABEL: @test6_expect_with_prob_0_1(
 ; CHECK: block0:
-; CHECK: br i1 %c0, label %block3, label %block1, !prof !1
+; CHECK: br i1 %c0, label %block3, label %block1, !prof [[PROF_MIN]]
 ; CHECK: block1:
-; CHECK: br i1 %c1, label %block3, label %block2, !prof !1
+; CHECK: br i1 %c1, label %block3, label %block2, !prof [[PROF_MIN]]
 ; CHECK: block3:
-; CHECK: br i1 %tobool, label %block4, label %block5, !prof !1
+; CHECK: br i1 %tobool, label %block4, label %block5, !prof [[PROF_MIN]]
 define void @test6_expect_with_prob_0_1(i8 %a0, i8 %a1, i8 %a2) {
 block0:
   %c0 = icmp eq i8 %a0, 1
   br i1 %c0, label %block3, label %block1
 
 block1:
   %c1 = icmp eq i8 %a1, 1
   br i1 %c1, label %block3, label %block2

 block4:
   call void @foo()
   br label %block5
 
 block5:
   ret void
 }
 
-; CHECK: !0 = !{!"branch_weights", i32 2147483647, i32 1}
-; CHECK: !1 = !{!"branch_weights", i32 1, i32 2147483647}
+; CHECK: [[PROF_TEST1]] = !{!"branch_weights", i32 1, i32 0}
+; CHECK: [[PROF_MAX]] = !{!"branch_weights", i32 2147483647, i32 1}
+; CHECK: [[PROF_TEST4]] = !{!"branch_weights", i32 0, i32 1}
+; CHECK: [[PROF_MIN]] = !{!"branch_weights", i32 1, i32 2147483647}

llvm/test/Transforms/SampleProfile/profile-inference.ll

b3:
   call void @llvm.pseudoprobe(i64 -907520326213521421, i64 3, i32 0, i64 -1)
   br label %b9
 ; CHECK:  edge b3 -> b9 probability is 0x80000000 / 0x80000000 = 100.00%
 ; CHECK2: - b3: float = {{.*}}, int = {{.*}}, count = 1
 
 b4:
   call void @llvm.pseudoprobe(i64 -907520326213521421, i64 4, i32 0, i64 -1)
   br i1 %cmp, label %b5, label %b6
-; CHECK:  edge b4 -> b5 probability is 0x80000000 / 0x80000000 = 100.00%
-; CHECK:  edge b4 -> b6 probability is 0x00000000 / 0x80000000 = 0.00%
+; CHECK:  edge b4 -> b5 probability is 0x7fef9fcb / 0x80000000 = 99.95%
+; CHECK:  edge b4 -> b6 probability is 0x00106035 / 0x80000000 = 0.05%
 ; CHECK2: - b4: float = {{.*}}, int = {{.*}}, count = 5992
 
 b5:
   call void @llvm.pseudoprobe(i64 -907520326213521421, i64 5, i32 0, i64 -1)
   br label %b7
 ; CHECK:  edge b5 -> b7 probability is 0x80000000 / 0x80000000 = 100.00%
-; CHECK2: - b5: float = {{.*}}, int = {{.*}}, count = 5992
+; CHECK2: - b5: float = {{.*}}, int = {{.*}}, count = 5989
 
 b6:
   call void @llvm.pseudoprobe(i64 -907520326213521421, i64 6, i32 0, i64 -1)
   br label %b8
 ; CHECK:  edge b6 -> b8 probability is 0x80000000 / 0x80000000 = 100.00%
-; CHECK2: - b6: float = {{.*}}, int = {{.*}}, count = 0
+; CHECK2: - b6: float = {{.*}}, int = {{.*}}, count = 3
 
 b7:
   call void @llvm.pseudoprobe(i64 -907520326213521421, i64 7, i32 0, i64 -1)
   br label %b8
 ; CHECK:  edge b7 -> b8 probability is 0x80000000 / 0x80000000 = 100.00%
-; CHECK2: - b7: float = {{.*}}, int = {{.*}}, count = 5992
+; CHECK2: - b7: float = {{.*}}, int = {{.*}}, count = 5989
 
 b8:
   call void @llvm.pseudoprobe(i64 -907520326213521421, i64 8, i32 0, i64 -1)
   br label %b9
 ; CHECK:  edge b8 -> b9 probability is 0x80000000 / 0x80000000 = 100.00%
 ; CHECK2: - b8: float = {{.*}}, int = {{.*}}, count = 5992
 
 b9: