[CodeGen] Improve likelihood attribute branch weights

[Mordante]

Summary

Bruno De Fraine discovered some issues with D85091. The branch weights generated for logical not and ternary conditional were wrong. The logical and and logical or differed from the code generated of __builtin_predict.

Adjusted the generated code for the likelihood to match __builtin_predict. The patch is based on Bruno's suggestions.

[bdf]

Looks good. Good idea to add tests to verify that we match __builtin_except().

[aaron.ballman]

clang/lib/CodeGen/CodeGenFunction.cpp

1622

Why LH_None in this case? (Also, given that it's the default value, should you skip passing this at all? Or should the parameter not have a default value, perhaps?)

1678–1679

This comment is now a bit stale.

[Mordante]

clang/lib/CodeGen/CodeGenFunction.cpp

1622

This should be LH_None since this is the likelihood of C ? T : F and there's no likelihood for this expression. When used in if(C ? T : F) the if can have an likelihood attribute for the if. That's why the other two calls in this block have a likelihood. I'll add comment to make it clear why this is correct. I felt using the explicit value here instead of the defaulted value was clearer. This would be the only call in this function not to explicitly use a value.

Since EmitBranchOnBoolExpr is also used at other places I prefer to keep the default argument.

1678–1679

Agreed, I'll remove it.

[aaron.ballman]

LGTM modulo comment nits.

clang/lib/CodeGen/CodeGenFunction.cpp

1622

Thank you for the explanation!

[jmorse]

Hi -- We (Sony) are running into a bit of difficulty with the test for this change, as it relies on the configuration of the -O1 optimisation pipeline. Would it be possible to reduce down to a frontend test, and then tests for whatever passes are to interpret the IR produced by clang?

[Mordante]

In D88363#2312129, @jmorse wrote:

Hi -- We (Sony) are running into a bit of difficulty with the test for this change, as it relies on the configuration of the -O1 optimisation pipeline. Would it be possible to reduce down to a frontend test, and then tests for whatever passes are to interpret the IR produced by clang?

Can you explain the kind of issues you're having?
Currently the code, like PGO, requires at least -O1 to be effective. At -O0 the attributes don't have any effect.
I looked at other CodeGen tests using __builtin_expect. They are using -O1 -disable-llvm-passes, would that solve your issue?

[jmorse]

In D88363#2317241, @Mordante wrote:

Can you explain the kind of issues you're having?

At the shallowest level, our -O1 produces different IR and fails the test, which is more or less our problem; however my understanding is that tests in the LLVM project / subprojects should aim to test as little amount of code as possible. Relying on all of -O1 makes it a brittle test -- changes to any optimisation pass enabled in -O1 could cause this test to fail spuriously.

Instead, I believe the test should be in two parts:

• One checking clang produces the correct /unoptimised/ IR output
• One or more checking that the consuming IR passes do-the-right-thing

An example is (some of) the TBAA tests -- as you suggest, they're using -O1 -disable-llvm-passes to check that clang produces stores with !tbaa metadata attached. Then over in the LLVM optimisation passes there are tests checking that GVN / LICM etc correctly consume TBAA metadata. Note that I'm unfamiliar with how branch weights work, but I believe the principle is the same.

[bdf]

In D88363#2319157, @jmorse wrote:

In D88363#2317241, @Mordante wrote:

Can you explain the kind of issues you're having?

At the shallowest level, our -O1 produces different IR and fails the test, which is more or less our problem; however my understanding is that tests in the LLVM project / subprojects should aim to test as little amount of code as possible. Relying on all of -O1 makes it a brittle test -- changes to any optimisation pass enabled in -O1 could cause this test to fail spuriously.

Instead, I believe the test should be in two parts:

• One checking clang produces the correct /unoptimised/ IR output
• One or more checking that the consuming IR passes do-the-right-thing

As I see, the intent of the test is not so much to verify a certain expected output, but more to verify that two styles of likelihood hints in C code produce the same code structure and branch weights. Theses styles are likely/unlikely-annotations, and use of __builtin_expect in the if condition. But the processing of these two is quite different:

• for likely/unlikely annotations, branch weights are added immediately in the initial CodeGen
• __builtin_expect is first translated straightforward to an expect intrinsic, then processed by a later lower-expect pass

To make the test less brittle, would it be possible to explicitly select only the optimization passes that are needed?

[Mordante]

In D88363#2319242, @bdf wrote:

In D88363#2319157, @jmorse wrote:

In D88363#2317241, @Mordante wrote:

Can you explain the kind of issues you're having?

At the shallowest level, our -O1 produces different IR and fails the test, which is more or less our problem; however my understanding is that tests in the LLVM project / subprojects should aim to test as little amount of code as possible. Relying on all of -O1 makes it a brittle test -- changes to any optimisation pass enabled in -O1 could cause this test to fail spuriously.

Instead, I believe the test should be in two parts:

• One checking clang produces the correct /unoptimised/ IR output
• One or more checking that the consuming IR passes do-the-right-thing

As I see, the intent of the test is not so much to verify a certain expected output, but more to verify that two styles of likelihood hints in C code produce the same code structure and branch weights. Theses styles are likely/unlikely-annotations, and use of __builtin_expect in the if condition. But the processing of these two is quite different:

• for likely/unlikely annotations, branch weights are added immediately in the initial CodeGen
• __builtin_expect is first translated straightforward to an expect intrinsic, then processed by a later lower-expect pass

To make the test less brittle, would it be possible to explicitly select only the optimization passes that are needed?

Indeed verifying the output of the likelihood attributes against __builtin_expect is exactly what's required. But I think I can make the test less brittle by using the following command. This only runs the lower expect pass, which lowers the __builtin_expect.
RUN: %clang_cc1 -O1 -disable-llvm-passes -emit-llvm %s -o - -triple=x86_64-linux-gnu | opt --lower-expect -S | FileCheck %s

I'll work on a patch to solve the issue.

[Mordante]

I created D89204 which hopefully fixes Sony's issue.

clang/lib/CodeGen/CGStmt.cpp

 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TargetInfo.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/Support/SaveAndRestore.h"
-#include "llvm/Transforms/Scalar/LowerExpectIntrinsic.h"
 
 using namespace clang;
 using namespace CodeGen;
 
 //===----------------------------------------------------------------------===//
 //                              Statement Emission
 //===----------------------------------------------------------------------===//
 
void CodeGenFunction::EmitIndirectGotoStmt(const IndirectGotoStmt &S) {
   llvm::BasicBlock *IndGotoBB = GetIndirectGotoBlock();
 
   // The first instruction in the block has to be the PHI for the switch dest,
   // add an entry for this branch.
   cast<llvm::PHINode>(IndGotoBB->begin())->addIncoming(V, CurBB);
 
   EmitBranch(IndGotoBB);
 }
-static Optional<std::pair<uint32_t, uint32_t>>
-getLikelihoodWeights(const IfStmt &If) {
-  switch (Stmt::getLikelihood(If.getThen(), If.getElse())) {
-  case Stmt::LH_Unlikely:
-    return std::pair<uint32_t, uint32_t>(llvm::UnlikelyBranchWeight,
-                                         llvm::LikelyBranchWeight);
-  case Stmt::LH_None:
-    return None;
-  case Stmt::LH_Likely:
-    return std::pair<uint32_t, uint32_t>(llvm::LikelyBranchWeight,
-                                         llvm::UnlikelyBranchWeight);
-  }
-  llvm_unreachable("Unknown Likelihood");
-}
 
 void CodeGenFunction::EmitIfStmt(const IfStmt &S) {
   // C99 6.8.4.1: The first substatement is executed if the expression compares
   // unequal to 0.  The condition must be a scalar type.
   LexicalScope ConditionScope(*this, S.getCond()->getSourceRange());
 
   if (S.getInit())
     EmitStmt(S.getInit());
void CodeGenFunction::EmitIfStmt(const IfStmt &S) {
   llvm::BasicBlock *ContBlock = createBasicBlock("if.end");
   llvm::BasicBlock *ElseBlock = ContBlock;
   if (S.getElse())
     ElseBlock = createBasicBlock("if.else");
 
   // Prefer the PGO based weights over the likelihood attribute.
   // When the build isn't optimized the metadata isn't used, so don't generate
   // it.
-  llvm::MDNode *Weights = nullptr;
+  Stmt::Likelihood LH = Stmt::LH_None;
   uint64_t Count = getProfileCount(S.getThen());
-  if (!Count && CGM.getCodeGenOpts().OptimizationLevel) {
-    Optional<std::pair<uint32_t, uint32_t>> LHW = getLikelihoodWeights(S);
-    if (LHW) {
-      llvm::MDBuilder MDHelper(CGM.getLLVMContext());
-      Weights = MDHelper.createBranchWeights(LHW->first, LHW->second);
-    }
-  }
-
-  EmitBranchOnBoolExpr(S.getCond(), ThenBlock, ElseBlock, Count, Weights);
+  if (!Count && CGM.getCodeGenOpts().OptimizationLevel)
+    LH = Stmt::getLikelihood(S.getThen(), S.getElse());
+  EmitBranchOnBoolExpr(S.getCond(), ThenBlock, ElseBlock, Count, LH);
 
   // Emit the 'then' code.
   EmitBlock(ThenBlock);
   incrementProfileCounter(&S);
   {
     RunCleanupsScope ThenScope(*this);
     EmitStmt(S.getThen());
   }

clang/lib/CodeGen/CodeGenFunction.h

public:
 
   /// EmitBranchOnBoolExpr - Emit a branch on a boolean condition (e.g. for an
   /// if statement) to the specified blocks.  Based on the condition, this might
   /// try to simplify the codegen of the conditional based on the branch.
   /// TrueCount should be the number of times we expect the condition to
   /// evaluate to true based on PGO data.
   void EmitBranchOnBoolExpr(const Expr *Cond, llvm::BasicBlock *TrueBlock,
                             llvm::BasicBlock *FalseBlock, uint64_t TrueCount,
-                            llvm::MDNode *Weights = nullptr);
+                            Stmt::Likelihood LH = Stmt::LH_None);
 
   /// Given an assignment `*LHS = RHS`, emit a test that checks if \p RHS is
   /// nonnull, if \p LHS is marked _Nonnull.
   void EmitNullabilityCheck(LValue LHS, llvm::Value *RHS, SourceLocation Loc);
 
   /// An enumeration which makes it easier to specify whether or not an
   /// operation is a subtraction.
   enum { NotSubtraction = false, IsSubtraction = true };

clang/lib/CodeGen/CodeGenFunction.cpp

 #include "llvm/Frontend/OpenMP/OMPIRBuilder.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/FPEnv.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/IR/Operator.h"
+#include "llvm/Transforms/Scalar/LowerExpectIntrinsic.h"
 #include "llvm/Transforms/Utils/PromoteMemToReg.h"
 using namespace clang;
 using namespace CodeGen;
 
 /// shouldEmitLifetimeMarkers - Decide whether we need emit the life-time
 /// markers.
 static bool shouldEmitLifetimeMarkers(const CodeGenOptions &CGOpts,
                                       const LangOptions &LangOpts) {
bool CodeGenFunction::ConstantFoldsToSimpleInteger(const Expr *Cond,
   llvm::APSInt Int = Result.Val.getInt();
   if (!AllowLabels && CodeGenFunction::ContainsLabel(Cond))
     return false;  // Contains a label.
 
   ResultInt = Int;
   return true;
 }
 
+static Optional<std::pair<uint32_t, uint32_t>>
+getLikelihoodWeights(Stmt::Likelihood LH) {
+  switch (LH) {
+  case Stmt::LH_Unlikely:
+    return std::pair<uint32_t, uint32_t>(llvm::UnlikelyBranchWeight,
+                                         llvm::LikelyBranchWeight);
+  case Stmt::LH_None:
+    return None;
+  case Stmt::LH_Likely:
+    return std::pair<uint32_t, uint32_t>(llvm::LikelyBranchWeight,
+                                         llvm::UnlikelyBranchWeight);
+  }
+  llvm_unreachable("Unknown Likelihood");
+}
+
 /// EmitBranchOnBoolExpr - Emit a branch on a boolean condition (e.g. for an if
 /// statement) to the specified blocks.  Based on the condition, this might try
 /// to simplify the codegen of the conditional based on the branch.
-/// \param Weights The weights determined by the likelihood attributes.
+/// \param LH The value of the likelihood attribute on the True branch.
 void CodeGenFunction::EmitBranchOnBoolExpr(const Expr *Cond,
                                            llvm::BasicBlock *TrueBlock,
                                            llvm::BasicBlock *FalseBlock,
                                            uint64_t TrueCount,
-                                           llvm::MDNode *Weights) {
+                                           Stmt::Likelihood LH) {
   Cond = Cond->IgnoreParens();
 
   if (const BinaryOperator *CondBOp = dyn_cast<BinaryOperator>(Cond)) {
 
     // Handle X && Y in a condition.
     if (CondBOp->getOpcode() == BO_LAnd) {
       // If we have "1 && X", simplify the code.  "0 && X" would have constant
       // folded if the case was simple enough.
       bool ConstantBool = false;
       if (ConstantFoldsToSimpleInteger(CondBOp->getLHS(), ConstantBool) &&
           ConstantBool) {
         // br(1 && X) -> br(X).
         incrementProfileCounter(CondBOp);
         return EmitBranchOnBoolExpr(CondBOp->getRHS(), TrueBlock, FalseBlock,
-                                    TrueCount, Weights);
+                                    TrueCount, LH);
       }
 
       // If we have "X && 1", simplify the code to use an uncond branch.
       // "X && 0" would have been constant folded to 0.
       if (ConstantFoldsToSimpleInteger(CondBOp->getRHS(), ConstantBool) &&
           ConstantBool) {
         // br(X && 1) -> br(X).
         return EmitBranchOnBoolExpr(CondBOp->getLHS(), TrueBlock, FalseBlock,
-                                    TrueCount, Weights);
+                                    TrueCount, LH);
       }
 
       // Emit the LHS as a conditional.  If the LHS conditional is false, we
       // want to jump to the FalseBlock.
       llvm::BasicBlock *LHSTrue = createBasicBlock("land.lhs.true");
       // The counter tells us how often we evaluate RHS, and all of TrueCount
       // can be propagated to that branch.
       uint64_t RHSCount = getProfileCount(CondBOp->getRHS());
 
       ConditionalEvaluation eval(*this);
       {
         ApplyDebugLocation DL(*this, Cond);
+        // Propagate the likelihood attribute like __builtin_expect
+        // __builtin_expect(X && Y, 1) -> X and Y are likely
+        // __builtin_expect(X && Y, 0) -> only Y is unlikely
         EmitBranchOnBoolExpr(CondBOp->getLHS(), LHSTrue, FalseBlock, RHSCount,
-                             Weights);
+                             LH == Stmt::LH_Unlikely ? Stmt::LH_None : LH);
         EmitBlock(LHSTrue);
       }
 
       incrementProfileCounter(CondBOp);
       setCurrentProfileCount(getProfileCount(CondBOp->getRHS()));
 
       // Any temporaries created here are conditional.
       eval.begin(*this);
       EmitBranchOnBoolExpr(CondBOp->getRHS(), TrueBlock, FalseBlock, TrueCount,
-                           Weights);
+                           LH);
       eval.end(*this);
 
       return;
     }
 
     if (CondBOp->getOpcode() == BO_LOr) {
       // If we have "0 || X", simplify the code.  "1 || X" would have constant
       // folded if the case was simple enough.
       bool ConstantBool = false;
       if (ConstantFoldsToSimpleInteger(CondBOp->getLHS(), ConstantBool) &&
           !ConstantBool) {
         // br(0 || X) -> br(X).
         incrementProfileCounter(CondBOp);
         return EmitBranchOnBoolExpr(CondBOp->getRHS(), TrueBlock, FalseBlock,
-                                    TrueCount, Weights);
+                                    TrueCount, LH);
       }
 
       // If we have "X || 0", simplify the code to use an uncond branch.
       // "X || 1" would have been constant folded to 1.
       if (ConstantFoldsToSimpleInteger(CondBOp->getRHS(), ConstantBool) &&
           !ConstantBool) {
         // br(X || 0) -> br(X).
         return EmitBranchOnBoolExpr(CondBOp->getLHS(), TrueBlock, FalseBlock,
-                                    TrueCount, Weights);
+                                    TrueCount, LH);
       }
 
       // Emit the LHS as a conditional.  If the LHS conditional is true, we
       // want to jump to the TrueBlock.
       llvm::BasicBlock *LHSFalse = createBasicBlock("lor.lhs.false");
       // We have the count for entry to the RHS and for the whole expression
       // being true, so we can divy up True count between the short circuit and
       // the RHS.
       uint64_t LHSCount =
           getCurrentProfileCount() - getProfileCount(CondBOp->getRHS());
       uint64_t RHSCount = TrueCount - LHSCount;
 
       ConditionalEvaluation eval(*this);
       {
+        // Propagate the likelihood attribute like __builtin_expect
+        // __builtin_expect(X || Y, 1) -> only Y is likely
+        // __builtin_expect(X || Y, 0) -> both X and Y are unlikely
         ApplyDebugLocation DL(*this, Cond);
         EmitBranchOnBoolExpr(CondBOp->getLHS(), TrueBlock, LHSFalse, LHSCount,
-                             Weights);
+                             LH == Stmt::LH_Likely ? Stmt::LH_None : LH);
         EmitBlock(LHSFalse);
       }
 
       incrementProfileCounter(CondBOp);
       setCurrentProfileCount(getProfileCount(CondBOp->getRHS()));
 
       // Any temporaries created here are conditional.
       eval.begin(*this);
       EmitBranchOnBoolExpr(CondBOp->getRHS(), TrueBlock, FalseBlock, RHSCount,
-                           Weights);
+                           LH);
 
       eval.end(*this);
 
       return;
     }
   }
 
   if (const UnaryOperator *CondUOp = dyn_cast<UnaryOperator>(Cond)) {
     // br(!x, t, f) -> br(x, f, t)
     if (CondUOp->getOpcode() == UO_LNot) {
       // Negate the count.
       uint64_t FalseCount = getCurrentProfileCount() - TrueCount;
+      // The values of the enum are chosen to make this negation possible.
+      LH = static_cast<Stmt::Likelihood>(-LH);
       // Negate the condition and swap the destination blocks.
       return EmitBranchOnBoolExpr(CondUOp->getSubExpr(), FalseBlock, TrueBlock,
-                                  FalseCount, Weights);
+                                  FalseCount, LH);
     }
   }
 
   if (const ConditionalOperator *CondOp = dyn_cast<ConditionalOperator>(Cond)) {
     // br(c ? x : y, t, f) -> br(c, br(x, t, f), br(y, t, f))
     llvm::BasicBlock *LHSBlock = createBasicBlock("cond.true");
     llvm::BasicBlock *RHSBlock = createBasicBlock("cond.false");
 
     ConditionalEvaluation cond(*this);
     EmitBranchOnBoolExpr(CondOp->getCond(), LHSBlock, RHSBlock,
-                         getProfileCount(CondOp), Weights);
+                         getProfileCount(CondOp), Stmt::LH_None);
 
     // When computing PGO branch weights, we only know the overall count for
     // the true block. This code is essentially doing tail duplication of the
     // naive code-gen, introducing new edges for which counts are not
     // available. Divide the counts proportionally between the LHS and RHS of
     // the conditional operator.
     uint64_t LHSScaledTrueCount = 0;
     if (TrueCount) {
       double LHSRatio =
           getProfileCount(CondOp) / (double)getCurrentProfileCount();
       LHSScaledTrueCount = TrueCount * LHSRatio;
     }
 
     cond.begin(*this);
     EmitBlock(LHSBlock);
     incrementProfileCounter(CondOp);
     {
       ApplyDebugLocation DL(*this, Cond);
       EmitBranchOnBoolExpr(CondOp->getLHS(), TrueBlock, FalseBlock,
-                           LHSScaledTrueCount, Weights);
+                           LHSScaledTrueCount, LH);
     }
     cond.end(*this);
 
     cond.begin(*this);
     EmitBlock(RHSBlock);
     EmitBranchOnBoolExpr(CondOp->getRHS(), TrueBlock, FalseBlock,
-                         TrueCount - LHSScaledTrueCount, Weights);
+                         TrueCount - LHSScaledTrueCount, LH);
     cond.end(*this);
 
     return;
   }
 
   if (const CXXThrowExpr *Throw = dyn_cast<CXXThrowExpr>(Cond)) {
     // Conditional operator handling can give us a throw expression as a
     // condition for a case like:
void CodeGenFunction::EmitBranchOnBoolExpr(const Expr *Cond,
   if (Call && CGM.getCodeGenOpts().OptimizationLevel != 0) {
     auto *FD = dyn_cast_or_null<FunctionDecl>(Call->getCalleeDecl());
     if (FD && FD->getBuiltinID() == Builtin::BI__builtin_unpredictable) {
       llvm::MDBuilder MDHelper(getLLVMContext());
       Unpredictable = MDHelper.createUnpredictable();
     }
   }
 
   // Create branch weights based on the number of times we get here and the
   // number of times the condition should be true.
+  llvm::MDNode *Weights = nullptr;
+  Optional<std::pair<uint32_t, uint32_t>> LHW = getLikelihoodWeights(LH);
+  if (LHW) {
+    llvm::MDBuilder MDHelper(CGM.getLLVMContext());
+    Weights = MDHelper.createBranchWeights(LHW->first, LHW->second);
+  }
   if (!Weights) {
     uint64_t CurrentCount = std::max(getCurrentProfileCount(), TrueCount);
     Weights = createProfileWeights(TrueCount, CurrentCount - TrueCount);
   }
 
   // Emit the code with the fully general case.
   llvm::Value *CondV;
   {

clang/test/CodeGenCXX/attr-likelihood-if-vs-builtin-expect.cpp

+// RUN: %clang_cc1 -O1 -emit-llvm %s -o - -triple=x86_64-linux-gnu | FileCheck %s
+
+// Verifies the output of __builtin_expect versus the output of the likelihood
+// attributes. They should generate the same probabilities for the branches.
+
+extern bool a();
+extern bool b();
+extern bool c();
+
+void ab1(int &i) {
+  // CHECK-LABEL: define{{.*}}ab1
+  // CHECK: br {{.*}} !prof !2
+  // CHECK: br {{.*}} !prof !2
+  // CHECK: br {{.*}} !prof !2
+  if (__builtin_expect(a() && b() && a(), 1)) {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void al(int &i) {
+  // CHECK-LABEL: define{{.*}}al
+  // CHECK: br {{.*}} !prof !2
+  // CHECK: br {{.*}} !prof !2
+  // CHECK: br {{.*}} !prof !2
+  if (a() && b() && c()) [[likely]] {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void ab0(int &i) {
+  // CHECK-LABEL: define{{.*}}ab0
+  // CHECK: br {{.*}}else{{$}}
+  // CHECK: br {{.*}}else{{$}}
+  // CHECK: br {{.*}} !prof !8
+  if (__builtin_expect(a() && b() && c(), 0)) {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void au(int &i) {
+  // CHECK-LABEL: define{{.*}}au
+  // CHECK: br {{.*}}else{{$}}
+  // CHECK: br {{.*}}else{{$}}
+  // CHECK: br {{.*}} !prof !8
+  if (a() && b() && c()) [[unlikely]] {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void ob1(int &i) {
+  // CHECK-LABEL: define{{.*}}ob1
+  // CHECK: br {{.*}}false{{$}}
+  // CHECK: br {{.*}}rhs{{$}}
+  // CHECK: br {{.*}} !prof !2
+  if (__builtin_expect(a() || b() || a(), 1)) {
+    i = 0;
+  } else {
+    --i;
+  }
+}
+
+void ol(int &i) {
+  // CHECK-LABEL: define{{.*}}ol
+  // CHECK: br {{.*}}false{{$}}
+  // CHECK: br {{.*}}false2{{$}}
+  // CHECK: br {{.*}} !prof !2
+  if (a() || b() || c()) [[likely]] {
+    i = 0;
+  } else {
+    --i;
+  }
+}
+
+void ob0(int &i) {
+  // CHECK-LABEL: define{{.*}}ob0
+  // CHECK: br {{.*}} !prof !8
+  // CHECK: br {{.*}} !prof !8
+  // CHECK: br {{.*}} !prof !8
+  if (__builtin_expect(a() || b() || c(), 0)) {
+    i = 0;
+  } else {
+    --i;
+  }
+}
+
+void ou(int &i) {
+  // CHECK-LABEL: define{{.*}}ou
+  // CHECK: br {{.*}} !prof !8
+  // CHECK: br {{.*}} !prof !8
+  // CHECK: br {{.*}} !prof !8
+  if (a() || b() || c()) [[unlikely]] {
+    i = 0;
+  } else {
+    --i;
+  }
+}
+
+void nb1(int &i) {
+  // CHECK-LABEL: define{{.*}}nb1
+  // CHECK: storemerge{{.*}} !prof !8
+  if (__builtin_expect(!a(), 1)) {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void nl(int &i) {
+  // CHECK-LABEL: define{{.*}}nl
+  // CHECK: storemerge{{.*}} !prof !8
+  if (!a()) [[likely]] {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void nb0(int &i) {
+  // CHECK-LABEL: define{{.*}}nb0
+  // CHECK: storemerge{{.*}} !prof !2
+  if (__builtin_expect(!a(), 0)) {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void nu(int &i) {
+  // CHECK-LABEL: define{{.*}}nu
+  // CHECK: storemerge{{.*}} !prof !2
+  if (!a()) [[unlikely]] {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void tb1(int &i) {
+  // CHECK-LABEL: define{{.*}}tb1
+  // CHECK: br {{.*}}false{{$}}
+  // CHECK: br {{.*}}end{{$}}
+  // CHECK: br {{.*}}end{{$}}
+  // CHECK: storemerge{{.*}} !prof !2
+  if (__builtin_expect(a() ? b() : c(), 1)) {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void tl(int &i) {
+  // CHECK-LABEL: define{{.*}}tl
+  // CHECK: br {{.*}}false{{$}}
+  // CHECK: br {{.*}}end{{$}}
+  // CHECK: br {{.*}}end{{$}}
+  // CHECK: storemerge{{.*}} !prof !2
+  if (bool d = a() ? b() : c()) [[likely]] {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void tl2(int &i) {
+  // CHECK-LABEL: define{{.*}}tl
+  // CHECK: br {{.*}}false{{$}}
+  // CHECK: br {{.*}} !prof !2
+  // CHECK: br {{.*}} !prof !2
+  if (a() ? b() : c()) [[likely]] {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void tb0(int &i) {
+  // CHECK-LABEL: define{{.*}}tb0
+  // CHECK: br {{.*}}false{{$}}
+  // CHECK: br {{.*}}end{{$}}
+  // CHECK: br {{.*}}end{{$}}
+  // CHECK: storemerge{{.*}} !prof !8
+  if (__builtin_expect(a() ? b() : c(), 0)) {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void tu(int &i) {
+  // CHECK-LABEL: define{{.*}}tu
+  // CHECK: br {{.*}}false{{$}}
+  // CHECK: br {{.*}}end{{$}}
+  // CHECK: br {{.*}}end{{$}}
+  // CHECK: storemerge{{.*}} !prof !8
+  if (bool d = a() ? b() : c()) [[unlikely]] {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+void tu2(int &i) {
+  // CHECK-LABEL: define{{.*}}tu
+  // CHECK: br {{.*}}false{{$}}
+  // CHECK: br {{.*}} !prof !8
+  // CHECK: br {{.*}} !prof !8
+  if (a() ? b() : c()) [[unlikely]] {
+    ++i;
+  } else {
+    --i;
+  }
+}
+
+// CHECK: !2 = !{!"branch_weights", i32 2000, i32 1}
+// CHECK: !8 = !{!"branch_weights", i32 1, i32 2000}