[CodeGen] recognize select form of and/ors when splitting branch conditions

[aqjune]

Summary

Recently a few patches are made to move towards using select i1 instead of and/or i1 to represent "a && b"/"a || b" in C/C++.
"a && b" in C/C++ does not evaluate b if a is false whereas 'and a, b' in IR evaluates b and uses its result regardless of the result of a.
This is problematic because it can cause miscompilation if b was an erroneous operation (https://llvm.org/pr48353).
In C/C++, the result is simply false because b is not evaluated, but in IR the result is poison.
The discussion at D93065 has more context about this.

This patch makes two branch-splitting optimizations (one in SelectionDAGBuilder, one in CodeGenPrepare) recognize
select form of and/or as well using m_LogicalAnd/Or.
Since it is CodeGen, I think this is semantically ok (at least as safe as what codegen already did).

[nikic]

This looks correct to me, as we're breaking the select up into two branches, which means the second condition will get short-circuited entirely. As far as I can see, the conditions don't get swapped or similar.

However, I think it's worthwhile to consider an alternative here: We could simply fold select -> and/or in CGP, and forget about this as far as CodeGen is concerned. This is not technically correct, because SDAG also has a poison concept, but it's kind of a moot point, as DAGCombiner will end up doing the transform anyway: https://github.com/llvm/llvm-project/blob/a485a59d2172daaee1d5e734da54fbb243f7d54c/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp#L9197-L9220 That's something we could revisit if we ever wanted to drop the corresponding DAGCombiner fold (but I believe our current stance is to ignore such issues in the SDAG layer, as poison reasoning is used much less in this area and we've not seen any practical poison-related miscompiles in SDAG).

llvm/lib/CodeGen/CodeGenPrepare.cpp

7868

I think this will make the optimization not work for a chain of logical and/or, as this restricts to m_BinOp.

[aqjune]

clang-format, address a comment

[aqjune]

I'm slightly worried that it will still preserve unsound IR-level transformation (because CodeGenPrepare is in IR). :( A bug in CodeGenPrepare will be hard to discover because opt -O<n> won't use it.
If DAGCombiner does that, would it be still okay in terms of performance? I see that SelectionDAGISel::CodeGenAndEmitDAG invokes CurDAG->Combine first.

[aemerson]

We do this condition splitting optimization in GlobalISel too, so that will also need fixing: see IRTranslator.cpp.

[aqjune]

Support GlobalISel

[aqjune]

Thanks! Added tests for globalisel as well

[nikic]

In D93853#2473140, @aqjune wrote:

I'm slightly worried that it will still preserve unsound IR-level transformation (because CodeGenPrepare is in IR). :( A bug in CodeGenPrepare will be hard to discover because opt -O<n> won't use it.
If DAGCombiner does that, would it be still okay in terms of performance? I see that SelectionDAGISel::CodeGenAndEmitDAG invokes CurDAG->Combine first.

Fair enough. CGP does call things like InstructionSimplify, so there is some potential for issues... I was concerned that keeping select would cause SDAG fallback in FastISel, but apparently that will happen anyway for non-branch uses of and/or (and branch uses are split).

So this LGTM.

llvm/lib/CodeGen/CodeGenPrepare.cpp

 //===- CodeGenPrepare.cpp - Prepare a function for code generation --------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
bool CodeGenPrepare::splitBranchCondition(Function &F, bool &ModifiedDT) {
 
   bool MadeChange = false;
   for (auto &BB : F) {
     // Does this BB end with the following?
     //   %cond1 = icmp|fcmp|binary instruction ...
     //   %cond2 = icmp|fcmp|binary instruction ...
     //   %cond.or = or|and i1 %cond1, cond2
     //   br i1 %cond.or label %dest1, label %dest2"
-    BinaryOperator *LogicOp;
+    Instruction *LogicOp;
     BasicBlock *TBB, *FBB;
-    if (!match(BB.getTerminator(), m_Br(m_OneUse(m_BinOp(LogicOp)), TBB, FBB)))
+    if (!match(BB.getTerminator(),
+               m_Br(m_OneUse(m_Instruction(LogicOp)), TBB, FBB)))
       continue;
 
     auto *Br1 = cast<BranchInst>(BB.getTerminator());
     if (Br1->getMetadata(LLVMContext::MD_unpredictable))
       continue;
 
     // The merging of mostly empty BB can cause a degenerate branch.
     if (TBB == FBB)
       continue;
 
     unsigned Opc;
     Value *Cond1, *Cond2;
-    if (match(LogicOp, m_And(m_OneUse(m_Value(Cond1)),
+    if (match(LogicOp, m_LogicalAnd(m_OneUse(m_Value(Cond1)),
                                     m_OneUse(m_Value(Cond2)))))
       Opc = Instruction::And;
-    else if (match(LogicOp, m_Or(m_OneUse(m_Value(Cond1)),
+    else if (match(LogicOp, m_LogicalOr(m_OneUse(m_Value(Cond1)),
                                         m_OneUse(m_Value(Cond2)))))
       Opc = Instruction::Or;
     else
       continue;
 
     if (!match(Cond1, m_CombineOr(m_Cmp(), m_BinOp())) ||
-        !match(Cond2, m_CombineOr(m_Cmp(), m_BinOp()))   )
+        !match(Cond2, m_CombineOr(m_Cmp(), m_BinOp())))
       continue;
 
     LLVM_DEBUG(dbgs() << "Before branch condition splitting\n"; BB.dump());
 
     // Create a new BB.
     auto *TmpBB =
         BasicBlock::Create(BB.getContext(), BB.getName() + ".cond.split",
                            BB.getParent(), BB.getNextNode());

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

 //===- SelectionDAGBuilder.cpp - Selection-DAG building -------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
void SelectionDAGBuilder::FindMergedConditions(const Value *Cond,
   if (match(Cond, m_OneUse(m_Not(m_Value(NotCond)))) &&
       InBlock(NotCond, CurBB->getBasicBlock())) {
     FindMergedConditions(NotCond, TBB, FBB, CurBB, SwitchBB, Opc, TProb, FProb,
                          !InvertCond);
     return;
   }
 
   const Instruction *BOp = dyn_cast<Instruction>(Cond);
+  const Value *BOpOp0, *BOpOp1;
   // Compute the effective opcode for Cond, taking into account whether it needs
   // to be inverted, e.g.
   //   and (not (or A, B)), C
   // gets lowered as
   //   and (and (not A, not B), C)
   unsigned BOpc = 0;
   if (BOp) {
-    BOpc = BOp->getOpcode();
+    BOpc = match(BOp, m_LogicalAnd(m_Value(BOpOp0), m_Value(BOpOp1))) ?
+        Instruction::And :
+        (match(BOp, m_LogicalOr(m_Value(BOpOp0), m_Value(BOpOp1))) ?
+         Instruction::Or : 0);
     if (InvertCond) {
       if (BOpc == Instruction::And)
         BOpc = Instruction::Or;
       else if (BOpc == Instruction::Or)
         BOpc = Instruction::And;
     }
   }
 
   // If this node is not part of the or/and tree, emit it as a branch.
-  if (!BOp || !(isa<BinaryOperator>(BOp) || isa<CmpInst>(BOp)) ||
-      BOpc != unsigned(Opc) || !BOp->hasOneUse() ||
-      BOp->getParent() != CurBB->getBasicBlock() ||
-      !InBlock(BOp->getOperand(0), CurBB->getBasicBlock()) ||
-      !InBlock(BOp->getOperand(1), CurBB->getBasicBlock())) {
+  // Note that all nodes in the tree should have same opcode.
+  bool BOpIsInOrAndTree = BOpc && BOpc == unsigned(Opc) && BOp->hasOneUse();
+  if (!BOpIsInOrAndTree || BOp->getParent() != CurBB->getBasicBlock() ||
+      !InBlock(BOpOp0, CurBB->getBasicBlock()) ||
+      !InBlock(BOpOp1, CurBB->getBasicBlock())) {
     EmitBranchForMergedCondition(Cond, TBB, FBB, CurBB, SwitchBB,
                                  TProb, FProb, InvertCond);
     return;
   }
 
   //  Create TmpBB after CurBB.
   MachineFunction::iterator BBI(CurBB);
   MachineFunction &MF = DAG.getMachineFunction();
if (Opc == Instruction::Or) {
     // A/(1+B) and 2B/(1+B). This choice assumes that
     //   TrueProb for BB1 == FalseProb for BB1 * TrueProb for TmpBB.
     // Another choice is to assume TrueProb for BB1 equals to TrueProb for
     // TmpBB, but the math is more complicated.
 
     auto NewTrueProb = TProb / 2;
     auto NewFalseProb = TProb / 2 + FProb;
     // Emit the LHS condition.
-    FindMergedConditions(BOp->getOperand(0), TBB, TmpBB, CurBB, SwitchBB, Opc,
+    FindMergedConditions(BOpOp0, TBB, TmpBB, CurBB, SwitchBB, Opc,
                          NewTrueProb, NewFalseProb, InvertCond);
 
     // Normalize A/2 and B to get A/(1+B) and 2B/(1+B).
     SmallVector<BranchProbability, 2> Probs{TProb / 2, FProb};
     BranchProbability::normalizeProbabilities(Probs.begin(), Probs.end());
     // Emit the RHS condition into TmpBB.
-    FindMergedConditions(BOp->getOperand(1), TBB, FBB, TmpBB, SwitchBB, Opc,
+    FindMergedConditions(BOpOp1, TBB, FBB, TmpBB, SwitchBB, Opc,
                          Probs[0], Probs[1], InvertCond);
   } else {
     assert(Opc == Instruction::And && "Unknown merge op!");
     // Codegen X & Y as:
     // BB1:
     //   jmp_if_X TmpBB
     //   jmp FBB
     // TmpBB:
if (Opc == Instruction::Or) {
     // Assuming the original probabilities are A and B, one choice is to set
     // BB1's probabilities to A+B/2 and B/2, and set TmpBB's probabilities to
     // 2A/(1+A) and B/(1+A). This choice assumes that FalseProb for BB1 ==
     // TrueProb for BB1 * FalseProb for TmpBB.
 
     auto NewTrueProb = TProb + FProb / 2;
     auto NewFalseProb = FProb / 2;
     // Emit the LHS condition.
-    FindMergedConditions(BOp->getOperand(0), TmpBB, FBB, CurBB, SwitchBB, Opc,
+    FindMergedConditions(BOpOp0, TmpBB, FBB, CurBB, SwitchBB, Opc,
                          NewTrueProb, NewFalseProb, InvertCond);
 
     // Normalize A and B/2 to get 2A/(1+A) and B/(1+A).
     SmallVector<BranchProbability, 2> Probs{TProb, FProb / 2};
     BranchProbability::normalizeProbabilities(Probs.begin(), Probs.end());
     // Emit the RHS condition into TmpBB.
-    FindMergedConditions(BOp->getOperand(1), TBB, FBB, TmpBB, SwitchBB, Opc,
+    FindMergedConditions(BOpOp1, TBB, FBB, TmpBB, SwitchBB, Opc,
                          Probs[0], Probs[1], InvertCond);
   }
 }
 
 /// If the set of cases should be emitted as a series of branches, return true.
 /// If we should emit this as a bunch of and/or'd together conditions, return
 /// false.
 bool
void SelectionDAGBuilder::visitBr(const BranchInst &I) {
   //     F = setle
   //     or C, F
   //     jnz foo
   // Emit:
   //     cmp A, B
   //     je foo
   //     cmp D, E
   //     jle foo
-  if (const BinaryOperator *BOp = dyn_cast<BinaryOperator>(CondVal)) {
-    Instruction::BinaryOps Opcode = BOp->getOpcode();
-    Value *Vec, *BOp0 = BOp->getOperand(0), *BOp1 = BOp->getOperand(1);
+  if (const Instruction *BOp = dyn_cast<Instruction>(CondVal)) {
+    Value *Vec;
+    const Value *BOp0 = nullptr, *BOp1 = nullptr;
     if (!DAG.getTargetLoweringInfo().isJumpExpensive() && BOp->hasOneUse() &&
-        !I.hasMetadata(LLVMContext::MD_unpredictable) &&
-        (Opcode == Instruction::And || Opcode == Instruction::Or) &&
+        !I.hasMetadata(LLVMContext::MD_unpredictable)) {
+      Instruction::BinaryOps Opcode;
+      if (match(BOp, m_LogicalAnd(m_Value(BOp0), m_Value(BOp1))))
+        Opcode = Instruction::And;
+      else if (match(BOp, m_LogicalOr(m_Value(BOp0), m_Value(BOp1))))
+        Opcode = Instruction::Or;
+
+      if (BOp0 && BOp1 &&
           !(match(BOp0, m_ExtractElt(m_Value(Vec), m_Value())) &&
             match(BOp1, m_ExtractElt(m_Specific(Vec), m_Value())))) {
-      FindMergedConditions(BOp, Succ0MBB, Succ1MBB, BrMBB, BrMBB,
-                           Opcode,
+        FindMergedConditions(BOp, Succ0MBB, Succ1MBB, BrMBB, BrMBB, Opcode,
                              getEdgeProbability(BrMBB, Succ0MBB),
                              getEdgeProbability(BrMBB, Succ1MBB),
                              /*InvertCond=*/false);
         // If the compares in later blocks need to use values not currently
         // exported from this block, export them now.  This block should always
         // be the first entry.
         assert(SL->SwitchCases[0].ThisBB == BrMBB && "Unexpected lowering!");
 
         // Allow some cases to be rejected.
         if (ShouldEmitAsBranches(SL->SwitchCases)) {
           for (unsigned i = 1, e = SL->SwitchCases.size(); i != e; ++i) {
             ExportFromCurrentBlock(SL->SwitchCases[i].CmpLHS);
             ExportFromCurrentBlock(SL->SwitchCases[i].CmpRHS);
           }
 
           // Emit the branch for this block.
           visitSwitchCase(SL->SwitchCases[0], BrMBB);
           SL->SwitchCases.erase(SL->SwitchCases.begin());
           return;
         }
 
         // Okay, we decided not to do this, remove any inserted MBB's and clear
         // SwitchCases.
         for (unsigned i = 1, e = SL->SwitchCases.size(); i != e; ++i)
           FuncInfo.MF->erase(SL->SwitchCases[i].ThisBB);
 
         SL->SwitchCases.clear();
       }
     }
+  }
 
   // Create a CaseBlock record representing this branch.
   CaseBlock CB(ISD::SETEQ, CondVal, ConstantInt::getTrue(*DAG.getContext()),
                nullptr, Succ0MBB, Succ1MBB, BrMBB, getCurSDLoc());
 
   // Use visitSwitchCase to actually insert the fast branch sequence for this
   // cond branch.
   visitSwitchCase(CB, BrMBB);

llvm/test/CodeGen/AArch64/arm64_32.ll

 true:
   call void(...) @bar()
   ret void
 
 false:
   ret void
 }
 
+define void @test_multiple_icmp_ptr_select(i8* %l, i8* %r) {
+; CHECK-LABEL: test_multiple_icmp_ptr_select:
+; CHECK: tbnz w0, #31, [[FALSEBB:LBB[0-9]+_[0-9]+]]
+; CHECK: tbnz w1, #31, [[FALSEBB]]
+  %tst1 = icmp sgt i8* %l, inttoptr (i32 -1 to i8*)
+  %tst2 = icmp sgt i8* %r, inttoptr (i32 -1 to i8*)
+  %tst = select i1 %tst1, i1 %tst2, i1 false
+  br i1 %tst, label %true, label %false
+
+true:
+  call void(...) @bar()
+  ret void
+
+false:
+  ret void
+}
+
 define { [18 x i8] }* @test_gep_nonpow2({ [18 x i8] }* %a0, i32 %a1) {
 ; CHECK-LABEL: test_gep_nonpow2:
 ; CHECK-OPT:      mov w[[SIZE:[0-9]+]], #18
 ; CHECK-OPT-NEXT: smaddl x0, w1, w[[SIZE]], x0
 ; CHECK-OPT-NEXT: ret
 
 ; CHECK-FAST:      mov w[[SIZE:[0-9]+]], #18
 ; CHECK-FAST-NEXT: smaddl [[TMP:x[0-9]+]], w1, w[[SIZE]], x0

llvm/test/CodeGen/AArch64/fast-isel-branch-cond-split.ll

 bb3:
   ret i64 0
 
 bb4:
   %2 = call i64 @bar()
   ret i64 %2
 }
 
+; CHECK-LABEL: test_or_select
+; CHECK:       cbnz w0, {{LBB[0-9]+_2}}
+; CHECK:       cbz w1, {{LBB[0-9]+_1}}
+define i64 @test_or_select(i32 %a, i32 %b) {
+bb1:
+  %0 = icmp eq i32 %a, 0
+  %1 = icmp eq i32 %b, 0
+  %or.cond = select i1 %0, i1 true, i1 %1
+  br i1 %or.cond, label %bb3, label %bb4, !prof !0
+
+bb3:
+  ret i64 0
+
+bb4:
+  %2 = call i64 @bar()
+  ret i64 %2
+}
+
 ; CHECK-LABEL: test_and
 ; CHECK:       cbnz w0, {{LBB[0-9]+_2}}
 ; CHECK:       cbz w1, {{LBB[0-9]+_1}}
 define i64 @test_and(i32 %a, i32 %b) {
 bb1:
   %0 = icmp ne i32 %a, 0
   %1 = icmp ne i32 %b, 0
   %or.cond = and i1 %0, %1
   br i1 %or.cond, label %bb4, label %bb3, !prof !1
 
 bb3:
   ret i64 0
 
 bb4:
   %2 = call i64 @bar()
   ret i64 %2
 }
 
+; CHECK-LABEL: test_and_select
+; CHECK:       cbnz w0, {{LBB[0-9]+_2}}
+; CHECK:       cbz w1, {{LBB[0-9]+_1}}
+define i64 @test_and_select(i32 %a, i32 %b) {
+bb1:
+  %0 = icmp ne i32 %a, 0
+  %1 = icmp ne i32 %b, 0
+  %or.cond = select i1 %0, i1 %1, i1 false
+  br i1 %or.cond, label %bb4, label %bb3, !prof !1
+
+bb3:
+  ret i64 0
+
+bb4:
+  %2 = call i64 @bar()
+  ret i64 %2
+}
+
 ; If the branch is unpredictable, don't add another branch.
 
 ; CHECK-LABEL: test_or_unpredictable
 ; CHECK:       cmp   w0, #0
 ; CHECK-NEXT:  cset  w8, eq
 ; CHECK-NEXT:  cmp   w1, #0
 ; CHECK-NEXT:  cset  w9, eq
 ; CHECK-NEXT:  orr   w8, w8, w9