[InstCombine] Transform X == 0 ? 0 : X * Y --> X * freeze(Y).

[fzhinkin]

Summary

Enabled folding optimization that was previously disabled by being incorrect.
To preserve correctness mul's operand that is not compared with zero in select's condition is now frozen.

Related bug: https://bugs.llvm.org/show_bug.cgi?id=51286

Correctness:
https://alive2.llvm.org/ce/z/bHef7J
https://alive2.llvm.org/ce/z/QcR7sf
https://alive2.llvm.org/ce/z/vvBLzt
https://alive2.llvm.org/ce/z/jGDXgq
https://alive2.llvm.org/ce/z/3Pe8Z4
https://alive2.llvm.org/ce/z/LGga8M
https://alive2.llvm.org/ce/z/CTG5fs

[lebedev.ri]

InstSimplify can not create new instructions
This needs to go into instcombine
While there, please let's create the multiply, it's less ugly.

[fzhinkin]

Moved fold optimization to instcombine, updated tests.

[fzhinkin]

In D108408#2955906, @lebedev.ri wrote:

InstSimplify can not create new instructions
This needs to go into instcombine
While there, please let's create the multiply, it's less ugly.

Thanks for suggestion!

[lebedev.ri]

1. what's wrong with vectors?
2. can't we preserve no-wrap flags?
3. for vectors, what if some element is not zero but undef? (see Constant::mergeUndefsWith())

[fzhinkin]

In D108408#2958323, @lebedev.ri wrote:

1. what's wrong with vectors?

There's nothing wrong, I'll support it, thanks!

2. can't we preserve no-wrap flags?

Definitely.

3. for vectors, what if some element is not zero but undef? (see Constant::mergeUndefsWith())

Not sure that I fully understand the issue.
If %a in the snippet below contains some undefs then the select may choose either 0 or %m depending on particular value of a (which could be 0 for undef elements).
Probably %a should be frozen to ensure that both icmp and mul will see the same a's value, but (if I understood it correctly) absence of freeze does not compromise the correctness.

define <2 x i4> @src(<2 x i4> %a, <2 x i4> %b) {
  %c = icmp eq <2 x i4> %a, zeroinitializer
  %m = mul <2 x i4> %a, %b
  %r = select <2 x i1> %c, <2 x i4> zeroinitializer, <2 x i4> %m
  ret <2 x i4> %r
}

Or you meant that the code could be folded even if icmp's RHS is not only the all-zeros vector, but a constant-vector containing some undefs?

[RKSimon]

llvm/test/Transforms/InstCombine/select.ll

3028

please can you not enumerate tests, and give them names that describe the purpose of the test - e.g. mul_select_eq_zero, mul_select_eq_zero_commute, mul_select_eq_zero_vector, mul_select_ne_zero

3046

I'm not sure this test does anything useful? instcombine will canonicalize the 0 to the RHS

[fzhinkin]

Enabled folding for vectors, copied flags for mul, renamed tests.

[lebedev.ri]

In D108408#2958735, @fzhinkin wrote:

In D108408#2958323, @lebedev.ri wrote:

1. what's wrong with vectors?

There's nothing wrong, I'll support it, thanks!

2. can't we preserve no-wrap flags?

Definitely.

3. for vectors, what if some element is not zero but undef? (see Constant::mergeUndefsWith())

Not sure that I fully understand the issue.
If %a in the snippet below contains some undefs then the select may choose either 0 or %m depending on particular value of a (which could be 0 for undef elements).
Probably %a should be frozen to ensure that both icmp and mul will see the same a's value, but (if I understood it correctly) absence of freeze does not compromise the correctness.

define <2 x i4> @src(<2 x i4> %a, <2 x i4> %b) {
  %c = icmp eq <2 x i4> %a, zeroinitializer
  %m = mul <2 x i4> %a, %b
  %r = select <2 x i1> %c, <2 x i4> zeroinitializer, <2 x i4> %m
  ret <2 x i4> %r
}

Or you meant that the code could be folded even if icmp's RHS is not only the all-zeros vector, but a constant-vector containing some undefs?

I mean that if in either zero constant vector, a particular element is undef, then in the other constant vector, the same element can be anything:
https://alive2.llvm.org/ce/z/-gt253
https://alive2.llvm.org/ce/z/3PAU3E

So i think you want if(!match(Constant::mergeUndefsWith(TrueVal, CondVal.getOperand(1)), m_Zero)) return;

[fzhinkin]

In D108408#2962950, @lebedev.ri wrote:

In D108408#2958735, @fzhinkin wrote:

In D108408#2958323, @lebedev.ri wrote:

1. what's wrong with vectors?

There's nothing wrong, I'll support it, thanks!

2. can't we preserve no-wrap flags?

Definitely.

3. for vectors, what if some element is not zero but undef? (see Constant::mergeUndefsWith())

Not sure that I fully understand the issue.
If %a in the snippet below contains some undefs then the select may choose either 0 or %m depending on particular value of a (which could be 0 for undef elements).
Probably %a should be frozen to ensure that both icmp and mul will see the same a's value, but (if I understood it correctly) absence of freeze does not compromise the correctness.

define <2 x i4> @src(<2 x i4> %a, <2 x i4> %b) {
  %c = icmp eq <2 x i4> %a, zeroinitializer
  %m = mul <2 x i4> %a, %b
  %r = select <2 x i1> %c, <2 x i4> zeroinitializer, <2 x i4> %m
  ret <2 x i4> %r
}

Or you meant that the code could be folded even if icmp's RHS is not only the all-zeros vector, but a constant-vector containing some undefs?

I mean that if in either zero constant vector, a particular element is undef, then in the other constant vector, the same element can be anything:
https://alive2.llvm.org/ce/z/-gt253
https://alive2.llvm.org/ce/z/3PAU3E

So i think you want if(!match(Constant::mergeUndefsWith(TrueVal, CondVal.getOperand(1)), m_Zero)) return;

Got it, thanks for explanation!

[fzhinkin]

Supproted undef values in both condition's and select's arguments.

[nikic]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

752

Should either check that the mul only has one use, or else make sure that other mul users get replaced with the new frozen form as well, to avoid having two muls.

[fzhinkin]

Replace original mul with mul using frozen input.

[nikic]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

758

What ensures that the other case is Y == CmpLHS? Couldn't this be comparing a completely unrelated value?

761

In InstCombine, replaceInstUsesWith should be used in place of replaceAllUsesWith for worklist management reasons.

[nikic]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

761

Also, can you please test the case where the other user is before the select? I think as written you're inserting the mul before the select, so it may not dominate other users. You would have to adjust the insertion point.

(This might also be one of the rare cases where modifying the mul in place to freeze one operand is more elegant than creating a new one and replacing users...)

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

741

Why match an arbitrary constant if this pattern requires a zero constant? Use m_Zero() or some variation of that.

742

Use ICmpInst::isEquality(Pred)

758

General note: each review comment like this one is a requirement for a regression test - either positive (ie, the transform works) or negative (the transform fails).

Instead of matching 2 arbitrary values with m_Mul(), consider using m_c_Mul(m_Specific(CmpLHS), m_Value(Y)).
(Adjust the variable names and/or code comments, so they line up.)

[fzhinkin]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

741

I'm capturing it to avoid bunch of casts to extract it from CondVal later on (to check if TrueVal merges with CmpRHS into zero). But as it rises some confusion I will use m_Zero here and will extract CmpRHS before actual use.

758

@nikic You're right, thanks!

@spatel thanks for suggestion!

761

Thanks! Mul's modification requires less code in that case. Not sure about elegance, but I'll stick with it.

[fzhinkin]

Cleaned up the code, updated tests.

[fzhinkin]

ping

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

746–749

I'm still confused about this: we want to match a zero constant as the TrueVal based on the block comment for this function, so why not use m_Zero()?
Usually, we can match vectors that include undef elements, but we may not be able to propagate those undefs through a transform (typically, we would replace with a ConstantInt::getNullValue() to get a "full" zero).

If the partial undef logic is complicated, I think that would be ok to defer to a follow-up patch. Ie, let's get the basic/common case right, then look at corner cases if that really matters.

[lebedev.ri]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

746–749

Please read previous review comments in this patch. We are going in circles here.

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

746–749

Ah, sorry I missed that. And now I see that @mul_select_eq_undef_vector is testing a non-zero constant.
I think it would be good to pre-commit the tests and add some explanatory comments, so that's easier to spot.
Also, the multi-use test would be easier to follow if it took the more common call void @use() pattern.

[fzhinkin]

Added comments to both transformation and tests, made multi-user test more readable.

[fzhinkin]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

746–749

it would be good to pre-commit the tests

Could you please explain what did you meant here?

As of other suggestions - I added comments to both code and tests and refactored multi-use test case to use @use_i32.

Thanks!

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

746–749

I mean we can commit the tests with the current (not transformed) output verified by the CHECK lines. Then, we apply the code change here and update the tests. That way, we show the diffs on the tests rather than just the new output.

If you do not have commit access, let me know. I can push the baseline tests for you.

[fzhinkin]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

746–749

I don't have commit access, so I'll very appreciate if you commit the baseline tests.
Besides changing checks I also removed tests' comments as comments does not make much sense without the transformation.

Here is the patch:

D108408-pre-commit-tests.patch5 KBDownload

.

Thanks!

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

746–749

Thanks - I pushed that:
745f82b8d909

So now you can update here by regenerating the CHECK lines and adding the test comments back.

[fzhinkin]

Rebased changes.

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

746–749

Thank you!

[spatel]

LGTM, but let's wait at least a day to commit in case there are any other comments or suggestions.

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

754

Should that be something like:
"...when it is a scalar undef value or a vector containing non-zero elements that are masked by undef elements in the compare constant."

[nikic]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

768

I believe this may fail if the multiplication is a constant expression (we should bail in that case).

769

It would be better to create the freeze using IRBuilder, because it will be queued for reprocessing that way. Or alternatively using InsertNewInstBefore, which also handles this correctly.

[fzhinkin]

Updated comments, fixed few bugs.

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

754

Yes, it looks better, thanks!

768

I was not able to reproduce that case w/ tests because all my const expressions were folded before invocation of newly added folding, but I'll add the check, thanks.

769

Thanks for suggestion, I'll use InsertNewInstBefore.

[nikic]

LG

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

768

The trick to that is usually embedding globals in to the constant expression, something like mul (i64 ptrtoint (i8* @g1 to i64). i64 ptrtoint (i8* @g2 to i64)) might do.

llvm/test/Transforms/InstCombine/select.ll

2888

As you can see, this doesn't actually check the ne case, as the icmp is canonicalize to eq first. I think it may be possible to prevent this by adding an extra use of the icmp.

[fzhinkin]

Fixed the test w/ icmp ne by preventing icmp's transformation.

[fzhinkin]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

768

It seems impossible to have ConstantExpr mul at the point where cast to Instruction is done:

• if mul is ConstantExpr then both its arguments are constant expressions too;
• to reach the cast one of the mul's operands should be compared with 0 by icmp and if both mul's operands are constant expressions then icmp should be a constant expression too;
• select's operands are icmp, mul and constant so select should be a constant expression.

So either mul is not a constant expression or the whole select is a constant expression. In latter case InstCombinerImpl::visitSelectInst will not be invoked.

llvm/test/Transforms/InstCombine/select.ll

2888

Thanks for pointing to it, fixed.

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

if (match(F, m_And(m_Value(X), m_APInt(NotC))) &&
     Constant *OrC = ConstantInt::get(Ty, *C);
     Value *NewSel = Builder.CreateSelect(Cond, OrC, Zero, "masksel", &Sel);
     return BinaryOperator::CreateOr(F, NewSel);
   }
 
   return nullptr;
 }
 
+//   select (x == 0), 0, x * y --> freeze(y) * x
+//   select (y == 0), 0, x * y --> freeze(x) * y
+//   select (x == 0), undef, x * y --> freeze(y) * x
+//   select (x == undef), 0, x * y --> freeze(y) * x
+// Usage of mul instead of 0 will make the result more poisonous,
+// so the operand that was not checked in the condition should be frozen.
+// The latter folding is applied only when a constant compared with x is
+// is a vector consisting of 0 and undefs. If a constant compared with x
+// is a scalar undefined value or undefined vector then an expression
+// should be already folded into a constant.
+static Instruction *foldSelectZeroOrMul(SelectInst &SI, InstCombinerImpl &IC) {
+  auto *CondVal = SI.getCondition();
+  auto *TrueVal = SI.getTrueValue();
+  auto *FalseVal = SI.getFalseValue();
+  Value *X, *Y;
+  ICmpInst::Predicate Predicate;
+
+  // Assuming that constant compared with zero is not undef (but it may be
+  // a vector with some undef elements). Otherwise (when a constant is undef)
+  // the select expression should be already simplified.
+  if (!match(CondVal, m_ICmp(Predicate, m_Value(X), m_Zero())) ||
+      !ICmpInst::isEquality(Predicate))
+    return nullptr;
+
+  if (Predicate == ICmpInst::ICMP_NE)
+    std::swap(TrueVal, FalseVal);
+
+  // Check that TrueVal is a constant instead of matching it with m_Zero()
+  // to handle the case when it is a scalar undef value or a vector containing
+  // non-zero elements that are masked by undef elements in the compare
+  // constant.
+  auto *TrueValC = dyn_cast<Constant>(TrueVal);
+  if (TrueValC == nullptr ||
+      !match(FalseVal, m_c_Mul(m_Specific(X), m_Value(Y))) ||
+      !isa<Instruction>(FalseVal))
+    return nullptr;
+
+  auto *ZeroC = cast<Constant>(cast<Instruction>(CondVal)->getOperand(1));
+  auto *MergedC = Constant::mergeUndefsWith(TrueValC, ZeroC);
+  // If X is compared with 0 then TrueVal could be either zero or undef.
+  // m_Zero match vectors containing some undef elements, but for scalars
+  // m_Undef should be used explicitly.
+  if (!match(MergedC, m_Zero()) && !match(MergedC, m_Undef()))
+    return nullptr;
+
+  auto *FalseValI = cast<Instruction>(FalseVal);
+  auto *FrY = IC.InsertNewInstBefore(new FreezeInst(Y, Y->getName() + ".fr"),
+                                     *FalseValI);
+  IC.replaceOperand(*FalseValI, FalseValI->getOperand(0) == Y ? 0 : 1, FrY);
+  return IC.replaceInstUsesWith(SI, FalseValI);
+}
+
 /// Transform patterns such as (a > b) ? a - b : 0 into usub.sat(a, b).
 /// There are 8 commuted/swapped variants of this pattern.
 /// TODO: Also support a - UMIN(a,b) patterns.
 static Value *canonicalizeSaturatedSubtract(const ICmpInst *ICI,
                                             const Value *TrueVal,
                                             const Value *FalseVal,
                                             InstCombiner::BuilderTy &Builder) {
   ICmpInst::Predicate Pred = ICI->getPredicate();
if (Instruction *Result = foldSelectInstWithICmp(SI, ICI))
       return Result;
 
   if (Instruction *Add = foldAddSubSelect(SI, Builder))
     return Add;
   if (Instruction *Add = foldOverflowingAddSubSelect(SI, Builder))
     return Add;
   if (Instruction *Or = foldSetClearBits(SI, Builder))
     return Or;
+  if (Instruction *Mul = foldSelectZeroOrMul(SI, *this))
+    return Mul;
 
   // Turn (select C, (op X, Y), (op X, Z)) -> (op X, (select C, Y, Z))
   auto *TI = dyn_cast<Instruction>(TrueVal);
   auto *FI = dyn_cast<Instruction>(FalseVal);
   if (TI && FI && TI->getOpcode() == FI->getOpcode())
     if (Instruction *IV = foldSelectOpOp(SI, TI, FI))
       return IV;
 

llvm/test/Transforms/InstCombine/select.ll

 define <2 x i1> @partial_false_undef_condval(<2 x i1> %x) {
 ; CHECK-LABEL: @partial_false_undef_condval(
 ; CHECK-NEXT:    ret <2 x i1> <i1 false, i1 poison>
 ;
   %r = select <2 x i1> <i1 false, i1 poison>, <2 x i1> %x, <2 x i1> <i1 false, i1 poison>
   ret <2 x i1> %r
 }
 
+; select (x == 0), 0, x * y --> freeze(y) * x
 define i32 @mul_select_eq_zero(i32 %x, i32 %y) {
 ; CHECK-LABEL: @mul_select_eq_zero(
-; CHECK-NEXT:    [[C:%.*]] = icmp eq i32 [[X:%.*]], 0
-; CHECK-NEXT:    [[M:%.*]] = mul i32 [[X]], [[Y:%.*]]
-; CHECK-NEXT:    [[R:%.*]] = select i1 [[C]], i32 0, i32 [[M]]
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-NEXT:    [[Y_FR:%.*]] = freeze i32 [[Y:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul i32 [[Y_FR]], [[X:%.*]]
+; CHECK-NEXT:    ret i32 [[M]]
 ;
   %c = icmp eq i32 %x, 0
   %m = mul i32 %x, %y
   %r = select i1 %c, i32 0, i32 %m
   ret i32 %r
 }
 
+; select (y == 0), 0, x * y --> freeze(x) * y
 define i32 @mul_select_eq_zero_commute(i32 %x, i32 %y) {
 ; CHECK-LABEL: @mul_select_eq_zero_commute(
-; CHECK-NEXT:    [[C:%.*]] = icmp eq i32 [[Y:%.*]], 0
-; CHECK-NEXT:    [[M:%.*]] = mul i32 [[X:%.*]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = select i1 [[C]], i32 0, i32 [[M]]
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-NEXT:    [[X_FR:%.*]] = freeze i32 [[X:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul i32 [[X_FR]], [[Y:%.*]]
+; CHECK-NEXT:    ret i32 [[M]]
 ;
   %c = icmp eq i32 %y, 0
   %m = mul i32 %x, %y
   %r = select i1 %c, i32 0, i32 %m
   ret i32 %r
 }
 
+; Check that mul's flags preserved during the transformation.
 define i32 @mul_select_eq_zero_copy_flags(i32 %x, i32 %y) {
 ; CHECK-LABEL: @mul_select_eq_zero_copy_flags(
-; CHECK-NEXT:    [[C:%.*]] = icmp eq i32 [[X:%.*]], 0
-; CHECK-NEXT:    [[M:%.*]] = mul nuw nsw i32 [[X]], [[Y:%.*]]
-; CHECK-NEXT:    [[R:%.*]] = select i1 [[C]], i32 0, i32 [[M]]
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-NEXT:    [[Y_FR:%.*]] = freeze i32 [[Y:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul nuw nsw i32 [[Y_FR]], [[X:%.*]]
+; CHECK-NEXT:    ret i32 [[M]]
 ;
   %c = icmp eq i32 %x, 0
   %m = mul nuw nsw i32 %x, %y
   %r = select i1 %c, i32 0, i32 %m
   ret i32 %r
 }
 
+; Check that the transformation could be applied after condition's inversion.
+; select (x != 0), x * y, 0 --> freeze(y) * x
 define i32 @mul_select_ne_zero(i32 %x, i32 %y) {
 ; CHECK-LABEL: @mul_select_ne_zero(
-; CHECK-NEXT:    [[C_NOT:%.*]] = icmp eq i32 [[X:%.*]], 0
-; CHECK-NEXT:    [[M:%.*]] = mul i32 [[X]], [[Y:%.*]]
-; CHECK-NEXT:    [[R:%.*]] = select i1 [[C_NOT]], i32 0, i32 [[M]]
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-NEXT:    [[C:%.*]] = icmp ne i32 [[X:%.*]], 0
+; CHECK-NEXT:    [[Y_FR:%.*]] = freeze i32 [[Y:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul i32 [[Y_FR]], [[X]]
+; CHECK-NEXT:    call void @use(i1 [[C]])
+; CHECK-NEXT:    ret i32 [[M]]
 ;
   %c = icmp ne i32 %x, 0
   %m = mul i32 %x, %y
   %r = select i1 %c, i32 %m, i32 0
+  call void @use(i1 %c)
   ret i32 %r
 }
 
+; Check that if one of a select's branches returns undef then
+; an expression could be folded into mul as if there was a 0 instead of undef.
+; select (x == 0), undef, x * y --> freeze(y) * x
 define i32 @mul_select_eq_zero_sel_undef(i32 %x, i32 %y) {
 ; CHECK-LABEL: @mul_select_eq_zero_sel_undef(
-; CHECK-NEXT:    [[C:%.*]] = icmp eq i32 [[X:%.*]], 0
-; CHECK-NEXT:    [[M:%.*]] = mul i32 [[X]], [[Y:%.*]]
-; CHECK-NEXT:    [[R:%.*]] = select i1 [[C]], i32 undef, i32 [[M]]
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-NEXT:    [[Y_FR:%.*]] = freeze i32 [[Y:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul i32 [[Y_FR]], [[X:%.*]]
+; CHECK-NEXT:    ret i32 [[M]]
 ;
   %c = icmp eq i32 %x, 0
   %m = mul i32 %x, %y
   %r = select i1 %c, i32 undef, i32 %m
   ret i32 %r
 }
 
+; Check that the transformation is applied disregard to a number
+; of expression's users.
 define i32 @mul_select_eq_zero_multiple_users(i32 %x, i32 %y) {
 ; CHECK-LABEL: @mul_select_eq_zero_multiple_users(
-; CHECK-NEXT:    [[M:%.*]] = mul i32 [[X:%.*]], [[Y:%.*]]
+; CHECK-NEXT:    [[Y_FR:%.*]] = freeze i32 [[Y:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul i32 [[Y_FR]], [[X:%.*]]
 ; CHECK-NEXT:    call void @use_i32(i32 [[M]])
-; CHECK-NEXT:    [[C:%.*]] = icmp eq i32 [[X]], 0
-; CHECK-NEXT:    [[R:%.*]] = select i1 [[C]], i32 0, i32 [[M]]
 ; CHECK-NEXT:    call void @use_i32(i32 [[M]])
-; CHECK-NEXT:    call void @use_i32(i32 [[R]])
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-NEXT:    call void @use_i32(i32 [[M]])
+; CHECK-NEXT:    ret i32 [[M]]
 ;
   %m = mul i32 %x, %y
   call void @use_i32(i32 %m)
   %c = icmp eq i32 %x, 0
   %r = select i1 %c, i32 0, i32 %m
   call void @use_i32(i32 %m)
   call void @use_i32(i32 %r)
   ret i32 %r
 }
 
+; Negative test: select's condition is unrelated to multiplied values,
+; so the transformation should not be applied.
 define i32 @mul_select_eq_zero_unrelated_condition(i32 %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @mul_select_eq_zero_unrelated_condition(
 ; CHECK-NEXT:    [[C:%.*]] = icmp eq i32 [[Z:%.*]], 0
 ; CHECK-NEXT:    [[M:%.*]] = mul i32 [[X:%.*]], [[Y:%.*]]
 ; CHECK-NEXT:    [[R:%.*]] = select i1 [[C]], i32 0, i32 [[M]]
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   %c = icmp eq i32 %z, 0
   %m = mul i32 %x, %y
   %r = select i1 %c, i32 0, i32 %m
   ret i32 %r
 }
 
+; select (<k x elt> x == 0), <k x elt> 0, <k x elt> x * y --> freeze(y) * x
 define <4 x i32> @mul_select_eq_zero_vector(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: @mul_select_eq_zero_vector(
-; CHECK-NEXT:    [[C:%.*]] = icmp eq <4 x i32> [[X:%.*]], zeroinitializer
-; CHECK-NEXT:    [[M:%.*]] = mul <4 x i32> [[X]], [[Y:%.*]]
-; CHECK-NEXT:    [[R:%.*]] = select <4 x i1> [[C]], <4 x i32> zeroinitializer, <4 x i32> [[M]]
-; CHECK-NEXT:    ret <4 x i32> [[R]]
+; CHECK-NEXT:    [[Y_FR:%.*]] = freeze <4 x i32> [[Y:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul <4 x i32> [[Y_FR]], [[X:%.*]]
+; CHECK-NEXT:    ret <4 x i32> [[M]]
 ;
   %c = icmp eq <4 x i32> %x, zeroinitializer
   %m = mul <4 x i32> %x, %y
   %r = select <4 x i1> %c, <4 x i32> zeroinitializer, <4 x i32> %m
   ret <4 x i32> %r
 }
 
+; Check that a select is folded into multiplication if condition's operand
+; is a vector consisting of zeros and undefs.
+; select (<k x elt> x == {0, undef, ...}), <k x elt> 0, <k x elt> x * y --> freeze(y) * x
 define <2 x i32> @mul_select_eq_undef_vector(<2 x i32> %x, <2 x i32> %y) {
 ; CHECK-LABEL: @mul_select_eq_undef_vector(
-; CHECK-NEXT:    [[C:%.*]] = icmp eq <2 x i32> [[X:%.*]], <i32 0, i32 undef>
-; CHECK-NEXT:    [[M:%.*]] = mul <2 x i32> [[X]], [[Y:%.*]]
-; CHECK-NEXT:    [[R:%.*]] = select <2 x i1> [[C]], <2 x i32> <i32 0, i32 42>, <2 x i32> [[M]]
-; CHECK-NEXT:    ret <2 x i32> [[R]]
+; CHECK-NEXT:    [[Y_FR:%.*]] = freeze <2 x i32> [[Y:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul <2 x i32> [[Y_FR]], [[X:%.*]]
+; CHECK-NEXT:    ret <2 x i32> [[M]]
 ;
   %c = icmp eq <2 x i32> %x, <i32 0, i32 undef>
   %m = mul <2 x i32> %x, %y
   %r = select <2 x i1> %c, <2 x i32> <i32 0, i32 42>, <2 x i32> %m
   ret <2 x i32> %r
 }
 
+; Check that a select is folded into multiplication if other select's operand
+; is a vector consisting of zeros and undefs.
+; select (<k x elt> x == 0), <k x elt> {0, undef, ...}, <k x elt> x * y --> freeze(y) * x
 define <2 x i32> @mul_select_eq_zero_sel_undef_vector(<2 x i32> %x, <2 x i32> %y) {
 ; CHECK-LABEL: @mul_select_eq_zero_sel_undef_vector(
-; CHECK-NEXT:    [[C:%.*]] = icmp eq <2 x i32> [[X:%.*]], zeroinitializer
-; CHECK-NEXT:    [[M:%.*]] = mul <2 x i32> [[X]], [[Y:%.*]]
-; CHECK-NEXT:    [[R:%.*]] = select <2 x i1> [[C]], <2 x i32> <i32 0, i32 undef>, <2 x i32> [[M]]
-; CHECK-NEXT:    ret <2 x i32> [[R]]
+; CHECK-NEXT:    [[Y_FR:%.*]] = freeze <2 x i32> [[Y:%.*]]
+; CHECK-NEXT:    [[M:%.*]] = mul <2 x i32> [[Y_FR]], [[X:%.*]]
+; CHECK-NEXT:    ret <2 x i32> [[M]]
 ;
   %c = icmp eq <2 x i32> %x, zeroinitializer
   %m = mul <2 x i32> %x, %y
   %r = select <2 x i1> %c, <2 x i32> <i32 0, i32 undef>, <2 x i32> %m
   ret <2 x i32> %r
 }
 
+; Negative test: select should not be folded into mul because
+; condition's operand and select's operand do not merge into zero vector.
 define <2 x i32> @mul_select_eq_undef_vector_not_merging_to_zero(<2 x i32> %x, <2 x i32> %y) {
 ; CHECK-LABEL: @mul_select_eq_undef_vector_not_merging_to_zero(
 ; CHECK-NEXT:    [[C:%.*]] = icmp eq <2 x i32> [[X:%.*]], <i32 0, i32 undef>
 ; CHECK-NEXT:    [[M:%.*]] = mul <2 x i32> [[X]], [[Y:%.*]]
 ; CHECK-NEXT:    [[R:%.*]] = select <2 x i1> [[C]], <2 x i32> <i32 1, i32 0>, <2 x i32> [[M]]
 ; CHECK-NEXT:    ret <2 x i32> [[R]]
 ;
   %c = icmp eq <2 x i32> %x, <i32 0, i32 undef>
   %m = mul <2 x i32> %x, %y
   %r = select <2 x i1> %c, <2 x i32> <i32 1, i32 0>, <2 x i32> %m
   ret <2 x i32> %r
 }
 
 declare void @use(i1)
 declare void @use_i8(i8)
 declare void @use_i32(i32)
 declare i32 @llvm.cttz.i32(i32, i1 immarg)