[Constants] Extend support for scalable-vector splats

[frasercrmck]

Summary

This patch extends the various "isXXX" functions of the Constant class
to include scalable-vector splats.

In several "isXXX" functions, code that was separately inspecting
ConstantVector and ConstantDataVector was unified to use
getSplatValue, which already includes support for said splats.

In the varous "isNotXXX" functions, code was added to check whether the
scalar splat value -- if any -- satisfies the predicate.

An extra fix for isNotMinSignedValue was included, as it previously
crashed when passed a scalable-vector type because it unconditionally
cast to FixedVectorType

These changes address numerous missed optimizations, a compiler crash
mentioned above and -- perhaps most egregiously -- an infinite loop in
InstCombine due to the compiler breaking canonical form when it failed
to pick up on a splat in a select instruction.

Test cases have been added to cover as many of these functions as
possible, though existing coverage is slim; it doesn't appear that there
are any in-tree uses of Constant::isNegativeZeroValue, for example.

[frasercrmck]

llvm/test/Transforms/InstCombine/sub.ll

862

This isn't combined to xor as above because the pattern in visitSub uses m_ImmConstant which matches Constant but explicitly not ConstantExpr. Still, it's a call to isNotMinSignedValue before it, checking that we don't mis-optimize.

[frasercrmck]

Ping.

[RKSimon]

llvm/lib/IR/Constants.cpp

204

I'm not really convinced it'd make a difference but we're not consistent at checking for splats before/after the FixedVectorType only tests - do you think we should be?

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

931

Could this be easily handled in a separate patch?

llvm/test/Transforms/InstCombine/sub.ll

862

Maybe mention that in a comment just above?

[frasercrmck]

llvm/lib/IR/Constants.cpp

204

Fair point. The lack of consistency I've introduced stemmed from the different styles going on in these functions in order to minimize changes. The functions that early-exited when !Fixed I added in the splat check before, else I added it after.

Checking for splats is adding another loop over the fixed elements so I'm leaning towards making splats come consistently afterwards, and so removing the !Fixed early-exit checks. That wouldn't penalize the fixed-length loops since they can early-exit if they detect the opposite of what they're checking for.

I guess in theory that would allow us to then only check for splats for scalable vectors, though that may look confusing and is adding extra complexity to the reader at (presumably) no benefit to compilation times.

Sound reasonable?

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

931

I'm concerned I won't be able to find a test case for this/that, depending on which goes in first, but yes logically they should be distinct changes so I'll give it a go.

llvm/test/Transforms/InstCombine/sub.ll

862

Aye good idea. I may make a note to look into that later. m_ImmConstant is surprisingly rare so we could probably go through them all and see if there's not a reason to "upgrade" them.

[frasercrmck]

• rebase
• add test fixme comment
• normalize check order to fixed-length then (scalable) splat
• consistently remove "this->" from these methods

[frasercrmck]

llvm/lib/IR/Constants.cpp

204

I've added that now to show what I'm meaning.

[frasercrmck]

• slice out change to dyn_castNegVal
• update test with FIXME accordingly

[frasercrmck]

llvm/test/Transforms/InstCombine/sub.ll

842

@RKSimon by removing the change to dyn_castNegVal I wasn't able to find a good test case for isNotMinSignedValue so I just left this as a FIXME here. This is now addressed in D103801 (which can go in after - I similarly wasn't able to find a test case for it without this patch).

[RKSimon]

LGTM - cheers

llvm/lib/IR/Constants.cpp

 
 using namespace llvm;
 using namespace PatternMatch;
 
 //===----------------------------------------------------------------------===//
 //                              Constant Class
 //===----------------------------------------------------------------------===//
 
 bool Constant::isNegativeZeroValue() const {
   // Floating point values have an explicit -0.0 value.
   if (const ConstantFP *CFP = dyn_cast<ConstantFP>(this))
     return CFP->isZero() && CFP->isNegative();
 
   // Equivalent for a vector of -0.0's.
-  if (const ConstantDataVector *CV = dyn_cast<ConstantDataVector>(this))
-    if (CV->getElementType()->isFloatingPointTy() && CV->isSplat())
-      if (CV->getElementAsAPFloat(0).isNegZero())
-        return true;
-
-  if (const ConstantVector *CV = dyn_cast<ConstantVector>(this))
-    if (ConstantFP *SplatCFP = dyn_cast_or_null<ConstantFP>(CV->getSplatValue()))
-      if (SplatCFP && SplatCFP->isZero() && SplatCFP->isNegative())
-        return true;
+  if (getType()->isVectorTy())
+    if (const auto *SplatCFP = dyn_cast_or_null<ConstantFP>(getSplatValue()))
+      return SplatCFP->isNegativeZeroValue();
 
   // We've already handled true FP case; any other FP vectors can't represent -0.0.
   if (getType()->isFPOrFPVectorTy())
     return false;
 
   // Otherwise, just use +0.0.
   return isNullValue();
 }
 
 // Return true iff this constant is positive zero (floating point), negative
 // zero (floating point), or a null value.
 bool Constant::isZeroValue() const {
   // Floating point values have an explicit -0.0 value.
   if (const ConstantFP *CFP = dyn_cast<ConstantFP>(this))
     return CFP->isZero();
 
-  // Equivalent for a vector of -0.0's.
-  if (const ConstantDataVector *CV = dyn_cast<ConstantDataVector>(this))
-    if (CV->getElementType()->isFloatingPointTy() && CV->isSplat())
-      if (CV->getElementAsAPFloat(0).isZero())
-        return true;
-
-  if (const ConstantVector *CV = dyn_cast<ConstantVector>(this))
-    if (ConstantFP *SplatCFP = dyn_cast_or_null<ConstantFP>(CV->getSplatValue()))
-      if (SplatCFP && SplatCFP->isZero())
-        return true;
+  // Check for constant splat vectors of 1 values.
+  if (getType()->isVectorTy())
+    if (const auto *SplatCFP = dyn_cast_or_null<ConstantFP>(getSplatValue()))
+      return SplatCFP->isZero();
 
   // Otherwise, just use +0.0.
   return isNullValue();
 }
 
 bool Constant::isNullValue() const {
   // 0 is null.
   if (const ConstantInt *CI = dyn_cast<ConstantInt>(this))
     return CI->isZero();
 
   // +0.0 is null.
   if (const ConstantFP *CFP = dyn_cast<ConstantFP>(this))
     // ppc_fp128 determine isZero using high order double only
     // Should check the bitwise value to make sure all bits are zero.
     return CFP->isExactlyValue(+0.0);
 
   // constant zero is zero for aggregates, cpnull is null for pointers, none for
   // tokens.
   return isa<ConstantAggregateZero>(this) || isa<ConstantPointerNull>(this) ||
          isa<ConstantTokenNone>(this);
 }
 
 bool Constant::isAllOnesValue() const {
   // Check for -1 integers
   if (const ConstantInt *CI = dyn_cast<ConstantInt>(this))
     return CI->isMinusOne();
 
   // Check for FP which are bitcasted from -1 integers
   if (const ConstantFP *CFP = dyn_cast<ConstantFP>(this))
     return CFP->getValueAPF().bitcastToAPInt().isAllOnesValue();
 
-  // Check for constant vectors which are splats of -1 values.
-  if (const ConstantVector *CV = dyn_cast<ConstantVector>(this))
-    if (Constant *Splat = CV->getSplatValue())
-      return Splat->isAllOnesValue();
-
-  // Check for constant vectors which are splats of -1 values.
-  if (const ConstantDataVector *CV = dyn_cast<ConstantDataVector>(this)) {
-    if (CV->isSplat()) {
-      if (CV->getElementType()->isFloatingPointTy())
-        return CV->getElementAsAPFloat(0).bitcastToAPInt().isAllOnesValue();
-      return CV->getElementAsAPInt(0).isAllOnesValue();
-    }
-  }
+  // Check for constant splat vectors of 1 values.
+  if (getType()->isVectorTy())
+    if (const auto *SplatVal = getSplatValue())
+      return SplatVal->isAllOnesValue();
 
   return false;
 }
 
 bool Constant::isOneValue() const {
   // Check for 1 integers
   if (const ConstantInt *CI = dyn_cast<ConstantInt>(this))
     return CI->isOne();
 
   // Check for FP which are bitcasted from 1 integers
   if (const ConstantFP *CFP = dyn_cast<ConstantFP>(this))
     return CFP->getValueAPF().bitcastToAPInt().isOneValue();
 
-  // Check for constant vectors which are splats of 1 values.
-  if (const ConstantVector *CV = dyn_cast<ConstantVector>(this))
-    if (Constant *Splat = CV->getSplatValue())
-      return Splat->isOneValue();
-
-  // Check for constant vectors which are splats of 1 values.
-  if (const ConstantDataVector *CV = dyn_cast<ConstantDataVector>(this)) {
-    if (CV->isSplat()) {
-      if (CV->getElementType()->isFloatingPointTy())
-        return CV->getElementAsAPFloat(0).bitcastToAPInt().isOneValue();
-      return CV->getElementAsAPInt(0).isOneValue();
-    }
-  }
+  // Check for constant splat vectors of 1 values.
+  if (getType()->isVectorTy())
+    if (const auto *SplatVal = getSplatValue())
+      return SplatVal->isOneValue();
 
   return false;
 }
 
 bool Constant::isNotOneValue() const {
   // Check for 1 integers
   if (const ConstantInt *CI = dyn_cast<ConstantInt>(this))
     return !CI->isOneValue();
 
   // Check for FP which are bitcasted from 1 integers
   if (const ConstantFP *CFP = dyn_cast<ConstantFP>(this))
     return !CFP->getValueAPF().bitcastToAPInt().isOneValue();
 
   // Check that vectors don't contain 1
-  if (auto *VTy = dyn_cast<VectorType>(this->getType())) {
-    unsigned NumElts = cast<FixedVectorType>(VTy)->getNumElements();
-    for (unsigned i = 0; i != NumElts; ++i) {
-      Constant *Elt = this->getAggregateElement(i);
+  if (auto *VTy = dyn_cast<FixedVectorType>(getType())) {
+    for (unsigned I = 0, E = VTy->getNumElements(); I != E; ++I) {
+      Constant *Elt = getAggregateElement(I);
       if (!Elt || !Elt->isNotOneValue())
         return false;
     }
     return true;
   }
 
+  // Check for splats that don't contain 1
+  if (getType()->isVectorTy())
+    if (const auto *SplatVal = getSplatValue())
+      return SplatVal->isNotOneValue();
+
   // It *may* contain 1, we can't tell.
   return false;
 }
 
 bool Constant::isMinSignedValue() const {
   // Check for INT_MIN integers
   if (const ConstantInt *CI = dyn_cast<ConstantInt>(this))
     return CI->isMinValue(/*isSigned=*/true);
 
   // Check for FP which are bitcasted from INT_MIN integers
   if (const ConstantFP *CFP = dyn_cast<ConstantFP>(this))
     return CFP->getValueAPF().bitcastToAPInt().isMinSignedValue();
 
-  // Check for constant vectors which are splats of INT_MIN values.
-  if (const ConstantVector *CV = dyn_cast<ConstantVector>(this))
-    if (Constant *Splat = CV->getSplatValue())
-      return Splat->isMinSignedValue();
-
-  // Check for constant vectors which are splats of INT_MIN values.
-  if (const ConstantDataVector *CV = dyn_cast<ConstantDataVector>(this)) {
-    if (CV->isSplat()) {
-      if (CV->getElementType()->isFloatingPointTy())
-        return CV->getElementAsAPFloat(0).bitcastToAPInt().isMinSignedValue();
-      return CV->getElementAsAPInt(0).isMinSignedValue();
-    }
-  }
+  // Check for splats of INT_MIN values.
+  if (getType()->isVectorTy())
+    if (const auto *SplatVal = getSplatValue())
+      return SplatVal->isMinSignedValue();
 
   return false;
 }
 
 bool Constant::isNotMinSignedValue() const {
   // Check for INT_MIN integers
   if (const ConstantInt *CI = dyn_cast<ConstantInt>(this))
     return !CI->isMinValue(/*isSigned=*/true);
 
   // Check for FP which are bitcasted from INT_MIN integers
   if (const ConstantFP *CFP = dyn_cast<ConstantFP>(this))
     return !CFP->getValueAPF().bitcastToAPInt().isMinSignedValue();
 
   // Check that vectors don't contain INT_MIN
-  if (auto *VTy = dyn_cast<VectorType>(this->getType())) {
-    unsigned NumElts = cast<FixedVectorType>(VTy)->getNumElements();
-    for (unsigned i = 0; i != NumElts; ++i) {
-      Constant *Elt = this->getAggregateElement(i);
+  if (auto *VTy = dyn_cast<FixedVectorType>(getType())) {
+    for (unsigned I = 0, E = VTy->getNumElements(); I != E; ++I) {
+      Constant *Elt = getAggregateElement(I);
       if (!Elt || !Elt->isNotMinSignedValue())
         return false;
     }
     return true;
   }
 
+  // Check for splats that aren't INT_MIN
+  if (getType()->isVectorTy())
+    if (const auto *SplatVal = getSplatValue())
+      return SplatVal->isNotMinSignedValue();
+
   // It *may* contain INT_MIN, we can't tell.
   return false;
 }
 
 bool Constant::isFiniteNonZeroFP() const {
   if (auto *CFP = dyn_cast<ConstantFP>(this))
     return CFP->getValueAPF().isFiniteNonZero();
-  auto *VTy = dyn_cast<FixedVectorType>(getType());
-  if (!VTy)
-    return false;
-  for (unsigned i = 0, e = VTy->getNumElements(); i != e; ++i) {
-    auto *CFP = dyn_cast_or_null<ConstantFP>(this->getAggregateElement(i));
+
+  if (auto *VTy = dyn_cast<FixedVectorType>(getType())) {
+    for (unsigned I = 0, E = VTy->getNumElements(); I != E; ++I) {
+      auto *CFP = dyn_cast_or_null<ConstantFP>(getAggregateElement(I));
       if (!CFP || !CFP->getValueAPF().isFiniteNonZero())
         return false;
     }
     return true;
   }
 
+  if (getType()->isVectorTy())
+    if (const auto *SplatCFP = dyn_cast_or_null<ConstantFP>(getSplatValue()))
+      return SplatCFP->isFiniteNonZeroFP();
+
+  // It *may* contain finite non-zero, we can't tell.
+  return false;
+}
+
 bool Constant::isNormalFP() const {
   if (auto *CFP = dyn_cast<ConstantFP>(this))
     return CFP->getValueAPF().isNormal();
-  auto *VTy = dyn_cast<FixedVectorType>(getType());
-  if (!VTy)
-    return false;
-  for (unsigned i = 0, e = VTy->getNumElements(); i != e; ++i) {
-    auto *CFP = dyn_cast_or_null<ConstantFP>(this->getAggregateElement(i));
+
+  if (auto *VTy = dyn_cast<FixedVectorType>(getType())) {
+    for (unsigned I = 0, E = VTy->getNumElements(); I != E; ++I) {
+      auto *CFP = dyn_cast_or_null<ConstantFP>(getAggregateElement(I));
       if (!CFP || !CFP->getValueAPF().isNormal())
         return false;
     }
     return true;
   }
 
+  if (getType()->isVectorTy())
+    if (const auto *SplatCFP = dyn_cast_or_null<ConstantFP>(getSplatValue()))
+      return SplatCFP->isNormalFP();
+
+  // It *may* contain a normal fp value, we can't tell.
+  return false;
+}
+
 bool Constant::hasExactInverseFP() const {
   if (auto *CFP = dyn_cast<ConstantFP>(this))
     return CFP->getValueAPF().getExactInverse(nullptr);
-  auto *VTy = dyn_cast<FixedVectorType>(getType());
-  if (!VTy)
-    return false;
-  for (unsigned i = 0, e = VTy->getNumElements(); i != e; ++i) {
-    auto *CFP = dyn_cast_or_null<ConstantFP>(this->getAggregateElement(i));
+
+  if (auto *VTy = dyn_cast<FixedVectorType>(getType())) {
+    for (unsigned I = 0, E = VTy->getNumElements(); I != E; ++I) {
+      auto *CFP = dyn_cast_or_null<ConstantFP>(getAggregateElement(I));
       if (!CFP || !CFP->getValueAPF().getExactInverse(nullptr))
         return false;
     }
     return true;
   }
 
+  if (getType()->isVectorTy())
+    if (const auto *SplatCFP = dyn_cast_or_null<ConstantFP>(getSplatValue()))
+      return SplatCFP->hasExactInverseFP();
+
+  // It *may* have an exact inverse fp value, we can't tell.
+  return false;
+}
+
 bool Constant::isNaN() const {
   if (auto *CFP = dyn_cast<ConstantFP>(this))
     return CFP->isNaN();
-  auto *VTy = dyn_cast<FixedVectorType>(getType());
-  if (!VTy)
-    return false;
-  for (unsigned i = 0, e = VTy->getNumElements(); i != e; ++i) {
-    auto *CFP = dyn_cast_or_null<ConstantFP>(this->getAggregateElement(i));
+
+  if (auto *VTy = dyn_cast<FixedVectorType>(getType())) {
+    for (unsigned I = 0, E = VTy->getNumElements(); I != E; ++I) {
+      auto *CFP = dyn_cast_or_null<ConstantFP>(getAggregateElement(I));
       if (!CFP || !CFP->isNaN())
         return false;
     }
     return true;
   }
 
+  if (getType()->isVectorTy())
+    if (const auto *SplatCFP = dyn_cast_or_null<ConstantFP>(getSplatValue()))
+      return SplatCFP->isNaN();
+
+  // It *may* be NaN, we can't tell.
+  return false;
+}
+
 bool Constant::isElementWiseEqual(Value *Y) const {
   // Are they fully identical?
   if (this == Y)
     return true;
 
   // The input value must be a vector constant with the same type.
   auto *VTy = dyn_cast<VectorType>(getType());
   if (!isa<Constant>(Y) || !VTy || VTy != Y->getType())

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

for (unsigned i = 0, e = CV->getNumOperands(); i != e; ++i) {
         continue;
 
       if (!isa<ConstantInt>(Elt))
         return nullptr;
     }
     return ConstantExpr::getNeg(CV);
   }
 
+  // Negate integer vector splats.
+  if (auto *CV = dyn_cast<Constant>(V))
+    if (CV->getType()->isVectorTy() &&
+        CV->getType()->getScalarType()->isIntegerTy() && CV->getSplatValue())
+      return ConstantExpr::getNeg(CV);
+
   return nullptr;
 }
 
 static Value *foldOperationIntoSelectOperand(Instruction &I, Value *SO,
                                              InstCombiner::BuilderTy &Builder) {
   if (auto *Cast = dyn_cast<CastInst>(&I))
     return Builder.CreateCast(Cast->getOpcode(), SO, I.getType());
 

llvm/test/Transforms/InstCombine/bitcast.ll

 define i32 @All111(i32 %in) {
 ; CHECK-LABEL: @All111(
 ; CHECK-NEXT:    ret i32 0
 ;
   %out = and i32 %in, xor (i32 bitcast (<1 x float> bitcast (i32 -1 to <1 x float>) to i32), i32 -1)
   ret i32 %out
 }
 
+define <vscale x 1 x i32> @ScalableAll111(<vscale x 1 x i32> %in) {
+; CHECK-LABEL: @ScalableAll111(
+; CHECK-NEXT:    ret <vscale x 1 x i32> [[IN:%.*]]
+;
+  %out = and <vscale x 1 x i32> %in, bitcast (<vscale x 2 x i16> shufflevector (<vscale x 2 x i16> insertelement (<vscale x 2 x i16> undef, i16 -1, i32 0), <vscale x 2 x i16> undef, <vscale x 2 x i32> zeroinitializer) to <vscale x 1 x i32>)
+  ret <vscale x 1 x i32> %out
+}
+
 define <2 x i16> @BitcastInsert(i32 %a) {
 ; CHECK-LABEL: @BitcastInsert(
 ; CHECK-NEXT:    [[R:%.*]] = bitcast i32 [[A:%.*]] to <2 x i16>
 ; CHECK-NEXT:    ret <2 x i16> [[R]]
 ;
   %v = insertelement <1 x i32> undef, i32 %a, i32 0
   %r = bitcast <1 x i32> %v to <2 x i16>
   ret <2 x i16> %r

llvm/test/Transforms/InstCombine/div.ll

 
 define <2 x i8> @sdiv_by_int_min_vec_splat_undef(<2 x i8> %x) {
 ; CHECK-LABEL: @sdiv_by_int_min_vec_splat_undef(
 ; CHECK-NEXT:    ret <2 x i8> poison
 ;
   %d = sdiv <2 x i8> %x, <i8 -128, i8 undef>
   ret <2 x i8> %d
 }
+
+define <2 x i8> @sdiv_by_negconst_v2i8(<2 x i8> %x) {
+; CHECK-LABEL: @sdiv_by_negconst_v2i8(
+; CHECK-NEXT:    [[DIV_NEG:%.*]] = sdiv <2 x i8> [[X:%.*]], <i8 108, i8 108>
+; CHECK-NEXT:    ret <2 x i8> [[DIV_NEG]]
+;
+  %div = sdiv <2 x i8> %x, <i8 -108, i8 -108>
+  %sub = sub <2 x i8> zeroinitializer, %div
+  ret <2 x i8> %sub
+}
+
+define <vscale x 2 x i8> @sdiv_by_negconst_nxv2i8(<vscale x 2 x i8> %x) {
+; CHECK-LABEL: @sdiv_by_negconst_nxv2i8(
+; CHECK-NEXT:    [[DIV_NEG:%.*]] = sdiv <vscale x 2 x i8> [[X:%.*]], shufflevector (<vscale x 2 x i8> insertelement (<vscale x 2 x i8> undef, i8 108, i32 0), <vscale x 2 x i8> undef, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    ret <vscale x 2 x i8> [[DIV_NEG]]
+;
+  %div = sdiv <vscale x 2 x i8> %x, shufflevector (<vscale x 2 x i8> insertelement (<vscale x 2 x i8> undef, i8 -108, i32 0), <vscale x 2 x i8> undef, <vscale x 2 x i32> zeroinitializer)
+  %sub = sub <vscale x 2 x i8> zeroinitializer, %div
+  ret <vscale x 2 x i8> %sub
+}
+
+define <2 x i8> @sdiv_by_minSigned_v2i8(<2 x i8> %x) {
+; CHECK-LABEL: @sdiv_by_minSigned_v2i8(
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp eq <2 x i8> [[X:%.*]], <i8 -128, i8 -128>
+; CHECK-NEXT:    [[DIV_NEG:%.*]] = sext <2 x i1> [[TMP1]] to <2 x i8>
+; CHECK-NEXT:    ret <2 x i8> [[DIV_NEG]]
+;
+  %div = sdiv <2 x i8> %x, <i8 -128, i8 -128>
+  %sub = sub <2 x i8> zeroinitializer, %div
+  ret <2 x i8> %sub
+}
+
+define <vscale x 2 x i8> @sdiv_by_minSigned_nxv2i8(<vscale x 2 x i8> %x) {
+; CHECK-LABEL: @sdiv_by_minSigned_nxv2i8(
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp eq <vscale x 2 x i8> [[X:%.*]], shufflevector (<vscale x 2 x i8> insertelement (<vscale x 2 x i8> undef, i8 -128, i32 0), <vscale x 2 x i8> undef, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    [[DIV_NEG:%.*]] = sext <vscale x 2 x i1> [[TMP1]] to <vscale x 2 x i8>
+; CHECK-NEXT:    ret <vscale x 2 x i8> [[DIV_NEG]]
+;
+  %div = sdiv <vscale x 2 x i8> %x, shufflevector (<vscale x 2 x i8> insertelement (<vscale x 2 x i8> undef, i8 -128, i32 0), <vscale x 2 x i8> undef, <vscale x 2 x i32> zeroinitializer)
+  %sub = sub <vscale x 2 x i8> zeroinitializer, %div
+  ret <vscale x 2 x i8> %sub
+}

llvm/test/Transforms/InstCombine/fdiv.ll

 ; CHECK-LABEL: @exact_inverse_splat(
 ; CHECK-NEXT:    [[DIV:%.*]] = fmul <2 x float> [[X:%.*]], <float 2.500000e-01, float 2.500000e-01>
 ; CHECK-NEXT:    ret <2 x float> [[DIV]]
 ;
   %div = fdiv <2 x float> %x, <float 4.0, float 4.0>
   ret <2 x float> %div
 }
 
+define <vscale x 2 x float> @exact_inverse_scalable_splat(<vscale x 2 x float> %x) {
+; CHECK-LABEL: @exact_inverse_scalable_splat(
+; CHECK-NEXT:    [[DIV:%.*]] = fmul <vscale x 2 x float> [[X:%.*]], shufflevector (<vscale x 2 x float> insertelement (<vscale x 2 x float> undef, float 2.500000e-01, i32 0), <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    ret <vscale x 2 x float> [[DIV]]
+;
+  %div = fdiv <vscale x 2 x float> %x, shufflevector (<vscale x 2 x float> insertelement (<vscale x 2 x float> undef, float 4.0, i32 0), <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer)
+  ret <vscale x 2 x float> %div
+}
+
 ; Fast math allows us to replace this fdiv.
 
 define <2 x float> @not_exact_but_allow_recip_splat(<2 x float> %x) {
 ; CHECK-LABEL: @not_exact_but_allow_recip_splat(
 ; CHECK-NEXT:    [[DIV:%.*]] = fmul arcp <2 x float> [[X:%.*]], <float 0x3FD5555560000000, float 0x3FD5555560000000>
 ; CHECK-NEXT:    ret <2 x float> [[DIV]]
 ;
   %div = fdiv arcp <2 x float> %x, <float 3.0, float 3.0>

llvm/test/Transforms/InstCombine/fmul.ll

 ; CHECK-NEXT:    [[T3:%.*]] = fadd reassoc float [[TMP1]], 6.000000e+00
 ; CHECK-NEXT:    ret float [[T3]]
 ;
   %t2 = fadd float %x, 2.0
   %t3 = fmul reassoc float %t2, 3.0
   ret float %t3
 }
 
+define <2 x float> @fmul_fadd_distribute_vec(<2 x float> %x) {
+; CHECK-LABEL: @fmul_fadd_distribute_vec(
+; CHECK-NEXT:    [[TMP1:%.*]] = fmul reassoc <2 x float> [[X:%.*]], <float 6.000000e+03, float 6.000000e+03>
+; CHECK-NEXT:    [[T3:%.*]] = fadd reassoc <2 x float> [[TMP1]], <float 1.200000e+07, float 1.200000e+07>
+; CHECK-NEXT:    ret <2 x float> [[T3]]
+;
+  %t1 = fadd <2 x float> <float 2.0e+3, float 2.0e+3>, %x
+  %t3 = fmul reassoc <2 x float> %t1, <float 6.0e+3, float 6.0e+3>
+  ret <2 x float> %t3
+}
+
+define <vscale x 2 x float> @fmul_fadd_distribute_scalablevec(<vscale x 2 x float> %x) {
+; CHECK-LABEL: @fmul_fadd_distribute_scalablevec(
+; CHECK-NEXT:    [[TMP1:%.*]] = fmul reassoc <vscale x 2 x float> [[X:%.*]], shufflevector (<vscale x 2 x float> insertelement (<vscale x 2 x float> undef, float 6.000000e+03, i32 0), <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    [[T3:%.*]] = fadd reassoc <vscale x 2 x float> [[TMP1]], shufflevector (<vscale x 2 x float> insertelement (<vscale x 2 x float> undef, float 1.200000e+07, i32 0), <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    ret <vscale x 2 x float> [[T3]]
+;
+  %t1 = fadd <vscale x 2 x float> shufflevector (<vscale x 2 x float> insertelement (<vscale x 2 x float> undef, float 2.0e+3, i32 0), <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer), %x
+  %t3 = fmul reassoc <vscale x 2 x float> %t1, shufflevector (<vscale x 2 x float> insertelement (<vscale x 2 x float> undef, float 6.0e+3, i32 0), <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer)
+
+
+  ret <vscale x 2 x float> %t3
+}
+
 ; (X - C1) * C2 --> (X * C2) - C1*C2
 
 define float @fmul_fsub_distribute1(float %x) {
 ; CHECK-LABEL: @fmul_fsub_distribute1(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fmul reassoc float [[X:%.*]], 3.000000e+00
 ; CHECK-NEXT:    [[T3:%.*]] = fadd reassoc float [[TMP1]], -6.000000e+00
 ; CHECK-NEXT:    ret float [[T3]]
 ;
;
   %mul = fmul fast double %sqr, %sel
   ret double %mul
 }
 
 ; fastmath => z * splat(0) = splat(0), even for scalable vectors
 define <vscale x 2 x float> @mul_scalable_splat_zero(<vscale x 2 x float> %z) {
 ; CHECK-LABEL: @mul_scalable_splat_zero(
 ; CHECK-NEXT:    ret <vscale x 2 x float> zeroinitializer
+;
   %shuf = shufflevector <vscale x 2 x float> insertelement (<vscale x 2 x float> undef, float 0.0, i32 0), <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
   %t3 = fmul fast <vscale x 2 x float> %shuf, %z
   ret <vscale x 2 x float> %t3
 }

llvm/test/Transforms/InstCombine/icmp-vec.ll

 ; CHECK-NEXT:    [[SPLATX:%.*]] = shufflevector <4 x i8> [[X:%.*]], <4 x i8> undef, <4 x i32> <i32 2, i32 2, i32 2, i32 2>
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp sgt <4 x i8> [[SPLATX]], <i8 43, i8 42, i8 42, i8 42>
 ; CHECK-NEXT:    ret <4 x i1> [[CMP]]
 ;
   %splatx = shufflevector <4 x i8> %x, <4 x i8> undef, <4 x i32> <i32 2, i32 2, i32 2, i32 2>
   %cmp = icmp sgt <4 x i8> %splatx, <i8 43, i8 42, i8 42, i8 42>
   ret <4 x i1> %cmp
 }
+
+; Check that we don't absorb the compare into the select, which is in the
+; canonical form of logical or.
+define <2 x i1> @icmp_logical_or_vec(<2 x i64> %x, <2 x i64> %y, <2 x i1> %falseval) {
+; CHECK-LABEL: @icmp_logical_or_vec(
+; CHECK-NEXT:    [[CMP_NE:%.*]] = icmp ne <2 x i64> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[SEL:%.*]] = select <2 x i1> [[CMP_NE]], <2 x i1> <i1 true, i1 true>, <2 x i1> [[FALSEVAL:%.*]]
+; CHECK-NEXT:    ret <2 x i1> [[SEL]]
+;
+  %cmp.ne = icmp ne <2 x i64> %x, zeroinitializer
+  %sel = select <2 x i1> %cmp.ne, <2 x i1> shufflevector (<2 x i1> insertelement (<2 x i1> undef, i1 true, i32 0), <2 x i1> undef, <2 x i32> zeroinitializer), <2 x i1> %falseval
+  ret <2 x i1> %sel
+}
+
+; The above, but for scalable vectors. Absorbing the compare into the select
+; and breaking the canonical form led to an infinite loop.
+define <vscale x 2 x i1> @icmp_logical_or_scalablevec(<vscale x 2 x i64> %x, <vscale x 2 x i64> %y, <vscale x 2 x i1> %falseval) {
+; CHECK-LABEL: @icmp_logical_or_scalablevec(
+; CHECK-NEXT:    [[CMP_NE:%.*]] = icmp ne <vscale x 2 x i64> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[SEL:%.*]] = select <vscale x 2 x i1> [[CMP_NE]], <vscale x 2 x i1> shufflevector (<vscale x 2 x i1> insertelement (<vscale x 2 x i1> undef, i1 true, i32 0), <vscale x 2 x i1> undef, <vscale x 2 x i32> zeroinitializer), <vscale x 2 x i1> [[FALSEVAL:%.*]]
+; CHECK-NEXT:    ret <vscale x 2 x i1> [[SEL]]
+;
+  %cmp.ne = icmp ne <vscale x 2 x i64> %x, zeroinitializer
+  %sel = select <vscale x 2 x i1> %cmp.ne, <vscale x 2 x i1> shufflevector (<vscale x 2 x i1> insertelement (<vscale x 2 x i1> undef, i1 true, i32 0), <vscale x 2 x i1> undef, <vscale x 2 x i32> zeroinitializer), <vscale x 2 x i1> %falseval
+  ret <vscale x 2 x i1> %sel
+}

llvm/test/Transforms/InstCombine/sub.ll

 ; CHECK-LABEL: @test44(
 ; CHECK-NEXT:    [[SUB:%.*]] = add nsw i32 [[X:%.*]], -32768
 ; CHECK-NEXT:    ret i32 [[SUB]]
 ;
   %sub = sub nsw i32 %x, 32768
   ret i32 %sub
 }
 
+define <2 x i32> @test44vec(<2 x i32> %x) {
+; CHECK-LABEL: @test44vec(
+; CHECK-NEXT:    [[SUB:%.*]] = add nsw <2 x i32> [[X:%.*]], <i32 -32768, i32 -32768>
+; CHECK-NEXT:    ret <2 x i32> [[SUB]]
+;
+  %sub = sub nsw <2 x i32> %x, <i32 32768, i32 32768>
+  ret <2 x i32> %sub
+}
+
+define <vscale x 2 x i32> @test44scalablevec(<vscale x 2 x i32> %x) {
+; CHECK-LABEL: @test44scalablevec(
+; CHECK-NEXT:    [[SUB:%.*]] = add nsw <vscale x 2 x i32> [[X:%.*]], shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> undef, i32 -32768, i32 0), <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    ret <vscale x 2 x i32> [[SUB]]
+;
+  %sub = sub nsw <vscale x 2 x i32> %x, shufflevector (<vscale x 2 x i32> insertelement (<vscale x 2 x i32> undef, i32 32768, i32 0), <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer)
+  ret <vscale x 2 x i32> %sub
+}
+
+define <2 x i16> @test44vecminval(<2 x i16> %x) {
+; CHECK-LABEL: @test44vecminval(
+; CHECK-NEXT:    [[SUB:%.*]] = xor <2 x i16> [[X:%.*]], <i16 -32768, i16 -32768>
+; CHECK-NEXT:    ret <2 x i16> [[SUB]]
+;
+  %sub = sub nsw <2 x i16> %x, <i16 -32768, i16 -32768>
+  ret <2 x i16> %sub
+}
+
+; FIXME: This isn't combined to xor as above because the pattern in visitSub
+; uses m_ImmConstant which matches Constant but (explicitly) not ConstantExpr.
+define <vscale x 2 x i16> @test44scalablevecminval(<vscale x 2 x i16> %x) {
+; CHECK-LABEL: @test44scalablevecminval(
+; CHECK-NEXT:    [[SUB:%.*]] = add <vscale x 2 x i16> [[X:%.*]], shufflevector (<vscale x 2 x i16> insertelement (<vscale x 2 x i16> undef, i16 -32768, i32 0), <vscale x 2 x i16> undef, <vscale x 2 x i32> zeroinitializer)
+; CHECK-NEXT:    ret <vscale x 2 x i16> [[SUB]]
+;
+  %sub = sub nsw <vscale x 2 x i16> %x, shufflevector (<vscale x 2 x i16> insertelement (<vscale x 2 x i16> undef, i16 -32768, i32 0), <vscale x 2 x i16> undef, <vscale x 2 x i32> zeroinitializer)
+  ret <vscale x 2 x i16> %sub
+}
+
 define i32 @test45(i32 %x, i32 %y) {
 ; CHECK-LABEL: @test45(
 ; CHECK-NEXT:    [[SUB:%.*]] = and i32 [[X:%.*]], [[Y:%.*]]
 ; CHECK-NEXT:    ret i32 [[SUB]]
 ;
   %or = or i32 %x, %y
   %xor = xor i32 %x, %y
   %sub = sub i32 %or, %xor