Stop instcombining propagating wider shufflevector arguments to predecessors.

[sheredom]

Summary

instcombine would propagate shufflevector insts that had wider output vectors onto predecessors, which would sometimes push undef's onto the divisor of a div/rem and result in bad codegen.

I've fixed this by just banning propagating shufflevector back if the result of the shufflevector is wider than the input vectors.

[spatel]

I found a case where instcombine would introduce an undef into the divisor of a div/rem instruction, which would in turn cause that instruction to be simplified away because 'if any vector component is undef, the whole div/rem is undef.'

I haven't stepped through any of the tests, but before we do that we need to correct this statement. The description only applies to integer ops, not FP. So we need to remove FP div/rem from this patch and/or show how those ops are causing different problems (and that would likely be a different patch).

Please minimize the tests that demonstrate the bug (don't need 'spir func' at least), and commit them to trunk with the current (buggy) output using the script at utils/update_test_checks.py to auto-generate the complete CHECK lines.

[sheredom]

Just to be clear you want me to commit the tests in a separate commit with the bad output first?

I'll remove the spir_func, but everything else in the tests is requied (instcombine will only attempt the opt if it sees the insertelement instructions as the input to the div/rem from what I can tell).

I don't have commit access to LLVM's trunk (long time user, finally able to be a contributor), do you want me to just file another review for just the test case?

[spatel]

In D52548#1246583, @sheredom wrote:

Just to be clear you want me to commit the tests in a separate commit with the bad output first?

Yes - that way we'll have evidence of the bogus output, so if the code patch that fixes those bugs gets reverted for some reason, we'll know that we've reintroduced potential miscompiles.

I'll remove the spir_func, but everything else in the tests is requied (instcombine will only attempt the opt if it sees the insertelement instructions as the input to the div/rem from what I can tell).

Sounds good - I couldn't tell just from looking how we trigger the problem.

I don't have commit access to LLVM's trunk (long time user, finally able to be a contributor), do you want me to just file another review for just the test case?

Great - welcome! Yes, if you don't have commit rights yet, just post the test file in another review, and I'll commit it on your behalf.

[sheredom]

See https://reviews.llvm.org/D52556 for just the test case with the bad output expected.

[sheredom]

Removed the fdiv/frem as they weren't currently inst simplifying the bad behaviour, and used the utils script to update the test case.

[lebedev.ri]

test/Transforms/InstCombine/stop_bad_undef_propagation.ll

3

You need to rebase this diff ontop of the D52556, so this diff actually shows the diff for the tests, too.

[spatel]

test/Transforms/InstCombine/stop_bad_undef_propagation.ll

3

I checked in the baseline at rL343140, so this can be rebased against trunk now. You'll want to regenerate the assertions using the script again because I made some cosmetic changes to the tests when I committed.

[sheredom]

Rebased on tip trunk.

[spatel]

I haven't actually stepped through the code changes yet, so let me know if I'm missing something. I'm just looking at the tests a bit closer now.

The problem is independent of any FP ops, so we can kill the int-to-fp-cast ops and the trailing fmul in all cases, and we still show the bug.
Going further, we don't even need the final shuffle in any of these tests?

define <3 x i32> @add(i32 %y, i32 %z) {
  %i0 = insertelement <2 x i32> undef, i32 %y, i32 0
  %i1 = insertelement <2 x i32> %i0, i32 %z, i32 1
  %a = udiv <2 x i32> %i1, <i32 255, i32 255>
  %ext = shufflevector <2 x i32> %a, <2 x i32> undef, <3 x i32> <i32 0, i32 1, i32 undef>
  ret <3 x i32> %ext
}

[spatel]

In D52548#1248149, @spatel wrote:

I haven't actually stepped through the code changes yet, so let me know if I'm missing something. I'm just looking at the tests a bit closer now.

The problem is independent of any FP ops, so we can kill the int-to-fp-cast ops and the trailing fmul in all cases, and we still show the bug.
Going further, we don't even need the final shuffle in any of these tests?

define <3 x i32> @add(i32 %y, i32 %z) {
  %i0 = insertelement <2 x i32> undef, i32 %y, i32 0
  %i1 = insertelement <2 x i32> %i0, i32 %z, i32 1
  %a = udiv <2 x i32> %i1, <i32 255, i32 255>
  %ext = shufflevector <2 x i32> %a, <2 x i32> undef, <3 x i32> <i32 0, i32 1, i32 undef>
  ret <3 x i32> %ext
}

And I gave away the punchline in that test name. :)
This is where I was heading, and I've now checked. Let's change the div to a seemingly safe 'add' instead:

define <3 x i32> @add(i32 %y, i32 %z) {
  %i0 = insertelement <2 x i32> undef, i32 %y, i32 0
  %i1 = insertelement <2 x i32> %i0, i32 %z, i32 1
  %a = add <2 x i32> %i1, <i32 255, i32 255>
  %ext = shufflevector <2 x i32> %a, <2 x i32> undef, <3 x i32> <i32 0, i32 1, i32 undef>
  ret <3 x i32> %ext
}

$ ./opt -instcombine badshuf.ll -S
define <3 x i32> @add(<3 x i32> %x, i32 %y, i32 %z) {
  %1 = insertelement <3 x i32> undef, i32 %y, i32 0
  %2 = insertelement <3 x i32> %1, i32 %z, i32 1
  %3 = add <3 x i32> %2, <i32 255, i32 255, i32 undef>
  ret <3 x i32> %3
}

That's probably not a desired canonicalization for any op. Yes, we eliminated a shuffle, but we created wider vector ops to make that happen, and that's not something we should do without a cost model.
So I think there's a simpler fix for the problem as demonstrated - don't call CanEvaluateShuffled() if this is shuffle that widens its inputs. There might still be a need for preventing undefs with div/rem, but we need a different test to show that.

[sheredom]

Changed the check to not ever push wider shufflevector stuff back onto predecessor instructions as per spatel's suggestion!

[lebedev.ri]

lib/Transforms/InstCombine/InstCombineVectorOps.cpp

874

I think this should to into lib/IR/Instructions.cpp, somewhere around the ShuffleVectorInst::isIdentityMask()?
(isWideningMask())

test/Transforms/InstCombine/stop_bad_undef_propagation.ll

4

Is it intentional that the test coverage changes?

[sheredom]

test/Transforms/InstCombine/stop_bad_undef_propagation.ll

4

Aye - @spatel noted that what I was testing was actually a specific part of a wider problem (that we shouldn't be widening shuffle vector predecessors), so I decided to reduce the test case as a result. With my change any predecessor of a shuffle vector that widens the input will not be optimized in this fashion.

[sheredom]

Move the shuffle widens check into Instructions.h, and call it increasesLength to match the existing changesLength call that was already on ShuffleVectorInst.

[sheredom]

lib/Transforms/InstCombine/InstCombineVectorOps.cpp

874

So I moved it into Instructions.h instead, and called it increasesLength because there was an existing method changesLength on ShuffleVectorInst so I thought better to match that!

[lebedev.ri]

include/llvm/IR/Instructions.h

2460 ↗

(On Diff #167446)

/// Examples: shufflevector <4 x n> A, <4 x n> B, <1,2,3>
///           shufflevector <4 x n> A, <4 x n> B, <1,2,3,4,5>

[sheredom]

include/llvm/IR/Instructions.h

2460 ↗

(On Diff #167446)

You want me to change the comment of the function I didn't touch? I generally try and avoid doing that, but if you think it is worth it I can!

[lebedev.ri]

include/llvm/IR/Instructions.h

2460 ↗

(On Diff #167446)

Yes, i think that comment for a slightly related function should be updated.

[spatel]

lib/Transforms/InstCombine/InstCombineVectorOps.cpp

917–919

Leave formatting changes out of this patch, so we're not distracting from the real diff.

937–938

Leave formatting changes out of this patch, so we're not distracting from the real diff.

958–959

Leave formatting changes out of this patch, so we're not distracting from the real diff.

1485

The length check should be before CanEvaluateShuffled() to be more efficient (CanEval... is recursive and potentially costly).

[sheredom]

Fix latest review comments.

[spatel]

Thanks - LGTM. As before, I'd prefer to have the baseline tests in place before the patch. Let me know if I should commit on your behalf.

[sheredom]

In D52548#1249085, @spatel wrote:

Thanks - LGTM. As before, I'd prefer to have the baseline tests in place before the patch. Let me know if I should commit on your behalf.

Yes please!

[jvesely]

• signature.asc848 BDownload
• vload-char-constant.cl3 KBDownload
• bad.link-0.ll14 KBDownload
• good.link-0.ll13 KBDownload

lib/Transforms/InstCombine/InstCombineVectorOps.cpp

Instruction *InstCombiner::visitInsertElementInst(InsertElementInst &IE) {
   // Turn a sequence of inserts that broadcasts a scalar into a single
   // insert + shufflevector.
   if (Instruction *Broadcast = foldInsSequenceIntoBroadcast(IE))
     return Broadcast;
 
   return nullptr;
 }
 
+/// Return true if a given LHS, undef RHS, and a Mask, would introduce an undef.
+static bool WouldIntroduceUndef(Value *LHS, ArrayRef<int> Mask) {
+  const int VWidth = LHS->getType()->getVectorNumElements();
+
+  for (int i = 0, e = Mask.size(); i != e; ++i) {
+    if ((VWidth <= Mask[i]) || (-1 == Mask[i]))
+      return true;
+  }
+
+  return false;
+}
+
 /// Return true if we can evaluate the specified expression tree if the vector
 /// elements were shuffled in a different order.
 static bool CanEvaluateShuffled(Value *V, ArrayRef<int> Mask,
-                                unsigned Depth = 5) {
+                                bool UndefBeingIntroduced, unsigned Depth = 5) {
   // We can always reorder the elements of a constant.
   if (isa<Constant>(V))
     return true;
 
   // We won't reorder vector arguments. No IPO here.
   Instruction *I = dyn_cast<Instruction>(V);
   if (!I) return false;
 
   // Two users may expect different orders of the elements. Don't try it.
   if (!I->hasOneUse())
     return false;
 
   if (Depth == 0) return false;
 
   switch (I->getOpcode()) {
+    case Instruction::UDiv:
+    case Instruction::SDiv:
+    case Instruction::URem:
+    case Instruction::SRem:
+      if (UndefBeingIntroduced)
+        return false;
+      LLVM_FALLTHROUGH;
     case Instruction::Add:
     case Instruction::FAdd:
     case Instruction::Sub:
     case Instruction::FSub:
     case Instruction::Mul:
     case Instruction::FMul:
-    case Instruction::UDiv:
-    case Instruction::SDiv:
     case Instruction::FDiv:
-    case Instruction::URem:
-    case Instruction::SRem:
     case Instruction::FRem:
     case Instruction::Shl:
     case Instruction::LShr:
     case Instruction::AShr:
     case Instruction::And:
     case Instruction::Or:
     case Instruction::Xor:
     case Instruction::ICmp:
     case Instruction::FCmp:
     case Instruction::Trunc:
     case Instruction::ZExt:
     case Instruction::SExt:
     case Instruction::FPToUI:
     case Instruction::FPToSI:
     case Instruction::UIToFP:
     case Instruction::SIToFP:
     case Instruction::FPTrunc:
     case Instruction::FPExt:
     case Instruction::GetElementPtr: {
       for (Value *Operand : I->operands()) {
-        if (!CanEvaluateShuffled(Operand, Mask, Depth-1))
+        if (!CanEvaluateShuffled(Operand, Mask, UndefBeingIntroduced,
+                                 Depth - 1))
           return false;
       }
       return true;
     }
     case Instruction::InsertElement: {
       ConstantInt *CI = dyn_cast<ConstantInt>(I->getOperand(2));
       if (!CI) return false;
       int ElementNumber = CI->getLimitedValue();
 
       // Verify that 'CI' does not occur twice in Mask. A single 'insertelement'
       // can't put an element into multiple indices.
       bool SeenOnce = false;
       for (int i = 0, e = Mask.size(); i != e; ++i) {
         if (Mask[i] == ElementNumber) {
           if (SeenOnce)
             return false;
           SeenOnce = true;
         }
       }
-      return CanEvaluateShuffled(I->getOperand(0), Mask, Depth-1);
+      return CanEvaluateShuffled(I->getOperand(0), Mask, UndefBeingIntroduced,
+                                 Depth - 1);
     }
   }
   return false;
 }
 
 /// Rebuild a new instruction just like 'I' but with the new operands given.
 /// In the event of type mismatch, the type of the operands is correct.
 static Value *buildNew(Instruction *I, ArrayRef<Value*> NewOps) {
if (VWidth == LHSWidth) {
     bool isLHSID, isRHSID;
     recognizeIdentityMask(Mask, isLHSID, isRHSID);
 
     // Eliminate identity shuffles.
     if (isLHSID) return replaceInstUsesWith(SVI, LHS);
     if (isRHSID) return replaceInstUsesWith(SVI, RHS);
   }
 
-  if (isa<UndefValue>(RHS) && CanEvaluateShuffled(LHS, Mask)) {
+  if (isa<UndefValue>(RHS) &&
+      CanEvaluateShuffled(LHS, Mask, WouldIntroduceUndef(LHS, Mask))) {
     Value *V = EvaluateInDifferentElementOrder(LHS, Mask);
     return replaceInstUsesWith(SVI, V);
   }
 
   // SROA generates shuffle+bitcast when the extracted sub-vector is bitcast to
   // a non-vector type. We can instead bitcast the original vector followed by
   // an extract of the desired element:
   //

test/Transforms/InstCombine/stop_bad_undef_propagation.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -instcombine -S | FileCheck %s
 
 ; For each of the div/rem integer instructions we test 3 different variants for
 ; how an undef value could be back propagated:
 ; 0) back propagation because of an undef shuffle vector mask
 ; 1) back propagation because of an undef value selected from RHS
 ; 2) back propagation because of an undef value selected from LHS
 
 define <3 x float> @udiv0(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @udiv0(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = udiv <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = uitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = udiv <2 x i32> %t1, <i32 255, i32 255>
   %t3 = uitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> %t3, <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @udiv1(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @udiv1(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = udiv <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = uitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = udiv <2 x i32> %t1, <i32 255, i32 255>
   %t3 = uitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> %t3, <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 2>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @udiv2(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @udiv2(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = udiv <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = uitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = udiv <2 x i32> %t1, <i32 255, i32 255>
   %t3 = uitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> undef, <2 x float> %t3, <3 x i32> <i32 2, i32 3, i32 0>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @sdiv0(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @sdiv0(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = sdiv <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = sitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = sdiv <2 x i32> %t1, <i32 255, i32 255>
   %t3 = sitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> %t3, <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @sdiv1(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @sdiv1(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = sdiv <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = sitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = sdiv <2 x i32> %t1, <i32 255, i32 255>
   %t3 = sitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> %t3, <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 2>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @sdiv2(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @sdiv2(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = sdiv <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = sitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = sdiv <2 x i32> %t1, <i32 255, i32 255>
   %t3 = sitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> undef, <2 x float> %t3, <3 x i32> <i32 2, i32 3, i32 0>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @urem0(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @urem0(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = urem <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = uitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = urem <2 x i32> %t1, <i32 255, i32 255>
   %t3 = uitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> %t3, <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @urem1(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @urem1(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = urem <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = uitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = urem <2 x i32> %t1, <i32 255, i32 255>
   %t3 = uitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> %t3, <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 2>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @urem2(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @urem2(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = urem <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = uitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = urem <2 x i32> %t1, <i32 255, i32 255>
   %t3 = uitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> undef, <2 x float> %t3, <3 x i32> <i32 2, i32 3, i32 0>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @srem0(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @srem0(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = srem <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = sitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = srem <2 x i32> %t1, <i32 255, i32 255>
   %t3 = sitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> %t3, <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @srem1(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @srem1(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = srem <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = sitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = srem <2 x i32> %t1, <i32 255, i32 255>
   %t3 = sitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> %t3, <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 2>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }
 
 define <3 x float> @srem2(<3 x float> %x, i32 %y, i32 %z) {
 ; CHECK-LABEL: @srem2(
-; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> <float 0.000000e+00, float 0.000000e+00, float undef>, <3 x i32> <i32 0, i32 3, i32 4>
+; CHECK-NEXT:    [[T0:%.*]] = insertelement <2 x i32> undef, i32 [[Y:%.*]], i32 0
+; CHECK-NEXT:    [[T1:%.*]] = insertelement <2 x i32> [[T0]], i32 [[Z:%.*]], i32 1
+; CHECK-NEXT:    [[T2:%.*]] = srem <2 x i32> [[T1]], <i32 255, i32 255>
+; CHECK-NEXT:    [[T3:%.*]] = sitofp <2 x i32> [[T2]] to <2 x float>
+; CHECK-NEXT:    [[T4:%.*]] = shufflevector <2 x float> [[T3]], <2 x float> undef, <3 x i32> <i32 0, i32 1, i32 undef>
+; CHECK-NEXT:    [[T5:%.*]] = shufflevector <3 x float> [[X:%.*]], <3 x float> [[T4]], <3 x i32> <i32 0, i32 3, i32 4>
 ; CHECK-NEXT:    [[T6:%.*]] = fmul <3 x float> [[T5]], [[X]]
 ; CHECK-NEXT:    ret <3 x float> [[T6]]
 ;
   %t0 = insertelement <2 x i32> undef, i32 %y, i32 0
   %t1 = insertelement <2 x i32> %t0, i32 %z, i32 1
   %t2 = srem <2 x i32> %t1, <i32 255, i32 255>
   %t3 = sitofp <2 x i32> %t2 to <2 x float>
   %t4 = shufflevector <2 x float> undef, <2 x float> %t3, <3 x i32> <i32 2, i32 3, i32 0>
   %t5 = shufflevector <3 x float> %x, <3 x float> %t4, <3 x i32> <i32 0, i32 3, i32 4>
   %t6 = fmul <3 x float> %x, %t5
   ret <3 x float> %t6
 }