[VectorCombine] Widening of partial vector loads

[lebedev.ri]

Summary

If we are loading some vector, and we know it will be legalized into a vector,
and occupy (potentially a number of) vector registers, iff we load less bytes
than the total size of the occupied vector registers, then the legalization
will have a hard time. At worst, the load will be scalarized, at least partially,
and scalar vector elements inserted forming the narrow vector.

But sometimes, if the vector will be widened, we can tell that we are allowed
to load those extra 'padding' elements, based on dereferenceability or alignment knowledge.

I think, this approach with asking the legalization about the final vector size
is the most straight-forward.

I've checked, and i believe this is endianness-agnostic as per alive2.

[lebedev.ri]

Streamline the code.

[RKSimon]

Thanks for looking at this - I'd delayed requesting something like this until we have a better idea of what the non-pow2 SLP IR from D57059 is likely to look like.

I also ended up wondering whether we should consider using the dereferencable data in DAGTypeLegalizer::GenWidenVectorLoads? As that would help with the float3 case on D106280 that raised concern

[lebedev.ri]

In D106399#2892863, @RKSimon wrote:

Thanks for looking at this - I'd delayed requesting something like this until we have a better idea of what the non-pow2 SLP IR from D57059 is likely to look like.

Even if SLP learns to emit wide-enough loads (which it should, regardless of non-pow2 vectorization support/etc),
i would guess we'd still want this, because here in IR we have much more information to deduce
the legality of such a transformation rather than leaving it up to the backend.

I also ended up wondering whether we should consider using the dereferencable data in DAGTypeLegalizer::GenWidenVectorLoads? As that would help with the float3 case on D106280 that raised concern

I actually tried looking at exactly that before doing this, without much success.

[spatel]

Thanks for working on this! I agree that this is useful independent of whatever we can/should do to improve SLP.

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

290

worth -> width

312–313

Use the unary variant of CreateShuffleVector here.
Could call this value ExtractSubvector or state that in the code comment. Can we always assume that the extract op is free, or should we add that potential cost into the equation?

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

590–593

Can we preserve the better alignment?

[spatel]

llvm/test/Transforms/VectorCombine/X86/load-widening.ll

1–2

Hmm...I don't think I've ever tried that experiment. :)
Did you confirm that the layout "wins" over the target to provide the coverage you expected?

[RKSimon]

D57059 looks to be performing masked loads when the wider range is dereferencable (costs permitting) - see the llvm/test/Transforms/SLPVectorizer/X86/dot-product.ll test changes

[lebedev.ri]

Thanks for taking a look!
Addressing nits.

In D106399#2893246, @RKSimon wrote:

D57059 looks to be performing masked loads when the wider range is dereferencable (costs permitting) - see the llvm/test/Transforms/SLPVectorizer/X86/dot-product.ll test changes

Then we'll also need a load unmasking transform.

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

290

This is not a typo, but okay.

312–313

That subvector we're extracting here will be legalized (widened) into the full vector we've just loaded,
so the shuffle just pretends that a few high elements of that legal vector don't exist.
That is the whole assumption behind the transformation here.
There isn't any actual subvector extraction here.
So i'm not really seeing why we'd need to check the cost of that.

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

590–593

Some other transformation does this, i'll take a look.

llvm/test/Transforms/VectorCombine/X86/load-widening.ll

1–2

https://alive2.llvm.org/ce/z/yAGpMV

[spatel]

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

312–313

Ok - please add a code comment with that explanation then. That way, we'll document why we don't factor cost of the shuffle.

llvm/test/Transforms/VectorCombine/X86/load-widening.ll

1–2

I'm not questioning the logic, just that there is no big-endian x86 target, so I don't know what is happening internally in LLVM in this situation.

I think it would be better to add a different test file for a target that actually does support both modes (AArch64 or PowerPC?). We still want an x86 test file to verify that SSE vs. AVX is behaving as expected though.

194

Can you explain the difference between this test and vec_with_7elts_256bits for an SSE target? It's not obvious to me why we are ok widening to 256-bit if that's not legal, but not wider than that.

[craig.topper]

Something to keep in mind. Loading more data than was stored can prevent store to load forwarding. If there is a store in the store buffer that hasn't written to the cache yet, its data can forward to a load instead of waiting until the store gets written to the cache. This doesn't work if the load is larger than the size of the store. The load will have to wait until the store gets to the cache to merge with the surrounding data.

[lebedev.ri]

llvm/test/Transforms/VectorCombine/X86/load-widening.ll

172

We widen this to either 2x XMM, or an 1x YMM, and we know we can do this as per deref info.

194

We need to widen this to either 3x XMM or 2x YMM, but we don't know we can load that many bytes.

There is another problem hiding here, iff we know we can load 3x XMM, we still try to widen to 4x XMM,
because that's what the legalizer told us, because it only knows how to double.

[lebedev.ri]

I guess i need to mimic X86TTIImpl::getMemoryOpCost() more.

llvm/include/llvm/Analysis/TargetTransformInfo.h

   InstructionCost getCallInstrCost(
       Function *F, Type *RetTy, ArrayRef<Type *> Tys,
       TTI::TargetCostKind CostKind = TTI::TCK_SizeAndLatency) const;
 
   /// \returns The number of pieces into which the provided type must be
   /// split during legalization. Zero is returned when the answer is unknown.
   unsigned getNumberOfParts(Type *Tp) const;
 
+  /// \returns The type of the piece into which the provided type must be
+  /// split during legalization.
+  Type *getLegalizedPartType(Type *Tp) const;
+
   /// \returns The cost of the address computation. For most targets this can be
   /// merged into the instruction indexing mode. Some targets might want to
   /// distinguish between address computation for memory operations on vector
   /// types and scalar types. Such targets should override this function.
   /// The 'SE' parameter holds pointer for the scalar evolution object which
   /// is used in order to get the Ptr step value in case of constant stride.
   /// The 'Ptr' parameter holds SCEV of the access pointer.
   InstructionCost getAddressComputationCost(Type *Ty,
virtual InstructionCost getExtendedAddReductionCost(
       TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput) = 0;
   virtual InstructionCost
   getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
                         TTI::TargetCostKind CostKind) = 0;
   virtual InstructionCost getCallInstrCost(Function *F, Type *RetTy,
                                            ArrayRef<Type *> Tys,
                                            TTI::TargetCostKind CostKind) = 0;
   virtual unsigned getNumberOfParts(Type *Tp) = 0;
+  virtual Type *getLegalizedPartType(Type *Tp) = 0;
   virtual InstructionCost
   getAddressComputationCost(Type *Ty, ScalarEvolution *SE, const SCEV *Ptr) = 0;
   virtual InstructionCost
   getCostOfKeepingLiveOverCall(ArrayRef<Type *> Tys) = 0;
   virtual bool getTgtMemIntrinsic(IntrinsicInst *Inst,
                                   MemIntrinsicInfo &Info) = 0;
   virtual unsigned getAtomicMemIntrinsicMaxElementSize() const = 0;
   virtual Value *getOrCreateResultFromMemIntrinsic(IntrinsicInst *Inst,
public:
   InstructionCost getCallInstrCost(Function *F, Type *RetTy,
                                    ArrayRef<Type *> Tys,
                                    TTI::TargetCostKind CostKind) override {
     return Impl.getCallInstrCost(F, RetTy, Tys, CostKind);
   }
   unsigned getNumberOfParts(Type *Tp) override {
     return Impl.getNumberOfParts(Tp);
   }
+  Type *getLegalizedPartType(Type *Tp) override {
+    return Impl.getLegalizedPartType(Tp);
+  }
   InstructionCost getAddressComputationCost(Type *Ty, ScalarEvolution *SE,
                                             const SCEV *Ptr) override {
     return Impl.getAddressComputationCost(Ty, SE, Ptr);
   }
   InstructionCost getCostOfKeepingLiveOverCall(ArrayRef<Type *> Tys) override {
     return Impl.getCostOfKeepingLiveOverCall(Tys);
   }
   bool getTgtMemIntrinsic(IntrinsicInst *Inst,

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
 
   InstructionCost getCallInstrCost(Function *F, Type *RetTy,
                                    ArrayRef<Type *> Tys,
                                    TTI::TargetCostKind CostKind) const {
     return 1;
   }
 
   unsigned getNumberOfParts(Type *Tp) const { return 0; }
+  Type *getLegalizedPartType(Type *Tp) const { return nullptr; }
 
   InstructionCost getAddressComputationCost(Type *Tp, ScalarEvolution *,
                                             const SCEV *) const {
     return 0;
   }
 
   InstructionCost getArithmeticReductionCost(unsigned, VectorType *,
                                              TTI::TargetCostKind) const {

llvm/include/llvm/CodeGen/BasicTTIImpl.h

   }
 
   unsigned getNumberOfParts(Type *Tp) {
     std::pair<InstructionCost, MVT> LT =
         getTLI()->getTypeLegalizationCost(DL, Tp);
     return *LT.first.getValue();
   }
 
+  Type *getLegalizedPartType(Type *Tp) {
+    std::pair<InstructionCost, MVT> LT =
+        getTLI()->getTypeLegalizationCost(DL, Tp);
+    return EVT(LT.second).getTypeForEVT(Tp->getContext());
+  }
+
   InstructionCost getAddressComputationCost(Type *Ty, ScalarEvolution *,
                                             const SCEV *) {
     return 0;
   }
 
   /// Try to calculate arithmetic and shuffle op costs for reduction intrinsics.
   /// We're assuming that reduction operation are performing the following way:
   ///

llvm/lib/Analysis/TargetTransformInfo.cpp

TargetTransformInfo::getCallInstrCost(Function *F, Type *RetTy,
   assert(Cost >= 0 && "TTI should not produce negative costs!");
   return Cost;
 }
 
 unsigned TargetTransformInfo::getNumberOfParts(Type *Tp) const {
   return TTIImpl->getNumberOfParts(Tp);
 }
 
+Type *TargetTransformInfo::getLegalizedPartType(Type *Tp) const {
+  return TTIImpl->getLegalizedPartType(Tp);
+}
+
 InstructionCost
 TargetTransformInfo::getAddressComputationCost(Type *Tp, ScalarEvolution *SE,
                                                const SCEV *Ptr) const {
   InstructionCost Cost = TTIImpl->getAddressComputationCost(Tp, SE, Ptr);
   assert(Cost >= 0 && "TTI should not produce negative costs!");
   return Cost;
 }
 

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

 #include "llvm/Transforms/Utils/Local.h"
 #include "llvm/Transforms/Vectorize.h"
 
 using namespace llvm;
 using namespace llvm::PatternMatch;
 
 #define DEBUG_TYPE "vector-combine"
 STATISTIC(NumVecLoad, "Number of vector loads formed");
+STATISTIC(NumVecLoadWiden, "Number of vector loads widened");
 STATISTIC(NumVecCmp, "Number of vector compares formed");
 STATISTIC(NumVecBO, "Number of vector binops formed");
 STATISTIC(NumVecCmpBO, "Number of vector compare + binop formed");
 STATISTIC(NumShufOfBitcast, "Number of shuffles moved after bitcast");
 STATISTIC(NumScalarBO, "Number of scalar binops formed");
 STATISTIC(NumScalarCmp, "Number of scalar compares formed");
 
 static cl::opt<bool> DisableVectorCombine(
private:
   Function &F;
   IRBuilder<> Builder;
   const TargetTransformInfo &TTI;
   const DominatorTree &DT;
   AAResults &AA;
   AssumptionCache &AC;
 
   bool vectorizeLoadInsert(Instruction &I);
+  bool widenPartialVectorLoad(Instruction &I);
   ExtractElementInst *getShuffleExtract(ExtractElementInst *Ext0,
                                         ExtractElementInst *Ext1,
                                         unsigned PreferredExtractIndex) const;
   bool isExtractExtractCheap(ExtractElementInst *Ext0, ExtractElementInst *Ext1,
                              unsigned Opcode,
                              ExtractElementInst *&ConvertToShuffle,
                              unsigned PreferredExtractIndex);
   void foldExtExtCmp(ExtractElementInst *Ext0, ExtractElementInst *Ext1,
bool VectorCombine::vectorizeLoadInsert(Instruction &I) {
   Value *VecLd = Builder.CreateAlignedLoad(MinVecTy, CastedPtr, Alignment);
   VecLd = Builder.CreateShuffleVector(VecLd, Mask);
 
   replaceValue(I, *VecLd);
   ++NumVecLoad;
   return true;
 }
 
+bool VectorCombine::widenPartialVectorLoad(Instruction &I) {
+  auto *Load = dyn_cast<LoadInst>(&I);
+  if (!Load)
+    return false;
+
+  // What vector type do we currently load?
+  auto *OrigVecTy = dyn_cast<FixedVectorType>(Load->getType());
+  if (!OrigVecTy)
+    return false;
+
+  const DataLayout &DL = I.getModule()->getDataLayout();
+
+  // Vector element type should be power-of-two sized and a multiple of byte.
+  auto *ScalarEltTy = OrigVecTy->getScalarType();
+  unsigned OrigNumElts = OrigVecTy->getNumElements();
+  unsigned NumBitsPerElt = DL.getTypeSizeInBits(ScalarEltTy);
+  unsigned EltTyByteSize = NumBitsPerElt / 8;
+  if (NumBitsPerElt % 8 != 0 || !isPowerOf2_32(NumBitsPerElt))
+    return false;
+
+  Value *OrigPtr = Load->getPointerOperand();
+  Align Alignment = Load->getAlign();
+  unsigned AS = Load->getPointerAddressSpace();
+
+  // How will that type be legalized? I.e. into what vector register
+  // will it be loaded, and how many registers will be occupied?
+  auto *LegalizedPartVecTy =
+      dyn_cast_or_null<FixedVectorType>(TTI.getLegalizedPartType(OrigVecTy));
+  unsigned NumOfLegalizedParts = TTI.getNumberOfParts(OrigVecTy);
+
+  // If it doesn't legalize into (a number of) vector registers, don't bother.
+  if (!LegalizedPartVecTy || !NumOfLegalizedParts)
+    return false;
+
+  unsigned OrigByteCount = DL.getTypeSizeInBits(OrigVecTy) / 8;
+  unsigned LegalizedByteCount =
+      NumOfLegalizedParts * DL.getTypeSizeInBits(LegalizedPartVecTy) / 8;
+  assert(LegalizedByteCount >= OrigByteCount &&
+         "Number of bytes-to-be-loaded shouldn't decrease!");
+
+  // Do we already load a multiple of the legalized type?
+  if (OrigByteCount == LegalizedByteCount)
+    return false;
+
+  // How many more elements would we need to load?
+  unsigned NumExtraBytes = LegalizedByteCount - OrigByteCount;
+  assert(
+      NumExtraBytes % EltTyByteSize == 0 &&
+      "Need to load extra byte count that isn't a multiple of element size?");
+  unsigned NumExtraElts = NumExtraBytes / EltTyByteSize;
+
+  auto *WideVecTy =
+      FixedVectorType::get(ScalarEltTy, OrigNumElts + NumExtraElts);
+  assert(DL.getTypeSizeInBits(WideVecTy) / 8 == LegalizedByteCount &&
+         "Failed to properly widen OrigVecTy to match the total legalized "
+         "vector size?");
+
+  // Okay, we currently load less than full worth of the legalized vectors.
+  // If we'd widen the load, would that be more costly than the current load?
+  InstructionCost OldLoadCost =
+      TTI.getMemoryOpCost(Instruction::Load, OrigVecTy, Alignment, AS);
+  InstructionCost NewLoadCost =
+      TTI.getMemoryOpCost(Instruction::Load, WideVecTy, Alignment, AS);
+  if (NewLoadCost > OldLoadCost)
+    return false;
+
+  // It would not be more costly. But can we perform such a wide load?
+  if (!isSafeToLoadUnconditionally(OrigPtr, WideVecTy, Align(1), DL, Load, &DT,
+                                   /*TLI=*/nullptr))
+    return false;
+
+  IRBuilder<> Builder(Load);
+  Value *CastedPtr =
+      Builder.CreateBitCast(OrigPtr, WideVecTy->getPointerTo(AS));
+  Value *WideVecLd = Builder.CreateAlignedLoad(WideVecTy, CastedPtr, Alignment);
+  // We loaded some extra elements, we only need the low NumElts ones.
+  // This is endiannes-insensitive.
+  SmallVector<int, 32> Mask(OrigNumElts);
+  std::iota(Mask.begin(), Mask.end(), 0);
+  Value *SmallVec =
+      Builder.CreateShuffleVector(WideVecLd, PoisonValue::get(WideVecTy), Mask);
+  replaceValue(I, *SmallVec);
+  ++NumVecLoadWiden;
+  return true;
+}
+
 /// Determine which, if any, of the inputs should be replaced by a shuffle
 /// followed by extract from a different index.
 ExtractElementInst *VectorCombine::getShuffleExtract(
     ExtractElementInst *Ext0, ExtractElementInst *Ext1,
     unsigned PreferredExtractIndex = InvalidIndex) const {
   assert(isa<ConstantInt>(Ext0->getIndexOperand()) &&
          isa<ConstantInt>(Ext1->getIndexOperand()) &&
          "Expected constant extract indexes");
for (BasicBlock &BB : F) {
     if (!DT.isReachableFromEntry(&BB))
       continue;
     // Use early increment range so that we can erase instructions in loop.
     for (Instruction &I : make_early_inc_range(BB)) {
       if (isa<DbgInfoIntrinsic>(I))
         continue;
       Builder.SetInsertPoint(&I);
       MadeChange |= vectorizeLoadInsert(I);
+      MadeChange |= widenPartialVectorLoad(I);
       MadeChange |= foldExtractExtract(I);
       MadeChange |= foldBitcastShuf(I);
       MadeChange |= scalarizeBinopOrCmp(I);
       MadeChange |= foldExtractedCmps(I);
       MadeChange |= scalarizeLoadExtract(I);
       MadeChange |= foldSingleElementStore(I);
     }
   }

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

;
   %result1 = insertelement <2 x float> %result0, float %t5, i32 1
   store <2 x float> %result1, <2 x float>* %resultptr, align 8
   ret void
 }
 
 define <4 x float> @load_v2f32_extract_insert_v4f32(<2 x float>* align 16 dereferenceable(16) %p) nofree nosync {
 ; CHECK-LABEL: @load_v2f32_extract_insert_v4f32(
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <2 x float>* [[P:%.*]] to <4 x float>*
-; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 16
-; CHECK-NEXT:    [[R:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <4 x i32> <i32 0, i32 undef, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 4
+; CHECK-NEXT:    [[L:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[S:%.*]] = extractelement <2 x float> [[L]], i32 0
+; CHECK-NEXT:    [[R:%.*]] = insertelement <4 x float> poison, float [[S]], i32 0
 ; CHECK-NEXT:    ret <4 x float> [[R]]
 ;
   %l = load <2 x float>, <2 x float>* %p, align 4
   %s = extractelement <2 x float> %l, i32 0
   %r = insertelement <4 x float> poison, float %s, i32 0
   ret <4 x float> %r
 }
 

llvm/test/Transforms/VectorCombine/X86/load-widening.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -vector-combine -S -mtriple=x86_64-- -mattr=sse2 --data-layout="e-m:e-i64:64-f80:128-n8:16:32:64-S128" | FileCheck %s --check-prefixes=CHECK
 ; RUN: opt < %s -vector-combine -S -mtriple=x86_64-- -mattr=avx2 --data-layout="e-m:e-i64:64-f80:128-n8:16:32:64-S128" | FileCheck %s --check-prefixes=CHECK
 ; RUN: opt < %s -vector-combine -S -mtriple=x86_64-- -mattr=sse2 --data-layout="E-m:e-i64:64-f80:128-n8:16:32:64-S128" | FileCheck %s --check-prefixes=CHECK
 ; RUN: opt < %s -vector-combine -S -mtriple=x86_64-- -mattr=avx2 --data-layout="E-m:e-i64:64-f80:128-n8:16:32:64-S128" | FileCheck %s --check-prefixes=CHECK
 
 ;-------------------------------------------------------------------------------
 ; Here we know we can load 128 bits as per dereferenceability and alignment.
 
 ; We don't widen scalar loads per-se.

 ; CHECK-NEXT:    ret <1 x float> [[R]]
 ;
   %r = load <1 x float>, <1 x float>* %p, align 16
   ret <1 x float> %r
 }
 
 define <2 x float> @vec_with_2elts(<2 x float>* align 16 dereferenceable(16) %p) {
 ; CHECK-LABEL: @vec_with_2elts(
-; CHECK-NEXT:    [[R:%.*]] = load <2 x float>, <2 x float>* [[P:%.*]], align 16
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <2 x float>* [[P:%.*]] to <4 x float>*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 16
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
 ; CHECK-NEXT:    ret <2 x float> [[R]]
 ;
   %r = load <2 x float>, <2 x float>* %p, align 16
   ret <2 x float> %r
 }
 
 define <3 x float> @vec_with_3elts(<3 x float>* align 16 dereferenceable(16) %p) {
 ; CHECK-LABEL: @vec_with_3elts(
-; CHECK-NEXT:    [[R:%.*]] = load <3 x float>, <3 x float>* [[P:%.*]], align 16
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <3 x float>* [[P:%.*]] to <4 x float>*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 16
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <3 x i32> <i32 0, i32 1, i32 2>
 ; CHECK-NEXT:    ret <3 x float> [[R]]
 ;
   %r = load <3 x float>, <3 x float>* %p, align 16
   ret <3 x float> %r
 }
 
 ; Full-vector load. All good already.
 define <4 x float> @vec_with_4elts(<4 x float>* align 16 dereferenceable(16) %p) {
;
   ret <5 x float> %r
 }
 
 ;-------------------------------------------------------------------------------
 
 ; We can load 128 bits, and the fact that it's underaligned isn't relevant.
 define <3 x float> @vec_with_3elts_underaligned(<3 x float>* align 8 dereferenceable(16) %p) {
 ; CHECK-LABEL: @vec_with_3elts_underaligned(
-; CHECK-NEXT:    [[R:%.*]] = load <3 x float>, <3 x float>* [[P:%.*]], align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <3 x float>* [[P:%.*]] to <4 x float>*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 8
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <3 x i32> <i32 0, i32 1, i32 2>
 ; CHECK-NEXT:    ret <3 x float> [[R]]
 ;
   %r = load <3 x float>, <3 x float>* %p, align 8
   ret <3 x float> %r
 }
 
 ; We don't know we can load 128 bits, but since it's aligned, we still can do wide load.
 ; FIXME: this should still get widened.

 ; CHECK-NEXT:    ret <1 x float> [[R]]
 ;
   %r = load <1 x float>, <1 x float>* %p, align 32
   ret <1 x float> %r
 }
 
 define <2 x float> @vec_with_2elts_256bits(<2 x float>* align 32 dereferenceable(32) %p) {
 ; CHECK-LABEL: @vec_with_2elts_256bits(
-; CHECK-NEXT:    [[R:%.*]] = load <2 x float>, <2 x float>* [[P:%.*]], align 32
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <2 x float>* [[P:%.*]] to <4 x float>*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 32
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
 ; CHECK-NEXT:    ret <2 x float> [[R]]
 ;
   %r = load <2 x float>, <2 x float>* %p, align 32
   ret <2 x float> %r
 }
 
 define <3 x float> @vec_with_3elts_256bits(<3 x float>* align 32 dereferenceable(32) %p) {
 ; CHECK-LABEL: @vec_with_3elts_256bits(
-; CHECK-NEXT:    [[R:%.*]] = load <3 x float>, <3 x float>* [[P:%.*]], align 32
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <3 x float>* [[P:%.*]] to <4 x float>*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 32
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <3 x i32> <i32 0, i32 1, i32 2>
 ; CHECK-NEXT:    ret <3 x float> [[R]]
 ;
   %r = load <3 x float>, <3 x float>* %p, align 32
   ret <3 x float> %r
 }
 
 define <4 x float> @vec_with_4elts_256bits(<4 x float>* align 32 dereferenceable(32) %p) {
 ; CHECK-LABEL: @vec_with_4elts_256bits(
 ; CHECK-NEXT:    [[R:%.*]] = load <4 x float>, <4 x float>* [[P:%.*]], align 32
 ; CHECK-NEXT:    ret <4 x float> [[R]]
 ;
   %r = load <4 x float>, <4 x float>* %p, align 32
   ret <4 x float> %r
 }
 
 define <5 x float> @vec_with_5elts_256bits(<5 x float>* align 32 dereferenceable(32) %p) {
 ; CHECK-LABEL: @vec_with_5elts_256bits(
-; CHECK-NEXT:    [[R:%.*]] = load <5 x float>, <5 x float>* [[P:%.*]], align 32
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <5 x float>* [[P:%.*]] to <8 x float>*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <8 x float>, <8 x float>* [[TMP1]], align 32
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <8 x float> [[TMP2]], <8 x float> poison, <5 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4>
 ; CHECK-NEXT:    ret <5 x float> [[R]]
 ;
   %r = load <5 x float>, <5 x float>* %p, align 32
   ret <5 x float> %r
 }
 
 define <6 x float> @vec_with_6elts_256bits(<6 x float>* align 32 dereferenceable(32) %p) {
 ; CHECK-LABEL: @vec_with_6elts_256bits(
-; CHECK-NEXT:    [[R:%.*]] = load <6 x float>, <6 x float>* [[P:%.*]], align 32
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <6 x float>* [[P:%.*]] to <8 x float>*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <8 x float>, <8 x float>* [[TMP1]], align 32
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <8 x float> [[TMP2]], <8 x float> poison, <6 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5>
 ; CHECK-NEXT:    ret <6 x float> [[R]]
 ;
   %r = load <6 x float>, <6 x float>* %p, align 32
   ret <6 x float> %r
 }
 
 define <7 x float> @vec_with_7elts_256bits(<7 x float>* align 32 dereferenceable(32) %p) {
 ; CHECK-LABEL: @vec_with_7elts_256bits(
-; CHECK-NEXT:    [[R:%.*]] = load <7 x float>, <7 x float>* [[P:%.*]], align 32
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <7 x float>* [[P:%.*]] to <8 x float>*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <8 x float>, <8 x float>* [[TMP1]], align 32
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <8 x float> [[TMP2]], <8 x float> poison, <7 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6>
 ; CHECK-NEXT:    ret <7 x float> [[R]]
 ;
   %r = load <7 x float>, <7 x float>* %p, align 32
   ret <7 x float> %r
 }
 
 ; Full-vector load. All good already.
 define <8 x float> @vec_with_8elts_256bits(<8 x float>* align 32 dereferenceable(32) %p) {
 ; CHECK-LABEL: @vec_with_8elts_256bits(
 ; CHECK-NEXT:    [[R:%.*]] = load <8 x float>, <8 x float>* [[P:%.*]], align 32
 ; CHECK-NEXT:    ret <8 x float> [[R]]
 ;
   %r = load <8 x float>, <8 x float>* %p, align 32
   ret <8 x float> %r
 }
 
 ; We can't tell if we can load more than 256 bits.
 define <9 x float> @vec_with_9elts_256bits(<9 x float>* align 32 dereferenceable(32) %p) {
 ; CHECK-LABEL: @vec_with_9elts_256bits(
 ; CHECK-NEXT:    [[R:%.*]] = load <9 x float>, <9 x float>* [[P:%.*]], align 32
 ; CHECK-NEXT:    ret <9 x float> [[R]]
 ;
   %r = load <9 x float>, <9 x float>* %p, align 32
   ret <9 x float> %r
 }
 

 ; CHECK-NEXT:    ret <2 x i24> [[R]]
 ;
   %r = load <2 x i24>, <2 x i24>* %p, align 16
   ret <2 x i24> %r
 }
 
 define <2 x float> @vec_with_2elts_addressspace(<2 x float> addrspace(2)* align 16 dereferenceable(16) %p) {
 ; CHECK-LABEL: @vec_with_2elts_addressspace(
-; CHECK-NEXT:    [[R:%.*]] = load <2 x float>, <2 x float> addrspace(2)* [[P:%.*]], align 16
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <2 x float> addrspace(2)* [[P:%.*]] to <4 x float> addrspace(2)*
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float> addrspace(2)* [[TMP1]], align 16
+; CHECK-NEXT:    [[R:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
 ; CHECK-NEXT:    ret <2 x float> [[R]]
 ;
   %r = load <2 x float>, <2 x float> addrspace(2)* %p, align 16
   ret <2 x float> %r
 }

llvm/test/Transforms/VectorCombine/X86/load.ll

;
   %result1 = insertelement <2 x float> %result0, float %t5, i32 1
   store <2 x float> %result1, <2 x float>* %resultptr, align 8
   ret void
 }
 
 define <4 x float> @load_v2f32_extract_insert_v4f32(<2 x float>* align 16 dereferenceable(16) %p) nofree nosync {
 ; CHECK-LABEL: @load_v2f32_extract_insert_v4f32(
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <2 x float>* [[P:%.*]] to <4 x float>*
-; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 16
-; CHECK-NEXT:    [[R:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <4 x i32> <i32 0, i32 undef, i32 undef, i32 undef>
+; CHECK-NEXT:    [[TMP2:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 4
+; CHECK-NEXT:    [[L:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT:    [[S:%.*]] = extractelement <2 x float> [[L]], i32 0
+; CHECK-NEXT:    [[R:%.*]] = insertelement <4 x float> undef, float [[S]], i32 0
 ; CHECK-NEXT:    ret <4 x float> [[R]]
 ;
   %l = load <2 x float>, <2 x float>* %p, align 4
   %s = extractelement <2 x float> %l, i32 0
   %r = insertelement <4 x float> undef, float %s, i32 0
   ret <4 x float> %r
 }