[SROA] Create additional vector type candidates based on store and load slices

[zhuhan0]

Summary

Second try at A-Wadhwani's https://reviews.llvm.org/D132096, which was reverted. The original patch had three issues:

• https://reviews.llvm.org/D134032, which bjope kindly fixed. That patch is merged into this one.
• GHI #57796. Fixed and added a test.
• GHI #57821. I believe this is an undefined behavior which is not the fault of the original patch. Please see the issue for more details.

Original diff summary:

This patch adds additional vector types to be considered when doing promotion in SROA, based on the types of the store and load slices. This provides more promotion opportunities, by potentially using an optimal "intermediate" vector type.

For example, the following code would currently not be promoted to a vector, since __m128i is a <2 x i64> vector.

#include <xmmintrin.h>

__m128i packfoo0(int a, int b, int c, int d) {
  int r[4] = {a, b, c, d};
  __m128i rm;
  std::memcpy(&rm, r, sizeof(rm));
  return rm;
}

packfoo0(int, int, int, int):
        mov     dword ptr [rsp - 24], edi
        mov     dword ptr [rsp - 20], esi
        mov     dword ptr [rsp - 16], edx
        mov     dword ptr [rsp - 12], ecx
        movaps  xmm0, xmmword ptr [rsp - 24]
        ret

By also considering the types of the elements, we could find that the <4 x i32> type would be valid for promotion, hence removing the memory accesses for this function. In other words, we can explore other new vector types, with the same size but different element types based on the load and store instructions from the Slices, which can provide us more promotion opportunities.

Additionally, the step for removing duplicate elements from the CandidateTys vector was not using an equality comparator, which has been fixed.

[arsenm]

llvm/lib/Transforms/Scalar/SROA.cpp

2030

The first isa<VectorType> should be redundant with the isValidElementType check

llvm/test/Transforms/SROA/vector-promotion.ll

960

Should add some vector of pointer tests

[aprantl]

Is one of the tests testing that debug info is updated correctly in the new code path?

[zhuhan0]

In D143225#4103038, @aprantl wrote:

Is one of the tests testing that debug info is updated correctly in the new code path?

I don't think so. Not familiar with debug info TBH. Let me take a stab at that.

[zhuhan0]

• Remove redundant isa<VectorType>. Add some vector of ptr tests.
• Add opt -passes=-debugify,sroa test to vector-promotion.ll

[zhuhan0]

llvm/test/Transforms/SROA/vector-promotion.ll

4

@aprantl How do you feel about this? I can pre-commit this to show the changes if that's better.

[zhuhan0]

Pre-committed tests in vector-promotion.ll

[zhuhan0]

Can I get some review on the newer version please? Thanks in advance!

[vangthao]

llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll

26

There seems to be a regression here.

[zhuhan0]

llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll

26

Hi, this is actually the result of a bug fix that we happened to put in in this diff (line 2001 - 2028 in the new version), not related with the optimization change. There was a bug where RankVectorTypes which is not an equivalence relation but used in std::unique. And when a non-equivalence relation is used, the behavior is undefined. Take test_zeroinit as an example, there are two candidates <4 x i32> and <8 x i16> during isVectorPromotionViable. But due to this bug, <8 x i16> was removed by CandidateTys.erase and <4 x i32> is not viable so the transformation failed unexpectedly. I could separate the bug fix if that's better. On the other hand, I don't think this bitcast addition is going to show up in the end after backend lowering anyways.

[mingmingl]

llvm/lib/Transforms/Scalar/SROA.cpp

2022–2029

(Take the liberty to chime in for my understanding)
Here Ty is inserted even if it doesn't have the exact size with P. Does it mean (in this patch) that slices (S) are considered for vector SROA even if it has different size than P?

[zhuhan0]

llvm/lib/Transforms/Scalar/SROA.cpp

2022–2029

This Ty is not a candidate type. It's a type that shows up in load or store instructions and it'll be used later for considering other types. For example, if one Ty is i16 and there is one candidate type <2 x i32>, the inner loop will be able to add another candidate type <4 x i16>. Slices are still only considered if they have the same size as P, because only two places insert candidate types and they both enforce such requirement (line 1954 - 1955 and 1969 - 1970.

[mingmingl]

llvm/lib/Transforms/Scalar/SROA.cpp

2022–2029

many thanks for the example! Running this over https://godbolt.org/z/1rqGd9z3E and was able to understand how it helps (basically having more candidates vector types for checkVectorTypeForPromotion, not only the vector types seen in original partition slices).

[zhuhan0]

Separate RankVectorTypes bug fix into another diff.

[zhuhan0]

Rebase

[MatzeB]

There is a curious change in types for test_memset, test_array_vector and test_array_vector2.

This appears to stem from the code added in https://github.com/llvm/llvm-project/commit/529eafd9beff233ba8debfc73e0b5c04cac36835 which I don't fully understand the intention right now. @lebedev.ri do you think it is good (or at least not bad) to use 8 x i16 in those tests?

Either way this seems outside the scope of this diff to discuss as it is pre-existing code. The change LGTM.

[zhuhan0]

Seems there's no objection. Landing this one.

[glandium]

This is causing crashes when building Skia. See https://reviews.llvm.org/D141188#4179444

[zhuhan0]

In D143225#4180342, @glandium wrote:

This is causing crashes when building Skia. See https://reviews.llvm.org/D141188#4179444

Sorry about that. Do you have steps to repro?

[glandium]

In D143225#4180438, @zhuhan0 wrote:

In D143225#4180342, @glandium wrote:

This is causing crashes when building Skia. See https://reviews.llvm.org/D141188#4179444

Sorry about that. Do you have steps to repro?

Here are the files that clang diagnostic created for me:
https://drive.google.com/file/d/1GAQhpwv2UYwJYdVO9L5GHLbC244W37FO/view?usp=sharing

You'll have to tweak paths in the .sh file, but that should be enough to reproduce on your end.

[zhuhan0]

In D143225#4180493, @glandium wrote:

In D143225#4180438, @zhuhan0 wrote:

In D143225#4180342, @glandium wrote:

This is causing crashes when building Skia. See https://reviews.llvm.org/D141188#4179444

Sorry about that. Do you have steps to repro?

Here are the files that clang diagnostic created for me:
https://drive.google.com/file/d/1GAQhpwv2UYwJYdVO9L5GHLbC244W37FO/view?usp=sharing

You'll have to tweak paths in the .sh file, but that should be enough to reproduce on your end.

Thanks! Fixed with https://github.com/llvm/llvm-project/commit/68a07c156e90e76a2129308fe28815e709276424.

[dyung]

Hi @zhuhan0 one of our internal tests started to fail and I bisected it back to your change. I've put details and repro up in issue 61318, can you take a look?

[dyung]

In D143225#4183616, @dyung wrote:

Hi @zhuhan0 one of our internal tests started to fail and I bisected it back to your change. I've put details and repro up in issue 61318, can you take a look?

Nevermind, I see this is the same issue I previously filed as issue #57821 which you believe to be due to undefined behavior in the spec. I'll have to discuss this further internally to see if perhaps an update to the test is warranted.

[zhuhan0]

In D143225#4183630, @dyung wrote:

In D143225#4183616, @dyung wrote:

Hi @zhuhan0 one of our internal tests started to fail and I bisected it back to your change. I've put details and repro up in issue 61318, can you take a look?

Nevermind, I see this is the same issue I previously filed as issue #57821 which you believe to be due to undefined behavior in the spec. I'll have to discuss this further internally to see if perhaps an update to the test is warranted.

Hi yes I have the investigation details in the old issue. Let me know if there's any question.

llvm/lib/Transforms/Scalar/SROA.cpp

 /// whole-vector loads and stores such that it could be promoted to a vector
 /// SSA value. We only can ensure this for a limited set of operations, and we
 /// don't want to do the rewrites unless we are confident that the result will
 /// be promotable, so we have an early test here.
 static VectorType *isVectorPromotionViable(Partition &P, const DataLayout &DL) {
   // Collect the candidate types for vector-based promotion. Also track whether
   // we have different element types.
   SmallVector<VectorType *, 4> CandidateTys;
+  SetVector<Type *> LoadStoreTys;
   Type *CommonEltTy = nullptr;
   VectorType *CommonVecPtrTy = nullptr;
   bool HaveVecPtrTy = false;
   bool HaveCommonEltTy = true;
   bool HaveCommonVecPtrTy = true;
   auto CheckCandidateType = [&](Type *Ty) {
     if (auto *VTy = dyn_cast<VectorType>(Ty)) {
       // Return if bitcast to vectors is different for total size in bits.
if (auto *VTy = dyn_cast<VectorType>(Ty)) {
         HaveVecPtrTy = true;
         if (!CommonVecPtrTy)
           CommonVecPtrTy = VTy;
         else if (CommonVecPtrTy != VTy)
           HaveCommonVecPtrTy = false;
       }
     }
   };
-  // Consider any loads or stores that are the exact size of the slice.
-  for (const Slice &S : P)
-    if (S.beginOffset() == P.beginOffset() &&
-        S.endOffset() == P.endOffset()) {
-      if (auto *LI = dyn_cast<LoadInst>(S.getUse()->getUser()))
-        CheckCandidateType(LI->getType());
-      else if (auto *SI = dyn_cast<StoreInst>(S.getUse()->getUser()))
-        CheckCandidateType(SI->getValueOperand()->getType());
-    }
+  // Put load and store types into a set for de-duplication.
+  for (const Slice &S : P) {
+    Type *Ty;
+      Ty = LI->getType();
+      Ty = SI->getValueOperand()->getType();
+    else
+      continue;
+    LoadStoreTys.insert(Ty);
+    // Consider any loads or stores that are the exact size of the slice.
+    if (S.beginOffset() == P.beginOffset() && S.endOffset() == P.endOffset())
+      CheckCandidateType(Ty);
+  }
+  // Consider additional vector types where the element type size is a
+  // multiple of load/store element size.
+  for (Type *Ty : LoadStoreTys) {
+    if (!VectorType::isValidElementType(Ty))
+      continue;
+    unsigned TypeSize = DL.getTypeSizeInBits(Ty).getFixedValue();
+    for (VectorType *&VTy : CandidateTys) {
+      unsigned VectorSize = DL.getTypeSizeInBits(VTy).getFixedValue();
+      unsigned ElementSize =
+          DL.getTypeSizeInBits(VTy->getElementType()).getFixedValue();
+      if (TypeSize != VectorSize && TypeSize != ElementSize &&
+          VectorSize % TypeSize == 0) {
+        VectorType *NewVTy = VectorType::get(Ty, VectorSize / TypeSize, false);
+        CheckCandidateType(NewVTy);
+      }
+    }
 
   // If we didn't find a vector type, nothing to do here.
   if (CandidateTys.empty())
     return nullptr;
 
   // Pointer-ness is sticky, if we had a vector-of-pointers candidate type,
   // then we should choose it, not some other alternative.
   // But, we can't perform a no-op pointer address space change via bitcast,

llvm/test/Transforms/SROA/pr57796.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-PRESERVE-CFG
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-MODIFY-CFG
+
+%struct.Value = type { %union.anon }
+%union.anon = type { <32 x i8> }
+
+@A = dso_local global i64 0, align 8
+
+; Make sure that sroa does not crash when dealing with an invalid vector
+; element type.
+define void @foo() {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[REF_TMP_I:%.*]] = alloca [[STRUCT_VALUE:%.*]], align 32
+; CHECK-NEXT:    call void @value_create(ptr sret([[STRUCT_VALUE]]) align 32 [[REF_TMP_I]])
+; CHECK-NEXT:    [[CALL_I:%.*]] = call align 32 ptr @value_set_type(ptr align 32 [[REF_TMP_I]])
+; CHECK-NEXT:    [[TMP0:%.*]] = load <32 x i8>, ptr [[CALL_I]], align 32
+; CHECK-NEXT:    [[REF_TMP_SROA_0_0_VEC_EXTRACT:%.*]] = shufflevector <32 x i8> [[TMP0]], <32 x i8> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <8 x i8> [[REF_TMP_SROA_0_0_VEC_EXTRACT]] to x86_mmx
+; CHECK-NEXT:    [[TMP2:%.*]] = call x86_mmx @llvm.x86.sse.pshuf.w(x86_mmx [[TMP1]], i8 0)
+; CHECK-NEXT:    store x86_mmx [[TMP2]], ptr @A, align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  %ref.tmp.i = alloca %struct.Value, align 32
+  %ref.tmp = alloca %struct.Value, align 32
+  call void @value_create(ptr sret(%struct.Value) align 32 %ref.tmp.i)
+  %call.i = call align 32 ptr @value_set_type(ptr align 32 %ref.tmp.i)
+  %0 = load <32 x i8>, ptr %call.i, align 32
+  store <32 x i8> %0, ptr %ref.tmp, align 32
+  %1 = load x86_mmx, ptr %ref.tmp, align 32
+  %2 = call x86_mmx @llvm.x86.sse.pshuf.w(x86_mmx %1, i8 0)
+  store x86_mmx %2, ptr @A, align 8
+  ret void
+}
+
+declare x86_mmx @llvm.x86.sse.pshuf.w(x86_mmx, i8 immarg)
+
+declare dso_local void @value_create(ptr sret(%struct.Value) align 32)
+
+declare dso_local align 32 ptr @value_set_type(ptr align 32)
+
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-MODIFY-CFG: {{.*}}
+; CHECK-PRESERVE-CFG: {{.*}}

llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll

 ; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
 ; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
 ; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %b_blockwise_copy = alloca %"struct.b", align 16
   store <8 x half> zeroinitializer, ptr %b_blockwise_copy, align 16
   %data = load <4 x float>, <4 x float>* undef
   store <4 x float> %data, ptr %b_blockwise_copy, align 16
   br label %bb
 
 bb:
   %load1 = load half, ptr %b_blockwise_copy, align 16
   %ptr2 = getelementptr inbounds i8, ptr %b_blockwise_copy, i64 2
   %load2 = load half, ptr %ptr2, align 16
   %ptr3 = getelementptr inbounds i8, ptr %b_blockwise_copy, i64 4
   %load3 = load half, ptr %ptr3, align 16
   ret void
 }
 
 define amdgpu_kernel void @test_memset() #0 {
 ; CHECK-LABEL: @test_memset(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %b_blockwise_copy = alloca %"struct.b", align 16
   call void @llvm.memset.p0.i64(ptr align 16 %b_blockwise_copy, i8 0, i64 16, i1 false)
   %data = load <4 x float>, <4 x float>* undef
   store <4 x float> %data, ptr %b_blockwise_copy, align 16
   br label %bb

 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x i32>, ptr undef, align 16
 ; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x i32> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    [[DATA2:%.*]] = load <4 x i32>, ptr undef, align 16
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <4 x i32> [[DATA2]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_4_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP1]], i32 0
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_4_16_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP1]], i32 0
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %b_blockwise_copy = alloca %"struct.f", align 16
   call void @llvm.memset.p0.i32(ptr align 16 %b_blockwise_copy, i8 0, i32 32, i1 false)
   %data = load <4 x i32>, <4 x i32>* undef
   store <4 x i32> %data, ptr %b_blockwise_copy, align 16
   %data2 = load <4 x i32>, <4 x i32>* undef

 }
 
 define amdgpu_kernel void @test_array_vector() #0 {
 ; CHECK-LABEL: @test_array_vector(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5:%.*]] = alloca <8 x half>, align 16
 ; CHECK-NEXT:    call void @llvm.memset.p0.i32(ptr align 16 [[B_BLOCKWISE_COPY_SROA_5]], i8 0, i32 16, i1 false)
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %b_blockwise_copy = alloca %"array.a", align 16
   call void @llvm.memset.p0.i32(ptr align 16 %b_blockwise_copy, i8 0, i32 32, i1 false)
   %data = load <4 x float>, <4 x float>* undef
   store <4 x float> %data, ptr %b_blockwise_copy, align 16
   br label %bb
 
 bb:
   %load1 = load half, ptr %b_blockwise_copy, align 16
   %ptr2 = getelementptr inbounds i8, ptr %b_blockwise_copy, i64 2
   %load2 = load half, ptr %ptr2, align 16
   %ptr3 = getelementptr inbounds i8, ptr %b_blockwise_copy, i64 4
   %load3 = load half, ptr %ptr3, align 16
   ret void
 }
 
 define amdgpu_kernel void @test_array_vector2() #0 {
 ; CHECK-LABEL: @test_array_vector2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_5:%.*]] = alloca <8 x half>, align 16
 ; CHECK-NEXT:    call void @llvm.memset.p0.i32(ptr align 16 [[B_BLOCKWISE_COPY_SROA_5]], i8 0, i32 16, i1 false)
 ; CHECK-NEXT:    [[DATA:%.*]] = load <4 x float>, ptr undef, align 16
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x half>
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x float> [[DATA]] to <8 x i16>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 0
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 1
-; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x half> [[TMP0]], i32 2
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_0_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_2_VEC_EXTRACT]] to half
+; CHECK-NEXT:    [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <8 x i16> [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16 [[B_BLOCKWISE_COPY_SROA_0_4_VEC_EXTRACT]] to half
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %b_blockwise_copy = alloca %"array.b", align 16
   call void @llvm.memset.p0.i32(ptr align 16 %b_blockwise_copy, i8 0, i32 32, i1 false)
   %data = load <4 x float>, <4 x float>* undef
   store <4 x float> %data, ptr %b_blockwise_copy, align 16
   br label %bb

llvm/test/Transforms/SROA/vector-promotion.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-PRESERVE-CFG
 ; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-MODIFY-CFG
 ; RUN: opt < %s -passes=debugify,sroa -S | FileCheck %s --check-prefix=DEBUG
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 %S1 = type { i64, [42 x float] }
 
 define i32 @test1(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: @test1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[A_SROA_0_8_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[X:%.*]], i32 2
;
   ret <4 x float> %vec
 }
 
 define <2 x i64> @test13(i32 %a, i32 %b, i32 %c, i32 %d) {
 ; Ensure that we can promote an alloca that needs to be
 ; cast to a different vector type
 ; CHECK-LABEL: @test13(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[X_SROA_0:%.*]] = alloca <2 x i64>, align 16
-; CHECK-NEXT:    store i32 [[A:%.*]], ptr [[X_SROA_0]], align 16
-; CHECK-NEXT:    [[X_SROA_0_4_X_TMP2_SROA_IDX1:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 4
-; CHECK-NEXT:    store i32 [[B:%.*]], ptr [[X_SROA_0_4_X_TMP2_SROA_IDX1]], align 4
-; CHECK-NEXT:    [[X_SROA_0_8_X_TMP3_SROA_IDX2:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 8
-; CHECK-NEXT:    store i32 [[C:%.*]], ptr [[X_SROA_0_8_X_TMP3_SROA_IDX2]], align 8
-; CHECK-NEXT:    [[X_SROA_0_12_X_TMP4_SROA_IDX3:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 12
-; CHECK-NEXT:    store i32 [[D:%.*]], ptr [[X_SROA_0_12_X_TMP4_SROA_IDX3]], align 4
-; CHECK-NEXT:    [[X_SROA_0_0_X_SROA_0_0_RESULT:%.*]] = load <2 x i64>, ptr [[X_SROA_0]], align 16
-; CHECK-NEXT:    ret <2 x i64> [[X_SROA_0_0_X_SROA_0_0_RESULT]]
+; CHECK-NEXT:    [[X_SROA_0_0_VEC_INSERT:%.*]] = insertelement <4 x i32> undef, i32 [[A:%.*]], i32 0
+; CHECK-NEXT:    [[X_SROA_0_4_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_0_VEC_INSERT]], i32 [[B:%.*]], i32 1
+; CHECK-NEXT:    [[X_SROA_0_8_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_4_VEC_INSERT]], i32 [[C:%.*]], i32 2
+; CHECK-NEXT:    [[X_SROA_0_12_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_8_VEC_INSERT]], i32 [[D:%.*]], i32 3
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x i32> [[X_SROA_0_12_VEC_INSERT]] to <2 x i64>
+; CHECK-NEXT:    ret <2 x i64> [[TMP0]]
 ;
 ; DEBUG-LABEL: @test13(
 ; DEBUG-NEXT:  entry:
-; DEBUG-NEXT:    [[X_SROA_0:%.*]] = alloca <2 x i64>, align 16, !dbg [[DBG354:![0-9]+]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META349:![0-9]+]], metadata !DIExpression()), !dbg [[DBG354]]
-; DEBUG-NEXT:    store i32 [[A:%.*]], ptr [[X_SROA_0]], align 16, !dbg [[DBG355:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META349:![0-9]+]], metadata !DIExpression()), !dbg [[DBG354:![0-9]+]]
+; DEBUG-NEXT:    [[X_SROA_0_0_VEC_INSERT:%.*]] = insertelement <4 x i32> undef, i32 [[A:%.*]], i32 0, !dbg [[DBG355:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META350:![0-9]+]], metadata !DIExpression()), !dbg [[DBG356:![0-9]+]]
-; DEBUG-NEXT:    [[X_SROA_0_4_X_TMP2_SROA_IDX1:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 4, !dbg [[DBG357:![0-9]+]]
-; DEBUG-NEXT:    store i32 [[B:%.*]], ptr [[X_SROA_0_4_X_TMP2_SROA_IDX1]], align 4, !dbg [[DBG357]]
+; DEBUG-NEXT:    [[X_SROA_0_4_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_0_VEC_INSERT]], i32 [[B:%.*]], i32 1, !dbg [[DBG357:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META351:![0-9]+]], metadata !DIExpression()), !dbg [[DBG358:![0-9]+]]
-; DEBUG-NEXT:    [[X_SROA_0_8_X_TMP3_SROA_IDX2:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 8, !dbg [[DBG359:![0-9]+]]
-; DEBUG-NEXT:    store i32 [[C:%.*]], ptr [[X_SROA_0_8_X_TMP3_SROA_IDX2]], align 8, !dbg [[DBG359]]
+; DEBUG-NEXT:    [[X_SROA_0_8_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_4_VEC_INSERT]], i32 [[C:%.*]], i32 2, !dbg [[DBG359:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META352:![0-9]+]], metadata !DIExpression()), !dbg [[DBG360:![0-9]+]]
-; DEBUG-NEXT:    [[X_SROA_0_12_X_TMP4_SROA_IDX3:%.*]] = getelementptr inbounds i8, ptr [[X_SROA_0]], i64 12, !dbg [[DBG361:![0-9]+]]
-; DEBUG-NEXT:    store i32 [[D:%.*]], ptr [[X_SROA_0_12_X_TMP4_SROA_IDX3]], align 4, !dbg [[DBG361]]
-; DEBUG-NEXT:    [[X_SROA_0_0_X_SROA_0_0_RESULT:%.*]] = load <2 x i64>, ptr [[X_SROA_0]], align 16, !dbg [[DBG362:![0-9]+]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata <2 x i64> [[X_SROA_0_0_X_SROA_0_0_RESULT]], metadata [[META353:![0-9]+]], metadata !DIExpression()), !dbg [[DBG362]]
-; DEBUG-NEXT:    ret <2 x i64> [[X_SROA_0_0_X_SROA_0_0_RESULT]], !dbg [[DBG363:![0-9]+]]
+; DEBUG-NEXT:    [[X_SROA_0_12_VEC_INSERT:%.*]] = insertelement <4 x i32> [[X_SROA_0_8_VEC_INSERT]], i32 [[D:%.*]], i32 3, !dbg [[DBG361:![0-9]+]]
+; DEBUG-NEXT:    [[TMP0:%.*]] = bitcast <4 x i32> [[X_SROA_0_12_VEC_INSERT]] to <2 x i64>, !dbg [[DBG362:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata <2 x i64> [[TMP0]], metadata [[META353:![0-9]+]], metadata !DIExpression()), !dbg [[DBG362]]
+; DEBUG-NEXT:    ret <2 x i64> [[TMP0]], !dbg [[DBG363:![0-9]+]]
 ;
 entry:
   %x = alloca [4 x i32]
   store i32 %a, ptr %x
   %x.tmp2 = getelementptr inbounds i32, ptr %x, i64 1
   store i32 %b, ptr %x.tmp2
   %x.tmp3 = getelementptr inbounds i32, ptr %x, i64 2
   store i32 %c, ptr %x.tmp3
   %x.tmp4 = getelementptr inbounds i32, ptr %x, i64 3
   store i32 %d, ptr %x.tmp4
   %result = load <2 x i64>, ptr %x
   ret <2 x i64> %result
 }
 
 define i32 @test14(<2 x i64> %x) {
 ; Ensure that we can promote an alloca that needs to be
 ; cast to a different vector type
 ; CHECK-LABEL: @test14(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[X_ADDR:%.*]] = alloca <2 x i64>, align 16
-; CHECK-NEXT:    store <2 x i64> [[X:%.*]], ptr [[X_ADDR]], align 16
-; CHECK-NEXT:    [[X_ADDR_0_A:%.*]] = load i32, ptr [[X_ADDR]], align 16
-; CHECK-NEXT:    [[X_ADDR_4_X_TMP2_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 4
-; CHECK-NEXT:    [[X_ADDR_4_B:%.*]] = load i32, ptr [[X_ADDR_4_X_TMP2_SROA_IDX]], align 4
-; CHECK-NEXT:    [[X_ADDR_8_X_TMP3_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 8
-; CHECK-NEXT:    [[X_ADDR_8_C:%.*]] = load i32, ptr [[X_ADDR_8_X_TMP3_SROA_IDX]], align 8
-; CHECK-NEXT:    [[X_ADDR_12_X_TMP4_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 12
-; CHECK-NEXT:    [[X_ADDR_12_D:%.*]] = load i32, ptr [[X_ADDR_12_X_TMP4_SROA_IDX]], align 4
-; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[X_ADDR_0_A]], [[X_ADDR_4_B]]
-; CHECK-NEXT:    [[ADD1:%.*]] = add i32 [[X_ADDR_8_C]], [[X_ADDR_12_D]]
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast <2 x i64> [[X:%.*]] to <4 x i32>
+; CHECK-NEXT:    [[X_ADDR_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 0
+; CHECK-NEXT:    [[X_ADDR_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 1
+; CHECK-NEXT:    [[X_ADDR_SROA_0_8_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 2
+; CHECK-NEXT:    [[X_ADDR_SROA_0_12_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 3
+; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[X_ADDR_SROA_0_0_VEC_EXTRACT]], [[X_ADDR_SROA_0_4_VEC_EXTRACT]]
+; CHECK-NEXT:    [[ADD1:%.*]] = add i32 [[X_ADDR_SROA_0_8_VEC_EXTRACT]], [[X_ADDR_SROA_0_12_VEC_EXTRACT]]
 ; CHECK-NEXT:    [[ADD2:%.*]] = add i32 [[ADD]], [[ADD1]]
 ; CHECK-NEXT:    ret i32 [[ADD2]]
 ;
 ; DEBUG-LABEL: @test14(
 ; DEBUG-NEXT:  entry:
-; DEBUG-NEXT:    [[X_ADDR:%.*]] = alloca <2 x i64>, align 16, !dbg [[DBG378:![0-9]+]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr [[X_ADDR]], metadata [[META366:![0-9]+]], metadata !DIExpression()), !dbg [[DBG378]]
-; DEBUG-NEXT:    store <2 x i64> [[X:%.*]], ptr [[X_ADDR]], align 16, !dbg [[DBG379:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META366:![0-9]+]], metadata !DIExpression()), !dbg [[DBG378:![0-9]+]]
+; DEBUG-NEXT:    [[TMP0:%.*]] = bitcast <2 x i64> [[X:%.*]] to <4 x i32>, !dbg [[DBG379:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META367:![0-9]+]], metadata !DIExpression()), !dbg [[DBG380:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_0_A:%.*]] = load i32, ptr [[X_ADDR]], align 16, !dbg [[DBG381:![0-9]+]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_0_A]], metadata [[META368:![0-9]+]], metadata !DIExpression()), !dbg [[DBG381]]
+; DEBUG-NEXT:    [[X_ADDR_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 0, !dbg [[DBG381:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_SROA_0_0_VEC_EXTRACT]], metadata [[META368:![0-9]+]], metadata !DIExpression()), !dbg [[DBG381]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META369:![0-9]+]], metadata !DIExpression()), !dbg [[DBG382:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_4_X_TMP2_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 4, !dbg [[DBG383:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_4_B:%.*]] = load i32, ptr [[X_ADDR_4_X_TMP2_SROA_IDX]], align 4, !dbg [[DBG383]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_4_B]], metadata [[META370:![0-9]+]], metadata !DIExpression()), !dbg [[DBG383]]
+; DEBUG-NEXT:    [[X_ADDR_SROA_0_4_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 1, !dbg [[DBG383:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_SROA_0_4_VEC_EXTRACT]], metadata [[META370:![0-9]+]], metadata !DIExpression()), !dbg [[DBG383]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META371:![0-9]+]], metadata !DIExpression()), !dbg [[DBG384:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_8_X_TMP3_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 8, !dbg [[DBG385:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_8_C:%.*]] = load i32, ptr [[X_ADDR_8_X_TMP3_SROA_IDX]], align 8, !dbg [[DBG385]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_8_C]], metadata [[META372:![0-9]+]], metadata !DIExpression()), !dbg [[DBG385]]
+; DEBUG-NEXT:    [[X_ADDR_SROA_0_8_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 2, !dbg [[DBG385:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_SROA_0_8_VEC_EXTRACT]], metadata [[META372:![0-9]+]], metadata !DIExpression()), !dbg [[DBG385]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata ptr undef, metadata [[META373:![0-9]+]], metadata !DIExpression()), !dbg [[DBG386:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_12_X_TMP4_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR]], i64 12, !dbg [[DBG387:![0-9]+]]
-; DEBUG-NEXT:    [[X_ADDR_12_D:%.*]] = load i32, ptr [[X_ADDR_12_X_TMP4_SROA_IDX]], align 4, !dbg [[DBG387]]
-; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_12_D]], metadata [[META374:![0-9]+]], metadata !DIExpression()), !dbg [[DBG387]]
-; DEBUG-NEXT:    [[ADD:%.*]] = add i32 [[X_ADDR_0_A]], [[X_ADDR_4_B]], !dbg [[DBG388:![0-9]+]]
+; DEBUG-NEXT:    [[X_ADDR_SROA_0_12_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[TMP0]], i32 3, !dbg [[DBG387:![0-9]+]]
+; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[X_ADDR_SROA_0_12_VEC_EXTRACT]], metadata [[META374:![0-9]+]], metadata !DIExpression()), !dbg [[DBG387]]
+; DEBUG-NEXT:    [[ADD:%.*]] = add i32 [[X_ADDR_SROA_0_0_VEC_EXTRACT]], [[X_ADDR_SROA_0_4_VEC_EXTRACT]], !dbg [[DBG388:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[ADD]], metadata [[META375:![0-9]+]], metadata !DIExpression()), !dbg [[DBG388]]
-; DEBUG-NEXT:    [[ADD1:%.*]] = add i32 [[X_ADDR_8_C]], [[X_ADDR_12_D]], !dbg [[DBG389:![0-9]+]]
+; DEBUG-NEXT:    [[ADD1:%.*]] = add i32 [[X_ADDR_SROA_0_8_VEC_EXTRACT]], [[X_ADDR_SROA_0_12_VEC_EXTRACT]], !dbg [[DBG389:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[ADD1]], metadata [[META376:![0-9]+]], metadata !DIExpression()), !dbg [[DBG389]]
 ; DEBUG-NEXT:    [[ADD2:%.*]] = add i32 [[ADD]], [[ADD1]], !dbg [[DBG390:![0-9]+]]
 ; DEBUG-NEXT:    call void @llvm.dbg.value(metadata i32 [[ADD2]], metadata [[META377:![0-9]+]], metadata !DIExpression()), !dbg [[DBG390]]
 ; DEBUG-NEXT:    ret i32 [[ADD2]], !dbg [[DBG391:![0-9]+]]
 ;
 entry:
   %x.addr = alloca <2 x i64>, align 16
   store <2 x i64> %x, <2 x i64>* %x.addr, align 16