This is an archive of the discontinued LLVM Phabricator instance.

Differential D145085

[RISCV] Lower fixed length interleaved accesses via vssegN/vlsegN
ClosedPublic

Authored by luke on Mar 1 2023, 10:33 AM.

Details

Reviewers
reames
craig.topper
luke957
Commits
rGec26c9cdc002: [RISCV] Lower fixed length interleaved accesses via vssegN/vlsegN
rGb95913e8c3a3: [RISCV] Lower fixed length interleaved accesses via vssegN/vlsegN
Summary

This enables the interleaved access pass on O1 and above, and causes
interleaving/deinterleaving shuffles of fixed length vectors with
stores/loads to be lowered into vssegN/vlsegN.

We need to be careful and make sure that we only lower vsseg/vlseg
whenever we know the fixed vector type will fit within the minimum vlen,
and that the interleaving factor is supported for the given LMUL.

Diff Detail

Event Timeline

luke created this revision.Mar 1 2023, 10:33 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 1 2023, 10:33 AM
Herald added subscribers: asb, pmatos, VincentWu and 28 others. · View Herald Transcript
luke requested review of this revision.Mar 1 2023, 10:33 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 1 2023, 10:33 AM
Herald added subscribers: llvm-commits, • pcwang-thead, eopXD, MaskRay. · View Herald Transcript
luke added a parent revision: D145022: [RISCV] Add vsseg intrinsic for fixed length vectors.Mar 1 2023, 10:33 AM
luke added inline comments.Mar 1 2023, 10:40 AM
llvm/lib/Target/RISCV/RISCVISelLowering.cpp
1805 ↗(On Diff #501571)

The idea here is that opaque pointer types will be lowered to i32 or i64 depending on the xlen, but the subtarget might only have ve32x when on rv64: Therefore ptr isn't a legal element type.
I originally intended to split this out into a separate change, but it doesn't actually affect any test cases since they seem to be covered by isTypeLegal, which is used in the gather/scatter lowering pass.

14991–15008 ↗(On Diff #501571)

The existence of the actual pseudo is checked here because we need to make sure we only lower when this condition from the spec is met:

The EMUL setting must be such that EMUL * NFIELDS ≤ 8, otherwise the instruction encoding is reserved.

It's hairy having to extract the value type, but this seems to be what the gather/scatter does here too.

Also, because we reject illegal types with !isTypeLegal(getValueType(DL, VTy)), we don't support segmented loads/stores of v3i32 etc.
Ideally we could still use them here and have them widened to v4i32 etc. later, but it's blocked on us needing the MVT to check for the existence of the pseudo.

Harbormaster completed remote builds in B216767: Diff 501571.Mar 1 2023, 11:11 AM
luke added a child revision: D145155: [RISCV] Enable interleaved access vectorization.Mar 2 2023, 4:36 AM
luke updated this revision to Diff 502083.Mar 3 2023, 2:39 AM

Add missing includes

Harbormaster completed remote builds in B217134: Diff 502083.Mar 3 2023, 3:29 AM
craig.topper added inline comments.Mar 3 2023, 10:44 AM
llvm/lib/Target/RISCV/RISCVISelLowering.cpp
14878 ↗(On Diff #502083)

Save the result of getValueType from the earlier call.

14881 ↗(On Diff #502083)

Would it be sufficient to just check LMul * Factor is <= 8 instead of looking up the pseudos?

14922 ↗(On Diff #502083)

static const?

14974 ↗(On Diff #502083)

static const

luke updated this revision to Diff 502266.Mar 3 2023, 2:43 PM

Address review comments

luke updated this revision to Diff 502269.Mar 3 2023, 2:57 PM

Remove unneeded include

Harbormaster completed remote builds in B217278: Diff 502269.Mar 3 2023, 4:51 PM
reames added inline comments.Mar 6 2023, 8:55 AM
llvm/lib/Target/RISCV/RISCVISelLowering.cpp
14878 ↗(On Diff #502269)

Ignoring fractional LMUL here is wrong. mf8 should count as LMUL1 here I believe. As you have it written here, you have it treated as LMUL8.

Also, please add a test for this case.

14879 ↗(On Diff #502269)

Style: return LMUL * Factor <= 8.

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-interleaved-access-zve32x.ll
5 ↗(On Diff #502269)

We could lower this as a wide load, plus a set of SEW=32 shuffles. I think the codegen here is distinctly sub-optimal. I'm not advocating for improving it now, but you might want to adjust the comment to be a bit more future proof.

p.s. This case is a really good catch.

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-interleaved-access-zvl32b.ll
1 ↗(On Diff #502269)

The naming on this test file doesn't make any sense to me.

If I'm gathering the intent of these tests correctly, they can be rewritten using wider types using V. I'd recommend doing so, and merging them into the base fixed-vectors-interleaved-access.ll test. If you think they really do need the zve32x, maybe into the test file named with that?

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-interleaved-access.ll
2 ↗(On Diff #502269)

Why only riscv32 here?

32 ↗(On Diff #502269)

Can you add a few more test variants here? I realize it's a bit duplicated with the transform test and the intrinsic codegen tests, but having the end to end codegen demonstrated for at least a few of the common factors seems worthwhile.

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-shufflevector-vnsrl.ll
7 ↗(On Diff #502269)

I'm concerned about the loss of test coverage here. I think it makes sense to add a runline which explicitly turns off interleave load/store formation. Or even just do so unconditionally for this file.

luke marked 3 inline comments as done.Mar 7 2023, 6:00 AM
luke updated this revision to Diff 503267.Mar 8 2023, 1:58 AM
luke marked an inline comment as done.

Address review comments

luke updated this revision to Diff 503268.Mar 8 2023, 1:59 AM

Address review comments

luke marked 6 inline comments as done.Mar 8 2023, 2:02 AM
Harbormaster completed remote builds in B218026: Diff 503268.Mar 8 2023, 2:49 AM
luke updated this revision to Diff 503441.Mar 8 2023, 10:34 AM

Update test case with codegen changes from rebase

Harbormaster completed remote builds in B218161: Diff 503441.Mar 8 2023, 12:06 PM
luke removed a child revision: D145155: [RISCV] Enable interleaved access vectorization.Mar 9 2023, 8:11 AM
luke updated this revision to Diff 506621.Mar 20 2023, 9:28 AM

Rebase and update tests

Herald added a subscriber: jobnoorman. · View Herald TranscriptMar 20 2023, 9:28 AM
Harbormaster completed remote builds in B220468: Diff 506621.Mar 20 2023, 10:04 AM
luke mentioned this in D146442: [RISCV][NFC] Make interleaved access test more vectorizable.Mar 20 2023, 10:39 AM
luke updated this revision to Diff 506653.Mar 20 2023, 11:03 AM

Remove unused argument in isLegalInterleavedAccessType and make it public, since it can be used by TTI cost model hooks later

Harbormaster completed remote builds in B220493: Diff 506653.Mar 20 2023, 12:05 PM
luke added a child revision: D146522: [RISCV] Model vlseg/vsseg in interleaved memory ops.Mar 21 2023, 6:06 AM
reames retitled this revision from [RISCV] Lower interleaved accesses to [RISCV] Lower fixed length interleaved accesses via vssegN/vlsegN.Mar 22 2023, 7:47 AM
luke mentioned this in rG06f16232b1b0: [RISCV][NFC] Make interleaved access test more vectorizable.Mar 22 2023, 9:02 AM
craig.topper added inline comments.Mar 24 2023, 5:10 PM
llvm/lib/Target/RISCV/RISCVISelLowering.cpp
14971 ↗(On Diff #506653)

You can return true for Fractional. This division can only produce 1 or 0. Factor is 2-8 and LMUL is 2, 4, or 8.

15062 ↗(On Diff #506653)

8->10 to over the maximum operands: 8 vectors + pointer + VL

llvm/lib/Target/RISCV/RISCVTargetMachine.cpp
289 ↗(On Diff #506653)

Merge these 3 ifs.

luke updated this revision to Diff 508551.Mar 27 2023, 3:01 AM

Address review comments

Harbormaster completed remote builds in B221935: Diff 508551.Mar 27 2023, 4:02 AM
luke updated this revision to Diff 509991.Mar 31 2023, 4:41 AM

Rebase

Harbormaster completed remote builds in B222983: Diff 509991.Mar 31 2023, 7:56 AM
craig.topper accepted this revision.Mar 31 2023, 9:45 PM

LGTM

This revision is now accepted and ready to land.Mar 31 2023, 9:45 PM
Closed by commit rGb95913e8c3a3: [RISCV] Lower fixed length interleaved accesses via vssegN/vlsegN (authored by luke). · Explain WhyApr 2 2023, 7:20 AM
This revision was automatically updated to reflect the committed changes.
luke added a commit: rGb95913e8c3a3: [RISCV] Lower fixed length interleaved accesses via vssegN/vlsegN.
luke added a reverting change: rG80f3be960355: Revert "[RISCV] Lower fixed length interleaved accesses via vssegN/vlsegN".Apr 2 2023, 7:56 AM
luke added a commit: rGec26c9cdc002: [RISCV] Lower fixed length interleaved accesses via vssegN/vlsegN.Apr 2 2023, 8:48 AM

Revision Contents

PathSize
llvm/
lib/
Target/
RISCV/
7 lines
12 lines
test/
Transforms/
LoopVectorize/
RISCV/
49 lines
554 lines
33 lines

Diff 503267

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

Show First 20 Lines • Show All 116 Lines • ▼ Show 20 Lines InstructionCost getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp,
ArrayRef<int> Mask, ArrayRef<int> Mask,
TTI::TargetCostKind CostKind, int Index, TTI::TargetCostKind CostKind, int Index,
VectorType *SubTp, VectorType *SubTp,
ArrayRef<const Value *> Args = std::nullopt); ArrayRef<const Value *> Args = std::nullopt);
InstructionCost getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA, InstructionCost getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
TTI::TargetCostKind CostKind); TTI::TargetCostKind CostKind);
InstructionCost getInterleavedMemoryOpCost(
unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
bool UseMaskForCond = false, bool UseMaskForGaps = false);
InstructionCost getGatherScatterOpCost(unsigned Opcode, Type *DataTy, InstructionCost getGatherScatterOpCost(unsigned Opcode, Type *DataTy,
const Value *Ptr, bool VariableMask, const Value *Ptr, bool VariableMask,
Align Alignment, Align Alignment,
TTI::TargetCostKind CostKind, TTI::TargetCostKind CostKind,
const Instruction *I); const Instruction *I);
InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src, InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
TTI::CastContextHint CCH, TTI::CastContextHint CCH,
▲ Show 20 Lines • Show All 142 Lines • ▼ Show 20 Lines unsigned getMaxInterleaveFactor(ElementCount VF) {
// Don't interleave if the loop has been vectorized with scalable vectors. // Don't interleave if the loop has been vectorized with scalable vectors.
if (VF.isScalable()) if (VF.isScalable())
return 1; return 1;
// If the loop will not be vectorized, don't interleave the loop. // If the loop will not be vectorized, don't interleave the loop.
// Let regular unroll to unroll the loop. // Let regular unroll to unroll the loop.
return VF.isScalar() ? 1 : ST->getMaxInterleaveFactor(); return VF.isScalar() ? 1 : ST->getMaxInterleaveFactor();
} }
bool enableInterleavedAccessVectorization() { return true; }
enum RISCVRegisterClass { GPRRC, FPRRC, VRRC }; enum RISCVRegisterClass { GPRRC, FPRRC, VRRC };
unsigned getNumberOfRegisters(unsigned ClassID) const { unsigned getNumberOfRegisters(unsigned ClassID) const {
switch (ClassID) { switch (ClassID) {
case RISCVRegisterClass::GPRRC: case RISCVRegisterClass::GPRRC:
// 31 = 32 GPR - x0 (zero register) // 31 = 32 GPR - x0 (zero register)
// FIXME: Should we exclude fixed registers like SP, TP or GP? // FIXME: Should we exclude fixed registers like SP, TP or GP?
return 31; return 31;
case RISCVRegisterClass::FPRRC: case RISCVRegisterClass::FPRRC:
▲ Show 20 Lines • Show All 48 Lines • Show Last 20 Lines

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp

Show First 20 Lines • Show All 332 Lines • ▼ Show 20 LinesRISCVTTIImpl::getMaskedMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
if (!isLegalMaskedLoadStore(Src, Alignment) || if (!isLegalMaskedLoadStore(Src, Alignment) ||
CostKind != TTI::TCK_RecipThroughput) CostKind != TTI::TCK_RecipThroughput)
return BaseT::getMaskedMemoryOpCost(Opcode, Src, Alignment, AddressSpace, return BaseT::getMaskedMemoryOpCost(Opcode, Src, Alignment, AddressSpace,
CostKind); CostKind);
return getMemoryOpCost(Opcode, Src, Alignment, AddressSpace, CostKind); return getMemoryOpCost(Opcode, Src, Alignment, AddressSpace, CostKind);
} }
InstructionCost RISCVTTIImpl::getInterleavedMemoryOpCost(
unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
bool UseMaskForCond, bool UseMaskForGaps) {
if (Factor <= TLI->getMaxSupportedInterleaveFactor() &&
CostKind == TTI::TCK_RecipThroughput)
return Factor;
return BaseT::getInterleavedMemoryOpCost(Opcode, VecTy, Factor, Indices,
Alignment, AddressSpace, CostKind,
UseMaskForCond, UseMaskForGaps);
}
InstructionCost RISCVTTIImpl::getGatherScatterOpCost( InstructionCost RISCVTTIImpl::getGatherScatterOpCost(
unsigned Opcode, Type *DataTy, const Value *Ptr, bool VariableMask, unsigned Opcode, Type *DataTy, const Value *Ptr, bool VariableMask,
Align Alignment, TTI::TargetCostKind CostKind, const Instruction *I) { Align Alignment, TTI::TargetCostKind CostKind, const Instruction *I) {
if (CostKind != TTI::TCK_RecipThroughput) if (CostKind != TTI::TCK_RecipThroughput)
return BaseT::getGatherScatterOpCost(Opcode, DataTy, Ptr, VariableMask, return BaseT::getGatherScatterOpCost(Opcode, DataTy, Ptr, VariableMask,
Alignment, CostKind, I); Alignment, CostKind, I);
if ((Opcode == Instruction::Load && if ((Opcode == Instruction::Load &&
▲ Show 20 Lines • Show All 1,146 Lines • Show Last 20 Lines

llvm/test/Transforms/LoopVectorize/RISCV/interleaved-accesses-expensive.ll

  • This file was added.
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
; RUN: opt < %s -passes=loop-vectorize -mtriple=riscv64 -mattr=+zve32x,+zvl1024b -S | FileCheck %s
; This element type isn't a supported SEW so this shouldn't be interleaved
define void @load_store_expensive(ptr %p) {
; CHECK-LABEL: @load_store_expensive(
; CHECK-NEXT: entry:
; CHECK-NEXT: br label [[LOOP:%.*]]
; CHECK: loop:
; CHECK-NEXT: [[I:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
; CHECK-NEXT: [[OFFSET0:%.*]] = shl i64 [[I]], 1
; CHECK-NEXT: [[Q0:%.*]] = getelementptr i64, ptr [[P:%.*]], i64 [[OFFSET0]]
; CHECK-NEXT: [[X0:%.*]] = load i64, ptr [[Q0]], align 4
; CHECK-NEXT: [[Y0:%.*]] = add i64 [[X0]], 1
; CHECK-NEXT: store i64 [[Y0]], ptr [[Q0]], align 4
; CHECK-NEXT: [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
; CHECK-NEXT: [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
; CHECK-NEXT: [[X1:%.*]] = load i64, ptr [[Q1]], align 4
; CHECK-NEXT: [[Y1:%.*]] = add i64 [[X1]], 2
; CHECK-NEXT: store i64 [[Y1]], ptr [[Q1]], align 4
; CHECK-NEXT: [[NEXTI]] = add i64 [[I]], 1
; CHECK-NEXT: [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
; CHECK-NEXT: br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
; CHECK: exit:
; CHECK-NEXT: ret void
;
entry:
br label %loop
loop:
%i = phi i64 [0, %entry], [%nexti, %loop]
%offset0 = shl i64 %i, 1
%q0 = getelementptr i64, ptr %p, i64 %offset0
%x0 = load i64, ptr %q0
%y0 = add i64 %x0, 1
store i64 %y0, ptr %q0
%offset1 = add i64 %offset0, 1
%q1 = getelementptr i64, ptr %p, i64 %offset1
%x1 = load i64, ptr %q1
%y1 = add i64 %x1, 2
store i64 %y1, ptr %q1
%nexti = add i64 %i, 1
%done = icmp eq i64 %nexti, 1024
br i1 %done, label %exit, label %loop
exit:
ret void
}

llvm/test/Transforms/LoopVectorize/RISCV/interleaved-accesses.ll

  • This file was added.
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
; RUN: opt < %s -passes=loop-vectorize -mtriple=riscv64 -mattr=+v -S | FileCheck %s
target triple = "riscv64"
define void @load_store_factor2(ptr %p) {
; CHECK-LABEL: @load_store_factor2(
; CHECK-NEXT: entry:
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_SCEVCHECK:%.*]]
; CHECK: vector.scevcheck:
; CHECK-NEXT: [[UGLYGEP:%.*]] = getelementptr i8, ptr [[P:%.*]], i64 8
; CHECK-NEXT: [[MUL:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 16, i64 1023)
; CHECK-NEXT: [[MUL_RESULT:%.*]] = extractvalue { i64, i1 } [[MUL]], 0
; CHECK-NEXT: [[MUL_OVERFLOW:%.*]] = extractvalue { i64, i1 } [[MUL]], 1
; CHECK-NEXT: [[TMP0:%.*]] = sub i64 0, [[MUL_RESULT]]
; CHECK-NEXT: [[TMP1:%.*]] = getelementptr i8, ptr [[UGLYGEP]], i64 [[MUL_RESULT]]
; CHECK-NEXT: [[TMP2:%.*]] = icmp ult ptr [[TMP1]], [[UGLYGEP]]
; CHECK-NEXT: [[TMP3:%.*]] = or i1 [[TMP2]], [[MUL_OVERFLOW]]
; CHECK-NEXT: [[MUL1:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 16, i64 1023)
; CHECK-NEXT: [[MUL_RESULT2:%.*]] = extractvalue { i64, i1 } [[MUL1]], 0
; CHECK-NEXT: [[MUL_OVERFLOW3:%.*]] = extractvalue { i64, i1 } [[MUL1]], 1
; CHECK-NEXT: [[TMP4:%.*]] = sub i64 0, [[MUL_RESULT2]]
; CHECK-NEXT: [[TMP5:%.*]] = getelementptr i8, ptr [[P]], i64 [[MUL_RESULT2]]
; CHECK-NEXT: [[TMP6:%.*]] = icmp ult ptr [[TMP5]], [[P]]
; CHECK-NEXT: [[TMP7:%.*]] = or i1 [[TMP6]], [[MUL_OVERFLOW3]]
; CHECK-NEXT: [[TMP8:%.*]] = or i1 [[TMP3]], [[TMP7]]
; CHECK-NEXT: br i1 [[TMP8]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP9:%.*]] = add i64 [[INDEX]], 0
; CHECK-NEXT: [[TMP10:%.*]] = shl i64 [[TMP9]], 1
; CHECK-NEXT: [[TMP11:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP10]]
; CHECK-NEXT: [[TMP12:%.*]] = getelementptr i64, ptr [[TMP11]], i32 0
; CHECK-NEXT: [[WIDE_VEC:%.*]] = load <4 x i64>, ptr [[TMP12]], align 4
; CHECK-NEXT: [[STRIDED_VEC:%.*]] = shufflevector <4 x i64> [[WIDE_VEC]], <4 x i64> poison, <2 x i32> <i32 0, i32 2>
; CHECK-NEXT: [[STRIDED_VEC4:%.*]] = shufflevector <4 x i64> [[WIDE_VEC]], <4 x i64> poison, <2 x i32> <i32 1, i32 3>
; CHECK-NEXT: [[TMP13:%.*]] = add <2 x i64> [[STRIDED_VEC]], <i64 1, i64 1>
; CHECK-NEXT: [[TMP14:%.*]] = add i64 [[TMP10]], 1
; CHECK-NEXT: [[TMP15:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP14]]
; CHECK-NEXT: [[TMP16:%.*]] = add <2 x i64> [[STRIDED_VEC4]], <i64 2, i64 2>
; CHECK-NEXT: [[TMP17:%.*]] = getelementptr i64, ptr [[TMP15]], i32 -1
; CHECK-NEXT: [[TMP18:%.*]] = shufflevector <2 x i64> [[TMP13]], <2 x i64> [[TMP16]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[INTERLEAVED_VEC:%.*]] = shufflevector <4 x i64> [[TMP18]], <4 x i64> poison, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
; CHECK-NEXT: store <4 x i64> [[INTERLEAVED_VEC]], ptr [[TMP17]], align 4
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 2
; CHECK-NEXT: [[TMP19:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024
; CHECK-NEXT: br i1 [[TMP19]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 1024, 1024
; CHECK-NEXT: br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 1024, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_SCEVCHECK]] ]
; CHECK-NEXT: br label [[LOOP:%.*]]
; CHECK: loop:
; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
; CHECK-NEXT: [[OFFSET0:%.*]] = shl i64 [[I]], 1
; CHECK-NEXT: [[Q0:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET0]]
; CHECK-NEXT: [[X0:%.*]] = load i64, ptr [[Q0]], align 4
; CHECK-NEXT: [[Y0:%.*]] = add i64 [[X0]], 1
; CHECK-NEXT: store i64 [[Y0]], ptr [[Q0]], align 4
; CHECK-NEXT: [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
; CHECK-NEXT: [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
; CHECK-NEXT: [[X1:%.*]] = load i64, ptr [[Q1]], align 4
; CHECK-NEXT: [[Y1:%.*]] = add i64 [[X1]], 2
; CHECK-NEXT: store i64 [[Y1]], ptr [[Q1]], align 4
; CHECK-NEXT: [[NEXTI]] = add i64 [[I]], 1
; CHECK-NEXT: [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
; CHECK-NEXT: br i1 [[DONE]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP3:![0-9]+]]
; CHECK: exit:
; CHECK-NEXT: ret void
;
entry:
br label %loop
loop:
%i = phi i64 [0, %entry], [%nexti, %loop]
%offset0 = shl i64 %i, 1
%q0 = getelementptr i64, ptr %p, i64 %offset0
%x0 = load i64, ptr %q0
%y0 = add i64 %x0, 1
store i64 %y0, ptr %q0
%offset1 = add i64 %offset0, 1
%q1 = getelementptr i64, ptr %p, i64 %offset1
%x1 = load i64, ptr %q1
%y1 = add i64 %x1, 2
store i64 %y1, ptr %q1
%nexti = add i64 %i, 1
%done = icmp eq i64 %nexti, 1024
br i1 %done, label %exit, label %loop
exit:
ret void
}
define void @load_store_factor3(ptr %p) {
; CHECK-LABEL: @load_store_factor3(
; CHECK-NEXT: entry:
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_SCEVCHECK:%.*]]
; CHECK: vector.scevcheck:
; CHECK-NEXT: [[UGLYGEP:%.*]] = getelementptr i8, ptr [[P:%.*]], i64 16
; CHECK-NEXT: [[MUL:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 24, i64 1023)
; CHECK-NEXT: [[MUL_RESULT:%.*]] = extractvalue { i64, i1 } [[MUL]], 0
; CHECK-NEXT: [[MUL_OVERFLOW:%.*]] = extractvalue { i64, i1 } [[MUL]], 1
; CHECK-NEXT: [[TMP0:%.*]] = sub i64 0, [[MUL_RESULT]]
; CHECK-NEXT: [[TMP1:%.*]] = getelementptr i8, ptr [[UGLYGEP]], i64 [[MUL_RESULT]]
; CHECK-NEXT: [[TMP2:%.*]] = icmp ult ptr [[TMP1]], [[UGLYGEP]]
; CHECK-NEXT: [[TMP3:%.*]] = or i1 [[TMP2]], [[MUL_OVERFLOW]]
; CHECK-NEXT: [[UGLYGEP1:%.*]] = getelementptr i8, ptr [[P]], i64 8
; CHECK-NEXT: [[MUL2:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 24, i64 1023)
; CHECK-NEXT: [[MUL_RESULT3:%.*]] = extractvalue { i64, i1 } [[MUL2]], 0
; CHECK-NEXT: [[MUL_OVERFLOW4:%.*]] = extractvalue { i64, i1 } [[MUL2]], 1
; CHECK-NEXT: [[TMP4:%.*]] = sub i64 0, [[MUL_RESULT3]]
; CHECK-NEXT: [[TMP5:%.*]] = getelementptr i8, ptr [[UGLYGEP1]], i64 [[MUL_RESULT3]]
; CHECK-NEXT: [[TMP6:%.*]] = icmp ult ptr [[TMP5]], [[UGLYGEP1]]
; CHECK-NEXT: [[TMP7:%.*]] = or i1 [[TMP6]], [[MUL_OVERFLOW4]]
; CHECK-NEXT: [[MUL5:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 24, i64 1023)
; CHECK-NEXT: [[MUL_RESULT6:%.*]] = extractvalue { i64, i1 } [[MUL5]], 0
; CHECK-NEXT: [[MUL_OVERFLOW7:%.*]] = extractvalue { i64, i1 } [[MUL5]], 1
; CHECK-NEXT: [[TMP8:%.*]] = sub i64 0, [[MUL_RESULT6]]
; CHECK-NEXT: [[TMP9:%.*]] = getelementptr i8, ptr [[P]], i64 [[MUL_RESULT6]]
; CHECK-NEXT: [[TMP10:%.*]] = icmp ult ptr [[TMP9]], [[P]]
; CHECK-NEXT: [[TMP11:%.*]] = or i1 [[TMP10]], [[MUL_OVERFLOW7]]
; CHECK-NEXT: [[TMP12:%.*]] = or i1 [[TMP3]], [[TMP7]]
; CHECK-NEXT: [[TMP13:%.*]] = or i1 [[TMP12]], [[TMP11]]
; CHECK-NEXT: br i1 [[TMP13]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP14:%.*]] = add i64 [[INDEX]], 0
; CHECK-NEXT: [[TMP15:%.*]] = mul i64 [[TMP14]], 3
; CHECK-NEXT: [[TMP16:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP15]]
; CHECK-NEXT: [[TMP17:%.*]] = getelementptr i64, ptr [[TMP16]], i32 0
; CHECK-NEXT: [[WIDE_VEC:%.*]] = load <6 x i64>, ptr [[TMP17]], align 4
; CHECK-NEXT: [[STRIDED_VEC:%.*]] = shufflevector <6 x i64> [[WIDE_VEC]], <6 x i64> poison, <2 x i32> <i32 0, i32 3>
; CHECK-NEXT: [[STRIDED_VEC8:%.*]] = shufflevector <6 x i64> [[WIDE_VEC]], <6 x i64> poison, <2 x i32> <i32 1, i32 4>
; CHECK-NEXT: [[STRIDED_VEC9:%.*]] = shufflevector <6 x i64> [[WIDE_VEC]], <6 x i64> poison, <2 x i32> <i32 2, i32 5>
; CHECK-NEXT: [[TMP18:%.*]] = add <2 x i64> [[STRIDED_VEC]], <i64 1, i64 1>
; CHECK-NEXT: [[TMP19:%.*]] = add i64 [[TMP15]], 1
; CHECK-NEXT: [[TMP20:%.*]] = add <2 x i64> [[STRIDED_VEC8]], <i64 2, i64 2>
; CHECK-NEXT: [[TMP21:%.*]] = add i64 [[TMP19]], 1
; CHECK-NEXT: [[TMP22:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP21]]
; CHECK-NEXT: [[TMP23:%.*]] = add <2 x i64> [[STRIDED_VEC9]], <i64 3, i64 3>
; CHECK-NEXT: [[TMP24:%.*]] = getelementptr i64, ptr [[TMP22]], i32 -2
; CHECK-NEXT: [[TMP25:%.*]] = shufflevector <2 x i64> [[TMP18]], <2 x i64> [[TMP20]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP26:%.*]] = shufflevector <2 x i64> [[TMP23]], <2 x i64> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
; CHECK-NEXT: [[TMP27:%.*]] = shufflevector <4 x i64> [[TMP25]], <4 x i64> [[TMP26]], <6 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5>
; CHECK-NEXT: [[INTERLEAVED_VEC:%.*]] = shufflevector <6 x i64> [[TMP27]], <6 x i64> poison, <6 x i32> <i32 0, i32 2, i32 4, i32 1, i32 3, i32 5>
; CHECK-NEXT: store <6 x i64> [[INTERLEAVED_VEC]], ptr [[TMP24]], align 4
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 2
; CHECK-NEXT: [[TMP28:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024
; CHECK-NEXT: br i1 [[TMP28]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 1024, 1024
; CHECK-NEXT: br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 1024, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_SCEVCHECK]] ]
; CHECK-NEXT: br label [[LOOP:%.*]]
; CHECK: loop:
; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
; CHECK-NEXT: [[OFFSET0:%.*]] = mul i64 [[I]], 3
; CHECK-NEXT: [[Q0:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET0]]
; CHECK-NEXT: [[X0:%.*]] = load i64, ptr [[Q0]], align 4
; CHECK-NEXT: [[Y0:%.*]] = add i64 [[X0]], 1
; CHECK-NEXT: store i64 [[Y0]], ptr [[Q0]], align 4
; CHECK-NEXT: [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
; CHECK-NEXT: [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
; CHECK-NEXT: [[X1:%.*]] = load i64, ptr [[Q1]], align 4
; CHECK-NEXT: [[Y1:%.*]] = add i64 [[X1]], 2
; CHECK-NEXT: store i64 [[Y1]], ptr [[Q1]], align 4
; CHECK-NEXT: [[OFFSET2:%.*]] = add i64 [[OFFSET1]], 1
; CHECK-NEXT: [[Q2:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET2]]
; CHECK-NEXT: [[X2:%.*]] = load i64, ptr [[Q2]], align 4
; CHECK-NEXT: [[Y2:%.*]] = add i64 [[X2]], 3
; CHECK-NEXT: store i64 [[Y2]], ptr [[Q2]], align 4
; CHECK-NEXT: [[NEXTI]] = add i64 [[I]], 1
; CHECK-NEXT: [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
; CHECK-NEXT: br i1 [[DONE]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP5:![0-9]+]]
; CHECK: exit:
; CHECK-NEXT: ret void
;
entry:
br label %loop
loop:
%i = phi i64 [0, %entry], [%nexti, %loop]
%offset0 = mul i64 %i, 3
%q0 = getelementptr i64, ptr %p, i64 %offset0
%x0 = load i64, ptr %q0
%y0 = add i64 %x0, 1
store i64 %y0, ptr %q0
%offset1 = add i64 %offset0, 1
%q1 = getelementptr i64, ptr %p, i64 %offset1
%x1 = load i64, ptr %q1
%y1 = add i64 %x1, 2
store i64 %y1, ptr %q1
%offset2 = add i64 %offset1, 1
%q2 = getelementptr i64, ptr %p, i64 %offset2
%x2 = load i64, ptr %q2
%y2 = add i64 %x2, 3
store i64 %y2, ptr %q2
%nexti = add i64 %i, 1
%done = icmp eq i64 %nexti, 1024
br i1 %done, label %exit, label %loop
exit:
ret void
}
define void @load_store_factor8(ptr %p) {
; CHECK-LABEL: @load_store_factor8(
; CHECK-NEXT: entry:
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_SCEVCHECK:%.*]]
; CHECK: vector.scevcheck:
; CHECK-NEXT: [[UGLYGEP:%.*]] = getelementptr i8, ptr [[P:%.*]], i64 56
; CHECK-NEXT: [[MUL:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 64, i64 1023)
; CHECK-NEXT: [[MUL_RESULT:%.*]] = extractvalue { i64, i1 } [[MUL]], 0
; CHECK-NEXT: [[MUL_OVERFLOW:%.*]] = extractvalue { i64, i1 } [[MUL]], 1
; CHECK-NEXT: [[TMP0:%.*]] = sub i64 0, [[MUL_RESULT]]
; CHECK-NEXT: [[TMP1:%.*]] = getelementptr i8, ptr [[UGLYGEP]], i64 [[MUL_RESULT]]
; CHECK-NEXT: [[TMP2:%.*]] = icmp ult ptr [[TMP1]], [[UGLYGEP]]
; CHECK-NEXT: [[TMP3:%.*]] = or i1 [[TMP2]], [[MUL_OVERFLOW]]
; CHECK-NEXT: [[UGLYGEP1:%.*]] = getelementptr i8, ptr [[P]], i64 48
; CHECK-NEXT: [[MUL2:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 64, i64 1023)
; CHECK-NEXT: [[MUL_RESULT3:%.*]] = extractvalue { i64, i1 } [[MUL2]], 0
; CHECK-NEXT: [[MUL_OVERFLOW4:%.*]] = extractvalue { i64, i1 } [[MUL2]], 1
; CHECK-NEXT: [[TMP4:%.*]] = sub i64 0, [[MUL_RESULT3]]
; CHECK-NEXT: [[TMP5:%.*]] = getelementptr i8, ptr [[UGLYGEP1]], i64 [[MUL_RESULT3]]
; CHECK-NEXT: [[TMP6:%.*]] = icmp ult ptr [[TMP5]], [[UGLYGEP1]]
; CHECK-NEXT: [[TMP7:%.*]] = or i1 [[TMP6]], [[MUL_OVERFLOW4]]
; CHECK-NEXT: [[UGLYGEP5:%.*]] = getelementptr i8, ptr [[P]], i64 40
; CHECK-NEXT: [[MUL6:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 64, i64 1023)
; CHECK-NEXT: [[MUL_RESULT7:%.*]] = extractvalue { i64, i1 } [[MUL6]], 0
; CHECK-NEXT: [[MUL_OVERFLOW8:%.*]] = extractvalue { i64, i1 } [[MUL6]], 1
; CHECK-NEXT: [[TMP8:%.*]] = sub i64 0, [[MUL_RESULT7]]
; CHECK-NEXT: [[TMP9:%.*]] = getelementptr i8, ptr [[UGLYGEP5]], i64 [[MUL_RESULT7]]
; CHECK-NEXT: [[TMP10:%.*]] = icmp ult ptr [[TMP9]], [[UGLYGEP5]]
; CHECK-NEXT: [[TMP11:%.*]] = or i1 [[TMP10]], [[MUL_OVERFLOW8]]
; CHECK-NEXT: [[UGLYGEP9:%.*]] = getelementptr i8, ptr [[P]], i64 32
; CHECK-NEXT: [[MUL10:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 64, i64 1023)
; CHECK-NEXT: [[MUL_RESULT11:%.*]] = extractvalue { i64, i1 } [[MUL10]], 0
; CHECK-NEXT: [[MUL_OVERFLOW12:%.*]] = extractvalue { i64, i1 } [[MUL10]], 1
; CHECK-NEXT: [[TMP12:%.*]] = sub i64 0, [[MUL_RESULT11]]
; CHECK-NEXT: [[TMP13:%.*]] = getelementptr i8, ptr [[UGLYGEP9]], i64 [[MUL_RESULT11]]
; CHECK-NEXT: [[TMP14:%.*]] = icmp ult ptr [[TMP13]], [[UGLYGEP9]]
; CHECK-NEXT: [[TMP15:%.*]] = or i1 [[TMP14]], [[MUL_OVERFLOW12]]
; CHECK-NEXT: [[UGLYGEP13:%.*]] = getelementptr i8, ptr [[P]], i64 24
; CHECK-NEXT: [[MUL14:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 64, i64 1023)
; CHECK-NEXT: [[MUL_RESULT15:%.*]] = extractvalue { i64, i1 } [[MUL14]], 0
; CHECK-NEXT: [[MUL_OVERFLOW16:%.*]] = extractvalue { i64, i1 } [[MUL14]], 1
; CHECK-NEXT: [[TMP16:%.*]] = sub i64 0, [[MUL_RESULT15]]
; CHECK-NEXT: [[TMP17:%.*]] = getelementptr i8, ptr [[UGLYGEP13]], i64 [[MUL_RESULT15]]
; CHECK-NEXT: [[TMP18:%.*]] = icmp ult ptr [[TMP17]], [[UGLYGEP13]]
; CHECK-NEXT: [[TMP19:%.*]] = or i1 [[TMP18]], [[MUL_OVERFLOW16]]
; CHECK-NEXT: [[UGLYGEP17:%.*]] = getelementptr i8, ptr [[P]], i64 16
; CHECK-NEXT: [[MUL18:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 64, i64 1023)
; CHECK-NEXT: [[MUL_RESULT19:%.*]] = extractvalue { i64, i1 } [[MUL18]], 0
; CHECK-NEXT: [[MUL_OVERFLOW20:%.*]] = extractvalue { i64, i1 } [[MUL18]], 1
; CHECK-NEXT: [[TMP20:%.*]] = sub i64 0, [[MUL_RESULT19]]
; CHECK-NEXT: [[TMP21:%.*]] = getelementptr i8, ptr [[UGLYGEP17]], i64 [[MUL_RESULT19]]
; CHECK-NEXT: [[TMP22:%.*]] = icmp ult ptr [[TMP21]], [[UGLYGEP17]]
; CHECK-NEXT: [[TMP23:%.*]] = or i1 [[TMP22]], [[MUL_OVERFLOW20]]
; CHECK-NEXT: [[UGLYGEP21:%.*]] = getelementptr i8, ptr [[P]], i64 8
; CHECK-NEXT: [[MUL22:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 64, i64 1023)
; CHECK-NEXT: [[MUL_RESULT23:%.*]] = extractvalue { i64, i1 } [[MUL22]], 0
; CHECK-NEXT: [[MUL_OVERFLOW24:%.*]] = extractvalue { i64, i1 } [[MUL22]], 1
; CHECK-NEXT: [[TMP24:%.*]] = sub i64 0, [[MUL_RESULT23]]
; CHECK-NEXT: [[TMP25:%.*]] = getelementptr i8, ptr [[UGLYGEP21]], i64 [[MUL_RESULT23]]
; CHECK-NEXT: [[TMP26:%.*]] = icmp ult ptr [[TMP25]], [[UGLYGEP21]]
; CHECK-NEXT: [[TMP27:%.*]] = or i1 [[TMP26]], [[MUL_OVERFLOW24]]
; CHECK-NEXT: [[MUL25:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 64, i64 1023)
; CHECK-NEXT: [[MUL_RESULT26:%.*]] = extractvalue { i64, i1 } [[MUL25]], 0
; CHECK-NEXT: [[MUL_OVERFLOW27:%.*]] = extractvalue { i64, i1 } [[MUL25]], 1
; CHECK-NEXT: [[TMP28:%.*]] = sub i64 0, [[MUL_RESULT26]]
; CHECK-NEXT: [[TMP29:%.*]] = getelementptr i8, ptr [[P]], i64 [[MUL_RESULT26]]
; CHECK-NEXT: [[TMP30:%.*]] = icmp ult ptr [[TMP29]], [[P]]
; CHECK-NEXT: [[TMP31:%.*]] = or i1 [[TMP30]], [[MUL_OVERFLOW27]]
; CHECK-NEXT: [[TMP32:%.*]] = or i1 [[TMP3]], [[TMP7]]
; CHECK-NEXT: [[TMP33:%.*]] = or i1 [[TMP32]], [[TMP11]]
; CHECK-NEXT: [[TMP34:%.*]] = or i1 [[TMP33]], [[TMP15]]
; CHECK-NEXT: [[TMP35:%.*]] = or i1 [[TMP34]], [[TMP19]]
; CHECK-NEXT: [[TMP36:%.*]] = or i1 [[TMP35]], [[TMP23]]
; CHECK-NEXT: [[TMP37:%.*]] = or i1 [[TMP36]], [[TMP27]]
; CHECK-NEXT: [[TMP38:%.*]] = or i1 [[TMP37]], [[TMP31]]
; CHECK-NEXT: br i1 [[TMP38]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP39:%.*]] = add i64 [[INDEX]], 0
; CHECK-NEXT: [[TMP40:%.*]] = shl i64 [[TMP39]], 3
; CHECK-NEXT: [[TMP41:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP40]]
; CHECK-NEXT: [[TMP42:%.*]] = getelementptr i64, ptr [[TMP41]], i32 0
; CHECK-NEXT: [[WIDE_VEC:%.*]] = load <16 x i64>, ptr [[TMP42]], align 4
; CHECK-NEXT: [[STRIDED_VEC:%.*]] = shufflevector <16 x i64> [[WIDE_VEC]], <16 x i64> poison, <2 x i32> <i32 0, i32 8>
; CHECK-NEXT: [[STRIDED_VEC28:%.*]] = shufflevector <16 x i64> [[WIDE_VEC]], <16 x i64> poison, <2 x i32> <i32 1, i32 9>
; CHECK-NEXT: [[STRIDED_VEC29:%.*]] = shufflevector <16 x i64> [[WIDE_VEC]], <16 x i64> poison, <2 x i32> <i32 2, i32 10>
; CHECK-NEXT: [[STRIDED_VEC30:%.*]] = shufflevector <16 x i64> [[WIDE_VEC]], <16 x i64> poison, <2 x i32> <i32 3, i32 11>
; CHECK-NEXT: [[STRIDED_VEC31:%.*]] = shufflevector <16 x i64> [[WIDE_VEC]], <16 x i64> poison, <2 x i32> <i32 4, i32 12>
; CHECK-NEXT: [[STRIDED_VEC32:%.*]] = shufflevector <16 x i64> [[WIDE_VEC]], <16 x i64> poison, <2 x i32> <i32 5, i32 13>
; CHECK-NEXT: [[STRIDED_VEC33:%.*]] = shufflevector <16 x i64> [[WIDE_VEC]], <16 x i64> poison, <2 x i32> <i32 6, i32 14>
; CHECK-NEXT: [[STRIDED_VEC34:%.*]] = shufflevector <16 x i64> [[WIDE_VEC]], <16 x i64> poison, <2 x i32> <i32 7, i32 15>
; CHECK-NEXT: [[TMP43:%.*]] = add <2 x i64> [[STRIDED_VEC]], <i64 1, i64 1>
; CHECK-NEXT: [[TMP44:%.*]] = add i64 [[TMP40]], 1
; CHECK-NEXT: [[TMP45:%.*]] = add <2 x i64> [[STRIDED_VEC28]], <i64 2, i64 2>
; CHECK-NEXT: [[TMP46:%.*]] = add i64 [[TMP44]], 1
; CHECK-NEXT: [[TMP47:%.*]] = add <2 x i64> [[STRIDED_VEC29]], <i64 3, i64 3>
; CHECK-NEXT: [[TMP48:%.*]] = add i64 [[TMP46]], 1
; CHECK-NEXT: [[TMP49:%.*]] = add <2 x i64> [[STRIDED_VEC30]], <i64 4, i64 4>
; CHECK-NEXT: [[TMP50:%.*]] = add i64 [[TMP48]], 1
; CHECK-NEXT: [[TMP51:%.*]] = add <2 x i64> [[STRIDED_VEC31]], <i64 5, i64 5>
; CHECK-NEXT: [[TMP52:%.*]] = add i64 [[TMP50]], 1
; CHECK-NEXT: [[TMP53:%.*]] = add <2 x i64> [[STRIDED_VEC32]], <i64 6, i64 6>
; CHECK-NEXT: [[TMP54:%.*]] = add i64 [[TMP52]], 1
; CHECK-NEXT: [[TMP55:%.*]] = add <2 x i64> [[STRIDED_VEC33]], <i64 7, i64 7>
; CHECK-NEXT: [[TMP56:%.*]] = add i64 [[TMP54]], 1
; CHECK-NEXT: [[TMP57:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP56]]
; CHECK-NEXT: [[TMP58:%.*]] = add <2 x i64> [[STRIDED_VEC34]], <i64 8, i64 8>
; CHECK-NEXT: [[TMP59:%.*]] = getelementptr i64, ptr [[TMP57]], i32 -7
; CHECK-NEXT: [[TMP60:%.*]] = shufflevector <2 x i64> [[TMP43]], <2 x i64> [[TMP45]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP61:%.*]] = shufflevector <2 x i64> [[TMP47]], <2 x i64> [[TMP49]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP62:%.*]] = shufflevector <2 x i64> [[TMP51]], <2 x i64> [[TMP53]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP63:%.*]] = shufflevector <2 x i64> [[TMP55]], <2 x i64> [[TMP58]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP64:%.*]] = shufflevector <4 x i64> [[TMP60]], <4 x i64> [[TMP61]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: [[TMP65:%.*]] = shufflevector <4 x i64> [[TMP62]], <4 x i64> [[TMP63]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: [[TMP66:%.*]] = shufflevector <8 x i64> [[TMP64]], <8 x i64> [[TMP65]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
; CHECK-NEXT: [[INTERLEAVED_VEC:%.*]] = shufflevector <16 x i64> [[TMP66]], <16 x i64> poison, <16 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14, i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
; CHECK-NEXT: store <16 x i64> [[INTERLEAVED_VEC]], ptr [[TMP59]], align 4
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 2
; CHECK-NEXT: [[TMP67:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024
; CHECK-NEXT: br i1 [[TMP67]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 1024, 1024
; CHECK-NEXT: br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 1024, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_SCEVCHECK]] ]
; CHECK-NEXT: br label [[LOOP:%.*]]
; CHECK: loop:
; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
; CHECK-NEXT: [[OFFSET0:%.*]] = shl i64 [[I]], 3
; CHECK-NEXT: [[Q0:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET0]]
; CHECK-NEXT: [[X0:%.*]] = load i64, ptr [[Q0]], align 4
; CHECK-NEXT: [[Y0:%.*]] = add i64 [[X0]], 1
; CHECK-NEXT: store i64 [[Y0]], ptr [[Q0]], align 4
; CHECK-NEXT: [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
; CHECK-NEXT: [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
; CHECK-NEXT: [[X1:%.*]] = load i64, ptr [[Q1]], align 4
; CHECK-NEXT: [[Y1:%.*]] = add i64 [[X1]], 2
; CHECK-NEXT: store i64 [[Y1]], ptr [[Q1]], align 4
; CHECK-NEXT: [[OFFSET2:%.*]] = add i64 [[OFFSET1]], 1
; CHECK-NEXT: [[Q2:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET2]]
; CHECK-NEXT: [[X2:%.*]] = load i64, ptr [[Q2]], align 4
; CHECK-NEXT: [[Y2:%.*]] = add i64 [[X2]], 3
; CHECK-NEXT: store i64 [[Y2]], ptr [[Q2]], align 4
; CHECK-NEXT: [[OFFSET3:%.*]] = add i64 [[OFFSET2]], 1
; CHECK-NEXT: [[Q3:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET3]]
; CHECK-NEXT: [[X3:%.*]] = load i64, ptr [[Q3]], align 4
; CHECK-NEXT: [[Y3:%.*]] = add i64 [[X3]], 4
; CHECK-NEXT: store i64 [[Y3]], ptr [[Q3]], align 4
; CHECK-NEXT: [[OFFSET4:%.*]] = add i64 [[OFFSET3]], 1
; CHECK-NEXT: [[Q4:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET4]]
; CHECK-NEXT: [[X4:%.*]] = load i64, ptr [[Q4]], align 4
; CHECK-NEXT: [[Y4:%.*]] = add i64 [[X4]], 5
; CHECK-NEXT: store i64 [[Y4]], ptr [[Q4]], align 4
; CHECK-NEXT: [[OFFSET5:%.*]] = add i64 [[OFFSET4]], 1
; CHECK-NEXT: [[Q5:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET5]]
; CHECK-NEXT: [[X5:%.*]] = load i64, ptr [[Q5]], align 4
; CHECK-NEXT: [[Y5:%.*]] = add i64 [[X5]], 6
; CHECK-NEXT: store i64 [[Y5]], ptr [[Q5]], align 4
; CHECK-NEXT: [[OFFSET6:%.*]] = add i64 [[OFFSET5]], 1
; CHECK-NEXT: [[Q6:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET6]]
; CHECK-NEXT: [[X6:%.*]] = load i64, ptr [[Q6]], align 4
; CHECK-NEXT: [[Y6:%.*]] = add i64 [[X6]], 7
; CHECK-NEXT: store i64 [[Y6]], ptr [[Q6]], align 4
; CHECK-NEXT: [[OFFSET7:%.*]] = add i64 [[OFFSET6]], 1
; CHECK-NEXT: [[Q7:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET7]]
; CHECK-NEXT: [[X7:%.*]] = load i64, ptr [[Q7]], align 4
; CHECK-NEXT: [[Y7:%.*]] = add i64 [[X7]], 8
; CHECK-NEXT: store i64 [[Y7]], ptr [[Q7]], align 4
; CHECK-NEXT: [[NEXTI]] = add i64 [[I]], 1
; CHECK-NEXT: [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
; CHECK-NEXT: br i1 [[DONE]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP7:![0-9]+]]
; CHECK: exit:
; CHECK-NEXT: ret void
;
entry:
br label %loop
loop:
%i = phi i64 [0, %entry], [%nexti, %loop]
%offset0 = shl i64 %i, 3
%q0 = getelementptr i64, ptr %p, i64 %offset0
%x0 = load i64, ptr %q0
%y0 = add i64 %x0, 1
store i64 %y0, ptr %q0
%offset1 = add i64 %offset0, 1
%q1 = getelementptr i64, ptr %p, i64 %offset1
%x1 = load i64, ptr %q1
%y1 = add i64 %x1, 2
store i64 %y1, ptr %q1
%offset2 = add i64 %offset1, 1
%q2 = getelementptr i64, ptr %p, i64 %offset2
%x2 = load i64, ptr %q2
%y2 = add i64 %x2, 3
store i64 %y2, ptr %q2
%offset3 = add i64 %offset2, 1
%q3 = getelementptr i64, ptr %p, i64 %offset3
%x3 = load i64, ptr %q3
%y3 = add i64 %x3, 4
store i64 %y3, ptr %q3
%offset4 = add i64 %offset3, 1
%q4 = getelementptr i64, ptr %p, i64 %offset4
%x4 = load i64, ptr %q4
%y4 = add i64 %x4, 5
store i64 %y4, ptr %q4
%offset5 = add i64 %offset4, 1
%q5 = getelementptr i64, ptr %p, i64 %offset5
%x5 = load i64, ptr %q5
%y5 = add i64 %x5, 6
store i64 %y5, ptr %q5
%offset6 = add i64 %offset5, 1
%q6 = getelementptr i64, ptr %p, i64 %offset6
%x6 = load i64, ptr %q6
%y6 = add i64 %x6, 7
store i64 %y6, ptr %q6
%offset7 = add i64 %offset6, 1
%q7 = getelementptr i64, ptr %p, i64 %offset7
%x7 = load i64, ptr %q7
%y7 = add i64 %x7, 8
store i64 %y7, ptr %q7
%nexti = add i64 %i, 1
%done = icmp eq i64 %nexti, 1024
br i1 %done, label %exit, label %loop
exit:
ret void
}
define void @combine_load_factor2(ptr %p) {
; CHECK-LABEL: @combine_load_factor2(
; CHECK-NEXT: entry:
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_SCEVCHECK:%.*]]
; CHECK: vector.scevcheck:
; CHECK-NEXT: [[UGLYGEP:%.*]] = getelementptr i8, ptr [[P:%.*]], i64 8
; CHECK-NEXT: [[MUL:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 16, i64 1023)
; CHECK-NEXT: [[MUL_RESULT:%.*]] = extractvalue { i64, i1 } [[MUL]], 0
; CHECK-NEXT: [[MUL_OVERFLOW:%.*]] = extractvalue { i64, i1 } [[MUL]], 1
; CHECK-NEXT: [[TMP0:%.*]] = sub i64 0, [[MUL_RESULT]]
; CHECK-NEXT: [[TMP1:%.*]] = getelementptr i8, ptr [[UGLYGEP]], i64 [[MUL_RESULT]]
; CHECK-NEXT: [[TMP2:%.*]] = icmp ult ptr [[TMP1]], [[UGLYGEP]]
; CHECK-NEXT: [[TMP3:%.*]] = or i1 [[TMP2]], [[MUL_OVERFLOW]]
; CHECK-NEXT: [[MUL1:%.*]] = call { i64, i1 } @llvm.umul.with.overflow.i64(i64 16, i64 1023)
; CHECK-NEXT: [[MUL_RESULT2:%.*]] = extractvalue { i64, i1 } [[MUL1]], 0
; CHECK-NEXT: [[MUL_OVERFLOW3:%.*]] = extractvalue { i64, i1 } [[MUL1]], 1
; CHECK-NEXT: [[TMP4:%.*]] = sub i64 0, [[MUL_RESULT2]]
; CHECK-NEXT: [[TMP5:%.*]] = getelementptr i8, ptr [[P]], i64 [[MUL_RESULT2]]
; CHECK-NEXT: [[TMP6:%.*]] = icmp ult ptr [[TMP5]], [[P]]
; CHECK-NEXT: [[TMP7:%.*]] = or i1 [[TMP6]], [[MUL_OVERFLOW3]]
; CHECK-NEXT: [[TMP8:%.*]] = or i1 [[TMP3]], [[TMP7]]
; CHECK-NEXT: br i1 [[TMP8]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_IND:%.*]] = phi <2 x i64> [ <i64 0, i64 1>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[STEP_ADD:%.*]] = add <2 x i64> [[VEC_IND]], <i64 2, i64 2>
; CHECK-NEXT: [[TMP9:%.*]] = shl <2 x i64> [[VEC_IND]], <i64 1, i64 1>
; CHECK-NEXT: [[TMP10:%.*]] = shl <2 x i64> [[STEP_ADD]], <i64 1, i64 1>
; CHECK-NEXT: [[TMP11:%.*]] = extractelement <2 x i64> [[TMP9]], i32 0
; CHECK-NEXT: [[TMP12:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP11]]
; CHECK-NEXT: [[TMP13:%.*]] = extractelement <2 x i64> [[TMP9]], i32 1
; CHECK-NEXT: [[TMP14:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP13]]
; CHECK-NEXT: [[TMP15:%.*]] = extractelement <2 x i64> [[TMP10]], i32 0
; CHECK-NEXT: [[TMP16:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP15]]
; CHECK-NEXT: [[TMP17:%.*]] = extractelement <2 x i64> [[TMP10]], i32 1
; CHECK-NEXT: [[TMP18:%.*]] = getelementptr i64, ptr [[P]], i64 [[TMP17]]
; CHECK-NEXT: [[TMP19:%.*]] = getelementptr i64, ptr [[TMP12]], i32 0
; CHECK-NEXT: [[TMP20:%.*]] = getelementptr i64, ptr [[TMP16]], i32 0
; CHECK-NEXT: [[WIDE_VEC:%.*]] = load <4 x i64>, ptr [[TMP19]], align 4
; CHECK-NEXT: [[WIDE_VEC5:%.*]] = load <4 x i64>, ptr [[TMP20]], align 4
; CHECK-NEXT: [[STRIDED_VEC:%.*]] = shufflevector <4 x i64> [[WIDE_VEC]], <4 x i64> poison, <2 x i32> <i32 0, i32 2>
; CHECK-NEXT: [[STRIDED_VEC6:%.*]] = shufflevector <4 x i64> [[WIDE_VEC5]], <4 x i64> poison, <2 x i32> <i32 0, i32 2>
; CHECK-NEXT: [[STRIDED_VEC7:%.*]] = shufflevector <4 x i64> [[WIDE_VEC]], <4 x i64> poison, <2 x i32> <i32 1, i32 3>
; CHECK-NEXT: [[STRIDED_VEC8:%.*]] = shufflevector <4 x i64> [[WIDE_VEC5]], <4 x i64> poison, <2 x i32> <i32 1, i32 3>
; CHECK-NEXT: [[TMP21:%.*]] = add <2 x i64> [[STRIDED_VEC]], [[STRIDED_VEC7]]
; CHECK-NEXT: [[TMP22:%.*]] = add <2 x i64> [[STRIDED_VEC6]], [[STRIDED_VEC8]]
; CHECK-NEXT: [[TMP23:%.*]] = extractelement <2 x i64> [[TMP21]], i32 0
; CHECK-NEXT: store i64 [[TMP23]], ptr [[TMP12]], align 4
; CHECK-NEXT: [[TMP24:%.*]] = extractelement <2 x i64> [[TMP21]], i32 1
; CHECK-NEXT: store i64 [[TMP24]], ptr [[TMP14]], align 4
; CHECK-NEXT: [[TMP25:%.*]] = extractelement <2 x i64> [[TMP22]], i32 0
; CHECK-NEXT: store i64 [[TMP25]], ptr [[TMP16]], align 4
; CHECK-NEXT: [[TMP26:%.*]] = extractelement <2 x i64> [[TMP22]], i32 1
; CHECK-NEXT: store i64 [[TMP26]], ptr [[TMP18]], align 4
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
; CHECK-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[STEP_ADD]], <i64 2, i64 2>
; CHECK-NEXT: [[TMP27:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024
; CHECK-NEXT: br i1 [[TMP27]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 1024, 1024
; CHECK-NEXT: br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 1024, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_SCEVCHECK]] ]
; CHECK-NEXT: br label [[LOOP:%.*]]
; CHECK: loop:
; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[NEXTI:%.*]], [[LOOP]] ]
; CHECK-NEXT: [[OFFSET0:%.*]] = shl i64 [[I]], 1
; CHECK-NEXT: [[Q0:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET0]]
; CHECK-NEXT: [[X0:%.*]] = load i64, ptr [[Q0]], align 4
; CHECK-NEXT: [[OFFSET1:%.*]] = add i64 [[OFFSET0]], 1
; CHECK-NEXT: [[Q1:%.*]] = getelementptr i64, ptr [[P]], i64 [[OFFSET1]]
; CHECK-NEXT: [[X1:%.*]] = load i64, ptr [[Q1]], align 4
; CHECK-NEXT: [[RES:%.*]] = add i64 [[X0]], [[X1]]
; CHECK-NEXT: store i64 [[RES]], ptr [[Q0]], align 4
; CHECK-NEXT: [[NEXTI]] = add i64 [[I]], 1
; CHECK-NEXT: [[DONE:%.*]] = icmp eq i64 [[NEXTI]], 1024
; CHECK-NEXT: br i1 [[DONE]], label [[EXIT]], label [[LOOP]], !llvm.loop [[LOOP9:![0-9]+]]
; CHECK: exit:
; CHECK-NEXT: ret void
;
entry:
br label %loop
loop:
%i = phi i64 [0, %entry], [%nexti, %loop]
%offset0 = shl i64 %i, 1
%q0 = getelementptr i64, ptr %p, i64 %offset0
%x0 = load i64, ptr %q0
%offset1 = add i64 %offset0, 1
%q1 = getelementptr i64, ptr %p, i64 %offset1
%x1 = load i64, ptr %q1
%res = add i64 %x0, %x1
store i64 %res, ptr %q0
%nexti = add i64 %i, 1
%done = icmp eq i64 %nexti, 1024
br i1 %done, label %exit, label %loop
exit:
ret void
}

llvm/test/Transforms/LoopVectorize/RISCV/zvl32b.ll

; NOTE: Assertions have been autogenerated by utils/update_test_checks.py ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
; RUN: opt < %s -passes=loop-vectorize -scalable-vectorization=on -mtriple riscv64-linux-gnu -mattr=+zve32f,+f -S 2>%t | FileCheck %s ; RUN: opt < %s -passes=loop-vectorize -scalable-vectorization=on -mtriple riscv64-linux-gnu -mattr=+zve32f,+f -S 2>%t | FileCheck %s
target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n64-S128" target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n64-S128"
target triple = "riscv64" target triple = "riscv64"
; We can't use scalable vectorization for Zvl32b due to RVVBitsPerBlock being ; We can't use scalable vectorization for Zvl32b due to RVVBitsPerBlock being
; 64. Since our vscale value is vlen/RVVBitsPerBlock this makes vscale 0. ; 64. Since our vscale value is vlen/RVVBitsPerBlock this makes vscale 0.
; Make sure we fall back to fixed vectorization instead. ; Make sure we fall back to fixed vectorization instead.
define void @vector_add_i16(ptr noalias nocapture %a, i16 %v, i64 %n) { define void @vector_add_i16(ptr noalias nocapture %a, i16 %v, i64 %n) {
; CHECK-LABEL: @vector_add_i16( ; CHECK-LABEL: @vector_add_i16(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]] ; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph: ; CHECK: vector.ph:
; CHECK-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <2 x i16> poison, i16 [[V:%.*]], i64 0 ; CHECK-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <2 x i16> poison, i16 [[V:%.*]], i64 0
; CHECK-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <2 x i16> [[BROADCAST_SPLATINSERT]], <2 x i16> poison, <2 x i32> zeroinitializer ; CHECK-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <2 x i16> [[BROADCAST_SPLATINSERT]], <2 x i16> poison, <2 x i32> zeroinitializer
; CHECK-NEXT: [[BROADCAST_SPLATINSERT3:%.*]] = insertelement <2 x i16> poison, i16 [[V]], i64 0 ; CHECK-NEXT: [[BROADCAST_SPLATINSERT4:%.*]] = insertelement <2 x i16> poison, i16 [[V]], i64 0
; CHECK-NEXT: [[BROADCAST_SPLAT4:%.*]] = shufflevector <2 x i16> [[BROADCAST_SPLATINSERT3]], <2 x i16> poison, <2 x i32> zeroinitializer ; CHECK-NEXT: [[BROADCAST_SPLAT5:%.*]] = shufflevector <2 x i16> [[BROADCAST_SPLATINSERT4]], <2 x i16> poison, <2 x i32> zeroinitializer
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]] ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body: ; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ] ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_IND:%.*]] = phi <2 x i64> [ <i64 0, i64 1>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[VECTOR_BODY]] ] ; CHECK-NEXT: [[VEC_IND:%.*]] = phi <2 x i64> [ <i64 0, i64 1>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[STEP_ADD:%.*]] = add <2 x i64> [[VEC_IND]], <i64 2, i64 2> ; CHECK-NEXT: [[STEP_ADD:%.*]] = add <2 x i64> [[VEC_IND]], <i64 2, i64 2>
; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[A:%.*]], <2 x i64> [[VEC_IND]] ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[A:%.*]], <2 x i64> [[VEC_IND]]
; CHECK-NEXT: [[TMP1:%.*]] = getelementptr inbounds i32, ptr [[A]], <2 x i64> [[STEP_ADD]] ; CHECK-NEXT: [[TMP1:%.*]] = getelementptr inbounds i32, ptr [[A]], <2 x i64> [[STEP_ADD]]
; CHECK-NEXT: [[WIDE_MASKED_GATHER:%.*]] = call <2 x i16> @llvm.masked.gather.v2i16.v2p0(<2 x ptr> [[TMP0]], i32 2, <2 x i1> <i1 true, i1 true>, <2 x i16> poison) ; CHECK-NEXT: [[TMP2:%.*]] = extractelement <2 x ptr> [[TMP0]], i32 0
; CHECK-NEXT: [[WIDE_MASKED_GATHER2:%.*]] = call <2 x i16> @llvm.masked.gather.v2i16.v2p0(<2 x ptr> [[TMP1]], i32 2, <2 x i1> <i1 true, i1 true>, <2 x i16> poison) ; CHECK-NEXT: [[TMP3:%.*]] = getelementptr i16, ptr [[TMP2]], i32 0
; CHECK-NEXT: [[TMP2:%.*]] = add <2 x i16> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]] ; CHECK-NEXT: [[TMP4:%.*]] = extractelement <2 x ptr> [[TMP1]], i32 0
; CHECK-NEXT: [[TMP3:%.*]] = add <2 x i16> [[WIDE_MASKED_GATHER2]], [[BROADCAST_SPLAT4]] ; CHECK-NEXT: [[TMP5:%.*]] = getelementptr i16, ptr [[TMP4]], i32 0
; CHECK-NEXT: call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> [[TMP2]], <2 x ptr> [[TMP0]], i32 2, <2 x i1> <i1 true, i1 true>) ; CHECK-NEXT: [[WIDE_VEC:%.*]] = load <4 x i16>, ptr [[TMP3]], align 2
; CHECK-NEXT: call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> [[TMP3]], <2 x ptr> [[TMP1]], i32 2, <2 x i1> <i1 true, i1 true>) ; CHECK-NEXT: [[WIDE_VEC2:%.*]] = load <4 x i16>, ptr [[TMP5]], align 2
; CHECK-NEXT: [[STRIDED_VEC:%.*]] = shufflevector <4 x i16> [[WIDE_VEC]], <4 x i16> poison, <2 x i32> <i32 0, i32 2>
; CHECK-NEXT: [[STRIDED_VEC3:%.*]] = shufflevector <4 x i16> [[WIDE_VEC2]], <4 x i16> poison, <2 x i32> <i32 0, i32 2>
; CHECK-NEXT: [[TMP6:%.*]] = add <2 x i16> [[STRIDED_VEC]], [[BROADCAST_SPLAT]]
; CHECK-NEXT: [[TMP7:%.*]] = add <2 x i16> [[STRIDED_VEC3]], [[BROADCAST_SPLAT5]]
; CHECK-NEXT: call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> [[TMP6]], <2 x ptr> [[TMP0]], i32 2, <2 x i1> <i1 true, i1 true>)
; CHECK-NEXT: call void @llvm.masked.scatter.v2i16.v2p0(<2 x i16> [[TMP7]], <2 x ptr> [[TMP1]], i32 2, <2 x i1> <i1 true, i1 true>)
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4 ; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
; CHECK-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[STEP_ADD]], <i64 2, i64 2> ; CHECK-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[STEP_ADD]], <i64 2, i64 2>
; CHECK-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024 ; CHECK-NEXT: [[TMP8:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1020
; CHECK-NEXT: br i1 [[TMP4]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]] ; CHECK-NEXT: br i1 [[TMP8]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
; CHECK: middle.block: ; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 1024, 1024 ; CHECK-NEXT: br label [[SCALAR_PH]]
; CHECK-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph: ; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 1024, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ] ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 1020, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
; CHECK-NEXT: br label [[FOR_BODY:%.*]] ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
; CHECK: for.body: ; CHECK: for.body:
; CHECK-NEXT: [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[FOR_BODY]] ] ; CHECK-NEXT: [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[FOR_BODY]] ]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[IV]] ; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[IV]]
; CHECK-NEXT: [[ELEM:%.*]] = load i16, ptr [[ARRAYIDX]], align 2 ; CHECK-NEXT: [[ELEM:%.*]] = load i16, ptr [[ARRAYIDX]], align 2
; CHECK-NEXT: [[ADD:%.*]] = add i16 [[ELEM]], [[V]] ; CHECK-NEXT: [[ADD:%.*]] = add i16 [[ELEM]], [[V]]
; CHECK-NEXT: store i16 [[ADD]], ptr [[ARRAYIDX]], align 2 ; CHECK-NEXT: store i16 [[ADD]], ptr [[ARRAYIDX]], align 2
; CHECK-NEXT: [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1 ; CHECK-NEXT: [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
; CHECK-NEXT: [[EXITCOND_NOT:%.*]] = icmp eq i64 [[IV_NEXT]], 1024 ; CHECK-NEXT: [[EXITCOND_NOT:%.*]] = icmp eq i64 [[IV_NEXT]], 1024
; CHECK-NEXT: br i1 [[EXITCOND_NOT]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]] ; CHECK-NEXT: br i1 [[EXITCOND_NOT]], label [[FOR_END:%.*]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
; CHECK: for.end: ; CHECK: for.end:
; CHECK-NEXT: ret void ; CHECK-NEXT: ret void
; ;
entry: entry:
br label %for.body br label %for.body
for.body: for.body:
%iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ] %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
Show All 11 Lines