This is an archive of the discontinued LLVM Phabricator instance.

Differential D103535

[mlir][ArmSVE] Add basic load/store operations
ClosedPublic

Authored by jsetoain on Jun 2 2021, 9:40 AM.

Details

Reviewers
aartbik
nicolasvasilache
ftynse
Commits
rG96ca2d92b52b: [mlir][ArmSVE] Add basic load/store operations
Summary

ArmSVE-specific memory operations are needed to generate end-to-end
code for as long as MLIR core doesn't support scalable vectors. This
instructions will be eventually unnecessary, for now they're required
for more complex testing.

Diff Detail

Event Timeline

jsetoain created this revision.Jun 2 2021, 9:40 AM
Herald added a reviewer: ftynse. · View Herald TranscriptJun 2 2021, 9:40 AM
Herald added subscribers: dcaballe, cota, teijeong and 17 others. · View Herald Transcript
jsetoain published this revision for review.Jun 2 2021, 9:42 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 2 2021, 9:42 AM
Herald added a subscriber: stephenneuendorffer. · View Herald Transcript
Harbormaster completed remote builds in B107272: Diff 349306.Jun 2 2021, 10:24 AM
ftynse requested changes to this revision.Jun 4 2021, 4:29 AM
ftynse added inline comments.
mlir/lib/Dialect/ArmSVE/Transforms/LegalizeForLLVMExport.cpp
123–126

I don't think this works. Derived patterns never call this function and override matchAndRewrite, the default implementation of which would have called match then rewrite, but it is not called either.

This revision now requires changes to proceed.Jun 4 2021, 4:29 AM
Matt added a subscriber: Matt.Jun 4 2021, 9:28 AM
Matt added inline comments.Jun 4 2021, 9:32 AM
mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td
379

s/on/to/

jsetoain updated this revision to Diff 349929.Jun 4 2021, 11:23 AM

Fixed legality check on memref

jsetoain marked 2 inline comments as done.Jun 4 2021, 11:27 AM
jsetoain added inline comments.
mlir/lib/Dialect/ArmSVE/Transforms/LegalizeForLLVMExport.cpp
123–126

This should fix it, now I make sure that the check is performed in the readAndRewrite method.

jsetoain updated this revision to Diff 349931.Jun 4 2021, 11:38 AM
jsetoain marked an inline comment as done.

Clang format...

Harbormaster completed remote builds in B107718: Diff 349931.Jun 4 2021, 12:12 PM
ftynse accepted this revision.Jun 7 2021, 1:22 AM
This revision is now accepted and ready to land.Jun 7 2021, 1:22 AM
aartbik accepted this revision.Jun 7 2021, 2:02 PM
aartbik added inline comments.
mlir/lib/Dialect/ArmSVE/Transforms/LegalizeForLLVMExport.cpp
114

super nit: the ing forms read a bit hard

How about:

The load operation is lowered to code that obtains ... and loads

150

same

jsetoain updated this revision to Diff 350839.Jun 9 2021, 3:15 AM

Rebase to main and address comments

jsetoain marked 2 inline comments as done.Jun 9 2021, 3:17 AM
Harbormaster completed remote builds in B108378: Diff 350839.Jun 9 2021, 3:53 AM
This revision was landed with ongoing or failed builds.Jun 9 2021, 8:04 AM
Closed by commit rG96ca2d92b52b: [mlir][ArmSVE] Add basic load/store operations (authored by jsetoain). · Explain Why
This revision was automatically updated to reflect the committed changes.
jsetoain added a commit: rG96ca2d92b52b: [mlir][ArmSVE] Add basic load/store operations.

Revision Contents

PathSize
mlir/
include/
mlir/
Dialect/
ArmSVE/
31 lines
lib/
Dialect/
ArmSVE/
Transforms/
81 lines
test/
Dialect/
ArmSVE/
29 lines
12 lines
Target/
LLVMIR/
78 lines

Diff 349929

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

Show First 20 Lines • Show All 349 Lines • ▼ Show 20 Lines let description = [{
times, once on each <4xi32> segment of the scalable vector. The vector_scale times, once on each <4xi32> segment of the scalable vector. The vector_scale
op can be used to calculate the step in vector-length agnostic (VLA) loops. op can be used to calculate the step in vector-length agnostic (VLA) loops.
}]; }];
let results = (outs Index:$res); let results = (outs Index:$res);
let assemblyFormat = let assemblyFormat =
"attr-dict `:` type($res)"; "attr-dict `:` type($res)";
} }
def ScalableLoadOp : ArmSVE_Op<"load">,
Arguments<(ins Arg<AnyMemRef, "", [MemRead]>:$base, Index:$index)>,
Results<(outs ScalableVectorOf<[AnyType]>:$result)> {
let summary = "Load scalable vector from memory";
let description = [{
Load a slice of memory into a scalable vector.
}];
let extraClassDeclaration = [{
MemRefType getMemRefType() {
return base().getType().cast<MemRefType>();
}
}];
let assemblyFormat = "$base `[` $index `]` attr-dict `:` "
"type($result) `from` type($base)";
}
def ScalableStoreOp : ArmSVE_Op<"store">,
Arguments<(ins Arg<AnyMemRef, "", [MemWrite]>:$base, Index:$index,
ScalableVectorOf<[AnyType]>:$value)> {
let summary = "Store scalable vector into memory";
let description = [{
Store a scalable vector on a slice of memory.
MattUnsubmitted
Done
ReplyInline Actions
let description = [{
- Store a scalable vector on a slice of memory.
+ Store a scalable vector to a slice of memory.
}];
let extraClassDeclaration = [{

s/on/to/

Matt: s/on/to/
}];
let extraClassDeclaration = [{
MemRefType getMemRefType() {
return base().getType().cast<MemRefType>();
}
}];
let assemblyFormat = "$value `,` $base `[` $index `]` attr-dict `:` "
"type($value) `to` type($base)";
}
def ScalableAddIOp : ScalableIOp<"addi", "addition", [Commutative]>; def ScalableAddIOp : ScalableIOp<"addi", "addition", [Commutative]>;
def ScalableAddFOp : ScalableFOp<"addf", "addition", [Commutative]>; def ScalableAddFOp : ScalableFOp<"addf", "addition", [Commutative]>;
def ScalableSubIOp : ScalableIOp<"subi", "subtraction">; def ScalableSubIOp : ScalableIOp<"subi", "subtraction">;
def ScalableSubFOp : ScalableFOp<"subf", "subtraction">; def ScalableSubFOp : ScalableFOp<"subf", "subtraction">;
▲ Show 20 Lines • Show All 104 Lines • Show Last 20 Lines

mlir/lib/Dialect/ArmSVE/Transforms/LegalizeForLLVMExport.cpp

Show First 20 Lines • Show All 105 Lines • ▼ Show 20 Lines OneToOneConvertToLLVMPattern<ScalableMaskedSDivIOp,
ScalableMaskedSDivIIntrOp>; ScalableMaskedSDivIIntrOp>;
using ScalableMaskedUDivIOpLowering = using ScalableMaskedUDivIOpLowering =
OneToOneConvertToLLVMPattern<ScalableMaskedUDivIOp, OneToOneConvertToLLVMPattern<ScalableMaskedUDivIOp,
ScalableMaskedUDivIIntrOp>; ScalableMaskedUDivIIntrOp>;
using ScalableMaskedDivFOpLowering = using ScalableMaskedDivFOpLowering =
OneToOneConvertToLLVMPattern<ScalableMaskedDivFOp, OneToOneConvertToLLVMPattern<ScalableMaskedDivFOp,
ScalableMaskedDivFIntrOp>; ScalableMaskedDivFIntrOp>;
// Load operation is lowered to obtaining a pointer to the indexed element
aartbikUnsubmitted
Done
ReplyInline Actions

super nit: the ing forms read a bit hard

How about:

The load operation is lowered to code that obtains ... and loads

aartbik: super nit: the ing forms read a bit hard How about: The load operation is lowered to code…
// and loading it.
struct ScalableLoadOpLowering
Lint: Pre-merge checks
Inline Actions

clang-format: please reformat the code

-struct ScalableLoadOpLowering
-    : public ConvertOpToLLVMPattern<ScalableLoadOp> {
+struct ScalableLoadOpLowering : public ConvertOpToLLVMPattern<ScalableLoadOp> {
Lint: Pre-merge checks: clang-format: please reformat the code ``` -struct ScalableLoadOpLowering - : public…
: public ConvertOpToLLVMPattern<ScalableLoadOp> {
using ConvertOpToLLVMPattern<ScalableLoadOp>::ConvertOpToLLVMPattern;
LogicalResult
matchAndRewrite(ScalableLoadOp loadOp, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override {
auto type = loadOp.getMemRefType();
if (!isConvertibleAndHasIdentityMaps(type))
return failure();
ftynseUnsubmitted
Done
ReplyInline Actions

I don't think this works. Derived patterns never call this function and override matchAndRewrite, the default implementation of which would have called match then rewrite, but it is not called either.

ftynse: I don't think this works. Derived patterns never call this function and override…
jsetoainAuthorUnsubmitted
Done
ReplyInline Actions

This should fix it, now I make sure that the check is performed in the readAndRewrite method.

jsetoain: This should fix it, now I make sure that the check is performed in the readAndRewrite method.
ScalableLoadOp::Adaptor transformed(operands);
LLVMTypeConverter converter(loadOp.getContext());
auto resultType = loadOp.result().getType();
LLVM::LLVMPointerType llvmDataTypePtr;
if (resultType.isa<VectorType>()) {
llvmDataTypePtr =
LLVM::LLVMPointerType::get(resultType.cast<VectorType>());
} else if (resultType.isa<ScalableVectorType>()) {
llvmDataTypePtr = LLVM::LLVMPointerType::get(
convertScalableVectorTypeToLLVM(resultType.cast<ScalableVectorType>(),
converter)
.getValue());
}
Value dataPtr =
getStridedElementPtr(loadOp.getLoc(), type, transformed.base(),
transformed.index(), rewriter);
Value bitCastedPtr = rewriter.create<LLVM::BitcastOp>(
loadOp.getLoc(), llvmDataTypePtr, dataPtr);
rewriter.replaceOpWithNewOp<LLVM::LoadOp>(loadOp, bitCastedPtr);
return success();
}
};
aartbikUnsubmitted
Done
ReplyInline Actions

same

aartbik: same
// Store operation is lowered to obtaining a pointer to the indexed element,
// and storing the given value to it.
struct ScalableStoreOpLowering
: public ConvertOpToLLVMPattern<ScalableStoreOp> {
using ConvertOpToLLVMPattern<ScalableStoreOp>::ConvertOpToLLVMPattern;
LogicalResult
matchAndRewrite(ScalableStoreOp storeOp, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override {
auto type = storeOp.getMemRefType();
if (!isConvertibleAndHasIdentityMaps(type))
return failure();
ScalableStoreOp::Adaptor transformed(operands);
LLVMTypeConverter converter(storeOp.getContext());
auto resultType = storeOp.value().getType();
LLVM::LLVMPointerType llvmDataTypePtr;
if (resultType.isa<VectorType>()) {
llvmDataTypePtr =
LLVM::LLVMPointerType::get(resultType.cast<VectorType>());
} else if (resultType.isa<ScalableVectorType>()) {
llvmDataTypePtr = LLVM::LLVMPointerType::get(
convertScalableVectorTypeToLLVM(resultType.cast<ScalableVectorType>(),
converter)
.getValue());
}
Value dataPtr =
getStridedElementPtr(storeOp.getLoc(), type, transformed.base(),
transformed.index(), rewriter);
Value bitCastedPtr = rewriter.create<LLVM::BitcastOp>(
storeOp.getLoc(), llvmDataTypePtr, dataPtr);
rewriter.replaceOpWithNewOp<LLVM::StoreOp>(storeOp, transformed.value(),
bitCastedPtr);
return success();
}
};
static void static void
populateBasicSVEArithmeticExportPatterns(LLVMTypeConverter &converter, populateBasicSVEArithmeticExportPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList &patterns) { OwningRewritePatternList &patterns) {
// clang-format off // clang-format off
patterns.add<OneToOneConvertToLLVMPattern<ScalableAddIOp, LLVM::AddOp>, patterns.add<OneToOneConvertToLLVMPattern<ScalableAddIOp, LLVM::AddOp>,
OneToOneConvertToLLVMPattern<ScalableAddFOp, LLVM::FAddOp>, OneToOneConvertToLLVMPattern<ScalableAddFOp, LLVM::FAddOp>,
OneToOneConvertToLLVMPattern<ScalableSubIOp, LLVM::SubOp>, OneToOneConvertToLLVMPattern<ScalableSubIOp, LLVM::SubOp>,
OneToOneConvertToLLVMPattern<ScalableSubFOp, LLVM::FSubOp>, OneToOneConvertToLLVMPattern<ScalableSubFOp, LLVM::FSubOp>,
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 Lines patterns.add<SdotOpLowering,
ScalableMaskedAddFOpLowering, ScalableMaskedAddFOpLowering,
ScalableMaskedSubIOpLowering, ScalableMaskedSubIOpLowering,
ScalableMaskedSubFOpLowering, ScalableMaskedSubFOpLowering,
ScalableMaskedMulIOpLowering, ScalableMaskedMulIOpLowering,
ScalableMaskedMulFOpLowering, ScalableMaskedMulFOpLowering,
ScalableMaskedSDivIOpLowering, ScalableMaskedSDivIOpLowering,
ScalableMaskedUDivIOpLowering, ScalableMaskedUDivIOpLowering,
ScalableMaskedDivFOpLowering>(converter); ScalableMaskedDivFOpLowering>(converter);
patterns.add<ScalableLoadOpLowering,
ScalableStoreOpLowering>(converter);
// clang-format on // clang-format on
populateBasicSVEArithmeticExportPatterns(converter, patterns); populateBasicSVEArithmeticExportPatterns(converter, patterns);
} }
void mlir::configureArmSVELegalizeForExportTarget( void mlir::configureArmSVELegalizeForExportTarget(
LLVMConversionTarget &target) { LLVMConversionTarget &target) {
// clang-format off // clang-format off
target.addLegalOp<SdotIntrOp, target.addLegalOp<SdotIntrOp,
Show All 18 Lines target.addIllegalOp<SdotOp,
ScalableMaskedAddIOp, ScalableMaskedAddIOp,
ScalableMaskedAddFOp, ScalableMaskedAddFOp,
ScalableMaskedSubIOp, ScalableMaskedSubIOp,
ScalableMaskedSubFOp, ScalableMaskedSubFOp,
ScalableMaskedMulIOp, ScalableMaskedMulIOp,
ScalableMaskedMulFOp, ScalableMaskedMulFOp,
ScalableMaskedSDivIOp, ScalableMaskedSDivIOp,
ScalableMaskedUDivIOp, ScalableMaskedUDivIOp,
ScalableMaskedDivFOp>(); ScalableMaskedDivFOp,
ScalableLoadOp,
ScalableStoreOp>();
// clang-format on // clang-format on
auto hasScalableVectorType = [](TypeRange types) { auto hasScalableVectorType = [](TypeRange types) {
for (Type type : types) for (Type type : types)
if (type.isa<arm_sve::ScalableVectorType>()) if (type.isa<arm_sve::ScalableVectorType>())
return true; return true;
return false; return false;
}; };
target.addDynamicallyLegalOp<FuncOp>([hasScalableVectorType](FuncOp op) { target.addDynamicallyLegalOp<FuncOp>([hasScalableVectorType](FuncOp op) {
Show All 10 Lines

mlir/test/Dialect/ArmSVE/memcpy.mlir

  • This file was added.
// RUN: mlir-opt %s -convert-vector-to-llvm="enable-arm-sve" | mlir-opt | FileCheck %s
// CHECK: memcopy([[SRC:%arg[0-9]+]]: memref<?xf32>, [[DST:%arg[0-9]+]]
func @memcopy(%src : memref<?xf32>, %dst : memref<?xf32>, %size : index) {
%c0 = constant 0 : index
%c4 = constant 4 : index
%vs = arm_sve.vector_scale : index
%step = muli %c4, %vs : index
// CHECK: scf.for [[LOOPIDX:%arg[0-9]+]] = {{.*}}
scf.for %i0 = %c0 to %size step %step {
// CHECK: [[SRCMRS:%[0-9]+]] = llvm.mlir.cast [[SRC]] : memref<?xf32> to !llvm.struct<(ptr<f32>
// CHECK: [[SRCIDX:%[0-9]+]] = llvm.mlir.cast [[LOOPIDX]] : index to i64
// CHECK: [[SRCMEM:%[0-9]+]] = llvm.extractvalue [[SRCMRS]][1] : !llvm.struct<(ptr<f32>
// CHECK-NEXT: [[SRCPTR:%[0-9]+]] = llvm.getelementptr [[SRCMEM]]{{.}}[[SRCIDX]]{{.}} : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
// CHECK-NEXT: [[SRCVPTR:%[0-9]+]] = llvm.bitcast [[SRCPTR]] : !llvm.ptr<f32> to !llvm.ptr<vec<? x 4 x f32>>
// CHECK-NEXT: [[LDVAL:%[0-9]+]] = llvm.load [[SRCVPTR]] : !llvm.ptr<vec<? x 4 x f32>>
%0 = arm_sve.load %src[%i0] : !arm_sve.vector<4xf32> from memref<?xf32>
// CHECK: [[DSTMRS:%[0-9]+]] = llvm.mlir.cast [[DST]] : memref<?xf32> to !llvm.struct<(ptr<f32>
// CHECK: [[DSTIDX:%[0-9]+]] = llvm.mlir.cast [[LOOPIDX]] : index to i64
// CHECK: [[DSTMEM:%[0-9]+]] = llvm.extractvalue [[DSTMRS]][1] : !llvm.struct<(ptr<f32>
// CHECK-NEXT: [[DSTPTR:%[0-9]+]] = llvm.getelementptr [[DSTMEM]]{{.}}[[DSTIDX]]{{.}} : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
// CHECK-NEXT: [[DSTVPTR:%[0-9]+]] = llvm.bitcast [[DSTPTR]] : !llvm.ptr<f32> to !llvm.ptr<vec<? x 4 x f32>>
// CHECK-NEXT: llvm.store [[LDVAL]], [[DSTVPTR]] : !llvm.ptr<vec<? x 4 x f32>>
arm_sve.store %0, %dst[%i0] : !arm_sve.vector<4xf32> to memref<?xf32>
}
return
}

mlir/test/Dialect/ArmSVE/roundtrip.mlir

Show First 20 Lines • Show All 97 Lines • ▼ Show 20 Linesfunc @arm_sve_masked_arithf(%a: !arm_sve.vector<4xf32>,
%2 = arm_sve.masked.subf %mask, %1, %d : !arm_sve.vector<4xi1>, %2 = arm_sve.masked.subf %mask, %1, %d : !arm_sve.vector<4xi1>,
!arm_sve.vector<4xf32> !arm_sve.vector<4xf32>
// CHECK: arm_sve.masked.divf {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector // CHECK: arm_sve.masked.divf {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector
%3 = arm_sve.masked.divf %mask, %2, %e : !arm_sve.vector<4xi1>, %3 = arm_sve.masked.divf %mask, %2, %e : !arm_sve.vector<4xi1>,
!arm_sve.vector<4xf32> !arm_sve.vector<4xf32>
return %3 : !arm_sve.vector<4xf32> return %3 : !arm_sve.vector<4xf32>
} }
func @arm_sve_memory(%v: !arm_sve.vector<4xi32>,
%m: memref<?xi32>)
-> !arm_sve.vector<4xi32>
{
%c0 = constant 0 : index
// CHECK: arm_sve.load {{.*}}: !arm_sve.vector<4xi32> from memref<?xi32>
%0 = arm_sve.load %m[%c0] : !arm_sve.vector<4xi32> from memref<?xi32>
// CHECK: arm_sve.store {{.*}}: !arm_sve.vector<4xi32> to memref<?xi32>
arm_sve.store %v, %m[%c0] : !arm_sve.vector<4xi32> to memref<?xi32>
return %0 : !arm_sve.vector<4xi32>
}
func @get_vector_scale() -> index { func @get_vector_scale() -> index {
// CHECK: arm_sve.vector_scale : index // CHECK: arm_sve.vector_scale : index
%0 = arm_sve.vector_scale : index %0 = arm_sve.vector_scale : index
return %0 : index return %0 : index
} }

mlir/test/Target/LLVMIR/arm-sve.mlir

Show First 20 Lines • Show All 133 Lines • ▼ Show 20 Linesllvm.func @arm_sve_arithf_masked(%arg0: !llvm.vec<? x 4 x f32>,
// CHECK: call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.nxv4f32 // CHECK: call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.nxv4f32
%3 = "arm_sve.intr.fdiv"(%arg5, %2, %arg4) : (!llvm.vec<? x 4 x i1>, %3 = "arm_sve.intr.fdiv"(%arg5, %2, %arg4) : (!llvm.vec<? x 4 x i1>,
!llvm.vec<? x 4 x f32>, !llvm.vec<? x 4 x f32>,
!llvm.vec<? x 4 x f32>) !llvm.vec<? x 4 x f32>)
-> !llvm.vec<? x 4 x f32> -> !llvm.vec<? x 4 x f32>
llvm.return %3 : !llvm.vec<? x 4 x f32> llvm.return %3 : !llvm.vec<? x 4 x f32>
} }
// CHECK-LABEL: define void @memcopy
llvm.func @memcopy(%arg0: !llvm.ptr<f32>, %arg1: !llvm.ptr<f32>,
%arg2: i64, %arg3: i64, %arg4: i64,
%arg5: !llvm.ptr<f32>, %arg6: !llvm.ptr<f32>,
%arg7: i64, %arg8: i64, %arg9: i64,
%arg10: i64) {
%0 = llvm.mlir.undef : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>, array<1 x i64>)>
%1 = llvm.insertvalue %arg0, %0[0] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%2 = llvm.insertvalue %arg1, %1[1] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%3 = llvm.insertvalue %arg2, %2[2] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%4 = llvm.insertvalue %arg3, %3[3, 0] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%5 = llvm.insertvalue %arg4, %4[4, 0] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%6 = llvm.mlir.undef : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%7 = llvm.insertvalue %arg5, %6[0] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%8 = llvm.insertvalue %arg6, %7[1] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%9 = llvm.insertvalue %arg7, %8[2] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%10 = llvm.insertvalue %arg8, %9[3, 0] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%11 = llvm.insertvalue %arg9, %10[4, 0] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
%12 = llvm.mlir.constant(0 : index) : i64
%13 = llvm.mlir.constant(4 : index) : i64
// CHECK: [[VL:%[0-9]+]] = call i64 @llvm.vscale.i64()
%14 = "arm_sve.vscale"() : () -> i64
// CHECK: mul i64 [[VL]], 4
%15 = llvm.mul %14, %13 : i64
llvm.br ^bb1(%12 : i64)
^bb1(%16: i64):
%17 = llvm.icmp "slt" %16, %arg10 : i64
llvm.cond_br %17, ^bb2, ^bb3
^bb2:
// CHECK: extractvalue { float*, float*, i64, [1 x i64], [1 x i64] }
%18 = llvm.extractvalue %5[1] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
// CHECK: etelementptr float, float*
%19 = llvm.getelementptr %18[%16] : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
// CHECK: bitcast float* %{{[0-9]+}} to <vscale x 4 x float>*
%20 = llvm.bitcast %19 : !llvm.ptr<f32> to !llvm.ptr<vec<? x 4 x f32>>
// CHECK: load <vscale x 4 x float>, <vscale x 4 x float>*
%21 = llvm.load %20 : !llvm.ptr<vec<? x 4 x f32>>
// CHECK: extractvalue { float*, float*, i64, [1 x i64], [1 x i64] }
%22 = llvm.extractvalue %11[1] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
array<1 x i64>,
array<1 x i64>)>
// CHECK: getelementptr float, float* %32
%23 = llvm.getelementptr %22[%16] : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
// CHECK: bitcast float* %33 to <vscale x 4 x float>*
%24 = llvm.bitcast %23 : !llvm.ptr<f32> to !llvm.ptr<vec<? x 4 x f32>>
// CHECK: store <vscale x 4 x float> %{{[0-9]+}}, <vscale x 4 x float>* %{{[0-9]+}}
llvm.store %21, %24 : !llvm.ptr<vec<? x 4 x f32>>
%25 = llvm.add %16, %15 : i64
llvm.br ^bb1(%25 : i64)
^bb3:
llvm.return
}
// CHECK-LABEL: define i64 @get_vector_scale() // CHECK-LABEL: define i64 @get_vector_scale()
llvm.func @get_vector_scale() -> i64 { llvm.func @get_vector_scale() -> i64 {
// CHECK: call i64 @llvm.vscale.i64() // CHECK: call i64 @llvm.vscale.i64()
%0 = "arm_sve.vscale"() : () -> i64 %0 = "arm_sve.vscale"() : () -> i64
llvm.return %0 : i64 llvm.return %0 : i64
} }