This is an archive of the discontinued LLVM Phabricator instance.

Differential D154707

[mlir][LLVM] Make `SplitStores` pattern capable of writing to sub-aggregates
ClosedPublic

Authored by zero9178 on Jul 7 2023, 5:29 AM.

Details

Reviewers
gysit
Dinistro
Moxinilian
ftynse
dcaballe
nicolasvasilache
Commits
rG7786449334d8: [mlir][LLVM] Make `SplitStores` pattern capable of writing to sub-aggregates
Summary

The pattern was previously only capable of storing into struct fields which are primitive types. If the struct contained a nested struct it immediately aborted the pattern rewrite.

This patch introduces the capability of recursively splitting stores into sub-aggregates as well. This is achieved by splitting an aggregate sized integer from the original store argument and letting repeated pattern applications further split it into field stores.

Additionally, the pattern is also capable of handling partial writes into aggregates, which is a pattern clang generates as well. Special care had to be taken to make sure no stores are created that weren't in the original code.

Diff Detail

Event Timeline

zero9178 created this revision.Jul 7 2023, 5:29 AM
Herald added a reviewer: ftynse. · View Herald TranscriptJul 7 2023, 5:29 AM
Herald added a reviewer: dcaballe. · View Herald Transcript
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: bviyer, Moerafaat, bzcheeseman and 21 others. · View Herald Transcript
zero9178 requested review of this revision.Jul 7 2023, 5:29 AM
Herald added a reviewer: nicolasvasilache. · View Herald TranscriptJul 7 2023, 5:29 AM
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: stephenneuendorffer, nicolasvasilache. · View Herald Transcript
zero9178 added a parent revision: D154587: [mlir][LLVM] Add `BitcastStores` type-consistency pattern.Jul 7 2023, 5:29 AM
Harbormaster completed remote builds in B243747: Diff 538101.Jul 7 2023, 6:14 AM
gysit added inline comments.Jul 9 2023, 9:14 AM
mlir/lib/Dialect/LLVMIR/Transforms/TypeConsistency.cpp
395

I guess there is a slight risk here that we may not be able to split the store into the nested array/struct?

515–522
zero9178 updated this revision to Diff 538522.Jul 10 2023, 12:19 AM
zero9178 marked an inline comment as done.

address review comments

mlir/lib/Dialect/LLVMIR/Transforms/TypeConsistency.cpp
395

Yes. This transformation is generally optimistic in that it always assumes that integer stores into structs it generates can be further split up. Checking otherwise would require doing a recursive check of the fields/elements of the aggregate.
That said, I think its a rather unlikely case that an integer store would create consistent stores into some fields followed by an inconsistent store into a subaggregate. And even if, the generated code stays correct.

Harbormaster completed remote builds in B244055: Diff 538522.Jul 10 2023, 12:33 AM
zero9178 updated this revision to Diff 538594.Jul 10 2023, 5:42 AM

add explicit check making sure that when splitting an integer only partially storing into an aggregate that it can also be split afterwards

zero9178 added a child revision: D154840: [mlir][LLVM] Add support for arrays in `SplitStores` pattern.Jul 10 2023, 6:15 AM
Harbormaster completed remote builds in B244109: Diff 538594.Jul 10 2023, 6:19 AM
gysit accepted this revision.Jul 10 2023, 6:27 AM

Thanks LGTM modulo ultra nit.

mlir/test/Dialect/LLVMIR/type-consistency.mlir
612
This revision is now accepted and ready to land.Jul 10 2023, 6:27 AM
Closed by commit rG7786449334d8: [mlir][LLVM] Make `SplitStores` pattern capable of writing to sub-aggregates (authored by zero9178). · Explain WhyJul 10 2023, 6:28 AM
This revision was automatically updated to reflect the committed changes.
zero9178 added a commit: rG7786449334d8: [mlir][LLVM] Make `SplitStores` pattern capable of writing to sub-aggregates.
zero9178 marked an inline comment as done.Jul 10 2023, 6:29 AM

Revision Contents

PathSize
mlir/
lib/
Dialect/
LLVMIR/
Transforms/
68 lines
test/
Dialect/
LLVMIR/
131 lines

Diff 538611

mlir/lib/Dialect/LLVMIR/Transforms/TypeConsistency.cpp

Show First 20 Lines • Show All 351 Lines • ▼ Show 20 Lines
/// operation writing `storeSize` bytes at `storeOffset` within the struct. /// operation writing `storeSize` bytes at `storeOffset` within the struct.
/// `storeOffset` is required to cleanly point to an immediate field within /// `storeOffset` is required to cleanly point to an immediate field within
/// the struct. /// the struct.
/// If the write operation were to write to any padding, write beyond the /// If the write operation were to write to any padding, write beyond the
/// struct, partially write to a field, or contains currently unsupported /// struct, partially write to a field, or contains currently unsupported
/// types, failure is returned. /// types, failure is returned.
static FailureOr<ArrayRef<Type>> static FailureOr<ArrayRef<Type>>
getWrittenToFields(const DataLayout &dataLayout, LLVMStructType structType, getWrittenToFields(const DataLayout &dataLayout, LLVMStructType structType,
int storeSize, unsigned storeOffset) { unsigned storeSize, unsigned storeOffset) {
ArrayRef<Type> body = structType.getBody(); ArrayRef<Type> body = structType.getBody();
unsigned currentOffset = 0; unsigned currentOffset = 0;
body = body.drop_until([&](Type type) { body = body.drop_until([&](Type type) {
if (!structType.isPacked()) { if (!structType.isPacked()) {
unsigned alignment = dataLayout.getTypeABIAlignment(type); unsigned alignment = dataLayout.getTypeABIAlignment(type);
currentOffset = llvm::alignTo(currentOffset, alignment); currentOffset = llvm::alignTo(currentOffset, alignment);
} }
// currentOffset is guaranteed to be equal to offset since offset is either // currentOffset is guaranteed to be equal to offset since offset is either
// 0 or stems from a type-consistent GEP indexing into just a single // 0 or stems from a type-consistent GEP indexing into just a single
// aggregate. // aggregate.
if (currentOffset == storeOffset) if (currentOffset == storeOffset)
return true; return true;
assert(currentOffset < storeOffset && assert(currentOffset < storeOffset &&
"storeOffset should cleanly point into an immediate field"); "storeOffset should cleanly point into an immediate field");
currentOffset += dataLayout.getTypeSize(type); currentOffset += dataLayout.getTypeSize(type);
return false; return false;
}); });
size_t exclusiveEnd = 0; size_t exclusiveEnd = 0;
for (; exclusiveEnd < body.size() && storeSize > 0; exclusiveEnd++) { for (; exclusiveEnd < body.size() && storeSize > 0; exclusiveEnd++) {
// Not yet recursively handling aggregates, only primitives.
if (!isa<IntegerType, FloatType>(body[exclusiveEnd]))
return failure();
if (!structType.isPacked()) { if (!structType.isPacked()) {
unsigned alignment = dataLayout.getTypeABIAlignment(body[exclusiveEnd]); unsigned alignment = dataLayout.getTypeABIAlignment(body[exclusiveEnd]);
// No padding allowed inbetween fields at this point in time. // No padding allowed inbetween fields at this point in time.
if (!llvm::isAligned(llvm::Align(alignment), currentOffset)) if (!llvm::isAligned(llvm::Align(alignment), currentOffset))
return failure(); return failure();
} }
unsigned fieldSize = dataLayout.getTypeSize(body[exclusiveEnd]); unsigned fieldSize = dataLayout.getTypeSize(body[exclusiveEnd]);
if (fieldSize > storeSize) {
// Partial writes into an aggregate are okay since subsequent pattern
// applications can further split these up into writes into the
// sub-elements.
gysitUnsubmitted
Not Done
ReplyInline Actions

I guess there is a slight risk here that we may not be able to split the store into the nested array/struct?

gysit: I guess there is a slight risk here that we may not be able to split the store into the nested…
zero9178AuthorUnsubmitted
Done
ReplyInline Actions

Yes. This transformation is generally optimistic in that it always assumes that integer stores into structs it generates can be further split up. Checking otherwise would require doing a recursive check of the fields/elements of the aggregate.
That said, I think its a rather unlikely case that an integer store would create consistent stores into some fields followed by an inconsistent store into a subaggregate. And even if, the generated code stays correct.

zero9178: Yes. This transformation is generally optimistic in that it always assumes that integer stores…
auto subStruct = dyn_cast<LLVMStructType>(body[exclusiveEnd]);
if (!subStruct)
return failure();
// Avoid splitting redundantly by making sure the store into the struct
// can actually be split.
if (failed(getWrittenToFields(dataLayout, subStruct, storeSize,
/*storeOffset=*/0)))
return failure();
return body.take_front(exclusiveEnd + 1);
}
currentOffset += fieldSize; currentOffset += fieldSize;
storeSize -= fieldSize; storeSize -= fieldSize;
} }
// If the storeSize is not 0 at this point we are either partially writing // If the storeSize is not 0 at this point we are writing past the aggregate
// into a field or writing past the aggregate as a whole. Abort. // as a whole. Abort.
if (storeSize != 0) if (storeSize > 0)
return failure(); return failure();
return body.take_front(exclusiveEnd); return body.take_front(exclusiveEnd);
} }
/// Splits a store of the vector `value` into `address` at `storeOffset` into /// Splits a store of the vector `value` into `address` at `storeOffset` into
/// multiple stores of each element with the goal of each generated store /// multiple stores of each element with the goal of each generated store
/// becoming type-consistent through subsequent pattern applications. /// becoming type-consistent through subsequent pattern applications.
static void splitVectorStore(const DataLayout &dataLayout, Location loc, static void splitVectorStore(const DataLayout &dataLayout, Location loc,
Show All 19 Linesstatic void splitVectorStore(const DataLayout &dataLayout, Location loc,
} }
} }
/// Splits a store of the integer `value` into `address` at `storeOffset` into /// Splits a store of the integer `value` into `address` at `storeOffset` into
/// multiple stores to each 'writtenFields', making each store operation /// multiple stores to each 'writtenFields', making each store operation
/// type-consistent. /// type-consistent.
static void splitIntegerStore(const DataLayout &dataLayout, Location loc, static void splitIntegerStore(const DataLayout &dataLayout, Location loc,
RewriterBase &rewriter, Value address, RewriterBase &rewriter, Value address,
Value value, unsigned storeOffset, Value value, unsigned storeSize,
unsigned storeOffset,
ArrayRef<Type> writtenToFields) { ArrayRef<Type> writtenToFields) {
unsigned currentOffset = storeOffset; unsigned currentOffset = storeOffset;
for (Type type : writtenToFields) { for (Type type : writtenToFields) {
unsigned fieldSize = dataLayout.getTypeSize(type); unsigned fieldSize = dataLayout.getTypeSize(type);
// Extract the data out of the integer by first shifting right and then // Extract the data out of the integer by first shifting right and then
// truncating it. // truncating it.
auto pos = rewriter.create<ConstantOp>( auto pos = rewriter.create<ConstantOp>(
loc, rewriter.getIntegerAttr(value.getType(), loc, rewriter.getIntegerAttr(value.getType(),
(currentOffset - storeOffset) * 8)); (currentOffset - storeOffset) * 8));
auto shrOp = rewriter.create<LShrOp>(loc, value, pos); auto shrOp = rewriter.create<LShrOp>(loc, value, pos);
IntegerType fieldIntType = rewriter.getIntegerType(fieldSize * 8); // If we are doing a partial write into a direct field the remaining
// `storeSize` will be less than the size of the field. We have to truncate
// to the `storeSize` to avoid creating a store that wasn't in the original
// code.
IntegerType fieldIntType =
rewriter.getIntegerType(std::min(fieldSize, storeSize) * 8);
Value valueToStore = rewriter.create<TruncOp>(loc, fieldIntType, shrOp); Value valueToStore = rewriter.create<TruncOp>(loc, fieldIntType, shrOp);
// We create an `i8` indexed GEP here as that is the easiest (offset is // We create an `i8` indexed GEP here as that is the easiest (offset is
// already known). Other patterns turn this into a type-consistent GEP. // already known). Other patterns turn this into a type-consistent GEP.
auto gepOp = auto gepOp =
rewriter.create<GEPOp>(loc, address.getType(), rewriter.getI8Type(), rewriter.create<GEPOp>(loc, address.getType(), rewriter.getI8Type(),
address, ArrayRef<GEPArg>{currentOffset}); address, ArrayRef<GEPArg>{currentOffset});
rewriter.create<StoreOp>(loc, valueToStore, gepOp); rewriter.create<StoreOp>(loc, valueToStore, gepOp);
// No need to care about padding here since we already checked previously // No need to care about padding here since we already checked previously
// that no padding exists in this range. // that no padding exists in this range.
currentOffset += fieldSize; currentOffset += fieldSize;
storeSize -= fieldSize;
} }
} }
LogicalResult SplitStores::matchAndRewrite(StoreOp store, LogicalResult SplitStores::matchAndRewrite(StoreOp store,
PatternRewriter &rewriter) const { PatternRewriter &rewriter) const {
Type sourceType = store.getValue().getType(); Type sourceType = store.getValue().getType();
if (!isa<IntegerType, VectorType>(sourceType)) { if (!isa<IntegerType, VectorType>(sourceType)) {
// We currently only support integer and vector sources. // We currently only support integer and vector sources.
return failure(); return failure();
} }
Type typeHint = isElementTypeInconsistent(store.getAddr(), sourceType); Type typeHint = isElementTypeInconsistent(store.getAddr(), sourceType);
if (!typeHint) { if (!typeHint) {
// Nothing to do, since it is already consistent. // Nothing to do, since it is already consistent.
return failure(); return failure();
} }
auto dataLayout = DataLayout::closest(store); auto dataLayout = DataLayout::closest(store);
unsigned storeSize = dataLayout.getTypeSize(sourceType);
unsigned offset = 0; unsigned offset = 0;
Value address = store.getAddr(); Value address = store.getAddr();
if (auto gepOp = address.getDefiningOp<GEPOp>()) { if (auto gepOp = address.getDefiningOp<GEPOp>()) {
// Currently only handle canonical GEPs with exactly two indices, // Currently only handle canonical GEPs with exactly two indices,
// indexing a single aggregate deep. // indexing a single aggregate deep.
// Recursing into sub-structs is left as a future exercise.
// If the GEP is not canonical we have to fail, otherwise we would not // If the GEP is not canonical we have to fail, otherwise we would not
// create type-consistent IR. // create type-consistent IR.
if (gepOp.getIndices().size() != 2 || if (gepOp.getIndices().size() != 2 ||
succeeded(getRequiredConsistentGEPType(gepOp))) succeeded(getRequiredConsistentGEPType(gepOp)))
return failure(); return failure();
// A GEP might point somewhere into the middle of an aggregate with the // If the size of the element indexed by the GEP is smaller than the store
// store storing into multiple adjacent elements. Destructure into // size, it is pointing into the middle of an aggregate with the store
// the base address with an offset. // storing into multiple adjacent elements. Destructure into the base
// address of the aggregate with a store offset.
if (storeSize > dataLayout.getTypeSize(gepOp.getResultPtrElementType())) {
std::optional<uint64_t> byteOffset = gepToByteOffset(dataLayout, gepOp); std::optional<uint64_t> byteOffset = gepToByteOffset(dataLayout, gepOp);
if (!byteOffset) if (!byteOffset)
return failure(); return failure();
gysitUnsubmitted
Done
ReplyInline Actions
return failure();
- // If the element's size the GEP is pointing at is smaller than the store
+ // If the size of the element indexed by the GEP is smaller than the store
// size, it is pointing into the middle of an aggregate with the store
gysit:
offset = *byteOffset; offset = *byteOffset;
typeHint = gepOp.getSourceElementType(); typeHint = gepOp.getSourceElementType();
address = gepOp.getBase(); address = gepOp.getBase();
} }
}
auto structType = typeHint.dyn_cast<LLVMStructType>(); auto structType = typeHint.dyn_cast<LLVMStructType>();
if (!structType) { if (!structType) {
// TODO: Handle array types in the future. // TODO: Handle array types in the future.
return failure(); return failure();
} }
FailureOr<ArrayRef<Type>> writtenToFields = FailureOr<ArrayRef<Type>> writtenToFields =
getWrittenToFields(dataLayout, structType, getWrittenToFields(dataLayout, structType, storeSize, offset);
/*storeSize=*/dataLayout.getTypeSize(sourceType),
/*storeOffset=*/offset);
if (failed(writtenToFields)) if (failed(writtenToFields))
return failure(); return failure();
if (writtenToFields->size() <= 1) { if (writtenToFields->size() <= 1) {
// Other patterns should take care of this case, we are only interested in // Other patterns should take care of this case, we are only interested in
// splitting field stores. // splitting field stores.
return failure(); return failure();
} }
if (isa<IntegerType>(sourceType)) { if (isa<IntegerType>(sourceType)) {
splitIntegerStore(dataLayout, store.getLoc(), rewriter, address, splitIntegerStore(dataLayout, store.getLoc(), rewriter, address,
store.getValue(), offset, *writtenToFields); store.getValue(), storeSize, offset, *writtenToFields);
rewriter.eraseOp(store); rewriter.eraseOp(store);
return success(); return success();
} }
// Add a reasonable bound to not split very large vectors that would end up // Add a reasonable bound to not split very large vectors that would end up
// generating lots of code. // generating lots of code.
if (dataLayout.getTypeSizeInBits(sourceType) > maxVectorSplitSize) if (dataLayout.getTypeSizeInBits(sourceType) > maxVectorSplitSize)
return failure(); return failure();
▲ Show 20 Lines • Show All 97 Lines • Show Last 20 Lines

mlir/test/Dialect/LLVMIR/type-consistency.mlir

Show First 20 Lines • Show All 487 Lines • ▼ Show 20 Linesllvm.func @gep_result_ptr_type_dynamic(%arg: i64) {
%3 = llvm.getelementptr %1[0, %arg, 0] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.array<2 x struct<"foo", (i64)>> %3 = llvm.getelementptr %1[0, %arg, 0] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.array<2 x struct<"foo", (i64)>>
// CHECK: %[[TOP_GEP:.*]] = llvm.getelementptr %[[ALLOCA]][0, %[[ARG]]] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.array<2 x struct<"foo", (i64)>> // CHECK: %[[TOP_GEP:.*]] = llvm.getelementptr %[[ALLOCA]][0, %[[ARG]]] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.array<2 x struct<"foo", (i64)>>
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[TOP_GEP]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i64)> // CHECK: %[[GEP:.*]] = llvm.getelementptr %[[TOP_GEP]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i64)>
// CHECK: llvm.store %[[ARG]], %[[GEP]] // CHECK: llvm.store %[[ARG]], %[[GEP]]
llvm.store %arg, %3 : i64, !llvm.ptr llvm.store %arg, %3 : i64, !llvm.ptr
// CHECK-NOT: llvm.store %[[ARG]], %[[ALLOCA]] // CHECK-NOT: llvm.store %[[ARG]], %[[ALLOCA]]
llvm.return llvm.return
} }
// -----
// CHECK-LABEL: llvm.func @overlapping_int_aggregate_store
// CHECK-SAME: %[[ARG:.*]]: i64
llvm.func @overlapping_int_aggregate_store(%arg: i64) {
// CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i64) : i64
// CHECK: %[[CST16:.*]] = llvm.mlir.constant(16 : i64) : i64
%0 = llvm.mlir.constant(1 : i32) : i32
// CHECK: %[[ALLOCA:.*]] = llvm.alloca %{{.*}} x !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)>
%1 = llvm.alloca %0 x !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)> : (i32) -> !llvm.ptr
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[ALLOCA]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)>
// CHECK: %[[SHR:.*]] = llvm.lshr %[[ARG]], %[[CST0]]
// CHECK: %[[TRUNC:.*]] = llvm.trunc %[[SHR]] : i64 to i16
// CHECK: llvm.store %[[TRUNC]], %[[GEP]]
// CHECK: %[[SHR:.*]] = llvm.lshr %[[ARG]], %[[CST16]] : i64
// CHECK: [[TRUNC:.*]] = llvm.trunc %[[SHR]] : i64 to i48
// CHECK: %[[TOP_GEP:.*]] = llvm.getelementptr %[[ALLOCA]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)>
// Normal integer splitting of [[TRUNC]] follows:
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[TOP_GEP]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16)>
// CHECK: llvm.store %{{.*}}, %[[GEP]]
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[TOP_GEP]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16)>
// CHECK: llvm.store %{{.*}}, %[[GEP]]
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[TOP_GEP]][0, 2] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16)>
// CHECK: llvm.store %{{.*}}, %[[GEP]]
llvm.store %arg, %1 : i64, !llvm.ptr
// CHECK-NOT: llvm.store %[[ARG]], %[[ALLOCA]]
llvm.return
}
// -----
// CHECK-LABEL: llvm.func @overlapping_vector_aggregate_store
// CHECK-SAME: %[[ARG:.*]]: vector<4xi16>
llvm.func @overlapping_vector_aggregate_store(%arg: vector<4 x i16>) {
// CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i32) : i32
// CHECK: %[[CST1:.*]] = llvm.mlir.constant(1 : i32) : i32
// CHECK: %[[CST2:.*]] = llvm.mlir.constant(2 : i32) : i32
// CHECK: %[[CST3:.*]] = llvm.mlir.constant(3 : i32) : i32
%0 = llvm.mlir.constant(1 : i32) : i32
// CHECK: %[[ALLOCA:.*]] = llvm.alloca %{{.*}} x !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)>
%1 = llvm.alloca %0 x !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)> : (i32) -> !llvm.ptr
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[ALLOCA]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)>
// CHECK: %[[EXTRACT:.*]] = llvm.extractelement %[[ARG]][%[[CST0]] : i32]
// CHECK: llvm.store %[[EXTRACT]], %[[GEP]]
// CHECK: %[[EXTRACT:.*]] = llvm.extractelement %[[ARG]][%[[CST1]] : i32]
// CHECK: %[[GEP0:.*]] = llvm.getelementptr %[[ALLOCA]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)>
// CHECK: %[[GEP1:.*]] = llvm.getelementptr %[[GEP0]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16)>
// CHECK: llvm.store %[[EXTRACT]], %[[GEP1]]
// CHECK: %[[EXTRACT:.*]] = llvm.extractelement %[[ARG]][%[[CST2]] : i32]
// CHECK: %[[GEP0:.*]] = llvm.getelementptr %[[ALLOCA]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)>
// CHECK: %[[GEP1:.*]] = llvm.getelementptr %[[GEP0]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16)>
// CHECK: llvm.store %[[EXTRACT]], %[[GEP1]]
// CHECK: %[[EXTRACT:.*]] = llvm.extractelement %[[ARG]][%[[CST3]] : i32]
// CHECK: %[[GEP0:.*]] = llvm.getelementptr %[[ALLOCA]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i16, struct<(i16, i16, i16)>)>
// CHECK: %[[GEP1:.*]] = llvm.getelementptr %[[GEP0]][0, 2] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16)>
// CHECK: llvm.store %[[EXTRACT]], %[[GEP1]]
llvm.store %arg, %1 : vector<4 x i16>, !llvm.ptr
// CHECK-NOT: llvm.store %[[ARG]], %[[ALLOCA]]
llvm.return
}
// -----
// CHECK-LABEL: llvm.func @partially_overlapping_aggregate_store
// CHECK-SAME: %[[ARG:.*]]: i64
llvm.func @partially_overlapping_aggregate_store(%arg: i64) {
// CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i64) : i64
// CHECK: %[[CST16:.*]] = llvm.mlir.constant(16 : i64) : i64
%0 = llvm.mlir.constant(1 : i32) : i32
// CHECK: %[[ALLOCA:.*]] = llvm.alloca %{{.*}} x !llvm.struct<"foo", (i16, struct<(i16, i16, i16, i16)>)>
%1 = llvm.alloca %0 x !llvm.struct<"foo", (i16, struct<(i16, i16, i16, i16)>)> : (i32) -> !llvm.ptr
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[ALLOCA]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i16, struct<(i16, i16, i16, i16)>)>
// CHECK: %[[SHR:.*]] = llvm.lshr %[[ARG]], %[[CST0]]
// CHECK: %[[TRUNC:.*]] = llvm.trunc %[[SHR]] : i64 to i16
// CHECK: llvm.store %[[TRUNC]], %[[GEP]]
// CHECK: %[[SHR:.*]] = llvm.lshr %[[ARG]], %[[CST16]] : i64
// CHECK: [[TRUNC:.*]] = llvm.trunc %[[SHR]] : i64 to i48
// CHECK: %[[TOP_GEP:.*]] = llvm.getelementptr %[[ALLOCA]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i16, struct<(i16, i16, i16, i16)>)>
// Normal integer splitting of [[TRUNC]] follows:
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[TOP_GEP]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16, i16)>
// CHECK: llvm.store %{{.*}}, %[[GEP]]
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[TOP_GEP]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16, i16)>
// CHECK: llvm.store %{{.*}}, %[[GEP]]
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[TOP_GEP]][0, 2] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(i16, i16, i16, i16)>
// CHECK: llvm.store %{{.*}}, %[[GEP]]
// It is important that there are no more stores at this point.
// Specifically a store into the fourth field of %[[TOP_GEP]] would
// incorrectly change the semantics of the code.
// CHECK-NOT: llvm.store %{{.*}}, %{{.*}}
llvm.store %arg, %1 : i64, !llvm.ptr
llvm.return
}
// -----
// Here a split is undesirable since the store does a partial store into the field.
gysitUnsubmitted
Done
ReplyInline Actions
// -----
- // Pattern here undesirable to split since the store does a partial store into the field.
+ // Here a split is undesirable since the store does a partial store into the field.
// CHECK-LABEL: llvm.func @undesirable_overlapping_aggregate_store
gysit:
// CHECK-LABEL: llvm.func @undesirable_overlapping_aggregate_store
// CHECK-SAME: %[[ARG:.*]]: i64
llvm.func @undesirable_overlapping_aggregate_store(%arg: i64) {
%0 = llvm.mlir.constant(1 : i32) : i32
// CHECK: %[[ALLOCA:.*]] = llvm.alloca %{{.*}} x !llvm.struct<"foo", (i32, i32, struct<(i64, i16, i16, i16)>)>
%1 = llvm.alloca %0 x !llvm.struct<"foo", (i32, i32, struct<(i64, i16, i16, i16)>)> : (i32) -> !llvm.ptr
// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[ALLOCA]][0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i32, i32, struct<(i64, i16, i16, i16)>)>
%2 = llvm.getelementptr %1[0, 1] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<"foo", (i32, i32, struct<(i64, i16, i16, i16)>)>
// CHECK: llvm.store %[[ARG]], %[[GEP]]
llvm.store %arg, %2 : i64, !llvm.ptr
llvm.return
}