This is an archive of the discontinued LLVM Phabricator instance.

Differential D72548

[mlir] NFC: Remove Value::operator* and Value::operator-> now that Value is properly value-typed.
ClosedPublic

Authored by rriddle on Jan 10 2020, 6:00 PM.

Details

Reviewers
jpienaar
mehdi_amini
mravishankar
antiagainst
nicolasvasilache
Commits
rG2bdf33cc4c73: [mlir] NFC: Remove Value::operator* and Value::operator-> now that Value is…
Summary

These were temporary methods used to simplify the transition.

Diff Detail

Event Timeline

rriddle created this revision.Jan 10 2020, 6:00 PM
Herald added a reviewer: mravishankar. · View Herald TranscriptJan 10 2020, 6:00 PM
Herald added a reviewer: antiagainst. · View Herald Transcript
Herald added a reviewer: nicolasvasilache. · View Herald Transcript
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: llvm-commits, aartbik, herhut and 11 others. · View Herald Transcript
merge_guards_bot added a subscriber: merge_guards_bot.Jan 10 2020, 6:17 PM

Unit tests: pass. 61744 tests passed, 0 failed and 780 were skipped.

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Harbormaster failed remote builds in B43746: Diff 237467!Jan 10 2020, 6:19 PM
antiagainst accepted this revision.Jan 11 2020, 5:18 AM
rriddle removed a reviewer: nicolasvasilache.Jan 11 2020, 8:50 AM
This revision is now accepted and ready to land.Jan 11 2020, 8:50 AM
Herald added a reviewer: nicolasvasilache. · View Herald TranscriptJan 11 2020, 8:50 AM
This revision now requires review to proceed.Jan 11 2020, 8:50 AM
This revision was not accepted when it landed; it landed in state Needs Review.Jan 11 2020, 9:06 AM
Closed by commit rG2bdf33cc4c73: [mlir] NFC: Remove Value::operator* and Value::operator-> now that Value is… (authored by rriddle). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathSize
mlir/
docs/
3 lines
2 lines
8 lines
Tutorials/
Toy/
6 lines
4 lines
examples/
toy/
Ch2/
mlir/
5 lines
Ch3/
mlir/
5 lines
2 lines
4 lines
Ch4/
mlir/
17 lines
2 lines
4 lines
Ch5/
mlir/
17 lines
2 lines
4 lines
Ch6/
mlir/
17 lines
2 lines
4 lines
Ch7/
mlir/
26 lines
4 lines
2 lines
4 lines
include/
mlir/
Analysis/
2 lines
Dialect/
AffineOps/
22 lines
2 lines
LLVMIR/
14 lines
Linalg/
EDSC/
2 lines
IR/
6 lines
4 lines
4 lines
Transforms/
2 lines
Utils/
2 lines
QuantOps/
2 lines
StandardOps/
12 lines
30 lines
VectorOps/
78 lines
4 lines
EDSC/
4 lines
2 lines
IR/
6 lines
22 lines
6 lines
6 lines
4 lines
19 lines
Quantizer/
Support/
2 lines
Transforms/
2 lines
lib/
Analysis/
4 lines
28 lines
2 lines
4 lines
18 lines
4 lines
12 lines
20 lines
2 lines
4 lines
Conversion/
GPUCommon/
4 lines
GPUToCUDA/
2 lines
GPUToNVVM/
26 lines
GPUToSPIRV/
4 lines
LinalgToLLVM/
6 lines
LoopsToGPU/
4 lines
StandardToLLVM/
46 lines
StandardToSPIRV/
22 lines
5 lines
VectorToLLVM/
14 lines
Dialect/
AffineOps/
99 lines
FxpMathOps/
Transforms/
10 lines
8 lines
GPU/
IR/
50 lines
Transforms/
8 lines
LLVMIR/
IR/
72 lines
Linalg/
Analysis/
14 lines
EDSC/
2 lines
IR/
36 lines
Transforms/
15 lines
4 lines
12 lines
14 lines
Utils/
6 lines
LoopOps/
19 lines
QuantOps/
IR/
4 lines
Transforms/
6 lines
SPIRV/
2 lines
160 lines
Serialization/
6 lines
8 lines
Transforms/
2 lines
StandardOps/
166 lines
6 lines
VectorOps/
109 lines
30 lines
EDSC/
12 lines
6 lines
IR/
16 lines
2 lines
2 lines
4 lines
24 lines
2 lines
4 lines
6 lines
6 lines
Parser/
14 lines
Quantizer/
Configurations/
18 lines
Support/
6 lines
Transforms/
8 lines
20 lines
Target/
LLVMIR/
2 lines
4 lines
Transforms/
6 lines
32 lines
22 lines
2 lines
2 lines
2 lines
8 lines
20 lines
Utils/
6 lines
8 lines
20 lines
4 lines
37 lines
6 lines
36 lines
23 lines
test/
lib/
TestDialect/
12 lines
2 lines
12 lines
Transforms/
4 lines
2 lines
2 lines
mlir-tblgen/
4 lines
4 lines
tools/
mlir-tblgen/
2 lines
14 lines
10 lines

Diff 237506

mlir/docs/DeclarativeRewrites.md

Show First 20 Lines • Show All 612 Lines • ▼ Show 20 Lines
Constraints can be placed on op arguments when matching. But sometimes we need Constraints can be placed on op arguments when matching. But sometimes we need
to also place constraints on the matched op's results or sometimes need to limit to also place constraints on the matched op's results or sometimes need to limit
the matching with some constraints that cover both the arguments and the the matching with some constraints that cover both the arguments and the
results. The third parameter to `Pattern` (and `Pat`) is for this purpose. results. The third parameter to `Pattern` (and `Pat`) is for this purpose.
For example, we can write For example, we can write
```tablegen ```tablegen
def HasNoUseOf: Constraint< def HasNoUseOf: Constraint<CPred<"$_self.use_empty()">, "has no use">;
CPred<"$_self->use_begin() == $_self->use_end()">, "has no use">;
def HasSameElementType : Constraint< def HasSameElementType : Constraint<
CPred<"$0.cast<ShapedType>().getElementType() == " CPred<"$0.cast<ShapedType>().getElementType() == "
"$1.cast<ShapedType>().getElementType()">, "$1.cast<ShapedType>().getElementType()">,
"has same element type">; "has same element type">;
def : Pattern<(TwoResultOp:$results $input), def : Pattern<(TwoResultOp:$results $input),
[(...), (...)], [(...), (...)],
▲ Show 20 Lines • Show All 60 Lines • Show Last 20 Lines

mlir/docs/OpDefinitions.md

Show First 20 Lines • Show All 728 Lines • ▼ Show 20 Lines* `$_op` will be replaced by the current operation so that you can access
information of the current operation. information of the current operation.
* `$_self` will be replaced with the entity this predicate is attached to. * `$_self` will be replaced with the entity this predicate is attached to.
E.g., `BoolAttr` is an attribute constraint that wraps a E.g., `BoolAttr` is an attribute constraint that wraps a
`CPred<"$_self.isa<BoolAttr>()">`. Then for `F32:$attr`,`$_self` will be `CPred<"$_self.isa<BoolAttr>()">`. Then for `F32:$attr`,`$_self` will be
replaced by `$attr`. For type constraints, it's a little bit special since replaced by `$attr`. For type constraints, it's a little bit special since
we want the constraints on each type definition reads naturally and we want we want the constraints on each type definition reads naturally and we want
to attach type constraints directly to an operand/result, `$_self` will be to attach type constraints directly to an operand/result, `$_self` will be
replaced by the operand/result's type. E.g., for `F32` in `F32:$operand`, its replaced by the operand/result's type. E.g., for `F32` in `F32:$operand`, its
`$_self` will be expanded as `getOperand(...)->getType()`. `$_self` will be expanded as `getOperand(...).getType()`.
TODO(b/130663252): Reconsider the leading symbol for special placeholders. TODO(b/130663252): Reconsider the leading symbol for special placeholders.
Eventually we want to allow referencing operand/result $-names; such $-names Eventually we want to allow referencing operand/result $-names; such $-names
can start with underscore. can start with underscore.
For example, to write an attribute `attr` is an `IntegerAttr`, in C++ you can For example, to write an attribute `attr` is an `IntegerAttr`, in C++ you can
just call `attr.isa<IntegerAttr>()`. The code can be wrapped in a `CPred` as just call `attr.isa<IntegerAttr>()`. The code can be wrapped in a `CPred` as
`$_self.isa<IntegerAttr>()`, with `$_self` as the special placeholder to be `$_self.isa<IntegerAttr>()`, with `$_self` as the special placeholder to be
▲ Show 20 Lines • Show All 465 Lines • Show Last 20 Lines

mlir/docs/QuickstartRewrites.md

Show First 20 Lines • Show All 115 Lines • ▼ Show 20 Lines
If this were a more advance pattern that the current framework could not express If this were a more advance pattern that the current framework could not express
as destination then one could use a general native code fallback method. This as destination then one could use a general native code fallback method. This
consists of defining a pattern as well as adding a C++ function to perform the consists of defining a pattern as well as adding a C++ function to perform the
replacement: replacement:
```tablegen ```tablegen
def createTFLLeakyRelu : NativeCodeCall< def createTFLLeakyRelu : NativeCodeCall<
"createTFLLeakyRelu($_builder, $0->getDefiningOp(), $1, $2)">; "createTFLLeakyRelu($_builder, $0.getDefiningOp(), $1, $2)">;
def : Pat<(TF_LeakyReluOp:$old_value, $arg, F32Attr:$a), def : Pat<(TF_LeakyReluOp:$old_value, $arg, F32Attr:$a),
(createTFLLeakyRelu $old_value, $arg, $a)>; (createTFLLeakyRelu $old_value, $arg, $a)>;
``` ```
```c++ ```c++
static Value createTFLLeakyRelu(PatternRewriter &rewriter, Operation *op, static Value createTFLLeakyRelu(PatternRewriter &rewriter, Operation *op,
Value operand, Attribute attr) { Value operand, Attribute attr) {
return rewriter.create<mlir::TFL::LeakyReluOp>( return rewriter.create<mlir::TFL::LeakyReluOp>(
op->getLoc(), operands[0]->getType(), /*arg=*/operands[0], op->getLoc(), operands[0].getType(), /*arg=*/operands[0],
/*alpha=*/attrs[0].cast<FloatAttr>()); /*alpha=*/attrs[0].cast<FloatAttr>());
} }
``` ```
This allows for arbitrarily complex builders. Input pattern side one can express This allows for arbitrarily complex builders. Input pattern side one can express
multi-op patterns with constraints on input operands and attributes. But input multi-op patterns with constraints on input operands and attributes. But input
patterns cannot yet express constraints across multiple operands/attributes. patterns cannot yet express constraints across multiple operands/attributes.
Show All 29 Lines ConvertTFLeakyRelu(MLIRContext *context)
: RewritePattern("tf.LeakyRelu", 1, context) {} : RewritePattern("tf.LeakyRelu", 1, context) {}
PatternMatchResult match(Operation *op) const override { PatternMatchResult match(Operation *op) const override {
return matchSuccess(); return matchSuccess();
} }
void rewrite(Operation *op, PatternRewriter &rewriter) const override { void rewrite(Operation *op, PatternRewriter &rewriter) const override {
rewriter.replaceOpWithNewOp<TFL::LeakyReluOp>( rewriter.replaceOpWithNewOp<TFL::LeakyReluOp>(
op, op->getResult(0)->getType(), op->getOperand(0), op, op->getResult(0).getType(), op->getOperand(0),
/*alpha=*/op->getAttrOfType<FloatAttr>("alpha")); /*alpha=*/op->getAttrOfType<FloatAttr>("alpha"));
} }
}; };
/// Single-step rewrite with "matchAndRewrite". This allows for performing the /// Single-step rewrite with "matchAndRewrite". This allows for performing the
/// rewrite immediately upon a successful match. /// rewrite immediately upon a successful match.
struct ConvertTFLeakyRelu : public RewritePattern { struct ConvertTFLeakyRelu : public RewritePattern {
ConvertTFLeakyRelu(MLIRContext *context) ConvertTFLeakyRelu(MLIRContext *context)
: RewritePattern("tf.LeakyRelu", 1, context) {} : RewritePattern("tf.LeakyRelu", 1, context) {}
PatternMatchResult matchAndRewrite(Operation *op, PatternMatchResult matchAndRewrite(Operation *op,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
rewriter.replaceOpWithNewOp<TFL::LeakyReluOp>( rewriter.replaceOpWithNewOp<TFL::LeakyReluOp>(
op, op->getResult(0)->getType(), op->getOperand(0), op, op->getResult(0).getType(), op->getOperand(0),
/*alpha=*/op->getAttrOfType<FloatAttr>("alpha")); /*alpha=*/op->getAttrOfType<FloatAttr>("alpha"));
return matchSuccess(); return matchSuccess();
} }
}; };
``` ```
In the C++ rewrite the static benefit of the rewrite pattern is specified at In the C++ rewrite the static benefit of the rewrite pattern is specified at
construction. While in the pattern generator a simple heuristic is currently construction. While in the pattern generator a simple heuristic is currently
▲ Show 20 Lines • Show All 53 Lines • Show Last 20 Lines

mlir/docs/Tutorials/Toy/Ch-3.md

Show First 20 Lines • Show All 86 Lines • ▼ Show 20 Linesstruct SimplifyRedundantTranspose : public mlir::OpRewritePattern<TransposeOp> {
/// argument is the orchestrator of the sequence of rewrites. It is expected /// argument is the orchestrator of the sequence of rewrites. It is expected
/// to interact with it to perform any changes to the IR from here. /// to interact with it to perform any changes to the IR from here.
mlir::PatternMatchResult mlir::PatternMatchResult
matchAndRewrite(TransposeOp op, matchAndRewrite(TransposeOp op,
mlir::PatternRewriter &rewriter) const override { mlir::PatternRewriter &rewriter) const override {
// Look through the input of the current transpose. // Look through the input of the current transpose.
mlir::Value transposeInput = op.getOperand(); mlir::Value transposeInput = op.getOperand();
TransposeOp transposeInputOp = TransposeOp transposeInputOp =
llvm::dyn_cast_or_null<TransposeOp>(transposeInput->getDefiningOp()); llvm::dyn_cast_or_null<TransposeOp>(transposeInput.getDefiningOp());
// If the input is defined by another Transpose, bingo! // If the input is defined by another Transpose, bingo!
if (!transposeInputOp) if (!transposeInputOp)
return matchFailure(); return matchFailure();
// Use the rewriter to perform the replacement // Use the rewriter to perform the replacement
rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp}); rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp});
return matchSuccess(); return matchSuccess();
} }
▲ Show 20 Lines • Show All 85 Lines • ▼ Show 20 Lines
can be found under path/to/BUILD/projects/mlir/examples/toy/Ch3/ToyCombine.inc. can be found under path/to/BUILD/projects/mlir/examples/toy/Ch3/ToyCombine.inc.
DRR also provides a method for adding argument constraints when the DRR also provides a method for adding argument constraints when the
transformation is conditional on some properties of the arguments and results. transformation is conditional on some properties of the arguments and results.
An example is a transformation that eliminates reshapes when they are redundant, An example is a transformation that eliminates reshapes when they are redundant,
i.e. when the input and output shapes are identical. i.e. when the input and output shapes are identical.
```tablegen ```tablegen
def TypesAreIdentical : Constraint<CPred<"$0->getType() == $1->getType()">>; def TypesAreIdentical : Constraint<CPred<"$0.getType() == $1.getType()">>;
def RedundantReshapeOptPattern : Pat< def RedundantReshapeOptPattern : Pat<
(ReshapeOp:$res $arg), (replaceWithValue $arg), (ReshapeOp:$res $arg), (replaceWithValue $arg),
[(TypesAreIdentical $res, $arg)]>; [(TypesAreIdentical $res, $arg)]>;
``` ```
Some optimizations may require additional transformations on instruction Some optimizations may require additional transformations on instruction
arguments. This is achieved using NativeCodeCall, which allows for more complex arguments. This is achieved using NativeCodeCall, which allows for more complex
transformations either by calling into a C++ helper function or by using inline transformations either by calling into a C++ helper function or by using inline
C++. An example of such an optimization is FoldConstantReshape, where we C++. An example of such an optimization is FoldConstantReshape, where we
optimize Reshape of a constant value by reshaping the constant in place and optimize Reshape of a constant value by reshaping the constant in place and
eliminating the reshape operation. eliminating the reshape operation.
```tablegen ```tablegen
def ReshapeConstant : NativeCodeCall<"$0.reshape(($1->getType()).cast<ShapedType>())">; def ReshapeConstant : NativeCodeCall<"$0.reshape(($1.getType()).cast<ShapedType>())">;
def FoldConstantReshapeOptPattern : Pat< def FoldConstantReshapeOptPattern : Pat<
(ReshapeOp:$res (ConstantOp $arg)), (ReshapeOp:$res (ConstantOp $arg)),
(ConstantOp (ReshapeConstant $arg, $res))>; (ConstantOp (ReshapeConstant $arg, $res))>;
``` ```
We demonstrate these reshape optimizations using the following We demonstrate these reshape optimizations using the following
trivialReshape.toy program: trivialReshape.toy program:
▲ Show 20 Lines • Show All 45 Lines • Show Last 20 Lines

mlir/docs/Tutorials/Toy/Ch-4.md

Show First 20 Lines • Show All 76 Lines • ▼ Show 20 Linesstruct ToyInlinerInterface : public DialectInlinerInterface {
void handleTerminator(Operation *op, void handleTerminator(Operation *op,
ArrayRef<Value> valuesToRepl) const final { ArrayRef<Value> valuesToRepl) const final {
// Only "toy.return" needs to be handled here. // Only "toy.return" needs to be handled here.
auto returnOp = cast<ReturnOp>(op); auto returnOp = cast<ReturnOp>(op);
// Replace the values directly with the return operands. // Replace the values directly with the return operands.
assert(returnOp.getNumOperands() == valuesToRepl.size()); assert(returnOp.getNumOperands() == valuesToRepl.size());
for (const auto &it : llvm::enumerate(returnOp.getOperands())) for (const auto &it : llvm::enumerate(returnOp.getOperands()))
valuesToRepl[it.index()]->replaceAllUsesWith(it.value()); valuesToRepl[it.index()].replaceAllUsesWith(it.value());
} }
}; };
``` ```
We then register our dialect interface directly on the Toy dialect, similarly to We then register our dialect interface directly on the Toy dialect, similarly to
how we did for operations. how we did for operations.
```c++ ```c++
▲ Show 20 Lines • Show All 211 Lines • ▼ Show 20 Lines
Each of these operations will then need to provide a definition for the Each of these operations will then need to provide a definition for the
`inferShapes()` method. As an example, for the mul op, the result shape is `inferShapes()` method. As an example, for the mul op, the result shape is
inferred as the shape of the inputs. inferred as the shape of the inputs.
```c++ ```c++
/// Infer the output shape of the MulOp, this is required by the shape inference /// Infer the output shape of the MulOp, this is required by the shape inference
/// interface. /// interface.
void MulOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void MulOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
``` ```
At this point, each of the necessary Toy operations provide a mechanism by which At this point, each of the necessary Toy operations provide a mechanism by which
to infer their output shapes. The ShapeInferencePass is a FunctionPass: it will to infer their output shapes. The ShapeInferencePass is a FunctionPass: it will
runs on each Function in isolation. MLIR also supports general runs on each Function in isolation. MLIR also supports general
[OperationPasses](../../WritingAPass.md#operation-pass) that run on any isolated [OperationPasses](../../WritingAPass.md#operation-pass) that run on any isolated
operation (i.e. other function-like operations), but here our module only operation (i.e. other function-like operations), but here our module only
contains functions, so there is no need to generalize to all operations. contains functions, so there is no need to generalize to all operations.
▲ Show 20 Lines • Show All 66 Lines • Show Last 20 Lines

mlir/examples/toy/Ch2/mlir/Dialect.cpp

Show First 20 Lines • Show All 48 Lines • ▼ Show 20 Linesvoid ConstantOp::build(mlir::Builder *builder, mlir::OperationState &state,
ConstantOp::build(builder, state, dataType, dataAttribute); ConstantOp::build(builder, state, dataType, dataAttribute);
} }
/// Verifier for the constant operation. This corresponds to the `::verify(...)` /// Verifier for the constant operation. This corresponds to the `::verify(...)`
/// in the op definition. /// in the op definition.
static mlir::LogicalResult verify(ConstantOp op) { static mlir::LogicalResult verify(ConstantOp op) {
// If the return type of the constant is not an unranked tensor, the shape // If the return type of the constant is not an unranked tensor, the shape
// must match the shape of the attribute holding the data. // must match the shape of the attribute holding the data.
auto resultType = auto resultType = op.getResult().getType().dyn_cast<mlir::RankedTensorType>();
op.getResult()->getType().dyn_cast<mlir::RankedTensorType>();
if (!resultType) if (!resultType)
return success(); return success();
// Check that the rank of the attribute type matches the rank of the constant // Check that the rank of the attribute type matches the rank of the constant
// result type. // result type.
auto attrType = op.value().getType().cast<mlir::TensorType>(); auto attrType = op.value().getType().cast<mlir::TensorType>();
if (attrType.getRank() != resultType.getRank()) { if (attrType.getRank() != resultType.getRank()) {
return op.emitOpError( return op.emitOpError(
▲ Show 20 Lines • Show All 86 Lines • ▼ Show 20 Lines
void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state, void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value value) { mlir::Value value) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands(value); state.addOperands(value);
} }
static mlir::LogicalResult verify(TransposeOp op) { static mlir::LogicalResult verify(TransposeOp op) {
auto inputType = op.getOperand()->getType().dyn_cast<RankedTensorType>(); auto inputType = op.getOperand().getType().dyn_cast<RankedTensorType>();
auto resultType = op.getType().dyn_cast<RankedTensorType>(); auto resultType = op.getType().dyn_cast<RankedTensorType>();
if (!inputType || !resultType) if (!inputType || !resultType)
return mlir::success(); return mlir::success();
auto inputShape = inputType.getShape(); auto inputShape = inputType.getShape();
if (!std::equal(inputShape.begin(), inputShape.end(), if (!std::equal(inputShape.begin(), inputShape.end(),
resultType.getShape().rbegin())) { resultType.getShape().rbegin())) {
return op.emitError() return op.emitError()
Show All 11 Lines

mlir/examples/toy/Ch3/mlir/Dialect.cpp

Show First 20 Lines • Show All 48 Lines • ▼ Show 20 Linesvoid ConstantOp::build(mlir::Builder *builder, mlir::OperationState &state,
ConstantOp::build(builder, state, dataType, dataAttribute); ConstantOp::build(builder, state, dataType, dataAttribute);
} }
/// Verifier for the constant operation. This corresponds to the `::verify(...)` /// Verifier for the constant operation. This corresponds to the `::verify(...)`
/// in the op definition. /// in the op definition.
static mlir::LogicalResult verify(ConstantOp op) { static mlir::LogicalResult verify(ConstantOp op) {
// If the return type of the constant is not an unranked tensor, the shape // If the return type of the constant is not an unranked tensor, the shape
// must match the shape of the attribute holding the data. // must match the shape of the attribute holding the data.
auto resultType = auto resultType = op.getResult().getType().dyn_cast<mlir::RankedTensorType>();
op.getResult()->getType().dyn_cast<mlir::RankedTensorType>();
if (!resultType) if (!resultType)
return success(); return success();
// Check that the rank of the attribute type matches the rank of the constant // Check that the rank of the attribute type matches the rank of the constant
// result type. // result type.
auto attrType = op.value().getType().cast<mlir::TensorType>(); auto attrType = op.value().getType().cast<mlir::TensorType>();
if (attrType.getRank() != resultType.getRank()) { if (attrType.getRank() != resultType.getRank()) {
return op.emitOpError( return op.emitOpError(
▲ Show 20 Lines • Show All 86 Lines • ▼ Show 20 Lines
void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state, void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value value) { mlir::Value value) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands(value); state.addOperands(value);
} }
static mlir::LogicalResult verify(TransposeOp op) { static mlir::LogicalResult verify(TransposeOp op) {
auto inputType = op.getOperand()->getType().dyn_cast<RankedTensorType>(); auto inputType = op.getOperand().getType().dyn_cast<RankedTensorType>();
auto resultType = op.getType().dyn_cast<RankedTensorType>(); auto resultType = op.getType().dyn_cast<RankedTensorType>();
if (!inputType || !resultType) if (!inputType || !resultType)
return mlir::success(); return mlir::success();
auto inputShape = inputType.getShape(); auto inputShape = inputType.getShape();
if (!std::equal(inputShape.begin(), inputShape.end(), if (!std::equal(inputShape.begin(), inputShape.end(),
resultType.getShape().rbegin())) { resultType.getShape().rbegin())) {
return op.emitError() return op.emitError()
Show All 11 Lines

mlir/examples/toy/Ch3/mlir/ToyCombine.cpp

Show All 35 Linesstruct SimplifyRedundantTranspose : public mlir::OpRewritePattern<TransposeOp> {
/// argument is the orchestrator of the sequence of rewrites. The pattern is /// argument is the orchestrator of the sequence of rewrites. The pattern is
/// expected to interact with it to perform any changes to the IR from here. /// expected to interact with it to perform any changes to the IR from here.
mlir::PatternMatchResult mlir::PatternMatchResult
matchAndRewrite(TransposeOp op, matchAndRewrite(TransposeOp op,
mlir::PatternRewriter &rewriter) const override { mlir::PatternRewriter &rewriter) const override {
// Look through the input of the current transpose. // Look through the input of the current transpose.
mlir::Value transposeInput = op.getOperand(); mlir::Value transposeInput = op.getOperand();
TransposeOp transposeInputOp = TransposeOp transposeInputOp =
llvm::dyn_cast_or_null<TransposeOp>(transposeInput->getDefiningOp()); llvm::dyn_cast_or_null<TransposeOp>(transposeInput.getDefiningOp());
// If the input is defined by another Transpose, bingo! // If the input is defined by another Transpose, bingo!
if (!transposeInputOp) if (!transposeInputOp)
return matchFailure(); return matchFailure();
// Use the rewriter to perform the replacement. // Use the rewriter to perform the replacement.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp}); rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp});
return matchSuccess(); return matchSuccess();
Show All 17 Lines

mlir/examples/toy/Ch3/mlir/ToyCombine.td

Show All 35 Lines
// Pattern-Match and Rewrite using Native Code Call // Pattern-Match and Rewrite using Native Code Call
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Native Code Calls may be used for more complex transformations using inline // Native Code Calls may be used for more complex transformations using inline
// C++ and C++ helper functions. // C++ and C++ helper functions.
// Reshape(Constant(x)) = x' // Reshape(Constant(x)) = x'
def ReshapeConstant : def ReshapeConstant :
NativeCodeCall<"$0.reshape(($1->getType()).cast<ShapedType>())">; NativeCodeCall<"$0.reshape(($1.getType()).cast<ShapedType>())">;
def FoldConstantReshapeOptPattern : Pat< def FoldConstantReshapeOptPattern : Pat<
(ReshapeOp:$res (ConstantOp $arg)), (ReshapeOp:$res (ConstantOp $arg)),
(ConstantOp (ReshapeConstant $arg, $res))>; (ConstantOp (ReshapeConstant $arg, $res))>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pattern-Match and Rewrite with Constraints // Pattern-Match and Rewrite with Constraints
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DRR allows for constraint checking when the transformation is conditional // DRR allows for constraint checking when the transformation is conditional
// on operand properties. // on operand properties.
// Reshape(x) = x, where input and output shapes are identical // Reshape(x) = x, where input and output shapes are identical
def TypesAreIdentical : Constraint<CPred<"$0->getType() == $1->getType()">>; def TypesAreIdentical : Constraint<CPred<"$0.getType() == $1.getType()">>;
def RedundantReshapeOptPattern : Pat< def RedundantReshapeOptPattern : Pat<
(ReshapeOp:$res $arg), (replaceWithValue $arg), (ReshapeOp:$res $arg), (replaceWithValue $arg),
[(TypesAreIdentical $res, $arg)]>; [(TypesAreIdentical $res, $arg)]>;
#endif // TOY_COMBINE #endif // TOY_COMBINE

mlir/examples/toy/Ch4/mlir/Dialect.cpp

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 Linesstruct ToyInlinerInterface : public DialectInlinerInterface {
void handleTerminator(Operation *op, void handleTerminator(Operation *op,
ArrayRef<Value> valuesToRepl) const final { ArrayRef<Value> valuesToRepl) const final {
// Only "toy.return" needs to be handled here. // Only "toy.return" needs to be handled here.
auto returnOp = cast<ReturnOp>(op); auto returnOp = cast<ReturnOp>(op);
// Replace the values directly with the return operands. // Replace the values directly with the return operands.
assert(returnOp.getNumOperands() == valuesToRepl.size()); assert(returnOp.getNumOperands() == valuesToRepl.size());
for (const auto &it : llvm::enumerate(returnOp.getOperands())) for (const auto &it : llvm::enumerate(returnOp.getOperands()))
valuesToRepl[it.index()]->replaceAllUsesWith(it.value()); valuesToRepl[it.index()].replaceAllUsesWith(it.value());
} }
/// Attempts to materialize a conversion for a type mismatch between a call /// Attempts to materialize a conversion for a type mismatch between a call
/// from this dialect, and a callable region. This method should generate an /// from this dialect, and a callable region. This method should generate an
/// operation that takes 'input' as the only operand, and produces a single /// operation that takes 'input' as the only operand, and produces a single
/// result of 'resultType'. If a conversion can not be generated, nullptr /// result of 'resultType'. If a conversion can not be generated, nullptr
/// should be returned. /// should be returned.
Operation *materializeCallConversion(OpBuilder &builder, Value input, Operation *materializeCallConversion(OpBuilder &builder, Value input,
Show All 34 Linesvoid ConstantOp::build(mlir::Builder *builder, mlir::OperationState &state,
ConstantOp::build(builder, state, dataType, dataAttribute); ConstantOp::build(builder, state, dataType, dataAttribute);
} }
/// Verifier for the constant operation. This corresponds to the `::verify(...)` /// Verifier for the constant operation. This corresponds to the `::verify(...)`
/// in the op definition. /// in the op definition.
static mlir::LogicalResult verify(ConstantOp op) { static mlir::LogicalResult verify(ConstantOp op) {
// If the return type of the constant is not an unranked tensor, the shape // If the return type of the constant is not an unranked tensor, the shape
// must match the shape of the attribute holding the data. // must match the shape of the attribute holding the data.
auto resultType = auto resultType = op.getResult().getType().dyn_cast<mlir::RankedTensorType>();
op.getResult()->getType().dyn_cast<mlir::RankedTensorType>();
if (!resultType) if (!resultType)
return success(); return success();
// Check that the rank of the attribute type matches the rank of the constant // Check that the rank of the attribute type matches the rank of the constant
// result type. // result type.
auto attrType = op.value().getType().cast<mlir::TensorType>(); auto attrType = op.value().getType().cast<mlir::TensorType>();
if (attrType.getRank() != resultType.getRank()) { if (attrType.getRank() != resultType.getRank()) {
return op.emitOpError( return op.emitOpError(
Show All 20 Lines
void AddOp::build(mlir::Builder *builder, mlir::OperationState &state, void AddOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value lhs, mlir::Value rhs) { mlir::Value lhs, mlir::Value rhs) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands({lhs, rhs}); state.addOperands({lhs, rhs});
} }
/// Infer the output shape of the AddOp, this is required by the shape inference /// Infer the output shape of the AddOp, this is required by the shape inference
/// interface. /// interface.
void AddOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void AddOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CastOp // CastOp
/// Infer the output shape of the CastOp, this is required by the shape /// Infer the output shape of the CastOp, this is required by the shape
/// inference interface. /// inference interface.
void CastOp::inferShapes() { getResult()->setType(getOperand()->getType()); } void CastOp::inferShapes() { getResult().setType(getOperand().getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// GenericCallOp // GenericCallOp
void GenericCallOp::build(mlir::Builder *builder, mlir::OperationState &state, void GenericCallOp::build(mlir::Builder *builder, mlir::OperationState &state,
StringRef callee, ArrayRef<mlir::Value> arguments) { StringRef callee, ArrayRef<mlir::Value> arguments) {
// Generic call always returns an unranked Tensor initially. // Generic call always returns an unranked Tensor initially.
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
Show All 17 Lines
void MulOp::build(mlir::Builder *builder, mlir::OperationState &state, void MulOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value lhs, mlir::Value rhs) { mlir::Value lhs, mlir::Value rhs) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands({lhs, rhs}); state.addOperands({lhs, rhs});
} }
/// Infer the output shape of the MulOp, this is required by the shape inference /// Infer the output shape of the MulOp, this is required by the shape inference
/// interface. /// interface.
void MulOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void MulOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ReturnOp // ReturnOp
static mlir::LogicalResult verify(ReturnOp op) { static mlir::LogicalResult verify(ReturnOp op) {
// We know that the parent operation is a function, because of the 'HasParent' // We know that the parent operation is a function, because of the 'HasParent'
// trait attached to the operation definition. // trait attached to the operation definition.
auto function = cast<FuncOp>(op.getParentOp()); auto function = cast<FuncOp>(op.getParentOp());
Show All 33 Lines
void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state, void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value value) { mlir::Value value) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands(value); state.addOperands(value);
} }
void TransposeOp::inferShapes() { void TransposeOp::inferShapes() {
auto arrayTy = getOperand()->getType().cast<RankedTensorType>(); auto arrayTy = getOperand().getType().cast<RankedTensorType>();
SmallVector<int64_t, 2> dims(llvm::reverse(arrayTy.getShape())); SmallVector<int64_t, 2> dims(llvm::reverse(arrayTy.getShape()));
getResult()->setType(RankedTensorType::get(dims, arrayTy.getElementType())); getResult().setType(RankedTensorType::get(dims, arrayTy.getElementType()));
} }
static mlir::LogicalResult verify(TransposeOp op) { static mlir::LogicalResult verify(TransposeOp op) {
auto inputType = op.getOperand()->getType().dyn_cast<RankedTensorType>(); auto inputType = op.getOperand().getType().dyn_cast<RankedTensorType>();
auto resultType = op.getType().dyn_cast<RankedTensorType>(); auto resultType = op.getType().dyn_cast<RankedTensorType>();
if (!inputType || !resultType) if (!inputType || !resultType)
return mlir::success(); return mlir::success();
auto inputShape = inputType.getShape(); auto inputShape = inputType.getShape();
if (!std::equal(inputShape.begin(), inputShape.end(), if (!std::equal(inputShape.begin(), inputShape.end(),
resultType.getShape().rbegin())) { resultType.getShape().rbegin())) {
return op.emitError() return op.emitError()
Show All 11 Lines

mlir/examples/toy/Ch4/mlir/ToyCombine.cpp

Show All 40 Linesstruct SimplifyRedundantTranspose : public mlir::OpRewritePattern<TransposeOp> {
/// argument is the orchestrator of the sequence of rewrites. The pattern is /// argument is the orchestrator of the sequence of rewrites. The pattern is
/// expected to interact with it to perform any changes to the IR from here. /// expected to interact with it to perform any changes to the IR from here.
mlir::PatternMatchResult mlir::PatternMatchResult
matchAndRewrite(TransposeOp op, matchAndRewrite(TransposeOp op,
mlir::PatternRewriter &rewriter) const override { mlir::PatternRewriter &rewriter) const override {
// Look through the input of the current transpose. // Look through the input of the current transpose.
mlir::Value transposeInput = op.getOperand(); mlir::Value transposeInput = op.getOperand();
TransposeOp transposeInputOp = TransposeOp transposeInputOp =
llvm::dyn_cast_or_null<TransposeOp>(transposeInput->getDefiningOp()); llvm::dyn_cast_or_null<TransposeOp>(transposeInput.getDefiningOp());
// If the input is defined by another Transpose, bingo! // If the input is defined by another Transpose, bingo!
if (!transposeInputOp) if (!transposeInputOp)
return matchFailure(); return matchFailure();
// Use the rewriter to perform the replacement. // Use the rewriter to perform the replacement.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp}); rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp});
return matchSuccess(); return matchSuccess();
Show All 17 Lines

mlir/examples/toy/Ch4/mlir/ToyCombine.td

Show All 35 Lines
// Pattern-Match and Rewrite using Native Code Call // Pattern-Match and Rewrite using Native Code Call
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Native Code Calls may be used for more complex transformations using inline // Native Code Calls may be used for more complex transformations using inline
// C++ and C++ helper functions. // C++ and C++ helper functions.
// Reshape(Constant(x)) = x' // Reshape(Constant(x)) = x'
def ReshapeConstant : def ReshapeConstant :
NativeCodeCall<"$0.reshape(($1->getType()).cast<ShapedType>())">; NativeCodeCall<"$0.reshape(($1.getType()).cast<ShapedType>())">;
def FoldConstantReshapeOptPattern : Pat< def FoldConstantReshapeOptPattern : Pat<
(ReshapeOp:$res (ConstantOp $arg)), (ReshapeOp:$res (ConstantOp $arg)),
(ConstantOp (ReshapeConstant $arg, $res))>; (ConstantOp (ReshapeConstant $arg, $res))>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pattern-Match and Rewrite with Constraints // Pattern-Match and Rewrite with Constraints
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DRR allows for constraint checking when the transformation is conditional // DRR allows for constraint checking when the transformation is conditional
// on operand properties. // on operand properties.
// Reshape(x) = x, where input and output shapes are identical // Reshape(x) = x, where input and output shapes are identical
def TypesAreIdentical : Constraint<CPred<"$0->getType() == $1->getType()">>; def TypesAreIdentical : Constraint<CPred<"$0.getType() == $1.getType()">>;
def RedundantReshapeOptPattern : Pat< def RedundantReshapeOptPattern : Pat<
(ReshapeOp:$res $arg), (replaceWithValue $arg), (ReshapeOp:$res $arg), (replaceWithValue $arg),
[(TypesAreIdentical $res, $arg)]>; [(TypesAreIdentical $res, $arg)]>;
#endif // TOY_COMBINE #endif // TOY_COMBINE

mlir/examples/toy/Ch5/mlir/Dialect.cpp

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 Linesstruct ToyInlinerInterface : public DialectInlinerInterface {
void handleTerminator(Operation *op, void handleTerminator(Operation *op,
ArrayRef<Value> valuesToRepl) const final { ArrayRef<Value> valuesToRepl) const final {
// Only "toy.return" needs to be handled here. // Only "toy.return" needs to be handled here.
auto returnOp = cast<ReturnOp>(op); auto returnOp = cast<ReturnOp>(op);
// Replace the values directly with the return operands. // Replace the values directly with the return operands.
assert(returnOp.getNumOperands() == valuesToRepl.size()); assert(returnOp.getNumOperands() == valuesToRepl.size());
for (const auto &it : llvm::enumerate(returnOp.getOperands())) for (const auto &it : llvm::enumerate(returnOp.getOperands()))
valuesToRepl[it.index()]->replaceAllUsesWith(it.value()); valuesToRepl[it.index()].replaceAllUsesWith(it.value());
} }
/// Attempts to materialize a conversion for a type mismatch between a call /// Attempts to materialize a conversion for a type mismatch between a call
/// from this dialect, and a callable region. This method should generate an /// from this dialect, and a callable region. This method should generate an
/// operation that takes 'input' as the only operand, and produces a single /// operation that takes 'input' as the only operand, and produces a single
/// result of 'resultType'. If a conversion can not be generated, nullptr /// result of 'resultType'. If a conversion can not be generated, nullptr
/// should be returned. /// should be returned.
Operation *materializeCallConversion(OpBuilder &builder, Value input, Operation *materializeCallConversion(OpBuilder &builder, Value input,
Show All 34 Linesvoid ConstantOp::build(mlir::Builder *builder, mlir::OperationState &state,
ConstantOp::build(builder, state, dataType, dataAttribute); ConstantOp::build(builder, state, dataType, dataAttribute);
} }
/// Verifier for the constant operation. This corresponds to the `::verify(...)` /// Verifier for the constant operation. This corresponds to the `::verify(...)`
/// in the op definition. /// in the op definition.
static mlir::LogicalResult verify(ConstantOp op) { static mlir::LogicalResult verify(ConstantOp op) {
// If the return type of the constant is not an unranked tensor, the shape // If the return type of the constant is not an unranked tensor, the shape
// must match the shape of the attribute holding the data. // must match the shape of the attribute holding the data.
auto resultType = auto resultType = op.getResult().getType().dyn_cast<mlir::RankedTensorType>();
op.getResult()->getType().dyn_cast<mlir::RankedTensorType>();
if (!resultType) if (!resultType)
return success(); return success();
// Check that the rank of the attribute type matches the rank of the constant // Check that the rank of the attribute type matches the rank of the constant
// result type. // result type.
auto attrType = op.value().getType().cast<mlir::TensorType>(); auto attrType = op.value().getType().cast<mlir::TensorType>();
if (attrType.getRank() != resultType.getRank()) { if (attrType.getRank() != resultType.getRank()) {
return op.emitOpError( return op.emitOpError(
Show All 20 Lines
void AddOp::build(mlir::Builder *builder, mlir::OperationState &state, void AddOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value lhs, mlir::Value rhs) { mlir::Value lhs, mlir::Value rhs) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands({lhs, rhs}); state.addOperands({lhs, rhs});
} }
/// Infer the output shape of the AddOp, this is required by the shape inference /// Infer the output shape of the AddOp, this is required by the shape inference
/// interface. /// interface.
void AddOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void AddOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CastOp // CastOp
/// Infer the output shape of the CastOp, this is required by the shape /// Infer the output shape of the CastOp, this is required by the shape
/// inference interface. /// inference interface.
void CastOp::inferShapes() { getResult()->setType(getOperand()->getType()); } void CastOp::inferShapes() { getResult().setType(getOperand().getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// GenericCallOp // GenericCallOp
void GenericCallOp::build(mlir::Builder *builder, mlir::OperationState &state, void GenericCallOp::build(mlir::Builder *builder, mlir::OperationState &state,
StringRef callee, ArrayRef<mlir::Value> arguments) { StringRef callee, ArrayRef<mlir::Value> arguments) {
// Generic call always returns an unranked Tensor initially. // Generic call always returns an unranked Tensor initially.
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
Show All 17 Lines
void MulOp::build(mlir::Builder *builder, mlir::OperationState &state, void MulOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value lhs, mlir::Value rhs) { mlir::Value lhs, mlir::Value rhs) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands({lhs, rhs}); state.addOperands({lhs, rhs});
} }
/// Infer the output shape of the MulOp, this is required by the shape inference /// Infer the output shape of the MulOp, this is required by the shape inference
/// interface. /// interface.
void MulOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void MulOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ReturnOp // ReturnOp
static mlir::LogicalResult verify(ReturnOp op) { static mlir::LogicalResult verify(ReturnOp op) {
// We know that the parent operation is a function, because of the 'HasParent' // We know that the parent operation is a function, because of the 'HasParent'
// trait attached to the operation definition. // trait attached to the operation definition.
auto function = cast<FuncOp>(op.getParentOp()); auto function = cast<FuncOp>(op.getParentOp());
Show All 33 Lines
void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state, void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value value) { mlir::Value value) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands(value); state.addOperands(value);
} }
void TransposeOp::inferShapes() { void TransposeOp::inferShapes() {
auto arrayTy = getOperand()->getType().cast<RankedTensorType>(); auto arrayTy = getOperand().getType().cast<RankedTensorType>();
SmallVector<int64_t, 2> dims(llvm::reverse(arrayTy.getShape())); SmallVector<int64_t, 2> dims(llvm::reverse(arrayTy.getShape()));
getResult()->setType(RankedTensorType::get(dims, arrayTy.getElementType())); getResult().setType(RankedTensorType::get(dims, arrayTy.getElementType()));
} }
static mlir::LogicalResult verify(TransposeOp op) { static mlir::LogicalResult verify(TransposeOp op) {
auto inputType = op.getOperand()->getType().dyn_cast<RankedTensorType>(); auto inputType = op.getOperand().getType().dyn_cast<RankedTensorType>();
auto resultType = op.getType().dyn_cast<RankedTensorType>(); auto resultType = op.getType().dyn_cast<RankedTensorType>();
if (!inputType || !resultType) if (!inputType || !resultType)
return mlir::success(); return mlir::success();
auto inputShape = inputType.getShape(); auto inputShape = inputType.getShape();
if (!std::equal(inputShape.begin(), inputShape.end(), if (!std::equal(inputShape.begin(), inputShape.end(),
resultType.getShape().rbegin())) { resultType.getShape().rbegin())) {
return op.emitError() return op.emitError()
Show All 11 Lines

mlir/examples/toy/Ch5/mlir/ToyCombine.cpp

Show All 40 Linesstruct SimplifyRedundantTranspose : public mlir::OpRewritePattern<TransposeOp> {
/// argument is the orchestrator of the sequence of rewrites. The pattern is /// argument is the orchestrator of the sequence of rewrites. The pattern is
/// expected to interact with it to perform any changes to the IR from here. /// expected to interact with it to perform any changes to the IR from here.
mlir::PatternMatchResult mlir::PatternMatchResult
matchAndRewrite(TransposeOp op, matchAndRewrite(TransposeOp op,
mlir::PatternRewriter &rewriter) const override { mlir::PatternRewriter &rewriter) const override {
// Look through the input of the current transpose. // Look through the input of the current transpose.
mlir::Value transposeInput = op.getOperand(); mlir::Value transposeInput = op.getOperand();
TransposeOp transposeInputOp = TransposeOp transposeInputOp =
llvm::dyn_cast_or_null<TransposeOp>(transposeInput->getDefiningOp()); llvm::dyn_cast_or_null<TransposeOp>(transposeInput.getDefiningOp());
// If the input is defined by another Transpose, bingo! // If the input is defined by another Transpose, bingo!
if (!transposeInputOp) if (!transposeInputOp)
return matchFailure(); return matchFailure();
// Use the rewriter to perform the replacement. // Use the rewriter to perform the replacement.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp}); rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp});
return matchSuccess(); return matchSuccess();
Show All 17 Lines

mlir/examples/toy/Ch5/mlir/ToyCombine.td

Show All 35 Lines
// Pattern-Match and Rewrite using Native Code Call // Pattern-Match and Rewrite using Native Code Call
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Native Code Calls may be used for more complex transformations using inline // Native Code Calls may be used for more complex transformations using inline
// C++ and C++ helper functions. // C++ and C++ helper functions.
// Reshape(Constant(x)) = x' // Reshape(Constant(x)) = x'
def ReshapeConstant : def ReshapeConstant :
NativeCodeCall<"$0.reshape(($1->getType()).cast<ShapedType>())">; NativeCodeCall<"$0.reshape(($1.getType()).cast<ShapedType>())">;
def FoldConstantReshapeOptPattern : Pat< def FoldConstantReshapeOptPattern : Pat<
(ReshapeOp:$res (ConstantOp $arg)), (ReshapeOp:$res (ConstantOp $arg)),
(ConstantOp (ReshapeConstant $arg, $res))>; (ConstantOp (ReshapeConstant $arg, $res))>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pattern-Match and Rewrite with Constraints // Pattern-Match and Rewrite with Constraints
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DRR allows for constraint checking when the transformation is conditional // DRR allows for constraint checking when the transformation is conditional
// on operand properties. // on operand properties.
// Reshape(x) = x, where input and output shapes are identical // Reshape(x) = x, where input and output shapes are identical
def TypesAreIdentical : Constraint<CPred<"$0->getType() == $1->getType()">>; def TypesAreIdentical : Constraint<CPred<"$0.getType() == $1.getType()">>;
def RedundantReshapeOptPattern : Pat< def RedundantReshapeOptPattern : Pat<
(ReshapeOp:$res $arg), (replaceWithValue $arg), (ReshapeOp:$res $arg), (replaceWithValue $arg),
[(TypesAreIdentical $res, $arg)]>; [(TypesAreIdentical $res, $arg)]>;
#endif // TOY_COMBINE #endif // TOY_COMBINE

mlir/examples/toy/Ch6/mlir/Dialect.cpp

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 Linesstruct ToyInlinerInterface : public DialectInlinerInterface {
void handleTerminator(Operation *op, void handleTerminator(Operation *op,
ArrayRef<Value> valuesToRepl) const final { ArrayRef<Value> valuesToRepl) const final {
// Only "toy.return" needs to be handled here. // Only "toy.return" needs to be handled here.
auto returnOp = cast<ReturnOp>(op); auto returnOp = cast<ReturnOp>(op);
// Replace the values directly with the return operands. // Replace the values directly with the return operands.
assert(returnOp.getNumOperands() == valuesToRepl.size()); assert(returnOp.getNumOperands() == valuesToRepl.size());
for (const auto &it : llvm::enumerate(returnOp.getOperands())) for (const auto &it : llvm::enumerate(returnOp.getOperands()))
valuesToRepl[it.index()]->replaceAllUsesWith(it.value()); valuesToRepl[it.index()].replaceAllUsesWith(it.value());
} }
/// Attempts to materialize a conversion for a type mismatch between a call /// Attempts to materialize a conversion for a type mismatch between a call
/// from this dialect, and a callable region. This method should generate an /// from this dialect, and a callable region. This method should generate an
/// operation that takes 'input' as the only operand, and produces a single /// operation that takes 'input' as the only operand, and produces a single
/// result of 'resultType'. If a conversion can not be generated, nullptr /// result of 'resultType'. If a conversion can not be generated, nullptr
/// should be returned. /// should be returned.
Operation *materializeCallConversion(OpBuilder &builder, Value input, Operation *materializeCallConversion(OpBuilder &builder, Value input,
Show All 34 Linesvoid ConstantOp::build(mlir::Builder *builder, mlir::OperationState &state,
ConstantOp::build(builder, state, dataType, dataAttribute); ConstantOp::build(builder, state, dataType, dataAttribute);
} }
/// Verifier for the constant operation. This corresponds to the `::verify(...)` /// Verifier for the constant operation. This corresponds to the `::verify(...)`
/// in the op definition. /// in the op definition.
static mlir::LogicalResult verify(ConstantOp op) { static mlir::LogicalResult verify(ConstantOp op) {
// If the return type of the constant is not an unranked tensor, the shape // If the return type of the constant is not an unranked tensor, the shape
// must match the shape of the attribute holding the data. // must match the shape of the attribute holding the data.
auto resultType = auto resultType = op.getResult().getType().dyn_cast<mlir::RankedTensorType>();
op.getResult()->getType().dyn_cast<mlir::RankedTensorType>();
if (!resultType) if (!resultType)
return success(); return success();
// Check that the rank of the attribute type matches the rank of the constant // Check that the rank of the attribute type matches the rank of the constant
// result type. // result type.
auto attrType = op.value().getType().cast<mlir::TensorType>(); auto attrType = op.value().getType().cast<mlir::TensorType>();
if (attrType.getRank() != resultType.getRank()) { if (attrType.getRank() != resultType.getRank()) {
return op.emitOpError( return op.emitOpError(
Show All 20 Lines
void AddOp::build(mlir::Builder *builder, mlir::OperationState &state, void AddOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value lhs, mlir::Value rhs) { mlir::Value lhs, mlir::Value rhs) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands({lhs, rhs}); state.addOperands({lhs, rhs});
} }
/// Infer the output shape of the AddOp, this is required by the shape inference /// Infer the output shape of the AddOp, this is required by the shape inference
/// interface. /// interface.
void AddOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void AddOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CastOp // CastOp
/// Infer the output shape of the CastOp, this is required by the shape /// Infer the output shape of the CastOp, this is required by the shape
/// inference interface. /// inference interface.
void CastOp::inferShapes() { getResult()->setType(getOperand()->getType()); } void CastOp::inferShapes() { getResult().setType(getOperand().getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// GenericCallOp // GenericCallOp
void GenericCallOp::build(mlir::Builder *builder, mlir::OperationState &state, void GenericCallOp::build(mlir::Builder *builder, mlir::OperationState &state,
StringRef callee, ArrayRef<mlir::Value> arguments) { StringRef callee, ArrayRef<mlir::Value> arguments) {
// Generic call always returns an unranked Tensor initially. // Generic call always returns an unranked Tensor initially.
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
Show All 17 Lines
void MulOp::build(mlir::Builder *builder, mlir::OperationState &state, void MulOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value lhs, mlir::Value rhs) { mlir::Value lhs, mlir::Value rhs) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands({lhs, rhs}); state.addOperands({lhs, rhs});
} }
/// Infer the output shape of the MulOp, this is required by the shape inference /// Infer the output shape of the MulOp, this is required by the shape inference
/// interface. /// interface.
void MulOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void MulOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ReturnOp // ReturnOp
static mlir::LogicalResult verify(ReturnOp op) { static mlir::LogicalResult verify(ReturnOp op) {
// We know that the parent operation is a function, because of the 'HasParent' // We know that the parent operation is a function, because of the 'HasParent'
// trait attached to the operation definition. // trait attached to the operation definition.
auto function = cast<FuncOp>(op.getParentOp()); auto function = cast<FuncOp>(op.getParentOp());
Show All 33 Lines
void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state, void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value value) { mlir::Value value) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands(value); state.addOperands(value);
} }
void TransposeOp::inferShapes() { void TransposeOp::inferShapes() {
auto arrayTy = getOperand()->getType().cast<RankedTensorType>(); auto arrayTy = getOperand().getType().cast<RankedTensorType>();
SmallVector<int64_t, 2> dims(llvm::reverse(arrayTy.getShape())); SmallVector<int64_t, 2> dims(llvm::reverse(arrayTy.getShape()));
getResult()->setType(RankedTensorType::get(dims, arrayTy.getElementType())); getResult().setType(RankedTensorType::get(dims, arrayTy.getElementType()));
} }
static mlir::LogicalResult verify(TransposeOp op) { static mlir::LogicalResult verify(TransposeOp op) {
auto inputType = op.getOperand()->getType().dyn_cast<RankedTensorType>(); auto inputType = op.getOperand().getType().dyn_cast<RankedTensorType>();
auto resultType = op.getType().dyn_cast<RankedTensorType>(); auto resultType = op.getType().dyn_cast<RankedTensorType>();
if (!inputType || !resultType) if (!inputType || !resultType)
return mlir::success(); return mlir::success();
auto inputShape = inputType.getShape(); auto inputShape = inputType.getShape();
if (!std::equal(inputShape.begin(), inputShape.end(), if (!std::equal(inputShape.begin(), inputShape.end(),
resultType.getShape().rbegin())) { resultType.getShape().rbegin())) {
return op.emitError() return op.emitError()
Show All 11 Lines

mlir/examples/toy/Ch6/mlir/ToyCombine.cpp

Show All 40 Linesstruct SimplifyRedundantTranspose : public mlir::OpRewritePattern<TransposeOp> {
/// argument is the orchestrator of the sequence of rewrites. The pattern is /// argument is the orchestrator of the sequence of rewrites. The pattern is
/// expected to interact with it to perform any changes to the IR from here. /// expected to interact with it to perform any changes to the IR from here.
mlir::PatternMatchResult mlir::PatternMatchResult
matchAndRewrite(TransposeOp op, matchAndRewrite(TransposeOp op,
mlir::PatternRewriter &rewriter) const override { mlir::PatternRewriter &rewriter) const override {
// Look through the input of the current transpose. // Look through the input of the current transpose.
mlir::Value transposeInput = op.getOperand(); mlir::Value transposeInput = op.getOperand();
TransposeOp transposeInputOp = TransposeOp transposeInputOp =
llvm::dyn_cast_or_null<TransposeOp>(transposeInput->getDefiningOp()); llvm::dyn_cast_or_null<TransposeOp>(transposeInput.getDefiningOp());
// If the input is defined by another Transpose, bingo! // If the input is defined by another Transpose, bingo!
if (!transposeInputOp) if (!transposeInputOp)
return matchFailure(); return matchFailure();
// Use the rewriter to perform the replacement. // Use the rewriter to perform the replacement.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp}); rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp});
return matchSuccess(); return matchSuccess();
Show All 17 Lines

mlir/examples/toy/Ch6/mlir/ToyCombine.td

Show All 35 Lines
// Pattern-Match and Rewrite using Native Code Call // Pattern-Match and Rewrite using Native Code Call
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Native Code Calls may be used for more complex transformations using inline // Native Code Calls may be used for more complex transformations using inline
// C++ and C++ helper functions. // C++ and C++ helper functions.
// Reshape(Constant(x)) = x' // Reshape(Constant(x)) = x'
def ReshapeConstant : def ReshapeConstant :
NativeCodeCall<"$0.reshape(($1->getType()).cast<ShapedType>())">; NativeCodeCall<"$0.reshape(($1.getType()).cast<ShapedType>())">;
def FoldConstantReshapeOptPattern : Pat< def FoldConstantReshapeOptPattern : Pat<
(ReshapeOp:$res (ConstantOp $arg)), (ReshapeOp:$res (ConstantOp $arg)),
(ConstantOp (ReshapeConstant $arg, $res))>; (ConstantOp (ReshapeConstant $arg, $res))>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pattern-Match and Rewrite with Constraints // Pattern-Match and Rewrite with Constraints
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DRR allows for constraint checking when the transformation is conditional // DRR allows for constraint checking when the transformation is conditional
// on operand properties. // on operand properties.
// Reshape(x) = x, where input and output shapes are identical // Reshape(x) = x, where input and output shapes are identical
def TypesAreIdentical : Constraint<CPred<"$0->getType() == $1->getType()">>; def TypesAreIdentical : Constraint<CPred<"$0.getType() == $1.getType()">>;
def RedundantReshapeOptPattern : Pat< def RedundantReshapeOptPattern : Pat<
(ReshapeOp:$res $arg), (replaceWithValue $arg), (ReshapeOp:$res $arg), (replaceWithValue $arg),
[(TypesAreIdentical $res, $arg)]>; [(TypesAreIdentical $res, $arg)]>;
#endif // TOY_COMBINE #endif // TOY_COMBINE

mlir/examples/toy/Ch7/mlir/Dialect.cpp

Show First 20 Lines • Show All 48 Lines • ▼ Show 20 Linesstruct ToyInlinerInterface : public DialectInlinerInterface {
void handleTerminator(Operation *op, void handleTerminator(Operation *op,
ArrayRef<Value> valuesToRepl) const final { ArrayRef<Value> valuesToRepl) const final {
// Only "toy.return" needs to be handled here. // Only "toy.return" needs to be handled here.
auto returnOp = cast<ReturnOp>(op); auto returnOp = cast<ReturnOp>(op);
// Replace the values directly with the return operands. // Replace the values directly with the return operands.
assert(returnOp.getNumOperands() == valuesToRepl.size()); assert(returnOp.getNumOperands() == valuesToRepl.size());
for (const auto &it : llvm::enumerate(returnOp.getOperands())) for (const auto &it : llvm::enumerate(returnOp.getOperands()))
valuesToRepl[it.index()]->replaceAllUsesWith(it.value()); valuesToRepl[it.index()].replaceAllUsesWith(it.value());
} }
/// Attempts to materialize a conversion for a type mismatch between a call /// Attempts to materialize a conversion for a type mismatch between a call
/// from this dialect, and a callable region. This method should generate an /// from this dialect, and a callable region. This method should generate an
/// operation that takes 'input' as the only operand, and produces a single /// operation that takes 'input' as the only operand, and produces a single
/// result of 'resultType'. If a conversion can not be generated, nullptr /// result of 'resultType'. If a conversion can not be generated, nullptr
/// should be returned. /// should be returned.
Operation *materializeCallConversion(OpBuilder &builder, Value input, Operation *materializeCallConversion(OpBuilder &builder, Value input,
▲ Show 20 Lines • Show All 100 Lines • ▼ Show 20 Lines for (const auto &it : llvm::zip(resultElementTypes, attrElementValues))
if (failed(verifyConstantForType(std::get<0>(it), std::get<1>(it), op))) if (failed(verifyConstantForType(std::get<0>(it), std::get<1>(it), op)))
return mlir::failure(); return mlir::failure();
return mlir::success(); return mlir::success();
} }
/// Verifier for the constant operation. This corresponds to the `::verify(...)` /// Verifier for the constant operation. This corresponds to the `::verify(...)`
/// in the op definition. /// in the op definition.
static mlir::LogicalResult verify(ConstantOp op) { static mlir::LogicalResult verify(ConstantOp op) {
return verifyConstantForType(op.getResult()->getType(), op.value(), op); return verifyConstantForType(op.getResult().getType(), op.value(), op);
} }
static mlir::LogicalResult verify(StructConstantOp op) { static mlir::LogicalResult verify(StructConstantOp op) {
return verifyConstantForType(op.getResult()->getType(), op.value(), op); return verifyConstantForType(op.getResult().getType(), op.value(), op);
} }
/// Infer the output shape of the ConstantOp, this is required by the shape /// Infer the output shape of the ConstantOp, this is required by the shape
/// inference interface. /// inference interface.
void ConstantOp::inferShapes() { getResult()->setType(value().getType()); } void ConstantOp::inferShapes() { getResult().setType(value().getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AddOp // AddOp
void AddOp::build(mlir::Builder *builder, mlir::OperationState &state, void AddOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value lhs, mlir::Value rhs) { mlir::Value lhs, mlir::Value rhs) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands({lhs, rhs}); state.addOperands({lhs, rhs});
} }
/// Infer the output shape of the AddOp, this is required by the shape inference /// Infer the output shape of the AddOp, this is required by the shape inference
/// interface. /// interface.
void AddOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void AddOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CastOp // CastOp
/// Infer the output shape of the CastOp, this is required by the shape /// Infer the output shape of the CastOp, this is required by the shape
/// inference interface. /// inference interface.
void CastOp::inferShapes() { getResult()->setType(getOperand()->getType()); } void CastOp::inferShapes() { getResult().setType(getOperand().getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// GenericCallOp // GenericCallOp
void GenericCallOp::build(mlir::Builder *builder, mlir::OperationState &state, void GenericCallOp::build(mlir::Builder *builder, mlir::OperationState &state,
StringRef callee, ArrayRef<mlir::Value> arguments) { StringRef callee, ArrayRef<mlir::Value> arguments) {
// Generic call always returns an unranked Tensor initially. // Generic call always returns an unranked Tensor initially.
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
Show All 17 Lines
void MulOp::build(mlir::Builder *builder, mlir::OperationState &state, void MulOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value lhs, mlir::Value rhs) { mlir::Value lhs, mlir::Value rhs) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands({lhs, rhs}); state.addOperands({lhs, rhs});
} }
/// Infer the output shape of the MulOp, this is required by the shape inference /// Infer the output shape of the MulOp, this is required by the shape inference
/// interface. /// interface.
void MulOp::inferShapes() { getResult()->setType(getOperand(0)->getType()); } void MulOp::inferShapes() { getResult().setType(getOperand(0).getType()); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ReturnOp // ReturnOp
static mlir::LogicalResult verify(ReturnOp op) { static mlir::LogicalResult verify(ReturnOp op) {
// We know that the parent operation is a function, because of the 'HasParent' // We know that the parent operation is a function, because of the 'HasParent'
// trait attached to the operation definition. // trait attached to the operation definition.
auto function = cast<FuncOp>(op.getParentOp()); auto function = cast<FuncOp>(op.getParentOp());
Show All 29 Lines
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// StructAccessOp // StructAccessOp
void StructAccessOp::build(mlir::Builder *b, mlir::OperationState &state, void StructAccessOp::build(mlir::Builder *b, mlir::OperationState &state,
mlir::Value input, size_t index) { mlir::Value input, size_t index) {
// Extract the result type from the input type. // Extract the result type from the input type.
StructType structTy = input->getType().cast<StructType>(); StructType structTy = input.getType().cast<StructType>();
assert(index < structTy.getNumElementTypes()); assert(index < structTy.getNumElementTypes());
mlir::Type resultType = structTy.getElementTypes()[index]; mlir::Type resultType = structTy.getElementTypes()[index];
// Call into the auto-generated build method. // Call into the auto-generated build method.
build(b, state, resultType, input, b->getI64IntegerAttr(index)); build(b, state, resultType, input, b->getI64IntegerAttr(index));
} }
static mlir::LogicalResult verify(StructAccessOp op) { static mlir::LogicalResult verify(StructAccessOp op) {
StructType structTy = op.input()->getType().cast<StructType>(); StructType structTy = op.input().getType().cast<StructType>();
size_t index = op.index().getZExtValue(); size_t index = op.index().getZExtValue();
if (index >= structTy.getNumElementTypes()) if (index >= structTy.getNumElementTypes())
return op.emitOpError() return op.emitOpError()
<< "index should be within the range of the input struct type"; << "index should be within the range of the input struct type";
mlir::Type resultType = op.getResult()->getType(); mlir::Type resultType = op.getResult().getType();
if (resultType != structTy.getElementTypes()[index]) if (resultType != structTy.getElementTypes()[index])
return op.emitOpError() << "must have the same result type as the struct " return op.emitOpError() << "must have the same result type as the struct "
"element referred to by the index"; "element referred to by the index";
return mlir::success(); return mlir::success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TransposeOp // TransposeOp
void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state, void TransposeOp::build(mlir::Builder *builder, mlir::OperationState &state,
mlir::Value value) { mlir::Value value) {
state.addTypes(UnrankedTensorType::get(builder->getF64Type())); state.addTypes(UnrankedTensorType::get(builder->getF64Type()));
state.addOperands(value); state.addOperands(value);
} }
void TransposeOp::inferShapes() { void TransposeOp::inferShapes() {
auto arrayTy = getOperand()->getType().cast<RankedTensorType>(); auto arrayTy = getOperand().getType().cast<RankedTensorType>();
SmallVector<int64_t, 2> dims(llvm::reverse(arrayTy.getShape())); SmallVector<int64_t, 2> dims(llvm::reverse(arrayTy.getShape()));
getResult()->setType(RankedTensorType::get(dims, arrayTy.getElementType())); getResult().setType(RankedTensorType::get(dims, arrayTy.getElementType()));
} }
static mlir::LogicalResult verify(TransposeOp op) { static mlir::LogicalResult verify(TransposeOp op) {
auto inputType = op.getOperand()->getType().dyn_cast<RankedTensorType>(); auto inputType = op.getOperand().getType().dyn_cast<RankedTensorType>();
auto resultType = op.getType().dyn_cast<RankedTensorType>(); auto resultType = op.getType().dyn_cast<RankedTensorType>();
if (!inputType || !resultType) if (!inputType || !resultType)
return mlir::success(); return mlir::success();
auto inputShape = inputType.getShape(); auto inputShape = inputType.getShape();
if (!std::equal(inputShape.begin(), inputShape.end(), if (!std::equal(inputShape.begin(), inputShape.end(),
resultType.getShape().rbegin())) { resultType.getShape().rbegin())) {
return op.emitError() return op.emitError()
▲ Show 20 Lines • Show All 146 Lines • Show Last 20 Lines

mlir/examples/toy/Ch7/mlir/MLIRGen.cpp

Show First 20 Lines • Show All 579 Lines • ▼ Show 20 Lines mlir::Value mlirGen(VarDeclExprAST &vardecl) {
// Handle the case where we are initializing a struct value. // Handle the case where we are initializing a struct value.
VarType varType = vardecl.getType(); VarType varType = vardecl.getType();
if (!varType.name.empty()) { if (!varType.name.empty()) {
// Check that the initializer type is the same as the variable // Check that the initializer type is the same as the variable
// declaration. // declaration.
mlir::Type type = getType(varType, vardecl.loc()); mlir::Type type = getType(varType, vardecl.loc());
if (!type) if (!type)
return nullptr; return nullptr;
if (type != value->getType()) { if (type != value.getType()) {
emitError(loc(vardecl.loc())) emitError(loc(vardecl.loc()))
<< "struct type of initializer is different than the variable " << "struct type of initializer is different than the variable "
"declaration. Got " "declaration. Got "
<< value->getType() << ", but expected " << type; << value.getType() << ", but expected " << type;
return nullptr; return nullptr;
} }
// Otherwise, we have the initializer value, but in case the variable was // Otherwise, we have the initializer value, but in case the variable was
// declared with specific shape, we emit a "reshape" operation. It will // declared with specific shape, we emit a "reshape" operation. It will
// get optimized out later as needed. // get optimized out later as needed.
} else if (!varType.shape.empty()) { } else if (!varType.shape.empty()) {
value = builder.create<ReshapeOp>(loc(vardecl.loc()), value = builder.create<ReshapeOp>(loc(vardecl.loc()),
▲ Show 20 Lines • Show All 74 Lines • Show Last 20 Lines

mlir/examples/toy/Ch7/mlir/ToyCombine.cpp

Show First 20 Lines • Show All 58 Lines • ▼ Show 20 Linesstruct SimplifyRedundantTranspose : public mlir::OpRewritePattern<TransposeOp> {
/// argument is the orchestrator of the sequence of rewrites. The pattern is /// argument is the orchestrator of the sequence of rewrites. The pattern is
/// expected to interact with it to perform any changes to the IR from here. /// expected to interact with it to perform any changes to the IR from here.
mlir::PatternMatchResult mlir::PatternMatchResult
matchAndRewrite(TransposeOp op, matchAndRewrite(TransposeOp op,
mlir::PatternRewriter &rewriter) const override { mlir::PatternRewriter &rewriter) const override {
// Look through the input of the current transpose. // Look through the input of the current transpose.
mlir::Value transposeInput = op.getOperand(); mlir::Value transposeInput = op.getOperand();
TransposeOp transposeInputOp = TransposeOp transposeInputOp =
llvm::dyn_cast_or_null<TransposeOp>(transposeInput->getDefiningOp()); llvm::dyn_cast_or_null<TransposeOp>(transposeInput.getDefiningOp());
// If the input is defined by another Transpose, bingo! // If the input is defined by another Transpose, bingo!
if (!transposeInputOp) if (!transposeInputOp)
return matchFailure(); return matchFailure();
// Use the rewriter to perform the replacement. // Use the rewriter to perform the replacement.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp}); rewriter.replaceOp(op, {transposeInputOp.getOperand()}, {transposeInputOp});
return matchSuccess(); return matchSuccess();
Show All 17 Lines

mlir/examples/toy/Ch7/mlir/ToyCombine.td

Show All 35 Lines
// Pattern-Match and Rewrite using Native Code Call // Pattern-Match and Rewrite using Native Code Call
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Native Code Calls may be used for more complex transformations using inline // Native Code Calls may be used for more complex transformations using inline
// C++ and C++ helper functions. // C++ and C++ helper functions.
// Reshape(Constant(x)) = x' // Reshape(Constant(x)) = x'
def ReshapeConstant : def ReshapeConstant :
NativeCodeCall<"$0.reshape(($1->getType()).cast<ShapedType>())">; NativeCodeCall<"$0.reshape(($1.getType()).cast<ShapedType>())">;
def FoldConstantReshapeOptPattern : Pat< def FoldConstantReshapeOptPattern : Pat<
(ReshapeOp:$res (ConstantOp $arg)), (ReshapeOp:$res (ConstantOp $arg)),
(ConstantOp (ReshapeConstant $arg, $res))>; (ConstantOp (ReshapeConstant $arg, $res))>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pattern-Match and Rewrite with Constraints // Pattern-Match and Rewrite with Constraints
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DRR allows for constraint checking when the transformation is conditional // DRR allows for constraint checking when the transformation is conditional
// on operand properties. // on operand properties.
// Reshape(x) = x, where input and output shapes are identical // Reshape(x) = x, where input and output shapes are identical
def TypesAreIdentical : Constraint<CPred<"$0->getType() == $1->getType()">>; def TypesAreIdentical : Constraint<CPred<"$0.getType() == $1.getType()">>;
def RedundantReshapeOptPattern : Pat< def RedundantReshapeOptPattern : Pat<
(ReshapeOp:$res $arg), (replaceWithValue $arg), (ReshapeOp:$res $arg), (replaceWithValue $arg),
[(TypesAreIdentical $res, $arg)]>; [(TypesAreIdentical $res, $arg)]>;
#endif // TOY_COMBINE #endif // TOY_COMBINE

mlir/include/mlir/Analysis/Dominance.h

Show First 20 Lines • Show All 63 Lines • ▼ Show 20 Lines bool dominates(Operation *a, Operation *b) {
return a == b || properlyDominates(a, b); return a == b || properlyDominates(a, b);
} }
/// Return true if value A properly dominates operation B. /// Return true if value A properly dominates operation B.
bool properlyDominates(Value a, Operation *b); bool properlyDominates(Value a, Operation *b);
/// Return true if operation A dominates operation B. /// Return true if operation A dominates operation B.
bool dominates(Value a, Operation *b) { bool dominates(Value a, Operation *b) {
return (Operation *)a->getDefiningOp() == b || properlyDominates(a, b); return (Operation *)a.getDefiningOp() == b || properlyDominates(a, b);
} }
/// Return true if the specified block A dominates block B. /// Return true if the specified block A dominates block B.
bool dominates(Block *a, Block *b) { bool dominates(Block *a, Block *b) {
return a == b || properlyDominates(a, b); return a == b || properlyDominates(a, b);
} }
/// Return true if the specified block A properly dominates block B. /// Return true if the specified block A properly dominates block B.
▲ Show 20 Lines • Show All 61 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/AffineOps/AffineOps.h

Show First 20 Lines • Show All 145 Lines • ▼ Show 20 Lines static void build(Builder *builder, OperationState &result, Value srcMemRef,
Value stride = nullptr, Value elementsPerStride = nullptr); Value stride = nullptr, Value elementsPerStride = nullptr);
/// Returns the operand index of the src memref. /// Returns the operand index of the src memref.
unsigned getSrcMemRefOperandIndex() { return 0; } unsigned getSrcMemRefOperandIndex() { return 0; }
/// Returns the source MemRefType for this DMA operation. /// Returns the source MemRefType for this DMA operation.
Value getSrcMemRef() { return getOperand(getSrcMemRefOperandIndex()); } Value getSrcMemRef() { return getOperand(getSrcMemRefOperandIndex()); }
MemRefType getSrcMemRefType() { MemRefType getSrcMemRefType() {
return getSrcMemRef()->getType().cast<MemRefType>(); return getSrcMemRef().getType().cast<MemRefType>();
} }
/// Returns the rank (number of indices) of the source MemRefType. /// Returns the rank (number of indices) of the source MemRefType.
unsigned getSrcMemRefRank() { return getSrcMemRefType().getRank(); } unsigned getSrcMemRefRank() { return getSrcMemRefType().getRank(); }
/// Returns the affine map used to access the src memref. /// Returns the affine map used to access the src memref.
AffineMap getSrcMap() { return getSrcMapAttr().getValue(); } AffineMap getSrcMap() { return getSrcMapAttr().getValue(); }
AffineMapAttr getSrcMapAttr() { AffineMapAttr getSrcMapAttr() {
return getAttr(getSrcMapAttrName()).cast<AffineMapAttr>(); return getAttr(getSrcMapAttrName()).cast<AffineMapAttr>();
} }
/// Returns the source memref affine map indices for this DMA operation. /// Returns the source memref affine map indices for this DMA operation.
operand_range getSrcIndices() { operand_range getSrcIndices() {
return {operand_begin() + getSrcMemRefOperandIndex() + 1, return {operand_begin() + getSrcMemRefOperandIndex() + 1,
operand_begin() + getSrcMemRefOperandIndex() + 1 + operand_begin() + getSrcMemRefOperandIndex() + 1 +
getSrcMap().getNumInputs()}; getSrcMap().getNumInputs()};
} }
/// Returns the memory space of the src memref. /// Returns the memory space of the src memref.
unsigned getSrcMemorySpace() { unsigned getSrcMemorySpace() {
return getSrcMemRef()->getType().cast<MemRefType>().getMemorySpace(); return getSrcMemRef().getType().cast<MemRefType>().getMemorySpace();
} }
/// Returns the operand index of the dst memref. /// Returns the operand index of the dst memref.
unsigned getDstMemRefOperandIndex() { unsigned getDstMemRefOperandIndex() {
return getSrcMemRefOperandIndex() + 1 + getSrcMap().getNumInputs(); return getSrcMemRefOperandIndex() + 1 + getSrcMap().getNumInputs();
} }
/// Returns the destination MemRefType for this DMA operations. /// Returns the destination MemRefType for this DMA operations.
Value getDstMemRef() { return getOperand(getDstMemRefOperandIndex()); } Value getDstMemRef() { return getOperand(getDstMemRefOperandIndex()); }
MemRefType getDstMemRefType() { MemRefType getDstMemRefType() {
return getDstMemRef()->getType().cast<MemRefType>(); return getDstMemRef().getType().cast<MemRefType>();
} }
/// Returns the rank (number of indices) of the destination MemRefType. /// Returns the rank (number of indices) of the destination MemRefType.
unsigned getDstMemRefRank() { unsigned getDstMemRefRank() {
return getDstMemRef()->getType().cast<MemRefType>().getRank(); return getDstMemRef().getType().cast<MemRefType>().getRank();
} }
/// Returns the memory space of the src memref. /// Returns the memory space of the src memref.
unsigned getDstMemorySpace() { unsigned getDstMemorySpace() {
return getDstMemRef()->getType().cast<MemRefType>().getMemorySpace(); return getDstMemRef().getType().cast<MemRefType>().getMemorySpace();
} }
/// Returns the affine map used to access the dst memref. /// Returns the affine map used to access the dst memref.
AffineMap getDstMap() { return getDstMapAttr().getValue(); } AffineMap getDstMap() { return getDstMapAttr().getValue(); }
AffineMapAttr getDstMapAttr() { AffineMapAttr getDstMapAttr() {
return getAttr(getDstMapAttrName()).cast<AffineMapAttr>(); return getAttr(getDstMapAttrName()).cast<AffineMapAttr>();
} }
/// Returns the destination memref indices for this DMA operation. /// Returns the destination memref indices for this DMA operation.
operand_range getDstIndices() { operand_range getDstIndices() {
return {operand_begin() + getDstMemRefOperandIndex() + 1, return {operand_begin() + getDstMemRefOperandIndex() + 1,
operand_begin() + getDstMemRefOperandIndex() + 1 + operand_begin() + getDstMemRefOperandIndex() + 1 +
getDstMap().getNumInputs()}; getDstMap().getNumInputs()};
} }
/// Returns the operand index of the tag memref. /// Returns the operand index of the tag memref.
unsigned getTagMemRefOperandIndex() { unsigned getTagMemRefOperandIndex() {
return getDstMemRefOperandIndex() + 1 + getDstMap().getNumInputs(); return getDstMemRefOperandIndex() + 1 + getDstMap().getNumInputs();
} }
/// Returns the Tag MemRef for this DMA operation. /// Returns the Tag MemRef for this DMA operation.
Value getTagMemRef() { return getOperand(getTagMemRefOperandIndex()); } Value getTagMemRef() { return getOperand(getTagMemRefOperandIndex()); }
MemRefType getTagMemRefType() { MemRefType getTagMemRefType() {
return getTagMemRef()->getType().cast<MemRefType>(); return getTagMemRef().getType().cast<MemRefType>();
} }
/// Returns the rank (number of indices) of the tag MemRefType. /// Returns the rank (number of indices) of the tag MemRefType.
unsigned getTagMemRefRank() { unsigned getTagMemRefRank() {
return getTagMemRef()->getType().cast<MemRefType>().getRank(); return getTagMemRef().getType().cast<MemRefType>().getRank();
} }
/// Returns the affine map used to access the tag memref. /// Returns the affine map used to access the tag memref.
AffineMap getTagMap() { return getTagMapAttr().getValue(); } AffineMap getTagMap() { return getTagMapAttr().getValue(); }
AffineMapAttr getTagMapAttr() { AffineMapAttr getTagMapAttr() {
return getAttr(getTagMapAttrName()).cast<AffineMapAttr>(); return getAttr(getTagMapAttrName()).cast<AffineMapAttr>();
} }
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Linespublic:
static void build(Builder *builder, OperationState &result, Value tagMemRef, static void build(Builder *builder, OperationState &result, Value tagMemRef,
AffineMap tagMap, ValueRange tagIndices, Value numElements); AffineMap tagMap, ValueRange tagIndices, Value numElements);
static StringRef getOperationName() { return "affine.dma_wait"; } static StringRef getOperationName() { return "affine.dma_wait"; }
// Returns the Tag MemRef associated with the DMA operation being waited on. // Returns the Tag MemRef associated with the DMA operation being waited on.
Value getTagMemRef() { return getOperand(0); } Value getTagMemRef() { return getOperand(0); }
MemRefType getTagMemRefType() { MemRefType getTagMemRefType() {
return getTagMemRef()->getType().cast<MemRefType>(); return getTagMemRef().getType().cast<MemRefType>();
} }
/// Returns the affine map used to access the tag memref. /// Returns the affine map used to access the tag memref.
AffineMap getTagMap() { return getTagMapAttr().getValue(); } AffineMap getTagMap() { return getTagMapAttr().getValue(); }
AffineMapAttr getTagMapAttr() { AffineMapAttr getTagMapAttr() {
return getAttr(getTagMapAttrName()).cast<AffineMapAttr>(); return getAttr(getTagMapAttrName()).cast<AffineMapAttr>();
} }
// Returns the tag memref index for this DMA operation. // Returns the tag memref index for this DMA operation.
operand_range getTagIndices() { operand_range getTagIndices() {
return {operand_begin() + 1, return {operand_begin() + 1,
operand_begin() + 1 + getTagMap().getNumInputs()}; operand_begin() + 1 + getTagMap().getNumInputs()};
} }
// Returns the rank (number of indices) of the tag memref. // Returns the rank (number of indices) of the tag memref.
unsigned getTagMemRefRank() { unsigned getTagMemRefRank() {
return getTagMemRef()->getType().cast<MemRefType>().getRank(); return getTagMemRef().getType().cast<MemRefType>().getRank();
} }
/// Returns the AffineMapAttr associated with 'memref'. /// Returns the AffineMapAttr associated with 'memref'.
NamedAttribute getAffineMapAttrForMemRef(Value memref) { NamedAttribute getAffineMapAttrForMemRef(Value memref) {
assert(memref == getTagMemRef()); assert(memref == getTagMemRef());
return {Identifier::get(getTagMapAttrName(), getContext()), return {Identifier::get(getTagMapAttrName(), getContext()),
getTagMapAttr()}; getTagMapAttr()};
} }
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Linespublic:
/// Returns the operand index of the memref. /// Returns the operand index of the memref.
unsigned getMemRefOperandIndex() { return 0; } unsigned getMemRefOperandIndex() { return 0; }
/// Get memref operand. /// Get memref operand.
Value getMemRef() { return getOperand(getMemRefOperandIndex()); } Value getMemRef() { return getOperand(getMemRefOperandIndex()); }
void setMemRef(Value value) { setOperand(getMemRefOperandIndex(), value); } void setMemRef(Value value) { setOperand(getMemRefOperandIndex(), value); }
MemRefType getMemRefType() { MemRefType getMemRefType() {
return getMemRef()->getType().cast<MemRefType>(); return getMemRef().getType().cast<MemRefType>();
} }
/// Get affine map operands. /// Get affine map operands.
operand_range getMapOperands() { return llvm::drop_begin(getOperands(), 1); } operand_range getMapOperands() { return llvm::drop_begin(getOperands(), 1); }
/// Returns the affine map used to index the memref for this operation. /// Returns the affine map used to index the memref for this operation.
AffineMap getAffineMap() { return getAffineMapAttr().getValue(); } AffineMap getAffineMap() { return getAffineMapAttr().getValue(); }
AffineMapAttr getAffineMapAttr() { AffineMapAttr getAffineMapAttr() {
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Linespublic:
/// Returns the operand index of the memref. /// Returns the operand index of the memref.
unsigned getMemRefOperandIndex() { return 1; } unsigned getMemRefOperandIndex() { return 1; }
/// Get memref operand. /// Get memref operand.
Value getMemRef() { return getOperand(getMemRefOperandIndex()); } Value getMemRef() { return getOperand(getMemRefOperandIndex()); }
void setMemRef(Value value) { setOperand(getMemRefOperandIndex(), value); } void setMemRef(Value value) { setOperand(getMemRefOperandIndex(), value); }
MemRefType getMemRefType() { MemRefType getMemRefType() {
return getMemRef()->getType().cast<MemRefType>(); return getMemRef().getType().cast<MemRefType>();
} }
/// Get affine map operands. /// Get affine map operands.
operand_range getMapOperands() { return llvm::drop_begin(getOperands(), 2); } operand_range getMapOperands() { return llvm::drop_begin(getOperands(), 2); }
/// Returns the affine map used to index the memref for this operation. /// Returns the affine map used to index the memref for this operation.
AffineMap getAffineMap() { return getAffineMapAttr().getValue(); } AffineMap getAffineMap() { return getAffineMapAttr().getValue(); }
AffineMapAttr getAffineMapAttr() { AffineMapAttr getAffineMapAttr() {
▲ Show 20 Lines • Show All 184 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/AffineOps/AffineOps.td

Show First 20 Lines • Show All 290 Lines • ▼ Show 20 Lines [{
result.addOperands(mapOperands); result.addOperands(mapOperands);
result.addAttribute(getLocalityHintAttrName(), localityHintAttr); result.addAttribute(getLocalityHintAttrName(), localityHintAttr);
result.addAttribute(getIsWriteAttrName(), isWriteAttr); result.addAttribute(getIsWriteAttrName(), isWriteAttr);
result.addAttribute(getIsDataCacheAttrName(), isDataCacheAttr); result.addAttribute(getIsDataCacheAttrName(), isDataCacheAttr);
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
MemRefType getMemRefType() { MemRefType getMemRefType() {
return memref()->getType().cast<MemRefType>(); return memref().getType().cast<MemRefType>();
} }
/// Returns the affine map used to index the memref for this operation. /// Returns the affine map used to index the memref for this operation.
AffineMap getAffineMap() { return getAffineMapAttr().getValue(); } AffineMap getAffineMap() { return getAffineMapAttr().getValue(); }
AffineMapAttr getAffineMapAttr() { AffineMapAttr getAffineMapAttr() {
return getAttr(getMapAttrName()).cast<AffineMapAttr>(); return getAttr(getMapAttrName()).cast<AffineMapAttr>();
} }
▲ Show 20 Lines • Show All 43 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td

Show First 20 Lines • Show All 172 Lines • ▼ Show 20 Linesdef LLVM_ICmpOp : LLVM_OneResultOp<"icmp", [NoSideEffect]>,
Arguments<(ins ICmpPredicate:$predicate, LLVM_Type:$lhs, Arguments<(ins ICmpPredicate:$predicate, LLVM_Type:$lhs,
LLVM_Type:$rhs)> { LLVM_Type:$rhs)> {
let llvmBuilder = [{ let llvmBuilder = [{
$res = builder.CreateICmp(getLLVMCmpPredicate($predicate), $lhs, $rhs); $res = builder.CreateICmp(getLLVMCmpPredicate($predicate), $lhs, $rhs);
}]; }];
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *b, OperationState &result, ICmpPredicate predicate, Value lhs, " "Builder *b, OperationState &result, ICmpPredicate predicate, Value lhs, "
"Value rhs", [{ "Value rhs", [{
LLVMDialect *dialect = &lhs->getType().cast<LLVMType>().getDialect(); LLVMDialect *dialect = &lhs.getType().cast<LLVMType>().getDialect();
build(b, result, LLVMType::getInt1Ty(dialect), build(b, result, LLVMType::getInt1Ty(dialect),
b->getI64IntegerAttr(static_cast<int64_t>(predicate)), lhs, rhs); b->getI64IntegerAttr(static_cast<int64_t>(predicate)), lhs, rhs);
}]>]; }]>];
let parser = [{ return parseCmpOp<ICmpPredicate>(parser, result); }]; let parser = [{ return parseCmpOp<ICmpPredicate>(parser, result); }];
let printer = [{ printICmpOp(p, *this); }]; let printer = [{ printICmpOp(p, *this); }];
} }
// Predicate for float comparisons // Predicate for float comparisons
Show All 30 Linesdef LLVM_FCmpOp : LLVM_OneResultOp<"fcmp", [NoSideEffect]>,
Arguments<(ins FCmpPredicate:$predicate, LLVM_Type:$lhs, Arguments<(ins FCmpPredicate:$predicate, LLVM_Type:$lhs,
LLVM_Type:$rhs)> { LLVM_Type:$rhs)> {
let llvmBuilder = [{ let llvmBuilder = [{
$res = builder.CreateFCmp(getLLVMCmpPredicate($predicate), $lhs, $rhs); $res = builder.CreateFCmp(getLLVMCmpPredicate($predicate), $lhs, $rhs);
}]; }];
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *b, OperationState &result, FCmpPredicate predicate, Value lhs, " "Builder *b, OperationState &result, FCmpPredicate predicate, Value lhs, "
"Value rhs", [{ "Value rhs", [{
LLVMDialect *dialect = &lhs->getType().cast<LLVMType>().getDialect(); LLVMDialect *dialect = &lhs.getType().cast<LLVMType>().getDialect();
build(b, result, LLVMType::getInt1Ty(dialect), build(b, result, LLVMType::getInt1Ty(dialect),
b->getI64IntegerAttr(static_cast<int64_t>(predicate)), lhs, rhs); b->getI64IntegerAttr(static_cast<int64_t>(predicate)), lhs, rhs);
}]>]; }]>];
let parser = [{ return parseCmpOp<FCmpPredicate>(parser, result); }]; let parser = [{ return parseCmpOp<FCmpPredicate>(parser, result); }];
let printer = [{ printFCmpOp(p, *this); }]; let printer = [{ printFCmpOp(p, *this); }];
} }
// Floating point binary operations. // Floating point binary operations.
▲ Show 20 Lines • Show All 43 Lines • ▼ Show 20 Linesdef LLVM_GEPOp : LLVM_OneResultOp<"getelementptr", [NoSideEffect]>,
let parser = [{ return parseGEPOp(parser, result); }]; let parser = [{ return parseGEPOp(parser, result); }];
let printer = [{ printGEPOp(p, *this); }]; let printer = [{ printGEPOp(p, *this); }];
} }
def LLVM_LoadOp : LLVM_OneResultOp<"load">, Arguments<(ins LLVM_Type:$addr)>, def LLVM_LoadOp : LLVM_OneResultOp<"load">, Arguments<(ins LLVM_Type:$addr)>,
LLVM_Builder<"$res = builder.CreateLoad($addr);"> { LLVM_Builder<"$res = builder.CreateLoad($addr);"> {
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *b, OperationState &result, Value addr", "Builder *b, OperationState &result, Value addr",
[{ [{
auto type = addr->getType().cast<LLVM::LLVMType>().getPointerElementTy(); auto type = addr.getType().cast<LLVM::LLVMType>().getPointerElementTy();
build(b, result, type, addr); build(b, result, type, addr);
}]>]; }]>];
let parser = [{ return parseLoadOp(parser, result); }]; let parser = [{ return parseLoadOp(parser, result); }];
let printer = [{ printLoadOp(p, *this); }]; let printer = [{ printLoadOp(p, *this); }];
} }
def LLVM_StoreOp : LLVM_ZeroResultOp<"store">, def LLVM_StoreOp : LLVM_ZeroResultOp<"store">,
Arguments<(ins LLVM_Type:$value, LLVM_Type:$addr)>, Arguments<(ins LLVM_Type:$value, LLVM_Type:$addr)>,
LLVM_Builder<"builder.CreateStore($value, $addr);"> { LLVM_Builder<"builder.CreateStore($value, $addr);"> {
▲ Show 20 Lines • Show All 76 Lines • ▼ Show 20 Linesdef LLVM_InsertValueOp : LLVM_OneResultOp<"insertvalue", [NoSideEffect]>,
string llvmBuilder = [{ string llvmBuilder = [{
$res = builder.CreateInsertValue($container, $value, $res = builder.CreateInsertValue($container, $value,
extractPosition($position)); extractPosition($position));
}]; }];
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *b, OperationState &result, Value container, Value value, " "Builder *b, OperationState &result, Value container, Value value, "
"ArrayAttr position", "ArrayAttr position",
[{ [{
build(b, result, container->getType(), container, value, position); build(b, result, container.getType(), container, value, position);
}]>]; }]>];
let parser = [{ return parseInsertValueOp(parser, result); }]; let parser = [{ return parseInsertValueOp(parser, result); }];
let printer = [{ printInsertValueOp(p, *this); }]; let printer = [{ printInsertValueOp(p, *this); }];
} }
def LLVM_ShuffleVectorOp def LLVM_ShuffleVectorOp
: LLVM_OneResultOp<"shufflevector", [NoSideEffect]>, : LLVM_OneResultOp<"shufflevector", [NoSideEffect]>,
Arguments<(ins LLVM_Type:$v1, LLVM_Type:$v2, ArrayAttr:$mask)>, Arguments<(ins LLVM_Type:$v1, LLVM_Type:$v2, ArrayAttr:$mask)>,
LLVM_Builder< LLVM_Builder<
"$res = builder.CreateShuffleVector($v1, $v2, extractPosition($mask));"> { "$res = builder.CreateShuffleVector($v1, $v2, extractPosition($mask));"> {
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *b, OperationState &result, Value v1, Value v2, " "Builder *b, OperationState &result, Value v1, Value v2, "
"ArrayAttr mask, ArrayRef<NamedAttribute> attrs = {}">]; "ArrayAttr mask, ArrayRef<NamedAttribute> attrs = {}">];
let verifier = [{ let verifier = [{
auto wrappedVectorType1 = v1()->getType().cast<LLVM::LLVMType>(); auto wrappedVectorType1 = v1().getType().cast<LLVM::LLVMType>();
auto wrappedVectorType2 = v2()->getType().cast<LLVM::LLVMType>(); auto wrappedVectorType2 = v2().getType().cast<LLVM::LLVMType>();
if (!wrappedVectorType2.getUnderlyingType()->isVectorTy()) if (!wrappedVectorType2.getUnderlyingType()->isVectorTy())
return emitOpError("expected LLVM IR Dialect vector type for operand #2"); return emitOpError("expected LLVM IR Dialect vector type for operand #2");
if (wrappedVectorType1.getVectorElementType() != if (wrappedVectorType1.getVectorElementType() !=
wrappedVectorType2.getVectorElementType()) wrappedVectorType2.getVectorElementType())
return emitOpError("expected matching LLVM IR Dialect element types"); return emitOpError("expected matching LLVM IR Dialect element types");
return success(); return success();
}]; }];
let parser = [{ return parseShuffleVectorOp(parser, result); }]; let parser = [{ return parseShuffleVectorOp(parser, result); }];
let printer = [{ printShuffleVectorOp(p, *this); }]; let printer = [{ printShuffleVectorOp(p, *this); }];
} }
// Misc operations. // Misc operations.
def LLVM_SelectOp def LLVM_SelectOp
: LLVM_OneResultOp<"select", [NoSideEffect]>, : LLVM_OneResultOp<"select", [NoSideEffect]>,
Arguments<(ins LLVM_Type:$condition, LLVM_Type:$trueValue, Arguments<(ins LLVM_Type:$condition, LLVM_Type:$trueValue,
LLVM_Type:$falseValue)>, LLVM_Type:$falseValue)>,
LLVM_Builder< LLVM_Builder<
"$res = builder.CreateSelect($condition, $trueValue, $falseValue);"> { "$res = builder.CreateSelect($condition, $trueValue, $falseValue);"> {
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *b, OperationState &result, Value condition, Value lhs, " "Builder *b, OperationState &result, Value condition, Value lhs, "
"Value rhs", [{ "Value rhs", [{
build(b, result, lhs->getType(), condition, lhs, rhs); build(b, result, lhs.getType(), condition, lhs, rhs);
}]>]; }]>];
let parser = [{ return parseSelectOp(parser, result); }]; let parser = [{ return parseSelectOp(parser, result); }];
let printer = [{ printSelectOp(p, *this); }]; let printer = [{ printSelectOp(p, *this); }];
} }
// Terminators. // Terminators.
def LLVM_BrOp : LLVM_TerminatorOp<"br", []> { def LLVM_BrOp : LLVM_TerminatorOp<"br", []> {
let parser = [{ return parseBrOp(parser, result); }]; let parser = [{ return parseBrOp(parser, result); }];
▲ Show 20 Lines • Show All 308 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Linalg/EDSC/Builders.h

Show First 20 Lines • Show All 50 Lines • ▼ Show 20 Linesstruct StructuredIndexed {
} }
operator Value() const /* implicit */ { return value; } operator Value() const /* implicit */ { return value; }
ArrayRef<AffineExpr> getExprs() { return exprs; } ArrayRef<AffineExpr> getExprs() { return exprs; }
private: private:
StructuredIndexed(Value v, ArrayRef<AffineExpr> indexings) StructuredIndexed(Value v, ArrayRef<AffineExpr> indexings)
: value(v), exprs(indexings.begin(), indexings.end()) { : value(v), exprs(indexings.begin(), indexings.end()) {
assert(v->getType().isa<MemRefType>() && "MemRefType expected"); assert(v.getType().isa<MemRefType>() && "MemRefType expected");
} }
StructuredIndexed(ValueHandle v, ArrayRef<AffineExpr> indexings) StructuredIndexed(ValueHandle v, ArrayRef<AffineExpr> indexings)
: StructuredIndexed(v.getValue(), indexings) {} : StructuredIndexed(v.getValue(), indexings) {}
Value value; Value value;
SmallVector<AffineExpr, 4> exprs; SmallVector<AffineExpr, 4> exprs;
}; };
▲ Show 20 Lines • Show All 161 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Linalg/IR/LinalgOps.td

Show First 20 Lines • Show All 159 Lines • ▼ Show 20 Lines let builders = [OpBuilder<
"ValueRange indexings">]; "ValueRange indexings">];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
enum { FirstIndexingOperand = 1 }; enum { FirstIndexingOperand = 1 };
unsigned getRank() { return getShapedType().getRank(); } unsigned getRank() { return getShapedType().getRank(); }
Type getElementType() { return getShapedType().getElementType(); } Type getElementType() { return getShapedType().getElementType(); }
ShapedType getShapedType() { return getType().cast<ShapedType>(); } ShapedType getShapedType() { return getType().cast<ShapedType>(); }
unsigned getBaseViewRank() { return getBaseViewType().getRank(); } unsigned getBaseViewRank() { return getBaseViewType().getRank(); }
ShapedType getBaseViewType() { return view()->getType().cast<ShapedType>();} ShapedType getBaseViewType() { return view().getType().cast<ShapedType>();}
// Get the underlying indexing at a given rank. // Get the underlying indexing at a given rank.
Value indexing(unsigned rank) { return *(indexings().begin() + rank); } Value indexing(unsigned rank) { return *(indexings().begin() + rank); }
// Get the subset of indexings that are of RangeType. // Get the subset of indexings that are of RangeType.
SmallVector<Value, 8> getRanges() { SmallVector<Value, 8> getRanges() {
SmallVector<Value, 8> res; SmallVector<Value, 8> res;
for (auto operand : indexings()) for (auto operand : indexings())
if (!operand->getType().isa<IndexType>()) if (!operand.getType().isa<IndexType>())
res.push_back(operand); res.push_back(operand);
return res; return res;
} }
}]; }];
} }
def Linalg_TransposeOp : Linalg_Op<"transpose", [NoSideEffect]>, def Linalg_TransposeOp : Linalg_Op<"transpose", [NoSideEffect]>,
Arguments<(ins AnyStridedMemRef:$view, AffineMapAttr:$permutation)>, Arguments<(ins AnyStridedMemRef:$view, AffineMapAttr:$permutation)>,
Show All 20 Lines if (!permutation().isPermutation())
return emitOpError("expected a permutation map"); return emitOpError("expected a permutation map");
if (permutation().getNumDims() != getShapedType().getRank()) if (permutation().getNumDims() != getShapedType().getRank())
return emitOpError("expected a permutation map of same rank as the view"); return emitOpError("expected a permutation map of same rank as the view");
return success(); return success();
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static StringRef getPermutationAttrName() { return "permutation"; } static StringRef getPermutationAttrName() { return "permutation"; }
ShapedType getShapedType() { return view()->getType().cast<ShapedType>(); } ShapedType getShapedType() { return view().getType().cast<ShapedType>(); }
}]; }];
} }
def Linalg_YieldOp : Linalg_Op<"yield", [NativeOpTrait<"IsTerminator">]>, def Linalg_YieldOp : Linalg_Op<"yield", [NativeOpTrait<"IsTerminator">]>,
Arguments<(ins Variadic<AnyType>:$values)> { Arguments<(ins Variadic<AnyType>:$values)> {
let summary = "Linalg yield operation"; let summary = "Linalg yield operation";
let description = [{ let description = [{
`linalg.yield` is a special terminator operation for blocks inside regions `linalg.yield` is a special terminator operation for blocks inside regions
Show All 12 Lines

mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td

Show First 20 Lines • Show All 230 Lines • ▼ Show 20 Lines let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value input, Value output", [{ "Builder *builder, OperationState &result, Value input, Value output", [{
return build( return build(
builder, result, input, output, AffineMapAttr(), AffineMapAttr()); builder, result, input, output, AffineMapAttr(), AffineMapAttr());
}]>]; }]>];
let extraClassDeclaration = libraryCallName # [{ let extraClassDeclaration = libraryCallName # [{
ArrayAttr indexing_maps(); ArrayAttr indexing_maps();
ArrayAttr iterator_types() { ArrayAttr iterator_types() {
unsigned nPar = input()->getType().cast<ShapedType>().getRank(); unsigned nPar = input().getType().cast<ShapedType>().getRank();
MLIRContext *ctx = getContext(); MLIRContext *ctx = getContext();
SmallVector<Attribute, 8> iters( SmallVector<Attribute, 8> iters(
nPar, StringAttr::get(getParallelIteratorTypeName(), ctx)); nPar, StringAttr::get(getParallelIteratorTypeName(), ctx));
return ArrayAttr::get(iters, ctx); return ArrayAttr::get(iters, ctx);
} }
}]; }];
let verifier = [{ return ::verify(*this); }]; let verifier = [{ return ::verify(*this); }];
} }
def FillOp : LinalgStructured_Op<"fill", [NInputs<0>, NOutputs<1>]> { def FillOp : LinalgStructured_Op<"fill", [NInputs<0>, NOutputs<1>]> {
let arguments = (ins AnyStridedMemRef:$input, let arguments = (ins AnyStridedMemRef:$input,
AnyTypeOf<[AnyFloat, AnyInteger, AnyVector]>:$value); AnyTypeOf<[AnyFloat, AnyInteger, AnyVector]>:$value);
let extraClassDeclaration = libraryCallName # [{ let extraClassDeclaration = libraryCallName # [{
ArrayAttr indexing_maps(); ArrayAttr indexing_maps();
ArrayAttr iterator_types() { ArrayAttr iterator_types() {
unsigned nPar = input()->getType().cast<ShapedType>().getRank(); unsigned nPar = input().getType().cast<ShapedType>().getRank();
MLIRContext *ctx = getContext(); MLIRContext *ctx = getContext();
SmallVector<Attribute, 8> iters( SmallVector<Attribute, 8> iters(
nPar, StringAttr::get(getParallelIteratorTypeName(), ctx)); nPar, StringAttr::get(getParallelIteratorTypeName(), ctx));
return ArrayAttr::get(iters, ctx); return ArrayAttr::get(iters, ctx);
} }
}]; }];
let verifier = [{ return ::verify(*this); }]; let verifier = [{ return ::verify(*this); }];
} }
▲ Show 20 Lines • Show All 418 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Linalg/IR/LinalgTraits.h

Show First 20 Lines • Show All 77 Lines • ▼ Show 20 Linespublic:
Optional<unsigned> getIndexOfInput(Value value) { Optional<unsigned> getIndexOfInput(Value value) {
auto it = llvm::find(getInputs(), value); auto it = llvm::find(getInputs(), value);
if (it != getInputs().end()) if (it != getInputs().end())
return it - getInputs().begin(); return it - getInputs().begin();
return llvm::None; return llvm::None;
} }
/// Return the `i`-th input buffer type. /// Return the `i`-th input buffer type.
ShapedType getInputShapedType(unsigned i) { ShapedType getInputShapedType(unsigned i) {
return getInput(i)->getType().template cast<ShapedType>(); return getInput(i).getType().template cast<ShapedType>();
} }
/// Return the range over inputs. /// Return the range over inputs.
Operation::operand_range getInputs() { Operation::operand_range getInputs() {
auto range = this->getOperation()->getOperands(); auto range = this->getOperation()->getOperands();
return {range.begin(), range.begin() + nInputs()}; return {range.begin(), range.begin() + nInputs()};
} }
/// Return the `i`-th output. /// Return the `i`-th output.
Value getOutput(unsigned i) { Value getOutput(unsigned i) {
return this->getOperation()->getOperand(nInputs() + i); return this->getOperation()->getOperand(nInputs() + i);
} }
/// Return the index of `value` in the list of output values if found, /// Return the index of `value` in the list of output values if found,
/// llvm::None otherwise. /// llvm::None otherwise.
Optional<unsigned> getIndexOfOutput(Value value) { Optional<unsigned> getIndexOfOutput(Value value) {
auto it = llvm::find(getOutputs(), value); auto it = llvm::find(getOutputs(), value);
if (it != getOutputs().end()) if (it != getOutputs().end())
return it - getOutputs().begin(); return it - getOutputs().begin();
return llvm::None; return llvm::None;
} }
/// Return the `i`-th output buffer type. /// Return the `i`-th output buffer type.
ShapedType getOutputShapedType(unsigned i) { ShapedType getOutputShapedType(unsigned i) {
return getOutput(i)->getType().template cast<ShapedType>(); return getOutput(i).getType().template cast<ShapedType>();
} }
/// Query whether the op has only MemRef input and outputs. /// Query whether the op has only MemRef input and outputs.
bool hasBufferSemantics() { bool hasBufferSemantics() {
return this->getOperation()->getNumResults() == 0 && return this->getOperation()->getNumResults() == 0 &&
llvm::all_of(getInputsAndOutputs(), llvm::all_of(getInputsAndOutputs(),
[](Value v) { return v.getType().isa<MemRefType>(); }); [](Value v) { return v.getType().isa<MemRefType>(); });
} }
/// Query the subset of input operands that are of ranked tensor type. /// Query the subset of input operands that are of ranked tensor type.
▲ Show 20 Lines • Show All 65 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Linalg/Transforms/LinalgTransformPatterns.td

Show All 31 Lines
class AffineMapDomainHasDim<int n> : CPred<[{ class AffineMapDomainHasDim<int n> : CPred<[{
op.getAttrOfType<ArrayAttr>(getIndexingMapsAttrName()).getValue()[0]. op.getAttrOfType<ArrayAttr>(getIndexingMapsAttrName()).getValue()[0].
cast<AffineMapAttr>().getValue().getNumDims() ==}] # n # [{}]>; cast<AffineMapAttr>().getValue().getNumDims() ==}] # n # [{}]>;
class HasOperandsOfType<string type>: CPred<[{ class HasOperandsOfType<string type>: CPred<[{
llvm::any_of(op.getOperands(), llvm::any_of(op.getOperands(),
[](Value v) { [](Value v) {
return dyn_cast_or_null<}] # type # [{>(v->getDefiningOp()); return dyn_cast_or_null<}] # type # [{>(v.getDefiningOp());
}) })
}]>; }]>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Linalg fusion patterns. // Linalg fusion patterns.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// //
// In the future, tile sizes should be derived from op properties + machine // In the future, tile sizes should be derived from op properties + machine
▲ Show 20 Lines • Show All 64 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Linalg/Utils/Utils.h

Show First 20 Lines • Show All 114 Lines • ▼ Show 20 Lines
/// Returns the linearized list of all view dimensions in a linalgOp. Applying /// Returns the linearized list of all view dimensions in a linalgOp. Applying
/// the inverse, concatenated loopToOperandRangeMaps to this list allows the /// the inverse, concatenated loopToOperandRangeMaps to this list allows the
/// derivation of loop ranges for any linalgOp. /// derivation of loop ranges for any linalgOp.
template <typename ConcreteOp> template <typename ConcreteOp>
SmallVector<Value, 8> getViewSizes(ConcreteOp linalgOp) { SmallVector<Value, 8> getViewSizes(ConcreteOp linalgOp) {
SmallVector<Value, 8> res; SmallVector<Value, 8> res;
for (auto v : linalgOp.getInputsAndOutputs()) { for (auto v : linalgOp.getInputsAndOutputs()) {
MemRefType t = v->getType().template cast<MemRefType>(); MemRefType t = v.getType().template cast<MemRefType>();
for (unsigned i = 0; i < t.getRank(); ++i) for (unsigned i = 0; i < t.getRank(); ++i)
res.push_back(edsc::intrinsics::dim(v, i)); res.push_back(edsc::intrinsics::dim(v, i));
} }
return res; return res;
} }
/// Returns the values obtained by applying `map` to the list of values. /// Returns the values obtained by applying `map` to the list of values.
/// When non-null, the optional pointer `folder` is used to call into the /// When non-null, the optional pointer `folder` is used to call into the
▲ Show 20 Lines • Show All 110 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/QuantOps/QuantOps.td

Show First 20 Lines • Show All 193 Lines • ▼ Show 20 Linesdef quant_StatisticsOp : quant_Op<"stats", [SameOperandsAndResultType]> {
let arguments = (ins let arguments = (ins
quant_RealValueType:$arg, quant_RealValueType:$arg,
ElementsAttr:$layerStats, ElementsAttr:$layerStats,
OptionalAttr<ElementsAttr>:$axisStats, OptionalAttr<ElementsAttr>:$axisStats,
OptionalAttr<I64Attr>:$axis); OptionalAttr<I64Attr>:$axis);
let results = (outs quant_RealValueType); let results = (outs quant_RealValueType);
let verifier = [{ let verifier = [{
auto tensorArg = arg()->getType().dyn_cast<TensorType>(); auto tensorArg = arg().getType().dyn_cast<TensorType>();
if (!tensorArg) return emitOpError("arg needs to be tensor type."); if (!tensorArg) return emitOpError("arg needs to be tensor type.");
// Verify layerStats attribute. // Verify layerStats attribute.
{ {
auto layerStatsType = layerStats().getType(); auto layerStatsType = layerStats().getType();
if (!layerStatsType.getElementType().isa<FloatType>()) { if (!layerStatsType.getElementType().isa<FloatType>()) {
return emitOpError( return emitOpError(
"layerStats must have a floating point element type"); "layerStats must have a floating point element type");
▲ Show 20 Lines • Show All 48 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/StandardOps/Ops.h

Show First 20 Lines • Show All 177 Lines • ▼ Show 20 Lines static void build(Builder *builder, OperationState &result, Value srcMemRef,
ValueRange destIndices, Value numElements, Value tagMemRef, ValueRange destIndices, Value numElements, Value tagMemRef,
ValueRange tagIndices, Value stride = nullptr, ValueRange tagIndices, Value stride = nullptr,
Value elementsPerStride = nullptr); Value elementsPerStride = nullptr);
// Returns the source MemRefType for this DMA operation. // Returns the source MemRefType for this DMA operation.
Value getSrcMemRef() { return getOperand(0); } Value getSrcMemRef() { return getOperand(0); }
// Returns the rank (number of indices) of the source MemRefType. // Returns the rank (number of indices) of the source MemRefType.
unsigned getSrcMemRefRank() { unsigned getSrcMemRefRank() {
return getSrcMemRef()->getType().cast<MemRefType>().getRank(); return getSrcMemRef().getType().cast<MemRefType>().getRank();
} }
// Returns the source memref indices for this DMA operation. // Returns the source memref indices for this DMA operation.
operand_range getSrcIndices() { operand_range getSrcIndices() {
return {getOperation()->operand_begin() + 1, return {getOperation()->operand_begin() + 1,
getOperation()->operand_begin() + 1 + getSrcMemRefRank()}; getOperation()->operand_begin() + 1 + getSrcMemRefRank()};
} }
// Returns the destination MemRefType for this DMA operations. // Returns the destination MemRefType for this DMA operations.
Value getDstMemRef() { return getOperand(1 + getSrcMemRefRank()); } Value getDstMemRef() { return getOperand(1 + getSrcMemRefRank()); }
// Returns the rank (number of indices) of the destination MemRefType. // Returns the rank (number of indices) of the destination MemRefType.
unsigned getDstMemRefRank() { unsigned getDstMemRefRank() {
return getDstMemRef()->getType().cast<MemRefType>().getRank(); return getDstMemRef().getType().cast<MemRefType>().getRank();
} }
unsigned getSrcMemorySpace() { unsigned getSrcMemorySpace() {
return getSrcMemRef()->getType().cast<MemRefType>().getMemorySpace(); return getSrcMemRef().getType().cast<MemRefType>().getMemorySpace();
} }
unsigned getDstMemorySpace() { unsigned getDstMemorySpace() {
return getDstMemRef()->getType().cast<MemRefType>().getMemorySpace(); return getDstMemRef().getType().cast<MemRefType>().getMemorySpace();
} }
// Returns the destination memref indices for this DMA operation. // Returns the destination memref indices for this DMA operation.
operand_range getDstIndices() { operand_range getDstIndices() {
return {getOperation()->operand_begin() + 1 + getSrcMemRefRank() + 1, return {getOperation()->operand_begin() + 1 + getSrcMemRefRank() + 1,
getOperation()->operand_begin() + 1 + getSrcMemRefRank() + 1 + getOperation()->operand_begin() + 1 + getSrcMemRefRank() + 1 +
getDstMemRefRank()}; getDstMemRefRank()};
} }
// Returns the number of elements being transferred by this DMA operation. // Returns the number of elements being transferred by this DMA operation.
Value getNumElements() { Value getNumElements() {
return getOperand(1 + getSrcMemRefRank() + 1 + getDstMemRefRank()); return getOperand(1 + getSrcMemRefRank() + 1 + getDstMemRefRank());
} }
// Returns the Tag MemRef for this DMA operation. // Returns the Tag MemRef for this DMA operation.
Value getTagMemRef() { Value getTagMemRef() {
return getOperand(1 + getSrcMemRefRank() + 1 + getDstMemRefRank() + 1); return getOperand(1 + getSrcMemRefRank() + 1 + getDstMemRefRank() + 1);
} }
// Returns the rank (number of indices) of the tag MemRefType. // Returns the rank (number of indices) of the tag MemRefType.
unsigned getTagMemRefRank() { unsigned getTagMemRefRank() {
return getTagMemRef()->getType().cast<MemRefType>().getRank(); return getTagMemRef().getType().cast<MemRefType>().getRank();
} }
// Returns the tag memref index for this DMA operation. // Returns the tag memref index for this DMA operation.
operand_range getTagIndices() { operand_range getTagIndices() {
unsigned tagIndexStartPos = unsigned tagIndexStartPos =
1 + getSrcMemRefRank() + 1 + getDstMemRefRank() + 1 + 1; 1 + getSrcMemRefRank() + 1 + getDstMemRefRank() + 1 + 1;
return {getOperation()->operand_begin() + tagIndexStartPos, return {getOperation()->operand_begin() + tagIndexStartPos,
getOperation()->operand_begin() + tagIndexStartPos + getOperation()->operand_begin() + tagIndexStartPos +
▲ Show 20 Lines • Show All 74 Lines • ▼ Show 20 Linespublic:
// Returns the tag memref index for this DMA operation. // Returns the tag memref index for this DMA operation.
operand_range getTagIndices() { operand_range getTagIndices() {
return {getOperation()->operand_begin() + 1, return {getOperation()->operand_begin() + 1,
getOperation()->operand_begin() + 1 + getTagMemRefRank()}; getOperation()->operand_begin() + 1 + getTagMemRefRank()};
} }
// Returns the rank (number of indices) of the tag memref. // Returns the rank (number of indices) of the tag memref.
unsigned getTagMemRefRank() { unsigned getTagMemRefRank() {
return getTagMemRef()->getType().cast<MemRefType>().getRank(); return getTagMemRef().getType().cast<MemRefType>().getRank();
} }
// Returns the number of elements transferred in the associated DMA operation. // Returns the number of elements transferred in the associated DMA operation.
Value getNumElements() { return getOperand(1 + getTagMemRefRank()); } Value getNumElements() { return getOperand(1 + getTagMemRefRank()); }
static ParseResult parse(OpAsmParser &parser, OperationState &result); static ParseResult parse(OpAsmParser &parser, OperationState &result);
void print(OpAsmPrinter &p); void print(OpAsmPrinter &p);
LogicalResult fold(ArrayRef<Attribute> cstOperands, LogicalResult fold(ArrayRef<Attribute> cstOperands,
Show All 18 Lines

mlir/include/mlir/Dialect/StandardOps/Ops.td

Show First 20 Lines • Show All 186 Lines • ▼ Show 20 Lines "ArrayRef<Value> operands, IntegerAttr alignment = IntegerAttr()", [{
result.types.push_back(memrefType); result.types.push_back(memrefType);
if (alignment) if (alignment)
result.addAttribute(getAlignmentAttrName(), alignment); result.addAttribute(getAlignmentAttrName(), alignment);
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static StringRef getAlignmentAttrName() { return "alignment"; } static StringRef getAlignmentAttrName() { return "alignment"; }
MemRefType getType() { return getResult()->getType().cast<MemRefType>(); } MemRefType getType() { return getResult().getType().cast<MemRefType>(); }
/// Returns the number of symbolic operands (the ones in square brackets), /// Returns the number of symbolic operands (the ones in square brackets),
/// which bind to the symbols of the memref's layout map. /// which bind to the symbols of the memref's layout map.
unsigned getNumSymbolicOperands() { unsigned getNumSymbolicOperands() {
return getNumOperands() - getType().getNumDynamicDims(); return getNumOperands() - getType().getNumDynamicDims();
} }
/// Returns the symbolic operands (the ones in square brackets), which bind /// Returns the symbolic operands (the ones in square brackets), which bind
▲ Show 20 Lines • Show All 116 Lines • ▼ Show 20 Linesdef CallIndirectOp : Std_Op<"call_indirect", [CallOpInterface]> {
let arguments = (ins FunctionType:$callee, Variadic<AnyType>:$operands); let arguments = (ins FunctionType:$callee, Variadic<AnyType>:$operands);
let results = (outs Variadic<AnyType>); let results = (outs Variadic<AnyType>);
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *, OperationState &result, Value callee," "Builder *, OperationState &result, Value callee,"
"ValueRange operands = {}", [{ "ValueRange operands = {}", [{
result.operands.push_back(callee); result.operands.push_back(callee);
result.addOperands(operands); result.addOperands(operands);
result.addTypes(callee->getType().cast<FunctionType>().getResults()); result.addTypes(callee.getType().cast<FunctionType>().getResults());
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Value getCallee() { return getOperand(0); } Value getCallee() { return getOperand(0); }
/// Get the argument operands to the called function. /// Get the argument operands to the called function.
operand_range getArgOperands() { operand_range getArgOperands() {
return {arg_operand_begin(), arg_operand_end()}; return {arg_operand_begin(), arg_operand_end()};
▲ Show 20 Lines • Show All 381 Lines • ▼ Show 20 Linesdef ExtractElementOp : Std_Op<"extract_element", [NoSideEffect]> {
let arguments = (ins AnyTypeOf<[AnyVector, AnyTensor]>:$aggregate, let arguments = (ins AnyTypeOf<[AnyVector, AnyTensor]>:$aggregate,
Variadic<Index>:$indices); Variadic<Index>:$indices);
let results = (outs AnyType); let results = (outs AnyType);
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value aggregate," "Builder *builder, OperationState &result, Value aggregate,"
"ValueRange indices = {}", [{ "ValueRange indices = {}", [{
auto resType = aggregate->getType().cast<ShapedType>() auto resType = aggregate.getType().cast<ShapedType>()
.getElementType(); .getElementType();
build(builder, result, resType, aggregate, indices); build(builder, result, resType, aggregate, indices);
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Value getAggregate() { return getOperand(0); } Value getAggregate() { return getOperand(0); }
operand_range getIndices() { operand_range getIndices() {
▲ Show 20 Lines • Show All 69 Lines • ▼ Show 20 Linesdef LoadOp : Std_Op<"load"> {
}]; }];
let arguments = (ins AnyMemRef:$memref, Variadic<Index>:$indices); let arguments = (ins AnyMemRef:$memref, Variadic<Index>:$indices);
let results = (outs AnyType); let results = (outs AnyType);
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *, OperationState &result, Value memref," "Builder *, OperationState &result, Value memref,"
"ValueRange indices = {}", [{ "ValueRange indices = {}", [{
auto memrefType = memref->getType().cast<MemRefType>(); auto memrefType = memref.getType().cast<MemRefType>();
result.addOperands(memref); result.addOperands(memref);
result.addOperands(indices); result.addOperands(indices);
result.types.push_back(memrefType.getElementType()); result.types.push_back(memrefType.getElementType());
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Value getMemRef() { return getOperand(0); } Value getMemRef() { return getOperand(0); }
void setMemRef(Value value) { setOperand(0, value); } void setMemRef(Value value) { setOperand(0, value); }
MemRefType getMemRefType() { MemRefType getMemRefType() {
return getMemRef()->getType().cast<MemRefType>(); return getMemRef().getType().cast<MemRefType>();
} }
operand_range getIndices() { return {operand_begin() + 1, operand_end()}; } operand_range getIndices() { return {operand_begin() + 1, operand_end()}; }
}]; }];
let hasFolder = 1; let hasFolder = 1;
} }
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Linesdef MemRefCastOp : CastOp<"memref_cast"> {
let results = (outs AnyRankedOrUnrankedMemRef); let results = (outs AnyRankedOrUnrankedMemRef);
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Return true if `a` and `b` are valid operand and result pairs for /// Return true if `a` and `b` are valid operand and result pairs for
/// the operation. /// the operation.
static bool areCastCompatible(Type a, Type b); static bool areCastCompatible(Type a, Type b);
/// The result of a memref_cast is always a memref. /// The result of a memref_cast is always a memref.
Type getType() { return getResult()->getType(); } Type getType() { return getResult().getType(); }
}]; }];
} }
def MulFOp : FloatArithmeticOp<"mulf"> { def MulFOp : FloatArithmeticOp<"mulf"> {
let summary = "floating point multiplication operation"; let summary = "floating point multiplication operation";
let hasFolder = 1; let hasFolder = 1;
} }
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines [{
result.addOperands(indices); result.addOperands(indices);
result.addAttribute("localityHint", hintAttr); result.addAttribute("localityHint", hintAttr);
result.addAttribute("isWrite", isWriteAttr); result.addAttribute("isWrite", isWriteAttr);
result.addAttribute("isDataCache", isDataCacheAttr); result.addAttribute("isDataCache", isDataCacheAttr);
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
MemRefType getMemRefType() { MemRefType getMemRefType() {
return memref()->getType().cast<MemRefType>(); return memref().getType().cast<MemRefType>();
} }
static StringRef getLocalityHintAttrName() { return "localityHint"; } static StringRef getLocalityHintAttrName() { return "localityHint"; }
static StringRef getIsWriteAttrName() { return "isWrite"; } static StringRef getIsWriteAttrName() { return "isWrite"; }
static StringRef getIsDataCacheAttrName() { return "isDataCache"; } static StringRef getIsDataCacheAttrName() { return "isDataCache"; }
}]; }];
let hasFolder = 1; let hasFolder = 1;
} }
▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Linesdef SelectOp : Std_Op<"select", [NoSideEffect, SameOperandsAndResultShape]> {
let arguments = (ins BoolLike:$condition, AnyType:$true_value, let arguments = (ins BoolLike:$condition, AnyType:$true_value,
AnyType:$false_value); AnyType:$false_value);
let results = (outs AnyType); let results = (outs AnyType);
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value condition," "Builder *builder, OperationState &result, Value condition,"
"Value trueValue, Value falseValue", [{ "Value trueValue, Value falseValue", [{
result.addOperands({condition, trueValue, falseValue}); result.addOperands({condition, trueValue, falseValue});
result.addTypes(trueValue->getType()); result.addTypes(trueValue.getType());
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Value getCondition() { return condition(); } Value getCondition() { return condition(); }
Value getTrueValue() { return true_value(); } Value getTrueValue() { return true_value(); }
Value getFalseValue() { return false_value(); } Value getFalseValue() { return false_value(); }
}]; }];
▲ Show 20 Lines • Show All 152 Lines • ▼ Show 20 Linesdef StoreOp : Std_Op<"store"> {
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Value getValueToStore() { return getOperand(0); } Value getValueToStore() { return getOperand(0); }
Value getMemRef() { return getOperand(1); } Value getMemRef() { return getOperand(1); }
void setMemRef(Value value) { setOperand(1, value); } void setMemRef(Value value) { setOperand(1, value); }
MemRefType getMemRefType() { MemRefType getMemRefType() {
return getMemRef()->getType().cast<MemRefType>(); return getMemRef().getType().cast<MemRefType>();
} }
operand_range getIndices() { operand_range getIndices() {
return {operand_begin() + 2, operand_end()}; return {operand_begin() + 2, operand_end()};
} }
}]; }];
let hasFolder = 1; let hasFolder = 1;
▲ Show 20 Lines • Show All 135 Lines • ▼ Show 20 Lines let builders = [
OpBuilder< OpBuilder<
"Builder *builder, OperationState &result, " "Builder *builder, OperationState &result, "
"Type resultType, Value source"> "Type resultType, Value source">
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Returns the type of the base memref operand. /// Returns the type of the base memref operand.
MemRefType getBaseMemRefType() { MemRefType getBaseMemRefType() {
return source()->getType().cast<MemRefType>(); return source().getType().cast<MemRefType>();
} }
/// The result of a subview is always a memref. /// The result of a subview is always a memref.
MemRefType getType() { return getResult()->getType().cast<MemRefType>(); } MemRefType getType() { return getResult().getType().cast<MemRefType>(); }
/// Returns as integer value the number of offset operands. /// Returns as integer value the number of offset operands.
int64_t getNumOffsets() { return llvm::size(offsets()); } int64_t getNumOffsets() { return llvm::size(offsets()); }
/// Returns as integer value the number of size operands. /// Returns as integer value the number of size operands.
int64_t getNumSizes() { return llvm::size(sizes()); } int64_t getNumSizes() { return llvm::size(sizes()); }
/// Returns as integer value the number of stride operands. /// Returns as integer value the number of stride operands.
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 Linesdef TensorCastOp : CastOp<"tensor_cast"> {
let results = (outs AnyTensor); let results = (outs AnyTensor);
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Return true if `a` and `b` are valid operand and result pairs for /// Return true if `a` and `b` are valid operand and result pairs for
/// the operation. /// the operation.
static bool areCastCompatible(Type a, Type b); static bool areCastCompatible(Type a, Type b);
/// The result of a tensor_cast is always a tensor. /// The result of a tensor_cast is always a tensor.
TensorType getType() { return getResult()->getType().cast<TensorType>(); } TensorType getType() { return getResult().getType().cast<TensorType>(); }
}]; }];
} }
def TensorLoadOp : Std_Op<"tensor_load", def TensorLoadOp : Std_Op<"tensor_load",
[SameOperandsAndResultShape, SameOperandsAndResultElementType]> { [SameOperandsAndResultShape, SameOperandsAndResultElementType]> {
let summary = "tensor load operation"; let summary = "tensor load operation";
let description = [{ let description = [{
The "tensor_load" operation creates a tensor from a memref, making an The "tensor_load" operation creates a tensor from a memref, making an
independent copy of the element data. The result value is a tensor whose independent copy of the element data. The result value is a tensor whose
shape and element type match the memref operand. shape and element type match the memref operand.
Produce a value of tensor<4x?xf32> type. Produce a value of tensor<4x?xf32> type.
%12 = tensor_load %10 : memref<4x?xf32, #layout, memspace0> %12 = tensor_load %10 : memref<4x?xf32, #layout, memspace0>
}]; }];
let arguments = (ins AnyMemRef); let arguments = (ins AnyMemRef);
let results = (outs AnyTensor); let results = (outs AnyTensor);
// TensorLoadOp is fully verified by traits. // TensorLoadOp is fully verified by traits.
let verifier = ?; let verifier = ?;
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value memref", [{ "Builder *builder, OperationState &result, Value memref", [{
auto memrefType = memref->getType().cast<MemRefType>(); auto memrefType = memref.getType().cast<MemRefType>();
auto resultType = RankedTensorType::get(memrefType.getShape(), auto resultType = RankedTensorType::get(memrefType.getShape(),
memrefType.getElementType()); memrefType.getElementType());
result.addOperands(memref); result.addOperands(memref);
result.addTypes(resultType); result.addTypes(resultType);
}]>]; }]>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// The result of a tensor_load is always a tensor. /// The result of a tensor_load is always a tensor.
TensorType getType() { return getResult()->getType().cast<TensorType>(); } TensorType getType() { return getResult().getType().cast<TensorType>(); }
}]; }];
} }
def TensorStoreOp : Std_Op<"tensor_store", def TensorStoreOp : Std_Op<"tensor_store",
[SameOperandsShape, SameOperandsElementType]> { [SameOperandsShape, SameOperandsElementType]> {
let summary = "tensor store operation"; let summary = "tensor store operation";
let description = [{ let description = [{
The "tensor_store" operation stores the contents of a tensor into a memref. The "tensor_store" operation stores the contents of a tensor into a memref.
▲ Show 20 Lines • Show All 81 Lines • ▼ Show 20 Linesdef ViewOp : Std_Op<"view", [NoSideEffect]> {
}]; }];
let arguments = (ins MemRefRankOf<[I8], [1]>:$source, let arguments = (ins MemRefRankOf<[I8], [1]>:$source,
Variadic<Index>:$operands); Variadic<Index>:$operands);
let results = (outs AnyMemRef); let results = (outs AnyMemRef);
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// The result of a view is always a memref. /// The result of a view is always a memref.
MemRefType getType() { return getResult()->getType().cast<MemRefType>(); } MemRefType getType() { return getResult().getType().cast<MemRefType>(); }
/// Returns the dynamic offset for this view operation if specified. /// Returns the dynamic offset for this view operation if specified.
/// Returns nullptr if no dynamic offset was specified. /// Returns nullptr if no dynamic offset was specified.
Value getDynamicOffset(); Value getDynamicOffset();
/// Returns the starting operand list position of the dynamic size operands. /// Returns the starting operand list position of the dynamic size operands.
unsigned getDynamicSizesOperandStart() { unsigned getDynamicSizesOperandStart() {
return getDynamicOffset() == nullptr ? 1 : 2; return getDynamicOffset() == nullptr ? 1 : 2;
▲ Show 20 Lines • Show All 50 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/VectorOps/VectorOps.td

Show First 20 Lines • Show All 117 Lines • ▼ Show 20 Lines Examples:
%5 = vector.contract #contraction_trait %0, %1, %2, %lhs_mask, %rhs_mask %5 = vector.contract #contraction_trait %0, %1, %2, %lhs_mask, %rhs_mask
: vector<7x8x16x15xf32>, vector<8x16x7x5xf32> into vector<8x15x8x5xf32> : vector<7x8x16x15xf32>, vector<8x16x7x5xf32> into vector<8x15x8x5xf32>
}]; }];
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value lhs, Value rhs, " "Builder *builder, OperationState &result, Value lhs, Value rhs, "
"Value acc, ArrayAttr indexingMaps, ArrayAttr iteratorTypes">]; "Value acc, ArrayAttr indexingMaps, ArrayAttr iteratorTypes">];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
VectorType getLhsType() { VectorType getLhsType() {
return lhs()->getType().cast<VectorType>(); return lhs().getType().cast<VectorType>();
} }
VectorType getRhsType() { VectorType getRhsType() {
return rhs()->getType().cast<VectorType>(); return rhs().getType().cast<VectorType>();
} }
VectorType getAccType() { VectorType getAccType() {
return acc()->getType().cast<VectorType>(); return acc().getType().cast<VectorType>();
} }
VectorType getLHSVectorMaskType() { VectorType getLHSVectorMaskType() {
if (llvm::size(masks()) != 2) return VectorType(); if (llvm::size(masks()) != 2) return VectorType();
return getOperand(3)->getType().cast<VectorType>(); return getOperand(3).getType().cast<VectorType>();
} }
VectorType getRHSVectorMaskType() { VectorType getRHSVectorMaskType() {
if (llvm::size(masks()) != 2) return VectorType(); if (llvm::size(masks()) != 2) return VectorType();
return getOperand(4)->getType().cast<VectorType>(); return getOperand(4).getType().cast<VectorType>();
} }
VectorType getResultType() { VectorType getResultType() {
return getResult()->getType().cast<VectorType>(); return getResult().getType().cast<VectorType>();
} }
ArrayRef<StringRef> getTraitAttrNames(); ArrayRef<StringRef> getTraitAttrNames();
SmallVector<AffineMap, 4> getIndexingMaps(); SmallVector<AffineMap, 4> getIndexingMaps();
static unsigned getAccOperandIndex() { return 2; } static unsigned getAccOperandIndex() { return 2; }
// Returns the bounds of each dimension in the iteration space spanned // Returns the bounds of each dimension in the iteration space spanned
// by the iterator types of this operation. // by the iterator types of this operation.
void getIterationBounds(SmallVectorImpl<int64_t> &iterationBounds); void getIterationBounds(SmallVectorImpl<int64_t> &iterationBounds);
▲ Show 20 Lines • Show All 41 Lines • ▼ Show 20 Lines let description = [{
Examples: Examples:
``` ```
%0 = constant 0.0 : f32 %0 = constant 0.0 : f32
%1 = vector.broadcast %0 : f32 to vector<16xf32> %1 = vector.broadcast %0 : f32 to vector<16xf32>
%2 = vector.broadcast %1 : vector<16xf32> to vector<4x16xf32> %2 = vector.broadcast %1 : vector<16xf32> to vector<4x16xf32>
``` ```
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Type getSourceType() { return source()->getType(); } Type getSourceType() { return source().getType(); }
VectorType getVectorType() { VectorType getVectorType() {
return vector()->getType().cast<VectorType>(); return vector().getType().cast<VectorType>();
} }
}]; }];
} }
def Vector_ShuffleOp : def Vector_ShuffleOp :
Vector_Op<"shuffle", [NoSideEffect, Vector_Op<"shuffle", [NoSideEffect,
PredOpTrait<"first operand v1 and result have same element type", PredOpTrait<"first operand v1 and result have same element type",
TCresVTEtIsSameAsOpBase<0, 0>>, TCresVTEtIsSameAsOpBase<0, 0>>,
Show All 31 Lines let description = [{
``` ```
}]; }];
let builders = [OpBuilder<"Builder *builder, OperationState &result," let builders = [OpBuilder<"Builder *builder, OperationState &result,"
"Value v1, Value v2, ArrayRef<int64_t>">]; "Value v1, Value v2, ArrayRef<int64_t>">];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static StringRef getMaskAttrName() { return "mask"; } static StringRef getMaskAttrName() { return "mask"; }
VectorType getV1VectorType() { VectorType getV1VectorType() {
return v1()->getType().cast<VectorType>(); return v1().getType().cast<VectorType>();
} }
VectorType getV2VectorType() { VectorType getV2VectorType() {
return v2()->getType().cast<VectorType>(); return v2().getType().cast<VectorType>();
} }
VectorType getVectorType() { VectorType getVectorType() {
return vector()->getType().cast<VectorType>(); return vector().getType().cast<VectorType>();
} }
}]; }];
} }
def Vector_ExtractElementOp : def Vector_ExtractElementOp :
Vector_Op<"extractelement", [NoSideEffect, Vector_Op<"extractelement", [NoSideEffect,
PredOpTrait<"operand and result have same element type", PredOpTrait<"operand and result have same element type",
TCresVTEtIsSameAsOpBase<0, 0>>]>, TCresVTEtIsSameAsOpBase<0, 0>>]>,
Show All 10 Lines let description = [{
Example: Example:
``` ```
%c = constant 15 : i32 %c = constant 15 : i32
%1 = vector.extractelement %0[%c : i32]: vector<16xf32> %1 = vector.extractelement %0[%c : i32]: vector<16xf32>
``` ```
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
VectorType getVectorType() { VectorType getVectorType() {
return vector()->getType().cast<VectorType>(); return vector().getType().cast<VectorType>();
} }
}]; }];
} }
def Vector_ExtractOp : def Vector_ExtractOp :
Vector_Op<"extract", [NoSideEffect, Vector_Op<"extract", [NoSideEffect,
PredOpTrait<"operand and result have same element type", PredOpTrait<"operand and result have same element type",
TCresVTEtIsSameAsOpBase<0, 0>>]>, TCresVTEtIsSameAsOpBase<0, 0>>]>,
Show All 11 Lines let description = [{
``` ```
}]; }];
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value source," "Builder *builder, OperationState &result, Value source,"
"ArrayRef<int64_t>">]; "ArrayRef<int64_t>">];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static StringRef getPositionAttrName() { return "position"; } static StringRef getPositionAttrName() { return "position"; }
VectorType getVectorType() { VectorType getVectorType() {
return vector()->getType().cast<VectorType>(); return vector().getType().cast<VectorType>();
} }
}]; }];
} }
def Vector_ExtractSlicesOp : def Vector_ExtractSlicesOp :
Vector_Op<"extract_slices", [NoSideEffect]>, Vector_Op<"extract_slices", [NoSideEffect]>,
Arguments<(ins AnyVector:$vector, I64ArrayAttr:$sizes, Arguments<(ins AnyVector:$vector, I64ArrayAttr:$sizes,
I64ArrayAttr:$strides)>, I64ArrayAttr:$strides)>,
Show All 28 Lines let description = [{
``` ```
}]; }];
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, TupleType tupleType, " # "Builder *builder, OperationState &result, TupleType tupleType, " #
"Value vector, ArrayRef<int64_t> sizes, " # "Value vector, ArrayRef<int64_t> sizes, " #
"ArrayRef<int64_t> strides">]; "ArrayRef<int64_t> strides">];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
VectorType getSourceVectorType() { VectorType getSourceVectorType() {
return vector()->getType().cast<VectorType>(); return vector().getType().cast<VectorType>();
} }
TupleType getResultTupleType() { TupleType getResultTupleType() {
return getResult()->getType().cast<TupleType>(); return getResult().getType().cast<TupleType>();
} }
void getSizes(SmallVectorImpl<int64_t> &results); void getSizes(SmallVectorImpl<int64_t> &results);
void getStrides(SmallVectorImpl<int64_t> &results); void getStrides(SmallVectorImpl<int64_t> &results);
static StringRef getSizesAttrName() { return "sizes"; } static StringRef getSizesAttrName() { return "sizes"; }
static StringRef getStridesAttrName() { return "strides"; } static StringRef getStridesAttrName() { return "strides"; }
}]; }];
} }
Show All 17 Lines let description = [{
Example: Example:
``` ```
%c = constant 15 : i32 %c = constant 15 : i32
%f = constant 0.0f : f32 %f = constant 0.0f : f32
%1 = vector.insertelement %f, %0[%c : i32]: vector<16xf32> %1 = vector.insertelement %f, %0[%c : i32]: vector<16xf32>
``` ```
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Type getSourceType() { return source()->getType(); } Type getSourceType() { return source().getType(); }
VectorType getDestVectorType() { VectorType getDestVectorType() {
return dest()->getType().cast<VectorType>(); return dest().getType().cast<VectorType>();
} }
}]; }];
} }
def Vector_InsertOp : def Vector_InsertOp :
Vector_Op<"insert", [NoSideEffect, Vector_Op<"insert", [NoSideEffect,
PredOpTrait<"source operand and result have same element type", PredOpTrait<"source operand and result have same element type",
TCresVTEtIsSameAsOpBase<0, 0>>, TCresVTEtIsSameAsOpBase<0, 0>>,
Show All 15 Lines ```
f32 into vector<4x8x16xf32> f32 into vector<4x8x16xf32>
``` ```
}]; }];
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value source, " # "Builder *builder, OperationState &result, Value source, " #
"Value dest, ArrayRef<int64_t>">]; "Value dest, ArrayRef<int64_t>">];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static StringRef getPositionAttrName() { return "position"; } static StringRef getPositionAttrName() { return "position"; }
Type getSourceType() { return source()->getType(); } Type getSourceType() { return source().getType(); }
VectorType getDestVectorType() { VectorType getDestVectorType() {
return dest()->getType().cast<VectorType>(); return dest().getType().cast<VectorType>();
} }
}]; }];
} }
def Vector_InsertSlicesOp : def Vector_InsertSlicesOp :
Vector_Op<"insert_slices", [NoSideEffect]>, Vector_Op<"insert_slices", [NoSideEffect]>,
Arguments<(ins TupleOf<[AnyVector]>:$vectors, I64ArrayAttr:$sizes, Arguments<(ins TupleOf<[AnyVector]>:$vectors, I64ArrayAttr:$sizes,
I64ArrayAttr:$strides)>, I64ArrayAttr:$strides)>,
Show All 28 Lines ```
%4 = vector.insert_slices %3, [2, 2], [1, 1] %4 = vector.insert_slices %3, [2, 2], [1, 1]
: tuple<vector<2x2xf32>, vector<2x1xf32>, : tuple<vector<2x2xf32>, vector<2x1xf32>,
vector<2x2xf32>, vector<2x1xf32>> into vector<4x3xf32> vector<2x2xf32>, vector<2x1xf32>> into vector<4x3xf32>
``` ```
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
TupleType getSourceTupleType() { TupleType getSourceTupleType() {
return vectors()->getType().cast<TupleType>(); return vectors().getType().cast<TupleType>();
} }
VectorType getResultVectorType() { VectorType getResultVectorType() {
return getResult()->getType().cast<VectorType>(); return getResult().getType().cast<VectorType>();
} }
void getSizes(SmallVectorImpl<int64_t> &results); void getSizes(SmallVectorImpl<int64_t> &results);
void getStrides(SmallVectorImpl<int64_t> &results); void getStrides(SmallVectorImpl<int64_t> &results);
static StringRef getSizesAttrName() { return "sizes"; } static StringRef getSizesAttrName() { return "sizes"; }
static StringRef getStridesAttrName() { return "strides"; } static StringRef getStridesAttrName() { return "strides"; }
}]; }];
} }
Show All 28 Linesdef Vector_InsertStridedSliceOp :
}]; }];
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value source, Value dest, " # "Builder *builder, OperationState &result, Value source, Value dest, " #
"ArrayRef<int64_t> offsets, ArrayRef<int64_t> strides">]; "ArrayRef<int64_t> offsets, ArrayRef<int64_t> strides">];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static StringRef getOffsetsAttrName() { return "offsets"; } static StringRef getOffsetsAttrName() { return "offsets"; }
static StringRef getStridesAttrName() { return "strides"; } static StringRef getStridesAttrName() { return "strides"; }
VectorType getSourceVectorType() { VectorType getSourceVectorType() {
return source()->getType().cast<VectorType>(); return source().getType().cast<VectorType>();
} }
VectorType getDestVectorType() { VectorType getDestVectorType() {
return dest()->getType().cast<VectorType>(); return dest().getType().cast<VectorType>();
} }
}]; }];
} }
def Vector_OuterProductOp : def Vector_OuterProductOp :
Vector_Op<"outerproduct", [NoSideEffect, SameOperandsAndResultElementType]>, Vector_Op<"outerproduct", [NoSideEffect, SameOperandsAndResultElementType]>,
Arguments<(ins AnyVector:$lhs, AnyVector:$rhs, Variadic<AnyVector>:$acc)>, Arguments<(ins AnyVector:$lhs, AnyVector:$rhs, Variadic<AnyVector>:$acc)>,
Results<(outs AnyVector)> { Results<(outs AnyVector)> {
Show All 12 Lines Examples
return %2: vector<4x8xf32> return %2: vector<4x8xf32>
%3 = vector.outerproduct %0, %1, %2: %3 = vector.outerproduct %0, %1, %2:
vector<4xf32>, vector<8xf32>, vector<4x8xf32> vector<4xf32>, vector<8xf32>, vector<4x8xf32>
return %3: vector<4x8xf32> return %3: vector<4x8xf32>
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
VectorType getOperandVectorTypeLHS() { VectorType getOperandVectorTypeLHS() {
return lhs()->getType().cast<VectorType>(); return lhs().getType().cast<VectorType>();
} }
VectorType getOperandVectorTypeRHS() { VectorType getOperandVectorTypeRHS() {
return rhs()->getType().cast<VectorType>(); return rhs().getType().cast<VectorType>();
} }
VectorType getOperandVectorTypeACC() { VectorType getOperandVectorTypeACC() {
return (llvm::size(acc()) == 0) ? VectorType() : return (llvm::size(acc()) == 0) ? VectorType() :
(*acc().begin())->getType().cast<VectorType>(); (*acc().begin()).getType().cast<VectorType>();
} }
VectorType getVectorType() { VectorType getVectorType() {
return getResult()->getType().cast<VectorType>(); return getResult().getType().cast<VectorType>();
} }
}]; }];
} }
// TODO(andydavis) Add transformation which decomposes ReshapeOp into an // TODO(andydavis) Add transformation which decomposes ReshapeOp into an
// optimized sequence of vector rotate/shuffle/select operations. // optimized sequence of vector rotate/shuffle/select operations.
def Vector_ReshapeOp : def Vector_ReshapeOp :
Vector_Op<"reshape", [AttrSizedOperandSegments, NoSideEffect]>, Vector_Op<"reshape", [AttrSizedOperandSegments, NoSideEffect]>,
▲ Show 20 Lines • Show All 83 Lines • ▼ Show 20 Lines Example
[i, -, -, -]], [i, -, -, -]],
[[j, k, l, m], [[j, k, l, m],
[n, o, p, q], [n, o, p, q],
[r, -, -, -]]] [r, -, -, -]]]
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
VectorType getInputVectorType() { VectorType getInputVectorType() {
return vector()->getType().cast<VectorType>(); return vector().getType().cast<VectorType>();
} }
VectorType getOutputVectorType() { VectorType getOutputVectorType() {
return getResult()->getType().cast<VectorType>(); return getResult().getType().cast<VectorType>();
} }
/// Returns as integer value the number of input shape operands. /// Returns as integer value the number of input shape operands.
int64_t getNumInputShapeSizes() { return input_shape().size(); } int64_t getNumInputShapeSizes() { return input_shape().size(); }
/// Returns as integer value the number of output shape operands. /// Returns as integer value the number of output shape operands.
int64_t getNumOutputShapeSizes() { return output_shape().size(); } int64_t getNumOutputShapeSizes() { return output_shape().size(); }
▲ Show 20 Lines • Show All 41 Lines • ▼ Show 20 Linesdef Vector_StridedSliceOp :
let builders = [OpBuilder< let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value source, " # "Builder *builder, OperationState &result, Value source, " #
"ArrayRef<int64_t> offsets, ArrayRef<int64_t> sizes, " # "ArrayRef<int64_t> offsets, ArrayRef<int64_t> sizes, " #
"ArrayRef<int64_t> strides">]; "ArrayRef<int64_t> strides">];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static StringRef getOffsetsAttrName() { return "offsets"; } static StringRef getOffsetsAttrName() { return "offsets"; }
static StringRef getSizesAttrName() { return "sizes"; } static StringRef getSizesAttrName() { return "sizes"; }
static StringRef getStridesAttrName() { return "strides"; } static StringRef getStridesAttrName() { return "strides"; }
VectorType getVectorType(){ return vector()->getType().cast<VectorType>(); } VectorType getVectorType(){ return vector().getType().cast<VectorType>(); }
void getOffsets(SmallVectorImpl<int64_t> &results); void getOffsets(SmallVectorImpl<int64_t> &results);
}]; }];
let hasCanonicalizer = 1; let hasCanonicalizer = 1;
} }
def Vector_TransferReadOp : def Vector_TransferReadOp :
Vector_Op<"transfer_read">, Vector_Op<"transfer_read">,
Arguments<(ins AnyMemRef:$memref, Variadic<Index>:$indices, Arguments<(ins AnyMemRef:$memref, Variadic<Index>:$indices,
▲ Show 20 Lines • Show All 122 Lines • ▼ Show 20 Lines let description = [{
%4 = vector.transfer_read %arg1[%c3, %c3], %vf0 %4 = vector.transfer_read %arg1[%c3, %c3], %vf0
{permutation_map = (d0, d1)->(d0, d1)} {permutation_map = (d0, d1)->(d0, d1)}
: memref<?x?xvector<4x3xf32>>, vector<1x1x4x3xf32> : memref<?x?xvector<4x3xf32>>, vector<1x1x4x3xf32>
``` ```
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
MemRefType getMemRefType() { MemRefType getMemRefType() {
return memref()->getType().cast<MemRefType>(); return memref().getType().cast<MemRefType>();
} }
VectorType getVectorType() { VectorType getVectorType() {
return vector()->getType().cast<VectorType>(); return vector().getType().cast<VectorType>();
} }
}]; }];
} }
def Vector_TransferWriteOp : def Vector_TransferWriteOp :
Vector_Op<"transfer_write">, Vector_Op<"transfer_write">,
Arguments<(ins AnyVector:$vector, AnyMemRef:$memref, Arguments<(ins AnyVector:$vector, AnyMemRef:$memref,
Variadic<Index>:$indices, Variadic<Index>:$indices,
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines let description = [{
vector.transfer_write %4, %arg1[%c3, %c3] vector.transfer_write %4, %arg1[%c3, %c3]
{permutation_map = (d0, d1)->(d0, d1)} {permutation_map = (d0, d1)->(d0, d1)}
: vector<1x1x4x3xf32>, memref<?x?xvector<4x3xf32>> : vector<1x1x4x3xf32>, memref<?x?xvector<4x3xf32>>
``` ```
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
VectorType getVectorType() { VectorType getVectorType() {
return vector()->getType().cast<VectorType>(); return vector().getType().cast<VectorType>();
} }
MemRefType getMemRefType() { MemRefType getMemRefType() {
return memref()->getType().cast<MemRefType>(); return memref().getType().cast<MemRefType>();
} }
}]; }];
} }
def Vector_TypeCastOp : def Vector_TypeCastOp :
Vector_Op<"type_cast", [NoSideEffect]>, Vector_Op<"type_cast", [NoSideEffect]>,
Arguments<(ins StaticShapeMemRefOf<[AnyType]>:$memref)>, Arguments<(ins StaticShapeMemRefOf<[AnyType]>:$memref)>,
Results<(outs AnyMemRef)> { Results<(outs AnyMemRef)> {
Show All 23 Lines let builders = [OpBuilder<
"Builder *builder, OperationState &result, Value source">]; "Builder *builder, OperationState &result, Value source">];
let parser = [{ let parser = [{
return impl::parseCastOp(parser, result); return impl::parseCastOp(parser, result);
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
MemRefType getMemRefType() { MemRefType getMemRefType() {
return memref()->getType().cast<MemRefType>(); return memref().getType().cast<MemRefType>();
} }
MemRefType getResultMemRefType() { MemRefType getResultMemRefType() {
return getResult()->getType().cast<MemRefType>(); return getResult().getType().cast<MemRefType>();
} }
}]; }];
} }
def Vector_ConstantMaskOp : def Vector_ConstantMaskOp :
Vector_Op<"constant_mask", [NoSideEffect]>, Vector_Op<"constant_mask", [NoSideEffect]>,
Arguments<(ins I64ArrayAttr:$mask_dim_sizes)>, Arguments<(ins I64ArrayAttr:$mask_dim_sizes)>,
Results<(outs VectorOf<[I1]>)> { Results<(outs VectorOf<[I1]>)> {
▲ Show 20 Lines • Show All 82 Lines • ▼ Show 20 Lines ```
%4 = vector.tuple %0, %1, %2, %3 %4 = vector.tuple %0, %1, %2, %3
: vector<2x2xf32>, vector<2x1xf32>, vector<2x2xf32>, vector<2x1xf32> : vector<2x2xf32>, vector<2x1xf32>, vector<2x2xf32>, vector<2x1xf32>
``` ```
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
TupleType getResultTupleType() { TupleType getResultTupleType() {
return getResult()->getType().cast<TupleType>(); return getResult().getType().cast<TupleType>();
} }
}]; }];
} }
def Vector_TupleGetOp : def Vector_TupleGetOp :
Vector_Op<"tuple_get", [NoSideEffect]>, Vector_Op<"tuple_get", [NoSideEffect]>,
Arguments<(ins TupleOf<[AnyVector]>:$vectors, APIntAttr:$index)>, Arguments<(ins TupleOf<[AnyVector]>:$vectors, APIntAttr:$index)>,
Results<(outs AnyVector)> { Results<(outs AnyVector)> {
Show All 13 Lines ```
%5 = vector.tuple_get %4, 1 %5 = vector.tuple_get %4, 1
: tuple<vector<2x2xf32>, vector<2x1xf32>, : tuple<vector<2x2xf32>, vector<2x1xf32>,
vector<2x2xf32>, vector<2x1xf32>> vector<2x2xf32>, vector<2x1xf32>>
``` ```
}]; }];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
VectorType getResultVectorType() { VectorType getResultVectorType() {
return getResult()->getType().cast<VectorType>(); return getResult().getType().cast<VectorType>();
} }
int64_t getIndex() { int64_t getIndex() {
return getAttrOfType<IntegerAttr>("index").getValue().getSExtValue(); return getAttrOfType<IntegerAttr>("index").getValue().getSExtValue();
} }
static StringRef getIndexAttrName() { return "index"; } static StringRef getIndexAttrName() { return "index"; }
}]; }];
} }
Show All 19 Lines ```
which only needs to provide a few printing methods (single which only needs to provide a few printing methods (single
value for all data types, opening/closing bracket, comma, value for all data types, opening/closing bracket, comma,
newline). newline).
``` ```
}]; }];
let verifier = ?; let verifier = ?;
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Type getPrintType() { Type getPrintType() {
return source()->getType(); return source().getType();
} }
}]; }];
} }
#endif // VECTOR_OPS #endif // VECTOR_OPS

mlir/include/mlir/Dialect/VectorOps/VectorTransformPatterns.td

Show All 10 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
#ifndef VECTOR_TRANSFORM_PATTERNS #ifndef VECTOR_TRANSFORM_PATTERNS
#define VECTOR_TRANSFORM_PATTERNS #define VECTOR_TRANSFORM_PATTERNS
include "mlir/IR/OpBase.td" include "mlir/IR/OpBase.td"
class HasShape<list<int> shape> : class HasShape<list<int> shape> :
CPred<"$0->getType().cast<ShapedType>().hasStaticShape({" # CPred<"$0.getType().cast<ShapedType>().hasStaticShape({" #
StrJoinInt<shape>.result # "})">; StrJoinInt<shape>.result # "})">;
class UnrollVectorOp<list<int> factors> : NativeCodeCall< class UnrollVectorOp<list<int> factors> : NativeCodeCall<
"unrollSingleResultOpMatchingType($_builder, $0->getDefiningOp(), " # "unrollSingleResultOpMatchingType($_builder, $0.getDefiningOp(), " #
"{" # StrJoinInt<factors>.result # "})">; "{" # StrJoinInt<factors>.result # "})">;
#endif // VECTOR_TRANSFORM_PATTERNS #endif // VECTOR_TRANSFORM_PATTERNS

mlir/include/mlir/EDSC/Builders.h

Show First 20 Lines • Show All 297 Lines • ▼ Show 20 Linespublic:
/// placeholder for an mlir::Value that will be constructed and captured at /// placeholder for an mlir::Value that will be constructed and captured at
/// some later point in the program. /// some later point in the program.
explicit ValueHandle(Type t) : t(t), v(nullptr) {} explicit ValueHandle(Type t) : t(t), v(nullptr) {}
/// A ValueHandle that is constructed from an mlir::Value is an "eager" /// A ValueHandle that is constructed from an mlir::Value is an "eager"
/// Value. An eager Value represents both the declaration and the definition /// Value. An eager Value represents both the declaration and the definition
/// (in the PL sense) of a placeholder for an mlir::Value that has already /// (in the PL sense) of a placeholder for an mlir::Value that has already
/// been constructed in the past and that is captured "now" in the program. /// been constructed in the past and that is captured "now" in the program.
explicit ValueHandle(Value v) : t(v->getType()), v(v) {} explicit ValueHandle(Value v) : t(v.getType()), v(v) {}
/// Builds a ConstantIndexOp of value `cst`. The constant is created at the /// Builds a ConstantIndexOp of value `cst`. The constant is created at the
/// current insertion point. /// current insertion point.
/// This implicit constructor is provided to each build an eager Value for a /// This implicit constructor is provided to each build an eager Value for a
/// constant at the current insertion point in the IR. An implicit constructor /// constant at the current insertion point in the IR. An implicit constructor
/// allows idiomatic expressions mixing ValueHandle and literals. /// allows idiomatic expressions mixing ValueHandle and literals.
ValueHandle(index_t cst); ValueHandle(index_t cst);
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Lines Value getValue() const {
return v; return v;
} }
bool hasType() const { return t != Type(); } bool hasType() const { return t != Type(); }
Type getType() const { return t; } Type getType() const { return t; }
Operation *getOperation() const { Operation *getOperation() const {
if (!v) if (!v)
return nullptr; return nullptr;
return v->getDefiningOp(); return v.getDefiningOp();
} }
protected: protected:
ValueHandle() : t(), v(nullptr) {} ValueHandle() : t(), v(nullptr) {}
Type t; Type t;
Value v; Value v;
}; };
▲ Show 20 Lines • Show All 162 Lines • Show Last 20 Lines

mlir/include/mlir/EDSC/Intrinsics.h

Show All 30 Lines
/// ```c++ /// ```c++
/// IndexHandle i, j, k; /// IndexHandle i, j, k;
/// ``` /// ```
struct IndexHandle : public ValueHandle { struct IndexHandle : public ValueHandle {
explicit IndexHandle() explicit IndexHandle()
: ValueHandle(ScopedContext::getBuilder().getIndexType()) {} : ValueHandle(ScopedContext::getBuilder().getIndexType()) {}
explicit IndexHandle(index_t v) : ValueHandle(v) {} explicit IndexHandle(index_t v) : ValueHandle(v) {}
explicit IndexHandle(Value v) : ValueHandle(v) { explicit IndexHandle(Value v) : ValueHandle(v) {
assert(v->getType() == ScopedContext::getBuilder().getIndexType() && assert(v.getType() == ScopedContext::getBuilder().getIndexType() &&
"Expected index type"); "Expected index type");
} }
explicit IndexHandle(ValueHandle v) : ValueHandle(v) { explicit IndexHandle(ValueHandle v) : ValueHandle(v) {
assert(v.getType() == ScopedContext::getBuilder().getIndexType() && assert(v.getType() == ScopedContext::getBuilder().getIndexType() &&
"Expected index type"); "Expected index type");
} }
IndexHandle &operator=(const ValueHandle &v) { IndexHandle &operator=(const ValueHandle &v) {
assert(v.getType() == ScopedContext::getBuilder().getIndexType() && assert(v.getType() == ScopedContext::getBuilder().getIndexType() &&
▲ Show 20 Lines • Show All 229 Lines • Show Last 20 Lines

mlir/include/mlir/IR/Matchers.h

Show First 20 Lines • Show All 80 Lines • ▼ Show 20 Linesstruct constant_int_op_binder {
/// Creates a matcher instance that binds the value to bv if match succeeds. /// Creates a matcher instance that binds the value to bv if match succeeds.
constant_int_op_binder(IntegerAttr::ValueType *bv) : bind_value(bv) {} constant_int_op_binder(IntegerAttr::ValueType *bv) : bind_value(bv) {}
bool match(Operation *op) { bool match(Operation *op) {
Attribute attr; Attribute attr;
if (!constant_op_binder<Attribute>(&attr).match(op)) if (!constant_op_binder<Attribute>(&attr).match(op))
return false; return false;
auto type = op->getResult(0)->getType(); auto type = op->getResult(0).getType();
if (type.isIntOrIndex()) { if (type.isIntOrIndex()) {
return attr_value_binder<IntegerAttr>(bind_value).match(attr); return attr_value_binder<IntegerAttr>(bind_value).match(attr);
} }
if (type.isa<VectorType>() || type.isa<RankedTensorType>()) { if (type.isa<VectorType>() || type.isa<RankedTensorType>()) {
if (auto splatAttr = attr.dyn_cast<SplatElementsAttr>()) { if (auto splatAttr = attr.dyn_cast<SplatElementsAttr>()) {
return attr_value_binder<IntegerAttr>(bind_value) return attr_value_binder<IntegerAttr>(bind_value)
.match(splatAttr.getSplatValue()); .match(splatAttr.getSplatValue());
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Lines
} }
/// Statically switch to an Operation matcher. /// Statically switch to an Operation matcher.
template <typename MatcherClass> template <typename MatcherClass>
typename std::enable_if_t<is_detected<detail::has_operation_or_value_matcher_t, typename std::enable_if_t<is_detected<detail::has_operation_or_value_matcher_t,
MatcherClass, Operation *>::value, MatcherClass, Operation *>::value,
bool> bool>
matchOperandOrValueAtIndex(Operation *op, unsigned idx, MatcherClass &matcher) { matchOperandOrValueAtIndex(Operation *op, unsigned idx, MatcherClass &matcher) {
if (auto defOp = op->getOperand(idx)->getDefiningOp()) if (auto defOp = op->getOperand(idx).getDefiningOp())
return matcher.match(defOp); return matcher.match(defOp);
return false; return false;
} }
/// Terminal matcher, always returns true. /// Terminal matcher, always returns true.
struct AnyValueMatcher { struct AnyValueMatcher {
bool match(Value op) const { return true; } bool match(Value op) const { return true; }
}; };
▲ Show 20 Lines • Show All 66 Lines • ▼ Show 20 Lines
inline detail::constant_int_not_value_matcher<0> m_NonZero() { inline detail::constant_int_not_value_matcher<0> m_NonZero() {
return detail::constant_int_not_value_matcher<0>(); return detail::constant_int_not_value_matcher<0>();
} }
/// Entry point for matching a pattern over a Value. /// Entry point for matching a pattern over a Value.
template <typename Pattern> template <typename Pattern>
inline bool matchPattern(Value value, const Pattern &pattern) { inline bool matchPattern(Value value, const Pattern &pattern) {
// TODO: handle other cases // TODO: handle other cases
if (auto *op = value->getDefiningOp()) if (auto *op = value.getDefiningOp())
return const_cast<Pattern &>(pattern).match(op); return const_cast<Pattern &>(pattern).match(op);
return false; return false;
} }
/// Entry point for matching a pattern over an Operation. /// Entry point for matching a pattern over an Operation.
template <typename Pattern> template <typename Pattern>
inline bool matchPattern(Operation *op, const Pattern &pattern) { inline bool matchPattern(Operation *op, const Pattern &pattern) {
return const_cast<Pattern &>(pattern).match(op); return const_cast<Pattern &>(pattern).match(op);
Show All 22 Lines

mlir/include/mlir/IR/OpBase.td

Show First 20 Lines • Show All 79 Lines • ▼ Show 20 Lines
// * `$_self` will be replaced with the entity this predicate is attached to. // * `$_self` will be replaced with the entity this predicate is attached to.
// E.g., `BoolAttr` is an attribute constraint that wraps a // E.g., `BoolAttr` is an attribute constraint that wraps a
// `CPred<"$_self.isa<BoolAttr>()">` (see the following sections for details). // `CPred<"$_self.isa<BoolAttr>()">` (see the following sections for details).
// Then for `F32:$attr`,`$_self` will be replaced by `$attr`. // Then for `F32:$attr`,`$_self` will be replaced by `$attr`.
// For type constraints, it's a little bit special since we want the // For type constraints, it's a little bit special since we want the
// constraints on each type definition reads naturally and we want to attach // constraints on each type definition reads naturally and we want to attach
// type constraints directly to an operand/result, $_self will be replaced // type constraints directly to an operand/result, $_self will be replaced
// by the operand/result's type. E.g., for `F32` in `F32:$operand`, its // by the operand/result's type. E.g., for `F32` in `F32:$operand`, its
// `$_self` will be expanded as `getOperand(...)->getType()`. // `$_self` will be expanded as `getOperand(...).getType()`.
class CPred<code pred> : Pred { class CPred<code pred> : Pred {
code predExpr = "(" # pred # ")"; code predExpr = "(" # pred # ")";
} }
// Kinds of predicate combiners. These must closely match the predicates // Kinds of predicate combiners. These must closely match the predicates
// implemented by the C++ backend (tblgen::PredCombinerKind). // implemented by the C++ backend (tblgen::PredCombinerKind).
class PredCombinerKind; class PredCombinerKind;
def PredCombinerAnd : PredCombinerKind; def PredCombinerAnd : PredCombinerKind;
▲ Show 20 Lines • Show All 1,498 Lines • ▼ Show 20 Linesclass Results<dag rets> {
dag results = rets; dag results = rets;
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Common value constraints // Common value constraints
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
def HasNoUseOf: Constraint< def HasNoUseOf: Constraint<
CPred<"$_self->use_begin() == $_self->use_end()">, "has no use">; CPred<"$_self.use_empty()">, "has no use">;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Common op type constraints // Common op type constraints
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// These traits are for verifying properties of an op that require knowledge of // These traits are for verifying properties of an op that require knowledge of
// multiple arguments or results. For verifying properties of a single argument // multiple arguments or results. For verifying properties of a single argument
// or result, prefer operand type constraints. // or result, prefer operand type constraints.
▲ Show 20 Lines • Show All 44 Lines • ▼ Show 20 Linesclass AllShapesMatch<list<string> names> :
AllMatchSameOperatorTrait<names, Shape<"_self">.result, "shape">; AllMatchSameOperatorTrait<names, Shape<"_self">.result, "shape">;
class AllTypesMatch<list<string> names> : class AllTypesMatch<list<string> names> :
AllMatchSameOperatorTrait<names, "$_self.getType()", "type">; AllMatchSameOperatorTrait<names, "$_self.getType()", "type">;
// Type Constraint operand `idx`'s Element type is `type`. // Type Constraint operand `idx`'s Element type is `type`.
class TCopVTEtIs<int idx, Type type> : And<[ class TCopVTEtIs<int idx, Type type> : And<[
CPred<"$_op.getNumOperands() > " # idx>, CPred<"$_op.getNumOperands() > " # idx>,
SubstLeaves<"$_self", "$_op.getOperand(" # idx # ")->getType()", SubstLeaves<"$_self", "$_op.getOperand(" # idx # ").getType()",
IsShapedTypePred>, IsShapedTypePred>,
SubstLeaves<"$_self", "getElementTypeOrSelf($_op.getOperand(" # idx # "))", SubstLeaves<"$_self", "getElementTypeOrSelf($_op.getOperand(" # idx # "))",
type.predicate>]>; type.predicate>]>;
// Predicate to verify that a named argument or result's element type matches a // Predicate to verify that a named argument or result's element type matches a
// given type. // given type.
class TypeIsPred<string name, Type type> : class TypeIsPred<string name, Type type> :
SubstLeaves<"$_self", "$" # name # ".getType()", type.predicate>; SubstLeaves<"$_self", "$" # name # ".getType()", type.predicate>;
Show All 10 Linesclass ElementTypeIs<string name, Type type> : PredOpTrait<
"'" # name # "' is " # type.description, ElementTypeIsPred<name, type>>; "'" # name # "' is " # type.description, ElementTypeIsPred<name, type>>;
// Predicate to verify that the i'th operand and the j'th operand have the same // Predicate to verify that the i'th operand and the j'th operand have the same
// elemental type. // elemental type.
// Type Constraint operand `i`'s Element type is Same As operand `j`'s Element // Type Constraint operand `i`'s Element type is Same As operand `j`'s Element
// type. // type.
class TCopVTEtIsSameAs<int i, int j> : And<[ class TCopVTEtIsSameAs<int i, int j> : And<[
CPred<"$_op.getNumOperands() > std::max(" # i # "u," # j # "u)">, CPred<"$_op.getNumOperands() > std::max(" # i # "u," # j # "u)">,
SubstLeaves<"$_self", "$_op.getOperand(" # i # ")->getType()", SubstLeaves<"$_self", "$_op.getOperand(" # i # ").getType()",
IsShapedTypePred>, IsShapedTypePred>,
SubstLeaves<"$_self", "$_op.getOperand(" # j # ")->getType()", SubstLeaves<"$_self", "$_op.getOperand(" # j # ").getType()",
IsShapedTypePred>, IsShapedTypePred>,
CPred<"mlir::getElementTypeOrSelf($_op.getOperand(" # i # ")) == " CPred<"mlir::getElementTypeOrSelf($_op.getOperand(" # i # ")) == "
"mlir::getElementTypeOrSelf($_op.getOperand(" # j # "))">]>; "mlir::getElementTypeOrSelf($_op.getOperand(" # j # "))">]>;
// Predicate to verify that the i'th result and the j'th operand exist and has // Predicate to verify that the i'th result and the j'th operand exist and has
// shaped types. // shaped types.
class TCOpResIsShapedTypePred<int i, int j> : And<[ class TCOpResIsShapedTypePred<int i, int j> : And<[
CPred<"$_op.getNumResults() > " # i>, CPred<"$_op.getNumResults() > " # i>,
CPred<"$_op.getNumOperands() > " # j>, CPred<"$_op.getNumOperands() > " # j>,
SubstLeaves<"$_self", "$_op.getResult(" # i # ")->getType()", SubstLeaves<"$_self", "$_op.getResult(" # i # ").getType()",
IsShapedTypePred>, IsShapedTypePred>,
SubstLeaves<"$_self", "$_op.getOperand(" # j # ")->getType()", SubstLeaves<"$_self", "$_op.getOperand(" # j # ").getType()",
IsShapedTypePred>]>; IsShapedTypePred>]>;
// Predicate to verify that the i'th result and the j'th operand have the same // Predicate to verify that the i'th result and the j'th operand have the same
// type. // type.
class TCresIsSameAsOpBase<int i, int j> : class TCresIsSameAsOpBase<int i, int j> :
CPred<"$_op.getResult(" # i # ")->getType() == " CPred<"$_op.getResult(" # i # ").getType() == "
"$_op.getOperand(" # j # ")->getType()">; "$_op.getOperand(" # j # ").getType()">;
// Basic Predicate to verify that the i'th result and the j'th operand have the // Basic Predicate to verify that the i'th result and the j'th operand have the
// same elemental type. // same elemental type.
class TCresVTEtIsSameAsOpBase<int i, int j> : class TCresVTEtIsSameAsOpBase<int i, int j> :
CPred<"getElementTypeOrSelf($_op.getResult(" # i # ")) == " CPred<"getElementTypeOrSelf($_op.getResult(" # i # ")) == "
"getElementTypeOrSelf($_op.getOperand(" # j # "))">; "getElementTypeOrSelf($_op.getOperand(" # j # "))">;
// Predicate to verify that the i'th result and the j'th operand have the same // Predicate to verify that the i'th result and the j'th operand have the same
// elemental type. // elemental type.
// Type Constraint result`i`'s Element type is Same As Operand `j`'s Element // Type Constraint result`i`'s Element type is Same As Operand `j`'s Element
// type. // type.
class TCresVTEtIsSameAsOp<int i, int j> : And<[ class TCresVTEtIsSameAsOp<int i, int j> : And<[
TCOpResIsShapedTypePred<i, j>, TCOpResIsShapedTypePred<i, j>,
TCresVTEtIsSameAsOpBase<i, j>]>; TCresVTEtIsSameAsOpBase<i, j>]>;
// Predicate to verify that the opId'th operand can be broadcasted to the type // Predicate to verify that the opId'th operand can be broadcasted to the type
// of the resId'th result. // of the resId'th result.
class TCOpIsBroadcastableToRes<int opId, int resId> : And<[ class TCOpIsBroadcastableToRes<int opId, int resId> : And<[
TCOpResIsShapedTypePred<opId, resId>, TCOpResIsShapedTypePred<opId, resId>,
CPred<"OpTrait::util::getBroadcastedType(" CPred<"OpTrait::util::getBroadcastedType("
"$_op.getOperand(" # opId # ")->getType(), " "$_op.getOperand(" # opId # ").getType(), "
"$_op.getResult(" # resId # ")->getType())">]>; "$_op.getResult(" # resId # ").getType())">]>;
// Predicate to verify that all the operands at the given `indices` // Predicate to verify that all the operands at the given `indices`
// have the same element type. // have the same element type.
// Type Constraint operands' Element type are all Same At the given `indices`. // Type Constraint operands' Element type are all Same At the given `indices`.
// We query the operands' types into a list and check they are all the same. // We query the operands' types into a list and check they are all the same.
// Precondition: // Precondition:
// 1) all operands involved are of shaped type and // 1) all operands involved are of shaped type and
// 2) the indices are not out of range. // 2) the indices are not out of range.
▲ Show 20 Lines • Show All 130 Lines • Show Last 20 Lines

mlir/include/mlir/IR/OpDefinition.h

Show First 20 Lines • Show All 544 Lines • ▼ Show 20 Linesstruct MultiResultTraitBase : public TraitBase<ConcreteType, TraitType> {
/// Replace all uses of results of this operation with the provided 'values'. /// Replace all uses of results of this operation with the provided 'values'.
/// 'values' may correspond to an existing operation, or a range of 'Value'. /// 'values' may correspond to an existing operation, or a range of 'Value'.
template <typename ValuesT> void replaceAllUsesWith(ValuesT &&values) { template <typename ValuesT> void replaceAllUsesWith(ValuesT &&values) {
this->getOperation()->replaceAllUsesWith(std::forward<ValuesT>(values)); this->getOperation()->replaceAllUsesWith(std::forward<ValuesT>(values));
} }
/// Return the type of the `i`-th result. /// Return the type of the `i`-th result.
Type getType(unsigned i) { return getResult(i)->getType(); } Type getType(unsigned i) { return getResult(i).getType(); }
/// Result iterator access. /// Result iterator access.
result_iterator result_begin() { result_iterator result_begin() {
return this->getOperation()->result_begin(); return this->getOperation()->result_begin();
} }
result_iterator result_end() { return this->getOperation()->result_end(); } result_iterator result_end() { return this->getOperation()->result_end(); }
result_range getResults() { return this->getOperation()->getResults(); } result_range getResults() { return this->getOperation()->getResults(); }
Show All 11 Lines
} // end namespace detail } // end namespace detail
/// This class provides return value APIs for ops that are known to have a /// This class provides return value APIs for ops that are known to have a
/// single result. /// single result.
template <typename ConcreteType> template <typename ConcreteType>
class OneResult : public TraitBase<ConcreteType, OneResult> { class OneResult : public TraitBase<ConcreteType, OneResult> {
public: public:
Value getResult() { return this->getOperation()->getResult(0); } Value getResult() { return this->getOperation()->getResult(0); }
Type getType() { return getResult()->getType(); } Type getType() { return getResult().getType(); }
/// Replace all uses of 'this' value with the new value, updating anything in /// Replace all uses of 'this' value with the new value, updating anything in
/// the IR that uses 'this' to use the other value instead. When this returns /// the IR that uses 'this' to use the other value instead. When this returns
/// there are zero uses of 'this'. /// there are zero uses of 'this'.
void replaceAllUsesWith(Value newValue) { void replaceAllUsesWith(Value newValue) {
getResult()->replaceAllUsesWith(newValue); getResult().replaceAllUsesWith(newValue);
} }
/// Replace all uses of 'this' value with the result of 'op'. /// Replace all uses of 'this' value with the result of 'op'.
void replaceAllUsesWith(Operation *op) { void replaceAllUsesWith(Operation *op) {
this->getOperation()->replaceAllUsesWith(op); this->getOperation()->replaceAllUsesWith(op);
} }
static LogicalResult verifyTrait(Operation *op) { static LogicalResult verifyTrait(Operation *op) {
▲ Show 20 Lines • Show All 630 Lines • Show Last 20 Lines

mlir/include/mlir/IR/OpImplementation.h

Show First 20 Lines • Show All 108 Lines • ▼ Show 20 Linespublic:
} }
/// Print the complete type of an operation in functional form. /// Print the complete type of an operation in functional form.
void printFunctionalType(Operation *op) { void printFunctionalType(Operation *op) {
auto &os = getStream(); auto &os = getStream();
os << "("; os << "(";
interleaveComma(op->getNonSuccessorOperands(), os, [&](Value operand) { interleaveComma(op->getNonSuccessorOperands(), os, [&](Value operand) {
if (operand) if (operand)
printType(operand->getType()); printType(operand.getType());
else else
os << "<<NULL>"; os << "<<NULL>";
}); });
os << ") -> "; os << ") -> ";
if (op->getNumResults() == 1 && if (op->getNumResults() == 1 &&
!op->getResult(0)->getType().isa<FunctionType>()) { !op->getResult(0).getType().isa<FunctionType>()) {
printType(op->getResult(0)->getType()); printType(op->getResult(0).getType());
} else { } else {
os << '('; os << '(';
interleaveComma(op->getResultTypes(), os); interleaveComma(op->getResultTypes(), os);
os << ')'; os << ')';
} }
} }
/// Print the given string as a symbol reference, i.e. a form representable by /// Print the given string as a symbol reference, i.e. a form representable by
▲ Show 20 Lines • Show All 534 Lines • Show Last 20 Lines

mlir/include/mlir/IR/Operation.h

Show First 20 Lines • Show All 143 Lines • ▼ Show 20 Linespublic:
template <typename ValuesT, template <typename ValuesT,
typename = decltype(std::declval<ValuesT>().begin())> typename = decltype(std::declval<ValuesT>().begin())>
void replaceAllUsesWith(ValuesT &&values) { void replaceAllUsesWith(ValuesT &&values) {
assert(std::distance(values.begin(), values.end()) == getNumResults() && assert(std::distance(values.begin(), values.end()) == getNumResults() &&
"expected 'values' to correspond 1-1 with the number of results"); "expected 'values' to correspond 1-1 with the number of results");
auto valueIt = values.begin(); auto valueIt = values.begin();
for (unsigned i = 0, e = getNumResults(); i != e; ++i) for (unsigned i = 0, e = getNumResults(); i != e; ++i)
getResult(i)->replaceAllUsesWith(*(valueIt++)); getResult(i).replaceAllUsesWith(*(valueIt++));
} }
/// Replace all uses of results of this operation with results of 'op'. /// Replace all uses of results of this operation with results of 'op'.
void replaceAllUsesWith(Operation *op) { void replaceAllUsesWith(Operation *op) {
assert(getNumResults() == op->getNumResults()); assert(getNumResults() == op->getNumResults());
for (unsigned i = 0, e = getNumResults(); i != e; ++i) for (unsigned i = 0, e = getNumResults(); i != e; ++i)
getResult(i)->replaceAllUsesWith(op->getResult(i)); getResult(i).replaceAllUsesWith(op->getResult(i));
} }
/// Destroys this operation and its subclass data. /// Destroys this operation and its subclass data.
void destroy(); void destroy();
/// This drops all operand uses from this operation, which is an essential /// This drops all operand uses from this operation, which is an essential
/// step in breaking cyclic dependences between references when they are to /// step in breaking cyclic dependences between references when they are to
/// be deleted. /// be deleted.
▲ Show 20 Lines • Show All 525 Lines • Show Last 20 Lines

mlir/include/mlir/IR/Value.h

Show First 20 Lines • Show All 84 Lines • ▼ Show 20 Linespublic:
template <typename U> U dyn_cast_or_null() const { template <typename U> U dyn_cast_or_null() const {
return (*this && isa<U>()) ? U(ownerAndKind) : U(nullptr); return (*this && isa<U>()) ? U(ownerAndKind) : U(nullptr);
} }
template <typename U> U cast() const { template <typename U> U cast() const {
assert(isa<U>()); assert(isa<U>());
return U(ownerAndKind); return U(ownerAndKind);
} }
/// Temporary methods to enable transition of Value to being used as a
/// value-type.
/// TODO(riverriddle) Remove these when all usages have been removed.
Value operator*() const { return *this; }
Value *operator->() const { return const_cast<Value *>(this); }
operator bool() const { return ownerAndKind.getPointer(); } operator bool() const { return ownerAndKind.getPointer(); }
bool operator==(const Value &other) const { bool operator==(const Value &other) const {
return ownerAndKind == other.ownerAndKind; return ownerAndKind == other.ownerAndKind;
} }
bool operator!=(const Value &other) const { return !(*this == other); } bool operator!=(const Value &other) const { return !(*this == other); }
/// Return the type of this value. /// Return the type of this value.
Type getType() const; Type getType() const;
Show All 10 Linespublic:
void setType(Type newType); void setType(Type newType);
/// If this value is the result of an operation, return the operation that /// If this value is the result of an operation, return the operation that
/// defines it. /// defines it.
Operation *getDefiningOp() const; Operation *getDefiningOp() const;
/// If this value is the result of an operation, use it as a location, /// If this value is the result of an operation, use it as a location,
/// otherwise return an unknown location. /// otherwise return an unknown location.
Location getLoc(); Location getLoc() const;
/// Return the Region in which this Value is defined. /// Return the Region in which this Value is defined.
Region *getParentRegion(); Region *getParentRegion();
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// UseLists // UseLists
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 97 Lines • ▼ Show 20 Lines
}; };
} // end namespace detail } // end namespace detail
/// Block arguments are values. /// Block arguments are values.
class BlockArgument : public Value { class BlockArgument : public Value {
public: public:
using Value::Value; using Value::Value;
/// Temporary methods to enable transition of Value to being used as a
/// value-type.
/// TODO(riverriddle) Remove this when all usages have been removed.
BlockArgument *operator->() { return this; }
static bool classof(Value value) { static bool classof(Value value) {
return value.getKind() == Kind::BlockArgument; return value.getKind() == Kind::BlockArgument;
} }
/// Returns the block that owns this argument. /// Returns the block that owns this argument.
Block *getOwner() const { return getImpl()->owner; } Block *getOwner() const { return getImpl()->owner; }
/// Return the type of this value. /// Return the type of this value.
Show All 31 Lines
// OpResult // OpResult
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// This is a value defined by a result of an operation. /// This is a value defined by a result of an operation.
class OpResult : public Value { class OpResult : public Value {
public: public:
using Value::Value; using Value::Value;
/// Temporary methods to enable transition of Value to being used as a
/// value-type.
/// TODO(riverriddle) Remove these when all usages have been removed.
OpResult operator*() { return *this; }
OpResult *operator->() { return this; }
static bool classof(Value value) { static bool classof(Value value) {
return value.getKind() != Kind::BlockArgument; return value.getKind() != Kind::BlockArgument;
} }
/// Returns the operation that owns this result. /// Returns the operation that owns this result.
Operation *getOwner() const; Operation *getOwner() const;
/// Returns the number of this result. /// Returns the number of this result.
▲ Show 20 Lines • Show All 86 Lines • Show Last 20 Lines

mlir/include/mlir/Quantizer/Support/ConstraintAnalysisGraph.h

Show First 20 Lines • Show All 215 Lines • ▼ Show 20 Lines
class CAGResultAnchor : public CAGAnchorNode { class CAGResultAnchor : public CAGAnchorNode {
public: public:
CAGResultAnchor(Operation *op, unsigned resultIdx); CAGResultAnchor(Operation *op, unsigned resultIdx);
static bool classof(const CAGNode *n) { static bool classof(const CAGNode *n) {
return n->getKind() == Kind::Anchor || n->getKind() == Kind::ResultAnchor; return n->getKind() == Kind::Anchor || n->getKind() == Kind::ResultAnchor;
} }
Operation *getOp() const final { return resultValue->getDefiningOp(); } Operation *getOp() const final { return resultValue.getDefiningOp(); }
Value getValue() const final { return resultValue; } Value getValue() const final { return resultValue; }
void printLabel(raw_ostream &os) const override; void printLabel(raw_ostream &os) const override;
private: private:
Value resultValue; Value resultValue;
}; };
▲ Show 20 Lines • Show All 128 Lines • Show Last 20 Lines

mlir/include/mlir/Transforms/RegionUtils.h

Show All 16 Lines
namespace mlir { namespace mlir {
/// Check if all values in the provided range are defined above the `limit` /// Check if all values in the provided range are defined above the `limit`
/// region. That is, if they are defined in a region that is a proper ancestor /// region. That is, if they are defined in a region that is a proper ancestor
/// of `limit`. /// of `limit`.
template <typename Range> template <typename Range>
bool areValuesDefinedAbove(Range values, Region &limit) { bool areValuesDefinedAbove(Range values, Region &limit) {
for (Value v : values) for (Value v : values)
if (!v->getParentRegion()->isProperAncestor(&limit)) if (!v.getParentRegion()->isProperAncestor(&limit))
return false; return false;
return true; return true;
} }
/// Replace all uses of `orig` within the given region with `replacement`. /// Replace all uses of `orig` within the given region with `replacement`.
void replaceAllUsesInRegionWith(Value orig, Value replacement, Region &region); void replaceAllUsesInRegionWith(Value orig, Value replacement, Region &region);
/// Calls `callback` for each use of a value within `region` or its descendants /// Calls `callback` for each use of a value within `region` or its descendants
Show All 28 Lines

mlir/lib/Analysis/AffineAnalysis.cpp

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 Linesvoid mlir::getReachableAffineApplyOps(
}; };
SmallVector<State, 4> worklist; SmallVector<State, 4> worklist;
for (auto operand : operands) { for (auto operand : operands) {
worklist.push_back({operand, 0}); worklist.push_back({operand, 0});
} }
while (!worklist.empty()) { while (!worklist.empty()) {
State &state = worklist.back(); State &state = worklist.back();
auto *opInst = state.value->getDefiningOp(); auto *opInst = state.value.getDefiningOp();
// Note: getDefiningOp will return nullptr if the operand is not an // Note: getDefiningOp will return nullptr if the operand is not an
// Operation (i.e. block argument), which is a terminator for the search. // Operation (i.e. block argument), which is a terminator for the search.
if (!isa_and_nonnull<AffineApplyOp>(opInst)) { if (!isa_and_nonnull<AffineApplyOp>(opInst)) {
worklist.pop_back(); worklist.pop_back();
continue; continue;
} }
if (state.operandIndex == 0) { if (state.operandIndex == 0) {
▲ Show 20 Lines • Show All 385 Lines • ▼ Show 20 LinesaddMemRefAccessConstraints(const AffineValueMap &srcAccessMap,
// Add equality constraints for any operands that are defined by constant ops. // Add equality constraints for any operands that are defined by constant ops.
auto addEqForConstOperands = [&](ArrayRef<Value> operands) { auto addEqForConstOperands = [&](ArrayRef<Value> operands) {
for (unsigned i = 0, e = operands.size(); i < e; ++i) { for (unsigned i = 0, e = operands.size(); i < e; ++i) {
if (isForInductionVar(operands[i])) if (isForInductionVar(operands[i]))
continue; continue;
auto symbol = operands[i]; auto symbol = operands[i];
assert(isValidSymbol(symbol)); assert(isValidSymbol(symbol));
// Check if the symbol is a constant. // Check if the symbol is a constant.
if (auto cOp = dyn_cast_or_null<ConstantIndexOp>(symbol->getDefiningOp())) if (auto cOp = dyn_cast_or_null<ConstantIndexOp>(symbol.getDefiningOp()))
dependenceDomain->setIdToConstant(valuePosMap.getSymPos(symbol), dependenceDomain->setIdToConstant(valuePosMap.getSymPos(symbol),
cOp.getValue()); cOp.getValue());
} }
}; };
// Add equality constraints for any src symbols defined by constant ops. // Add equality constraints for any src symbols defined by constant ops.
addEqForConstOperands(srcOperands); addEqForConstOperands(srcOperands);
// Add equality constraints for any dst symbols defined by constant ops. // Add equality constraints for any dst symbols defined by constant ops.
▲ Show 20 Lines • Show All 420 Lines • Show Last 20 Lines

mlir/lib/Analysis/AffineStructures.cpp

Show First 20 Lines • Show All 579 Lines • ▼ Show 20 Linesstatic void mergeAndAlignIds(unsigned offset, FlatAffineConstraints *A,
SmallVector<Value, 4> aDimValues, aSymValues; SmallVector<Value, 4> aDimValues, aSymValues;
A->getIdValues(offset, A->getNumDimIds(), &aDimValues); A->getIdValues(offset, A->getNumDimIds(), &aDimValues);
A->getIdValues(A->getNumDimIds(), A->getNumDimAndSymbolIds(), &aSymValues); A->getIdValues(A->getNumDimIds(), A->getNumDimAndSymbolIds(), &aSymValues);
{ {
// Merge dims from A into B. // Merge dims from A into B.
unsigned d = offset; unsigned d = offset;
for (auto aDimValue : aDimValues) { for (auto aDimValue : aDimValues) {
unsigned loc; unsigned loc;
if (B->findId(*aDimValue, &loc)) { if (B->findId(aDimValue, &loc)) {
assert(loc >= offset && "A's dim appears in B's aligned range"); assert(loc >= offset && "A's dim appears in B's aligned range");
assert(loc < B->getNumDimIds() && assert(loc < B->getNumDimIds() &&
"A's dim appears in B's non-dim position"); "A's dim appears in B's non-dim position");
swapId(B, d, loc); swapId(B, d, loc);
} else { } else {
B->addDimId(d); B->addDimId(d);
B->setIdValue(d, aDimValue); B->setIdValue(d, aDimValue);
} }
d++; d++;
} }
// Dimensions that are in B, but not in A, are added at the end. // Dimensions that are in B, but not in A, are added at the end.
for (unsigned t = A->getNumDimIds(), e = B->getNumDimIds(); t < e; t++) { for (unsigned t = A->getNumDimIds(), e = B->getNumDimIds(); t < e; t++) {
A->addDimId(A->getNumDimIds()); A->addDimId(A->getNumDimIds());
A->setIdValue(A->getNumDimIds() - 1, B->getIdValue(t)); A->setIdValue(A->getNumDimIds() - 1, B->getIdValue(t));
} }
} }
{ {
// Merge symbols: merge A's symbols into B first. // Merge symbols: merge A's symbols into B first.
unsigned s = B->getNumDimIds(); unsigned s = B->getNumDimIds();
for (auto aSymValue : aSymValues) { for (auto aSymValue : aSymValues) {
unsigned loc; unsigned loc;
if (B->findId(*aSymValue, &loc)) { if (B->findId(aSymValue, &loc)) {
assert(loc >= B->getNumDimIds() && loc < B->getNumDimAndSymbolIds() && assert(loc >= B->getNumDimIds() && loc < B->getNumDimAndSymbolIds() &&
"A's symbol appears in B's non-symbol position"); "A's symbol appears in B's non-symbol position");
swapId(B, s, loc); swapId(B, s, loc);
} else { } else {
B->addSymbolId(s - B->getNumDimIds()); B->addSymbolId(s - B->getNumDimIds());
B->setIdValue(s, aSymValue); B->setIdValue(s, aSymValue);
} }
s++; s++;
▲ Show 20 Lines • Show All 58 Lines • ▼ Show 20 Lines for (unsigned r = 0, e = flatExprs.size(); r < e; r++) {
// eqToAdd is the equality corresponding to the flattened affine expression. // eqToAdd is the equality corresponding to the flattened affine expression.
SmallVector<int64_t, 8> eqToAdd(getNumCols(), 0); SmallVector<int64_t, 8> eqToAdd(getNumCols(), 0);
// Set the coefficient for this result to one. // Set the coefficient for this result to one.
eqToAdd[r] = 1; eqToAdd[r] = 1;
// Dims and symbols. // Dims and symbols.
for (unsigned i = 0, e = vMap->getNumOperands(); i < e; i++) { for (unsigned i = 0, e = vMap->getNumOperands(); i < e; i++) {
unsigned loc; unsigned loc;
bool ret = findId(*vMap->getOperand(i), &loc); bool ret = findId(vMap->getOperand(i), &loc);
assert(ret && "value map's id can't be found"); assert(ret && "value map's id can't be found");
(void)ret; (void)ret;
// Negate 'eq[r]' since the newly added dimension will be set to this one. // Negate 'eq[r]' since the newly added dimension will be set to this one.
eqToAdd[loc] = -flatExpr[i]; eqToAdd[loc] = -flatExpr[i];
} }
// Local vars common to eq and localCst are at the beginning. // Local vars common to eq and localCst are at the beginning.
unsigned j = getNumDimIds() + getNumSymbolIds(); unsigned j = getNumDimIds() + getNumSymbolIds();
unsigned end = flatExpr.size() - 1; unsigned end = flatExpr.size() - 1;
▲ Show 20 Lines • Show All 104 Lines • ▼ Show 20 Linesvoid FlatAffineConstraints::convertLoopIVSymbolsToDims() {
// Gather all symbols which are loop IVs. // Gather all symbols which are loop IVs.
SmallVector<Value, 4> loopIVs; SmallVector<Value, 4> loopIVs;
for (unsigned i = getNumDimIds(), e = getNumDimAndSymbolIds(); i < e; i++) { for (unsigned i = getNumDimIds(), e = getNumDimAndSymbolIds(); i < e; i++) {
if (ids[i].hasValue() && getForInductionVarOwner(ids[i].getValue())) if (ids[i].hasValue() && getForInductionVarOwner(ids[i].getValue()))
loopIVs.push_back(ids[i].getValue()); loopIVs.push_back(ids[i].getValue());
} }
// Turn each symbol in 'loopIVs' into a dim identifier. // Turn each symbol in 'loopIVs' into a dim identifier.
for (auto iv : loopIVs) { for (auto iv : loopIVs) {
turnSymbolIntoDim(this, *iv); turnSymbolIntoDim(this, iv);
} }
} }
void FlatAffineConstraints::addInductionVarOrTerminalSymbol(Value id) { void FlatAffineConstraints::addInductionVarOrTerminalSymbol(Value id) {
if (containsId(*id)) if (containsId(id))
return; return;
// Caller is expected to fully compose map/operands if necessary. // Caller is expected to fully compose map/operands if necessary.
assert((isTopLevelValue(id) || isForInductionVar(id)) && assert((isTopLevelValue(id) || isForInductionVar(id)) &&
"non-terminal symbol / loop IV expected"); "non-terminal symbol / loop IV expected");
// Outer loop IVs could be used in forOp's bounds. // Outer loop IVs could be used in forOp's bounds.
if (auto loop = getForInductionVarOwner(id)) { if (auto loop = getForInductionVarOwner(id)) {
addDimId(getNumDimIds(), id); addDimId(getNumDimIds(), id);
if (failed(this->addAffineForOpDomain(loop))) if (failed(this->addAffineForOpDomain(loop)))
LLVM_DEBUG( LLVM_DEBUG(
loop.emitWarning("failed to add domain info to constraint system")); loop.emitWarning("failed to add domain info to constraint system"));
return; return;
} }
// Add top level symbol. // Add top level symbol.
addSymbolId(getNumSymbolIds(), id); addSymbolId(getNumSymbolIds(), id);
// Check if the symbol is a constant. // Check if the symbol is a constant.
if (auto constOp = dyn_cast_or_null<ConstantIndexOp>(id->getDefiningOp())) if (auto constOp = dyn_cast_or_null<ConstantIndexOp>(id.getDefiningOp()))
setIdToConstant(*id, constOp.getValue()); setIdToConstant(id, constOp.getValue());
} }
LogicalResult FlatAffineConstraints::addAffineForOpDomain(AffineForOp forOp) { LogicalResult FlatAffineConstraints::addAffineForOpDomain(AffineForOp forOp) {
unsigned pos; unsigned pos;
// Pre-condition for this method. // Pre-condition for this method.
if (!findId(*forOp.getInductionVar(), &pos)) { if (!findId(forOp.getInductionVar(), &pos)) {
assert(false && "Value not found"); assert(false && "Value not found");
return failure(); return failure();
} }
int64_t step = forOp.getStep(); int64_t step = forOp.getStep();
if (step != 1) { if (step != 1) {
if (!forOp.hasConstantLowerBound()) if (!forOp.hasConstantLowerBound())
forOp.emitWarning("domain conservatively approximated"); forOp.emitWarning("domain conservatively approximated");
▲ Show 20 Lines • Show All 930 Lines • ▼ Show 20 LinesFlatAffineConstraints::addLowerOrUpperBound(unsigned pos, AffineMap boundMap,
} }
// Merge and align with localVarCst. // Merge and align with localVarCst.
if (localVarCst.getNumLocalIds() > 0) { if (localVarCst.getNumLocalIds() > 0) {
// Set values for localVarCst. // Set values for localVarCst.
localVarCst.setIdValues(0, localVarCst.getNumDimAndSymbolIds(), operands); localVarCst.setIdValues(0, localVarCst.getNumDimAndSymbolIds(), operands);
for (auto operand : operands) { for (auto operand : operands) {
unsigned pos; unsigned pos;
if (findId(*operand, &pos)) { if (findId(operand, &pos)) {
if (pos >= getNumDimIds() && pos < getNumDimAndSymbolIds()) { if (pos >= getNumDimIds() && pos < getNumDimAndSymbolIds()) {
// If the local var cst has this as a dim, turn it into its symbol. // If the local var cst has this as a dim, turn it into its symbol.
turnDimIntoSymbol(&localVarCst, *operand); turnDimIntoSymbol(&localVarCst, operand);
} else if (pos < getNumDimIds()) { } else if (pos < getNumDimIds()) {
// Or vice versa. // Or vice versa.
turnSymbolIntoDim(&localVarCst, *operand); turnSymbolIntoDim(&localVarCst, operand);
} }
} }
} }
mergeAndAlignIds(/*offset=*/0, this, &localVarCst); mergeAndAlignIds(/*offset=*/0, this, &localVarCst);
append(localVarCst); append(localVarCst);
} }
// Record positions of the operands in the constraint system. Need to do // Record positions of the operands in the constraint system. Need to do
// this here since the constraint system changes after a bound is added. // this here since the constraint system changes after a bound is added.
SmallVector<unsigned, 8> positions; SmallVector<unsigned, 8> positions;
unsigned numOperands = operands.size(); unsigned numOperands = operands.size();
for (auto operand : operands) { for (auto operand : operands) {
unsigned pos; unsigned pos;
if (!findId(*operand, &pos)) if (!findId(operand, &pos))
assert(0 && "expected to be found"); assert(0 && "expected to be found");
positions.push_back(pos); positions.push_back(pos);
} }
for (const auto &flatExpr : flatExprs) { for (const auto &flatExpr : flatExprs) {
SmallVector<int64_t, 4> ineq(getNumCols(), 0); SmallVector<int64_t, 4> ineq(getNumCols(), 0);
ineq[pos] = lower ? 1 : -1; ineq[pos] = lower ? 1 : -1;
// Dims and symbols. // Dims and symbols.
Show All 30 LinesLogicalResult FlatAffineConstraints::addSliceBounds(ArrayRef<Value> values,
ArrayRef<AffineMap> lbMaps, ArrayRef<AffineMap> lbMaps,
ArrayRef<AffineMap> ubMaps, ArrayRef<AffineMap> ubMaps,
ArrayRef<Value> operands) { ArrayRef<Value> operands) {
assert(values.size() == lbMaps.size()); assert(values.size() == lbMaps.size());
assert(lbMaps.size() == ubMaps.size()); assert(lbMaps.size() == ubMaps.size());
for (unsigned i = 0, e = lbMaps.size(); i < e; ++i) { for (unsigned i = 0, e = lbMaps.size(); i < e; ++i) {
unsigned pos; unsigned pos;
if (!findId(*values[i], &pos)) if (!findId(values[i], &pos))
continue; continue;
AffineMap lbMap = lbMaps[i]; AffineMap lbMap = lbMaps[i];
AffineMap ubMap = ubMaps[i]; AffineMap ubMap = ubMaps[i];
assert(!lbMap || lbMap.getNumInputs() == operands.size()); assert(!lbMap || lbMap.getNumInputs() == operands.size());
assert(!ubMap || ubMap.getNumInputs() == operands.size()); assert(!ubMap || ubMap.getNumInputs() == operands.size());
// Check if this slice is just an equality along this dimension. // Check if this slice is just an equality along this dimension.
▲ Show 20 Lines • Show All 839 Lines • ▼ Show 20 Linesvoid FlatAffineConstraints::projectOut(unsigned pos, unsigned num) {
GCDTightenInequalities(); GCDTightenInequalities();
// Normalize constraints after tightening since the latter impacts this, but // Normalize constraints after tightening since the latter impacts this, but
// not the other way round. // not the other way round.
normalizeConstraintsByGCD(); normalizeConstraintsByGCD();
} }
void FlatAffineConstraints::projectOut(Value id) { void FlatAffineConstraints::projectOut(Value id) {
unsigned pos; unsigned pos;
bool ret = findId(*id, &pos); bool ret = findId(id, &pos);
assert(ret); assert(ret);
(void)ret; (void)ret;
FourierMotzkinEliminate(pos); FourierMotzkinEliminate(pos);
} }
void FlatAffineConstraints::clearConstraints() { void FlatAffineConstraints::clearConstraints() {
equalities.clear(); equalities.clear();
inequalities.clear(); inequalities.clear();
▲ Show 20 Lines • Show All 140 Lines • Show Last 20 Lines

mlir/lib/Analysis/CallGraph.cpp

Show First 20 Lines • Show All 173 Lines • ▼ Show 20 LinesCallGraphNode *CallGraph::resolveCallable(CallInterfaceCallable callable,
Operation *from) const { Operation *from) const {
// Get the callee operation from the callable. // Get the callee operation from the callable.
Operation *callee; Operation *callee;
if (auto symbolRef = callable.dyn_cast<SymbolRefAttr>()) if (auto symbolRef = callable.dyn_cast<SymbolRefAttr>())
// TODO(riverriddle) Support nested references. // TODO(riverriddle) Support nested references.
callee = SymbolTable::lookupNearestSymbolFrom(from, callee = SymbolTable::lookupNearestSymbolFrom(from,
symbolRef.getRootReference()); symbolRef.getRootReference());
else else
callee = callable.get<Value>()->getDefiningOp(); callee = callable.get<Value>().getDefiningOp();
// If the callee is non-null and is a valid callable object, try to get the // If the callee is non-null and is a valid callable object, try to get the
// called region from it. // called region from it.
if (callee && callee->getNumRegions()) { if (callee && callee->getNumRegions()) {
if (auto callableOp = dyn_cast_or_null<CallableOpInterface>(callee)) { if (auto callableOp = dyn_cast_or_null<CallableOpInterface>(callee)) {
if (auto *node = lookupNode(callableOp.getCallableRegion(callable))) if (auto *node = lookupNode(callableOp.getCallableRegion(callable)))
return node; return node;
} }
▲ Show 20 Lines • Show All 66 Lines • Show Last 20 Lines

mlir/lib/Analysis/Dominance.cpp

Show First 20 Lines • Show All 113 Lines • ▼ Show 20 Linesbool DominanceInfo::properlyDominates(Operation *a, Operation *b) {
} }
// If the blocks are different, check if a's block dominates b's. // If the blocks are different, check if a's block dominates b's.
return properlyDominates(aBlock, bBlock); return properlyDominates(aBlock, bBlock);
} }
/// Return true if value A properly dominates operation B. /// Return true if value A properly dominates operation B.
bool DominanceInfo::properlyDominates(Value a, Operation *b) { bool DominanceInfo::properlyDominates(Value a, Operation *b) {
if (auto *aOp = a->getDefiningOp()) { if (auto *aOp = a.getDefiningOp()) {
// The values defined by an operation do *not* dominate any nested // The values defined by an operation do *not* dominate any nested
// operations. // operations.
if (aOp->getParentRegion() != b->getParentRegion() && aOp->isAncestor(b)) if (aOp->getParentRegion() != b->getParentRegion() && aOp->isAncestor(b))
return false; return false;
return properlyDominates(aOp, b); return properlyDominates(aOp, b);
} }
// block arguments properly dominate all operations in their own block, so // block arguments properly dominate all operations in their own block, so
// we use a dominates check here, not a properlyDominates check. // we use a dominates check here, not a properlyDominates check.
return dominates(a.cast<BlockArgument>()->getOwner(), b->getBlock()); return dominates(a.cast<BlockArgument>().getOwner(), b->getBlock());
} }
DominanceInfoNode *DominanceInfo::getNode(Block *a) { DominanceInfoNode *DominanceInfo::getNode(Block *a) {
auto *region = a->getParent(); auto *region = a->getParent();
assert(dominanceInfos.count(region) != 0); assert(dominanceInfos.count(region) != 0);
return dominanceInfos[region]->getNode(a); return dominanceInfos[region]->getNode(a);
} }
Show All 31 Lines

mlir/lib/Analysis/Liveness.cpp

Show All 39 Lines for (Operation &operation : *block)
for (Value result : operation.getResults()) { for (Value result : operation.getResults()) {
defValues.insert(result); defValues.insert(result);
// Check whether this value will be in the outValues // Check whether this value will be in the outValues
// set (its uses escape this block). Due to the SSA // set (its uses escape this block). Due to the SSA
// properties of the program, the uses must occur after // properties of the program, the uses must occur after
// the definition. Therefore, we do not have to check // the definition. Therefore, we do not have to check
// additional conditions to detect an escaping value. // additional conditions to detect an escaping value.
for (OpOperand &use : result->getUses()) for (OpOperand &use : result.getUses())
if (use.getOwner()->getBlock() != block) { if (use.getOwner()->getBlock() != block) {
outValues.insert(result); outValues.insert(result);
break; break;
} }
} }
// Check all operations for used operands. // Check all operations for used operands.
for (Operation &operation : block->getOperations()) for (Operation &operation : block->getOperations())
▲ Show 20 Lines • Show All 109 Lines • ▼ Show 20 Lines
/// Gets liveness info (if any) for the given value. /// Gets liveness info (if any) for the given value.
Liveness::OperationListT Liveness::resolveLiveness(Value value) const { Liveness::OperationListT Liveness::resolveLiveness(Value value) const {
OperationListT result; OperationListT result;
SmallPtrSet<Block *, 32> visited; SmallPtrSet<Block *, 32> visited;
SmallVector<Block *, 8> toProcess; SmallVector<Block *, 8> toProcess;
// Start with the defining block // Start with the defining block
Block *currentBlock; Block *currentBlock;
if (Operation *defOp = value->getDefiningOp()) if (Operation *defOp = value.getDefiningOp())
currentBlock = defOp->getBlock(); currentBlock = defOp->getBlock();
else else
currentBlock = value.cast<BlockArgument>()->getOwner(); currentBlock = value.cast<BlockArgument>().getOwner();
toProcess.push_back(currentBlock); toProcess.push_back(currentBlock);
visited.insert(currentBlock); visited.insert(currentBlock);
// Start with all associated blocks // Start with all associated blocks
for (OpOperand &use : value->getUses()) { for (OpOperand &use : value.getUses()) {
Block *useBlock = use.getOwner()->getBlock(); Block *useBlock = use.getOwner()->getBlock();
if (visited.insert(useBlock).second) if (visited.insert(useBlock).second)
toProcess.push_back(useBlock); toProcess.push_back(useBlock);
} }
while (!toProcess.empty()) { while (!toProcess.empty()) {
// Get block and block liveness information. // Get block and block liveness information.
Block *block = toProcess.back(); Block *block = toProcess.back();
▲ Show 20 Lines • Show All 73 Lines • ▼ Show 20 Lines for (Block &block : region) {
operationIds.insert({&operation, operationIds.size()}); operationIds.insert({&operation, operationIds.size()});
for (Value result : operation.getResults()) for (Value result : operation.getResults())
valueIds.insert({result, valueIds.size()}); valueIds.insert({result, valueIds.size()});
} }
} }
// Local printing helpers // Local printing helpers
auto printValueRef = [&](Value value) { auto printValueRef = [&](Value value) {
if (Operation *defOp = value->getDefiningOp()) if (Operation *defOp = value.getDefiningOp())
os << "val_" << defOp->getName(); os << "val_" << defOp->getName();
else { else {
auto blockArg = value.cast<BlockArgument>(); auto blockArg = value.cast<BlockArgument>();
os << "arg" << blockArg->getArgNumber() << "@" os << "arg" << blockArg.getArgNumber() << "@"
<< blockIds[blockArg->getOwner()]; << blockIds[blockArg.getOwner()];
} }
os << " "; os << " ";
}; };
auto printValueRefs = [&](const ValueSetT &values) { auto printValueRefs = [&](const ValueSetT &values) {
std::vector<Value> orderedValues(values.begin(), values.end()); std::vector<Value> orderedValues(values.begin(), values.end());
std::sort(orderedValues.begin(), orderedValues.end(), std::sort(orderedValues.begin(), orderedValues.end(),
[&](Value left, Value right) { [&](Value left, Value right) {
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines
/// Returns true if the given value is in the live-out set. /// Returns true if the given value is in the live-out set.
bool LivenessBlockInfo::isLiveOut(Value value) const { bool LivenessBlockInfo::isLiveOut(Value value) const {
return outValues.count(value); return outValues.count(value);
} }
/// Gets the start operation for the given value /// Gets the start operation for the given value
/// (must be referenced in this block). /// (must be referenced in this block).
Operation *LivenessBlockInfo::getStartOperation(Value value) const { Operation *LivenessBlockInfo::getStartOperation(Value value) const {
Operation *definingOp = value->getDefiningOp(); Operation *definingOp = value.getDefiningOp();
// The given value is either live-in or is defined // The given value is either live-in or is defined
// in the scope of this block. // in the scope of this block.
if (isLiveIn(value) || !definingOp) if (isLiveIn(value) || !definingOp)
return &block->front(); return &block->front();
return definingOp; return definingOp;
} }
/// Gets the end operation for the given value using the start operation /// Gets the end operation for the given value using the start operation
/// provided (must be referenced in this block). /// provided (must be referenced in this block).
Operation *LivenessBlockInfo::getEndOperation(Value value, Operation *LivenessBlockInfo::getEndOperation(Value value,
Operation *startOperation) const { Operation *startOperation) const {
// The given value is either dying in this block or live-out. // The given value is either dying in this block or live-out.
if (isLiveOut(value)) if (isLiveOut(value))
return &block->back(); return &block->back();
// Resolve the last operation (must exist by definition). // Resolve the last operation (must exist by definition).
Operation *endOperation = startOperation; Operation *endOperation = startOperation;
for (OpOperand &use : value->getUses()) { for (OpOperand &use : value.getUses()) {
Operation *useOperation = use.getOwner(); Operation *useOperation = use.getOwner();
// Check whether the use is in our block and after // Check whether the use is in our block and after
// the current end operation. // the current end operation.
if (useOperation->getBlock() == block && if (useOperation->getBlock() == block &&
endOperation->isBeforeInBlock(useOperation)) endOperation->isBeforeInBlock(useOperation))
endOperation = useOperation; endOperation = useOperation;
} }
return endOperation; return endOperation;
} }

mlir/lib/Analysis/LoopAnalysis.cpp

Show First 20 Lines • Show All 160 Lines • ▼ Show 20 Lines
/// Prerequisites: /// Prerequisites:
/// 1. `iv` and `index` of the proper type; /// 1. `iv` and `index` of the proper type;
/// 2. at most one reachable AffineApplyOp from index; /// 2. at most one reachable AffineApplyOp from index;
/// ///
/// Returns false in cases with more than one AffineApplyOp, this is /// Returns false in cases with more than one AffineApplyOp, this is
/// conservative. /// conservative.
static bool isAccessIndexInvariant(Value iv, Value index) { static bool isAccessIndexInvariant(Value iv, Value index) {
assert(isForInductionVar(iv) && "iv must be a AffineForOp"); assert(isForInductionVar(iv) && "iv must be a AffineForOp");
assert(index->getType().isa<IndexType>() && "index must be of IndexType"); assert(index.getType().isa<IndexType>() && "index must be of IndexType");
SmallVector<Operation *, 4> affineApplyOps; SmallVector<Operation *, 4> affineApplyOps;
getReachableAffineApplyOps({index}, affineApplyOps); getReachableAffineApplyOps({index}, affineApplyOps);
if (affineApplyOps.empty()) { if (affineApplyOps.empty()) {
// Pointer equality test because of Value pointer semantics. // Pointer equality test because of Value pointer semantics.
return index != iv; return index != iv;
} }
▲ Show 20 Lines • Show All 190 Lines • ▼ Show 20 Lines for (auto it : llvm::enumerate(llvm::reverse(forBody->getOperations()))) {
// Remember the shift of this operation. // Remember the shift of this operation.
uint64_t shift = shifts[index]; uint64_t shift = shifts[index];
forBodyShift.try_emplace(&op, shift); forBodyShift.try_emplace(&op, shift);
// Validate the results of this operation if it were to be shifted. // Validate the results of this operation if it were to be shifted.
for (unsigned i = 0, e = op.getNumResults(); i < e; ++i) { for (unsigned i = 0, e = op.getNumResults(); i < e; ++i) {
Value result = op.getResult(i); Value result = op.getResult(i);
for (auto *user : result->getUsers()) { for (auto *user : result.getUsers()) {
// If an ancestor operation doesn't lie in the block of forOp, // If an ancestor operation doesn't lie in the block of forOp,
// there is no shift to check. // there is no shift to check.
if (auto *ancOp = forBody->findAncestorOpInBlock(*user)) { if (auto *ancOp = forBody->findAncestorOpInBlock(*user)) {
assert(forBodyShift.count(ancOp) > 0 && "ancestor expected in map"); assert(forBodyShift.count(ancOp) > 0 && "ancestor expected in map");
if (shift != forBodyShift[ancOp]) if (shift != forBodyShift[ancOp])
return false; return false;
} }
} }
} }
} }
return true; return true;
} }

mlir/lib/Analysis/SliceAnalysis.cpp

Show All 37 Linesstatic void getForwardSliceImpl(Operation *op,
// Evaluate whether we should keep this use. // Evaluate whether we should keep this use.
// This is useful in particular to implement scoping; i.e. return the // This is useful in particular to implement scoping; i.e. return the
// transitive forwardSlice in the current scope. // transitive forwardSlice in the current scope.
if (!filter(op)) { if (!filter(op)) {
return; return;
} }
if (auto forOp = dyn_cast<AffineForOp>(op)) { if (auto forOp = dyn_cast<AffineForOp>(op)) {
for (auto *ownerInst : forOp.getInductionVar()->getUsers()) for (auto *ownerInst : forOp.getInductionVar().getUsers())
if (forwardSlice->count(ownerInst) == 0) if (forwardSlice->count(ownerInst) == 0)
getForwardSliceImpl(ownerInst, forwardSlice, filter); getForwardSliceImpl(ownerInst, forwardSlice, filter);
} else if (auto forOp = dyn_cast<loop::ForOp>(op)) { } else if (auto forOp = dyn_cast<loop::ForOp>(op)) {
for (auto *ownerInst : forOp.getInductionVar()->getUsers()) for (auto *ownerInst : forOp.getInductionVar().getUsers())
if (forwardSlice->count(ownerInst) == 0) if (forwardSlice->count(ownerInst) == 0)
getForwardSliceImpl(ownerInst, forwardSlice, filter); getForwardSliceImpl(ownerInst, forwardSlice, filter);
} else { } else {
assert(op->getNumRegions() == 0 && "unexpected generic op with regions"); assert(op->getNumRegions() == 0 && "unexpected generic op with regions");
assert(op->getNumResults() <= 1 && "unexpected multiple results"); assert(op->getNumResults() <= 1 && "unexpected multiple results");
if (op->getNumResults() > 0) { if (op->getNumResults() > 0) {
for (auto *ownerInst : op->getResult(0)->getUsers()) for (auto *ownerInst : op->getResult(0).getUsers())
if (forwardSlice->count(ownerInst) == 0) if (forwardSlice->count(ownerInst) == 0)
getForwardSliceImpl(ownerInst, forwardSlice, filter); getForwardSliceImpl(ownerInst, forwardSlice, filter);
} }
} }
forwardSlice->insert(op); forwardSlice->insert(op);
} }
Show All 34 Lines if (auto blockArg = operand.dyn_cast<BlockArgument>()) {
if (auto affIv = getForInductionVarOwner(operand)) { if (auto affIv = getForInductionVarOwner(operand)) {
auto *affOp = affIv.getOperation(); auto *affOp = affIv.getOperation();
if (backwardSlice->count(affOp) == 0) if (backwardSlice->count(affOp) == 0)
getBackwardSliceImpl(affOp, backwardSlice, filter); getBackwardSliceImpl(affOp, backwardSlice, filter);
} else if (auto loopIv = loop::getForInductionVarOwner(operand)) { } else if (auto loopIv = loop::getForInductionVarOwner(operand)) {
auto *loopOp = loopIv.getOperation(); auto *loopOp = loopIv.getOperation();
if (backwardSlice->count(loopOp) == 0) if (backwardSlice->count(loopOp) == 0)
getBackwardSliceImpl(loopOp, backwardSlice, filter); getBackwardSliceImpl(loopOp, backwardSlice, filter);
} else if (blockArg->getOwner() != } else if (blockArg.getOwner() !=
&op->getParentOfType<FuncOp>().getBody().front()) { &op->getParentOfType<FuncOp>().getBody().front()) {
op->emitError("unsupported CF for operand ") << en.index(); op->emitError("unsupported CF for operand ") << en.index();
llvm_unreachable("Unsupported control flow"); llvm_unreachable("Unsupported control flow");
} }
continue; continue;
} }
auto *op = operand->getDefiningOp(); auto *op = operand.getDefiningOp();
if (backwardSlice->count(op) == 0) { if (backwardSlice->count(op) == 0) {
getBackwardSliceImpl(op, backwardSlice, filter); getBackwardSliceImpl(op, backwardSlice, filter);
} }
} }
backwardSlice->insert(op); backwardSlice->insert(op);
} }
▲ Show 20 Lines • Show All 44 Lines • ▼ Show 20 Linesstruct DFSState {
SmallVector<Operation *, 16> topologicalCounts; SmallVector<Operation *, 16> topologicalCounts;
DenseSet<Operation *> seen; DenseSet<Operation *> seen;
}; };
} // namespace } // namespace
static void DFSPostorder(Operation *current, DFSState *state) { static void DFSPostorder(Operation *current, DFSState *state) {
assert(current->getNumResults() <= 1 && "NYI: multi-result"); assert(current->getNumResults() <= 1 && "NYI: multi-result");
if (current->getNumResults() > 0) { if (current->getNumResults() > 0) {
for (auto &u : current->getResult(0)->getUses()) { for (auto &u : current->getResult(0).getUses()) {
auto *op = u.getOwner(); auto *op = u.getOwner();
DFSPostorder(op, state); DFSPostorder(op, state);
} }
} }
bool inserted; bool inserted;
using IterTy = decltype(state->seen.begin()); using IterTy = decltype(state->seen.begin());
IterTy iter; IterTy iter;
std::tie(iter, inserted) = state->seen.insert(current); std::tie(iter, inserted) = state->seen.insert(current);
Show All 29 Lines

mlir/lib/Analysis/Utils.cpp

Show First 20 Lines • Show All 53 Lines • ▼ Show 20 LinesComputationSliceState::getAsConstraints(FlatAffineConstraints *cst) {
SmallVector<Value, 4> values(ivs); SmallVector<Value, 4> values(ivs);
// Append 'ivs' then 'operands' to 'values'. // Append 'ivs' then 'operands' to 'values'.
values.append(lbOperands[0].begin(), lbOperands[0].end()); values.append(lbOperands[0].begin(), lbOperands[0].end());
cst->reset(numDims, numSymbols, 0, values); cst->reset(numDims, numSymbols, 0, values);
// Add loop bound constraints for values which are loop IVs and equality // Add loop bound constraints for values which are loop IVs and equality
// constraints for symbols which are constants. // constraints for symbols which are constants.
for (const auto &value : values) { for (const auto &value : values) {
assert(cst->containsId(*value) && "value expected to be present"); assert(cst->containsId(value) && "value expected to be present");
if (isValidSymbol(value)) { if (isValidSymbol(value)) {
// Check if the symbol is a constant. // Check if the symbol is a constant.
if (auto cOp = dyn_cast_or_null<ConstantIndexOp>(value->getDefiningOp())) if (auto cOp = dyn_cast_or_null<ConstantIndexOp>(value.getDefiningOp()))
cst->setIdToConstant(*value, cOp.getValue()); cst->setIdToConstant(value, cOp.getValue());
} else if (auto loop = getForInductionVarOwner(value)) { } else if (auto loop = getForInductionVarOwner(value)) {
if (failed(cst->addAffineForOpDomain(loop))) if (failed(cst->addAffineForOpDomain(loop)))
return failure(); return failure();
} }
} }
// Add slices bounds on 'ivs' using maps 'lbs'/'ubs' with 'lbOperands[0]' // Add slices bounds on 'ivs' using maps 'lbs'/'ubs' with 'lbOperands[0]'
LogicalResult ret = cst->addSliceBounds(ivs, lbs, ubs, lbOperands[0]); LogicalResult ret = cst->addSliceBounds(ivs, lbs, ubs, lbOperands[0]);
assert(succeeded(ret) && assert(succeeded(ret) &&
"should not fail as we never have semi-affine slice maps"); "should not fail as we never have semi-affine slice maps");
(void)ret; (void)ret;
return success(); return success();
} }
// Clears state bounds and operand state. // Clears state bounds and operand state.
void ComputationSliceState::clearBounds() { void ComputationSliceState::clearBounds() {
lbs.clear(); lbs.clear();
ubs.clear(); ubs.clear();
lbOperands.clear(); lbOperands.clear();
ubOperands.clear(); ubOperands.clear();
} }
unsigned MemRefRegion::getRank() const { unsigned MemRefRegion::getRank() const {
return memref->getType().cast<MemRefType>().getRank(); return memref.getType().cast<MemRefType>().getRank();
} }
Optional<int64_t> MemRefRegion::getConstantBoundingSizeAndShape( Optional<int64_t> MemRefRegion::getConstantBoundingSizeAndShape(
SmallVectorImpl<int64_t> *shape, std::vector<SmallVector<int64_t, 4>> *lbs, SmallVectorImpl<int64_t> *shape, std::vector<SmallVector<int64_t, 4>> *lbs,
SmallVectorImpl<int64_t> *lbDivisors) const { SmallVectorImpl<int64_t> *lbDivisors) const {
auto memRefType = memref->getType().cast<MemRefType>(); auto memRefType = memref.getType().cast<MemRefType>();
unsigned rank = memRefType.getRank(); unsigned rank = memRefType.getRank();
if (shape) if (shape)
shape->reserve(rank); shape->reserve(rank);
assert(rank == cst.getNumDimIds() && "inconsistent memref region"); assert(rank == cst.getNumDimIds() && "inconsistent memref region");
// Find a constant upper bound on the extent of this memref region along each // Find a constant upper bound on the extent of this memref region along each
// dimension. // dimension.
▲ Show 20 Lines • Show All 117 Lines • ▼ Show 20 Lines if (auto loop = getForInductionVarOwner(operand)) {
// conditionals will be handled when the latter supports it. // conditionals will be handled when the latter supports it.
if (failed(cst.addAffineForOpDomain(loop))) if (failed(cst.addAffineForOpDomain(loop)))
return failure(); return failure();
} else { } else {
// Has to be a valid symbol. // Has to be a valid symbol.
auto symbol = operand; auto symbol = operand;
assert(isValidSymbol(symbol)); assert(isValidSymbol(symbol));
// Check if the symbol is a constant. // Check if the symbol is a constant.
if (auto *op = symbol->getDefiningOp()) { if (auto *op = symbol.getDefiningOp()) {
if (auto constOp = dyn_cast<ConstantIndexOp>(op)) { if (auto constOp = dyn_cast<ConstantIndexOp>(op)) {
cst.setIdToConstant(*symbol, constOp.getValue()); cst.setIdToConstant(symbol, constOp.getValue());
} }
} }
} }
} }
// Add lower/upper bounds on loop IVs using bounds from 'sliceState'. // Add lower/upper bounds on loop IVs using bounds from 'sliceState'.
if (sliceState != nullptr) { if (sliceState != nullptr) {
// Add dim and symbol slice operands. // Add dim and symbol slice operands.
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 Lines cst.constantFoldIdRange(/*pos=*/cst.getNumDimIds(),
/*num=*/cst.getNumSymbolIds()); /*num=*/cst.getNumSymbolIds());
assert(cst.getNumDimIds() == rank && "unexpected MemRefRegion format"); assert(cst.getNumDimIds() == rank && "unexpected MemRefRegion format");
// Add upper/lower bounds for each memref dimension with static size // Add upper/lower bounds for each memref dimension with static size
// to guard against potential over-approximation from projection. // to guard against potential over-approximation from projection.
// TODO(andydavis) Support dynamic memref dimensions. // TODO(andydavis) Support dynamic memref dimensions.
if (addMemRefDimBounds) { if (addMemRefDimBounds) {
auto memRefType = memref->getType().cast<MemRefType>(); auto memRefType = memref.getType().cast<MemRefType>();
for (unsigned r = 0; r < rank; r++) { for (unsigned r = 0; r < rank; r++) {
cst.addConstantLowerBound(r, 0); cst.addConstantLowerBound(r, 0);
int64_t dimSize = memRefType.getDimSize(r); int64_t dimSize = memRefType.getDimSize(r);
if (ShapedType::isDynamic(dimSize)) if (ShapedType::isDynamic(dimSize))
continue; continue;
cst.addConstantUpperBound(r, dimSize - 1); cst.addConstantUpperBound(r, dimSize - 1);
} }
} }
Show All 15 Lines if (elementType.isIntOrFloat()) {
sizeInBits = sizeInBits =
vectorType.getElementTypeBitWidth() * vectorType.getNumElements(); vectorType.getElementTypeBitWidth() * vectorType.getNumElements();
} }
return llvm::divideCeil(sizeInBits, 8); return llvm::divideCeil(sizeInBits, 8);
} }
// Returns the size of the region. // Returns the size of the region.
Optional<int64_t> MemRefRegion::getRegionSize() { Optional<int64_t> MemRefRegion::getRegionSize() {
auto memRefType = memref->getType().cast<MemRefType>(); auto memRefType = memref.getType().cast<MemRefType>();
auto layoutMaps = memRefType.getAffineMaps(); auto layoutMaps = memRefType.getAffineMaps();
if (layoutMaps.size() > 1 || if (layoutMaps.size() > 1 ||
(layoutMaps.size() == 1 && !layoutMaps[0].isIdentity())) { (layoutMaps.size() == 1 && !layoutMaps[0].isIdentity())) {
LLVM_DEBUG(llvm::dbgs() << "Non-identity layout map not yet supported\n"); LLVM_DEBUG(llvm::dbgs() << "Non-identity layout map not yet supported\n");
return false; return false;
} }
▲ Show 20 Lines • Show All 512 Lines • ▼ Show 20 Lines if (auto loadOp = dyn_cast<AffineLoadOp>(loadOrStoreOpInst)) {
indices.reserve(storeMemrefType.getRank()); indices.reserve(storeMemrefType.getRank());
for (auto index : storeOp.getMapOperands()) { for (auto index : storeOp.getMapOperands()) {
indices.push_back(index); indices.push_back(index);
} }
} }
} }
unsigned MemRefAccess::getRank() const { unsigned MemRefAccess::getRank() const {
return memref->getType().cast<MemRefType>().getRank(); return memref.getType().cast<MemRefType>().getRank();
} }
bool MemRefAccess::isStore() const { return isa<AffineStoreOp>(opInst); } bool MemRefAccess::isStore() const { return isa<AffineStoreOp>(opInst); }
/// Returns the nesting depth of this statement, i.e., the number of loops /// Returns the nesting depth of this statement, i.e., the number of loops
/// surrounding this statement. /// surrounding this statement.
unsigned mlir::getNestingDepth(Operation &op) { unsigned mlir::getNestingDepth(Operation &op) {
Operation *currOp = &op; Operation *currOp = &op;
▲ Show 20 Lines • Show All 142 Lines • Show Last 20 Lines

mlir/lib/Analysis/VectorAnalysis.cpp

Show First 20 Lines • Show All 192 Lines • ▼ Show 20 Lines if (auto read = dyn_cast<vector::TransferReadOp>(op)) {
mustDivide = true; mustDivide = true;
} else if (op.getNumResults() == 0) { } else if (op.getNumResults() == 0) {
if (!isa<ReturnOp>(op)) { if (!isa<ReturnOp>(op)) {
op.emitError("NYI: assuming only return operations can have 0 " op.emitError("NYI: assuming only return operations can have 0 "
" results at this point"); " results at this point");
} }
return false; return false;
} else if (op.getNumResults() == 1) { } else if (op.getNumResults() == 1) {
if (auto v = op.getResult(0)->getType().dyn_cast<VectorType>()) { if (auto v = op.getResult(0).getType().dyn_cast<VectorType>()) {
superVectorType = v; superVectorType = v;
} else { } else {
// Not a vector type. // Not a vector type.
return false; return false;
} }
} else { } else {
// Not a vector.transfer and has more than 1 result, fail hard for now to // Not a vector.transfer and has more than 1 result, fail hard for now to
// wake us up when something changes. // wake us up when something changes.
Show All 23 Lines

mlir/lib/Analysis/Verifier.cpp

Show First 20 Lines • Show All 124 Lines • ▼ Show 20 LinesLogicalResult OperationVerifier::verifyRegion(Region &region) {
for (auto &block : region) for (auto &block : region)
if (failed(verifyBlock(block))) if (failed(verifyBlock(block)))
return failure(); return failure();
return success(); return success();
} }
LogicalResult OperationVerifier::verifyBlock(Block &block) { LogicalResult OperationVerifier::verifyBlock(Block &block) {
for (auto arg : block.getArguments()) for (auto arg : block.getArguments())
if (arg->getOwner() != &block) if (arg.getOwner() != &block)
return emitError(block, "block argument not owned by block"); return emitError(block, "block argument not owned by block");
// Verify that this block has a terminator. // Verify that this block has a terminator.
if (block.empty()) if (block.empty())
return emitError(block, "block with no terminator"); return emitError(block, "block with no terminator");
// Verify the non-terminator operations separately so that we can verify // Verify the non-terminator operations separately so that we can verify
// they has no successors. // they has no successors.
▲ Show 20 Lines • Show All 94 Lines • ▼ Show 20 LinesLogicalResult OperationVerifier::verifyDominance(Operation &op) {
for (unsigned operandNo = 0, e = op.getNumOperands(); operandNo != e; for (unsigned operandNo = 0, e = op.getNumOperands(); operandNo != e;
++operandNo) { ++operandNo) {
auto operand = op.getOperand(operandNo); auto operand = op.getOperand(operandNo);
if (domInfo->properlyDominates(operand, &op)) if (domInfo->properlyDominates(operand, &op))
continue; continue;
auto diag = op.emitError("operand #") auto diag = op.emitError("operand #")
<< operandNo << " does not dominate this use"; << operandNo << " does not dominate this use";
if (auto *useOp = operand->getDefiningOp()) if (auto *useOp = operand.getDefiningOp())
diag.attachNote(useOp->getLoc()) << "operand defined here"; diag.attachNote(useOp->getLoc()) << "operand defined here";
return failure(); return failure();
} }
// Verify the dominance of each of the nested blocks within this operation. // Verify the dominance of each of the nested blocks within this operation.
for (auto &region : op.getRegions()) for (auto &region : op.getRegions())
if (failed(verifyDominance(region))) if (failed(verifyDominance(region)))
return failure(); return failure();
Show All 14 Lines

mlir/lib/Conversion/GPUCommon/OpToFuncCallLowering.h

Show All 38 Lines matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
using LLVM::LLVMFuncOp; using LLVM::LLVMFuncOp;
using LLVM::LLVMType; using LLVM::LLVMType;
static_assert( static_assert(
std::is_base_of<OpTrait::OneResult<SourceOp>, SourceOp>::value, std::is_base_of<OpTrait::OneResult<SourceOp>, SourceOp>::value,
"expected single result op"); "expected single result op");
LLVMType resultType = lowering.convertType(op->getResult(0)->getType()) LLVMType resultType = lowering.convertType(op->getResult(0).getType())
.template cast<LLVM::LLVMType>(); .template cast<LLVM::LLVMType>();
LLVMType funcType = getFunctionType(resultType, operands); LLVMType funcType = getFunctionType(resultType, operands);
StringRef funcName = getFunctionName(resultType); StringRef funcName = getFunctionName(resultType);
if (funcName.empty()) if (funcName.empty())
return matchFailure(); return matchFailure();
LLVMFuncOp funcOp = appendOrGetFuncOp(funcName, funcType, op); LLVMFuncOp funcOp = appendOrGetFuncOp(funcName, funcType, op);
auto callOp = rewriter.create<LLVM::CallOp>( auto callOp = rewriter.create<LLVM::CallOp>(
op->getLoc(), resultType, rewriter.getSymbolRefAttr(funcOp), operands); op->getLoc(), resultType, rewriter.getSymbolRefAttr(funcOp), operands);
rewriter.replaceOp(op, {callOp.getResult(0)}); rewriter.replaceOp(op, {callOp.getResult(0)});
return matchSuccess(); return matchSuccess();
} }
private: private:
LLVM::LLVMType getFunctionType(LLVM::LLVMType resultType, LLVM::LLVMType getFunctionType(LLVM::LLVMType resultType,
ArrayRef<Value> operands) const { ArrayRef<Value> operands) const {
using LLVM::LLVMType; using LLVM::LLVMType;
SmallVector<LLVMType, 1> operandTypes; SmallVector<LLVMType, 1> operandTypes;
for (Value operand : operands) { for (Value operand : operands) {
operandTypes.push_back(operand->getType().cast<LLVMType>()); operandTypes.push_back(operand.getType().cast<LLVMType>());
} }
return LLVMType::getFunctionTy(resultType, operandTypes, return LLVMType::getFunctionTy(resultType, operandTypes,
/*isVarArg=*/false); /*isVarArg=*/false);
} }
StringRef getFunctionName(LLVM::LLVMType type) const { StringRef getFunctionName(LLVM::LLVMType type) const {
if (type.isFloatTy()) if (type.isFloatTy())
return f32Func; return f32Func;
Show All 25 Lines

mlir/lib/Conversion/GPUToCUDA/ConvertLaunchFuncToCudaCalls.cpp

Show First 20 Lines • Show All 247 Lines • ▼ Show 20 LinesValue GpuLaunchFuncToCudaCallsPass::setupParamsArray(gpu::LaunchFuncOp launchOp,
// Provision twice as much for the `array` to allow up to one level of // Provision twice as much for the `array` to allow up to one level of
// indirection for each argument. // indirection for each argument.
auto arraySize = builder.create<LLVM::ConstantOp>( auto arraySize = builder.create<LLVM::ConstantOp>(
loc, getInt32Type(), builder.getI32IntegerAttr(numKernelOperands)); loc, getInt32Type(), builder.getI32IntegerAttr(numKernelOperands));
auto array = builder.create<LLVM::AllocaOp>(loc, getPointerPointerType(), auto array = builder.create<LLVM::AllocaOp>(loc, getPointerPointerType(),
arraySize, /*alignment=*/0); arraySize, /*alignment=*/0);
for (unsigned idx = 0; idx < numKernelOperands; ++idx) { for (unsigned idx = 0; idx < numKernelOperands; ++idx) {
auto operand = launchOp.getKernelOperand(idx); auto operand = launchOp.getKernelOperand(idx);
auto llvmType = operand->getType().cast<LLVM::LLVMType>(); auto llvmType = operand.getType().cast<LLVM::LLVMType>();
Value memLocation = builder.create<LLVM::AllocaOp>( Value memLocation = builder.create<LLVM::AllocaOp>(
loc, llvmType.getPointerTo(), one, /*alignment=*/1); loc, llvmType.getPointerTo(), one, /*alignment=*/1);
builder.create<LLVM::StoreOp>(loc, operand, memLocation); builder.create<LLVM::StoreOp>(loc, operand, memLocation);
auto casted = auto casted =
builder.create<LLVM::BitcastOp>(loc, getPointerType(), memLocation); builder.create<LLVM::BitcastOp>(loc, getPointerType(), memLocation);
// Assume all struct arguments come from MemRef. If this assumption does not // Assume all struct arguments come from MemRef. If this assumption does not
// hold anymore then we `launchOp` to lower from MemRefType and not after // hold anymore then we `launchOp` to lower from MemRefType and not after
▲ Show 20 Lines • Show All 160 Lines • Show Last 20 Lines

mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp

Show First 20 Lines • Show All 60 Lines • ▼ Show 20 Linesstruct GPUAllReduceOpLowering : public LLVMOpLowering {
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
Location loc = op->getLoc(); Location loc = op->getLoc();
Value operand = operands.front(); Value operand = operands.front();
// TODO(csigg): Generalize to other types of accumulation. // TODO(csigg): Generalize to other types of accumulation.
assert(op->getOperand(0)->getType().isIntOrFloat()); assert(op->getOperand(0).getType().isIntOrFloat());
// Create the reduction using an accumulator factory. // Create the reduction using an accumulator factory.
AccumulatorFactory factory = AccumulatorFactory factory =
getFactory(cast<gpu::AllReduceOp>(op), operand); getFactory(cast<gpu::AllReduceOp>(op), operand);
assert(factory && "failed to create accumulator factory"); assert(factory && "failed to create accumulator factory");
Value result = createBlockReduce(loc, operand, factory, rewriter); Value result = createBlockReduce(loc, operand, factory, rewriter);
rewriter.replaceOp(op, {result}); rewriter.replaceOp(op, {result});
return matchSuccess(); return matchSuccess();
} }
private: private:
/// Returns an accumulator factory using either the op attribute or the body /// Returns an accumulator factory using either the op attribute or the body
/// region. /// region.
AccumulatorFactory getFactory(gpu::AllReduceOp allReduce, AccumulatorFactory getFactory(gpu::AllReduceOp allReduce,
Value operand) const { Value operand) const {
if (!allReduce.body().empty()) { if (!allReduce.body().empty()) {
return getFactory(allReduce.body()); return getFactory(allReduce.body());
} }
if (allReduce.op()) { if (allReduce.op()) {
auto type = operand->getType().cast<LLVM::LLVMType>(); auto type = operand.getType().cast<LLVM::LLVMType>();
return getFactory(*allReduce.op(), type.getUnderlyingType()); return getFactory(*allReduce.op(), type.getUnderlyingType());
} }
return AccumulatorFactory(); return AccumulatorFactory();
} }
/// Returns an accumulator factory that clones the body. The body's entry /// Returns an accumulator factory that clones the body. The body's entry
/// block is expected to have 2 arguments. The gpu.yield return the /// block is expected to have 2 arguments. The gpu.yield return the
/// accumulated value of the same type. /// accumulated value of the same type.
Show All 23 Lines return AccumulatorFactory([&](Location loc, Value lhs, Value rhs,
rewriter.setInsertionPointToEnd(block); rewriter.setInsertionPointToEnd(block);
rewriter.replaceOpWithNewOp<LLVM::BrOp>( rewriter.replaceOpWithNewOp<LLVM::BrOp>(
terminator, ArrayRef<Value>{}, llvm::makeArrayRef(split), terminator, ArrayRef<Value>{}, llvm::makeArrayRef(split),
ValueRange(terminator->getOperand(0))); ValueRange(terminator->getOperand(0)));
} }
// Return accumulator result. // Return accumulator result.
rewriter.setInsertionPointToStart(split); rewriter.setInsertionPointToStart(split);
return split->addArgument(lhs->getType()); return split->addArgument(lhs.getType());
}); });
} }
/// Returns an accumulator factory that creates an op specified by opName. /// Returns an accumulator factory that creates an op specified by opName.
AccumulatorFactory getFactory(StringRef opName, llvm::Type *type) const { AccumulatorFactory getFactory(StringRef opName, llvm::Type *type) const {
if (type->isVectorTy() || type->isArrayTy()) if (type->isVectorTy() || type->isArrayTy())
return getFactory(opName, type->getSequentialElementType()); return getFactory(opName, type->getSequentialElementType());
Show All 10 Lines AccumulatorFactory getFactory(StringRef opName, llvm::Type *type) const {
return AccumulatorFactory(); return AccumulatorFactory();
} }
/// Returns an accumulator factory that creates an op of type T. /// Returns an accumulator factory that creates an op of type T.
template <typename T> AccumulatorFactory getFactory() const { template <typename T> AccumulatorFactory getFactory() const {
return [](Location loc, Value lhs, Value rhs, return [](Location loc, Value lhs, Value rhs,
ConversionPatternRewriter &rewriter) { ConversionPatternRewriter &rewriter) {
return rewriter.create<T>(loc, lhs->getType(), lhs, rhs); return rewriter.create<T>(loc, lhs.getType(), lhs, rhs);
}; };
} }
/// Creates an all_reduce across the block. /// Creates an all_reduce across the block.
/// ///
/// First reduce the elements within a warp. The first thread of each warp /// First reduce the elements within a warp. The first thread of each warp
/// writes the intermediate result to shared memory. After synchronizing the /// writes the intermediate result to shared memory. After synchronizing the
/// block, the first warp reduces the values from shared memory. The result /// block, the first warp reduces the values from shared memory. The result
Show All 26 Linesprivate:
/// ^continue2: /// ^continue2:
/// nvvm.barrier0 /// nvvm.barrier0
/// %result = llvm.load %result_ptr /// %result = llvm.load %result_ptr
/// return %result /// return %result
/// ///
Value createBlockReduce(Location loc, Value operand, Value createBlockReduce(Location loc, Value operand,
AccumulatorFactory &accumFactory, AccumulatorFactory &accumFactory,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
auto type = operand->getType().cast<LLVM::LLVMType>(); auto type = operand.getType().cast<LLVM::LLVMType>();
// Create shared memory array to store the warp reduction. // Create shared memory array to store the warp reduction.
auto module = operand->getDefiningOp()->getParentOfType<ModuleOp>(); auto module = operand.getDefiningOp()->getParentOfType<ModuleOp>();
assert(module && "op must belong to a module"); assert(module && "op must belong to a module");
Value sharedMemPtr = Value sharedMemPtr =
createSharedMemoryArray(loc, module, type, kWarpSize, rewriter); createSharedMemoryArray(loc, module, type, kWarpSize, rewriter);
Value zero = rewriter.create<LLVM::ConstantOp>( Value zero = rewriter.create<LLVM::ConstantOp>(
loc, int32Type, rewriter.getI32IntegerAttr(0u)); loc, int32Type, rewriter.getI32IntegerAttr(0u));
Value laneId = rewriter.create<NVVM::LaneIdOp>(loc, int32Type); Value laneId = rewriter.create<NVVM::LaneIdOp>(loc, int32Type);
Value isFirstLane = rewriter.create<LLVM::ICmpOp>( Value isFirstLane = rewriter.create<LLVM::ICmpOp>(
▲ Show 20 Lines • Show All 78 Lines • ▼ Show 20 Lines void createIf(Location loc, ConversionPatternRewriter &rewriter,
rewriter.setInsertionPointToStart(elseBlock); rewriter.setInsertionPointToStart(elseBlock);
auto elseOperands = elseOpsFactory(); auto elseOperands = elseOpsFactory();
addBranch(elseOperands); addBranch(elseOperands);
assert(thenOperands.size() == elseOperands.size()); assert(thenOperands.size() == elseOperands.size());
rewriter.setInsertionPointToStart(continueBlock); rewriter.setInsertionPointToStart(continueBlock);
for (auto operand : thenOperands) for (auto operand : thenOperands)
continueBlock->addArgument(operand->getType()); continueBlock->addArgument(operand.getType());
} }
/// Shortcut for createIf with empty else block and no block operands. /// Shortcut for createIf with empty else block and no block operands.
template <typename Factory> template <typename Factory>
void createPredicatedBlock(Location loc, ConversionPatternRewriter &rewriter, void createPredicatedBlock(Location loc, ConversionPatternRewriter &rewriter,
Value condition, Value condition,
Factory &&predicatedOpsFactory) const { Factory &&predicatedOpsFactory) const {
createIf( createIf(
Show All 9 Linesprivate:
/// The first lane returns the result, all others return values are undefined. /// The first lane returns the result, all others return values are undefined.
Value createWarpReduce(Location loc, Value activeWidth, Value laneId, Value createWarpReduce(Location loc, Value activeWidth, Value laneId,
Value operand, AccumulatorFactory accumFactory, Value operand, AccumulatorFactory accumFactory,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
Value warpSize = rewriter.create<LLVM::ConstantOp>( Value warpSize = rewriter.create<LLVM::ConstantOp>(
loc, int32Type, rewriter.getI32IntegerAttr(kWarpSize)); loc, int32Type, rewriter.getI32IntegerAttr(kWarpSize));
Value isPartialWarp = rewriter.create<LLVM::ICmpOp>( Value isPartialWarp = rewriter.create<LLVM::ICmpOp>(
loc, LLVM::ICmpPredicate::slt, activeWidth, warpSize); loc, LLVM::ICmpPredicate::slt, activeWidth, warpSize);
auto type = operand->getType().cast<LLVM::LLVMType>(); auto type = operand.getType().cast<LLVM::LLVMType>();
createIf( createIf(
loc, rewriter, isPartialWarp, loc, rewriter, isPartialWarp,
// Generate reduction over a (potentially) partial warp. // Generate reduction over a (potentially) partial warp.
[&] { [&] {
Value value = operand; Value value = operand;
Value one = rewriter.create<LLVM::ConstantOp>( Value one = rewriter.create<LLVM::ConstantOp>(
loc, int32Type, rewriter.getI32IntegerAttr(1)); loc, int32Type, rewriter.getI32IntegerAttr(1));
▲ Show 20 Lines • Show All 115 Lines • ▼ Show 20 Lines Value numThreadsWithSmallerWarpId =
rewriter.create<LLVM::AndOp>(loc, threadIdx, threadIdxMask); rewriter.create<LLVM::AndOp>(loc, threadIdx, threadIdxMask);
return rewriter.create<LLVM::SubOp>(loc, blockSize, return rewriter.create<LLVM::SubOp>(loc, blockSize,
numThreadsWithSmallerWarpId); numThreadsWithSmallerWarpId);
} }
/// Returns value divided by the warp size (i.e. 32). /// Returns value divided by the warp size (i.e. 32).
Value getDivideByWarpSize(Value value, Value getDivideByWarpSize(Value value,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
auto loc = value->getLoc(); auto loc = value.getLoc();
auto warpSize = rewriter.create<LLVM::ConstantOp>( auto warpSize = rewriter.create<LLVM::ConstantOp>(
loc, int32Type, rewriter.getI32IntegerAttr(kWarpSize)); loc, int32Type, rewriter.getI32IntegerAttr(kWarpSize));
return rewriter.create<LLVM::SDivOp>(loc, int32Type, value, warpSize); return rewriter.create<LLVM::SDivOp>(loc, int32Type, value, warpSize);
} }
LLVM::LLVMType int32Type; LLVM::LLVMType int32Type;
static constexpr int kWarpSize = 32; static constexpr int kWarpSize = 32;
Show All 22 Linesstruct GPUShuffleOpLowering : public LLVMOpLowering {
/// !llvm<"{ float, i1 }"> /// !llvm<"{ float, i1 }">
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
Location loc = op->getLoc(); Location loc = op->getLoc();
gpu::ShuffleOpOperandAdaptor adaptor(operands); gpu::ShuffleOpOperandAdaptor adaptor(operands);
auto dialect = lowering.getDialect(); auto dialect = lowering.getDialect();
auto valueTy = adaptor.value()->getType().cast<LLVM::LLVMType>(); auto valueTy = adaptor.value().getType().cast<LLVM::LLVMType>();
auto int32Type = LLVM::LLVMType::getInt32Ty(dialect); auto int32Type = LLVM::LLVMType::getInt32Ty(dialect);
auto predTy = LLVM::LLVMType::getInt1Ty(dialect); auto predTy = LLVM::LLVMType::getInt1Ty(dialect);
auto resultTy = LLVM::LLVMType::getStructTy(dialect, {valueTy, predTy}); auto resultTy = LLVM::LLVMType::getStructTy(dialect, {valueTy, predTy});
Value one = rewriter.create<LLVM::ConstantOp>( Value one = rewriter.create<LLVM::ConstantOp>(
loc, int32Type, rewriter.getI32IntegerAttr(1)); loc, int32Type, rewriter.getI32IntegerAttr(1));
// Bit mask of active lanes: `(1 << activeWidth) - 1`. // Bit mask of active lanes: `(1 << activeWidth) - 1`.
Value activeMask = rewriter.create<LLVM::SubOp>( Value activeMask = rewriter.create<LLVM::SubOp>(
Show All 31 Lines matchAndRewrite(Operation *op, ArrayRef<Value> operands,
auto gpuFuncOp = cast<gpu::GPUFuncOp>(op); auto gpuFuncOp = cast<gpu::GPUFuncOp>(op);
Location loc = gpuFuncOp.getLoc(); Location loc = gpuFuncOp.getLoc();
SmallVector<LLVM::GlobalOp, 3> workgroupBuffers; SmallVector<LLVM::GlobalOp, 3> workgroupBuffers;
workgroupBuffers.reserve(gpuFuncOp.getNumWorkgroupAttributions()); workgroupBuffers.reserve(gpuFuncOp.getNumWorkgroupAttributions());
for (auto en : llvm::enumerate(gpuFuncOp.getWorkgroupAttributions())) { for (auto en : llvm::enumerate(gpuFuncOp.getWorkgroupAttributions())) {
Value attribution = en.value(); Value attribution = en.value();
auto type = attribution->getType().dyn_cast<MemRefType>(); auto type = attribution.getType().dyn_cast<MemRefType>();
assert(type && type.hasStaticShape() && "unexpected type in attribution"); assert(type && type.hasStaticShape() && "unexpected type in attribution");
uint64_t numElements = type.getNumElements(); uint64_t numElements = type.getNumElements();
auto elementType = auto elementType =
lowering.convertType(type.getElementType()).cast<LLVM::LLVMType>(); lowering.convertType(type.getElementType()).cast<LLVM::LLVMType>();
auto arrayType = LLVM::LLVMType::getArrayTy(elementType, numElements); auto arrayType = LLVM::LLVMType::getArrayTy(elementType, numElements);
std::string name = std::string name =
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Lines auto llvmFuncOp = rewriter.create<LLVM::LLVMFuncOp>(
loc, elementType.getPointerTo(global.addr_space().getZExtValue()), loc, elementType.getPointerTo(global.addr_space().getZExtValue()),
address, ArrayRef<Value>{zero, zero}); address, ArrayRef<Value>{zero, zero});
// Build a memref descriptor pointing to the buffer to plug with the // Build a memref descriptor pointing to the buffer to plug with the
// existing memref infrastructure. This may use more registers than // existing memref infrastructure. This may use more registers than
// otherwise necessary given that memref sizes are fixed, but we can try // otherwise necessary given that memref sizes are fixed, but we can try
// and canonicalize that away later. // and canonicalize that away later.
Value attribution = gpuFuncOp.getWorkgroupAttributions()[en.index()]; Value attribution = gpuFuncOp.getWorkgroupAttributions()[en.index()];
auto type = attribution->getType().cast<MemRefType>(); auto type = attribution.getType().cast<MemRefType>();
auto descr = MemRefDescriptor::fromStaticShape(rewriter, loc, lowering, auto descr = MemRefDescriptor::fromStaticShape(rewriter, loc, lowering,
type, memory); type, memory);
signatureConversion.remapInput(numProperArguments + en.index(), descr); signatureConversion.remapInput(numProperArguments + en.index(), descr);
} }
// Rewrite private memory attributions to alloca'ed buffers. // Rewrite private memory attributions to alloca'ed buffers.
unsigned numWorkgroupAttributions = unsigned numWorkgroupAttributions =
gpuFuncOp.getNumWorkgroupAttributions(); gpuFuncOp.getNumWorkgroupAttributions();
auto int64Ty = LLVM::LLVMType::getInt64Ty(lowering.getDialect()); auto int64Ty = LLVM::LLVMType::getInt64Ty(lowering.getDialect());
for (auto en : llvm::enumerate(gpuFuncOp.getPrivateAttributions())) { for (auto en : llvm::enumerate(gpuFuncOp.getPrivateAttributions())) {
Value attribution = en.value(); Value attribution = en.value();
auto type = attribution->getType().cast<MemRefType>(); auto type = attribution.getType().cast<MemRefType>();
assert(type && type.hasStaticShape() && assert(type && type.hasStaticShape() &&
"unexpected type in attribution"); "unexpected type in attribution");
// Explicitly drop memory space when lowering private memory // Explicitly drop memory space when lowering private memory
// attributions since NVVM models it as `alloca`s in the default // attributions since NVVM models it as `alloca`s in the default
// memory space and does not support `alloca`s with addrspace(5). // memory space and does not support `alloca`s with addrspace(5).
auto ptrType = lowering.convertType(type.getElementType()) auto ptrType = lowering.convertType(type.getElementType())
.cast<LLVM::LLVMType>() .cast<LLVM::LLVMType>()
▲ Show 20 Lines • Show All 116 Lines • Show Last 20 Lines

mlir/lib/Conversion/GPUToSPIRV/ConvertGPUToSPIRV.cpp

Show First 20 Lines • Show All 121 Lines • ▼ Show 20 LinesForOpConversion::matchAndRewrite(loop::ForOp forOp, ArrayRef<Value> operands,
OpBuilder::InsertionGuard guard(rewriter); OpBuilder::InsertionGuard guard(rewriter);
// Create the block for the header. // Create the block for the header.
auto header = new Block(); auto header = new Block();
// Insert the header. // Insert the header.
loopOp.body().getBlocks().insert(std::next(loopOp.body().begin(), 1), header); loopOp.body().getBlocks().insert(std::next(loopOp.body().begin(), 1), header);
// Create the new induction variable to use. // Create the new induction variable to use.
BlockArgument newIndVar = BlockArgument newIndVar =
header->addArgument(forOperands.lowerBound()->getType()); header->addArgument(forOperands.lowerBound().getType());
Block *body = forOp.getBody(); Block *body = forOp.getBody();
// Apply signature conversion to the body of the forOp. It has a single block, // Apply signature conversion to the body of the forOp. It has a single block,
// with argument which is the induction variable. That has to be replaced with // with argument which is the induction variable. That has to be replaced with
// the new induction variable. // the new induction variable.
TypeConverter::SignatureConversion signatureConverter( TypeConverter::SignatureConversion signatureConverter(
body->getNumArguments()); body->getNumArguments());
signatureConverter.remapInput(0, newIndVar); signatureConverter.remapInput(0, newIndVar);
Show All 22 LinesForOpConversion::matchAndRewrite(loop::ForOp forOp, ArrayRef<Value> operands,
// Generate instructions to increment the step of the induction variable and // Generate instructions to increment the step of the induction variable and
// branch to the header. // branch to the header.
Block *continueBlock = loopOp.getContinueBlock(); Block *continueBlock = loopOp.getContinueBlock();
rewriter.setInsertionPointToEnd(continueBlock); rewriter.setInsertionPointToEnd(continueBlock);
// Add the step to the induction variable and branch to the header. // Add the step to the induction variable and branch to the header.
Value updatedIndVar = rewriter.create<spirv::IAddOp>( Value updatedIndVar = rewriter.create<spirv::IAddOp>(
loc, newIndVar->getType(), newIndVar, forOperands.step()); loc, newIndVar.getType(), newIndVar, forOperands.step());
rewriter.create<spirv::BranchOp>(loc, header, updatedIndVar); rewriter.create<spirv::BranchOp>(loc, header, updatedIndVar);
rewriter.eraseOp(forOp); rewriter.eraseOp(forOp);
return matchSuccess(); return matchSuccess();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Builtins. // Builtins.
▲ Show 20 Lines • Show All 182 Lines • Show Last 20 Lines

mlir/lib/Conversion/LinalgToLLVM/LinalgToLLVM.cpp

Show First 20 Lines • Show All 146 Lines • ▼ Show 20 Linespublic:
explicit RangeOpConversion(MLIRContext *context, LLVMTypeConverter &lowering_) explicit RangeOpConversion(MLIRContext *context, LLVMTypeConverter &lowering_)
: LLVMOpLowering(RangeOp::getOperationName(), context, lowering_) {} : LLVMOpLowering(RangeOp::getOperationName(), context, lowering_) {}
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto rangeOp = cast<RangeOp>(op); auto rangeOp = cast<RangeOp>(op);
auto rangeDescriptorTy = auto rangeDescriptorTy =
convertLinalgType(rangeOp.getResult()->getType(), lowering); convertLinalgType(rangeOp.getResult().getType(), lowering);
edsc::ScopedContext context(rewriter, op->getLoc()); edsc::ScopedContext context(rewriter, op->getLoc());
// Fill in an aggregate value of the descriptor. // Fill in an aggregate value of the descriptor.
RangeOpOperandAdaptor adaptor(operands); RangeOpOperandAdaptor adaptor(operands);
Value desc = llvm_undef(rangeDescriptorTy); Value desc = llvm_undef(rangeDescriptorTy);
desc = insertvalue(desc, adaptor.min(), rewriter.getI64ArrayAttr(0)); desc = insertvalue(desc, adaptor.min(), rewriter.getI64ArrayAttr(0));
desc = insertvalue(desc, adaptor.max(), rewriter.getI64ArrayAttr(1)); desc = insertvalue(desc, adaptor.max(), rewriter.getI64ArrayAttr(1));
▲ Show 20 Lines • Show All 82 Lines • ▼ Show 20 Lines auto pos = [&rewriter](ArrayRef<int64_t> values) {
return rewriter.getI64ArrayAttr(values); return rewriter.getI64ArrayAttr(values);
}; };
// Compute base offset. // Compute base offset.
Value baseOffset = baseDesc.offset(); Value baseOffset = baseDesc.offset();
for (int i = 0, e = memRefType.getRank(); i < e; ++i) { for (int i = 0, e = memRefType.getRank(); i < e; ++i) {
Value indexing = adaptor.indexings()[i]; Value indexing = adaptor.indexings()[i];
Value min = indexing; Value min = indexing;
if (sliceOp.indexing(i)->getType().isa<RangeType>()) if (sliceOp.indexing(i).getType().isa<RangeType>())
min = extractvalue(int64Ty, indexing, pos(0)); min = extractvalue(int64Ty, indexing, pos(0));
baseOffset = add(baseOffset, mul(min, strides[i])); baseOffset = add(baseOffset, mul(min, strides[i]));
} }
// Insert the base and aligned pointers. // Insert the base and aligned pointers.
desc.setAllocatedPtr(baseDesc.allocatedPtr()); desc.setAllocatedPtr(baseDesc.allocatedPtr());
desc.setAlignedPtr(baseDesc.alignedPtr()); desc.setAlignedPtr(baseDesc.alignedPtr());
// Insert base offset. // Insert base offset.
desc.setOffset(baseOffset); desc.setOffset(baseOffset);
// Corner case, no sizes or strides: early return the descriptor. // Corner case, no sizes or strides: early return the descriptor.
if (sliceOp.getShapedType().getRank() == 0) if (sliceOp.getShapedType().getRank() == 0)
return rewriter.replaceOp(op, {desc}), matchSuccess(); return rewriter.replaceOp(op, {desc}), matchSuccess();
Value zero = Value zero =
constant(int64Ty, rewriter.getIntegerAttr(rewriter.getIndexType(), 0)); constant(int64Ty, rewriter.getIntegerAttr(rewriter.getIndexType(), 0));
// Compute and insert view sizes (max - min along the range) and strides. // Compute and insert view sizes (max - min along the range) and strides.
// Skip the non-range operands as they will be projected away from the view. // Skip the non-range operands as they will be projected away from the view.
int numNewDims = 0; int numNewDims = 0;
for (auto en : llvm::enumerate(sliceOp.indexings())) { for (auto en : llvm::enumerate(sliceOp.indexings())) {
Value indexing = en.value(); Value indexing = en.value();
if (indexing->getType().isa<RangeType>()) { if (indexing.getType().isa<RangeType>()) {
int rank = en.index(); int rank = en.index();
Value rangeDescriptor = adaptor.indexings()[rank]; Value rangeDescriptor = adaptor.indexings()[rank];
Value min = extractvalue(int64Ty, rangeDescriptor, pos(0)); Value min = extractvalue(int64Ty, rangeDescriptor, pos(0));
Value max = extractvalue(int64Ty, rangeDescriptor, pos(1)); Value max = extractvalue(int64Ty, rangeDescriptor, pos(1));
Value step = extractvalue(int64Ty, rangeDescriptor, pos(2)); Value step = extractvalue(int64Ty, rangeDescriptor, pos(2));
Value baseSize = baseDesc.size(rank); Value baseSize = baseDesc.size(rank);
// Bound upper by base view upper bound. // Bound upper by base view upper bound.
▲ Show 20 Lines • Show All 313 Lines • Show Last 20 Lines

mlir/lib/Conversion/LoopsToGPU/LoopsToGPU.cpp

Show First 20 Lines • Show All 209 Lines • ▼ Show 20 Linesstruct LoopToGpuConverter {
SmallVector<Value, 6> ivs; SmallVector<Value, 6> ivs;
// Steps of the loops mapped to blocks or threads. // Steps of the loops mapped to blocks or threads.
SmallVector<Value, 6> steps; SmallVector<Value, 6> steps;
}; };
} // namespace } // namespace
// Return true if the value is obviously a constant "one". // Return true if the value is obviously a constant "one".
static bool isConstantOne(Value value) { static bool isConstantOne(Value value) {
if (auto def = dyn_cast_or_null<ConstantIndexOp>(value->getDefiningOp())) if (auto def = dyn_cast_or_null<ConstantIndexOp>(value.getDefiningOp()))
return def.getValue() == 1; return def.getValue() == 1;
return false; return false;
} }
// Collect ranges, bounds, steps and induction variables in preparation for // Collect ranges, bounds, steps and induction variables in preparation for
// mapping a loop nest of depth "numLoops" rooted at "forOp" to a GPU kernel. // mapping a loop nest of depth "numLoops" rooted at "forOp" to a GPU kernel.
// This may fail if the IR for computing loop bounds cannot be constructed, for // This may fail if the IR for computing loop bounds cannot be constructed, for
// example if an affine loop uses semi-affine maps. Return the last loop to be // example if an affine loop uses semi-affine maps. Return the last loop to be
▲ Show 20 Lines • Show All 225 Lines • ▼ Show 20 Lines Value id =
? getDim3Value(launchOp.getBlockIds(), en.index()) ? getDim3Value(launchOp.getBlockIds(), en.index())
: getDim3Value(launchOp.getThreadIds(), en.index() - numBlockDims); : getDim3Value(launchOp.getThreadIds(), en.index() - numBlockDims);
Value step = steps[en.index()]; Value step = steps[en.index()];
if (!isConstantOne(step)) if (!isConstantOne(step))
id = builder.create<MulIOp>(rootForOp.getLoc(), step, id); id = builder.create<MulIOp>(rootForOp.getLoc(), step, id);
Value ivReplacement = Value ivReplacement =
builder.create<AddIOp>(rootForOp.getLoc(), *lbArgumentIt, id); builder.create<AddIOp>(rootForOp.getLoc(), *lbArgumentIt, id);
en.value()->replaceAllUsesWith(ivReplacement); en.value().replaceAllUsesWith(ivReplacement);
replaceAllUsesInRegionWith(steps[en.index()], *stepArgumentIt, replaceAllUsesInRegionWith(steps[en.index()], *stepArgumentIt,
launchOp.body()); launchOp.body());
std::advance(lbArgumentIt, 1); std::advance(lbArgumentIt, 1);
std::advance(stepArgumentIt, 1); std::advance(stepArgumentIt, 1);
} }
// Remap the values defined outside the body to use kernel arguments instead. // Remap the values defined outside the body to use kernel arguments instead.
// The list of kernel arguments also contains the lower bounds for loops at // The list of kernel arguments also contains the lower bounds for loops at
▲ Show 20 Lines • Show All 60 Lines • Show Last 20 Lines

mlir/lib/Conversion/StandardToLLVM/ConvertStandardToLLVM.cpp

Show First 20 Lines • Show All 243 Lines • ▼ Show 20 LinesLLVMOpLowering::LLVMOpLowering(StringRef rootOpName, MLIRContext *context,
PatternBenefit benefit) PatternBenefit benefit)
: ConversionPattern(rootOpName, benefit, context), lowering(lowering_) {} : ConversionPattern(rootOpName, benefit, context), lowering(lowering_) {}
/*============================================================================*/ /*============================================================================*/
/* StructBuilder implementation */ /* StructBuilder implementation */
/*============================================================================*/ /*============================================================================*/
StructBuilder::StructBuilder(Value v) : value(v) { StructBuilder::StructBuilder(Value v) : value(v) {
assert(value != nullptr && "value cannot be null"); assert(value != nullptr && "value cannot be null");
structType = value->getType().cast<LLVM::LLVMType>(); structType = value.getType().cast<LLVM::LLVMType>();
} }
Value StructBuilder::extractPtr(OpBuilder &builder, Location loc, Value StructBuilder::extractPtr(OpBuilder &builder, Location loc,
unsigned pos) { unsigned pos) {
Type type = structType.cast<LLVM::LLVMType>().getStructElementType(pos); Type type = structType.cast<LLVM::LLVMType>().getStructElementType(pos);
return builder.create<LLVM::ExtractValueOp>(loc, type, value, return builder.create<LLVM::ExtractValueOp>(loc, type, value,
builder.getI64ArrayAttr(pos)); builder.getI64ArrayAttr(pos));
} }
void StructBuilder::setPtr(OpBuilder &builder, Location loc, unsigned pos, void StructBuilder::setPtr(OpBuilder &builder, Location loc, unsigned pos,
Value ptr) { Value ptr) {
value = builder.create<LLVM::InsertValueOp>(loc, structType, value, ptr, value = builder.create<LLVM::InsertValueOp>(loc, structType, value, ptr,
builder.getI64ArrayAttr(pos)); builder.getI64ArrayAttr(pos));
} }
/*============================================================================*/ /*============================================================================*/
/* MemRefDescriptor implementation */ /* MemRefDescriptor implementation */
/*============================================================================*/ /*============================================================================*/
/// Construct a helper for the given descriptor value. /// Construct a helper for the given descriptor value.
MemRefDescriptor::MemRefDescriptor(Value descriptor) MemRefDescriptor::MemRefDescriptor(Value descriptor)
: StructBuilder(descriptor) { : StructBuilder(descriptor) {
assert(value != nullptr && "value cannot be null"); assert(value != nullptr && "value cannot be null");
indexType = value->getType().cast<LLVM::LLVMType>().getStructElementType( indexType = value.getType().cast<LLVM::LLVMType>().getStructElementType(
kOffsetPosInMemRefDescriptor); kOffsetPosInMemRefDescriptor);
} }
/// Builds IR creating an `undef` value of the descriptor type. /// Builds IR creating an `undef` value of the descriptor type.
MemRefDescriptor MemRefDescriptor::undef(OpBuilder &builder, Location loc, MemRefDescriptor MemRefDescriptor::undef(OpBuilder &builder, Location loc,
Type descriptorType) { Type descriptorType) {
Value descriptor = Value descriptor =
▲ Show 20 Lines • Show All 123 Lines • ▼ Show 20 Lines
/// Builds IR inserting the pos-th stride into the descriptor /// Builds IR inserting the pos-th stride into the descriptor
void MemRefDescriptor::setConstantStride(OpBuilder &builder, Location loc, void MemRefDescriptor::setConstantStride(OpBuilder &builder, Location loc,
unsigned pos, uint64_t stride) { unsigned pos, uint64_t stride) {
setStride(builder, loc, pos, setStride(builder, loc, pos,
createIndexAttrConstant(builder, loc, indexType, stride)); createIndexAttrConstant(builder, loc, indexType, stride));
} }
LLVM::LLVMType MemRefDescriptor::getElementType() { LLVM::LLVMType MemRefDescriptor::getElementType() {
return value->getType().cast<LLVM::LLVMType>().getStructElementType( return value.getType().cast<LLVM::LLVMType>().getStructElementType(
kAlignedPtrPosInMemRefDescriptor); kAlignedPtrPosInMemRefDescriptor);
} }
/*============================================================================*/ /*============================================================================*/
/* UnrankedMemRefDescriptor implementation */ /* UnrankedMemRefDescriptor implementation */
/*============================================================================*/ /*============================================================================*/
/// Construct a helper for the given descriptor value. /// Construct a helper for the given descriptor value.
▲ Show 20 Lines • Show All 244 Lines • ▼ Show 20 Lines if (numResults == 1)
return rewriter.replaceOp(op, newOp.getOperation()->getResult(0)), return rewriter.replaceOp(op, newOp.getOperation()->getResult(0)),
this->matchSuccess(); this->matchSuccess();
// Otherwise, it had been converted to an operation producing a structure. // Otherwise, it had been converted to an operation producing a structure.
// Extract individual results from the structure and return them as list. // Extract individual results from the structure and return them as list.
SmallVector<Value, 4> results; SmallVector<Value, 4> results;
results.reserve(numResults); results.reserve(numResults);
for (unsigned i = 0; i < numResults; ++i) { for (unsigned i = 0; i < numResults; ++i) {
auto type = this->lowering.convertType(op->getResult(i)->getType()); auto type = this->lowering.convertType(op->getResult(i).getType());
results.push_back(rewriter.create<LLVM::ExtractValueOp>( results.push_back(rewriter.create<LLVM::ExtractValueOp>(
op->getLoc(), type, newOp.getOperation()->getResult(0), op->getLoc(), type, newOp.getOperation()->getResult(0),
rewriter.getI64ArrayAttr(i))); rewriter.getI64ArrayAttr(i)));
} }
rewriter.replaceOp(op, results); rewriter.replaceOp(op, results);
return this->matchSuccess(); return this->matchSuccess();
} }
}; };
Show All 33 Lines static_assert(
std::is_base_of<OpTrait::OneResult<SourceOp>, SourceOp>::value, std::is_base_of<OpTrait::OneResult<SourceOp>, SourceOp>::value,
"expected single result op"); "expected single result op");
static_assert(std::is_base_of<OpTrait::SameOperandsAndResultType<SourceOp>, static_assert(std::is_base_of<OpTrait::SameOperandsAndResultType<SourceOp>,
SourceOp>::value, SourceOp>::value,
"expected same operands and result type"); "expected same operands and result type");
// Cannot convert ops if their operands are not of LLVM type. // Cannot convert ops if their operands are not of LLVM type.
for (Value operand : operands) { for (Value operand : operands) {
if (!operand || !operand->getType().isa<LLVM::LLVMType>()) if (!operand || !operand.getType().isa<LLVM::LLVMType>())
return this->matchFailure(); return this->matchFailure();
} }
auto loc = op->getLoc(); auto loc = op->getLoc();
auto llvmArrayTy = operands[0]->getType().cast<LLVM::LLVMType>(); auto llvmArrayTy = operands[0].getType().cast<LLVM::LLVMType>();
if (!llvmArrayTy.isArrayTy()) { if (!llvmArrayTy.isArrayTy()) {
auto newOp = rewriter.create<TargetOp>( auto newOp = rewriter.create<TargetOp>(
op->getLoc(), operands[0]->getType(), operands, op->getAttrs()); op->getLoc(), operands[0].getType(), operands, op->getAttrs());
rewriter.replaceOp(op, newOp.getResult()); rewriter.replaceOp(op, newOp.getResult());
return this->matchSuccess(); return this->matchSuccess();
} }
auto vectorType = op->getResult(0)->getType().dyn_cast<VectorType>(); auto vectorType = op->getResult(0).getType().dyn_cast<VectorType>();
if (!vectorType) if (!vectorType)
return this->matchFailure(); return this->matchFailure();
auto vectorTypeInfo = extractNDVectorTypeInfo(vectorType, this->lowering); auto vectorTypeInfo = extractNDVectorTypeInfo(vectorType, this->lowering);
auto llvmVectorTy = vectorTypeInfo.llvmVectorTy; auto llvmVectorTy = vectorTypeInfo.llvmVectorTy;
if (!llvmVectorTy || llvmArrayTy != vectorTypeInfo.llvmArrayTy) if (!llvmVectorTy || llvmArrayTy != vectorTypeInfo.llvmArrayTy)
return this->matchFailure(); return this->matchFailure();
Value desc = rewriter.create<LLVM::UndefOp>(loc, llvmArrayTy); Value desc = rewriter.create<LLVM::UndefOp>(loc, llvmArrayTy);
▲ Show 20 Lines • Show All 278 Lines • ▼ Show 20 Lines if (!useAlloca && alignment != 0) {
// offset = (align - (ptr % align))% align // offset = (align - (ptr % align))% align
Value intVal = rewriter.create<LLVM::PtrToIntOp>( Value intVal = rewriter.create<LLVM::PtrToIntOp>(
loc, this->getIndexType(), allocated); loc, this->getIndexType(), allocated);
Value ptrModAlign = Value ptrModAlign =
rewriter.create<LLVM::URemOp>(loc, intVal, alignmentValue); rewriter.create<LLVM::URemOp>(loc, intVal, alignmentValue);
Value subbed = Value subbed =
rewriter.create<LLVM::SubOp>(loc, alignmentValue, ptrModAlign); rewriter.create<LLVM::SubOp>(loc, alignmentValue, ptrModAlign);
Value offset = rewriter.create<LLVM::URemOp>(loc, subbed, alignmentValue); Value offset = rewriter.create<LLVM::URemOp>(loc, subbed, alignmentValue);
Value aligned = rewriter.create<LLVM::GEPOp>(loc, allocated->getType(), Value aligned = rewriter.create<LLVM::GEPOp>(loc, allocated.getType(),
allocated, offset); allocated, offset);
bitcastAligned = rewriter.create<LLVM::BitcastOp>( bitcastAligned = rewriter.create<LLVM::BitcastOp>(
loc, elementPtrType, ArrayRef<Value>(aligned)); loc, elementPtrType, ArrayRef<Value>(aligned));
} }
memRefDescriptor.setAlignedPtr(rewriter, loc, bitcastAligned); memRefDescriptor.setAlignedPtr(rewriter, loc, bitcastAligned);
// Field 3: Offset in aligned pointer. // Field 3: Offset in aligned pointer.
memRefDescriptor.setOffset(rewriter, loc, memRefDescriptor.setOffset(rewriter, loc,
▲ Show 20 Lines • Show All 83 Lines • ▼ Show 20 Lines matchAndRewrite(Operation *op, ArrayRef<Value> operands,
// Otherwise, it had been converted to an operation producing a structure. // Otherwise, it had been converted to an operation producing a structure.
// Extract individual results from the structure and return them as list. // Extract individual results from the structure and return them as list.
// TODO(aminim, ntv, riverriddle, zinenko): this seems like patching around // TODO(aminim, ntv, riverriddle, zinenko): this seems like patching around
// a particular interaction between MemRefType and CallOp lowering. Find a // a particular interaction between MemRefType and CallOp lowering. Find a
// way to avoid special casing. // way to avoid special casing.
SmallVector<Value, 4> results; SmallVector<Value, 4> results;
results.reserve(numResults); results.reserve(numResults);
for (unsigned i = 0; i < numResults; ++i) { for (unsigned i = 0; i < numResults; ++i) {
auto type = this->lowering.convertType(op->getResult(i)->getType()); auto type = this->lowering.convertType(op->getResult(i).getType());
results.push_back(rewriter.create<LLVM::ExtractValueOp>( results.push_back(rewriter.create<LLVM::ExtractValueOp>(
op->getLoc(), type, newOp.getOperation()->getResult(0), op->getLoc(), type, newOp.getOperation()->getResult(0),
rewriter.getI64ArrayAttr(i))); rewriter.getI64ArrayAttr(i)));
} }
rewriter.replaceOp(op, results); rewriter.replaceOp(op, results);
return this->matchSuccess(); return this->matchSuccess();
} }
▲ Show 20 Lines • Show All 58 Lines • ▼ Show 20 Linesstruct TanhOpLowering : public LLVMLegalizationPattern<TanhOp> {
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
using LLVMFuncOpT = LLVM::LLVMFuncOp; using LLVMFuncOpT = LLVM::LLVMFuncOp;
using LLVMTypeT = LLVM::LLVMType; using LLVMTypeT = LLVM::LLVMType;
OperandAdaptor<TanhOp> transformed(operands); OperandAdaptor<TanhOp> transformed(operands);
LLVMTypeT operandType = LLVMTypeT operandType =
transformed.operand()->getType().dyn_cast_or_null<LLVM::LLVMType>(); transformed.operand().getType().dyn_cast_or_null<LLVM::LLVMType>();
if (!operandType) if (!operandType)
return matchFailure(); return matchFailure();
std::string functionName; std::string functionName;
if (operandType.isFloatTy()) if (operandType.isFloatTy())
functionName = "tanhf"; functionName = "tanhf";
else if (operandType.isDoubleTy()) else if (operandType.isDoubleTy())
Show All 25 Linesstruct TanhOpLowering : public LLVMLegalizationPattern<TanhOp> {
} }
}; };
struct MemRefCastOpLowering : public LLVMLegalizationPattern<MemRefCastOp> { struct MemRefCastOpLowering : public LLVMLegalizationPattern<MemRefCastOp> {
using LLVMLegalizationPattern<MemRefCastOp>::LLVMLegalizationPattern; using LLVMLegalizationPattern<MemRefCastOp>::LLVMLegalizationPattern;
PatternMatchResult match(Operation *op) const override { PatternMatchResult match(Operation *op) const override {
auto memRefCastOp = cast<MemRefCastOp>(op); auto memRefCastOp = cast<MemRefCastOp>(op);
Type srcType = memRefCastOp.getOperand()->getType(); Type srcType = memRefCastOp.getOperand().getType();
Type dstType = memRefCastOp.getType(); Type dstType = memRefCastOp.getType();
if (srcType.isa<MemRefType>() && dstType.isa<MemRefType>()) { if (srcType.isa<MemRefType>() && dstType.isa<MemRefType>()) {
MemRefType sourceType = MemRefType sourceType =
memRefCastOp.getOperand()->getType().cast<MemRefType>(); memRefCastOp.getOperand().getType().cast<MemRefType>();
MemRefType targetType = memRefCastOp.getType().cast<MemRefType>(); MemRefType targetType = memRefCastOp.getType().cast<MemRefType>();
return (isSupportedMemRefType(targetType) && return (isSupportedMemRefType(targetType) &&
isSupportedMemRefType(sourceType)) isSupportedMemRefType(sourceType))
? matchSuccess() ? matchSuccess()
: matchFailure(); : matchFailure();
} }
// At least one of the operands is unranked type // At least one of the operands is unranked type
assert(srcType.isa<UnrankedMemRefType>() || assert(srcType.isa<UnrankedMemRefType>() ||
dstType.isa<UnrankedMemRefType>()); dstType.isa<UnrankedMemRefType>());
// Unranked to unranked cast is disallowed // Unranked to unranked cast is disallowed
return !(srcType.isa<UnrankedMemRefType>() && return !(srcType.isa<UnrankedMemRefType>() &&
dstType.isa<UnrankedMemRefType>()) dstType.isa<UnrankedMemRefType>())
? matchSuccess() ? matchSuccess()
: matchFailure(); : matchFailure();
} }
void rewrite(Operation *op, ArrayRef<Value> operands, void rewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto memRefCastOp = cast<MemRefCastOp>(op); auto memRefCastOp = cast<MemRefCastOp>(op);
OperandAdaptor<MemRefCastOp> transformed(operands); OperandAdaptor<MemRefCastOp> transformed(operands);
auto srcType = memRefCastOp.getOperand()->getType(); auto srcType = memRefCastOp.getOperand().getType();
auto dstType = memRefCastOp.getType(); auto dstType = memRefCastOp.getType();
auto targetStructType = lowering.convertType(memRefCastOp.getType()); auto targetStructType = lowering.convertType(memRefCastOp.getType());
auto loc = op->getLoc(); auto loc = op->getLoc();
if (srcType.isa<MemRefType>() && dstType.isa<MemRefType>()) { if (srcType.isa<MemRefType>() && dstType.isa<MemRefType>()) {
// memref_cast is defined for source and destination memref types with the // memref_cast is defined for source and destination memref types with the
// same element type, same mappings, same address space and same rank. // same element type, same mappings, same address space and same rank.
// Therefore a simple bitcast suffices. If not it is undefined behavior. // Therefore a simple bitcast suffices. If not it is undefined behavior.
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Lines
struct DimOpLowering : public LLVMLegalizationPattern<DimOp> { struct DimOpLowering : public LLVMLegalizationPattern<DimOp> {
using LLVMLegalizationPattern<DimOp>::LLVMLegalizationPattern; using LLVMLegalizationPattern<DimOp>::LLVMLegalizationPattern;
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto dimOp = cast<DimOp>(op); auto dimOp = cast<DimOp>(op);
OperandAdaptor<DimOp> transformed(operands); OperandAdaptor<DimOp> transformed(operands);
MemRefType type = dimOp.getOperand()->getType().cast<MemRefType>(); MemRefType type = dimOp.getOperand().getType().cast<MemRefType>();
auto shape = type.getShape(); auto shape = type.getShape();
int64_t index = dimOp.getIndex(); int64_t index = dimOp.getIndex();
// Extract dynamic size from the memref descriptor. // Extract dynamic size from the memref descriptor.
if (ShapedType::isDynamic(shape[index])) if (ShapedType::isDynamic(shape[index]))
rewriter.replaceOp(op, {MemRefDescriptor(transformed.memrefOrTensor()) rewriter.replaceOp(op, {MemRefDescriptor(transformed.memrefOrTensor())
.size(rewriter, op->getLoc(), index)}); .size(rewriter, op->getLoc(), index)});
else else
▲ Show 20 Lines • Show All 163 Lines • ▼ Show 20 Linesstruct IndexCastOpLowering : public LLVMLegalizationPattern<IndexCastOp> {
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
IndexCastOpOperandAdaptor transformed(operands); IndexCastOpOperandAdaptor transformed(operands);
auto indexCastOp = cast<IndexCastOp>(op); auto indexCastOp = cast<IndexCastOp>(op);
auto targetType = auto targetType =
this->lowering.convertType(indexCastOp.getResult()->getType()) this->lowering.convertType(indexCastOp.getResult().getType())
.cast<LLVM::LLVMType>(); .cast<LLVM::LLVMType>();
auto sourceType = transformed.in()->getType().cast<LLVM::LLVMType>(); auto sourceType = transformed.in().getType().cast<LLVM::LLVMType>();
unsigned targetBits = targetType.getUnderlyingType()->getIntegerBitWidth(); unsigned targetBits = targetType.getUnderlyingType()->getIntegerBitWidth();
unsigned sourceBits = sourceType.getUnderlyingType()->getIntegerBitWidth(); unsigned sourceBits = sourceType.getUnderlyingType()->getIntegerBitWidth();
if (targetBits == sourceBits) if (targetBits == sourceBits)
rewriter.replaceOp(op, transformed.in()); rewriter.replaceOp(op, transformed.in());
else if (targetBits < sourceBits) else if (targetBits < sourceBits)
rewriter.replaceOpWithNewOp<LLVM::TruncOp>(op, targetType, rewriter.replaceOpWithNewOp<LLVM::TruncOp>(op, targetType,
transformed.in()); transformed.in());
Show All 16 Linesstruct CmpIOpLowering : public LLVMLegalizationPattern<CmpIOp> {
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto cmpiOp = cast<CmpIOp>(op); auto cmpiOp = cast<CmpIOp>(op);
CmpIOpOperandAdaptor transformed(operands); CmpIOpOperandAdaptor transformed(operands);
rewriter.replaceOpWithNewOp<LLVM::ICmpOp>( rewriter.replaceOpWithNewOp<LLVM::ICmpOp>(
op, lowering.convertType(cmpiOp.getResult()->getType()), op, lowering.convertType(cmpiOp.getResult().getType()),
rewriter.getI64IntegerAttr(static_cast<int64_t>( rewriter.getI64IntegerAttr(static_cast<int64_t>(
convertCmpPredicate<LLVM::ICmpPredicate>(cmpiOp.getPredicate()))), convertCmpPredicate<LLVM::ICmpPredicate>(cmpiOp.getPredicate()))),
transformed.lhs(), transformed.rhs()); transformed.lhs(), transformed.rhs());
return matchSuccess(); return matchSuccess();
} }
}; };
struct CmpFOpLowering : public LLVMLegalizationPattern<CmpFOp> { struct CmpFOpLowering : public LLVMLegalizationPattern<CmpFOp> {
using LLVMLegalizationPattern<CmpFOp>::LLVMLegalizationPattern; using LLVMLegalizationPattern<CmpFOp>::LLVMLegalizationPattern;
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto cmpfOp = cast<CmpFOp>(op); auto cmpfOp = cast<CmpFOp>(op);
CmpFOpOperandAdaptor transformed(operands); CmpFOpOperandAdaptor transformed(operands);
rewriter.replaceOpWithNewOp<LLVM::FCmpOp>( rewriter.replaceOpWithNewOp<LLVM::FCmpOp>(
op, lowering.convertType(cmpfOp.getResult()->getType()), op, lowering.convertType(cmpfOp.getResult().getType()),
rewriter.getI64IntegerAttr(static_cast<int64_t>( rewriter.getI64IntegerAttr(static_cast<int64_t>(
convertCmpPredicate<LLVM::FCmpPredicate>(cmpfOp.getPredicate()))), convertCmpPredicate<LLVM::FCmpPredicate>(cmpfOp.getPredicate()))),
transformed.lhs(), transformed.rhs()); transformed.lhs(), transformed.rhs());
return matchSuccess(); return matchSuccess();
} }
}; };
▲ Show 20 Lines • Show All 207 Lines • ▼ Show 20 Lines SmallVector<Value, 2> dynamicSizes(
std::next(operands.begin(), 1 + viewOp.getNumOffsets()), std::next(operands.begin(), 1 + viewOp.getNumOffsets()),
std::next(operands.begin(), std::next(operands.begin(),
1 + viewOp.getNumOffsets() + viewOp.getNumSizes())); 1 + viewOp.getNumOffsets() + viewOp.getNumSizes()));
SmallVector<Value, 2> dynamicStrides( SmallVector<Value, 2> dynamicStrides(
std::next(operands.begin(), std::next(operands.begin(),
1 + viewOp.getNumOffsets() + viewOp.getNumSizes()), 1 + viewOp.getNumOffsets() + viewOp.getNumSizes()),
operands.end()); operands.end());
auto sourceMemRefType = viewOp.source()->getType().cast<MemRefType>(); auto sourceMemRefType = viewOp.source().getType().cast<MemRefType>();
auto sourceElementTy = auto sourceElementTy =
lowering.convertType(sourceMemRefType.getElementType()) lowering.convertType(sourceMemRefType.getElementType())
.dyn_cast_or_null<LLVM::LLVMType>(); .dyn_cast_or_null<LLVM::LLVMType>();
auto viewMemRefType = viewOp.getType(); auto viewMemRefType = viewOp.getType();
auto targetElementTy = lowering.convertType(viewMemRefType.getElementType()) auto targetElementTy = lowering.convertType(viewMemRefType.getElementType())
.dyn_cast<LLVM::LLVMType>(); .dyn_cast<LLVM::LLVMType>();
auto targetDescTy = auto targetDescTy =
▲ Show 20 Lines • Show All 350 Lines • ▼ Show 20 Lines
Value LLVMTypeConverter::promoteOneMemRefDescriptor(Location loc, Value operand, Value LLVMTypeConverter::promoteOneMemRefDescriptor(Location loc, Value operand,
OpBuilder &builder) { OpBuilder &builder) {
auto *context = builder.getContext(); auto *context = builder.getContext();
auto int64Ty = LLVM::LLVMType::getInt64Ty(getDialect()); auto int64Ty = LLVM::LLVMType::getInt64Ty(getDialect());
auto indexType = IndexType::get(context); auto indexType = IndexType::get(context);
// Alloca with proper alignment. We do not expect optimizations of this // Alloca with proper alignment. We do not expect optimizations of this
// alloca op and so we omit allocating at the entry block. // alloca op and so we omit allocating at the entry block.
auto ptrType = operand->getType().cast<LLVM::LLVMType>().getPointerTo(); auto ptrType = operand.getType().cast<LLVM::LLVMType>().getPointerTo();
Value one = builder.create<LLVM::ConstantOp>(loc, int64Ty, Value one = builder.create<LLVM::ConstantOp>(loc, int64Ty,
IntegerAttr::get(indexType, 1)); IntegerAttr::get(indexType, 1));
Value allocated = Value allocated =
builder.create<LLVM::AllocaOp>(loc, ptrType, one, /*alignment=*/0); builder.create<LLVM::AllocaOp>(loc, ptrType, one, /*alignment=*/0);
// Store into the alloca'ed descriptor. // Store into the alloca'ed descriptor.
builder.create<LLVM::StoreOp>(loc, operand, allocated); builder.create<LLVM::StoreOp>(loc, operand, allocated);
return allocated; return allocated;
} }
SmallVector<Value, 4> SmallVector<Value, 4>
LLVMTypeConverter::promoteMemRefDescriptors(Location loc, ValueRange opOperands, LLVMTypeConverter::promoteMemRefDescriptors(Location loc, ValueRange opOperands,
ValueRange operands, ValueRange operands,
OpBuilder &builder) { OpBuilder &builder) {
SmallVector<Value, 4> promotedOperands; SmallVector<Value, 4> promotedOperands;
promotedOperands.reserve(operands.size()); promotedOperands.reserve(operands.size());
for (auto it : llvm::zip(opOperands, operands)) { for (auto it : llvm::zip(opOperands, operands)) {
auto operand = std::get<0>(it); auto operand = std::get<0>(it);
auto llvmOperand = std::get<1>(it); auto llvmOperand = std::get<1>(it);
if (!operand->getType().isa<MemRefType>() && if (!operand.getType().isa<MemRefType>() &&
!operand->getType().isa<UnrankedMemRefType>()) { !operand.getType().isa<UnrankedMemRefType>()) {
promotedOperands.push_back(operand); promotedOperands.push_back(operand);
continue; continue;
} }
promotedOperands.push_back( promotedOperands.push_back(
promoteOneMemRefDescriptor(loc, llvmOperand, builder)); promoteOneMemRefDescriptor(loc, llvmOperand, builder));
} }
return promotedOperands; return promotedOperands;
} }
▲ Show 20 Lines • Show All 74 Lines • Show Last 20 Lines

mlir/lib/Conversion/StandardToSPIRV/ConvertStandardToSPIRV.cpp

Show First 20 Lines • Show All 68 Lines • ▼ Show 20 Lines
class IntegerOpConversion final : public SPIRVOpLowering<StdOp> { class IntegerOpConversion final : public SPIRVOpLowering<StdOp> {
public: public:
using SPIRVOpLowering<StdOp>::SPIRVOpLowering; using SPIRVOpLowering<StdOp>::SPIRVOpLowering;
PatternMatchResult PatternMatchResult
matchAndRewrite(StdOp operation, ArrayRef<Value> operands, matchAndRewrite(StdOp operation, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto resultType = auto resultType =
this->typeConverter.convertType(operation.getResult()->getType()); this->typeConverter.convertType(operation.getResult().getType());
rewriter.template replaceOpWithNewOp<SPIRVOp>( rewriter.template replaceOpWithNewOp<SPIRVOp>(
operation, resultType, operands, ArrayRef<NamedAttribute>()); operation, resultType, operands, ArrayRef<NamedAttribute>());
return this->matchSuccess(); return this->matchSuccess();
} }
}; };
/// Convert load -> spv.LoadOp. The operands of the replaced operation are of /// Convert load -> spv.LoadOp. The operands of the replaced operation are of
/// IndexType while that of the replacement operation are of type i32. This is /// IndexType while that of the replacement operation are of type i32. This is
▲ Show 20 Lines • Show All 87 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ConstantOp with index type. // ConstantOp with index type.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
PatternMatchResult ConstantIndexOpConversion::matchAndRewrite( PatternMatchResult ConstantIndexOpConversion::matchAndRewrite(
ConstantOp constIndexOp, ArrayRef<Value> operands, ConstantOp constIndexOp, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
if (!constIndexOp.getResult()->getType().isa<IndexType>()) { if (!constIndexOp.getResult().getType().isa<IndexType>()) {
return matchFailure(); return matchFailure();
} }
// The attribute has index type which is not directly supported in // The attribute has index type which is not directly supported in
// SPIR-V. Get the integer value and create a new IntegerAttr. // SPIR-V. Get the integer value and create a new IntegerAttr.
auto constAttr = constIndexOp.value().dyn_cast<IntegerAttr>(); auto constAttr = constIndexOp.value().dyn_cast<IntegerAttr>();
if (!constAttr) { if (!constAttr) {
return matchFailure(); return matchFailure();
} }
// Use the bitwidth set in the value attribute to decide the result type // Use the bitwidth set in the value attribute to decide the result type
// of the SPIR-V constant operation since SPIR-V does not support index // of the SPIR-V constant operation since SPIR-V does not support index
// types. // types.
auto constVal = constAttr.getValue(); auto constVal = constAttr.getValue();
auto constValType = constAttr.getType().dyn_cast<IndexType>(); auto constValType = constAttr.getType().dyn_cast<IndexType>();
if (!constValType) { if (!constValType) {
return matchFailure(); return matchFailure();
} }
auto spirvConstType = auto spirvConstType =
typeConverter.convertType(constIndexOp.getResult()->getType()); typeConverter.convertType(constIndexOp.getResult().getType());
auto spirvConstVal = auto spirvConstVal =
rewriter.getIntegerAttr(spirvConstType, constAttr.getInt()); rewriter.getIntegerAttr(spirvConstType, constAttr.getInt());
rewriter.replaceOpWithNewOp<spirv::ConstantOp>(constIndexOp, spirvConstType, rewriter.replaceOpWithNewOp<spirv::ConstantOp>(constIndexOp, spirvConstType,
spirvConstVal); spirvConstVal);
return matchSuccess(); return matchSuccess();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CmpFOp // CmpFOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
PatternMatchResult PatternMatchResult
CmpFOpConversion::matchAndRewrite(CmpFOp cmpFOp, ArrayRef<Value> operands, CmpFOpConversion::matchAndRewrite(CmpFOp cmpFOp, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
CmpFOpOperandAdaptor cmpFOpOperands(operands); CmpFOpOperandAdaptor cmpFOpOperands(operands);
switch (cmpFOp.getPredicate()) { switch (cmpFOp.getPredicate()) {
#define DISPATCH(cmpPredicate, spirvOp) \ #define DISPATCH(cmpPredicate, spirvOp) \
case cmpPredicate: \ case cmpPredicate: \
rewriter.replaceOpWithNewOp<spirvOp>( \ rewriter.replaceOpWithNewOp<spirvOp>(cmpFOp, cmpFOp.getResult().getType(), \
cmpFOp, cmpFOp.getResult()->getType(), cmpFOpOperands.lhs(), \ cmpFOpOperands.lhs(), \
cmpFOpOperands.rhs()); \ cmpFOpOperands.rhs()); \
return matchSuccess(); return matchSuccess();
// Ordered. // Ordered.
DISPATCH(CmpFPredicate::OEQ, spirv::FOrdEqualOp); DISPATCH(CmpFPredicate::OEQ, spirv::FOrdEqualOp);
DISPATCH(CmpFPredicate::OGT, spirv::FOrdGreaterThanOp); DISPATCH(CmpFPredicate::OGT, spirv::FOrdGreaterThanOp);
DISPATCH(CmpFPredicate::OGE, spirv::FOrdGreaterThanEqualOp); DISPATCH(CmpFPredicate::OGE, spirv::FOrdGreaterThanEqualOp);
DISPATCH(CmpFPredicate::OLT, spirv::FOrdLessThanOp); DISPATCH(CmpFPredicate::OLT, spirv::FOrdLessThanOp);
DISPATCH(CmpFPredicate::OLE, spirv::FOrdLessThanEqualOp); DISPATCH(CmpFPredicate::OLE, spirv::FOrdLessThanEqualOp);
Show All 21 Lines
PatternMatchResult PatternMatchResult
CmpIOpConversion::matchAndRewrite(CmpIOp cmpIOp, ArrayRef<Value> operands, CmpIOpConversion::matchAndRewrite(CmpIOp cmpIOp, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
CmpIOpOperandAdaptor cmpIOpOperands(operands); CmpIOpOperandAdaptor cmpIOpOperands(operands);
switch (cmpIOp.getPredicate()) { switch (cmpIOp.getPredicate()) {
#define DISPATCH(cmpPredicate, spirvOp) \ #define DISPATCH(cmpPredicate, spirvOp) \
case cmpPredicate: \ case cmpPredicate: \
rewriter.replaceOpWithNewOp<spirvOp>( \ rewriter.replaceOpWithNewOp<spirvOp>(cmpIOp, cmpIOp.getResult().getType(), \
cmpIOp, cmpIOp.getResult()->getType(), cmpIOpOperands.lhs(), \ cmpIOpOperands.lhs(), \
cmpIOpOperands.rhs()); \ cmpIOpOperands.rhs()); \
return matchSuccess(); return matchSuccess();
DISPATCH(CmpIPredicate::eq, spirv::IEqualOp); DISPATCH(CmpIPredicate::eq, spirv::IEqualOp);
DISPATCH(CmpIPredicate::ne, spirv::INotEqualOp); DISPATCH(CmpIPredicate::ne, spirv::INotEqualOp);
DISPATCH(CmpIPredicate::slt, spirv::SLessThanOp); DISPATCH(CmpIPredicate::slt, spirv::SLessThanOp);
DISPATCH(CmpIPredicate::sle, spirv::SLessThanEqualOp); DISPATCH(CmpIPredicate::sle, spirv::SLessThanEqualOp);
DISPATCH(CmpIPredicate::sgt, spirv::SGreaterThanOp); DISPATCH(CmpIPredicate::sgt, spirv::SGreaterThanOp);
DISPATCH(CmpIPredicate::sge, spirv::SGreaterThanEqualOp); DISPATCH(CmpIPredicate::sge, spirv::SGreaterThanEqualOp);
Show All 11 Lines
// LoadOp // LoadOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
PatternMatchResult PatternMatchResult
LoadOpConversion::matchAndRewrite(LoadOp loadOp, ArrayRef<Value> operands, LoadOpConversion::matchAndRewrite(LoadOp loadOp, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
LoadOpOperandAdaptor loadOperands(operands); LoadOpOperandAdaptor loadOperands(operands);
auto loadPtr = getElementPtr(rewriter, typeConverter, loadOp.getLoc(), auto loadPtr = getElementPtr(rewriter, typeConverter, loadOp.getLoc(),
loadOp.memref()->getType().cast<MemRefType>(), loadOp.memref().getType().cast<MemRefType>(),
loadOperands.memref(), loadOperands.indices()); loadOperands.memref(), loadOperands.indices());
rewriter.replaceOpWithNewOp<spirv::LoadOp>(loadOp, loadPtr, rewriter.replaceOpWithNewOp<spirv::LoadOp>(loadOp, loadPtr,
/*memory_access =*/nullptr, /*memory_access =*/nullptr,
/*alignment =*/nullptr); /*alignment =*/nullptr);
return matchSuccess(); return matchSuccess();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 29 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
PatternMatchResult PatternMatchResult
StoreOpConversion::matchAndRewrite(StoreOp storeOp, ArrayRef<Value> operands, StoreOpConversion::matchAndRewrite(StoreOp storeOp, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
StoreOpOperandAdaptor storeOperands(operands); StoreOpOperandAdaptor storeOperands(operands);
auto storePtr = auto storePtr =
getElementPtr(rewriter, typeConverter, storeOp.getLoc(), getElementPtr(rewriter, typeConverter, storeOp.getLoc(),
storeOp.memref()->getType().cast<MemRefType>(), storeOp.memref().getType().cast<MemRefType>(),
storeOperands.memref(), storeOperands.indices()); storeOperands.memref(), storeOperands.indices());
rewriter.replaceOpWithNewOp<spirv::StoreOp>(storeOp, storePtr, rewriter.replaceOpWithNewOp<spirv::StoreOp>(storeOp, storePtr,
storeOperands.value(), storeOperands.value(),
/*memory_access =*/nullptr, /*memory_access =*/nullptr,
/*alignment =*/nullptr); /*alignment =*/nullptr);
return matchSuccess(); return matchSuccess();
} }
Show All 21 Lines

mlir/lib/Conversion/StandardToSPIRV/LegalizeStandardForSPIRV.cpp

Show First 20 Lines • Show All 104 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Folding SubViewOp and LoadOp. // Folding SubViewOp and LoadOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
PatternMatchResult PatternMatchResult
LoadOpOfSubViewFolder::matchAndRewrite(LoadOp loadOp, LoadOpOfSubViewFolder::matchAndRewrite(LoadOp loadOp,
PatternRewriter &rewriter) const { PatternRewriter &rewriter) const {
auto subViewOp = auto subViewOp = dyn_cast_or_null<SubViewOp>(loadOp.memref().getDefiningOp());
dyn_cast_or_null<SubViewOp>(loadOp.memref()->getDefiningOp());
if (!subViewOp) { if (!subViewOp) {
return matchFailure(); return matchFailure();
} }
SmallVector<Value, 4> sourceIndices; SmallVector<Value, 4> sourceIndices;
if (failed(resolveSourceIndices(loadOp.getLoc(), rewriter, subViewOp, if (failed(resolveSourceIndices(loadOp.getLoc(), rewriter, subViewOp,
loadOp.indices(), sourceIndices))) loadOp.indices(), sourceIndices)))
return matchFailure(); return matchFailure();
rewriter.replaceOpWithNewOp<LoadOp>(loadOp, subViewOp.source(), rewriter.replaceOpWithNewOp<LoadOp>(loadOp, subViewOp.source(),
sourceIndices); sourceIndices);
return matchSuccess(); return matchSuccess();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Folding SubViewOp and StoreOp. // Folding SubViewOp and StoreOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
PatternMatchResult PatternMatchResult
StoreOpOfSubViewFolder::matchAndRewrite(StoreOp storeOp, StoreOpOfSubViewFolder::matchAndRewrite(StoreOp storeOp,
PatternRewriter &rewriter) const { PatternRewriter &rewriter) const {
auto subViewOp = auto subViewOp =
dyn_cast_or_null<SubViewOp>(storeOp.memref()->getDefiningOp()); dyn_cast_or_null<SubViewOp>(storeOp.memref().getDefiningOp());
if (!subViewOp) { if (!subViewOp) {
return matchFailure(); return matchFailure();
} }
SmallVector<Value, 4> sourceIndices; SmallVector<Value, 4> sourceIndices;
if (failed(resolveSourceIndices(storeOp.getLoc(), rewriter, subViewOp, if (failed(resolveSourceIndices(storeOp.getLoc(), rewriter, subViewOp,
storeOp.indices(), sourceIndices))) storeOp.indices(), sourceIndices)))
return matchFailure(); return matchFailure();
Show All 37 Lines

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp

Show First 20 Lines • Show All 362 Lines • ▼ Show 20 Linespublic:
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto loc = op->getLoc(); auto loc = op->getLoc();
auto adaptor = vector::ExtractOpOperandAdaptor(operands); auto adaptor = vector::ExtractOpOperandAdaptor(operands);
auto extractOp = cast<vector::ExtractOp>(op); auto extractOp = cast<vector::ExtractOp>(op);
auto vectorType = extractOp.getVectorType(); auto vectorType = extractOp.getVectorType();
auto resultType = extractOp.getResult()->getType(); auto resultType = extractOp.getResult().getType();
auto llvmResultType = lowering.convertType(resultType); auto llvmResultType = lowering.convertType(resultType);
auto positionArrayAttr = extractOp.position(); auto positionArrayAttr = extractOp.position();
// Bail if result type cannot be lowered. // Bail if result type cannot be lowered.
if (!llvmResultType) if (!llvmResultType)
return matchFailure(); return matchFailure();
// One-shot extraction of vector from array (only requires extractvalue). // One-shot extraction of vector from array (only requires extractvalue).
▲ Show 20 Lines • Show All 262 Lines • ▼ Show 20 Lines explicit VectorOuterProductOpConversion(MLIRContext *context,
typeConverter) {} typeConverter) {}
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto loc = op->getLoc(); auto loc = op->getLoc();
auto adaptor = vector::OuterProductOpOperandAdaptor(operands); auto adaptor = vector::OuterProductOpOperandAdaptor(operands);
auto *ctx = op->getContext(); auto *ctx = op->getContext();
auto vLHS = adaptor.lhs()->getType().cast<LLVM::LLVMType>(); auto vLHS = adaptor.lhs().getType().cast<LLVM::LLVMType>();
auto vRHS = adaptor.rhs()->getType().cast<LLVM::LLVMType>(); auto vRHS = adaptor.rhs().getType().cast<LLVM::LLVMType>();
auto rankLHS = vLHS.getUnderlyingType()->getVectorNumElements(); auto rankLHS = vLHS.getUnderlyingType()->getVectorNumElements();
auto rankRHS = vRHS.getUnderlyingType()->getVectorNumElements(); auto rankRHS = vRHS.getUnderlyingType()->getVectorNumElements();
auto llvmArrayOfVectType = lowering.convertType( auto llvmArrayOfVectType = lowering.convertType(
cast<vector::OuterProductOp>(op).getResult()->getType()); cast<vector::OuterProductOp>(op).getResult().getType());
Value desc = rewriter.create<LLVM::UndefOp>(loc, llvmArrayOfVectType); Value desc = rewriter.create<LLVM::UndefOp>(loc, llvmArrayOfVectType);
Value a = adaptor.lhs(), b = adaptor.rhs(); Value a = adaptor.lhs(), b = adaptor.rhs();
Value acc = adaptor.acc().empty() ? nullptr : adaptor.acc().front(); Value acc = adaptor.acc().empty() ? nullptr : adaptor.acc().front();
SmallVector<Value, 8> lhs, accs; SmallVector<Value, 8> lhs, accs;
lhs.reserve(rankLHS); lhs.reserve(rankLHS);
accs.reserve(rankLHS); accs.reserve(rankLHS);
for (unsigned d = 0, e = rankLHS; d < e; ++d) { for (unsigned d = 0, e = rankLHS; d < e; ++d) {
// shufflevector explicitly requires i32. // shufflevector explicitly requires i32.
Show All 30 Lines explicit VectorTypeCastOpConversion(MLIRContext *context,
typeConverter) {} typeConverter) {}
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto loc = op->getLoc(); auto loc = op->getLoc();
vector::TypeCastOp castOp = cast<vector::TypeCastOp>(op); vector::TypeCastOp castOp = cast<vector::TypeCastOp>(op);
MemRefType sourceMemRefType = MemRefType sourceMemRefType =
castOp.getOperand()->getType().cast<MemRefType>(); castOp.getOperand().getType().cast<MemRefType>();
MemRefType targetMemRefType = MemRefType targetMemRefType =
castOp.getResult()->getType().cast<MemRefType>(); castOp.getResult().getType().cast<MemRefType>();
// Only static shape casts supported atm. // Only static shape casts supported atm.
if (!sourceMemRefType.hasStaticShape() || if (!sourceMemRefType.hasStaticShape() ||
!targetMemRefType.hasStaticShape()) !targetMemRefType.hasStaticShape())
return matchFailure(); return matchFailure();
auto llvmSourceDescriptorTy = auto llvmSourceDescriptorTy =
operands[0]->getType().dyn_cast<LLVM::LLVMType>(); operands[0].getType().dyn_cast<LLVM::LLVMType>();
if (!llvmSourceDescriptorTy || !llvmSourceDescriptorTy.isStructTy()) if (!llvmSourceDescriptorTy || !llvmSourceDescriptorTy.isStructTy())
return matchFailure(); return matchFailure();
MemRefDescriptor sourceMemRef(operands[0]); MemRefDescriptor sourceMemRef(operands[0]);
auto llvmTargetDescriptorTy = lowering.convertType(targetMemRefType) auto llvmTargetDescriptorTy = lowering.convertType(targetMemRefType)
.dyn_cast_or_null<LLVM::LLVMType>(); .dyn_cast_or_null<LLVM::LLVMType>();
if (!llvmTargetDescriptorTy || !llvmTargetDescriptorTy.isStructTy()) if (!llvmTargetDescriptorTy || !llvmTargetDescriptorTy.isStructTy())
return matchFailure(); return matchFailure();
▲ Show 20 Lines • Show All 274 Lines • Show Last 20 Lines

mlir/lib/Dialect/AffineOps/AffineOps.cpp

Show First 20 Lines • Show All 102 Lines • ▼ Show 20 Linesstatic bool isFunctionRegion(Region *region) {
return llvm::isa<FuncOp>(region->getParentOp()); return llvm::isa<FuncOp>(region->getParentOp());
} }
/// A utility function to check if a value is defined at the top level of a /// A utility function to check if a value is defined at the top level of a
/// function. A value of index type defined at the top level is always a valid /// function. A value of index type defined at the top level is always a valid
/// symbol. /// symbol.
bool mlir::isTopLevelValue(Value value) { bool mlir::isTopLevelValue(Value value) {
if (auto arg = value.dyn_cast<BlockArgument>()) if (auto arg = value.dyn_cast<BlockArgument>())
return isFunctionRegion(arg->getOwner()->getParent()); return isFunctionRegion(arg.getOwner()->getParent());
return isFunctionRegion(value->getDefiningOp()->getParentRegion()); return isFunctionRegion(value.getDefiningOp()->getParentRegion());
} }
// Value can be used as a dimension id if it is valid as a symbol, or // Value can be used as a dimension id if it is valid as a symbol, or
// it is an induction variable, or it is a result of affine apply operation // it is an induction variable, or it is a result of affine apply operation
// with dimension id arguments. // with dimension id arguments.
bool mlir::isValidDim(Value value) { bool mlir::isValidDim(Value value) {
// The value must be an index type. // The value must be an index type.
if (!value->getType().isIndex()) if (!value.getType().isIndex())
return false; return false;
if (auto *op = value->getDefiningOp()) { if (auto *op = value.getDefiningOp()) {
// Top level operation or constant operation is ok. // Top level operation or constant operation is ok.
if (isFunctionRegion(op->getParentRegion()) || isa<ConstantOp>(op)) if (isFunctionRegion(op->getParentRegion()) || isa<ConstantOp>(op))
return true; return true;
// Affine apply operation is ok if all of its operands are ok. // Affine apply operation is ok if all of its operands are ok.
if (auto applyOp = dyn_cast<AffineApplyOp>(op)) if (auto applyOp = dyn_cast<AffineApplyOp>(op))
return applyOp.isValidDim(); return applyOp.isValidDim();
// The dim op is okay if its operand memref/tensor is defined at the top // The dim op is okay if its operand memref/tensor is defined at the top
// level. // level.
if (auto dimOp = dyn_cast<DimOp>(op)) if (auto dimOp = dyn_cast<DimOp>(op))
return isTopLevelValue(dimOp.getOperand()); return isTopLevelValue(dimOp.getOperand());
return false; return false;
} }
// This value has to be a block argument for a FuncOp or an affine.for. // This value has to be a block argument for a FuncOp or an affine.for.
auto *parentOp = value.cast<BlockArgument>()->getOwner()->getParentOp(); auto *parentOp = value.cast<BlockArgument>().getOwner()->getParentOp();
return isa<FuncOp>(parentOp) || isa<AffineForOp>(parentOp); return isa<FuncOp>(parentOp) || isa<AffineForOp>(parentOp);
} }
/// Returns true if the 'index' dimension of the `memref` defined by /// Returns true if the 'index' dimension of the `memref` defined by
/// `memrefDefOp` is a statically shaped one or defined using a valid symbol. /// `memrefDefOp` is a statically shaped one or defined using a valid symbol.
template <typename AnyMemRefDefOp> template <typename AnyMemRefDefOp>
bool isMemRefSizeValidSymbol(AnyMemRefDefOp memrefDefOp, unsigned index) { bool isMemRefSizeValidSymbol(AnyMemRefDefOp memrefDefOp, unsigned index) {
auto memRefType = memrefDefOp.getType(); auto memRefType = memrefDefOp.getType();
Show All 11 Linesstatic bool isDimOpValidSymbol(DimOp dimOp) {
// The dim op is okay if its operand memref/tensor is defined at the top // The dim op is okay if its operand memref/tensor is defined at the top
// level. // level.
if (isTopLevelValue(dimOp.getOperand())) if (isTopLevelValue(dimOp.getOperand()))
return true; return true;
// The dim op is also okay if its operand memref/tensor is a view/subview // The dim op is also okay if its operand memref/tensor is a view/subview
// whose corresponding size is a valid symbol. // whose corresponding size is a valid symbol.
unsigned index = dimOp.getIndex(); unsigned index = dimOp.getIndex();
if (auto viewOp = dyn_cast<ViewOp>(dimOp.getOperand()->getDefiningOp())) if (auto viewOp = dyn_cast<ViewOp>(dimOp.getOperand().getDefiningOp()))
return isMemRefSizeValidSymbol<ViewOp>(viewOp, index); return isMemRefSizeValidSymbol<ViewOp>(viewOp, index);
if (auto subViewOp = dyn_cast<SubViewOp>(dimOp.getOperand()->getDefiningOp())) if (auto subViewOp = dyn_cast<SubViewOp>(dimOp.getOperand().getDefiningOp()))
return isMemRefSizeValidSymbol<SubViewOp>(subViewOp, index); return isMemRefSizeValidSymbol<SubViewOp>(subViewOp, index);
if (auto allocOp = dyn_cast<AllocOp>(dimOp.getOperand()->getDefiningOp())) if (auto allocOp = dyn_cast<AllocOp>(dimOp.getOperand().getDefiningOp()))
return isMemRefSizeValidSymbol<AllocOp>(allocOp, index); return isMemRefSizeValidSymbol<AllocOp>(allocOp, index);
return false; return false;
} }
// Value can be used as a symbol if it is a constant, or it is defined at // Value can be used as a symbol if it is a constant, or it is defined at
// the top level, or it is a result of affine apply operation with symbol // the top level, or it is a result of affine apply operation with symbol
// arguments, or a result of the dim op on a memref satisfying certain // arguments, or a result of the dim op on a memref satisfying certain
// constraints. // constraints.
bool mlir::isValidSymbol(Value value) { bool mlir::isValidSymbol(Value value) {
// The value must be an index type. // The value must be an index type.
if (!value->getType().isIndex()) if (!value.getType().isIndex())
return false; return false;
if (auto *op = value->getDefiningOp()) { if (auto *op = value.getDefiningOp()) {
// Top level operation or constant operation is ok. // Top level operation or constant operation is ok.
if (isFunctionRegion(op->getParentRegion()) || isa<ConstantOp>(op)) if (isFunctionRegion(op->getParentRegion()) || isa<ConstantOp>(op))
return true; return true;
// Affine apply operation is ok if all of its operands are ok. // Affine apply operation is ok if all of its operands are ok.
if (auto applyOp = dyn_cast<AffineApplyOp>(op)) if (auto applyOp = dyn_cast<AffineApplyOp>(op))
return applyOp.isValidSymbol(); return applyOp.isValidSymbol();
if (auto dimOp = dyn_cast<DimOp>(op)) { if (auto dimOp = dyn_cast<DimOp>(op)) {
return isDimOpValidSymbol(dimOp); return isDimOpValidSymbol(dimOp);
▲ Show 20 Lines • Show All 86 Lines • ▼ Show 20 Lines return emitOpError(
"operand count and affine map dimension and symbol count must match"); "operand count and affine map dimension and symbol count must match");
// Verify that all operands are of `index` type. // Verify that all operands are of `index` type.
for (Type t : getOperandTypes()) { for (Type t : getOperandTypes()) {
if (!t.isIndex()) if (!t.isIndex())
return emitOpError("operands must be of type 'index'"); return emitOpError("operands must be of type 'index'");
} }
if (!getResult()->getType().isIndex()) if (!getResult().getType().isIndex())
return emitOpError("result must be of type 'index'"); return emitOpError("result must be of type 'index'");
// Verify that the map only produces one result. // Verify that the map only produces one result.
if (map.getNumResults() != 1) if (map.getNumResults() != 1)
return emitOpError("mapping must produce one value"); return emitOpError("mapping must produce one value");
return success(); return success();
} }
Show All 32 Lines
AffineDimExpr AffineApplyNormalizer::renumberOneDim(Value v) { AffineDimExpr AffineApplyNormalizer::renumberOneDim(Value v) {
DenseMap<Value, unsigned>::iterator iterPos; DenseMap<Value, unsigned>::iterator iterPos;
bool inserted = false; bool inserted = false;
std::tie(iterPos, inserted) = std::tie(iterPos, inserted) =
dimValueToPosition.insert(std::make_pair(v, dimValueToPosition.size())); dimValueToPosition.insert(std::make_pair(v, dimValueToPosition.size()));
if (inserted) { if (inserted) {
reorderedDims.push_back(v); reorderedDims.push_back(v);
} }
return getAffineDimExpr(iterPos->second, v->getContext()) return getAffineDimExpr(iterPos->second, v.getContext())
.cast<AffineDimExpr>(); .cast<AffineDimExpr>();
} }
AffineMap AffineApplyNormalizer::renumber(const AffineApplyNormalizer &other) { AffineMap AffineApplyNormalizer::renumber(const AffineApplyNormalizer &other) {
SmallVector<AffineExpr, 8> dimRemapping; SmallVector<AffineExpr, 8> dimRemapping;
for (auto v : other.reorderedDims) { for (auto v : other.reorderedDims) {
auto kvp = other.dimValueToPosition.find(v); auto kvp = other.dimValueToPosition.find(v);
if (dimRemapping.size() <= kvp->second) if (dimRemapping.size() <= kvp->second)
Show All 16 Lines return map.replaceDimsAndSymbols(dimRemapping, symRemapping,
concatenatedSymbols.size()); concatenatedSymbols.size());
} }
// Gather the positions of the operands that are produced by an AffineApplyOp. // Gather the positions of the operands that are produced by an AffineApplyOp.
static llvm::SetVector<unsigned> static llvm::SetVector<unsigned>
indicesFromAffineApplyOp(ArrayRef<Value> operands) { indicesFromAffineApplyOp(ArrayRef<Value> operands) {
llvm::SetVector<unsigned> res; llvm::SetVector<unsigned> res;
for (auto en : llvm::enumerate(operands)) for (auto en : llvm::enumerate(operands))
if (isa_and_nonnull<AffineApplyOp>(en.value()->getDefiningOp())) if (isa_and_nonnull<AffineApplyOp>(en.value().getDefiningOp()))
res.insert(en.index()); res.insert(en.index());
return res; return res;
} }
// Support the special case of a symbol coming from an AffineApplyOp that needs // Support the special case of a symbol coming from an AffineApplyOp that needs
// to be composed into the current AffineApplyOp. // to be composed into the current AffineApplyOp.
// This case is handled by rewriting all such symbols into dims for the purpose // This case is handled by rewriting all such symbols into dims for the purpose
// of allowing mathematical AffineMap composition. // of allowing mathematical AffineMap composition.
▲ Show 20 Lines • Show All 105 Lines • ▼ Show 20 LinesAffineApplyNormalizer::AffineApplyNormalizer(AffineMap map,
// We fully spell out the 2 cases below. In this particular instance a little // We fully spell out the 2 cases below. In this particular instance a little
// code duplication greatly improves readability. // code duplication greatly improves readability.
// Note that the first branch would disappear if we only supported full // Note that the first branch would disappear if we only supported full
// composition (i.e. infinite kMaxAffineApplyDepth). // composition (i.e. infinite kMaxAffineApplyDepth).
if (!furtherCompose) { if (!furtherCompose) {
// 1. Only dispatch dims or symbols. // 1. Only dispatch dims or symbols.
for (auto en : llvm::enumerate(operands)) { for (auto en : llvm::enumerate(operands)) {
auto t = en.value(); auto t = en.value();
assert(t->getType().isIndex()); assert(t.getType().isIndex());
bool isDim = (en.index() < map.getNumDims()); bool isDim = (en.index() < map.getNumDims());
if (isDim) { if (isDim) {
// a. The mathematical composition of AffineMap composes dims. // a. The mathematical composition of AffineMap composes dims.
auxiliaryExprs.push_back(renumberOneDim(t)); auxiliaryExprs.push_back(renumberOneDim(t));
} else { } else {
// b. The mathematical composition of AffineMap concatenates symbols. // b. The mathematical composition of AffineMap concatenates symbols.
// We do the same for symbol operands. // We do the same for symbol operands.
concatenatedSymbols.push_back(t); concatenatedSymbols.push_back(t);
} }
} }
} else { } else {
assert(numDimsBeforeRewrite <= operands.size()); assert(numDimsBeforeRewrite <= operands.size());
// 2. Compose AffineApplyOps and dispatch dims or symbols. // 2. Compose AffineApplyOps and dispatch dims or symbols.
for (unsigned i = 0, e = operands.size(); i < e; ++i) { for (unsigned i = 0, e = operands.size(); i < e; ++i) {
auto t = operands[i]; auto t = operands[i];
auto affineApply = dyn_cast_or_null<AffineApplyOp>(t->getDefiningOp()); auto affineApply = dyn_cast_or_null<AffineApplyOp>(t.getDefiningOp());
if (affineApply) { if (affineApply) {
// a. Compose affine.apply operations. // a. Compose affine.apply operations.
LLVM_DEBUG(affineApply.getOperation()->print( LLVM_DEBUG(affineApply.getOperation()->print(
dbgs() << "\nCompose AffineApplyOp recursively: ")); dbgs() << "\nCompose AffineApplyOp recursively: "));
AffineMap affineApplyMap = affineApply.getAffineMap(); AffineMap affineApplyMap = affineApply.getAffineMap();
SmallVector<Value, 8> affineApplyOperands( SmallVector<Value, 8> affineApplyOperands(
affineApply.getOperands().begin(), affineApply.getOperands().end()); affineApply.getOperands().begin(), affineApply.getOperands().end());
AffineApplyNormalizer normalizer(affineApplyMap, affineApplyOperands); AffineApplyNormalizer normalizer(affineApplyMap, affineApplyOperands);
▲ Show 20 Lines • Show All 68 Lines • ▼ Show 20 Linesstatic void composeAffineMapAndOperands(AffineMap *map,
*map = normalizedMap; *map = normalizedMap;
*operands = normalizedOperands; *operands = normalizedOperands;
assert(*map); assert(*map);
} }
void mlir::fullyComposeAffineMapAndOperands(AffineMap *map, void mlir::fullyComposeAffineMapAndOperands(AffineMap *map,
SmallVectorImpl<Value> *operands) { SmallVectorImpl<Value> *operands) {
while (llvm::any_of(*operands, [](Value v) { while (llvm::any_of(*operands, [](Value v) {
return isa_and_nonnull<AffineApplyOp>(v->getDefiningOp()); return isa_and_nonnull<AffineApplyOp>(v.getDefiningOp());
})) { })) {
composeAffineMapAndOperands(map, operands); composeAffineMapAndOperands(map, operands);
} }
} }
AffineApplyOp mlir::makeComposedAffineApply(OpBuilder &b, Location loc, AffineApplyOp mlir::makeComposedAffineApply(OpBuilder &b, Location loc,
AffineMap map, AffineMap map,
ArrayRef<Value> operands) { ArrayRef<Value> operands) {
▲ Show 20 Lines • Show All 214 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// This is a common class used for patterns of the form /// This is a common class used for patterns of the form
/// "someop(memrefcast) -> someop". It folds the source of any memref_cast /// "someop(memrefcast) -> someop". It folds the source of any memref_cast
/// into the root operation directly. /// into the root operation directly.
static LogicalResult foldMemRefCast(Operation *op) { static LogicalResult foldMemRefCast(Operation *op) {
bool folded = false; bool folded = false;
for (OpOperand &operand : op->getOpOperands()) { for (OpOperand &operand : op->getOpOperands()) {
auto cast = dyn_cast_or_null<MemRefCastOp>(operand.get()->getDefiningOp()); auto cast = dyn_cast_or_null<MemRefCastOp>(operand.get().getDefiningOp());
if (cast && !cast.getOperand()->getType().isa<UnrankedMemRefType>()) { if (cast && !cast.getOperand().getType().isa<UnrankedMemRefType>()) {
operand.set(cast.getOperand()); operand.set(cast.getOperand());
folded = true; folded = true;
} }
} }
return success(folded); return success(folded);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 19 Linesvoid AffineDmaStartOp::build(Builder *builder, OperationState &result,
result.addOperands(tagIndices); result.addOperands(tagIndices);
result.addOperands(numElements); result.addOperands(numElements);
if (stride) { if (stride) {
result.addOperands({stride, elementsPerStride}); result.addOperands({stride, elementsPerStride});
} }
} }
void AffineDmaStartOp::print(OpAsmPrinter &p) { void AffineDmaStartOp::print(OpAsmPrinter &p) {
p << "affine.dma_start " << *getSrcMemRef() << '['; p << "affine.dma_start " << getSrcMemRef() << '[';
p.printAffineMapOfSSAIds(getSrcMapAttr(), getSrcIndices()); p.printAffineMapOfSSAIds(getSrcMapAttr(), getSrcIndices());
p << "], " << *getDstMemRef() << '['; p << "], " << getDstMemRef() << '[';
p.printAffineMapOfSSAIds(getDstMapAttr(), getDstIndices()); p.printAffineMapOfSSAIds(getDstMapAttr(), getDstIndices());
p << "], " << *getTagMemRef() << '['; p << "], " << getTagMemRef() << '[';
p.printAffineMapOfSSAIds(getTagMapAttr(), getTagIndices()); p.printAffineMapOfSSAIds(getTagMapAttr(), getTagIndices());
p << "], " << *getNumElements(); p << "], " << getNumElements();
if (isStrided()) { if (isStrided()) {
p << ", " << *getStride(); p << ", " << getStride();
p << ", " << *getNumElementsPerStride(); p << ", " << getNumElementsPerStride();
} }
p << " : " << getSrcMemRefType() << ", " << getDstMemRefType() << ", " p << " : " << getSrcMemRefType() << ", " << getDstMemRefType() << ", "
<< getTagMemRefType(); << getTagMemRefType();
} }
// Parse AffineDmaStartOp. // Parse AffineDmaStartOp.
// Ex: // Ex:
// affine.dma_start %src[%i, %j], %dst[%k, %l], %tag[%index], %size, // affine.dma_start %src[%i, %j], %dst[%k, %l], %tag[%index], %size,
▲ Show 20 Lines • Show All 69 Lines • ▼ Show 20 Lines if (srcMapOperands.size() != srcMapAttr.getValue().getNumInputs() ||
dstMapOperands.size() != dstMapAttr.getValue().getNumInputs() || dstMapOperands.size() != dstMapAttr.getValue().getNumInputs() ||
tagMapOperands.size() != tagMapAttr.getValue().getNumInputs()) tagMapOperands.size() != tagMapAttr.getValue().getNumInputs())
return parser.emitError(parser.getNameLoc(), return parser.emitError(parser.getNameLoc(),
"memref operand count not equal to map.numInputs"); "memref operand count not equal to map.numInputs");
return success(); return success();
} }
LogicalResult AffineDmaStartOp::verify() { LogicalResult AffineDmaStartOp::verify() {
if (!getOperand(getSrcMemRefOperandIndex())->getType().isa<MemRefType>()) if (!getOperand(getSrcMemRefOperandIndex()).getType().isa<MemRefType>())
return emitOpError("expected DMA source to be of memref type"); return emitOpError("expected DMA source to be of memref type");
if (!getOperand(getDstMemRefOperandIndex())->getType().isa<MemRefType>()) if (!getOperand(getDstMemRefOperandIndex()).getType().isa<MemRefType>())
return emitOpError("expected DMA destination to be of memref type"); return emitOpError("expected DMA destination to be of memref type");
if (!getOperand(getTagMemRefOperandIndex())->getType().isa<MemRefType>()) if (!getOperand(getTagMemRefOperandIndex()).getType().isa<MemRefType>())
return emitOpError("expected DMA tag to be of memref type"); return emitOpError("expected DMA tag to be of memref type");
// DMAs from different memory spaces supported. // DMAs from different memory spaces supported.
if (getSrcMemorySpace() == getDstMemorySpace()) { if (getSrcMemorySpace() == getDstMemorySpace()) {
return emitOpError("DMA should be between different memory spaces"); return emitOpError("DMA should be between different memory spaces");
} }
unsigned numInputsAllMaps = getSrcMap().getNumInputs() + unsigned numInputsAllMaps = getSrcMap().getNumInputs() +
getDstMap().getNumInputs() + getDstMap().getNumInputs() +
getTagMap().getNumInputs(); getTagMap().getNumInputs();
if (getNumOperands() != numInputsAllMaps + 3 + 1 && if (getNumOperands() != numInputsAllMaps + 3 + 1 &&
getNumOperands() != numInputsAllMaps + 3 + 1 + 2) { getNumOperands() != numInputsAllMaps + 3 + 1 + 2) {
return emitOpError("incorrect number of operands"); return emitOpError("incorrect number of operands");
} }
for (auto idx : getSrcIndices()) { for (auto idx : getSrcIndices()) {
if (!idx->getType().isIndex()) if (!idx.getType().isIndex())
return emitOpError("src index to dma_start must have 'index' type"); return emitOpError("src index to dma_start must have 'index' type");
if (!isValidAffineIndexOperand(idx)) if (!isValidAffineIndexOperand(idx))
return emitOpError("src index must be a dimension or symbol identifier"); return emitOpError("src index must be a dimension or symbol identifier");
} }
for (auto idx : getDstIndices()) { for (auto idx : getDstIndices()) {
if (!idx->getType().isIndex()) if (!idx.getType().isIndex())
return emitOpError("dst index to dma_start must have 'index' type"); return emitOpError("dst index to dma_start must have 'index' type");
if (!isValidAffineIndexOperand(idx)) if (!isValidAffineIndexOperand(idx))
return emitOpError("dst index must be a dimension or symbol identifier"); return emitOpError("dst index must be a dimension or symbol identifier");
} }
for (auto idx : getTagIndices()) { for (auto idx : getTagIndices()) {
if (!idx->getType().isIndex()) if (!idx.getType().isIndex())
return emitOpError("tag index to dma_start must have 'index' type"); return emitOpError("tag index to dma_start must have 'index' type");
if (!isValidAffineIndexOperand(idx)) if (!isValidAffineIndexOperand(idx))
return emitOpError("tag index must be a dimension or symbol identifier"); return emitOpError("tag index must be a dimension or symbol identifier");
} }
return success(); return success();
} }
LogicalResult AffineDmaStartOp::fold(ArrayRef<Attribute> cstOperands, LogicalResult AffineDmaStartOp::fold(ArrayRef<Attribute> cstOperands,
Show All 12 Linesvoid AffineDmaWaitOp::build(Builder *builder, OperationState &result,
ValueRange tagIndices, Value numElements) { ValueRange tagIndices, Value numElements) {
result.addOperands(tagMemRef); result.addOperands(tagMemRef);
result.addAttribute(getTagMapAttrName(), AffineMapAttr::get(tagMap)); result.addAttribute(getTagMapAttrName(), AffineMapAttr::get(tagMap));
result.addOperands(tagIndices); result.addOperands(tagIndices);
result.addOperands(numElements); result.addOperands(numElements);
} }
void AffineDmaWaitOp::print(OpAsmPrinter &p) { void AffineDmaWaitOp::print(OpAsmPrinter &p) {
p << "affine.dma_wait " << *getTagMemRef() << '['; p << "affine.dma_wait " << getTagMemRef() << '[';
SmallVector<Value, 2> operands(getTagIndices()); SmallVector<Value, 2> operands(getTagIndices());
p.printAffineMapOfSSAIds(getTagMapAttr(), operands); p.printAffineMapOfSSAIds(getTagMapAttr(), operands);
p << "], "; p << "], ";
p.printOperand(getNumElements()); p.printOperand(getNumElements());
p << " : " << getTagMemRef()->getType(); p << " : " << getTagMemRef().getType();
} }
// Parse AffineDmaWaitOp. // Parse AffineDmaWaitOp.
// Eg: // Eg:
// affine.dma_wait %tag[%index], %num_elements // affine.dma_wait %tag[%index], %num_elements
// : memref<1 x i32, (d0) -> (d0), 4> // : memref<1 x i32, (d0) -> (d0), 4>
// //
ParseResult AffineDmaWaitOp::parse(OpAsmParser &parser, ParseResult AffineDmaWaitOp::parse(OpAsmParser &parser,
Show All 22 LinesParseResult AffineDmaWaitOp::parse(OpAsmParser &parser,
if (tagMapOperands.size() != tagMapAttr.getValue().getNumInputs()) if (tagMapOperands.size() != tagMapAttr.getValue().getNumInputs())
return parser.emitError(parser.getNameLoc(), return parser.emitError(parser.getNameLoc(),
"tag memref operand count != to map.numInputs"); "tag memref operand count != to map.numInputs");
return success(); return success();
} }
LogicalResult AffineDmaWaitOp::verify() { LogicalResult AffineDmaWaitOp::verify() {
if (!getOperand(0)->getType().isa<MemRefType>()) if (!getOperand(0).getType().isa<MemRefType>())
return emitOpError("expected DMA tag to be of memref type"); return emitOpError("expected DMA tag to be of memref type");
for (auto idx : getTagIndices()) { for (auto idx : getTagIndices()) {
if (!idx->getType().isIndex()) if (!idx.getType().isIndex())
return emitOpError("index to dma_wait must have 'index' type"); return emitOpError("index to dma_wait must have 'index' type");
if (!isValidAffineIndexOperand(idx)) if (!isValidAffineIndexOperand(idx))
return emitOpError("index must be a dimension or symbol identifier"); return emitOpError("index must be a dimension or symbol identifier");
} }
return success(); return success();
} }
LogicalResult AffineDmaWaitOp::fold(ArrayRef<Attribute> cstOperands, LogicalResult AffineDmaWaitOp::fold(ArrayRef<Attribute> cstOperands,
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Linesvoid AffineForOp::build(Builder *builder, OperationState &result, int64_t lb,
auto ubMap = AffineMap::getConstantMap(ub, builder->getContext()); auto ubMap = AffineMap::getConstantMap(ub, builder->getContext());
return build(builder, result, {}, lbMap, {}, ubMap, step); return build(builder, result, {}, lbMap, {}, ubMap, step);
} }
static LogicalResult verify(AffineForOp op) { static LogicalResult verify(AffineForOp op) {
// Check that the body defines as single block argument for the induction // Check that the body defines as single block argument for the induction
// variable. // variable.
auto *body = op.getBody(); auto *body = op.getBody();
if (body->getNumArguments() != 1 || if (body->getNumArguments() != 1 || !body->getArgument(0).getType().isIndex())
!body->getArgument(0)->getType().isIndex())
return op.emitOpError( return op.emitOpError(
"expected body to have a single index argument for the " "expected body to have a single index argument for the "
"induction variable"); "induction variable");
// Verify that there are enough operands for the bounds. // Verify that there are enough operands for the bounds.
AffineMap lowerBoundMap = op.getLowerBoundMap(), AffineMap lowerBoundMap = op.getLowerBoundMap(),
upperBoundMap = op.getUpperBoundMap(); upperBoundMap = op.getUpperBoundMap();
if (op.getNumOperands() != if (op.getNumOperands() !=
▲ Show 20 Lines • Show All 412 Lines • ▼ Show 20 Lines if (getOperand(i) != getOperand(numOperands + i))
return false; return false;
} }
return true; return true;
} }
Region &AffineForOp::getLoopBody() { return region(); } Region &AffineForOp::getLoopBody() { return region(); }
bool AffineForOp::isDefinedOutsideOfLoop(Value value) { bool AffineForOp::isDefinedOutsideOfLoop(Value value) {
return !region().isAncestor(value->getParentRegion()); return !region().isAncestor(value.getParentRegion());
} }
LogicalResult AffineForOp::moveOutOfLoop(ArrayRef<Operation *> ops) { LogicalResult AffineForOp::moveOutOfLoop(ArrayRef<Operation *> ops) {
for (auto *op : ops) for (auto *op : ops)
op->moveBefore(*this); op->moveBefore(*this);
return success(); return success();
} }
/// Returns if the provided value is the induction variable of a AffineForOp. /// Returns if the provided value is the induction variable of a AffineForOp.
bool mlir::isForInductionVar(Value val) { bool mlir::isForInductionVar(Value val) {
return getForInductionVarOwner(val) != AffineForOp(); return getForInductionVarOwner(val) != AffineForOp();
} }
/// Returns the loop parent of an induction variable. If the provided value is /// Returns the loop parent of an induction variable. If the provided value is
/// not an induction variable, then return nullptr. /// not an induction variable, then return nullptr.
AffineForOp mlir::getForInductionVarOwner(Value val) { AffineForOp mlir::getForInductionVarOwner(Value val) {
auto ivArg = val.dyn_cast<BlockArgument>(); auto ivArg = val.dyn_cast<BlockArgument>();
if (!ivArg || !ivArg->getOwner()) if (!ivArg || !ivArg.getOwner())
return AffineForOp(); return AffineForOp();
auto *containingInst = ivArg->getOwner()->getParent()->getParentOp(); auto *containingInst = ivArg.getOwner()->getParent()->getParentOp();
return dyn_cast<AffineForOp>(containingInst); return dyn_cast<AffineForOp>(containingInst);
} }
/// Extracts the induction variables from a list of AffineForOps and returns /// Extracts the induction variables from a list of AffineForOps and returns
/// them. /// them.
void mlir::extractForInductionVars(ArrayRef<AffineForOp> forInsts, void mlir::extractForInductionVars(ArrayRef<AffineForOp> forInsts,
SmallVectorImpl<Value> *ivs) { SmallVectorImpl<Value> *ivs) {
ivs->reserve(forInsts.size()); ivs->reserve(forInsts.size());
▲ Show 20 Lines • Show All 154 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void AffineLoadOp::build(Builder *builder, OperationState &result, void AffineLoadOp::build(Builder *builder, OperationState &result,
AffineMap map, ValueRange operands) { AffineMap map, ValueRange operands) {
assert(operands.size() == 1 + map.getNumInputs() && "inconsistent operands"); assert(operands.size() == 1 + map.getNumInputs() && "inconsistent operands");
result.addOperands(operands); result.addOperands(operands);
if (map) if (map)
result.addAttribute(getMapAttrName(), AffineMapAttr::get(map)); result.addAttribute(getMapAttrName(), AffineMapAttr::get(map));
auto memrefType = operands[0]->getType().cast<MemRefType>(); auto memrefType = operands[0].getType().cast<MemRefType>();
result.types.push_back(memrefType.getElementType()); result.types.push_back(memrefType.getElementType());
} }
void AffineLoadOp::build(Builder *builder, OperationState &result, Value memref, void AffineLoadOp::build(Builder *builder, OperationState &result, Value memref,
AffineMap map, ValueRange mapOperands) { AffineMap map, ValueRange mapOperands) {
assert(map.getNumInputs() == mapOperands.size() && "inconsistent index info"); assert(map.getNumInputs() == mapOperands.size() && "inconsistent index info");
result.addOperands(memref); result.addOperands(memref);
result.addOperands(mapOperands); result.addOperands(mapOperands);
auto memrefType = memref->getType().cast<MemRefType>(); auto memrefType = memref.getType().cast<MemRefType>();
result.addAttribute(getMapAttrName(), AffineMapAttr::get(map)); result.addAttribute(getMapAttrName(), AffineMapAttr::get(map));
result.types.push_back(memrefType.getElementType()); result.types.push_back(memrefType.getElementType());
} }
void AffineLoadOp::build(Builder *builder, OperationState &result, Value memref, void AffineLoadOp::build(Builder *builder, OperationState &result, Value memref,
ValueRange indices) { ValueRange indices) {
auto memrefType = memref->getType().cast<MemRefType>(); auto memrefType = memref.getType().cast<MemRefType>();
auto rank = memrefType.getRank(); auto rank = memrefType.getRank();
// Create identity map for memrefs with at least one dimension or () -> () // Create identity map for memrefs with at least one dimension or () -> ()
// for zero-dimensional memrefs. // for zero-dimensional memrefs.
auto map = rank ? builder->getMultiDimIdentityMap(rank) auto map = rank ? builder->getMultiDimIdentityMap(rank)
: builder->getEmptyAffineMap(); : builder->getEmptyAffineMap();
build(builder, result, memref, map, indices); build(builder, result, memref, map, indices);
} }
Show All 12 Lines return failure(
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperand(memrefInfo, type, result.operands) || parser.resolveOperand(memrefInfo, type, result.operands) ||
parser.resolveOperands(mapOperands, indexTy, result.operands) || parser.resolveOperands(mapOperands, indexTy, result.operands) ||
parser.addTypeToList(type.getElementType(), result.types)); parser.addTypeToList(type.getElementType(), result.types));
} }
void AffineLoadOp::print(OpAsmPrinter &p) { void AffineLoadOp::print(OpAsmPrinter &p) {
p << "affine.load " << *getMemRef() << '['; p << "affine.load " << getMemRef() << '[';
if (AffineMapAttr mapAttr = getAttrOfType<AffineMapAttr>(getMapAttrName())) if (AffineMapAttr mapAttr = getAttrOfType<AffineMapAttr>(getMapAttrName()))
p.printAffineMapOfSSAIds(mapAttr, getMapOperands()); p.printAffineMapOfSSAIds(mapAttr, getMapOperands());
p << ']'; p << ']';
p.printOptionalAttrDict(getAttrs(), /*elidedAttrs=*/{getMapAttrName()}); p.printOptionalAttrDict(getAttrs(), /*elidedAttrs=*/{getMapAttrName()});
p << " : " << getMemRefType(); p << " : " << getMemRefType();
} }
LogicalResult AffineLoadOp::verify() { LogicalResult AffineLoadOp::verify() {
Show All 10 Lines if (map.getNumInputs() != getNumOperands() - 1)
return emitOpError("expects as many subscripts as affine map inputs"); return emitOpError("expects as many subscripts as affine map inputs");
} else { } else {
if (getMemRefType().getRank() != getNumOperands() - 1) if (getMemRefType().getRank() != getNumOperands() - 1)
return emitOpError( return emitOpError(
"expects the number of subscripts to be equal to memref rank"); "expects the number of subscripts to be equal to memref rank");
} }
for (auto idx : getMapOperands()) { for (auto idx : getMapOperands()) {
if (!idx->getType().isIndex()) if (!idx.getType().isIndex())
return emitOpError("index to load must have 'index' type"); return emitOpError("index to load must have 'index' type");
if (!isValidAffineIndexOperand(idx)) if (!isValidAffineIndexOperand(idx))
return emitOpError("index must be a dimension or symbol identifier"); return emitOpError("index must be a dimension or symbol identifier");
} }
return success(); return success();
} }
void AffineLoadOp::getCanonicalizationPatterns( void AffineLoadOp::getCanonicalizationPatterns(
Show All 21 Linesvoid AffineStoreOp::build(Builder *builder, OperationState &result,
result.addOperands(mapOperands); result.addOperands(mapOperands);
result.addAttribute(getMapAttrName(), AffineMapAttr::get(map)); result.addAttribute(getMapAttrName(), AffineMapAttr::get(map));
} }
// Use identity map. // Use identity map.
void AffineStoreOp::build(Builder *builder, OperationState &result, void AffineStoreOp::build(Builder *builder, OperationState &result,
Value valueToStore, Value memref, Value valueToStore, Value memref,
ValueRange indices) { ValueRange indices) {
auto memrefType = memref->getType().cast<MemRefType>(); auto memrefType = memref.getType().cast<MemRefType>();
auto rank = memrefType.getRank(); auto rank = memrefType.getRank();
// Create identity map for memrefs with at least one dimension or () -> () // Create identity map for memrefs with at least one dimension or () -> ()
// for zero-dimensional memrefs. // for zero-dimensional memrefs.
auto map = rank ? builder->getMultiDimIdentityMap(rank) auto map = rank ? builder->getMultiDimIdentityMap(rank)
: builder->getEmptyAffineMap(); : builder->getEmptyAffineMap();
build(builder, result, valueToStore, memref, map, indices); build(builder, result, valueToStore, memref, map, indices);
} }
Show All 14 Lines return failure(parser.parseOperand(storeValueInfo) || parser.parseComma() ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperand(storeValueInfo, type.getElementType(), parser.resolveOperand(storeValueInfo, type.getElementType(),
result.operands) || result.operands) ||
parser.resolveOperand(memrefInfo, type, result.operands) || parser.resolveOperand(memrefInfo, type, result.operands) ||
parser.resolveOperands(mapOperands, indexTy, result.operands)); parser.resolveOperands(mapOperands, indexTy, result.operands));
} }
void AffineStoreOp::print(OpAsmPrinter &p) { void AffineStoreOp::print(OpAsmPrinter &p) {
p << "affine.store " << *getValueToStore(); p << "affine.store " << getValueToStore();
p << ", " << *getMemRef() << '['; p << ", " << getMemRef() << '[';
if (AffineMapAttr mapAttr = getAttrOfType<AffineMapAttr>(getMapAttrName())) if (AffineMapAttr mapAttr = getAttrOfType<AffineMapAttr>(getMapAttrName()))
p.printAffineMapOfSSAIds(mapAttr, getMapOperands()); p.printAffineMapOfSSAIds(mapAttr, getMapOperands());
p << ']'; p << ']';
p.printOptionalAttrDict(getAttrs(), /*elidedAttrs=*/{getMapAttrName()}); p.printOptionalAttrDict(getAttrs(), /*elidedAttrs=*/{getMapAttrName()});
p << " : " << getMemRefType(); p << " : " << getMemRefType();
} }
LogicalResult AffineStoreOp::verify() { LogicalResult AffineStoreOp::verify() {
// First operand must have same type as memref element type. // First operand must have same type as memref element type.
if (getValueToStore()->getType() != getMemRefType().getElementType()) if (getValueToStore().getType() != getMemRefType().getElementType())
return emitOpError("first operand must have same type memref element type"); return emitOpError("first operand must have same type memref element type");
auto mapAttr = getAttrOfType<AffineMapAttr>(getMapAttrName()); auto mapAttr = getAttrOfType<AffineMapAttr>(getMapAttrName());
if (mapAttr) { if (mapAttr) {
AffineMap map = mapAttr.getValue(); AffineMap map = mapAttr.getValue();
if (map.getNumResults() != getMemRefType().getRank()) if (map.getNumResults() != getMemRefType().getRank())
return emitOpError("affine.store affine map num results must equal" return emitOpError("affine.store affine map num results must equal"
" memref rank"); " memref rank");
if (map.getNumInputs() != getNumOperands() - 2) if (map.getNumInputs() != getNumOperands() - 2)
return emitOpError("expects as many subscripts as affine map inputs"); return emitOpError("expects as many subscripts as affine map inputs");
} else { } else {
if (getMemRefType().getRank() != getNumOperands() - 2) if (getMemRefType().getRank() != getNumOperands() - 2)
return emitOpError( return emitOpError(
"expects the number of subscripts to be equal to memref rank"); "expects the number of subscripts to be equal to memref rank");
} }
for (auto idx : getMapOperands()) { for (auto idx : getMapOperands()) {
if (!idx->getType().isIndex()) if (!idx.getType().isIndex())
return emitOpError("index to store must have 'index' type"); return emitOpError("index to store must have 'index' type");
if (!isValidAffineIndexOperand(idx)) if (!isValidAffineIndexOperand(idx))
return emitOpError("index must be a dimension or symbol identifier"); return emitOpError("index must be a dimension or symbol identifier");
} }
return success(); return success();
} }
void AffineStoreOp::getCanonicalizationPatterns( void AffineStoreOp::getCanonicalizationPatterns(
▲ Show 20 Lines • Show All 128 Lines • ▼ Show 20 Linesstatic ParseResult parseAffinePrefetchOp(OpAsmParser &parser,
result.addAttribute( result.addAttribute(
AffinePrefetchOp::getIsDataCacheAttrName(), AffinePrefetchOp::getIsDataCacheAttrName(),
parser.getBuilder().getBoolAttr(cacheType.equals("data"))); parser.getBuilder().getBoolAttr(cacheType.equals("data")));
return success(); return success();
} }
void print(OpAsmPrinter &p, AffinePrefetchOp op) { void print(OpAsmPrinter &p, AffinePrefetchOp op) {
p << AffinePrefetchOp::getOperationName() << " " << *op.memref() << '['; p << AffinePrefetchOp::getOperationName() << " " << op.memref() << '[';
AffineMapAttr mapAttr = op.getAttrOfType<AffineMapAttr>(op.getMapAttrName()); AffineMapAttr mapAttr = op.getAttrOfType<AffineMapAttr>(op.getMapAttrName());
if (mapAttr) { if (mapAttr) {
SmallVector<Value, 2> operands(op.getMapOperands()); SmallVector<Value, 2> operands(op.getMapOperands());
p.printAffineMapOfSSAIds(mapAttr, operands); p.printAffineMapOfSSAIds(mapAttr, operands);
} }
p << ']' << ", " << (op.isWrite() ? "write" : "read") << ", " p << ']' << ", " << (op.isWrite() ? "write" : "read") << ", "
<< "locality<" << op.localityHint() << ">, " << "locality<" << op.localityHint() << ">, "
<< (op.isDataCache() ? "data" : "instr"); << (op.isDataCache() ? "data" : "instr");
▲ Show 20 Lines • Show All 46 Lines • Show Last 20 Lines

mlir/lib/Dialect/FxpMathOps/Transforms/LowerUniformRealMath.cpp

Show First 20 Lines • Show All 41 Lines • ▼ Show 20 Linesstatic Value emitUniformPerLayerDequantize(Location loc, Value input,
PatternRewriter &rewriter) { PatternRewriter &rewriter) {
// Pre-conditions. // Pre-conditions.
if (!elementType.isSigned()) { if (!elementType.isSigned()) {
// TODO: Support unsigned storage type. // TODO: Support unsigned storage type.
emitWarning(loc, "unimplemented: dequantize signed uniform"); emitWarning(loc, "unimplemented: dequantize signed uniform");
return nullptr; return nullptr;
} }
Type storageType = elementType.castToStorageType(input->getType()); Type storageType = elementType.castToStorageType(input.getType());
Type realType = elementType.castToExpressedType(input->getType()); Type realType = elementType.castToExpressedType(input.getType());
Type intermediateType = Type intermediateType =
castElementType(storageType, IntegerType::get(32, rewriter.getContext())); castElementType(storageType, IntegerType::get(32, rewriter.getContext()));
assert(storageType && "cannot cast to storage type"); assert(storageType && "cannot cast to storage type");
assert(realType && "cannot cast to expressed type"); assert(realType && "cannot cast to expressed type");
// Cast to storage type. // Cast to storage type.
input = rewriter.create<StorageCastOp>(loc, storageType, input); input = rewriter.create<StorageCastOp>(loc, storageType, input);
Show All 25 LinesemitUniformPerAxisDequantize(Location loc, Value input,
// TODO: Support per-axis dequantize. // TODO: Support per-axis dequantize.
rewriter.getContext()->getDiagEngine().emit(loc, DiagnosticSeverity::Warning) rewriter.getContext()->getDiagEngine().emit(loc, DiagnosticSeverity::Warning)
<< "unimplemented: per-axis uniform dequantization"; << "unimplemented: per-axis uniform dequantization";
return nullptr; return nullptr;
} }
static Value emitDequantize(Location loc, Value input, static Value emitDequantize(Location loc, Value input,
PatternRewriter &rewriter) { PatternRewriter &rewriter) {
Type inputType = input->getType(); Type inputType = input.getType();
QuantizedType qElementType = QuantizedType qElementType =
QuantizedType::getQuantizedElementType(inputType); QuantizedType::getQuantizedElementType(inputType);
if (auto uperLayerElementType = if (auto uperLayerElementType =
qElementType.dyn_cast_or_null<UniformQuantizedType>()) { qElementType.dyn_cast_or_null<UniformQuantizedType>()) {
return emitUniformPerLayerDequantize(loc, input, uperLayerElementType, return emitUniformPerLayerDequantize(loc, input, uperLayerElementType,
rewriter); rewriter);
} else if (auto uperAxisElementType = } else if (auto uperAxisElementType =
qElementType.dyn_cast_or_null<UniformQuantizedPerAxisType>()) { qElementType.dyn_cast_or_null<UniformQuantizedPerAxisType>()) {
return emitUniformPerAxisDequantize(loc, input, uperAxisElementType, return emitUniformPerAxisDequantize(loc, input, uperAxisElementType,
rewriter); rewriter);
} else { } else {
return nullptr; return nullptr;
} }
} }
namespace { namespace {
struct UniformDequantizePattern : public OpRewritePattern<DequantizeCastOp> { struct UniformDequantizePattern : public OpRewritePattern<DequantizeCastOp> {
using OpRewritePattern<DequantizeCastOp>::OpRewritePattern; using OpRewritePattern<DequantizeCastOp>::OpRewritePattern;
PatternMatchResult matchAndRewrite(DequantizeCastOp op, PatternMatchResult matchAndRewrite(DequantizeCastOp op,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
Type inputType = op.arg()->getType(); Type inputType = op.arg().getType();
Type outputType = op.getResult()->getType(); Type outputType = op.getResult().getType();
QuantizedType inputElementType = QuantizedType inputElementType =
QuantizedType::getQuantizedElementType(inputType); QuantizedType::getQuantizedElementType(inputType);
Type expressedOutputType = inputElementType.castToExpressedType(inputType); Type expressedOutputType = inputElementType.castToExpressedType(inputType);
if (expressedOutputType != outputType) { if (expressedOutputType != outputType) {
// Not a valid uniform cast. // Not a valid uniform cast.
return matchFailure(); return matchFailure();
} }
▲ Show 20 Lines • Show All 268 Lines • Show Last 20 Lines

mlir/lib/Dialect/FxpMathOps/Transforms/UniformKernelUtils.h

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 Lines
} }
/// Helper class for operating on binary operations where all operands /// Helper class for operating on binary operations where all operands
/// and the result are a UniformQuantizedType. /// and the result are a UniformQuantizedType.
struct UniformBinaryOpInfo { struct UniformBinaryOpInfo {
UniformBinaryOpInfo(Operation *op, Value lhs, Value rhs, UniformBinaryOpInfo(Operation *op, Value lhs, Value rhs,
Optional<APFloat> clampMin, Optional<APFloat> clampMax) Optional<APFloat> clampMin, Optional<APFloat> clampMax)
: op(op), lhs(lhs), rhs(rhs), clampMin(clampMin), clampMax(clampMax), : op(op), lhs(lhs), rhs(rhs), clampMin(clampMin), clampMax(clampMax),
lhsType(getUniformElementType(lhs->getType())), lhsType(getUniformElementType(lhs.getType())),
rhsType(getUniformElementType(rhs->getType())), rhsType(getUniformElementType(rhs.getType())),
resultType(getUniformElementType(*op->result_type_begin())), resultType(getUniformElementType(*op->result_type_begin())),
lhsStorageType(quant::QuantizedType::castToStorageType(lhs->getType())), lhsStorageType(quant::QuantizedType::castToStorageType(lhs.getType())),
rhsStorageType(quant::QuantizedType::castToStorageType(rhs->getType())), rhsStorageType(quant::QuantizedType::castToStorageType(rhs.getType())),
resultStorageType( resultStorageType(
quant::QuantizedType::castToStorageType(*op->result_type_begin())) { quant::QuantizedType::castToStorageType(*op->result_type_begin())) {
} }
/// Returns whether this info is valid (all types defined, etc). /// Returns whether this info is valid (all types defined, etc).
bool isValid() const { bool isValid() const {
return lhsType && rhsType && resultType && lhsStorageType && return lhsType && rhsType && resultType && lhsStorageType &&
rhsStorageType && resultStorageType; rhsStorageType && resultStorageType;
▲ Show 20 Lines • Show All 159 Lines • Show Last 20 Lines

mlir/lib/Dialect/GPU/IR/GPUDialect.cpp

Show First 20 Lines • Show All 104 Lines • ▼ Show 20 Lines if (expectedNumArguments != actualNumArguments)
<< expectedNumArguments; << expectedNumArguments;
// Due to the ordering of the current impl of lowering and LLVMLowering, // Due to the ordering of the current impl of lowering and LLVMLowering,
// type checks need to be temporarily disabled. // type checks need to be temporarily disabled.
// TODO(ntv,zinenko,herhut): reactivate checks once "changing gpu.launchFunc // TODO(ntv,zinenko,herhut): reactivate checks once "changing gpu.launchFunc
// to encode target module" has landed. // to encode target module" has landed.
// auto functionType = kernelFunc.getType(); // auto functionType = kernelFunc.getType();
// for (unsigned i = 0; i < numKernelFuncArgs; ++i) { // for (unsigned i = 0; i < numKernelFuncArgs; ++i) {
// if (getKernelOperand(i)->getType() != functionType.getInput(i)) { // if (getKernelOperand(i).getType() != functionType.getInput(i)) {
// return emitOpError("type of function argument ") // return emitOpError("type of function argument ")
// << i << " does not match"; // << i << " does not match";
// } // }
// } // }
return success(); return success();
}); });
Show All 10 Lines
static LogicalResult verifyAllReduce(gpu::AllReduceOp allReduce) { static LogicalResult verifyAllReduce(gpu::AllReduceOp allReduce) {
if (allReduce.body().empty() != allReduce.op().hasValue()) if (allReduce.body().empty() != allReduce.op().hasValue())
return allReduce.emitError( return allReduce.emitError(
"expected either an op attribute or a non-empty body"); "expected either an op attribute or a non-empty body");
if (!allReduce.body().empty()) { if (!allReduce.body().empty()) {
if (allReduce.body().front().getNumArguments() != 2) if (allReduce.body().front().getNumArguments() != 2)
return allReduce.emitError("expected two region arguments"); return allReduce.emitError("expected two region arguments");
for (auto argument : allReduce.body().front().getArguments()) { for (auto argument : allReduce.body().front().getArguments()) {
if (argument->getType() != allReduce.getType()) if (argument.getType() != allReduce.getType())
return allReduce.emitError("incorrect region argument type"); return allReduce.emitError("incorrect region argument type");
} }
unsigned yieldCount = 0; unsigned yieldCount = 0;
for (Block &block : allReduce.body()) { for (Block &block : allReduce.body()) {
if (auto yield = dyn_cast<gpu::YieldOp>(block.getTerminator())) { if (auto yield = dyn_cast<gpu::YieldOp>(block.getTerminator())) {
if (yield.getNumOperands() != 1) if (yield.getNumOperands() != 1)
return allReduce.emitError("expected one gpu.yield operand"); return allReduce.emitError("expected one gpu.yield operand");
if (yield.getOperand(0)->getType() != allReduce.getType()) if (yield.getOperand(0).getType() != allReduce.getType())
return allReduce.emitError("incorrect gpu.yield type"); return allReduce.emitError("incorrect gpu.yield type");
++yieldCount; ++yieldCount;
} }
} }
if (yieldCount == 0) if (yieldCount == 0)
return allReduce.emitError("expected gpu.yield op in region"); return allReduce.emitError("expected gpu.yield op in region");
} }
return success(); return success();
} }
static LogicalResult verifyShuffleOp(gpu::ShuffleOp shuffleOp) { static LogicalResult verifyShuffleOp(gpu::ShuffleOp shuffleOp) {
auto type = shuffleOp.value()->getType(); auto type = shuffleOp.value().getType();
if (shuffleOp.result()->getType() != type) { if (shuffleOp.result().getType() != type) {
return shuffleOp.emitOpError() return shuffleOp.emitOpError()
<< "requires the same type for value operand and result"; << "requires the same type for value operand and result";
} }
if (!type.isIntOrFloat() || type.getIntOrFloatBitWidth() != 32) { if (!type.isIntOrFloat() || type.getIntOrFloatBitWidth() != 32) {
return shuffleOp.emitOpError() return shuffleOp.emitOpError()
<< "requires value operand type to be f32 or i32"; << "requires value operand type to be f32 or i32";
} }
return success(); return success();
} }
static void printShuffleOp(OpAsmPrinter &p, ShuffleOp op) { static void printShuffleOp(OpAsmPrinter &p, ShuffleOp op) {
p << ShuffleOp::getOperationName() << ' '; p << ShuffleOp::getOperationName() << ' ' << op.getOperands() << ' '
p.printOperands(op.getOperands()); << op.mode() << " : " << op.value().getType();
p << ' ' << op.mode() << " : ";
p.printType(op.value()->getType());
} }
static ParseResult parseShuffleOp(OpAsmParser &parser, OperationState &state) { static ParseResult parseShuffleOp(OpAsmParser &parser, OperationState &state) {
SmallVector<OpAsmParser::OperandType, 3> operandInfo; SmallVector<OpAsmParser::OperandType, 3> operandInfo;
if (parser.parseOperandList(operandInfo, 3)) if (parser.parseOperandList(operandInfo, 3))
return failure(); return failure();
StringRef mode; StringRef mode;
Show All 11 Lines if (parser.parseColonType(valueType) ||
return failure(); return failure();
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// LaunchOp // LaunchOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static SmallVector<Type, 4> getValueTypes(ValueRange values) {
SmallVector<Type, 4> types;
types.reserve(values.size());
for (Value v : values)
types.push_back(v->getType());
return types;
}
void LaunchOp::build(Builder *builder, OperationState &result, Value gridSizeX, void LaunchOp::build(Builder *builder, OperationState &result, Value gridSizeX,
Value gridSizeY, Value gridSizeZ, Value blockSizeX, Value gridSizeY, Value gridSizeZ, Value blockSizeX,
Value blockSizeY, Value blockSizeZ, ValueRange operands) { Value blockSizeY, Value blockSizeZ, ValueRange operands) {
// Add grid and block sizes as op operands, followed by the data operands. // Add grid and block sizes as op operands, followed by the data operands.
result.addOperands( result.addOperands(
{gridSizeX, gridSizeY, gridSizeZ, blockSizeX, blockSizeY, blockSizeZ}); {gridSizeX, gridSizeY, gridSizeZ, blockSizeX, blockSizeY, blockSizeZ});
result.addOperands(operands); result.addOperands(operands);
// Create a kernel body region with kNumConfigRegionAttributes + N arguments, // Create a kernel body region with kNumConfigRegionAttributes + N arguments,
// where the first kNumConfigRegionAttributes arguments have `index` type and // where the first kNumConfigRegionAttributes arguments have `index` type and
// the rest have the same types as the data operands. // the rest have the same types as the data operands.
Region *kernelRegion = result.addRegion(); Region *kernelRegion = result.addRegion();
Block *body = new Block(); Block *body = new Block();
body->addArguments( body->addArguments(
std::vector<Type>(kNumConfigRegionAttributes, builder->getIndexType())); std::vector<Type>(kNumConfigRegionAttributes, builder->getIndexType()));
body->addArguments(getValueTypes(operands)); body->addArguments(llvm::to_vector<4>(operands.getTypes()));
kernelRegion->push_back(body); kernelRegion->push_back(body);
} }
KernelDim3 LaunchOp::getBlockIds() { KernelDim3 LaunchOp::getBlockIds() {
assert(!body().getBlocks().empty() && "FuncOp body must not be empty."); assert(!body().getBlocks().empty() && "FuncOp body must not be empty.");
auto args = body().getBlocks().front().getArguments(); auto args = body().getBlocks().front().getArguments();
return KernelDim3{args[0], args[1], args[2]}; return KernelDim3{args[0], args[1], args[2]};
} }
▲ Show 20 Lines • Show All 68 Lines • ▼ Show 20 Lines
} }
// Pretty-print the kernel grid/block size assignment as // Pretty-print the kernel grid/block size assignment as
// (%iter-x, %iter-y, %iter-z) in // (%iter-x, %iter-y, %iter-z) in
// (%size-x = %ssa-use, %size-y = %ssa-use, %size-z = %ssa-use) // (%size-x = %ssa-use, %size-y = %ssa-use, %size-z = %ssa-use)
// where %size-* and %iter-* will correspond to the body region arguments. // where %size-* and %iter-* will correspond to the body region arguments.
static void printSizeAssignment(OpAsmPrinter &p, KernelDim3 size, static void printSizeAssignment(OpAsmPrinter &p, KernelDim3 size,
ValueRange operands, KernelDim3 ids) { ValueRange operands, KernelDim3 ids) {
p << '(' << *ids.x << ", " << *ids.y << ", " << *ids.z << ") in ("; p << '(' << ids.x << ", " << ids.y << ", " << ids.z << ") in (";
p << *size.x << " = " << *operands[0] << ", "; p << size.x << " = " << operands[0] << ", ";
p << *size.y << " = " << *operands[1] << ", "; p << size.y << " = " << operands[1] << ", ";
p << *size.z << " = " << *operands[2] << ')'; p << size.z << " = " << operands[2] << ')';
} }
void printLaunchOp(OpAsmPrinter &p, LaunchOp op) { void printLaunchOp(OpAsmPrinter &p, LaunchOp op) {
ValueRange operands = op.getOperands(); ValueRange operands = op.getOperands();
// Print the launch configuration. // Print the launch configuration.
p << LaunchOp::getOperationName() << ' ' << op.getBlocksKeyword(); p << LaunchOp::getOperationName() << ' ' << op.getBlocksKeyword();
printSizeAssignment(p, op.getGridSize(), operands.take_front(3), printSizeAssignment(p, op.getGridSize(), operands.take_front(3),
op.getBlockIds()); op.getBlockIds());
p << ' ' << op.getThreadsKeyword(); p << ' ' << op.getThreadsKeyword();
printSizeAssignment(p, op.getBlockSize(), operands.slice(3, 3), printSizeAssignment(p, op.getBlockSize(), operands.slice(3, 3),
op.getThreadIds()); op.getThreadIds());
// From now on, the first kNumConfigOperands operands corresponding to grid // From now on, the first kNumConfigOperands operands corresponding to grid
// and block sizes are irrelevant, so we can drop them. // and block sizes are irrelevant, so we can drop them.
operands = operands.drop_front(LaunchOp::kNumConfigOperands); operands = operands.drop_front(LaunchOp::kNumConfigOperands);
// Print the data argument remapping. // Print the data argument remapping.
if (!op.body().empty() && !operands.empty()) { if (!op.body().empty() && !operands.empty()) {
p << ' ' << op.getArgsKeyword() << '('; p << ' ' << op.getArgsKeyword() << '(';
Block *entryBlock = &op.body().front(); Block *entryBlock = &op.body().front();
interleaveComma(llvm::seq<int>(0, operands.size()), p, [&](int i) { interleaveComma(llvm::seq<int>(0, operands.size()), p, [&](int i) {
p << *entryBlock->getArgument(LaunchOp::kNumConfigRegionAttributes + i) p << entryBlock->getArgument(LaunchOp::kNumConfigRegionAttributes + i)
<< " = " << *operands[i]; << " = " << operands[i];
}); });
p << ") "; p << ") ";
} }
// Print the types of data arguments. // Print the types of data arguments.
if (!operands.empty()) if (!operands.empty())
p << ": " << operands.getTypes(); p << ": " << operands.getTypes();
▲ Show 20 Lines • Show All 133 Lines • ▼ Show 20 Lines PatternMatchResult matchAndRewrite(LaunchOp launchOp,
// the traversal in the inverse order to simplify index arithmetics when // the traversal in the inverse order to simplify index arithmetics when
// dropping arguments. // dropping arguments.
auto operands = launchOp.getKernelOperandValues(); auto operands = launchOp.getKernelOperandValues();
auto kernelArgs = launchOp.getKernelArguments(); auto kernelArgs = launchOp.getKernelArguments();
bool found = false; bool found = false;
for (unsigned i = operands.size(); i > 0; --i) { for (unsigned i = operands.size(); i > 0; --i) {
unsigned index = i - 1; unsigned index = i - 1;
Value operand = operands[index]; Value operand = operands[index];
if (!isa_and_nonnull<ConstantOp>(operand->getDefiningOp())) if (!isa_and_nonnull<ConstantOp>(operand.getDefiningOp()))
continue; continue;
found = true; found = true;
Value internalConstant = Value internalConstant =
rewriter.clone(*operand->getDefiningOp())->getResult(0); rewriter.clone(*operand.getDefiningOp())->getResult(0);
Value kernelArg = *std::next(kernelArgs.begin(), index); Value kernelArg = *std::next(kernelArgs.begin(), index);
kernelArg->replaceAllUsesWith(internalConstant); kernelArg.replaceAllUsesWith(internalConstant);
launchOp.eraseKernelArgument(index); launchOp.eraseKernelArgument(index);
} }
if (!found) { if (!found) {
rewriter.cancelRootUpdate(launchOp); rewriter.cancelRootUpdate(launchOp);
return matchFailure(); return matchFailure();
} }
▲ Show 20 Lines • Show All 230 Lines • ▼ Show 20 Lines
static void printAttributions(OpAsmPrinter &p, StringRef keyword, static void printAttributions(OpAsmPrinter &p, StringRef keyword,
ArrayRef<BlockArgument> values) { ArrayRef<BlockArgument> values) {
if (values.empty()) if (values.empty())
return; return;
p << ' ' << keyword << '('; p << ' ' << keyword << '(';
interleaveComma(values, p, interleaveComma(values, p,
[&p](BlockArgument v) { p << *v << " : " << v->getType(); }); [&p](BlockArgument v) { p << v << " : " << v.getType(); });
p << ')'; p << ')';
} }
/// Prints a GPU Func op. /// Prints a GPU Func op.
void printGPUFuncOp(OpAsmPrinter &p, GPUFuncOp op) { void printGPUFuncOp(OpAsmPrinter &p, GPUFuncOp op) {
p << GPUFuncOp::getOperationName() << ' '; p << GPUFuncOp::getOperationName() << ' ';
p.printSymbolName(op.getName()); p.printSymbolName(op.getName());
Show All 33 Lines return emitOpError("requires '" + getTypeAttrName() +
"' attribute of function type"); "' attribute of function type");
return success(); return success();
} }
static LogicalResult verifyAttributions(Operation *op, static LogicalResult verifyAttributions(Operation *op,
ArrayRef<BlockArgument> attributions, ArrayRef<BlockArgument> attributions,
unsigned memorySpace) { unsigned memorySpace) {
for (Value v : attributions) { for (Value v : attributions) {
auto type = v->getType().dyn_cast<MemRefType>(); auto type = v.getType().dyn_cast<MemRefType>();
if (!type) if (!type)
return op->emitOpError() << "expected memref type in attribution"; return op->emitOpError() << "expected memref type in attribution";
if (type.getMemorySpace() != memorySpace) { if (type.getMemorySpace() != memorySpace) {
return op->emitOpError() return op->emitOpError()
<< "expected memory space " << memorySpace << " in attribution"; << "expected memory space " << memorySpace << " in attribution";
} }
} }
return success(); return success();
} }
/// Verifies the body of the function. /// Verifies the body of the function.
LogicalResult GPUFuncOp::verifyBody() { LogicalResult GPUFuncOp::verifyBody() {
unsigned numFuncArguments = getNumArguments(); unsigned numFuncArguments = getNumArguments();
unsigned numWorkgroupAttributions = getNumWorkgroupAttributions(); unsigned numWorkgroupAttributions = getNumWorkgroupAttributions();
unsigned numBlockArguments = front().getNumArguments(); unsigned numBlockArguments = front().getNumArguments();
if (numBlockArguments < numFuncArguments + numWorkgroupAttributions) if (numBlockArguments < numFuncArguments + numWorkgroupAttributions)
return emitOpError() << "expected at least " return emitOpError() << "expected at least "
<< numFuncArguments + numWorkgroupAttributions << numFuncArguments + numWorkgroupAttributions
<< " arguments to body region"; << " arguments to body region";
ArrayRef<Type> funcArgTypes = getType().getInputs(); ArrayRef<Type> funcArgTypes = getType().getInputs();
for (unsigned i = 0; i < numFuncArguments; ++i) { for (unsigned i = 0; i < numFuncArguments; ++i) {
Type blockArgType = front().getArgument(i)->getType(); Type blockArgType = front().getArgument(i).getType();
if (funcArgTypes[i] != blockArgType) if (funcArgTypes[i] != blockArgType)
return emitOpError() << "expected body region argument #" << i return emitOpError() << "expected body region argument #" << i
<< " to be of type " << funcArgTypes[i] << ", got " << " to be of type " << funcArgTypes[i] << ", got "
<< blockArgType; << blockArgType;
} }
if (failed(verifyAttributions(getOperation(), getWorkgroupAttributions(), if (failed(verifyAttributions(getOperation(), getWorkgroupAttributions(),
GPUDialect::getWorkgroupAddressSpace())) || GPUDialect::getWorkgroupAddressSpace())) ||
Show All 14 Lines

mlir/lib/Dialect/GPU/Transforms/KernelOutlining.cpp

Show All 39 Linesstatic void injectGpuIndexOperations(Location loc, Region &body) {
SmallVector<Value, 12> indexOps; SmallVector<Value, 12> indexOps;
createForAllDimensions<gpu::BlockIdOp>(builder, loc, indexOps); createForAllDimensions<gpu::BlockIdOp>(builder, loc, indexOps);
createForAllDimensions<gpu::ThreadIdOp>(builder, loc, indexOps); createForAllDimensions<gpu::ThreadIdOp>(builder, loc, indexOps);
createForAllDimensions<gpu::GridDimOp>(builder, loc, indexOps); createForAllDimensions<gpu::GridDimOp>(builder, loc, indexOps);
createForAllDimensions<gpu::BlockDimOp>(builder, loc, indexOps); createForAllDimensions<gpu::BlockDimOp>(builder, loc, indexOps);
// Replace the leading 12 function args with the respective thread/block index // Replace the leading 12 function args with the respective thread/block index
// operations. Iterate backwards since args are erased and indices change. // operations. Iterate backwards since args are erased and indices change.
for (int i = 11; i >= 0; --i) { for (int i = 11; i >= 0; --i) {
firstBlock.getArgument(i)->replaceAllUsesWith(indexOps[i]); firstBlock.getArgument(i).replaceAllUsesWith(indexOps[i]);
firstBlock.eraseArgument(i); firstBlock.eraseArgument(i);
} }
} }
static bool isInliningBeneficiary(Operation *op) { static bool isInliningBeneficiary(Operation *op) {
return isa<ConstantOp>(op) || isa<DimOp>(op); return isa<ConstantOp>(op) || isa<DimOp>(op);
} }
// Move arguments of the given kernel function into the function if this reduces // Move arguments of the given kernel function into the function if this reduces
// the number of kernel arguments. // the number of kernel arguments.
static gpu::LaunchFuncOp inlineBeneficiaryOps(gpu::GPUFuncOp kernelFunc, static gpu::LaunchFuncOp inlineBeneficiaryOps(gpu::GPUFuncOp kernelFunc,
gpu::LaunchFuncOp launch) { gpu::LaunchFuncOp launch) {
OpBuilder kernelBuilder(kernelFunc.getBody()); OpBuilder kernelBuilder(kernelFunc.getBody());
auto &firstBlock = kernelFunc.getBody().front(); auto &firstBlock = kernelFunc.getBody().front();
SmallVector<Value, 8> newLaunchArgs; SmallVector<Value, 8> newLaunchArgs;
BlockAndValueMapping map; BlockAndValueMapping map;
for (int i = 0, e = launch.getNumKernelOperands(); i < e; ++i) { for (int i = 0, e = launch.getNumKernelOperands(); i < e; ++i) {
map.map(launch.getKernelOperand(i), kernelFunc.getArgument(i)); map.map(launch.getKernelOperand(i), kernelFunc.getArgument(i));
} }
for (int i = launch.getNumKernelOperands() - 1; i >= 0; --i) { for (int i = launch.getNumKernelOperands() - 1; i >= 0; --i) {
auto operandOp = launch.getKernelOperand(i)->getDefiningOp(); auto operandOp = launch.getKernelOperand(i).getDefiningOp();
if (!operandOp || !isInliningBeneficiary(operandOp)) { if (!operandOp || !isInliningBeneficiary(operandOp)) {
newLaunchArgs.push_back(launch.getKernelOperand(i)); newLaunchArgs.push_back(launch.getKernelOperand(i));
continue; continue;
} }
// Only inline operations that do not create new arguments. // Only inline operations that do not create new arguments.
if (!llvm::all_of(operandOp->getOperands(), if (!llvm::all_of(operandOp->getOperands(),
[map](Value value) { return map.contains(value); })) { [map](Value value) { return map.contains(value); })) {
continue; continue;
} }
auto clone = kernelBuilder.clone(*operandOp, map); auto clone = kernelBuilder.clone(*operandOp, map);
firstBlock.getArgument(i)->replaceAllUsesWith(clone->getResult(0)); firstBlock.getArgument(i).replaceAllUsesWith(clone->getResult(0));
firstBlock.eraseArgument(i); firstBlock.eraseArgument(i);
} }
if (newLaunchArgs.size() == launch.getNumKernelOperands()) if (newLaunchArgs.size() == launch.getNumKernelOperands())
return launch; return launch;
std::reverse(newLaunchArgs.begin(), newLaunchArgs.end()); std::reverse(newLaunchArgs.begin(), newLaunchArgs.end());
OpBuilder LaunchBuilder(launch); OpBuilder LaunchBuilder(launch);
SmallVector<Type, 8> newArgumentTypes; SmallVector<Type, 8> newArgumentTypes;
newArgumentTypes.reserve(firstBlock.getNumArguments()); newArgumentTypes.reserve(firstBlock.getNumArguments());
for (auto value : firstBlock.getArguments()) { for (auto value : firstBlock.getArguments()) {
newArgumentTypes.push_back(value->getType()); newArgumentTypes.push_back(value.getType());
} }
kernelFunc.setType(LaunchBuilder.getFunctionType(newArgumentTypes, {})); kernelFunc.setType(LaunchBuilder.getFunctionType(newArgumentTypes, {}));
auto newLaunch = LaunchBuilder.create<gpu::LaunchFuncOp>( auto newLaunch = LaunchBuilder.create<gpu::LaunchFuncOp>(
launch.getLoc(), kernelFunc, launch.getGridSizeOperandValues(), launch.getLoc(), kernelFunc, launch.getGridSizeOperandValues(),
launch.getBlockSizeOperandValues(), newLaunchArgs); launch.getBlockSizeOperandValues(), newLaunchArgs);
launch.erase(); launch.erase();
return newLaunch; return newLaunch;
} }
▲ Show 20 Lines • Show All 120 Lines • Show Last 20 Lines

mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp

Show All 29 Lines
#include "mlir/Dialect/LLVMIR/LLVMOpsEnums.cpp.inc" #include "mlir/Dialect/LLVMIR/LLVMOpsEnums.cpp.inc"
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::CmpOp. // Printing/parsing for LLVM::CmpOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printICmpOp(OpAsmPrinter &p, ICmpOp &op) { static void printICmpOp(OpAsmPrinter &p, ICmpOp &op) {
p << op.getOperationName() << " \"" << stringifyICmpPredicate(op.predicate()) p << op.getOperationName() << " \"" << stringifyICmpPredicate(op.predicate())
<< "\" " << *op.getOperand(0) << ", " << *op.getOperand(1); << "\" " << op.getOperand(0) << ", " << op.getOperand(1);
p.printOptionalAttrDict(op.getAttrs(), {"predicate"}); p.printOptionalAttrDict(op.getAttrs(), {"predicate"});
p << " : " << op.lhs()->getType(); p << " : " << op.lhs().getType();
} }
static void printFCmpOp(OpAsmPrinter &p, FCmpOp &op) { static void printFCmpOp(OpAsmPrinter &p, FCmpOp &op) {
p << op.getOperationName() << " \"" << stringifyFCmpPredicate(op.predicate()) p << op.getOperationName() << " \"" << stringifyFCmpPredicate(op.predicate())
<< "\" " << *op.getOperand(0) << ", " << *op.getOperand(1); << "\" " << op.getOperand(0) << ", " << op.getOperand(1);
p.printOptionalAttrDict(op.getAttrs(), {"predicate"}); p.printOptionalAttrDict(op.getAttrs(), {"predicate"});
p << " : " << op.lhs()->getType(); p << " : " << op.lhs().getType();
} }
// <operation> ::= `llvm.icmp` string-literal ssa-use `,` ssa-use // <operation> ::= `llvm.icmp` string-literal ssa-use `,` ssa-use
// attribute-dict? `:` type // attribute-dict? `:` type
// <operation> ::= `llvm.fcmp` string-literal ssa-use `,` ssa-use // <operation> ::= `llvm.fcmp` string-literal ssa-use `,` ssa-use
// attribute-dict? `:` type // attribute-dict? `:` type
template <typename CmpPredicateType> template <typename CmpPredicateType>
static ParseResult parseCmpOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseCmpOp(OpAsmParser &parser, OperationState &result) {
▲ Show 20 Lines • Show All 59 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::AllocaOp. // Printing/parsing for LLVM::AllocaOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printAllocaOp(OpAsmPrinter &p, AllocaOp &op) { static void printAllocaOp(OpAsmPrinter &p, AllocaOp &op) {
auto elemTy = op.getType().cast<LLVM::LLVMType>().getPointerElementTy(); auto elemTy = op.getType().cast<LLVM::LLVMType>().getPointerElementTy();
auto funcTy = FunctionType::get({op.arraySize()->getType()}, {op.getType()}, auto funcTy = FunctionType::get({op.arraySize().getType()}, {op.getType()},
op.getContext()); op.getContext());
p << op.getOperationName() << ' ' << *op.arraySize() << " x " << elemTy; p << op.getOperationName() << ' ' << op.arraySize() << " x " << elemTy;
if (op.alignment().hasValue() && op.alignment()->getSExtValue() != 0) if (op.alignment().hasValue() && op.alignment()->getSExtValue() != 0)
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
else else
p.printOptionalAttrDict(op.getAttrs(), {"alignment"}); p.printOptionalAttrDict(op.getAttrs(), {"alignment"});
p << " : " << funcTy; p << " : " << funcTy;
} }
// <operation> ::= `llvm.alloca` ssa-use `x` type attribute-dict? // <operation> ::= `llvm.alloca` ssa-use `x` type attribute-dict?
Show All 28 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::GEPOp. // Printing/parsing for LLVM::GEPOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printGEPOp(OpAsmPrinter &p, GEPOp &op) { static void printGEPOp(OpAsmPrinter &p, GEPOp &op) {
SmallVector<Type, 8> types(op.getOperandTypes()); SmallVector<Type, 8> types(op.getOperandTypes());
auto funcTy = FunctionType::get(types, op.getType(), op.getContext()); auto funcTy = FunctionType::get(types, op.getType(), op.getContext());
p << op.getOperationName() << ' ' << *op.base() << '[' p << op.getOperationName() << ' ' << op.base() << '['
<< op.getOperands().drop_front() << ']'; << op.getOperands().drop_front() << ']';
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << funcTy; p << " : " << funcTy;
} }
// <operation> ::= `llvm.getelementptr` ssa-use `[` ssa-use-list `]` // <operation> ::= `llvm.getelementptr` ssa-use `[` ssa-use-list `]`
// attribute-dict? `:` type // attribute-dict? `:` type
static ParseResult parseGEPOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseGEPOp(OpAsmParser &parser, OperationState &result) {
Show All 27 Linesstatic ParseResult parseGEPOp(OpAsmParser &parser, OperationState &result) {
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::LoadOp. // Printing/parsing for LLVM::LoadOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printLoadOp(OpAsmPrinter &p, LoadOp &op) { static void printLoadOp(OpAsmPrinter &p, LoadOp &op) {
p << op.getOperationName() << ' ' << *op.addr(); p << op.getOperationName() << ' ' << op.addr();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.addr()->getType(); p << " : " << op.addr().getType();
} }
// Extract the pointee type from the LLVM pointer type wrapped in MLIR. Return // Extract the pointee type from the LLVM pointer type wrapped in MLIR. Return
// the resulting type wrapped in MLIR, or nullptr on error. // the resulting type wrapped in MLIR, or nullptr on error.
static Type getLoadStoreElementType(OpAsmParser &parser, Type type, static Type getLoadStoreElementType(OpAsmParser &parser, Type type,
llvm::SMLoc trailingTypeLoc) { llvm::SMLoc trailingTypeLoc) {
auto llvmTy = type.dyn_cast<LLVM::LLVMType>(); auto llvmTy = type.dyn_cast<LLVM::LLVMType>();
if (!llvmTy) if (!llvmTy)
Show All 25 Linesstatic ParseResult parseLoadOp(OpAsmParser &parser, OperationState &result) {
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::StoreOp. // Printing/parsing for LLVM::StoreOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printStoreOp(OpAsmPrinter &p, StoreOp &op) { static void printStoreOp(OpAsmPrinter &p, StoreOp &op) {
p << op.getOperationName() << ' ' << *op.value() << ", " << *op.addr(); p << op.getOperationName() << ' ' << op.value() << ", " << op.addr();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.addr()->getType(); p << " : " << op.addr().getType();
} }
// <operation> ::= `llvm.store` ssa-use `,` ssa-use attribute-dict? `:` type // <operation> ::= `llvm.store` ssa-use `,` ssa-use attribute-dict? `:` type
static ParseResult parseStoreOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseStoreOp(OpAsmParser &parser, OperationState &result) {
SmallVector<NamedAttribute, 4> attrs; SmallVector<NamedAttribute, 4> attrs;
OpAsmParser::OperandType addr, value; OpAsmParser::OperandType addr, value;
Type type; Type type;
llvm::SMLoc trailingTypeLoc; llvm::SMLoc trailingTypeLoc;
Show All 25 Linesstatic void printCallOp(OpAsmPrinter &p, CallOp &op) {
bool isDirect = callee.hasValue(); bool isDirect = callee.hasValue();
// Print the direct callee if present as a function attribute, or an indirect // Print the direct callee if present as a function attribute, or an indirect
// callee (first operand) otherwise. // callee (first operand) otherwise.
p << op.getOperationName() << ' '; p << op.getOperationName() << ' ';
if (isDirect) if (isDirect)
p.printSymbolName(callee.getValue()); p.printSymbolName(callee.getValue());
else else
p << *op.getOperand(0); p << op.getOperand(0);
p << '(' << op.getOperands().drop_front(isDirect ? 0 : 1) << ')'; p << '(' << op.getOperands().drop_front(isDirect ? 0 : 1) << ')';
p.printOptionalAttrDict(op.getAttrs(), {"callee"}); p.printOptionalAttrDict(op.getAttrs(), {"callee"});
// Reconstruct the function MLIR function type from operand and result types. // Reconstruct the function MLIR function type from operand and result types.
SmallVector<Type, 1> resultTypes(op.getResultTypes()); SmallVector<Type, 1> resultTypes(op.getResultTypes());
SmallVector<Type, 8> argTypes( SmallVector<Type, 8> argTypes(
llvm::drop_begin(op.getOperandTypes(), isDirect ? 0 : 1)); llvm::drop_begin(op.getOperandTypes(), isDirect ? 0 : 1));
▲ Show 20 Lines • Show All 91 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::ExtractElementOp. // Printing/parsing for LLVM::ExtractElementOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Expects vector to be of wrapped LLVM vector type and position to be of // Expects vector to be of wrapped LLVM vector type and position to be of
// wrapped LLVM i32 type. // wrapped LLVM i32 type.
void LLVM::ExtractElementOp::build(Builder *b, OperationState &result, void LLVM::ExtractElementOp::build(Builder *b, OperationState &result,
Value vector, Value position, Value vector, Value position,
ArrayRef<NamedAttribute> attrs) { ArrayRef<NamedAttribute> attrs) {
auto wrappedVectorType = vector->getType().cast<LLVM::LLVMType>(); auto wrappedVectorType = vector.getType().cast<LLVM::LLVMType>();
auto llvmType = wrappedVectorType.getVectorElementType(); auto llvmType = wrappedVectorType.getVectorElementType();
build(b, result, llvmType, vector, position); build(b, result, llvmType, vector, position);
result.addAttributes(attrs); result.addAttributes(attrs);
} }
static void printExtractElementOp(OpAsmPrinter &p, ExtractElementOp &op) { static void printExtractElementOp(OpAsmPrinter &p, ExtractElementOp &op) {
p << op.getOperationName() << ' ' << *op.vector() << "[" << *op.position() p << op.getOperationName() << ' ' << op.vector() << "[" << op.position()
<< " : " << op.position()->getType() << "]"; << " : " << op.position().getType() << "]";
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.vector()->getType(); p << " : " << op.vector().getType();
} }
// <operation> ::= `llvm.extractelement` ssa-use `, ` ssa-use // <operation> ::= `llvm.extractelement` ssa-use `, ` ssa-use
// attribute-dict? `:` type // attribute-dict? `:` type
static ParseResult parseExtractElementOp(OpAsmParser &parser, static ParseResult parseExtractElementOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
llvm::SMLoc loc; llvm::SMLoc loc;
OpAsmParser::OperandType vector, position; OpAsmParser::OperandType vector, position;
Show All 15 Linesstatic ParseResult parseExtractElementOp(OpAsmParser &parser,
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::ExtractValueOp. // Printing/parsing for LLVM::ExtractValueOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printExtractValueOp(OpAsmPrinter &p, ExtractValueOp &op) { static void printExtractValueOp(OpAsmPrinter &p, ExtractValueOp &op) {
p << op.getOperationName() << ' ' << *op.container() << op.position(); p << op.getOperationName() << ' ' << op.container() << op.position();
p.printOptionalAttrDict(op.getAttrs(), {"position"}); p.printOptionalAttrDict(op.getAttrs(), {"position"});
p << " : " << op.container()->getType(); p << " : " << op.container().getType();
} }
// Extract the type at `position` in the wrapped LLVM IR aggregate type // Extract the type at `position` in the wrapped LLVM IR aggregate type
// `containerType`. Position is an integer array attribute where each value // `containerType`. Position is an integer array attribute where each value
// is a zero-based position of the element in the aggregate type. Return the // is a zero-based position of the element in the aggregate type. Return the
// resulting type wrapped in MLIR, or nullptr on error. // resulting type wrapped in MLIR, or nullptr on error.
static LLVM::LLVMType getInsertExtractValueElementType(OpAsmParser &parser, static LLVM::LLVMType getInsertExtractValueElementType(OpAsmParser &parser,
Type containerType, Type containerType,
▲ Show 20 Lines • Show All 73 Lines • ▼ Show 20 Linesstatic ParseResult parseExtractValueOp(OpAsmParser &parser,
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::InsertElementOp. // Printing/parsing for LLVM::InsertElementOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printInsertElementOp(OpAsmPrinter &p, InsertElementOp &op) { static void printInsertElementOp(OpAsmPrinter &p, InsertElementOp &op) {
p << op.getOperationName() << ' ' << *op.value() << ", " << *op.vector() p << op.getOperationName() << ' ' << op.value() << ", " << op.vector() << "["
<< "[" << *op.position() << " : " << op.position()->getType() << "]"; << op.position() << " : " << op.position().getType() << "]";
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.vector()->getType(); p << " : " << op.vector().getType();
} }
// <operation> ::= `llvm.insertelement` ssa-use `,` ssa-use `,` ssa-use // <operation> ::= `llvm.insertelement` ssa-use `,` ssa-use `,` ssa-use
// attribute-dict? `:` type // attribute-dict? `:` type
static ParseResult parseInsertElementOp(OpAsmParser &parser, static ParseResult parseInsertElementOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
llvm::SMLoc loc; llvm::SMLoc loc;
OpAsmParser::OperandType vector, value, position; OpAsmParser::OperandType vector, value, position;
Show All 24 Linesstatic ParseResult parseInsertElementOp(OpAsmParser &parser,
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::InsertValueOp. // Printing/parsing for LLVM::InsertValueOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printInsertValueOp(OpAsmPrinter &p, InsertValueOp &op) { static void printInsertValueOp(OpAsmPrinter &p, InsertValueOp &op) {
p << op.getOperationName() << ' ' << *op.value() << ", " << *op.container() p << op.getOperationName() << ' ' << op.value() << ", " << op.container()
<< op.position(); << op.position();
p.printOptionalAttrDict(op.getAttrs(), {"position"}); p.printOptionalAttrDict(op.getAttrs(), {"position"});
p << " : " << op.container()->getType(); p << " : " << op.container().getType();
} }
// <operation> ::= `llvm.insertvaluevalue` ssa-use `,` ssa-use // <operation> ::= `llvm.insertvaluevalue` ssa-use `,` ssa-use
// `[` integer-literal (`,` integer-literal)* `]` // `[` integer-literal (`,` integer-literal)* `]`
// attribute-dict? `:` type // attribute-dict? `:` type
static ParseResult parseInsertValueOp(OpAsmParser &parser, static ParseResult parseInsertValueOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
OpAsmParser::OperandType container, value; OpAsmParser::OperandType container, value;
Show All 23 Linesstatic ParseResult parseInsertValueOp(OpAsmParser &parser,
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::SelectOp. // Printing/parsing for LLVM::SelectOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printSelectOp(OpAsmPrinter &p, SelectOp &op) { static void printSelectOp(OpAsmPrinter &p, SelectOp &op) {
p << op.getOperationName() << ' ' << *op.condition() << ", " p << op.getOperationName() << ' ' << op.condition() << ", " << op.trueValue()
<< *op.trueValue() << ", " << *op.falseValue(); << ", " << op.falseValue();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.condition()->getType() << ", " << op.trueValue()->getType(); p << " : " << op.condition().getType() << ", " << op.trueValue().getType();
} }
// <operation> ::= `llvm.select` ssa-use `,` ssa-use `,` ssa-use // <operation> ::= `llvm.select` ssa-use `,` ssa-use `,` ssa-use
// attribute-dict? `:` type, type // attribute-dict? `:` type, type
static ParseResult parseSelectOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseSelectOp(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType condition, trueValue, falseValue; OpAsmParser::OperandType condition, trueValue, falseValue;
Type conditionType, argType; Type conditionType, argType;
Show All 37 Linesstatic ParseResult parseBrOp(OpAsmParser &parser, OperationState &result) {
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::CondBrOp. // Printing/parsing for LLVM::CondBrOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printCondBrOp(OpAsmPrinter &p, CondBrOp &op) { static void printCondBrOp(OpAsmPrinter &p, CondBrOp &op) {
p << op.getOperationName() << ' ' << *op.getOperand(0) << ", "; p << op.getOperationName() << ' ' << op.getOperand(0) << ", ";
p.printSuccessorAndUseList(op.getOperation(), 0); p.printSuccessorAndUseList(op.getOperation(), 0);
p << ", "; p << ", ";
p.printSuccessorAndUseList(op.getOperation(), 1); p.printSuccessorAndUseList(op.getOperation(), 1);
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
} }
// <operation> ::= `llvm.cond_br` ssa-use `,` // <operation> ::= `llvm.cond_br` ssa-use `,`
// bb-id (`[` ssa-use-and-type-list `]`)? `,` // bb-id (`[` ssa-use-and-type-list `]`)? `,`
Show All 30 Lines
static void printReturnOp(OpAsmPrinter &p, ReturnOp &op) { static void printReturnOp(OpAsmPrinter &p, ReturnOp &op) {
p << op.getOperationName(); p << op.getOperationName();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
assert(op.getNumOperands() <= 1); assert(op.getNumOperands() <= 1);
if (op.getNumOperands() == 0) if (op.getNumOperands() == 0)
return; return;
p << ' ' << *op.getOperand(0) << " : " << op.getOperand(0)->getType(); p << ' ' << op.getOperand(0) << " : " << op.getOperand(0).getType();
} }
// <operation> ::= `llvm.return` ssa-use-list attribute-dict? `:` // <operation> ::= `llvm.return` ssa-use-list attribute-dict? `:`
// type-list-no-parens // type-list-no-parens
static ParseResult parseReturnOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseReturnOp(OpAsmParser &parser, OperationState &result) {
SmallVector<OpAsmParser::OperandType, 1> operands; SmallVector<OpAsmParser::OperandType, 1> operands;
Type type; Type type;
Show All 11 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::UndefOp. // Printing/parsing for LLVM::UndefOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printUndefOp(OpAsmPrinter &p, UndefOp &op) { static void printUndefOp(OpAsmPrinter &p, UndefOp &op) {
p << op.getOperationName(); p << op.getOperationName();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.res()->getType(); p << " : " << op.res().getType();
} }
// <operation> ::= `llvm.mlir.undef` attribute-dict? : type // <operation> ::= `llvm.mlir.undef` attribute-dict? : type
static ParseResult parseUndefOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseUndefOp(OpAsmParser &parser, OperationState &result) {
Type type; Type type;
if (parser.parseOptionalAttrDict(result.attributes) || if (parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(type)) parser.parseColonType(type))
Show All 14 LinesGlobalOp AddressOfOp::getGlobal() {
assert(module && "unexpected operation outside of a module"); assert(module && "unexpected operation outside of a module");
return dyn_cast_or_null<LLVM::GlobalOp>( return dyn_cast_or_null<LLVM::GlobalOp>(
mlir::SymbolTable::lookupSymbolIn(module, global_name())); mlir::SymbolTable::lookupSymbolIn(module, global_name()));
} }
static void printAddressOfOp(OpAsmPrinter &p, AddressOfOp op) { static void printAddressOfOp(OpAsmPrinter &p, AddressOfOp op) {
p << op.getOperationName() << " @" << op.global_name(); p << op.getOperationName() << " @" << op.global_name();
p.printOptionalAttrDict(op.getAttrs(), {"global_name"}); p.printOptionalAttrDict(op.getAttrs(), {"global_name"});
p << " : " << op.getResult()->getType(); p << " : " << op.getResult().getType();
} }
static ParseResult parseAddressOfOp(OpAsmParser &parser, static ParseResult parseAddressOfOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
Attribute symRef; Attribute symRef;
Type type; Type type;
if (parser.parseAttribute(symRef, "global_name", result.attributes) || if (parser.parseAttribute(symRef, "global_name", result.attributes) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(type) || parser.addTypeToList(type, result.types)) parser.parseColonType(type) || parser.addTypeToList(type, result.types))
return failure(); return failure();
if (!symRef.isa<SymbolRefAttr>()) if (!symRef.isa<SymbolRefAttr>())
return parser.emitError(parser.getNameLoc(), "expected symbol reference"); return parser.emitError(parser.getNameLoc(), "expected symbol reference");
return success(); return success();
} }
static LogicalResult verify(AddressOfOp op) { static LogicalResult verify(AddressOfOp op) {
auto global = op.getGlobal(); auto global = op.getGlobal();
if (!global) if (!global)
return op.emitOpError( return op.emitOpError(
"must reference a global defined by 'llvm.mlir.global'"); "must reference a global defined by 'llvm.mlir.global'");
if (global.getType().getPointerTo(global.addr_space().getZExtValue()) != if (global.getType().getPointerTo(global.addr_space().getZExtValue()) !=
op.getResult()->getType()) op.getResult().getType())
return op.emitOpError( return op.emitOpError(
"the type must be a pointer to the type of the referred global"); "the type must be a pointer to the type of the referred global");
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::ConstantOp. // Printing/parsing for LLVM::ConstantOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void printConstantOp(OpAsmPrinter &p, ConstantOp &op) { static void printConstantOp(OpAsmPrinter &p, ConstantOp &op) {
p << op.getOperationName() << '(' << op.value() << ')'; p << op.getOperationName() << '(' << op.value() << ')';
p.printOptionalAttrDict(op.getAttrs(), {"value"}); p.printOptionalAttrDict(op.getAttrs(), {"value"});
p << " : " << op.res()->getType(); p << " : " << op.res().getType();
} }
// <operation> ::= `llvm.mlir.constant` `(` attribute `)` attribute-list? : type // <operation> ::= `llvm.mlir.constant` `(` attribute `)` attribute-list? : type
static ParseResult parseConstantOp(OpAsmParser &parser, static ParseResult parseConstantOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
Attribute valueAttr; Attribute valueAttr;
Type type; Type type;
▲ Show 20 Lines • Show All 213 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Printing/parsing for LLVM::ShuffleVectorOp. // Printing/parsing for LLVM::ShuffleVectorOp.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Expects vector to be of wrapped LLVM vector type and position to be of // Expects vector to be of wrapped LLVM vector type and position to be of
// wrapped LLVM i32 type. // wrapped LLVM i32 type.
void LLVM::ShuffleVectorOp::build(Builder *b, OperationState &result, Value v1, void LLVM::ShuffleVectorOp::build(Builder *b, OperationState &result, Value v1,
Value v2, ArrayAttr mask, Value v2, ArrayAttr mask,
ArrayRef<NamedAttribute> attrs) { ArrayRef<NamedAttribute> attrs) {
auto wrappedContainerType1 = v1->getType().cast<LLVM::LLVMType>(); auto wrappedContainerType1 = v1.getType().cast<LLVM::LLVMType>();
auto vType = LLVMType::getVectorTy( auto vType = LLVMType::getVectorTy(
wrappedContainerType1.getVectorElementType(), mask.size()); wrappedContainerType1.getVectorElementType(), mask.size());
build(b, result, vType, v1, v2, mask); build(b, result, vType, v1, v2, mask);
result.addAttributes(attrs); result.addAttributes(attrs);
} }
static void printShuffleVectorOp(OpAsmPrinter &p, ShuffleVectorOp &op) { static void printShuffleVectorOp(OpAsmPrinter &p, ShuffleVectorOp &op) {
p << op.getOperationName() << ' ' << *op.v1() << ", " << *op.v2() << " " p << op.getOperationName() << ' ' << op.v1() << ", " << op.v2() << " "
<< op.mask(); << op.mask();
p.printOptionalAttrDict(op.getAttrs(), {"mask"}); p.printOptionalAttrDict(op.getAttrs(), {"mask"});
p << " : " << op.v1()->getType() << ", " << op.v2()->getType(); p << " : " << op.v1().getType() << ", " << op.v2().getType();
} }
// <operation> ::= `llvm.shufflevector` ssa-use `, ` ssa-use // <operation> ::= `llvm.shufflevector` ssa-use `, ` ssa-use
// `[` integer-literal (`,` integer-literal)* `]` // `[` integer-literal (`,` integer-literal)* `]`
// attribute-dict? `:` type // attribute-dict? `:` type
static ParseResult parseShuffleVectorOp(OpAsmParser &parser, static ParseResult parseShuffleVectorOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
llvm::SMLoc loc; llvm::SMLoc loc;
▲ Show 20 Lines • Show All 241 Lines • ▼ Show 20 Linesstatic LogicalResult verify(LLVMFuncOp op) {
if (op.isVarArg()) if (op.isVarArg())
return op.emitOpError("only external functions can be variadic"); return op.emitOpError("only external functions can be variadic");
auto *funcType = cast<llvm::FunctionType>(op.getType().getUnderlyingType()); auto *funcType = cast<llvm::FunctionType>(op.getType().getUnderlyingType());
unsigned numArguments = funcType->getNumParams(); unsigned numArguments = funcType->getNumParams();
Block &entryBlock = op.front(); Block &entryBlock = op.front();
for (unsigned i = 0; i < numArguments; ++i) { for (unsigned i = 0; i < numArguments; ++i) {
Type argType = entryBlock.getArgument(i)->getType(); Type argType = entryBlock.getArgument(i).getType();
auto argLLVMType = argType.dyn_cast<LLVMType>(); auto argLLVMType = argType.dyn_cast<LLVMType>();
if (!argLLVMType) if (!argLLVMType)
return op.emitOpError("entry block argument #") return op.emitOpError("entry block argument #")
<< i << " is not of LLVM type"; << i << " is not of LLVM type";
if (funcType->getParamType(i) != argLLVMType.getUnderlyingType()) if (funcType->getParamType(i) != argLLVMType.getUnderlyingType())
return op.emitOpError("the type of entry block argument #") return op.emitOpError("the type of entry block argument #")
<< i << " does not match the function signature"; << i << " does not match the function signature";
} }
▲ Show 20 Lines • Show All 350 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Analysis/DependenceAnalysis.cpp

Show First 20 Lines • Show All 42 Lines • ▼ Show 20 Lines
#endif #endif
Value Aliases::find(Value v) { Value Aliases::find(Value v) {
if (v.isa<BlockArgument>()) if (v.isa<BlockArgument>())
return v; return v;
auto it = aliases.find(v); auto it = aliases.find(v);
if (it != aliases.end()) { if (it != aliases.end()) {
assert(it->getSecond()->getType().isa<MemRefType>() && "Memref expected"); assert(it->getSecond().getType().isa<MemRefType>() && "Memref expected");
return it->getSecond(); return it->getSecond();
} }
while (true) { while (true) {
if (v.isa<BlockArgument>()) if (v.isa<BlockArgument>())
return v; return v;
if (auto alloc = dyn_cast_or_null<AllocOp>(v->getDefiningOp())) { if (auto alloc = dyn_cast_or_null<AllocOp>(v.getDefiningOp())) {
if (isStrided(alloc.getType())) if (isStrided(alloc.getType()))
return alloc.getResult(); return alloc.getResult();
} }
if (auto slice = dyn_cast_or_null<SliceOp>(v->getDefiningOp())) { if (auto slice = dyn_cast_or_null<SliceOp>(v.getDefiningOp())) {
auto it = aliases.insert(std::make_pair(v, find(slice.view()))); auto it = aliases.insert(std::make_pair(v, find(slice.view())));
return it.first->second; return it.first->second;
} }
if (auto view = dyn_cast_or_null<ViewOp>(v->getDefiningOp())) { if (auto view = dyn_cast_or_null<ViewOp>(v.getDefiningOp())) {
auto it = aliases.insert(std::make_pair(v, view.source())); auto it = aliases.insert(std::make_pair(v, view.source()));
return it.first->second; return it.first->second;
} }
if (auto view = dyn_cast_or_null<SubViewOp>(v->getDefiningOp())) { if (auto view = dyn_cast_or_null<SubViewOp>(v.getDefiningOp())) {
v = view.source(); v = view.source();
continue; continue;
} }
llvm::errs() << "View alias analysis reduces to: " << *v << "\n"; llvm::errs() << "View alias analysis reduces to: " << v << "\n";
llvm_unreachable("unsupported view alias case"); llvm_unreachable("unsupported view alias case");
} }
} }
LinalgDependenceGraph LinalgDependenceGraph
LinalgDependenceGraph::buildDependenceGraph(Aliases &aliases, FuncOp f) { LinalgDependenceGraph::buildDependenceGraph(Aliases &aliases, FuncOp f) {
SmallVector<Operation *, 8> linalgOps; SmallVector<Operation *, 8> linalgOps;
f.walk([&](LinalgOp op) { linalgOps.push_back(op); }); f.walk([&](LinalgOp op) { linalgOps.push_back(op); });
▲ Show 20 Lines • Show All 136 Lines • ▼ Show 20 Lines for (auto dependence : getDependencesFrom(src, dt)) {
// Skip if not interleaved. // Skip if not interleaved.
if (interimPos >= dstPos || interimPos <= srcPos) if (interimPos >= dstPos || interimPos <= srcPos)
continue; continue;
if (view && !aliases.alias(view, dependence.indexingView)) if (view && !aliases.alias(view, dependence.indexingView))
continue; continue;
auto *op = dependence.dependentOpView.op; auto *op = dependence.dependentOpView.op;
LLVM_DEBUG(dbgs() << "\n***Found covering dependence of type " LLVM_DEBUG(dbgs() << "\n***Found covering dependence of type "
<< toStringRef(dt) << ": " << *src << " -> " << *op << toStringRef(dt) << ": " << *src << " -> " << *op
<< " on " << *dependence.indexingView); << " on " << dependence.indexingView);
res.push_back(op); res.push_back(op);
} }
} }
return res; return res;
} }

mlir/lib/Dialect/Linalg/EDSC/Builders.cpp

Show First 20 Lines • Show All 82 Lines • ▼ Show 20 LinesOperation *mlir::edsc::makeGenericLinalgOp(
// clang-format on // clang-format on
using namespace edsc; using namespace edsc;
SmallVector<Type, 4> blockTypes; SmallVector<Type, 4> blockTypes;
blockTypes.reserve(values.size()); blockTypes.reserve(values.size());
for (auto it : llvm::enumerate(values)) for (auto it : llvm::enumerate(values))
blockTypes.push_back((it.index() < nViews) blockTypes.push_back((it.index() < nViews)
? getElementTypeOrSelf(it.value()) ? getElementTypeOrSelf(it.value())
: it.value()->getType()); : it.value().getType());
assert(op->getRegions().front().empty()); assert(op->getRegions().front().empty());
op->getRegions().front().push_front(new Block); op->getRegions().front().push_front(new Block);
OpBuilder bb(op->getRegions().front()); OpBuilder bb(op->getRegions().front());
ScopedContext scope(bb, op->getLoc()); ScopedContext scope(bb, op->getLoc());
BlockHandle b; BlockHandle b;
auto handles = makeValueHandles(blockTypes); auto handles = makeValueHandles(blockTypes);
BlockBuilder(&b, makeHandlePointers(MutableArrayRef<ValueHandle>(handles)))( BlockBuilder(&b, makeHandlePointers(MutableArrayRef<ValueHandle>(handles)))(
▲ Show 20 Lines • Show All 148 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp

Show First 20 Lines • Show All 114 Lines • ▼ Show 20 Linestemplate <> LogicalResult verifyBlockArgs(GenericOp op, Block &block) {
auto nViews = op.getNumInputsAndOutputs(); auto nViews = op.getNumInputsAndOutputs();
auto nInputViews = op.getNumInputs(); auto nInputViews = op.getNumInputs();
if (block.getNumArguments() != nViews) if (block.getNumArguments() != nViews)
return op.emitOpError( return op.emitOpError(
"expected number of block arguments to match number of views"); "expected number of block arguments to match number of views");
for (unsigned i = 0; i < nViews; ++i) { for (unsigned i = 0; i < nViews; ++i) {
auto viewType = op.getShapedType(i); auto viewType = op.getShapedType(i);
if (viewType.getElementType() != block.getArgument(i)->getType()) if (viewType.getElementType() != block.getArgument(i).getType())
return op.emitOpError("expected block argument ") return op.emitOpError("expected block argument ")
<< i << " of the same type as elemental type of " << i << " of the same type as elemental type of "
<< ((i < nInputViews) ? "input " : "output ") << ((i < nInputViews) ? "input " : "output ")
<< "view: " << viewType; << "view: " << viewType;
} }
return success(); return success();
} }
template <> LogicalResult verifyBlockArgs(IndexedGenericOp op, Block &block) { template <> LogicalResult verifyBlockArgs(IndexedGenericOp op, Block &block) {
auto nInputViews = op.getNumInputs(); auto nInputViews = op.getNumInputs();
auto nLoops = op.getNumLoops(); auto nLoops = op.getNumLoops();
auto nViews = op.getNumInputsAndOutputs(); auto nViews = op.getNumInputsAndOutputs();
if (block.getNumArguments() != nViews + nLoops) if (block.getNumArguments() != nViews + nLoops)
return op.emitOpError( return op.emitOpError(
"expected number of block arguments to match number of views + " "expected number of block arguments to match number of views + "
"number of loops"); "number of loops");
for (unsigned i = 0; i < nLoops; ++i) { for (unsigned i = 0; i < nLoops; ++i) {
if (!block.getArgument(i)->getType().isIndex()) if (!block.getArgument(i).getType().isIndex())
return op.emitOpError("expected block argument ") return op.emitOpError("expected block argument ")
<< i << " to be of IndexType"; << i << " to be of IndexType";
} }
for (unsigned i = 0; i < nViews; ++i) { for (unsigned i = 0; i < nViews; ++i) {
unsigned memrefArgIndex = i + nLoops; unsigned memrefArgIndex = i + nLoops;
auto viewType = op.getShapedType(i); auto viewType = op.getShapedType(i);
if (viewType.getElementType() != if (viewType.getElementType() !=
block.getArgument(memrefArgIndex)->getType()) block.getArgument(memrefArgIndex).getType())
return op.emitOpError("expected block argument ") return op.emitOpError("expected block argument ")
<< memrefArgIndex << " of the same type as elemental type of " << memrefArgIndex << " of the same type as elemental type of "
<< ((i < nInputViews) ? "input " : "output ") << ((i < nInputViews) ? "input " : "output ")
<< "view: " << viewType; << "view: " << viewType;
} }
return success(); return success();
} }
▲ Show 20 Lines • Show All 149 Lines • ▼ Show 20 Lines
static LogicalResult verify(GenericOp op) { return verifyGenericOp(op); } static LogicalResult verify(GenericOp op) { return verifyGenericOp(op); }
static LogicalResult verify(IndexedGenericOp op) { return verifyGenericOp(op); } static LogicalResult verify(IndexedGenericOp op) { return verifyGenericOp(op); }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// RangeOp // RangeOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, RangeOp op) { static void print(OpAsmPrinter &p, RangeOp op) {
p << op.getOperationName() << " " << *op.min() << ":" << *op.max() << ":" p << op.getOperationName() << " " << op.min() << ":" << op.max() << ":"
<< *op.step(); << op.step();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getResult()->getType(); p << " : " << op.getResult().getType();
} }
static ParseResult parseRangeOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseRangeOp(OpAsmParser &parser, OperationState &result) {
SmallVector<OpAsmParser::OperandType, 3> rangeInfo(3); SmallVector<OpAsmParser::OperandType, 3> rangeInfo(3);
RangeType type; RangeType type;
auto indexTy = parser.getBuilder().getIndexType(); auto indexTy = parser.getBuilder().getIndexType();
return failure(parser.parseOperand(rangeInfo[0]) || parser.parseColon() || return failure(parser.parseOperand(rangeInfo[0]) || parser.parseColon() ||
parser.parseOperand(rangeInfo[1]) || parser.parseColon() || parser.parseOperand(rangeInfo[1]) || parser.parseColon() ||
▲ Show 20 Lines • Show All 207 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SliceOp // SliceOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void mlir::linalg::SliceOp::build(Builder *b, OperationState &result, void mlir::linalg::SliceOp::build(Builder *b, OperationState &result,
Value base, ValueRange indexings) { Value base, ValueRange indexings) {
result.addOperands(base); result.addOperands(base);
result.addOperands(indexings); result.addOperands(indexings);
auto memRefType = base->getType().cast<MemRefType>(); auto memRefType = base.getType().cast<MemRefType>();
int64_t offset; int64_t offset;
SmallVector<int64_t, 4> strides; SmallVector<int64_t, 4> strides;
auto res = getStridesAndOffset(memRefType, strides, offset); auto res = getStridesAndOffset(memRefType, strides, offset);
assert(succeeded(res) && strides.size() == indexings.size()); assert(succeeded(res) && strides.size() == indexings.size());
(void)res; (void)res;
unsigned rank = memRefType.getRank(); unsigned rank = memRefType.getRank();
// TODO(ntv): propagate static size and stride information when available. // TODO(ntv): propagate static size and stride information when available.
SmallVector<int64_t, 4> sizes(rank, -1); // -1 encodes dynamic size. SmallVector<int64_t, 4> sizes(rank, -1); // -1 encodes dynamic size.
Type elementType = memRefType.getElementType(); Type elementType = memRefType.getElementType();
result.addTypes({MemRefType::get( result.addTypes({MemRefType::get(
sizes, elementType, sizes, elementType,
{makeStridedLinearLayoutMap(strides, offset, b->getContext())}, {makeStridedLinearLayoutMap(strides, offset, b->getContext())},
memRefType.getMemorySpace())}); memRefType.getMemorySpace())});
} }
static void print(OpAsmPrinter &p, SliceOp op) { static void print(OpAsmPrinter &p, SliceOp op) {
auto indexings = op.indexings(); auto indexings = op.indexings();
p << SliceOp::getOperationName() << " " << *op.view() << "[" << indexings p << SliceOp::getOperationName() << " " << op.view() << "[" << indexings
<< "] "; << "] ";
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getBaseViewType(); p << " : " << op.getBaseViewType();
if (!indexings.empty()) if (!indexings.empty())
p << ", " << op.indexings().getTypes(); p << ", " << op.indexings().getTypes();
p << ", " << op.getType(); p << ", " << op.getType();
} }
Show All 22 Lines
static LogicalResult verify(SliceOp op) { static LogicalResult verify(SliceOp op) {
unsigned rank = op.getBaseViewRank(); unsigned rank = op.getBaseViewRank();
if (rank != llvm::size(op.indexings())) if (rank != llvm::size(op.indexings()))
return op.emitOpError("expected ") return op.emitOpError("expected ")
<< rank << " indexings, got " << llvm::size(op.indexings()); << rank << " indexings, got " << llvm::size(op.indexings());
unsigned index = 0; unsigned index = 0;
for (auto indexing : op.indexings()) { for (auto indexing : op.indexings()) {
if (indexing->getType().isa<IndexType>()) if (indexing.getType().isa<IndexType>())
--rank; --rank;
++index; ++index;
} }
if (op.getRank() != rank) if (op.getRank() != rank)
return op.emitOpError() << "expected rank of the view(" << op.getRank() return op.emitOpError() << "expected rank of the view(" << op.getRank()
<< ") to be the number of ranges(" << rank << ")"; << ") to be the number of ranges(" << rank << ")";
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TransposeOp // TransposeOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void mlir::linalg::TransposeOp::build(Builder *b, OperationState &result, void mlir::linalg::TransposeOp::build(Builder *b, OperationState &result,
Value view, AffineMapAttr permutation, Value view, AffineMapAttr permutation,
ArrayRef<NamedAttribute> attrs) { ArrayRef<NamedAttribute> attrs) {
auto permutationMap = permutation.getValue(); auto permutationMap = permutation.getValue();
assert(permutationMap); assert(permutationMap);
auto memRefType = view->getType().cast<MemRefType>(); auto memRefType = view.getType().cast<MemRefType>();
auto rank = memRefType.getRank(); auto rank = memRefType.getRank();
auto originalSizes = memRefType.getShape(); auto originalSizes = memRefType.getShape();
// Compute permuted sizes. // Compute permuted sizes.
SmallVector<int64_t, 4> sizes(rank, 0); SmallVector<int64_t, 4> sizes(rank, 0);
for (auto en : llvm::enumerate(permutationMap.getResults())) for (auto en : llvm::enumerate(permutationMap.getResults()))
sizes[en.index()] = sizes[en.index()] =
originalSizes[en.value().cast<AffineDimExpr>().getPosition()]; originalSizes[en.value().cast<AffineDimExpr>().getPosition()];
Show All 9 Linesvoid mlir::linalg::TransposeOp::build(Builder *b, OperationState &result,
auto resultType = MemRefType::get(sizes, memRefType.getElementType(), map, auto resultType = MemRefType::get(sizes, memRefType.getElementType(), map,
memRefType.getMemorySpace()); memRefType.getMemorySpace());
build(b, result, resultType, view, attrs); build(b, result, resultType, view, attrs);
result.addAttribute(TransposeOp::getPermutationAttrName(), permutation); result.addAttribute(TransposeOp::getPermutationAttrName(), permutation);
} }
static void print(OpAsmPrinter &p, TransposeOp op) { static void print(OpAsmPrinter &p, TransposeOp op) {
p << op.getOperationName() << " " << *op.view() << " " << op.permutation(); p << op.getOperationName() << " " << op.view() << " " << op.permutation();
p.printOptionalAttrDict(op.getAttrs(), p.printOptionalAttrDict(op.getAttrs(),
{TransposeOp::getPermutationAttrName()}); {TransposeOp::getPermutationAttrName()});
p << " : " << op.view()->getType(); p << " : " << op.view().getType();
} }
static ParseResult parseTransposeOp(OpAsmParser &parser, static ParseResult parseTransposeOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
OpAsmParser::OperandType view; OpAsmParser::OperandType view;
AffineMapAttr permutation; AffineMapAttr permutation;
MemRefType type; MemRefType type;
return failure(parser.parseOperand(view) || return failure(parser.parseOperand(view) ||
Show All 34 LinesLogicalResult verifyYield(YieldOp op, GenericOpType genericOp) {
// The operand number and types must match the view element types. // The operand number and types must match the view element types.
auto nOutputViews = genericOp.getNumOutputs(); auto nOutputViews = genericOp.getNumOutputs();
if (op.getNumOperands() != nOutputViews) if (op.getNumOperands() != nOutputViews)
return op.emitOpError("expected ") return op.emitOpError("expected ")
<< nOutputViews << " operand to match enclosing linalg.generic op"; << nOutputViews << " operand to match enclosing linalg.generic op";
for (unsigned i = 0; i != nOutputViews; ++i) { for (unsigned i = 0; i != nOutputViews; ++i) {
auto elementType = genericOp.getOutputShapedType(i).getElementType(); auto elementType = genericOp.getOutputShapedType(i).getElementType();
if (op.getOperand(i)->getType() != elementType) if (op.getOperand(i).getType() != elementType)
return op.emitOpError("type of return operand ") return op.emitOpError("type of return operand ")
<< i << " (" << op.getOperand(i)->getType() << i << " (" << op.getOperand(i).getType()
<< ") doesn't match view element type (" << elementType << ")"; << ") doesn't match view element type (" << elementType << ")";
} }
return success(); return success();
} }
static LogicalResult verify(YieldOp op) { static LogicalResult verify(YieldOp op) {
auto *parentOp = op.getParentOp(); auto *parentOp = op.getParentOp();
if (parentOp->getNumRegions() != 1 || parentOp->getRegion(0).empty()) if (parentOp->getNumRegions() != 1 || parentOp->getRegion(0).empty())
▲ Show 20 Lines • Show All 48 Lines • ▼ Show 20 Lines return failure(
parser.parseOperandList(ops, OpAsmParser::Delimiter::Paren) || parser.parseOperandList(ops, OpAsmParser::Delimiter::Paren) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonTypeList(types) || parser.parseColonTypeList(types) ||
parser.resolveOperands(ops, types, parser.getNameLoc(), result.operands)); parser.resolveOperands(ops, types, parser.getNameLoc(), result.operands));
} }
static LogicalResult verify(FillOp op) { static LogicalResult verify(FillOp op) {
auto viewType = op.getOutputShapedType(0); auto viewType = op.getOutputShapedType(0);
auto fillType = op.value()->getType(); auto fillType = op.value().getType();
if (viewType.getElementType() != fillType) if (viewType.getElementType() != fillType)
return op.emitOpError("expects fill type to match view elemental type"); return op.emitOpError("expects fill type to match view elemental type");
return success(); return success();
} }
static LogicalResult verify(CopyOp op) { static LogicalResult verify(CopyOp op) {
auto outputViewType = op.getOutputShapedType(0); auto outputViewType = op.getOutputShapedType(0);
auto inputViewType = op.getInputShapedType(0); auto inputViewType = op.getInputShapedType(0);
Show All 34 Lines if (attrs.size() != op.getNumWindowLoops())
return op.emitOpError("expects num ") return op.emitOpError("expects num ")
<< strideOrDilation << strideOrDilation
<< "s equal to number of window dimensions: " << attrs.size() << "s equal to number of window dimensions: " << attrs.size()
<< " vs " << op.getNumWindowLoops(); << " vs " << op.getNumWindowLoops();
return success(); return success();
} }
static LogicalResult verify(ConvOp op) { static LogicalResult verify(ConvOp op) {
auto oType = op.output()->getType().cast<MemRefType>(); auto oType = op.output().getType().cast<MemRefType>();
auto fType = op.filter()->getType().cast<MemRefType>(); auto fType = op.filter().getType().cast<MemRefType>();
auto iType = op.input()->getType().cast<MemRefType>(); auto iType = op.input().getType().cast<MemRefType>();
if (oType.getElementType() != iType.getElementType() || if (oType.getElementType() != iType.getElementType() ||
oType.getElementType() != fType.getElementType()) oType.getElementType() != fType.getElementType())
return op.emitOpError("expects memref elemental types to match"); return op.emitOpError("expects memref elemental types to match");
if (oType.getRank() != iType.getRank() || oType.getRank() != fType.getRank()) if (oType.getRank() != iType.getRank() || oType.getRank() != fType.getRank())
return op.emitOpError("expects memref ranks to match"); return op.emitOpError("expects memref ranks to match");
if (auto strides = op.strides()) { if (auto strides = op.strides()) {
if (failed( if (failed(
verifyStrideOrDilation(op, strides->getValue(), /*isStride=*/true))) verifyStrideOrDilation(op, strides->getValue(), /*isStride=*/true)))
▲ Show 20 Lines • Show All 221 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

Show First 20 Lines • Show All 127 Lines • ▼ Show 20 Lines for (auto en : llvm::enumerate(ios)) {
LLVM_DEBUG(dbgs() << "getViewDefiningLoopRange I/O idx: " << idx << "\n"); LLVM_DEBUG(dbgs() << "getViewDefiningLoopRange I/O idx: " << idx << "\n");
LLVM_DEBUG(dbgs() << "getViewDefiningLoopRange map: " << map << "\n"); LLVM_DEBUG(dbgs() << "getViewDefiningLoopRange map: " << map << "\n");
Value view = en.value(); Value view = en.value();
SmallVector<Value, 8> viewRanges(map.getNumResults(), nullptr); SmallVector<Value, 8> viewRanges(map.getNumResults(), nullptr);
for (auto en2 : llvm::enumerate(map.getResults())) { for (auto en2 : llvm::enumerate(map.getResults())) {
if (loopDepth == en2.value().cast<AffineDimExpr>().getPosition()) { if (loopDepth == en2.value().cast<AffineDimExpr>().getPosition()) {
LLVM_DEBUG(dbgs() << "getViewDefiningLoopRange loopDepth: " << loopDepth LLVM_DEBUG(dbgs() << "getViewDefiningLoopRange loopDepth: " << loopDepth
<< "\n"); << "\n");
LLVM_DEBUG(dbgs() << "getViewDefiningLoopRange view: " << *view LLVM_DEBUG(dbgs() << "getViewDefiningLoopRange view: " << view << "\n");
<< "\n");
return ViewDimension{view, static_cast<unsigned>(en2.index())}; return ViewDimension{view, static_cast<unsigned>(en2.index())};
} }
} }
} }
llvm_unreachable("Expect to be able to extract a view defining loop range"); llvm_unreachable("Expect to be able to extract a view defining loop range");
} }
static LinalgOp fuse(Value producedView, LinalgOp producer, LinalgOp consumer, static LinalgOp fuse(Value producedView, LinalgOp producer, LinalgOp consumer,
unsigned consumerIdx, unsigned producerIdx, unsigned consumerIdx, unsigned producerIdx,
OperationFolder *folder) { OperationFolder *folder) {
auto subView = dyn_cast_or_null<SubViewOp>( auto subView = dyn_cast_or_null<SubViewOp>(
consumer.getInput(consumerIdx)->getDefiningOp()); consumer.getInput(consumerIdx).getDefiningOp());
auto slice = dyn_cast_or_null<SliceOp>( auto slice =
consumer.getInput(consumerIdx)->getDefiningOp()); dyn_cast_or_null<SliceOp>(consumer.getInput(consumerIdx).getDefiningOp());
assert(subView || slice); assert(subView || slice);
(void)subView; (void)subView;
(void)slice; (void)slice;
// loopToOperandRangesMaps are permutations-only by construction: // loopToOperandRangesMaps are permutations-only by construction:
// we can always identify a data dimension with a (at least one) loop // we can always identify a data dimension with a (at least one) loop
// dimension. // dimension.
AffineMap producerMap = AffineMap producerMap =
▲ Show 20 Lines • Show All 107 Lines • ▼ Show 20 Lines if (consumer.getInput(consumerIdx) != consumedView)
continue; continue;
// Consumer consumes this view, `isStructurallyFusableProducer` also checks // Consumer consumes this view, `isStructurallyFusableProducer` also checks
// whether it is a strict subview of the producer view. // whether it is a strict subview of the producer view.
auto producedView = dependence.dependentOpView.view; auto producedView = dependence.dependentOpView.view;
auto producerIdx = producer.getIndexOfOutput(producedView).getValue(); auto producerIdx = producer.getIndexOfOutput(producedView).getValue();
// `consumerIdx` and `producerIdx` exist by construction. // `consumerIdx` and `producerIdx` exist by construction.
LLVM_DEBUG(dbgs() << "\nRAW producer: " << *producer.getOperation() LLVM_DEBUG(dbgs() << "\nRAW producer: " << *producer.getOperation()
<< " view: " << *producedView << " view: " << producedView
<< " output index: " << producerIdx); << " output index: " << producerIdx);
// Must be a subview or a slice to guarantee there are loops we can fuse // Must be a subview or a slice to guarantee there are loops we can fuse
// into. // into.
auto subView = dyn_cast_or_null<SubViewOp>(consumedView->getDefiningOp()); auto subView = dyn_cast_or_null<SubViewOp>(consumedView.getDefiningOp());
auto slice = dyn_cast_or_null<SliceOp>(consumedView->getDefiningOp()); auto slice = dyn_cast_or_null<SliceOp>(consumedView.getDefiningOp());
if (!subView && !slice) { if (!subView && !slice) {
LLVM_DEBUG(dbgs() << "\nNot fusable (not a subview or slice)"); LLVM_DEBUG(dbgs() << "\nNot fusable (not a subview or slice)");
continue; continue;
} }
// Simple fusability checks. // Simple fusability checks.
if (!isFusableInto(graph, consumer, consumedView, producer)) if (!isFusableInto(graph, consumer, consumedView, producer))
continue; continue;
▲ Show 20 Lines • Show All 57 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Transforms/LinalgTransforms.cpp

Show First 20 Lines • Show All 160 Lines • ▼ Show 20 Linesmlir::linalg::vectorizeGenericLinalgOpPrecondition(Operation *op) {
// TODO(ntv): This is in fact much more general than just vectorization for // TODO(ntv): This is in fact much more general than just vectorization for
// matmul ops. // matmul ops.
auto genericOp = dyn_cast<linalg::GenericOp>(op); auto genericOp = dyn_cast<linalg::GenericOp>(op);
if (!genericOp || !isMatmul(genericOp)) if (!genericOp || !isMatmul(genericOp))
return failure(); return failure();
// TODO(ntv): non-identity layout. // TODO(ntv): non-identity layout.
auto isStaticMemRefWithIdentityLayout = [](Value v) { auto isStaticMemRefWithIdentityLayout = [](Value v) {
auto m = v->getType().dyn_cast<MemRefType>(); auto m = v.getType().dyn_cast<MemRefType>();
if (!m || !m.hasStaticShape() || !m.getAffineMaps().empty()) if (!m || !m.hasStaticShape() || !m.getAffineMaps().empty())
return false; return false;
return true; return true;
}; };
if (!llvm::all_of(genericOp.getInputsAndOutputs(), if (!llvm::all_of(genericOp.getInputsAndOutputs(),
isStaticMemRefWithIdentityLayout)) isStaticMemRefWithIdentityLayout))
return failure(); return failure();
return success(); return success();
▲ Show 20 Lines • Show All 98 Lines • ▼ Show 20 Lines LLVM_DEBUG(dbgs() << "\n[" DEBUG_TYPE "]: Promote subviews for linalg op: "
<< *op << ":\n"); << *op << ":\n");
assert(succeeded(promoteSubviewsLinalgOpPrecondition(op)) && assert(succeeded(promoteSubviewsLinalgOpPrecondition(op)) &&
"DRR failure case must be a precondition"); "DRR failure case must be a precondition");
LinalgOp linOp = cast<LinalgOp>(op); LinalgOp linOp = cast<LinalgOp>(op);
SetVector<Value> subViews; SetVector<Value> subViews;
for (auto it : linOp.getInputsAndOutputs()) for (auto it : linOp.getInputsAndOutputs())
if (auto sv = dyn_cast_or_null<SubViewOp>(it->getDefiningOp())) if (auto sv = dyn_cast_or_null<SubViewOp>(it.getDefiningOp()))
subViews.insert(sv); subViews.insert(sv);
if (!subViews.empty()) { if (!subViews.empty()) {
promoteSubViewOperands(rewriter, linOp, subViews); promoteSubViewOperands(rewriter, linOp, subViews);
return {}; return {};
} }
llvm_unreachable("DRR failure case must be a precondition"); llvm_unreachable("DRR failure case must be a precondition");
} }

mlir/lib/Dialect/Linalg/Transforms/Promotion.cpp

Show First 20 Lines • Show All 41 Lines • ▼ Show 20 Lines
static llvm::cl::OptionCategory clOptionsCategory(DEBUG_TYPE " options"); static llvm::cl::OptionCategory clOptionsCategory(DEBUG_TYPE " options");
static llvm::cl::opt<bool> clPromoteDynamic( static llvm::cl::opt<bool> clPromoteDynamic(
"test-linalg-promote-dynamic", "test-linalg-promote-dynamic",
llvm::cl::desc("Test generation of dynamic promoted buffers"), llvm::cl::desc("Test generation of dynamic promoted buffers"),
llvm::cl::cat(clOptionsCategory), llvm::cl::init(false)); llvm::cl::cat(clOptionsCategory), llvm::cl::init(false));
static Value allocBuffer(Type elementType, Value size, bool dynamicBuffers) { static Value allocBuffer(Type elementType, Value size, bool dynamicBuffers) {
auto *ctx = size->getContext(); auto *ctx = size.getContext();
auto width = llvm::divideCeil(elementType.getIntOrFloatBitWidth(), 8); auto width = llvm::divideCeil(elementType.getIntOrFloatBitWidth(), 8);
if (!dynamicBuffers) if (!dynamicBuffers)
if (auto cst = dyn_cast_or_null<ConstantIndexOp>(size->getDefiningOp())) if (auto cst = dyn_cast_or_null<ConstantIndexOp>(size.getDefiningOp()))
return alloc( return alloc(
MemRefType::get(width * cst.getValue(), IntegerType::get(8, ctx))); MemRefType::get(width * cst.getValue(), IntegerType::get(8, ctx)));
Value mul = muli(constant_index(width), size); Value mul = muli(constant_index(width), size);
return alloc(MemRefType::get(-1, IntegerType::get(8, ctx)), mul); return alloc(MemRefType::get(-1, IntegerType::get(8, ctx)), mul);
} }
// Performs promotion of a `subView` into a local buffer of the size of the // Performs promotion of a `subView` into a local buffer of the size of the
// *ranges* of the `subView`. This produces a buffer whose size may be bigger // *ranges* of the `subView`. This produces a buffer whose size may be bigger
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Linesmlir::linalg::promoteSubViews(OpBuilder &b, Location loc,
if (subViews.empty()) if (subViews.empty())
return {}; return {};
ScopedContext scope(b, loc); ScopedContext scope(b, loc);
SmallVector<PromotionInfo, 8> res; SmallVector<PromotionInfo, 8> res;
res.reserve(subViews.size()); res.reserve(subViews.size());
DenseMap<Value, PromotionInfo> promotionInfoMap; DenseMap<Value, PromotionInfo> promotionInfoMap;
for (auto v : subViews) { for (auto v : subViews) {
SubViewOp subView = cast<SubViewOp>(v->getDefiningOp()); SubViewOp subView = cast<SubViewOp>(v.getDefiningOp());
auto viewType = subView.getType(); auto viewType = subView.getType();
// TODO(ntv): support more cases than just float. // TODO(ntv): support more cases than just float.
if (!viewType.getElementType().isa<FloatType>()) if (!viewType.getElementType().isa<FloatType>())
continue; continue;
auto promotionInfo = auto promotionInfo =
promoteFullTileBuffer(b, loc, subView, dynamicBuffers, folder); promoteFullTileBuffer(b, loc, subView, dynamicBuffers, folder);
promotionInfoMap.insert(std::make_pair(subView.getResult(), promotionInfo)); promotionInfoMap.insert(std::make_pair(subView.getResult(), promotionInfo));
res.push_back(promotionInfo); res.push_back(promotionInfo);
} }
for (auto v : subViews) { for (auto v : subViews) {
SubViewOp subView = cast<SubViewOp>(v->getDefiningOp()); SubViewOp subView = cast<SubViewOp>(v.getDefiningOp());
auto info = promotionInfoMap.find(v); auto info = promotionInfoMap.find(v);
if (info == promotionInfoMap.end()) if (info == promotionInfoMap.end())
continue; continue;
// TODO(ntv): value to fill with should be related to the operation. // TODO(ntv): value to fill with should be related to the operation.
// For now, just use APFloat(0.0f). // For now, just use APFloat(0.0f).
auto t = subView.getType().getElementType().cast<FloatType>(); auto t = subView.getType().getElementType().cast<FloatType>();
Value fillVal = constant_float(folder, APFloat(0.0f), t); Value fillVal = constant_float(folder, APFloat(0.0f), t);
// TODO(ntv): fill is only necessary if `promotionInfo` has a full local // TODO(ntv): fill is only necessary if `promotionInfo` has a full local
// view that is different from the partial local view and we are on the // view that is different from the partial local view and we are on the
// boundary. // boundary.
fill(info->second.fullLocalView, fillVal); fill(info->second.fullLocalView, fillVal);
} }
for (auto v : subViews) { for (auto v : subViews) {
auto info = promotionInfoMap.find(v); auto info = promotionInfoMap.find(v);
if (info == promotionInfoMap.end()) if (info == promotionInfoMap.end())
continue; continue;
copy(cast<SubViewOp>(v->getDefiningOp()), info->second.partialLocalView); copy(cast<SubViewOp>(v.getDefiningOp()), info->second.partialLocalView);
} }
return res; return res;
} }
LinalgOp mlir::linalg::promoteSubViewOperands(OpBuilder &b, LinalgOp op, LinalgOp mlir::linalg::promoteSubViewOperands(OpBuilder &b, LinalgOp op,
SetVector<Value> subViews, SetVector<Value> subViews,
bool dynamicBuffers, bool dynamicBuffers,
OperationFolder *folder) { OperationFolder *folder) {
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Linesstatic void promoteSubViews(FuncOp f, bool dynamicBuffers) {
SmallVector<LinalgOp, 8> toErase; SmallVector<LinalgOp, 8> toErase;
OperationFolder folder(f.getContext()); OperationFolder folder(f.getContext());
f.walk([dynamicBuffers, &folder, &toErase](LinalgOp op) { f.walk([dynamicBuffers, &folder, &toErase](LinalgOp op) {
// TODO(ntv) some heuristic here to decide what to promote. Atm it is all or // TODO(ntv) some heuristic here to decide what to promote. Atm it is all or
// nothing. // nothing.
SetVector<Value> subViews; SetVector<Value> subViews;
OpBuilder b(op); OpBuilder b(op);
for (auto it : op.getInputsAndOutputs()) for (auto it : op.getInputsAndOutputs())
if (auto sv = dyn_cast_or_null<SubViewOp>(it->getDefiningOp())) if (auto sv = dyn_cast_or_null<SubViewOp>(it.getDefiningOp()))
subViews.insert(sv); subViews.insert(sv);
if (!subViews.empty()) { if (!subViews.empty()) {
promoteSubViewOperands(b, op, subViews, dynamicBuffers, &folder); promoteSubViewOperands(b, op, subViews, dynamicBuffers, &folder);
toErase.push_back(op); toErase.push_back(op);
} }
}); });
for (auto op : toErase) for (auto op : toErase)
op.erase(); op.erase();
Show All 24 Lines

mlir/lib/Dialect/Linalg/Transforms/Tiling.cpp

Show All 39 Lines
static llvm::cl::OptionCategory clOptionsCategory(DEBUG_TYPE " options"); static llvm::cl::OptionCategory clOptionsCategory(DEBUG_TYPE " options");
static llvm::cl::list<unsigned> static llvm::cl::list<unsigned>
clTileSizes("linalg-tile-sizes", clTileSizes("linalg-tile-sizes",
llvm::cl::desc("Tile sizes by which to tile linalg operations"), llvm::cl::desc("Tile sizes by which to tile linalg operations"),
llvm::cl::ZeroOrMore, llvm::cl::MiscFlags::CommaSeparated, llvm::cl::ZeroOrMore, llvm::cl::MiscFlags::CommaSeparated,
llvm::cl::cat(clOptionsCategory)); llvm::cl::cat(clOptionsCategory));
static bool isZero(Value v) { static bool isZero(Value v) {
return isa_and_nonnull<ConstantIndexOp>(v->getDefiningOp()) && return isa_and_nonnull<ConstantIndexOp>(v.getDefiningOp()) &&
cast<ConstantIndexOp>(v->getDefiningOp()).getValue() == 0; cast<ConstantIndexOp>(v.getDefiningOp()).getValue() == 0;
} }
using LoopIndexToRangeIndexMap = DenseMap<int, int>; using LoopIndexToRangeIndexMap = DenseMap<int, int>;
// Creates a number of ranges equal to the number of non-zero in `tileSizes`. // Creates a number of ranges equal to the number of non-zero in `tileSizes`.
// One for each loop of the LinalgOp that is tiled. The `tileSizes` argument has // One for each loop of the LinalgOp that is tiled. The `tileSizes` argument has
// one entry per surrounding loop. It uses zero as the convention that a // one entry per surrounding loop. It uses zero as the convention that a
// particular loop is not tiled. This convention simplifies implementations by // particular loop is not tiled. This convention simplifies implementations by
▲ Show 20 Lines • Show All 138 Lines • ▼ Show 20 Lines for (unsigned i = 0; i < indexedGenericOp.getNumLoops(); ++i) {
auto rangeIndex = loopIndexToRangeIndex.find(i); auto rangeIndex = loopIndexToRangeIndex.find(i);
if (rangeIndex == loopIndexToRangeIndex.end()) if (rangeIndex == loopIndexToRangeIndex.end())
continue; continue;
Value oldIndex = block.getArgument(i); Value oldIndex = block.getArgument(i);
// Offset the index argument `i` by the value of the corresponding induction // Offset the index argument `i` by the value of the corresponding induction
// variable and replace all uses of the previous value. // variable and replace all uses of the previous value.
Value newIndex = b.create<AddIOp>(indexedGenericOp.getLoc(), oldIndex, Value newIndex = b.create<AddIOp>(indexedGenericOp.getLoc(), oldIndex,
pivs[rangeIndex->second]->getValue()); pivs[rangeIndex->second]->getValue());
for (auto &use : oldIndex->getUses()) { for (auto &use : oldIndex.getUses()) {
if (use.getOwner() == newIndex->getDefiningOp()) if (use.getOwner() == newIndex.getDefiningOp())
continue; continue;
use.set(newIndex); use.set(newIndex);
} }
} }
} }
static bool isTiled(AffineExpr expr, ArrayRef<Value> tileSizes) { static bool isTiled(AffineExpr expr, ArrayRef<Value> tileSizes) {
if (!expr) if (!expr)
Show All 39 LinesmakeTiledViews(OpBuilder &b, Location loc, LinalgOp linalgOp,
auto *op = linalgOp.getOperation(); auto *op = linalgOp.getOperation();
SmallVector<Value, 4> res; SmallVector<Value, 4> res;
res.reserve(op->getNumOperands()); res.reserve(op->getNumOperands());
auto viewIteratorBegin = linalgOp.getInputsAndOutputs().begin(); auto viewIteratorBegin = linalgOp.getInputsAndOutputs().begin();
for (unsigned viewIndex = 0; viewIndex < linalgOp.getNumInputsAndOutputs(); for (unsigned viewIndex = 0; viewIndex < linalgOp.getNumInputsAndOutputs();
++viewIndex) { ++viewIndex) {
Value view = *(viewIteratorBegin + viewIndex); Value view = *(viewIteratorBegin + viewIndex);
unsigned rank = view->getType().cast<MemRefType>().getRank(); unsigned rank = view.getType().cast<MemRefType>().getRank();
auto map = loopToOperandRangesMaps(linalgOp)[viewIndex]; auto map = loopToOperandRangesMaps(linalgOp)[viewIndex];
// If the view is not tiled, we can use it as is. // If the view is not tiled, we can use it as is.
if (!isTiled(map, tileSizes)) { if (!isTiled(map, tileSizes)) {
res.push_back(view); res.push_back(view);
continue; continue;
} }
// Construct a new subview for the tile. // Construct a new subview for the tile.
Show All 24 Lines for (unsigned viewIndex = 0; viewIndex < linalgOp.getNumInputsAndOutputs();
res.push_back(b.create<SubViewOp>(loc, view, offsets, sizes, strides)); res.push_back(b.create<SubViewOp>(loc, view, offsets, sizes, strides));
} }
// Traverse the mins/maxes and erase those that don't have uses left. // Traverse the mins/maxes and erase those that don't have uses left.
// This is a special type of folding that we only apply when `folder` is // This is a special type of folding that we only apply when `folder` is
// defined. // defined.
if (folder) if (folder)
for (auto v : llvm::concat<Value>(lbs, subViewSizes)) for (auto v : llvm::concat<Value>(lbs, subViewSizes))
if (v->use_empty()) if (v.use_empty())
v->getDefiningOp()->erase(); v.getDefiningOp()->erase();
return res; return res;
} }
Optional<TiledLinalgOp> Optional<TiledLinalgOp>
mlir::linalg::tileLinalgOp(OpBuilder &b, LinalgOp op, ArrayRef<Value> tileSizes, mlir::linalg::tileLinalgOp(OpBuilder &b, LinalgOp op, ArrayRef<Value> tileSizes,
ArrayRef<unsigned> permutation, ArrayRef<unsigned> permutation,
OperationFolder *folder) { OperationFolder *folder) {
▲ Show 20 Lines • Show All 150 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Utils/Utils.cpp

Show All 29 Lines
using namespace mlir::edsc::intrinsics; using namespace mlir::edsc::intrinsics;
using namespace mlir::linalg; using namespace mlir::linalg;
using namespace mlir::linalg::intrinsics; using namespace mlir::linalg::intrinsics;
using namespace mlir::loop; using namespace mlir::loop;
mlir::edsc::LoopRangeBuilder::LoopRangeBuilder(ValueHandle *iv, mlir::edsc::LoopRangeBuilder::LoopRangeBuilder(ValueHandle *iv,
ValueHandle range) { ValueHandle range) {
assert(range.getType() && "expected !linalg.range type"); assert(range.getType() && "expected !linalg.range type");
assert(range.getValue()->getDefiningOp() && assert(range.getValue().getDefiningOp() &&
"need operations to extract range parts"); "need operations to extract range parts");
auto rangeOp = cast<RangeOp>(range.getValue()->getDefiningOp()); auto rangeOp = cast<RangeOp>(range.getValue().getDefiningOp());
auto lb = rangeOp.min(); auto lb = rangeOp.min();
auto ub = rangeOp.max(); auto ub = rangeOp.max();
auto step = rangeOp.step(); auto step = rangeOp.step();
auto forOp = OperationHandle::createOp<ForOp>(lb, ub, step); auto forOp = OperationHandle::createOp<ForOp>(lb, ub, step);
*iv = ValueHandle(forOp.getInductionVar()); *iv = ValueHandle(forOp.getInductionVar());
auto *body = forOp.getBody(); auto *body = forOp.getBody();
enter(body, /*prev=*/1); enter(body, /*prev=*/1);
} }
▲ Show 20 Lines • Show All 114 Lines • ▼ Show 20 Lines
mlir::linalg::getAssumedNonViewOperands(LinalgOp linalgOp) { mlir::linalg::getAssumedNonViewOperands(LinalgOp linalgOp) {
auto *op = linalgOp.getOperation(); auto *op = linalgOp.getOperation();
unsigned numViews = linalgOp.getNumInputsAndOutputs(); unsigned numViews = linalgOp.getNumInputsAndOutputs();
unsigned nOperands = op->getNumOperands() - numViews; unsigned nOperands = op->getNumOperands() - numViews;
SmallVector<Value, 4> res; SmallVector<Value, 4> res;
res.reserve(nOperands); res.reserve(nOperands);
for (unsigned i = 0; i < nOperands; ++i) { for (unsigned i = 0; i < nOperands; ++i) {
res.push_back(op->getOperand(numViews + i)); res.push_back(op->getOperand(numViews + i));
auto t = res.back()->getType(); auto t = res.back().getType();
(void)t; (void)t;
assert((t.isIntOrIndexOrFloat() || t.isa<VectorType>()) && assert((t.isIntOrIndexOrFloat() || t.isa<VectorType>()) &&
"expected scalar or vector type"); "expected scalar or vector type");
} }
return res; return res;
} }

mlir/lib/Dialect/LoopOps/LoopOps.cpp

Show First 20 Lines • Show All 63 Lines • ▼ Show 20 Linesvoid ForOp::build(Builder *builder, OperationState &result, Value lb, Value ub,
Value step) { Value step) {
result.addOperands({lb, ub, step}); result.addOperands({lb, ub, step});
Region *bodyRegion = result.addRegion(); Region *bodyRegion = result.addRegion();
ForOp::ensureTerminator(*bodyRegion, *builder, result.location); ForOp::ensureTerminator(*bodyRegion, *builder, result.location);
bodyRegion->front().addArgument(builder->getIndexType()); bodyRegion->front().addArgument(builder->getIndexType());
} }
LogicalResult verify(ForOp op) { LogicalResult verify(ForOp op) {
if (auto cst = dyn_cast_or_null<ConstantIndexOp>(op.step()->getDefiningOp())) if (auto cst = dyn_cast_or_null<ConstantIndexOp>(op.step().getDefiningOp()))
if (cst.getValue() <= 0) if (cst.getValue() <= 0)
return op.emitOpError("constant step operand must be positive"); return op.emitOpError("constant step operand must be positive");
// Check that the body defines as single block argument for the induction // Check that the body defines as single block argument for the induction
// variable. // variable.
auto *body = op.getBody(); auto *body = op.getBody();
if (body->getNumArguments() != 1 || if (body->getNumArguments() != 1 || !body->getArgument(0).getType().isIndex())
!body->getArgument(0)->getType().isIndex())
return op.emitOpError("expected body to have a single index argument for " return op.emitOpError("expected body to have a single index argument for "
"the induction variable"); "the induction variable");
return success(); return success();
} }
static void print(OpAsmPrinter &p, ForOp op) { static void print(OpAsmPrinter &p, ForOp op) {
p << op.getOperationName() << " " << *op.getInductionVar() << " = " p << op.getOperationName() << " " << op.getInductionVar() << " = "
<< *op.lowerBound() << " to " << *op.upperBound() << " step " << *op.step(); << op.lowerBound() << " to " << op.upperBound() << " step " << op.step();
p.printRegion(op.region(), p.printRegion(op.region(),
/*printEntryBlockArgs=*/false, /*printEntryBlockArgs=*/false,
/*printBlockTerminators=*/false); /*printBlockTerminators=*/false);
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
} }
static ParseResult parseForOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseForOp(OpAsmParser &parser, OperationState &result) {
auto &builder = parser.getBuilder(); auto &builder = parser.getBuilder();
Show All 24 Lines if (parser.parseOptionalAttrDict(result.attributes))
return failure(); return failure();
return success(); return success();
} }
Region &ForOp::getLoopBody() { return region(); } Region &ForOp::getLoopBody() { return region(); }
bool ForOp::isDefinedOutsideOfLoop(Value value) { bool ForOp::isDefinedOutsideOfLoop(Value value) {
return !region().isAncestor(value->getParentRegion()); return !region().isAncestor(value.getParentRegion());
} }
LogicalResult ForOp::moveOutOfLoop(ArrayRef<Operation *> ops) { LogicalResult ForOp::moveOutOfLoop(ArrayRef<Operation *> ops) {
for (auto *op : ops) for (auto op : ops)
op->moveBefore(this->getOperation()); op->moveBefore(this->getOperation());
return success(); return success();
} }
ForOp mlir::loop::getForInductionVarOwner(Value val) { ForOp mlir::loop::getForInductionVarOwner(Value val) {
auto ivArg = val.dyn_cast<BlockArgument>(); auto ivArg = val.dyn_cast<BlockArgument>();
if (!ivArg) if (!ivArg)
return ForOp(); return ForOp();
assert(ivArg->getOwner() && "unlinked block argument"); assert(ivArg.getOwner() && "unlinked block argument");
auto *containingInst = ivArg->getOwner()->getParentOp(); auto *containingInst = ivArg.getOwner()->getParentOp();
return dyn_cast_or_null<ForOp>(containingInst); return dyn_cast_or_null<ForOp>(containingInst);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// IfOp // IfOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void IfOp::build(Builder *builder, OperationState &result, Value cond, void IfOp::build(Builder *builder, OperationState &result, Value cond,
▲ Show 20 Lines • Show All 48 Lines • ▼ Show 20 Linesstatic ParseResult parseIfOp(OpAsmParser &parser, OperationState &result) {
// Parse the optional attribute list. // Parse the optional attribute list.
if (parser.parseOptionalAttrDict(result.attributes)) if (parser.parseOptionalAttrDict(result.attributes))
return failure(); return failure();
return success(); return success();
} }
static void print(OpAsmPrinter &p, IfOp op) { static void print(OpAsmPrinter &p, IfOp op) {
p << IfOp::getOperationName() << " " << *op.condition(); p << IfOp::getOperationName() << " " << op.condition();
p.printRegion(op.thenRegion(), p.printRegion(op.thenRegion(),
/*printEntryBlockArgs=*/false, /*printEntryBlockArgs=*/false,
/*printBlockTerminators=*/false); /*printBlockTerminators=*/false);
// Print the 'else' regions if it exists and has a block. // Print the 'else' regions if it exists and has a block.
auto &elseRegion = op.elseRegion(); auto &elseRegion = op.elseRegion();
if (!elseRegion.empty()) { if (!elseRegion.empty()) {
p << " else"; p << " else";
Show All 14 Lines

mlir/lib/Dialect/QuantOps/IR/QuantOps.cpp

Show All 30 Lines
#define GET_OP_LIST #define GET_OP_LIST
#include "mlir/Dialect/QuantOps/QuantOps.cpp.inc" #include "mlir/Dialect/QuantOps/QuantOps.cpp.inc"
>(); >();
} }
OpFoldResult StorageCastOp::fold(ArrayRef<Attribute> operands) { OpFoldResult StorageCastOp::fold(ArrayRef<Attribute> operands) {
/// Matches x -> [scast -> scast] -> y, replacing the second scast with the /// Matches x -> [scast -> scast] -> y, replacing the second scast with the
/// value of x if the casts invert each other. /// value of x if the casts invert each other.
auto srcScastOp = dyn_cast_or_null<StorageCastOp>(arg()->getDefiningOp()); auto srcScastOp = dyn_cast_or_null<StorageCastOp>(arg().getDefiningOp());
if (!srcScastOp || srcScastOp.arg()->getType() != getType()) if (!srcScastOp || srcScastOp.arg().getType() != getType())
return OpFoldResult(); return OpFoldResult();
return srcScastOp.arg(); return srcScastOp.arg();
} }
#define GET_OP_CLASSES #define GET_OP_CLASSES
#include "mlir/Dialect/QuantOps/QuantOps.cpp.inc" #include "mlir/Dialect/QuantOps/QuantOps.cpp.inc"

mlir/lib/Dialect/QuantOps/Transforms/ConvertConst.cpp

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 LinesQuantizedConstRewrite::matchAndRewrite(QuantizeCastOp qbarrier,
// Is the operand a constant? // Is the operand a constant?
if (!matchPattern(qbarrier.arg(), m_Constant(&value))) { if (!matchPattern(qbarrier.arg(), m_Constant(&value))) {
return matchFailure(); return matchFailure();
} }
// Does the qbarrier convert to a quantized type. This will not be true // Does the qbarrier convert to a quantized type. This will not be true
// if a quantized type has not yet been chosen or if the cast to an equivalent // if a quantized type has not yet been chosen or if the cast to an equivalent
// storage type is not supported. // storage type is not supported.
Type qbarrierResultType = qbarrier.getResult()->getType(); Type qbarrierResultType = qbarrier.getResult().getType();
QuantizedType quantizedElementType = QuantizedType quantizedElementType =
QuantizedType::getQuantizedElementType(qbarrierResultType); QuantizedType::getQuantizedElementType(qbarrierResultType);
if (!quantizedElementType) { if (!quantizedElementType) {
return matchFailure(); return matchFailure();
} }
if (!QuantizedType::castToStorageType(qbarrierResultType)) { if (!QuantizedType::castToStorageType(qbarrierResultType)) {
return matchFailure(); return matchFailure();
} }
// Is the operand type compatible with the expressed type of the quantized // Is the operand type compatible with the expressed type of the quantized
// type? This will not be true if the qbarrier is superfluous (converts // type? This will not be true if the qbarrier is superfluous (converts
// from and to a quantized type). // from and to a quantized type).
if (!quantizedElementType.isCompatibleExpressedType( if (!quantizedElementType.isCompatibleExpressedType(
qbarrier.arg()->getType())) { qbarrier.arg().getType())) {
return matchFailure(); return matchFailure();
} }
// Is the constant value a type expressed in a way that we support? // Is the constant value a type expressed in a way that we support?
if (!value.isa<FloatAttr>() && !value.isa<DenseElementsAttr>() && if (!value.isa<FloatAttr>() && !value.isa<DenseElementsAttr>() &&
!value.isa<SparseElementsAttr>()) { !value.isa<SparseElementsAttr>()) {
return matchFailure(); return matchFailure();
} }
Type newConstValueType; Type newConstValueType;
auto newConstValue = auto newConstValue =
quantizeAttr(value, quantizedElementType, newConstValueType); quantizeAttr(value, quantizedElementType, newConstValueType);
if (!newConstValue) { if (!newConstValue) {
return matchFailure(); return matchFailure();
} }
// When creating the new const op, use a fused location that combines the // When creating the new const op, use a fused location that combines the
// original const and the qbarrier that led to the quantization. // original const and the qbarrier that led to the quantization.
auto fusedLoc = FusedLoc::get( auto fusedLoc = FusedLoc::get(
{qbarrier.arg()->getDefiningOp()->getLoc(), qbarrier.getLoc()}, {qbarrier.arg().getDefiningOp()->getLoc(), qbarrier.getLoc()},
rewriter.getContext()); rewriter.getContext());
auto newConstOp = auto newConstOp =
rewriter.create<ConstantOp>(fusedLoc, newConstValueType, newConstValue); rewriter.create<ConstantOp>(fusedLoc, newConstValueType, newConstValue);
rewriter.replaceOpWithNewOp<StorageCastOp>({qbarrier.arg()}, qbarrier, rewriter.replaceOpWithNewOp<StorageCastOp>({qbarrier.arg()}, qbarrier,
qbarrier.getType(), newConstOp); qbarrier.getType(), newConstOp);
return matchSuccess(); return matchSuccess();
} }
Show All 15 Lines

mlir/lib/Dialect/SPIRV/SPIRVDialect.cpp

Show First 20 Lines • Show All 98 Lines • ▼ Show 20 Lines void handleTerminator(Operation *op,
// Only spv.ReturnValue needs to be handled here. // Only spv.ReturnValue needs to be handled here.
auto retValOp = dyn_cast<spirv::ReturnValueOp>(op); auto retValOp = dyn_cast<spirv::ReturnValueOp>(op);
if (!retValOp) if (!retValOp)
return; return;
// Replace the values directly with the return operands. // Replace the values directly with the return operands.
assert(valuesToRepl.size() == 1 && assert(valuesToRepl.size() == 1 &&
"spv.ReturnValue expected to only handle one result"); "spv.ReturnValue expected to only handle one result");
valuesToRepl.front()->replaceAllUsesWith(retValOp.value()); valuesToRepl.front().replaceAllUsesWith(retValOp.value());
} }
}; };
} // namespace } // namespace
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SPIR-V Dialect // SPIR-V Dialect
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 584 Lines • Show Last 20 Lines

mlir/lib/Dialect/SPIRV/SPIRVOps.cpp

Show First 20 Lines • Show All 161 Lines • ▼ Show 20 Lines if (auto memAccess = loadStoreOp.memory_access()) {
} }
printer << "]"; printer << "]";
} }
elidedAttrs.push_back(spirv::attributeName<spirv::StorageClass>()); elidedAttrs.push_back(spirv::attributeName<spirv::StorageClass>());
} }
static LogicalResult verifyCastOp(Operation *op, static LogicalResult verifyCastOp(Operation *op,
bool requireSameBitWidth = true) { bool requireSameBitWidth = true) {
Type operandType = op->getOperand(0)->getType(); Type operandType = op->getOperand(0).getType();
Type resultType = op->getResult(0)->getType(); Type resultType = op->getResult(0).getType();
// ODS checks that result type and operand type have the same shape. // ODS checks that result type and operand type have the same shape.
if (auto vectorType = operandType.dyn_cast<VectorType>()) { if (auto vectorType = operandType.dyn_cast<VectorType>()) {
operandType = vectorType.getElementType(); operandType = vectorType.getElementType();
resultType = resultType.cast<VectorType>().getElementType(); resultType = resultType.cast<VectorType>().getElementType();
} }
auto operandTypeBitWidth = operandType.getIntOrFloatBitWidth(); auto operandTypeBitWidth = operandType.getIntOrFloatBitWidth();
▲ Show 20 Lines • Show All 86 Lines • ▼ Show 20 Lines
template <typename LoadStoreOpTy> template <typename LoadStoreOpTy>
static LogicalResult verifyLoadStorePtrAndValTypes(LoadStoreOpTy op, Value ptr, static LogicalResult verifyLoadStorePtrAndValTypes(LoadStoreOpTy op, Value ptr,
Value val) { Value val) {
// ODS already checks ptr is spirv::PointerType. Just check that the pointee // ODS already checks ptr is spirv::PointerType. Just check that the pointee
// type of the pointer and the type of the value are the same // type of the pointer and the type of the value are the same
// //
// TODO(ravishankarm): Check that the value type satisfies restrictions of // TODO(ravishankarm): Check that the value type satisfies restrictions of
// SPIR-V OpLoad/OpStore operations // SPIR-V OpLoad/OpStore operations
if (val->getType() != if (val.getType() !=
ptr->getType().cast<spirv::PointerType>().getPointeeType()) { ptr.getType().cast<spirv::PointerType>().getPointeeType()) {
return op.emitOpError("mismatch in result type and pointer type"); return op.emitOpError("mismatch in result type and pointer type");
} }
return success(); return success();
} }
static ParseResult parseVariableDecorations(OpAsmParser &parser, static ParseResult parseVariableDecorations(OpAsmParser &parser,
OperationState &state) { OperationState &state) {
auto builtInName = auto builtInName =
▲ Show 20 Lines • Show All 208 Lines • ▼ Show 20 Lines
} }
static void printBitFieldExtractOp(Operation *op, OpAsmPrinter &printer) { static void printBitFieldExtractOp(Operation *op, OpAsmPrinter &printer) {
printer << op->getName() << ' ' << op->getOperands() << " : " printer << op->getName() << ' ' << op->getOperands() << " : "
<< op->getOperandTypes(); << op->getOperandTypes();
} }
static LogicalResult verifyBitFieldExtractOp(Operation *op) { static LogicalResult verifyBitFieldExtractOp(Operation *op) {
if (op->getOperand(0)->getType() != op->getResult(0)->getType()) { if (op->getOperand(0).getType() != op->getResult(0).getType()) {
return op->emitError("expected the same type for the first operand and " return op->emitError("expected the same type for the first operand and "
"result, but provided ") "result, but provided ")
<< op->getOperand(0)->getType() << " and " << op->getOperand(0).getType() << " and "
<< op->getResult(0)->getType(); << op->getResult(0).getType();
} }
return success(); return success();
} }
// Parses an atomic update op. If the update op does not take a value (like // Parses an atomic update op. If the update op does not take a value (like
// AtomicIIncrement) `hasValue` must be false. // AtomicIIncrement) `hasValue` must be false.
static ParseResult parseAtomicUpdateOp(OpAsmParser &parser, static ParseResult parseAtomicUpdateOp(OpAsmParser &parser,
OperationState &state, bool hasValue) { OperationState &state, bool hasValue) {
Show All 29 Linesstatic void printAtomicUpdateOp(Operation *op, OpAsmPrinter &printer) {
auto scopeAttr = op->getAttrOfType<IntegerAttr>(kMemoryScopeAttrName); auto scopeAttr = op->getAttrOfType<IntegerAttr>(kMemoryScopeAttrName);
printer << spirv::stringifyScope( printer << spirv::stringifyScope(
static_cast<spirv::Scope>(scopeAttr.getInt())) static_cast<spirv::Scope>(scopeAttr.getInt()))
<< "\" \""; << "\" \"";
auto memorySemanticsAttr = op->getAttrOfType<IntegerAttr>(kSemanticsAttrName); auto memorySemanticsAttr = op->getAttrOfType<IntegerAttr>(kSemanticsAttrName);
printer << spirv::stringifyMemorySemantics( printer << spirv::stringifyMemorySemantics(
static_cast<spirv::MemorySemantics>( static_cast<spirv::MemorySemantics>(
memorySemanticsAttr.getInt())) memorySemanticsAttr.getInt()))
<< "\" " << op->getOperands() << " : " << "\" " << op->getOperands() << " : " << op->getOperand(0).getType();
<< op->getOperand(0)->getType();
} }
// Verifies an atomic update op. // Verifies an atomic update op.
static LogicalResult verifyAtomicUpdateOp(Operation *op) { static LogicalResult verifyAtomicUpdateOp(Operation *op) {
auto ptrType = op->getOperand(0)->getType().cast<spirv::PointerType>(); auto ptrType = op->getOperand(0).getType().cast<spirv::PointerType>();
auto elementType = ptrType.getPointeeType(); auto elementType = ptrType.getPointeeType();
if (!elementType.isa<IntegerType>()) if (!elementType.isa<IntegerType>())
return op->emitOpError( return op->emitOpError(
"pointer operand must point to an integer value, found ") "pointer operand must point to an integer value, found ")
<< elementType; << elementType;
if (op->getNumOperands() > 1) { if (op->getNumOperands() > 1) {
auto valueType = op->getOperand(1)->getType(); auto valueType = op->getOperand(1).getType();
if (valueType != elementType) if (valueType != elementType)
return op->emitOpError("expected value to have the same type as the " return op->emitOpError("expected value to have the same type as the "
"pointer operand's pointee type ") "pointer operand's pointee type ")
<< elementType << ", but found " << valueType; << elementType << ", but found " << valueType;
} }
return success(); return success();
} }
Show All 17 Lines if (parser.parseOperand(operandInfo) || parser.parseColonType(type) ||
parser.resolveOperands(operandInfo, type, state.operands)) { parser.resolveOperands(operandInfo, type, state.operands)) {
return failure(); return failure();
} }
state.addTypes(type); state.addTypes(type);
return success(); return success();
} }
static void printUnaryOp(Operation *unaryOp, OpAsmPrinter &printer) { static void printUnaryOp(Operation *unaryOp, OpAsmPrinter &printer) {
printer << unaryOp->getName() << ' ' << *unaryOp->getOperand(0) << " : " printer << unaryOp->getName() << ' ' << unaryOp->getOperand(0) << " : "
<< unaryOp->getOperand(0)->getType(); << unaryOp->getOperand(0).getType();
} }
/// Result of a logical op must be a scalar or vector of boolean type. /// Result of a logical op must be a scalar or vector of boolean type.
static Type getUnaryOpResultType(Builder &builder, Type operandType) { static Type getUnaryOpResultType(Builder &builder, Type operandType) {
Type resultType = builder.getIntegerType(1); Type resultType = builder.getIntegerType(1);
if (auto vecType = operandType.dyn_cast<VectorType>()) { if (auto vecType = operandType.dyn_cast<VectorType>()) {
return VectorType::get(vecType.getNumElements(), resultType); return VectorType::get(vecType.getNumElements(), resultType);
} }
Show All 21 Lines if (parser.parseOperandList(ops, 2) || parser.parseColonType(type) ||
return failure(); return failure();
} }
result.addTypes(getUnaryOpResultType(parser.getBuilder(), type)); result.addTypes(getUnaryOpResultType(parser.getBuilder(), type));
return success(); return success();
} }
static void printLogicalOp(Operation *logicalOp, OpAsmPrinter &printer) { static void printLogicalOp(Operation *logicalOp, OpAsmPrinter &printer) {
printer << logicalOp->getName() << ' ' << logicalOp->getOperands() << " : " printer << logicalOp->getName() << ' ' << logicalOp->getOperands() << " : "
<< logicalOp->getOperand(0)->getType(); << logicalOp->getOperand(0).getType();
} }
static ParseResult parseShiftOp(OpAsmParser &parser, OperationState &state) { static ParseResult parseShiftOp(OpAsmParser &parser, OperationState &state) {
SmallVector<OpAsmParser::OperandType, 2> operandInfo; SmallVector<OpAsmParser::OperandType, 2> operandInfo;
Type baseType; Type baseType;
Type shiftType; Type shiftType;
auto loc = parser.getCurrentLocation(); auto loc = parser.getCurrentLocation();
if (parser.parseOperandList(operandInfo, 2) || parser.parseColon() || if (parser.parseOperandList(operandInfo, 2) || parser.parseColon() ||
parser.parseType(baseType) || parser.parseComma() || parser.parseType(baseType) || parser.parseComma() ||
parser.parseType(shiftType) || parser.parseType(shiftType) ||
parser.resolveOperands(operandInfo, {baseType, shiftType}, loc, parser.resolveOperands(operandInfo, {baseType, shiftType}, loc,
state.operands)) { state.operands)) {
return failure(); return failure();
} }
state.addTypes(baseType); state.addTypes(baseType);
return success(); return success();
} }
static void printShiftOp(Operation *op, OpAsmPrinter &printer) { static void printShiftOp(Operation *op, OpAsmPrinter &printer) {
Value base = op->getOperand(0); Value base = op->getOperand(0);
Value shift = op->getOperand(1); Value shift = op->getOperand(1);
printer << op->getName() << ' ' << *base << ", " << *shift << " : " printer << op->getName() << ' ' << base << ", " << shift << " : "
<< base->getType() << ", " << shift->getType(); << base.getType() << ", " << shift.getType();
} }
static LogicalResult verifyShiftOp(Operation *op) { static LogicalResult verifyShiftOp(Operation *op) {
if (op->getOperand(0)->getType() != op->getResult(0)->getType()) { if (op->getOperand(0).getType() != op->getResult(0).getType()) {
return op->emitError("expected the same type for the first operand and " return op->emitError("expected the same type for the first operand and "
"result, but provided ") "result, but provided ")
<< op->getOperand(0)->getType() << " and " << op->getOperand(0).getType() << " and "
<< op->getResult(0)->getType(); << op->getResult(0).getType();
} }
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.AccessChainOp // spv.AccessChainOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 21 Lines for (auto indexSSA : indices) {
if (!cType) { if (!cType) {
emitError(baseLoc, emitError(baseLoc,
"'spv.AccessChain' op cannot extract from non-composite type ") "'spv.AccessChain' op cannot extract from non-composite type ")
<< resultType << " with index " << index; << resultType << " with index " << index;
return nullptr; return nullptr;
} }
index = 0; index = 0;
if (resultType.isa<spirv::StructType>()) { if (resultType.isa<spirv::StructType>()) {
Operation *op = indexSSA->getDefiningOp(); Operation *op = indexSSA.getDefiningOp();
if (!op) { if (!op) {
emitError(baseLoc, "'spv.AccessChain' op index must be an " emitError(baseLoc, "'spv.AccessChain' op index must be an "
"integer spv.constant to access " "integer spv.constant to access "
"element of spv.struct"); "element of spv.struct");
return nullptr; return nullptr;
} }
// TODO(denis0x0D): this should be relaxed to allow // TODO(denis0x0D): this should be relaxed to allow
Show All 13 Lines for (auto indexSSA : indices) {
} }
resultType = cType.getElementType(index); resultType = cType.getElementType(index);
} }
return spirv::PointerType::get(resultType, resultStorageClass); return spirv::PointerType::get(resultType, resultStorageClass);
} }
void spirv::AccessChainOp::build(Builder *builder, OperationState &state, void spirv::AccessChainOp::build(Builder *builder, OperationState &state,
Value basePtr, ValueRange indices) { Value basePtr, ValueRange indices) {
auto type = getElementPtrType(basePtr->getType(), indices, state.location); auto type = getElementPtrType(basePtr.getType(), indices, state.location);
assert(type && "Unable to deduce return type based on basePtr and indices"); assert(type && "Unable to deduce return type based on basePtr and indices");
build(builder, state, type, basePtr, indices); build(builder, state, type, basePtr, indices);
} }
static ParseResult parseAccessChainOp(OpAsmParser &parser, static ParseResult parseAccessChainOp(OpAsmParser &parser,
OperationState &state) { OperationState &state) {
OpAsmParser::OperandType ptrInfo; OpAsmParser::OperandType ptrInfo;
SmallVector<OpAsmParser::OperandType, 4> indicesInfo; SmallVector<OpAsmParser::OperandType, 4> indicesInfo;
Show All 17 Lines if (!resultType) {
return failure(); return failure();
} }
state.addTypes(resultType); state.addTypes(resultType);
return success(); return success();
} }
static void print(spirv::AccessChainOp op, OpAsmPrinter &printer) { static void print(spirv::AccessChainOp op, OpAsmPrinter &printer) {
printer << spirv::AccessChainOp::getOperationName() << ' ' << *op.base_ptr() printer << spirv::AccessChainOp::getOperationName() << ' ' << op.base_ptr()
<< '[' << op.indices() << "] : " << op.base_ptr()->getType(); << '[' << op.indices() << "] : " << op.base_ptr().getType();
} }
static LogicalResult verify(spirv::AccessChainOp accessChainOp) { static LogicalResult verify(spirv::AccessChainOp accessChainOp) {
SmallVector<Value, 4> indices(accessChainOp.indices().begin(), SmallVector<Value, 4> indices(accessChainOp.indices().begin(),
accessChainOp.indices().end()); accessChainOp.indices().end());
auto resultType = getElementPtrType(accessChainOp.base_ptr()->getType(), auto resultType = getElementPtrType(accessChainOp.base_ptr().getType(),
indices, accessChainOp.getLoc()); indices, accessChainOp.getLoc());
if (!resultType) { if (!resultType) {
return failure(); return failure();
} }
auto providedResultType = auto providedResultType =
accessChainOp.getType().dyn_cast<spirv::PointerType>(); accessChainOp.getType().dyn_cast<spirv::PointerType>();
if (!providedResultType) { if (!providedResultType) {
Show All 16 Lines
/// `spirv::AccessChainOp` operation. /// `spirv::AccessChainOp` operation.
struct CombineChainedAccessChain struct CombineChainedAccessChain
: public OpRewritePattern<spirv::AccessChainOp> { : public OpRewritePattern<spirv::AccessChainOp> {
using OpRewritePattern<spirv::AccessChainOp>::OpRewritePattern; using OpRewritePattern<spirv::AccessChainOp>::OpRewritePattern;
PatternMatchResult matchAndRewrite(spirv::AccessChainOp accessChainOp, PatternMatchResult matchAndRewrite(spirv::AccessChainOp accessChainOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
auto parentAccessChainOp = dyn_cast_or_null<spirv::AccessChainOp>( auto parentAccessChainOp = dyn_cast_or_null<spirv::AccessChainOp>(
accessChainOp.base_ptr()->getDefiningOp()); accessChainOp.base_ptr().getDefiningOp());
if (!parentAccessChainOp) { if (!parentAccessChainOp) {
return matchFailure(); return matchFailure();
} }
// Combine indices. // Combine indices.
SmallVector<Value, 4> indices(parentAccessChainOp.indices()); SmallVector<Value, 4> indices(parentAccessChainOp.indices());
indices.append(accessChainOp.indices().begin(), indices.append(accessChainOp.indices().begin(),
▲ Show 20 Lines • Show All 43 Lines • ▼ Show 20 Linesstatic void print(spirv::AddressOfOp addressOfOp, OpAsmPrinter &printer) {
SmallVector<StringRef, 4> elidedAttrs; SmallVector<StringRef, 4> elidedAttrs;
printer << spirv::AddressOfOp::getOperationName(); printer << spirv::AddressOfOp::getOperationName();
// Print symbol name. // Print symbol name.
printer << ' '; printer << ' ';
printer.printSymbolName(addressOfOp.variable()); printer.printSymbolName(addressOfOp.variable());
// Print the type. // Print the type.
printer << " : " << addressOfOp.pointer()->getType(); printer << " : " << addressOfOp.pointer().getType();
} }
static LogicalResult verify(spirv::AddressOfOp addressOfOp) { static LogicalResult verify(spirv::AddressOfOp addressOfOp) {
auto moduleOp = addressOfOp.getParentOfType<spirv::ModuleOp>(); auto moduleOp = addressOfOp.getParentOfType<spirv::ModuleOp>();
auto varOp = auto varOp =
moduleOp.lookupSymbol<spirv::GlobalVariableOp>(addressOfOp.variable()); moduleOp.lookupSymbol<spirv::GlobalVariableOp>(addressOfOp.variable());
if (!varOp) { if (!varOp) {
return addressOfOp.emitOpError("expected spv.globalVariable symbol"); return addressOfOp.emitOpError("expected spv.globalVariable symbol");
} }
if (addressOfOp.pointer()->getType() != varOp.type()) { if (addressOfOp.pointer().getType() != varOp.type()) {
return addressOfOp.emitOpError( return addressOfOp.emitOpError(
"result type mismatch with the referenced global variable's type"); "result type mismatch with the referenced global variable's type");
} }
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.AtomicCompareExchangeWeak // spv.AtomicCompareExchangeWeak
Show All 31 Lines
} }
static void print(spirv::AtomicCompareExchangeWeakOp atomOp, static void print(spirv::AtomicCompareExchangeWeakOp atomOp,
OpAsmPrinter &printer) { OpAsmPrinter &printer) {
printer << spirv::AtomicCompareExchangeWeakOp::getOperationName() << " \"" printer << spirv::AtomicCompareExchangeWeakOp::getOperationName() << " \""
<< stringifyScope(atomOp.memory_scope()) << "\" \"" << stringifyScope(atomOp.memory_scope()) << "\" \""
<< stringifyMemorySemantics(atomOp.equal_semantics()) << "\" \"" << stringifyMemorySemantics(atomOp.equal_semantics()) << "\" \""
<< stringifyMemorySemantics(atomOp.unequal_semantics()) << "\" " << stringifyMemorySemantics(atomOp.unequal_semantics()) << "\" "
<< atomOp.getOperands() << " : " << atomOp.pointer()->getType(); << atomOp.getOperands() << " : " << atomOp.pointer().getType();
} }
static LogicalResult verify(spirv::AtomicCompareExchangeWeakOp atomOp) { static LogicalResult verify(spirv::AtomicCompareExchangeWeakOp atomOp) {
// According to the spec: // According to the spec:
// "The type of Value must be the same as Result Type. The type of the value // "The type of Value must be the same as Result Type. The type of the value
// pointed to by Pointer must be the same as Result Type. This type must also // pointed to by Pointer must be the same as Result Type. This type must also
// match the type of Comparator." // match the type of Comparator."
if (atomOp.getType() != atomOp.value()->getType()) if (atomOp.getType() != atomOp.value().getType())
return atomOp.emitOpError("value operand must have the same type as the op " return atomOp.emitOpError("value operand must have the same type as the op "
"result, but found ") "result, but found ")
<< atomOp.value()->getType() << " vs " << atomOp.getType(); << atomOp.value().getType() << " vs " << atomOp.getType();
if (atomOp.getType() != atomOp.comparator()->getType()) if (atomOp.getType() != atomOp.comparator().getType())
return atomOp.emitOpError( return atomOp.emitOpError(
"comparator operand must have the same type as the op " "comparator operand must have the same type as the op "
"result, but found ") "result, but found ")
<< atomOp.comparator()->getType() << " vs " << atomOp.getType(); << atomOp.comparator().getType() << " vs " << atomOp.getType();
Type pointeeType = Type pointeeType =
atomOp.pointer()->getType().cast<spirv::PointerType>().getPointeeType(); atomOp.pointer().getType().cast<spirv::PointerType>().getPointeeType();
if (atomOp.getType() != pointeeType) if (atomOp.getType() != pointeeType)
return atomOp.emitOpError( return atomOp.emitOpError(
"pointer operand's pointee type must have the same " "pointer operand's pointee type must have the same "
"as the op result type, but found ") "as the op result type, but found ")
<< pointeeType << " vs " << atomOp.getType(); << pointeeType << " vs " << atomOp.getType();
// TODO(antiagainst): Unequal cannot be set to Release or Acquire and Release. // TODO(antiagainst): Unequal cannot be set to Release or Acquire and Release.
// In addition, Unequal cannot be set to a stronger memory-order then Equal. // In addition, Unequal cannot be set to a stronger memory-order then Equal.
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.BitcastOp // spv.BitcastOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(spirv::BitcastOp bitcastOp) { static LogicalResult verify(spirv::BitcastOp bitcastOp) {
// TODO: The SPIR-V spec validation rules are different for different // TODO: The SPIR-V spec validation rules are different for different
// versions. // versions.
auto operandType = bitcastOp.operand()->getType(); auto operandType = bitcastOp.operand().getType();
auto resultType = bitcastOp.result()->getType(); auto resultType = bitcastOp.result().getType();
if (operandType == resultType) { if (operandType == resultType) {
return bitcastOp.emitError( return bitcastOp.emitError(
"result type must be different from operand type"); "result type must be different from operand type");
} }
if (operandType.isa<spirv::PointerType>() && if (operandType.isa<spirv::PointerType>() &&
!resultType.isa<spirv::PointerType>()) { !resultType.isa<spirv::PointerType>()) {
return bitcastOp.emitError( return bitcastOp.emitError(
"unhandled bit cast conversion from pointer type to non-pointer type"); "unhandled bit cast conversion from pointer type to non-pointer type");
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Linesstatic ParseResult parseBitFieldInsertOp(OpAsmParser &parser,
state.addTypes(baseType); state.addTypes(baseType);
return success(); return success();
} }
static void print(spirv::BitFieldInsertOp bitFieldInsertOp, static void print(spirv::BitFieldInsertOp bitFieldInsertOp,
OpAsmPrinter &printer) { OpAsmPrinter &printer) {
printer << spirv::BitFieldInsertOp::getOperationName() << ' ' printer << spirv::BitFieldInsertOp::getOperationName() << ' '
<< bitFieldInsertOp.getOperands() << " : " << bitFieldInsertOp.getOperands() << " : "
<< bitFieldInsertOp.base()->getType() << ", " << bitFieldInsertOp.base().getType() << ", "
<< bitFieldInsertOp.offset()->getType() << ", " << bitFieldInsertOp.offset().getType() << ", "
<< bitFieldInsertOp.count()->getType(); << bitFieldInsertOp.count().getType();
} }
static LogicalResult verify(spirv::BitFieldInsertOp bitFieldOp) { static LogicalResult verify(spirv::BitFieldInsertOp bitFieldOp) {
auto baseType = bitFieldOp.base()->getType(); auto baseType = bitFieldOp.base().getType();
auto insertType = bitFieldOp.insert()->getType(); auto insertType = bitFieldOp.insert().getType();
auto resultType = bitFieldOp.getResult()->getType(); auto resultType = bitFieldOp.getResult().getType();
if ((baseType != insertType) || (baseType != resultType)) { if ((baseType != insertType) || (baseType != resultType)) {
return bitFieldOp.emitError("expected the same type for the base operand, " return bitFieldOp.emitError("expected the same type for the base operand, "
"insert operand, and " "insert operand, and "
"result, but provided ") "result, but provided ")
<< baseType << ", " << insertType << " and " << resultType; << baseType << ", " << insertType << " and " << resultType;
} }
return success(); return success();
▲ Show 20 Lines • Show All 148 Lines • ▼ Show 20 Linesstatic ParseResult parseCompositeConstructOp(OpAsmParser &parser,
state.addTypes(type); state.addTypes(type);
return parser.resolveOperands(operands, elementTypes, loc, state.operands); return parser.resolveOperands(operands, elementTypes, loc, state.operands);
} }
static void print(spirv::CompositeConstructOp compositeConstructOp, static void print(spirv::CompositeConstructOp compositeConstructOp,
OpAsmPrinter &printer) { OpAsmPrinter &printer) {
printer << spirv::CompositeConstructOp::getOperationName() << " " printer << spirv::CompositeConstructOp::getOperationName() << " "
<< compositeConstructOp.constituents() << " : " << compositeConstructOp.constituents() << " : "
<< compositeConstructOp.getResult()->getType(); << compositeConstructOp.getResult().getType();
} }
static LogicalResult verify(spirv::CompositeConstructOp compositeConstructOp) { static LogicalResult verify(spirv::CompositeConstructOp compositeConstructOp) {
auto cType = compositeConstructOp.getType().cast<spirv::CompositeType>(); auto cType = compositeConstructOp.getType().cast<spirv::CompositeType>();
SmallVector<Value, 4> constituents(compositeConstructOp.constituents()); SmallVector<Value, 4> constituents(compositeConstructOp.constituents());
if (constituents.size() != cType.getNumElements()) { if (constituents.size() != cType.getNumElements()) {
return compositeConstructOp.emitError( return compositeConstructOp.emitError(
"has incorrect number of operands: expected ") "has incorrect number of operands: expected ")
<< cType.getNumElements() << ", but provided " << cType.getNumElements() << ", but provided "
<< constituents.size(); << constituents.size();
} }
for (auto index : llvm::seq<uint32_t>(0, constituents.size())) { for (auto index : llvm::seq<uint32_t>(0, constituents.size())) {
if (constituents[index]->getType() != cType.getElementType(index)) { if (constituents[index].getType() != cType.getElementType(index)) {
return compositeConstructOp.emitError( return compositeConstructOp.emitError(
"operand type mismatch: expected operand type ") "operand type mismatch: expected operand type ")
<< cType.getElementType(index) << ", but provided " << cType.getElementType(index) << ", but provided "
<< constituents[index]->getType(); << constituents[index].getType();
} }
} }
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.CompositeExtractOp // spv.CompositeExtractOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void spirv::CompositeExtractOp::build(Builder *builder, OperationState &state, void spirv::CompositeExtractOp::build(Builder *builder, OperationState &state,
Value composite, Value composite,
ArrayRef<int32_t> indices) { ArrayRef<int32_t> indices) {
auto indexAttr = builder->getI32ArrayAttr(indices); auto indexAttr = builder->getI32ArrayAttr(indices);
auto elementType = auto elementType =
getElementType(composite->getType(), indexAttr, state.location); getElementType(composite.getType(), indexAttr, state.location);
if (!elementType) { if (!elementType) {
return; return;
} }
build(builder, state, elementType, composite, indexAttr); build(builder, state, elementType, composite, indexAttr);
} }
static ParseResult parseCompositeExtractOp(OpAsmParser &parser, static ParseResult parseCompositeExtractOp(OpAsmParser &parser,
OperationState &state) { OperationState &state) {
Show All 17 Linesstatic ParseResult parseCompositeExtractOp(OpAsmParser &parser,
} }
state.addTypes(resultType); state.addTypes(resultType);
return success(); return success();
} }
static void print(spirv::CompositeExtractOp compositeExtractOp, static void print(spirv::CompositeExtractOp compositeExtractOp,
OpAsmPrinter &printer) { OpAsmPrinter &printer) {
printer << spirv::CompositeExtractOp::getOperationName() << ' ' printer << spirv::CompositeExtractOp::getOperationName() << ' '
<< *compositeExtractOp.composite() << compositeExtractOp.indices() << compositeExtractOp.composite() << compositeExtractOp.indices()
<< " : " << compositeExtractOp.composite()->getType(); << " : " << compositeExtractOp.composite().getType();
} }
static LogicalResult verify(spirv::CompositeExtractOp compExOp) { static LogicalResult verify(spirv::CompositeExtractOp compExOp) {
auto indicesArrayAttr = compExOp.indices().dyn_cast<ArrayAttr>(); auto indicesArrayAttr = compExOp.indices().dyn_cast<ArrayAttr>();
auto resultType = getElementType(compExOp.composite()->getType(), auto resultType = getElementType(compExOp.composite().getType(),
indicesArrayAttr, compExOp.getLoc()); indicesArrayAttr, compExOp.getLoc());
if (!resultType) if (!resultType)
return failure(); return failure();
if (resultType != compExOp.getType()) { if (resultType != compExOp.getType()) {
return compExOp.emitOpError("invalid result type: expected ") return compExOp.emitOpError("invalid result type: expected ")
<< resultType << " but provided " << compExOp.getType(); << resultType << " but provided " << compExOp.getType();
} }
Show All 30 Lines return failure(
parser.resolveOperands(operands, {objectType, compositeType}, loc, parser.resolveOperands(operands, {objectType, compositeType}, loc,
state.operands) || state.operands) ||
parser.addTypesToList(compositeType, state.types)); parser.addTypesToList(compositeType, state.types));
} }
static LogicalResult verify(spirv::CompositeInsertOp compositeInsertOp) { static LogicalResult verify(spirv::CompositeInsertOp compositeInsertOp) {
auto indicesArrayAttr = compositeInsertOp.indices().dyn_cast<ArrayAttr>(); auto indicesArrayAttr = compositeInsertOp.indices().dyn_cast<ArrayAttr>();
auto objectType = auto objectType =
getElementType(compositeInsertOp.composite()->getType(), indicesArrayAttr, getElementType(compositeInsertOp.composite().getType(), indicesArrayAttr,
compositeInsertOp.getLoc()); compositeInsertOp.getLoc());
if (!objectType) if (!objectType)
return failure(); return failure();
if (objectType != compositeInsertOp.object()->getType()) { if (objectType != compositeInsertOp.object().getType()) {
return compositeInsertOp.emitOpError("object operand type should be ") return compositeInsertOp.emitOpError("object operand type should be ")
<< objectType << ", but found " << objectType << ", but found "
<< compositeInsertOp.object()->getType(); << compositeInsertOp.object().getType();
} }
if (compositeInsertOp.composite()->getType() != compositeInsertOp.getType()) { if (compositeInsertOp.composite().getType() != compositeInsertOp.getType()) {
return compositeInsertOp.emitOpError("result type should be the same as " return compositeInsertOp.emitOpError("result type should be the same as "
"the composite type, but found ") "the composite type, but found ")
<< compositeInsertOp.composite()->getType() << " vs " << compositeInsertOp.composite().getType() << " vs "
<< compositeInsertOp.getType(); << compositeInsertOp.getType();
} }
return success(); return success();
} }
static void print(spirv::CompositeInsertOp compositeInsertOp, static void print(spirv::CompositeInsertOp compositeInsertOp,
OpAsmPrinter &printer) { OpAsmPrinter &printer) {
printer << spirv::CompositeInsertOp::getOperationName() << " " printer << spirv::CompositeInsertOp::getOperationName() << " "
<< *compositeInsertOp.object() << ", " << compositeInsertOp.object() << ", " << compositeInsertOp.composite()
<< *compositeInsertOp.composite() << compositeInsertOp.indices() << compositeInsertOp.indices() << " : "
<< " : " << compositeInsertOp.object()->getType() << " into " << compositeInsertOp.object().getType() << " into "
<< compositeInsertOp.composite()->getType(); << compositeInsertOp.composite().getType();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.constant // spv.constant
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static ParseResult parseConstantOp(OpAsmParser &parser, OperationState &state) { static ParseResult parseConstantOp(OpAsmParser &parser, OperationState &state) {
Attribute value; Attribute value;
▲ Show 20 Lines • Show All 342 Lines • ▼ Show 20 Linesstatic LogicalResult verify(spirv::FunctionCallOp functionCallOp) {
if (functionType.getNumInputs() != functionCallOp.getNumOperands()) { if (functionType.getNumInputs() != functionCallOp.getNumOperands()) {
return functionCallOp.emitOpError( return functionCallOp.emitOpError(
"has incorrect number of operands for callee: expected ") "has incorrect number of operands for callee: expected ")
<< functionType.getNumInputs() << ", but provided " << functionType.getNumInputs() << ", but provided "
<< functionCallOp.getNumOperands(); << functionCallOp.getNumOperands();
} }
for (uint32_t i = 0, e = functionType.getNumInputs(); i != e; ++i) { for (uint32_t i = 0, e = functionType.getNumInputs(); i != e; ++i) {
if (functionCallOp.getOperand(i)->getType() != functionType.getInput(i)) { if (functionCallOp.getOperand(i).getType() != functionType.getInput(i)) {
return functionCallOp.emitOpError( return functionCallOp.emitOpError(
"operand type mismatch: expected operand type ") "operand type mismatch: expected operand type ")
<< functionType.getInput(i) << ", but provided " << functionType.getInput(i) << ", but provided "
<< functionCallOp.getOperand(i)->getType() << functionCallOp.getOperand(i).getType() << " for operand number "
<< " for operand number " << i; << i;
} }
} }
if (functionType.getNumResults() != functionCallOp.getNumResults()) { if (functionType.getNumResults() != functionCallOp.getNumResults()) {
return functionCallOp.emitOpError( return functionCallOp.emitOpError(
"has incorrect number of results has for callee: expected ") "has incorrect number of results has for callee: expected ")
<< functionType.getNumResults() << ", but provided " << functionType.getNumResults() << ", but provided "
<< functionCallOp.getNumResults(); << functionCallOp.getNumResults();
} }
if (functionCallOp.getNumResults() && if (functionCallOp.getNumResults() &&
(functionCallOp.getResult(0)->getType() != functionType.getResult(0))) { (functionCallOp.getResult(0).getType() != functionType.getResult(0))) {
return functionCallOp.emitOpError("result type mismatch: expected ") return functionCallOp.emitOpError("result type mismatch: expected ")
<< functionType.getResult(0) << ", but provided " << functionType.getResult(0) << ", but provided "
<< functionCallOp.getResult(0)->getType(); << functionCallOp.getResult(0).getType();
} }
return success(); return success();
} }
CallInterfaceCallable spirv::FunctionCallOp::getCallableForCallee() { CallInterfaceCallable spirv::FunctionCallOp::getCallableForCallee() {
return getAttrOfType<SymbolRefAttr>(kCallee); return getAttrOfType<SymbolRefAttr>(kCallee);
} }
▲ Show 20 Lines • Show All 211 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.LoadOp // spv.LoadOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void spirv::LoadOp::build(Builder *builder, OperationState &state, void spirv::LoadOp::build(Builder *builder, OperationState &state,
Value basePtr, IntegerAttr memory_access, Value basePtr, IntegerAttr memory_access,
IntegerAttr alignment) { IntegerAttr alignment) {
auto ptrType = basePtr->getType().cast<spirv::PointerType>(); auto ptrType = basePtr.getType().cast<spirv::PointerType>();
build(builder, state, ptrType.getPointeeType(), basePtr, memory_access, build(builder, state, ptrType.getPointeeType(), basePtr, memory_access,
alignment); alignment);
} }
static ParseResult parseLoadOp(OpAsmParser &parser, OperationState &state) { static ParseResult parseLoadOp(OpAsmParser &parser, OperationState &state) {
// Parse the storage class specification // Parse the storage class specification
spirv::StorageClass storageClass; spirv::StorageClass storageClass;
OpAsmParser::OperandType ptrInfo; OpAsmParser::OperandType ptrInfo;
Show All 14 Linesstatic ParseResult parseLoadOp(OpAsmParser &parser, OperationState &state) {
state.addTypes(elementType); state.addTypes(elementType);
return success(); return success();
} }
static void print(spirv::LoadOp loadOp, OpAsmPrinter &printer) { static void print(spirv::LoadOp loadOp, OpAsmPrinter &printer) {
auto *op = loadOp.getOperation(); auto *op = loadOp.getOperation();
SmallVector<StringRef, 4> elidedAttrs; SmallVector<StringRef, 4> elidedAttrs;
StringRef sc = stringifyStorageClass( StringRef sc = stringifyStorageClass(
loadOp.ptr()->getType().cast<spirv::PointerType>().getStorageClass()); loadOp.ptr().getType().cast<spirv::PointerType>().getStorageClass());
printer << spirv::LoadOp::getOperationName() << " \"" << sc << "\" " printer << spirv::LoadOp::getOperationName() << " \"" << sc << "\" "
<< loadOp.ptr(); << loadOp.ptr();
printMemoryAccessAttribute(loadOp, printer, elidedAttrs); printMemoryAccessAttribute(loadOp, printer, elidedAttrs);
printer.printOptionalAttrDict(op->getAttrs(), elidedAttrs); printer.printOptionalAttrDict(op->getAttrs(), elidedAttrs);
printer << " : " << loadOp.getType(); printer << " : " << loadOp.getType();
} }
▲ Show 20 Lines • Show All 411 Lines • ▼ Show 20 Lines if (parser.parseAttribute(constRefAttr, Type(), kSpecConstAttrName,
return failure(); return failure();
} }
return parser.addTypeToList(type, state.types); return parser.addTypeToList(type, state.types);
} }
static void print(spirv::ReferenceOfOp referenceOfOp, OpAsmPrinter &printer) { static void print(spirv::ReferenceOfOp referenceOfOp, OpAsmPrinter &printer) {
printer << spirv::ReferenceOfOp::getOperationName() << ' '; printer << spirv::ReferenceOfOp::getOperationName() << ' ';
printer.printSymbolName(referenceOfOp.spec_const()); printer.printSymbolName(referenceOfOp.spec_const());
printer << " : " << referenceOfOp.reference()->getType(); printer << " : " << referenceOfOp.reference().getType();
} }
static LogicalResult verify(spirv::ReferenceOfOp referenceOfOp) { static LogicalResult verify(spirv::ReferenceOfOp referenceOfOp) {
auto moduleOp = referenceOfOp.getParentOfType<spirv::ModuleOp>(); auto moduleOp = referenceOfOp.getParentOfType<spirv::ModuleOp>();
auto specConstOp = auto specConstOp =
moduleOp.lookupSymbol<spirv::SpecConstantOp>(referenceOfOp.spec_const()); moduleOp.lookupSymbol<spirv::SpecConstantOp>(referenceOfOp.spec_const());
if (!specConstOp) { if (!specConstOp) {
return referenceOfOp.emitOpError("expected spv.specConstant symbol"); return referenceOfOp.emitOpError("expected spv.specConstant symbol");
} }
if (referenceOfOp.reference()->getType() != if (referenceOfOp.reference().getType() !=
specConstOp.default_value().getType()) { specConstOp.default_value().getType()) {
return referenceOfOp.emitOpError("result type mismatch with the referenced " return referenceOfOp.emitOpError("result type mismatch with the referenced "
"specialization constant's type"); "specialization constant's type");
} }
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 20 Linesstatic ParseResult parseReturnValueOp(OpAsmParser &parser,
Type retValType; Type retValType;
return failure(parser.parseOperand(retValInfo) || return failure(parser.parseOperand(retValInfo) ||
parser.parseColonType(retValType) || parser.parseColonType(retValType) ||
parser.resolveOperand(retValInfo, retValType, state.operands)); parser.resolveOperand(retValInfo, retValType, state.operands));
} }
static void print(spirv::ReturnValueOp retValOp, OpAsmPrinter &printer) { static void print(spirv::ReturnValueOp retValOp, OpAsmPrinter &printer) {
printer << spirv::ReturnValueOp::getOperationName() << ' ' << retValOp.value() printer << spirv::ReturnValueOp::getOperationName() << ' ' << retValOp.value()
<< " : " << retValOp.value()->getType(); << " : " << retValOp.value().getType();
} }
static LogicalResult verify(spirv::ReturnValueOp retValOp) { static LogicalResult verify(spirv::ReturnValueOp retValOp) {
auto funcOp = retValOp.getParentOfType<FuncOp>(); auto funcOp = retValOp.getParentOfType<FuncOp>();
auto numFnResults = funcOp.getType().getNumResults(); auto numFnResults = funcOp.getType().getNumResults();
if (numFnResults != 1) if (numFnResults != 1)
return retValOp.emitOpError( return retValOp.emitOpError(
"returns 1 value but enclosing function requires ") "returns 1 value but enclosing function requires ")
<< numFnResults << " results"; << numFnResults << " results";
auto operandType = retValOp.value()->getType(); auto operandType = retValOp.value().getType();
auto fnResultType = funcOp.getType().getResult(0); auto fnResultType = funcOp.getType().getResult(0);
if (operandType != fnResultType) if (operandType != fnResultType)
return retValOp.emitOpError(" return value's type (") return retValOp.emitOpError(" return value's type (")
<< operandType << ") mismatch with function's result type (" << operandType << ") mismatch with function's result type ("
<< fnResultType << ")"; << fnResultType << ")";
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.Select // spv.Select
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void spirv::SelectOp::build(Builder *builder, OperationState &state, Value cond, void spirv::SelectOp::build(Builder *builder, OperationState &state, Value cond,
Value trueValue, Value falseValue) { Value trueValue, Value falseValue) {
build(builder, state, trueValue->getType(), cond, trueValue, falseValue); build(builder, state, trueValue.getType(), cond, trueValue, falseValue);
} }
static ParseResult parseSelectOp(OpAsmParser &parser, OperationState &state) { static ParseResult parseSelectOp(OpAsmParser &parser, OperationState &state) {
OpAsmParser::OperandType condition; OpAsmParser::OperandType condition;
SmallVector<OpAsmParser::OperandType, 2> operands; SmallVector<OpAsmParser::OperandType, 2> operands;
SmallVector<Type, 2> types; SmallVector<Type, 2> types;
auto loc = parser.getCurrentLocation(); auto loc = parser.getCurrentLocation();
if (parser.parseOperand(condition) || parser.parseComma() || if (parser.parseOperand(condition) || parser.parseComma() ||
Show All 9 Lines if (parser.resolveOperand(condition, types[0], state.operands) ||
parser.resolveOperands(operands, types[1], state.operands)) { parser.resolveOperands(operands, types[1], state.operands)) {
return failure(); return failure();
} }
return parser.addTypesToList(types[1], state.types); return parser.addTypesToList(types[1], state.types);
} }
static void print(spirv::SelectOp op, OpAsmPrinter &printer) { static void print(spirv::SelectOp op, OpAsmPrinter &printer) {
printer << spirv::SelectOp::getOperationName() << " " << op.getOperands() printer << spirv::SelectOp::getOperationName() << " " << op.getOperands()
<< " : " << op.condition()->getType() << ", " << " : " << op.condition().getType() << ", " << op.result().getType();
<< op.result()->getType();
} }
static LogicalResult verify(spirv::SelectOp op) { static LogicalResult verify(spirv::SelectOp op) {
auto resultTy = op.result()->getType(); auto resultTy = op.result().getType();
if (op.true_value()->getType() != resultTy) { if (op.true_value().getType() != resultTy) {
return op.emitOpError("result type and true value type must be the same"); return op.emitOpError("result type and true value type must be the same");
} }
if (op.false_value()->getType() != resultTy) { if (op.false_value().getType() != resultTy) {
return op.emitOpError("result type and false value type must be the same"); return op.emitOpError("result type and false value type must be the same");
} }
if (auto conditionTy = op.condition()->getType().dyn_cast<VectorType>()) { if (auto conditionTy = op.condition().getType().dyn_cast<VectorType>()) {
auto resultVectorTy = resultTy.dyn_cast<VectorType>(); auto resultVectorTy = resultTy.dyn_cast<VectorType>();
if (!resultVectorTy) { if (!resultVectorTy) {
return op.emitOpError("result expected to be of vector type when " return op.emitOpError("result expected to be of vector type when "
"condition is of vector type"); "condition is of vector type");
} }
if (resultVectorTy.getNumElements() != conditionTy.getNumElements()) { if (resultVectorTy.getNumElements() != conditionTy.getNumElements()) {
return op.emitOpError("result should have the same number of elements as " return op.emitOpError("result should have the same number of elements as "
"the condition when condition is of vector type"); "the condition when condition is of vector type");
▲ Show 20 Lines • Show All 152 Lines • ▼ Show 20 Lines PatternMatchResult matchAndRewrite(spirv::SelectionOp selectionOp,
auto trueValue = getSrcValue(trueBlock); auto trueValue = getSrcValue(trueBlock);
auto falseValue = getSrcValue(falseBlock); auto falseValue = getSrcValue(falseBlock);
auto ptrValue = getDstPtr(trueBlock); auto ptrValue = getDstPtr(trueBlock);
auto storeOpAttributes = auto storeOpAttributes =
cast<spirv::StoreOp>(trueBlock->front()).getOperation()->getAttrs(); cast<spirv::StoreOp>(trueBlock->front()).getOperation()->getAttrs();
auto selectOp = rewriter.create<spirv::SelectOp>( auto selectOp = rewriter.create<spirv::SelectOp>(
selectionOp.getLoc(), trueValue->getType(), brConditionalOp.condition(), selectionOp.getLoc(), trueValue.getType(), brConditionalOp.condition(),
trueValue, falseValue); trueValue, falseValue);
rewriter.create<spirv::StoreOp>(selectOp.getLoc(), ptrValue, rewriter.create<spirv::StoreOp>(selectOp.getLoc(), ptrValue,
selectOp.getResult(), storeOpAttributes); selectOp.getResult(), storeOpAttributes);
// `spv.selection` is not needed anymore. // `spv.selection` is not needed anymore.
rewriter.eraseOp(op); rewriter.eraseOp(op);
return matchSuccess(); return matchSuccess();
} }
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines if (!trueBrStoreOp || !trueBrBranchOp || !falseBrStoreOp ||
!falseBrBranchOp) { !falseBrBranchOp) {
return matchFailure(); return matchFailure();
} }
// Check that each `spv.Store` uses the same pointer, memory access // Check that each `spv.Store` uses the same pointer, memory access
// attributes and a valid type of the value. // attributes and a valid type of the value.
if ((trueBrStoreOp.ptr() != falseBrStoreOp.ptr()) || if ((trueBrStoreOp.ptr() != falseBrStoreOp.ptr()) ||
!isSameAttrList(trueBrStoreOp, falseBrStoreOp) || !isSameAttrList(trueBrStoreOp, falseBrStoreOp) ||
!isValidType(trueBrStoreOp.value()->getType())) { !isValidType(trueBrStoreOp.value().getType())) {
return matchFailure(); return matchFailure();
} }
if ((trueBrBranchOp.getOperation()->getSuccessor(0) != mergeBlock) || if ((trueBrBranchOp.getOperation()->getSuccessor(0) != mergeBlock) ||
(falseBrBranchOp.getOperation()->getSuccessor(0) != mergeBlock)) { (falseBrBranchOp.getOperation()->getSuccessor(0) != mergeBlock)) {
return matchFailure(); return matchFailure();
} }
▲ Show 20 Lines • Show All 89 Lines • ▼ Show 20 Linesstatic ParseResult parseStoreOp(OpAsmParser &parser, OperationState &state) {
} }
return success(); return success();
} }
static void print(spirv::StoreOp storeOp, OpAsmPrinter &printer) { static void print(spirv::StoreOp storeOp, OpAsmPrinter &printer) {
auto *op = storeOp.getOperation(); auto *op = storeOp.getOperation();
SmallVector<StringRef, 4> elidedAttrs; SmallVector<StringRef, 4> elidedAttrs;
StringRef sc = stringifyStorageClass( StringRef sc = stringifyStorageClass(
storeOp.ptr()->getType().cast<spirv::PointerType>().getStorageClass()); storeOp.ptr().getType().cast<spirv::PointerType>().getStorageClass());
printer << spirv::StoreOp::getOperationName() << " \"" << sc << "\" " printer << spirv::StoreOp::getOperationName() << " \"" << sc << "\" "
<< storeOp.ptr() << ", " << storeOp.value(); << storeOp.ptr() << ", " << storeOp.value();
printMemoryAccessAttribute(storeOp, printer, elidedAttrs); printMemoryAccessAttribute(storeOp, printer, elidedAttrs);
printer << " : " << storeOp.value()->getType(); printer << " : " << storeOp.value().getType();
printer.printOptionalAttrDict(op->getAttrs(), elidedAttrs); printer.printOptionalAttrDict(op->getAttrs(), elidedAttrs);
} }
static LogicalResult verify(spirv::StoreOp storeOp) { static LogicalResult verify(spirv::StoreOp storeOp) {
// SPIR-V spec : "Pointer is the pointer to store through. Its type must be an // SPIR-V spec : "Pointer is the pointer to store through. Its type must be an
// OpTypePointer whose Type operand is the same as the type of Object." // OpTypePointer whose Type operand is the same as the type of Object."
if (failed(verifyLoadStorePtrAndValTypes(storeOp, storeOp.ptr(), if (failed(verifyLoadStorePtrAndValTypes(storeOp, storeOp.ptr(),
storeOp.value()))) { storeOp.value()))) {
▲ Show 20 Lines • Show All 123 Lines • ▼ Show 20 Linesstatic LogicalResult verify(spirv::VariableOp varOp) {
// object. It cannot be Generic. It must be the same as the Storage Class // object. It cannot be Generic. It must be the same as the Storage Class
// operand of the Result Type." // operand of the Result Type."
if (varOp.storage_class() != spirv::StorageClass::Function) { if (varOp.storage_class() != spirv::StorageClass::Function) {
return varOp.emitOpError( return varOp.emitOpError(
"can only be used to model function-level variables. Use " "can only be used to model function-level variables. Use "
"spv.globalVariable for module-level variables."); "spv.globalVariable for module-level variables.");
} }
auto pointerType = varOp.pointer()->getType().cast<spirv::PointerType>(); auto pointerType = varOp.pointer().getType().cast<spirv::PointerType>();
if (varOp.storage_class() != pointerType.getStorageClass()) if (varOp.storage_class() != pointerType.getStorageClass())
return varOp.emitOpError( return varOp.emitOpError(
"storage class must match result pointer's storage class"); "storage class must match result pointer's storage class");
if (varOp.getNumOperands() != 0) { if (varOp.getNumOperands() != 0) {
// SPIR-V spec: "Initializer must be an <id> from a constant instruction or // SPIR-V spec: "Initializer must be an <id> from a constant instruction or
// a global (module scope) OpVariable instruction". // a global (module scope) OpVariable instruction".
auto *initOp = varOp.getOperand(0)->getDefiningOp(); auto *initOp = varOp.getOperand(0).getDefiningOp();
if (!initOp || !(isa<spirv::ConstantOp>(initOp) || // for normal constant if (!initOp || !(isa<spirv::ConstantOp>(initOp) || // for normal constant
isa<spirv::ReferenceOfOp>(initOp) || // for spec constant isa<spirv::ReferenceOfOp>(initOp) || // for spec constant
isa<spirv::AddressOfOp>(initOp))) isa<spirv::AddressOfOp>(initOp)))
return varOp.emitOpError("initializer must be the result of a " return varOp.emitOpError("initializer must be the result of a "
"constant or spv.globalVariable op"); "constant or spv.globalVariable op");
} }
// TODO(antiagainst): generate these strings using ODS. // TODO(antiagainst): generate these strings using ODS.
Show All 33 Lines

mlir/lib/Dialect/SPIRV/Serialization/Deserializer.cpp

Show First 20 Lines • Show All 1,769 Lines • ▼ Show 20 Lines for (auto *block : constructBlocks) {
// Create a block and insert it before the selection/loop merge block in the // Create a block and insert it before the selection/loop merge block in the
// SelectionOp/LoopOp's region. // SelectionOp/LoopOp's region.
auto *newBlock = builder.createBlock(&body.back()); auto *newBlock = builder.createBlock(&body.back());
mapper.map(block, newBlock); mapper.map(block, newBlock);
LLVM_DEBUG(llvm::dbgs() << "[cf] cloned block " << newBlock LLVM_DEBUG(llvm::dbgs() << "[cf] cloned block " << newBlock
<< " from block " << block << "\n"); << " from block " << block << "\n");
if (!isFnEntryBlock(block)) { if (!isFnEntryBlock(block)) {
for (BlockArgument blockArg : block->getArguments()) { for (BlockArgument blockArg : block->getArguments()) {
auto newArg = newBlock->addArgument(blockArg->getType()); auto newArg = newBlock->addArgument(blockArg.getType());
mapper.map(blockArg, newArg); mapper.map(blockArg, newArg);
LLVM_DEBUG(llvm::dbgs() << "[cf] remapped block argument " << blockArg LLVM_DEBUG(llvm::dbgs() << "[cf] remapped block argument " << blockArg
<< " to " << newArg << '\n'); << " to " << newArg << '\n');
} }
} else { } else {
LLVM_DEBUG(llvm::dbgs() LLVM_DEBUG(llvm::dbgs()
<< "[cf] block " << block << " is a function entry block\n"); << "[cf] block " << block << " is a function entry block\n");
} }
Show All 24 LinesLogicalResult ControlFlowStructurizer::structurizeImpl() {
headerBlock->replaceAllUsesWith(mergeBlock); headerBlock->replaceAllUsesWith(mergeBlock);
if (isLoop) { if (isLoop) {
// The loop selection/loop header block may have block arguments. Since now // The loop selection/loop header block may have block arguments. Since now
// we place the selection/loop op inside the old merge block, we need to // we place the selection/loop op inside the old merge block, we need to
// make sure the old merge block has the same block argument list. // make sure the old merge block has the same block argument list.
assert(mergeBlock->args_empty() && "OpPhi in loop merge block unsupported"); assert(mergeBlock->args_empty() && "OpPhi in loop merge block unsupported");
for (BlockArgument blockArg : headerBlock->getArguments()) { for (BlockArgument blockArg : headerBlock->getArguments()) {
mergeBlock->addArgument(blockArg->getType()); mergeBlock->addArgument(blockArg.getType());
} }
// If the loop header block has block arguments, make sure the spv.branch op // If the loop header block has block arguments, make sure the spv.branch op
// matches. // matches.
SmallVector<Value, 4> blockArgs; SmallVector<Value, 4> blockArgs;
if (!headerBlock->args_empty()) if (!headerBlock->args_empty())
blockArgs = {mergeBlock->args_begin(), mergeBlock->args_end()}; blockArgs = {mergeBlock->args_begin(), mergeBlock->args_end()};
▲ Show 20 Lines • Show All 367 Lines • ▼ Show 20 Lines if (wordIndex < words.size()) {
wordIndex++; wordIndex++;
} }
if (wordIndex != words.size()) { if (wordIndex != words.size()) {
return emitError(unknownLoc, return emitError(unknownLoc,
"found more operands than expected when deserializing " "found more operands than expected when deserializing "
"spirv::BitcastOp, only ") "spirv::BitcastOp, only ")
<< wordIndex << " of " << words.size() << " processed"; << wordIndex << " of " << words.size() << " processed";
} }
if (resultTypes[0] == operands[0]->getType() && if (resultTypes[0] == operands[0].getType() &&
resultTypes[0].isa<IntegerType>()) { resultTypes[0].isa<IntegerType>()) {
// TODO(b/130356985): This check is added to ignore error in Op verification // TODO(b/130356985): This check is added to ignore error in Op verification
// due to both signed and unsigned integers mapping to the same // due to both signed and unsigned integers mapping to the same
// type. Without this check this method is same as what is auto-generated. // type. Without this check this method is same as what is auto-generated.
valueMap[valueID] = operands[0]; valueMap[valueID] = operands[0];
return success(); return success();
} }
▲ Show 20 Lines • Show All 212 Lines • Show Last 20 Lines

mlir/lib/Dialect/SPIRV/Serialization/Serializer.cpp

Show First 20 Lines • Show All 501 Lines • ▼ Show 20 Lines
} }
void Serializer::printValueIDMap(raw_ostream &os) { void Serializer::printValueIDMap(raw_ostream &os) {
os << "\n= Value <id> Map =\n\n"; os << "\n= Value <id> Map =\n\n";
for (auto valueIDPair : valueIDMap) { for (auto valueIDPair : valueIDMap) {
Value val = valueIDPair.first; Value val = valueIDPair.first;
os << " " << val << " " os << " " << val << " "
<< "id = " << valueIDPair.second << ' '; << "id = " << valueIDPair.second << ' ';
if (auto *op = val->getDefiningOp()) { if (auto *op = val.getDefiningOp()) {
os << "from op '" << op->getName() << "'"; os << "from op '" << op->getName() << "'";
} else if (auto arg = val.dyn_cast<BlockArgument>()) { } else if (auto arg = val.dyn_cast<BlockArgument>()) {
Block *block = arg->getOwner(); Block *block = arg.getOwner();
os << "from argument of block " << block << ' '; os << "from argument of block " << block << ' ';
os << " in op '" << block->getParentOp()->getName() << "'"; os << " in op '" << block->getParentOp()->getName() << "'";
} }
os << '\n'; os << '\n';
} }
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 187 Lines • ▼ Show 20 LinesLogicalResult Serializer::processFuncOp(FuncOp op) {
// Add function name. // Add function name.
if (failed(processName(funcID, op.getName()))) { if (failed(processName(funcID, op.getName()))) {
return failure(); return failure();
} }
// Declare the parameters. // Declare the parameters.
for (auto arg : op.getArguments()) { for (auto arg : op.getArguments()) {
uint32_t argTypeID = 0; uint32_t argTypeID = 0;
if (failed(processType(op.getLoc(), arg->getType(), argTypeID))) { if (failed(processType(op.getLoc(), arg.getType(), argTypeID))) {
return failure(); return failure();
} }
auto argValueID = getNextID(); auto argValueID = getNextID();
valueIDMap[arg] = argValueID; valueIDMap[arg] = argValueID;
encodeInstructionInto(functionHeader, spirv::Opcode::OpFunctionParameter, encodeInstructionInto(functionHeader, spirv::Opcode::OpFunctionParameter,
{argTypeID, argValueID}); {argTypeID, argValueID});
} }
▲ Show 20 Lines • Show All 666 Lines • ▼ Show 20 LinesLogicalResult Serializer::emitPhiForBlockArguments(Block *block) {
} }
// Then create OpPhi instruction for each of the block argument. // Then create OpPhi instruction for each of the block argument.
for (auto argIndex : llvm::seq<unsigned>(0, block->getNumArguments())) { for (auto argIndex : llvm::seq<unsigned>(0, block->getNumArguments())) {
BlockArgument arg = block->getArgument(argIndex); BlockArgument arg = block->getArgument(argIndex);
// Get the type <id> and result <id> for this OpPhi instruction. // Get the type <id> and result <id> for this OpPhi instruction.
uint32_t phiTypeID = 0; uint32_t phiTypeID = 0;
if (failed(processType(arg->getLoc(), arg->getType(), phiTypeID))) if (failed(processType(arg.getLoc(), arg.getType(), phiTypeID)))
return failure(); return failure();
uint32_t phiID = getNextID(); uint32_t phiID = getNextID();
LLVM_DEBUG(llvm::dbgs() << "[phi] for block argument #" << argIndex << ' ' LLVM_DEBUG(llvm::dbgs() << "[phi] for block argument #" << argIndex << ' '
<< arg << " (id = " << phiID << ")\n"); << arg << " (id = " << phiID << ")\n");
SmallVector<uint32_t, 8> phiArgs; SmallVector<uint32_t, 8> phiArgs;
phiArgs.push_back(phiTypeID); phiArgs.push_back(phiTypeID);
▲ Show 20 Lines • Show All 413 Lines • Show Last 20 Lines

mlir/lib/Dialect/SPIRV/Transforms/DecorateSPIRVCompositeTypeLayoutPass.cpp

Show First 20 Lines • Show All 94 Lines • ▼ Show 20 Linesvoid DecorateSPIRVCompositeTypeLayoutPass::runOnModule() {
target.addLegalOp<FuncOp>(); target.addLegalOp<FuncOp>();
target.addDynamicallyLegalOp<spirv::GlobalVariableOp>( target.addDynamicallyLegalOp<spirv::GlobalVariableOp>(
[](spirv::GlobalVariableOp op) { [](spirv::GlobalVariableOp op) {
return VulkanLayoutUtils::isLegalType(op.type()); return VulkanLayoutUtils::isLegalType(op.type());
}); });
// Change the type for the direct users. // Change the type for the direct users.
target.addDynamicallyLegalOp<spirv::AddressOfOp>([](spirv::AddressOfOp op) { target.addDynamicallyLegalOp<spirv::AddressOfOp>([](spirv::AddressOfOp op) {
return VulkanLayoutUtils::isLegalType(op.pointer()->getType()); return VulkanLayoutUtils::isLegalType(op.pointer().getType());
}); });
// TODO: Change the type for the indirect users such as spv.Load, spv.Store, // TODO: Change the type for the indirect users such as spv.Load, spv.Store,
// spv.FunctionCall and so on. // spv.FunctionCall and so on.
for (auto spirvModule : module.getOps<spirv::ModuleOp>()) { for (auto spirvModule : module.getOps<spirv::ModuleOp>()) {
if (failed(applyFullConversion(spirvModule, target, patterns))) { if (failed(applyFullConversion(spirvModule, target, patterns))) {
signalPassFailure(); signalPassFailure();
Show All 12 Lines

mlir/lib/Dialect/StandardOps/Ops.cpp

Show First 20 Lines • Show All 73 Lines • ▼ Show 20 Linesstruct StdInlinerInterface : public DialectInlinerInterface {
void handleTerminator(Operation *op, void handleTerminator(Operation *op,
ArrayRef<Value> valuesToRepl) const final { ArrayRef<Value> valuesToRepl) const final {
// Only "std.return" needs to be handled here. // Only "std.return" needs to be handled here.
auto returnOp = cast<ReturnOp>(op); auto returnOp = cast<ReturnOp>(op);
// Replace the values directly with the return operands. // Replace the values directly with the return operands.
assert(returnOp.getNumOperands() == valuesToRepl.size()); assert(returnOp.getNumOperands() == valuesToRepl.size());
for (const auto &it : llvm::enumerate(returnOp.getOperands())) for (const auto &it : llvm::enumerate(returnOp.getOperands()))
valuesToRepl[it.index()]->replaceAllUsesWith(it.value()); valuesToRepl[it.index()].replaceAllUsesWith(it.value());
} }
}; };
} // end anonymous namespace } // end anonymous namespace
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// StandardOpsDialect // StandardOpsDialect
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// A custom unary operation printer that omits the "std." prefix from the /// A custom unary operation printer that omits the "std." prefix from the
/// operation names. /// operation names.
static void printStandardUnaryOp(Operation *op, OpAsmPrinter &p) { static void printStandardUnaryOp(Operation *op, OpAsmPrinter &p) {
assert(op->getNumOperands() == 1 && "unary op should have one operand"); assert(op->getNumOperands() == 1 && "unary op should have one operand");
assert(op->getNumResults() == 1 && "unary op should have one result"); assert(op->getNumResults() == 1 && "unary op should have one result");
int stdDotLen = StandardOpsDialect::getDialectNamespace().size() + 1; int stdDotLen = StandardOpsDialect::getDialectNamespace().size() + 1;
p << op->getName().getStringRef().drop_front(stdDotLen) << ' ' p << op->getName().getStringRef().drop_front(stdDotLen) << ' '
<< *op->getOperand(0); << op->getOperand(0);
p.printOptionalAttrDict(op->getAttrs()); p.printOptionalAttrDict(op->getAttrs());
p << " : " << op->getOperand(0)->getType(); p << " : " << op->getOperand(0).getType();
} }
/// A custom binary operation printer that omits the "std." prefix from the /// A custom binary operation printer that omits the "std." prefix from the
/// operation names. /// operation names.
static void printStandardBinaryOp(Operation *op, OpAsmPrinter &p) { static void printStandardBinaryOp(Operation *op, OpAsmPrinter &p) {
assert(op->getNumOperands() == 2 && "binary op should have two operands"); assert(op->getNumOperands() == 2 && "binary op should have two operands");
assert(op->getNumResults() == 1 && "binary op should have one result"); assert(op->getNumResults() == 1 && "binary op should have one result");
// If not all the operand and result types are the same, just use the // If not all the operand and result types are the same, just use the
// generic assembly form to avoid omitting information in printing. // generic assembly form to avoid omitting information in printing.
auto resultType = op->getResult(0)->getType(); auto resultType = op->getResult(0).getType();
if (op->getOperand(0)->getType() != resultType || if (op->getOperand(0).getType() != resultType ||
op->getOperand(1)->getType() != resultType) { op->getOperand(1).getType() != resultType) {
p.printGenericOp(op); p.printGenericOp(op);
return; return;
} }
int stdDotLen = StandardOpsDialect::getDialectNamespace().size() + 1; int stdDotLen = StandardOpsDialect::getDialectNamespace().size() + 1;
p << op->getName().getStringRef().drop_front(stdDotLen) << ' ' p << op->getName().getStringRef().drop_front(stdDotLen) << ' '
<< *op->getOperand(0) << ", " << *op->getOperand(1); << op->getOperand(0) << ", " << op->getOperand(1);
p.printOptionalAttrDict(op->getAttrs()); p.printOptionalAttrDict(op->getAttrs());
// Now we can output only one type for all operands and the result. // Now we can output only one type for all operands and the result.
p << " : " << op->getResult(0)->getType(); p << " : " << op->getResult(0).getType();
} }
/// A custom cast operation printer that omits the "std." prefix from the /// A custom cast operation printer that omits the "std." prefix from the
/// operation names. /// operation names.
static void printStandardCastOp(Operation *op, OpAsmPrinter &p) { static void printStandardCastOp(Operation *op, OpAsmPrinter &p) {
int stdDotLen = StandardOpsDialect::getDialectNamespace().size() + 1; int stdDotLen = StandardOpsDialect::getDialectNamespace().size() + 1;
p << op->getName().getStringRef().drop_front(stdDotLen) << ' ' p << op->getName().getStringRef().drop_front(stdDotLen) << ' '
<< *op->getOperand(0) << " : " << op->getOperand(0)->getType() << " to " << op->getOperand(0) << " : " << op->getOperand(0).getType() << " to "
<< op->getResult(0)->getType(); << op->getResult(0).getType();
} }
/// A custom cast operation verifier. /// A custom cast operation verifier.
template <typename T> static LogicalResult verifyCastOp(T op) { template <typename T> static LogicalResult verifyCastOp(T op) {
auto opType = op.getOperand()->getType(); auto opType = op.getOperand().getType();
auto resType = op.getType(); auto resType = op.getType();
if (!T::areCastCompatible(opType, resType)) if (!T::areCastCompatible(opType, resType))
return op.emitError("operand type ") << opType << " and result type " return op.emitError("operand type ") << opType << " and result type "
<< resType << " are cast incompatible"; << resType << " are cast incompatible";
return success(); return success();
} }
▲ Show 20 Lines • Show All 56 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// This is a common class used for patterns of the form /// This is a common class used for patterns of the form
/// "someop(memrefcast) -> someop". It folds the source of any memref_cast /// "someop(memrefcast) -> someop". It folds the source of any memref_cast
/// into the root operation directly. /// into the root operation directly.
static LogicalResult foldMemRefCast(Operation *op) { static LogicalResult foldMemRefCast(Operation *op) {
bool folded = false; bool folded = false;
for (OpOperand &operand : op->getOpOperands()) { for (OpOperand &operand : op->getOpOperands()) {
auto cast = dyn_cast_or_null<MemRefCastOp>(operand.get()->getDefiningOp()); auto cast = dyn_cast_or_null<MemRefCastOp>(operand.get().getDefiningOp());
if (cast && !cast.getOperand()->getType().isa<UnrankedMemRefType>()) { if (cast && !cast.getOperand().getType().isa<UnrankedMemRefType>()) {
operand.set(cast.getOperand()); operand.set(cast.getOperand());
folded = true; folded = true;
} }
} }
return success(folded); return success(folded);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Lines if (numDimOperands != type.getNumDynamicDims())
return parser.emitError(parser.getNameLoc()) return parser.emitError(parser.getNameLoc())
<< "dimension operand count does not equal memref dynamic dimension " << "dimension operand count does not equal memref dynamic dimension "
"count"; "count";
result.types.push_back(type); result.types.push_back(type);
return success(); return success();
} }
static LogicalResult verify(AllocOp op) { static LogicalResult verify(AllocOp op) {
auto memRefType = op.getResult()->getType().dyn_cast<MemRefType>(); auto memRefType = op.getResult().getType().dyn_cast<MemRefType>();
if (!memRefType) if (!memRefType)
return op.emitOpError("result must be a memref"); return op.emitOpError("result must be a memref");
unsigned numSymbols = 0; unsigned numSymbols = 0;
if (!memRefType.getAffineMaps().empty()) { if (!memRefType.getAffineMaps().empty()) {
// Store number of symbols used in affine map (used in subsequent check). // Store number of symbols used in affine map (used in subsequent check).
AffineMap affineMap = memRefType.getAffineMaps()[0]; AffineMap affineMap = memRefType.getAffineMaps()[0];
numSymbols = affineMap.getNumSymbols(); numSymbols = affineMap.getNumSymbols();
Show All 40 Lines PatternMatchResult matchAndRewrite(AllocOp alloc,
unsigned dynamicDimPos = 0; unsigned dynamicDimPos = 0;
for (unsigned dim = 0, e = memrefType.getRank(); dim < e; ++dim) { for (unsigned dim = 0, e = memrefType.getRank(); dim < e; ++dim) {
int64_t dimSize = memrefType.getDimSize(dim); int64_t dimSize = memrefType.getDimSize(dim);
// If this is already static dimension, keep it. // If this is already static dimension, keep it.
if (dimSize != -1) { if (dimSize != -1) {
newShapeConstants.push_back(dimSize); newShapeConstants.push_back(dimSize);
continue; continue;
} }
auto *defOp = alloc.getOperand(dynamicDimPos)->getDefiningOp(); auto *defOp = alloc.getOperand(dynamicDimPos).getDefiningOp();
if (auto constantIndexOp = dyn_cast_or_null<ConstantIndexOp>(defOp)) { if (auto constantIndexOp = dyn_cast_or_null<ConstantIndexOp>(defOp)) {
// Dynamic shape dimension will be folded. // Dynamic shape dimension will be folded.
newShapeConstants.push_back(constantIndexOp.getValue()); newShapeConstants.push_back(constantIndexOp.getValue());
// Record to check for zero uses later below. // Record to check for zero uses later below.
droppedOperands.push_back(constantIndexOp); droppedOperands.push_back(constantIndexOp);
} else { } else {
// Dynamic shape dimension not folded; copy operand from old memref. // Dynamic shape dimension not folded; copy operand from old memref.
newShapeConstants.push_back(-1); newShapeConstants.push_back(-1);
▲ Show 20 Lines • Show All 134 Lines • ▼ Show 20 Lines return op.emitOpError() << "'" << fnAttr.getValue()
<< "' does not reference a valid function"; << "' does not reference a valid function";
// Verify that the operand and result types match the callee. // Verify that the operand and result types match the callee.
auto fnType = fn.getType(); auto fnType = fn.getType();
if (fnType.getNumInputs() != op.getNumOperands()) if (fnType.getNumInputs() != op.getNumOperands())
return op.emitOpError("incorrect number of operands for callee"); return op.emitOpError("incorrect number of operands for callee");
for (unsigned i = 0, e = fnType.getNumInputs(); i != e; ++i) for (unsigned i = 0, e = fnType.getNumInputs(); i != e; ++i)
if (op.getOperand(i)->getType() != fnType.getInput(i)) if (op.getOperand(i).getType() != fnType.getInput(i))
return op.emitOpError("operand type mismatch"); return op.emitOpError("operand type mismatch");
if (fnType.getNumResults() != op.getNumResults()) if (fnType.getNumResults() != op.getNumResults())
return op.emitOpError("incorrect number of results for callee"); return op.emitOpError("incorrect number of results for callee");
for (unsigned i = 0, e = fnType.getNumResults(); i != e; ++i) for (unsigned i = 0, e = fnType.getNumResults(); i != e; ++i)
if (op.getResult(i)->getType() != fnType.getResult(i)) if (op.getResult(i).getType() != fnType.getResult(i))
return op.emitOpError("result type mismatch"); return op.emitOpError("result type mismatch");
return success(); return success();
} }
FunctionType CallOp::getCalleeType() { FunctionType CallOp::getCalleeType() {
SmallVector<Type, 4> resultTypes(getResultTypes()); SmallVector<Type, 4> resultTypes(getResultTypes());
SmallVector<Type, 8> argTypes(getOperandTypes()); SmallVector<Type, 8> argTypes(getOperandTypes());
Show All 40 Lines return failure(
parser.resolveOperands(operands, calleeType.getInputs(), operandsLoc, parser.resolveOperands(operands, calleeType.getInputs(), operandsLoc,
result.operands) || result.operands) ||
parser.addTypesToList(calleeType.getResults(), result.types)); parser.addTypesToList(calleeType.getResults(), result.types));
} }
static void print(OpAsmPrinter &p, CallIndirectOp op) { static void print(OpAsmPrinter &p, CallIndirectOp op) {
p << "call_indirect " << op.getCallee() << '(' << op.getArgOperands() << ')'; p << "call_indirect " << op.getCallee() << '(' << op.getArgOperands() << ')';
p.printOptionalAttrDict(op.getAttrs(), /*elidedAttrs=*/{"callee"}); p.printOptionalAttrDict(op.getAttrs(), /*elidedAttrs=*/{"callee"});
p << " : " << op.getCallee()->getType(); p << " : " << op.getCallee().getType();
} }
static LogicalResult verify(CallIndirectOp op) { static LogicalResult verify(CallIndirectOp op) {
// The callee must be a function. // The callee must be a function.
auto fnType = op.getCallee()->getType().dyn_cast<FunctionType>(); auto fnType = op.getCallee().getType().dyn_cast<FunctionType>();
if (!fnType) if (!fnType)
return op.emitOpError("callee must have function type"); return op.emitOpError("callee must have function type");
// Verify that the operand and result types match the callee. // Verify that the operand and result types match the callee.
if (fnType.getNumInputs() != op.getNumOperands() - 1) if (fnType.getNumInputs() != op.getNumOperands() - 1)
return op.emitOpError("incorrect number of operands for callee"); return op.emitOpError("incorrect number of operands for callee");
for (unsigned i = 0, e = fnType.getNumInputs(); i != e; ++i) for (unsigned i = 0, e = fnType.getNumInputs(); i != e; ++i)
if (op.getOperand(i + 1)->getType() != fnType.getInput(i)) if (op.getOperand(i + 1).getType() != fnType.getInput(i))
return op.emitOpError("operand type mismatch"); return op.emitOpError("operand type mismatch");
if (fnType.getNumResults() != op.getNumResults()) if (fnType.getNumResults() != op.getNumResults())
return op.emitOpError("incorrect number of results for callee"); return op.emitOpError("incorrect number of results for callee");
for (unsigned i = 0, e = fnType.getNumResults(); i != e; ++i) for (unsigned i = 0, e = fnType.getNumResults(); i != e; ++i)
if (op.getResult(i)->getType() != fnType.getResult(i)) if (op.getResult(i).getType() != fnType.getResult(i))
return op.emitOpError("result type mismatch"); return op.emitOpError("result type mismatch");
return success(); return success();
} }
void CallIndirectOp::getCanonicalizationPatterns( void CallIndirectOp::getCanonicalizationPatterns(
OwningRewritePatternList &results, MLIRContext *context) { OwningRewritePatternList &results, MLIRContext *context) {
results.insert<SimplifyIndirectCallWithKnownCallee>(context); results.insert<SimplifyIndirectCallWithKnownCallee>(context);
Show All 25 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CmpIOp // CmpIOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void buildCmpIOp(Builder *build, OperationState &result, static void buildCmpIOp(Builder *build, OperationState &result,
CmpIPredicate predicate, Value lhs, Value rhs) { CmpIPredicate predicate, Value lhs, Value rhs) {
result.addOperands({lhs, rhs}); result.addOperands({lhs, rhs});
result.types.push_back(getI1SameShape(build, lhs->getType())); result.types.push_back(getI1SameShape(build, lhs.getType()));
result.addAttribute( result.addAttribute(
CmpIOp::getPredicateAttrName(), CmpIOp::getPredicateAttrName(),
build->getI64IntegerAttr(static_cast<int64_t>(predicate))); build->getI64IntegerAttr(static_cast<int64_t>(predicate)));
} }
static ParseResult parseCmpIOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseCmpIOp(OpAsmParser &parser, OperationState &result) {
SmallVector<OpAsmParser::OperandType, 2> ops; SmallVector<OpAsmParser::OperandType, 2> ops;
SmallVector<NamedAttribute, 4> attrs; SmallVector<NamedAttribute, 4> attrs;
Show All 35 Linesstatic void print(OpAsmPrinter &p, CmpIOp op) {
Builder b(op.getContext()); Builder b(op.getContext());
auto predicateValue = auto predicateValue =
op.getAttrOfType<IntegerAttr>(CmpIOp::getPredicateAttrName()).getInt(); op.getAttrOfType<IntegerAttr>(CmpIOp::getPredicateAttrName()).getInt();
p << '"' << stringifyCmpIPredicate(static_cast<CmpIPredicate>(predicateValue)) p << '"' << stringifyCmpIPredicate(static_cast<CmpIPredicate>(predicateValue))
<< '"' << ", " << op.lhs() << ", " << op.rhs(); << '"' << ", " << op.lhs() << ", " << op.rhs();
p.printOptionalAttrDict(op.getAttrs(), p.printOptionalAttrDict(op.getAttrs(),
/*elidedAttrs=*/{CmpIOp::getPredicateAttrName()}); /*elidedAttrs=*/{CmpIOp::getPredicateAttrName()});
p << " : " << op.lhs()->getType(); p << " : " << op.lhs().getType();
} }
// Compute `lhs` `pred` `rhs`, where `pred` is one of the known integer // Compute `lhs` `pred` `rhs`, where `pred` is one of the known integer
// comparison predicates. // comparison predicates.
static bool applyCmpPredicate(CmpIPredicate predicate, const APInt &lhs, static bool applyCmpPredicate(CmpIPredicate predicate, const APInt &lhs,
const APInt &rhs) { const APInt &rhs) {
switch (predicate) { switch (predicate) {
case CmpIPredicate::eq: case CmpIPredicate::eq:
▲ Show 20 Lines • Show All 84 Lines • ▼ Show 20 Lines return llvm::StringSwitch<CmpFPredicate>(name)
.Case("uno", CmpFPredicate::UNO) .Case("uno", CmpFPredicate::UNO)
.Case("true", CmpFPredicate::AlwaysTrue) .Case("true", CmpFPredicate::AlwaysTrue)
.Default(CmpFPredicate::NumPredicates); .Default(CmpFPredicate::NumPredicates);
} }
static void buildCmpFOp(Builder *build, OperationState &result, static void buildCmpFOp(Builder *build, OperationState &result,
CmpFPredicate predicate, Value lhs, Value rhs) { CmpFPredicate predicate, Value lhs, Value rhs) {
result.addOperands({lhs, rhs}); result.addOperands({lhs, rhs});
result.types.push_back(getI1SameShape(build, lhs->getType())); result.types.push_back(getI1SameShape(build, lhs.getType()));
result.addAttribute( result.addAttribute(
CmpFOp::getPredicateAttrName(), CmpFOp::getPredicateAttrName(),
build->getI64IntegerAttr(static_cast<int64_t>(predicate))); build->getI64IntegerAttr(static_cast<int64_t>(predicate)));
} }
static ParseResult parseCmpFOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseCmpFOp(OpAsmParser &parser, OperationState &result) {
SmallVector<OpAsmParser::OperandType, 2> ops; SmallVector<OpAsmParser::OperandType, 2> ops;
SmallVector<NamedAttribute, 4> attrs; SmallVector<NamedAttribute, 4> attrs;
Show All 38 Lines auto predicateValue =
op.getAttrOfType<IntegerAttr>(CmpFOp::getPredicateAttrName()).getInt(); op.getAttrOfType<IntegerAttr>(CmpFOp::getPredicateAttrName()).getInt();
assert(predicateValue >= static_cast<int>(CmpFPredicate::FirstValidValue) && assert(predicateValue >= static_cast<int>(CmpFPredicate::FirstValidValue) &&
predicateValue < static_cast<int>(CmpFPredicate::NumPredicates) && predicateValue < static_cast<int>(CmpFPredicate::NumPredicates) &&
"unknown predicate index"); "unknown predicate index");
p << '"' << getCmpFPredicateNames()[predicateValue] << '"' << ", " << op.lhs() p << '"' << getCmpFPredicateNames()[predicateValue] << '"' << ", " << op.lhs()
<< ", " << op.rhs(); << ", " << op.rhs();
p.printOptionalAttrDict(op.getAttrs(), p.printOptionalAttrDict(op.getAttrs(),
/*elidedAttrs=*/{CmpFOp::getPredicateAttrName()}); /*elidedAttrs=*/{CmpFOp::getPredicateAttrName()});
p << " : " << op.lhs()->getType(); p << " : " << op.lhs().getType();
} }
static LogicalResult verify(CmpFOp op) { static LogicalResult verify(CmpFOp op) {
auto predicateAttr = auto predicateAttr =
op.getAttrOfType<IntegerAttr>(CmpFOp::getPredicateAttrName()); op.getAttrOfType<IntegerAttr>(CmpFOp::getPredicateAttrName());
if (!predicateAttr) if (!predicateAttr)
return op.emitOpError("requires an integer attribute named 'predicate'"); return op.emitOpError("requires an integer attribute named 'predicate'");
auto predicate = predicateAttr.getInt(); auto predicate = predicateAttr.getInt();
▲ Show 20 Lines • Show All 282 Lines • ▼ Show 20 Lines
} }
void ConstantFloatOp::build(Builder *builder, OperationState &result, void ConstantFloatOp::build(Builder *builder, OperationState &result,
const APFloat &value, FloatType type) { const APFloat &value, FloatType type) {
ConstantOp::build(builder, result, type, builder->getFloatAttr(type, value)); ConstantOp::build(builder, result, type, builder->getFloatAttr(type, value));
} }
bool ConstantFloatOp::classof(Operation *op) { bool ConstantFloatOp::classof(Operation *op) {
return ConstantOp::classof(op) && return ConstantOp::classof(op) && op->getResult(0).getType().isa<FloatType>();
op->getResult(0)->getType().isa<FloatType>();
} }
/// ConstantIntOp only matches values whose result type is an IntegerType. /// ConstantIntOp only matches values whose result type is an IntegerType.
bool ConstantIntOp::classof(Operation *op) { bool ConstantIntOp::classof(Operation *op) {
return ConstantOp::classof(op) && return ConstantOp::classof(op) &&
op->getResult(0)->getType().isa<IntegerType>(); op->getResult(0).getType().isa<IntegerType>();
} }
void ConstantIntOp::build(Builder *builder, OperationState &result, void ConstantIntOp::build(Builder *builder, OperationState &result,
int64_t value, unsigned width) { int64_t value, unsigned width) {
Type type = builder->getIntegerType(width); Type type = builder->getIntegerType(width);
ConstantOp::build(builder, result, type, ConstantOp::build(builder, result, type,
builder->getIntegerAttr(type, value)); builder->getIntegerAttr(type, value));
} }
/// Build a constant int op producing an integer with the specified type, /// Build a constant int op producing an integer with the specified type,
/// which must be an integer type. /// which must be an integer type.
void ConstantIntOp::build(Builder *builder, OperationState &result, void ConstantIntOp::build(Builder *builder, OperationState &result,
int64_t value, Type type) { int64_t value, Type type) {
assert(type.isa<IntegerType>() && "ConstantIntOp can only have integer type"); assert(type.isa<IntegerType>() && "ConstantIntOp can only have integer type");
ConstantOp::build(builder, result, type, ConstantOp::build(builder, result, type,
builder->getIntegerAttr(type, value)); builder->getIntegerAttr(type, value));
} }
/// ConstantIndexOp only matches values whose result type is Index. /// ConstantIndexOp only matches values whose result type is Index.
bool ConstantIndexOp::classof(Operation *op) { bool ConstantIndexOp::classof(Operation *op) {
return ConstantOp::classof(op) && op->getResult(0)->getType().isIndex(); return ConstantOp::classof(op) && op->getResult(0).getType().isIndex();
} }
void ConstantIndexOp::build(Builder *builder, OperationState &result, void ConstantIndexOp::build(Builder *builder, OperationState &result,
int64_t value) { int64_t value) {
Type type = builder->getIndexType(); Type type = builder->getIndexType();
ConstantOp::build(builder, result, type, ConstantOp::build(builder, result, type,
builder->getIntegerAttr(type, value)); builder->getIntegerAttr(type, value));
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DeallocOp // DeallocOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace { namespace {
/// Fold Dealloc operations that are deallocating an AllocOp that is only used /// Fold Dealloc operations that are deallocating an AllocOp that is only used
/// by other Dealloc operations. /// by other Dealloc operations.
struct SimplifyDeadDealloc : public OpRewritePattern<DeallocOp> { struct SimplifyDeadDealloc : public OpRewritePattern<DeallocOp> {
using OpRewritePattern<DeallocOp>::OpRewritePattern; using OpRewritePattern<DeallocOp>::OpRewritePattern;
PatternMatchResult matchAndRewrite(DeallocOp dealloc, PatternMatchResult matchAndRewrite(DeallocOp dealloc,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
// Check that the memref operand's defining operation is an AllocOp. // Check that the memref operand's defining operation is an AllocOp.
Value memref = dealloc.memref(); Value memref = dealloc.memref();
if (!isa_and_nonnull<AllocOp>(memref->getDefiningOp())) if (!isa_and_nonnull<AllocOp>(memref.getDefiningOp()))
return matchFailure(); return matchFailure();
// Check that all of the uses of the AllocOp are other DeallocOps. // Check that all of the uses of the AllocOp are other DeallocOps.
for (auto *user : memref->getUsers()) for (auto *user : memref.getUsers())
if (!isa<DeallocOp>(user)) if (!isa<DeallocOp>(user))
return matchFailure(); return matchFailure();
// Erase the dealloc operation. // Erase the dealloc operation.
rewriter.eraseOp(dealloc); rewriter.eraseOp(dealloc);
return matchSuccess(); return matchSuccess();
} }
}; };
} // end anonymous namespace. } // end anonymous namespace.
static void print(OpAsmPrinter &p, DeallocOp op) { static void print(OpAsmPrinter &p, DeallocOp op) {
p << "dealloc " << *op.memref() << " : " << op.memref()->getType(); p << "dealloc " << op.memref() << " : " << op.memref().getType();
} }
static ParseResult parseDeallocOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseDeallocOp(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType memrefInfo; OpAsmParser::OperandType memrefInfo;
MemRefType type; MemRefType type;
return failure(parser.parseOperand(memrefInfo) || return failure(parser.parseOperand(memrefInfo) ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperand(memrefInfo, type, result.operands)); parser.resolveOperand(memrefInfo, type, result.operands));
} }
static LogicalResult verify(DeallocOp op) { static LogicalResult verify(DeallocOp op) {
if (!op.memref()->getType().isa<MemRefType>()) if (!op.memref().getType().isa<MemRefType>())
return op.emitOpError("operand must be a memref"); return op.emitOpError("operand must be a memref");
return success(); return success();
} }
void DeallocOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void DeallocOp::getCanonicalizationPatterns(OwningRewritePatternList &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyDeadDealloc>(context); results.insert<SimplifyDeadDealloc>(context);
} }
LogicalResult DeallocOp::fold(ArrayRef<Attribute> cstOperands, LogicalResult DeallocOp::fold(ArrayRef<Attribute> cstOperands,
SmallVectorImpl<OpFoldResult> &results) { SmallVectorImpl<OpFoldResult> &results) {
/// dealloc(memrefcast) -> dealloc /// dealloc(memrefcast) -> dealloc
return foldMemRefCast(*this); return foldMemRefCast(*this);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DimOp // DimOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, DimOp op) { static void print(OpAsmPrinter &p, DimOp op) {
p << "dim " << *op.getOperand() << ", " << op.getIndex(); p << "dim " << op.getOperand() << ", " << op.getIndex();
p.printOptionalAttrDict(op.getAttrs(), /*elidedAttrs=*/{"index"}); p.printOptionalAttrDict(op.getAttrs(), /*elidedAttrs=*/{"index"});
p << " : " << op.getOperand()->getType(); p << " : " << op.getOperand().getType();
} }
static ParseResult parseDimOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseDimOp(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType operandInfo; OpAsmParser::OperandType operandInfo;
IntegerAttr indexAttr; IntegerAttr indexAttr;
Type type; Type type;
Type indexType = parser.getBuilder().getIndexType(); Type indexType = parser.getBuilder().getIndexType();
return failure( return failure(
parser.parseOperand(operandInfo) || parser.parseComma() || parser.parseOperand(operandInfo) || parser.parseComma() ||
parser.parseAttribute(indexAttr, indexType, "index", result.attributes) || parser.parseAttribute(indexAttr, indexType, "index", result.attributes) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperand(operandInfo, type, result.operands) || parser.resolveOperand(operandInfo, type, result.operands) ||
parser.addTypeToList(indexType, result.types)); parser.addTypeToList(indexType, result.types));
} }
static LogicalResult verify(DimOp op) { static LogicalResult verify(DimOp op) {
// Check that we have an integer index operand. // Check that we have an integer index operand.
auto indexAttr = op.getAttrOfType<IntegerAttr>("index"); auto indexAttr = op.getAttrOfType<IntegerAttr>("index");
if (!indexAttr) if (!indexAttr)
return op.emitOpError("requires an integer attribute named 'index'"); return op.emitOpError("requires an integer attribute named 'index'");
int64_t index = indexAttr.getValue().getSExtValue(); int64_t index = indexAttr.getValue().getSExtValue();
auto type = op.getOperand()->getType(); auto type = op.getOperand().getType();
if (auto tensorType = type.dyn_cast<RankedTensorType>()) { if (auto tensorType = type.dyn_cast<RankedTensorType>()) {
if (index >= tensorType.getRank()) if (index >= tensorType.getRank())
return op.emitOpError("index is out of range"); return op.emitOpError("index is out of range");
} else if (auto memrefType = type.dyn_cast<MemRefType>()) { } else if (auto memrefType = type.dyn_cast<MemRefType>()) {
if (index >= memrefType.getRank()) if (index >= memrefType.getRank())
return op.emitOpError("index is out of range"); return op.emitOpError("index is out of range");
} else if (type.isa<UnrankedTensorType>()) { } else if (type.isa<UnrankedTensorType>()) {
// ok, assumed to be in-range. // ok, assumed to be in-range.
} else { } else {
return op.emitOpError("requires an operand with tensor or memref type"); return op.emitOpError("requires an operand with tensor or memref type");
} }
return success(); return success();
} }
OpFoldResult DimOp::fold(ArrayRef<Attribute> operands) { OpFoldResult DimOp::fold(ArrayRef<Attribute> operands) {
// Constant fold dim when the size along the index referred to is a constant. // Constant fold dim when the size along the index referred to is a constant.
auto opType = memrefOrTensor()->getType(); auto opType = memrefOrTensor().getType();
int64_t indexSize = -1; int64_t indexSize = -1;
if (auto tensorType = opType.dyn_cast<RankedTensorType>()) if (auto tensorType = opType.dyn_cast<RankedTensorType>())
indexSize = tensorType.getShape()[getIndex()]; indexSize = tensorType.getShape()[getIndex()];
else if (auto memrefType = opType.dyn_cast<MemRefType>()) else if (auto memrefType = opType.dyn_cast<MemRefType>())
indexSize = memrefType.getShape()[getIndex()]; indexSize = memrefType.getShape()[getIndex()];
if (!ShapedType::isDynamic(indexSize)) if (!ShapedType::isDynamic(indexSize))
return IntegerAttr::get(IndexType::get(getContext()), indexSize); return IntegerAttr::get(IndexType::get(getContext()), indexSize);
// Fold dim to the size argument for an AllocOp/ViewOp/SubViewOp. // Fold dim to the size argument for an AllocOp/ViewOp/SubViewOp.
auto memrefType = opType.dyn_cast<MemRefType>(); auto memrefType = opType.dyn_cast<MemRefType>();
if (!memrefType) if (!memrefType)
return {}; return {};
// The size at getIndex() is now a dynamic size of a memref. // The size at getIndex() is now a dynamic size of a memref.
auto memref = memrefOrTensor()->getDefiningOp(); auto memref = memrefOrTensor().getDefiningOp();
if (auto alloc = dyn_cast_or_null<AllocOp>(memref)) if (auto alloc = dyn_cast_or_null<AllocOp>(memref))
return *(alloc.getDynamicSizes().begin() + return *(alloc.getDynamicSizes().begin() +
memrefType.getDynamicDimIndex(getIndex())); memrefType.getDynamicDimIndex(getIndex()));
if (auto view = dyn_cast_or_null<ViewOp>(memref)) if (auto view = dyn_cast_or_null<ViewOp>(memref))
return *(view.getDynamicSizes().begin() + return *(view.getDynamicSizes().begin() +
memrefType.getDynamicDimIndex(getIndex())); memrefType.getDynamicDimIndex(getIndex()));
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Linesvoid DmaStartOp::build(Builder *builder, OperationState &result,
result.addOperands(destIndices); result.addOperands(destIndices);
result.addOperands({numElements, tagMemRef}); result.addOperands({numElements, tagMemRef});
result.addOperands(tagIndices); result.addOperands(tagIndices);
if (stride) if (stride)
result.addOperands({stride, elementsPerStride}); result.addOperands({stride, elementsPerStride});
} }
void DmaStartOp::print(OpAsmPrinter &p) { void DmaStartOp::print(OpAsmPrinter &p) {
p << "dma_start " << *getSrcMemRef() << '[' << getSrcIndices() << "], " p << "dma_start " << getSrcMemRef() << '[' << getSrcIndices() << "], "
<< *getDstMemRef() << '[' << getDstIndices() << "], " << *getNumElements() << getDstMemRef() << '[' << getDstIndices() << "], " << getNumElements()
<< ", " << *getTagMemRef() << '[' << getTagIndices() << ']'; << ", " << getTagMemRef() << '[' << getTagIndices() << ']';
if (isStrided()) if (isStrided())
p << ", " << *getStride() << ", " << *getNumElementsPerStride(); p << ", " << getStride() << ", " << getNumElementsPerStride();
p.printOptionalAttrDict(getAttrs()); p.printOptionalAttrDict(getAttrs());
p << " : " << getSrcMemRef()->getType(); p << " : " << getSrcMemRef().getType() << ", " << getDstMemRef().getType()
p << ", " << getDstMemRef()->getType(); << ", " << getTagMemRef().getType();
p << ", " << getTagMemRef()->getType();
} }
// Parse DmaStartOp. // Parse DmaStartOp.
// Ex: // Ex:
// %dma_id = dma_start %src[%i, %j], %dst[%k, %l], %size, // %dma_id = dma_start %src[%i, %j], %dst[%k, %l], %size,
// %tag[%index], %stride, %num_elt_per_stride : // %tag[%index], %stride, %num_elt_per_stride :
// : memref<3076 x f32, 0>, // : memref<3076 x f32, 0>,
// memref<1024 x f32, 2>, // memref<1024 x f32, 2>,
▲ Show 20 Lines • Show All 113 Lines • ▼ Show 20 Linesvoid DmaWaitOp::build(Builder *builder, OperationState &result, Value tagMemRef,
result.addOperands(tagIndices); result.addOperands(tagIndices);
result.addOperands(numElements); result.addOperands(numElements);
} }
void DmaWaitOp::print(OpAsmPrinter &p) { void DmaWaitOp::print(OpAsmPrinter &p) {
p << "dma_wait " << getTagMemRef() << '[' << getTagIndices() << "], " p << "dma_wait " << getTagMemRef() << '[' << getTagIndices() << "], "
<< getNumElements(); << getNumElements();
p.printOptionalAttrDict(getAttrs()); p.printOptionalAttrDict(getAttrs());
p << " : " << getTagMemRef()->getType(); p << " : " << getTagMemRef().getType();
} }
// Parse DmaWaitOp. // Parse DmaWaitOp.
// Eg: // Eg:
// dma_wait %tag[%index], %num_elements : memref<1 x i32, (d0) -> (d0), 4> // dma_wait %tag[%index], %num_elements : memref<1 x i32, (d0) -> (d0), 4>
// //
ParseResult DmaWaitOp::parse(OpAsmParser &parser, OperationState &result) { ParseResult DmaWaitOp::parse(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType tagMemrefInfo; OpAsmParser::OperandType tagMemrefInfo;
Show All 30 LinesLogicalResult DmaWaitOp::fold(ArrayRef<Attribute> cstOperands,
return foldMemRefCast(*this); return foldMemRefCast(*this);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ExtractElementOp // ExtractElementOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, ExtractElementOp op) { static void print(OpAsmPrinter &p, ExtractElementOp op) {
p << "extract_element " << *op.getAggregate() << '[' << op.getIndices(); p << "extract_element " << op.getAggregate() << '[' << op.getIndices();
p << ']'; p << ']';
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getAggregate()->getType(); p << " : " << op.getAggregate().getType();
} }
static ParseResult parseExtractElementOp(OpAsmParser &parser, static ParseResult parseExtractElementOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
OpAsmParser::OperandType aggregateInfo; OpAsmParser::OperandType aggregateInfo;
SmallVector<OpAsmParser::OperandType, 4> indexInfo; SmallVector<OpAsmParser::OperandType, 4> indexInfo;
ShapedType type; ShapedType type;
auto indexTy = parser.getBuilder().getIndexType(); auto indexTy = parser.getBuilder().getIndexType();
return failure( return failure(
parser.parseOperand(aggregateInfo) || parser.parseOperand(aggregateInfo) ||
parser.parseOperandList(indexInfo, OpAsmParser::Delimiter::Square) || parser.parseOperandList(indexInfo, OpAsmParser::Delimiter::Square) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperand(aggregateInfo, type, result.operands) || parser.resolveOperand(aggregateInfo, type, result.operands) ||
parser.resolveOperands(indexInfo, indexTy, result.operands) || parser.resolveOperands(indexInfo, indexTy, result.operands) ||
parser.addTypeToList(type.getElementType(), result.types)); parser.addTypeToList(type.getElementType(), result.types));
} }
static LogicalResult verify(ExtractElementOp op) { static LogicalResult verify(ExtractElementOp op) {
auto aggregateType = op.getAggregate()->getType().cast<ShapedType>(); auto aggregateType = op.getAggregate().getType().cast<ShapedType>();
// This should be possible with tablegen type constraints // This should be possible with tablegen type constraints
if (op.getType() != aggregateType.getElementType()) if (op.getType() != aggregateType.getElementType())
return op.emitOpError("result type must match element type of aggregate"); return op.emitOpError("result type must match element type of aggregate");
// Verify the # indices match if we have a ranked type. // Verify the # indices match if we have a ranked type.
if (aggregateType.hasRank() && if (aggregateType.hasRank() &&
aggregateType.getRank() != op.getNumOperands() - 1) aggregateType.getRank() != op.getNumOperands() - 1)
Show All 40 Lines return (a.isIndex() && b.isa<IntegerType>()) ||
(a.isa<IntegerType>() && b.isIndex()); (a.isa<IntegerType>() && b.isIndex());
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// LoadOp // LoadOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, LoadOp op) { static void print(OpAsmPrinter &p, LoadOp op) {
p << "load " << *op.getMemRef() << '[' << op.getIndices() << ']'; p << "load " << op.getMemRef() << '[' << op.getIndices() << ']';
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getMemRefType(); p << " : " << op.getMemRefType();
} }
static ParseResult parseLoadOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseLoadOp(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType memrefInfo; OpAsmParser::OperandType memrefInfo;
SmallVector<OpAsmParser::OperandType, 4> indexInfo; SmallVector<OpAsmParser::OperandType, 4> indexInfo;
MemRefType type; MemRefType type;
▲ Show 20 Lines • Show All 130 Lines • ▼ Show 20 Lines return constFoldBinaryOp<IntegerAttr>(operands,
[](APInt a, APInt b) { return a * b; }); [](APInt a, APInt b) { return a * b; });
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// PrefetchOp // PrefetchOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, PrefetchOp op) { static void print(OpAsmPrinter &p, PrefetchOp op) {
p << PrefetchOp::getOperationName() << " " << *op.memref() << '['; p << PrefetchOp::getOperationName() << " " << op.memref() << '[';
p.printOperands(op.indices()); p.printOperands(op.indices());
p << ']' << ", " << (op.isWrite() ? "write" : "read"); p << ']' << ", " << (op.isWrite() ? "write" : "read");
p << ", locality<" << op.localityHint(); p << ", locality<" << op.localityHint();
p << ">, " << (op.isDataCache() ? "data" : "instr"); p << ">, " << (op.isDataCache() ? "data" : "instr");
p.printOptionalAttrDict( p.printOptionalAttrDict(
op.getAttrs(), op.getAttrs(),
/*elidedAttrs=*/{"localityHint", "isWrite", "isDataCache"}); /*elidedAttrs=*/{"localityHint", "isWrite", "isDataCache"});
p << " : " << op.getMemRefType(); p << " : " << op.getMemRefType();
▲ Show 20 Lines • Show All 53 Lines • ▼ Show 20 LinesLogicalResult PrefetchOp::fold(ArrayRef<Attribute> cstOperands,
return foldMemRefCast(*this); return foldMemRefCast(*this);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// RankOp // RankOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, RankOp op) { static void print(OpAsmPrinter &p, RankOp op) {
p << "rank " << *op.getOperand() << " : " << op.getOperand()->getType(); p << "rank " << op.getOperand() << " : " << op.getOperand().getType();
} }
static ParseResult parseRankOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseRankOp(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType operandInfo; OpAsmParser::OperandType operandInfo;
Type type; Type type;
Type indexType = parser.getBuilder().getIndexType(); Type indexType = parser.getBuilder().getIndexType();
return failure(parser.parseOperand(operandInfo) || return failure(parser.parseOperand(operandInfo) ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperand(operandInfo, type, result.operands) || parser.resolveOperand(operandInfo, type, result.operands) ||
parser.addTypeToList(indexType, result.types)); parser.addTypeToList(indexType, result.types));
} }
OpFoldResult RankOp::fold(ArrayRef<Attribute> operands) { OpFoldResult RankOp::fold(ArrayRef<Attribute> operands) {
// Constant fold rank when the rank of the tensor is known. // Constant fold rank when the rank of the tensor is known.
auto type = getOperand()->getType(); auto type = getOperand().getType();
if (auto tensorType = type.dyn_cast<RankedTensorType>()) if (auto tensorType = type.dyn_cast<RankedTensorType>())
return IntegerAttr::get(IndexType::get(getContext()), tensorType.getRank()); return IntegerAttr::get(IndexType::get(getContext()), tensorType.getRank());
return IntegerAttr(); return IntegerAttr();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SignedRemIOp // SignedRemIOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Linesstatic LogicalResult verify(ReturnOp op) {
// The operand number and types must match the function signature. // The operand number and types must match the function signature.
const auto &results = function.getType().getResults(); const auto &results = function.getType().getResults();
if (op.getNumOperands() != results.size()) if (op.getNumOperands() != results.size())
return op.emitOpError("has ") return op.emitOpError("has ")
<< op.getNumOperands() << op.getNumOperands()
<< " operands, but enclosing function returns " << results.size(); << " operands, but enclosing function returns " << results.size();
for (unsigned i = 0, e = results.size(); i != e; ++i) for (unsigned i = 0, e = results.size(); i != e; ++i)
if (op.getOperand(i)->getType() != results[i]) if (op.getOperand(i).getType() != results[i])
return op.emitError() return op.emitError()
<< "type of return operand " << i << " (" << "type of return operand " << i << " ("
<< op.getOperand(i)->getType() << op.getOperand(i).getType()
<< ") doesn't match function result type (" << results[i] << ")"; << ") doesn't match function result type (" << results[i] << ")";
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SIToFPOp // SIToFPOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 23 Linesstatic ParseResult parseSelectOp(OpAsmParser &parser, OperationState &result) {
SmallVector<Type, 3> types = {i1Type, type, type}; SmallVector<Type, 3> types = {i1Type, type, type};
return failure(parser.resolveOperands(ops, types, parser.getNameLoc(), return failure(parser.resolveOperands(ops, types, parser.getNameLoc(),
result.operands) || result.operands) ||
parser.addTypeToList(type, result.types)); parser.addTypeToList(type, result.types));
} }
static void print(OpAsmPrinter &p, SelectOp op) { static void print(OpAsmPrinter &p, SelectOp op) {
p << "select " << op.getOperands() << " : " << op.getTrueValue()->getType(); p << "select " << op.getOperands() << " : " << op.getTrueValue().getType();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
} }
static LogicalResult verify(SelectOp op) { static LogicalResult verify(SelectOp op) {
auto trueType = op.getTrueValue()->getType(); auto trueType = op.getTrueValue().getType();
auto falseType = op.getFalseValue()->getType(); auto falseType = op.getFalseValue().getType();
if (trueType != falseType) if (trueType != falseType)
return op.emitOpError( return op.emitOpError(
"requires 'true' and 'false' arguments to be of the same type"); "requires 'true' and 'false' arguments to be of the same type");
return success(); return success();
} }
Show All 12 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SignExtendIOp // SignExtendIOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(SignExtendIOp op) { static LogicalResult verify(SignExtendIOp op) {
// Get the scalar type (which is either directly the type of the operand // Get the scalar type (which is either directly the type of the operand
// or the vector's/tensor's element type. // or the vector's/tensor's element type.
auto srcType = getElementTypeOrSelf(op.getOperand()->getType()); auto srcType = getElementTypeOrSelf(op.getOperand().getType());
auto dstType = getElementTypeOrSelf(op.getType()); auto dstType = getElementTypeOrSelf(op.getType());
// For now, index is forbidden for the source and the destination type. // For now, index is forbidden for the source and the destination type.
if (srcType.isa<IndexType>()) if (srcType.isa<IndexType>())
return op.emitError() << srcType << " is not a valid operand type"; return op.emitError() << srcType << " is not a valid operand type";
if (dstType.isa<IndexType>()) if (dstType.isa<IndexType>())
return op.emitError() << dstType << " is not a valid result type"; return op.emitError() << dstType << " is not a valid result type";
if (srcType.cast<IntegerType>().getWidth() >= if (srcType.cast<IntegerType>().getWidth() >=
dstType.cast<IntegerType>().getWidth()) dstType.cast<IntegerType>().getWidth())
return op.emitError("result type ") return op.emitError("result type ")
<< dstType << " must be wider than operand type " << srcType; << dstType << " must be wider than operand type " << srcType;
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SplatOp // SplatOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, SplatOp op) { static void print(OpAsmPrinter &p, SplatOp op) {
p << "splat " << *op.getOperand(); p << "splat " << op.getOperand();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getType(); p << " : " << op.getType();
} }
static ParseResult parseSplatOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseSplatOp(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType splatValueInfo; OpAsmParser::OperandType splatValueInfo;
ShapedType shapedType; ShapedType shapedType;
return failure(parser.parseOperand(splatValueInfo) || return failure(parser.parseOperand(splatValueInfo) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(shapedType) || parser.parseColonType(shapedType) ||
parser.resolveOperand(splatValueInfo, parser.resolveOperand(splatValueInfo,
shapedType.getElementType(), shapedType.getElementType(),
result.operands) || result.operands) ||
parser.addTypeToList(shapedType, result.types)); parser.addTypeToList(shapedType, result.types));
} }
static LogicalResult verify(SplatOp op) { static LogicalResult verify(SplatOp op) {
// TODO: we could replace this by a trait. // TODO: we could replace this by a trait.
if (op.getOperand()->getType() != if (op.getOperand().getType() !=
op.getType().cast<ShapedType>().getElementType()) op.getType().cast<ShapedType>().getElementType())
return op.emitError("operand should be of elemental type of result type"); return op.emitError("operand should be of elemental type of result type");
return success(); return success();
} }
// Constant folding hook for SplatOp. // Constant folding hook for SplatOp.
OpFoldResult SplatOp::fold(ArrayRef<Attribute> operands) { OpFoldResult SplatOp::fold(ArrayRef<Attribute> operands) {
Show All 12 LinesOpFoldResult SplatOp::fold(ArrayRef<Attribute> operands) {
return SplatElementsAttr::get(shapedType, {constOperand}); return SplatElementsAttr::get(shapedType, {constOperand});
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// StoreOp // StoreOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, StoreOp op) { static void print(OpAsmPrinter &p, StoreOp op) {
p << "store " << *op.getValueToStore(); p << "store " << op.getValueToStore();
p << ", " << *op.getMemRef() << '[' << op.getIndices() << ']'; p << ", " << op.getMemRef() << '[' << op.getIndices() << ']';
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getMemRefType(); p << " : " << op.getMemRefType();
} }
static ParseResult parseStoreOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseStoreOp(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType storeValueInfo; OpAsmParser::OperandType storeValueInfo;
OpAsmParser::OperandType memrefInfo; OpAsmParser::OperandType memrefInfo;
SmallVector<OpAsmParser::OperandType, 4> indexInfo; SmallVector<OpAsmParser::OperandType, 4> indexInfo;
Show All 9 Lines return failure(
parser.resolveOperand(storeValueInfo, memrefType.getElementType(), parser.resolveOperand(storeValueInfo, memrefType.getElementType(),
result.operands) || result.operands) ||
parser.resolveOperand(memrefInfo, memrefType, result.operands) || parser.resolveOperand(memrefInfo, memrefType, result.operands) ||
parser.resolveOperands(indexInfo, indexTy, result.operands)); parser.resolveOperands(indexInfo, indexTy, result.operands));
} }
static LogicalResult verify(StoreOp op) { static LogicalResult verify(StoreOp op) {
// First operand must have same type as memref element type. // First operand must have same type as memref element type.
if (op.getValueToStore()->getType() != op.getMemRefType().getElementType()) if (op.getValueToStore().getType() != op.getMemRefType().getElementType())
return op.emitOpError( return op.emitOpError(
"first operand must have same type memref element type"); "first operand must have same type memref element type");
if (op.getNumOperands() != 2 + op.getMemRefType().getRank()) if (op.getNumOperands() != 2 + op.getMemRefType().getRank())
return op.emitOpError("store index operand count not equal to memref rank"); return op.emitOpError("store index operand count not equal to memref rank");
return success(); return success();
} }
▲ Show 20 Lines • Show All 104 Lines • ▼ Show 20 Linesstatic Type getTensorTypeFromMemRefType(Builder &b, Type type) {
return b.getNoneType(); return b.getNoneType();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TensorLoadOp // TensorLoadOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, TensorLoadOp op) { static void print(OpAsmPrinter &p, TensorLoadOp op) {
p << "tensor_load " << *op.getOperand(); p << "tensor_load " << op.getOperand();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getOperand()->getType(); p << " : " << op.getOperand().getType();
} }
static ParseResult parseTensorLoadOp(OpAsmParser &parser, static ParseResult parseTensorLoadOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
OpAsmParser::OperandType op; OpAsmParser::OperandType op;
Type type; Type type;
return failure(parser.parseOperand(op) || return failure(parser.parseOperand(op) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperand(op, type, result.operands) || parser.resolveOperand(op, type, result.operands) ||
parser.addTypeToList( parser.addTypeToList(
getTensorTypeFromMemRefType(parser.getBuilder(), type), getTensorTypeFromMemRefType(parser.getBuilder(), type),
result.types)); result.types));
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TensorStoreOp // TensorStoreOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, TensorStoreOp op) { static void print(OpAsmPrinter &p, TensorStoreOp op) {
p << "tensor_store " << *op.tensor() << ", " << *op.memref(); p << "tensor_store " << op.tensor() << ", " << op.memref();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.memref()->getType(); p << " : " << op.memref().getType();
} }
static ParseResult parseTensorStoreOp(OpAsmParser &parser, static ParseResult parseTensorStoreOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
SmallVector<OpAsmParser::OperandType, 2> ops; SmallVector<OpAsmParser::OperandType, 2> ops;
Type type; Type type;
llvm::SMLoc loc = parser.getCurrentLocation(); llvm::SMLoc loc = parser.getCurrentLocation();
return failure( return failure(
parser.parseOperandList(ops, /*requiredOperandCount=*/2) || parser.parseOperandList(ops, /*requiredOperandCount=*/2) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperands( parser.resolveOperands(
ops, {getTensorTypeFromMemRefType(parser.getBuilder(), type), type}, ops, {getTensorTypeFromMemRefType(parser.getBuilder(), type), type},
loc, result.operands)); loc, result.operands));
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TruncateIOp // TruncateIOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(TruncateIOp op) { static LogicalResult verify(TruncateIOp op) {
auto srcType = getElementTypeOrSelf(op.getOperand()->getType()); auto srcType = getElementTypeOrSelf(op.getOperand().getType());
auto dstType = getElementTypeOrSelf(op.getType()); auto dstType = getElementTypeOrSelf(op.getType());
if (srcType.isa<IndexType>()) if (srcType.isa<IndexType>())
return op.emitError() << srcType << " is not a valid operand type"; return op.emitError() << srcType << " is not a valid operand type";
if (dstType.isa<IndexType>()) if (dstType.isa<IndexType>())
return op.emitError() << dstType << " is not a valid result type"; return op.emitError() << dstType << " is not a valid result type";
if (srcType.cast<IntegerType>().getWidth() <= if (srcType.cast<IntegerType>().getWidth() <=
Show All 29 Lines return failure(
parser.resolveOperand(srcInfo, srcType, result.operands) || parser.resolveOperand(srcInfo, srcType, result.operands) ||
parser.resolveOperands(offsetInfo, indexType, result.operands) || parser.resolveOperands(offsetInfo, indexType, result.operands) ||
parser.resolveOperands(sizesInfo, indexType, result.operands) || parser.resolveOperands(sizesInfo, indexType, result.operands) ||
parser.parseKeywordType("to", dstType) || parser.parseKeywordType("to", dstType) ||
parser.addTypeToList(dstType, result.types)); parser.addTypeToList(dstType, result.types));
} }
static void print(OpAsmPrinter &p, ViewOp op) { static void print(OpAsmPrinter &p, ViewOp op) {
p << op.getOperationName() << ' ' << *op.getOperand(0) << '['; p << op.getOperationName() << ' ' << op.getOperand(0) << '[';
auto dynamicOffset = op.getDynamicOffset(); auto dynamicOffset = op.getDynamicOffset();
if (dynamicOffset != nullptr) if (dynamicOffset != nullptr)
p.printOperand(dynamicOffset); p.printOperand(dynamicOffset);
p << "][" << op.getDynamicSizes() << ']'; p << "][" << op.getDynamicSizes() << ']';
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getOperand(0)->getType() << " to " << op.getType(); p << " : " << op.getOperand(0).getType() << " to " << op.getType();
} }
Value ViewOp::getDynamicOffset() { Value ViewOp::getDynamicOffset() {
int64_t offset; int64_t offset;
SmallVector<int64_t, 4> strides; SmallVector<int64_t, 4> strides;
auto result = auto result =
succeeded(mlir::getStridesAndOffset(getType(), strides, offset)); succeeded(mlir::getStridesAndOffset(getType(), strides, offset));
assert(result); assert(result);
Show All 15 Lines for (int i = rank - 2; i >= 0; --i) {
// If stride at dim 'i' is not dynamic, return error. // If stride at dim 'i' is not dynamic, return error.
if (dynamicStrides && strides[i] != MemRefType::getDynamicStrideOrOffset()) if (dynamicStrides && strides[i] != MemRefType::getDynamicStrideOrOffset())
return failure(); return failure();
} }
return success(); return success();
} }
static LogicalResult verify(ViewOp op) { static LogicalResult verify(ViewOp op) {
auto baseType = op.getOperand(0)->getType().cast<MemRefType>(); auto baseType = op.getOperand(0).getType().cast<MemRefType>();
auto viewType = op.getResult()->getType().cast<MemRefType>(); auto viewType = op.getResult().getType().cast<MemRefType>();
// The base memref should have identity layout map (or none). // The base memref should have identity layout map (or none).
if (baseType.getAffineMaps().size() > 1 || if (baseType.getAffineMaps().size() > 1 ||
(baseType.getAffineMaps().size() == 1 && (baseType.getAffineMaps().size() == 1 &&
!baseType.getAffineMaps()[0].isIdentity())) !baseType.getAffineMaps()[0].isIdentity()))
return op.emitError("unsupported map for base memref type ") << baseType; return op.emitError("unsupported map for base memref type ") << baseType;
// The base memref and the view memref should be in the same memory space. // The base memref and the view memref should be in the same memory space.
▲ Show 20 Lines • Show All 53 Lines • ▼ Show 20 Lines PatternMatchResult matchAndRewrite(ViewOp viewOp,
SmallVector<Value, 4> newOperands; SmallVector<Value, 4> newOperands;
SmallVector<Value, 4> droppedOperands; SmallVector<Value, 4> droppedOperands;
// Fold dynamic offset operand if it is produced by a constant. // Fold dynamic offset operand if it is produced by a constant.
auto dynamicOffset = viewOp.getDynamicOffset(); auto dynamicOffset = viewOp.getDynamicOffset();
int64_t newOffset = oldOffset; int64_t newOffset = oldOffset;
unsigned dynamicOffsetOperandCount = 0; unsigned dynamicOffsetOperandCount = 0;
if (dynamicOffset != nullptr) { if (dynamicOffset != nullptr) {
auto *defOp = dynamicOffset->getDefiningOp(); auto *defOp = dynamicOffset.getDefiningOp();
if (auto constantIndexOp = dyn_cast_or_null<ConstantIndexOp>(defOp)) { if (auto constantIndexOp = dyn_cast_or_null<ConstantIndexOp>(defOp)) {
// Dynamic offset will be folded into the map. // Dynamic offset will be folded into the map.
newOffset = constantIndexOp.getValue(); newOffset = constantIndexOp.getValue();
droppedOperands.push_back(dynamicOffset); droppedOperands.push_back(dynamicOffset);
} else { } else {
// Unable to fold dynamic offset. Add it to 'newOperands' list. // Unable to fold dynamic offset. Add it to 'newOperands' list.
newOperands.push_back(dynamicOffset); newOperands.push_back(dynamicOffset);
dynamicOffsetOperandCount = 1; dynamicOffsetOperandCount = 1;
} }
} }
// Fold any dynamic dim operands which are produced by a constant. // Fold any dynamic dim operands which are produced by a constant.
SmallVector<int64_t, 4> newShapeConstants; SmallVector<int64_t, 4> newShapeConstants;
newShapeConstants.reserve(memrefType.getRank()); newShapeConstants.reserve(memrefType.getRank());
unsigned dynamicDimPos = viewOp.getDynamicSizesOperandStart(); unsigned dynamicDimPos = viewOp.getDynamicSizesOperandStart();
unsigned rank = memrefType.getRank(); unsigned rank = memrefType.getRank();
for (unsigned dim = 0, e = rank; dim < e; ++dim) { for (unsigned dim = 0, e = rank; dim < e; ++dim) {
int64_t dimSize = memrefType.getDimSize(dim); int64_t dimSize = memrefType.getDimSize(dim);
// If this is already static dimension, keep it. // If this is already static dimension, keep it.
if (!ShapedType::isDynamic(dimSize)) { if (!ShapedType::isDynamic(dimSize)) {
newShapeConstants.push_back(dimSize); newShapeConstants.push_back(dimSize);
continue; continue;
} }
auto *defOp = viewOp.getOperand(dynamicDimPos)->getDefiningOp(); auto *defOp = viewOp.getOperand(dynamicDimPos).getDefiningOp();
if (auto constantIndexOp = dyn_cast_or_null<ConstantIndexOp>(defOp)) { if (auto constantIndexOp = dyn_cast_or_null<ConstantIndexOp>(defOp)) {
// Dynamic shape dimension will be folded. // Dynamic shape dimension will be folded.
newShapeConstants.push_back(constantIndexOp.getValue()); newShapeConstants.push_back(constantIndexOp.getValue());
// Record to check for zero uses later below. // Record to check for zero uses later below.
droppedOperands.push_back(constantIndexOp); droppedOperands.push_back(constantIndexOp);
} else { } else {
// Dynamic shape dimension not folded; copy operand from old memref. // Dynamic shape dimension not folded; copy operand from old memref.
newShapeConstants.push_back(dimSize); newShapeConstants.push_back(dimSize);
▲ Show 20 Lines • Show All 95 Lines • ▼ Show 20 Lines return MemRefType::get(sizes, elementType, stridedLayout,
memRefType.getMemorySpace()); memRefType.getMemorySpace());
} }
void mlir::SubViewOp::build(Builder *b, OperationState &result, Value source, void mlir::SubViewOp::build(Builder *b, OperationState &result, Value source,
ValueRange offsets, ValueRange sizes, ValueRange offsets, ValueRange sizes,
ValueRange strides, Type resultType, ValueRange strides, Type resultType,
ArrayRef<NamedAttribute> attrs) { ArrayRef<NamedAttribute> attrs) {
if (!resultType) if (!resultType)
resultType = inferSubViewResultType(source->getType().cast<MemRefType>()); resultType = inferSubViewResultType(source.getType().cast<MemRefType>());
auto segmentAttr = b->getI32VectorAttr( auto segmentAttr = b->getI32VectorAttr(
{1, static_cast<int>(offsets.size()), static_cast<int32_t>(sizes.size()), {1, static_cast<int>(offsets.size()), static_cast<int32_t>(sizes.size()),
static_cast<int32_t>(strides.size())}); static_cast<int32_t>(strides.size())});
build(b, result, resultType, source, offsets, sizes, strides, segmentAttr); build(b, result, resultType, source, offsets, sizes, strides, segmentAttr);
result.addAttributes(attrs); result.addAttributes(attrs);
} }
void mlir::SubViewOp::build(Builder *b, OperationState &result, Type resultType, void mlir::SubViewOp::build(Builder *b, OperationState &result, Type resultType,
Show All 30 Lines return failure(
parser.resolveOperands(offsetsInfo, indexType, result.operands) || parser.resolveOperands(offsetsInfo, indexType, result.operands) ||
parser.resolveOperands(sizesInfo, indexType, result.operands) || parser.resolveOperands(sizesInfo, indexType, result.operands) ||
parser.resolveOperands(stridesInfo, indexType, result.operands) || parser.resolveOperands(stridesInfo, indexType, result.operands) ||
parser.parseKeywordType("to", dstType) || parser.parseKeywordType("to", dstType) ||
parser.addTypeToList(dstType, result.types)); parser.addTypeToList(dstType, result.types));
} }
static void print(OpAsmPrinter &p, SubViewOp op) { static void print(OpAsmPrinter &p, SubViewOp op) {
p << op.getOperationName() << ' ' << *op.getOperand(0) << '[' << op.offsets() p << op.getOperationName() << ' ' << op.getOperand(0) << '[' << op.offsets()
<< "][" << op.sizes() << "][" << op.strides() << ']'; << "][" << op.sizes() << "][" << op.strides() << ']';
SmallVector<StringRef, 1> elidedAttrs = { SmallVector<StringRef, 1> elidedAttrs = {
SubViewOp::getOperandSegmentSizeAttr()}; SubViewOp::getOperandSegmentSizeAttr()};
p.printOptionalAttrDict(op.getAttrs(), elidedAttrs); p.printOptionalAttrDict(op.getAttrs(), elidedAttrs);
p << " : " << op.getOperand(0)->getType() << " to " << op.getType(); p << " : " << op.getOperand(0).getType() << " to " << op.getType();
} }
static LogicalResult verify(SubViewOp op) { static LogicalResult verify(SubViewOp op) {
auto baseType = op.getBaseMemRefType().cast<MemRefType>(); auto baseType = op.getBaseMemRefType().cast<MemRefType>();
auto subViewType = op.getType(); auto subViewType = op.getType();
// The rank of the base and result subview must match. // The rank of the base and result subview must match.
if (baseType.getRank() != subViewType.getRank()) { if (baseType.getRank() != subViewType.getRank()) {
▲ Show 20 Lines • Show All 97 Lines • ▼ Show 20 Lines if (stride.value() == MemRefType::getDynamicStrideOrOffset() &&
"expected result type to have dynamic stride along a dimension if " "expected result type to have dynamic stride along a dimension if "
"the base memref type has dynamic stride along that dimension"); "the base memref type has dynamic stride along that dimension");
} }
} }
return success(); return success();
} }
raw_ostream &mlir::operator<<(raw_ostream &os, SubViewOp::Range &range) { raw_ostream &mlir::operator<<(raw_ostream &os, SubViewOp::Range &range) {
return os << "range " << *range.offset << ":" << *range.size << ":" return os << "range " << range.offset << ":" << range.size << ":"
<< *range.stride; << range.stride;
} }
SmallVector<SubViewOp::Range, 8> SubViewOp::getRanges() { SmallVector<SubViewOp::Range, 8> SubViewOp::getRanges() {
SmallVector<Range, 8> res; SmallVector<Range, 8> res;
unsigned rank = getType().getRank(); unsigned rank = getType().getRank();
res.reserve(rank); res.reserve(rank);
for (unsigned i = 0; i < rank; ++i) for (unsigned i = 0; i < rank; ++i)
res.emplace_back(Range{*(offsets().begin() + i), *(sizes().begin() + i), res.emplace_back(Range{*(offsets().begin() + i), *(sizes().begin() + i),
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines PatternMatchResult matchAndRewrite(SubViewOp subViewOp,
if (subViewType.hasStaticShape() || if (subViewType.hasStaticShape() ||
llvm::any_of(subViewOp.sizes(), [](Value operand) { llvm::any_of(subViewOp.sizes(), [](Value operand) {
return !matchPattern(operand, m_ConstantIndex()); return !matchPattern(operand, m_ConstantIndex());
})) { })) {
return matchFailure(); return matchFailure();
} }
SmallVector<int64_t, 4> staticShape(subViewOp.getNumSizes()); SmallVector<int64_t, 4> staticShape(subViewOp.getNumSizes());
for (auto size : llvm::enumerate(subViewOp.sizes())) { for (auto size : llvm::enumerate(subViewOp.sizes())) {
auto defOp = size.value()->getDefiningOp(); auto defOp = size.value().getDefiningOp();
assert(defOp); assert(defOp);
staticShape[size.index()] = cast<ConstantIndexOp>(defOp).getValue(); staticShape[size.index()] = cast<ConstantIndexOp>(defOp).getValue();
} }
MemRefType newMemRefType = MemRefType::get( MemRefType newMemRefType = MemRefType::get(
staticShape, subViewType.getElementType(), subViewType.getAffineMaps(), staticShape, subViewType.getElementType(), subViewType.getAffineMaps(),
subViewType.getMemorySpace()); subViewType.getMemorySpace());
auto newSubViewOp = rewriter.create<SubViewOp>( auto newSubViewOp = rewriter.create<SubViewOp>(
subViewOp.getLoc(), subViewOp.source(), subViewOp.offsets(), subViewOp.getLoc(), subViewOp.source(), subViewOp.offsets(),
Show All 29 Lines if (failed(getStridesAndOffset(subViewOp.getBaseMemRefType(), baseStrides,
llvm::any_of(subViewOp.strides(), [](Value stride) { llvm::any_of(subViewOp.strides(), [](Value stride) {
return !matchPattern(stride, m_ConstantIndex()); return !matchPattern(stride, m_ConstantIndex());
})) { })) {
return matchFailure(); return matchFailure();
} }
SmallVector<int64_t, 4> staticStrides(subViewOp.getNumStrides()); SmallVector<int64_t, 4> staticStrides(subViewOp.getNumStrides());
for (auto stride : llvm::enumerate(subViewOp.strides())) { for (auto stride : llvm::enumerate(subViewOp.strides())) {
auto defOp = stride.value()->getDefiningOp(); auto defOp = stride.value().getDefiningOp();
assert(defOp); assert(defOp);
assert(baseStrides[stride.index()] > 0); assert(baseStrides[stride.index()] > 0);
staticStrides[stride.index()] = staticStrides[stride.index()] =
cast<ConstantIndexOp>(defOp).getValue() * baseStrides[stride.index()]; cast<ConstantIndexOp>(defOp).getValue() * baseStrides[stride.index()];
} }
AffineMap layoutMap = makeStridedLinearLayoutMap( AffineMap layoutMap = makeStridedLinearLayoutMap(
staticStrides, resultOffset, rewriter.getContext()); staticStrides, resultOffset, rewriter.getContext());
MemRefType newMemRefType = MemRefType newMemRefType =
Show All 34 Lines if (failed(getStridesAndOffset(subViewOp.getBaseMemRefType(), baseStrides,
llvm::any_of(subViewOp.offsets(), [](Value stride) { llvm::any_of(subViewOp.offsets(), [](Value stride) {
return !matchPattern(stride, m_ConstantIndex()); return !matchPattern(stride, m_ConstantIndex());
})) { })) {
return matchFailure(); return matchFailure();
} }
auto staticOffset = baseOffset; auto staticOffset = baseOffset;
for (auto offset : llvm::enumerate(subViewOp.offsets())) { for (auto offset : llvm::enumerate(subViewOp.offsets())) {
auto defOp = offset.value()->getDefiningOp(); auto defOp = offset.value().getDefiningOp();
assert(defOp); assert(defOp);
assert(baseStrides[offset.index()] > 0); assert(baseStrides[offset.index()] > 0);
staticOffset += staticOffset +=
cast<ConstantIndexOp>(defOp).getValue() * baseStrides[offset.index()]; cast<ConstantIndexOp>(defOp).getValue() * baseStrides[offset.index()];
} }
AffineMap layoutMap = makeStridedLinearLayoutMap( AffineMap layoutMap = makeStridedLinearLayoutMap(
resultStrides, staticOffset, rewriter.getContext()); resultStrides, staticOffset, rewriter.getContext());
Show All 18 Lines results.insert<SubViewOpShapeFolder, SubViewOpStrideFolder,
SubViewOpOffsetFolder>(context); SubViewOpOffsetFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ZeroExtendIOp // ZeroExtendIOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(ZeroExtendIOp op) { static LogicalResult verify(ZeroExtendIOp op) {
auto srcType = getElementTypeOrSelf(op.getOperand()->getType()); auto srcType = getElementTypeOrSelf(op.getOperand().getType());
auto dstType = getElementTypeOrSelf(op.getType()); auto dstType = getElementTypeOrSelf(op.getType());
if (srcType.isa<IndexType>()) if (srcType.isa<IndexType>())
return op.emitError() << srcType << " is not a valid operand type"; return op.emitError() << srcType << " is not a valid operand type";
if (dstType.isa<IndexType>()) if (dstType.isa<IndexType>())
return op.emitError() << dstType << " is not a valid result type"; return op.emitError() << dstType << " is not a valid result type";
if (srcType.cast<IntegerType>().getWidth() >= if (srcType.cast<IntegerType>().getWidth() >=
Show All 35 Lines

mlir/lib/Dialect/Traits.cpp

Show First 20 Lines • Show All 157 Lines • ▼ Show 20 Lines
} }
LogicalResult OpTrait::impl::verifyCompatibleOperandBroadcast(Operation *op) { LogicalResult OpTrait::impl::verifyCompatibleOperandBroadcast(Operation *op) {
assert(op->getNumOperands() == 2 && assert(op->getNumOperands() == 2 &&
"only support broadcast check on two operands"); "only support broadcast check on two operands");
assert(op->getNumResults() == 1 && assert(op->getNumResults() == 1 &&
"only support broadcast check on one result"); "only support broadcast check on one result");
auto type1 = op->getOperand(0)->getType(); auto type1 = op->getOperand(0).getType();
auto type2 = op->getOperand(1)->getType(); auto type2 = op->getOperand(1).getType();
auto retType = op->getResult(0)->getType(); auto retType = op->getResult(0).getType();
// We forbid broadcasting vector and tensor. // We forbid broadcasting vector and tensor.
if (hasBothVectorAndTensorType({type1, type2, retType})) if (hasBothVectorAndTensorType({type1, type2, retType}))
return op->emitError("cannot broadcast vector with tensor"); return op->emitError("cannot broadcast vector with tensor");
if (retType.isa<UnrankedTensorType>()) if (retType.isa<UnrankedTensorType>())
return success(); return success();
Show All 35 Lines

mlir/lib/Dialect/VectorOps/VectorOps.cpp

Show First 20 Lines • Show All 61 Lines • ▼ Show 20 Lines
// ContractionOp // ContractionOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void vector::ContractionOp::build(Builder *builder, OperationState &result, void vector::ContractionOp::build(Builder *builder, OperationState &result,
Value lhs, Value rhs, Value acc, Value lhs, Value rhs, Value acc,
ArrayAttr indexingMaps, ArrayAttr indexingMaps,
ArrayAttr iteratorTypes) { ArrayAttr iteratorTypes) {
result.addOperands({lhs, rhs, acc}); result.addOperands({lhs, rhs, acc});
result.addTypes(acc->getType()); result.addTypes(acc.getType());
result.addAttribute(getIndexingMapsAttrName(), indexingMaps); result.addAttribute(getIndexingMapsAttrName(), indexingMaps);
result.addAttribute(getIteratorTypesAttrName(), iteratorTypes); result.addAttribute(getIteratorTypesAttrName(), iteratorTypes);
} }
static ParseResult parseContractionOp(OpAsmParser &parser, static ParseResult parseContractionOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
OpAsmParser::OperandType lhsInfo; OpAsmParser::OperandType lhsInfo;
OpAsmParser::OperandType rhsInfo; OpAsmParser::OperandType rhsInfo;
▲ Show 20 Lines • Show All 41 Lines • ▼ Show 20 Linesstatic void print(OpAsmPrinter &p, ContractionOp op) {
llvm::StringSet<> traitAttrsSet; llvm::StringSet<> traitAttrsSet;
traitAttrsSet.insert(attrNames.begin(), attrNames.end()); traitAttrsSet.insert(attrNames.begin(), attrNames.end());
SmallVector<NamedAttribute, 8> attrs; SmallVector<NamedAttribute, 8> attrs;
for (auto attr : op.getAttrs()) for (auto attr : op.getAttrs())
if (traitAttrsSet.count(attr.first.strref()) > 0) if (traitAttrsSet.count(attr.first.strref()) > 0)
attrs.push_back(attr); attrs.push_back(attr);
auto dictAttr = DictionaryAttr::get(attrs, op.getContext()); auto dictAttr = DictionaryAttr::get(attrs, op.getContext());
p << op.getOperationName() << " " << dictAttr << " " << *op.lhs() << ", "; p << op.getOperationName() << " " << dictAttr << " " << op.lhs() << ", ";
p << *op.rhs() << ", " << *op.acc(); p << op.rhs() << ", " << op.acc();
if (op.masks().size() == 2) if (op.masks().size() == 2)
p << ", " << op.masks(); p << ", " << op.masks();
p.printOptionalAttrDict(op.getAttrs(), attrNames); p.printOptionalAttrDict(op.getAttrs(), attrNames);
p << " : " << op.lhs()->getType() << ", " << op.rhs()->getType() << " into " p << " : " << op.lhs().getType() << ", " << op.rhs().getType() << " into "
<< op.getResultType(); << op.getResultType();
} }
static bool verifyDimMap(VectorType lhsType, VectorType rhsType, static bool verifyDimMap(VectorType lhsType, VectorType rhsType,
const std::vector<std::pair<int64_t, int64_t>> &map) { const std::vector<std::pair<int64_t, int64_t>> &map) {
for (auto &dimPair : map) { for (auto &dimPair : map) {
if (dimPair.first < 0 || dimPair.first >= lhsType.getRank() || if (dimPair.first < 0 || dimPair.first >= lhsType.getRank() ||
dimPair.second < 0 || dimPair.second >= rhsType.getRank() || dimPair.second < 0 || dimPair.second >= rhsType.getRank() ||
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 Lines for (auto it : llvm::enumerate(op.indexing_maps())) {
auto index = it.index(); auto index = it.index();
auto map = it.value().cast<AffineMapAttr>().getValue(); auto map = it.value().cast<AffineMapAttr>().getValue();
if (map.getNumSymbols() != 0) if (map.getNumSymbols() != 0)
return op.emitOpError("expected indexing map ") return op.emitOpError("expected indexing map ")
<< index << " to have no symbols"; << index << " to have no symbols";
if (map.getNumDims() != numIterators) if (map.getNumDims() != numIterators)
return op.emitOpError("expected indexing map ") return op.emitOpError("expected indexing map ")
<< index << " to have " << numIterators << " number of inputs"; << index << " to have " << numIterators << " number of inputs";
auto operandType = op.getOperand(index)->getType().cast<VectorType>(); auto operandType = op.getOperand(index).getType().cast<VectorType>();
unsigned rank = operandType.getShape().size(); unsigned rank = operandType.getShape().size();
if (map.getNumResults() != rank) if (map.getNumResults() != rank)
return op.emitOpError("expected indexing map ") return op.emitOpError("expected indexing map ")
<< index << " to have " << rank << " number of outputs"; << index << " to have " << rank << " number of outputs";
if (!map.isProjectedPermutation()) if (!map.isProjectedPermutation())
return op.emitOpError("expected indexing map ") return op.emitOpError("expected indexing map ")
<< index << " to be a projected permutation of its inputs"; << index << " to be a projected permutation of its inputs";
} }
▲ Show 20 Lines • Show All 123 Lines • ▼ Show 20 LinesSmallVector<AffineMap, 4> ContractionOp::getIndexingMaps() {
return res; return res;
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ExtractElementOp // ExtractElementOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, vector::ExtractElementOp op) { static void print(OpAsmPrinter &p, vector::ExtractElementOp op) {
p << op.getOperationName() << " " << *op.vector() << "[" << *op.position() p << op.getOperationName() << " " << op.vector() << "[" << op.position()
<< " : " << op.position()->getType() << "]"; << " : " << op.position().getType() << "]";
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.vector()->getType(); p << " : " << op.vector().getType();
} }
static ParseResult parseExtractElementOp(OpAsmParser &parser, static ParseResult parseExtractElementOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
OpAsmParser::OperandType vector, position; OpAsmParser::OperandType vector, position;
Type positionType; Type positionType;
VectorType vectorType; VectorType vectorType;
if (parser.parseOperand(vector) || parser.parseLSquare() || if (parser.parseOperand(vector) || parser.parseLSquare() ||
Show All 27 Linesstatic Type inferExtractOpResultType(VectorType vectorType,
return VectorType::get(vectorType.getShape().drop_front(position.size()), return VectorType::get(vectorType.getShape().drop_front(position.size()),
vectorType.getElementType()); vectorType.getElementType());
} }
void vector::ExtractOp::build(Builder *builder, OperationState &result, void vector::ExtractOp::build(Builder *builder, OperationState &result,
Value source, ArrayRef<int64_t> position) { Value source, ArrayRef<int64_t> position) {
result.addOperands(source); result.addOperands(source);
auto positionAttr = getVectorSubscriptAttr(*builder, position); auto positionAttr = getVectorSubscriptAttr(*builder, position);
result.addTypes(inferExtractOpResultType(source->getType().cast<VectorType>(), result.addTypes(inferExtractOpResultType(source.getType().cast<VectorType>(),
positionAttr)); positionAttr));
result.addAttribute(getPositionAttrName(), positionAttr); result.addAttribute(getPositionAttrName(), positionAttr);
} }
static void print(OpAsmPrinter &p, vector::ExtractOp op) { static void print(OpAsmPrinter &p, vector::ExtractOp op) {
p << op.getOperationName() << " " << *op.vector() << op.position(); p << op.getOperationName() << " " << op.vector() << op.position();
p.printOptionalAttrDict(op.getAttrs(), {"position"}); p.printOptionalAttrDict(op.getAttrs(), {"position"});
p << " : " << op.vector()->getType(); p << " : " << op.vector().getType();
} }
static ParseResult parseExtractOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseExtractOp(OpAsmParser &parser, OperationState &result) {
llvm::SMLoc attributeLoc, typeLoc; llvm::SMLoc attributeLoc, typeLoc;
SmallVector<NamedAttribute, 4> attrs; SmallVector<NamedAttribute, 4> attrs;
OpAsmParser::OperandType vector; OpAsmParser::OperandType vector;
Type type; Type type;
Attribute attr; Attribute attr;
▲ Show 20 Lines • Show All 72 Lines • ▼ Show 20 Lines return failure(
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(vectorType) || parser.parseColonType(vectorType) ||
parser.parseKeywordType("into", resultTupleType) || parser.parseKeywordType("into", resultTupleType) ||
parser.resolveOperand(operandInfo, vectorType, result.operands) || parser.resolveOperand(operandInfo, vectorType, result.operands) ||
parser.addTypeToList(resultTupleType, result.types)); parser.addTypeToList(resultTupleType, result.types));
} }
static void print(OpAsmPrinter &p, ExtractSlicesOp op) { static void print(OpAsmPrinter &p, ExtractSlicesOp op) {
p << op.getOperationName() << ' ' << *op.vector() << ", "; p << op.getOperationName() << ' ' << op.vector() << ", ";
p << op.sizes() << ", " << op.strides(); p << op.sizes() << ", " << op.strides();
p.printOptionalAttrDict( p.printOptionalAttrDict(
op.getAttrs(), op.getAttrs(),
/*elidedAttrs=*/{ExtractSlicesOp::getSizesAttrName(), /*elidedAttrs=*/{ExtractSlicesOp::getSizesAttrName(),
ExtractSlicesOp::getStridesAttrName()}); ExtractSlicesOp::getStridesAttrName()});
p << " : " << op.vector()->getType(); p << " : " << op.vector().getType();
p << " into " << op.getResultTupleType(); p << " into " << op.getResultTupleType();
} }
static LogicalResult static LogicalResult
isValidExtractOrInsertSlicesType(Operation *op, VectorType vectorType, isValidExtractOrInsertSlicesType(Operation *op, VectorType vectorType,
TupleType tupleType, ArrayRef<int64_t> sizes, TupleType tupleType, ArrayRef<int64_t> sizes,
ArrayRef<int64_t> strides) { ArrayRef<int64_t> strides) {
// Check for non-unit strides. // Check for non-unit strides.
▲ Show 20 Lines • Show All 76 Lines • ▼ Show 20 Linesvoid ExtractSlicesOp::getStrides(SmallVectorImpl<int64_t> &results) {
populateFromInt64AttrArray(strides(), results); populateFromInt64AttrArray(strides(), results);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// BroadcastOp // BroadcastOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, BroadcastOp op) { static void print(OpAsmPrinter &p, BroadcastOp op) {
p << op.getOperationName() << " " << *op.source() << " : " p << op.getOperationName() << " " << op.source() << " : "
<< op.getSourceType() << " to " << op.getVectorType(); << op.getSourceType() << " to " << op.getVectorType();
} }
static LogicalResult verify(BroadcastOp op) { static LogicalResult verify(BroadcastOp op) {
VectorType srcVectorType = op.getSourceType().dyn_cast<VectorType>(); VectorType srcVectorType = op.getSourceType().dyn_cast<VectorType>();
VectorType dstVectorType = op.getVectorType(); VectorType dstVectorType = op.getVectorType();
// Scalar to vector broadcast is always valid. A vector // Scalar to vector broadcast is always valid. A vector
// to vector broadcast needs some additional checking. // to vector broadcast needs some additional checking.
Show All 31 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ShuffleOp // ShuffleOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void ShuffleOp::build(Builder *builder, OperationState &result, Value v1, void ShuffleOp::build(Builder *builder, OperationState &result, Value v1,
Value v2, ArrayRef<int64_t> mask) { Value v2, ArrayRef<int64_t> mask) {
result.addOperands({v1, v2}); result.addOperands({v1, v2});
auto maskAttr = getVectorSubscriptAttr(*builder, mask); auto maskAttr = getVectorSubscriptAttr(*builder, mask);
result.addTypes(v1->getType()); result.addTypes(v1.getType());
result.addAttribute(getMaskAttrName(), maskAttr); result.addAttribute(getMaskAttrName(), maskAttr);
} }
static void print(OpAsmPrinter &p, ShuffleOp op) { static void print(OpAsmPrinter &p, ShuffleOp op) {
p << op.getOperationName() << " " << *op.v1() << ", " << *op.v2() << " " p << op.getOperationName() << " " << op.v1() << ", " << op.v2() << " "
<< op.mask(); << op.mask();
p.printOptionalAttrDict(op.getAttrs(), {ShuffleOp::getMaskAttrName()}); p.printOptionalAttrDict(op.getAttrs(), {ShuffleOp::getMaskAttrName()});
p << " : " << op.v1()->getType() << ", " << op.v2()->getType(); p << " : " << op.v1().getType() << ", " << op.v2().getType();
} }
static LogicalResult verify(ShuffleOp op) { static LogicalResult verify(ShuffleOp op) {
VectorType resultType = op.getVectorType(); VectorType resultType = op.getVectorType();
VectorType v1Type = op.getV1VectorType(); VectorType v1Type = op.getV1VectorType();
VectorType v2Type = op.getV2VectorType(); VectorType v2Type = op.getV2VectorType();
// Verify ranks. // Verify ranks.
int64_t resRank = resultType.getRank(); int64_t resRank = resultType.getRank();
▲ Show 20 Lines • Show All 58 Lines • ▼ Show 20 Linesstatic ParseResult parseShuffleOp(OpAsmParser &parser, OperationState &result) {
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// InsertElementOp // InsertElementOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, InsertElementOp op) { static void print(OpAsmPrinter &p, InsertElementOp op) {
p << op.getOperationName() << " " << *op.source() << ", " << *op.dest() << "[" p << op.getOperationName() << " " << op.source() << ", " << op.dest() << "["
<< *op.position() << " : " << op.position()->getType() << "]"; << op.position() << " : " << op.position().getType() << "]";
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.dest()->getType(); p << " : " << op.dest().getType();
} }
static ParseResult parseInsertElementOp(OpAsmParser &parser, static ParseResult parseInsertElementOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
OpAsmParser::OperandType source, dest, position; OpAsmParser::OperandType source, dest, position;
Type positionType; Type positionType;
VectorType destType; VectorType destType;
if (parser.parseOperand(source) || parser.parseComma() || if (parser.parseOperand(source) || parser.parseComma() ||
Show All 21 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// InsertOp // InsertOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void InsertOp::build(Builder *builder, OperationState &result, Value source, void InsertOp::build(Builder *builder, OperationState &result, Value source,
Value dest, ArrayRef<int64_t> position) { Value dest, ArrayRef<int64_t> position) {
result.addOperands({source, dest}); result.addOperands({source, dest});
auto positionAttr = getVectorSubscriptAttr(*builder, position); auto positionAttr = getVectorSubscriptAttr(*builder, position);
result.addTypes(dest->getType()); result.addTypes(dest.getType());
result.addAttribute(getPositionAttrName(), positionAttr); result.addAttribute(getPositionAttrName(), positionAttr);
} }
static void print(OpAsmPrinter &p, InsertOp op) { static void print(OpAsmPrinter &p, InsertOp op) {
p << op.getOperationName() << " " << *op.source() << ", " << *op.dest() p << op.getOperationName() << " " << op.source() << ", " << op.dest()
<< op.position(); << op.position();
p.printOptionalAttrDict(op.getAttrs(), {InsertOp::getPositionAttrName()}); p.printOptionalAttrDict(op.getAttrs(), {InsertOp::getPositionAttrName()});
p << " : " << op.getSourceType() << " into " << op.getDestVectorType(); p << " : " << op.getSourceType() << " into " << op.getDestVectorType();
} }
static ParseResult parseInsertOp(OpAsmParser &parser, OperationState &result) { static ParseResult parseInsertOp(OpAsmParser &parser, OperationState &result) {
SmallVector<NamedAttribute, 4> attrs; SmallVector<NamedAttribute, 4> attrs;
OpAsmParser::OperandType source, dest; OpAsmParser::OperandType source, dest;
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 Lines return failure(
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(tupleType) || parser.parseColonType(tupleType) ||
parser.parseKeywordType("into", resultVectorType) || parser.parseKeywordType("into", resultVectorType) ||
parser.resolveOperand(operandInfo, tupleType, result.operands) || parser.resolveOperand(operandInfo, tupleType, result.operands) ||
parser.addTypeToList(resultVectorType, result.types)); parser.addTypeToList(resultVectorType, result.types));
} }
static void print(OpAsmPrinter &p, InsertSlicesOp op) { static void print(OpAsmPrinter &p, InsertSlicesOp op) {
p << op.getOperationName() << ' ' << *op.vectors() << ", "; p << op.getOperationName() << ' ' << op.vectors() << ", ";
p << op.sizes() << ", " << op.strides(); p << op.sizes() << ", " << op.strides();
p.printOptionalAttrDict( p.printOptionalAttrDict(
op.getAttrs(), op.getAttrs(),
/*elidedAttrs=*/{InsertSlicesOp::getSizesAttrName(), /*elidedAttrs=*/{InsertSlicesOp::getSizesAttrName(),
InsertSlicesOp::getStridesAttrName()}); InsertSlicesOp::getStridesAttrName()});
p << " : " << op.vectors()->getType(); p << " : " << op.vectors().getType();
p << " into " << op.getResultVectorType(); p << " into " << op.getResultVectorType();
} }
static LogicalResult verify(InsertSlicesOp op) { static LogicalResult verify(InsertSlicesOp op) {
SmallVector<int64_t, 4> sizes; SmallVector<int64_t, 4> sizes;
op.getSizes(sizes); op.getSizes(sizes);
SmallVector<int64_t, 4> strides; SmallVector<int64_t, 4> strides;
op.getStrides(strides); op.getStrides(strides);
Show All 16 Lines
void InsertStridedSliceOp::build(Builder *builder, OperationState &result, void InsertStridedSliceOp::build(Builder *builder, OperationState &result,
Value source, Value dest, Value source, Value dest,
ArrayRef<int64_t> offsets, ArrayRef<int64_t> offsets,
ArrayRef<int64_t> strides) { ArrayRef<int64_t> strides) {
result.addOperands({source, dest}); result.addOperands({source, dest});
auto offsetsAttr = getVectorSubscriptAttr(*builder, offsets); auto offsetsAttr = getVectorSubscriptAttr(*builder, offsets);
auto stridesAttr = getVectorSubscriptAttr(*builder, strides); auto stridesAttr = getVectorSubscriptAttr(*builder, strides);
result.addTypes(dest->getType()); result.addTypes(dest.getType());
result.addAttribute(getOffsetsAttrName(), offsetsAttr); result.addAttribute(getOffsetsAttrName(), offsetsAttr);
result.addAttribute(getStridesAttrName(), stridesAttr); result.addAttribute(getStridesAttrName(), stridesAttr);
} }
static void print(OpAsmPrinter &p, InsertStridedSliceOp op) { static void print(OpAsmPrinter &p, InsertStridedSliceOp op) {
p << op.getOperationName() << " " << *op.source() << ", " << *op.dest() p << op.getOperationName() << " " << op.source() << ", " << op.dest() << " ";
<< " ";
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getSourceVectorType() << " into " << op.getDestVectorType(); p << " : " << op.getSourceVectorType() << " into " << op.getDestVectorType();
} }
static ParseResult parseInsertStridedSliceOp(OpAsmParser &parser, static ParseResult parseInsertStridedSliceOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
OpAsmParser::OperandType source, dest; OpAsmParser::OperandType source, dest;
VectorType sourceVectorType, destVectorType; VectorType sourceVectorType, destVectorType;
▲ Show 20 Lines • Show All 135 Lines • ▼ Show 20 Linesstatic LogicalResult verify(InsertStridedSliceOp op) {
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// OuterProductOp // OuterProductOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, OuterProductOp op) { static void print(OpAsmPrinter &p, OuterProductOp op) {
p << op.getOperationName() << " " << *op.lhs() << ", " << *op.rhs(); p << op.getOperationName() << " " << op.lhs() << ", " << op.rhs();
if (!op.acc().empty()) if (!op.acc().empty())
p << ", " << op.acc(); p << ", " << op.acc();
p << " : " << op.lhs()->getType() << ", " << op.rhs()->getType(); p << " : " << op.lhs().getType() << ", " << op.rhs().getType();
} }
static ParseResult parseOuterProductOp(OpAsmParser &parser, static ParseResult parseOuterProductOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
SmallVector<OpAsmParser::OperandType, 3> operandsInfo; SmallVector<OpAsmParser::OperandType, 3> operandsInfo;
Type tLHS, tRHS; Type tLHS, tRHS;
if (parser.parseOperandList(operandsInfo) || parser.parseColonType(tLHS) || if (parser.parseOperandList(operandsInfo) || parser.parseColonType(tLHS) ||
parser.parseComma() || parser.parseType(tRHS)) parser.parseComma() || parser.parseType(tRHS))
Show All 34 Linesstatic LogicalResult verify(OuterProductOp op) {
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ReshapeOp // ReshapeOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, ReshapeOp op) { static void print(OpAsmPrinter &p, ReshapeOp op) {
p << op.getOperationName() << " " << *op.vector() << ", [" << op.input_shape() p << op.getOperationName() << " " << op.vector() << ", [" << op.input_shape()
<< "], [" << op.output_shape() << "], " << op.fixed_vector_sizes(); << "], [" << op.output_shape() << "], " << op.fixed_vector_sizes();
SmallVector<StringRef, 2> elidedAttrs = { SmallVector<StringRef, 2> elidedAttrs = {
ReshapeOp::getOperandSegmentSizeAttr(), ReshapeOp::getOperandSegmentSizeAttr(),
ReshapeOp::getFixedVectorSizesAttrName()}; ReshapeOp::getFixedVectorSizesAttrName()};
p.printOptionalAttrDict(op.getAttrs(), elidedAttrs); p.printOptionalAttrDict(op.getAttrs(), elidedAttrs);
p << " : " << op.getInputVectorType() << " to " << op.getOutputVectorType(); p << " : " << op.getInputVectorType() << " to " << op.getOutputVectorType();
} }
▲ Show 20 Lines • Show All 73 Lines • ▼ Show 20 Lines for (unsigned i = 0; i < numFixedVectorSizes; ++i) {
if (fixedVectorSizes[i] != outputVectorType.getShape()[index]) if (fixedVectorSizes[i] != outputVectorType.getShape()[index])
return op.emitError("fixed vector size must match output vector for dim ") return op.emitError("fixed vector size must match output vector for dim ")
<< i; << i;
} }
// If all shape operands are produced by constant ops, verify that product // If all shape operands are produced by constant ops, verify that product
// of dimensions for input/output shape match. // of dimensions for input/output shape match.
auto isDefByConstant = [](Value operand) { auto isDefByConstant = [](Value operand) {
return isa_and_nonnull<ConstantIndexOp>(operand->getDefiningOp()); return isa_and_nonnull<ConstantIndexOp>(operand.getDefiningOp());
}; };
if (llvm::all_of(op.input_shape(), isDefByConstant) && if (llvm::all_of(op.input_shape(), isDefByConstant) &&
llvm::all_of(op.output_shape(), isDefByConstant)) { llvm::all_of(op.output_shape(), isDefByConstant)) {
int64_t numInputElements = 1; int64_t numInputElements = 1;
for (auto operand : op.input_shape()) for (auto operand : op.input_shape())
numInputElements *= numInputElements *=
cast<ConstantIndexOp>(operand->getDefiningOp()).getValue(); cast<ConstantIndexOp>(operand.getDefiningOp()).getValue();
int64_t numOutputElements = 1; int64_t numOutputElements = 1;
for (auto operand : op.output_shape()) for (auto operand : op.output_shape())
numOutputElements *= numOutputElements *=
cast<ConstantIndexOp>(operand->getDefiningOp()).getValue(); cast<ConstantIndexOp>(operand.getDefiningOp()).getValue();
if (numInputElements != numOutputElements) if (numInputElements != numOutputElements)
return op.emitError("product of input and output shape sizes must match"); return op.emitError("product of input and output shape sizes must match");
} }
return success(); return success();
} }
void ReshapeOp::getFixedVectorSizes(SmallVectorImpl<int64_t> &results) { void ReshapeOp::getFixedVectorSizes(SmallVectorImpl<int64_t> &results) {
populateFromInt64AttrArray(fixed_vector_sizes(), results); populateFromInt64AttrArray(fixed_vector_sizes(), results);
Show All 24 Lines
void StridedSliceOp::build(Builder *builder, OperationState &result, void StridedSliceOp::build(Builder *builder, OperationState &result,
Value source, ArrayRef<int64_t> offsets, Value source, ArrayRef<int64_t> offsets,
ArrayRef<int64_t> sizes, ArrayRef<int64_t> strides) { ArrayRef<int64_t> sizes, ArrayRef<int64_t> strides) {
result.addOperands(source); result.addOperands(source);
auto offsetsAttr = getVectorSubscriptAttr(*builder, offsets); auto offsetsAttr = getVectorSubscriptAttr(*builder, offsets);
auto sizesAttr = getVectorSubscriptAttr(*builder, sizes); auto sizesAttr = getVectorSubscriptAttr(*builder, sizes);
auto stridesAttr = getVectorSubscriptAttr(*builder, strides); auto stridesAttr = getVectorSubscriptAttr(*builder, strides);
result.addTypes( result.addTypes(
inferStridedSliceOpResultType(source->getType().cast<VectorType>(), inferStridedSliceOpResultType(source.getType().cast<VectorType>(),
offsetsAttr, sizesAttr, stridesAttr)); offsetsAttr, sizesAttr, stridesAttr));
result.addAttribute(getOffsetsAttrName(), offsetsAttr); result.addAttribute(getOffsetsAttrName(), offsetsAttr);
result.addAttribute(getSizesAttrName(), sizesAttr); result.addAttribute(getSizesAttrName(), sizesAttr);
result.addAttribute(getStridesAttrName(), stridesAttr); result.addAttribute(getStridesAttrName(), stridesAttr);
} }
static void print(OpAsmPrinter &p, StridedSliceOp op) { static void print(OpAsmPrinter &p, StridedSliceOp op) {
p << op.getOperationName() << " " << *op.vector(); p << op.getOperationName() << " " << op.vector();
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.vector()->getType() << " to " << op.getResult()->getType(); p << " : " << op.vector().getType() << " to " << op.getResult().getType();
} }
static ParseResult parseStridedSliceOp(OpAsmParser &parser, static ParseResult parseStridedSliceOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
llvm::SMLoc attributeLoc, typeLoc; llvm::SMLoc attributeLoc, typeLoc;
OpAsmParser::OperandType vector; OpAsmParser::OperandType vector;
VectorType vectorType, resultVectorType; VectorType vectorType, resultVectorType;
return failure(parser.parseOperand(vector) || return failure(parser.parseOperand(vector) ||
Show All 33 Lines if (failed(isIntegerArrayAttrSmallerThanShape(op, offsets, shape, offName)) ||
/*halfOpen=*/false)) || /*halfOpen=*/false)) ||
failed(isSumOfIntegerArrayAttrConfinedToShape(op, offsets, sizes, shape, failed(isSumOfIntegerArrayAttrConfinedToShape(op, offsets, sizes, shape,
offName, sizesName, offName, sizesName,
/*halfOpen=*/false))) /*halfOpen=*/false)))
return failure(); return failure();
auto resultType = inferStridedSliceOpResultType( auto resultType = inferStridedSliceOpResultType(
op.getVectorType(), op.offsets(), op.sizes(), op.strides()); op.getVectorType(), op.offsets(), op.sizes(), op.strides());
if (op.getResult()->getType() != resultType) { if (op.getResult().getType() != resultType) {
op.emitOpError("expected result type to be ") << resultType; op.emitOpError("expected result type to be ") << resultType;
return failure(); return failure();
} }
return success(); return success();
} }
void StridedSliceOp::getOffsets(SmallVectorImpl<int64_t> &results) { void StridedSliceOp::getOffsets(SmallVectorImpl<int64_t> &results) {
populateFromInt64AttrArray(offsets(), results); populateFromInt64AttrArray(offsets(), results);
} }
namespace { namespace {
// Pattern to rewrite a StridedSliceOp(ConstantMaskOp) -> ConstantMaskOp. // Pattern to rewrite a StridedSliceOp(ConstantMaskOp) -> ConstantMaskOp.
class StridedSliceConstantMaskFolder final class StridedSliceConstantMaskFolder final
: public OpRewritePattern<StridedSliceOp> { : public OpRewritePattern<StridedSliceOp> {
public: public:
using OpRewritePattern<StridedSliceOp>::OpRewritePattern; using OpRewritePattern<StridedSliceOp>::OpRewritePattern;
PatternMatchResult matchAndRewrite(StridedSliceOp stridedSliceOp, PatternMatchResult matchAndRewrite(StridedSliceOp stridedSliceOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
// Return if 'stridedSliceOp' operand is not defined by a ConstantMaskOp. // Return if 'stridedSliceOp' operand is not defined by a ConstantMaskOp.
auto defOp = stridedSliceOp.vector()->getDefiningOp(); auto defOp = stridedSliceOp.vector().getDefiningOp();
auto constantMaskOp = dyn_cast_or_null<ConstantMaskOp>(defOp); auto constantMaskOp = dyn_cast_or_null<ConstantMaskOp>(defOp);
if (!constantMaskOp) if (!constantMaskOp)
return matchFailure(); return matchFailure();
// Return if 'stridedSliceOp' has non-unit strides. // Return if 'stridedSliceOp' has non-unit strides.
if (llvm::any_of(stridedSliceOp.strides(), [](Attribute attr) { if (llvm::any_of(stridedSliceOp.strides(), [](Attribute attr) {
return attr.cast<IntegerAttr>().getInt() != 1; return attr.cast<IntegerAttr>().getInt() != 1;
})) }))
return matchFailure(); return matchFailure();
Show All 20 Lines PatternMatchResult matchAndRewrite(StridedSliceOp stridedSliceOp,
} }
// If any of 'sliceMaskDimSizes' are zero, then set all to zero (masked // If any of 'sliceMaskDimSizes' are zero, then set all to zero (masked
// region is a conjunction of mask dim intervals). // region is a conjunction of mask dim intervals).
if (llvm::any_of(sliceMaskDimSizes, [](int64_t sz) { return sz == 0; })) if (llvm::any_of(sliceMaskDimSizes, [](int64_t sz) { return sz == 0; }))
sliceMaskDimSizes.assign(maskDimSizes.size(), 0); sliceMaskDimSizes.assign(maskDimSizes.size(), 0);
// Replace 'stridedSliceOp' with ConstantMaskOp with sliced mask region. // Replace 'stridedSliceOp' with ConstantMaskOp with sliced mask region.
rewriter.replaceOpWithNewOp<ConstantMaskOp>( rewriter.replaceOpWithNewOp<ConstantMaskOp>(
stridedSliceOp, stridedSliceOp.getResult()->getType(), stridedSliceOp, stridedSliceOp.getResult().getType(),
vector::getVectorSubscriptAttr(rewriter, sliceMaskDimSizes)); vector::getVectorSubscriptAttr(rewriter, sliceMaskDimSizes));
return matchSuccess(); return matchSuccess();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void StridedSliceOp::getCanonicalizationPatterns( void StridedSliceOp::getCanonicalizationPatterns(
▲ Show 20 Lines • Show All 121 Lines • ▼ Show 20 Lines return failure(
result.operands) || result.operands) ||
parser.addTypeToList(vectorType, result.types)); parser.addTypeToList(vectorType, result.types));
} }
static LogicalResult verify(TransferReadOp op) { static LogicalResult verify(TransferReadOp op) {
// Consistency of elemental types in memref and vector. // Consistency of elemental types in memref and vector.
MemRefType memrefType = op.getMemRefType(); MemRefType memrefType = op.getMemRefType();
VectorType vectorType = op.getVectorType(); VectorType vectorType = op.getVectorType();
auto paddingType = op.padding()->getType(); auto paddingType = op.padding().getType();
auto permutationMap = op.permutation_map(); auto permutationMap = op.permutation_map();
auto memrefElementType = memrefType.getElementType(); auto memrefElementType = memrefType.getElementType();
if (static_cast<int64_t>(op.indices().size()) != memrefType.getRank()) if (static_cast<int64_t>(op.indices().size()) != memrefType.getRank())
return op.emitOpError("requires ") << memrefType.getRank() << " indices"; return op.emitOpError("requires ") << memrefType.getRank() << " indices";
if (failed(verifyTransferOp(op.getOperation(), memrefType, vectorType, if (failed(verifyTransferOp(op.getOperation(), memrefType, vectorType,
permutationMap))) permutationMap)))
Show All 20 Linesstatic LogicalResult verify(TransferReadOp op) {
return verifyPermutationMap(permutationMap, return verifyPermutationMap(permutationMap,
[&op](Twine t) { return op.emitOpError(t); }); [&op](Twine t) { return op.emitOpError(t); });
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TransferWriteOp // TransferWriteOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, TransferWriteOp op) { static void print(OpAsmPrinter &p, TransferWriteOp op) {
p << op.getOperationName() << " " << *op.vector() << ", " << *op.memref() p << op.getOperationName() << " " << op.vector() << ", " << op.memref() << "["
<< "[" << op.indices() << "]"; << op.indices() << "]";
p.printOptionalAttrDict(op.getAttrs()); p.printOptionalAttrDict(op.getAttrs());
p << " : " << op.getVectorType() << ", " << op.getMemRefType(); p << " : " << op.getVectorType() << ", " << op.getMemRefType();
} }
ParseResult parseTransferWriteOp(OpAsmParser &parser, OperationState &result) { ParseResult parseTransferWriteOp(OpAsmParser &parser, OperationState &result) {
llvm::SMLoc typesLoc; llvm::SMLoc typesLoc;
OpAsmParser::OperandType storeValueInfo; OpAsmParser::OperandType storeValueInfo;
OpAsmParser::OperandType memRefInfo; OpAsmParser::OperandType memRefInfo;
Show All 38 Lines
static MemRefType inferVectorTypeCastResultType(MemRefType t) { static MemRefType inferVectorTypeCastResultType(MemRefType t) {
return MemRefType::get({}, VectorType::get(t.getShape(), t.getElementType())); return MemRefType::get({}, VectorType::get(t.getShape(), t.getElementType()));
} }
void TypeCastOp::build(Builder *builder, OperationState &result, Value source) { void TypeCastOp::build(Builder *builder, OperationState &result, Value source) {
result.addOperands(source); result.addOperands(source);
result.addTypes( result.addTypes(
inferVectorTypeCastResultType(source->getType().cast<MemRefType>())); inferVectorTypeCastResultType(source.getType().cast<MemRefType>()));
} }
static void print(OpAsmPrinter &p, TypeCastOp op) { static void print(OpAsmPrinter &p, TypeCastOp op) {
auto type = op.getOperand()->getType().cast<MemRefType>(); auto type = op.getOperand().getType().cast<MemRefType>();
p << op.getOperationName() << ' ' << *op.memref() << " : " << type << " to " p << op.getOperationName() << ' ' << op.memref() << " : " << type << " to "
<< inferVectorTypeCastResultType(type); << inferVectorTypeCastResultType(type);
} }
static LogicalResult verify(TypeCastOp op) { static LogicalResult verify(TypeCastOp op) {
auto resultType = inferVectorTypeCastResultType(op.getMemRefType()); auto resultType = inferVectorTypeCastResultType(op.getMemRefType());
if (op.getResultMemRefType() != resultType) if (op.getResultMemRefType() != resultType)
return op.emitOpError("expects result type to be: ") << resultType; return op.emitOpError("expects result type to be: ") << resultType;
return success(); return success();
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Linesstatic ParseResult parseTupleGetOp(OpAsmParser &parser,
if (indexAttr.getInt() < 0 || if (indexAttr.getInt() < 0 ||
indexAttr.getInt() >= static_cast<int64_t>(tupleType.size())) indexAttr.getInt() >= static_cast<int64_t>(tupleType.size()))
return failure(); return failure();
parser.addTypeToList(tupleType.getType(indexAttr.getInt()), result.types); parser.addTypeToList(tupleType.getType(indexAttr.getInt()), result.types);
return success(); return success();
} }
static void print(OpAsmPrinter &p, TupleGetOp op) { static void print(OpAsmPrinter &p, TupleGetOp op) {
p << op.getOperationName() << ' ' << *op.getOperand() << ", " << op.index(); p << op.getOperationName() << ' ' << op.getOperand() << ", " << op.index();
p.printOptionalAttrDict(op.getAttrs(), p.printOptionalAttrDict(op.getAttrs(),
/*elidedAttrs=*/{TupleGetOp::getIndexAttrName()}); /*elidedAttrs=*/{TupleGetOp::getIndexAttrName()});
p << " : " << op.getOperand()->getType(); p << " : " << op.getOperand().getType();
} }
static LogicalResult verify(TupleGetOp op) { static LogicalResult verify(TupleGetOp op) {
auto tupleType = op.getOperand()->getType().cast<TupleType>(); auto tupleType = op.getOperand().getType().cast<TupleType>();
if (op.getIndex() < 0 || if (op.getIndex() < 0 ||
op.getIndex() >= static_cast<int64_t>(tupleType.size())) op.getIndex() >= static_cast<int64_t>(tupleType.size()))
return op.emitOpError("tuple get index out of range"); return op.emitOpError("tuple get index out of range");
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ConstantMaskOp // ConstantMaskOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
ParseResult parseConstantMaskOp(OpAsmParser &parser, OperationState &result) { ParseResult parseConstantMaskOp(OpAsmParser &parser, OperationState &result) {
Type resultType; Type resultType;
ArrayAttr maskDimSizesAttr; ArrayAttr maskDimSizesAttr;
StringRef attrName = ConstantMaskOp::getMaskDimSizesAttrName(); StringRef attrName = ConstantMaskOp::getMaskDimSizesAttrName();
return failure( return failure(
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseAttribute(maskDimSizesAttr, attrName, result.attributes) || parser.parseAttribute(maskDimSizesAttr, attrName, result.attributes) ||
parser.parseColonType(resultType) || parser.parseColonType(resultType) ||
parser.addTypeToList(resultType, result.types)); parser.addTypeToList(resultType, result.types));
} }
static void print(OpAsmPrinter &p, ConstantMaskOp op) { static void print(OpAsmPrinter &p, ConstantMaskOp op) {
p << op.getOperationName() << ' ' << op.mask_dim_sizes() << " : " p << op.getOperationName() << ' ' << op.mask_dim_sizes() << " : "
<< op.getResult()->getType(); << op.getResult().getType();
} }
static LogicalResult verify(ConstantMaskOp &op) { static LogicalResult verify(ConstantMaskOp &op) {
// Verify that array attr size matches the rank of the vector result. // Verify that array attr size matches the rank of the vector result.
auto resultType = op.getResult()->getType().cast<VectorType>(); auto resultType = op.getResult().getType().cast<VectorType>();
if (static_cast<int64_t>(op.mask_dim_sizes().size()) != resultType.getRank()) if (static_cast<int64_t>(op.mask_dim_sizes().size()) != resultType.getRank())
return op.emitOpError( return op.emitOpError(
"must specify array attr of size equal vector result rank"); "must specify array attr of size equal vector result rank");
// Verify that each array attr element is in bounds of corresponding vector // Verify that each array attr element is in bounds of corresponding vector
// result dimension size. // result dimension size.
auto resultShape = resultType.getShape(); auto resultShape = resultType.getShape();
SmallVector<int64_t, 4> maskDimSizes; SmallVector<int64_t, 4> maskDimSizes;
for (auto it : llvm::enumerate(op.mask_dim_sizes())) { for (auto it : llvm::enumerate(op.mask_dim_sizes())) {
Show All 31 Lines
static void print(OpAsmPrinter &p, CreateMaskOp op) { static void print(OpAsmPrinter &p, CreateMaskOp op) {
p << op.getOperationName() << ' ' << op.operands() << " : " << op.getType(); p << op.getOperationName() << ' ' << op.operands() << " : " << op.getType();
} }
static LogicalResult verify(CreateMaskOp op) { static LogicalResult verify(CreateMaskOp op) {
// Verify that an operand was specified for each result vector each dimension. // Verify that an operand was specified for each result vector each dimension.
if (op.getNumOperands() != if (op.getNumOperands() !=
op.getResult()->getType().cast<VectorType>().getRank()) op.getResult().getType().cast<VectorType>().getRank())
return op.emitOpError( return op.emitOpError(
"must specify an operand for each result vector dimension"); "must specify an operand for each result vector dimension");
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// PrintOp // PrintOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
ParseResult parsePrintOp(OpAsmParser &parser, OperationState &result) { ParseResult parsePrintOp(OpAsmParser &parser, OperationState &result) {
OpAsmParser::OperandType source; OpAsmParser::OperandType source;
Type sourceType; Type sourceType;
return failure(parser.parseOperand(source) || return failure(parser.parseOperand(source) ||
parser.parseColonType(sourceType) || parser.parseColonType(sourceType) ||
parser.resolveOperand(source, sourceType, result.operands)); parser.resolveOperand(source, sourceType, result.operands));
} }
static void print(OpAsmPrinter &p, PrintOp op) { static void print(OpAsmPrinter &p, PrintOp op) {
p << op.getOperationName() << ' ' << *op.source() << " : " p << op.getOperationName() << ' ' << op.source() << " : "
<< op.getPrintType(); << op.getPrintType();
} }
namespace { namespace {
// Pattern to rewrite a CreateMaskOp with a ConstantMaskOp. // Pattern to rewrite a CreateMaskOp with a ConstantMaskOp.
class CreateMaskFolder final : public OpRewritePattern<CreateMaskOp> { class CreateMaskFolder final : public OpRewritePattern<CreateMaskOp> {
public: public:
using OpRewritePattern<CreateMaskOp>::OpRewritePattern; using OpRewritePattern<CreateMaskOp>::OpRewritePattern;
PatternMatchResult matchAndRewrite(CreateMaskOp createMaskOp, PatternMatchResult matchAndRewrite(CreateMaskOp createMaskOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
// Return if any of 'createMaskOp' operands are not defined by a constant. // Return if any of 'createMaskOp' operands are not defined by a constant.
auto is_not_def_by_constant = [](Value operand) { auto is_not_def_by_constant = [](Value operand) {
return !isa_and_nonnull<ConstantIndexOp>(operand->getDefiningOp()); return !isa_and_nonnull<ConstantIndexOp>(operand.getDefiningOp());
}; };
if (llvm::any_of(createMaskOp.operands(), is_not_def_by_constant)) if (llvm::any_of(createMaskOp.operands(), is_not_def_by_constant))
return matchFailure(); return matchFailure();
// Gather constant mask dimension sizes. // Gather constant mask dimension sizes.
SmallVector<int64_t, 4> maskDimSizes; SmallVector<int64_t, 4> maskDimSizes;
for (auto operand : createMaskOp.operands()) { for (auto operand : createMaskOp.operands()) {
auto defOp = operand->getDefiningOp(); auto defOp = operand.getDefiningOp();
maskDimSizes.push_back(cast<ConstantIndexOp>(defOp).getValue()); maskDimSizes.push_back(cast<ConstantIndexOp>(defOp).getValue());
} }
// Replace 'createMaskOp' with ConstantMaskOp. // Replace 'createMaskOp' with ConstantMaskOp.
rewriter.replaceOpWithNewOp<ConstantMaskOp>( rewriter.replaceOpWithNewOp<ConstantMaskOp>(
createMaskOp, createMaskOp.getResult()->getType(), createMaskOp, createMaskOp.getResult().getType(),
vector::getVectorSubscriptAttr(rewriter, maskDimSizes)); vector::getVectorSubscriptAttr(rewriter, maskDimSizes));
return matchSuccess(); return matchSuccess();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void CreateMaskOp::getCanonicalizationPatterns( void CreateMaskOp::getCanonicalizationPatterns(
Show All 17 Lines

mlir/lib/Dialect/VectorOps/VectorTransforms.cpp

Show First 20 Lines • Show All 189 Lines • ▼ Show 20 Linesstatic void initUnrolledVectorState(VectorType vectorType, Value initValue,
state.slicesTuple = nullptr; state.slicesTuple = nullptr;
if (initValue != nullptr) { if (initValue != nullptr) {
// Create ExtractSlicesOp. // Create ExtractSlicesOp.
SmallVector<int64_t, 4> sizes(state.unrolledShape); SmallVector<int64_t, 4> sizes(state.unrolledShape);
SmallVector<int64_t, 4> strides(state.unrollFactors.size(), 1); SmallVector<int64_t, 4> strides(state.unrollFactors.size(), 1);
auto tupleType = auto tupleType =
generateExtractSlicesOpResultType(vectorType, sizes, strides, builder); generateExtractSlicesOpResultType(vectorType, sizes, strides, builder);
state.slicesTuple = builder.create<vector::ExtractSlicesOp>( state.slicesTuple = builder.create<vector::ExtractSlicesOp>(
initValue->getLoc(), tupleType, initValue, sizes, strides); initValue.getLoc(), tupleType, initValue, sizes, strides);
} }
} }
// Computes and returns the linear index of the unrolled vector at // Computes and returns the linear index of the unrolled vector at
// 'vectorOffsets' within the vector represented by 'state'. // 'vectorOffsets' within the vector represented by 'state'.
static int64_t static int64_t
getUnrolledVectorLinearIndex(UnrolledVectorState &state, getUnrolledVectorLinearIndex(UnrolledVectorState &state,
ArrayRef<int64_t> vectorOffsets, ArrayRef<int64_t> vectorOffsets,
Show All 20 Linesstatic Value getOrCreateUnrolledVectorSlice(
// Compute linear index of 'sliceOffsets' w.r.t 'state.basis'. // Compute linear index of 'sliceOffsets' w.r.t 'state.basis'.
int64_t sliceLinearIndex = int64_t sliceLinearIndex =
getUnrolledVectorLinearIndex(state, vectorOffsets, indexMap); getUnrolledVectorLinearIndex(state, vectorOffsets, indexMap);
assert(sliceLinearIndex < static_cast<int64_t>(cache.size())); assert(sliceLinearIndex < static_cast<int64_t>(cache.size()));
auto valueSlice = cache[sliceLinearIndex]; auto valueSlice = cache[sliceLinearIndex];
if (valueSlice == nullptr) { if (valueSlice == nullptr) {
// Return tuple element at 'sliceLinearIndex'. // Return tuple element at 'sliceLinearIndex'.
auto tupleIndex = builder.getI64IntegerAttr(sliceLinearIndex); auto tupleIndex = builder.getI64IntegerAttr(sliceLinearIndex);
auto initValueType = initValue->getType().cast<VectorType>(); auto initValueType = initValue.getType().cast<VectorType>();
auto vectorType = auto vectorType =
VectorType::get(state.unrolledShape, initValueType.getElementType()); VectorType::get(state.unrolledShape, initValueType.getElementType());
// Initialize 'cache' with slice from 'initValue'. // Initialize 'cache' with slice from 'initValue'.
valueSlice = builder.create<vector::TupleGetOp>( valueSlice = builder.create<vector::TupleGetOp>(
loc, vectorType, state.slicesTuple, tupleIndex); loc, vectorType, state.slicesTuple, tupleIndex);
// Store value back to 'cache'. // Store value back to 'cache'.
cache[sliceLinearIndex] = valueSlice; cache[sliceLinearIndex] = valueSlice;
} }
▲ Show 20 Lines • Show All 62 Lines • ▼ Show 20 Lines
// TODO(andydavis) Generalize this to support structured ops beyond // TODO(andydavis) Generalize this to support structured ops beyond
// vector ContractionOp, and merge it with 'unrollSingleResultOpMatchingType' // vector ContractionOp, and merge it with 'unrollSingleResultOpMatchingType'
static Value unrollSingleResultStructuredOp(Operation *op, static Value unrollSingleResultStructuredOp(Operation *op,
ArrayRef<int64_t> iterationBounds, ArrayRef<int64_t> iterationBounds,
std::vector<VectorState> &vectors, std::vector<VectorState> &vectors,
unsigned resultIndex, unsigned resultIndex,
ArrayRef<int64_t> targetShape, ArrayRef<int64_t> targetShape,
PatternRewriter &builder) { PatternRewriter &builder) {
auto shapedType = op->getResult(0)->getType().dyn_cast_or_null<ShapedType>(); auto shapedType = op->getResult(0).getType().dyn_cast_or_null<ShapedType>();
if (!shapedType || !shapedType.hasStaticShape()) if (!shapedType || !shapedType.hasStaticShape())
assert(false && "Expected a statically shaped result type"); assert(false && "Expected a statically shaped result type");
// Compute unroll factors for 'iterationBounds' based on 'targetShape' // Compute unroll factors for 'iterationBounds' based on 'targetShape'
auto maybeUnrollFactors = shapeRatio(iterationBounds, targetShape); auto maybeUnrollFactors = shapeRatio(iterationBounds, targetShape);
if (!maybeUnrollFactors.hasValue()) if (!maybeUnrollFactors.hasValue())
assert(false && "Failed to compute unroll factors for target shape"); assert(false && "Failed to compute unroll factors for target shape");
auto unrollFactors = *maybeUnrollFactors; auto unrollFactors = *maybeUnrollFactors;
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines for (unsigned i = 0; i < numUnrolledInstances; ++i) {
// Update result cache at 'linearIndex'. // Update result cache at 'linearIndex'.
caches[resultIndex][linearIndex] = resultVector; caches[resultIndex][linearIndex] = resultVector;
} }
// Create TupleOp of unrolled result vectors. // Create TupleOp of unrolled result vectors.
SmallVector<Type, 4> vectorTupleTypes(resultValueState.numInstances); SmallVector<Type, 4> vectorTupleTypes(resultValueState.numInstances);
SmallVector<Value, 4> vectorTupleValues(resultValueState.numInstances); SmallVector<Value, 4> vectorTupleValues(resultValueState.numInstances);
for (unsigned i = 0; i < resultValueState.numInstances; ++i) { for (unsigned i = 0; i < resultValueState.numInstances; ++i) {
vectorTupleTypes[i] = caches[resultIndex][i]->getType().cast<VectorType>(); vectorTupleTypes[i] = caches[resultIndex][i].getType().cast<VectorType>();
vectorTupleValues[i] = caches[resultIndex][i]; vectorTupleValues[i] = caches[resultIndex][i];
} }
TupleType tupleType = builder.getTupleType(vectorTupleTypes); TupleType tupleType = builder.getTupleType(vectorTupleTypes);
Value tupleOp = builder.create<vector::TupleOp>(op->getLoc(), tupleType, Value tupleOp = builder.create<vector::TupleOp>(op->getLoc(), tupleType,
vectorTupleValues); vectorTupleValues);
// Create InsertSlicesOp(Tuple(result_vectors)). // Create InsertSlicesOp(Tuple(result_vectors)).
auto resultVectorType = op->getResult(0)->getType().cast<VectorType>(); auto resultVectorType = op->getResult(0).getType().cast<VectorType>();
SmallVector<int64_t, 4> sizes(resultValueState.unrolledShape); SmallVector<int64_t, 4> sizes(resultValueState.unrolledShape);
SmallVector<int64_t, 4> strides(resultValueState.unrollFactors.size(), 1); SmallVector<int64_t, 4> strides(resultValueState.unrollFactors.size(), 1);
Value insertSlicesOp = builder.create<vector::InsertSlicesOp>( Value insertSlicesOp = builder.create<vector::InsertSlicesOp>(
op->getLoc(), resultVectorType, tupleOp, builder.getI64ArrayAttr(sizes), op->getLoc(), resultVectorType, tupleOp, builder.getI64ArrayAttr(sizes),
builder.getI64ArrayAttr(strides)); builder.getI64ArrayAttr(strides));
return insertSlicesOp; return insertSlicesOp;
} }
static void getVectorContractionOpUnrollState( static void getVectorContractionOpUnrollState(
vector::ContractionOp contractionOp, ArrayRef<int64_t> targetShape, vector::ContractionOp contractionOp, ArrayRef<int64_t> targetShape,
SmallVectorImpl<int64_t> &iterationBounds, SmallVectorImpl<int64_t> &iterationBounds,
std::vector<VectorState> &vectors, unsigned &resultIndex) { std::vector<VectorState> &vectors, unsigned &resultIndex) {
// Get contraction op iteration bounds. // Get contraction op iteration bounds.
contractionOp.getIterationBounds(iterationBounds); contractionOp.getIterationBounds(iterationBounds);
assert(iterationBounds.size() == targetShape.size()); assert(iterationBounds.size() == targetShape.size());
// Get map from iteration space index to lhs/rhs/result shape index. // Get map from iteration space index to lhs/rhs/result shape index.
std::vector<DenseMap<int64_t, int64_t>> iterationIndexMapList; std::vector<DenseMap<int64_t, int64_t>> iterationIndexMapList;
contractionOp.getIterationIndexMap(iterationIndexMapList); contractionOp.getIterationIndexMap(iterationIndexMapList);
unsigned numIterators = iterationIndexMapList.size(); unsigned numIterators = iterationIndexMapList.size();
vectors.resize(numIterators); vectors.resize(numIterators);
unsigned accOperandIndex = vector::ContractionOp::getAccOperandIndex(); unsigned accOperandIndex = vector::ContractionOp::getAccOperandIndex();
for (unsigned i = 0; i < numIterators; ++i) { for (unsigned i = 0; i < numIterators; ++i) {
vectors[i].type = contractionOp.getOperand(i)->getType().cast<VectorType>(); vectors[i].type = contractionOp.getOperand(i).getType().cast<VectorType>();
vectors[i].indexMap = iterationIndexMapList[i]; vectors[i].indexMap = iterationIndexMapList[i];
vectors[i].operandIndex = i; vectors[i].operandIndex = i;
vectors[i].isAcc = i == accOperandIndex ? true : false; vectors[i].isAcc = i == accOperandIndex ? true : false;
} }
if (llvm::size(contractionOp.masks()) == 2) { if (llvm::size(contractionOp.masks()) == 2) {
// Add vectors for lhs/rhs vector mask arguments. Masks have the // Add vectors for lhs/rhs vector mask arguments. Masks have the
// same vector shape lhs/rhs args, so copy their index maps. // same vector shape lhs/rhs args, so copy their index maps.
Show All 9 Lines
} }
static void static void
getVectorElementwiseOpUnrollState(Operation *op, ArrayRef<int64_t> targetShape, getVectorElementwiseOpUnrollState(Operation *op, ArrayRef<int64_t> targetShape,
SmallVectorImpl<int64_t> &iterationBounds, SmallVectorImpl<int64_t> &iterationBounds,
std::vector<VectorState> &vectors, std::vector<VectorState> &vectors,
unsigned &resultIndex) { unsigned &resultIndex) {
// Verify that operation and operands all have the same vector shape. // Verify that operation and operands all have the same vector shape.
auto resultType = op->getResult(0)->getType().dyn_cast_or_null<VectorType>(); auto resultType = op->getResult(0).getType().dyn_cast_or_null<VectorType>();
assert(resultType && "Expected op with vector result type"); assert(resultType && "Expected op with vector result type");
auto resultShape = resultType.getShape(); auto resultShape = resultType.getShape();
// Verify that all operands have the same vector type as result. // Verify that all operands have the same vector type as result.
assert(llvm::all_of(op->getOperandTypes(), assert(llvm::all_of(op->getOperandTypes(),
[=](Type type) { return type == resultType; })); [=](Type type) { return type == resultType; }));
// Populate 'iterationBounds' with 'resultShape' for elementwise operations. // Populate 'iterationBounds' with 'resultShape' for elementwise operations.
iterationBounds.assign(resultShape.begin(), resultShape.end()); iterationBounds.assign(resultShape.begin(), resultShape.end());
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines auto offsets = zipMap([](int64_t v1, int64_t v2) { return v1 * v2; },
vectorOffsets, sizes); vectorOffsets, sizes);
// Compute 'sliceIndices' by adding 'sliceOffsets[i]' to 'indices[i]'. // Compute 'sliceIndices' by adding 'sliceOffsets[i]' to 'indices[i]'.
SmallVector<Value, 4> sliceIndices(numSliceIndices); SmallVector<Value, 4> sliceIndices(numSliceIndices);
for (auto it : llvm::enumerate(indices)) { for (auto it : llvm::enumerate(indices)) {
auto expr = getAffineDimExpr(0, ctx) + auto expr = getAffineDimExpr(0, ctx) +
getAffineConstantExpr(offsets[it.index()], ctx); getAffineConstantExpr(offsets[it.index()], ctx);
auto map = AffineMap::get(/*dimCount=*/1, /*symbolCount=*/0, expr); auto map = AffineMap::get(/*dimCount=*/1, /*symbolCount=*/0, expr);
sliceIndices[it.index()] = rewriter.create<AffineApplyOp>( sliceIndices[it.index()] = rewriter.create<AffineApplyOp>(
it.value()->getLoc(), map, ArrayRef<Value>(it.value())); it.value().getLoc(), map, ArrayRef<Value>(it.value()));
} }
// Call 'fn' to generate slice 'i' at 'sliceIndices'. // Call 'fn' to generate slice 'i' at 'sliceIndices'.
fn(i, sliceIndices); fn(i, sliceIndices);
} }
} }
// Splits vector TransferReadOp into smaller TransferReadOps based on slicing // Splits vector TransferReadOp into smaller TransferReadOps based on slicing
// scheme of its unique ExtractSlicesOp user. // scheme of its unique ExtractSlicesOp user.
struct SplitTransferReadOp : public OpRewritePattern<vector::TransferReadOp> { struct SplitTransferReadOp : public OpRewritePattern<vector::TransferReadOp> {
using OpRewritePattern<vector::TransferReadOp>::OpRewritePattern; using OpRewritePattern<vector::TransferReadOp>::OpRewritePattern;
PatternMatchResult matchAndRewrite(vector::TransferReadOp xferReadOp, PatternMatchResult matchAndRewrite(vector::TransferReadOp xferReadOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
// TODO(andydavis, ntv) Support spliting TransferReadOp with non-identity // TODO(andydavis, ntv) Support spliting TransferReadOp with non-identity
// permutation maps. Repurpose code from MaterializeVectors transformation. // permutation maps. Repurpose code from MaterializeVectors transformation.
if (!xferReadOp.permutation_map().isIdentity()) if (!xferReadOp.permutation_map().isIdentity())
return matchFailure(); return matchFailure();
// Return unless the unique 'xferReadOp' user is an ExtractSlicesOp. // Return unless the unique 'xferReadOp' user is an ExtractSlicesOp.
Value xferReadResult = xferReadOp.getResult(); Value xferReadResult = xferReadOp.getResult();
auto extractSlicesOp = auto extractSlicesOp =
dyn_cast<vector::ExtractSlicesOp>(*xferReadResult->getUsers().begin()); dyn_cast<vector::ExtractSlicesOp>(*xferReadResult.getUsers().begin());
if (!xferReadResult->hasOneUse() || !extractSlicesOp) if (!xferReadResult.hasOneUse() || !extractSlicesOp)
return matchFailure(); return matchFailure();
// Get 'sizes' and 'strides' parameters from ExtractSlicesOp user. // Get 'sizes' and 'strides' parameters from ExtractSlicesOp user.
auto sourceVectorType = extractSlicesOp.getSourceVectorType(); auto sourceVectorType = extractSlicesOp.getSourceVectorType();
auto resultTupleType = extractSlicesOp.getResultTupleType(); auto resultTupleType = extractSlicesOp.getResultTupleType();
SmallVector<int64_t, 4> sizes; SmallVector<int64_t, 4> sizes;
extractSlicesOp.getSizes(sizes); extractSlicesOp.getSizes(sizes);
SmallVector<int64_t, 4> strides; SmallVector<int64_t, 4> strides;
Show All 33 Linesstruct SplitTransferWriteOp : public OpRewritePattern<vector::TransferWriteOp> {
PatternMatchResult matchAndRewrite(vector::TransferWriteOp xferWriteOp, PatternMatchResult matchAndRewrite(vector::TransferWriteOp xferWriteOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
// TODO(andydavis, ntv) Support spliting TransferWriteOp with non-identity // TODO(andydavis, ntv) Support spliting TransferWriteOp with non-identity
// permutation maps. Repurpose code from MaterializeVectors transformation. // permutation maps. Repurpose code from MaterializeVectors transformation.
if (!xferWriteOp.permutation_map().isIdentity()) if (!xferWriteOp.permutation_map().isIdentity())
return matchFailure(); return matchFailure();
// Return unless the 'xferWriteOp' 'vector' operand is an 'InsertSlicesOp'. // Return unless the 'xferWriteOp' 'vector' operand is an 'InsertSlicesOp'.
auto *vectorDefOp = xferWriteOp.vector()->getDefiningOp(); auto *vectorDefOp = xferWriteOp.vector().getDefiningOp();
auto insertSlicesOp = dyn_cast_or_null<vector::InsertSlicesOp>(vectorDefOp); auto insertSlicesOp = dyn_cast_or_null<vector::InsertSlicesOp>(vectorDefOp);
if (!insertSlicesOp) if (!insertSlicesOp)
return matchFailure(); return matchFailure();
// Get TupleOp operand of 'insertSlicesOp'. // Get TupleOp operand of 'insertSlicesOp'.
auto tupleOp = dyn_cast_or_null<vector::TupleOp>( auto tupleOp = dyn_cast_or_null<vector::TupleOp>(
insertSlicesOp.vectors()->getDefiningOp()); insertSlicesOp.vectors().getDefiningOp());
if (!tupleOp) if (!tupleOp)
return matchFailure(); return matchFailure();
// Get 'sizes' and 'strides' parameters from InsertSlicesOp user. // Get 'sizes' and 'strides' parameters from InsertSlicesOp user.
auto sourceTupleType = insertSlicesOp.getSourceTupleType(); auto sourceTupleType = insertSlicesOp.getSourceTupleType();
auto resultVectorType = insertSlicesOp.getResultVectorType(); auto resultVectorType = insertSlicesOp.getResultVectorType();
SmallVector<int64_t, 4> sizes; SmallVector<int64_t, 4> sizes;
insertSlicesOp.getSizes(sizes); insertSlicesOp.getSizes(sizes);
Show All 23 Lines
// -> User(Producer) // -> User(Producer)
struct TupleGetFolderOp : public OpRewritePattern<vector::TupleGetOp> { struct TupleGetFolderOp : public OpRewritePattern<vector::TupleGetOp> {
using OpRewritePattern<vector::TupleGetOp>::OpRewritePattern; using OpRewritePattern<vector::TupleGetOp>::OpRewritePattern;
PatternMatchResult matchAndRewrite(vector::TupleGetOp tupleGetOp, PatternMatchResult matchAndRewrite(vector::TupleGetOp tupleGetOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
// Return if 'tupleGetOp.vectors' arg was not defined by ExtractSlicesOp. // Return if 'tupleGetOp.vectors' arg was not defined by ExtractSlicesOp.
auto extractSlicesOp = dyn_cast_or_null<vector::ExtractSlicesOp>( auto extractSlicesOp = dyn_cast_or_null<vector::ExtractSlicesOp>(
tupleGetOp.vectors()->getDefiningOp()); tupleGetOp.vectors().getDefiningOp());
if (!extractSlicesOp) if (!extractSlicesOp)
return matchFailure(); return matchFailure();
// Return if 'extractSlicesOp.vector' arg was not defined by InsertSlicesOp. // Return if 'extractSlicesOp.vector' arg was not defined by InsertSlicesOp.
auto insertSlicesOp = dyn_cast_or_null<vector::InsertSlicesOp>( auto insertSlicesOp = dyn_cast_or_null<vector::InsertSlicesOp>(
extractSlicesOp.vector()->getDefiningOp()); extractSlicesOp.vector().getDefiningOp());
if (!insertSlicesOp) if (!insertSlicesOp)
return matchFailure(); return matchFailure();
// Return if 'insertSlicesOp.vectors' arg was not defined by TupleOp. // Return if 'insertSlicesOp.vectors' arg was not defined by TupleOp.
auto tupleOp = dyn_cast_or_null<vector::TupleOp>( auto tupleOp = dyn_cast_or_null<vector::TupleOp>(
insertSlicesOp.vectors()->getDefiningOp()); insertSlicesOp.vectors().getDefiningOp());
if (!tupleOp) if (!tupleOp)
return matchFailure(); return matchFailure();
// Forward Value from 'tupleOp' at 'tupleGetOp.index'. // Forward Value from 'tupleOp' at 'tupleGetOp.index'.
Value tupleValue = tupleOp.getOperand(tupleGetOp.getIndex()); Value tupleValue = tupleOp.getOperand(tupleGetOp.getIndex());
rewriter.replaceOp(tupleGetOp, tupleValue); rewriter.replaceOp(tupleGetOp, tupleValue);
return matchSuccess(); return matchSuccess();
} }
Show All 9 Lines

mlir/lib/EDSC/Builders.cpp

Show First 20 Lines • Show All 63 Lines • ▼ Show 20 Lines
MLIRContext *mlir::edsc::ScopedContext::getContext() { MLIRContext *mlir::edsc::ScopedContext::getContext() {
return getBuilder().getContext(); return getBuilder().getContext();
} }
mlir::edsc::ValueHandle::ValueHandle(index_t cst) { mlir::edsc::ValueHandle::ValueHandle(index_t cst) {
auto &b = ScopedContext::getBuilder(); auto &b = ScopedContext::getBuilder();
auto loc = ScopedContext::getLocation(); auto loc = ScopedContext::getLocation();
v = b.create<ConstantIndexOp>(loc, cst.v).getResult(); v = b.create<ConstantIndexOp>(loc, cst.v).getResult();
t = v->getType(); t = v.getType();
} }
ValueHandle &mlir::edsc::ValueHandle::operator=(const ValueHandle &other) { ValueHandle &mlir::edsc::ValueHandle::operator=(const ValueHandle &other) {
assert(t == other.t && "Wrong type capture"); assert(t == other.t && "Wrong type capture");
assert(!v && "ValueHandle has already been captured, use a new name!"); assert(!v && "ValueHandle has already been captured, use a new name!");
v = other.v; v = other.v;
return *this; return *this;
} }
▲ Show 20 Lines • Show All 53 Lines • ▼ Show 20 Lines
} }
static Optional<ValueHandle> emitStaticFor(ArrayRef<ValueHandle> lbs, static Optional<ValueHandle> emitStaticFor(ArrayRef<ValueHandle> lbs,
ArrayRef<ValueHandle> ubs, ArrayRef<ValueHandle> ubs,
int64_t step) { int64_t step) {
if (lbs.size() != 1 || ubs.size() != 1) if (lbs.size() != 1 || ubs.size() != 1)
return Optional<ValueHandle>(); return Optional<ValueHandle>();
auto *lbDef = lbs.front().getValue()->getDefiningOp(); auto *lbDef = lbs.front().getValue().getDefiningOp();
auto *ubDef = ubs.front().getValue()->getDefiningOp(); auto *ubDef = ubs.front().getValue().getDefiningOp();
if (!lbDef || !ubDef) if (!lbDef || !ubDef)
return Optional<ValueHandle>(); return Optional<ValueHandle>();
auto lbConst = dyn_cast<ConstantIndexOp>(lbDef); auto lbConst = dyn_cast<ConstantIndexOp>(lbDef);
auto ubConst = dyn_cast<ConstantIndexOp>(ubDef); auto ubConst = dyn_cast<ConstantIndexOp>(ubDef);
if (!lbConst || !ubConst) if (!lbConst || !ubConst)
return Optional<ValueHandle>(); return Optional<ValueHandle>();
▲ Show 20 Lines • Show All 148 Lines • ▼ Show 20 Linesstatic ValueHandle createBinaryHandle(ValueHandle lhs, ValueHandle rhs) {
return ValueHandle::create<Op>(lhs.getValue(), rhs.getValue()); return ValueHandle::create<Op>(lhs.getValue(), rhs.getValue());
} }
static std::pair<AffineExpr, Value> static std::pair<AffineExpr, Value>
categorizeValueByAffineType(MLIRContext *context, Value val, unsigned &numDims, categorizeValueByAffineType(MLIRContext *context, Value val, unsigned &numDims,
unsigned &numSymbols) { unsigned &numSymbols) {
AffineExpr d; AffineExpr d;
Value resultVal = nullptr; Value resultVal = nullptr;
if (auto constant = dyn_cast_or_null<ConstantIndexOp>(val->getDefiningOp())) { if (auto constant = dyn_cast_or_null<ConstantIndexOp>(val.getDefiningOp())) {
d = getAffineConstantExpr(constant.getValue(), context); d = getAffineConstantExpr(constant.getValue(), context);
} else if (isValidSymbol(val) && !isValidDim(val)) { } else if (isValidSymbol(val) && !isValidDim(val)) {
d = getAffineSymbolExpr(numSymbols++, context); d = getAffineSymbolExpr(numSymbols++, context);
resultVal = val; resultVal = val;
} else { } else {
d = getAffineDimExpr(numDims++, context); d = getAffineDimExpr(numDims++, context);
resultVal = val; resultVal = val;
} }
Show All 22 Linesstatic ValueHandle createBinaryIndexHandle(
// TODO: createOrFold when available. // TODO: createOrFold when available.
return ValueHandle::createComposedAffineApply(map, operands); return ValueHandle::createComposedAffineApply(map, operands);
} }
template <typename IOp, typename FOp> template <typename IOp, typename FOp>
static ValueHandle createBinaryHandle( static ValueHandle createBinaryHandle(
ValueHandle lhs, ValueHandle rhs, ValueHandle lhs, ValueHandle rhs,
function_ref<AffineExpr(AffineExpr, AffineExpr)> affCombiner) { function_ref<AffineExpr(AffineExpr, AffineExpr)> affCombiner) {
auto thisType = lhs.getValue()->getType(); auto thisType = lhs.getValue().getType();
auto thatType = rhs.getValue()->getType(); auto thatType = rhs.getValue().getType();
assert(thisType == thatType && "cannot mix types in operators"); assert(thisType == thatType && "cannot mix types in operators");
(void)thisType; (void)thisType;
(void)thatType; (void)thatType;
if (thisType.isIndex()) { if (thisType.isIndex()) {
return createBinaryIndexHandle(lhs, rhs, affCombiner); return createBinaryIndexHandle(lhs, rhs, affCombiner);
} else if (thisType.isa<IntegerType>()) { } else if (thisType.isa<IntegerType>()) {
return createBinaryHandle<IOp>(lhs, rhs); return createBinaryHandle<IOp>(lhs, rhs);
} else if (thisType.isa<FloatType>()) { } else if (thisType.isa<FloatType>()) {
▲ Show 20 Lines • Show All 131 Lines • Show Last 20 Lines

mlir/lib/EDSC/Helpers.cpp

//===- Helpers.cpp - MLIR Declarative Helper Functionality ----------------===// //===- Helpers.cpp - MLIR Declarative Helper Functionality ----------------===//
// //
// Part of the MLIR Project, under the Apache License v2.0 with LLVM Exceptions. // Part of the MLIR Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information. // See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
#include "mlir/EDSC/Helpers.h" #include "mlir/EDSC/Helpers.h"
#include "mlir/Dialect/StandardOps/Ops.h" #include "mlir/Dialect/StandardOps/Ops.h"
#include "mlir/IR/AffineExpr.h" #include "mlir/IR/AffineExpr.h"
using namespace mlir; using namespace mlir;
using namespace mlir::edsc; using namespace mlir::edsc;
static SmallVector<ValueHandle, 8> getMemRefSizes(Value memRef) { static SmallVector<ValueHandle, 8> getMemRefSizes(Value memRef) {
MemRefType memRefType = memRef->getType().cast<MemRefType>(); MemRefType memRefType = memRef.getType().cast<MemRefType>();
assert(isStrided(memRefType) && "Expected strided MemRef type"); assert(isStrided(memRefType) && "Expected strided MemRef type");
SmallVector<ValueHandle, 8> res; SmallVector<ValueHandle, 8> res;
res.reserve(memRefType.getShape().size()); res.reserve(memRefType.getShape().size());
const auto &shape = memRefType.getShape(); const auto &shape = memRefType.getShape();
for (unsigned idx = 0, n = shape.size(); idx < n; ++idx) { for (unsigned idx = 0, n = shape.size(); idx < n; ++idx) {
if (shape[idx] == -1) { if (shape[idx] == -1) {
res.push_back(ValueHandle::create<DimOp>(memRef, idx)); res.push_back(ValueHandle::create<DimOp>(memRef, idx));
} else { } else {
res.push_back(static_cast<index_t>(shape[idx])); res.push_back(static_cast<index_t>(shape[idx]));
} }
} }
return res; return res;
} }
mlir::edsc::MemRefView::MemRefView(Value v) : base(v) { mlir::edsc::MemRefView::MemRefView(Value v) : base(v) {
assert(v->getType().isa<MemRefType>() && "MemRefType expected"); assert(v.getType().isa<MemRefType>() && "MemRefType expected");
auto memrefSizeValues = getMemRefSizes(v); auto memrefSizeValues = getMemRefSizes(v);
for (auto &size : memrefSizeValues) { for (auto &size : memrefSizeValues) {
lbs.push_back(static_cast<index_t>(0)); lbs.push_back(static_cast<index_t>(0));
ubs.push_back(size); ubs.push_back(size);
steps.push_back(1); steps.push_back(1);
} }
} }
mlir::edsc::VectorView::VectorView(Value v) : base(v) { mlir::edsc::VectorView::VectorView(Value v) : base(v) {
auto vectorType = v->getType().cast<VectorType>(); auto vectorType = v.getType().cast<VectorType>();
for (auto s : vectorType.getShape()) { for (auto s : vectorType.getShape()) {
lbs.push_back(static_cast<index_t>(0)); lbs.push_back(static_cast<index_t>(0));
ubs.push_back(static_cast<index_t>(s)); ubs.push_back(static_cast<index_t>(s));
steps.push_back(1); steps.push_back(1);
} }
} }

mlir/lib/IR/AsmPrinter.cpp

Show First 20 Lines • Show All 399 Lines • ▼ Show 20 Linesvoid AliasState::visitOperation(Operation *op) {
// Visit all the types used in the operation. // Visit all the types used in the operation.
for (auto type : op->getOperandTypes()) for (auto type : op->getOperandTypes())
visitType(type); visitType(type);
for (auto type : op->getResultTypes()) for (auto type : op->getResultTypes())
visitType(type); visitType(type);
for (auto &region : op->getRegions()) for (auto &region : op->getRegions())
for (auto &block : region) for (auto &block : region)
for (auto arg : block.getArguments()) for (auto arg : block.getArguments())
visitType(arg->getType()); visitType(arg.getType());
// Visit each of the attributes. // Visit each of the attributes.
for (auto elt : op->getAttrs()) for (auto elt : op->getAttrs())
visitAttribute(elt.second); visitAttribute(elt.second);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SSANameState // SSANameState
▲ Show 20 Lines • Show All 193 Lines • ▼ Show 20 Linesvoid SSANameState::numberValuesInRegion(
} }
} }
void SSANameState::numberValuesInBlock( void SSANameState::numberValuesInBlock(
Block &block, Block &block,
DialectInterfaceCollection<OpAsmDialectInterface> &interfaces) { DialectInterfaceCollection<OpAsmDialectInterface> &interfaces) {
auto setArgNameFn = [&](Value arg, StringRef name) { auto setArgNameFn = [&](Value arg, StringRef name) {
assert(!valueIDs.count(arg) && "arg numbered multiple times"); assert(!valueIDs.count(arg) && "arg numbered multiple times");
assert(arg.cast<BlockArgument>()->getOwner() == &block && assert(arg.cast<BlockArgument>().getOwner() == &block &&
"arg not defined in 'block'"); "arg not defined in 'block'");
setValueName(arg, name); setValueName(arg, name);
}; };
bool isEntryBlock = block.isEntryBlock(); bool isEntryBlock = block.isEntryBlock();
if (isEntryBlock) { if (isEntryBlock) {
if (auto *op = block.getParentOp()) { if (auto *op = block.getParentOp()) {
if (auto asmInterface = interfaces.getInterfaceFor(op->getDialect())) if (auto asmInterface = interfaces.getInterfaceFor(op->getDialect()))
Show All 27 Linesvoid SSANameState::numberValuesInOp(
if (numResults == 0) if (numResults == 0)
return; return;
Value resultBegin = op.getResult(0); Value resultBegin = op.getResult(0);
// Function used to set the special result names for the operation. // Function used to set the special result names for the operation.
SmallVector<int, 2> resultGroups(/*Size=*/1, /*Value=*/0); SmallVector<int, 2> resultGroups(/*Size=*/1, /*Value=*/0);
auto setResultNameFn = [&](Value result, StringRef name) { auto setResultNameFn = [&](Value result, StringRef name) {
assert(!valueIDs.count(result) && "result numbered multiple times"); assert(!valueIDs.count(result) && "result numbered multiple times");
assert(result->getDefiningOp() == &op && "result not defined by 'op'"); assert(result.getDefiningOp() == &op && "result not defined by 'op'");
setValueName(result, name); setValueName(result, name);
// Record the result number for groups not anchored at 0. // Record the result number for groups not anchored at 0.
if (int resultNo = result.cast<OpResult>()->getResultNumber()) if (int resultNo = result.cast<OpResult>().getResultNumber())
resultGroups.push_back(resultNo); resultGroups.push_back(resultNo);
}; };
if (OpAsmOpInterface asmInterface = dyn_cast<OpAsmOpInterface>(&op)) if (OpAsmOpInterface asmInterface = dyn_cast<OpAsmOpInterface>(&op))
asmInterface.getAsmResultNames(setResultNameFn); asmInterface.getAsmResultNames(setResultNameFn);
else if (auto *asmInterface = interfaces.getInterfaceFor(op.getDialect())) else if (auto *asmInterface = interfaces.getInterfaceFor(op.getDialect()))
asmInterface->getAsmResultNames(&op, setResultNameFn); asmInterface->getAsmResultNames(&op, setResultNameFn);
// If the first result wasn't numbered, give it a default number. // If the first result wasn't numbered, give it a default number.
if (valueIDs.try_emplace(resultBegin, nextValueID).second) if (valueIDs.try_emplace(resultBegin, nextValueID).second)
++nextValueID; ++nextValueID;
// If this operation has multiple result groups, mark it. // If this operation has multiple result groups, mark it.
if (resultGroups.size() != 1) { if (resultGroups.size() != 1) {
llvm::array_pod_sort(resultGroups.begin(), resultGroups.end()); llvm::array_pod_sort(resultGroups.begin(), resultGroups.end());
opResultGroups.try_emplace(&op, std::move(resultGroups)); opResultGroups.try_emplace(&op, std::move(resultGroups));
} }
} }
void SSANameState::getResultIDAndNumber(OpResult result, Value &lookupValue, void SSANameState::getResultIDAndNumber(OpResult result, Value &lookupValue,
Optional<int> &lookupResultNo) const { Optional<int> &lookupResultNo) const {
Operation *owner = result->getOwner(); Operation *owner = result.getOwner();
if (owner->getNumResults() == 1) if (owner->getNumResults() == 1)
return; return;
int resultNo = result->getResultNumber(); int resultNo = result.getResultNumber();
// If this operation has multiple result groups, we will need to find the // If this operation has multiple result groups, we will need to find the
// one corresponding to this result. // one corresponding to this result.
auto resultGroupIt = opResultGroups.find(owner); auto resultGroupIt = opResultGroups.find(owner);
if (resultGroupIt == opResultGroups.end()) { if (resultGroupIt == opResultGroups.end()) {
// If not, just use the first result. // If not, just use the first result.
lookupResultNo = resultNo; lookupResultNo = resultNo;
lookupValue = owner->getResult(0); lookupValue = owner->getResult(0);
▲ Show 20 Lines • Show All 1,305 Lines • ▼ Show 20 Lines if (printBlockArgs) {
printBlockName(block); printBlockName(block);
// Print the argument list if non-empty. // Print the argument list if non-empty.
if (!block->args_empty()) { if (!block->args_empty()) {
os << '('; os << '(';
interleaveComma(block->getArguments(), [&](BlockArgument arg) { interleaveComma(block->getArguments(), [&](BlockArgument arg) {
printValueID(arg); printValueID(arg);
os << ": "; os << ": ";
printType(arg->getType()); printType(arg.getType());
}); });
os << ')'; os << ')';
} }
os << ':'; os << ':';
// Print out some context information about the predecessors of this block. // Print out some context information about the predecessors of this block.
if (!block->getParent()) { if (!block->getParent()) {
os << "\t// block is not in a region!"; os << "\t// block is not in a region!";
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Linesvoid OperationPrinter::printSuccessorAndUseList(Operation *term,
if (succOperands.begin() == succOperands.end()) if (succOperands.begin() == succOperands.end())
return; return;
os << '('; os << '(';
interleaveComma(succOperands, interleaveComma(succOperands,
[this](Value operand) { printValueID(operand); }); [this](Value operand) { printValueID(operand); });
os << " : "; os << " : ";
interleaveComma(succOperands, interleaveComma(succOperands,
[this](Value operand) { printType(operand->getType()); }); [this](Value operand) { printType(operand.getType()); });
os << ')'; os << ')';
} }
void OperationPrinter::printRegion(Region &region, bool printEntryBlockArgs, void OperationPrinter::printRegion(Region &region, bool printEntryBlockArgs,
bool printBlockTerminators) { bool printBlockTerminators) {
os << " {\n"; os << " {\n";
if (!region.empty()) { if (!region.empty()) {
auto *entryBlock = &region.front(); auto *entryBlock = &region.front();
▲ Show 20 Lines • Show All 175 Lines • Show Last 20 Lines

mlir/lib/IR/Block.cpp

Show First 20 Lines • Show All 85 Lines • ▼ Show 20 Lines
/// they are to be deleted. /// they are to be deleted.
void Block::dropAllReferences() { void Block::dropAllReferences() {
for (Operation &i : *this) for (Operation &i : *this)
i.dropAllReferences(); i.dropAllReferences();
} }
void Block::dropAllDefinedValueUses() { void Block::dropAllDefinedValueUses() {
for (auto arg : getArguments()) for (auto arg : getArguments())
arg->dropAllUses(); arg.dropAllUses();
for (auto &op : *this) for (auto &op : *this)
op.dropAllDefinedValueUses(); op.dropAllDefinedValueUses();
dropAllUses(); dropAllUses();
} }
/// Returns true if the ordering of the child operations is valid, false /// Returns true if the ordering of the child operations is valid, false
/// otherwise. /// otherwise.
bool Block::isOpOrderValid() { return parentValidOpOrderPair.getInt(); } bool Block::isOpOrderValid() { return parentValidOpOrderPair.getInt(); }
▲ Show 20 Lines • Show All 186 Lines • Show Last 20 Lines

mlir/lib/IR/Builders.cpp

Show First 20 Lines • Show All 371 Lines • ▼ Show 20 Lines for (auto &it : llvm::enumerate(foldResults)) {
// Otherwise, try to materialize a constant operation. // Otherwise, try to materialize a constant operation.
if (!dialect) if (!dialect)
return cleanupFailure(); return cleanupFailure();
// Ask the dialect to materialize a constant operation for this value. // Ask the dialect to materialize a constant operation for this value.
Attribute attr = it.value().get<Attribute>(); Attribute attr = it.value().get<Attribute>();
auto *constOp = dialect->materializeConstant( auto *constOp = dialect->materializeConstant(
cstBuilder, attr, op->getResult(it.index())->getType(), op->getLoc()); cstBuilder, attr, op->getResult(it.index()).getType(), op->getLoc());
if (!constOp) { if (!constOp) {
// Erase any generated constants. // Erase any generated constants.
for (Operation *cst : generatedConstants) for (Operation *cst : generatedConstants)
cst->erase(); cst->erase();
return cleanupFailure(); return cleanupFailure();
} }
assert(matchPattern(constOp, m_Constant(&attr))); assert(matchPattern(constOp, m_Constant(&attr)));
Show All 10 Lines

mlir/lib/IR/Function.cpp

Show First 20 Lines • Show All 90 Lines • ▼ Show 20 Lines if (isExternal())
return success(); return success();
// Verify that the argument list of the function and the arg list of the entry // Verify that the argument list of the function and the arg list of the entry
// block line up. The trait already verified that the number of arguments is // block line up. The trait already verified that the number of arguments is
// the same between the signature and the block. // the same between the signature and the block.
auto fnInputTypes = getType().getInputs(); auto fnInputTypes = getType().getInputs();
Block &entryBlock = front(); Block &entryBlock = front();
for (unsigned i = 0, e = entryBlock.getNumArguments(); i != e; ++i) for (unsigned i = 0, e = entryBlock.getNumArguments(); i != e; ++i)
if (fnInputTypes[i] != entryBlock.getArgument(i)->getType()) if (fnInputTypes[i] != entryBlock.getArgument(i).getType())
return emitOpError("type of entry block argument #") return emitOpError("type of entry block argument #")
<< i << '(' << entryBlock.getArgument(i)->getType() << i << '(' << entryBlock.getArgument(i).getType()
<< ") must match the type of the corresponding argument in " << ") must match the type of the corresponding argument in "
<< "function signature(" << fnInputTypes[i] << ')'; << "function signature(" << fnInputTypes[i] << ')';
return success(); return success();
} }
void FuncOp::eraseArguments(ArrayRef<unsigned> argIndices) { void FuncOp::eraseArguments(ArrayRef<unsigned> argIndices) {
auto oldType = getType(); auto oldType = getType();
▲ Show 20 Lines • Show All 104 Lines • Show Last 20 Lines

mlir/lib/IR/Operation.cpp

Show First 20 Lines • Show All 803 Lines • ▼ Show 20 Lines
} }
LogicalResult OpTrait::impl::verifySameTypeOperands(Operation *op) { LogicalResult OpTrait::impl::verifySameTypeOperands(Operation *op) {
// Zero or one operand always have the "same" type. // Zero or one operand always have the "same" type.
unsigned nOperands = op->getNumOperands(); unsigned nOperands = op->getNumOperands();
if (nOperands < 2) if (nOperands < 2)
return success(); return success();
auto type = op->getOperand(0)->getType(); auto type = op->getOperand(0).getType();
for (auto opType : llvm::drop_begin(op->getOperandTypes(), 1)) for (auto opType : llvm::drop_begin(op->getOperandTypes(), 1))
if (opType != type) if (opType != type)
return op->emitOpError() << "requires all operands to have the same type"; return op->emitOpError() << "requires all operands to have the same type";
return success(); return success();
} }
LogicalResult OpTrait::impl::verifyZeroResult(Operation *op) { LogicalResult OpTrait::impl::verifyZeroResult(Operation *op) {
if (op->getNumResults() != 0) if (op->getNumResults() != 0)
Show All 21 Lines return op->emitOpError()
<< "expected " << numOperands << " or more results"; << "expected " << numOperands << " or more results";
return success(); return success();
} }
LogicalResult OpTrait::impl::verifySameOperandsShape(Operation *op) { LogicalResult OpTrait::impl::verifySameOperandsShape(Operation *op) {
if (failed(verifyAtLeastNOperands(op, 1))) if (failed(verifyAtLeastNOperands(op, 1)))
return failure(); return failure();
auto type = op->getOperand(0)->getType(); auto type = op->getOperand(0).getType();
for (auto opType : llvm::drop_begin(op->getOperandTypes(), 1)) { for (auto opType : llvm::drop_begin(op->getOperandTypes(), 1)) {
if (failed(verifyCompatibleShape(opType, type))) if (failed(verifyCompatibleShape(opType, type)))
return op->emitOpError() << "requires the same shape for all operands"; return op->emitOpError() << "requires the same shape for all operands";
} }
return success(); return success();
} }
LogicalResult OpTrait::impl::verifySameOperandsAndResultShape(Operation *op) { LogicalResult OpTrait::impl::verifySameOperandsAndResultShape(Operation *op) {
if (failed(verifyAtLeastNOperands(op, 1)) || if (failed(verifyAtLeastNOperands(op, 1)) ||
failed(verifyAtLeastNResults(op, 1))) failed(verifyAtLeastNResults(op, 1)))
return failure(); return failure();
auto type = op->getOperand(0)->getType(); auto type = op->getOperand(0).getType();
for (auto resultType : op->getResultTypes()) { for (auto resultType : op->getResultTypes()) {
if (failed(verifyCompatibleShape(resultType, type))) if (failed(verifyCompatibleShape(resultType, type)))
return op->emitOpError() return op->emitOpError()
<< "requires the same shape for all operands and results"; << "requires the same shape for all operands and results";
} }
for (auto opType : llvm::drop_begin(op->getOperandTypes(), 1)) { for (auto opType : llvm::drop_begin(op->getOperandTypes(), 1)) {
if (failed(verifyCompatibleShape(opType, type))) if (failed(verifyCompatibleShape(opType, type)))
return op->emitOpError() return op->emitOpError()
Show All 40 LinesOpTrait::impl::verifySameOperandsAndResultElementType(Operation *op) {
return success(); return success();
} }
LogicalResult OpTrait::impl::verifySameOperandsAndResultType(Operation *op) { LogicalResult OpTrait::impl::verifySameOperandsAndResultType(Operation *op) {
if (failed(verifyAtLeastNOperands(op, 1)) || if (failed(verifyAtLeastNOperands(op, 1)) ||
failed(verifyAtLeastNResults(op, 1))) failed(verifyAtLeastNResults(op, 1)))
return failure(); return failure();
auto type = op->getResult(0)->getType(); auto type = op->getResult(0).getType();
auto elementType = getElementTypeOrSelf(type); auto elementType = getElementTypeOrSelf(type);
for (auto resultType : llvm::drop_begin(op->getResultTypes(), 1)) { for (auto resultType : llvm::drop_begin(op->getResultTypes(), 1)) {
if (getElementTypeOrSelf(resultType) != elementType || if (getElementTypeOrSelf(resultType) != elementType ||
failed(verifyCompatibleShape(resultType, type))) failed(verifyCompatibleShape(resultType, type)))
return op->emitOpError() return op->emitOpError()
<< "requires the same type for all operands and results"; << "requires the same type for all operands and results";
} }
for (auto opType : op->getOperandTypes()) { for (auto opType : op->getOperandTypes()) {
Show All 12 Linesstatic LogicalResult verifySuccessor(Operation *op, unsigned succNo) {
if (operandCount != destBB->getNumArguments()) if (operandCount != destBB->getNumArguments())
return op->emitError() << "branch has " << operandCount return op->emitError() << "branch has " << operandCount
<< " operands for successor #" << succNo << " operands for successor #" << succNo
<< ", but target block has " << ", but target block has "
<< destBB->getNumArguments(); << destBB->getNumArguments();
auto operandIt = operands.begin(); auto operandIt = operands.begin();
for (unsigned i = 0, e = operandCount; i != e; ++i, ++operandIt) { for (unsigned i = 0, e = operandCount; i != e; ++i, ++operandIt) {
if ((*operandIt)->getType() != destBB->getArgument(i)->getType()) if ((*operandIt).getType() != destBB->getArgument(i).getType())
return op->emitError() << "type mismatch for bb argument #" << i return op->emitError() << "type mismatch for bb argument #" << i
<< " of successor #" << succNo; << " of successor #" << succNo;
} }
return success(); return success();
} }
static LogicalResult verifyTerminatorSuccessors(Operation *op) { static LogicalResult verifyTerminatorSuccessors(Operation *op) {
▲ Show 20 Lines • Show All 93 Lines • ▼ Show 20 Lines
// BinaryOp implementation // BinaryOp implementation
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// These functions are out-of-line implementations of the methods in BinaryOp, // These functions are out-of-line implementations of the methods in BinaryOp,
// which avoids them being template instantiated/duplicated. // which avoids them being template instantiated/duplicated.
void impl::buildBinaryOp(Builder *builder, OperationState &result, Value lhs, void impl::buildBinaryOp(Builder *builder, OperationState &result, Value lhs,
Value rhs) { Value rhs) {
assert(lhs->getType() == rhs->getType()); assert(lhs.getType() == rhs.getType());
result.addOperands({lhs, rhs}); result.addOperands({lhs, rhs});
result.types.push_back(lhs->getType()); result.types.push_back(lhs.getType());
} }
ParseResult impl::parseOneResultSameOperandTypeOp(OpAsmParser &parser, ParseResult impl::parseOneResultSameOperandTypeOp(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
SmallVector<OpAsmParser::OperandType, 2> ops; SmallVector<OpAsmParser::OperandType, 2> ops;
Type type; Type type;
return failure(parser.parseOperandList(ops) || return failure(parser.parseOperandList(ops) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(type) || parser.parseColonType(type) ||
parser.resolveOperands(ops, type, result.operands) || parser.resolveOperands(ops, type, result.operands) ||
parser.addTypeToList(type, result.types)); parser.addTypeToList(type, result.types));
} }
void impl::printOneResultOp(Operation *op, OpAsmPrinter &p) { void impl::printOneResultOp(Operation *op, OpAsmPrinter &p) {
assert(op->getNumResults() == 1 && "op should have one result"); assert(op->getNumResults() == 1 && "op should have one result");
// If not all the operand and result types are the same, just use the // If not all the operand and result types are the same, just use the
// generic assembly form to avoid omitting information in printing. // generic assembly form to avoid omitting information in printing.
auto resultType = op->getResult(0)->getType(); auto resultType = op->getResult(0).getType();
if (llvm::any_of(op->getOperandTypes(), if (llvm::any_of(op->getOperandTypes(),
[&](Type type) { return type != resultType; })) { [&](Type type) { return type != resultType; })) {
p.printGenericOp(op); p.printGenericOp(op);
return; return;
} }
p << op->getName() << ' '; p << op->getName() << ' ';
p.printOperands(op->getOperands()); p.printOperands(op->getOperands());
Show All 19 Lines return failure(parser.parseOperand(srcInfo) ||
parser.parseOptionalAttrDict(result.attributes) || parser.parseOptionalAttrDict(result.attributes) ||
parser.parseColonType(srcType) || parser.parseColonType(srcType) ||
parser.resolveOperand(srcInfo, srcType, result.operands) || parser.resolveOperand(srcInfo, srcType, result.operands) ||
parser.parseKeywordType("to", dstType) || parser.parseKeywordType("to", dstType) ||
parser.addTypeToList(dstType, result.types)); parser.addTypeToList(dstType, result.types));
} }
void impl::printCastOp(Operation *op, OpAsmPrinter &p) { void impl::printCastOp(Operation *op, OpAsmPrinter &p) {
p << op->getName() << ' ' << *op->getOperand(0); p << op->getName() << ' ' << op->getOperand(0);
p.printOptionalAttrDict(op->getAttrs()); p.printOptionalAttrDict(op->getAttrs());
p << " : " << op->getOperand(0)->getType() << " to " p << " : " << op->getOperand(0).getType() << " to "
<< op->getResult(0)->getType(); << op->getResult(0).getType();
} }
Value impl::foldCastOp(Operation *op) { Value impl::foldCastOp(Operation *op) {
// Identity cast // Identity cast
if (op->getOperand(0)->getType() == op->getResult(0)->getType()) if (op->getOperand(0).getType() == op->getResult(0).getType())
return op->getOperand(0); return op->getOperand(0);
return nullptr; return nullptr;
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Misc. utils // Misc. utils
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 16 Lines

mlir/lib/IR/PatternMatch.cpp

Show First 20 Lines • Show All 108 Lines • ▼ Show 20 Linesvoid PatternRewriter::mergeBlocks(Block *source, Block *dest,
assert(llvm::all_of(source->getPredecessors(), assert(llvm::all_of(source->getPredecessors(),
[dest](Block *succ) { return succ == dest; }) && [dest](Block *succ) { return succ == dest; }) &&
"expected 'source' to have no predecessors or only 'dest'"); "expected 'source' to have no predecessors or only 'dest'");
assert(argValues.size() == source->getNumArguments() && assert(argValues.size() == source->getNumArguments() &&
"incorrect # of argument replacement values"); "incorrect # of argument replacement values");
// Replace all of the successor arguments with the provided values. // Replace all of the successor arguments with the provided values.
for (auto it : llvm::zip(source->getArguments(), argValues)) for (auto it : llvm::zip(source->getArguments(), argValues))
std::get<0>(it)->replaceAllUsesWith(std::get<1>(it)); std::get<0>(it).replaceAllUsesWith(std::get<1>(it));
// Splice the operations of the 'source' block into the 'dest' block and erase // Splice the operations of the 'source' block into the 'dest' block and erase
// it. // it.
dest->getOperations().splice(dest->end(), source->getOperations()); dest->getOperations().splice(dest->end(), source->getOperations());
source->dropAllUses(); source->dropAllUses();
source->erase(); source->erase();
} }
▲ Show 20 Lines • Show All 79 Lines • Show Last 20 Lines

mlir/lib/IR/Region.cpp

Show First 20 Lines • Show All 78 Lines • ▼ Show 20 Lines for (Block &block : *this) {
Block *newBlock = new Block(); Block *newBlock = new Block();
mapper.map(&block, newBlock); mapper.map(&block, newBlock);
// Clone the block arguments. The user might be deleting arguments to the // Clone the block arguments. The user might be deleting arguments to the
// block by specifying them in the mapper. If so, we don't add the // block by specifying them in the mapper. If so, we don't add the
// argument to the cloned block. // argument to the cloned block.
for (auto arg : block.getArguments()) for (auto arg : block.getArguments())
if (!mapper.contains(arg)) if (!mapper.contains(arg))
mapper.map(arg, newBlock->addArgument(arg->getType())); mapper.map(arg, newBlock->addArgument(arg.getType()));
// Clone and remap the operations within this block. // Clone and remap the operations within this block.
for (auto &op : block) for (auto &op : block)
newBlock->push_back(op.clone(mapper)); newBlock->push_back(op.clone(mapper));
dest->getBlocks().insert(destPos, newBlock); dest->getBlocks().insert(destPos, newBlock);
} }
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Lines for (Block &block : *pendingRegions.pop_back_val()) {
if (!operand) { if (!operand) {
if (noteLoc) if (noteLoc)
op.emitOpError("block's operand not defined").attachNote(noteLoc); op.emitOpError("block's operand not defined").attachNote(noteLoc);
return false; return false;
} }
// Check that any value that is used by an operation is defined in the // Check that any value that is used by an operation is defined in the
// same region as either an operation result or a block argument. // same region as either an operation result or a block argument.
if (operand->getParentRegion()->isProperAncestor(&limit)) { if (operand.getParentRegion()->isProperAncestor(&limit)) {
if (noteLoc) { if (noteLoc) {
op.emitOpError("using value defined outside the region") op.emitOpError("using value defined outside the region")
.attachNote(noteLoc) .attachNote(noteLoc)
<< "required by region isolation constraints"; << "required by region isolation constraints";
} }
return false; return false;
} }
} }
▲ Show 20 Lines • Show All 73 Lines • Show Last 20 Lines

mlir/lib/IR/TypeUtilities.cpp

Show All 19 Lines
Type mlir::getElementTypeOrSelf(Type type) { Type mlir::getElementTypeOrSelf(Type type) {
if (auto st = type.dyn_cast<ShapedType>()) if (auto st = type.dyn_cast<ShapedType>())
return st.getElementType(); return st.getElementType();
return type; return type;
} }
Type mlir::getElementTypeOrSelf(Value val) { Type mlir::getElementTypeOrSelf(Value val) {
return getElementTypeOrSelf(val->getType()); return getElementTypeOrSelf(val.getType());
} }
Type mlir::getElementTypeOrSelf(Attribute attr) { Type mlir::getElementTypeOrSelf(Attribute attr) {
return getElementTypeOrSelf(attr.getType()); return getElementTypeOrSelf(attr.getType());
} }
SmallVector<Type, 10> mlir::getFlattenedTypes(TupleType t) { SmallVector<Type, 10> mlir::getFlattenedTypes(TupleType t) {
SmallVector<Type, 10> fTypes; SmallVector<Type, 10> fTypes;
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Lines
} }
OperandElementTypeIterator::OperandElementTypeIterator( OperandElementTypeIterator::OperandElementTypeIterator(
Operation::operand_iterator it) Operation::operand_iterator it)
: llvm::mapped_iterator<Operation::operand_iterator, Type (*)(Value)>( : llvm::mapped_iterator<Operation::operand_iterator, Type (*)(Value)>(
it, &unwrap) {} it, &unwrap) {}
Type OperandElementTypeIterator::unwrap(Value value) { Type OperandElementTypeIterator::unwrap(Value value) {
return value->getType().cast<ShapedType>().getElementType(); return value.getType().cast<ShapedType>().getElementType();
} }
ResultElementTypeIterator::ResultElementTypeIterator( ResultElementTypeIterator::ResultElementTypeIterator(
Operation::result_iterator it) Operation::result_iterator it)
: llvm::mapped_iterator<Operation::result_iterator, Type (*)(Value)>( : llvm::mapped_iterator<Operation::result_iterator, Type (*)(Value)>(
it, &unwrap) {} it, &unwrap) {}
Type ResultElementTypeIterator::unwrap(Value value) { Type ResultElementTypeIterator::unwrap(Value value) {
return value->getType().cast<ShapedType>().getElementType(); return value.getType().cast<ShapedType>().getElementType();
} }

mlir/lib/IR/Value.cpp

Show First 20 Lines • Show All 64 Lines • ▼ Show 20 Linesvoid Value::setType(Type newType) {
newTypes[resultNo] = newType; newTypes[resultNo] = newType;
owner->resultType = TupleType::get(newTypes, newType.getContext()); owner->resultType = TupleType::get(newTypes, newType.getContext());
} }
/// If this value is the result of an Operation, return the operation that /// If this value is the result of an Operation, return the operation that
/// defines it. /// defines it.
Operation *Value::getDefiningOp() const { Operation *Value::getDefiningOp() const {
if (auto result = dyn_cast<OpResult>()) if (auto result = dyn_cast<OpResult>())
return result->getOwner(); return result.getOwner();
return nullptr; return nullptr;
} }
Location Value::getLoc() { Location Value::getLoc() const {
if (auto *op = getDefiningOp()) if (auto *op = getDefiningOp())
return op->getLoc(); return op->getLoc();
return UnknownLoc::get(getContext()); return UnknownLoc::get(getContext());
} }
/// Return the Region in which this Value is defined. /// Return the Region in which this Value is defined.
Region *Value::getParentRegion() { Region *Value::getParentRegion() {
if (auto *op = getDefiningOp()) if (auto *op = getDefiningOp())
return op->getParentRegion(); return op->getParentRegion();
return cast<BlockArgument>()->getOwner()->getParent(); return cast<BlockArgument>().getOwner()->getParent();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Value::UseLists // Value::UseLists
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Provide the use list that is attached to this value. /// Provide the use list that is attached to this value.
IRObjectWithUseList<OpOperand> *Value::getUseList() const { IRObjectWithUseList<OpOperand> *Value::getUseList() const {
▲ Show 20 Lines • Show All 137 Lines • Show Last 20 Lines

mlir/lib/Parser/Parser.cpp

Show First 20 Lines • Show All 3,250 Lines • ▼ Show 20 Linesprivate:
ModuleOp moduleOp; ModuleOp moduleOp;
}; };
} // end anonymous namespace } // end anonymous namespace
OperationParser::~OperationParser() { OperationParser::~OperationParser() {
for (auto &fwd : forwardRefPlaceholders) { for (auto &fwd : forwardRefPlaceholders) {
// Drop all uses of undefined forward declared reference and destroy // Drop all uses of undefined forward declared reference and destroy
// defining operation. // defining operation.
fwd.first->dropAllUses(); fwd.first.dropAllUses();
fwd.first->getDefiningOp()->destroy(); fwd.first.getDefiningOp()->destroy();
} }
} }
/// After parsing is finished, this function must be called to see if there are /// After parsing is finished, this function must be called to see if there are
/// any remaining issues. /// any remaining issues.
ParseResult OperationParser::finalize() { ParseResult OperationParser::finalize() {
// Check for any forward references that are left. If we find any, error // Check for any forward references that are left. If we find any, error
// out. // out.
▲ Show 20 Lines • Show All 77 Lines • ▼ Show 20 Lines if (!isForwardRefPlaceholder(existing)) {
.append("redefinition of SSA value '", useInfo.name, "'") .append("redefinition of SSA value '", useInfo.name, "'")
.attachNote(getEncodedSourceLocation(entries[useInfo.number].second)) .attachNote(getEncodedSourceLocation(entries[useInfo.number].second))
.append("previously defined here"); .append("previously defined here");
} }
// If it was a forward reference, update everything that used it to use // If it was a forward reference, update everything that used it to use
// the actual definition instead, delete the forward ref, and remove it // the actual definition instead, delete the forward ref, and remove it
// from our set of forward references we track. // from our set of forward references we track.
existing->replaceAllUsesWith(value); existing.replaceAllUsesWith(value);
existing->getDefiningOp()->destroy(); existing.getDefiningOp()->destroy();
forwardRefPlaceholders.erase(existing); forwardRefPlaceholders.erase(existing);
} }
/// Record this definition for the current scope. /// Record this definition for the current scope.
entries[useInfo.number] = {value, useInfo.loc}; entries[useInfo.number] = {value, useInfo.loc};
recordDefinition(useInfo.name); recordDefinition(useInfo.name);
return success(); return success();
} }
▲ Show 20 Lines • Show All 43 Lines • ▼ Show 20 Lines
/// it specifies. This returns null on failure. /// it specifies. This returns null on failure.
Value OperationParser::resolveSSAUse(SSAUseInfo useInfo, Type type) { Value OperationParser::resolveSSAUse(SSAUseInfo useInfo, Type type) {
auto &entries = getSSAValueEntry(useInfo.name); auto &entries = getSSAValueEntry(useInfo.name);
// If we have already seen a value of this name, return it. // If we have already seen a value of this name, return it.
if (useInfo.number < entries.size() && entries[useInfo.number].first) { if (useInfo.number < entries.size() && entries[useInfo.number].first) {
auto result = entries[useInfo.number].first; auto result = entries[useInfo.number].first;
// Check that the type matches the other uses. // Check that the type matches the other uses.
if (result->getType() == type) if (result.getType() == type)
return result; return result;
emitError(useInfo.loc, "use of value '") emitError(useInfo.loc, "use of value '")
.append(useInfo.name, .append(useInfo.name,
"' expects different type than prior uses: ", type, " vs ", "' expects different type than prior uses: ", type, " vs ",
result->getType()) result.getType())
.attachNote(getEncodedSourceLocation(entries[useInfo.number].second)) .attachNote(getEncodedSourceLocation(entries[useInfo.number].second))
.append("prior use here"); .append("prior use here");
return nullptr; return nullptr;
} }
// Make sure we have enough slots for this. // Make sure we have enough slots for this.
if (entries.size() <= useInfo.number) if (entries.size() <= useInfo.number)
entries.resize(useInfo.number + 1); entries.resize(useInfo.number + 1);
▲ Show 20 Lines • Show All 1,110 Lines • ▼ Show 20 Lines return parseSSADefOrUseAndType(
return addDefinition(useInfo, owner->addArgument(type)); return addDefinition(useInfo, owner->addArgument(type));
// Otherwise, ensure that this argument has already been created. // Otherwise, ensure that this argument has already been created.
if (nextArgument >= owner->getNumArguments()) if (nextArgument >= owner->getNumArguments())
return emitError("too many arguments specified in argument list"); return emitError("too many arguments specified in argument list");
// Finally, make sure the existing argument has the correct type. // Finally, make sure the existing argument has the correct type.
auto arg = owner->getArgument(nextArgument++); auto arg = owner->getArgument(nextArgument++);
if (arg->getType() != type) if (arg.getType() != type)
return emitError("argument and block argument type mismatch"); return emitError("argument and block argument type mismatch");
return addDefinition(useInfo, arg); return addDefinition(useInfo, arg);
}); });
}); });
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Top-level entity parsing. // Top-level entity parsing.
▲ Show 20 Lines • Show All 269 Lines • Show Last 20 Lines

mlir/lib/Quantizer/Configurations/FxpMathConfig.cpp

Show First 20 Lines • Show All 100 Lines • ▼ Show 20 Linesstruct FxpMathTargetConfigImpl : public FxpMathTargetConfig {
void addValueIdentityOpByName(StringRef opName) override { void addValueIdentityOpByName(StringRef opName) override {
addOpHandlerByName( addOpHandlerByName(
opName, opName,
std::bind(&FxpMathTargetConfigImpl::handleValueIdentity, this, _1, _2)); std::bind(&FxpMathTargetConfigImpl::handleValueIdentity, this, _1, _2));
} }
void handleValueIdentity(Operation *op, CAGSlice &cag) const { void handleValueIdentity(Operation *op, CAGSlice &cag) const {
assert(op->getNumResults() == 1); assert(op->getNumResults() == 1);
if (!isHandledType(op->getResult(0)->getType())) if (!isHandledType(op->getResult(0).getType()))
return; return;
auto resultNode = cag.getResultAnchor(op, 0); auto resultNode = cag.getResultAnchor(op, 0);
resultNode->setTypeTransformRule( resultNode->setTypeTransformRule(
CAGAnchorNode::TypeTransformRule::DirectStorage); CAGAnchorNode::TypeTransformRule::DirectStorage);
for (unsigned opIdx = 0, e = op->getNumOperands(); opIdx < e; ++opIdx) { for (unsigned opIdx = 0, e = op->getNumOperands(); opIdx < e; ++opIdx) {
if (!isHandledType(op->getOperand(opIdx)->getType())) if (!isHandledType(op->getOperand(opIdx).getType()))
continue; continue;
auto operandNode = cag.getOperandAnchor(op, opIdx); auto operandNode = cag.getOperandAnchor(op, opIdx);
operandNode->setTypeTransformRule( operandNode->setTypeTransformRule(
CAGAnchorNode::TypeTransformRule::DirectStorage); CAGAnchorNode::TypeTransformRule::DirectStorage);
UniformConstraintsBuilder(cag).coupleAnchors(operandNode, resultNode); UniformConstraintsBuilder(cag).coupleAnchors(operandNode, resultNode);
} }
} }
void handleConstant(Operation *op, CAGSlice &cag) const { void handleConstant(Operation *op, CAGSlice &cag) const {
if (!isHandledType(op->getResult(0)->getType())) if (!isHandledType(op->getResult(0).getType()))
return; return;
auto resultNode = cag.getResultAnchor(op, 0); auto resultNode = cag.getResultAnchor(op, 0);
resultNode->setTypeTransformRule( resultNode->setTypeTransformRule(
CAGAnchorNode::TypeTransformRule::ExpressedOnly); CAGAnchorNode::TypeTransformRule::ExpressedOnly);
Attribute valueAttr; Attribute valueAttr;
if (!matchPattern(op, m_Constant(&valueAttr))) { if (!matchPattern(op, m_Constant(&valueAttr))) {
return; return;
} }
AttributeTensorStatistics stats(valueAttr); AttributeTensorStatistics stats(valueAttr);
TensorAxisStatistics layerStats; TensorAxisStatistics layerStats;
if (!stats.get(layerStats)) { if (!stats.get(layerStats)) {
op->emitOpError("could not compute statistics"); op->emitOpError("could not compute statistics");
return; return;
} }
UniformConstraintsBuilder(cag).applyStats(resultNode, layerStats); UniformConstraintsBuilder(cag).applyStats(resultNode, layerStats);
} }
void handleTerminal(Operation *op, CAGSlice &cag) const { void handleTerminal(Operation *op, CAGSlice &cag) const {
if (!isHandledType(op->getOperand(0)->getType())) if (!isHandledType(op->getOperand(0).getType()))
return; return;
auto operandNode = cag.getOperandAnchor(op, 0); auto operandNode = cag.getOperandAnchor(op, 0);
operandNode->setTypeTransformRule( operandNode->setTypeTransformRule(
CAGAnchorNode::TypeTransformRule::ExpressedOnly); CAGAnchorNode::TypeTransformRule::ExpressedOnly);
} }
void handleStats(Operation *op, CAGSlice &cag) const { void handleStats(Operation *op, CAGSlice &cag) const {
if (!isHandledType(op->getResult(0)->getType())) if (!isHandledType(op->getResult(0).getType()))
return; return;
auto argNode = cag.getOperandAnchor(op, 0); auto argNode = cag.getOperandAnchor(op, 0);
auto resultNode = cag.getResultAnchor(op, 0); auto resultNode = cag.getResultAnchor(op, 0);
UniformConstraintsBuilder(cag).coupleAnchors(argNode, resultNode); UniformConstraintsBuilder(cag).coupleAnchors(argNode, resultNode);
TensorAxisStatistics layerStats; TensorAxisStatistics layerStats;
auto statsOp = cast<quant::StatisticsOp>(op); auto statsOp = cast<quant::StatisticsOp>(op);
auto layerStatsAttr = statsOp.layerStats(); auto layerStatsAttr = statsOp.layerStats();
layerStats.minValue = layerStats.minValue =
layerStatsAttr.getValue<FloatAttr>(0).getValueAsDouble(); layerStatsAttr.getValue<FloatAttr>(0).getValueAsDouble();
layerStats.maxValue = layerStats.maxValue =
layerStatsAttr.getValue<FloatAttr>(1).getValueAsDouble(); layerStatsAttr.getValue<FloatAttr>(1).getValueAsDouble();
UniformConstraintsBuilder(cag).applyStats(resultNode, layerStats); UniformConstraintsBuilder(cag).applyStats(resultNode, layerStats);
} }
void handleAdd(Operation *op, CAGSlice &cag) const { void handleAdd(Operation *op, CAGSlice &cag) const {
if (!isHandledType(op->getResult(0)->getType())) if (!isHandledType(op->getResult(0).getType()))
return; return;
auto lhs = cag.getOperandAnchor(op, 0); auto lhs = cag.getOperandAnchor(op, 0);
auto rhs = cag.getOperandAnchor(op, 1); auto rhs = cag.getOperandAnchor(op, 1);
auto resultNode = cag.getResultAnchor(op, 0); auto resultNode = cag.getResultAnchor(op, 0);
// Add supports 8/16 bit math. // Add supports 8/16 bit math.
llvm::SmallBitVector disableMask = llvm::SmallBitVector disableMask =
getCandidateTypeDisabledExceptMask({q8, q16}); getCandidateTypeDisabledExceptMask({q8, q16});
lhs->getUniformMetadata().disabledCandidateTypes = disableMask; lhs->getUniformMetadata().disabledCandidateTypes = disableMask;
rhs->getUniformMetadata().disabledCandidateTypes = disableMask; rhs->getUniformMetadata().disabledCandidateTypes = disableMask;
resultNode->getUniformMetadata().disabledCandidateTypes = disableMask; resultNode->getUniformMetadata().disabledCandidateTypes = disableMask;
// NOTE: We couple the add such that the scale/zeroPoint match between // NOTE: We couple the add such that the scale/zeroPoint match between
// both args and the result. This is overly constrained in that it is // both args and the result. This is overly constrained in that it is
// possible to write efficient add kernels with a bit more freedom (i.e. // possible to write efficient add kernels with a bit more freedom (i.e.
// zeroPoints can vary, scales can differ by a power of two, etc). // zeroPoints can vary, scales can differ by a power of two, etc).
// However, fully coupled yields the simples solutions on the fast path. // However, fully coupled yields the simples solutions on the fast path.
// Further efficiency can be had by constraining the zeroPoint to 0, but // Further efficiency can be had by constraining the zeroPoint to 0, but
// there isn't a constraint for this yet (and there are tradeoffs). // there isn't a constraint for this yet (and there are tradeoffs).
UniformConstraintsBuilder(cag).coupleAnchors(lhs, resultNode); UniformConstraintsBuilder(cag).coupleAnchors(lhs, resultNode);
UniformConstraintsBuilder(cag).coupleAnchors(rhs, resultNode); UniformConstraintsBuilder(cag).coupleAnchors(rhs, resultNode);
addRealMathOptionalConstraints(op, resultNode, cag); addRealMathOptionalConstraints(op, resultNode, cag);
} }
void handleMul(Operation *op, CAGSlice &cag) const { void handleMul(Operation *op, CAGSlice &cag) const {
if (!isHandledType(op->getResult(0)->getType())) if (!isHandledType(op->getResult(0).getType()))
return; return;
auto lhs = cag.getOperandAnchor(op, 0); auto lhs = cag.getOperandAnchor(op, 0);
auto rhs = cag.getOperandAnchor(op, 1); auto rhs = cag.getOperandAnchor(op, 1);
auto resultNode = cag.getResultAnchor(op, 0); auto resultNode = cag.getResultAnchor(op, 0);
// Mul supports 8/16 bit math. // Mul supports 8/16 bit math.
llvm::SmallBitVector disableMask = llvm::SmallBitVector disableMask =
getCandidateTypeDisabledExceptMask({q8, q16}); getCandidateTypeDisabledExceptMask({q8, q16});
lhs->getUniformMetadata().disabledCandidateTypes = disableMask; lhs->getUniformMetadata().disabledCandidateTypes = disableMask;
rhs->getUniformMetadata().disabledCandidateTypes = disableMask; rhs->getUniformMetadata().disabledCandidateTypes = disableMask;
resultNode->getUniformMetadata().disabledCandidateTypes = disableMask; resultNode->getUniformMetadata().disabledCandidateTypes = disableMask;
addRealMathOptionalConstraints(op, resultNode, cag); addRealMathOptionalConstraints(op, resultNode, cag);
} }
void handleMatMul(Operation *op, CAGSlice &cag) const { void handleMatMul(Operation *op, CAGSlice &cag) const {
if (!isHandledType(op->getResult(0)->getType())) if (!isHandledType(op->getResult(0).getType()))
return; return;
auto lhs = cag.getOperandAnchor(op, 0); auto lhs = cag.getOperandAnchor(op, 0);
auto rhs = cag.getOperandAnchor(op, 1); auto rhs = cag.getOperandAnchor(op, 1);
auto resultNode = cag.getResultAnchor(op, 0); auto resultNode = cag.getResultAnchor(op, 0);
// Mul supports 8/16 bit math. // Mul supports 8/16 bit math.
llvm::SmallBitVector disableMask = llvm::SmallBitVector disableMask =
getCandidateTypeDisabledExceptMask({q8, q16}); getCandidateTypeDisabledExceptMask({q8, q16});
lhs->getUniformMetadata().disabledCandidateTypes = disableMask; lhs->getUniformMetadata().disabledCandidateTypes = disableMask;
rhs->getUniformMetadata().disabledCandidateTypes = disableMask; rhs->getUniformMetadata().disabledCandidateTypes = disableMask;
resultNode->getUniformMetadata().disabledCandidateTypes = disableMask; resultNode->getUniformMetadata().disabledCandidateTypes = disableMask;
addRealMathOptionalConstraints(op, resultNode, cag); addRealMathOptionalConstraints(op, resultNode, cag);
} }
void handleMatMulBias(Operation *op, CAGSlice &cag) const { void handleMatMulBias(Operation *op, CAGSlice &cag) const {
if (!isHandledType(op->getResult(0)->getType())) if (!isHandledType(op->getResult(0).getType()))
return; return;
auto lhs = cag.getOperandAnchor(op, 0); auto lhs = cag.getOperandAnchor(op, 0);
auto rhs = cag.getOperandAnchor(op, 1); auto rhs = cag.getOperandAnchor(op, 1);
auto bias = cag.getOperandAnchor(op, 2); auto bias = cag.getOperandAnchor(op, 2);
bias->getUniformMetadata().disabledCandidateTypes = bias->getUniformMetadata().disabledCandidateTypes =
getCandidateTypeDisabledExceptMask({q32ExplicitFixedPoint}); getCandidateTypeDisabledExceptMask({q32ExplicitFixedPoint});
Show All 38 Lines

mlir/lib/Quantizer/Support/ConstraintAnalysisGraph.cpp

Show All 31 Lines
void CAGNode::addOutgoing(CAGNode *toNode) { void CAGNode::addOutgoing(CAGNode *toNode) {
if (!llvm::is_contained(outgoing, toNode)) { if (!llvm::is_contained(outgoing, toNode)) {
outgoing.push_back(toNode); outgoing.push_back(toNode);
toNode->incoming.push_back(this); toNode->incoming.push_back(this);
} }
} }
CAGOperandAnchor::CAGOperandAnchor(Operation *op, unsigned operandIdx) CAGOperandAnchor::CAGOperandAnchor(Operation *op, unsigned operandIdx)
: CAGAnchorNode(Kind::OperandAnchor, op->getOperand(operandIdx)->getType()), : CAGAnchorNode(Kind::OperandAnchor, op->getOperand(operandIdx).getType()),
op(op), operandIdx(operandIdx) {} op(op), operandIdx(operandIdx) {}
CAGResultAnchor::CAGResultAnchor(Operation *op, unsigned resultIdx) CAGResultAnchor::CAGResultAnchor(Operation *op, unsigned resultIdx)
: CAGAnchorNode(Kind::ResultAnchor, op->getResult(resultIdx)->getType()), : CAGAnchorNode(Kind::ResultAnchor, op->getResult(resultIdx).getType()),
resultValue(op->getResult(resultIdx)) {} resultValue(op->getResult(resultIdx)) {}
CAGSlice::CAGSlice(SolverContext &context) : context(context) {} CAGSlice::CAGSlice(SolverContext &context) : context(context) {}
CAGSlice::~CAGSlice() { llvm::DeleteContainerPointers(allNodes); } CAGSlice::~CAGSlice() { llvm::DeleteContainerPointers(allNodes); }
CAGOperandAnchor *CAGSlice::getOperandAnchor(Operation *op, CAGOperandAnchor *CAGSlice::getOperandAnchor(Operation *op,
unsigned operandIdx) { unsigned operandIdx) {
assert(operandIdx < op->getNumOperands() && "illegal operand index"); assert(operandIdx < op->getNumOperands() && "illegal operand index");
Show All 36 Lines
void CAGSlice::enumerateImpliedConnections( void CAGSlice::enumerateImpliedConnections(
std::function<void(CAGAnchorNode *from, CAGAnchorNode *to)> callback) { std::function<void(CAGAnchorNode *from, CAGAnchorNode *to)> callback) {
// Discover peer identity pairs (i.e. implied edges from Result->Operand and // Discover peer identity pairs (i.e. implied edges from Result->Operand and
// Arg->Call). Use an intermediate vector so that the callback can modify. // Arg->Call). Use an intermediate vector so that the callback can modify.
std::vector<std::pair<CAGAnchorNode *, CAGAnchorNode *>> impliedPairs; std::vector<std::pair<CAGAnchorNode *, CAGAnchorNode *>> impliedPairs;
for (auto &resultAnchorPair : resultAnchors) { for (auto &resultAnchorPair : resultAnchors) {
CAGResultAnchor *resultAnchor = resultAnchorPair.second; CAGResultAnchor *resultAnchor = resultAnchorPair.second;
Value resultValue = resultAnchor->getValue(); Value resultValue = resultAnchor->getValue();
for (auto &use : resultValue->getUses()) { for (auto &use : resultValue.getUses()) {
Operation *operandOp = use.getOwner(); Operation *operandOp = use.getOwner();
unsigned operandIdx = use.getOperandNumber(); unsigned operandIdx = use.getOperandNumber();
auto foundIt = operandAnchors.find(std::make_pair(operandOp, operandIdx)); auto foundIt = operandAnchors.find(std::make_pair(operandOp, operandIdx));
if (foundIt != operandAnchors.end()) { if (foundIt != operandAnchors.end()) {
impliedPairs.push_back(std::make_pair(resultAnchor, foundIt->second)); impliedPairs.push_back(std::make_pair(resultAnchor, foundIt->second));
} }
} }
} }
▲ Show 20 Lines • Show All 67 Lines • Show Last 20 Lines

mlir/lib/Quantizer/Transforms/AddDefaultStatsTestPass.cpp

Show First 20 Lines • Show All 60 Lines • ▼ Show 20 Lines
} }
void AddDefaultStatsPass::runWithConfig(SolverContext &solverContext, void AddDefaultStatsPass::runWithConfig(SolverContext &solverContext,
const TargetConfiguration &config) { const TargetConfiguration &config) {
auto func = getFunction(); auto func = getFunction();
// Insert stats for each argument. // Insert stats for each argument.
for (auto arg : func.getArguments()) { for (auto arg : func.getArguments()) {
if (!config.isHandledType(arg->getType())) if (!config.isHandledType(arg.getType()))
continue; continue;
OpBuilder b(func.getBody()); OpBuilder b(func.getBody());
APFloat minValue(-1.0f); APFloat minValue(-1.0f);
APFloat maxValue(1.0f); APFloat maxValue(1.0f);
ElementsAttr layerStats = DenseFPElementsAttr::get( ElementsAttr layerStats = DenseFPElementsAttr::get(
RankedTensorType::get({2}, b.getF32Type()), {minValue, maxValue}); RankedTensorType::get({2}, b.getF32Type()), {minValue, maxValue});
auto statsOp = b.create<StatisticsOp>(func.getLoc(), arg, layerStats, auto statsOp = b.create<StatisticsOp>(func.getLoc(), arg, layerStats,
nullptr, nullptr); nullptr, nullptr);
arg->replaceAllUsesWith(statsOp); arg.replaceAllUsesWith(statsOp);
// StatsOp contained a use to 'arg' so make sure to reset it after replacing // StatsOp contained a use to 'arg' so make sure to reset it after replacing
// all of the uses of 'arg'. // all of the uses of 'arg'.
statsOp.getOperation()->replaceUsesOfWith(statsOp, arg); statsOp.getOperation()->replaceUsesOfWith(statsOp, arg);
} }
// Walk the ops and insert stats. // Walk the ops and insert stats.
func.walk([&](Operation *op) { func.walk([&](Operation *op) {
if (!config.isRequireStatsOp(op)) { if (!config.isRequireStatsOp(op)) {
return; return;
} }
assert(op->getNumResults() == 1); assert(op->getNumResults() == 1);
auto originalResult = op->getResult(0); auto originalResult = op->getResult(0);
if (!config.isHandledType(originalResult->getType())) if (!config.isHandledType(originalResult.getType()))
return; return;
OpBuilder b(op->getBlock(), ++op->getIterator()); OpBuilder b(op->getBlock(), ++op->getIterator());
APFloat minValue(-1.0f); APFloat minValue(-1.0f);
APFloat maxValue(1.0f); APFloat maxValue(1.0f);
ElementsAttr layerStats = DenseFPElementsAttr::get( ElementsAttr layerStats = DenseFPElementsAttr::get(
RankedTensorType::get({2}, b.getF32Type()), {minValue, maxValue}); RankedTensorType::get({2}, b.getF32Type()), {minValue, maxValue});
auto statsOp = b.create<StatisticsOp>(op->getLoc(), op->getResult(0), auto statsOp = b.create<StatisticsOp>(op->getLoc(), op->getResult(0),
layerStats, nullptr, nullptr); layerStats, nullptr, nullptr);
originalResult->replaceAllUsesWith(statsOp); originalResult.replaceAllUsesWith(statsOp);
// StatsOp contained a use to 'op' so make sure to reset it after replacing // StatsOp contained a use to 'op' so make sure to reset it after replacing
// all of the uses of 'op'. // all of the uses of 'op'.
statsOp.getOperation()->replaceUsesOfWith(statsOp, originalResult); statsOp.getOperation()->replaceUsesOfWith(statsOp, originalResult);
}); });
} }
std::unique_ptr<OpPassBase<FuncOp>> std::unique_ptr<OpPassBase<FuncOp>>
mlir::quantizer::createAddDefaultStatsPass() { mlir::quantizer::createAddDefaultStatsPass() {
return std::make_unique<AddDefaultStatsPass>(); return std::make_unique<AddDefaultStatsPass>();
} }
static PassRegistration<AddDefaultStatsPass> pass( static PassRegistration<AddDefaultStatsPass> pass(
"quantizer-add-default-stats-test", "quantizer-add-default-stats-test",
"Adds default (dummy) statistics to all ops that can benefit from " "Adds default (dummy) statistics to all ops that can benefit from "
"runtime statistics. This is meant to help in early stage bootstrapping."); "runtime statistics. This is meant to help in early stage bootstrapping.");

mlir/lib/Quantizer/Transforms/InferQuantizedTypesPass.cpp

Show First 20 Lines • Show All 178 Lines • ▼ Show 20 Linesvoid InferQuantizedTypesPass::transformOperandType(CAGOperandAnchor *anchor,
SmallVector<Value, 1> removeValuesIfDead; SmallVector<Value, 1> removeValuesIfDead;
// Because we've already run the result transforms at this phase, it is // Because we've already run the result transforms at this phase, it is
// very likely that inputValue points to a dcast op whose input matches // very likely that inputValue points to a dcast op whose input matches
// our type. We detect that situation and route around just to save some // our type. We detect that situation and route around just to save some
// bulk in the IR. // bulk in the IR.
Value newTypedInputValue = inputValue; Value newTypedInputValue = inputValue;
auto inputDcastOp = auto inputDcastOp =
dyn_cast_or_null<DequantizeCastOp>(inputValue->getDefiningOp()); dyn_cast_or_null<DequantizeCastOp>(inputValue.getDefiningOp());
if (inputDcastOp && inputDcastOp.arg()->getType() == newType) { if (inputDcastOp && inputDcastOp.arg().getType() == newType) {
// Can just use the dcast's input value. // Can just use the dcast's input value.
newTypedInputValue = inputDcastOp.arg(); newTypedInputValue = inputDcastOp.arg();
removeValuesIfDead.push_back(inputDcastOp); removeValuesIfDead.push_back(inputDcastOp);
} else { } else {
// Need to synthesize a qcast. // Need to synthesize a qcast.
newTypedInputValue = newTypedInputValue =
b.create<QuantizeCastOp>(op->getLoc(), newType, inputValue); b.create<QuantizeCastOp>(op->getLoc(), newType, inputValue);
} }
Show All 18 Lines case CAGAnchorNode::TypeTransformRule::ExpressedOnly:
anchor->getOp()->setOperand( anchor->getOp()->setOperand(
anchor->getOperandIdx(), anchor->getOperandIdx(),
b.create<DequantizeCastOp>(op->getLoc(), anchor->getOriginalType(), b.create<DequantizeCastOp>(op->getLoc(), anchor->getOriginalType(),
newTypedInputValue)); newTypedInputValue));
break; break;
} }
for (Value removeValueIfDead : removeValuesIfDead) { for (Value removeValueIfDead : removeValuesIfDead) {
if (removeValueIfDead->use_empty()) { if (removeValueIfDead.use_empty()) {
removeValueIfDead->getDefiningOp()->erase(); removeValueIfDead.getDefiningOp()->erase();
} }
} }
} }
void InferQuantizedTypesPass::transformResultType(CAGResultAnchor *anchor, void InferQuantizedTypesPass::transformResultType(CAGResultAnchor *anchor,
Type newType) { Type newType) {
Value origResultValue = anchor->getValue(); Value origResultValue = anchor->getValue();
Operation *op = origResultValue->getDefiningOp(); Operation *op = origResultValue.getDefiningOp();
OpBuilder b(op->getBlock(), ++Block::iterator(op)); OpBuilder b(op->getBlock(), ++Block::iterator(op));
Value replacedResultValue = nullptr; Value replacedResultValue = nullptr;
Value newResultValue = nullptr; Value newResultValue = nullptr;
switch (anchor->getTypeTransformRule()) { switch (anchor->getTypeTransformRule()) {
case CAGAnchorNode::TypeTransformRule::Direct: case CAGAnchorNode::TypeTransformRule::Direct:
origResultValue->setType(newType); origResultValue.setType(newType);
replacedResultValue = newResultValue = b.create<DequantizeCastOp>( replacedResultValue = newResultValue = b.create<DequantizeCastOp>(
op->getLoc(), anchor->getOriginalType(), origResultValue); op->getLoc(), anchor->getOriginalType(), origResultValue);
break; break;
case CAGAnchorNode::TypeTransformRule::DirectStorage: { case CAGAnchorNode::TypeTransformRule::DirectStorage: {
Type storageType = QuantizedType::castToStorageType(newType); Type storageType = QuantizedType::castToStorageType(newType);
if (failed(validateTypeConversion(storageType, newType, op))) if (failed(validateTypeConversion(storageType, newType, op)))
return; return;
origResultValue->setType(storageType); origResultValue.setType(storageType);
replacedResultValue = replacedResultValue =
b.create<StorageCastOp>(op->getLoc(), newType, origResultValue); b.create<StorageCastOp>(op->getLoc(), newType, origResultValue);
newResultValue = b.create<DequantizeCastOp>( newResultValue = b.create<DequantizeCastOp>(
op->getLoc(), anchor->getOriginalType(), replacedResultValue); op->getLoc(), anchor->getOriginalType(), replacedResultValue);
break; break;
} }
case CAGAnchorNode::TypeTransformRule::ExpressedOnly: case CAGAnchorNode::TypeTransformRule::ExpressedOnly:
Show All 9 Lines if (replacedResultValue) {
// Transform: // Transform:
// origResultValue --> replaceResultValue -> newResultValue // origResultValue --> replaceResultValue -> newResultValue
// \-> [original uses] // \-> [original uses]
// To: // To:
// origResultValue -> replaceResultValue -> // origResultValue -> replaceResultValue ->
// newResultValue -> [original uses] // newResultValue -> [original uses]
// Note that replaceResultValue may equal newResultValue or there may // Note that replaceResultValue may equal newResultValue or there may
// be operands between the two. // be operands between the two.
origResultValue->replaceAllUsesWith(newResultValue); origResultValue.replaceAllUsesWith(newResultValue);
replacedResultValue->getDefiningOp()->replaceUsesOfWith(newResultValue, replacedResultValue.getDefiningOp()->replaceUsesOfWith(newResultValue,
origResultValue); origResultValue);
} }
} }
std::unique_ptr<OpPassBase<ModuleOp>> std::unique_ptr<OpPassBase<ModuleOp>>
mlir::quantizer::createInferQuantizedTypesPass( mlir::quantizer::createInferQuantizedTypesPass(
SolverContext &solverContext, const TargetConfiguration &config) { SolverContext &solverContext, const TargetConfiguration &config) {
return std::make_unique<InferQuantizedTypesPass>(solverContext, config); return std::make_unique<InferQuantizedTypesPass>(solverContext, config);
} }
static PassRegistration<InferQuantizedTypesPass> static PassRegistration<InferQuantizedTypesPass>
pass("quantizer-infer-quantized-types", pass("quantizer-infer-quantized-types",
"Infers quantized types for a module"); "Infers quantized types for a module");

mlir/lib/Target/LLVMIR/ConvertFromLLVMIR.cpp

Show First 20 Lines • Show All 558 Lines • ▼ Show 20 LinesLogicalResult Importer::processFunction(llvm::Function *f) {
} }
// Now that all instructions are guaranteed to have been visited, ensure // Now that all instructions are guaranteed to have been visited, ensure
// any unknown uses we encountered are remapped. // any unknown uses we encountered are remapped.
for (auto &llvmAndUnknown : unknownInstMap) { for (auto &llvmAndUnknown : unknownInstMap) {
assert(instMap.count(llvmAndUnknown.first)); assert(instMap.count(llvmAndUnknown.first));
Value newValue = instMap[llvmAndUnknown.first]; Value newValue = instMap[llvmAndUnknown.first];
Value oldValue = llvmAndUnknown.second->getResult(0); Value oldValue = llvmAndUnknown.second->getResult(0);
oldValue->replaceAllUsesWith(newValue); oldValue.replaceAllUsesWith(newValue);
llvmAndUnknown.second->erase(); llvmAndUnknown.second->erase();
} }
return success(); return success();
} }
LogicalResult Importer::processBasicBlock(llvm::BasicBlock *bb, Block *block) { LogicalResult Importer::processBasicBlock(llvm::BasicBlock *bb, Block *block) {
b.setInsertionPointToStart(block); b.setInsertionPointToStart(block);
for (llvm::Instruction &inst : *bb) { for (llvm::Instruction &inst : *bb) {
▲ Show 20 Lines • Show All 53 Lines • Show Last 20 Lines

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

Show First 20 Lines • Show All 234 Lines • ▼ Show 20 LinesLogicalResult ModuleTranslation::convertBlock(Block &bb, bool ignoreArguments) {
// This step is omitted if "ignoreArguments" is set. The arguments of the // This step is omitted if "ignoreArguments" is set. The arguments of the
// first block have been already made available through the remapping of // first block have been already made available through the remapping of
// LLVM function arguments. // LLVM function arguments.
if (!ignoreArguments) { if (!ignoreArguments) {
auto predecessors = bb.getPredecessors(); auto predecessors = bb.getPredecessors();
unsigned numPredecessors = unsigned numPredecessors =
std::distance(predecessors.begin(), predecessors.end()); std::distance(predecessors.begin(), predecessors.end());
for (auto arg : bb.getArguments()) { for (auto arg : bb.getArguments()) {
auto wrappedType = arg->getType().dyn_cast<LLVM::LLVMType>(); auto wrappedType = arg.getType().dyn_cast<LLVM::LLVMType>();
if (!wrappedType) if (!wrappedType)
return emitError(bb.front().getLoc(), return emitError(bb.front().getLoc(),
"block argument does not have an LLVM type"); "block argument does not have an LLVM type");
llvm::Type *type = wrappedType.getUnderlyingType(); llvm::Type *type = wrappedType.getUnderlyingType();
llvm::PHINode *phi = builder.CreatePHI(type, numPredecessors); llvm::PHINode *phi = builder.CreatePHI(type, numPredecessors);
valueMapping[arg] = phi; valueMapping[arg] = phi;
} }
} }
▲ Show 20 Lines • Show All 159 Lines • ▼ Show 20 LinesLogicalResult ModuleTranslation::convertOneFunction(LLVMFuncOp func) {
unsigned int argIdx = 0; unsigned int argIdx = 0;
for (auto kvp : llvm::zip(func.getArguments(), llvmFunc->args())) { for (auto kvp : llvm::zip(func.getArguments(), llvmFunc->args())) {
llvm::Argument &llvmArg = std::get<1>(kvp); llvm::Argument &llvmArg = std::get<1>(kvp);
BlockArgument mlirArg = std::get<0>(kvp); BlockArgument mlirArg = std::get<0>(kvp);
if (auto attr = func.getArgAttrOfType<BoolAttr>(argIdx, "llvm.noalias")) { if (auto attr = func.getArgAttrOfType<BoolAttr>(argIdx, "llvm.noalias")) {
// NB: Attribute already verified to be boolean, so check if we can indeed // NB: Attribute already verified to be boolean, so check if we can indeed
// attach the attribute to this argument, based on its type. // attach the attribute to this argument, based on its type.
auto argTy = mlirArg->getType().dyn_cast<LLVM::LLVMType>(); auto argTy = mlirArg.getType().dyn_cast<LLVM::LLVMType>();
if (!argTy.getUnderlyingType()->isPointerTy()) if (!argTy.getUnderlyingType()->isPointerTy())
return func.emitError( return func.emitError(
"llvm.noalias attribute attached to LLVM non-pointer argument"); "llvm.noalias attribute attached to LLVM non-pointer argument");
if (attr.getValue()) if (attr.getValue())
llvmArg.addAttr(llvm::Attribute::AttrKind::NoAlias); llvmArg.addAttr(llvm::Attribute::AttrKind::NoAlias);
} }
valueMapping[mlirArg] = &llvmArg; valueMapping[mlirArg] = &llvmArg;
argIdx++; argIdx++;
▲ Show 20 Lines • Show All 89 Lines • Show Last 20 Lines

mlir/lib/Transforms/AffineLoopInvariantCodeMotion.cpp

Show First 20 Lines • Show All 85 Lines • ▼ Show 20 Linesbool isOpLoopInvariant(Operation &op, Value indVar,
} else if (isa<AffineDmaStartOp>(op) || isa<AffineDmaWaitOp>(op)) { } else if (isa<AffineDmaStartOp>(op) || isa<AffineDmaWaitOp>(op)) {
// TODO(asabne): Support DMA ops. // TODO(asabne): Support DMA ops.
return false; return false;
} else if (!isa<ConstantOp>(op)) { } else if (!isa<ConstantOp>(op)) {
if (isMemRefDereferencingOp(op)) { if (isMemRefDereferencingOp(op)) {
Value memref = isa<AffineLoadOp>(op) Value memref = isa<AffineLoadOp>(op)
? cast<AffineLoadOp>(op).getMemRef() ? cast<AffineLoadOp>(op).getMemRef()
: cast<AffineStoreOp>(op).getMemRef(); : cast<AffineStoreOp>(op).getMemRef();
for (auto *user : memref->getUsers()) { for (auto *user : memref.getUsers()) {
// If this memref has a user that is a DMA, give up because these // If this memref has a user that is a DMA, give up because these
// operations write to this memref. // operations write to this memref.
if (isa<AffineDmaStartOp>(op) || isa<AffineDmaWaitOp>(op)) { if (isa<AffineDmaStartOp>(op) || isa<AffineDmaWaitOp>(op)) {
return false; return false;
} }
// If the memref used by the load/store is used in a store elsewhere in // If the memref used by the load/store is used in a store elsewhere in
// the loop nest, we do not hoist. Similarly, if the memref used in a // the loop nest, we do not hoist. Similarly, if the memref used in a
// load is also being stored too, we do not hoist the load. // load is also being stored too, we do not hoist the load.
Show All 14 Lines if (isa<AffineIfOp>(op)) {
// Insert this op in the defined ops list. // Insert this op in the defined ops list.
definedOps.insert(&op); definedOps.insert(&op);
if (op.getNumOperands() == 0 && !isa<AffineTerminatorOp>(op)) { if (op.getNumOperands() == 0 && !isa<AffineTerminatorOp>(op)) {
LLVM_DEBUG(llvm::dbgs() << "\nNon-constant op with 0 operands\n"); LLVM_DEBUG(llvm::dbgs() << "\nNon-constant op with 0 operands\n");
return false; return false;
} }
for (unsigned int i = 0; i < op.getNumOperands(); ++i) { for (unsigned int i = 0; i < op.getNumOperands(); ++i) {
auto *operandSrc = op.getOperand(i)->getDefiningOp(); auto *operandSrc = op.getOperand(i).getDefiningOp();
LLVM_DEBUG( LLVM_DEBUG(
op.getOperand(i)->print(llvm::dbgs() << "\nIterating on operand\n")); op.getOperand(i).print(llvm::dbgs() << "\nIterating on operand\n"));
// If the loop IV is the operand, this op isn't loop invariant. // If the loop IV is the operand, this op isn't loop invariant.
if (indVar == op.getOperand(i)) { if (indVar == op.getOperand(i)) {
LLVM_DEBUG(llvm::dbgs() << "\nLoop IV is the operand\n"); LLVM_DEBUG(llvm::dbgs() << "\nLoop IV is the operand\n");
return false; return false;
} }
if (operandSrc != nullptr) { if (operandSrc != nullptr) {
▲ Show 20 Lines • Show All 103 Lines • Show Last 20 Lines

mlir/lib/Transforms/DialectConversion.cpp

Show First 20 Lines • Show All 221 Lines • ▼ Show 20 Lines for (Block &block : region) {
// Check if this block was converted. // Check if this block was converted.
auto it = conversionInfo.find(&block); auto it = conversionInfo.find(&block);
if (it == conversionInfo.end()) if (it == conversionInfo.end())
return; return;
// Drop all uses of the original arguments and delete the original block. // Drop all uses of the original arguments and delete the original block.
Block *origBlock = it->second.origBlock; Block *origBlock = it->second.origBlock;
for (BlockArgument arg : origBlock->getArguments()) for (BlockArgument arg : origBlock->getArguments())
arg->dropAllUses(); arg.dropAllUses();
conversionInfo.erase(it); conversionInfo.erase(it);
} }
} }
} }
void ArgConverter::discardRewrites(Block *block) { void ArgConverter::discardRewrites(Block *block) {
auto it = conversionInfo.find(block); auto it = conversionInfo.find(block);
if (it == conversionInfo.end()) if (it == conversionInfo.end())
return; return;
Block *origBlock = it->second.origBlock; Block *origBlock = it->second.origBlock;
// Drop all uses of the new block arguments and replace uses of the new block. // Drop all uses of the new block arguments and replace uses of the new block.
for (int i = block->getNumArguments() - 1; i >= 0; --i) for (int i = block->getNumArguments() - 1; i >= 0; --i)
block->getArgument(i)->dropAllUses(); block->getArgument(i).dropAllUses();
block->replaceAllUsesWith(origBlock); block->replaceAllUsesWith(origBlock);
// Move the operations back the original block and the delete the new block. // Move the operations back the original block and the delete the new block.
origBlock->getOperations().splice(origBlock->end(), block->getOperations()); origBlock->getOperations().splice(origBlock->end(), block->getOperations());
origBlock->moveBefore(block); origBlock->moveBefore(block);
block->erase(); block->erase();
conversionInfo.erase(it); conversionInfo.erase(it);
Show All 11 Lines for (unsigned i = 0, e = origBlock->getNumArguments(); i != e; ++i) {
BlockArgument origArg = origBlock->getArgument(i); BlockArgument origArg = origBlock->getArgument(i);
// Handle the case of a 1->0 value mapping. // Handle the case of a 1->0 value mapping.
if (!argInfo) { if (!argInfo) {
// If a replacement value was given for this argument, use that to // If a replacement value was given for this argument, use that to
// replace all uses. // replace all uses.
auto argReplacementValue = mapping.lookupOrDefault(origArg); auto argReplacementValue = mapping.lookupOrDefault(origArg);
if (argReplacementValue != origArg) { if (argReplacementValue != origArg) {
origArg->replaceAllUsesWith(argReplacementValue); origArg.replaceAllUsesWith(argReplacementValue);
continue; continue;
} }
// If there are any dangling uses then replace the argument with one // If there are any dangling uses then replace the argument with one
// generated by the type converter. This is necessary as the cast must // generated by the type converter. This is necessary as the cast must
// persist in the IR after conversion. // persist in the IR after conversion.
if (!origArg->use_empty()) { if (!origArg.use_empty()) {
rewriter.setInsertionPointToStart(newBlock); rewriter.setInsertionPointToStart(newBlock);
auto *newOp = typeConverter->materializeConversion( auto *newOp = typeConverter->materializeConversion(
rewriter, origArg->getType(), llvm::None, loc); rewriter, origArg.getType(), llvm::None, loc);
origArg->replaceAllUsesWith(newOp->getResult(0)); origArg.replaceAllUsesWith(newOp->getResult(0));
} }
continue; continue;
} }
// If mapping is 1-1, replace the remaining uses and drop the cast // If mapping is 1-1, replace the remaining uses and drop the cast
// operation. // operation.
// FIXME(riverriddle) This should check that the result type and operand // FIXME(riverriddle) This should check that the result type and operand
// type are the same, otherwise it should force a conversion to be // type are the same, otherwise it should force a conversion to be
// materialized. // materialized.
if (argInfo->newArgSize == 1) { if (argInfo->newArgSize == 1) {
origArg->replaceAllUsesWith( origArg.replaceAllUsesWith(
mapping.lookupOrDefault(newBlock->getArgument(argInfo->newArgIdx))); mapping.lookupOrDefault(newBlock->getArgument(argInfo->newArgIdx)));
continue; continue;
} }
// Otherwise this is a 1->N value mapping. // Otherwise this is a 1->N value mapping.
Value castValue = argInfo->castValue; Value castValue = argInfo->castValue;
assert(argInfo->newArgSize > 1 && castValue && "expected 1->N mapping"); assert(argInfo->newArgSize > 1 && castValue && "expected 1->N mapping");
// If the argument is still used, replace it with the generated cast. // If the argument is still used, replace it with the generated cast.
if (!origArg->use_empty()) if (!origArg.use_empty())
origArg->replaceAllUsesWith(mapping.lookupOrDefault(castValue)); origArg.replaceAllUsesWith(mapping.lookupOrDefault(castValue));
// If all users of the cast were removed, we can drop it. Otherwise, keep // If all users of the cast were removed, we can drop it. Otherwise, keep
// the operation alive and let the user handle any remaining usages. // the operation alive and let the user handle any remaining usages.
if (castValue->use_empty()) if (castValue.use_empty())
castValue->getDefiningOp()->erase(); castValue.getDefiningOp()->erase();
} }
} }
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Conversion // Conversion
Block *ArgConverter::convertSignature(Block *block, Block *ArgConverter::convertSignature(Block *block,
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Lines if (inputMap->size == 1) {
info.argInfo[i] = ConvertedArgInfo(inputMap->inputNo, inputMap->size); info.argInfo[i] = ConvertedArgInfo(inputMap->inputNo, inputMap->size);
continue; continue;
} }
// Otherwise, this is a 1->N mapping. Call into the provided type converter // Otherwise, this is a 1->N mapping. Call into the provided type converter
// to pack the new values. // to pack the new values.
auto replArgs = newArgs.slice(inputMap->inputNo, inputMap->size); auto replArgs = newArgs.slice(inputMap->inputNo, inputMap->size);
Operation *cast = typeConverter->materializeConversion( Operation *cast = typeConverter->materializeConversion(
rewriter, origArg->getType(), replArgs, loc); rewriter, origArg.getType(), replArgs, loc);
assert(cast->getNumResults() == 1 && assert(cast->getNumResults() == 1 &&
cast->getNumOperands() == replArgs.size()); cast->getNumOperands() == replArgs.size());
mapping.map(origArg, cast->getResult(0)); mapping.map(origArg, cast->getResult(0));
info.argInfo[i] = info.argInfo[i] =
ConvertedArgInfo(inputMap->inputNo, inputMap->size, cast->getResult(0)); ConvertedArgInfo(inputMap->inputNo, inputMap->size, cast->getResult(0));
} }
// Remove the original block from the region and return the new one. // Remove the original block from the region and return the new one.
▲ Show 20 Lines • Show All 320 Lines • ▼ Show 20 Lines for (auto *op : llvm::reverse(createdOps))
op->erase(); op->erase();
} }
void ConversionPatternRewriterImpl::applyRewrites() { void ConversionPatternRewriterImpl::applyRewrites() {
// Apply all of the rewrites replacements requested during conversion. // Apply all of the rewrites replacements requested during conversion.
for (auto &repl : replacements) { for (auto &repl : replacements) {
for (unsigned i = 0, e = repl.newValues.size(); i != e; ++i) { for (unsigned i = 0, e = repl.newValues.size(); i != e; ++i) {
if (auto newValue = repl.newValues[i]) if (auto newValue = repl.newValues[i])
repl.op->getResult(i)->replaceAllUsesWith( repl.op->getResult(i).replaceAllUsesWith(
mapping.lookupOrDefault(newValue)); mapping.lookupOrDefault(newValue));
} }
// If this operation defines any regions, drop any pending argument // If this operation defines any regions, drop any pending argument
// rewrites. // rewrites.
if (argConverter.typeConverter && repl.op->getNumRegions()) if (argConverter.typeConverter && repl.op->getNumRegions())
argConverter.notifyOpRemoved(repl.op); argConverter.notifyOpRemoved(repl.op);
} }
▲ Show 20 Lines • Show All 137 Lines • ▼ Show 20 Lines
/// Apply a signature conversion to the entry block of the given region. /// Apply a signature conversion to the entry block of the given region.
Block *ConversionPatternRewriter::applySignatureConversion( Block *ConversionPatternRewriter::applySignatureConversion(
Region *region, TypeConverter::SignatureConversion &conversion) { Region *region, TypeConverter::SignatureConversion &conversion) {
return impl->applySignatureConversion(region, conversion); return impl->applySignatureConversion(region, conversion);
} }
void ConversionPatternRewriter::replaceUsesOfBlockArgument(BlockArgument from, void ConversionPatternRewriter::replaceUsesOfBlockArgument(BlockArgument from,
Value to) { Value to) {
for (auto &u : from->getUses()) { for (auto &u : from.getUses()) {
if (u.getOwner() == to->getDefiningOp()) if (u.getOwner() == to.getDefiningOp())
continue; continue;
u.getOwner()->replaceUsesOfWith(from, to); u.getOwner()->replaceUsesOfWith(from, to);
} }
impl->mapping.map(impl->mapping.lookupOrDefault(from), to); impl->mapping.map(impl->mapping.lookupOrDefault(from), to);
} }
/// Return the converted value that replaces 'key'. Return 'key' if there is /// Return the converted value that replaces 'key'. Return 'key' if there is
/// no such a converted value. /// no such a converted value.
▲ Show 20 Lines • Show All 784 Lines • ▼ Show 20 Lines
/// This function converts the type signature of the given block, by invoking /// This function converts the type signature of the given block, by invoking
/// 'convertSignatureArg' for each argument. This function should return a valid /// 'convertSignatureArg' for each argument. This function should return a valid
/// conversion for the signature on success, None otherwise. /// conversion for the signature on success, None otherwise.
auto TypeConverter::convertBlockSignature(Block *block) auto TypeConverter::convertBlockSignature(Block *block)
-> Optional<SignatureConversion> { -> Optional<SignatureConversion> {
SignatureConversion conversion(block->getNumArguments()); SignatureConversion conversion(block->getNumArguments());
for (unsigned i = 0, e = block->getNumArguments(); i != e; ++i) for (unsigned i = 0, e = block->getNumArguments(); i != e; ++i)
if (failed(convertSignatureArg(i, block->getArgument(i)->getType(), if (failed(convertSignatureArg(i, block->getArgument(i).getType(),
conversion))) conversion)))
return llvm::None; return llvm::None;
return conversion; return conversion;
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ConversionTarget // ConversionTarget
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 170 Lines • Show Last 20 Lines

mlir/lib/Transforms/LoopFusion.cpp

Show First 20 Lines • Show All 295 Lines • ▼ Show 20 Linespublic:
} }
// Returns true if node 'id' writes to any memref which escapes (or is an // Returns true if node 'id' writes to any memref which escapes (or is an
// argument to) the function/block. Returns false otherwise. // argument to) the function/block. Returns false otherwise.
bool writesToLiveInOrEscapingMemrefs(unsigned id) { bool writesToLiveInOrEscapingMemrefs(unsigned id) {
Node *node = getNode(id); Node *node = getNode(id);
for (auto *storeOpInst : node->stores) { for (auto *storeOpInst : node->stores) {
auto memref = cast<AffineStoreOp>(storeOpInst).getMemRef(); auto memref = cast<AffineStoreOp>(storeOpInst).getMemRef();
auto *op = memref->getDefiningOp(); auto *op = memref.getDefiningOp();
// Return true if 'memref' is a block argument. // Return true if 'memref' is a block argument.
if (!op) if (!op)
return true; return true;
// Return true if any use of 'memref' escapes the function. // Return true if any use of 'memref' escapes the function.
for (auto *user : memref->getUsers()) for (auto *user : memref.getUsers())
if (!isMemRefDereferencingOp(*user)) if (!isMemRefDereferencingOp(*user))
return true; return true;
} }
return false; return false;
} }
// Returns the unique AffineStoreOp in `node` that meets all the following: // Returns the unique AffineStoreOp in `node` that meets all the following:
// *) store is the only one that writes to a function-local memref live out // *) store is the only one that writes to a function-local memref live out
▲ Show 20 Lines • Show All 67 Lines • ▼ Show 20 Lines bool hasEdge(unsigned srcId, unsigned dstId, Value value = nullptr) {
return hasOutEdge && hasInEdge; return hasOutEdge && hasInEdge;
} }
// Adds an edge from node 'srcId' to node 'dstId' for 'value'. // Adds an edge from node 'srcId' to node 'dstId' for 'value'.
void addEdge(unsigned srcId, unsigned dstId, Value value) { void addEdge(unsigned srcId, unsigned dstId, Value value) {
if (!hasEdge(srcId, dstId, value)) { if (!hasEdge(srcId, dstId, value)) {
outEdges[srcId].push_back({dstId, value}); outEdges[srcId].push_back({dstId, value});
inEdges[dstId].push_back({srcId, value}); inEdges[dstId].push_back({srcId, value});
if (value->getType().isa<MemRefType>()) if (value.getType().isa<MemRefType>())
memrefEdgeCount[value]++; memrefEdgeCount[value]++;
} }
} }
// Removes an edge from node 'srcId' to node 'dstId' for 'value'. // Removes an edge from node 'srcId' to node 'dstId' for 'value'.
void removeEdge(unsigned srcId, unsigned dstId, Value value) { void removeEdge(unsigned srcId, unsigned dstId, Value value) {
assert(inEdges.count(dstId) > 0); assert(inEdges.count(dstId) > 0);
assert(outEdges.count(srcId) > 0); assert(outEdges.count(srcId) > 0);
if (value->getType().isa<MemRefType>()) { if (value.getType().isa<MemRefType>()) {
assert(memrefEdgeCount.count(value) > 0); assert(memrefEdgeCount.count(value) > 0);
memrefEdgeCount[value]--; memrefEdgeCount[value]--;
} }
// Remove 'srcId' from 'inEdges[dstId]'. // Remove 'srcId' from 'inEdges[dstId]'.
for (auto it = inEdges[dstId].begin(); it != inEdges[dstId].end(); ++it) { for (auto it = inEdges[dstId].begin(); it != inEdges[dstId].end(); ++it) {
if ((*it).id == srcId && (*it).value == value) { if ((*it).id == srcId && (*it).value == value) {
inEdges[dstId].erase(it); inEdges[dstId].erase(it);
break; break;
▲ Show 20 Lines • Show All 218 Lines • ▼ Show 20 Linespublic:
} }
// Calls 'callback' for each edge in 'edges' which carries a memref // Calls 'callback' for each edge in 'edges' which carries a memref
// dependence. // dependence.
void forEachMemRefEdge(ArrayRef<Edge> edges, void forEachMemRefEdge(ArrayRef<Edge> edges,
const std::function<void(Edge)> &callback) { const std::function<void(Edge)> &callback) {
for (auto &edge : edges) { for (auto &edge : edges) {
// Skip if 'edge' is not a memref dependence edge. // Skip if 'edge' is not a memref dependence edge.
if (!edge.value->getType().isa<MemRefType>()) if (!edge.value.getType().isa<MemRefType>())
continue; continue;
assert(nodes.count(edge.id) > 0); assert(nodes.count(edge.id) > 0);
// Skip if 'edge.id' is not a loop nest. // Skip if 'edge.id' is not a loop nest.
if (!isa<AffineForOp>(getNode(edge.id)->op)) if (!isa<AffineForOp>(getNode(edge.id)->op))
continue; continue;
// Visit current input edge 'edge'. // Visit current input edge 'edge'.
callback(edge); callback(edge);
} }
▲ Show 20 Lines • Show All 84 Lines • ▼ Show 20 Linesbool MemRefDependenceGraph::init(FuncOp f) {
// Add dependence edges between nodes which produce SSA values and their // Add dependence edges between nodes which produce SSA values and their
// users. // users.
for (auto &idAndNode : nodes) { for (auto &idAndNode : nodes) {
const Node &node = idAndNode.second; const Node &node = idAndNode.second;
if (!node.loads.empty() || !node.stores.empty()) if (!node.loads.empty() || !node.stores.empty())
continue; continue;
auto *opInst = node.op; auto *opInst = node.op;
for (auto value : opInst->getResults()) { for (auto value : opInst->getResults()) {
for (auto *user : value->getUsers()) { for (auto *user : value.getUsers()) {
SmallVector<AffineForOp, 4> loops; SmallVector<AffineForOp, 4> loops;
getLoopIVs(*user, &loops); getLoopIVs(*user, &loops);
if (loops.empty()) if (loops.empty())
continue; continue;
assert(forToNodeMap.count(loops[0].getOperation()) > 0); assert(forToNodeMap.count(loops[0].getOperation()) > 0);
unsigned userLoopNestId = forToNodeMap[loops[0].getOperation()]; unsigned userLoopNestId = forToNodeMap[loops[0].getOperation()];
addEdge(node.id, userLoopNestId, value); addEdge(node.id, userLoopNestId, value);
} }
▲ Show 20 Lines • Show All 144 Lines • ▼ Show 20 Linesstatic Value createPrivateMemRef(AffineForOp forOp, Operation *srcStoreOpInst,
auto *forInst = forOp.getOperation(); auto *forInst = forOp.getOperation();
// Create builder to insert alloc op just before 'forOp'. // Create builder to insert alloc op just before 'forOp'.
OpBuilder b(forInst); OpBuilder b(forInst);
// Builder to create constants at the top level. // Builder to create constants at the top level.
OpBuilder top(forInst->getParentOfType<FuncOp>().getBody()); OpBuilder top(forInst->getParentOfType<FuncOp>().getBody());
// Create new memref type based on slice bounds. // Create new memref type based on slice bounds.
auto oldMemRef = cast<AffineStoreOp>(srcStoreOpInst).getMemRef(); auto oldMemRef = cast<AffineStoreOp>(srcStoreOpInst).getMemRef();
auto oldMemRefType = oldMemRef->getType().cast<MemRefType>(); auto oldMemRefType = oldMemRef.getType().cast<MemRefType>();
unsigned rank = oldMemRefType.getRank(); unsigned rank = oldMemRefType.getRank();
// Compute MemRefRegion for 'srcStoreOpInst' at depth 'dstLoopDepth'. // Compute MemRefRegion for 'srcStoreOpInst' at depth 'dstLoopDepth'.
MemRefRegion region(srcStoreOpInst->getLoc()); MemRefRegion region(srcStoreOpInst->getLoc());
bool validRegion = succeeded(region.compute(srcStoreOpInst, dstLoopDepth)); bool validRegion = succeeded(region.compute(srcStoreOpInst, dstLoopDepth));
(void)validRegion; (void)validRegion;
assert(validRegion && "unexpected memref region failure"); assert(validRegion && "unexpected memref region failure");
SmallVector<int64_t, 4> newShape; SmallVector<int64_t, 4> newShape;
▲ Show 20 Lines • Show All 737 Lines • ▼ Show 20 Lines while (!worklist.empty()) {
// TODO(andydavis) Use union of memref write regions to compute // TODO(andydavis) Use union of memref write regions to compute
// private memref footprint. // private memref footprint.
auto newMemRef = createPrivateMemRef( auto newMemRef = createPrivateMemRef(
dstAffineForOp, storesForMemref[0], bestDstLoopDepth, dstAffineForOp, storesForMemref[0], bestDstLoopDepth,
fastMemorySpace, localBufSizeThreshold); fastMemorySpace, localBufSizeThreshold);
visitedMemrefs.insert(newMemRef); visitedMemrefs.insert(newMemRef);
// Create new node in dependence graph for 'newMemRef' alloc op. // Create new node in dependence graph for 'newMemRef' alloc op.
unsigned newMemRefNodeId = unsigned newMemRefNodeId =
mdg->addNode(newMemRef->getDefiningOp()); mdg->addNode(newMemRef.getDefiningOp());
// Add edge from 'newMemRef' node to dstNode. // Add edge from 'newMemRef' node to dstNode.
mdg->addEdge(newMemRefNodeId, dstId, newMemRef); mdg->addEdge(newMemRefNodeId, dstId, newMemRef);
} }
// Collect dst loop stats after memref privatization transformation. // Collect dst loop stats after memref privatization transformation.
LoopNestStateCollector dstLoopCollector; LoopNestStateCollector dstLoopCollector;
dstLoopCollector.collect(dstAffineForOp.getOperation()); dstLoopCollector.collect(dstAffineForOp.getOperation());
▲ Show 20 Lines • Show All 163 Lines • ▼ Show 20 Lines auto canFuseWithSibNode = [&](Node *sibNode, Value memref) {
if (storeMemrefs.size() != 1) if (storeMemrefs.size() != 1)
return false; return false;
return true; return true;
}; };
// Search for siblings which load the same memref function argument. // Search for siblings which load the same memref function argument.
auto fn = dstNode->op->getParentOfType<FuncOp>(); auto fn = dstNode->op->getParentOfType<FuncOp>();
for (unsigned i = 0, e = fn.getNumArguments(); i != e; ++i) { for (unsigned i = 0, e = fn.getNumArguments(); i != e; ++i) {
for (auto *user : fn.getArgument(i)->getUsers()) { for (auto *user : fn.getArgument(i).getUsers()) {
if (auto loadOp = dyn_cast<AffineLoadOp>(user)) { if (auto loadOp = dyn_cast<AffineLoadOp>(user)) {
// Gather loops surrounding 'use'. // Gather loops surrounding 'use'.
SmallVector<AffineForOp, 4> loops; SmallVector<AffineForOp, 4> loops;
getLoopIVs(*user, &loops); getLoopIVs(*user, &loops);
// Skip 'use' if it is not within a loop nest. // Skip 'use' if it is not within a loop nest.
if (loops.empty()) if (loops.empty())
continue; continue;
Node *sibNode = mdg->getForOpNode(loops[0]); Node *sibNode = mdg->getForOpNode(loops[0]);
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Linespublic:
// Clean up any allocs with no users. // Clean up any allocs with no users.
void eraseUnusedMemRefAllocations() { void eraseUnusedMemRefAllocations() {
for (auto &pair : mdg->memrefEdgeCount) { for (auto &pair : mdg->memrefEdgeCount) {
if (pair.second > 0) if (pair.second > 0)
continue; continue;
auto memref = pair.first; auto memref = pair.first;
// Skip if there exist other uses (return operation or function calls). // Skip if there exist other uses (return operation or function calls).
if (!memref->use_empty()) if (!memref.use_empty())
continue; continue;
// Use list expected to match the dep graph info. // Use list expected to match the dep graph info.
auto *op = memref->getDefiningOp(); auto *op = memref.getDefiningOp();
if (isa_and_nonnull<AllocOp>(op)) if (isa_and_nonnull<AllocOp>(op))
op->erase(); op->erase();
} }
} }
}; };
} // end anonymous namespace } // end anonymous namespace
Show All 22 Lines

mlir/lib/Transforms/LoopInvariantCodeMotion.cpp

Show First 20 Lines • Show All 78 Lines • ▼ Show 20 Linesstatic LogicalResult moveLoopInvariantCode(LoopLikeOpInterface looplike,
// We use two collections here as we need to preserve the order for insertion // We use two collections here as we need to preserve the order for insertion
// and this is easiest. // and this is easiest.
SmallPtrSet<Operation *, 8> willBeMovedSet; SmallPtrSet<Operation *, 8> willBeMovedSet;
SmallVector<Operation *, 8> opsToMove; SmallVector<Operation *, 8> opsToMove;
// Helper to check whether an operation is loop invariant wrt. SSA properties. // Helper to check whether an operation is loop invariant wrt. SSA properties.
auto isDefinedOutsideOfBody = [&](Value value) { auto isDefinedOutsideOfBody = [&](Value value) {
auto definingOp = value->getDefiningOp(); auto definingOp = value.getDefiningOp();
return (definingOp && !!willBeMovedSet.count(definingOp)) || return (definingOp && !!willBeMovedSet.count(definingOp)) ||
looplike.isDefinedOutsideOfLoop(value); looplike.isDefinedOutsideOfLoop(value);
}; };
// Do not use walk here, as we do not want to go into nested regions and hoist // Do not use walk here, as we do not want to go into nested regions and hoist
// operations from there. These regions might have semantics unknown to this // operations from there. These regions might have semantics unknown to this
// rewriting. If the nested regions are loops, they will have been processed. // rewriting. If the nested regions are loops, they will have been processed.
for (auto &block : loopBody) { for (auto &block : loopBody) {
▲ Show 20 Lines • Show All 45 Lines • Show Last 20 Lines

mlir/lib/Transforms/LoopTiling.cpp

Show First 20 Lines • Show All 235 Lines • ▼ Show 20 Lines if (!cst.isHyperRectangular(0, width)) {
rootAffineForOp.emitError("tiled code generation unimplemented for the " rootAffineForOp.emitError("tiled code generation unimplemented for the "
"non-hyperrectangular case"); "non-hyperrectangular case");
return failure(); return failure();
} }
constructTiledIndexSetHyperRect(origLoops, newLoops, tileSizes); constructTiledIndexSetHyperRect(origLoops, newLoops, tileSizes);
// In this case, the point loop IVs just replace the original ones. // In this case, the point loop IVs just replace the original ones.
for (unsigned i = 0; i < width; i++) { for (unsigned i = 0; i < width; i++) {
origLoopIVs[i]->replaceAllUsesWith(newLoops[i + width].getInductionVar()); origLoopIVs[i].replaceAllUsesWith(newLoops[i + width].getInductionVar());
} }
// Erase the old loop nest. // Erase the old loop nest.
rootAffineForOp.erase(); rootAffineForOp.erase();
return success(); return success();
} }
▲ Show 20 Lines • Show All 150 Lines • Show Last 20 Lines

mlir/lib/Transforms/LoopUnrollAndJam.cpp

Show First 20 Lines • Show All 205 Lines • ▼ Show 20 Lines for (unsigned i = unrollJamFactor - 1; i >= 1; --i) {
BlockAndValueMapping operandMapping; BlockAndValueMapping operandMapping;
for (auto &subBlock : subBlocks) { for (auto &subBlock : subBlocks) {
// Builder to insert unroll-jammed bodies. Insert right at the end of // Builder to insert unroll-jammed bodies. Insert right at the end of
// sub-block. // sub-block.
OpBuilder builder(subBlock.first->getBlock(), std::next(subBlock.second)); OpBuilder builder(subBlock.first->getBlock(), std::next(subBlock.second));
// If the induction variable is used, create a remapping to the value for // If the induction variable is used, create a remapping to the value for
// this unrolled instance. // this unrolled instance.
if (!forOpIV->use_empty()) { if (!forOpIV.use_empty()) {
// iv' = iv + i, i = 1 to unrollJamFactor-1. // iv' = iv + i, i = 1 to unrollJamFactor-1.
auto d0 = builder.getAffineDimExpr(0); auto d0 = builder.getAffineDimExpr(0);
auto bumpMap = AffineMap::get(1, 0, {d0 + i * step}); auto bumpMap = AffineMap::get(1, 0, {d0 + i * step});
auto ivUnroll = auto ivUnroll =
builder.create<AffineApplyOp>(forInst->getLoc(), bumpMap, forOpIV); builder.create<AffineApplyOp>(forInst->getLoc(), bumpMap, forOpIV);
operandMapping.map(forOpIV, ivUnroll); operandMapping.map(forOpIV, ivUnroll);
} }
// Clone the sub-block being unroll-jammed. // Clone the sub-block being unroll-jammed.
Show All 13 Lines

mlir/lib/Transforms/MemRefDataFlowOpt.cpp

Show First 20 Lines • Show All 87 Lines • ▼ Show 20 Lines
void MemRefDataFlowOpt::forwardStoreToLoad(AffineLoadOp loadOp) { void MemRefDataFlowOpt::forwardStoreToLoad(AffineLoadOp loadOp) {
Operation *loadOpInst = loadOp.getOperation(); Operation *loadOpInst = loadOp.getOperation();
// First pass over the use list to get minimum number of surrounding // First pass over the use list to get minimum number of surrounding
// loops common between the load op and the store op, with min taken across // loops common between the load op and the store op, with min taken across
// all store ops. // all store ops.
SmallVector<Operation *, 8> storeOps; SmallVector<Operation *, 8> storeOps;
unsigned minSurroundingLoops = getNestingDepth(*loadOpInst); unsigned minSurroundingLoops = getNestingDepth(*loadOpInst);
for (auto *user : loadOp.getMemRef()->getUsers()) { for (auto *user : loadOp.getMemRef().getUsers()) {
auto storeOp = dyn_cast<AffineStoreOp>(user); auto storeOp = dyn_cast<AffineStoreOp>(user);
if (!storeOp) if (!storeOp)
continue; continue;
auto *storeOpInst = storeOp.getOperation(); auto *storeOpInst = storeOp.getOperation();
unsigned nsLoops = getNumCommonSurroundingLoops(*loadOpInst, *storeOpInst); unsigned nsLoops = getNumCommonSurroundingLoops(*loadOpInst, *storeOpInst);
minSurroundingLoops = std::min(nsLoops, minSurroundingLoops); minSurroundingLoops = std::min(nsLoops, minSurroundingLoops);
storeOps.push_back(storeOpInst); storeOps.push_back(storeOpInst);
} }
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Lines for (auto *loadOp : loadOpsToErase) {
loadOp->erase(); loadOp->erase();
} }
// Check if the store fwd'ed memrefs are now left with only stores and can // Check if the store fwd'ed memrefs are now left with only stores and can
// thus be completely deleted. Note: the canonicalize pass should be able // thus be completely deleted. Note: the canonicalize pass should be able
// to do this as well, but we'll do it here since we collected these anyway. // to do this as well, but we'll do it here since we collected these anyway.
for (auto memref : memrefsToErase) { for (auto memref : memrefsToErase) {
// If the memref hasn't been alloc'ed in this function, skip. // If the memref hasn't been alloc'ed in this function, skip.
Operation *defInst = memref->getDefiningOp(); Operation *defInst = memref.getDefiningOp();
if (!defInst || !isa<AllocOp>(defInst)) if (!defInst || !isa<AllocOp>(defInst))
// TODO(mlir-team): if the memref was returned by a 'call' operation, we // TODO(mlir-team): if the memref was returned by a 'call' operation, we
// could still erase it if the call had no side-effects. // could still erase it if the call had no side-effects.
continue; continue;
if (llvm::any_of(memref->getUsers(), [&](Operation *ownerInst) { if (llvm::any_of(memref.getUsers(), [&](Operation *ownerInst) {
return (!isa<AffineStoreOp>(ownerInst) && !isa<DeallocOp>(ownerInst)); return (!isa<AffineStoreOp>(ownerInst) && !isa<DeallocOp>(ownerInst));
})) }))
continue; continue;
// Erase all stores, the dealloc, and the alloc on the memref. // Erase all stores, the dealloc, and the alloc on the memref.
for (auto *user : llvm::make_early_inc_range(memref->getUsers())) for (auto *user : llvm::make_early_inc_range(memref.getUsers()))
user->erase(); user->erase();
defInst->erase(); defInst->erase();
} }
} }
static PassRegistration<MemRefDataFlowOpt> static PassRegistration<MemRefDataFlowOpt>
pass("memref-dataflow-opt", "Perform store/load forwarding for memrefs"); pass("memref-dataflow-opt", "Perform store/load forwarding for memrefs");

mlir/lib/Transforms/PipelineDataTransfer.cpp

Show First 20 Lines • Show All 72 Lines • ▼ Show 20 Lines auto doubleShape = [&](MemRefType oldMemRefType) -> MemRefType {
newShape[0] = 2; newShape[0] = 2;
std::copy(oldShape.begin(), oldShape.end(), newShape.begin() + 1); std::copy(oldShape.begin(), oldShape.end(), newShape.begin() + 1);
auto newMemRefType = auto newMemRefType =
MemRefType::get(newShape, oldMemRefType.getElementType(), {}, MemRefType::get(newShape, oldMemRefType.getElementType(), {},
oldMemRefType.getMemorySpace()); oldMemRefType.getMemorySpace());
return newMemRefType; return newMemRefType;
}; };
auto oldMemRefType = oldMemRef->getType().cast<MemRefType>(); auto oldMemRefType = oldMemRef.getType().cast<MemRefType>();
auto newMemRefType = doubleShape(oldMemRefType); auto newMemRefType = doubleShape(oldMemRefType);
// The double buffer is allocated right before 'forInst'. // The double buffer is allocated right before 'forInst'.
auto *forInst = forOp.getOperation(); auto *forInst = forOp.getOperation();
OpBuilder bOuter(forInst); OpBuilder bOuter(forInst);
// Put together alloc operands for any dynamic dimensions of the memref. // Put together alloc operands for any dynamic dimensions of the memref.
SmallVector<Value, 4> allocOperands; SmallVector<Value, 4> allocOperands;
unsigned dynamicDimCount = 0; unsigned dynamicDimCount = 0;
▲ Show 20 Lines • Show All 110 Lines • ▼ Show 20 Lines for (; it != outgoingDmaOps.end(); ++it) {
break; break;
} }
if (it != outgoingDmaOps.end()) if (it != outgoingDmaOps.end())
continue; continue;
// We only double buffer if the buffer is not live out of loop. // We only double buffer if the buffer is not live out of loop.
auto memref = dmaStartOp.getOperand(dmaStartOp.getFasterMemPos()); auto memref = dmaStartOp.getOperand(dmaStartOp.getFasterMemPos());
bool escapingUses = false; bool escapingUses = false;
for (auto *user : memref->getUsers()) { for (auto *user : memref.getUsers()) {
// We can double buffer regardless of dealloc's outside the loop. // We can double buffer regardless of dealloc's outside the loop.
if (isa<DeallocOp>(user)) if (isa<DeallocOp>(user))
continue; continue;
if (!forOp.getBody()->findAncestorOpInBlock(*user)) { if (!forOp.getBody()->findAncestorOpInBlock(*user)) {
LLVM_DEBUG(llvm::dbgs() LLVM_DEBUG(llvm::dbgs()
<< "can't pipeline: buffer is live out of loop\n";); << "can't pipeline: buffer is live out of loop\n";);
escapingUses = true; escapingUses = true;
break; break;
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Lines if (!doubleBuffer(oldMemRef, forOp)) {
return; return;
} }
// If the old memref has no more uses, remove its 'dead' alloc if it was // If the old memref has no more uses, remove its 'dead' alloc if it was
// alloc'ed. (note: DMA buffers are rarely function live-in; but a 'dim' // alloc'ed. (note: DMA buffers are rarely function live-in; but a 'dim'
// operation could have been used on it if it was dynamically shaped in // operation could have been used on it if it was dynamically shaped in
// order to create the double buffer above.) // order to create the double buffer above.)
// '-canonicalize' does this in a more general way, but we'll anyway do the // '-canonicalize' does this in a more general way, but we'll anyway do the
// simple/common case so that the output / test cases looks clear. // simple/common case so that the output / test cases looks clear.
if (auto *allocInst = oldMemRef->getDefiningOp()) { if (auto *allocInst = oldMemRef.getDefiningOp()) {
if (oldMemRef->use_empty()) { if (oldMemRef.use_empty()) {
allocInst->erase(); allocInst->erase();
} else if (oldMemRef->hasOneUse()) { } else if (oldMemRef.hasOneUse()) {
if (auto dealloc = dyn_cast<DeallocOp>(*oldMemRef->user_begin())) { if (auto dealloc = dyn_cast<DeallocOp>(*oldMemRef.user_begin())) {
dealloc.erase(); dealloc.erase();
allocInst->erase(); allocInst->erase();
} }
} }
} }
} }
// Double the buffers for tag memrefs. // Double the buffers for tag memrefs.
for (auto &pair : startWaitPairs) { for (auto &pair : startWaitPairs) {
auto *dmaFinishInst = pair.second; auto *dmaFinishInst = pair.second;
Value oldTagMemRef = Value oldTagMemRef =
dmaFinishInst->getOperand(getTagMemRefPos(*dmaFinishInst)); dmaFinishInst->getOperand(getTagMemRefPos(*dmaFinishInst));
if (!doubleBuffer(oldTagMemRef, forOp)) { if (!doubleBuffer(oldTagMemRef, forOp)) {
LLVM_DEBUG(llvm::dbgs() << "tag double buffering failed\n";); LLVM_DEBUG(llvm::dbgs() << "tag double buffering failed\n";);
return; return;
} }
// If the old tag has no uses or a single dealloc use, remove it. // If the old tag has no uses or a single dealloc use, remove it.
// (canonicalization handles more complex cases). // (canonicalization handles more complex cases).
if (auto *tagAllocInst = oldTagMemRef->getDefiningOp()) { if (auto *tagAllocInst = oldTagMemRef.getDefiningOp()) {
if (oldTagMemRef->use_empty()) { if (oldTagMemRef.use_empty()) {
tagAllocInst->erase(); tagAllocInst->erase();
} else if (oldTagMemRef->hasOneUse()) { } else if (oldTagMemRef.hasOneUse()) {
if (auto dealloc = dyn_cast<DeallocOp>(*oldTagMemRef->user_begin())) { if (auto dealloc = dyn_cast<DeallocOp>(*oldTagMemRef.user_begin())) {
dealloc.erase(); dealloc.erase();
tagAllocInst->erase(); tagAllocInst->erase();
} }
} }
} }
} }
// Double buffering would have invalidated all the old DMA start/wait insts. // Double buffering would have invalidated all the old DMA start/wait insts.
▲ Show 20 Lines • Show All 64 Lines • Show Last 20 Lines

mlir/lib/Transforms/Utils/FoldUtils.cpp

Show First 20 Lines • Show All 91 Lines • ▼ Show 20 Lines if (preReplaceAction)
preReplaceAction(op); preReplaceAction(op);
// Check to see if the operation was just updated in place. // Check to see if the operation was just updated in place.
if (results.empty()) if (results.empty())
return success(); return success();
// Otherwise, replace all of the result values and erase the operation. // Otherwise, replace all of the result values and erase the operation.
for (unsigned i = 0, e = results.size(); i != e; ++i) for (unsigned i = 0, e = results.size(); i != e; ++i)
op->getResult(i)->replaceAllUsesWith(results[i]); op->getResult(i).replaceAllUsesWith(results[i]);
op->erase(); op->erase();
return success(); return success();
} }
/// Notifies that the given constant `op` should be remove from this /// Notifies that the given constant `op` should be remove from this
/// OperationFolder's internal bookkeeping. /// OperationFolder's internal bookkeeping.
void OperationFolder::notifyRemoval(Operation *op) { void OperationFolder::notifyRemoval(Operation *op) {
// Check to see if this operation is uniqued within the folder. // Check to see if this operation is uniqued within the folder.
auto it = referencedDialects.find(op); auto it = referencedDialects.find(op);
if (it == referencedDialects.end()) if (it == referencedDialects.end())
return; return;
// Get the constant value for this operation, this is the value that was used // Get the constant value for this operation, this is the value that was used
// to unique the operation internally. // to unique the operation internally.
Attribute constValue; Attribute constValue;
matchPattern(op, m_Constant(&constValue)); matchPattern(op, m_Constant(&constValue));
assert(constValue); assert(constValue);
// Get the constant map that this operation was uniqued in. // Get the constant map that this operation was uniqued in.
auto &uniquedConstants = foldScopes[getInsertionRegion(interfaces, op)]; auto &uniquedConstants = foldScopes[getInsertionRegion(interfaces, op)];
// Erase all of the references to this operation. // Erase all of the references to this operation.
auto type = op->getResult(0)->getType(); auto type = op->getResult(0).getType();
for (auto *dialect : it->second) for (auto *dialect : it->second)
uniquedConstants.erase(std::make_tuple(dialect, constValue, type)); uniquedConstants.erase(std::make_tuple(dialect, constValue, type));
referencedDialects.erase(it); referencedDialects.erase(it);
} }
/// Tries to perform folding on the given `op`. If successful, populates /// Tries to perform folding on the given `op`. If successful, populates
/// `results` with the results of the folding. /// `results` with the results of the folding.
LogicalResult OperationFolder::tryToFold( LogicalResult OperationFolder::tryToFold(
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Lines if (auto repl = foldResults[i].dyn_cast<Value>()) {
continue; continue;
} }
// Check to see if there is a canonicalized version of this constant. // Check to see if there is a canonicalized version of this constant.
auto res = op->getResult(i); auto res = op->getResult(i);
Attribute attrRepl = foldResults[i].get<Attribute>(); Attribute attrRepl = foldResults[i].get<Attribute>();
if (auto *constOp = if (auto *constOp =
tryGetOrCreateConstant(uniquedConstants, dialect, builder, attrRepl, tryGetOrCreateConstant(uniquedConstants, dialect, builder, attrRepl,
res->getType(), op->getLoc())) { res.getType(), op->getLoc())) {
results.push_back(constOp->getResult(0)); results.push_back(constOp->getResult(0));
continue; continue;
} }
// If materialization fails, cleanup any operations generated for the // If materialization fails, cleanup any operations generated for the
// previous results and return failure. // previous results and return failure.
for (Operation &op : llvm::make_early_inc_range( for (Operation &op : llvm::make_early_inc_range(
llvm::make_range(entry.begin(), builder.getInsertionPoint()))) { llvm::make_range(entry.begin(), builder.getInsertionPoint()))) {
notifyRemoval(&op); notifyRemoval(&op);
▲ Show 20 Lines • Show All 53 Lines • Show Last 20 Lines

mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp

Show First 20 Lines • Show All 93 Lines • ▼ Show 20 Lines void notifyOperationRemoved(Operation *op) override {
}); });
} }
// When the root of a pattern is about to be replaced, it can trigger // When the root of a pattern is about to be replaced, it can trigger
// simplifications to its users - make sure to add them to the worklist // simplifications to its users - make sure to add them to the worklist
// before the root is changed. // before the root is changed.
void notifyRootReplaced(Operation *op) override { void notifyRootReplaced(Operation *op) override {
for (auto result : op->getResults()) for (auto result : op->getResults())
for (auto *user : result->getUsers()) for (auto *user : result.getUsers())
addToWorklist(user); addToWorklist(user);
} }
private: private:
// Look over the provided operands for any defining operations that should // Look over the provided operands for any defining operations that should
// be re-added to the worklist. This function should be called when an // be re-added to the worklist. This function should be called when an
// operation is modified or removed, as it may trigger further // operation is modified or removed, as it may trigger further
// simplifications. // simplifications.
template <typename Operands> void addToWorklist(Operands &&operands) { template <typename Operands> void addToWorklist(Operands &&operands) {
for (Value operand : operands) { for (Value operand : operands) {
// If the use count of this operand is now < 2, we re-add the defining // If the use count of this operand is now < 2, we re-add the defining
// operation to the worklist. // operation to the worklist.
// TODO(riverriddle) This is based on the fact that zero use operations // TODO(riverriddle) This is based on the fact that zero use operations
// may be deleted, and that single use values often have more // may be deleted, and that single use values often have more
// canonicalization opportunities. // canonicalization opportunities.
if (!operand->use_empty() && !operand->hasOneUse()) if (!operand.use_empty() && !operand.hasOneUse())
continue; continue;
if (auto *defInst = operand->getDefiningOp()) if (auto *defInst = operand.getDefiningOp())
addToWorklist(defInst); addToWorklist(defInst);
} }
} }
/// The low-level pattern matcher. /// The low-level pattern matcher.
RewritePatternMatcher matcher; RewritePatternMatcher matcher;
/// The worklist for this transformation keeps track of the operations that /// The worklist for this transformation keeps track of the operations that
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines while (!worklist.empty()) {
originalOperands.assign(op->operand_begin(), op->operand_end()); originalOperands.assign(op->operand_begin(), op->operand_end());
auto preReplaceAction = [&](Operation *op) { auto preReplaceAction = [&](Operation *op) {
// Add the operands to the worklist for visitation. // Add the operands to the worklist for visitation.
addToWorklist(originalOperands); addToWorklist(originalOperands);
// Add all the users of the result to the worklist so we make sure // Add all the users of the result to the worklist so we make sure
// to revisit them. // to revisit them.
for (auto result : op->getResults()) for (auto result : op->getResults())
for (auto *operand : result->getUsers()) for (auto *operand : result.getUsers())
addToWorklist(operand); addToWorklist(operand);
notifyOperationRemoved(op); notifyOperationRemoved(op);
}; };
// Try to fold this op. // Try to fold this op.
if (succeeded(folder.tryToFold(op, collectOps, preReplaceAction))) { if (succeeded(folder.tryToFold(op, collectOps, preReplaceAction))) {
changed |= true; changed |= true;
▲ Show 20 Lines • Show All 55 Lines • Show Last 20 Lines

mlir/lib/Transforms/Utils/InliningUtils.cpp

Show First 20 Lines • Show All 193 Lines • ▼ Show 20 Lines if (std::next(newBlocks.begin()) == newBlocks.end()) {
// Merge the post insert block into the cloned entry block. // Merge the post insert block into the cloned entry block.
firstNewBlock->getOperations().splice(firstNewBlock->end(), firstNewBlock->getOperations().splice(firstNewBlock->end(),
postInsertBlock->getOperations()); postInsertBlock->getOperations());
postInsertBlock->erase(); postInsertBlock->erase();
} else { } else {
// Otherwise, there were multiple blocks inlined. Add arguments to the post // Otherwise, there were multiple blocks inlined. Add arguments to the post
// insertion block to represent the results to replace. // insertion block to represent the results to replace.
for (Value resultToRepl : resultsToReplace) { for (Value resultToRepl : resultsToReplace) {
resultToRepl->replaceAllUsesWith( resultToRepl.replaceAllUsesWith(
postInsertBlock->addArgument(resultToRepl->getType())); postInsertBlock->addArgument(resultToRepl.getType()));
} }
/// Handle the terminators for each of the new blocks. /// Handle the terminators for each of the new blocks.
for (auto &newBlock : newBlocks) for (auto &newBlock : newBlocks)
interface.handleTerminator(newBlock.getTerminator(), postInsertBlock); interface.handleTerminator(newBlock.getTerminator(), postInsertBlock);
} }
// Splice the instructions of the inlined entry block into the insert block. // Splice the instructions of the inlined entry block into the insert block.
Show All 21 Lines if (inlinedOperands.size() != entryBlock->getNumArguments())
return failure(); return failure();
// Map the provided call operands to the arguments of the region. // Map the provided call operands to the arguments of the region.
BlockAndValueMapping mapper; BlockAndValueMapping mapper;
for (unsigned i = 0, e = inlinedOperands.size(); i != e; ++i) { for (unsigned i = 0, e = inlinedOperands.size(); i != e; ++i) {
// Verify that the types of the provided values match the function argument // Verify that the types of the provided values match the function argument
// types. // types.
BlockArgument regionArg = entryBlock->getArgument(i); BlockArgument regionArg = entryBlock->getArgument(i);
if (inlinedOperands[i]->getType() != regionArg->getType()) if (inlinedOperands[i].getType() != regionArg.getType())
return failure(); return failure();
mapper.map(regionArg, inlinedOperands[i]); mapper.map(regionArg, inlinedOperands[i]);
} }
// Call into the main region inliner function. // Call into the main region inliner function.
return inlineRegion(interface, src, inlinePoint, mapper, resultsToReplace, return inlineRegion(interface, src, inlinePoint, mapper, resultsToReplace,
inlineLoc, shouldCloneInlinedRegion); inlineLoc, shouldCloneInlinedRegion);
} }
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 LinesLogicalResult mlir::inlineCall(InlinerInterface &interface,
// A set of cast operations generated to matchup the signature of the region // A set of cast operations generated to matchup the signature of the region
// with the signature of the call. // with the signature of the call.
SmallVector<Operation *, 4> castOps; SmallVector<Operation *, 4> castOps;
castOps.reserve(callOperands.size() + callResults.size()); castOps.reserve(callOperands.size() + callResults.size());
// Functor used to cleanup generated state on failure. // Functor used to cleanup generated state on failure.
auto cleanupState = [&] { auto cleanupState = [&] {
for (auto *op : castOps) { for (auto *op : castOps) {
op->getResult(0)->replaceAllUsesWith(op->getOperand(0)); op->getResult(0).replaceAllUsesWith(op->getOperand(0));
op->erase(); op->erase();
} }
return failure(); return failure();
}; };
// Builder used for any conversion operations that need to be materialized. // Builder used for any conversion operations that need to be materialized.
OpBuilder castBuilder(call); OpBuilder castBuilder(call);
Location castLoc = call.getLoc(); Location castLoc = call.getLoc();
auto *callInterface = interface.getInterfaceFor(call.getDialect()); auto *callInterface = interface.getInterfaceFor(call.getDialect());
// Map the provided call operands to the arguments of the region. // Map the provided call operands to the arguments of the region.
BlockAndValueMapping mapper; BlockAndValueMapping mapper;
for (unsigned i = 0, e = callOperands.size(); i != e; ++i) { for (unsigned i = 0, e = callOperands.size(); i != e; ++i) {
BlockArgument regionArg = entryBlock->getArgument(i); BlockArgument regionArg = entryBlock->getArgument(i);
Value operand = callOperands[i]; Value operand = callOperands[i];
// If the call operand doesn't match the expected region argument, try to // If the call operand doesn't match the expected region argument, try to
// generate a cast. // generate a cast.
Type regionArgType = regionArg->getType(); Type regionArgType = regionArg.getType();
if (operand->getType() != regionArgType) { if (operand.getType() != regionArgType) {
if (!(operand = materializeConversion(callInterface, castOps, castBuilder, if (!(operand = materializeConversion(callInterface, castOps, castBuilder,
operand, regionArgType, castLoc))) operand, regionArgType, castLoc)))
return cleanupState(); return cleanupState();
} }
mapper.map(regionArg, operand); mapper.map(regionArg, operand);
} }
// Ensure that the resultant values of the call, match the callable. // Ensure that the resultant values of the call, match the callable.
castBuilder.setInsertionPointAfter(call); castBuilder.setInsertionPointAfter(call);
for (unsigned i = 0, e = callResults.size(); i != e; ++i) { for (unsigned i = 0, e = callResults.size(); i != e; ++i) {
Value callResult = callResults[i]; Value callResult = callResults[i];
if (callResult->getType() == callableResultTypes[i]) if (callResult.getType() == callableResultTypes[i])
continue; continue;
// Generate a conversion that will produce the original type, so that the IR // Generate a conversion that will produce the original type, so that the IR
// is still valid after the original call gets replaced. // is still valid after the original call gets replaced.
Value castResult = Value castResult =
materializeConversion(callInterface, castOps, castBuilder, callResult, materializeConversion(callInterface, castOps, castBuilder, callResult,
callResult->getType(), castLoc); callResult.getType(), castLoc);
if (!castResult) if (!castResult)
return cleanupState(); return cleanupState();
callResult->replaceAllUsesWith(castResult); callResult.replaceAllUsesWith(castResult);
castResult->getDefiningOp()->replaceUsesOfWith(castResult, callResult); castResult.getDefiningOp()->replaceUsesOfWith(castResult, callResult);
} }
// Attempt to inline the call. // Attempt to inline the call.
if (failed(inlineRegion(interface, src, call, mapper, callResults, if (failed(inlineRegion(interface, src, call, mapper, callResults,
call.getLoc(), shouldCloneInlinedRegion))) call.getLoc(), shouldCloneInlinedRegion)))
return cleanupState(); return cleanupState();
return success(); return success();
} }

mlir/lib/Transforms/Utils/LoopFusionUtils.cpp

Show First 20 Lines • Show All 107 Lines • ▼ Show 20 Lines opX->walk([&](Operation *op) {
if (isa<AffineLoadOp>(op) || isa<AffineStoreOp>(op)) { if (isa<AffineLoadOp>(op) || isa<AffineStoreOp>(op)) {
if (isDependentLoadOrStoreOp(op, values)) { if (isDependentLoadOrStoreOp(op, values)) {
lastDepOp = opX; lastDepOp = opX;
return WalkResult::interrupt(); return WalkResult::interrupt();
} }
return WalkResult::advance(); return WalkResult::advance();
} }
for (auto value : op->getResults()) { for (auto value : op->getResults()) {
for (auto user : value->getUsers()) { for (auto user : value.getUsers()) {
SmallVector<AffineForOp, 4> loops; SmallVector<AffineForOp, 4> loops;
// Check if any loop in loop nest surrounding 'user' is 'opB'. // Check if any loop in loop nest surrounding 'user' is 'opB'.
getLoopIVs(*user, &loops); getLoopIVs(*user, &loops);
if (llvm::is_contained(loops, cast<AffineForOp>(opB))) { if (llvm::is_contained(loops, cast<AffineForOp>(opB))) {
lastDepOp = opX; lastDepOp = opX;
return WalkResult::interrupt(); return WalkResult::interrupt();
} }
} }
▲ Show 20 Lines • Show All 317 Lines • ▼ Show 20 Lines srcForOp.walk([&](Operation *op) {
} }
}); });
// Subtract out any store ops in single-iteration src slice loop nest. // Subtract out any store ops in single-iteration src slice loop nest.
if (storeCount > 0) if (storeCount > 0)
computeCostMap[insertPointParent] = -storeCount; computeCostMap[insertPointParent] = -storeCount;
// Subtract out any load users of 'storeMemrefs' nested below // Subtract out any load users of 'storeMemrefs' nested below
// 'insertPointParent'. // 'insertPointParent'.
for (auto value : storeMemrefs) { for (auto value : storeMemrefs) {
for (auto *user : value->getUsers()) { for (auto *user : value.getUsers()) {
if (auto loadOp = dyn_cast<AffineLoadOp>(user)) { if (auto loadOp = dyn_cast<AffineLoadOp>(user)) {
SmallVector<AffineForOp, 4> loops; SmallVector<AffineForOp, 4> loops;
// Check if any loop in loop nest surrounding 'user' is // Check if any loop in loop nest surrounding 'user' is
// 'insertPointParent'. // 'insertPointParent'.
getLoopIVs(*user, &loops); getLoopIVs(*user, &loops);
if (llvm::is_contained(loops, cast<AffineForOp>(insertPointParent))) { if (llvm::is_contained(loops, cast<AffineForOp>(insertPointParent))) {
if (auto forOp = if (auto forOp =
dyn_cast_or_null<AffineForOp>(user->getParentOp())) { dyn_cast_or_null<AffineForOp>(user->getParentOp())) {
Show All 22 Lines

mlir/lib/Transforms/Utils/LoopUtils.cpp

Show First 20 Lines • Show All 91 Lines • ▼ Show 20 Linesvoid mlir::getCleanupLoopLowerBound(AffineForOp forOp, unsigned unrollFactor,
operands->append(bumpValues.begin(), bumpValues.end()); operands->append(bumpValues.begin(), bumpValues.end());
*map = AffineMap::get(1 + tripCountMap.getNumResults(), 0, newUbExprs); *map = AffineMap::get(1 + tripCountMap.getNumResults(), 0, newUbExprs);
// Simplify the map + operands. // Simplify the map + operands.
fullyComposeAffineMapAndOperands(map, operands); fullyComposeAffineMapAndOperands(map, operands);
*map = simplifyAffineMap(*map); *map = simplifyAffineMap(*map);
canonicalizeMapAndOperands(map, operands); canonicalizeMapAndOperands(map, operands);
// Remove any affine.apply's that became dead from the simplification above. // Remove any affine.apply's that became dead from the simplification above.
for (auto v : bumpValues) { for (auto v : bumpValues) {
if (v->use_empty()) { if (v.use_empty())
v->getDefiningOp()->erase(); v.getDefiningOp()->erase();
}
} }
if (lb.use_empty()) if (lb.use_empty())
lb.erase(); lb.erase();
} }
/// Promotes the loop body of a forOp to its containing block if the forOp /// Promotes the loop body of a forOp to its containing block if the forOp
/// was known to have a single iteration. /// was known to have a single iteration.
// TODO(bondhugula): extend this for arbitrary affine bounds. // TODO(bondhugula): extend this for arbitrary affine bounds.
LogicalResult mlir::promoteIfSingleIteration(AffineForOp forOp) { LogicalResult mlir::promoteIfSingleIteration(AffineForOp forOp) {
Optional<uint64_t> tripCount = getConstantTripCount(forOp); Optional<uint64_t> tripCount = getConstantTripCount(forOp);
if (!tripCount.hasValue() || tripCount.getValue() != 1) if (!tripCount.hasValue() || tripCount.getValue() != 1)
return failure(); return failure();
// TODO(mlir-team): there is no builder for a max. // TODO(mlir-team): there is no builder for a max.
if (forOp.getLowerBoundMap().getNumResults() != 1) if (forOp.getLowerBoundMap().getNumResults() != 1)
return failure(); return failure();
// Replaces all IV uses to its single iteration value. // Replaces all IV uses to its single iteration value.
auto iv = forOp.getInductionVar(); auto iv = forOp.getInductionVar();
Operation *op = forOp.getOperation(); Operation *op = forOp.getOperation();
if (!iv->use_empty()) { if (!iv.use_empty()) {
if (forOp.hasConstantLowerBound()) { if (forOp.hasConstantLowerBound()) {
OpBuilder topBuilder(op->getParentOfType<FuncOp>().getBody()); OpBuilder topBuilder(op->getParentOfType<FuncOp>().getBody());
auto constOp = topBuilder.create<ConstantIndexOp>( auto constOp = topBuilder.create<ConstantIndexOp>(
forOp.getLoc(), forOp.getConstantLowerBound()); forOp.getLoc(), forOp.getConstantLowerBound());
iv->replaceAllUsesWith(constOp); iv.replaceAllUsesWith(constOp);
} else { } else {
AffineBound lb = forOp.getLowerBound(); AffineBound lb = forOp.getLowerBound();
SmallVector<Value, 4> lbOperands(lb.operand_begin(), lb.operand_end()); SmallVector<Value, 4> lbOperands(lb.operand_begin(), lb.operand_end());
OpBuilder builder(op->getBlock(), Block::iterator(op)); OpBuilder builder(op->getBlock(), Block::iterator(op));
if (lb.getMap() == builder.getDimIdentityMap()) { if (lb.getMap() == builder.getDimIdentityMap()) {
// No need of generating an affine.apply. // No need of generating an affine.apply.
iv->replaceAllUsesWith(lbOperands[0]); iv.replaceAllUsesWith(lbOperands[0]);
} else { } else {
auto affineApplyOp = builder.create<AffineApplyOp>( auto affineApplyOp = builder.create<AffineApplyOp>(
op->getLoc(), lb.getMap(), lbOperands); op->getLoc(), lb.getMap(), lbOperands);
iv->replaceAllUsesWith(affineApplyOp); iv.replaceAllUsesWith(affineApplyOp);
} }
} }
} }
// Move the loop body operations, except for terminator, to the loop's // Move the loop body operations, except for terminator, to the loop's
// containing block. // containing block.
auto *block = op->getBlock(); auto *block = op->getBlock();
forOp.getBody()->getOperations().back().erase(); forOp.getBody()->getOperations().back().erase();
block->getOperations().splice(Block::iterator(op), block->getOperations().splice(Block::iterator(op),
Show All 39 LinesgenerateLoop(AffineMap lbMap, AffineMap ubMap,
for (auto it = instGroupQueue.begin() + offset, e = instGroupQueue.end(); for (auto it = instGroupQueue.begin() + offset, e = instGroupQueue.end();
it != e; ++it) { it != e; ++it) {
uint64_t shift = it->first; uint64_t shift = it->first;
auto insts = it->second; auto insts = it->second;
// All 'same shift' operations get added with their operands being // All 'same shift' operations get added with their operands being
// remapped to results of cloned operations, and their IV used remapped. // remapped to results of cloned operations, and their IV used remapped.
// Generate the remapping if the shift is not zero: remappedIV = newIV - // Generate the remapping if the shift is not zero: remappedIV = newIV -
// shift. // shift.
if (!srcIV->use_empty() && shift != 0) { if (!srcIV.use_empty() && shift != 0) {
auto ivRemap = bodyBuilder.create<AffineApplyOp>( auto ivRemap = bodyBuilder.create<AffineApplyOp>(
srcForInst.getLoc(), srcForInst.getLoc(),
bodyBuilder.getSingleDimShiftAffineMap( bodyBuilder.getSingleDimShiftAffineMap(
-static_cast<int64_t>(srcForInst.getStep() * shift)), -static_cast<int64_t>(srcForInst.getStep() * shift)),
loopChunkIV); loopChunkIV);
operandMap.map(srcIV, ivRemap); operandMap.map(srcIV, ivRemap);
} else { } else {
operandMap.map(srcIV, loopChunkIV); operandMap.map(srcIV, loopChunkIV);
▲ Show 20 Lines • Show All 265 Lines • ▼ Show 20 LinesLogicalResult mlir::loopUnrollByFactor(AffineForOp forOp,
// Unroll the contents of 'forOp' (append unrollFactor-1 additional copies). // Unroll the contents of 'forOp' (append unrollFactor-1 additional copies).
auto forOpIV = forOp.getInductionVar(); auto forOpIV = forOp.getInductionVar();
for (unsigned i = 1; i < unrollFactor; i++) { for (unsigned i = 1; i < unrollFactor; i++) {
BlockAndValueMapping operandMap; BlockAndValueMapping operandMap;
// If the induction variable is used, create a remapping to the value for // If the induction variable is used, create a remapping to the value for
// this unrolled instance. // this unrolled instance.
if (!forOpIV->use_empty()) { if (!forOpIV.use_empty()) {
// iv' = iv + 1/2/3...unrollFactor-1; // iv' = iv + 1/2/3...unrollFactor-1;
auto d0 = builder.getAffineDimExpr(0); auto d0 = builder.getAffineDimExpr(0);
auto bumpMap = AffineMap::get(1, 0, {d0 + i * step}); auto bumpMap = AffineMap::get(1, 0, {d0 + i * step});
auto ivUnroll = auto ivUnroll =
builder.create<AffineApplyOp>(forOp.getLoc(), bumpMap, forOpIV); builder.create<AffineApplyOp>(forOp.getLoc(), bumpMap, forOpIV);
operandMap.map(forOpIV, ivUnroll); operandMap.map(forOpIV, ivUnroll);
} }
▲ Show 20 Lines • Show All 344 Lines • ▼ Show 20 Lines
} }
// Build the IR that performs ceil division of a positive value by a constant: // Build the IR that performs ceil division of a positive value by a constant:
// ceildiv(a, B) = divis(a + (B-1), B) // ceildiv(a, B) = divis(a + (B-1), B)
// where divis is rounding-to-zero division. // where divis is rounding-to-zero division.
static Value ceilDivPositive(OpBuilder &builder, Location loc, Value dividend, static Value ceilDivPositive(OpBuilder &builder, Location loc, Value dividend,
int64_t divisor) { int64_t divisor) {
assert(divisor > 0 && "expected positive divisor"); assert(divisor > 0 && "expected positive divisor");
assert(dividend->getType().isIndex() && "expected index-typed value"); assert(dividend.getType().isIndex() && "expected index-typed value");
Value divisorMinusOneCst = builder.create<ConstantIndexOp>(loc, divisor - 1); Value divisorMinusOneCst = builder.create<ConstantIndexOp>(loc, divisor - 1);
Value divisorCst = builder.create<ConstantIndexOp>(loc, divisor); Value divisorCst = builder.create<ConstantIndexOp>(loc, divisor);
Value sum = builder.create<AddIOp>(loc, dividend, divisorMinusOneCst); Value sum = builder.create<AddIOp>(loc, dividend, divisorMinusOneCst);
return builder.create<SignedDivIOp>(loc, sum, divisorCst); return builder.create<SignedDivIOp>(loc, sum, divisorCst);
} }
// Build the IR that performs ceil division of a positive value by another // Build the IR that performs ceil division of a positive value by another
// positive value: // positive value:
// ceildiv(a, b) = divis(a + (b - 1), b) // ceildiv(a, b) = divis(a + (b - 1), b)
// where divis is rounding-to-zero division. // where divis is rounding-to-zero division.
static Value ceilDivPositive(OpBuilder &builder, Location loc, Value dividend, static Value ceilDivPositive(OpBuilder &builder, Location loc, Value dividend,
Value divisor) { Value divisor) {
assert(dividend->getType().isIndex() && "expected index-typed value"); assert(dividend.getType().isIndex() && "expected index-typed value");
Value cstOne = builder.create<ConstantIndexOp>(loc, 1); Value cstOne = builder.create<ConstantIndexOp>(loc, 1);
Value divisorMinusOne = builder.create<SubIOp>(loc, divisor, cstOne); Value divisorMinusOne = builder.create<SubIOp>(loc, divisor, cstOne);
Value sum = builder.create<AddIOp>(loc, dividend, divisorMinusOne); Value sum = builder.create<AddIOp>(loc, dividend, divisorMinusOne);
return builder.create<SignedDivIOp>(loc, sum, divisor); return builder.create<SignedDivIOp>(loc, sum, divisor);
} }
// Hoist the ops within `outer` that appear before `inner`. // Hoist the ops within `outer` that appear before `inner`.
▲ Show 20 Lines • Show All 102 Lines • ▼ Show 20 LinesTileLoops mlir::extractFixedOuterLoops(loop::ForOp rootForOp,
return tileLoops; return tileLoops;
} }
// Replaces all uses of `orig` with `replacement` except if the user is listed // Replaces all uses of `orig` with `replacement` except if the user is listed
// in `exceptions`. // in `exceptions`.
static void static void
replaceAllUsesExcept(Value orig, Value replacement, replaceAllUsesExcept(Value orig, Value replacement,
const SmallPtrSetImpl<Operation *> &exceptions) { const SmallPtrSetImpl<Operation *> &exceptions) {
for (auto &use : llvm::make_early_inc_range(orig->getUses())) { for (auto &use : llvm::make_early_inc_range(orig.getUses())) {
if (exceptions.count(use.getOwner()) == 0) if (exceptions.count(use.getOwner()) == 0)
use.set(replacement); use.set(replacement);
} }
} }
// Transform a loop with a strictly positive step // Transform a loop with a strictly positive step
// for %i = %lb to %ub step %s // for %i = %lb to %ub step %s
// into a 0-based loop with step 1 // into a 0-based loop with step 1
// for %ii = 0 to ceildiv(%ub - %lb, %s) step 1 { // for %ii = 0 to ceildiv(%ub - %lb, %s) step 1 {
// %i = %ii * %s + %lb // %i = %ii * %s + %lb
// Insert the induction variable remapping in the body of `inner`, which is // Insert the induction variable remapping in the body of `inner`, which is
// expected to be either `loop` or another loop perfectly nested under `loop`. // expected to be either `loop` or another loop perfectly nested under `loop`.
// Insert the definition of new bounds immediate before `outer`, which is // Insert the definition of new bounds immediate before `outer`, which is
// expected to be either `loop` or its parent in the loop nest. // expected to be either `loop` or its parent in the loop nest.
static void normalizeLoop(loop::ForOp loop, loop::ForOp outer, static void normalizeLoop(loop::ForOp loop, loop::ForOp outer,
loop::ForOp inner) { loop::ForOp inner) {
OpBuilder builder(outer); OpBuilder builder(outer);
Location loc = loop.getLoc(); Location loc = loop.getLoc();
// Check if the loop is already known to have a constant zero lower bound or // Check if the loop is already known to have a constant zero lower bound or
// a constant one step. // a constant one step.
bool isZeroBased = false; bool isZeroBased = false;
if (auto ubCst = if (auto ubCst =
dyn_cast_or_null<ConstantIndexOp>(loop.lowerBound()->getDefiningOp())) dyn_cast_or_null<ConstantIndexOp>(loop.lowerBound().getDefiningOp()))
isZeroBased = ubCst.getValue() == 0; isZeroBased = ubCst.getValue() == 0;
bool isStepOne = false; bool isStepOne = false;
if (auto stepCst = if (auto stepCst =
dyn_cast_or_null<ConstantIndexOp>(loop.step()->getDefiningOp())) dyn_cast_or_null<ConstantIndexOp>(loop.step().getDefiningOp()))
isStepOne = stepCst.getValue() == 1; isStepOne = stepCst.getValue() == 1;
if (isZeroBased && isStepOne) if (isZeroBased && isStepOne)
return; return;
// Compute the number of iterations the loop executes: ceildiv(ub - lb, step) // Compute the number of iterations the loop executes: ceildiv(ub - lb, step)
// assuming the step is strictly positive. Update the bounds and the step // assuming the step is strictly positive. Update the bounds and the step
// of the loop to go from 0 to the number of iterations, if necessary. // of the loop to go from 0 to the number of iterations, if necessary.
Show All 20 Linesstatic void normalizeLoop(loop::ForOp loop, loop::ForOp outer,
// from the "normalized" one. // from the "normalized" one.
builder.setInsertionPointToStart(inner.getBody()); builder.setInsertionPointToStart(inner.getBody());
Value scaled = Value scaled =
isStepOne ? loop.getInductionVar() isStepOne ? loop.getInductionVar()
: builder.create<MulIOp>(loc, loop.getInductionVar(), step); : builder.create<MulIOp>(loc, loop.getInductionVar(), step);
Value shifted = Value shifted =
isZeroBased ? scaled : builder.create<AddIOp>(loc, scaled, lb); isZeroBased ? scaled : builder.create<AddIOp>(loc, scaled, lb);
SmallPtrSet<Operation *, 2> preserve{scaled->getDefiningOp(), SmallPtrSet<Operation *, 2> preserve{scaled.getDefiningOp(),
shifted->getDefiningOp()}; shifted.getDefiningOp()};
replaceAllUsesExcept(loop.getInductionVar(), shifted, preserve); replaceAllUsesExcept(loop.getInductionVar(), shifted, preserve);
} }
void mlir::coalesceLoops(MutableArrayRef<loop::ForOp> loops) { void mlir::coalesceLoops(MutableArrayRef<loop::ForOp> loops) {
if (loops.size() < 2) if (loops.size() < 2)
return; return;
loop::ForOp innermost = loops.back(); loop::ForOp innermost = loops.back();
▲ Show 20 Lines • Show All 123 Lines • ▼ Show 20 Linesstatic void getMultiLevelStrides(const MemRefRegion &region,
ArrayRef<int64_t> bufferShape, ArrayRef<int64_t> bufferShape,
SmallVectorImpl<StrideInfo> *strideInfos) { SmallVectorImpl<StrideInfo> *strideInfos) {
if (bufferShape.size() <= 1) if (bufferShape.size() <= 1)
return; return;
int64_t numEltPerStride = 1; int64_t numEltPerStride = 1;
int64_t stride = 1; int64_t stride = 1;
for (int d = bufferShape.size() - 1; d >= 1; d--) { for (int d = bufferShape.size() - 1; d >= 1; d--) {
int64_t dimSize = region.memref->getType().cast<MemRefType>().getDimSize(d); int64_t dimSize = region.memref.getType().cast<MemRefType>().getDimSize(d);
stride *= dimSize; stride *= dimSize;
numEltPerStride *= bufferShape[d]; numEltPerStride *= bufferShape[d];
// A stride is needed only if the region has a shorter extent than the // A stride is needed only if the region has a shorter extent than the
// memref along the dimension *and* has an extent greater than one along the // memref along the dimension *and* has an extent greater than one along the
// next major dimension. // next major dimension.
if (bufferShape[d] < dimSize && bufferShape[d - 1] > 1) { if (bufferShape[d] < dimSize && bufferShape[d - 1] > 1) {
strideInfos->push_back({stride, numEltPerStride}); strideInfos->push_back({stride, numEltPerStride});
} }
▲ Show 20 Lines • Show All 103 Lines • ▼ Show 20 Linesstatic LogicalResult generateCopy(
OpBuilder &b = region.isWrite() ? epilogue : prologue; OpBuilder &b = region.isWrite() ? epilogue : prologue;
// Builder to create constants at the top level. // Builder to create constants at the top level.
auto func = copyPlacementBlock->getParent()->getParentOfType<FuncOp>(); auto func = copyPlacementBlock->getParent()->getParentOfType<FuncOp>();
OpBuilder top(func.getBody()); OpBuilder top(func.getBody());
auto loc = region.loc; auto loc = region.loc;
auto memref = region.memref; auto memref = region.memref;
auto memRefType = memref->getType().cast<MemRefType>(); auto memRefType = memref.getType().cast<MemRefType>();
auto layoutMaps = memRefType.getAffineMaps(); auto layoutMaps = memRefType.getAffineMaps();
if (layoutMaps.size() > 1 || if (layoutMaps.size() > 1 ||
(layoutMaps.size() == 1 && !layoutMaps[0].isIdentity())) { (layoutMaps.size() == 1 && !layoutMaps[0].isIdentity())) {
LLVM_DEBUG(llvm::dbgs() << "Non-identity layout map not yet supported\n"); LLVM_DEBUG(llvm::dbgs() << "Non-identity layout map not yet supported\n");
return failure(); return failure();
} }
▲ Show 20 Lines • Show All 248 Lines • ▼ Show 20 Linesstatic bool getFullMemRefAsRegion(Operation *opInst, unsigned numParamLoopIVs,
} else if (auto storeOp = dyn_cast<AffineStoreOp>(opInst)) { } else if (auto storeOp = dyn_cast<AffineStoreOp>(opInst)) {
rank = storeOp.getMemRefType().getRank(); rank = storeOp.getMemRefType().getRank();
region->memref = storeOp.getMemRef(); region->memref = storeOp.getMemRef();
region->setWrite(true); region->setWrite(true);
} else { } else {
assert(false && "expected load or store op"); assert(false && "expected load or store op");
return false; return false;
} }
auto memRefType = region->memref->getType().cast<MemRefType>(); auto memRefType = region->memref.getType().cast<MemRefType>();
if (!memRefType.hasStaticShape()) if (!memRefType.hasStaticShape())
return false; return false;
auto *regionCst = region->getConstraints(); auto *regionCst = region->getConstraints();
// Just get the first numSymbols IVs, which the memref region is parametric // Just get the first numSymbols IVs, which the memref region is parametric
// on. // on.
SmallVector<AffineForOp, 4> ivs; SmallVector<AffineForOp, 4> ivs;
▲ Show 20 Lines • Show All 208 Lines • Show Last 20 Lines

mlir/lib/Transforms/Utils/RegionUtils.cpp

Show All 14 Lines
#include "llvm/ADT/DepthFirstIterator.h" #include "llvm/ADT/DepthFirstIterator.h"
#include "llvm/ADT/PostOrderIterator.h" #include "llvm/ADT/PostOrderIterator.h"
#include "llvm/ADT/SmallSet.h" #include "llvm/ADT/SmallSet.h"
using namespace mlir; using namespace mlir;
void mlir::replaceAllUsesInRegionWith(Value orig, Value replacement, void mlir::replaceAllUsesInRegionWith(Value orig, Value replacement,
Region &region) { Region &region) {
for (auto &use : llvm::make_early_inc_range(orig->getUses())) { for (auto &use : llvm::make_early_inc_range(orig.getUses())) {
if (region.isAncestor(use.getOwner()->getParentRegion())) if (region.isAncestor(use.getOwner()->getParentRegion()))
use.set(replacement); use.set(replacement);
} }
} }
void mlir::visitUsedValuesDefinedAbove( void mlir::visitUsedValuesDefinedAbove(
Region &region, Region &limit, function_ref<void(OpOperand *)> callback) { Region &region, Region &limit, function_ref<void(OpOperand *)> callback) {
assert(limit.isAncestor(&region) && assert(limit.isAncestor(&region) &&
"expected isolation limit to be an ancestor of the given region"); "expected isolation limit to be an ancestor of the given region");
// Collect proper ancestors of `limit` upfront to avoid traversing the region // Collect proper ancestors of `limit` upfront to avoid traversing the region
// tree for every value. // tree for every value.
SmallPtrSet<Region *, 4> properAncestors; SmallPtrSet<Region *, 4> properAncestors;
for (auto *reg = limit.getParentRegion(); reg != nullptr; for (auto *reg = limit.getParentRegion(); reg != nullptr;
reg = reg->getParentRegion()) { reg = reg->getParentRegion()) {
properAncestors.insert(reg); properAncestors.insert(reg);
} }
region.walk([callback, &properAncestors](Operation *op) { region.walk([callback, &properAncestors](Operation *op) {
for (OpOperand &operand : op->getOpOperands()) for (OpOperand &operand : op->getOpOperands())
// Callback on values defined in a proper ancestor of region. // Callback on values defined in a proper ancestor of region.
if (properAncestors.count(operand.get()->getParentRegion())) if (properAncestors.count(operand.get().getParentRegion()))
callback(&operand); callback(&operand);
}); });
} }
void mlir::visitUsedValuesDefinedAbove( void mlir::visitUsedValuesDefinedAbove(
MutableArrayRef<Region> regions, function_ref<void(OpOperand *)> callback) { MutableArrayRef<Region> regions, function_ref<void(OpOperand *)> callback) {
for (Region &region : regions) for (Region &region : regions)
visitUsedValuesDefinedAbove(region, region, callback); visitUsedValuesDefinedAbove(region, region, callback);
▲ Show 20 Lines • Show All 121 Lines • ▼ Show 20 Lines if (owner->isKnownTerminator()) {
if (auto arg = owner->getSuccessorBlockArgument(operandIndex)) if (auto arg = owner->getSuccessorBlockArgument(operandIndex))
return !liveMap.wasProvenLive(*arg); return !liveMap.wasProvenLive(*arg);
return false; return false;
} }
return false; return false;
} }
static void processValue(Value value, LiveMap &liveMap) { static void processValue(Value value, LiveMap &liveMap) {
bool provedLive = llvm::any_of(value->getUses(), [&](OpOperand &use) { bool provedLive = llvm::any_of(value.getUses(), [&](OpOperand &use) {
if (isUseSpeciallyKnownDead(use, liveMap)) if (isUseSpeciallyKnownDead(use, liveMap))
return false; return false;
return liveMap.wasProvenLive(use.getOwner()); return liveMap.wasProvenLive(use.getOwner());
}); });
if (provedLive) if (provedLive)
liveMap.setProvedLive(value); liveMap.setProvedLive(value);
} }
▲ Show 20 Lines • Show All 157 Lines • Show Last 20 Lines

mlir/lib/Transforms/Utils/Utils.cpp

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Lines
// Perform the replacement in `op`. // Perform the replacement in `op`.
LogicalResult mlir::replaceAllMemRefUsesWith(Value oldMemRef, Value newMemRef, LogicalResult mlir::replaceAllMemRefUsesWith(Value oldMemRef, Value newMemRef,
Operation *op, Operation *op,
ArrayRef<Value> extraIndices, ArrayRef<Value> extraIndices,
AffineMap indexRemap, AffineMap indexRemap,
ArrayRef<Value> extraOperands, ArrayRef<Value> extraOperands,
ArrayRef<Value> symbolOperands) { ArrayRef<Value> symbolOperands) {
unsigned newMemRefRank = newMemRef->getType().cast<MemRefType>().getRank(); unsigned newMemRefRank = newMemRef.getType().cast<MemRefType>().getRank();
(void)newMemRefRank; // unused in opt mode (void)newMemRefRank; // unused in opt mode
unsigned oldMemRefRank = oldMemRef->getType().cast<MemRefType>().getRank(); unsigned oldMemRefRank = oldMemRef.getType().cast<MemRefType>().getRank();
(void)oldMemRefRank; // unused in opt mode (void)oldMemRefRank; // unused in opt mode
if (indexRemap) { if (indexRemap) {
assert(indexRemap.getNumSymbols() == symbolOperands.size() && assert(indexRemap.getNumSymbols() == symbolOperands.size() &&
"symbolic operand count mismatch"); "symbolic operand count mismatch");
assert(indexRemap.getNumInputs() == assert(indexRemap.getNumInputs() ==
extraOperands.size() + oldMemRefRank + symbolOperands.size()); extraOperands.size() + oldMemRefRank + symbolOperands.size());
assert(indexRemap.getNumResults() + extraIndices.size() == newMemRefRank); assert(indexRemap.getNumResults() + extraIndices.size() == newMemRefRank);
} else { } else {
assert(oldMemRefRank + extraIndices.size() == newMemRefRank); assert(oldMemRefRank + extraIndices.size() == newMemRefRank);
} }
// Assert same elemental type. // Assert same elemental type.
assert(oldMemRef->getType().cast<MemRefType>().getElementType() == assert(oldMemRef.getType().cast<MemRefType>().getElementType() ==
newMemRef->getType().cast<MemRefType>().getElementType()); newMemRef.getType().cast<MemRefType>().getElementType());
if (!isMemRefDereferencingOp(*op)) if (!isMemRefDereferencingOp(*op))
// Failure: memref used in a non-dereferencing context (potentially // Failure: memref used in a non-dereferencing context (potentially
// escapes); no replacement in these cases. // escapes); no replacement in these cases.
return failure(); return failure();
SmallVector<unsigned, 2> usePositions; SmallVector<unsigned, 2> usePositions;
for (const auto &opEntry : llvm::enumerate(op->getOperands())) { for (const auto &opEntry : llvm::enumerate(op->getOperands())) {
▲ Show 20 Lines • Show All 67 Lines • ▼ Show 20 Lines if (indexRemap &&
remapOutputs.append(remapOperands.begin(), remapOperands.end()); remapOutputs.append(remapOperands.begin(), remapOperands.end());
} }
SmallVector<Value, 4> newMapOperands; SmallVector<Value, 4> newMapOperands;
newMapOperands.reserve(newMemRefRank); newMapOperands.reserve(newMemRefRank);
// Prepend 'extraIndices' in 'newMapOperands'. // Prepend 'extraIndices' in 'newMapOperands'.
for (auto extraIndex : extraIndices) { for (auto extraIndex : extraIndices) {
assert(extraIndex->getDefiningOp()->getNumResults() == 1 && assert(extraIndex.getDefiningOp()->getNumResults() == 1 &&
"single result op's expected to generate these indices"); "single result op's expected to generate these indices");
assert((isValidDim(extraIndex) || isValidSymbol(extraIndex)) && assert((isValidDim(extraIndex) || isValidSymbol(extraIndex)) &&
"invalid memory op index"); "invalid memory op index");
newMapOperands.push_back(extraIndex); newMapOperands.push_back(extraIndex);
} }
// Append 'remapOutputs' to 'newMapOperands'. // Append 'remapOutputs' to 'newMapOperands'.
newMapOperands.append(remapOutputs.begin(), remapOutputs.end()); newMapOperands.append(remapOutputs.begin(), remapOutputs.end());
// Create new fully composed AffineMap for new op to be created. // Create new fully composed AffineMap for new op to be created.
assert(newMapOperands.size() == newMemRefRank); assert(newMapOperands.size() == newMemRefRank);
auto newMap = builder.getMultiDimIdentityMap(newMemRefRank); auto newMap = builder.getMultiDimIdentityMap(newMemRefRank);
// TODO(b/136262594) Avoid creating/deleting temporary AffineApplyOps here. // TODO(b/136262594) Avoid creating/deleting temporary AffineApplyOps here.
fullyComposeAffineMapAndOperands(&newMap, &newMapOperands); fullyComposeAffineMapAndOperands(&newMap, &newMapOperands);
newMap = simplifyAffineMap(newMap); newMap = simplifyAffineMap(newMap);
canonicalizeMapAndOperands(&newMap, &newMapOperands); canonicalizeMapAndOperands(&newMap, &newMapOperands);
// Remove any affine.apply's that became dead as a result of composition. // Remove any affine.apply's that became dead as a result of composition.
for (auto value : affineApplyOps) for (auto value : affineApplyOps)
if (value->use_empty()) if (value.use_empty())
value->getDefiningOp()->erase(); value.getDefiningOp()->erase();
// Construct the new operation using this memref. // Construct the new operation using this memref.
OperationState state(op->getLoc(), op->getName()); OperationState state(op->getLoc(), op->getName());
state.setOperandListToResizable(op->hasResizableOperandsList()); state.setOperandListToResizable(op->hasResizableOperandsList());
state.operands.reserve(op->getNumOperands() + extraIndices.size()); state.operands.reserve(op->getNumOperands() + extraIndices.size());
// Insert the non-memref operands. // Insert the non-memref operands.
state.operands.append(op->operand_begin(), state.operands.append(op->operand_begin(),
op->operand_begin() + memRefOperandPos); op->operand_begin() + memRefOperandPos);
// Insert the new memref value. // Insert the new memref value.
state.operands.push_back(newMemRef); state.operands.push_back(newMemRef);
// Insert the new memref map operands. // Insert the new memref map operands.
state.operands.append(newMapOperands.begin(), newMapOperands.end()); state.operands.append(newMapOperands.begin(), newMapOperands.end());
// Insert the remaining operands unmodified. // Insert the remaining operands unmodified.
state.operands.append(op->operand_begin() + memRefOperandPos + 1 + state.operands.append(op->operand_begin() + memRefOperandPos + 1 +
oldMapNumInputs, oldMapNumInputs,
op->operand_end()); op->operand_end());
// Result types don't change. Both memref's are of the same elemental type. // Result types don't change. Both memref's are of the same elemental type.
state.types.reserve(op->getNumResults()); state.types.reserve(op->getNumResults());
for (auto result : op->getResults()) for (auto result : op->getResults())
state.types.push_back(result->getType()); state.types.push_back(result.getType());
// Add attribute for 'newMap', other Attributes do not change. // Add attribute for 'newMap', other Attributes do not change.
auto newMapAttr = AffineMapAttr::get(newMap); auto newMapAttr = AffineMapAttr::get(newMap);
for (auto namedAttr : op->getAttrs()) { for (auto namedAttr : op->getAttrs()) {
if (namedAttr.first == oldMapAttrPair.first) { if (namedAttr.first == oldMapAttrPair.first) {
state.attributes.push_back({namedAttr.first, newMapAttr}); state.attributes.push_back({namedAttr.first, newMapAttr});
} else { } else {
state.attributes.push_back(namedAttr); state.attributes.push_back(namedAttr);
Show All 10 Lines
LogicalResult mlir::replaceAllMemRefUsesWith(Value oldMemRef, Value newMemRef, LogicalResult mlir::replaceAllMemRefUsesWith(Value oldMemRef, Value newMemRef,
ArrayRef<Value> extraIndices, ArrayRef<Value> extraIndices,
AffineMap indexRemap, AffineMap indexRemap,
ArrayRef<Value> extraOperands, ArrayRef<Value> extraOperands,
ArrayRef<Value> symbolOperands, ArrayRef<Value> symbolOperands,
Operation *domInstFilter, Operation *domInstFilter,
Operation *postDomInstFilter) { Operation *postDomInstFilter) {
unsigned newMemRefRank = newMemRef->getType().cast<MemRefType>().getRank(); unsigned newMemRefRank = newMemRef.getType().cast<MemRefType>().getRank();
(void)newMemRefRank; // unused in opt mode (void)newMemRefRank; // unused in opt mode
unsigned oldMemRefRank = oldMemRef->getType().cast<MemRefType>().getRank(); unsigned oldMemRefRank = oldMemRef.getType().cast<MemRefType>().getRank();
(void)oldMemRefRank; (void)oldMemRefRank;
if (indexRemap) { if (indexRemap) {
assert(indexRemap.getNumSymbols() == symbolOperands.size() && assert(indexRemap.getNumSymbols() == symbolOperands.size() &&
"symbol operand count mismatch"); "symbol operand count mismatch");
assert(indexRemap.getNumInputs() == assert(indexRemap.getNumInputs() ==
extraOperands.size() + oldMemRefRank + symbolOperands.size()); extraOperands.size() + oldMemRefRank + symbolOperands.size());
assert(indexRemap.getNumResults() + extraIndices.size() == newMemRefRank); assert(indexRemap.getNumResults() + extraIndices.size() == newMemRefRank);
} else { } else {
assert(oldMemRefRank + extraIndices.size() == newMemRefRank); assert(oldMemRefRank + extraIndices.size() == newMemRefRank);
} }
// Assert same elemental type. // Assert same elemental type.
assert(oldMemRef->getType().cast<MemRefType>().getElementType() == assert(oldMemRef.getType().cast<MemRefType>().getElementType() ==
newMemRef->getType().cast<MemRefType>().getElementType()); newMemRef.getType().cast<MemRefType>().getElementType());
std::unique_ptr<DominanceInfo> domInfo; std::unique_ptr<DominanceInfo> domInfo;
std::unique_ptr<PostDominanceInfo> postDomInfo; std::unique_ptr<PostDominanceInfo> postDomInfo;
if (domInstFilter) if (domInstFilter)
domInfo = std::make_unique<DominanceInfo>( domInfo = std::make_unique<DominanceInfo>(
domInstFilter->getParentOfType<FuncOp>()); domInstFilter->getParentOfType<FuncOp>());
if (postDomInstFilter) if (postDomInstFilter)
postDomInfo = std::make_unique<PostDominanceInfo>( postDomInfo = std::make_unique<PostDominanceInfo>(
postDomInstFilter->getParentOfType<FuncOp>()); postDomInstFilter->getParentOfType<FuncOp>());
// Walk all uses of old memref; collect ops to perform replacement. We use a // Walk all uses of old memref; collect ops to perform replacement. We use a
// DenseSet since an operation could potentially have multiple uses of a // DenseSet since an operation could potentially have multiple uses of a
// memref (although rare), and the replacement later is going to erase ops. // memref (although rare), and the replacement later is going to erase ops.
DenseSet<Operation *> opsToReplace; DenseSet<Operation *> opsToReplace;
for (auto *op : oldMemRef->getUsers()) { for (auto *op : oldMemRef.getUsers()) {
// Skip this use if it's not dominated by domInstFilter. // Skip this use if it's not dominated by domInstFilter.
if (domInstFilter && !domInfo->dominates(domInstFilter, op)) if (domInstFilter && !domInfo->dominates(domInstFilter, op))
continue; continue;
// Skip this use if it's not post-dominated by postDomInstFilter. // Skip this use if it's not post-dominated by postDomInstFilter.
if (postDomInstFilter && !postDomInfo->postDominates(postDomInstFilter, op)) if (postDomInstFilter && !postDomInfo->postDominates(postDomInstFilter, op))
continue; continue;
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Lines
/// uses besides this opInst; otherwise returns the list of affine.apply /// uses besides this opInst; otherwise returns the list of affine.apply
/// operations created in output argument `sliceOps`. /// operations created in output argument `sliceOps`.
void mlir::createAffineComputationSlice( void mlir::createAffineComputationSlice(
Operation *opInst, SmallVectorImpl<AffineApplyOp> *sliceOps) { Operation *opInst, SmallVectorImpl<AffineApplyOp> *sliceOps) {
// Collect all operands that are results of affine apply ops. // Collect all operands that are results of affine apply ops.
SmallVector<Value, 4> subOperands; SmallVector<Value, 4> subOperands;
subOperands.reserve(opInst->getNumOperands()); subOperands.reserve(opInst->getNumOperands());
for (auto operand : opInst->getOperands()) for (auto operand : opInst->getOperands())
if (isa_and_nonnull<AffineApplyOp>(operand->getDefiningOp())) if (isa_and_nonnull<AffineApplyOp>(operand.getDefiningOp()))
subOperands.push_back(operand); subOperands.push_back(operand);
// Gather sequence of AffineApplyOps reachable from 'subOperands'. // Gather sequence of AffineApplyOps reachable from 'subOperands'.
SmallVector<Operation *, 4> affineApplyOps; SmallVector<Operation *, 4> affineApplyOps;
getReachableAffineApplyOps(subOperands, affineApplyOps); getReachableAffineApplyOps(subOperands, affineApplyOps);
// Skip transforming if there are no affine maps to compose. // Skip transforming if there are no affine maps to compose.
if (affineApplyOps.empty()) if (affineApplyOps.empty())
return; return;
// Check if all uses of the affine apply op's lie only in this op op, in // Check if all uses of the affine apply op's lie only in this op op, in
// which case there would be nothing to do. // which case there would be nothing to do.
bool localized = true; bool localized = true;
for (auto *op : affineApplyOps) { for (auto *op : affineApplyOps) {
for (auto result : op->getResults()) { for (auto result : op->getResults()) {
for (auto *user : result->getUsers()) { for (auto *user : result.getUsers()) {
if (user != opInst) { if (user != opInst) {
localized = false; localized = false;
break; break;
} }
} }
} }
} }
if (localized) if (localized)
▲ Show 20 Lines • Show All 104 Lines • ▼ Show 20 Lines if (failed(replaceAllMemRefUsesWith(oldMemRef, /*newMemRef=*/newAlloc,
/*extraOperands=*/{}, /*extraOperands=*/{},
/*symbolOperands=*/symbolOperands))) { /*symbolOperands=*/symbolOperands))) {
// If it failed (due to escapes for example), bail out. // If it failed (due to escapes for example), bail out.
newAlloc.erase(); newAlloc.erase();
return failure(); return failure();
} }
// Replace any uses of the original alloc op and erase it. All remaining uses // Replace any uses of the original alloc op and erase it. All remaining uses
// have to be dealloc's; RAMUW above would've failed otherwise. // have to be dealloc's; RAMUW above would've failed otherwise.
assert(std::all_of(oldMemRef->user_begin(), oldMemRef->user_end(), assert(llvm::all_of(oldMemRef.getUsers(),
[](Operation *op) { return isa<DeallocOp>(op); })); [](Operation *op) { return isa<DeallocOp>(op); }));
oldMemRef->replaceAllUsesWith(newAlloc); oldMemRef.replaceAllUsesWith(newAlloc);
allocOp.erase(); allocOp.erase();
return success(); return success();
} }

mlir/lib/Transforms/Vectorize.cpp

Show First 20 Lines • Show All 791 Lines • ▼ Show 20 Lines
/// Additionally, they may have ssa-values operands which come from outside the /// Additionally, they may have ssa-values operands which come from outside the
/// scope of the current pattern. /// scope of the current pattern.
/// Such special cases force us to delay the vectorization of the stores until /// Such special cases force us to delay the vectorization of the stores until
/// the last step. Here we merely register the store operation. /// the last step. Here we merely register the store operation.
template <typename LoadOrStoreOpPointer> template <typename LoadOrStoreOpPointer>
static LogicalResult vectorizeRootOrTerminal(Value iv, static LogicalResult vectorizeRootOrTerminal(Value iv,
LoadOrStoreOpPointer memoryOp, LoadOrStoreOpPointer memoryOp,
VectorizationState *state) { VectorizationState *state) {
auto memRefType = memoryOp.getMemRef()->getType().template cast<MemRefType>(); auto memRefType = memoryOp.getMemRef().getType().template cast<MemRefType>();
auto elementType = memRefType.getElementType(); auto elementType = memRefType.getElementType();
// TODO(ntv): ponder whether we want to further vectorize a vector value. // TODO(ntv): ponder whether we want to further vectorize a vector value.
assert(VectorType::isValidElementType(elementType) && assert(VectorType::isValidElementType(elementType) &&
"Not a valid vector element type"); "Not a valid vector element type");
auto vectorType = VectorType::get(state->strategy->vectorSizes, elementType); auto vectorType = VectorType::get(state->strategy->vectorSizes, elementType);
// Materialize a MemRef with 1 vector. // Materialize a MemRef with 1 vector.
▲ Show 20 Lines • Show All 167 Lines • ▼ Show 20 Lines
/// One such example is top level function constants that need to be splatted. /// One such example is top level function constants that need to be splatted.
/// ///
/// Returns an operand that has been vectorized to match `state`'s strategy if /// Returns an operand that has been vectorized to match `state`'s strategy if
/// vectorization is possible with the above logic. Returns nullptr otherwise. /// vectorization is possible with the above logic. Returns nullptr otherwise.
/// ///
/// TODO(ntv): handle more complex cases. /// TODO(ntv): handle more complex cases.
static Value vectorizeOperand(Value operand, Operation *op, static Value vectorizeOperand(Value operand, Operation *op,
VectorizationState *state) { VectorizationState *state) {
LLVM_DEBUG(dbgs() << "\n[early-vect]vectorize operand: "); LLVM_DEBUG(dbgs() << "\n[early-vect]vectorize operand: " << operand);
LLVM_DEBUG(operand->print(dbgs()));
// 1. If this value has already been vectorized this round, we are done. // 1. If this value has already been vectorized this round, we are done.
if (state->vectorizedSet.count(operand->getDefiningOp()) > 0) { if (state->vectorizedSet.count(operand.getDefiningOp()) > 0) {
LLVM_DEBUG(dbgs() << " -> already vector operand"); LLVM_DEBUG(dbgs() << " -> already vector operand");
return operand; return operand;
} }
// 1.b. Delayed on-demand replacement of a use. // 1.b. Delayed on-demand replacement of a use.
// Note that we cannot just call replaceAllUsesWith because it may result // Note that we cannot just call replaceAllUsesWith because it may result
// in ops with mixed types, for ops whose operands have not all yet // in ops with mixed types, for ops whose operands have not all yet
// been vectorized. This would be invalid IR. // been vectorized. This would be invalid IR.
auto it = state->replacementMap.find(operand); auto it = state->replacementMap.find(operand);
if (it != state->replacementMap.end()) { if (it != state->replacementMap.end()) {
auto res = it->second; auto res = it->second;
LLVM_DEBUG(dbgs() << "-> delayed replacement by: "); LLVM_DEBUG(dbgs() << "-> delayed replacement by: " << res);
LLVM_DEBUG(res->print(dbgs()));
return res; return res;
} }
// 2. TODO(ntv): broadcast needed. // 2. TODO(ntv): broadcast needed.
if (operand->getType().isa<VectorType>()) { if (operand.getType().isa<VectorType>()) {
LLVM_DEBUG(dbgs() << "-> non-vectorizable"); LLVM_DEBUG(dbgs() << "-> non-vectorizable");
return nullptr; return nullptr;
} }
// 3. vectorize constant. // 3. vectorize constant.
if (auto constant = dyn_cast<ConstantOp>(operand->getDefiningOp())) { if (auto constant = dyn_cast<ConstantOp>(operand.getDefiningOp())) {
return vectorizeConstant( return vectorizeConstant(
op, constant, op, constant,
VectorType::get(state->strategy->vectorSizes, operand->getType())); VectorType::get(state->strategy->vectorSizes, operand.getType()));
} }
// 4. currently non-vectorizable. // 4. currently non-vectorizable.
LLVM_DEBUG(dbgs() << "-> non-vectorizable"); LLVM_DEBUG(dbgs() << "-> non-vectorizable: " << operand);
LLVM_DEBUG(operand->print(dbgs()));
return nullptr; return nullptr;
} }
/// Encodes Operation-specific behavior for vectorization. In general we assume /// Encodes Operation-specific behavior for vectorization. In general we assume
/// that all operands of an op must be vectorized but this is not always true. /// that all operands of an op must be vectorized but this is not always true.
/// In the future, it would be nice to have a trait that describes how a /// In the future, it would be nice to have a trait that describes how a
/// particular operation vectorizes. For now we implement the case distinction /// particular operation vectorizes. For now we implement the case distinction
/// here. /// here.
▲ Show 20 Lines • Show All 44 Lines • ▼ Show 20 Lines if (auto store = dyn_cast<AffineStoreOp>(opInst)) {
return res; return res;
} }
if (opInst->getNumRegions() != 0) if (opInst->getNumRegions() != 0)
return nullptr; return nullptr;
SmallVector<Type, 8> vectorTypes; SmallVector<Type, 8> vectorTypes;
for (auto v : opInst->getResults()) { for (auto v : opInst->getResults()) {
vectorTypes.push_back( vectorTypes.push_back(
VectorType::get(state->strategy->vectorSizes, v->getType())); VectorType::get(state->strategy->vectorSizes, v.getType()));
} }
SmallVector<Value, 8> vectorOperands; SmallVector<Value, 8> vectorOperands;
for (auto v : opInst->getOperands()) { for (auto v : opInst->getOperands()) {
vectorOperands.push_back(vectorizeOperand(v, opInst, state)); vectorOperands.push_back(vectorizeOperand(v, opInst, state));
} }
// Check whether a single operand is null. If so, vectorization failed. // Check whether a single operand is null. If so, vectorization failed.
bool success = llvm::all_of(vectorOperands, [](Value op) { return op; }); bool success = llvm::all_of(vectorOperands, [](Value op) { return op; });
if (!success) { if (!success) {
▲ Show 20 Lines • Show All 89 Lines • ▼ Show 20 Linesstatic LogicalResult vectorizeRootMatch(NestedMatch m,
/// Sets up error handling for this root loop. This is how the root match /// Sets up error handling for this root loop. This is how the root match
/// maintains a clone for handling failure and restores the proper state via /// maintains a clone for handling failure and restores the proper state via
/// RAII. /// RAII.
auto *loopInst = loop.getOperation(); auto *loopInst = loop.getOperation();
OpBuilder builder(loopInst); OpBuilder builder(loopInst);
auto clonedLoop = cast<AffineForOp>(builder.clone(*loopInst)); auto clonedLoop = cast<AffineForOp>(builder.clone(*loopInst));
struct Guard { struct Guard {
LogicalResult failure() { LogicalResult failure() {
loop.getInductionVar()->replaceAllUsesWith(clonedLoop.getInductionVar()); loop.getInductionVar().replaceAllUsesWith(clonedLoop.getInductionVar());
loop.erase(); loop.erase();
return mlir::failure(); return mlir::failure();
} }
LogicalResult success() { LogicalResult success() {
clonedLoop.erase(); clonedLoop.erase();
return mlir::success(); return mlir::success();
} }
AffineForOp loop; AffineForOp loop;
▲ Show 20 Lines • Show All 102 Lines • Show Last 20 Lines

mlir/test/lib/TestDialect/TestDialect.cpp

Show First 20 Lines • Show All 94 Lines • ▼ Show 20 Lines void handleTerminator(Operation *op,
// Only handle "test.return" here. // Only handle "test.return" here.
auto returnOp = dyn_cast<TestReturnOp>(op); auto returnOp = dyn_cast<TestReturnOp>(op);
if (!returnOp) if (!returnOp)
return; return;
// Replace the values directly with the return operands. // Replace the values directly with the return operands.
assert(returnOp.getNumOperands() == valuesToRepl.size()); assert(returnOp.getNumOperands() == valuesToRepl.size());
for (const auto &it : llvm::enumerate(returnOp.getOperands())) for (const auto &it : llvm::enumerate(returnOp.getOperands()))
valuesToRepl[it.index()]->replaceAllUsesWith(it.value()); valuesToRepl[it.index()].replaceAllUsesWith(it.value());
} }
/// Attempt to materialize a conversion for a type mismatch between a call /// Attempt to materialize a conversion for a type mismatch between a call
/// from this dialect, and a callable region. This method should generate an /// from this dialect, and a callable region. This method should generate an
/// operation that takes 'input' as the only operand, and produces a single /// operation that takes 'input' as the only operand, and produces a single
/// result of 'resultType'. If a conversion can not be generated, nullptr /// result of 'resultType'. If a conversion can not be generated, nullptr
/// should be returned. /// should be returned.
Operation *materializeCallConversion(OpBuilder &builder, Value input, Operation *materializeCallConversion(OpBuilder &builder, Value input,
Type resultType, Type resultType,
Location conversionLoc) const final { Location conversionLoc) const final {
// Only allow conversion for i16/i32 types. // Only allow conversion for i16/i32 types.
if (!(resultType.isInteger(16) || resultType.isInteger(32)) || if (!(resultType.isInteger(16) || resultType.isInteger(32)) ||
!(input->getType().isInteger(16) || input->getType().isInteger(32))) !(input.getType().isInteger(16) || input.getType().isInteger(32)))
return nullptr; return nullptr;
return builder.create<TestCastOp>(conversionLoc, resultType, input); return builder.create<TestCastOp>(conversionLoc, resultType, input);
} }
}; };
} // end anonymous namespace } // end anonymous namespace
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TestDialect // TestDialect
▲ Show 20 Lines • Show All 168 Lines • ▼ Show 20 LinesLogicalResult TestOpWithVariadicResultsAndFolder::fold(
} }
return success(); return success();
} }
LogicalResult mlir::OpWithInferTypeInterfaceOp::inferReturnTypes( LogicalResult mlir::OpWithInferTypeInterfaceOp::inferReturnTypes(
llvm::Optional<Location> location, ValueRange operands, llvm::Optional<Location> location, ValueRange operands,
ArrayRef<NamedAttribute> attributes, RegionRange regions, ArrayRef<NamedAttribute> attributes, RegionRange regions,
SmallVectorImpl<Type> &inferedReturnTypes) { SmallVectorImpl<Type> &inferedReturnTypes) {
if (operands[0]->getType() != operands[1]->getType()) { if (operands[0].getType() != operands[1].getType()) {
return emitOptionalError(location, "operand type mismatch ", return emitOptionalError(location, "operand type mismatch ",
operands[0]->getType(), " vs ", operands[0].getType(), " vs ",
operands[1]->getType()); operands[1].getType());
} }
inferedReturnTypes.assign({operands[0]->getType()}); inferedReturnTypes.assign({operands[0].getType()});
return success(); return success();
} }
// Static initialization for Test dialect registration. // Static initialization for Test dialect registration.
static mlir::DialectRegistration<mlir::TestDialect> testDialect; static mlir::DialectRegistration<mlir::TestDialect> testDialect;
#include "TestOpEnums.cpp.inc" #include "TestOpEnums.cpp.inc"
#define GET_OP_CLASSES #define GET_OP_CLASSES
#include "TestOps.cpp.inc" #include "TestOps.cpp.inc"

mlir/test/lib/TestDialect/TestOps.td

Show First 20 Lines • Show All 645 Lines • ▼ Show 20 Linesdef OpSymbolBindingC : TEST_Op<"symbol_binding_c", []> {
let arguments = (ins I32:$operand); let arguments = (ins I32:$operand);
let results = (outs I32); let results = (outs I32);
let builders = OpSymbolBindingB.builders; let builders = OpSymbolBindingB.builders;
} }
def OpSymbolBindingD : TEST_Op<"symbol_binding_d", []> { def OpSymbolBindingD : TEST_Op<"symbol_binding_d", []> {
let arguments = (ins I32:$input1, I32:$input2, I64Attr:$attr); let arguments = (ins I32:$input1, I32:$input2, I64Attr:$attr);
let results = (outs I32); let results = (outs I32);
} }
def HasOneUse: Constraint<CPred<"$0->hasOneUse()">, "has one use">; def HasOneUse: Constraint<CPred<"$0.hasOneUse()">, "has one use">;
def : Pattern< def : Pattern<
// Bind to source pattern op operand/attribute/result // Bind to source pattern op operand/attribute/result
(OpSymbolBindingA:$res_a $operand, $attr), [ (OpSymbolBindingA:$res_a $operand, $attr), [
// Bind to auxiliary op result // Bind to auxiliary op result
(OpSymbolBindingC:$res_c (OpSymbolBindingB:$res_b $operand)), (OpSymbolBindingC:$res_c (OpSymbolBindingB:$res_b $operand)),
// Use bound symbols in resultant ops // Use bound symbols in resultant ops
(OpSymbolBindingD $res_b, $res_c, $attr)], (OpSymbolBindingD $res_b, $res_c, $attr)],
▲ Show 20 Lines • Show All 385 Lines • Show Last 20 Lines

mlir/test/lib/TestDialect/TestPatterns.cpp

Show All 17 Lines
} }
static void createOpI(PatternRewriter &rewriter, Value input) { static void createOpI(PatternRewriter &rewriter, Value input) {
rewriter.create<OpI>(rewriter.getUnknownLoc(), input); rewriter.create<OpI>(rewriter.getUnknownLoc(), input);
} }
void handleNoResultOp(PatternRewriter &rewriter, OpSymbolBindingNoResult op) { void handleNoResultOp(PatternRewriter &rewriter, OpSymbolBindingNoResult op) {
// Turn the no result op to a one-result op. // Turn the no result op to a one-result op.
rewriter.create<OpSymbolBindingB>(op.getLoc(), op.operand()->getType(), rewriter.create<OpSymbolBindingB>(op.getLoc(), op.operand().getType(),
op.operand()); op.operand());
} }
namespace { namespace {
#include "TestPatterns.inc" #include "TestPatterns.inc"
} // end anonymous namespace } // end anonymous namespace
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 142 Lines • ▼ Show 20 Lines matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
Region &region = op->getRegion(0); Region &region = op->getRegion(0);
Block *entry = &region.front(); Block *entry = &region.front();
// Convert the original entry arguments. // Convert the original entry arguments.
TypeConverter::SignatureConversion result(entry->getNumArguments()); TypeConverter::SignatureConversion result(entry->getNumArguments());
for (unsigned i = 0, e = entry->getNumArguments(); i != e; ++i) for (unsigned i = 0, e = entry->getNumArguments(); i != e; ++i)
if (failed(converter.convertSignatureArg( if (failed(converter.convertSignatureArg(
i, entry->getArgument(i)->getType(), result))) i, entry->getArgument(i).getType(), result)))
return matchFailure(); return matchFailure();
// Convert the region signature and just drop the operation. // Convert the region signature and just drop the operation.
rewriter.applySignatureConversion(&region, result); rewriter.applySignatureConversion(&region, result);
rewriter.eraseOp(op); rewriter.eraseOp(op);
return matchSuccess(); return matchSuccess();
} }
Show All 15 Lines
/// This pattern handles the case of a split return value. /// This pattern handles the case of a split return value.
struct TestSplitReturnType : public ConversionPattern { struct TestSplitReturnType : public ConversionPattern {
TestSplitReturnType(MLIRContext *ctx) TestSplitReturnType(MLIRContext *ctx)
: ConversionPattern("test.return", 1, ctx) {} : ConversionPattern("test.return", 1, ctx) {}
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const final { ConversionPatternRewriter &rewriter) const final {
// Check for a return of F32. // Check for a return of F32.
if (op->getNumOperands() != 1 || !op->getOperand(0)->getType().isF32()) if (op->getNumOperands() != 1 || !op->getOperand(0).getType().isF32())
return matchFailure(); return matchFailure();
// Check if the first operation is a cast operation, if it is we use the // Check if the first operation is a cast operation, if it is we use the
// results directly. // results directly.
auto *defOp = operands[0]->getDefiningOp(); auto *defOp = operands[0].getDefiningOp();
if (auto packerOp = llvm::dyn_cast_or_null<TestCastOp>(defOp)) { if (auto packerOp = llvm::dyn_cast_or_null<TestCastOp>(defOp)) {
rewriter.replaceOpWithNewOp<TestReturnOp>(op, packerOp.getOperands()); rewriter.replaceOpWithNewOp<TestReturnOp>(op, packerOp.getOperands());
return matchSuccess(); return matchSuccess();
} }
// Otherwise, fail to match. // Otherwise, fail to match.
return matchFailure(); return matchFailure();
} }
▲ Show 20 Lines • Show All 41 Lines • ▼ Show 20 Lines
}; };
struct TestUpdateConsumerType : public ConversionPattern { struct TestUpdateConsumerType : public ConversionPattern {
TestUpdateConsumerType(MLIRContext *ctx) TestUpdateConsumerType(MLIRContext *ctx)
: ConversionPattern("test.type_consumer", 1, ctx) {} : ConversionPattern("test.type_consumer", 1, ctx) {}
PatternMatchResult PatternMatchResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands, matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const final { ConversionPatternRewriter &rewriter) const final {
// Verify that the incoming operand has been successfully remapped to F64. // Verify that the incoming operand has been successfully remapped to F64.
if (!operands[0]->getType().isF64()) if (!operands[0].getType().isF64())
return matchFailure(); return matchFailure();
rewriter.replaceOpWithNewOp<TestTypeConsumerOp>(op, operands[0]); rewriter.replaceOpWithNewOp<TestTypeConsumerOp>(op, operands[0]);
return matchSuccess(); return matchSuccess();
} }
}; };
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Non-Root Replacement Rewrite Testing // Non-Root Replacement Rewrite Testing
▲ Show 20 Lines • Show All 86 Lines • ▼ Show 20 Lines void runOnModule() override {
}); });
target.addDynamicallyLegalOp<FuncOp>( target.addDynamicallyLegalOp<FuncOp>(
[&](FuncOp op) { return converter.isSignatureLegal(op.getType()); }); [&](FuncOp op) { return converter.isSignatureLegal(op.getType()); });
// Expect the type_producer/type_consumer operations to only operate on f64. // Expect the type_producer/type_consumer operations to only operate on f64.
target.addDynamicallyLegalOp<TestTypeProducerOp>( target.addDynamicallyLegalOp<TestTypeProducerOp>(
[](TestTypeProducerOp op) { return op.getType().isF64(); }); [](TestTypeProducerOp op) { return op.getType().isF64(); });
target.addDynamicallyLegalOp<TestTypeConsumerOp>([](TestTypeConsumerOp op) { target.addDynamicallyLegalOp<TestTypeConsumerOp>([](TestTypeConsumerOp op) {
return op.getOperand()->getType().isF64(); return op.getOperand().getType().isF64();
}); });
// Check support for marking certain operations as recursively legal. // Check support for marking certain operations as recursively legal.
target.markOpRecursivelyLegal<FuncOp, ModuleOp>([](Operation *op) { target.markOpRecursivelyLegal<FuncOp, ModuleOp>([](Operation *op) {
return static_cast<bool>( return static_cast<bool>(
op->getAttrOfType<UnitAttr>("test.recursively_legal")); op->getAttrOfType<UnitAttr>("test.recursively_legal"));
}); });
▲ Show 20 Lines • Show All 113 Lines • Show Last 20 Lines

mlir/test/lib/Transforms/TestInlining.cpp

Show All 31 Lines void runOnFunction() override {
function.walk([&](CallIndirectOp caller) { callers.push_back(caller); }); function.walk([&](CallIndirectOp caller) { callers.push_back(caller); });
// Build the inliner interface. // Build the inliner interface.
InlinerInterface interface(&getContext()); InlinerInterface interface(&getContext());
// Try to inline each of the call operations. // Try to inline each of the call operations.
for (auto caller : callers) { for (auto caller : callers) {
auto callee = dyn_cast_or_null<FunctionalRegionOp>( auto callee = dyn_cast_or_null<FunctionalRegionOp>(
caller.getCallee()->getDefiningOp()); caller.getCallee().getDefiningOp());
if (!callee) if (!callee)
continue; continue;
// Inline the functional region operation, but only clone the internal // Inline the functional region operation, but only clone the internal
// region if there is more than one use. // region if there is more than one use.
if (failed(inlineRegion( if (failed(inlineRegion(
interface, &callee.body(), caller, interface, &callee.body(), caller,
llvm::to_vector<8>(caller.getArgOperands()), llvm::to_vector<8>(caller.getArgOperands()),
SmallVector<Value, 8>(caller.getResults()), caller.getLoc(), SmallVector<Value, 8>(caller.getResults()), caller.getLoc(),
/*shouldCloneInlinedRegion=*/!callee.getResult()->hasOneUse()))) /*shouldCloneInlinedRegion=*/!callee.getResult().hasOneUse())))
continue; continue;
// If the inlining was successful then erase the call and callee if // If the inlining was successful then erase the call and callee if
// possible. // possible.
caller.erase(); caller.erase();
if (callee.use_empty()) if (callee.use_empty())
callee.erase(); callee.erase();
} }
} }
}; };
} // end anonymous namespace } // end anonymous namespace
static PassRegistration<Inliner> pass("test-inline", static PassRegistration<Inliner> pass("test-inline",
"Test inlining region calls"); "Test inlining region calls");

mlir/test/lib/Transforms/TestMemRefStrideCalculation.cpp

Show All 19 Linesstruct TestMemRefStrideCalculation
void runOnFunction() override; void runOnFunction() override;
}; };
} // end anonymous namespace } // end anonymous namespace
// Traverse AllocOp and compute strides of each MemRefType independently. // Traverse AllocOp and compute strides of each MemRefType independently.
void TestMemRefStrideCalculation::runOnFunction() { void TestMemRefStrideCalculation::runOnFunction() {
llvm::outs() << "Testing: " << getFunction().getName() << "\n"; llvm::outs() << "Testing: " << getFunction().getName() << "\n";
getFunction().walk([&](AllocOp allocOp) { getFunction().walk([&](AllocOp allocOp) {
auto memrefType = allocOp.getResult()->getType().cast<MemRefType>(); auto memrefType = allocOp.getResult().getType().cast<MemRefType>();
int64_t offset; int64_t offset;
SmallVector<int64_t, 4> strides; SmallVector<int64_t, 4> strides;
if (failed(getStridesAndOffset(memrefType, strides, offset))) { if (failed(getStridesAndOffset(memrefType, strides, offset))) {
llvm::outs() << "MemRefType " << memrefType << " cannot be converted to " llvm::outs() << "MemRefType " << memrefType << " cannot be converted to "
<< "strided form\n"; << "strided form\n";
return; return;
} }
llvm::outs() << "MemRefType offset: "; llvm::outs() << "MemRefType offset: ";
Show All 18 Lines

mlir/test/lib/Transforms/TestVectorizationUtils.cpp

Show First 20 Lines • Show All 110 Lines • ▼ Show 20 Linesvoid VectorizerTestPass::testVectorShapeRatio(llvm::raw_ostream &outs) {
SmallVector<NestedMatch, 8> matches; SmallVector<NestedMatch, 8> matches;
pat.match(f, &matches); pat.match(f, &matches);
for (auto m : matches) { for (auto m : matches) {
auto *opInst = m.getMatchedOperation(); auto *opInst = m.getMatchedOperation();
// This is a unit test that only checks and prints shape ratio. // This is a unit test that only checks and prints shape ratio.
// As a consequence we write only Ops with a single return type for the // As a consequence we write only Ops with a single return type for the
// purpose of this test. If we need to test more intricate behavior in the // purpose of this test. If we need to test more intricate behavior in the
// future we can always extend. // future we can always extend.
auto superVectorType = opInst->getResult(0)->getType().cast<VectorType>(); auto superVectorType = opInst->getResult(0).getType().cast<VectorType>();
auto ratio = shapeRatio(superVectorType, subVectorType); auto ratio = shapeRatio(superVectorType, subVectorType);
if (!ratio.hasValue()) { if (!ratio.hasValue()) {
opInst->emitRemark("NOT MATCHED"); opInst->emitRemark("NOT MATCHED");
} else { } else {
outs << "\nmatched: " << *opInst << " with shape ratio: "; outs << "\nmatched: " << *opInst << " with shape ratio: ";
interleaveComma(MutableArrayRef<int64_t>(*ratio), outs); interleaveComma(MutableArrayRef<int64_t>(*ratio), outs);
} }
} }
▲ Show 20 Lines • Show All 165 Lines • Show Last 20 Lines

mlir/test/mlir-tblgen/op-result.td

Show All 20 Linesdef OpB : NS_Op<"same_input_output_type_op", [SameOperandsAndResultType]> {
let arguments = (ins I32:$x); let arguments = (ins I32:$x);
let results = (outs I32:$y); let results = (outs I32:$y);
} }
// CHECK-LABEL: OpB definitions // CHECK-LABEL: OpB definitions
// CHECK: void OpB::build(Builder *tblgen_builder, OperationState &tblgen_state, Type y, Value x) // CHECK: void OpB::build(Builder *tblgen_builder, OperationState &tblgen_state, Type y, Value x)
// CHECK: tblgen_state.addTypes(y); // CHECK: tblgen_state.addTypes(y);
// CHECK: void OpB::build(Builder *tblgen_builder, OperationState &tblgen_state, Value x) // CHECK: void OpB::build(Builder *tblgen_builder, OperationState &tblgen_state, Value x)
// CHECK: tblgen_state.addTypes({x->getType()}); // CHECK: tblgen_state.addTypes({x.getType()});
def OpC : NS_Op<"three_normal_result_op", []> { def OpC : NS_Op<"three_normal_result_op", []> {
let results = (outs I32:$x, /*unnamed*/I32, I32:$z); let results = (outs I32:$x, /*unnamed*/I32, I32:$z);
} }
// CHECK-LABEL: OpC definitions // CHECK-LABEL: OpC definitions
// CHECK: void OpC::build(Builder *tblgen_builder, OperationState &tblgen_state, Type x, Type resultType1, Type z) // CHECK: void OpC::build(Builder *tblgen_builder, OperationState &tblgen_state, Type x, Type resultType1, Type z)
// CHECK-NEXT: tblgen_state.addTypes(x) // CHECK-NEXT: tblgen_state.addTypes(x)
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 Lines
// pack to set result type // pack to set result type
// --- // ---
def OpK : NS_Op<"only_input_is_variadic_with_same_value_type_op", [SameOperandsAndResultType]> { def OpK : NS_Op<"only_input_is_variadic_with_same_value_type_op", [SameOperandsAndResultType]> {
let arguments = (ins Variadic<AnyTensor>:$input); let arguments = (ins Variadic<AnyTensor>:$input);
let results = (outs AnyTensor:$result); let results = (outs AnyTensor:$result);
} }
// CHECK-LABEL: OpK::build(Builder *tblgen_builder, OperationState &tblgen_state, ValueRange input) // CHECK-LABEL: OpK::build(Builder *tblgen_builder, OperationState &tblgen_state, ValueRange input)
// CHECK: tblgen_state.addTypes({input.front()->getType()}); // CHECK: tblgen_state.addTypes({input.front().getType()});

mlir/test/mlir-tblgen/predicate.td

Show All 11 Linesdef I32OrF32 : Type<CPred<"$_self.isInteger(32) || $_self.isF32()">,
"32-bit integer or floating-point type">; "32-bit integer or floating-point type">;
def OpA : NS_Op<"op_for_CPred_containing_multiple_same_placeholder", []> { def OpA : NS_Op<"op_for_CPred_containing_multiple_same_placeholder", []> {
let arguments = (ins I32OrF32:$x); let arguments = (ins I32OrF32:$x);
} }
// CHECK-LABEL: OpA::verify // CHECK-LABEL: OpA::verify
// CHECK: for (Value v : getODSOperands(0)) { // CHECK: for (Value v : getODSOperands(0)) {
// CHECK: if (!((v->getType().isInteger(32) || v->getType().isF32()))) // CHECK: if (!((v.getType().isInteger(32) || v.getType().isF32())))
def OpB : NS_Op<"op_for_And_PredOpTrait", [ def OpB : NS_Op<"op_for_And_PredOpTrait", [
PredOpTrait<"both first and second holds", PredOpTrait<"both first and second holds",
And<[CPred<"first">, CPred<"second">]>>]> { And<[CPred<"first">, CPred<"second">]>>]> {
} }
// CHECK-LABEL: OpB::verify // CHECK-LABEL: OpB::verify
// CHECK: if (!(((first)) && ((second)))) // CHECK: if (!(((first)) && ((second))))
▲ Show 20 Lines • Show All 57 Lines • ▼ Show 20 Lines
// CHECK: return emitOpError("failed to verify that operands indexed at 0, 2, 3 should all have the same type"); // CHECK: return emitOpError("failed to verify that operands indexed at 0, 2, 3 should all have the same type");
def OpK : NS_Op<"op_for_AnyTensorOf", []> { def OpK : NS_Op<"op_for_AnyTensorOf", []> {
let arguments = (ins TensorOf<[F32, I32]>:$x); let arguments = (ins TensorOf<[F32, I32]>:$x);
} }
// CHECK-LABEL: OpK::verify // CHECK-LABEL: OpK::verify
// CHECK: for (Value v : getODSOperands(0)) { // CHECK: for (Value v : getODSOperands(0)) {
// CHECK: if (!(((v->getType().isa<TensorType>())) && (((v->getType().cast<ShapedType>().getElementType().isF32())) || ((v->getType().cast<ShapedType>().getElementType().isInteger(32)))))) // CHECK: if (!(((v.getType().isa<TensorType>())) && (((v.getType().cast<ShapedType>().getElementType().isF32())) || ((v.getType().cast<ShapedType>().getElementType().isInteger(32))))))

mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp

Show First 20 Lines • Show All 127 Lines • ▼ Show 20 Lines if (isOperandName(op, name)) {
? formatv("lookupValues(op.{0}())", name) ? formatv("lookupValues(op.{0}())", name)
: formatv("valueMapping.lookup(op.{0}())", name); : formatv("valueMapping.lookup(op.{0}())", name);
bs << result; bs << result;
} else if (isAttributeName(op, name)) { } else if (isAttributeName(op, name)) {
bs << formatv("op.{0}()", name); bs << formatv("op.{0}()", name);
} else if (isResultName(op, name)) { } else if (isResultName(op, name)) {
bs << formatv("valueMapping[op.{0}()]", name); bs << formatv("valueMapping[op.{0}()]", name);
} else if (name == "_resultType") { } else if (name == "_resultType") {
bs << "op.getResult()->getType().cast<LLVM::LLVMType>()." bs << "op.getResult().getType().cast<LLVM::LLVMType>()."
"getUnderlyingType()"; "getUnderlyingType()";
} else if (name == "_hasResult") { } else if (name == "_hasResult") {
bs << "opInst.getNumResults() == 1"; bs << "opInst.getNumResults() == 1";
} else if (name == "_location") { } else if (name == "_location") {
bs << "opInst.getLoc()"; bs << "opInst.getLoc()";
} else if (name == "_numOperands") { } else if (name == "_numOperands") {
bs << "opInst.getNumOperands()"; bs << "opInst.getNumOperands()";
} else if (name == "$") { } else if (name == "$") {
Show All 32 Lines

mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp

Show First 20 Lines • Show All 688 Lines • ▼ Show 20 Linesvoid OpEmitter::genUseOperandAsResultTypeCollectiveParamBuilder() {
// Create the correct number of regions // Create the correct number of regions
if (int numRegions = op.getNumRegions()) { if (int numRegions = op.getNumRegions()) {
for (int i = 0; i < numRegions; ++i) for (int i = 0; i < numRegions; ++i)
m.body() << " (void)" << builderOpState << ".addRegion();\n"; m.body() << " (void)" << builderOpState << ".addRegion();\n";
} }
// Result types // Result types
SmallVector<std::string, 2> resultTypes(numResults, "operands[0]->getType()"); SmallVector<std::string, 2> resultTypes(numResults, "operands[0].getType()");
body << " " << builderOpState << ".addTypes({" body << " " << builderOpState << ".addTypes({"
<< llvm::join(resultTypes, ", ") << "});\n\n"; << llvm::join(resultTypes, ", ") << "});\n\n";
} }
void OpEmitter::genInferedTypeCollectiveParamBuilder() { void OpEmitter::genInferedTypeCollectiveParamBuilder() {
// TODO(jpienaar): Expand to support regions. // TODO(jpienaar): Expand to support regions.
const char *params = const char *params =
"Builder *builder, OperationState &{0}, " "Builder *builder, OperationState &{0}, "
Show All 22 Linesvoid OpEmitter::genUseOperandAsResultTypeSeparateParamBuilder() {
auto numResults = op.getNumResults(); auto numResults = op.getNumResults();
if (numResults == 0) if (numResults == 0)
return; return;
// Push all result types to the operation state // Push all result types to the operation state
const char *index = op.getOperand(0).isVariadic() ? ".front()" : ""; const char *index = op.getOperand(0).isVariadic() ? ".front()" : "";
std::string resultType = std::string resultType =
formatv("{0}{1}->getType()", getArgumentName(op, 0), index).str(); formatv("{0}{1}.getType()", getArgumentName(op, 0), index).str();
m.body() << " " << builderOpState << ".addTypes({" << resultType; m.body() << " " << builderOpState << ".addTypes({" << resultType;
for (int i = 1; i != numResults; ++i) for (int i = 1; i != numResults; ++i)
m.body() << ", " << resultType; m.body() << ", " << resultType;
m.body() << "});\n\n"; m.body() << "});\n\n";
} }
void OpEmitter::genUseAttrAsResultTypeBuilder() { void OpEmitter::genUseAttrAsResultTypeBuilder() {
std::string params = std::string params =
▲ Show 20 Lines • Show All 369 Lines • ▼ Show 20 Lines verifyCtx.addSubst(namedAttr.name,
formatv("this->getAttr(\"{0}\")", namedAttr.name)); formatv("this->getAttr(\"{0}\")", namedAttr.name));
for (int i = 0, e = op.getNumOperands(); i < e; ++i) { for (int i = 0, e = op.getNumOperands(); i < e; ++i) {
auto &value = op.getOperand(i); auto &value = op.getOperand(i);
// Skip from from first variadic operands for now. Else getOperand index // Skip from from first variadic operands for now. Else getOperand index
// used below doesn't match. // used below doesn't match.
if (value.isVariadic()) if (value.isVariadic())
break; break;
if (!value.name.empty()) if (!value.name.empty())
verifyCtx.addSubst( verifyCtx.addSubst(value.name,
value.name, formatv("(*this->getOperation()->getOperand({0}))", i)); formatv("this->getOperation()->getOperand({0})", i));
} }
for (int i = 0, e = op.getNumResults(); i < e; ++i) { for (int i = 0, e = op.getNumResults(); i < e; ++i) {
auto &value = op.getResult(i); auto &value = op.getResult(i);
// Skip from from first variadic results for now. Else getResult index used // Skip from from first variadic results for now. Else getResult index used
// below doesn't match. // below doesn't match.
if (value.isVariadic()) if (value.isVariadic())
break; break;
if (!value.name.empty()) if (!value.name.empty())
verifyCtx.addSubst(value.name, verifyCtx.addSubst(value.name,
formatv("(*this->getOperation()->getResult({0}))", i)); formatv("this->getOperation()->getResult({0})", i));
} }
// Verify the attributes have the correct type. // Verify the attributes have the correct type.
for (const auto &namedAttr : op.getAttributes()) { for (const auto &namedAttr : op.getAttributes()) {
const auto &attr = namedAttr.attr; const auto &attr = namedAttr.attr;
if (attr.isDerivedAttr()) if (attr.isDerivedAttr())
continue; continue;
▲ Show 20 Lines • Show All 90 Lines • ▼ Show 20 Lines body << formatv(" for (Value v : getODS{0}{1}s({2})) {{\n",
valueKind.substr(0, 1).upper(), valueKind.substr(1), valueKind.substr(0, 1).upper(), valueKind.substr(1),
staticValue.index()); staticValue.index());
auto constraint = staticValue.value().constraint; auto constraint = staticValue.value().constraint;
body << " (void)v;\n" body << " (void)v;\n"
<< " if (!(" << " if (!("
<< tgfmt(constraint.getConditionTemplate(), << tgfmt(constraint.getConditionTemplate(),
&fctx.withSelf("v->getType()")) &fctx.withSelf("v.getType()"))
<< ")) {\n" << ")) {\n"
<< formatv(" return emitOpError(\"{0} #\") << index " << formatv(" return emitOpError(\"{0} #\") << index "
"<< \" must be {1}, but got \" << v->getType();\n", "<< \" must be {1}, but got \" << v.getType();\n",
valueKind, constraint.getDescription()) valueKind, constraint.getDescription())
<< " }\n" // if << " }\n" // if
<< " ++index;\n" << " ++index;\n"
<< " }\n"; // for << " }\n"; // for
} }
body << " }\n"; body << " }\n";
} }
▲ Show 20 Lines • Show All 242 Lines • Show Last 20 Lines

mlir/tools/mlir-tblgen/RewriterGen.cpp

Show First 20 Lines • Show All 241 Lines • ▼ Show 20 Lines if (DagNode argTree = tree.getArgAsNestedDag(i)) {
op.getOperationName(), i); op.getOperationName(), i);
PrintFatalError(loc, error); PrintFatalError(loc, error);
} }
} }
os.indent(indent) << "{\n"; os.indent(indent) << "{\n";
os.indent(indent + 2) << formatv( os.indent(indent + 2) << formatv(
"auto *op{0} = " "auto *op{0} = "
"(*castedOp{1}.getODSOperands({2}).begin())->getDefiningOp();\n", "(*castedOp{1}.getODSOperands({2}).begin()).getDefiningOp();\n",
depth + 1, depth, i); depth + 1, depth, i);
emitOpMatch(argTree, depth + 1); emitOpMatch(argTree, depth + 1);
os.indent(indent + 2) os.indent(indent + 2)
<< formatv("tblgen_ops[{0}] = op{1};\n", ++opCounter, depth + 1); << formatv("tblgen_ops[{0}] = op{1};\n", ++opCounter, depth + 1);
os.indent(indent) << "}\n"; os.indent(indent) << "}\n";
continue; continue;
} }
Show All 31 Lines if (!matcher.isUnspecified()) {
if (operand->constraint != matcher.getAsConstraint()) { if (operand->constraint != matcher.getAsConstraint()) {
if (operand->isVariadic()) { if (operand->isVariadic()) {
auto error = formatv( auto error = formatv(
"further constrain op {0}'s variadic operand #{1} unsupported now", "further constrain op {0}'s variadic operand #{1} unsupported now",
op.getOperationName(), argIndex); op.getOperationName(), argIndex);
PrintFatalError(loc, error); PrintFatalError(loc, error);
} }
auto self = auto self =
formatv("(*castedOp{0}.getODSOperands({1}).begin())->getType()", formatv("(*castedOp{0}.getODSOperands({1}).begin()).getType()", depth,
depth, argIndex); argIndex);
os.indent(indent) << "if (!(" os.indent(indent) << "if (!("
<< tgfmt(matcher.getConditionTemplate(), << tgfmt(matcher.getConditionTemplate(),
&fmtCtx.withSelf(self)) &fmtCtx.withSelf(self))
<< ")) return matchFailure();\n"; << ")) return matchFailure();\n";
} }
} }
// Capture the value // Capture the value
▲ Show 20 Lines • Show All 73 Lines • ▼ Show 20 Linesvoid PatternEmitter::emitMatchLogic(DagNode tree) {
for (auto &appliedConstraint : pattern.getConstraints()) { for (auto &appliedConstraint : pattern.getConstraints()) {
auto &constraint = appliedConstraint.constraint; auto &constraint = appliedConstraint.constraint;
auto &entities = appliedConstraint.entities; auto &entities = appliedConstraint.entities;
auto condition = constraint.getConditionTemplate(); auto condition = constraint.getConditionTemplate();
auto cmd = "if (!({0})) return matchFailure();\n"; auto cmd = "if (!({0})) return matchFailure();\n";
if (isa<TypeConstraint>(constraint)) { if (isa<TypeConstraint>(constraint)) {
auto self = formatv("({0}->getType())", auto self = formatv("({0}.getType())",
symbolInfoMap.getValueAndRangeUse(entities.front())); symbolInfoMap.getValueAndRangeUse(entities.front()));
os.indent(4) << formatv(cmd, os.indent(4) << formatv(cmd,
tgfmt(condition, &fmtCtx.withSelf(self.str()))); tgfmt(condition, &fmtCtx.withSelf(self.str())));
} else if (isa<AttrConstraint>(constraint)) { } else if (isa<AttrConstraint>(constraint)) {
PrintFatalError( PrintFatalError(
loc, "cannot use AttrConstraint in Pattern multi-entity constraints"); loc, "cannot use AttrConstraint in Pattern multi-entity constraints");
} else { } else {
// TODO(b/138794486): replace formatv arguments with the exact specified // TODO(b/138794486): replace formatv arguments with the exact specified
▲ Show 20 Lines • Show All 416 Lines • ▼ Show 20 Linesstd::string PatternEmitter::handleOpCreation(DagNode tree, int resultIndex,
// Then prepare the result types. We need to specify the types for all // Then prepare the result types. We need to specify the types for all
// results. // results.
os.indent(6) << formatv( os.indent(6) << formatv(
"SmallVector<Type, 4> tblgen_types; (void)tblgen_types;\n"); "SmallVector<Type, 4> tblgen_types; (void)tblgen_types;\n");
int numResults = resultOp.getNumResults(); int numResults = resultOp.getNumResults();
if (numResults != 0) { if (numResults != 0) {
for (int i = 0; i < numResults; ++i) for (int i = 0; i < numResults; ++i)
os.indent(6) << formatv("for (auto v : castedOp0.getODSResults({0})) {{" os.indent(6) << formatv("for (auto v : castedOp0.getODSResults({0})) {{"
"tblgen_types.push_back(v->getType()); }\n", "tblgen_types.push_back(v.getType()); }\n",
resultIndex + i); resultIndex + i);
} }
os.indent(6) << formatv("{0} = rewriter.create<{1}>(loc, tblgen_types, " os.indent(6) << formatv("{0} = rewriter.create<{1}>(loc, tblgen_types, "
"tblgen_values, tblgen_attrs);\n", "tblgen_values, tblgen_attrs);\n",
valuePackName, resultOp.getQualCppClassName()); valuePackName, resultOp.getQualCppClassName());
os.indent(4) << "}\n"; os.indent(4) << "}\n";
return resultValue; return resultValue;
} }
▲ Show 20 Lines • Show All 214 Lines • Show Last 20 Lines