This is an archive of the discontinued LLVM Phabricator instance.

Differential D99127

[PatternMatch] Big mechanical rename OwningRewritePatternList -> RewritePatternSet and insert -> add. NFC
ClosedPublic

Authored by lattner on Mar 22 2021, 5:00 PM.

Details

Reviewers
mravishankar
antiagainst
rriddle
aartbik
jpienaar
ftynse
silvas
jdoerfert
nicolasvasilache
herhut
Commits
rGdc4e913be9c3: [PatternMatch] Big mechanical rename OwningRewritePatternList ->…
Summary

This doesn't change APIs, this just cleans up the many in-tree uses of these
names to use the new preferred names. We'll keep the old names around for a
couple weeks to help transitions.

Diff Detail

Event Timeline

lattner created this revision.Mar 22 2021, 5:00 PM
Herald added a reviewer: mravishankar. · View Herald TranscriptMar 22 2021, 5:00 PM
Herald added a reviewer: antiagainst. · View Herald Transcript
Herald added a reviewer: rriddle. · View Herald Transcript
Herald added a reviewer: aartbik. · View Herald Transcript
Herald added a reviewer: jpienaar. · View Herald Transcript
Herald added a reviewer: ftynse. · View Herald Transcript
Herald added a reviewer: silvas. · View Herald Transcript
Herald added a reviewer: aartbik. · View Herald Transcript
Herald added a reviewer: aartbik. · View Herald Transcript
Herald added subscribers: dcaballe, cota, mravishankar and 24 others. · View Herald Transcript
lattner requested review of this revision.Mar 22 2021, 5:00 PM
Herald added a reviewer: jdoerfert. · View Herald TranscriptMar 22 2021, 5:00 PM
Herald added a reviewer: nicolasvasilache. · View Herald Transcript
Herald added a reviewer: herhut. · View Herald Transcript
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: sstefan1, stephenneuendorffer, nicolasvasilache. · View Herald Transcript
rriddle accepted this revision.Mar 22 2021, 5:11 PM

Looks good! (Didn't look at all of them, but I trust that you renamed+clang-formatted correctly).

Can you also update the naming of https://github.com/llvm/llvm-project/blob/main/mlir/include/mlir/Rewrite/FrozenRewritePatternList.h for consistency while you are at it?

lattner added a comment.Mar 22 2021, 5:19 PM

Sounds good, I'll update that in a separate patch.

rriddle added a comment.Mar 22 2021, 5:20 PM
In D99127#2643137, @lattner wrote:

Sounds good, I'll update that in a separate patch.

Great, thanks!

lattner added a child revision: D99130: Tidy up some docs..Mar 22 2021, 5:20 PM
This revision was not accepted when it landed; it landed in state Needs Review.Mar 22 2021, 5:21 PM
This revision was landed with ongoing or failed builds.
Closed by commit rGdc4e913be9c3: [PatternMatch] Big mechanical rename OwningRewritePatternList ->… (authored by lattner). · Explain Why
This revision was automatically updated to reflect the committed changes.
lattner added a commit: rGdc4e913be9c3: [PatternMatch] Big mechanical rename OwningRewritePatternList ->….
Harbormaster completed remote builds in B95127: Diff 332476.Mar 22 2021, 5:58 PM

Revision Contents

PathSize
mlir/
docs/
12 lines
4 lines
12 lines
Tutorials/
2 lines
Toy/
4 lines
4 lines
4 lines
examples/
toy/
Ch3/
mlir/
10 lines
Ch4/
mlir/
10 lines
Ch5/
mlir/
6 lines
10 lines
Ch6/
mlir/
6 lines
4 lines
10 lines
Ch7/
mlir/
6 lines
4 lines
10 lines
include/
mlir/
Conversion/
AffineToStandard/
5 lines
ArmSVEToLLVM/
2 lines
AsyncToLLVM/
2 lines
ComplexToLLVM/
4 lines
GPUToNVVM/
2 lines
GPUToROCDL/
2 lines
GPUToSPIRV/
2 lines
LinalgToLLVM/
5 lines
LinalgToSPIRV/
2 lines
LinalgToStandard/
3 lines
OpenMPToLLVM/
2 lines
SCFToGPU/
2 lines
SCFToSPIRV/
2 lines
SCFToStandard/
2 lines
SPIRVToLLVM/
6 lines
ShapeToStandard/
5 lines
StandardToLLVM/
14 lines
StandardToSPIRV/
6 lines
TosaToLinalg/
2 lines
TosaToSCF/
2 lines
TosaToStandard/
5 lines
VectorToLLVM/
6 lines
VectorToROCDL/
4 lines
VectorToSCF/
2 lines
VectorToSPIRV/
2 lines
Dialect/
AMX/
4 lines
AVX512/
4 lines
GPU/
4 lines
Linalg/
16 lines
Transforms/
37 lines
31 lines
SCF/
2 lines
SPIRV/
IR/
2 lines
Transforms/
2 lines
Shape/
Transforms/
6 lines
StandardOps/
Transforms/
7 lines
8 lines
4 lines
Tensor/
Transforms/
2 lines
Vector/
18 lines
2 lines
IR/
2 lines
4 lines
2 lines
Rewrite/
2 lines
Transforms/
2 lines
12 lines
lib/
Conversion/
AffineToStandard/
11 lines
ArmSVEToLLVM/
8 lines
AsyncToLLVM/
31 lines
ComplexToLLVM/
6 lines
GPUCommon/
22 lines
GPUToNVVM/
96 lines
GPUToROCDL/
88 lines
GPUToSPIRV/
4 lines
2 lines
LinalgToLLVM/
8 lines
LinalgToSPIRV/
5 lines
2 lines
LinalgToStandard/
8 lines
OpenMPToLLVM/
10 lines
SCFToGPU/
5 lines
2 lines
SCFToOpenMP/
4 lines
SCFToSPIRV/
4 lines
2 lines
SCFToStandard/
9 lines
SPIRVToLLVM/
4 lines
12 lines
2 lines
ShapeToStandard/
10 lines
6 lines
StandardToLLVM/
24 lines
StandardToSPIRV/
12 lines
14 lines
2 lines
TosaToLinalg/
4 lines
2 lines
TosaToSCF/
6 lines
2 lines
TosaToStandard/
8 lines
2 lines
VectorToLLVM/
62 lines
4 lines
VectorToROCDL/
8 lines
VectorToSCF/
9 lines
VectorToSPIRV/
14 lines
2 lines
Dialect/
AMX/
Transforms/
6 lines
AVX512/
Transforms/
10 lines
Affine/
IR/
44 lines
Transforms/
2 lines
2 lines
Utils/
2 lines
Async/
Transforms/
4 lines
10 lines
GPU/
Transforms/
4 lines
Linalg/
IR/
27 lines
Transforms/
8 lines
16 lines
12 lines
18 lines
6 lines
38 lines
14 lines
2 lines
6 lines
8 lines
4 lines
20 lines
22 lines
Math/
Transforms/
6 lines
MemRef/
IR/
40 lines
Quant/
Transforms/
4 lines
4 lines
SCF/
17 lines
Transforms/
2 lines
4 lines
SPIRV/
IR/
25 lines
15 lines
Transforms/
9 lines
4 lines
6 lines
Shape/
IR/
60 lines
Transforms/
2 lines
7 lines
6 lines
4 lines
StandardOps/
IR/
41 lines
Transforms/
8 lines
10 lines
8 lines
2 lines
18 lines
4 lines
Tensor/
IR/
16 lines
Transforms/
8 lines
Tosa/
Transforms/
24 lines
Vector/
76 lines
46 lines
Rewrite/
3 lines
Transforms/
8 lines
2 lines
Utils/
8 lines
2 lines
test/
lib/
Dialect/
Affine/
2 lines
SPIRV/
8 lines
2 lines
Test/
10 lines
51 lines
2 lines
Tosa/
6 lines
Rewrite/
2 lines
Transforms/
14 lines
4 lines
2 lines
2 lines
2 lines
20 lines
90 lines
2 lines
4 lines
40 lines
mlir-tblgen/
2 lines
tools/
mlir-linalg-ods-gen/
7 lines
6 lines
mlir-tblgen/
2 lines
4 lines
unittests/
Rewrite/
6 lines

Diff 332480

mlir/docs/Bufferization.md

Show First 20 Lines • Show All 150 Lines • ▼ Show 20 Lines
See [the talk](#the-talk) for more details on how to write these patterns. See [the talk](#the-talk) for more details on how to write these patterns.
The The
[pass itself](https://github.com/llvm/llvm-project/blob/bc8acf2ce8ad6e8c9b1d97b2e02d3f4ad26e1d9d/mlir/lib/Dialect/Tensor/Transforms/Bufferize.cpp#L57) [pass itself](https://github.com/llvm/llvm-project/blob/bc8acf2ce8ad6e8c9b1d97b2e02d3f4ad26e1d9d/mlir/lib/Dialect/Tensor/Transforms/Bufferize.cpp#L57)
is very small, and follows the basic pattern of any dialect conversion pass. is very small, and follows the basic pattern of any dialect conversion pass.
``` ```
void mlir::populateTensorBufferizePatterns( void mlir::populateTensorBufferizePatterns(
MLIRContext *context, BufferizeTypeConverter &typeConverter, BufferizeTypeConverter &typeConverter, RewritePatternSet &patterns) {
OwningRewritePatternList &patterns) { patterns.add<BufferizeCastOp, BufferizeExtractOp>(typeConverter,
patterns.insert<BufferizeCastOp, BufferizeExtractOp>(typeConverter, context); patterns.getContext());
} }
struct TensorBufferizePass : public TensorBufferizeBase<TensorBufferizePass> { struct TensorBufferizePass : public TensorBufferizeBase<TensorBufferizePass> {
void runOnFunction() override { void runOnFunction() override {
auto *context = &getContext(); auto *context = &getContext();
BufferizeTypeConverter typeConverter; BufferizeTypeConverter typeConverter;
OwningRewritePatternList patterns; RewritePatternSet patterns(context);
ConversionTarget target(*context); ConversionTarget target(*context);
populateTensorBufferizePatterns(context, typeConverter, patterns); populateTensorBufferizePatterns(typeConverter, patterns);
target.addIllegalOp<tensor::CastOp, tensor::ExtractOp>(); target.addIllegalOp<tensor::CastOp, tensor::ExtractOp>();
target.addLegalDialect<StandardOpsDialect>(); target.addLegalDialect<StandardOpsDialect>();
if (failed( if (failed(
applyPartialConversion(getFunction(), target, std::move(patterns)))) applyPartialConversion(getFunction(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
}; };
``` ```
The pass has all the hallmarks of a dialect conversion pass that does type The pass has all the hallmarks of a dialect conversion pass that does type
conversions: a `TypeConverter`, a `OwningRewritePatternList`, and a conversions: a `TypeConverter`, a `RewritePatternSet`, and a
`ConversionTarget`, and a call to `applyPartialConversion`. Note that a function `ConversionTarget`, and a call to `applyPartialConversion`. Note that a function
`populateTensorBufferizePatterns` is separated, so that power users can use the `populateTensorBufferizePatterns` is separated, so that power users can use the
patterns independently, if necessary (such as to combine multiple sets of patterns independently, if necessary (such as to combine multiple sets of
conversion patterns into a single conversion call, for performance). conversion patterns into a single conversion call, for performance).
One convenient utility provided by the MLIR bufferization infrastructure is the One convenient utility provided by the MLIR bufferization infrastructure is the
`BufferizeTypeConverter`, which comes pre-loaded with the necessary conversions `BufferizeTypeConverter`, which comes pre-loaded with the necessary conversions
and materializations between `tensor` and `memref`. and materializations between `tensor` and `memref`.
▲ Show 20 Lines • Show All 86 Lines • Show Last 20 Lines

mlir/docs/Canonicalization.md

Show First 20 Lines • Show All 73 Lines • ▼ Show 20 Lines
def MyOp : ... { def MyOp : ... {
let hasCanonicalizer = 1; let hasCanonicalizer = 1;
} }
``` ```
Canonicalization patterns can then be provided in the source file: Canonicalization patterns can then be provided in the source file:
```c++ ```c++
void MyOp::getCanonicalizationPatterns(OwningRewritePatternList &patterns, void MyOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
patterns.insert<...>(...); patterns.add<...>(...);
} }
``` ```
See the [quickstart guide](Tutorials/QuickstartRewrites.md) for information on See the [quickstart guide](Tutorials/QuickstartRewrites.md) for information on
defining operation rewrites. defining operation rewrites.
### Canonicalizing with `fold` ### Canonicalizing with `fold`
▲ Show 20 Lines • Show All 118 Lines • Show Last 20 Lines

mlir/docs/PatternRewriter.md

Show First 20 Lines • Show All 148 Lines • ▼ Show 20 Lines
of the same functionality present within an `OpBuilder`. This includes operation of the same functionality present within an `OpBuilder`. This includes operation
creation, as well as many useful attribute and type construction methods. creation, as well as many useful attribute and type construction methods.
## Pattern Application ## Pattern Application
After a set of patterns have been defined, they are collected and provided to a After a set of patterns have been defined, they are collected and provided to a
specific driver for application. A driver consists of several high levels parts: specific driver for application. A driver consists of several high levels parts:
* Input `OwningRewritePatternList` * Input `RewritePatternSet`
The input patterns to a driver are provided in the form of an The input patterns to a driver are provided in the form of an
`OwningRewritePatternList`. This class provides a simplified API for building a `RewritePatternSet`. This class provides a simplified API for building a
list of patterns. list of patterns.
* Driver-specific `PatternRewriter` * Driver-specific `PatternRewriter`
To ensure that the driver state does not become invalidated by IR mutations To ensure that the driver state does not become invalidated by IR mutations
within the pattern rewriters, a driver must provide a `PatternRewriter` instance within the pattern rewriters, a driver must provide a `PatternRewriter` instance
with the necessary hooks overridden. If a driver does not need to hook into with the necessary hooks overridden. If a driver does not need to hook into
certain mutations, a default implementation is provided that will perform the certain mutations, a default implementation is provided that will perform the
mutation directly. mutation directly.
* Pattern Application and Cost Model * Pattern Application and Cost Model
Each driver is responsible for defining its own operation visitation order as Each driver is responsible for defining its own operation visitation order as
well as pattern cost model, but the final application is performed via a well as pattern cost model, but the final application is performed via a
`PatternApplicator` class. This class takes as input the `PatternApplicator` class. This class takes as input the
`OwningRewritePatternList` and transforms the patterns based upon a provided `RewritePatternSet` and transforms the patterns based upon a provided
cost model. This cost model computes a final benefit for a given pattern, using cost model. This cost model computes a final benefit for a given pattern, using
whatever driver specific information necessary. After a cost model has been whatever driver specific information necessary. After a cost model has been
computed, the driver may begin to match patterns against operations using computed, the driver may begin to match patterns against operations using
`PatternApplicator::matchAndRewrite`. `PatternApplicator::matchAndRewrite`.
An example is shown below: An example is shown below:
```c++ ```c++
class MyPattern : public RewritePattern { class MyPattern : public RewritePattern {
public: public:
MyPattern(PatternBenefit benefit, MLIRContext *context) MyPattern(PatternBenefit benefit, MLIRContext *context)
: RewritePattern(MyOp::getOperationName(), benefit, context) {} : RewritePattern(MyOp::getOperationName(), benefit, context) {}
}; };
/// Populate the pattern list. /// Populate the pattern list.
void collectMyPatterns(OwningRewritePatternList &patterns, MLIRContext *ctx) { void collectMyPatterns(RewritePatternSet &patterns, MLIRContext *ctx) {
patterns.insert<MyPattern>(/*benefit=*/1, ctx); patterns.add<MyPattern>(/*benefit=*/1, ctx);
} }
/// Define a custom PatternRewriter for use by the driver. /// Define a custom PatternRewriter for use by the driver.
class MyPatternRewriter : public PatternRewriter { class MyPatternRewriter : public PatternRewriter {
public: public:
MyPatternRewriter(MLIRContext *ctx) : PatternRewriter(ctx) {} MyPatternRewriter(MLIRContext *ctx) : PatternRewriter(ctx) {}
/// Override the necessary PatternRewriter hooks here. /// Override the necessary PatternRewriter hooks here.
}; };
/// Apply the custom driver to `op`. /// Apply the custom driver to `op`.
void applyMyPatternDriver(Operation *op, void applyMyPatternDriver(Operation *op,
const OwningRewritePatternList &patterns) { const RewritePatternSet &patterns) {
// Initialize the custom PatternRewriter. // Initialize the custom PatternRewriter.
MyPatternRewriter rewriter(op->getContext()); MyPatternRewriter rewriter(op->getContext());
// Create the applicator and apply our cost model. // Create the applicator and apply our cost model.
PatternApplicator applicator(patterns); PatternApplicator applicator(patterns);
applicator.applyCostModel([](const Pattern &pattern) { applicator.applyCostModel([](const Pattern &pattern) {
// Apply a default cost model. // Apply a default cost model.
// Note: This is just for demonstration, if the default cost model is truly // Note: This is just for demonstration, if the default cost model is truly
▲ Show 20 Lines • Show All 42 Lines • Show Last 20 Lines

mlir/docs/Tutorials/QuickstartRewrites.md

Show First 20 Lines • Show All 149 Lines • ▼ Show 20 Lines
set(LLVM_TARGET_DEFINITIONS <name-of-the-td-file>) set(LLVM_TARGET_DEFINITIONS <name-of-the-td-file>)
mlir_tablegen(<name-of-the-generated-inc-file> -gen-rewriters) mlir_tablegen(<name-of-the-generated-inc-file> -gen-rewriters)
add_public_tablegen_target(<name-of-the-cmake-target>) add_public_tablegen_target(<name-of-the-cmake-target>)
``` ```
Then you can `#include` the generated file in any C++ implementation file you Then you can `#include` the generated file in any C++ implementation file you
like. (You will also need to make sure the library depends on the CMake target like. (You will also need to make sure the library depends on the CMake target
defined in the above.) The generated file will have a `populateWithGenerated( defined in the above.) The generated file will have a `populateWithGenerated(
OwningRewritePatternList &patterns)` function that you can RewritePatternSet &patterns)` function that you can
use to collect all the generated patterns inside `patterns` and then use use to collect all the generated patterns inside `patterns` and then use
`patterns` in any pass you would like. `patterns` in any pass you would like.
### C++ rewrite specification ### C++ rewrite specification
In case patterns are not sufficient there is also the fully C++ way of In case patterns are not sufficient there is also the fully C++ way of
expressing a rewrite: expressing a rewrite:
▲ Show 20 Lines • Show All 102 Lines • Show Last 20 Lines

mlir/docs/Tutorials/Toy/Ch-3.md

Show First 20 Lines • Show All 108 Lines • ▼ Show 20 Lines
defined by operations in a greedy, iterative manner. To ensure that the defined by operations in a greedy, iterative manner. To ensure that the
canonicalization pass applies our new transform, we set canonicalization pass applies our new transform, we set
[hasCanonicalizer = 1](../../OpDefinitions.md#hascanonicalizer) and register the [hasCanonicalizer = 1](../../OpDefinitions.md#hascanonicalizer) and register the
pattern with the canonicalization framework. pattern with the canonicalization framework.
```c++ ```c++
// Register our patterns for rewrite by the Canonicalization framework. // Register our patterns for rewrite by the Canonicalization framework.
void TransposeOp::getCanonicalizationPatterns( void TransposeOp::getCanonicalizationPatterns(
OwningRewritePatternList &results, MLIRContext *context) { RewritePatternSet &results, MLIRContext *context) {
results.insert<SimplifyRedundantTranspose>(context); results.add<SimplifyRedundantTranspose>(context);
} }
``` ```
We also need to update our main file, `toyc.cpp`, to add an optimization We also need to update our main file, `toyc.cpp`, to add an optimization
pipeline. In MLIR, the optimizations are run through a `PassManager` in a pipeline. In MLIR, the optimizations are run through a `PassManager` in a
similar way to LLVM: similar way to LLVM:
```c++ ```c++
▲ Show 20 Lines • Show All 136 Lines • Show Last 20 Lines

mlir/docs/Tutorials/Toy/Ch-5.md

Show First 20 Lines • Show All 141 Lines • ▼ Show 20 Lines
Now we can prepare the list of patterns to use during the lowering process: Now we can prepare the list of patterns to use during the lowering process:
```c++ ```c++
void ToyToAffineLoweringPass::runOnFunction() { void ToyToAffineLoweringPass::runOnFunction() {
... ...
// Now that the conversion target has been defined, we just need to provide // Now that the conversion target has been defined, we just need to provide
// the set of patterns that will lower the Toy operations. // the set of patterns that will lower the Toy operations.
mlir::OwningRewritePatternList patterns; mlir::RewritePatternSet patterns;
patterns.insert<..., TransposeOpLowering>(&getContext()); patterns.add<..., TransposeOpLowering>(&getContext());
... ...
``` ```
## Partial Lowering ## Partial Lowering
Once the patterns have been defined, we can perform the actual lowering. The Once the patterns have been defined, we can perform the actual lowering. The
`DialectConversion` framework provides several different modes of lowering, but, `DialectConversion` framework provides several different modes of lowering, but,
▲ Show 20 Lines • Show All 187 Lines • Show Last 20 Lines

mlir/docs/Tutorials/Toy/Ch-6.md

Show First 20 Lines • Show All 84 Lines • ▼ Show 20 Lines
Now that the conversion target has been defined, we need to provide the patterns Now that the conversion target has been defined, we need to provide the patterns
used for lowering. At this point in the compilation process, we have a used for lowering. At this point in the compilation process, we have a
combination of `toy`, `affine`, and `std` operations. Luckily, the `std` and combination of `toy`, `affine`, and `std` operations. Luckily, the `std` and
`affine` dialects already provide the set of patterns needed to transform them `affine` dialects already provide the set of patterns needed to transform them
into LLVM dialect. These patterns allow for lowering the IR in multiple stages into LLVM dialect. These patterns allow for lowering the IR in multiple stages
by relying on [transitive lowering](../../../getting_started/Glossary.md#transitive-lowering). by relying on [transitive lowering](../../../getting_started/Glossary.md#transitive-lowering).
```c++ ```c++
mlir::OwningRewritePatternList patterns; mlir::RewritePatternSet patterns;
mlir::populateAffineToStdConversionPatterns(patterns, &getContext()); mlir::populateAffineToStdConversionPatterns(patterns, &getContext());
mlir::populateLoopToStdConversionPatterns(patterns, &getContext()); mlir::populateLoopToStdConversionPatterns(patterns, &getContext());
mlir::populateStdToLLVMConversionPatterns(typeConverter, patterns); mlir::populateStdToLLVMConversionPatterns(typeConverter, patterns);
// The only remaining operation, to lower from the `toy` dialect, is the // The only remaining operation, to lower from the `toy` dialect, is the
// PrintOp. // PrintOp.
patterns.insert<PrintOpLowering>(&getContext()); patterns.add<PrintOpLowering>(&getContext());
``` ```
### Full Lowering ### Full Lowering
We want to completely lower to LLVM, so we use a `FullConversion`. This ensures We want to completely lower to LLVM, so we use a `FullConversion`. This ensures
that only legal operations will remain after the conversion. that only legal operations will remain after the conversion.
```c++ ```c++
▲ Show 20 Lines • Show All 220 Lines • Show Last 20 Lines

mlir/examples/toy/Ch3/mlir/ToyCombine.cpp

Show First 20 Lines • Show All 48 Lines • ▼ Show 20 Lines matchAndRewrite(TransposeOp op,
// Otherwise, we have a redundant transpose. Use the rewriter. // Otherwise, we have a redundant transpose. Use the rewriter.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}); rewriter.replaceOp(op, {transposeInputOp.getOperand()});
return success(); return success();
} }
}; };
/// Register our patterns as "canonicalization" patterns on the TransposeOp so /// Register our patterns as "canonicalization" patterns on the TransposeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void TransposeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void TransposeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyRedundantTranspose>(context); results.add<SimplifyRedundantTranspose>(context);
} }
/// Register our patterns as "canonicalization" patterns on the ReshapeOp so /// Register our patterns as "canonicalization" patterns on the ReshapeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void ReshapeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ReshapeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ReshapeReshapeOptPattern, RedundantReshapeOptPattern, results.add<ReshapeReshapeOptPattern, RedundantReshapeOptPattern,
FoldConstantReshapeOptPattern>(context); FoldConstantReshapeOptPattern>(context);
} }

mlir/examples/toy/Ch4/mlir/ToyCombine.cpp

Show First 20 Lines • Show All 48 Lines • ▼ Show 20 Lines matchAndRewrite(TransposeOp op,
// Otherwise, we have a redundant transpose. Use the rewriter. // Otherwise, we have a redundant transpose. Use the rewriter.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}); rewriter.replaceOp(op, {transposeInputOp.getOperand()});
return success(); return success();
} }
}; };
/// Register our patterns as "canonicalization" patterns on the TransposeOp so /// Register our patterns as "canonicalization" patterns on the TransposeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void TransposeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void TransposeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyRedundantTranspose>(context); results.add<SimplifyRedundantTranspose>(context);
} }
/// Register our patterns as "canonicalization" patterns on the ReshapeOp so /// Register our patterns as "canonicalization" patterns on the ReshapeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void ReshapeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ReshapeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ReshapeReshapeOptPattern, RedundantReshapeOptPattern, results.add<ReshapeReshapeOptPattern, RedundantReshapeOptPattern,
FoldConstantReshapeOptPattern>(context); FoldConstantReshapeOptPattern>(context);
} }

mlir/examples/toy/Ch5/mlir/LowerToAffineLoops.cpp

Show First 20 Lines • Show All 291 Lines • ▼ Show 20 Linesvoid ToyToAffineLoweringPass::runOnFunction() {
// if any of these operations are *not* converted. Given that we actually want // if any of these operations are *not* converted. Given that we actually want
// a partial lowering, we explicitly mark the Toy operations that don't want // a partial lowering, we explicitly mark the Toy operations that don't want
// to lower, `toy.print`, as `legal`. // to lower, `toy.print`, as `legal`.
target.addIllegalDialect<toy::ToyDialect>(); target.addIllegalDialect<toy::ToyDialect>();
target.addLegalOp<toy::PrintOp>(); target.addLegalOp<toy::PrintOp>();
// Now that the conversion target has been defined, we just need to provide // Now that the conversion target has been defined, we just need to provide
// the set of patterns that will lower the Toy operations. // the set of patterns that will lower the Toy operations.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
patterns.insert<AddOpLowering, ConstantOpLowering, MulOpLowering, patterns.add<AddOpLowering, ConstantOpLowering, MulOpLowering,
ReturnOpLowering, TransposeOpLowering>(&getContext()); ReturnOpLowering, TransposeOpLowering>(&getContext());
// With the target and rewrite patterns defined, we can now attempt the // With the target and rewrite patterns defined, we can now attempt the
// conversion. The conversion will signal failure if any of our `illegal` // conversion. The conversion will signal failure if any of our `illegal`
// operations were not converted successfully. // operations were not converted successfully.
if (failed( if (failed(
applyPartialConversion(getFunction(), target, std::move(patterns)))) applyPartialConversion(getFunction(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
/// Create a pass for lowering operations in the `Affine` and `Std` dialects, /// Create a pass for lowering operations in the `Affine` and `Std` dialects,
/// for a subset of the Toy IR (e.g. matmul). /// for a subset of the Toy IR (e.g. matmul).
std::unique_ptr<Pass> mlir::toy::createLowerToAffinePass() { std::unique_ptr<Pass> mlir::toy::createLowerToAffinePass() {
return std::make_unique<ToyToAffineLoweringPass>(); return std::make_unique<ToyToAffineLoweringPass>();
} }

mlir/examples/toy/Ch5/mlir/ToyCombine.cpp

Show First 20 Lines • Show All 48 Lines • ▼ Show 20 Lines matchAndRewrite(TransposeOp op,
// Otherwise, we have a redundant transpose. Use the rewriter. // Otherwise, we have a redundant transpose. Use the rewriter.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}); rewriter.replaceOp(op, {transposeInputOp.getOperand()});
return success(); return success();
} }
}; };
/// Register our patterns as "canonicalization" patterns on the TransposeOp so /// Register our patterns as "canonicalization" patterns on the TransposeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void TransposeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void TransposeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyRedundantTranspose>(context); results.add<SimplifyRedundantTranspose>(context);
} }
/// Register our patterns as "canonicalization" patterns on the ReshapeOp so /// Register our patterns as "canonicalization" patterns on the ReshapeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void ReshapeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ReshapeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ReshapeReshapeOptPattern, RedundantReshapeOptPattern, results.add<ReshapeReshapeOptPattern, RedundantReshapeOptPattern,
FoldConstantReshapeOptPattern>(context); FoldConstantReshapeOptPattern>(context);
} }

mlir/examples/toy/Ch6/mlir/LowerToAffineLoops.cpp

Show First 20 Lines • Show All 290 Lines • ▼ Show 20 Linesvoid ToyToAffineLoweringPass::runOnFunction() {
// if any of these operations are *not* converted. Given that we actually want // if any of these operations are *not* converted. Given that we actually want
// a partial lowering, we explicitly mark the Toy operations that don't want // a partial lowering, we explicitly mark the Toy operations that don't want
// to lower, `toy.print`, as `legal`. // to lower, `toy.print`, as `legal`.
target.addIllegalDialect<toy::ToyDialect>(); target.addIllegalDialect<toy::ToyDialect>();
target.addLegalOp<toy::PrintOp>(); target.addLegalOp<toy::PrintOp>();
// Now that the conversion target has been defined, we just need to provide // Now that the conversion target has been defined, we just need to provide
// the set of patterns that will lower the Toy operations. // the set of patterns that will lower the Toy operations.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
patterns.insert<AddOpLowering, ConstantOpLowering, MulOpLowering, patterns.add<AddOpLowering, ConstantOpLowering, MulOpLowering,
ReturnOpLowering, TransposeOpLowering>(&getContext()); ReturnOpLowering, TransposeOpLowering>(&getContext());
// With the target and rewrite patterns defined, we can now attempt the // With the target and rewrite patterns defined, we can now attempt the
// conversion. The conversion will signal failure if any of our `illegal` // conversion. The conversion will signal failure if any of our `illegal`
// operations were not converted successfully. // operations were not converted successfully.
if (failed( if (failed(
applyPartialConversion(getFunction(), target, std::move(patterns)))) applyPartialConversion(getFunction(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
/// Create a pass for lowering operations in the `Affine` and `Std` dialects, /// Create a pass for lowering operations in the `Affine` and `Std` dialects,
/// for a subset of the Toy IR (e.g. matmul). /// for a subset of the Toy IR (e.g. matmul).
std::unique_ptr<Pass> mlir::toy::createLowerToAffinePass() { std::unique_ptr<Pass> mlir::toy::createLowerToAffinePass() {
return std::make_unique<ToyToAffineLoweringPass>(); return std::make_unique<ToyToAffineLoweringPass>();
} }

mlir/examples/toy/Ch6/mlir/LowerToLLVM.cpp

Show First 20 Lines • Show All 185 Lines • ▼ Show 20 Linesvoid ToyToLLVMLoweringPass::runOnOperation() {
// Now that the conversion target has been defined, we need to provide the // Now that the conversion target has been defined, we need to provide the
// patterns used for lowering. At this point of the compilation process, we // patterns used for lowering. At this point of the compilation process, we
// have a combination of `toy`, `affine`, and `std` operations. Luckily, there // have a combination of `toy`, `affine`, and `std` operations. Luckily, there
// are already exists a set of patterns to transform `affine` and `std` // are already exists a set of patterns to transform `affine` and `std`
// dialects. These patterns lowering in multiple stages, relying on transitive // dialects. These patterns lowering in multiple stages, relying on transitive
// lowerings. Transitive lowering, or A->B->C lowering, is when multiple // lowerings. Transitive lowering, or A->B->C lowering, is when multiple
// patterns must be applied to fully transform an illegal operation into a // patterns must be applied to fully transform an illegal operation into a
// set of legal ones. // set of legal ones.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateAffineToStdConversionPatterns(patterns); populateAffineToStdConversionPatterns(patterns);
populateLoopToStdConversionPatterns(patterns); populateLoopToStdConversionPatterns(patterns);
populateStdToLLVMConversionPatterns(typeConverter, patterns); populateStdToLLVMConversionPatterns(typeConverter, patterns);
// The only remaining operation to lower from the `toy` dialect, is the // The only remaining operation to lower from the `toy` dialect, is the
// PrintOp. // PrintOp.
patterns.insert<PrintOpLowering>(&getContext()); patterns.add<PrintOpLowering>(&getContext());
// We want to completely lower to LLVM, so we use a `FullConversion`. This // We want to completely lower to LLVM, so we use a `FullConversion`. This
// ensures that only legal operations will remain after the conversion. // ensures that only legal operations will remain after the conversion.
auto module = getOperation(); auto module = getOperation();
if (failed(applyFullConversion(module, target, std::move(patterns)))) if (failed(applyFullConversion(module, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
/// Create a pass for lowering operations the remaining `Toy` operations, as /// Create a pass for lowering operations the remaining `Toy` operations, as
/// well as `Affine` and `Std`, to the LLVM dialect for codegen. /// well as `Affine` and `Std`, to the LLVM dialect for codegen.
std::unique_ptr<mlir::Pass> mlir::toy::createLowerToLLVMPass() { std::unique_ptr<mlir::Pass> mlir::toy::createLowerToLLVMPass() {
return std::make_unique<ToyToLLVMLoweringPass>(); return std::make_unique<ToyToLLVMLoweringPass>();
} }

mlir/examples/toy/Ch6/mlir/ToyCombine.cpp

Show First 20 Lines • Show All 48 Lines • ▼ Show 20 Lines matchAndRewrite(TransposeOp op,
// Otherwise, we have a redundant transpose. Use the rewriter. // Otherwise, we have a redundant transpose. Use the rewriter.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}); rewriter.replaceOp(op, {transposeInputOp.getOperand()});
return success(); return success();
} }
}; };
/// Register our patterns as "canonicalization" patterns on the TransposeOp so /// Register our patterns as "canonicalization" patterns on the TransposeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void TransposeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void TransposeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyRedundantTranspose>(context); results.add<SimplifyRedundantTranspose>(context);
} }
/// Register our patterns as "canonicalization" patterns on the ReshapeOp so /// Register our patterns as "canonicalization" patterns on the ReshapeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void ReshapeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ReshapeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ReshapeReshapeOptPattern, RedundantReshapeOptPattern, results.add<ReshapeReshapeOptPattern, RedundantReshapeOptPattern,
FoldConstantReshapeOptPattern>(context); FoldConstantReshapeOptPattern>(context);
} }

mlir/examples/toy/Ch7/mlir/LowerToAffineLoops.cpp

Show First 20 Lines • Show All 291 Lines • ▼ Show 20 Linesvoid ToyToAffineLoweringPass::runOnFunction() {
// if any of these operations are *not* converted. Given that we actually want // if any of these operations are *not* converted. Given that we actually want
// a partial lowering, we explicitly mark the Toy operations that don't want // a partial lowering, we explicitly mark the Toy operations that don't want
// to lower, `toy.print`, as `legal`. // to lower, `toy.print`, as `legal`.
target.addIllegalDialect<toy::ToyDialect>(); target.addIllegalDialect<toy::ToyDialect>();
target.addLegalOp<toy::PrintOp>(); target.addLegalOp<toy::PrintOp>();
// Now that the conversion target has been defined, we just need to provide // Now that the conversion target has been defined, we just need to provide
// the set of patterns that will lower the Toy operations. // the set of patterns that will lower the Toy operations.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
patterns.insert<AddOpLowering, ConstantOpLowering, MulOpLowering, patterns.add<AddOpLowering, ConstantOpLowering, MulOpLowering,
ReturnOpLowering, TransposeOpLowering>(&getContext()); ReturnOpLowering, TransposeOpLowering>(&getContext());
// With the target and rewrite patterns defined, we can now attempt the // With the target and rewrite patterns defined, we can now attempt the
// conversion. The conversion will signal failure if any of our `illegal` // conversion. The conversion will signal failure if any of our `illegal`
// operations were not converted successfully. // operations were not converted successfully.
if (failed( if (failed(
applyPartialConversion(getFunction(), target, std::move(patterns)))) applyPartialConversion(getFunction(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
/// Create a pass for lowering operations in the `Affine` and `Std` dialects, /// Create a pass for lowering operations in the `Affine` and `Std` dialects,
/// for a subset of the Toy IR (e.g. matmul). /// for a subset of the Toy IR (e.g. matmul).
std::unique_ptr<Pass> mlir::toy::createLowerToAffinePass() { std::unique_ptr<Pass> mlir::toy::createLowerToAffinePass() {
return std::make_unique<ToyToAffineLoweringPass>(); return std::make_unique<ToyToAffineLoweringPass>();
} }

mlir/examples/toy/Ch7/mlir/LowerToLLVM.cpp

Show First 20 Lines • Show All 185 Lines • ▼ Show 20 Linesvoid ToyToLLVMLoweringPass::runOnOperation() {
// Now that the conversion target has been defined, we need to provide the // Now that the conversion target has been defined, we need to provide the
// patterns used for lowering. At this point of the compilation process, we // patterns used for lowering. At this point of the compilation process, we
// have a combination of `toy`, `affine`, and `std` operations. Luckily, there // have a combination of `toy`, `affine`, and `std` operations. Luckily, there
// are already exists a set of patterns to transform `affine` and `std` // are already exists a set of patterns to transform `affine` and `std`
// dialects. These patterns lowering in multiple stages, relying on transitive // dialects. These patterns lowering in multiple stages, relying on transitive
// lowerings. Transitive lowering, or A->B->C lowering, is when multiple // lowerings. Transitive lowering, or A->B->C lowering, is when multiple
// patterns must be applied to fully transform an illegal operation into a // patterns must be applied to fully transform an illegal operation into a
// set of legal ones. // set of legal ones.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateAffineToStdConversionPatterns(patterns); populateAffineToStdConversionPatterns(patterns);
populateLoopToStdConversionPatterns(patterns); populateLoopToStdConversionPatterns(patterns);
populateStdToLLVMConversionPatterns(typeConverter, patterns); populateStdToLLVMConversionPatterns(typeConverter, patterns);
// The only remaining operation to lower from the `toy` dialect, is the // The only remaining operation to lower from the `toy` dialect, is the
// PrintOp. // PrintOp.
patterns.insert<PrintOpLowering>(&getContext()); patterns.add<PrintOpLowering>(&getContext());
// We want to completely lower to LLVM, so we use a `FullConversion`. This // We want to completely lower to LLVM, so we use a `FullConversion`. This
// ensures that only legal operations will remain after the conversion. // ensures that only legal operations will remain after the conversion.
auto module = getOperation(); auto module = getOperation();
if (failed(applyFullConversion(module, target, std::move(patterns)))) if (failed(applyFullConversion(module, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
/// Create a pass for lowering operations the remaining `Toy` operations, as /// Create a pass for lowering operations the remaining `Toy` operations, as
/// well as `Affine` and `Std`, to the LLVM dialect for codegen. /// well as `Affine` and `Std`, to the LLVM dialect for codegen.
std::unique_ptr<mlir::Pass> mlir::toy::createLowerToLLVMPass() { std::unique_ptr<mlir::Pass> mlir::toy::createLowerToLLVMPass() {
return std::make_unique<ToyToLLVMLoweringPass>(); return std::make_unique<ToyToLLVMLoweringPass>();
} }

mlir/examples/toy/Ch7/mlir/ToyCombine.cpp

Show First 20 Lines • Show All 66 Lines • ▼ Show 20 Lines matchAndRewrite(TransposeOp op,
// Otherwise, we have a redundant transpose. Use the rewriter. // Otherwise, we have a redundant transpose. Use the rewriter.
rewriter.replaceOp(op, {transposeInputOp.getOperand()}); rewriter.replaceOp(op, {transposeInputOp.getOperand()});
return success(); return success();
} }
}; };
/// Register our patterns as "canonicalization" patterns on the TransposeOp so /// Register our patterns as "canonicalization" patterns on the TransposeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void TransposeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void TransposeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyRedundantTranspose>(context); results.add<SimplifyRedundantTranspose>(context);
} }
/// Register our patterns as "canonicalization" patterns on the ReshapeOp so /// Register our patterns as "canonicalization" patterns on the ReshapeOp so
/// that they can be picked up by the Canonicalization framework. /// that they can be picked up by the Canonicalization framework.
void ReshapeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ReshapeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ReshapeReshapeOptPattern, RedundantReshapeOptPattern, results.add<ReshapeReshapeOptPattern, RedundantReshapeOptPattern,
FoldConstantReshapeOptPattern>(context); FoldConstantReshapeOptPattern>(context);
} }

mlir/include/mlir/Conversion/AffineToStandard/AffineToStandard.h

Show All 36 Lines
Optional<SmallVector<Value, 8>> expandAffineMap(OpBuilder &builder, Optional<SmallVector<Value, 8>> expandAffineMap(OpBuilder &builder,
Location loc, Location loc,
AffineMap affineMap, AffineMap affineMap,
ValueRange operands); ValueRange operands);
/// Collect a set of patterns to convert from the Affine dialect to the Standard /// Collect a set of patterns to convert from the Affine dialect to the Standard
/// dialect, in particular convert structured affine control flow into CFG /// dialect, in particular convert structured affine control flow into CFG
/// branch-based control flow. /// branch-based control flow.
void populateAffineToStdConversionPatterns(OwningRewritePatternList &patterns); void populateAffineToStdConversionPatterns(RewritePatternSet &patterns);
/// Collect a set of patterns to convert vector-related Affine ops to the Vector /// Collect a set of patterns to convert vector-related Affine ops to the Vector
/// dialect. /// dialect.
void populateAffineToVectorConversionPatterns( void populateAffineToVectorConversionPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
/// Emit code that computes the lower bound of the given affine loop using /// Emit code that computes the lower bound of the given affine loop using
/// standard arithmetic operations. /// standard arithmetic operations.
Value lowerAffineLowerBound(AffineForOp op, OpBuilder &builder); Value lowerAffineLowerBound(AffineForOp op, OpBuilder &builder);
/// Emit code that computes the upper bound of the given affine loop using /// Emit code that computes the upper bound of the given affine loop using
/// standard arithmetic operations. /// standard arithmetic operations.
Value lowerAffineUpperBound(AffineForOp op, OpBuilder &builder); Value lowerAffineUpperBound(AffineForOp op, OpBuilder &builder);
Show All 9 Lines

mlir/include/mlir/Conversion/ArmSVEToLLVM/ArmSVEToLLVM.h

Show All 11 Lines
namespace mlir { namespace mlir {
class LLVMTypeConverter; class LLVMTypeConverter;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
/// Collect a set of patterns to convert from the ArmSVE dialect to LLVM. /// Collect a set of patterns to convert from the ArmSVE dialect to LLVM.
void populateArmSVEToLLVMConversionPatterns(LLVMTypeConverter &converter, void populateArmSVEToLLVMConversionPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_ARMSVETOLLVM_ARMSVETOLLVM_H_ #endif // MLIR_CONVERSION_ARMSVETOLLVM_ARMSVETOLLVM_H_

mlir/include/mlir/Conversion/AsyncToLLVM/AsyncToLLVM.h

Show All 28 Lines
/// ///
/// A "structural" type conversion is one where the underlying ops are /// A "structural" type conversion is one where the underlying ops are
/// completely agnostic to the actual types involved and simply need to update /// completely agnostic to the actual types involved and simply need to update
/// their types. An example of this is async.execute -- the async.execute op and /// their types. An example of this is async.execute -- the async.execute op and
/// the corresponding async.yield ops need to update their types accordingly to /// the corresponding async.yield ops need to update their types accordingly to
/// the TypeConverter, but otherwise don't care what type conversions are /// the TypeConverter, but otherwise don't care what type conversions are
/// happening. /// happening.
void populateAsyncStructuralTypeConversionsAndLegality( void populateAsyncStructuralTypeConversionsAndLegality(
TypeConverter &typeConverter, OwningRewritePatternList &patterns, TypeConverter &typeConverter, RewritePatternSet &patterns,
ConversionTarget &target); ConversionTarget &target);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_ASYNCTOLLVM_ASYNCTOLLVM_H #endif // MLIR_CONVERSION_ASYNCTOLLVM_ASYNCTOLLVM_H

mlir/include/mlir/Conversion/ComplexToLLVM/ComplexToLLVM.h

Show All 12 Lines
namespace mlir { namespace mlir {
class MLIRContext; class MLIRContext;
class ModuleOp; class ModuleOp;
template <typename T> template <typename T>
class OperationPass; class OperationPass;
/// Populate the given list with patterns that convert from Complex to LLVM. /// Populate the given list with patterns that convert from Complex to LLVM.
void populateComplexToLLVMConversionPatterns( void populateComplexToLLVMConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Create a pass to convert Complex operations to the LLVMIR dialect. /// Create a pass to convert Complex operations to the LLVMIR dialect.
std::unique_ptr<OperationPass<ModuleOp>> createConvertComplexToLLVMPass(); std::unique_ptr<OperationPass<ModuleOp>> createConvertComplexToLLVMPass();
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_COMPLEXTOLLVM_COMPLEXTOLLVM_H_ #endif // MLIR_CONVERSION_COMPLEXTOLLVM_COMPLEXTOLLVM_H_

mlir/include/mlir/Conversion/GPUToNVVM/GPUToNVVMPass.h

Show All 23 Lines
class GPUModuleOp; class GPUModuleOp;
} }
/// Configure target to convert from the GPU dialect to NVVM. /// Configure target to convert from the GPU dialect to NVVM.
void configureGpuToNVVMConversionLegality(ConversionTarget &target); void configureGpuToNVVMConversionLegality(ConversionTarget &target);
/// Collect a set of patterns to convert from the GPU dialect to NVVM. /// Collect a set of patterns to convert from the GPU dialect to NVVM.
void populateGpuToNVVMConversionPatterns(LLVMTypeConverter &converter, void populateGpuToNVVMConversionPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Creates a pass that lowers GPU dialect operations to NVVM counterparts. The /// Creates a pass that lowers GPU dialect operations to NVVM counterparts. The
/// index bitwidth used for the lowering of the device side index computations /// index bitwidth used for the lowering of the device side index computations
/// is configurable. /// is configurable.
std::unique_ptr<OperationPass<gpu::GPUModuleOp>> createLowerGpuOpsToNVVMOpsPass( std::unique_ptr<OperationPass<gpu::GPUModuleOp>> createLowerGpuOpsToNVVMOpsPass(
unsigned indexBitwidth = kDeriveIndexBitwidthFromDataLayout); unsigned indexBitwidth = kDeriveIndexBitwidthFromDataLayout);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_GPUTONVVM_GPUTONVVMPASS_H_ #endif // MLIR_CONVERSION_GPUTONVVM_GPUTONVVMPASS_H_

mlir/include/mlir/Conversion/GPUToROCDL/GPUToROCDLPass.h

Show All 20 Lines
class OperationPass; class OperationPass;
namespace gpu { namespace gpu {
class GPUModuleOp; class GPUModuleOp;
} // namespace gpu } // namespace gpu
/// Collect a set of patterns to convert from the GPU dialect to ROCDL. /// Collect a set of patterns to convert from the GPU dialect to ROCDL.
void populateGpuToROCDLConversionPatterns(LLVMTypeConverter &converter, void populateGpuToROCDLConversionPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Configure target to convert from the GPU dialect to ROCDL. /// Configure target to convert from the GPU dialect to ROCDL.
void configureGpuToROCDLConversionLegality(ConversionTarget &target); void configureGpuToROCDLConversionLegality(ConversionTarget &target);
/// Creates a pass that lowers GPU dialect operations to ROCDL counterparts. The /// Creates a pass that lowers GPU dialect operations to ROCDL counterparts. The
/// index bitwidth used for the lowering of the device side index computations /// index bitwidth used for the lowering of the device side index computations
/// is configurable. /// is configurable.
std::unique_ptr<OperationPass<gpu::GPUModuleOp>> std::unique_ptr<OperationPass<gpu::GPUModuleOp>>
createLowerGpuOpsToROCDLOpsPass( createLowerGpuOpsToROCDLOpsPass(
unsigned indexBitwidth = kDeriveIndexBitwidthFromDataLayout); unsigned indexBitwidth = kDeriveIndexBitwidthFromDataLayout);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_GPUTOROCDL_GPUTOROCDLPASS_H_ #endif // MLIR_CONVERSION_GPUTOROCDL_GPUTOROCDLPASS_H_

mlir/include/mlir/Conversion/GPUToSPIRV/GPUToSPIRV.h

Show All 16 Lines
namespace mlir { namespace mlir {
class SPIRVTypeConverter; class SPIRVTypeConverter;
/// Appends to a pattern list additional patterns for translating GPU Ops to /// Appends to a pattern list additional patterns for translating GPU Ops to
/// SPIR-V ops. For a gpu.func to be converted, it should have a /// SPIR-V ops. For a gpu.func to be converted, it should have a
/// spv.entry_point_abi attribute. /// spv.entry_point_abi attribute.
void populateGPUToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateGPUToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_GPUTOSPIRV_GPUTOSPIRV_H #endif // MLIR_CONVERSION_GPUTOSPIRV_GPUTOSPIRV_H

mlir/include/mlir/Conversion/LinalgToLLVM/LinalgToLLVM.h

//===- LinalgToLLVM.h - Utils to convert from the linalg dialect ----------===// //===- LinalgToLLVM.h - Utils to convert from the linalg dialect ----------===//
// //
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information. // See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
#ifndef MLIR_CONVERSION_LINALGTOLLVM_LINALGTOLLVM_H_ #ifndef MLIR_CONVERSION_LINALGTOLLVM_LINALGTOLLVM_H_
#define MLIR_CONVERSION_LINALGTOLLVM_LINALGTOLLVM_H_ #define MLIR_CONVERSION_LINALGTOLLVM_LINALGTOLLVM_H_
#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h" #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
#include "mlir/Transforms/DialectConversion.h" #include "mlir/Transforms/DialectConversion.h"
namespace mlir { namespace mlir {
class MLIRContext; class MLIRContext;
class ModuleOp; class ModuleOp;
template <typename T> class OperationPass; template <typename T>
class OperationPass;
/// Populate the given list with patterns that convert from Linalg to LLVM. /// Populate the given list with patterns that convert from Linalg to LLVM.
void populateLinalgToLLVMConversionPatterns(LLVMTypeConverter &converter, void populateLinalgToLLVMConversionPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Create a pass to convert Linalg operations to the LLVMIR dialect. /// Create a pass to convert Linalg operations to the LLVMIR dialect.
std::unique_ptr<OperationPass<ModuleOp>> createConvertLinalgToLLVMPass(); std::unique_ptr<OperationPass<ModuleOp>> createConvertLinalgToLLVMPass();
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_LINALGTOLLVM_LINALGTOLLVM_H_ #endif // MLIR_CONVERSION_LINALGTOLLVM_LINALGTOLLVM_H_

mlir/include/mlir/Conversion/LinalgToSPIRV/LinalgToSPIRV.h

Show All 16 Lines
class MLIRContext; class MLIRContext;
class SPIRVTypeConverter; class SPIRVTypeConverter;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
/// Appends to a pattern list additional patterns for translating Linalg ops to /// Appends to a pattern list additional patterns for translating Linalg ops to
/// SPIR-V ops. /// SPIR-V ops.
void populateLinalgToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateLinalgToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_LINALGTOSPIRV_LINALGTOSPIRV_H #endif // MLIR_CONVERSION_LINALGTOSPIRV_LINALGTOSPIRV_H

mlir/include/mlir/Conversion/LinalgToStandard/LinalgToStandard.h

Show First 20 Lines • Show All 63 Lines • ▼ Show 20 Linesclass IndexedGenericOpToLibraryCallRewrite
: public OpRewritePattern<IndexedGenericOp> { : public OpRewritePattern<IndexedGenericOp> {
public: public:
using OpRewritePattern<IndexedGenericOp>::OpRewritePattern; using OpRewritePattern<IndexedGenericOp>::OpRewritePattern;
LogicalResult matchAndRewrite(IndexedGenericOp op, LogicalResult matchAndRewrite(IndexedGenericOp op,
PatternRewriter &rewriter) const override; PatternRewriter &rewriter) const override;
}; };
/// Populate the given list with patterns that convert from Linalg to Standard. /// Populate the given list with patterns that convert from Linalg to Standard.
void populateLinalgToStandardConversionPatterns( void populateLinalgToStandardConversionPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
} // namespace linalg } // namespace linalg
/// Create a pass to convert Linalg operations to the Standard dialect. /// Create a pass to convert Linalg operations to the Standard dialect.
std::unique_ptr<OperationPass<ModuleOp>> createConvertLinalgToStandardPass(); std::unique_ptr<OperationPass<ModuleOp>> createConvertLinalgToStandardPass();
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_LINALGTOSTANDARD_LINALGTOSTANDARD_H_ #endif // MLIR_CONVERSION_LINALGTOSTANDARD_LINALGTOSTANDARD_H_

mlir/include/mlir/Conversion/OpenMPToLLVM/ConvertOpenMPToLLVM.h

Show All 15 Lines
class ModuleOp; class ModuleOp;
template <typename T> template <typename T>
class OperationPass; class OperationPass;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
/// Populate the given list with patterns that convert from OpenMP to LLVM. /// Populate the given list with patterns that convert from OpenMP to LLVM.
void populateOpenMPToLLVMConversionPatterns(LLVMTypeConverter &converter, void populateOpenMPToLLVMConversionPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Create a pass to convert OpenMP operations to the LLVMIR dialect. /// Create a pass to convert OpenMP operations to the LLVMIR dialect.
std::unique_ptr<OperationPass<ModuleOp>> createConvertOpenMPToLLVMPass(); std::unique_ptr<OperationPass<ModuleOp>> createConvertOpenMPToLLVMPass();
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_OPENMPTOLLVM_OPENMPTOLLVM_H_ #endif // MLIR_CONVERSION_OPENMPTOLLVM_OPENMPTOLLVM_H_

mlir/include/mlir/Conversion/SCFToGPU/SCFToGPU.h

Show All 37 Lines
// detection followed by an affine.parallel -> scf.parallel -> gpu.launch // detection followed by an affine.parallel -> scf.parallel -> gpu.launch
// conversion // conversion
LogicalResult convertAffineLoopNestToGPULaunch(AffineForOp forOp, LogicalResult convertAffineLoopNestToGPULaunch(AffineForOp forOp,
unsigned numBlockDims, unsigned numBlockDims,
unsigned numThreadDims); unsigned numThreadDims);
/// Adds the conversion pattern from `scf.parallel` to `gpu.launch` to the /// Adds the conversion pattern from `scf.parallel` to `gpu.launch` to the
/// provided pattern list. /// provided pattern list.
void populateParallelLoopToGPUPatterns(OwningRewritePatternList &patterns); void populateParallelLoopToGPUPatterns(RewritePatternSet &patterns);
/// Configures the rewrite target such that only `scf.parallel` operations that /// Configures the rewrite target such that only `scf.parallel` operations that
/// are not rewritten by the provided patterns are legal. /// are not rewritten by the provided patterns are legal.
void configureParallelLoopToGPULegality(ConversionTarget &target); void configureParallelLoopToGPULegality(ConversionTarget &target);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_SCFTOGPU_SCFTOGPU_H_ #endif // MLIR_CONVERSION_SCFTOGPU_SCFTOGPU_H_

mlir/include/mlir/Conversion/SCFToSPIRV/SCFToSPIRV.h

Show All 31 Lines
private: private:
std::unique_ptr<ScfToSPIRVContextImpl> impl; std::unique_ptr<ScfToSPIRVContextImpl> impl;
}; };
/// Collects a set of patterns to lower from scf.for, scf.if, and /// Collects a set of patterns to lower from scf.for, scf.if, and
/// loop.terminator to CFG operations within the SPIR-V dialect. /// loop.terminator to CFG operations within the SPIR-V dialect.
void populateSCFToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateSCFToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
ScfToSPIRVContext &scfToSPIRVContext, ScfToSPIRVContext &scfToSPIRVContext,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_SCFTOSPIRV_SCFTOSPIRV_H_ #endif // MLIR_CONVERSION_SCFTOSPIRV_SCFTOSPIRV_H_

mlir/include/mlir/Conversion/SCFToStandard/SCFToStandard.h

Show All 16 Lines
class Pass; class Pass;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
/// Collect a set of patterns to lower from scf.for, scf.if, and /// Collect a set of patterns to lower from scf.for, scf.if, and
/// loop.terminator to CFG operations within the Standard dialect, in particular /// loop.terminator to CFG operations within the Standard dialect, in particular
/// convert structured control flow into CFG branch-based control flow. /// convert structured control flow into CFG branch-based control flow.
void populateLoopToStdConversionPatterns(OwningRewritePatternList &patterns); void populateLoopToStdConversionPatterns(RewritePatternSet &patterns);
/// Creates a pass to convert scf.for, scf.if and loop.terminator ops to CFG. /// Creates a pass to convert scf.for, scf.if and loop.terminator ops to CFG.
std::unique_ptr<Pass> createLowerToCFGPass(); std::unique_ptr<Pass> createLowerToCFGPass();
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_SCFTOSTANDARD_SCFTOSTANDARD_H_ #endif // MLIR_CONVERSION_SCFTOSTANDARD_SCFTOSTANDARD_H_

mlir/include/mlir/Conversion/SPIRVToLLVM/SPIRVToLLVM.h

Show All 35 Lines
/// both exist. /// both exist.
void encodeBindAttribute(ModuleOp module); void encodeBindAttribute(ModuleOp module);
/// Populates type conversions with additional SPIR-V types. /// Populates type conversions with additional SPIR-V types.
void populateSPIRVToLLVMTypeConversion(LLVMTypeConverter &typeConverter); void populateSPIRVToLLVMTypeConversion(LLVMTypeConverter &typeConverter);
/// Populates the given list with patterns that convert from SPIR-V to LLVM. /// Populates the given list with patterns that convert from SPIR-V to LLVM.
void populateSPIRVToLLVMConversionPatterns(LLVMTypeConverter &typeConverter, void populateSPIRVToLLVMConversionPatterns(LLVMTypeConverter &typeConverter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Populates the given list with patterns for function conversion from SPIR-V /// Populates the given list with patterns for function conversion from SPIR-V
/// to LLVM. /// to LLVM.
void populateSPIRVToLLVMFunctionConversionPatterns( void populateSPIRVToLLVMFunctionConversionPatterns(
LLVMTypeConverter &typeConverter, OwningRewritePatternList &patterns); LLVMTypeConverter &typeConverter, RewritePatternSet &patterns);
/// Populates the given patterns for module conversion from SPIR-V to LLVM. /// Populates the given patterns for module conversion from SPIR-V to LLVM.
void populateSPIRVToLLVMModuleConversionPatterns( void populateSPIRVToLLVMModuleConversionPatterns(
LLVMTypeConverter &typeConverter, OwningRewritePatternList &patterns); LLVMTypeConverter &typeConverter, RewritePatternSet &patterns);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_SPIRVTOLLVM_SPIRVTOLLVM_H #endif // MLIR_CONVERSION_SPIRVTOLLVM_SPIRVTOLLVM_H

mlir/include/mlir/Conversion/ShapeToStandard/ShapeToStandard.h

Show All 14 Lines
class FuncOp; class FuncOp;
class ModuleOp; class ModuleOp;
template <typename T> template <typename T>
class OperationPass; class OperationPass;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
void populateShapeToStandardConversionPatterns( void populateShapeToStandardConversionPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
std::unique_ptr<OperationPass<ModuleOp>> createConvertShapeToStandardPass(); std::unique_ptr<OperationPass<ModuleOp>> createConvertShapeToStandardPass();
void populateConvertShapeConstraintsConversionPatterns( void populateConvertShapeConstraintsConversionPatterns(
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
std::unique_ptr<OperationPass<FuncOp>> createConvertShapeConstraintsPass(); std::unique_ptr<OperationPass<FuncOp>> createConvertShapeConstraintsPass();
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_SHAPETOSTANDARD_SHAPETOSTANDARD_H_ #endif // MLIR_CONVERSION_SHAPETOSTANDARD_SHAPETOSTANDARD_H_

mlir/include/mlir/Conversion/StandardToLLVM/ConvertStandardToLLVMPass.h

Show First 20 Lines • Show All 43 Lines • ▼ Show 20 Lines static const LowerToLLVMOptions &getDefaultOptions() {
static LowerToLLVMOptions options; static LowerToLLVMOptions options;
return options; return options;
} }
}; };
/// Collect a set of patterns to convert memory-related operations from the /// Collect a set of patterns to convert memory-related operations from the
/// Standard dialect to the LLVM dialect, excluding non-memory-related /// Standard dialect to the LLVM dialect, excluding non-memory-related
/// operations and FuncOp. /// operations and FuncOp.
void populateStdToLLVMMemoryConversionPatterns( void populateStdToLLVMMemoryConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Collect a set of patterns to convert from the Standard dialect to the LLVM /// Collect a set of patterns to convert from the Standard dialect to the LLVM
/// dialect, excluding the memory-related operations. /// dialect, excluding the memory-related operations.
void populateStdToLLVMNonMemoryConversionPatterns( void populateStdToLLVMNonMemoryConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Collect the default pattern to convert a FuncOp to the LLVM dialect. If /// Collect the default pattern to convert a FuncOp to the LLVM dialect. If
/// `emitCWrappers` is set, the pattern will also produce functions /// `emitCWrappers` is set, the pattern will also produce functions
/// that pass memref descriptors by pointer-to-structure in addition to the /// that pass memref descriptors by pointer-to-structure in addition to the
/// default unpacked form. /// default unpacked form.
void populateStdToLLVMFuncOpConversionPattern( void populateStdToLLVMFuncOpConversionPattern(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Collect the patterns to convert from the Standard dialect to LLVM. The /// Collect the patterns to convert from the Standard dialect to LLVM. The
/// conversion patterns capture the LLVMTypeConverter and the LowerToLLVMOptions /// conversion patterns capture the LLVMTypeConverter and the LowerToLLVMOptions
/// by reference meaning the references have to remain alive during the entire /// by reference meaning the references have to remain alive during the entire
/// pattern lifetime. /// pattern lifetime.
void populateStdToLLVMConversionPatterns(LLVMTypeConverter &converter, void populateStdToLLVMConversionPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Creates a pass to convert the Standard dialect into the LLVMIR dialect. /// Creates a pass to convert the Standard dialect into the LLVMIR dialect.
/// stdlib malloc/free is used by default for allocating memrefs allocated with /// stdlib malloc/free is used by default for allocating memrefs allocated with
/// memref.alloc, while LLVM's alloca is used for those allocated with /// memref.alloc, while LLVM's alloca is used for those allocated with
/// memref.alloca. /// memref.alloca.
std::unique_ptr<OperationPass<ModuleOp>> std::unique_ptr<OperationPass<ModuleOp>>
createLowerToLLVMPass(const LowerToLLVMOptions &options = createLowerToLLVMPass(const LowerToLLVMOptions &options =
LowerToLLVMOptions::getDefaultOptions()); LowerToLLVMOptions::getDefaultOptions());
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_STANDARDTOLLVM_CONVERTSTANDARDTOLLVMPASS_H_ #endif // MLIR_CONVERSION_STANDARDTOLLVM_CONVERTSTANDARDTOLLVMPASS_H_

mlir/include/mlir/Conversion/StandardToSPIRV/StandardToSPIRV.h

Show All 16 Lines
namespace mlir { namespace mlir {
class SPIRVTypeConverter; class SPIRVTypeConverter;
/// Appends to a pattern list additional patterns for translating standard ops /// Appends to a pattern list additional patterns for translating standard ops
/// to SPIR-V ops. Also adds the patterns to legalize ops not directly /// to SPIR-V ops. Also adds the patterns to legalize ops not directly
/// translated to SPIR-V dialect. /// translated to SPIR-V dialect.
void populateStandardToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateStandardToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Appends to a pattern list additional patterns for translating tensor ops /// Appends to a pattern list additional patterns for translating tensor ops
/// to SPIR-V ops. /// to SPIR-V ops.
/// ///
/// Note: Normally tensors will be stored in buffers before converting to /// Note: Normally tensors will be stored in buffers before converting to
/// SPIR-V, given that is how a large amount of data is sent to the GPU. /// SPIR-V, given that is how a large amount of data is sent to the GPU.
/// However, SPIR-V supports converting from tensors directly too. This is /// However, SPIR-V supports converting from tensors directly too. This is
/// for the cases where the tensor just contains a small amount of elements /// for the cases where the tensor just contains a small amount of elements
/// and it makes sense to directly inline them as a small data array in the /// and it makes sense to directly inline them as a small data array in the
/// shader. To handle this, internally the conversion might create new local /// shader. To handle this, internally the conversion might create new local
/// variables. SPIR-V consumers in GPU drivers may or may not optimize that /// variables. SPIR-V consumers in GPU drivers may or may not optimize that
/// away. So this has implications over register pressure. Therefore, a /// away. So this has implications over register pressure. Therefore, a
/// threshold is used to control when the patterns should kick in. /// threshold is used to control when the patterns should kick in.
void populateTensorToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateTensorToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
int64_t byteCountThreshold, int64_t byteCountThreshold,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Appends to a pattern list patterns to legalize ops that are not directly /// Appends to a pattern list patterns to legalize ops that are not directly
/// lowered to SPIR-V. /// lowered to SPIR-V.
void populateStdLegalizationPatternsForSPIRVLowering( void populateStdLegalizationPatternsForSPIRVLowering(
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_STANDARDTOSPIRV_STANDARDTOSPIRV_H #endif // MLIR_CONVERSION_STANDARDTOSPIRV_STANDARDTOSPIRV_H

mlir/include/mlir/Conversion/TosaToLinalg/TosaToLinalg.h

Show All 22 Lines
/// Populates passes to convert from TOSA to Linalg on buffers. At the end of /// Populates passes to convert from TOSA to Linalg on buffers. At the end of
/// the pass, the function will only contain linalg ops or standard ops if the /// the pass, the function will only contain linalg ops or standard ops if the
/// pipeline succeeds. /// pipeline succeeds.
void addTosaToLinalgOnTensorsPasses(OpPassManager &pm); void addTosaToLinalgOnTensorsPasses(OpPassManager &pm);
/// Populates conversion passes from TOSA dialect to Linalg dialect. /// Populates conversion passes from TOSA dialect to Linalg dialect.
void populateTosaToLinalgOnTensorsConversionPatterns( void populateTosaToLinalgOnTensorsConversionPatterns(
OwningRewritePatternList *patterns); RewritePatternSet *patterns);
} // namespace tosa } // namespace tosa
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_TOSATOLINALG_TOSATOLINALG_H #endif // MLIR_CONVERSION_TOSATOLINALG_TOSATOLINALG_H

mlir/include/mlir/Conversion/TosaToSCF/TosaToSCF.h

Show All 14 Lines
#include "mlir/Pass/Pass.h" #include "mlir/Pass/Pass.h"
namespace mlir { namespace mlir {
namespace tosa { namespace tosa {
std::unique_ptr<Pass> createTosaToSCF(); std::unique_ptr<Pass> createTosaToSCF();
void populateTosaToSCFConversionPatterns(OwningRewritePatternList *patterns); void populateTosaToSCFConversionPatterns(RewritePatternSet *patterns);
/// Populates passes to convert from TOSA to SCF. /// Populates passes to convert from TOSA to SCF.
void addTosaToSCFPasses(OpPassManager &pm); void addTosaToSCFPasses(OpPassManager &pm);
} // namespace tosa } // namespace tosa
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_TOSATOSCF_TOSATOSCF_H #endif // MLIR_CONVERSION_TOSATOSCF_TOSATOSCF_H

mlir/include/mlir/Conversion/TosaToStandard/TosaToStandard.h

Show All 14 Lines
#include "mlir/Pass/Pass.h" #include "mlir/Pass/Pass.h"
namespace mlir { namespace mlir {
namespace tosa { namespace tosa {
std::unique_ptr<Pass> createTosaToStandard(); std::unique_ptr<Pass> createTosaToStandard();
void populateTosaToStandardConversionPatterns( void populateTosaToStandardConversionPatterns(RewritePatternSet *patterns);
OwningRewritePatternList *patterns);
void populateTosaRescaleToStandardConversionPatterns( void populateTosaRescaleToStandardConversionPatterns(
OwningRewritePatternList *patterns); RewritePatternSet *patterns);
/// Populates passes to convert from TOSA to Standard. /// Populates passes to convert from TOSA to Standard.
void addTosaToStandardPasses(OpPassManager &pm); void addTosaToStandardPasses(OpPassManager &pm);
} // namespace tosa } // namespace tosa
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_TOSATOSTANDARD_TOSATOSTANDARD_H #endif // MLIR_CONVERSION_TOSATOSTANDARD_TOSATOSTANDARD_H

mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h

Show First 20 Lines • Show All 56 Lines • ▼ Show 20 Linesstruct LowerVectorToLLVMOptions {
bool enableArmSVE; bool enableArmSVE;
bool enableAMX; bool enableAMX;
bool enableAVX512; bool enableAVX512;
}; };
/// Collect a set of patterns to convert from Vector contractions to LLVM Matrix /// Collect a set of patterns to convert from Vector contractions to LLVM Matrix
/// Intrinsics. To lower to assembly, the LLVM flag -lower-matrix-intrinsics /// Intrinsics. To lower to assembly, the LLVM flag -lower-matrix-intrinsics
/// will be needed when invoking LLVM. /// will be needed when invoking LLVM.
void populateVectorToLLVMMatrixConversionPatterns( void populateVectorToLLVMMatrixConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Collect a set of patterns to convert from the Vector dialect to LLVM. /// Collect a set of patterns to convert from the Vector dialect to LLVM.
void populateVectorToLLVMConversionPatterns( void populateVectorToLLVMConversionPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns, LLVMTypeConverter &converter, RewritePatternSet &patterns,
bool reassociateFPReductions = false, bool enableIndexOptimizations = true); bool reassociateFPReductions = false, bool enableIndexOptimizations = true);
/// Create a pass to convert vector operations to the LLVMIR dialect. /// Create a pass to convert vector operations to the LLVMIR dialect.
std::unique_ptr<OperationPass<ModuleOp>> createConvertVectorToLLVMPass( std::unique_ptr<OperationPass<ModuleOp>> createConvertVectorToLLVMPass(
const LowerVectorToLLVMOptions &options = LowerVectorToLLVMOptions()); const LowerVectorToLLVMOptions &options = LowerVectorToLLVMOptions());
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_VECTORTOLLVM_CONVERTVECTORTOLLVM_H_ #endif // MLIR_CONVERSION_VECTORTOLLVM_CONVERTVECTORTOLLVM_H_

mlir/include/mlir/Conversion/VectorToROCDL/VectorToROCDL.h

Show All 13 Lines
class LLVMTypeConverter; class LLVMTypeConverter;
class ModuleOp; class ModuleOp;
template <typename OpT> template <typename OpT>
class OperationPass; class OperationPass;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
/// Collect a set of patterns to convert from the GPU dialect to ROCDL. /// Collect a set of patterns to convert from the GPU dialect to ROCDL.
void populateVectorToROCDLConversionPatterns( void populateVectorToROCDLConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Create a pass to convert vector operations to the ROCDL dialect. /// Create a pass to convert vector operations to the ROCDL dialect.
std::unique_ptr<OperationPass<ModuleOp>> createConvertVectorToROCDLPass(); std::unique_ptr<OperationPass<ModuleOp>> createConvertVectorToROCDLPass();
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_VECTORTOROCDL_VECTORTOROCDL_H_ #endif // MLIR_CONVERSION_VECTORTOROCDL_VECTORTOROCDL_H_

mlir/include/mlir/Conversion/VectorToSCF/VectorToSCF.h

Show First 20 Lines • Show All 157 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(Operation *op,
PatternRewriter &rewriter) const override; PatternRewriter &rewriter) const override;
/// See description of `VectorTransferToSCFOptions`. /// See description of `VectorTransferToSCFOptions`.
VectorTransferToSCFOptions options; VectorTransferToSCFOptions options;
}; };
/// Collect a set of patterns to convert from the Vector dialect to SCF + std. /// Collect a set of patterns to convert from the Vector dialect to SCF + std.
void populateVectorToSCFConversionPatterns( void populateVectorToSCFConversionPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns,
const VectorTransferToSCFOptions &options = VectorTransferToSCFOptions()); const VectorTransferToSCFOptions &options = VectorTransferToSCFOptions());
/// Create a pass to convert a subset of vector ops to SCF. /// Create a pass to convert a subset of vector ops to SCF.
std::unique_ptr<Pass> createConvertVectorToSCFPass( std::unique_ptr<Pass> createConvertVectorToSCFPass(
const VectorTransferToSCFOptions &options = VectorTransferToSCFOptions()); const VectorTransferToSCFOptions &options = VectorTransferToSCFOptions());
} // namespace mlir } // namespace mlir
#endif // MLIR_CONVERSION_VECTORTOSCF_VECTORTOSCF_H_ #endif // MLIR_CONVERSION_VECTORTOSCF_VECTORTOSCF_H_

mlir/include/mlir/Conversion/VectorToSPIRV/VectorToSPIRV.h

Show All 15 Lines
#include "mlir/Transforms/DialectConversion.h" #include "mlir/Transforms/DialectConversion.h"
namespace mlir { namespace mlir {
class SPIRVTypeConverter; class SPIRVTypeConverter;
/// Appends to a pattern list additional patterns for translating Vector Ops to /// Appends to a pattern list additional patterns for translating Vector Ops to
/// SPIR-V ops. /// SPIR-V ops.
void populateVectorToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateVectorToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
} // namespace mlir } // namespace mlir
#endif // MLIR_INCLUDE_MLIR_CONVERSION_VECTORTOSPIRV_VECTORTOSPIRV_H #endif // MLIR_INCLUDE_MLIR_CONVERSION_VECTORTOSPIRV_VECTORTOSPIRV_H

mlir/include/mlir/Dialect/AMX/Transforms.h

Show All 12 Lines
class LLVMConversionTarget; class LLVMConversionTarget;
class LLVMTypeConverter; class LLVMTypeConverter;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
/// Collect a set of patterns to lower AMX ops to ops that map to LLVM /// Collect a set of patterns to lower AMX ops to ops that map to LLVM
/// intrinsics. /// intrinsics.
void populateAMXLegalizeForLLVMExportPatterns( void populateAMXLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Configure the target to support lowering AMX ops to ops that map to LLVM /// Configure the target to support lowering AMX ops to ops that map to LLVM
/// intrinsics. /// intrinsics.
void configureAMXLegalizeForExportTarget(LLVMConversionTarget &target); void configureAMXLegalizeForExportTarget(LLVMConversionTarget &target);
} // namespace mlir } // namespace mlir
#endif // MLIR_DIALECT_AMX_TRANSFORMS_H #endif // MLIR_DIALECT_AMX_TRANSFORMS_H

mlir/include/mlir/Dialect/AVX512/Transforms.h

Show All 12 Lines
class LLVMConversionTarget; class LLVMConversionTarget;
class LLVMTypeConverter; class LLVMTypeConverter;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
/// Collect a set of patterns to lower AVX512 ops to ops that map to LLVM /// Collect a set of patterns to lower AVX512 ops to ops that map to LLVM
/// intrinsics. /// intrinsics.
void populateAVX512LegalizeForLLVMExportPatterns( void populateAVX512LegalizeForLLVMExportPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Configure the target to support lowering AVX512 ops to ops that map to LLVM /// Configure the target to support lowering AVX512 ops to ops that map to LLVM
/// intrinsics. /// intrinsics.
void configureAVX512LegalizeForExportTarget(LLVMConversionTarget &target); void configureAVX512LegalizeForExportTarget(LLVMConversionTarget &target);
} // namespace mlir } // namespace mlir
#endif // MLIR_DIALECT_AVX512_TRANSFORMS_H #endif // MLIR_DIALECT_AVX512_TRANSFORMS_H

mlir/include/mlir/Dialect/GPU/Passes.h

Show All 25 Lines
/// Replaces `gpu.launch` with `gpu.launch_func` by moving the region into /// Replaces `gpu.launch` with `gpu.launch_func` by moving the region into
/// a separate kernel function. /// a separate kernel function.
std::unique_ptr<OperationPass<ModuleOp>> createGpuKernelOutliningPass(); std::unique_ptr<OperationPass<ModuleOp>> createGpuKernelOutliningPass();
/// Rewrites a function region so that GPU ops execute asynchronously. /// Rewrites a function region so that GPU ops execute asynchronously.
std::unique_ptr<OperationPass<FuncOp>> createGpuAsyncRegionPass(); std::unique_ptr<OperationPass<FuncOp>> createGpuAsyncRegionPass();
/// Collect a set of patterns to rewrite all-reduce ops within the GPU dialect. /// Collect a set of patterns to rewrite all-reduce ops within the GPU dialect.
void populateGpuAllReducePatterns(OwningRewritePatternList &patterns); void populateGpuAllReducePatterns(RewritePatternSet &patterns);
/// Collect all patterns to rewrite ops within the GPU dialect. /// Collect all patterns to rewrite ops within the GPU dialect.
inline void populateGpuRewritePatterns(OwningRewritePatternList &patterns) { inline void populateGpuRewritePatterns(RewritePatternSet &patterns) {
populateGpuAllReducePatterns(patterns); populateGpuAllReducePatterns(patterns);
} }
namespace gpu { namespace gpu {
/// Returns the default annotation name for GPU binary blobs. /// Returns the default annotation name for GPU binary blobs.
std::string getDefaultGpuBinaryAnnotation(); std::string getDefaultGpuBinaryAnnotation();
/// Base pass class to serialize kernel functions through LLVM into /// Base pass class to serialize kernel functions through LLVM into
▲ Show 20 Lines • Show All 56 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Linalg/Passes.h

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Lines
std::unique_ptr<OperationPass<FuncOp>> createConvertLinalgToAffineLoopsPass(); std::unique_ptr<OperationPass<FuncOp>> createConvertLinalgToAffineLoopsPass();
/// Create a pass to convert Linalg operations which work on tensors to use /// Create a pass to convert Linalg operations which work on tensors to use
/// buffers instead. /// buffers instead.
std::unique_ptr<OperationPass<FuncOp>> createLinalgBufferizePass(); std::unique_ptr<OperationPass<FuncOp>> createLinalgBufferizePass();
/// Populate patterns that convert `ElementwiseMappable` ops to linalg /// Populate patterns that convert `ElementwiseMappable` ops to linalg
/// parallel loops. /// parallel loops.
void populateElementwiseToLinalgConversionPatterns( void populateElementwiseToLinalgConversionPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
/// Create a pass to conver named Linalg operations to Linalg generic /// Create a pass to conver named Linalg operations to Linalg generic
/// operations. /// operations.
std::unique_ptr<OperationPass<FuncOp>> createLinalgGeneralizationPass(); std::unique_ptr<OperationPass<FuncOp>> createLinalgGeneralizationPass();
/// Create a pass to convert Linalg operations to equivalent operations that /// Create a pass to convert Linalg operations to equivalent operations that
/// work on primitive types, if possible. /// work on primitive types, if possible.
std::unique_ptr<Pass> createLinalgDetensorizePass(); std::unique_ptr<Pass> createLinalgDetensorizePass();
/// Patterns to fold an expanding (collapsing) tensor_reshape operation with its /// Patterns to fold an expanding (collapsing) tensor_reshape operation with its
/// producer (consumer) generic operation by expanding the dimensionality of the /// producer (consumer) generic operation by expanding the dimensionality of the
/// loop in the generic op. /// loop in the generic op.
void populateFoldReshapeOpsByExpansionPatterns( void populateFoldReshapeOpsByExpansionPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
/// Patterns to fold a collapsing (expanding) tensor_reshape operation with its /// Patterns to fold a collapsing (expanding) tensor_reshape operation with its
/// producer (consumer) generic/indexed_generic operation by linearizing the /// producer (consumer) generic/indexed_generic operation by linearizing the
/// indexing map used to access the source (target) of the reshape operation in /// indexing map used to access the source (target) of the reshape operation in
/// the generic/indexed_generic operation. /// the generic/indexed_generic operation.
void populateFoldReshapeOpsByLinearizationPatterns( void populateFoldReshapeOpsByLinearizationPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
/// Patterns to fold a collapsing (expanding) tensor_reshape operation with its /// Patterns to fold a collapsing (expanding) tensor_reshape operation with its
/// producer (consumer) generic/indexed_generic operation by linearizing the /// producer (consumer) generic/indexed_generic operation by linearizing the
/// indexing map used to access the source (target) of the reshape operation in /// indexing map used to access the source (target) of the reshape operation in
/// the generic/indexed_generic operation. The patterns are applied only when /// the generic/indexed_generic operation. The patterns are applied only when
/// the tensor reshape involved is collapsing (introducing) unit-extent /// the tensor reshape involved is collapsing (introducing) unit-extent
/// dimensions. /// dimensions.
void populateFoldUnitDimsReshapeOpsByLinearizationPatterns( void populateFoldUnitDimsReshapeOpsByLinearizationPatterns(
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Patterns for fusing linalg operation on tensors. /// Patterns for fusing linalg operation on tensors.
void populateLinalgTensorOpsFusionPatterns(OwningRewritePatternList &patterns); void populateLinalgTensorOpsFusionPatterns(RewritePatternSet &patterns);
/// Patterns to fold unit-extent dimensions in operands/results of linalg ops on /// Patterns to fold unit-extent dimensions in operands/results of linalg ops on
/// tensors. /// tensors.
void populateLinalgFoldUnitExtentDimsPatterns( void populateLinalgFoldUnitExtentDimsPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Registration // Registration
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Generate the code for registering passes. /// Generate the code for registering passes.
#define GEN_PASS_REGISTRATION #define GEN_PASS_REGISTRATION
#include "mlir/Dialect/Linalg/Passes.h.inc" #include "mlir/Dialect/Linalg/Passes.h.inc"
} // namespace mlir } // namespace mlir
#endif // MLIR_DIALECT_LINALG_PASSES_H_ #endif // MLIR_DIALECT_LINALG_PASSES_H_

mlir/include/mlir/Dialect/Linalg/Transforms/CodegenStrategy.h

Show All 18 Lines
namespace linalg { namespace linalg {
/// Abstract Transformation class applied in a sequence that also handles state /// Abstract Transformation class applied in a sequence that also handles state
/// through markers. /// through markers.
struct Transformation { struct Transformation {
explicit Transformation(linalg::LinalgTransformationFilter::FilterFunction f) explicit Transformation(linalg::LinalgTransformationFilter::FilterFunction f)
: filter(f) {} : filter(f) {}
virtual ~Transformation() = default; virtual ~Transformation() = default;
virtual OwningRewritePatternList virtual RewritePatternSet
buildRewritePatterns(MLIRContext *context, buildRewritePatterns(MLIRContext *context,
linalg::LinalgTransformationFilter m) = 0; linalg::LinalgTransformationFilter m) = 0;
linalg::LinalgTransformationFilter::FilterFunction filter = nullptr; linalg::LinalgTransformationFilter::FilterFunction filter = nullptr;
}; };
/// SFINAE: Enqueue helper for ConcreteOpType that have a `getOperationName`. /// SFINAE: Enqueue helper for ConcreteOpType that have a `getOperationName`.
template <template <typename> class PatternType, typename ConcreteOpType, template <template <typename> class PatternType, typename ConcreteOpType,
typename OptionsType, typename OptionsType,
typename = std::enable_if_t<std::is_member_function_pointer< typename = std::enable_if_t<std::is_member_function_pointer<
decltype(&ConcreteOpType::getOperationName)>::value>> decltype(&ConcreteOpType::getOperationName)>::value>>
void sfinae_enqueue(OwningRewritePatternList &patternList, OptionsType options, void sfinae_enqueue(RewritePatternSet &patternList, OptionsType options,
StringRef opName, linalg::LinalgTransformationFilter m) { StringRef opName, linalg::LinalgTransformationFilter m) {
assert(opName == ConcreteOpType::getOperationName() && assert(opName == ConcreteOpType::getOperationName() &&
"explicit name must match ConcreteOpType::getOperationName"); "explicit name must match ConcreteOpType::getOperationName");
patternList.insert<PatternType<ConcreteOpType>>(patternList.getContext(), patternList.add<PatternType<ConcreteOpType>>(patternList.getContext(),
options, m); options, m);
} }
/// SFINAE: Enqueue helper for OpType that do not have a `getOperationName` /// SFINAE: Enqueue helper for OpType that do not have a `getOperationName`
/// (e.g. LinalgOp, other interfaces, Operation*). /// (e.g. LinalgOp, other interfaces, Operation*).
template <template <typename> class PatternType, typename OpType, template <template <typename> class PatternType, typename OpType,
typename OptionsType> typename OptionsType>
void sfinae_enqueue(OwningRewritePatternList &patternList, OptionsType options, void sfinae_enqueue(RewritePatternSet &patternList, OptionsType options,
StringRef opName, linalg::LinalgTransformationFilter m) { StringRef opName, linalg::LinalgTransformationFilter m) {
assert(!opName.empty() && "opName must not be empty"); assert(!opName.empty() && "opName must not be empty");
patternList.insert<PatternType<OpType>>(opName, patternList.getContext(), patternList.add<PatternType<OpType>>(opName, patternList.getContext(),
options, m); options, m);
} }
template <typename PatternType, typename OpType, typename OptionsType> template <typename PatternType, typename OpType, typename OptionsType>
void enqueue(OwningRewritePatternList &patternList, OptionsType options, void enqueue(RewritePatternSet &patternList, OptionsType options,
StringRef opName, linalg::LinalgTransformationFilter m) { StringRef opName, linalg::LinalgTransformationFilter m) {
if (!opName.empty()) if (!opName.empty())
patternList.insert<PatternType>(opName, patternList.getContext(), options, patternList.add<PatternType>(opName, patternList.getContext(), options, m);
m);
else else
patternList.insert<PatternType>(m.addOpFilter<OpType>(), options); patternList.add<PatternType>(m.addOpFilter<OpType>(), options);
} }
/// Promotion transformation enqueues a particular stage-1 pattern for /// Promotion transformation enqueues a particular stage-1 pattern for
/// `Tile<LinalgOpType>`with the appropriate `options`. /// `Tile<LinalgOpType>`with the appropriate `options`.
template <typename LinalgOpType> template <typename LinalgOpType>
struct Tile : public Transformation { struct Tile : public Transformation {
explicit Tile(linalg::LinalgTilingOptions options, explicit Tile(linalg::LinalgTilingOptions options,
linalg::LinalgTransformationFilter::FilterFunction f = nullptr) linalg::LinalgTransformationFilter::FilterFunction f = nullptr)
: Transformation(f), opName(LinalgOpType::getOperationName()), : Transformation(f), opName(LinalgOpType::getOperationName()),
options(options) {} options(options) {}
Tile(StringRef name, linalg::LinalgTilingOptions options, Tile(StringRef name, linalg::LinalgTilingOptions options,
linalg::LinalgTransformationFilter::FilterFunction f = nullptr) linalg::LinalgTransformationFilter::FilterFunction f = nullptr)
: Transformation(f), opName(name), options(options) {} : Transformation(f), opName(name), options(options) {}
OwningRewritePatternList RewritePatternSet
buildRewritePatterns(MLIRContext *context, buildRewritePatterns(MLIRContext *context,
linalg::LinalgTransformationFilter m) override { linalg::LinalgTransformationFilter m) override {
OwningRewritePatternList tilingPatterns(context); RewritePatternSet tilingPatterns(context);
sfinae_enqueue<linalg::LinalgTilingPattern, LinalgOpType>( sfinae_enqueue<linalg::LinalgTilingPattern, LinalgOpType>(
tilingPatterns, options, opName, m); tilingPatterns, options, opName, m);
return tilingPatterns; return tilingPatterns;
} }
private: private:
std::string opName; std::string opName;
linalg::LinalgTilingOptions options; linalg::LinalgTilingOptions options;
}; };
/// Promotion transformation enqueues a particular stage-1 pattern for /// Promotion transformation enqueues a particular stage-1 pattern for
/// `Promote<LinalgOpType>`with the appropriate `options`. /// `Promote<LinalgOpType>`with the appropriate `options`.
template <typename LinalgOpType> template <typename LinalgOpType>
struct Promote : public Transformation { struct Promote : public Transformation {
explicit Promote( explicit Promote(
linalg::LinalgPromotionOptions options, linalg::LinalgPromotionOptions options,
linalg::LinalgTransformationFilter::FilterFunction f = nullptr) linalg::LinalgTransformationFilter::FilterFunction f = nullptr)
: Transformation(f), opName(LinalgOpType::getOperationName()), : Transformation(f), opName(LinalgOpType::getOperationName()),
options(options) {} options(options) {}
Promote(StringRef name, linalg::LinalgPromotionOptions options, Promote(StringRef name, linalg::LinalgPromotionOptions options,
linalg::LinalgTransformationFilter::FilterFunction f = nullptr) linalg::LinalgTransformationFilter::FilterFunction f = nullptr)
: Transformation(f), opName(name), options(options) {} : Transformation(f), opName(name), options(options) {}
OwningRewritePatternList RewritePatternSet
buildRewritePatterns(MLIRContext *context, buildRewritePatterns(MLIRContext *context,
linalg::LinalgTransformationFilter m) override { linalg::LinalgTransformationFilter m) override {
OwningRewritePatternList promotionPatterns(context); RewritePatternSet promotionPatterns(context);
sfinae_enqueue<linalg::LinalgPromotionPattern, LinalgOpType>( sfinae_enqueue<linalg::LinalgPromotionPattern, LinalgOpType>(
promotionPatterns, options, opName, m); promotionPatterns, options, opName, m);
return promotionPatterns; return promotionPatterns;
} }
private: private:
std::string opName; std::string opName;
linalg::LinalgPromotionOptions options; linalg::LinalgPromotionOptions options;
}; };
/// Vectorization transformation enqueues a particular stage-1 pattern for /// Vectorization transformation enqueues a particular stage-1 pattern for
/// `LinalgVectorizationPattern<LinalgOpType>` as well as copy to vector /// `LinalgVectorizationPattern<LinalgOpType>` as well as copy to vector
/// transfer rewrite forwarding patterns. /// transfer rewrite forwarding patterns.
template <typename LinalgOpType = LinalgOp> template <typename LinalgOpType = LinalgOp>
struct Vectorize : public Transformation { struct Vectorize : public Transformation {
explicit Vectorize( explicit Vectorize(
linalg::LinalgVectorizationOptions options, linalg::LinalgVectorizationOptions options,
linalg::LinalgTransformationFilter::FilterFunction f = nullptr) linalg::LinalgTransformationFilter::FilterFunction f = nullptr)
: Transformation(f), opName(), options(options) {} : Transformation(f), opName(), options(options) {}
Vectorize(StringRef name, linalg::LinalgVectorizationOptions options, Vectorize(StringRef name, linalg::LinalgVectorizationOptions options,
linalg::LinalgTransformationFilter::FilterFunction f = nullptr) linalg::LinalgTransformationFilter::FilterFunction f = nullptr)
: Transformation(f), opName(name), options(options) {} : Transformation(f), opName(name), options(options) {}
OwningRewritePatternList RewritePatternSet
buildRewritePatterns(MLIRContext *context, buildRewritePatterns(MLIRContext *context,
linalg::LinalgTransformationFilter m) override { linalg::LinalgTransformationFilter m) override {
OwningRewritePatternList vectorizationPatterns(context); RewritePatternSet vectorizationPatterns(context);
enqueue<linalg::LinalgVectorizationPattern, LinalgOpType>( enqueue<linalg::LinalgVectorizationPattern, LinalgOpType>(
vectorizationPatterns, options, opName, m); vectorizationPatterns, options, opName, m);
vectorizationPatterns.insert<linalg::LinalgCopyVTRForwardingPattern, vectorizationPatterns.add<linalg::LinalgCopyVTRForwardingPattern,
linalg::LinalgCopyVTWForwardingPattern>( linalg::LinalgCopyVTWForwardingPattern>(
context, /*benefit=*/2); context, /*benefit=*/2);
return vectorizationPatterns; return vectorizationPatterns;
} }
private: private:
std::string opName; std::string opName;
linalg::LinalgVectorizationOptions options; linalg::LinalgVectorizationOptions options;
}; };
▲ Show 20 Lines • Show All 190 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h

Show All 27 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Transformations exposed as function calls. // Transformations exposed as function calls.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
using LinalgLoops = SmallVector<Operation *, 4>; using LinalgLoops = SmallVector<Operation *, 4>;
/// Populates patterns for vectorization of all ConvN-D ops. /// Populates patterns for vectorization of all ConvN-D ops.
void populateConvVectorizationPatterns( void populateConvVectorizationPatterns(
MLIRContext *context, SmallVectorImpl<OwningRewritePatternList> &patterns, MLIRContext *context, SmallVectorImpl<RewritePatternSet> &patterns,
ArrayRef<int64_t> tileSizes); ArrayRef<int64_t> tileSizes);
/// Populates the given list with patterns to bufferize linalg ops. /// Populates the given list with patterns to bufferize linalg ops.
void populateLinalgBufferizePatterns(BufferizeTypeConverter &converter, void populateLinalgBufferizePatterns(BufferizeTypeConverter &converter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Performs standalone tiling of a single LinalgOp by `tileSizes`. /// Performs standalone tiling of a single LinalgOp by `tileSizes`.
/// and permute the loop nest according to `interchangeVector` /// and permute the loop nest according to `interchangeVector`
/// The permutation is expressed as a list of integers that specify /// The permutation is expressed as a list of integers that specify
/// the new ordering of the loop nest. The length of `interchangeVector` /// the new ordering of the loop nest. The length of `interchangeVector`
/// must be equal to the length of `tileSizes`. /// must be equal to the length of `tileSizes`.
/// An empty vector is interpreted as the identity permutation and the /// An empty vector is interpreted as the identity permutation and the
/// transformation returns early. /// transformation returns early.
▲ Show 20 Lines • Show All 386 Lines • ▼ Show 20 Lines setPaddingValueComputationFunction(PaddingValueComputationFunction fun) {
paddingValueComputationFunction = std::move(fun); paddingValueComputationFunction = std::move(fun);
return *this; return *this;
} }
}; };
/// Canonicalization patterns relevant to apply after tiling patterns. These are /// Canonicalization patterns relevant to apply after tiling patterns. These are
/// applied automatically by the tiling pass but need to be applied manually /// applied automatically by the tiling pass but need to be applied manually
/// when tiling is called programmatically. /// when tiling is called programmatically.
OwningRewritePatternList RewritePatternSet getLinalgTilingCanonicalizationPatterns(MLIRContext *ctx);
getLinalgTilingCanonicalizationPatterns(MLIRContext *ctx); void populateLinalgTilingCanonicalizationPatterns(RewritePatternSet &patterns);
void populateLinalgTilingCanonicalizationPatterns(
OwningRewritePatternList &patterns);
/// Base pattern that applied the tiling transformation specified by `options`. /// Base pattern that applied the tiling transformation specified by `options`.
/// Abort and return failure in 2 cases: /// Abort and return failure in 2 cases:
/// 1. if the tiling specification is invalid and tiling fails to occur. /// 1. if the tiling specification is invalid and tiling fails to occur.
/// 2. if tiling occurs but `options.paddingValueComputationFunction` is set /// 2. if tiling occurs but `options.paddingValueComputationFunction` is set
/// and some operand shape cannot be bounded statically. /// and some operand shape cannot be bounded statically.
struct LinalgBaseTilingPattern : public RewritePattern { struct LinalgBaseTilingPattern : public RewritePattern {
// Entry point to match any LinalgOp OpInterface. // Entry point to match any LinalgOp OpInterface.
▲ Show 20 Lines • Show All 229 Lines • ▼ Show 20 Lines
using detect_has_get_operation_name = using detect_has_get_operation_name =
llvm::is_detected<has_get_operation_name, T>; llvm::is_detected<has_get_operation_name, T>;
/// SFINAE helper for single C++ op with a `getOperationName` method. /// SFINAE helper for single C++ op with a `getOperationName` method.
template < template <
typename OpType, typename OpType,
typename = std::enable_if_t<detect_has_get_operation_name<OpType>::value>, typename = std::enable_if_t<detect_has_get_operation_name<OpType>::value>,
typename = void> typename = void>
void insertVectorizationPatternImpl(OwningRewritePatternList &patternList, void insertVectorizationPatternImpl(RewritePatternSet &patternList,
linalg::LinalgVectorizationOptions options, linalg::LinalgVectorizationOptions options,
linalg::LinalgTransformationFilter f) { linalg::LinalgTransformationFilter f) {
patternList.insert<linalg::LinalgVectorizationPattern>( patternList.add<linalg::LinalgVectorizationPattern>(
OpType::getOperationName(), patternList.getContext(), options, f); OpType::getOperationName(), patternList.getContext(), options, f);
} }
/// SFINAE helper for single C++ class without a `getOperationName` method (e.g. /// SFINAE helper for single C++ class without a `getOperationName` method (e.g.
/// an OpInterface). /// an OpInterface).
template <typename OpType, typename = std::enable_if_t< template <typename OpType, typename = std::enable_if_t<
!detect_has_get_operation_name<OpType>::value>> !detect_has_get_operation_name<OpType>::value>>
void insertVectorizationPatternImpl(OwningRewritePatternList &patternList, void insertVectorizationPatternImpl(RewritePatternSet &patternList,
linalg::LinalgVectorizationOptions options, linalg::LinalgVectorizationOptions options,
linalg::LinalgTransformationFilter f) { linalg::LinalgTransformationFilter f) {
patternList.insert<linalg::LinalgVectorizationPattern>( patternList.add<linalg::LinalgVectorizationPattern>(f.addOpFilter<OpType>(),
f.addOpFilter<OpType>(), options); options);
} }
/// Variadic helper function to insert vectorization patterns for C++ ops. /// Variadic helper function to insert vectorization patterns for C++ ops.
template <typename... OpTypes> template <typename... OpTypes>
void insertVectorizationPatterns(OwningRewritePatternList &patternList, void insertVectorizationPatterns(RewritePatternSet &patternList,
linalg::LinalgVectorizationOptions options, linalg::LinalgVectorizationOptions options,
linalg::LinalgTransformationFilter f = linalg::LinalgTransformationFilter f =
linalg::LinalgTransformationFilter()) { linalg::LinalgTransformationFilter()) {
// FIXME: In c++17 this can be simplified by using 'fold expressions'. // FIXME: In c++17 this can be simplified by using 'fold expressions'.
(void)std::initializer_list<int>{ (void)std::initializer_list<int>{
0, 0,
(insertVectorizationPatternImpl<OpTypes>(patternList, options, f), 0)...}; (insertVectorizationPatternImpl<OpTypes>(patternList, options, f), 0)...};
} }
▲ Show 20 Lines • Show All 62 Lines • ▼ Show 20 Linesprivate:
SmallVector<unsigned, 4> interchangeVector; SmallVector<unsigned, 4> interchangeVector;
}; };
/// Linalg generalization patterns /// Linalg generalization patterns
/// Populates `patterns` with patterns to convert spec-generated named ops to /// Populates `patterns` with patterns to convert spec-generated named ops to
/// linalg.generic ops. /// linalg.generic ops.
void populateLinalgNamedOpsGeneralizationPatterns( void populateLinalgNamedOpsGeneralizationPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns,
LinalgTransformationFilter filter = LinalgTransformationFilter()); LinalgTransformationFilter filter = LinalgTransformationFilter());
/// Populates `patterns` with patterns to convert linalg.conv ops to /// Populates `patterns` with patterns to convert linalg.conv ops to
/// linalg.generic ops. /// linalg.generic ops.
void populateLinalgConvGeneralizationPatterns( void populateLinalgConvGeneralizationPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns,
LinalgTransformationFilter filter = LinalgTransformationFilter()); LinalgTransformationFilter filter = LinalgTransformationFilter());
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Op-specific patterns. // Op-specific patterns.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// PadTensorOp does not implement the LinalgStructuredOpInterface `LinalgOp`, /// PadTensorOp does not implement the LinalgStructuredOpInterface `LinalgOp`,
/// it needs a specific pattern to vectorize. /// it needs a specific pattern to vectorize.
▲ Show 20 Lines • Show All 244 Lines • ▼ Show 20 Linesstruct SparsificationOptions {
unsigned vectorLength; unsigned vectorLength;
SparseIntType ptrType; SparseIntType ptrType;
SparseIntType indType; SparseIntType indType;
bool fastOutput; // experimental: fast output buffers bool fastOutput; // experimental: fast output buffers
}; };
/// Sets up sparsification rewriting rules with the given options. /// Sets up sparsification rewriting rules with the given options.
void populateSparsificationPatterns( void populateSparsificationPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns,
const SparsificationOptions &options = SparsificationOptions()); const SparsificationOptions &options = SparsificationOptions());
/// Sets up sparsification conversion rules with the given options. /// Sets up sparsification conversion rules with the given options.
void populateSparsificationConversionPatterns( void populateSparsificationConversionPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
} // namespace linalg } // namespace linalg
} // namespace mlir } // namespace mlir
#endif // DIALECT_LINALG_TRANSFORMS_TRANSFORMS_H_ #endif // DIALECT_LINALG_TRANSFORMS_TRANSFORMS_H_

mlir/include/mlir/Dialect/SCF/Transforms.h

Show First 20 Lines • Show All 55 Lines • ▼ Show 20 Lines
/// the ops to get converted properly. /// the ops to get converted properly.
/// ///
/// A "structural" type conversion is one where the underlying ops are /// A "structural" type conversion is one where the underlying ops are
/// completely agnostic to the actual types involved and simply need to update /// completely agnostic to the actual types involved and simply need to update
/// their types. An example of this is scf.if -- the scf.if op and the /// their types. An example of this is scf.if -- the scf.if op and the
/// corresponding scf.yield ops need to update their types accordingly to the /// corresponding scf.yield ops need to update their types accordingly to the
/// TypeConverter, but otherwise don't care what type conversions are happening. /// TypeConverter, but otherwise don't care what type conversions are happening.
void populateSCFStructuralTypeConversionsAndLegality( void populateSCFStructuralTypeConversionsAndLegality(
TypeConverter &typeConverter, OwningRewritePatternList &patterns, TypeConverter &typeConverter, RewritePatternSet &patterns,
ConversionTarget &target); ConversionTarget &target);
} // namespace scf } // namespace scf
} // namespace mlir } // namespace mlir
#endif // MLIR_DIALECT_SCF_TRANSFORMS_H_ #endif // MLIR_DIALECT_SCF_TRANSFORMS_H_

mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLSLCanonicalization.h

Show All 18 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// GLSL canonicalization patterns // GLSL canonicalization patterns
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace mlir { namespace mlir {
namespace spirv { namespace spirv {
void populateSPIRVGLSLCanonicalizationPatterns( void populateSPIRVGLSLCanonicalizationPatterns(
mlir::OwningRewritePatternList &results); mlir::RewritePatternSet &results);
} // namespace spirv } // namespace spirv
} // namespace mlir } // namespace mlir
#endif // MLIR_DIALECT_SPIRV_IR_SPIRVGLSLCANONICALIZATION_H_ #endif // MLIR_DIALECT_SPIRV_IR_SPIRVGLSLCANONICALIZATION_H_

mlir/include/mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h

Show First 20 Lines • Show All 62 Lines • ▼ Show 20 Linesprivate:
spirv::TargetEnv targetEnv; spirv::TargetEnv targetEnv;
}; };
/// Appends to a pattern list additional patterns for translating the builtin /// Appends to a pattern list additional patterns for translating the builtin
/// `func` op to the SPIR-V dialect. These patterns do not handle shader /// `func` op to the SPIR-V dialect. These patterns do not handle shader
/// interface/ABI; they convert function parameters to be of SPIR-V allowed /// interface/ABI; they convert function parameters to be of SPIR-V allowed
/// types. /// types.
void populateBuiltinFuncToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateBuiltinFuncToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
namespace spirv { namespace spirv {
class AccessChainOp; class AccessChainOp;
class FuncOp; class FuncOp;
class SPIRVConversionTarget : public ConversionTarget { class SPIRVConversionTarget : public ConversionTarget {
public: public:
/// Creates a SPIR-V conversion target for the given target environment. /// Creates a SPIR-V conversion target for the given target environment.
▲ Show 20 Lines • Show All 50 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Shape/Transforms/Passes.h

Show All 22 Lines
namespace mlir { namespace mlir {
/// Creates an instance of the ShapeToShapeLowering pass that legalizes Shape /// Creates an instance of the ShapeToShapeLowering pass that legalizes Shape
/// dialect to be convertible to Standard. For example, `shape.num_elements` get /// dialect to be convertible to Standard. For example, `shape.num_elements` get
/// transformed to `shape.reduce`, which can be lowered to SCF and Standard. /// transformed to `shape.reduce`, which can be lowered to SCF and Standard.
std::unique_ptr<Pass> createShapeToShapeLowering(); std::unique_ptr<Pass> createShapeToShapeLowering();
/// Collects a set of patterns to rewrite ops within the Shape dialect. /// Collects a set of patterns to rewrite ops within the Shape dialect.
void populateShapeRewritePatterns(OwningRewritePatternList &patterns); void populateShapeRewritePatterns(RewritePatternSet &patterns);
// Collects a set of patterns to replace all constraints with passing witnesses. // Collects a set of patterns to replace all constraints with passing witnesses.
// This is intended to then allow all ShapeConstraint related ops and data to // This is intended to then allow all ShapeConstraint related ops and data to
// have no effects and allow them to be freely removed such as through // have no effects and allow them to be freely removed such as through
// canonicalization and dead code elimination. // canonicalization and dead code elimination.
// //
// After this pass, no cstr_ operations exist. // After this pass, no cstr_ operations exist.
void populateRemoveShapeConstraintsPatterns(OwningRewritePatternList &patterns); void populateRemoveShapeConstraintsPatterns(RewritePatternSet &patterns);
std::unique_ptr<FunctionPass> createRemoveShapeConstraintsPass(); std::unique_ptr<FunctionPass> createRemoveShapeConstraintsPass();
/// Populates patterns for shape dialect structural type conversions and sets up /// Populates patterns for shape dialect structural type conversions and sets up
/// the provided ConversionTarget with the appropriate legality configuration /// the provided ConversionTarget with the appropriate legality configuration
/// for the ops to get converted properly. /// for the ops to get converted properly.
/// ///
/// A "structural" type conversion is one where the underlying ops are /// A "structural" type conversion is one where the underlying ops are
/// completely agnostic to the actual types involved and simply need to update /// completely agnostic to the actual types involved and simply need to update
/// their types consistently. An example of this is shape.assuming -- the /// their types consistently. An example of this is shape.assuming -- the
/// shape.assuming op and the corresponding shape.assuming_yield op need to have /// shape.assuming op and the corresponding shape.assuming_yield op need to have
/// consistent types, but the exact types don't matter. So all that we need to /// consistent types, but the exact types don't matter. So all that we need to
/// do for a structural type conversion is to update both of their types /// do for a structural type conversion is to update both of their types
/// consistently to the new types prescribed by the TypeConverter. /// consistently to the new types prescribed by the TypeConverter.
void populateShapeStructuralTypeConversionsAndLegality( void populateShapeStructuralTypeConversionsAndLegality(
TypeConverter &typeConverter, OwningRewritePatternList &patterns, TypeConverter &typeConverter, RewritePatternSet &patterns,
ConversionTarget &target); ConversionTarget &target);
// Bufferizes shape dialect ops. // Bufferizes shape dialect ops.
// //
// Note that most shape dialect ops must be converted to std before // Note that most shape dialect ops must be converted to std before
// bufferization happens, as they are intended to be bufferized at the std // bufferization happens, as they are intended to be bufferized at the std
// level. // level.
std::unique_ptr<FunctionPass> createShapeBufferizePass(); std::unique_ptr<FunctionPass> createShapeBufferizePass();
Show All 12 Lines

mlir/include/mlir/Dialect/StandardOps/Transforms/DecomposeCallGraphTypes.h

Show First 20 Lines • Show All 75 Lines • ▼ Show 20 Lines wrapDecomposeValueConversionCallback(FnT &&callback) {
}; };
} }
SmallVector<DecomposeValueConversionCallFn, 2> decomposeValueConversions; SmallVector<DecomposeValueConversionCallFn, 2> decomposeValueConversions;
}; };
/// Populates the patterns needed to drive the conversion process for /// Populates the patterns needed to drive the conversion process for
/// decomposing call graph types with the given `ValueDecomposer`. /// decomposing call graph types with the given `ValueDecomposer`.
void populateDecomposeCallGraphTypesPatterns( void populateDecomposeCallGraphTypesPatterns(MLIRContext *context,
MLIRContext *context, TypeConverter &typeConverter, TypeConverter &typeConverter,
ValueDecomposer &decomposer, OwningRewritePatternList &patterns); ValueDecomposer &decomposer,
RewritePatternSet &patterns);
} // end namespace mlir } // end namespace mlir
#endif // MLIR_DIALECT_STANDARDOPS_TRANSFORMS_DECOMPOSECALLGRAPHTYPES_H #endif // MLIR_DIALECT_STANDARDOPS_TRANSFORMS_DECOMPOSECALLGRAPHTYPES_H

mlir/include/mlir/Dialect/StandardOps/Transforms/FuncConversions.h

Show All 19 Lines
class MLIRContext; class MLIRContext;
class Operation; class Operation;
class TypeConverter; class TypeConverter;
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
/// Add a pattern to the given pattern list to convert the operand and result /// Add a pattern to the given pattern list to convert the operand and result
/// types of a CallOp with the given type converter. /// types of a CallOp with the given type converter.
void populateCallOpTypeConversionPattern(OwningRewritePatternList &patterns, void populateCallOpTypeConversionPattern(RewritePatternSet &patterns,
TypeConverter &converter); TypeConverter &converter);
/// Add a pattern to the given pattern list to rewrite branch operations to use /// Add a pattern to the given pattern list to rewrite branch operations to use
/// operands that have been legalized by the conversion framework. This can only /// operands that have been legalized by the conversion framework. This can only
/// be done if the branch operation implements the BranchOpInterface. Only /// be done if the branch operation implements the BranchOpInterface. Only
/// needed for partial conversions. /// needed for partial conversions.
void populateBranchOpInterfaceTypeConversionPattern( void populateBranchOpInterfaceTypeConversionPattern(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, TypeConverter &converter); TypeConverter &converter);
/// Return true if op is a BranchOpInterface op whose operands are all legal /// Return true if op is a BranchOpInterface op whose operands are all legal
/// according to converter. /// according to converter.
bool isLegalForBranchOpInterfaceTypeConversionPattern(Operation *op, bool isLegalForBranchOpInterfaceTypeConversionPattern(Operation *op,
TypeConverter &converter); TypeConverter &converter);
/// Add a pattern to the given pattern list to rewrite `return` ops to use /// Add a pattern to the given pattern list to rewrite `return` ops to use
/// operands that have been legalized by the conversion framework. /// operands that have been legalized by the conversion framework.
void populateReturnOpTypeConversionPattern(OwningRewritePatternList &patterns, void populateReturnOpTypeConversionPattern(RewritePatternSet &patterns,
TypeConverter &converter); TypeConverter &converter);
/// For ReturnLike ops (except `return`), return True. If op is a `return` && /// For ReturnLike ops (except `return`), return True. If op is a `return` &&
/// returnOpAlwaysLegal is false, legalize op according to converter. Otherwise, /// returnOpAlwaysLegal is false, legalize op according to converter. Otherwise,
/// return false. /// return false.
bool isLegalForReturnOpTypeConversionPattern(Operation *op, bool isLegalForReturnOpTypeConversionPattern(Operation *op,
TypeConverter &converter, TypeConverter &converter,
bool returnOpAlwaysLegal = false); bool returnOpAlwaysLegal = false);
Show All 10 Lines

mlir/include/mlir/Dialect/StandardOps/Transforms/Passes.h

Show All 17 Lines
#include "mlir/Transforms/Bufferize.h" #include "mlir/Transforms/Bufferize.h"
namespace mlir { namespace mlir {
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
void populateStdBufferizePatterns(BufferizeTypeConverter &typeConverter, void populateStdBufferizePatterns(BufferizeTypeConverter &typeConverter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Creates an instance of std bufferization pass. /// Creates an instance of std bufferization pass.
std::unique_ptr<Pass> createStdBufferizePass(); std::unique_ptr<Pass> createStdBufferizePass();
/// Creates an instance of func bufferization pass. /// Creates an instance of func bufferization pass.
std::unique_ptr<Pass> createFuncBufferizePass(); std::unique_ptr<Pass> createFuncBufferizePass();
/// Creates an instance of tensor constant bufferization pass. /// Creates an instance of tensor constant bufferization pass.
std::unique_ptr<Pass> createTensorConstantBufferizePass(); std::unique_ptr<Pass> createTensorConstantBufferizePass();
/// Creates an instance of the StdExpand pass that legalizes Std /// Creates an instance of the StdExpand pass that legalizes Std
/// dialect ops to be convertible to LLVM. For example, /// dialect ops to be convertible to LLVM. For example,
/// `std.ceildivi_signed` gets transformed to a number of std operations, /// `std.ceildivi_signed` gets transformed to a number of std operations,
/// which can be lowered to LLVM; `memref.reshape` gets converted to /// which can be lowered to LLVM; `memref.reshape` gets converted to
/// `memref_reinterpret_cast`. /// `memref_reinterpret_cast`.
std::unique_ptr<Pass> createStdExpandOpsPass(); std::unique_ptr<Pass> createStdExpandOpsPass();
/// Collects a set of patterns to rewrite ops within the Std dialect. /// Collects a set of patterns to rewrite ops within the Std dialect.
void populateStdExpandOpsPatterns(OwningRewritePatternList &patterns); void populateStdExpandOpsPatterns(RewritePatternSet &patterns);
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Registration // Registration
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Generate the code for registering passes. /// Generate the code for registering passes.
#define GEN_PASS_REGISTRATION #define GEN_PASS_REGISTRATION
#include "mlir/Dialect/StandardOps/Transforms/Passes.h.inc" #include "mlir/Dialect/StandardOps/Transforms/Passes.h.inc"
} // end namespace mlir } // end namespace mlir
#endif // MLIR_DIALECT_STANDARD_TRANSFORMS_PASSES_H_ #endif // MLIR_DIALECT_STANDARD_TRANSFORMS_PASSES_H_

mlir/include/mlir/Dialect/Tensor/Transforms/Passes.h

Show All 12 Lines
#include "mlir/Transforms/Bufferize.h" #include "mlir/Transforms/Bufferize.h"
namespace mlir { namespace mlir {
class RewritePatternSet; class RewritePatternSet;
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
void populateTensorBufferizePatterns(BufferizeTypeConverter &typeConverter, void populateTensorBufferizePatterns(BufferizeTypeConverter &typeConverter,
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Creates an instance of `tensor` dialect bufferization pass. /// Creates an instance of `tensor` dialect bufferization pass.
std::unique_ptr<Pass> createTensorBufferizePass(); std::unique_ptr<Pass> createTensorBufferizePass();
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Registration // Registration
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 9 Lines

mlir/include/mlir/Dialect/Vector/VectorOps.h

Show All 34 Lines
class VectorDialect; class VectorDialect;
namespace detail { namespace detail {
struct BitmaskEnumStorage; struct BitmaskEnumStorage;
} // namespace detail } // namespace detail
/// Collect a set of vector-to-vector canonicalization patterns. /// Collect a set of vector-to-vector canonicalization patterns.
void populateVectorToVectorCanonicalizationPatterns( void populateVectorToVectorCanonicalizationPatterns(
OwningRewritePatternList &patterns); RewritePatternSet &patterns);
/// Collect a set of vector-to-vector transformation patterns. /// Collect a set of vector-to-vector transformation patterns.
void populateVectorToVectorTransformationPatterns( void populateVectorToVectorTransformationPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
/// Collect a set of patterns to split transfer read/write ops. /// Collect a set of patterns to split transfer read/write ops.
/// ///
/// These patterns unrolls transfer read/write ops if the vector consumers/ /// These patterns unrolls transfer read/write ops if the vector consumers/
/// producers are extract/insert slices op. Transfer ops can map to hardware /// producers are extract/insert slices op. Transfer ops can map to hardware
/// load/store functionalities, where the vector size matters for bandwith /// load/store functionalities, where the vector size matters for bandwith
/// considerations. So these patterns should be collected separately, instead /// considerations. So these patterns should be collected separately, instead
/// of being generic canonicalization patterns. Also one can let the /// of being generic canonicalization patterns. Also one can let the
/// `ignoreFilter` to return true to fail matching for fine-grained control. /// `ignoreFilter` to return true to fail matching for fine-grained control.
void populateSplitVectorTransferPatterns( void populateSplitVectorTransferPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns,
std::function<bool(Operation *)> ignoreFilter = nullptr); std::function<bool(Operation *)> ignoreFilter = nullptr);
/// Collect a set of leading one dimension removal patterns. /// Collect a set of leading one dimension removal patterns.
/// ///
/// These patterns insert vector.shape_cast to remove leading one dimensions /// These patterns insert vector.shape_cast to remove leading one dimensions
/// to expose more canonical forms of read/write/insert/extract operations. /// to expose more canonical forms of read/write/insert/extract operations.
/// With them, there are more chances that we can cancel out extract-insert /// With them, there are more chances that we can cancel out extract-insert
/// pairs or forward write-read pairs. /// pairs or forward write-read pairs.
void populateCastAwayVectorLeadingOneDimPatterns( void populateCastAwayVectorLeadingOneDimPatterns(RewritePatternSet &patterns);
OwningRewritePatternList &patterns);
/// Collect a set of patterns that bubble up/down bitcast ops. /// Collect a set of patterns that bubble up/down bitcast ops.
/// ///
/// These patterns move vector.bitcast ops to be before insert ops or after /// These patterns move vector.bitcast ops to be before insert ops or after
/// extract ops where suitable. With them, bitcast will happen on smaller /// extract ops where suitable. With them, bitcast will happen on smaller
/// vectors and there are more chances to share extract/insert ops. /// vectors and there are more chances to share extract/insert ops.
void populateBubbleVectorBitCastOpPatterns(OwningRewritePatternList &patterns); void populateBubbleVectorBitCastOpPatterns(RewritePatternSet &patterns);
/// Collect a set of vector slices transformation patterns: /// Collect a set of vector slices transformation patterns:
/// ExtractSlicesOpLowering, InsertSlicesOpLowering /// ExtractSlicesOpLowering, InsertSlicesOpLowering
/// Useful for clients that want to express all vector "slices" /// Useful for clients that want to express all vector "slices"
/// ops in terms of more elementary vector "slice" ops. If all /// ops in terms of more elementary vector "slice" ops. If all
/// "produced" tuple values are "consumed" (the most common /// "produced" tuple values are "consumed" (the most common
/// use for "slices" ops), this lowering removes all tuple related /// use for "slices" ops), this lowering removes all tuple related
/// operations as well (through DCE and folding). If tuple values /// operations as well (through DCE and folding). If tuple values
/// "leak" coming in, however, some tuple related ops will remain. /// "leak" coming in, however, some tuple related ops will remain.
void populateVectorSlicesLoweringPatterns(OwningRewritePatternList &patterns); void populateVectorSlicesLoweringPatterns(RewritePatternSet &patterns);
/// Collect a set of transfer read/write lowering patterns. /// Collect a set of transfer read/write lowering patterns.
/// ///
/// These patterns lower transfer ops to simpler ops like `vector.load`, /// These patterns lower transfer ops to simpler ops like `vector.load`,
/// `vector.store` and `vector.broadcast`. /// `vector.store` and `vector.broadcast`.
void populateVectorTransferLoweringPatterns(OwningRewritePatternList &patterns); void populateVectorTransferLoweringPatterns(RewritePatternSet &patterns);
/// An attribute that specifies the combining function for `vector.contract`, /// An attribute that specifies the combining function for `vector.contract`,
/// and `vector.reduction`. /// and `vector.reduction`.
class CombiningKindAttr class CombiningKindAttr
: public Attribute::AttrBase<CombiningKindAttr, Attribute, : public Attribute::AttrBase<CombiningKindAttr, Attribute,
detail::BitmaskEnumStorage> { detail::BitmaskEnumStorage> {
public: public:
using Base::Base; using Base::Base;
▲ Show 20 Lines • Show All 67 Lines • ▼ Show 20 Lines
/// ShapeCastOp2DDownCastRewritePattern, /// ShapeCastOp2DDownCastRewritePattern,
/// ShapeCastOp2DUpCastRewritePattern /// ShapeCastOp2DUpCastRewritePattern
/// BroadcastOpLowering, /// BroadcastOpLowering,
/// TransposeOpLowering /// TransposeOpLowering
/// OuterproductOpLowering /// OuterproductOpLowering
/// These transformation express higher level vector ops in terms of more /// These transformation express higher level vector ops in terms of more
/// elementary extraction, insertion, reduction, product, and broadcast ops. /// elementary extraction, insertion, reduction, product, and broadcast ops.
void populateVectorContractLoweringPatterns( void populateVectorContractLoweringPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns,
VectorTransformsOptions vectorTransformOptions = VectorTransformsOptions()); VectorTransformsOptions vectorTransformOptions = VectorTransformsOptions());
/// Returns the integer type required for subscripts in the vector dialect. /// Returns the integer type required for subscripts in the vector dialect.
IntegerType getVectorSubscriptType(Builder &builder); IntegerType getVectorSubscriptType(Builder &builder);
/// Returns an integer array attribute containing the given values using /// Returns an integer array attribute containing the given values using
/// the integer type required for subscripts in the vector dialect. /// the integer type required for subscripts in the vector dialect.
ArrayAttr getVectorSubscriptAttr(Builder &b, ArrayRef<int64_t> values); ArrayAttr getVectorSubscriptAttr(Builder &b, ArrayRef<int64_t> values);
Show All 16 Lines

mlir/include/mlir/Dialect/Vector/VectorTransforms.h

Show All 21 Lines
namespace scf { namespace scf {
class IfOp; class IfOp;
} // namespace scf } // namespace scf
/// Collect a set of patterns to convert from the Vector dialect to itself. /// Collect a set of patterns to convert from the Vector dialect to itself.
/// Should be merged with populateVectorToSCFLoweringPattern. /// Should be merged with populateVectorToSCFLoweringPattern.
void populateVectorToVectorConversionPatterns( void populateVectorToVectorConversionPatterns(
MLIRContext *context, OwningRewritePatternList &patterns, MLIRContext *context, RewritePatternSet &patterns,
ArrayRef<int64_t> coarseVectorShape = {}, ArrayRef<int64_t> coarseVectorShape = {},
ArrayRef<int64_t> fineVectorShape = {}); ArrayRef<int64_t> fineVectorShape = {});
namespace vector { namespace vector {
/// Entry point for unrolling declarative pattern rewrites. /// Entry point for unrolling declarative pattern rewrites.
/// `op` is unrolled to the `targetShape` as follows, for each of its operands: /// `op` is unrolled to the `targetShape` as follows, for each of its operands:
/// 1. the unrolled type `unrolledVectorType` and number of unrolled instances /// 1. the unrolled type `unrolledVectorType` and number of unrolled instances
▲ Show 20 Lines • Show All 417 Lines • Show Last 20 Lines

mlir/include/mlir/IR/OpDefinition.h

Show First 20 Lines • Show All 179 Lines • ▼ Show 20 Linespublic:
RetT walk(FnT &&callback) { RetT walk(FnT &&callback) {
return state->walk<Order>(std::forward<FnT>(callback)); return state->walk<Order>(std::forward<FnT>(callback));
} }
// These are default implementations of customization hooks. // These are default implementations of customization hooks.
public: public:
/// This hook returns any canonicalization pattern rewrites that the operation /// This hook returns any canonicalization pattern rewrites that the operation
/// supports, for use by the canonicalization pass. /// supports, for use by the canonicalization pass.
static void getCanonicalizationPatterns(OwningRewritePatternList &results, static void getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) {} MLIRContext *context) {}
protected: protected:
/// If the concrete type didn't implement a custom verifier hook, just fall /// If the concrete type didn't implement a custom verifier hook, just fall
/// back to this one which accepts everything. /// back to this one which accepts everything.
LogicalResult verify() { return success(); } LogicalResult verify() { return success(); }
/// Unless overridden, the custom assembly form of an op is always rejected. /// Unless overridden, the custom assembly form of an op is always rejected.
▲ Show 20 Lines • Show All 1,597 Lines • Show Last 20 Lines

mlir/include/mlir/IR/OperationSupport.h

Show First 20 Lines • Show All 61 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// This is a "type erased" representation of a registered operation. This /// This is a "type erased" representation of a registered operation. This
/// should only be used by things like the AsmPrinter and other things that need /// should only be used by things like the AsmPrinter and other things that need
/// to be parameterized by generic operation hooks. Most user code should use /// to be parameterized by generic operation hooks. Most user code should use
/// the concrete operation types. /// the concrete operation types.
class AbstractOperation { class AbstractOperation {
public: public:
using GetCanonicalizationPatternsFn = void (*)(OwningRewritePatternList &, using GetCanonicalizationPatternsFn = void (*)(RewritePatternSet &,
MLIRContext *); MLIRContext *);
using FoldHookFn = LogicalResult (*)(Operation *, ArrayRef<Attribute>, using FoldHookFn = LogicalResult (*)(Operation *, ArrayRef<Attribute>,
SmallVectorImpl<OpFoldResult> &); SmallVectorImpl<OpFoldResult> &);
using HasTraitFn = bool (*)(TypeID); using HasTraitFn = bool (*)(TypeID);
using ParseAssemblyFn = ParseResult (*)(OpAsmParser &, OperationState &); using ParseAssemblyFn = ParseResult (*)(OpAsmParser &, OperationState &);
using PrintAssemblyFn = void (*)(Operation *, OpAsmPrinter &); using PrintAssemblyFn = void (*)(Operation *, OpAsmPrinter &);
using VerifyInvariantsFn = LogicalResult (*)(Operation *); using VerifyInvariantsFn = LogicalResult (*)(Operation *);
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Linespublic:
/// generalized constant folding. /// generalized constant folding.
LogicalResult foldHook(Operation *op, ArrayRef<Attribute> operands, LogicalResult foldHook(Operation *op, ArrayRef<Attribute> operands,
SmallVectorImpl<OpFoldResult> &results) const { SmallVectorImpl<OpFoldResult> &results) const {
return foldHookFn(op, operands, results); return foldHookFn(op, operands, results);
} }
/// This hook returns any canonicalization pattern rewrites that the operation /// This hook returns any canonicalization pattern rewrites that the operation
/// supports, for use by the canonicalization pass. /// supports, for use by the canonicalization pass.
void getCanonicalizationPatterns(OwningRewritePatternList &results, void getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) const { MLIRContext *context) const {
return getCanonicalizationPatternsFn(results, context); return getCanonicalizationPatternsFn(results, context);
} }
/// Returns an instance of the concept object for the given interface if it /// Returns an instance of the concept object for the given interface if it
/// was registered to this operation, null otherwise. This should not be used /// was registered to this operation, null otherwise. This should not be used
/// directly. /// directly.
template <typename T> template <typename T>
▲ Show 20 Lines • Show All 759 Lines • Show Last 20 Lines

mlir/include/mlir/IR/PatternMatch.h

Show First 20 Lines • Show All 888 Lines • ▼ Show 20 Lines addImpl(Args &&... args) {
pdlPatterns.mergeIn(T(std::forward<Args>(args)...)); pdlPatterns.mergeIn(T(std::forward<Args>(args)...));
} }
MLIRContext *const context; MLIRContext *const context;
NativePatternListT nativePatterns; NativePatternListT nativePatterns;
PDLPatternModule pdlPatterns; PDLPatternModule pdlPatterns;
}; };
// TODO: OwningRewritePatternList is soft-deprecated and will be removed in the // TODO: RewritePatternSet is soft-deprecated and will be removed in the
// future. // future.
using OwningRewritePatternList = RewritePatternSet; using OwningRewritePatternList = RewritePatternSet;
} // end namespace mlir } // end namespace mlir
#endif // MLIR_PATTERN_MATCH_H #endif // MLIR_PATTERN_MATCH_H

mlir/include/mlir/Rewrite/FrozenRewritePatternList.h

Show All 21 Lines
/// this class share the same compiled pattern list, allowing for a reduction in /// this class share the same compiled pattern list, allowing for a reduction in
/// the number of duplicated patterns that need to be created. /// the number of duplicated patterns that need to be created.
class FrozenRewritePatternList { class FrozenRewritePatternList {
using NativePatternListT = std::vector<std::unique_ptr<RewritePattern>>; using NativePatternListT = std::vector<std::unique_ptr<RewritePattern>>;
public: public:
/// Freeze the patterns held in `patterns`, and take ownership. /// Freeze the patterns held in `patterns`, and take ownership.
FrozenRewritePatternList(); FrozenRewritePatternList();
FrozenRewritePatternList(OwningRewritePatternList &&patterns); FrozenRewritePatternList(RewritePatternSet &&patterns);
FrozenRewritePatternList(FrozenRewritePatternList &&patterns) = default; FrozenRewritePatternList(FrozenRewritePatternList &&patterns) = default;
FrozenRewritePatternList(const FrozenRewritePatternList &patterns) = default; FrozenRewritePatternList(const FrozenRewritePatternList &patterns) = default;
FrozenRewritePatternList & FrozenRewritePatternList &
operator=(const FrozenRewritePatternList &patterns) = default; operator=(const FrozenRewritePatternList &patterns) = default;
FrozenRewritePatternList & FrozenRewritePatternList &
operator=(FrozenRewritePatternList &&patterns) = default; operator=(FrozenRewritePatternList &&patterns) = default;
~FrozenRewritePatternList(); ~FrozenRewritePatternList();
Show All 33 Lines

mlir/include/mlir/Transforms/Bufferize.h

Show First 20 Lines • Show All 50 Lines • ▼ Show 20 Lines
/// such as via the patterns in /// such as via the patterns in
/// populateEliminateBufferizeMaterializationsPatterns. /// populateEliminateBufferizeMaterializationsPatterns.
void populateBufferizeMaterializationLegality(ConversionTarget &target); void populateBufferizeMaterializationLegality(ConversionTarget &target);
/// Populate patterns to eliminate bufferize materializations. /// Populate patterns to eliminate bufferize materializations.
/// ///
/// In particular, these are the tensor_load/buffer_cast ops. /// In particular, these are the tensor_load/buffer_cast ops.
void populateEliminateBufferizeMaterializationsPatterns( void populateEliminateBufferizeMaterializationsPatterns(
BufferizeTypeConverter &typeConverter, OwningRewritePatternList &patterns); BufferizeTypeConverter &typeConverter, RewritePatternSet &patterns);
} // end namespace mlir } // end namespace mlir
#endif // MLIR_TRANSFORMS_BUFFERIZE_H #endif // MLIR_TRANSFORMS_BUFFERIZE_H

mlir/include/mlir/Transforms/DialectConversion.h

Show First 20 Lines • Show All 417 Lines • ▼ Show 20 Lines
private: private:
using ConversionPattern::matchAndRewrite; using ConversionPattern::matchAndRewrite;
}; };
/// Add a pattern to the given pattern list to convert the signature of a /// Add a pattern to the given pattern list to convert the signature of a
/// FunctionLike op with the given type converter. This only supports /// FunctionLike op with the given type converter. This only supports
/// FunctionLike ops which use FunctionType to represent their type. /// FunctionLike ops which use FunctionType to represent their type.
void populateFunctionLikeTypeConversionPattern( void populateFunctionLikeTypeConversionPattern(StringRef functionLikeOpName,
StringRef functionLikeOpName, OwningRewritePatternList &patterns, RewritePatternSet &patterns,
TypeConverter &converter); TypeConverter &converter);
template <typename FuncOpT> template <typename FuncOpT>
void populateFunctionLikeTypeConversionPattern( void populateFunctionLikeTypeConversionPattern(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, TypeConverter &converter) { TypeConverter &converter) {
populateFunctionLikeTypeConversionPattern(FuncOpT::getOperationName(), populateFunctionLikeTypeConversionPattern(FuncOpT::getOperationName(),
patterns, converter); patterns, converter);
} }
/// Add a pattern to the given pattern list to convert the signature of a FuncOp /// Add a pattern to the given pattern list to convert the signature of a FuncOp
/// with the given type converter. /// with the given type converter.
void populateFuncOpTypeConversionPattern(OwningRewritePatternList &patterns, void populateFuncOpTypeConversionPattern(RewritePatternSet &patterns,
TypeConverter &converter); TypeConverter &converter);
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Conversion PatternRewriter // Conversion PatternRewriter
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace detail { namespace detail {
struct ConversionPatternRewriterImpl; struct ConversionPatternRewriterImpl;
▲ Show 20 Lines • Show All 432 Lines • Show Last 20 Lines

mlir/lib/Conversion/AffineToStandard/AffineToStandard.cpp

Show First 20 Lines • Show All 740 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(AffineVectorStoreOp op,
rewriter.replaceOpWithNewOp<vector::StoreOp>( rewriter.replaceOpWithNewOp<vector::StoreOp>(
op, op.getValueToStore(), op.getMemRef(), *maybeExpandedMap); op, op.getValueToStore(), op.getMemRef(), *maybeExpandedMap);
return success(); return success();
} }
}; };
} // end namespace } // end namespace
void mlir::populateAffineToStdConversionPatterns( void mlir::populateAffineToStdConversionPatterns(RewritePatternSet &patterns) {
OwningRewritePatternList &patterns) {
// clang-format off // clang-format off
patterns.insert< patterns.add<
AffineApplyLowering, AffineApplyLowering,
AffineDmaStartLowering, AffineDmaStartLowering,
AffineDmaWaitLowering, AffineDmaWaitLowering,
AffineLoadLowering, AffineLoadLowering,
AffineMinLowering, AffineMinLowering,
AffineMaxLowering, AffineMaxLowering,
AffineParallelLowering, AffineParallelLowering,
AffinePrefetchLowering, AffinePrefetchLowering,
AffineStoreLowering, AffineStoreLowering,
AffineForLowering, AffineForLowering,
AffineIfLowering, AffineIfLowering,
AffineYieldOpLowering>(patterns.getContext()); AffineYieldOpLowering>(patterns.getContext());
// clang-format on // clang-format on
} }
void mlir::populateAffineToVectorConversionPatterns( void mlir::populateAffineToVectorConversionPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
// clang-format off // clang-format off
patterns.insert< patterns.add<
AffineVectorLoadLowering, AffineVectorLoadLowering,
AffineVectorStoreLowering>(patterns.getContext()); AffineVectorStoreLowering>(patterns.getContext());
// clang-format on // clang-format on
} }
namespace { namespace {
class LowerAffinePass : public ConvertAffineToStandardBase<LowerAffinePass> { class LowerAffinePass : public ConvertAffineToStandardBase<LowerAffinePass> {
void runOnOperation() override { void runOnOperation() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateAffineToStdConversionPatterns(patterns); populateAffineToStdConversionPatterns(patterns);
populateAffineToVectorConversionPatterns(patterns); populateAffineToVectorConversionPatterns(patterns);
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addLegalDialect<memref::MemRefDialect, scf::SCFDialect, target.addLegalDialect<memref::MemRefDialect, scf::SCFDialect,
StandardOpsDialect, VectorDialect>(); StandardOpsDialect, VectorDialect>();
if (failed(applyPartialConversion(getOperation(), target, if (failed(applyPartialConversion(getOperation(), target,
std::move(patterns)))) std::move(patterns))))
signalPassFailure(); signalPassFailure();
Show All 9 Lines

mlir/lib/Conversion/ArmSVEToLLVM/ArmSVEToLLVM.cpp

Show First 20 Lines • Show All 90 Lines • ▼ Show 20 Linesstatic Optional<Value> addUnrealizedCast(OpBuilder &builder,
if (inputs.size() != 1 || if (inputs.size() != 1 ||
!inputs[0].getType().isa<LLVM::LLVMScalableVectorType>()) !inputs[0].getType().isa<LLVM::LLVMScalableVectorType>())
return Value(); return Value();
return builder.create<UnrealizedConversionCastOp>(loc, svType, inputs) return builder.create<UnrealizedConversionCastOp>(loc, svType, inputs)
.getResult(0); .getResult(0);
} }
/// Populate the given list with patterns that convert from ArmSVE to LLVM. /// Populate the given list with patterns that convert from ArmSVE to LLVM.
void mlir::populateArmSVEToLLVMConversionPatterns( void mlir::populateArmSVEToLLVMConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
converter.addConversion([&converter](ScalableVectorType svType) { converter.addConversion([&converter](ScalableVectorType svType) {
return convertScalableVectorTypeToLLVM(svType, converter); return convertScalableVectorTypeToLLVM(svType, converter);
}); });
converter.addSourceMaterialization(addUnrealizedCast); converter.addSourceMaterialization(addUnrealizedCast);
// clang-format off // clang-format off
patterns.insert<ForwardOperands<CallOp>, patterns.add<ForwardOperands<CallOp>,
ForwardOperands<CallIndirectOp>, ForwardOperands<CallIndirectOp>,
ForwardOperands<ReturnOp>>(converter, ForwardOperands<ReturnOp>>(converter,
&converter.getContext()); &converter.getContext());
patterns.insert<SdotOpLowering, patterns.add<SdotOpLowering,
SmmlaOpLowering, SmmlaOpLowering,
UdotOpLowering, UdotOpLowering,
UmmlaOpLowering, UmmlaOpLowering,
VectorScaleOpLowering>(converter); VectorScaleOpLowering>(converter);
// clang-format on // clang-format on
} }

mlir/lib/Conversion/AsyncToLLVM/AsyncToLLVM.cpp

Show First 20 Lines • Show All 869 Lines • ▼ Show 20 Linesvoid ConvertAsyncToLLVMPass::runOnOperation() {
addAsyncRuntimeApiDeclarations(module); addAsyncRuntimeApiDeclarations(module);
addCRuntimeDeclarations(module); addCRuntimeDeclarations(module);
// Lower async.runtime and async.coro operations to Async Runtime API and // Lower async.runtime and async.coro operations to Async Runtime API and
// LLVM coroutine intrinsics. // LLVM coroutine intrinsics.
// Convert async dialect types and operations to LLVM dialect. // Convert async dialect types and operations to LLVM dialect.
AsyncRuntimeTypeConverter converter; AsyncRuntimeTypeConverter converter;
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
// We use conversion to LLVM type to lower async.runtime load and store // We use conversion to LLVM type to lower async.runtime load and store
// operations. // operations.
LLVMTypeConverter llvmConverter(ctx); LLVMTypeConverter llvmConverter(ctx);
llvmConverter.addConversion(AsyncRuntimeTypeConverter::convertAsyncTypes); llvmConverter.addConversion(AsyncRuntimeTypeConverter::convertAsyncTypes);
// Convert async types in function signatures and function calls. // Convert async types in function signatures and function calls.
populateFuncOpTypeConversionPattern(patterns, converter); populateFuncOpTypeConversionPattern(patterns, converter);
populateCallOpTypeConversionPattern(patterns, converter); populateCallOpTypeConversionPattern(patterns, converter);
// Convert return operations inside async.execute regions. // Convert return operations inside async.execute regions.
patterns.insert<ReturnOpOpConversion>(converter, ctx); patterns.add<ReturnOpOpConversion>(converter, ctx);
// Lower async.runtime operations to the async runtime API calls. // Lower async.runtime operations to the async runtime API calls.
patterns.insert<RuntimeSetAvailableOpLowering, RuntimeAwaitOpLowering, patterns.add<RuntimeSetAvailableOpLowering, RuntimeAwaitOpLowering,
RuntimeAwaitAndResumeOpLowering, RuntimeResumeOpLowering, RuntimeAwaitAndResumeOpLowering, RuntimeResumeOpLowering,
RuntimeAddToGroupOpLowering, RuntimeAddRefOpLowering, RuntimeAddToGroupOpLowering, RuntimeAddRefOpLowering,
RuntimeDropRefOpLowering>(converter, ctx); RuntimeDropRefOpLowering>(converter, ctx);
// Lower async.runtime operations that rely on LLVM type converter to convert // Lower async.runtime operations that rely on LLVM type converter to convert
// from async value payload type to the LLVM type. // from async value payload type to the LLVM type.
patterns.insert<RuntimeCreateOpLowering, RuntimeStoreOpLowering, patterns.add<RuntimeCreateOpLowering, RuntimeStoreOpLowering,
RuntimeLoadOpLowering>(llvmConverter, ctx); RuntimeLoadOpLowering>(llvmConverter, ctx);
// Lower async coroutine operations to LLVM coroutine intrinsics. // Lower async coroutine operations to LLVM coroutine intrinsics.
patterns.insert<CoroIdOpConversion, CoroBeginOpConversion, patterns
CoroFreeOpConversion, CoroEndOpConversion, .add<CoroIdOpConversion, CoroBeginOpConversion, CoroFreeOpConversion,
CoroSaveOpConversion, CoroSuspendOpConversion>(converter, CoroEndOpConversion, CoroSaveOpConversion, CoroSuspendOpConversion>(
ctx); converter, ctx);
ConversionTarget target(*ctx); ConversionTarget target(*ctx);
target.addLegalOp<ConstantOp>(); target.addLegalOp<ConstantOp>();
target.addLegalDialect<LLVM::LLVMDialect>(); target.addLegalDialect<LLVM::LLVMDialect>();
// All operations from Async dialect must be lowered to the runtime API and // All operations from Async dialect must be lowered to the runtime API and
// LLVM intrinsics calls. // LLVM intrinsics calls.
target.addIllegalDialect<AsyncDialect>(); target.addIllegalDialect<AsyncDialect>();
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Lines
}; };
} // namespace } // namespace
std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertAsyncToLLVMPass() { std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertAsyncToLLVMPass() {
return std::make_unique<ConvertAsyncToLLVMPass>(); return std::make_unique<ConvertAsyncToLLVMPass>();
} }
void mlir::populateAsyncStructuralTypeConversionsAndLegality( void mlir::populateAsyncStructuralTypeConversionsAndLegality(
TypeConverter &typeConverter, OwningRewritePatternList &patterns, TypeConverter &typeConverter, RewritePatternSet &patterns,
ConversionTarget &target) { ConversionTarget &target) {
typeConverter.addConversion([&](TokenType type) { return type; }); typeConverter.addConversion([&](TokenType type) { return type; });
typeConverter.addConversion([&](ValueType type) { typeConverter.addConversion([&](ValueType type) {
return ValueType::get(typeConverter.convertType(type.getValueType())); return ValueType::get(typeConverter.convertType(type.getValueType()));
}); });
patterns patterns.add<ConvertExecuteOpTypes, ConvertAwaitOpTypes, ConvertYieldOpTypes>(
.insert<ConvertExecuteOpTypes, ConvertAwaitOpTypes, ConvertYieldOpTypes>(
typeConverter, patterns.getContext()); typeConverter, patterns.getContext());
target.addDynamicallyLegalOp<AwaitOp, ExecuteOp, async::YieldOp>( target.addDynamicallyLegalOp<AwaitOp, ExecuteOp, async::YieldOp>(
[&](Operation *op) { return typeConverter.isLegal(op); }); [&](Operation *op) { return typeConverter.isLegal(op); });
} }

mlir/lib/Conversion/ComplexToLLVM/ComplexToLLVM.cpp

Show First 20 Lines • Show All 252 Lines • ▼ Show 20 Lines matchAndRewrite(complex::SubOp op, ArrayRef<Value> operands,
rewriter.replaceOp(op, {result}); rewriter.replaceOp(op, {result});
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateComplexToLLVMConversionPatterns( void mlir::populateComplexToLLVMConversionPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, RewritePatternSet &patterns) {
// clang-format off // clang-format off
patterns.insert< patterns.add<
AbsOpConversion, AbsOpConversion,
AddOpConversion, AddOpConversion,
CreateOpConversion, CreateOpConversion,
DivOpConversion, DivOpConversion,
ImOpConversion, ImOpConversion,
MulOpConversion, MulOpConversion,
ReOpConversion, ReOpConversion,
SubOpConversion SubOpConversion
>(converter); >(converter);
// clang-format on // clang-format on
} }
namespace { namespace {
struct ConvertComplexToLLVMPass struct ConvertComplexToLLVMPass
: public ConvertComplexToLLVMBase<ConvertComplexToLLVMPass> { : public ConvertComplexToLLVMBase<ConvertComplexToLLVMPass> {
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void ConvertComplexToLLVMPass::runOnOperation() { void ConvertComplexToLLVMPass::runOnOperation() {
auto module = getOperation(); auto module = getOperation();
// Convert to the LLVM IR dialect using the converter defined above. // Convert to the LLVM IR dialect using the converter defined above.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
LLVMTypeConverter converter(&getContext()); LLVMTypeConverter converter(&getContext());
populateComplexToLLVMConversionPatterns(converter, patterns); populateComplexToLLVMConversionPatterns(converter, patterns);
LLVMConversionTarget target(getContext()); LLVMConversionTarget target(getContext());
target.addLegalOp<ModuleOp, FuncOp>(); target.addLegalOp<ModuleOp, FuncOp>();
target.addLegalOp<LLVM::DialectCastOp>(); target.addLegalOp<LLVM::DialectCastOp>();
target.addIllegalDialect<complex::ComplexDialect>(); target.addIllegalDialect<complex::ComplexDialect>();
if (failed(applyPartialConversion(module, target, std::move(patterns)))) if (failed(applyPartialConversion(module, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> std::unique_ptr<OperationPass<ModuleOp>>
mlir::createConvertComplexToLLVMPass() { mlir::createConvertComplexToLLVMPass() {
return std::make_unique<ConvertComplexToLLVMPass>(); return std::make_unique<ConvertComplexToLLVMPass>();
} }

mlir/lib/Conversion/GPUCommon/GPUToLLVMConversion.cpp

Show First 20 Lines • Show All 302 Lines • ▼ Show 20 Linesprivate:
LogicalResult LogicalResult
matchAndRewrite(gpu::MemcpyOp memcpyOp, ArrayRef<Value> operands, matchAndRewrite(gpu::MemcpyOp memcpyOp, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override; ConversionPatternRewriter &rewriter) const override;
}; };
} // namespace } // namespace
void GpuToLLVMConversionPass::runOnOperation() { void GpuToLLVMConversionPass::runOnOperation() {
LLVMTypeConverter converter(&getContext()); LLVMTypeConverter converter(&getContext());
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
LLVMConversionTarget target(getContext()); LLVMConversionTarget target(getContext());
populateVectorToLLVMConversionPatterns(converter, patterns); populateVectorToLLVMConversionPatterns(converter, patterns);
populateStdToLLVMConversionPatterns(converter, patterns); populateStdToLLVMConversionPatterns(converter, patterns);
populateAsyncStructuralTypeConversionsAndLegality(converter, patterns, populateAsyncStructuralTypeConversionsAndLegality(converter, patterns,
target); target);
converter.addConversion( converter.addConversion(
[context = &converter.getContext()](gpu::AsyncTokenType type) -> Type { [context = &converter.getContext()](gpu::AsyncTokenType type) -> Type {
return LLVM::LLVMPointerType::get(IntegerType::get(context, 8)); return LLVM::LLVMPointerType::get(IntegerType::get(context, 8));
}); });
patterns.insert<ConvertAllocOpToGpuRuntimeCallPattern, patterns.add<ConvertAllocOpToGpuRuntimeCallPattern,
ConvertDeallocOpToGpuRuntimeCallPattern, ConvertDeallocOpToGpuRuntimeCallPattern,
ConvertHostRegisterOpToGpuRuntimeCallPattern, ConvertHostRegisterOpToGpuRuntimeCallPattern,
ConvertMemcpyOpToGpuRuntimeCallPattern, ConvertMemcpyOpToGpuRuntimeCallPattern,
ConvertWaitAsyncOpToGpuRuntimeCallPattern, ConvertWaitAsyncOpToGpuRuntimeCallPattern,
ConvertWaitOpToGpuRuntimeCallPattern, ConvertWaitOpToGpuRuntimeCallPattern,
ConvertAsyncYieldToGpuRuntimeCallPattern>(converter); ConvertAsyncYieldToGpuRuntimeCallPattern>(converter);
patterns.insert<ConvertLaunchFuncOpToGpuRuntimeCallPattern>( patterns.add<ConvertLaunchFuncOpToGpuRuntimeCallPattern>(converter,
converter, gpuBinaryAnnotation); gpuBinaryAnnotation);
patterns.insert<EraseGpuModuleOpPattern>(&converter.getContext()); patterns.add<EraseGpuModuleOpPattern>(&converter.getContext());
if (failed( if (failed(
applyPartialConversion(getOperation(), target, std::move(patterns)))) applyPartialConversion(getOperation(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
LLVM::CallOp FunctionCallBuilder::create(Location loc, OpBuilder &builder, LLVM::CallOp FunctionCallBuilder::create(Location loc, OpBuilder &builder,
ArrayRef<Value> arguments) const { ArrayRef<Value> arguments) const {
▲ Show 20 Lines • Show All 460 Lines • Show Last 20 Lines

mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp

Show First 20 Lines • Show All 119 Lines • ▼ Show 20 Lines void runOnOperation() override {
LLVMTypeConverter converter(m.getContext(), options); LLVMTypeConverter converter(m.getContext(), options);
converter.addConversion([&](MemRefType type) -> Optional<Type> { converter.addConversion([&](MemRefType type) -> Optional<Type> {
if (type.getMemorySpaceAsInt() != if (type.getMemorySpaceAsInt() !=
gpu::GPUDialect::getPrivateAddressSpace()) gpu::GPUDialect::getPrivateAddressSpace())
return llvm::None; return llvm::None;
return converter.convertType(MemRefType::Builder(type).setMemorySpace(0)); return converter.convertType(MemRefType::Builder(type).setMemorySpace(0));
}); });
OwningRewritePatternList patterns(m.getContext()); RewritePatternSet patterns(m.getContext());
OwningRewritePatternList llvmPatterns(m.getContext()); RewritePatternSet llvmPatterns(m.getContext());
// Apply in-dialect lowering first. In-dialect lowering will replace ops // Apply in-dialect lowering first. In-dialect lowering will replace ops
// which need to be lowered further, which is not supported by a single // which need to be lowered further, which is not supported by a single
// conversion pass. // conversion pass.
populateGpuRewritePatterns(patterns); populateGpuRewritePatterns(patterns);
(void)applyPatternsAndFoldGreedily(m, std::move(patterns)); (void)applyPatternsAndFoldGreedily(m, std::move(patterns));
populateStdToLLVMConversionPatterns(converter, llvmPatterns); populateStdToLLVMConversionPatterns(converter, llvmPatterns);
Show All 15 Linesvoid mlir::configureGpuToNVVMConversionLegality(ConversionTarget &target) {
target.addIllegalOp<LLVM::CosOp, LLVM::ExpOp, LLVM::FAbsOp, LLVM::FCeilOp, target.addIllegalOp<LLVM::CosOp, LLVM::ExpOp, LLVM::FAbsOp, LLVM::FCeilOp,
LLVM::FFloorOp, LLVM::LogOp, LLVM::Log10Op, LLVM::Log2Op, LLVM::FFloorOp, LLVM::LogOp, LLVM::Log10Op, LLVM::Log2Op,
LLVM::PowOp, LLVM::SinOp, LLVM::SqrtOp>(); LLVM::PowOp, LLVM::SinOp, LLVM::SqrtOp>();
// TODO: Remove once we support replacing non-root ops. // TODO: Remove once we support replacing non-root ops.
target.addLegalOp<gpu::YieldOp, gpu::GPUModuleOp, gpu::ModuleEndOp>(); target.addLegalOp<gpu::YieldOp, gpu::GPUModuleOp, gpu::ModuleEndOp>();
} }
void mlir::populateGpuToNVVMConversionPatterns( void mlir::populateGpuToNVVMConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
populateWithGenerated(patterns); populateWithGenerated(patterns);
patterns patterns
.insert<GPUIndexIntrinsicOpLowering<gpu::ThreadIdOp, NVVM::ThreadIdXOp, .add<GPUIndexIntrinsicOpLowering<gpu::ThreadIdOp, NVVM::ThreadIdXOp,
NVVM::ThreadIdYOp, NVVM::ThreadIdZOp>, NVVM::ThreadIdYOp, NVVM::ThreadIdZOp>,
GPUIndexIntrinsicOpLowering<gpu::BlockDimOp, NVVM::BlockDimXOp, GPUIndexIntrinsicOpLowering<gpu::BlockDimOp, NVVM::BlockDimXOp,
NVVM::BlockDimYOp, NVVM::BlockDimZOp>, NVVM::BlockDimYOp, NVVM::BlockDimZOp>,
GPUIndexIntrinsicOpLowering<gpu::BlockIdOp, NVVM::BlockIdXOp, GPUIndexIntrinsicOpLowering<gpu::BlockIdOp, NVVM::BlockIdXOp,
NVVM::BlockIdYOp, NVVM::BlockIdZOp>, NVVM::BlockIdYOp, NVVM::BlockIdZOp>,
GPUIndexIntrinsicOpLowering<gpu::GridDimOp, NVVM::GridDimXOp, GPUIndexIntrinsicOpLowering<gpu::GridDimOp, NVVM::GridDimXOp,
NVVM::GridDimYOp, NVVM::GridDimZOp>, NVVM::GridDimYOp, NVVM::GridDimZOp>,
GPUShuffleOpLowering, GPUReturnOpLowering>(converter); GPUShuffleOpLowering, GPUReturnOpLowering>(converter);
// Explicitly drop memory space when lowering private memory // Explicitly drop memory space when lowering private memory
// attributions since NVVM models it as `alloca`s in the default // attributions since NVVM models it as `alloca`s in the default
// memory space and does not support `alloca`s with addrspace(5). // memory space and does not support `alloca`s with addrspace(5).
patterns.insert<GPUFuncOpLowering>( patterns.add<GPUFuncOpLowering>(
converter, /*allocaAddrSpace=*/0, converter, /*allocaAddrSpace=*/0,
Identifier::get(NVVM::NVVMDialect::getKernelFuncAttrName(), Identifier::get(NVVM::NVVMDialect::getKernelFuncAttrName(),
&converter.getContext())); &converter.getContext()));
patterns.insert<OpToFuncCallLowering<AbsFOp>>(converter, "__nv_fabsf", patterns.add<OpToFuncCallLowering<AbsFOp>>(converter, "__nv_fabsf",
"__nv_fabs"); "__nv_fabs");
patterns.insert<OpToFuncCallLowering<math::AtanOp>>(converter, "__nv_atanf", patterns.add<OpToFuncCallLowering<math::AtanOp>>(converter, "__nv_atanf",
"__nv_atan"); "__nv_atan");
patterns.insert<OpToFuncCallLowering<math::Atan2Op>>(converter, "__nv_atan2f", patterns.add<OpToFuncCallLowering<math::Atan2Op>>(converter, "__nv_atan2f",
"__nv_atan2"); "__nv_atan2");
patterns.insert<OpToFuncCallLowering<CeilFOp>>(converter, "__nv_ceilf", patterns.add<OpToFuncCallLowering<CeilFOp>>(converter, "__nv_ceilf",
"__nv_ceil"); "__nv_ceil");
patterns.insert<OpToFuncCallLowering<math::CosOp>>(converter, "__nv_cosf", patterns.add<OpToFuncCallLowering<math::CosOp>>(converter, "__nv_cosf",
"__nv_cos"); "__nv_cos");
patterns.insert<OpToFuncCallLowering<math::ExpOp>>(converter, "__nv_expf", patterns.add<OpToFuncCallLowering<math::ExpOp>>(converter, "__nv_expf",
"__nv_exp"); "__nv_exp");
patterns.insert<OpToFuncCallLowering<math::ExpM1Op>>(converter, "__nv_expm1f", patterns.add<OpToFuncCallLowering<math::ExpM1Op>>(converter, "__nv_expm1f",
"__nv_expm1"); "__nv_expm1");
patterns.insert<OpToFuncCallLowering<FloorFOp>>(converter, "__nv_floorf", patterns.add<OpToFuncCallLowering<FloorFOp>>(converter, "__nv_floorf",
"__nv_floor"); "__nv_floor");
patterns.insert<OpToFuncCallLowering<math::LogOp>>(converter, "__nv_logf", patterns.add<OpToFuncCallLowering<math::LogOp>>(converter, "__nv_logf",
"__nv_log"); "__nv_log");
patterns.insert<OpToFuncCallLowering<math::Log1pOp>>(converter, "__nv_log1pf", patterns.add<OpToFuncCallLowering<math::Log1pOp>>(converter, "__nv_log1pf",
"__nv_log1p"); "__nv_log1p");
patterns.insert<OpToFuncCallLowering<math::Log10Op>>(converter, "__nv_log10f", patterns.add<OpToFuncCallLowering<math::Log10Op>>(converter, "__nv_log10f",
"__nv_log10"); "__nv_log10");
patterns.insert<OpToFuncCallLowering<math::Log2Op>>(converter, "__nv_log2f", patterns.add<OpToFuncCallLowering<math::Log2Op>>(converter, "__nv_log2f",
"__nv_log2"); "__nv_log2");
patterns.insert<OpToFuncCallLowering<math::PowFOp>>(converter, "__nv_powf", patterns.add<OpToFuncCallLowering<math::PowFOp>>(converter, "__nv_powf",
"__nv_pow"); "__nv_pow");
patterns.insert<OpToFuncCallLowering<math::RsqrtOp>>(converter, "__nv_rsqrtf", patterns.add<OpToFuncCallLowering<math::RsqrtOp>>(converter, "__nv_rsqrtf",
"__nv_rsqrt"); "__nv_rsqrt");
patterns.insert<OpToFuncCallLowering<math::SinOp>>(converter, "__nv_sinf", patterns.add<OpToFuncCallLowering<math::SinOp>>(converter, "__nv_sinf",
"__nv_sin"); "__nv_sin");
patterns.insert<OpToFuncCallLowering<math::SqrtOp>>(converter, "__nv_sqrtf", patterns.add<OpToFuncCallLowering<math::SqrtOp>>(converter, "__nv_sqrtf",
"__nv_sqrt"); "__nv_sqrt");
patterns.insert<OpToFuncCallLowering<math::TanhOp>>(converter, "__nv_tanhf", patterns.add<OpToFuncCallLowering<math::TanhOp>>(converter, "__nv_tanhf",
"__nv_tanh"); "__nv_tanh");
} }
std::unique_ptr<OperationPass<gpu::GPUModuleOp>> std::unique_ptr<OperationPass<gpu::GPUModuleOp>>
mlir::createLowerGpuOpsToNVVMOpsPass(unsigned indexBitwidth) { mlir::createLowerGpuOpsToNVVMOpsPass(unsigned indexBitwidth) {
return std::make_unique<LowerGpuOpsToNVVMOpsPass>(indexBitwidth); return std::make_unique<LowerGpuOpsToNVVMOpsPass>(indexBitwidth);
} }

mlir/lib/Conversion/GPUToROCDL/LowerGpuOpsToROCDLOps.cpp

Show First 20 Lines • Show All 54 Lines • ▼ Show 20 Lines void runOnOperation() override {
/// Customize the bitwidth used for the device side index computations. /// Customize the bitwidth used for the device side index computations.
LowerToLLVMOptions options = {/*useBarePtrCallConv =*/false, LowerToLLVMOptions options = {/*useBarePtrCallConv =*/false,
/*emitCWrappers =*/true, /*emitCWrappers =*/true,
/*indexBitwidth =*/indexBitwidth, /*indexBitwidth =*/indexBitwidth,
/*useAlignedAlloc =*/false}; /*useAlignedAlloc =*/false};
LLVMTypeConverter converter(m.getContext(), options); LLVMTypeConverter converter(m.getContext(), options);
OwningRewritePatternList patterns(m.getContext()); RewritePatternSet patterns(m.getContext());
OwningRewritePatternList llvmPatterns(m.getContext()); RewritePatternSet llvmPatterns(m.getContext());
populateGpuRewritePatterns(patterns); populateGpuRewritePatterns(patterns);
(void)applyPatternsAndFoldGreedily(m, std::move(patterns)); (void)applyPatternsAndFoldGreedily(m, std::move(patterns));
populateVectorToLLVMConversionPatterns(converter, llvmPatterns); populateVectorToLLVMConversionPatterns(converter, llvmPatterns);
populateVectorToROCDLConversionPatterns(converter, llvmPatterns); populateVectorToROCDLConversionPatterns(converter, llvmPatterns);
populateStdToLLVMConversionPatterns(converter, llvmPatterns); populateStdToLLVMConversionPatterns(converter, llvmPatterns);
populateGpuToROCDLConversionPatterns(converter, llvmPatterns); populateGpuToROCDLConversionPatterns(converter, llvmPatterns);
Show All 14 Linesvoid mlir::configureGpuToROCDLConversionLegality(ConversionTarget &target) {
target.addIllegalOp<LLVM::CosOp, LLVM::ExpOp, LLVM::FAbsOp, LLVM::FCeilOp, target.addIllegalOp<LLVM::CosOp, LLVM::ExpOp, LLVM::FAbsOp, LLVM::FCeilOp,
LLVM::FFloorOp, LLVM::LogOp, LLVM::Log10Op, LLVM::Log2Op, LLVM::FFloorOp, LLVM::LogOp, LLVM::Log10Op, LLVM::Log2Op,
LLVM::PowOp, LLVM::SinOp, LLVM::SqrtOp>(); LLVM::PowOp, LLVM::SinOp, LLVM::SqrtOp>();
// TODO: Remove once we support replacing non-root ops. // TODO: Remove once we support replacing non-root ops.
target.addLegalOp<gpu::YieldOp, gpu::GPUModuleOp, gpu::ModuleEndOp>(); target.addLegalOp<gpu::YieldOp, gpu::GPUModuleOp, gpu::ModuleEndOp>();
} }
void mlir::populateGpuToROCDLConversionPatterns( void mlir::populateGpuToROCDLConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
populateWithGenerated(patterns); populateWithGenerated(patterns);
patterns.insert< patterns
GPUIndexIntrinsicOpLowering<gpu::ThreadIdOp, ROCDL::ThreadIdXOp, .add<GPUIndexIntrinsicOpLowering<gpu::ThreadIdOp, ROCDL::ThreadIdXOp,
ROCDL::ThreadIdYOp, ROCDL::ThreadIdZOp>, ROCDL::ThreadIdYOp, ROCDL::ThreadIdZOp>,
GPUIndexIntrinsicOpLowering<gpu::BlockDimOp, ROCDL::BlockDimXOp, GPUIndexIntrinsicOpLowering<gpu::BlockDimOp, ROCDL::BlockDimXOp,
ROCDL::BlockDimYOp, ROCDL::BlockDimZOp>, ROCDL::BlockDimYOp, ROCDL::BlockDimZOp>,
GPUIndexIntrinsicOpLowering<gpu::BlockIdOp, ROCDL::BlockIdXOp, GPUIndexIntrinsicOpLowering<gpu::BlockIdOp, ROCDL::BlockIdXOp,
ROCDL::BlockIdYOp, ROCDL::BlockIdZOp>, ROCDL::BlockIdYOp, ROCDL::BlockIdZOp>,
GPUIndexIntrinsicOpLowering<gpu::GridDimOp, ROCDL::GridDimXOp, GPUIndexIntrinsicOpLowering<gpu::GridDimOp, ROCDL::GridDimXOp,
ROCDL::GridDimYOp, ROCDL::GridDimZOp>, ROCDL::GridDimYOp, ROCDL::GridDimZOp>,
GPUReturnOpLowering>(converter); GPUReturnOpLowering>(converter);
patterns.insert<GPUFuncOpLowering>( patterns.add<GPUFuncOpLowering>(
converter, /*allocaAddrSpace=*/5, converter, /*allocaAddrSpace=*/5,
Identifier::get(ROCDL::ROCDLDialect::getKernelFuncAttrName(), Identifier::get(ROCDL::ROCDLDialect::getKernelFuncAttrName(),
&converter.getContext())); &converter.getContext()));
patterns.insert<OpToFuncCallLowering<AbsFOp>>(converter, "__ocml_fabs_f32", patterns.add<OpToFuncCallLowering<AbsFOp>>(converter, "__ocml_fabs_f32",
"__ocml_fabs_f64"); "__ocml_fabs_f64");
patterns.insert<OpToFuncCallLowering<math::AtanOp>>( patterns.add<OpToFuncCallLowering<math::AtanOp>>(converter, "__ocml_atan_f32",
converter, "__ocml_atan_f32", "__ocml_atan_f64"); "__ocml_atan_f64");
patterns.insert<OpToFuncCallLowering<math::Atan2Op>>( patterns.add<OpToFuncCallLowering<math::Atan2Op>>(
converter, "__ocml_atan2_f32", "__ocml_atan2_f64"); converter, "__ocml_atan2_f32", "__ocml_atan2_f64");
patterns.insert<OpToFuncCallLowering<CeilFOp>>(converter, "__ocml_ceil_f32", patterns.add<OpToFuncCallLowering<CeilFOp>>(converter, "__ocml_ceil_f32",
"__ocml_ceil_f64"); "__ocml_ceil_f64");
patterns.insert<OpToFuncCallLowering<math::CosOp>>( patterns.add<OpToFuncCallLowering<math::CosOp>>(converter, "__ocml_cos_f32",
converter, "__ocml_cos_f32", "__ocml_cos_f64"); "__ocml_cos_f64");
patterns.insert<OpToFuncCallLowering<math::ExpOp>>( patterns.add<OpToFuncCallLowering<math::ExpOp>>(converter, "__ocml_exp_f32",
converter, "__ocml_exp_f32", "__ocml_exp_f64"); "__ocml_exp_f64");
patterns.insert<OpToFuncCallLowering<math::ExpM1Op>>( patterns.add<OpToFuncCallLowering<math::ExpM1Op>>(
converter, "__ocml_expm1_f32", "__ocml_expm1_f64"); converter, "__ocml_expm1_f32", "__ocml_expm1_f64");
patterns.insert<OpToFuncCallLowering<FloorFOp>>(converter, "__ocml_floor_f32", patterns.add<OpToFuncCallLowering<FloorFOp>>(converter, "__ocml_floor_f32",
"__ocml_floor_f64"); "__ocml_floor_f64");
patterns.insert<OpToFuncCallLowering<math::LogOp>>( patterns.add<OpToFuncCallLowering<math::LogOp>>(converter, "__ocml_log_f32",
converter, "__ocml_log_f32", "__ocml_log_f64"); "__ocml_log_f64");
patterns.insert<OpToFuncCallLowering<math::Log10Op>>( patterns.add<OpToFuncCallLowering<math::Log10Op>>(
converter, "__ocml_log10_f32", "__ocml_log10_f64"); converter, "__ocml_log10_f32", "__ocml_log10_f64");
patterns.insert<OpToFuncCallLowering<math::Log1pOp>>( patterns.add<OpToFuncCallLowering<math::Log1pOp>>(
converter, "__ocml_log1p_f32", "__ocml_log1p_f64"); converter, "__ocml_log1p_f32", "__ocml_log1p_f64");
patterns.insert<OpToFuncCallLowering<math::Log2Op>>( patterns.add<OpToFuncCallLowering<math::Log2Op>>(converter, "__ocml_log2_f32",
converter, "__ocml_log2_f32", "__ocml_log2_f64"); "__ocml_log2_f64");
patterns.insert<OpToFuncCallLowering<math::PowFOp>>( patterns.add<OpToFuncCallLowering<math::PowFOp>>(converter, "__ocml_pow_f32",
converter, "__ocml_pow_f32", "__ocml_pow_f64"); "__ocml_pow_f64");
patterns.insert<OpToFuncCallLowering<math::RsqrtOp>>( patterns.add<OpToFuncCallLowering<math::RsqrtOp>>(
converter, "__ocml_rsqrt_f32", "__ocml_rsqrt_f64"); converter, "__ocml_rsqrt_f32", "__ocml_rsqrt_f64");
patterns.insert<OpToFuncCallLowering<math::SinOp>>( patterns.add<OpToFuncCallLowering<math::SinOp>>(converter, "__ocml_sin_f32",
converter, "__ocml_sin_f32", "__ocml_sin_f64"); "__ocml_sin_f64");
patterns.insert<OpToFuncCallLowering<math::SqrtOp>>( patterns.add<OpToFuncCallLowering<math::SqrtOp>>(converter, "__ocml_sqrt_f32",
converter, "__ocml_sqrt_f32", "__ocml_sqrt_f64"); "__ocml_sqrt_f64");
patterns.insert<OpToFuncCallLowering<math::TanhOp>>( patterns.add<OpToFuncCallLowering<math::TanhOp>>(converter, "__ocml_tanh_f32",
converter, "__ocml_tanh_f32", "__ocml_tanh_f64"); "__ocml_tanh_f64");
} }
std::unique_ptr<OperationPass<gpu::GPUModuleOp>> std::unique_ptr<OperationPass<gpu::GPUModuleOp>>
mlir::createLowerGpuOpsToROCDLOpsPass(unsigned indexBitwidth) { mlir::createLowerGpuOpsToROCDLOpsPass(unsigned indexBitwidth) {
return std::make_unique<LowerGpuOpsToROCDLOpsPass>(indexBitwidth); return std::make_unique<LowerGpuOpsToROCDLOpsPass>(indexBitwidth);
} }

mlir/lib/Conversion/GPUToSPIRV/GPUToSPIRV.cpp

Show First 20 Lines • Show All 324 Lines • ▼ Show 20 Lines
// GPU To SPIRV Patterns. // GPU To SPIRV Patterns.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace { namespace {
#include "GPUToSPIRV.cpp.inc" #include "GPUToSPIRV.cpp.inc"
} }
void mlir::populateGPUToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void mlir::populateGPUToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
populateWithGenerated(patterns); populateWithGenerated(patterns);
patterns.insert< patterns.add<
GPUFuncOpConversion, GPUModuleConversion, GPUReturnOpConversion, GPUFuncOpConversion, GPUModuleConversion, GPUReturnOpConversion,
LaunchConfigConversion<gpu::BlockIdOp, spirv::BuiltIn::WorkgroupId>, LaunchConfigConversion<gpu::BlockIdOp, spirv::BuiltIn::WorkgroupId>,
LaunchConfigConversion<gpu::GridDimOp, spirv::BuiltIn::NumWorkgroups>, LaunchConfigConversion<gpu::GridDimOp, spirv::BuiltIn::NumWorkgroups>,
LaunchConfigConversion<gpu::ThreadIdOp, LaunchConfigConversion<gpu::ThreadIdOp,
spirv::BuiltIn::LocalInvocationId>, spirv::BuiltIn::LocalInvocationId>,
SingleDimLaunchConfigConversion<gpu::SubgroupIdOp, SingleDimLaunchConfigConversion<gpu::SubgroupIdOp,
spirv::BuiltIn::SubgroupId>, spirv::BuiltIn::SubgroupId>,
SingleDimLaunchConfigConversion<gpu::NumSubgroupsOp, SingleDimLaunchConfigConversion<gpu::NumSubgroupsOp,
spirv::BuiltIn::NumSubgroups>, spirv::BuiltIn::NumSubgroups>,
SingleDimLaunchConfigConversion<gpu::SubgroupSizeOp, SingleDimLaunchConfigConversion<gpu::SubgroupSizeOp,
spirv::BuiltIn::SubgroupSize>, spirv::BuiltIn::SubgroupSize>,
WorkGroupSizeConversion>(typeConverter, patterns.getContext()); WorkGroupSizeConversion>(typeConverter, patterns.getContext());
} }

mlir/lib/Conversion/GPUToSPIRV/GPUToSPIRVPass.cpp

Show First 20 Lines • Show All 51 Lines • ▼ Show 20 Lines module.walk([&builder, &kernelModules](gpu::GPUModuleOp moduleOp) {
kernelModules.push_back(builder.clone(*moduleOp.getOperation())); kernelModules.push_back(builder.clone(*moduleOp.getOperation()));
}); });
auto targetAttr = spirv::lookupTargetEnvOrDefault(module); auto targetAttr = spirv::lookupTargetEnvOrDefault(module);
std::unique_ptr<ConversionTarget> target = std::unique_ptr<ConversionTarget> target =
spirv::SPIRVConversionTarget::get(targetAttr); spirv::SPIRVConversionTarget::get(targetAttr);
SPIRVTypeConverter typeConverter(targetAttr); SPIRVTypeConverter typeConverter(targetAttr);
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
populateGPUToSPIRVPatterns(typeConverter, patterns); populateGPUToSPIRVPatterns(typeConverter, patterns);
populateStandardToSPIRVPatterns(typeConverter, patterns); populateStandardToSPIRVPatterns(typeConverter, patterns);
if (failed(applyFullConversion(kernelModules, *target, std::move(patterns)))) if (failed(applyFullConversion(kernelModules, *target, std::move(patterns))))
return signalPassFailure(); return signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertGPUToSPIRVPass() { std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertGPUToSPIRVPass() {
return std::make_unique<GPUToSPIRVPass>(); return std::make_unique<GPUToSPIRVPass>();
} }

mlir/lib/Conversion/LinalgToLLVM/LinalgToLLVM.cpp

Show First 20 Lines • Show All 194 Lines • ▼ Show 20 Lines matchAndRewrite(linalg::YieldOp op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
rewriter.replaceOpWithNewOp<LLVM::ReturnOp>(op, operands); rewriter.replaceOpWithNewOp<LLVM::ReturnOp>(op, operands);
return success(); return success();
} }
}; };
} // namespace } // namespace
/// Populate the given list with patterns that convert from Linalg to LLVM. /// Populate the given list with patterns that convert from Linalg to LLVM.
void mlir::populateLinalgToLLVMConversionPatterns( void mlir::populateLinalgToLLVMConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<RangeOpConversion, ReshapeOpConversion, YieldOpConversion>( patterns.add<RangeOpConversion, ReshapeOpConversion, YieldOpConversion>(
converter); converter);
// Populate the type conversions for the linalg types. // Populate the type conversions for the linalg types.
converter.addConversion( converter.addConversion(
[&](RangeType type) { return convertRangeType(type, converter); }); [&](RangeType type) { return convertRangeType(type, converter); });
} }
namespace { namespace {
struct ConvertLinalgToLLVMPass struct ConvertLinalgToLLVMPass
: public ConvertLinalgToLLVMBase<ConvertLinalgToLLVMPass> { : public ConvertLinalgToLLVMBase<ConvertLinalgToLLVMPass> {
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void ConvertLinalgToLLVMPass::runOnOperation() { void ConvertLinalgToLLVMPass::runOnOperation() {
auto module = getOperation(); auto module = getOperation();
// Convert to the LLVM IR dialect using the converter defined above. // Convert to the LLVM IR dialect using the converter defined above.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
LLVMTypeConverter converter(&getContext()); LLVMTypeConverter converter(&getContext());
populateLinalgToLLVMConversionPatterns(converter, patterns); populateLinalgToLLVMConversionPatterns(converter, patterns);
LLVMConversionTarget target(getContext()); LLVMConversionTarget target(getContext());
target.addIllegalOp<RangeOp>(); target.addIllegalOp<RangeOp>();
target.addLegalOp<ModuleOp, ModuleTerminatorOp, LLVM::DialectCastOp>(); target.addLegalOp<ModuleOp, ModuleTerminatorOp, LLVM::DialectCastOp>();
if (failed(applyPartialConversion(module, target, std::move(patterns)))) if (failed(applyPartialConversion(module, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertLinalgToLLVMPass() { std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertLinalgToLLVMPass() {
return std::make_unique<ConvertLinalgToLLVMPass>(); return std::make_unique<ConvertLinalgToLLVMPass>();
} }

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

Show First 20 Lines • Show All 198 Lines • ▼ Show 20 Lines#undef CREATE_ATOMIC_BIN_OP
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pattern population // Pattern population
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void mlir::populateLinalgToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void mlir::populateLinalgToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<SingleWorkgroupReduction>(typeConverter, patterns.add<SingleWorkgroupReduction>(typeConverter, patterns.getContext());
patterns.getContext());
} }

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRVPass.cpp

Show All 24 Linesvoid LinalgToSPIRVPass::runOnOperation() {
MLIRContext *context = &getContext(); MLIRContext *context = &getContext();
ModuleOp module = getOperation(); ModuleOp module = getOperation();
auto targetAttr = spirv::lookupTargetEnvOrDefault(module); auto targetAttr = spirv::lookupTargetEnvOrDefault(module);
std::unique_ptr<ConversionTarget> target = std::unique_ptr<ConversionTarget> target =
spirv::SPIRVConversionTarget::get(targetAttr); spirv::SPIRVConversionTarget::get(targetAttr);
SPIRVTypeConverter typeConverter(targetAttr); SPIRVTypeConverter typeConverter(targetAttr);
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
populateLinalgToSPIRVPatterns(typeConverter, patterns); populateLinalgToSPIRVPatterns(typeConverter, patterns);
populateBuiltinFuncToSPIRVPatterns(typeConverter, patterns); populateBuiltinFuncToSPIRVPatterns(typeConverter, patterns);
// Allow builtin ops. // Allow builtin ops.
target->addLegalOp<ModuleOp, ModuleTerminatorOp>(); target->addLegalOp<ModuleOp, ModuleTerminatorOp>();
target->addDynamicallyLegalOp<FuncOp>([&](FuncOp op) { target->addDynamicallyLegalOp<FuncOp>([&](FuncOp op) {
return typeConverter.isSignatureLegal(op.getType()) && return typeConverter.isSignatureLegal(op.getType()) &&
typeConverter.isLegal(&op.getBody()); typeConverter.isLegal(&op.getBody());
Show All 9 Lines

mlir/lib/Conversion/LinalgToStandard/LinalgToStandard.cpp

Show First 20 Lines • Show All 186 Lines • ▼ Show 20 Linesmlir::linalg::IndexedGenericOpToLibraryCallRewrite::matchAndRewrite(
rewriter.replaceOpWithNewOp<mlir::CallOp>( rewriter.replaceOpWithNewOp<mlir::CallOp>(
op, libraryCallName.getValue(), TypeRange(), op, libraryCallName.getValue(), TypeRange(),
createTypeCanonicalizedMemRefOperands(rewriter, op.getLoc(), operands)); createTypeCanonicalizedMemRefOperands(rewriter, op.getLoc(), operands));
return success(); return success();
} }
/// Populate the given list with patterns that convert from Linalg to Standard. /// Populate the given list with patterns that convert from Linalg to Standard.
void mlir::linalg::populateLinalgToStandardConversionPatterns( void mlir::linalg::populateLinalgToStandardConversionPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
// TODO: ConvOp conversion needs to export a descriptor with relevant // TODO: ConvOp conversion needs to export a descriptor with relevant
// attribute values such as kernel striding and dilation. // attribute values such as kernel striding and dilation.
// clang-format off // clang-format off
patterns.insert< patterns.add<
CopyOpToLibraryCallRewrite, CopyOpToLibraryCallRewrite,
CopyTransposeRewrite, CopyTransposeRewrite,
IndexedGenericOpToLibraryCallRewrite>(patterns.getContext()); IndexedGenericOpToLibraryCallRewrite>(patterns.getContext());
patterns.insert<LinalgOpToLibraryCallRewrite>(); patterns.add<LinalgOpToLibraryCallRewrite>();
// clang-format on // clang-format on
} }
namespace { namespace {
struct ConvertLinalgToStandardPass struct ConvertLinalgToStandardPass
: public ConvertLinalgToStandardBase<ConvertLinalgToStandardPass> { : public ConvertLinalgToStandardBase<ConvertLinalgToStandardPass> {
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void ConvertLinalgToStandardPass::runOnOperation() { void ConvertLinalgToStandardPass::runOnOperation() {
auto module = getOperation(); auto module = getOperation();
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addLegalDialect<AffineDialect, memref::MemRefDialect, scf::SCFDialect, target.addLegalDialect<AffineDialect, memref::MemRefDialect, scf::SCFDialect,
StandardOpsDialect>(); StandardOpsDialect>();
target.addLegalOp<ModuleOp, FuncOp, ModuleTerminatorOp, ReturnOp>(); target.addLegalOp<ModuleOp, FuncOp, ModuleTerminatorOp, ReturnOp>();
target.addLegalOp<linalg::ReshapeOp, linalg::RangeOp>(); target.addLegalOp<linalg::ReshapeOp, linalg::RangeOp>();
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateLinalgToStandardConversionPatterns(patterns); populateLinalgToStandardConversionPatterns(patterns);
if (failed(applyFullConversion(module, target, std::move(patterns)))) if (failed(applyFullConversion(module, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> std::unique_ptr<OperationPass<ModuleOp>>
mlir::createConvertLinalgToStandardPass() { mlir::createConvertLinalgToStandardPass() {
return std::make_unique<ConvertLinalgToStandardPass>(); return std::make_unique<ConvertLinalgToStandardPass>();
} }

mlir/lib/Conversion/OpenMPToLLVM/OpenMPToLLVM.cpp

Show All 35 Lines if (failed(rewriter.convertRegionTypes(&newOp.region(),
return failure(); return failure();
rewriter.eraseOp(curOp); rewriter.eraseOp(curOp);
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateOpenMPToLLVMConversionPatterns( void mlir::populateOpenMPToLLVMConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<RegionOpConversion<omp::ParallelOp>, patterns.add<RegionOpConversion<omp::ParallelOp>,
RegionOpConversion<omp::WsLoopOp>>(converter); RegionOpConversion<omp::WsLoopOp>>(converter);
} }
namespace { namespace {
struct ConvertOpenMPToLLVMPass struct ConvertOpenMPToLLVMPass
: public ConvertOpenMPToLLVMBase<ConvertOpenMPToLLVMPass> { : public ConvertOpenMPToLLVMBase<ConvertOpenMPToLLVMPass> {
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void ConvertOpenMPToLLVMPass::runOnOperation() { void ConvertOpenMPToLLVMPass::runOnOperation() {
auto module = getOperation(); auto module = getOperation();
// Convert to OpenMP operations with LLVM IR dialect // Convert to OpenMP operations with LLVM IR dialect
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
LLVMTypeConverter converter(&getContext()); LLVMTypeConverter converter(&getContext());
populateStdToLLVMConversionPatterns(converter, patterns); populateStdToLLVMConversionPatterns(converter, patterns);
populateOpenMPToLLVMConversionPatterns(converter, patterns); populateOpenMPToLLVMConversionPatterns(converter, patterns);
LLVMConversionTarget target(getContext()); LLVMConversionTarget target(getContext());
target.addDynamicallyLegalOp<omp::ParallelOp, omp::WsLoopOp>( target.addDynamicallyLegalOp<omp::ParallelOp, omp::WsLoopOp>(
[&](Operation *op) { return converter.isLegal(&op->getRegion(0)); }); [&](Operation *op) { return converter.isLegal(&op->getRegion(0)); });
target.addLegalOp<omp::TerminatorOp, omp::TaskyieldOp, omp::FlushOp, target.addLegalOp<omp::TerminatorOp, omp::TaskyieldOp, omp::FlushOp,
omp::BarrierOp, omp::TaskwaitOp>(); omp::BarrierOp, omp::TaskwaitOp>();
if (failed(applyPartialConversion(module, target, std::move(patterns)))) if (failed(applyPartialConversion(module, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertOpenMPToLLVMPass() { std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertOpenMPToLLVMPass() {
return std::make_unique<ConvertOpenMPToLLVMPass>(); return std::make_unique<ConvertOpenMPToLLVMPass>();
} }

mlir/lib/Conversion/SCFToGPU/SCFToGPU.cpp

Show First 20 Lines • Show All 636 Lines • ▼ Show 20 LinesParallelToGpuLaunchLowering::matchAndRewrite(ParallelOp parallelOp,
for (auto bound : launchBounds) for (auto bound : launchBounds)
launchOp.setOperand(getLaunchOpArgumentNum(std::get<0>(bound)), launchOp.setOperand(getLaunchOpArgumentNum(std::get<0>(bound)),
std::get<1>(bound)); std::get<1>(bound));
rewriter.eraseOp(parallelOp); rewriter.eraseOp(parallelOp);
return success(); return success();
} }
void mlir::populateParallelLoopToGPUPatterns( void mlir::populateParallelLoopToGPUPatterns(RewritePatternSet &patterns) {
OwningRewritePatternList &patterns) { patterns.add<ParallelToGpuLaunchLowering>(patterns.getContext());
patterns.insert<ParallelToGpuLaunchLowering>(patterns.getContext());
} }
void mlir::configureParallelLoopToGPULegality(ConversionTarget &target) { void mlir::configureParallelLoopToGPULegality(ConversionTarget &target) {
target.addLegalDialect<memref::MemRefDialect>(); target.addLegalDialect<memref::MemRefDialect>();
target.addDynamicallyLegalOp<scf::ParallelOp>([](scf::ParallelOp parallelOp) { target.addDynamicallyLegalOp<scf::ParallelOp>([](scf::ParallelOp parallelOp) {
return !parallelOp->getAttr(gpu::getMappingAttrName()); return !parallelOp->getAttr(gpu::getMappingAttrName());
}); });
} }

mlir/lib/Conversion/SCFToGPU/SCFToGPUPass.cpp

Show First 20 Lines • Show All 41 Lines • ▼ Show 20 Lines for (Operation &op : llvm::make_early_inc_range(getFunction().getOps())) {
} }
} }
} }
}; };
struct ParallelLoopToGpuPass struct ParallelLoopToGpuPass
: public ConvertParallelLoopToGpuBase<ParallelLoopToGpuPass> { : public ConvertParallelLoopToGpuBase<ParallelLoopToGpuPass> {
void runOnOperation() override { void runOnOperation() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateParallelLoopToGPUPatterns(patterns); populateParallelLoopToGPUPatterns(patterns);
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.markUnknownOpDynamicallyLegal([](Operation *) { return true; }); target.markUnknownOpDynamicallyLegal([](Operation *) { return true; });
configureParallelLoopToGPULegality(target); configureParallelLoopToGPULegality(target);
if (failed(applyPartialConversion(getOperation(), target, if (failed(applyPartialConversion(getOperation(), target,
std::move(patterns)))) std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
Show All 15 Lines

mlir/lib/Conversion/SCFToOpenMP/SCFToOpenMP.cpp

Show First 20 Lines • Show All 84 Lines • ▼ Show 20 Lines
/// Applies the conversion patterns in the given function. /// Applies the conversion patterns in the given function.
static LogicalResult applyPatterns(FuncOp func) { static LogicalResult applyPatterns(FuncOp func) {
ConversionTarget target(*func.getContext()); ConversionTarget target(*func.getContext());
target.addIllegalOp<scf::ParallelOp>(); target.addIllegalOp<scf::ParallelOp>();
target.addDynamicallyLegalOp<scf::YieldOp>( target.addDynamicallyLegalOp<scf::YieldOp>(
[](scf::YieldOp op) { return !isa<scf::ParallelOp>(op->getParentOp()); }); [](scf::YieldOp op) { return !isa<scf::ParallelOp>(op->getParentOp()); });
target.addLegalDialect<omp::OpenMPDialect>(); target.addLegalDialect<omp::OpenMPDialect>();
OwningRewritePatternList patterns(func.getContext()); RewritePatternSet patterns(func.getContext());
patterns.insert<ParallelOpLowering>(func.getContext()); patterns.add<ParallelOpLowering>(func.getContext());
FrozenRewritePatternList frozen(std::move(patterns)); FrozenRewritePatternList frozen(std::move(patterns));
return applyPartialConversion(func, target, frozen); return applyPartialConversion(func, target, frozen);
} }
/// A pass converting SCF operations to OpenMP operations. /// A pass converting SCF operations to OpenMP operations.
struct SCFToOpenMPPass : public ConvertSCFToOpenMPBase<SCFToOpenMPPass> { struct SCFToOpenMPPass : public ConvertSCFToOpenMPBase<SCFToOpenMPPass> {
/// Pass entry point. /// Pass entry point.
void runOnFunction() override { void runOnFunction() override {
Show All 11 Lines

mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRV.cpp

Show First 20 Lines • Show All 315 Lines • ▼ Show 20 Lines
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Hooks // Hooks
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void mlir::populateSCFToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void mlir::populateSCFToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
ScfToSPIRVContext &scfToSPIRVContext, ScfToSPIRVContext &scfToSPIRVContext,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<ForOpConversion, IfOpConversion, TerminatorOpConversion>( patterns.add<ForOpConversion, IfOpConversion, TerminatorOpConversion>(
patterns.getContext(), typeConverter, scfToSPIRVContext.getImpl()); patterns.getContext(), typeConverter, scfToSPIRVContext.getImpl());
} }

mlir/lib/Conversion/SCFToSPIRV/SCFToSPIRVPass.cpp

Show All 31 Linesvoid SCFToSPIRVPass::runOnOperation() {
ModuleOp module = getOperation(); ModuleOp module = getOperation();
auto targetAttr = spirv::lookupTargetEnvOrDefault(module); auto targetAttr = spirv::lookupTargetEnvOrDefault(module);
std::unique_ptr<ConversionTarget> target = std::unique_ptr<ConversionTarget> target =
spirv::SPIRVConversionTarget::get(targetAttr); spirv::SPIRVConversionTarget::get(targetAttr);
SPIRVTypeConverter typeConverter(targetAttr); SPIRVTypeConverter typeConverter(targetAttr);
ScfToSPIRVContext scfContext; ScfToSPIRVContext scfContext;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
populateSCFToSPIRVPatterns(typeConverter, scfContext, patterns); populateSCFToSPIRVPatterns(typeConverter, scfContext, patterns);
populateStandardToSPIRVPatterns(typeConverter, patterns); populateStandardToSPIRVPatterns(typeConverter, patterns);
populateBuiltinFuncToSPIRVPatterns(typeConverter, patterns); populateBuiltinFuncToSPIRVPatterns(typeConverter, patterns);
if (failed(applyPartialConversion(module, *target, std::move(patterns)))) if (failed(applyPartialConversion(module, *target, std::move(patterns))))
return signalPassFailure(); return signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertSCFToSPIRVPass() { std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertSCFToSPIRVPass() {
return std::make_unique<SCFToSPIRVPass>(); return std::make_unique<SCFToSPIRVPass>();
} }

mlir/lib/Conversion/SCFToStandard/SCFToStandard.cpp

Show First 20 Lines • Show All 562 Lines • ▼ Show 20 LinesDoWhileLowering::matchAndRewrite(WhileOp whileOp,
// Replace the op with values "yielded" from the "before" region, which are // Replace the op with values "yielded" from the "before" region, which are
// visible by dominance. // visible by dominance.
rewriter.replaceOp(whileOp, condOp.args()); rewriter.replaceOp(whileOp, condOp.args());
return success(); return success();
} }
void mlir::populateLoopToStdConversionPatterns( void mlir::populateLoopToStdConversionPatterns(RewritePatternSet &patterns) {
OwningRewritePatternList &patterns) { patterns.add<ForLowering, IfLowering, ParallelLowering, WhileLowering>(
patterns.insert<ForLowering, IfLowering, ParallelLowering, WhileLowering>(
patterns.getContext()); patterns.getContext());
patterns.insert<DoWhileLowering>(patterns.getContext(), /*benefit=*/2); patterns.add<DoWhileLowering>(patterns.getContext(), /*benefit=*/2);
} }
void SCFToStandardPass::runOnOperation() { void SCFToStandardPass::runOnOperation() {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateLoopToStdConversionPatterns(patterns); populateLoopToStdConversionPatterns(patterns);
// Configure conversion to lower out scf.for, scf.if, scf.parallel and // Configure conversion to lower out scf.for, scf.if, scf.parallel and
// scf.while. Anything else is fine. // scf.while. Anything else is fine.
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addIllegalOp<scf::ForOp, scf::IfOp, scf::ParallelOp, scf::WhileOp>(); target.addIllegalOp<scf::ForOp, scf::IfOp, scf::ParallelOp, scf::WhileOp>();
target.markUnknownOpDynamicallyLegal([](Operation *) { return true; }); target.markUnknownOpDynamicallyLegal([](Operation *) { return true; });
if (failed( if (failed(
applyPartialConversion(getOperation(), target, std::move(patterns)))) applyPartialConversion(getOperation(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
std::unique_ptr<Pass> mlir::createLowerToCFGPass() { std::unique_ptr<Pass> mlir::createLowerToCFGPass() {
return std::make_unique<SCFToStandardPass>(); return std::make_unique<SCFToStandardPass>();
} }

mlir/lib/Conversion/SPIRVToLLVM/ConvertLaunchFuncToLLVMCalls.cpp

Show First 20 Lines • Show All 272 Lines • ▼ Show 20 Lines void runOnOperation() override {
// Specify options to lower Standard to LLVM and pull in the conversion // Specify options to lower Standard to LLVM and pull in the conversion
// patterns. // patterns.
LowerToLLVMOptions options = { LowerToLLVMOptions options = {
/*useBarePtrCallConv=*/false, /*useBarePtrCallConv=*/false,
/*emitCWrappers=*/true, /*emitCWrappers=*/true,
/*indexBitwidth=*/kDeriveIndexBitwidthFromDataLayout}; /*indexBitwidth=*/kDeriveIndexBitwidthFromDataLayout};
auto *context = module.getContext(); auto *context = module.getContext();
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
LLVMTypeConverter typeConverter(context, options); LLVMTypeConverter typeConverter(context, options);
populateStdToLLVMConversionPatterns(typeConverter, patterns); populateStdToLLVMConversionPatterns(typeConverter, patterns);
patterns.insert<GPULaunchLowering>(typeConverter); patterns.add<GPULaunchLowering>(typeConverter);
// Pull in SPIR-V type conversion patterns to convert SPIR-V global // Pull in SPIR-V type conversion patterns to convert SPIR-V global
// variable's type to LLVM dialect type. // variable's type to LLVM dialect type.
populateSPIRVToLLVMTypeConversion(typeConverter); populateSPIRVToLLVMTypeConversion(typeConverter);
ConversionTarget target(*context); ConversionTarget target(*context);
target.addLegalDialect<LLVM::LLVMDialect>(); target.addLegalDialect<LLVM::LLVMDialect>();
if (failed(applyPartialConversion(module, target, std::move(patterns)))) if (failed(applyPartialConversion(module, target, std::move(patterns))))
Show All 14 Lines

mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp

Show First 20 Lines • Show All 1,379 Lines • ▼ Show 20 Lines typeConverter.addConversion([&](spirv::RuntimeArrayType type) {
return convertRuntimeArrayType(type, typeConverter); return convertRuntimeArrayType(type, typeConverter);
}); });
typeConverter.addConversion([&](spirv::StructType type) { typeConverter.addConversion([&](spirv::StructType type) {
return convertStructType(type, typeConverter); return convertStructType(type, typeConverter);
}); });
} }
void mlir::populateSPIRVToLLVMConversionPatterns( void mlir::populateSPIRVToLLVMConversionPatterns(
LLVMTypeConverter &typeConverter, OwningRewritePatternList &patterns) { LLVMTypeConverter &typeConverter, RewritePatternSet &patterns) {
patterns.insert< patterns.add<
// Arithmetic ops // Arithmetic ops
DirectConversionPattern<spirv::IAddOp, LLVM::AddOp>, DirectConversionPattern<spirv::IAddOp, LLVM::AddOp>,
DirectConversionPattern<spirv::IMulOp, LLVM::MulOp>, DirectConversionPattern<spirv::IMulOp, LLVM::MulOp>,
DirectConversionPattern<spirv::ISubOp, LLVM::SubOp>, DirectConversionPattern<spirv::ISubOp, LLVM::SubOp>,
DirectConversionPattern<spirv::FAddOp, LLVM::FAddOp>, DirectConversionPattern<spirv::FAddOp, LLVM::FAddOp>,
DirectConversionPattern<spirv::FDivOp, LLVM::FDivOp>, DirectConversionPattern<spirv::FDivOp, LLVM::FDivOp>,
DirectConversionPattern<spirv::FMulOp, LLVM::FMulOp>, DirectConversionPattern<spirv::FMulOp, LLVM::FMulOp>,
DirectConversionPattern<spirv::FNegateOp, LLVM::FNegOp>, DirectConversionPattern<spirv::FNegateOp, LLVM::FNegOp>,
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Lines patterns.add<
ShiftPattern<spirv::ShiftRightLogicalOp, LLVM::LShrOp>, ShiftPattern<spirv::ShiftRightLogicalOp, LLVM::LShrOp>,
ShiftPattern<spirv::ShiftLeftLogicalOp, LLVM::ShlOp>, ShiftPattern<spirv::ShiftLeftLogicalOp, LLVM::ShlOp>,
// Return ops // Return ops
ReturnPattern, ReturnValuePattern>(patterns.getContext(), typeConverter); ReturnPattern, ReturnValuePattern>(patterns.getContext(), typeConverter);
} }
void mlir::populateSPIRVToLLVMFunctionConversionPatterns( void mlir::populateSPIRVToLLVMFunctionConversionPatterns(
LLVMTypeConverter &typeConverter, OwningRewritePatternList &patterns) { LLVMTypeConverter &typeConverter, RewritePatternSet &patterns) {
patterns.insert<FuncConversionPattern>(patterns.getContext(), typeConverter); patterns.add<FuncConversionPattern>(patterns.getContext(), typeConverter);
} }
void mlir::populateSPIRVToLLVMModuleConversionPatterns( void mlir::populateSPIRVToLLVMModuleConversionPatterns(
LLVMTypeConverter &typeConverter, OwningRewritePatternList &patterns) { LLVMTypeConverter &typeConverter, RewritePatternSet &patterns) {
patterns.insert<ModuleConversionPattern, ModuleEndConversionPattern>( patterns.add<ModuleConversionPattern, ModuleEndConversionPattern>(
patterns.getContext(), typeConverter); patterns.getContext(), typeConverter);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pre-conversion hooks // Pre-conversion hooks
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Hook for descriptor set and binding number encoding. /// Hook for descriptor set and binding number encoding.
Show All 34 Lines

mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVMPass.cpp

Show All 30 Lines
void ConvertSPIRVToLLVMPass::runOnOperation() { void ConvertSPIRVToLLVMPass::runOnOperation() {
MLIRContext *context = &getContext(); MLIRContext *context = &getContext();
ModuleOp module = getOperation(); ModuleOp module = getOperation();
LLVMTypeConverter converter(&getContext()); LLVMTypeConverter converter(&getContext());
// Encode global variable's descriptor set and binding if they exist. // Encode global variable's descriptor set and binding if they exist.
encodeBindAttribute(module); encodeBindAttribute(module);
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
populateSPIRVToLLVMTypeConversion(converter); populateSPIRVToLLVMTypeConversion(converter);
populateSPIRVToLLVMModuleConversionPatterns(converter, patterns); populateSPIRVToLLVMModuleConversionPatterns(converter, patterns);
populateSPIRVToLLVMConversionPatterns(converter, patterns); populateSPIRVToLLVMConversionPatterns(converter, patterns);
populateSPIRVToLLVMFunctionConversionPatterns(converter, patterns); populateSPIRVToLLVMFunctionConversionPatterns(converter, patterns);
ConversionTarget target(*context); ConversionTarget target(*context);
Show All 14 Lines

mlir/lib/Conversion/ShapeToStandard/ConvertShapeConstraints.cpp

Show All 31 Lines LogicalResult matchAndRewrite(shape::CstrRequireOp op,
rewriter.create<AssertOp>(op.getLoc(), op.pred(), op.msgAttr()); rewriter.create<AssertOp>(op.getLoc(), op.pred(), op.msgAttr());
rewriter.replaceOpWithNewOp<shape::ConstWitnessOp>(op, true); rewriter.replaceOpWithNewOp<shape::ConstWitnessOp>(op, true);
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateConvertShapeConstraintsConversionPatterns( void mlir::populateConvertShapeConstraintsConversionPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<CstrBroadcastableToRequire>(patterns.getContext()); patterns.add<CstrBroadcastableToRequire>(patterns.getContext());
patterns.insert<CstrEqToRequire>(patterns.getContext()); patterns.add<CstrEqToRequire>(patterns.getContext());
patterns.insert<ConvertCstrRequireOp>(patterns.getContext()); patterns.add<ConvertCstrRequireOp>(patterns.getContext());
} }
namespace { namespace {
// This pass eliminates shape constraints from the program, converting them to // This pass eliminates shape constraints from the program, converting them to
// eager (side-effecting) error handling code. After eager error handling code // eager (side-effecting) error handling code. After eager error handling code
// is emitted, witnesses are satisfied, so they are replace with // is emitted, witnesses are satisfied, so they are replace with
// `shape.const_witness true`. // `shape.const_witness true`.
class ConvertShapeConstraints class ConvertShapeConstraints
: public ConvertShapeConstraintsBase<ConvertShapeConstraints> { : public ConvertShapeConstraintsBase<ConvertShapeConstraints> {
void runOnOperation() override { void runOnOperation() override {
auto func = getOperation(); auto func = getOperation();
auto *context = &getContext(); auto *context = &getContext();
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
populateConvertShapeConstraintsConversionPatterns(patterns); populateConvertShapeConstraintsConversionPatterns(patterns);
if (failed(applyPatternsAndFoldGreedily(func, std::move(patterns)))) if (failed(applyPatternsAndFoldGreedily(func, std::move(patterns))))
return signalPassFailure(); return signalPassFailure();
} }
}; };
} // namespace } // namespace
std::unique_ptr<OperationPass<FuncOp>> std::unique_ptr<OperationPass<FuncOp>>
mlir::createConvertShapeConstraintsPass() { mlir::createConvertShapeConstraintsPass() {
return std::make_unique<ConvertShapeConstraints>(); return std::make_unique<ConvertShapeConstraints>();
} }

mlir/lib/Conversion/ShapeToStandard/ShapeToStandard.cpp

Show First 20 Lines • Show All 672 Lines • ▼ Show 20 Linesvoid ConvertShapeToStandardPass::runOnOperation() {
// Setup target legality. // Setup target legality.
MLIRContext &ctx = getContext(); MLIRContext &ctx = getContext();
ConversionTarget target(ctx); ConversionTarget target(ctx);
target.addLegalDialect<memref::MemRefDialect, StandardOpsDialect, SCFDialect, target.addLegalDialect<memref::MemRefDialect, StandardOpsDialect, SCFDialect,
tensor::TensorDialect>(); tensor::TensorDialect>();
target.addLegalOp<CstrRequireOp, FuncOp, ModuleOp, ModuleTerminatorOp>(); target.addLegalOp<CstrRequireOp, FuncOp, ModuleOp, ModuleTerminatorOp>();
// Setup conversion patterns. // Setup conversion patterns.
OwningRewritePatternList patterns(&ctx); RewritePatternSet patterns(&ctx);
populateShapeToStandardConversionPatterns(patterns); populateShapeToStandardConversionPatterns(patterns);
// Apply conversion. // Apply conversion.
auto module = getOperation(); auto module = getOperation();
if (failed(applyPartialConversion(module, target, std::move(patterns)))) if (failed(applyPartialConversion(module, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
void mlir::populateShapeToStandardConversionPatterns( void mlir::populateShapeToStandardConversionPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
// clang-format off // clang-format off
populateWithGenerated(patterns); populateWithGenerated(patterns);
patterns.insert< patterns.add<
AnyOpConversion, AnyOpConversion,
BinaryOpConversion<AddOp, AddIOp>, BinaryOpConversion<AddOp, AddIOp>,
BinaryOpConversion<MulOp, MulIOp>, BinaryOpConversion<MulOp, MulIOp>,
BroadcastOpConverter, BroadcastOpConverter,
ConstShapeOpConverter, ConstShapeOpConverter,
ConstSizeOpConversion, ConstSizeOpConversion,
IsBroadcastableOpConverter, IsBroadcastableOpConverter,
GetExtentOpConverter, GetExtentOpConverter,
Show All 13 Lines

mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp

Show First 20 Lines • Show All 3,850 Lines • ▼ Show 20 Lines for (auto *it : opsToErase)
rewriter.eraseOp(it); rewriter.eraseOp(it);
} }
}; };
} // namespace } // namespace
/// Collect a set of patterns to convert from the Standard dialect to LLVM. /// Collect a set of patterns to convert from the Standard dialect to LLVM.
void mlir::populateStdToLLVMNonMemoryConversionPatterns( void mlir::populateStdToLLVMNonMemoryConversionPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, RewritePatternSet &patterns) {
// FIXME: this should be tablegen'ed // FIXME: this should be tablegen'ed
// clang-format off // clang-format off
patterns.insert< patterns.add<
AbsFOpLowering, AbsFOpLowering,
AddFOpLowering, AddFOpLowering,
AddIOpLowering, AddIOpLowering,
AllocaOpLowering, AllocaOpLowering,
AndOpLowering, AndOpLowering,
AssertOpLowering, AssertOpLowering,
AtomicRMWOpLowering, AtomicRMWOpLowering,
BranchOpLowering, BranchOpLowering,
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Lines patterns.add<
UnsignedRemIOpLowering, UnsignedRemIOpLowering,
UnsignedShiftRightOpLowering, UnsignedShiftRightOpLowering,
XOrOpLowering, XOrOpLowering,
ZeroExtendIOpLowering>(converter); ZeroExtendIOpLowering>(converter);
// clang-format on // clang-format on
} }
void mlir::populateStdToLLVMMemoryConversionPatterns( void mlir::populateStdToLLVMMemoryConversionPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, RewritePatternSet &patterns) {
// clang-format off // clang-format off
patterns.insert< patterns.add<
AssumeAlignmentOpLowering, AssumeAlignmentOpLowering,
DeallocOpLowering, DeallocOpLowering,
DimOpLowering, DimOpLowering,
GlobalMemrefOpLowering, GlobalMemrefOpLowering,
GetGlobalMemrefOpLowering, GetGlobalMemrefOpLowering,
LoadOpLowering, LoadOpLowering,
MemRefCastOpLowering, MemRefCastOpLowering,
MemRefReinterpretCastOpLowering, MemRefReinterpretCastOpLowering,
MemRefReshapeOpLowering, MemRefReshapeOpLowering,
RankOpLowering, RankOpLowering,
StoreOpLowering, StoreOpLowering,
SubViewOpLowering, SubViewOpLowering,
TransposeOpLowering, TransposeOpLowering,
ViewOpLowering>(converter); ViewOpLowering>(converter);
// clang-format on // clang-format on
if (converter.getOptions().useAlignedAlloc) if (converter.getOptions().useAlignedAlloc)
patterns.insert<AlignedAllocOpLowering>(converter); patterns.add<AlignedAllocOpLowering>(converter);
else else
patterns.insert<AllocOpLowering>(converter); patterns.add<AllocOpLowering>(converter);
} }
void mlir::populateStdToLLVMFuncOpConversionPattern( void mlir::populateStdToLLVMFuncOpConversionPattern(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, RewritePatternSet &patterns) {
if (converter.getOptions().useBarePtrCallConv) if (converter.getOptions().useBarePtrCallConv)
patterns.insert<BarePtrFuncOpConversion>(converter); patterns.add<BarePtrFuncOpConversion>(converter);
else else
patterns.insert<FuncOpConversion>(converter); patterns.add<FuncOpConversion>(converter);
} }
void mlir::populateStdToLLVMConversionPatterns( void mlir::populateStdToLLVMConversionPatterns(LLVMTypeConverter &converter,
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
populateStdToLLVMFuncOpConversionPattern(converter, patterns); populateStdToLLVMFuncOpConversionPattern(converter, patterns);
populateStdToLLVMNonMemoryConversionPatterns(converter, patterns); populateStdToLLVMNonMemoryConversionPatterns(converter, patterns);
populateStdToLLVMMemoryConversionPatterns(converter, patterns); populateStdToLLVMMemoryConversionPatterns(converter, patterns);
} }
/// Convert a non-empty list of types to be returned from a function into a /// Convert a non-empty list of types to be returned from a function into a
/// supported LLVM IR type. In particular, if more than one value is returned, /// supported LLVM IR type. In particular, if more than one value is returned,
/// create an LLVM IR structure type with elements that correspond to each of /// create an LLVM IR structure type with elements that correspond to each of
▲ Show 20 Lines • Show All 103 Lines • ▼ Show 20 Lines void runOnOperation() override {
ModuleOp m = getOperation(); ModuleOp m = getOperation();
LowerToLLVMOptions options = {useBarePtrCallConv, emitCWrappers, LowerToLLVMOptions options = {useBarePtrCallConv, emitCWrappers,
indexBitwidth, useAlignedAlloc, indexBitwidth, useAlignedAlloc,
llvm::DataLayout(this->dataLayout)}; llvm::DataLayout(this->dataLayout)};
LLVMTypeConverter typeConverter(&getContext(), options); LLVMTypeConverter typeConverter(&getContext(), options);
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateStdToLLVMConversionPatterns(typeConverter, patterns); populateStdToLLVMConversionPatterns(typeConverter, patterns);
LLVMConversionTarget target(getContext()); LLVMConversionTarget target(getContext());
if (failed(applyPartialConversion(m, target, std::move(patterns)))) if (failed(applyPartialConversion(m, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
m->setAttr(LLVM::LLVMDialect::getDataLayoutAttrName(), m->setAttr(LLVM::LLVMDialect::getDataLayoutAttrName(),
StringAttr::get(m.getContext(), this->dataLayout)); StringAttr::get(m.getContext(), this->dataLayout));
} }
Show All 16 Lines

mlir/lib/Conversion/StandardToSPIRV/LegalizeStandardForSPIRV.cpp

Show First 20 Lines • Show All 187 Lines • ▼ Show 20 LinesStoreOpOfSubViewFolder<OpTy>::matchAndRewrite(OpTy storeOp,
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Hook for adding patterns. // Hook for adding patterns.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void mlir::populateStdLegalizationPatternsForSPIRVLowering( void mlir::populateStdLegalizationPatternsForSPIRVLowering(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<LoadOpOfSubViewFolder<memref::LoadOp>, patterns.add<LoadOpOfSubViewFolder<memref::LoadOp>,
LoadOpOfSubViewFolder<vector::TransferReadOp>, LoadOpOfSubViewFolder<vector::TransferReadOp>,
StoreOpOfSubViewFolder<memref::StoreOp>, StoreOpOfSubViewFolder<memref::StoreOp>,
StoreOpOfSubViewFolder<vector::TransferWriteOp>>( StoreOpOfSubViewFolder<vector::TransferWriteOp>>(
patterns.getContext()); patterns.getContext());
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pass for testing just the legalization patterns. // Pass for testing just the legalization patterns.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace { namespace {
struct SPIRVLegalization final struct SPIRVLegalization final
: public LegalizeStandardForSPIRVBase<SPIRVLegalization> { : public LegalizeStandardForSPIRVBase<SPIRVLegalization> {
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void SPIRVLegalization::runOnOperation() { void SPIRVLegalization::runOnOperation() {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateStdLegalizationPatternsForSPIRVLowering(patterns); populateStdLegalizationPatternsForSPIRVLowering(patterns);
(void)applyPatternsAndFoldGreedily(getOperation()->getRegions(), (void)applyPatternsAndFoldGreedily(getOperation()->getRegions(),
std::move(patterns)); std::move(patterns));
} }
std::unique_ptr<Pass> mlir::createLegalizeStdOpsForSPIRVLoweringPass() { std::unique_ptr<Pass> mlir::createLegalizeStdOpsForSPIRVLoweringPass() {
return std::make_unique<SPIRVLegalization>(); return std::make_unique<SPIRVLegalization>();
} }

mlir/lib/Conversion/StandardToSPIRV/StandardToSPIRV.cpp

Show First 20 Lines • Show All 1,219 Lines • ▼ Show 20 Lines
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Pattern population // Pattern population
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace mlir { namespace mlir {
void populateStandardToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateStandardToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
MLIRContext *context = patterns.getContext(); MLIRContext *context = patterns.getContext();
patterns.insert< patterns.add<
// Math dialect operations. // Math dialect operations.
// TODO: Move to separate pass. // TODO: Move to separate pass.
UnaryAndBinaryOpPattern<math::CosOp, spirv::GLSLCosOp>, UnaryAndBinaryOpPattern<math::CosOp, spirv::GLSLCosOp>,
UnaryAndBinaryOpPattern<math::ExpOp, spirv::GLSLExpOp>, UnaryAndBinaryOpPattern<math::ExpOp, spirv::GLSLExpOp>,
UnaryAndBinaryOpPattern<math::LogOp, spirv::GLSLLogOp>, UnaryAndBinaryOpPattern<math::LogOp, spirv::GLSLLogOp>,
UnaryAndBinaryOpPattern<math::RsqrtOp, spirv::GLSLInverseSqrtOp>, UnaryAndBinaryOpPattern<math::RsqrtOp, spirv::GLSLInverseSqrtOp>,
UnaryAndBinaryOpPattern<math::SinOp, spirv::GLSLSinOp>, UnaryAndBinaryOpPattern<math::SinOp, spirv::GLSLSinOp>,
UnaryAndBinaryOpPattern<math::SqrtOp, spirv::GLSLSqrtOp>, UnaryAndBinaryOpPattern<math::SqrtOp, spirv::GLSLSqrtOp>,
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Lines patterns.add<
TypeCastingOpPattern<TruncateIOp, spirv::SConvertOp>, TypeCastingOpPattern<TruncateIOp, spirv::SConvertOp>,
TypeCastingOpPattern<FPToSIOp, spirv::ConvertFToSOp>, TypeCastingOpPattern<FPToSIOp, spirv::ConvertFToSOp>,
TypeCastingOpPattern<FPExtOp, spirv::FConvertOp>, TypeCastingOpPattern<FPExtOp, spirv::FConvertOp>,
TypeCastingOpPattern<FPTruncOp, spirv::FConvertOp>>(typeConverter, TypeCastingOpPattern<FPTruncOp, spirv::FConvertOp>>(typeConverter,
context); context);
// Give CmpFOpNanKernelPattern a higher benefit so it can prevail when Kernel // Give CmpFOpNanKernelPattern a higher benefit so it can prevail when Kernel
// capability is available. // capability is available.
patterns.insert<CmpFOpNanKernelPattern>(typeConverter, context, patterns.add<CmpFOpNanKernelPattern>(typeConverter, context,
/*benefit=*/2); /*benefit=*/2);
} }
void populateTensorToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void populateTensorToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
int64_t byteCountThreshold, int64_t byteCountThreshold,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<TensorExtractPattern>(typeConverter, patterns.getContext(), patterns.add<TensorExtractPattern>(typeConverter, patterns.getContext(),
byteCountThreshold); byteCountThreshold);
} }
} // namespace mlir } // namespace mlir

mlir/lib/Conversion/StandardToSPIRV/StandardToSPIRVPass.cpp

Show All 29 Linesvoid ConvertStandardToSPIRVPass::runOnOperation() {
MLIRContext *context = &getContext(); MLIRContext *context = &getContext();
ModuleOp module = getOperation(); ModuleOp module = getOperation();
auto targetAttr = spirv::lookupTargetEnvOrDefault(module); auto targetAttr = spirv::lookupTargetEnvOrDefault(module);
std::unique_ptr<ConversionTarget> target = std::unique_ptr<ConversionTarget> target =
spirv::SPIRVConversionTarget::get(targetAttr); spirv::SPIRVConversionTarget::get(targetAttr);
SPIRVTypeConverter typeConverter(targetAttr); SPIRVTypeConverter typeConverter(targetAttr);
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
populateStandardToSPIRVPatterns(typeConverter, patterns); populateStandardToSPIRVPatterns(typeConverter, patterns);
populateTensorToSPIRVPatterns(typeConverter, populateTensorToSPIRVPatterns(typeConverter,
/*byteCountThreshold=*/64, patterns); /*byteCountThreshold=*/64, patterns);
populateBuiltinFuncToSPIRVPatterns(typeConverter, patterns); populateBuiltinFuncToSPIRVPatterns(typeConverter, patterns);
if (failed(applyPartialConversion(module, *target, std::move(patterns)))) if (failed(applyPartialConversion(module, *target, std::move(patterns))))
return signalPassFailure(); return signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> std::unique_ptr<OperationPass<ModuleOp>>
mlir::createConvertStandardToSPIRVPass() { mlir::createConvertStandardToSPIRVPass() {
return std::make_unique<ConvertStandardToSPIRVPass>(); return std::make_unique<ConvertStandardToSPIRVPass>();
} }

mlir/lib/Conversion/TosaToLinalg/TosaToLinalg.cpp

Show First 20 Lines • Show All 1,010 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(tosa::ReverseOp op,
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::tosa::populateTosaToLinalgOnTensorsConversionPatterns( void mlir::tosa::populateTosaToLinalgOnTensorsConversionPatterns(
OwningRewritePatternList *patterns) { RewritePatternSet *patterns) {
patterns->insert< patterns->add<
PointwiseConverter<tosa::AddOp>, PointwiseConverter<tosa::SubOp>, PointwiseConverter<tosa::AddOp>, PointwiseConverter<tosa::SubOp>,
PointwiseConverter<tosa::MulOp>, PointwiseConverter<tosa::NegateOp>, PointwiseConverter<tosa::MulOp>, PointwiseConverter<tosa::NegateOp>,
PointwiseConverter<tosa::PowOp>, PointwiseConverter<tosa::RsqrtOp>, PointwiseConverter<tosa::PowOp>, PointwiseConverter<tosa::RsqrtOp>,
PointwiseConverter<tosa::LogOp>, PointwiseConverter<tosa::ExpOp>, PointwiseConverter<tosa::LogOp>, PointwiseConverter<tosa::ExpOp>,
PointwiseConverter<tosa::AbsOp>, PointwiseConverter<tosa::TanhOp>, PointwiseConverter<tosa::AbsOp>, PointwiseConverter<tosa::TanhOp>,
PointwiseConverter<tosa::BitwiseAndOp>, PointwiseConverter<tosa::BitwiseAndOp>,
PointwiseConverter<tosa::BitwiseOrOp>, PointwiseConverter<tosa::BitwiseOrOp>,
PointwiseConverter<tosa::BitwiseXorOp>, PointwiseConverter<tosa::BitwiseXorOp>,
Show All 18 Lines

mlir/lib/Conversion/TosaToLinalg/TosaToLinalgPass.cpp

Show All 31 Linesstruct TosaToLinalgOnTensors
: public TosaToLinalgOnTensorsBase<TosaToLinalgOnTensors> { : public TosaToLinalgOnTensorsBase<TosaToLinalgOnTensors> {
public: public:
void getDependentDialects(DialectRegistry &registry) const override { void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<linalg::LinalgDialect, math::MathDialect, registry.insert<linalg::LinalgDialect, math::MathDialect,
memref::MemRefDialect, StandardOpsDialect>(); memref::MemRefDialect, StandardOpsDialect>();
} }
void runOnFunction() override { void runOnFunction() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addLegalDialect<linalg::LinalgDialect, memref::MemRefDialect, target.addLegalDialect<linalg::LinalgDialect, memref::MemRefDialect,
StandardOpsDialect>(); StandardOpsDialect>();
target.addIllegalDialect<tosa::TosaDialect>(); target.addIllegalDialect<tosa::TosaDialect>();
// Not every TOSA op can be legalized to linalg. // Not every TOSA op can be legalized to linalg.
target.addLegalOp<tosa::ApplyScaleOp>(); target.addLegalOp<tosa::ApplyScaleOp>();
target.addLegalOp<tosa::IfOp>(); target.addLegalOp<tosa::IfOp>();
Show All 21 Lines

mlir/lib/Conversion/TosaToSCF/TosaToSCF.cpp

Show First 20 Lines • Show All 97 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(tosa::WhileOp op,
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::tosa::populateTosaToSCFConversionPatterns( void mlir::tosa::populateTosaToSCFConversionPatterns(
OwningRewritePatternList *patterns) { RewritePatternSet *patterns) {
patterns->insert<IfOpConverter>(patterns->getContext()); patterns->add<IfOpConverter>(patterns->getContext());
patterns->insert<WhileOpConverter>(patterns->getContext()); patterns->add<WhileOpConverter>(patterns->getContext());
} }

mlir/lib/Conversion/TosaToSCF/TosaToSCFPass.cpp

Show All 23 Lines
using namespace mlir; using namespace mlir;
using namespace tosa; using namespace tosa;
namespace { namespace {
struct TosaToSCF : public TosaToSCFBase<TosaToSCF> { struct TosaToSCF : public TosaToSCFBase<TosaToSCF> {
public: public:
void runOnOperation() override { void runOnOperation() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addLegalDialect<tensor::TensorDialect, scf::SCFDialect>(); target.addLegalDialect<tensor::TensorDialect, scf::SCFDialect>();
target.addIllegalOp<tosa::IfOp, tosa::WhileOp>(); target.addIllegalOp<tosa::IfOp, tosa::WhileOp>();
target.markUnknownOpDynamicallyLegal([](Operation *) { return true; }); target.markUnknownOpDynamicallyLegal([](Operation *) { return true; });
auto *op = getOperation(); auto *op = getOperation();
mlir::tosa::populateTosaToSCFConversionPatterns(&patterns); mlir::tosa::populateTosaToSCFConversionPatterns(&patterns);
if (failed(applyPartialConversion(op, target, std::move(patterns)))) if (failed(applyPartialConversion(op, target, std::move(patterns))))
Show All 12 Lines

mlir/lib/Conversion/TosaToStandard/TosaToStandard.cpp

Show First 20 Lines • Show All 148 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(tosa::ApplyScaleOp op,
rewriter.replaceOp(op, result32); rewriter.replaceOp(op, result32);
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::tosa::populateTosaToStandardConversionPatterns( void mlir::tosa::populateTosaToStandardConversionPatterns(
OwningRewritePatternList *patterns) { RewritePatternSet *patterns) {
patterns->insert<ApplyScaleOpConverter, ConstOpConverter, SliceOpConverter>( patterns->add<ApplyScaleOpConverter, ConstOpConverter, SliceOpConverter>(
patterns->getContext()); patterns->getContext());
} }
void mlir::tosa::populateTosaRescaleToStandardConversionPatterns( void mlir::tosa::populateTosaRescaleToStandardConversionPatterns(
OwningRewritePatternList *patterns) { RewritePatternSet *patterns) {
patterns->insert<ApplyScaleOpConverter>(patterns->getContext()); patterns->add<ApplyScaleOpConverter>(patterns->getContext());
} }

mlir/lib/Conversion/TosaToStandard/TosaToStandardPass.cpp

Show All 23 Lines
using namespace mlir; using namespace mlir;
using namespace tosa; using namespace tosa;
namespace { namespace {
struct TosaToStandard : public TosaToStandardBase<TosaToStandard> { struct TosaToStandard : public TosaToStandardBase<TosaToStandard> {
public: public:
void runOnOperation() override { void runOnOperation() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addIllegalOp<tosa::ConstOp>(); target.addIllegalOp<tosa::ConstOp>();
target.addIllegalOp<tosa::SliceOp>(); target.addIllegalOp<tosa::SliceOp>();
target.addIllegalOp<tosa::ApplyScaleOp>(); target.addIllegalOp<tosa::ApplyScaleOp>();
target.addLegalDialect<StandardOpsDialect>(); target.addLegalDialect<StandardOpsDialect>();
mlir::tosa::populateTosaToStandardConversionPatterns(&patterns); mlir::tosa::populateTosaToStandardConversionPatterns(&patterns);
if (failed(applyPartialConversion(getOperation(), target, if (failed(applyPartialConversion(getOperation(), target,
Show All 13 Lines

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp

Show First 20 Lines • Show All 1,476 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(ExtractStridedSliceOp op,
return success(); return success();
} }
}; };
} // namespace } // namespace
/// Populate the given list with patterns that convert from Vector to LLVM. /// Populate the given list with patterns that convert from Vector to LLVM.
void mlir::populateVectorToLLVMConversionPatterns( void mlir::populateVectorToLLVMConversionPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns, LLVMTypeConverter &converter, RewritePatternSet &patterns,
bool reassociateFPReductions, bool enableIndexOptimizations) { bool reassociateFPReductions, bool enableIndexOptimizations) {
MLIRContext *ctx = converter.getDialect()->getContext(); MLIRContext *ctx = converter.getDialect()->getContext();
// clang-format off patterns.add<VectorFMAOpNDRewritePattern,
patterns.insert<VectorFMAOpNDRewritePattern,
VectorInsertStridedSliceOpDifferentRankRewritePattern, VectorInsertStridedSliceOpDifferentRankRewritePattern,
VectorInsertStridedSliceOpSameRankRewritePattern, VectorInsertStridedSliceOpSameRankRewritePattern,
VectorExtractStridedSliceOpConversion>(ctx); VectorExtractStridedSliceOpConversion>(ctx);
patterns.insert<VectorReductionOpConversion>( patterns.add<VectorReductionOpConversion>(converter, reassociateFPReductions);
converter, reassociateFPReductions); patterns.add<VectorCreateMaskOpConversion,
patterns.insert<VectorCreateMaskOpConversion,
VectorTransferConversion<TransferReadOp>, VectorTransferConversion<TransferReadOp>,
VectorTransferConversion<TransferWriteOp>>( VectorTransferConversion<TransferWriteOp>>(
converter, enableIndexOptimizations); converter, enableIndexOptimizations);
patterns patterns
.insert<VectorBitCastOpConversion, .add<VectorBitCastOpConversion, VectorShuffleOpConversion,
VectorShuffleOpConversion, VectorExtractElementOpConversion, VectorExtractOpConversion,
VectorExtractElementOpConversion, VectorFMAOp1DConversion, VectorInsertElementOpConversion,
VectorExtractOpConversion, VectorInsertOpConversion, VectorPrintOpConversion,
VectorFMAOp1DConversion,
VectorInsertElementOpConversion,
VectorInsertOpConversion,
VectorPrintOpConversion,
VectorTypeCastOpConversion, VectorTypeCastOpConversion,
VectorLoadStoreConversion<vector::LoadOp, VectorLoadStoreConversion<vector::LoadOp, vector::LoadOpAdaptor>,
vector::LoadOpAdaptor>,
VectorLoadStoreConversion<vector::MaskedLoadOp, VectorLoadStoreConversion<vector::MaskedLoadOp,
vector::MaskedLoadOpAdaptor>, vector::MaskedLoadOpAdaptor>,
VectorLoadStoreConversion<vector::StoreOp, VectorLoadStoreConversion<vector::StoreOp, vector::StoreOpAdaptor>,
vector::StoreOpAdaptor>,
VectorLoadStoreConversion<vector::MaskedStoreOp, VectorLoadStoreConversion<vector::MaskedStoreOp,
vector::MaskedStoreOpAdaptor>, vector::MaskedStoreOpAdaptor>,
VectorGatherOpConversion, VectorGatherOpConversion, VectorScatterOpConversion,
VectorScatterOpConversion, VectorExpandLoadOpConversion, VectorCompressStoreOpConversion>(
VectorExpandLoadOpConversion, converter);
VectorCompressStoreOpConversion>(converter);
// clang-format on
} }
void mlir::populateVectorToLLVMMatrixConversionPatterns( void mlir::populateVectorToLLVMMatrixConversionPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, RewritePatternSet &patterns) {
patterns.insert<VectorMatmulOpConversion>(converter); patterns.add<VectorMatmulOpConversion>(converter);
patterns.insert<VectorFlatTransposeOpConversion>(converter); patterns.add<VectorFlatTransposeOpConversion>(converter);
} }

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

Show First 20 Lines • Show All 55 Lines • ▼ Show 20 Linesstruct LowerVectorToLLVMPass
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void LowerVectorToLLVMPass::runOnOperation() { void LowerVectorToLLVMPass::runOnOperation() {
// Perform progressive lowering of operations on slices and // Perform progressive lowering of operations on slices and
// all contraction operations. Also applies folding and DCE. // all contraction operations. Also applies folding and DCE.
{ {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateVectorToVectorCanonicalizationPatterns(patterns); populateVectorToVectorCanonicalizationPatterns(patterns);
populateVectorSlicesLoweringPatterns(patterns); populateVectorSlicesLoweringPatterns(patterns);
populateVectorContractLoweringPatterns(patterns); populateVectorContractLoweringPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
} }
// Convert to the LLVM IR dialect. // Convert to the LLVM IR dialect.
LLVMTypeConverter converter(&getContext()); LLVMTypeConverter converter(&getContext());
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateVectorToLLVMMatrixConversionPatterns(converter, patterns); populateVectorToLLVMMatrixConversionPatterns(converter, patterns);
populateVectorToLLVMConversionPatterns( populateVectorToLLVMConversionPatterns(
converter, patterns, reassociateFPReductions, enableIndexOptimizations); converter, patterns, reassociateFPReductions, enableIndexOptimizations);
populateVectorToLLVMMatrixConversionPatterns(converter, patterns); populateVectorToLLVMMatrixConversionPatterns(converter, patterns);
// Architecture specific augmentations. // Architecture specific augmentations.
LLVMConversionTarget target(getContext()); LLVMConversionTarget target(getContext());
target.addLegalOp<LLVM::DialectCastOp>(); target.addLegalOp<LLVM::DialectCastOp>();
▲ Show 20 Lines • Show All 49 Lines • Show Last 20 Lines

mlir/lib/Conversion/VectorToROCDL/VectorToROCDL.cpp

Show First 20 Lines • Show All 138 Lines • ▼ Show 20 Lines matchAndRewrite(ConcreteOp xferOp, ArrayRef<Value> operands,
return replaceTransferOpWithMubuf( return replaceTransferOpWithMubuf(
rewriter, operands, *this->getTypeConverter(), loc, xferOp, vecTy, rewriter, operands, *this->getTypeConverter(), loc, xferOp, vecTy,
dwordConfig, int32Zero, int32Zero, int1False, int1False); dwordConfig, int32Zero, int32Zero, int1False, int1False);
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void mlir::populateVectorToROCDLConversionPatterns( void mlir::populateVectorToROCDLConversionPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, RewritePatternSet &patterns) {
patterns.insert<VectorTransferConversion<TransferReadOp>, patterns.add<VectorTransferConversion<TransferReadOp>,
VectorTransferConversion<TransferWriteOp>>(converter); VectorTransferConversion<TransferWriteOp>>(converter);
} }
namespace { namespace {
struct LowerVectorToROCDLPass struct LowerVectorToROCDLPass
: public ConvertVectorToROCDLBase<LowerVectorToROCDLPass> { : public ConvertVectorToROCDLBase<LowerVectorToROCDLPass> {
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void LowerVectorToROCDLPass::runOnOperation() { void LowerVectorToROCDLPass::runOnOperation() {
LLVMTypeConverter converter(&getContext()); LLVMTypeConverter converter(&getContext());
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateVectorToROCDLConversionPatterns(converter, patterns); populateVectorToROCDLConversionPatterns(converter, patterns);
populateStdToLLVMConversionPatterns(converter, patterns); populateStdToLLVMConversionPatterns(converter, patterns);
LLVMConversionTarget target(getContext()); LLVMConversionTarget target(getContext());
target.addLegalDialect<ROCDL::ROCDLDialect>(); target.addLegalDialect<ROCDL::ROCDLDialect>();
if (failed( if (failed(
applyPartialConversion(getOperation(), target, std::move(patterns)))) applyPartialConversion(getOperation(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> std::unique_ptr<OperationPass<ModuleOp>>
mlir::createConvertVectorToROCDLPass() { mlir::createConvertVectorToROCDLPass() {
return std::make_unique<LowerVectorToROCDLPass>(); return std::make_unique<LowerVectorToROCDLPass>();
} }

mlir/lib/Conversion/VectorToSCF/VectorToSCF.cpp

Show First 20 Lines • Show All 688 Lines • ▼ Show 20 LinesLogicalResult VectorTransferRewriter<TransferWriteOp>::matchAndRewrite(
}); });
// 3. Erase. // 3. Erase.
rewriter.eraseOp(op); rewriter.eraseOp(op);
return success(); return success();
} }
void populateVectorToSCFConversionPatterns( void populateVectorToSCFConversionPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns, const VectorTransferToSCFOptions &options) {
const VectorTransferToSCFOptions &options) { patterns.add<VectorTransferRewriter<vector::TransferReadOp>,
patterns.insert<VectorTransferRewriter<vector::TransferReadOp>,
VectorTransferRewriter<vector::TransferWriteOp>>( VectorTransferRewriter<vector::TransferWriteOp>>(
options, patterns.getContext()); options, patterns.getContext());
} }
} // namespace mlir } // namespace mlir
namespace { namespace {
struct ConvertVectorToSCFPass struct ConvertVectorToSCFPass
: public ConvertVectorToSCFBase<ConvertVectorToSCFPass> { : public ConvertVectorToSCFBase<ConvertVectorToSCFPass> {
ConvertVectorToSCFPass() = default; ConvertVectorToSCFPass() = default;
ConvertVectorToSCFPass(const VectorTransferToSCFOptions &options) { ConvertVectorToSCFPass(const VectorTransferToSCFOptions &options) {
this->fullUnroll = options.unroll; this->fullUnroll = options.unroll;
} }
void runOnFunction() override { void runOnFunction() override {
OwningRewritePatternList patterns(getFunction().getContext()); RewritePatternSet patterns(getFunction().getContext());
populateVectorToSCFConversionPatterns( populateVectorToSCFConversionPatterns(
patterns, VectorTransferToSCFOptions().setUnroll(fullUnroll)); patterns, VectorTransferToSCFOptions().setUnroll(fullUnroll));
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
} }
}; };
} // namespace } // namespace
std::unique_ptr<Pass> std::unique_ptr<Pass>
mlir::createConvertVectorToSCFPass(const VectorTransferToSCFOptions &options) { mlir::createConvertVectorToSCFPass(const VectorTransferToSCFOptions &options) {
return std::make_unique<ConvertVectorToSCFPass>(options); return std::make_unique<ConvertVectorToSCFPass>(options);
} }

mlir/lib/Conversion/VectorToSPIRV/VectorToSPIRV.cpp

Show First 20 Lines • Show All 236 Lines • ▼ Show 20 Lines matchAndRewrite(vector::InsertStridedSliceOp insertOp,
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateVectorToSPIRVPatterns(SPIRVTypeConverter &typeConverter, void mlir::populateVectorToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<VectorBitcastConvert, VectorBroadcastConvert, patterns.add<VectorBitcastConvert, VectorBroadcastConvert,
VectorExtractElementOpConvert, VectorExtractOpConvert, VectorExtractElementOpConvert, VectorExtractOpConvert,
VectorExtractStridedSliceOpConvert, VectorFmaOpConvert, VectorExtractStridedSliceOpConvert, VectorFmaOpConvert,
VectorInsertElementOpConvert, VectorInsertOpConvert, VectorInsertElementOpConvert, VectorInsertOpConvert,
VectorInsertStridedSliceOpConvert>(typeConverter, VectorInsertStridedSliceOpConvert>(typeConverter,
patterns.getContext()); patterns.getContext());
} }

mlir/lib/Conversion/VectorToSPIRV/VectorToSPIRVPass.cpp

Show All 31 Linesvoid LowerVectorToSPIRVPass::runOnOperation() {
MLIRContext *context = &getContext(); MLIRContext *context = &getContext();
ModuleOp module = getOperation(); ModuleOp module = getOperation();
auto targetAttr = spirv::lookupTargetEnvOrDefault(module); auto targetAttr = spirv::lookupTargetEnvOrDefault(module);
std::unique_ptr<ConversionTarget> target = std::unique_ptr<ConversionTarget> target =
spirv::SPIRVConversionTarget::get(targetAttr); spirv::SPIRVConversionTarget::get(targetAttr);
SPIRVTypeConverter typeConverter(targetAttr); SPIRVTypeConverter typeConverter(targetAttr);
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
populateVectorToSPIRVPatterns(typeConverter, patterns); populateVectorToSPIRVPatterns(typeConverter, patterns);
target->addLegalOp<ModuleOp, ModuleTerminatorOp>(); target->addLegalOp<ModuleOp, ModuleTerminatorOp>();
target->addLegalOp<FuncOp>(); target->addLegalOp<FuncOp>();
if (failed(applyFullConversion(module, *target, std::move(patterns)))) if (failed(applyFullConversion(module, *target, std::move(patterns))))
return signalPassFailure(); return signalPassFailure();
} }
std::unique_ptr<OperationPass<ModuleOp>> std::unique_ptr<OperationPass<ModuleOp>>
mlir::createConvertVectorToSPIRVPass() { mlir::createConvertVectorToSPIRVPass() {
return std::make_unique<LowerVectorToSPIRVPass>(); return std::make_unique<LowerVectorToSPIRVPass>();
} }

mlir/lib/Dialect/AMX/Transforms/LegalizeForLLVMExport.cpp

Show First 20 Lines • Show All 210 Lines • ▼ Show 20 Lines else
adaptor.lhs(), adaptor.rhs()); adaptor.lhs(), adaptor.rhs());
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateAMXLegalizeForLLVMExportPatterns( void mlir::populateAMXLegalizeForLLVMExportPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, RewritePatternSet &patterns) {
patterns.insert<TileZeroConversion, TileLoadConversion, TileStoreConversion, patterns.add<TileZeroConversion, TileLoadConversion, TileStoreConversion,
TileMulFConversion, TileMulIConversion>(converter); TileMulFConversion, TileMulIConversion>(converter);
} }
void mlir::configureAMXLegalizeForExportTarget(LLVMConversionTarget &target) { void mlir::configureAMXLegalizeForExportTarget(LLVMConversionTarget &target) {
target.addLegalOp<x86_amx_tilezero, x86_amx_tileloadd64, x86_amx_tilestored64, target.addLegalOp<x86_amx_tilezero, x86_amx_tileloadd64, x86_amx_tilestored64,
x86_amx_tdpbf16ps, x86_amx_tdpbssd, x86_amx_tdpbsud, x86_amx_tdpbf16ps, x86_amx_tdpbssd, x86_amx_tdpbsud,
x86_amx_tdpbusd, x86_amx_tdpbuud>(); x86_amx_tdpbusd, x86_amx_tdpbuud>();
target.addIllegalOp<TileZeroOp, TileLoadOp, TileStoreOp, TileMulIOp, target.addIllegalOp<TileZeroOp, TileLoadOp, TileStoreOp, TileMulIOp,
TileMulFOp>(); TileMulFOp>();
} }

mlir/lib/Dialect/AVX512/Transforms/LegalizeForLLVMExport.cpp

Show First 20 Lines • Show All 99 Lines • ▼ Show 20 Lines
/// A container for op association entries facilitating the configuration of /// A container for op association entries facilitating the configuration of
/// dialect conversion. /// dialect conversion.
template <typename... Args> template <typename... Args>
struct RegistryImpl { struct RegistryImpl {
/// Registers the patterns specializing the "main" op to one of the /// Registers the patterns specializing the "main" op to one of the
/// "intrinsic" ops depending on elemental type. /// "intrinsic" ops depending on elemental type.
static void registerPatterns(LLVMTypeConverter &converter, static void registerPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns patterns
.insert<LowerToIntrinsic<typename Args::MainOp, typename Args::Intr32Op, .add<LowerToIntrinsic<typename Args::MainOp, typename Args::Intr32Op,
typename Args::Intr64Op>...>(converter); typename Args::Intr64Op>...>(converter);
} }
/// Configures the conversion target to lower out "main" ops. /// Configures the conversion target to lower out "main" ops.
static void configureTarget(LLVMConversionTarget &target) { static void configureTarget(LLVMConversionTarget &target) {
target.addIllegalOp<typename Args::MainOp...>(); target.addIllegalOp<typename Args::MainOp...>();
target.addLegalOp<typename Args::Intr32Op...>(); target.addLegalOp<typename Args::Intr32Op...>();
target.addLegalOp<typename Args::Intr64Op...>(); target.addLegalOp<typename Args::Intr64Op...>();
} }
}; };
using Registry = RegistryImpl< using Registry = RegistryImpl<
RegEntry<MaskRndScaleOp, MaskRndScalePSIntrOp, MaskRndScalePDIntrOp>, RegEntry<MaskRndScaleOp, MaskRndScalePSIntrOp, MaskRndScalePDIntrOp>,
RegEntry<MaskScaleFOp, MaskScaleFPSIntrOp, MaskScaleFPDIntrOp>, RegEntry<MaskScaleFOp, MaskScaleFPSIntrOp, MaskScaleFPDIntrOp>,
RegEntry<Vp2IntersectOp, Vp2IntersectDIntrOp, Vp2IntersectQIntrOp>>; RegEntry<Vp2IntersectOp, Vp2IntersectDIntrOp, Vp2IntersectQIntrOp>>;
} // namespace } // namespace
/// Populate the given list with patterns that convert from AVX512 to LLVM. /// Populate the given list with patterns that convert from AVX512 to LLVM.
void mlir::populateAVX512LegalizeForLLVMExportPatterns( void mlir::populateAVX512LegalizeForLLVMExportPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, RewritePatternSet &patterns) {
Registry::registerPatterns(converter, patterns); Registry::registerPatterns(converter, patterns);
patterns.insert<MaskCompressOpConversion>(converter); patterns.add<MaskCompressOpConversion>(converter);
} }
void mlir::configureAVX512LegalizeForExportTarget( void mlir::configureAVX512LegalizeForExportTarget(
LLVMConversionTarget &target) { LLVMConversionTarget &target) {
Registry::configureTarget(target); Registry::configureTarget(target);
target.addLegalOp<MaskCompressIntrOp>(); target.addLegalOp<MaskCompressIntrOp>();
target.addIllegalOp<MaskCompressOp>(); target.addIllegalOp<MaskCompressOp>();
} }

mlir/lib/Dialect/Affine/IR/AffineOps.cpp

Show First 20 Lines • Show All 899 Lines • ▼ Show 20 Lines
template <typename AffineOpTy> template <typename AffineOpTy>
void SimplifyAffineOp<AffineOpTy>::replaceAffineOp( void SimplifyAffineOp<AffineOpTy>::replaceAffineOp(
PatternRewriter &rewriter, AffineOpTy op, AffineMap map, PatternRewriter &rewriter, AffineOpTy op, AffineMap map,
ArrayRef<Value> mapOperands) const { ArrayRef<Value> mapOperands) const {
rewriter.replaceOpWithNewOp<AffineOpTy>(op, map, mapOperands); rewriter.replaceOpWithNewOp<AffineOpTy>(op, map, mapOperands);
} }
} // end anonymous namespace. } // end anonymous namespace.
void AffineApplyOp::getCanonicalizationPatterns( void AffineApplyOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyAffineOp<AffineApplyOp>>(context); results.add<SimplifyAffineOp<AffineApplyOp>>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Common canonicalization pattern support logic // Common canonicalization pattern support logic
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// This is a common class used for patterns of the form /// This is a common class used for patterns of the form
/// "someop(memrefcast) -> someop". It folds the source of any memref.cast /// "someop(memrefcast) -> someop". It folds the source of any memref.cast
▲ Show 20 Lines • Show All 687 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(AffineForOp forOp,
if (!llvm::hasSingleElement(*forOp.getBody())) if (!llvm::hasSingleElement(*forOp.getBody()))
return failure(); return failure();
rewriter.eraseOp(forOp); rewriter.eraseOp(forOp);
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void AffineForOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void AffineForOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<AffineForEmptyLoopFolder>(context); results.add<AffineForEmptyLoopFolder>(context);
} }
LogicalResult AffineForOp::fold(ArrayRef<Attribute> operands, LogicalResult AffineForOp::fold(ArrayRef<Attribute> operands,
SmallVectorImpl<OpFoldResult> &results) { SmallVectorImpl<OpFoldResult> &results) {
bool folded = succeeded(foldLoopBounds(*this)); bool folded = succeeded(foldLoopBounds(*this));
folded |= succeeded(canonicalizeLoopBounds(*this)); folded |= succeeded(canonicalizeLoopBounds(*this));
return success(folded); return success(folded);
} }
▲ Show 20 Lines • Show All 403 Lines • ▼ Show 20 Lines if (operands.size() < getIntegerSet().getNumInputs() ||
set.getNumSymbols() > getIntegerSet().getNumSymbols()) { set.getNumSymbols() > getIntegerSet().getNumSymbols()) {
setConditional(set, operands); setConditional(set, operands);
return success(); return success();
} }
return failure(); return failure();
} }
void AffineIfOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void AffineIfOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyDeadElse>(context); results.add<SimplifyDeadElse>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AffineLoadOp // AffineLoadOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void AffineLoadOp::build(OpBuilder &builder, OperationState &result, void AffineLoadOp::build(OpBuilder &builder, OperationState &result,
AffineMap map, ValueRange operands) { AffineMap map, ValueRange operands) {
▲ Show 20 Lines • Show All 97 Lines • ▼ Show 20 Lines if (failed(verifyMemoryOpIndexing(
op->getAttrOfType<AffineMapAttr>(op.getMapAttrName()), op->getAttrOfType<AffineMapAttr>(op.getMapAttrName()),
op.getMapOperands(), memrefType, op.getMapOperands(), memrefType,
/*numIndexOperands=*/op.getNumOperands() - 1))) /*numIndexOperands=*/op.getNumOperands() - 1)))
return failure(); return failure();
return success(); return success();
} }
void AffineLoadOp::getCanonicalizationPatterns( void AffineLoadOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyAffineOp<AffineLoadOp>>(context); results.add<SimplifyAffineOp<AffineLoadOp>>(context);
} }
OpFoldResult AffineLoadOp::fold(ArrayRef<Attribute> cstOperands) { OpFoldResult AffineLoadOp::fold(ArrayRef<Attribute> cstOperands) {
/// load(memrefcast) -> load /// load(memrefcast) -> load
if (succeeded(foldMemRefCast(*this))) if (succeeded(foldMemRefCast(*this)))
return getResult(); return getResult();
return OpFoldResult(); return OpFoldResult();
} }
▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Lines if (failed(verifyMemoryOpIndexing(
op->getAttrOfType<AffineMapAttr>(op.getMapAttrName()), op->getAttrOfType<AffineMapAttr>(op.getMapAttrName()),
op.getMapOperands(), memrefType, op.getMapOperands(), memrefType,
/*numIndexOperands=*/op.getNumOperands() - 2))) /*numIndexOperands=*/op.getNumOperands() - 2)))
return failure(); return failure();
return success(); return success();
} }
void AffineStoreOp::getCanonicalizationPatterns( void AffineStoreOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyAffineOp<AffineStoreOp>>(context); results.add<SimplifyAffineOp<AffineStoreOp>>(context);
} }
LogicalResult AffineStoreOp::fold(ArrayRef<Attribute> cstOperands, LogicalResult AffineStoreOp::fold(ArrayRef<Attribute> cstOperands,
SmallVectorImpl<OpFoldResult> &results) { SmallVectorImpl<OpFoldResult> &results) {
/// store(memrefcast) -> store /// store(memrefcast) -> store
return foldMemRefCast(*this); return foldMemRefCast(*this);
} }
▲ Show 20 Lines • Show All 80 Lines • ▼ Show 20 Lines
// //
// %0 = affine.min (d0) -> (1000, d0 + 512) (%i0) // %0 = affine.min (d0) -> (1000, d0 + 512) (%i0)
// //
OpFoldResult AffineMinOp::fold(ArrayRef<Attribute> operands) { OpFoldResult AffineMinOp::fold(ArrayRef<Attribute> operands) {
return foldMinMaxOp(*this, operands); return foldMinMaxOp(*this, operands);
} }
void AffineMinOp::getCanonicalizationPatterns( void AffineMinOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<SimplifyAffineOp<AffineMinOp>>(context); patterns.add<SimplifyAffineOp<AffineMinOp>>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AffineMaxOp // AffineMaxOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// //
// %0 = affine.max (d0) -> (1000, d0 + 512) (%i0) // %0 = affine.max (d0) -> (1000, d0 + 512) (%i0)
// //
OpFoldResult AffineMaxOp::fold(ArrayRef<Attribute> operands) { OpFoldResult AffineMaxOp::fold(ArrayRef<Attribute> operands) {
return foldMinMaxOp(*this, operands); return foldMinMaxOp(*this, operands);
} }
void AffineMaxOp::getCanonicalizationPatterns( void AffineMaxOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<SimplifyAffineOp<AffineMaxOp>>(context); patterns.add<SimplifyAffineOp<AffineMaxOp>>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AffinePrefetchOp // AffinePrefetchOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// //
// affine.prefetch %0[%i, %j + 5], read, locality<3>, data : memref<400x400xi32> // affine.prefetch %0[%i, %j + 5], read, locality<3>, data : memref<400x400xi32>
▲ Show 20 Lines • Show All 81 Lines • ▼ Show 20 Linesstatic LogicalResult verify(AffinePrefetchOp op) {
Region *scope = getAffineScope(op); Region *scope = getAffineScope(op);
for (auto idx : op.getMapOperands()) { for (auto idx : op.getMapOperands()) {
if (!isValidAffineIndexOperand(idx, scope)) if (!isValidAffineIndexOperand(idx, scope))
return op.emitOpError("index must be a dimension or symbol identifier"); return op.emitOpError("index must be a dimension or symbol identifier");
} }
return success(); return success();
} }
void AffinePrefetchOp::getCanonicalizationPatterns( void AffinePrefetchOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
// prefetch(memrefcast) -> prefetch // prefetch(memrefcast) -> prefetch
results.insert<SimplifyAffineOp<AffinePrefetchOp>>(context); results.add<SimplifyAffineOp<AffinePrefetchOp>>(context);
} }
LogicalResult AffinePrefetchOp::fold(ArrayRef<Attribute> cstOperands, LogicalResult AffinePrefetchOp::fold(ArrayRef<Attribute> cstOperands,
SmallVectorImpl<OpFoldResult> &results) { SmallVectorImpl<OpFoldResult> &results) {
/// prefetch(memrefcast) -> prefetch /// prefetch(memrefcast) -> prefetch
return foldMemRefCast(*this); return foldMemRefCast(*this);
} }
▲ Show 20 Lines • Show All 592 Lines • Show Last 20 Lines

mlir/lib/Dialect/Affine/Transforms/AffineDataCopyGeneration.cpp

Show First 20 Lines • Show All 221 Lines • ▼ Show 20 Lines nest->walk([&](Operation *op) {
(void)promoteIfSingleIteration(forOp); (void)promoteIfSingleIteration(forOp);
else if (isa<AffineLoadOp, AffineStoreOp>(op)) else if (isa<AffineLoadOp, AffineStoreOp>(op))
copyOps.push_back(op); copyOps.push_back(op);
}); });
// Promoting single iteration loops could lead to simplification of // Promoting single iteration loops could lead to simplification of
// contained load's/store's, and the latter could anyway also be // contained load's/store's, and the latter could anyway also be
// canonicalized. // canonicalized.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
AffineLoadOp::getCanonicalizationPatterns(patterns, &getContext()); AffineLoadOp::getCanonicalizationPatterns(patterns, &getContext());
AffineStoreOp::getCanonicalizationPatterns(patterns, &getContext()); AffineStoreOp::getCanonicalizationPatterns(patterns, &getContext());
FrozenRewritePatternList frozenPatterns(std::move(patterns)); FrozenRewritePatternList frozenPatterns(std::move(patterns));
for (Operation *op : copyOps) for (Operation *op : copyOps)
(void)applyOpPatternsAndFold(op, frozenPatterns); (void)applyOpPatternsAndFold(op, frozenPatterns);
} }

mlir/lib/Dialect/Affine/Transforms/SimplifyAffineStructures.cpp

Show First 20 Lines • Show All 73 Lines • ▼ Show 20 Lines
std::unique_ptr<OperationPass<FuncOp>> std::unique_ptr<OperationPass<FuncOp>>
mlir::createSimplifyAffineStructuresPass() { mlir::createSimplifyAffineStructuresPass() {
return std::make_unique<SimplifyAffineStructures>(); return std::make_unique<SimplifyAffineStructures>();
} }
void SimplifyAffineStructures::runOnFunction() { void SimplifyAffineStructures::runOnFunction() {
auto func = getFunction(); auto func = getFunction();
simplifiedAttributes.clear(); simplifiedAttributes.clear();
OwningRewritePatternList patterns(func.getContext()); RewritePatternSet patterns(func.getContext());
AffineForOp::getCanonicalizationPatterns(patterns, func.getContext()); AffineForOp::getCanonicalizationPatterns(patterns, func.getContext());
AffineIfOp::getCanonicalizationPatterns(patterns, func.getContext()); AffineIfOp::getCanonicalizationPatterns(patterns, func.getContext());
AffineApplyOp::getCanonicalizationPatterns(patterns, func.getContext()); AffineApplyOp::getCanonicalizationPatterns(patterns, func.getContext());
FrozenRewritePatternList frozenPatterns(std::move(patterns)); FrozenRewritePatternList frozenPatterns(std::move(patterns));
func.walk([&](Operation *op) { func.walk([&](Operation *op) {
for (auto attr : op->getAttrs()) { for (auto attr : op->getAttrs()) {
if (auto mapAttr = attr.second.dyn_cast<AffineMapAttr>()) if (auto mapAttr = attr.second.dyn_cast<AffineMapAttr>())
simplifyAndUpdateAttribute(op, attr.first, mapAttr); simplifyAndUpdateAttribute(op, attr.first, mapAttr);
Show All 10 Lines

mlir/lib/Dialect/Affine/Utils/Utils.cpp

Show First 20 Lines • Show All 182 Lines • ▼ Show 20 LinesLogicalResult mlir::hoistAffineIfOp(AffineIfOp ifOp, bool *folded) {
if (ifOp.getNumResults() != 0) if (ifOp.getNumResults() != 0)
return failure(); return failure();
// Apply canonicalization patterns and folding - this is necessary for the // Apply canonicalization patterns and folding - this is necessary for the
// hoisting check to be correct (operands should be composed), and to be more // hoisting check to be correct (operands should be composed), and to be more
// effective (no unused operands). Since the pattern rewriter's folding is // effective (no unused operands). Since the pattern rewriter's folding is
// entangled with application of patterns, we may fold/end up erasing the op, // entangled with application of patterns, we may fold/end up erasing the op,
// in which case we return with `folded` being set. // in which case we return with `folded` being set.
OwningRewritePatternList patterns(ifOp.getContext()); RewritePatternSet patterns(ifOp.getContext());
AffineIfOp::getCanonicalizationPatterns(patterns, ifOp.getContext()); AffineIfOp::getCanonicalizationPatterns(patterns, ifOp.getContext());
bool erased; bool erased;
FrozenRewritePatternList frozenPatterns(std::move(patterns)); FrozenRewritePatternList frozenPatterns(std::move(patterns));
(void)applyOpPatternsAndFold(ifOp, frozenPatterns, &erased); (void)applyOpPatternsAndFold(ifOp, frozenPatterns, &erased);
if (erased) { if (erased) {
if (folded) if (folded)
*folded = true; *folded = true;
return failure(); return failure();
▲ Show 20 Lines • Show All 56 Lines • Show Last 20 Lines

mlir/lib/Dialect/Async/Transforms/AsyncParallelFor.cpp

Show First 20 Lines • Show All 264 Lines • ▼ Show 20 LinesAsyncParallelForRewrite::matchAndRewrite(scf::ParallelOp op,
rewriter.eraseOp(op); rewriter.eraseOp(op);
return success(); return success();
} }
void AsyncParallelForPass::runOnFunction() { void AsyncParallelForPass::runOnFunction() {
MLIRContext *ctx = &getContext(); MLIRContext *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
patterns.insert<AsyncParallelForRewrite>(ctx, numConcurrentAsyncExecute); patterns.add<AsyncParallelForRewrite>(ctx, numConcurrentAsyncExecute);
if (failed(applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)))) if (failed(applyPatternsAndFoldGreedily(getFunction(), std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
std::unique_ptr<OperationPass<FuncOp>> mlir::createAsyncParallelForPass() { std::unique_ptr<OperationPass<FuncOp>> mlir::createAsyncParallelForPass() {
return std::make_unique<AsyncParallelForPass>(); return std::make_unique<AsyncParallelForPass>();
} }
std::unique_ptr<OperationPass<FuncOp>> std::unique_ptr<OperationPass<FuncOp>>
mlir::createAsyncParallelForPass(int numWorkerThreads) { mlir::createAsyncParallelForPass(int numWorkerThreads) {
return std::make_unique<AsyncParallelForPass>(numWorkerThreads); return std::make_unique<AsyncParallelForPass>(numWorkerThreads);
} }

mlir/lib/Dialect/Async/Transforms/AsyncToAsyncRuntime.cpp

Show First 20 Lines • Show All 479 Lines • ▼ Show 20 Linesvoid AsyncToAsyncRuntimePass::runOnOperation() {
LLVM_DEBUG({ LLVM_DEBUG({
llvm::dbgs() << "Outlined " << outlinedFunctions.size() llvm::dbgs() << "Outlined " << outlinedFunctions.size()
<< " functions built from async.execute operations\n"; << " functions built from async.execute operations\n";
}); });
// Lower async operations to async.runtime operations. // Lower async operations to async.runtime operations.
MLIRContext *ctx = module->getContext(); MLIRContext *ctx = module->getContext();
OwningRewritePatternList asyncPatterns(ctx); RewritePatternSet asyncPatterns(ctx);
// Async lowering does not use type converter because it must preserve all // Async lowering does not use type converter because it must preserve all
// types for async.runtime operations. // types for async.runtime operations.
asyncPatterns.insert<CreateGroupOpLowering, AddToGroupOpLowering>(ctx); asyncPatterns.add<CreateGroupOpLowering, AddToGroupOpLowering>(ctx);
asyncPatterns.insert<AwaitTokenOpLowering, AwaitValueOpLowering, asyncPatterns.add<AwaitTokenOpLowering, AwaitValueOpLowering,
AwaitAllOpLowering, YieldOpLowering>(ctx, AwaitAllOpLowering, YieldOpLowering>(ctx,
outlinedFunctions); outlinedFunctions);
// All high level async operations must be lowered to the runtime operations. // All high level async operations must be lowered to the runtime operations.
ConversionTarget runtimeTarget(*ctx); ConversionTarget runtimeTarget(*ctx);
runtimeTarget.addLegalDialect<AsyncDialect>(); runtimeTarget.addLegalDialect<AsyncDialect>();
runtimeTarget.addIllegalOp<CreateGroupOp, AddToGroupOp>(); runtimeTarget.addIllegalOp<CreateGroupOp, AddToGroupOp>();
runtimeTarget.addIllegalOp<ExecuteOp, AwaitOp, AwaitAllOp, async::YieldOp>(); runtimeTarget.addIllegalOp<ExecuteOp, AwaitOp, AwaitAllOp, async::YieldOp>();
if (failed(applyPartialConversion(module, runtimeTarget, if (failed(applyPartialConversion(module, runtimeTarget,
Show All 9 Lines

mlir/lib/Dialect/GPU/Transforms/AllReduceLowering.cpp

Show First 20 Lines • Show All 395 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(Operation *op,
}; };
while (funcOp.walk(callback).wasInterrupted()) { while (funcOp.walk(callback).wasInterrupted()) {
} }
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateGpuAllReducePatterns(OwningRewritePatternList &patterns) { void mlir::populateGpuAllReducePatterns(RewritePatternSet &patterns) {
patterns.insert<GpuAllReduceConversion>(patterns.getContext()); patterns.add<GpuAllReduceConversion>(patterns.getContext());
} }

mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp

Show First 20 Lines • Show All 836 Lines • ▼ Show 20 Lines Value initTensor = rewriter.create<InitTensorOp>(
loc, resultShapeValues, reshapeOp.getResultType().getElementType()); loc, resultShapeValues, reshapeOp.getResultType().getElementType());
rewriter.replaceOpWithNewOp<tensor::CastOp>( rewriter.replaceOpWithNewOp<tensor::CastOp>(
reshapeOp, reshapeOp.getResultType(), initTensor); reshapeOp, reshapeOp.getResultType(), initTensor);
return success(); return success();
} }
}; };
} // namespace } // namespace
void InitTensorOp::getCanonicalizationPatterns( void InitTensorOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results results.add<FoldInitTensorWithSubTensorOp, FoldInitTensorWithTensorReshapeOp,
.insert<FoldInitTensorWithSubTensorOp, FoldInitTensorWithTensorReshapeOp,
ReplaceDimOfInitTensorOp, ReplaceStaticShapeDims>(context); ReplaceDimOfInitTensorOp, ReplaceStaticShapeDims>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// PadTensorOp // PadTensorOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Extract int64_t values from the assumed ArrayAttr of IntegerAttr. /// Extract int64_t values from the assumed ArrayAttr of IntegerAttr.
▲ Show 20 Lines • Show All 684 Lines • ▼ Show 20 Linesstatic LogicalResult verify(ReshapeOp op) {
auto maps = getAffineMaps(op.reassociation()); auto maps = getAffineMaps(op.reassociation());
MemRefType expectedType = computeReshapeCollapsedType(expandedType, maps); MemRefType expectedType = computeReshapeCollapsedType(expandedType, maps);
if (collapsedType != expectedType) if (collapsedType != expectedType)
return op.emitOpError("expected collapsed type to be ") return op.emitOpError("expected collapsed type to be ")
<< expectedType << ", but got " << collapsedType; << expectedType << ", but got " << collapsedType;
return success(); return success();
} }
void ReshapeOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ReshapeOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<CollapseReshapeOps<ReshapeOp>>(context); results.add<CollapseReshapeOps<ReshapeOp>>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TensorReshapeOp // TensorReshapeOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Compute the RankedTensorType obtained by applying `reassociation` to `type`. /// Compute the RankedTensorType obtained by applying `reassociation` to `type`.
static RankedTensorType static RankedTensorType
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Lines rewriter.replaceOp(dimOp,
rewriter, dimOp.getLoc(), *dimIndex, reshapeOp.src(), rewriter, dimOp.getLoc(), *dimIndex, reshapeOp.src(),
reshapeOp.getResultType().getShape(), reshapeOp.getResultType().getShape(),
reshapeOp.getReassociationMaps())); reshapeOp.getReassociationMaps()));
return success(); return success();
} }
}; };
} // namespace } // namespace
void TensorReshapeOp::getCanonicalizationPatterns( void TensorReshapeOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<CollapseReshapeOps<TensorReshapeOp>, FoldReshapeWithConstant, results.add<CollapseReshapeOps<TensorReshapeOp>, FoldReshapeWithConstant,
ReplaceDimOfReshapeOpResult>(context); ReplaceDimOfReshapeOpResult>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// YieldOp // YieldOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, linalg::YieldOp op) { static void print(OpAsmPrinter &p, linalg::YieldOp op) {
p << op.getOperationName(); p << op.getOperationName();
▲ Show 20 Lines • Show All 973 Lines • ▼ Show 20 Lines if (returnedArgs.size() != genericOp.getOperation()->getNumResults())
return failure(); return failure();
rewriter.replaceOp(genericOp, returnedArgs); rewriter.replaceOp(genericOp, returnedArgs);
return success(); return success();
} }
}; };
} // namespace } // namespace
#define CANONICALIZERS_AND_FOLDERS(XXX) \ #define CANONICALIZERS_AND_FOLDERS(XXX) \
void XXX::getCanonicalizationPatterns(OwningRewritePatternList &results, \ void XXX::getCanonicalizationPatterns(RewritePatternSet &results, \
MLIRContext *context) { \ MLIRContext *context) { \
results.insert<DeduplicateInputs, EraseDeadLinalgOp, FoldTensorCastOp, \ results.add<DeduplicateInputs, EraseDeadLinalgOp, FoldTensorCastOp, \
RemoveIdentityLinalgOps>(); \ RemoveIdentityLinalgOps>(); \
results.insert<ReplaceDimOfLinalgOpResult>(context); \ results.add<ReplaceDimOfLinalgOpResult>(context); \
} \ } \
\ \
LogicalResult XXX::fold(ArrayRef<Attribute>, \ LogicalResult XXX::fold(ArrayRef<Attribute>, \
SmallVectorImpl<OpFoldResult> &) { \ SmallVectorImpl<OpFoldResult> &) { \
return foldMemRefCast(*this); \ return foldMemRefCast(*this); \
} }
CANONICALIZERS_AND_FOLDERS(ConvOp) CANONICALIZERS_AND_FOLDERS(ConvOp)
Show All 10 Lines

mlir/lib/Dialect/Linalg/Transforms/Bufferize.cpp

Show First 20 Lines • Show All 317 Lines • ▼ Show 20 Lines void runOnOperation() override {
// Mark all Linalg operations illegal as long as they work on tensors. // Mark all Linalg operations illegal as long as they work on tensors.
auto isLegalOperation = [&](Operation *op) { auto isLegalOperation = [&](Operation *op) {
return typeConverter.isLegal(op); return typeConverter.isLegal(op);
}; };
target.addDynamicallyLegalDialect<linalg::LinalgDialect>(isLegalOperation); target.addDynamicallyLegalDialect<linalg::LinalgDialect>(isLegalOperation);
target.addDynamicallyLegalOp<ConstantOp>(isLegalOperation); target.addDynamicallyLegalOp<ConstantOp>(isLegalOperation);
OwningRewritePatternList patterns(&context); RewritePatternSet patterns(&context);
populateLinalgBufferizePatterns(typeConverter, patterns); populateLinalgBufferizePatterns(typeConverter, patterns);
if (failed(applyPartialConversion(getOperation(), target, if (failed(applyPartialConversion(getOperation(), target,
std::move(patterns)))) std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
std::unique_ptr<OperationPass<FuncOp>> mlir::createLinalgBufferizePass() { std::unique_ptr<OperationPass<FuncOp>> mlir::createLinalgBufferizePass() {
return std::make_unique<LinalgBufferizePass>(); return std::make_unique<LinalgBufferizePass>();
} }
void mlir::linalg::populateLinalgBufferizePatterns( void mlir::linalg::populateLinalgBufferizePatterns(
BufferizeTypeConverter &typeConverter, OwningRewritePatternList &patterns) { BufferizeTypeConverter &typeConverter, RewritePatternSet &patterns) {
patterns.insert<BufferizeAnyLinalgOp>(typeConverter); patterns.add<BufferizeAnyLinalgOp>(typeConverter);
// TODO: Drop this once tensor constants work in standard. // TODO: Drop this once tensor constants work in standard.
// clang-format off // clang-format off
patterns.insert< patterns.add<
BufferizeFillOp, BufferizeFillOp,
BufferizeInitTensorOp, BufferizeInitTensorOp,
SubTensorOpConverter, SubTensorOpConverter,
SubTensorInsertOpConverter SubTensorInsertOpConverter
>(typeConverter, patterns.getContext()); >(typeConverter, patterns.getContext());
// clang-format on // clang-format on
} }

mlir/lib/Dialect/Linalg/Transforms/CodegenStrategy.cpp

Show All 39 Lines auto marker = (currentState == zeroState)
? linalg::LinalgTransformationFilter( ? linalg::LinalgTransformationFilter(
t->filter, ArrayRef<Identifier>{}, nextState) t->filter, ArrayRef<Identifier>{}, nextState)
: linalg::LinalgTransformationFilter( : linalg::LinalgTransformationFilter(
t->filter, currentState, nextState); t->filter, currentState, nextState);
stage1Patterns.emplace_back(t->buildRewritePatterns(context, marker)); stage1Patterns.emplace_back(t->buildRewritePatterns(context, marker));
currentState = nextState; currentState = nextState;
} }
OwningRewritePatternList stage2Patterns = RewritePatternSet stage2Patterns =
linalg::getLinalgTilingCanonicalizationPatterns(context); linalg::getLinalgTilingCanonicalizationPatterns(context);
stage2Patterns.insert<AffineMinSCFCanonicalizationPattern>(context); stage2Patterns.add<AffineMinSCFCanonicalizationPattern>(context);
auto stage3Transforms = [&](Operation *op) { auto stage3Transforms = [&](Operation *op) {
// Some of these may be too aggressive as a stage 3 that is applied on each // Some of these may be too aggressive as a stage 3 that is applied on each
// stage 1 application and may have to be split out to post staged patterns // stage 1 application and may have to be split out to post staged patterns
// application (in which case they could just be passes, TBD). // application (in which case they could just be passes, TBD).
if (lateCodegenStrategyOptions.enableLICM) { if (lateCodegenStrategyOptions.enableLICM) {
op->walk([&](LoopLikeOpInterface loopLike) { op->walk([&](LoopLikeOpInterface loopLike) {
LLVM_DEBUG(loopLike.print(llvm::dbgs() << "\nOriginal loop:\n")); LLVM_DEBUG(loopLike.print(llvm::dbgs() << "\nOriginal loop:\n"));
Show All 12 Lines (void)linalg::applyStagedPatterns(
func, stage1Patterns, std::move(stage2Patterns), stage3Transforms); func, stage1Patterns, std::move(stage2Patterns), stage3Transforms);
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Post staged patterns transforms // Post staged patterns transforms
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Programmatic splitting of slow/fast path vector transfers. // Programmatic splitting of slow/fast path vector transfers.
if (lateCodegenStrategyOptions.enableVectorTransferPartialRewrite) { if (lateCodegenStrategyOptions.enableVectorTransferPartialRewrite) {
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
patterns.insert<vector::VectorTransferFullPartialRewriter>( patterns.add<vector::VectorTransferFullPartialRewriter>(
context, vectorTransformsOptions); context, vectorTransformsOptions);
(void)applyPatternsAndFoldGreedily(func, std::move(patterns)); (void)applyPatternsAndFoldGreedily(func, std::move(patterns));
} }
// Programmatic controlled lowering of vector.contract only. // Programmatic controlled lowering of vector.contract only.
if (lateCodegenStrategyOptions.enableVectorContractLowering) { if (lateCodegenStrategyOptions.enableVectorContractLowering) {
OwningRewritePatternList vectorContractLoweringPatterns(context); RewritePatternSet vectorContractLoweringPatterns(context);
vectorContractLoweringPatterns vectorContractLoweringPatterns
.insert<ContractionOpToOuterProductOpLowering, .add<ContractionOpToOuterProductOpLowering,
ContractionOpToMatmulOpLowering, ContractionOpLowering>( ContractionOpToMatmulOpLowering, ContractionOpLowering>(
vectorTransformsOptions, context); vectorTransformsOptions, context);
(void)applyPatternsAndFoldGreedily( (void)applyPatternsAndFoldGreedily(
func, std::move(vectorContractLoweringPatterns)); func, std::move(vectorContractLoweringPatterns));
} }
// Programmatic controlled lowering of vector.transfer only. // Programmatic controlled lowering of vector.transfer only.
if (lateCodegenStrategyOptions.enableVectorToSCFConversion) { if (lateCodegenStrategyOptions.enableVectorToSCFConversion) {
OwningRewritePatternList vectorToLoopsPatterns(context); RewritePatternSet vectorToLoopsPatterns(context);
populateVectorToSCFConversionPatterns(vectorToLoopsPatterns, populateVectorToSCFConversionPatterns(vectorToLoopsPatterns,
vectorToSCFOptions); vectorToSCFOptions);
(void)applyPatternsAndFoldGreedily(func, std::move(vectorToLoopsPatterns)); (void)applyPatternsAndFoldGreedily(func, std::move(vectorToLoopsPatterns));
} }
// Ensure we drop the marker in the end. // Ensure we drop the marker in the end.
func.walk([](LinalgOp op) { func.walk([](LinalgOp op) {
op->removeAttr(LinalgTransforms::kLinalgTransformMarker); op->removeAttr(LinalgTransforms::kLinalgTransformMarker);
}); });
} }

mlir/lib/Dialect/Linalg/Transforms/Detensorize.cpp

Show First 20 Lines • Show All 157 Lines • ▼ Show 20 Linesstruct ExtractFromReshapeFromElements
} }
}; };
/// @see LinalgDetensorize in Linalg/Passes.td for more details. /// @see LinalgDetensorize in Linalg/Passes.td for more details.
struct LinalgDetensorize : public LinalgDetensorizeBase<LinalgDetensorize> { struct LinalgDetensorize : public LinalgDetensorizeBase<LinalgDetensorize> {
void runOnFunction() override { void runOnFunction() override {
auto *context = &getContext(); auto *context = &getContext();
DetensorizeTypeConverter typeConverter; DetensorizeTypeConverter typeConverter;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
ConversionTarget target(*context); ConversionTarget target(*context);
target.addDynamicallyLegalOp<GenericOp>([&](GenericOp op) { target.addDynamicallyLegalOp<GenericOp>([&](GenericOp op) {
// If any of the operands or results cannot be detensored (i.e. they are // If any of the operands or results cannot be detensored (i.e. they are
// all legal according the DetensorizeTypeConverter), the op is considered // all legal according the DetensorizeTypeConverter), the op is considered
// legal and won't be detensored. // legal and won't be detensored.
return llvm::any_of(op.getShapedOperandTypes(), return llvm::any_of(op.getShapedOperandTypes(),
[&](ShapedType shapedType) { [&](ShapedType shapedType) {
Show All 14 Lines void runOnFunction() override {
target.markUnknownOpDynamicallyLegal([&](Operation *op) { target.markUnknownOpDynamicallyLegal([&](Operation *op) {
return isNotBranchOpInterfaceOrReturnLikeOp(op) || return isNotBranchOpInterfaceOrReturnLikeOp(op) ||
isLegalForBranchOpInterfaceTypeConversionPattern(op, isLegalForBranchOpInterfaceTypeConversionPattern(op,
typeConverter) || typeConverter) ||
isLegalForReturnOpTypeConversionPattern( isLegalForReturnOpTypeConversionPattern(
op, typeConverter, /*returnOpAlwaysLegal*/ true); op, typeConverter, /*returnOpAlwaysLegal*/ true);
}); });
patterns.insert<DetensorizeGenericOp>(typeConverter, context); patterns.add<DetensorizeGenericOp>(typeConverter, context);
patterns.insert<FunctionNonEntryBlockConversion>(FuncOp::getOperationName(), patterns.add<FunctionNonEntryBlockConversion>(FuncOp::getOperationName(),
context, typeConverter); context, typeConverter);
// Since non-entry block arguments get detensorized, we also need to update // Since non-entry block arguments get detensorized, we also need to update
// the control flow inside the function to reflect the correct types. // the control flow inside the function to reflect the correct types.
populateBranchOpInterfaceTypeConversionPattern(patterns, typeConverter); populateBranchOpInterfaceTypeConversionPattern(patterns, typeConverter);
if (failed(applyFullConversion(getFunction(), target, std::move(patterns)))) if (failed(applyFullConversion(getFunction(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
OwningRewritePatternList canonPatterns(context); RewritePatternSet canonPatterns(context);
canonPatterns.insert<ExtractFromReshapeFromElements>(context); canonPatterns.add<ExtractFromReshapeFromElements>(context);
if (failed(applyPatternsAndFoldGreedily(getFunction(), if (failed(applyPatternsAndFoldGreedily(getFunction(),
std::move(canonPatterns)))) std::move(canonPatterns))))
signalPassFailure(); signalPassFailure();
} }
}; };
} // namespace } // namespace
std::unique_ptr<Pass> mlir::createLinalgDetensorizePass() { std::unique_ptr<Pass> mlir::createLinalgDetensorizePass() {
return std::make_unique<LinalgDetensorize>(); return std::make_unique<LinalgDetensorize>();
} }

mlir/lib/Dialect/Linalg/Transforms/DropUnitDims.cpp

Show First 20 Lines • Show All 484 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(TensorReshapeOp reshapeOp,
return success(); return success();
} }
}; };
} // namespace } // namespace
/// Patterns that are used to canonicalize the use of unit-extent dims for /// Patterns that are used to canonicalize the use of unit-extent dims for
/// broadcasting. /// broadcasting.
void mlir::populateLinalgFoldUnitExtentDimsPatterns( void mlir::populateLinalgFoldUnitExtentDimsPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
auto *context = patterns.getContext(); auto *context = patterns.getContext();
patterns patterns.add<FoldUnitDimLoops<GenericOp>, FoldUnitDimLoops<IndexedGenericOp>,
.insert<FoldUnitDimLoops<GenericOp>, FoldUnitDimLoops<IndexedGenericOp>,
ReplaceUnitExtentTensors<GenericOp>, ReplaceUnitExtentTensors<GenericOp>,
ReplaceUnitExtentTensors<IndexedGenericOp>>(context); ReplaceUnitExtentTensors<IndexedGenericOp>>(context);
TensorReshapeOp::getCanonicalizationPatterns(patterns, context); TensorReshapeOp::getCanonicalizationPatterns(patterns, context);
patterns.insert<FoldReshapeOpWithUnitExtent>(context); patterns.add<FoldReshapeOpWithUnitExtent>(context);
populateFoldUnitDimsReshapeOpsByLinearizationPatterns(patterns); populateFoldUnitDimsReshapeOpsByLinearizationPatterns(patterns);
} }
namespace { namespace {
/// Pass that removes unit-extent dims within generic ops. /// Pass that removes unit-extent dims within generic ops.
struct LinalgFoldUnitExtentDimsPass struct LinalgFoldUnitExtentDimsPass
: public LinalgFoldUnitExtentDimsBase<LinalgFoldUnitExtentDimsPass> { : public LinalgFoldUnitExtentDimsBase<LinalgFoldUnitExtentDimsPass> {
void runOnFunction() override { void runOnFunction() override {
FuncOp funcOp = getFunction(); FuncOp funcOp = getFunction();
MLIRContext *context = funcOp.getContext(); MLIRContext *context = funcOp.getContext();
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
if (foldOneTripLoopsOnly) if (foldOneTripLoopsOnly)
patterns.insert<FoldUnitDimLoops<GenericOp>, patterns
FoldUnitDimLoops<IndexedGenericOp>>(context); .add<FoldUnitDimLoops<GenericOp>, FoldUnitDimLoops<IndexedGenericOp>>(
context);
else else
populateLinalgFoldUnitExtentDimsPatterns(patterns); populateLinalgFoldUnitExtentDimsPatterns(patterns);
(void)applyPatternsAndFoldGreedily(funcOp.getBody(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(funcOp.getBody(), std::move(patterns));
} }
}; };
} // namespace } // namespace
std::unique_ptr<OperationPass<FuncOp>> std::unique_ptr<OperationPass<FuncOp>>
mlir::createLinalgFoldUnitExtentDimsPass() { mlir::createLinalgFoldUnitExtentDimsPass() {
return std::make_unique<LinalgFoldUnitExtentDimsPass>(); return std::make_unique<LinalgFoldUnitExtentDimsPass>();
} }

mlir/lib/Dialect/Linalg/Transforms/ElementwiseToLinalg.cpp

Show First 20 Lines • Show All 110 Lines • ▼ Show 20 Lines rewriter.replaceOpWithNewOp<linalg::GenericOp>(
builder.create<linalg::YieldOp>(loc, scalarOp->getResults()); builder.create<linalg::YieldOp>(loc, scalarOp->getResults());
}); });
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateElementwiseToLinalgConversionPatterns( void mlir::populateElementwiseToLinalgConversionPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<ConvertAnyElementwiseMappableOpOnRankedTensors>(); patterns.add<ConvertAnyElementwiseMappableOpOnRankedTensors>();
} }
namespace { namespace {
class ConvertElementwiseToLinalgPass class ConvertElementwiseToLinalgPass
: public ConvertElementwiseToLinalgBase<ConvertElementwiseToLinalgPass> { : public ConvertElementwiseToLinalgBase<ConvertElementwiseToLinalgPass> {
void runOnFunction() final { void runOnFunction() final {
auto func = getOperation(); auto func = getOperation();
auto *context = &getContext(); auto *context = &getContext();
ConversionTarget target(*context); ConversionTarget target(*context);
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
populateElementwiseToLinalgConversionPatterns(patterns); populateElementwiseToLinalgConversionPatterns(patterns);
target.markUnknownOpDynamicallyLegal([](Operation *op) { target.markUnknownOpDynamicallyLegal([](Operation *op) {
return !isElementwiseMappableOpOnRankedTensors(op); return !isElementwiseMappableOpOnRankedTensors(op);
}); });
if (failed(applyPartialConversion(func, target, std::move(patterns)))) if (failed(applyPartialConversion(func, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
}; };
} // namespace } // namespace
std::unique_ptr<OperationPass<FuncOp>> std::unique_ptr<OperationPass<FuncOp>>
mlir::createConvertElementwiseToLinalgPass() { mlir::createConvertElementwiseToLinalgPass() {
return std::make_unique<ConvertElementwiseToLinalgPass>(); return std::make_unique<ConvertElementwiseToLinalgPass>();
} }

mlir/lib/Dialect/Linalg/Transforms/FusionOnTensors.cpp

Show First 20 Lines • Show All 1,127 Lines • ▼ Show 20 Linesstruct FuseTensorOps : public OpRewritePattern<LinalgOpTy> {
} }
}; };
/// Pass that fuses generic ops on tensors. Used only for testing. /// Pass that fuses generic ops on tensors. Used only for testing.
struct FusionOfTensorOpsPass struct FusionOfTensorOpsPass
: public LinalgFusionOfTensorOpsBase<FusionOfTensorOpsPass> { : public LinalgFusionOfTensorOpsBase<FusionOfTensorOpsPass> {
void runOnOperation() override { void runOnOperation() override {
Operation *op = getOperation(); Operation *op = getOperation();
OwningRewritePatternList patterns(op->getContext()); RewritePatternSet patterns(op->getContext());
populateLinalgTensorOpsFusionPatterns(patterns); populateLinalgTensorOpsFusionPatterns(patterns);
(void)applyPatternsAndFoldGreedily(op->getRegions(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(op->getRegions(), std::move(patterns));
} }
}; };
/// Pass to test folding of reshape op with generic/indexed_generic ops by /// Pass to test folding of reshape op with generic/indexed_generic ops by
/// linearization. /// linearization.
struct FoldReshapeOpsByLinearizationPass struct FoldReshapeOpsByLinearizationPass
: public LinalgFoldReshapeOpsByLinearizationBase< : public LinalgFoldReshapeOpsByLinearizationBase<
FoldReshapeOpsByLinearizationPass> { FoldReshapeOpsByLinearizationPass> {
void runOnOperation() override { void runOnOperation() override {
Operation *op = getOperation(); Operation *op = getOperation();
OwningRewritePatternList patterns(op->getContext()); RewritePatternSet patterns(op->getContext());
populateFoldReshapeOpsByLinearizationPatterns(patterns); populateFoldReshapeOpsByLinearizationPatterns(patterns);
(void)applyPatternsAndFoldGreedily(op->getRegions(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(op->getRegions(), std::move(patterns));
} }
}; };
} // namespace } // namespace
void mlir::populateFoldReshapeOpsByLinearizationPatterns( void mlir::populateFoldReshapeOpsByLinearizationPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<FoldProducerReshapeOpByLinearization<GenericOp, false>, patterns.add<FoldProducerReshapeOpByLinearization<GenericOp, false>,
FoldProducerReshapeOpByLinearization<IndexedGenericOp, false>, FoldProducerReshapeOpByLinearization<IndexedGenericOp, false>,
FoldConsumerReshapeOpByLinearization<false>>( FoldConsumerReshapeOpByLinearization<false>>(
patterns.getContext()); patterns.getContext());
} }
void mlir::populateFoldUnitDimsReshapeOpsByLinearizationPatterns( void mlir::populateFoldUnitDimsReshapeOpsByLinearizationPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<FoldProducerReshapeOpByLinearization<GenericOp, true>, patterns.add<FoldProducerReshapeOpByLinearization<GenericOp, true>,
FoldProducerReshapeOpByLinearization<IndexedGenericOp, true>, FoldProducerReshapeOpByLinearization<IndexedGenericOp, true>,
FoldConsumerReshapeOpByLinearization<true>>( FoldConsumerReshapeOpByLinearization<true>>(
patterns.getContext()); patterns.getContext());
} }
void mlir::populateFoldReshapeOpsByExpansionPatterns( void mlir::populateFoldReshapeOpsByExpansionPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<FoldReshapeWithGenericOpByExpansion, patterns.add<FoldReshapeWithGenericOpByExpansion,
FoldWithProducerReshapeOpByExpansion<GenericOp>, FoldWithProducerReshapeOpByExpansion<GenericOp>,
FoldWithProducerReshapeOpByExpansion<IndexedGenericOp>>( FoldWithProducerReshapeOpByExpansion<IndexedGenericOp>>(
patterns.getContext()); patterns.getContext());
} }
void mlir::populateLinalgTensorOpsFusionPatterns( void mlir::populateLinalgTensorOpsFusionPatterns(RewritePatternSet &patterns) {
OwningRewritePatternList &patterns) {
auto *context = patterns.getContext(); auto *context = patterns.getContext();
patterns.insert<FuseTensorOps<GenericOp>, FuseTensorOps<IndexedGenericOp>, patterns
FoldSplatConstants<GenericOp>, .add<FuseTensorOps<GenericOp>, FuseTensorOps<IndexedGenericOp>,
FoldSplatConstants<IndexedGenericOp>>(context); FoldSplatConstants<GenericOp>, FoldSplatConstants<IndexedGenericOp>>(
context);
populateFoldReshapeOpsByExpansionPatterns(patterns); populateFoldReshapeOpsByExpansionPatterns(patterns);
GenericOp::getCanonicalizationPatterns(patterns, context); GenericOp::getCanonicalizationPatterns(patterns, context);
IndexedGenericOp::getCanonicalizationPatterns(patterns, context); IndexedGenericOp::getCanonicalizationPatterns(patterns, context);
TensorReshapeOp::getCanonicalizationPatterns(patterns, context); TensorReshapeOp::getCanonicalizationPatterns(patterns, context);
} }
std::unique_ptr<Pass> mlir::createLinalgFusionOfTensorOpsPass() { std::unique_ptr<Pass> mlir::createLinalgFusionOfTensorOpsPass() {
return std::make_unique<FusionOfTensorOpsPass>(); return std::make_unique<FusionOfTensorOpsPass>();
} }
std::unique_ptr<Pass> mlir::createFoldReshapeOpsByLinearizationPass() { std::unique_ptr<Pass> mlir::createFoldReshapeOpsByLinearizationPass() {
return std::make_unique<FoldReshapeOpsByLinearizationPass>(); return std::make_unique<FoldReshapeOpsByLinearizationPass>();
} }

mlir/lib/Dialect/Linalg/Transforms/Generalization.cpp

Show First 20 Lines • Show All 137 Lines • ▼ Show 20 Linesstruct LinalgGeneralizationPass
: public LinalgGeneralizationBase<LinalgGeneralizationPass> { : public LinalgGeneralizationBase<LinalgGeneralizationPass> {
void runOnFunction() override; void runOnFunction() override;
}; };
} // namespace } // namespace
void LinalgGeneralizationPass::runOnFunction() { void LinalgGeneralizationPass::runOnFunction() {
FuncOp func = getFunction(); FuncOp func = getFunction();
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
linalg::populateLinalgConvGeneralizationPatterns(patterns); linalg::populateLinalgConvGeneralizationPatterns(patterns);
linalg::populateLinalgNamedOpsGeneralizationPatterns(patterns); linalg::populateLinalgNamedOpsGeneralizationPatterns(patterns);
(void)applyPatternsAndFoldGreedily(func.getBody(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(func.getBody(), std::move(patterns));
} }
linalg::GenericOp GeneralizeConvOp::createGenericOp(linalg::ConvOp convOp, linalg::GenericOp GeneralizeConvOp::createGenericOp(linalg::ConvOp convOp,
OpBuilder &builder) const { OpBuilder &builder) const {
SmallVector<AffineMap, 4> indexingMaps = convOp.getIndexingMaps(); SmallVector<AffineMap, 4> indexingMaps = convOp.getIndexingMaps();
auto iterators = auto iterators =
llvm::to_vector<4>(convOp.iterator_types().getAsValueRange<StringAttr>()); llvm::to_vector<4>(convOp.iterator_types().getAsValueRange<StringAttr>());
return builder.create<linalg::GenericOp>( return builder.create<linalg::GenericOp>(
convOp.getLoc(), /*resultTensorTypes=*/ArrayRef<Type>(), convOp.getLoc(), /*resultTensorTypes=*/ArrayRef<Type>(),
convOp.getInputBuffers(), convOp.getOutputBuffers(), indexingMaps, convOp.getInputBuffers(), convOp.getOutputBuffers(), indexingMaps,
iterators, iterators,
[](OpBuilder &bodyBuilder, Location bodyLoc, ValueRange bodyArgs) { [](OpBuilder &bodyBuilder, Location bodyLoc, ValueRange bodyArgs) {
Value mul = Value mul =
bodyBuilder.create<MulFOp>(bodyLoc, bodyArgs[0], bodyArgs[1]); bodyBuilder.create<MulFOp>(bodyLoc, bodyArgs[0], bodyArgs[1]);
Value add = bodyBuilder.create<AddFOp>(bodyLoc, mul, bodyArgs[2]); Value add = bodyBuilder.create<AddFOp>(bodyLoc, mul, bodyArgs[2]);
bodyBuilder.create<linalg::YieldOp>(bodyLoc, add); bodyBuilder.create<linalg::YieldOp>(bodyLoc, add);
}); });
} }
void mlir::linalg::populateLinalgConvGeneralizationPatterns( void mlir::linalg::populateLinalgConvGeneralizationPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns, linalg::LinalgTransformationFilter marker) {
linalg::LinalgTransformationFilter marker) { patterns.add<GeneralizeConvOp>(patterns.getContext(), marker);
patterns.insert<GeneralizeConvOp>(patterns.getContext(), marker);
} }
void mlir::linalg::populateLinalgNamedOpsGeneralizationPatterns( void mlir::linalg::populateLinalgNamedOpsGeneralizationPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns, linalg::LinalgTransformationFilter marker) {
linalg::LinalgTransformationFilter marker) { patterns.add<LinalgNamedOpGeneralizationPattern>(patterns.getContext(),
patterns.insert<LinalgNamedOpGeneralizationPattern>(patterns.getContext(),
marker); marker);
} }
std::unique_ptr<OperationPass<FuncOp>> mlir::createLinalgGeneralizationPass() { std::unique_ptr<OperationPass<FuncOp>> mlir::createLinalgGeneralizationPass() {
return std::make_unique<LinalgGeneralizationPass>(); return std::make_unique<LinalgGeneralizationPass>();
} }

mlir/lib/Dialect/Linalg/Transforms/Hoisting.cpp

Show First 20 Lines • Show All 372 Lines • ▼ Show 20 Lines func.walk([&](scf::ForOp forOp) {
// Need to interrupt and restart: erasing the loop messes up the walk. // Need to interrupt and restart: erasing the loop messes up the walk.
return WalkResult::interrupt(); return WalkResult::interrupt();
} }
return WalkResult::advance(); return WalkResult::advance();
}); });
// Apply canonicalization so the newForOp + yield folds immediately, thus // Apply canonicalization so the newForOp + yield folds immediately, thus
// cleaning up the IR and potentially enabling more hoisting. // cleaning up the IR and potentially enabling more hoisting.
if (changed) { if (changed) {
OwningRewritePatternList patterns(func->getContext()); RewritePatternSet patterns(func->getContext());
scf::ForOp::getCanonicalizationPatterns(patterns, func->getContext()); scf::ForOp::getCanonicalizationPatterns(patterns, func->getContext());
(void)applyPatternsAndFoldGreedily(func, std::move(patterns)); (void)applyPatternsAndFoldGreedily(func, std::move(patterns));
} }
} }
} }
void mlir::linalg::hoistRedundantVectorTransfers(FuncOp func) { void mlir::linalg::hoistRedundantVectorTransfers(FuncOp func) {
bool changed = true; bool changed = true;
▲ Show 20 Lines • Show All 353 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Transforms/Loops.cpp

Show First 20 Lines • Show All 539 Lines • ▼ Show 20 Lines
struct FoldAffineOp; struct FoldAffineOp;
} // namespace } // namespace
template <typename LoopType> template <typename LoopType>
static void lowerLinalgToLoopsImpl(FuncOp funcOp, static void lowerLinalgToLoopsImpl(FuncOp funcOp,
ArrayRef<unsigned> interchangeVector) { ArrayRef<unsigned> interchangeVector) {
MLIRContext *context = funcOp.getContext(); MLIRContext *context = funcOp.getContext();
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
patterns.insert<LinalgRewritePattern<LoopType>>(interchangeVector); patterns.add<LinalgRewritePattern<LoopType>>(interchangeVector);
memref::DimOp::getCanonicalizationPatterns(patterns, context); memref::DimOp::getCanonicalizationPatterns(patterns, context);
AffineApplyOp::getCanonicalizationPatterns(patterns, context); AffineApplyOp::getCanonicalizationPatterns(patterns, context);
patterns.insert<FoldAffineOp>(context); patterns.add<FoldAffineOp>(context);
// Just apply the patterns greedily. // Just apply the patterns greedily.
(void)applyPatternsAndFoldGreedily(funcOp, std::move(patterns)); (void)applyPatternsAndFoldGreedily(funcOp, std::move(patterns));
} }
namespace { namespace {
/// Local folding pattern for AffineApplyOp that we can apply greedily. /// Local folding pattern for AffineApplyOp that we can apply greedily.
/// This replaces AffineApplyOp by the proper value in cases where the /// This replaces AffineApplyOp by the proper value in cases where the
/// associated map is trivial. /// associated map is trivial.
▲ Show 20 Lines • Show All 117 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Transforms/SparseLowering.cpp

Show First 20 Lines • Show All 131 Lines • ▼ Show 20 Linespublic:
} }
}; };
} // namespace } // namespace
/// Populates the given patterns list with conversion rules required for /// Populates the given patterns list with conversion rules required for
/// the sparsification of linear algebra operations. /// the sparsification of linear algebra operations.
void linalg::populateSparsificationConversionPatterns( void linalg::populateSparsificationConversionPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<TensorFromPointerConverter, TensorToDimSizeConverter, patterns.add<TensorFromPointerConverter, TensorToDimSizeConverter,
TensorToPointersConverter, TensorToIndicesConverter, TensorToPointersConverter, TensorToIndicesConverter,
TensorToValuesConverter>(patterns.getContext()); TensorToValuesConverter>(patterns.getContext());
} }

mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp

Show First 20 Lines • Show All 1,355 Lines • ▼ Show 20 Linesprivate:
linalg::SparsificationOptions options; linalg::SparsificationOptions options;
}; };
} // namespace } // namespace
/// Populates the given patterns list with rewriting rules required for /// Populates the given patterns list with rewriting rules required for
/// the sparsification of linear algebra operations. /// the sparsification of linear algebra operations.
void linalg::populateSparsificationPatterns( void linalg::populateSparsificationPatterns(
OwningRewritePatternList &patterns, const SparsificationOptions &options) { RewritePatternSet &patterns, const SparsificationOptions &options) {
patterns.insert<GenericOpSparsifier>(patterns.getContext(), options); patterns.add<GenericOpSparsifier>(patterns.getContext(), options);
} }

mlir/lib/Dialect/Linalg/Transforms/Tiling.cpp

Show First 20 Lines • Show All 505 Lines • ▼ Show 20 Lines
namespace { namespace {
/// Helper classes for type list expansion. /// Helper classes for type list expansion.
template <typename... OpTypes> template <typename... OpTypes>
class CanonicalizationPatternList; class CanonicalizationPatternList;
template <> template <>
class CanonicalizationPatternList<> { class CanonicalizationPatternList<> {
public: public:
static void insert(OwningRewritePatternList &patterns) {} static void insert(RewritePatternSet &patterns) {}
}; };
template <typename OpTy, typename... OpTypes> template <typename OpTy, typename... OpTypes>
class CanonicalizationPatternList<OpTy, OpTypes...> { class CanonicalizationPatternList<OpTy, OpTypes...> {
public: public:
static void insert(OwningRewritePatternList &patterns) { static void insert(RewritePatternSet &patterns) {
OpTy::getCanonicalizationPatterns(patterns, patterns.getContext()); OpTy::getCanonicalizationPatterns(patterns, patterns.getContext());
CanonicalizationPatternList<OpTypes...>::insert(patterns); CanonicalizationPatternList<OpTypes...>::insert(patterns);
} }
}; };
/// Helper classes for type list expansion. /// Helper classes for type list expansion.
template <typename... OpTypes> template <typename... OpTypes>
class RewritePatternList; class RewritePatternList;
template <> template <>
class RewritePatternList<> { class RewritePatternList<> {
public: public:
static void insert(OwningRewritePatternList &patterns, static void insert(RewritePatternSet &patterns,
const LinalgTilingOptions &options) {} const LinalgTilingOptions &options) {}
}; };
template <typename OpTy, typename... OpTypes> template <typename OpTy, typename... OpTypes>
class RewritePatternList<OpTy, OpTypes...> { class RewritePatternList<OpTy, OpTypes...> {
public: public:
static void insert(OwningRewritePatternList &patterns, static void insert(RewritePatternSet &patterns,
const LinalgTilingOptions &options) { const LinalgTilingOptions &options) {
auto *ctx = patterns.getContext(); auto *ctx = patterns.getContext();
patterns.insert<LinalgTilingPattern<OpTy>>( patterns.add<LinalgTilingPattern<OpTy>>(
ctx, options, ctx, options,
LinalgTransformationFilter(ArrayRef<Identifier>{}, LinalgTransformationFilter(ArrayRef<Identifier>{},
Identifier::get("tiled", ctx))); Identifier::get("tiled", ctx)));
RewritePatternList<OpTypes...>::insert(patterns, options); RewritePatternList<OpTypes...>::insert(patterns, options);
} }
}; };
} // namespace } // namespace
OwningRewritePatternList RewritePatternSet
mlir::linalg::getLinalgTilingCanonicalizationPatterns(MLIRContext *ctx) { mlir::linalg::getLinalgTilingCanonicalizationPatterns(MLIRContext *ctx) {
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
populateLinalgTilingCanonicalizationPatterns(patterns); populateLinalgTilingCanonicalizationPatterns(patterns);
return patterns; return patterns;
} }
void mlir::linalg::populateLinalgTilingCanonicalizationPatterns( void mlir::linalg::populateLinalgTilingCanonicalizationPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
auto *ctx = patterns.getContext(); auto *ctx = patterns.getContext();
AffineApplyOp::getCanonicalizationPatterns(patterns, ctx); AffineApplyOp::getCanonicalizationPatterns(patterns, ctx);
AffineForOp::getCanonicalizationPatterns(patterns, ctx); AffineForOp::getCanonicalizationPatterns(patterns, ctx);
AffineMinOp::getCanonicalizationPatterns(patterns, ctx); AffineMinOp::getCanonicalizationPatterns(patterns, ctx);
AffineMaxOp::getCanonicalizationPatterns(patterns, ctx); AffineMaxOp::getCanonicalizationPatterns(patterns, ctx);
scf::ForOp::getCanonicalizationPatterns(patterns, ctx); scf::ForOp::getCanonicalizationPatterns(patterns, ctx);
scf::ParallelOp::getCanonicalizationPatterns(patterns, ctx); scf::ParallelOp::getCanonicalizationPatterns(patterns, ctx);
ConstantIndexOp::getCanonicalizationPatterns(patterns, ctx); ConstantIndexOp::getCanonicalizationPatterns(patterns, ctx);
SubTensorOp::getCanonicalizationPatterns(patterns, ctx); SubTensorOp::getCanonicalizationPatterns(patterns, ctx);
memref::SubViewOp::getCanonicalizationPatterns(patterns, ctx); memref::SubViewOp::getCanonicalizationPatterns(patterns, ctx);
tensor::CastOp::getCanonicalizationPatterns(patterns, ctx); tensor::CastOp::getCanonicalizationPatterns(patterns, ctx);
memref::ViewOp::getCanonicalizationPatterns(patterns, ctx); memref::ViewOp::getCanonicalizationPatterns(patterns, ctx);
CanonicalizationPatternList< CanonicalizationPatternList<
#define GET_OP_LIST #define GET_OP_LIST
#include "mlir/Dialect/Linalg/IR/LinalgStructuredOps.cpp.inc" #include "mlir/Dialect/Linalg/IR/LinalgStructuredOps.cpp.inc"
>::insert(patterns); >::insert(patterns);
} }
/// Populate the given list with patterns that apply Linalg tiling. /// Populate the given list with patterns that apply Linalg tiling.
static void insertTilingPatterns(OwningRewritePatternList &patterns, static void insertTilingPatterns(RewritePatternSet &patterns,
const LinalgTilingOptions &options) { const LinalgTilingOptions &options) {
RewritePatternList<GenericOp, IndexedGenericOp, RewritePatternList<GenericOp, IndexedGenericOp,
#define GET_OP_LIST #define GET_OP_LIST
#include "mlir/Dialect/Linalg/IR/LinalgStructuredOps.cpp.inc" #include "mlir/Dialect/Linalg/IR/LinalgStructuredOps.cpp.inc"
>::insert(patterns, options); >::insert(patterns, options);
} }
static void applyTilingToLoopPatterns(LinalgTilingLoopType loopType, static void applyTilingToLoopPatterns(LinalgTilingLoopType loopType,
FuncOp funcOp, FuncOp funcOp,
ArrayRef<int64_t> tileSizes) { ArrayRef<int64_t> tileSizes) {
auto options = auto options =
LinalgTilingOptions().setTileSizes(tileSizes).setLoopType(loopType); LinalgTilingOptions().setTileSizes(tileSizes).setLoopType(loopType);
MLIRContext *ctx = funcOp.getContext(); MLIRContext *ctx = funcOp.getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
insertTilingPatterns(patterns, options); insertTilingPatterns(patterns, options);
(void)applyPatternsAndFoldGreedily(funcOp, std::move(patterns)); (void)applyPatternsAndFoldGreedily(funcOp, std::move(patterns));
(void)applyPatternsAndFoldGreedily( (void)applyPatternsAndFoldGreedily(
funcOp, getLinalgTilingCanonicalizationPatterns(ctx)); funcOp, getLinalgTilingCanonicalizationPatterns(ctx));
// Drop the marker. // Drop the marker.
funcOp.walk([](LinalgOp op) { funcOp.walk([](LinalgOp op) {
op->removeAttr(LinalgTransforms::kLinalgTransformMarker); op->removeAttr(LinalgTransforms::kLinalgTransformMarker);
}); });
Show All 37 Lines

mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp

Show First 20 Lines • Show All 570 Lines • ▼ Show 20 LinesLogicalResult ConvOpVectorization<ConvOp, N>::matchAndRewrite(
return success(); return success();
} }
using ConvOpConst = ConvOpVectorization<ConvWOp, 1>; using ConvOpConst = ConvOpVectorization<ConvWOp, 1>;
/// Inserts tiling, promotion and vectorization pattern for ConvOp /// Inserts tiling, promotion and vectorization pattern for ConvOp
/// conversion into corresponding pattern lists. /// conversion into corresponding pattern lists.
template <typename ConvOp, unsigned N> template <typename ConvOp, unsigned N>
static void static void populateVectorizationPatterns(
populateVectorizationPatterns(OwningRewritePatternList &tilingPatterns, RewritePatternSet &tilingPatterns, RewritePatternSet &promotionPatterns,
OwningRewritePatternList &promotionPatterns, RewritePatternSet &vectorizationPatterns, ArrayRef<int64_t> tileSizes) {
OwningRewritePatternList &vectorizationPatterns,
ArrayRef<int64_t> tileSizes) {
auto *context = tilingPatterns.getContext(); auto *context = tilingPatterns.getContext();
if (tileSizes.size() < N) if (tileSizes.size() < N)
return; return;
constexpr static StringRef kTiledMarker = "TILED"; constexpr static StringRef kTiledMarker = "TILED";
constexpr static StringRef kPromotedMarker = "PROMOTED"; constexpr static StringRef kPromotedMarker = "PROMOTED";
tilingPatterns.insert<LinalgTilingPattern<ConvOp>>( tilingPatterns.add<LinalgTilingPattern<ConvOp>>(
context, LinalgTilingOptions().setTileSizes(tileSizes), context, LinalgTilingOptions().setTileSizes(tileSizes),
LinalgTransformationFilter(ArrayRef<Identifier>{}, LinalgTransformationFilter(ArrayRef<Identifier>{},
Identifier::get(kTiledMarker, context))); Identifier::get(kTiledMarker, context)));
promotionPatterns.insert<LinalgPromotionPattern<ConvOp>>( promotionPatterns.add<LinalgPromotionPattern<ConvOp>>(
context, LinalgPromotionOptions().setUseFullTileBuffersByDefault(true), context, LinalgPromotionOptions().setUseFullTileBuffersByDefault(true),
LinalgTransformationFilter(Identifier::get(kTiledMarker, context), LinalgTransformationFilter(Identifier::get(kTiledMarker, context),
Identifier::get(kPromotedMarker, context))); Identifier::get(kPromotedMarker, context)));
SmallVector<bool, 4> mask(N); SmallVector<bool, 4> mask(N);
int offset = tileSizes.size() - N; int offset = tileSizes.size() - N;
std::transform(tileSizes.begin() + offset, tileSizes.end(), mask.begin(), std::transform(tileSizes.begin() + offset, tileSizes.end(), mask.begin(),
[](int64_t i) -> bool { return i > 1; }); [](int64_t i) -> bool { return i > 1; });
vectorizationPatterns.insert<ConvOpVectorization<ConvOp, N>>(context, mask); vectorizationPatterns.add<ConvOpVectorization<ConvOp, N>>(context, mask);
} }
void mlir::linalg::populateConvVectorizationPatterns( void mlir::linalg::populateConvVectorizationPatterns(
MLIRContext *context, SmallVectorImpl<OwningRewritePatternList> &patterns, MLIRContext *context, SmallVectorImpl<RewritePatternSet> &patterns,
ArrayRef<int64_t> tileSizes) { ArrayRef<int64_t> tileSizes) {
OwningRewritePatternList tiling(context); RewritePatternSet tiling(context);
OwningRewritePatternList promotion(context); RewritePatternSet promotion(context);
OwningRewritePatternList vectorization(context); RewritePatternSet vectorization(context);
populateVectorizationPatterns<ConvWOp, 1>(tiling, promotion, vectorization, populateVectorizationPatterns<ConvWOp, 1>(tiling, promotion, vectorization,
tileSizes); tileSizes);
populateVectorizationPatterns<ConvNWCOp, 3>(tiling, promotion, vectorization, populateVectorizationPatterns<ConvNWCOp, 3>(tiling, promotion, vectorization,
tileSizes); tileSizes);
populateVectorizationPatterns<ConvInputNWCFilterWCFOp, 3>( populateVectorizationPatterns<ConvInputNWCFilterWCFOp, 3>(
tiling, promotion, vectorization, tileSizes); tiling, promotion, vectorization, tileSizes);
▲ Show 20 Lines • Show All 212 Lines • Show Last 20 Lines

mlir/lib/Dialect/Math/Transforms/PolynomialApproximation.cpp

Show First 20 Lines • Show All 526 Lines • ▼ Show 20 LinesExpApproximation::matchAndRewrite(math::ExpOp op,
rewriter.replaceOp(op, expY); rewriter.replaceOp(op, expY);
return success(); return success();
} }
//----------------------------------------------------------------------------// //----------------------------------------------------------------------------//
void mlir::populateMathPolynomialApproximationPatterns( void mlir::populateMathPolynomialApproximationPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<TanhApproximation, LogApproximation, Log2Approximation, patterns.add<TanhApproximation, LogApproximation, Log2Approximation,
ExpApproximation>(patterns.getContext()); ExpApproximation>(patterns.getContext());
} }

mlir/lib/Dialect/MemRef/IR/MemRefOps.cpp

Show First 20 Lines • Show All 203 Lines • ▼ Show 20 Lines if (alloc.use_empty()) {
rewriter.eraseOp(alloc); rewriter.eraseOp(alloc);
return success(); return success();
} }
return failure(); return failure();
} }
}; };
} // end anonymous namespace. } // end anonymous namespace.
void AllocOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void AllocOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyAllocConst<AllocOp>, SimplifyDeadAlloc>(context); results.add<SimplifyAllocConst<AllocOp>, SimplifyDeadAlloc>(context);
} }
void AllocaOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void AllocaOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyAllocConst<AllocaOp>>(context); results.add<SimplifyAllocConst<AllocaOp>>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AssumeAlignmentOp // AssumeAlignmentOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(AssumeAlignmentOp op) { static LogicalResult verify(AssumeAlignmentOp op) {
unsigned alignment = op.alignment(); unsigned alignment = op.alignment();
▲ Show 20 Lines • Show All 57 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(BufferCastOp bufferCast,
rewriter.replaceOpWithNewOp<CastOp>(bufferCast, bufferCast.getType(), rewriter.replaceOpWithNewOp<CastOp>(bufferCast, bufferCast.getType(),
tensorLoad.memref()); tensorLoad.memref());
return success(); return success();
} }
}; };
} // namespace } // namespace
void BufferCastOp::getCanonicalizationPatterns( void BufferCastOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<BufferCast, TensorLoadToMemRef>(context); results.add<BufferCast, TensorLoadToMemRef>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CastOp // CastOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Determines whether MemRef_CastOp casts to a more dynamic version of the /// Determines whether MemRef_CastOp casts to a more dynamic version of the
/// source memref. This is useful to to fold a memref.cast into a consuming op /// source memref. This is useful to to fold a memref.cast into a consuming op
▲ Show 20 Lines • Show All 189 Lines • ▼ Show 20 Lines
} // end anonymous namespace. } // end anonymous namespace.
static LogicalResult verify(DeallocOp op) { static LogicalResult verify(DeallocOp op) {
if (!op.memref().getType().isa<MemRefType>()) if (!op.memref().getType().isa<MemRefType>())
return op.emitOpError("operand must be a memref"); return op.emitOpError("operand must be a memref");
return success(); return success();
} }
void DeallocOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void DeallocOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyDeadDealloc>(context); results.add<SimplifyDeadDealloc>(context);
} }
LogicalResult DeallocOp::fold(ArrayRef<Attribute> cstOperands, LogicalResult DeallocOp::fold(ArrayRef<Attribute> cstOperands,
SmallVectorImpl<OpFoldResult> &results) { SmallVectorImpl<OpFoldResult> &results) {
/// dealloc(memrefcast) -> dealloc /// dealloc(memrefcast) -> dealloc
return foldMemRefCast(*this); return foldMemRefCast(*this);
} }
▲ Show 20 Lines • Show All 160 Lines • ▼ Show 20 Lines if (!castOp)
return failure(); return failure();
Value newSource = castOp.getOperand(); Value newSource = castOp.getOperand();
rewriter.replaceOpWithNewOp<DimOp>(dimOp, newSource, dimOp.index()); rewriter.replaceOpWithNewOp<DimOp>(dimOp, newSource, dimOp.index());
return success(); return success();
} }
}; };
} // end anonymous namespace. } // end anonymous namespace.
void DimOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void DimOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<DimOfMemRefReshape, DimOfCastOp<BufferCastOp>, results.add<DimOfMemRefReshape, DimOfCastOp<BufferCastOp>,
DimOfCastOp<tensor::CastOp>>(context); DimOfCastOp<tensor::CastOp>>(context);
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// DmaStartOp // DmaStartOp
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
void DmaStartOp::build(OpBuilder &builder, OperationState &result, void DmaStartOp::build(OpBuilder &builder, OperationState &result,
Value srcMemRef, ValueRange srcIndices, Value destMemRef, Value srcMemRef, ValueRange srcIndices, Value destMemRef,
▲ Show 20 Lines • Show All 372 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(LoadOp load,
rewriter.replaceOpWithNewOp<tensor::ExtractOp>(load, buffercast.tensor(), rewriter.replaceOpWithNewOp<tensor::ExtractOp>(load, buffercast.tensor(),
load.indices()); load.indices());
return success(); return success();
} }
}; };
} // end anonymous namespace. } // end anonymous namespace.
void LoadOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void LoadOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<LoadOfBufferCast>(context); results.add<LoadOfBufferCast>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// PrefetchOp // PrefetchOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static void print(OpAsmPrinter &p, PrefetchOp op) { static void print(OpAsmPrinter &p, PrefetchOp op) {
p << PrefetchOp::getOperationName() << " " << op.memref() << '['; p << PrefetchOp::getOperationName() << " " << op.memref() << '[';
▲ Show 20 Lines • Show All 714 Lines • ▼ Show 20 Lines
/// A canonicalizer wrapper to replace SubViewOps. /// A canonicalizer wrapper to replace SubViewOps.
struct SubViewCanonicalizer { struct SubViewCanonicalizer {
void operator()(PatternRewriter &rewriter, SubViewOp op, SubViewOp newOp) { void operator()(PatternRewriter &rewriter, SubViewOp op, SubViewOp newOp) {
rewriter.replaceOpWithNewOp<CastOp>(op, newOp, op.getType()); rewriter.replaceOpWithNewOp<CastOp>(op, newOp, op.getType());
} }
}; };
void SubViewOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void SubViewOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<OpWithOffsetSizesAndStridesConstantArgumentFolder< results.add<OpWithOffsetSizesAndStridesConstantArgumentFolder<
SubViewOp, SubViewCanonicalizer>, SubViewOp, SubViewCanonicalizer>,
SubViewOpMemRefCastFolder>(context); SubViewOpMemRefCastFolder>(context);
} }
OpFoldResult SubViewOp::fold(ArrayRef<Attribute> operands) { OpFoldResult SubViewOp::fold(ArrayRef<Attribute> operands) {
auto resultShapedType = getResult().getType().cast<ShapedType>(); auto resultShapedType = getResult().getType().cast<ShapedType>();
auto sourceShapedType = source().getType().cast<ShapedType>(); auto sourceShapedType = source().getType().cast<ShapedType>();
if (resultShapedType.hasStaticShape() && if (resultShapedType.hasStaticShape() &&
resultShapedType == sourceShapedType) { resultShapedType == sourceShapedType) {
▲ Show 20 Lines • Show All 262 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(ViewOp viewOp,
rewriter.replaceOpWithNewOp<ViewOp>(viewOp, viewOp.getType(), allocOperand, rewriter.replaceOpWithNewOp<ViewOp>(viewOp, viewOp.getType(), allocOperand,
viewOp.byte_shift(), viewOp.sizes()); viewOp.byte_shift(), viewOp.sizes());
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void ViewOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ViewOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ViewOpShapeFolder, ViewOpMemrefCastFolder>(context); results.add<ViewOpShapeFolder, ViewOpMemrefCastFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TableGen'd op method definitions // TableGen'd op method definitions
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
#define GET_OP_CLASSES #define GET_OP_CLASSES
#include "mlir/Dialect/MemRef/IR/MemRefOps.cpp.inc" #include "mlir/Dialect/MemRef/IR/MemRefOps.cpp.inc"

mlir/lib/Dialect/Quant/Transforms/ConvertConst.cpp

Show First 20 Lines • Show All 85 Lines • ▼ Show 20 LinesQuantizedConstRewrite::matchAndRewrite(QuantizeCastOp qbarrier,
auto newConstOp = auto newConstOp =
rewriter.create<ConstantOp>(fusedLoc, newConstValueType, newConstValue); rewriter.create<ConstantOp>(fusedLoc, newConstValueType, newConstValue);
rewriter.replaceOpWithNewOp<StorageCastOp>(qbarrier, qbarrier.getType(), rewriter.replaceOpWithNewOp<StorageCastOp>(qbarrier, qbarrier.getType(),
newConstOp); newConstOp);
return success(); return success();
} }
void ConvertConstPass::runOnFunction() { void ConvertConstPass::runOnFunction() {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
auto func = getFunction(); auto func = getFunction();
auto *context = &getContext(); auto *context = &getContext();
patterns.insert<QuantizedConstRewrite>(context); patterns.add<QuantizedConstRewrite>(context);
(void)applyPatternsAndFoldGreedily(func, std::move(patterns)); (void)applyPatternsAndFoldGreedily(func, std::move(patterns));
} }
std::unique_ptr<OperationPass<FuncOp>> mlir::quant::createConvertConstPass() { std::unique_ptr<OperationPass<FuncOp>> mlir::quant::createConvertConstPass() {
return std::make_unique<ConvertConstPass>(); return std::make_unique<ConvertConstPass>();
} }

mlir/lib/Dialect/Quant/Transforms/ConvertSimQuant.cpp

Show First 20 Lines • Show All 119 Lines • ▼ Show 20 Linespublic:
} }
}; };
} // namespace } // namespace
void ConvertSimulatedQuantPass::runOnFunction() { void ConvertSimulatedQuantPass::runOnFunction() {
bool hadFailure = false; bool hadFailure = false;
auto func = getFunction(); auto func = getFunction();
OwningRewritePatternList patterns(func.getContext()); RewritePatternSet patterns(func.getContext());
auto ctx = func.getContext(); auto ctx = func.getContext();
patterns.insert<ConstFakeQuantRewrite, ConstFakeQuantPerAxisRewrite>( patterns.add<ConstFakeQuantRewrite, ConstFakeQuantPerAxisRewrite>(
ctx, &hadFailure); ctx, &hadFailure);
(void)applyPatternsAndFoldGreedily(func, std::move(patterns)); (void)applyPatternsAndFoldGreedily(func, std::move(patterns));
if (hadFailure) if (hadFailure)
signalPassFailure(); signalPassFailure();
} }
std::unique_ptr<OperationPass<FuncOp>> std::unique_ptr<OperationPass<FuncOp>>
mlir::quant::createConvertSimulatedQuantPass() { mlir::quant::createConvertSimulatedQuantPass() {
return std::make_unique<ConvertSimulatedQuantPass>(); return std::make_unique<ConvertSimulatedQuantPass>();
} }

mlir/lib/Dialect/SCF/SCF.cpp

Show First 20 Lines • Show All 697 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(ForOp forOp,
rewriter.replaceOpWithIf(forOp, newResults, [&](OpOperand &op) { rewriter.replaceOpWithIf(forOp, newResults, [&](OpOperand &op) {
return op.get() != newResults[idx++]; return op.get() != newResults[idx++];
}); });
return success(); return success();
} }
}; };
} // namespace } // namespace
void ForOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ForOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ForOpIterArgsFolder, SimplifyTrivialLoops, results.add<ForOpIterArgsFolder, SimplifyTrivialLoops,
LastTensorLoadCanonicalization>(context); LastTensorLoadCanonicalization>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// IfOp // IfOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void IfOp::build(OpBuilder &builder, OperationState &result, Value cond, void IfOp::build(OpBuilder &builder, OperationState &result, Value cond,
bool withElseRegion) { bool withElseRegion) {
▲ Show 20 Lines • Show All 250 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(IfOp op,
} }
rewriter.replaceOp(op, results); rewriter.replaceOp(op, results);
return success(); return success();
} }
}; };
} // namespace } // namespace
void IfOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void IfOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<RemoveUnusedResults, RemoveStaticCondition, results.add<RemoveUnusedResults, RemoveStaticCondition,
ConvertTrivialIfToSelect>(context); ConvertTrivialIfToSelect>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ParallelOp // ParallelOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void ParallelOp::build( void ParallelOp::build(
OpBuilder &builder, OperationState &result, ValueRange lowerBounds, OpBuilder &builder, OperationState &result, ValueRange lowerBounds,
▲ Show 20 Lines • Show All 282 Lines • ▼ Show 20 Lines for (auto dim : llvm::zip(op.lowerBound(), op.upperBound())) {
} }
} }
return failure(); return failure();
} }
}; };
} // namespace } // namespace
void ParallelOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ParallelOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<CollapseSingleIterationLoops, RemoveEmptyParallelLoops>( results.add<CollapseSingleIterationLoops, RemoveEmptyParallelLoops>(context);
context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ReduceOp // ReduceOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void ReduceOp::build( void ReduceOp::build(
OpBuilder &builder, OperationState &result, Value operand, OpBuilder &builder, OperationState &result, Value operand,
▲ Show 20 Lines • Show All 233 Lines • Show Last 20 Lines

mlir/lib/Dialect/SCF/Transforms/Bufferize.cpp

Show All 19 Lines
namespace { namespace {
struct SCFBufferizePass : public SCFBufferizeBase<SCFBufferizePass> { struct SCFBufferizePass : public SCFBufferizeBase<SCFBufferizePass> {
void runOnFunction() override { void runOnFunction() override {
auto func = getOperation(); auto func = getOperation();
auto *context = &getContext(); auto *context = &getContext();
BufferizeTypeConverter typeConverter; BufferizeTypeConverter typeConverter;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
ConversionTarget target(*context); ConversionTarget target(*context);
populateBufferizeMaterializationLegality(target); populateBufferizeMaterializationLegality(target);
populateSCFStructuralTypeConversionsAndLegality(typeConverter, patterns, populateSCFStructuralTypeConversionsAndLegality(typeConverter, patterns,
target); target);
if (failed(applyPartialConversion(func, target, std::move(patterns)))) if (failed(applyPartialConversion(func, target, std::move(patterns))))
return signalPassFailure(); return signalPassFailure();
}; };
}; };
} // end anonymous namespace } // end anonymous namespace
std::unique_ptr<Pass> mlir::createSCFBufferizePass() { std::unique_ptr<Pass> mlir::createSCFBufferizePass() {
return std::make_unique<SCFBufferizePass>(); return std::make_unique<SCFBufferizePass>();
} }

mlir/lib/Dialect/SCF/Transforms/StructuralTypeConversions.cpp

Show First 20 Lines • Show All 128 Lines • ▼ Show 20 Lines matchAndRewrite(scf::YieldOp op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
rewriter.replaceOpWithNewOp<scf::YieldOp>(op, operands); rewriter.replaceOpWithNewOp<scf::YieldOp>(op, operands);
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::scf::populateSCFStructuralTypeConversionsAndLegality( void mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
TypeConverter &typeConverter, OwningRewritePatternList &patterns, TypeConverter &typeConverter, RewritePatternSet &patterns,
ConversionTarget &target) { ConversionTarget &target) {
patterns.insert<ConvertForOpTypes, ConvertIfOpTypes, ConvertYieldOpTypes>( patterns.add<ConvertForOpTypes, ConvertIfOpTypes, ConvertYieldOpTypes>(
typeConverter, patterns.getContext()); typeConverter, patterns.getContext());
target.addDynamicallyLegalOp<ForOp, IfOp>([&](Operation *op) { target.addDynamicallyLegalOp<ForOp, IfOp>([&](Operation *op) {
return typeConverter.isLegal(op->getResultTypes()); return typeConverter.isLegal(op->getResultTypes());
}); });
target.addDynamicallyLegalOp<scf::YieldOp>([&](scf::YieldOp op) { target.addDynamicallyLegalOp<scf::YieldOp>([&](scf::YieldOp op) {
// We only have conversions for a subset of ops that use scf.yield // We only have conversions for a subset of ops that use scf.yield
// terminators. // terminators.
if (!isa<ForOp, IfOp>(op->getParentOp())) if (!isa<ForOp, IfOp>(op->getParentOp()))
return true; return true;
return typeConverter.isLegal(op.getOperandTypes()); return typeConverter.isLegal(op.getOperandTypes());
}); });
} }

mlir/lib/Dialect/SPIRV/IR/SPIRVCanonicalization.cpp

Show First 20 Lines • Show All 105 Lines • ▼ Show 20 Lines rewriter.replaceOpWithNewOp<spirv::AccessChainOp>(
accessChainOp, parentAccessChainOp.base_ptr(), indices); accessChainOp, parentAccessChainOp.base_ptr(), indices);
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void spirv::AccessChainOp::getCanonicalizationPatterns( void spirv::AccessChainOp::getCanonicalizationPatterns(
OwningRewritePatternList &results, MLIRContext *context) { RewritePatternSet &results, MLIRContext *context) {
results.insert<CombineChainedAccessChain>(context); results.add<CombineChainedAccessChain>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.BitcastOp // spv.BitcastOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void spirv::BitcastOp::getCanonicalizationPatterns( void spirv::BitcastOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<ConvertChainedBitcast>(context); results.add<ConvertChainedBitcast>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.CompositeExtractOp // spv.CompositeExtractOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
OpFoldResult spirv::CompositeExtractOp::fold(ArrayRef<Attribute> operands) { OpFoldResult spirv::CompositeExtractOp::fold(ArrayRef<Attribute> operands) {
assert(operands.size() == 1 && "spv.CompositeExtract expects one operand"); assert(operands.size() == 1 && "spv.CompositeExtract expects one operand");
▲ Show 20 Lines • Show All 92 Lines • ▼ Show 20 LinesOpFoldResult spirv::LogicalAndOp::fold(ArrayRef<Attribute> operands) {
return Attribute(); return Attribute();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.LogicalNot // spv.LogicalNot
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void spirv::LogicalNotOp::getCanonicalizationPatterns( void spirv::LogicalNotOp::getCanonicalizationPatterns(
OwningRewritePatternList &results, MLIRContext *context) { RewritePatternSet &results, MLIRContext *context) {
results.insert<ConvertLogicalNotOfIEqual, ConvertLogicalNotOfINotEqual, results
ConvertLogicalNotOfLogicalEqual, .add<ConvertLogicalNotOfIEqual, ConvertLogicalNotOfINotEqual,
ConvertLogicalNotOfLogicalNotEqual>(context); ConvertLogicalNotOfLogicalEqual, ConvertLogicalNotOfLogicalNotEqual>(
context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spv.LogicalOr // spv.LogicalOr
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
OpFoldResult spirv::LogicalOrOp::fold(ArrayRef<Attribute> operands) { OpFoldResult spirv::LogicalOrOp::fold(ArrayRef<Attribute> operands) {
assert(operands.size() == 2 && "spv.LogicalOr should take two operands"); assert(operands.size() == 2 && "spv.LogicalOr should take two operands");
▲ Show 20 Lines • Show All 165 Lines • ▼ Show 20 Lines if ((trueBrBranchOp->getSuccessor(0) != mergeBlock) ||
(falseBrBranchOp->getSuccessor(0) != mergeBlock)) { (falseBrBranchOp->getSuccessor(0) != mergeBlock)) {
return failure(); return failure();
} }
return success(); return success();
} }
} // end anonymous namespace } // end anonymous namespace
void spirv::SelectionOp::getCanonicalizationPatterns( void spirv::SelectionOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<ConvertSelectionOpToSelect>(context); results.add<ConvertSelectionOpToSelect>(context);
} }

mlir/lib/Dialect/SPIRV/IR/SPIRVGLSLCanonicalization.cpp

Show All 16 Lines
using namespace mlir; using namespace mlir;
namespace { namespace {
#include "SPIRVCanonicalization.inc" #include "SPIRVCanonicalization.inc"
} // end anonymous namespace } // end anonymous namespace
namespace mlir { namespace mlir {
namespace spirv { namespace spirv {
void populateSPIRVGLSLCanonicalizationPatterns( void populateSPIRVGLSLCanonicalizationPatterns(RewritePatternSet &results) {
OwningRewritePatternList &results) { results.add<ConvertComparisonIntoClampSPV_FOrdLessThanOp,
results.insert<ConvertComparisonIntoClampSPV_FOrdLessThanOp,
ConvertComparisonIntoClampSPV_FOrdLessThanEqualOp, ConvertComparisonIntoClampSPV_FOrdLessThanEqualOp,
ConvertComparisonIntoClampSPV_SLessThanOp, ConvertComparisonIntoClampSPV_SLessThanOp,
ConvertComparisonIntoClampSPV_SLessThanEqualOp, ConvertComparisonIntoClampSPV_SLessThanEqualOp,
ConvertComparisonIntoClampSPV_ULessThanOp, ConvertComparisonIntoClampSPV_ULessThanOp,
ConvertComparisonIntoClampSPV_ULessThanEqualOp>( ConvertComparisonIntoClampSPV_ULessThanEqualOp>(
results.getContext()); results.getContext());
} }
} // namespace spirv } // namespace spirv
} // namespace mlir } // namespace mlir

mlir/lib/Dialect/SPIRV/Transforms/DecorateCompositeTypeLayoutPass.cpp

Show First 20 Lines • Show All 68 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(spirv::AddressOfOp op,
rewriter.replaceOpWithNewOp<spirv::AddressOfOp>( rewriter.replaceOpWithNewOp<spirv::AddressOfOp>(
op, varOp.type(), rewriter.getSymbolRefAttr(varName)); op, varOp.type(), rewriter.getSymbolRefAttr(varName));
return success(); return success();
} }
}; };
} // namespace } // namespace
static void static void populateSPIRVLayoutInfoPatterns(RewritePatternSet &patterns) {
populateSPIRVLayoutInfoPatterns(OwningRewritePatternList &patterns) { patterns.add<SPIRVGlobalVariableOpLayoutInfoDecoration,
patterns.insert<SPIRVGlobalVariableOpLayoutInfoDecoration,
SPIRVAddressOfOpLayoutInfoDecoration>(patterns.getContext()); SPIRVAddressOfOpLayoutInfoDecoration>(patterns.getContext());
} }
namespace { namespace {
class DecorateSPIRVCompositeTypeLayoutPass class DecorateSPIRVCompositeTypeLayoutPass
: public SPIRVCompositeTypeLayoutBase< : public SPIRVCompositeTypeLayoutBase<
DecorateSPIRVCompositeTypeLayoutPass> { DecorateSPIRVCompositeTypeLayoutPass> {
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void DecorateSPIRVCompositeTypeLayoutPass::runOnOperation() { void DecorateSPIRVCompositeTypeLayoutPass::runOnOperation() {
auto module = getOperation(); auto module = getOperation();
OwningRewritePatternList patterns(module.getContext()); RewritePatternSet patterns(module.getContext());
populateSPIRVLayoutInfoPatterns(patterns); populateSPIRVLayoutInfoPatterns(patterns);
ConversionTarget target(*(module.getContext())); ConversionTarget target(*(module.getContext()));
target.addLegalDialect<spirv::SPIRVDialect>(); target.addLegalDialect<spirv::SPIRVDialect>();
target.addLegalOp<FuncOp>(); target.addLegalOp<FuncOp>();
target.addDynamicallyLegalOp<spirv::GlobalVariableOp>( target.addDynamicallyLegalOp<spirv::GlobalVariableOp>(
[](spirv::GlobalVariableOp op) { [](spirv::GlobalVariableOp op) {
return VulkanLayoutUtils::isLegalType(op.type()); return VulkanLayoutUtils::isLegalType(op.type());
}); });
Show All 18 Lines

mlir/lib/Dialect/SPIRV/Transforms/LowerABIAttributesPass.cpp

Show First 20 Lines • Show All 240 Lines • ▼ Show 20 Linesvoid LowerABIAttributesPass::runOnOperation() {
typeConverter.addSourceMaterialization([](OpBuilder &builder, typeConverter.addSourceMaterialization([](OpBuilder &builder,
spirv::PointerType type, spirv::PointerType type,
ValueRange inputs, Location loc) { ValueRange inputs, Location loc) {
if (inputs.size() != 1 || !inputs[0].getType().isa<spirv::PointerType>()) if (inputs.size() != 1 || !inputs[0].getType().isa<spirv::PointerType>())
return Value(); return Value();
return builder.create<spirv::BitcastOp>(loc, type, inputs[0]).getResult(); return builder.create<spirv::BitcastOp>(loc, type, inputs[0]).getResult();
}); });
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
patterns.insert<ProcessInterfaceVarABI>(typeConverter, context); patterns.add<ProcessInterfaceVarABI>(typeConverter, context);
ConversionTarget target(*context); ConversionTarget target(*context);
// "Legal" function ops should have no interface variable ABI attributes. // "Legal" function ops should have no interface variable ABI attributes.
target.addDynamicallyLegalOp<spirv::FuncOp>([&](spirv::FuncOp op) { target.addDynamicallyLegalOp<spirv::FuncOp>([&](spirv::FuncOp op) {
StringRef attrName = spirv::getInterfaceVarABIAttrName(); StringRef attrName = spirv::getInterfaceVarABIAttrName();
for (unsigned i = 0, e = op.getNumArguments(); i < e; ++i) for (unsigned i = 0, e = op.getNumArguments(); i < e; ++i)
if (op.getArgAttr(i, attrName)) if (op.getArgAttr(i, attrName))
return false; return false;
Show All 31 Lines

mlir/lib/Dialect/SPIRV/Transforms/SPIRVConversion.cpp

Show First 20 Lines • Show All 508 Lines • ▼ Show 20 Lines rewriter.inlineRegionBefore(funcOp.getBody(), newFuncOp.getBody(),
newFuncOp.end()); newFuncOp.end());
if (failed(rewriter.convertRegionTypes( if (failed(rewriter.convertRegionTypes(
&newFuncOp.getBody(), *getTypeConverter(), &signatureConverter))) &newFuncOp.getBody(), *getTypeConverter(), &signatureConverter)))
return failure(); return failure();
rewriter.eraseOp(funcOp); rewriter.eraseOp(funcOp);
return success(); return success();
} }
void mlir::populateBuiltinFuncToSPIRVPatterns( void mlir::populateBuiltinFuncToSPIRVPatterns(SPIRVTypeConverter &typeConverter,
SPIRVTypeConverter &typeConverter, OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<FuncOpConversion>(typeConverter, patterns.getContext()); patterns.add<FuncOpConversion>(typeConverter, patterns.getContext());
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Builtin Variables // Builtin Variables
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static spirv::GlobalVariableOp getBuiltinVariable(Block &body, static spirv::GlobalVariableOp getBuiltinVariable(Block &body,
spirv::BuiltIn builtin) { spirv::BuiltIn builtin) {
▲ Show 20 Lines • Show All 240 Lines • Show Last 20 Lines

mlir/lib/Dialect/Shape/IR/Shape.cpp

Show First 20 Lines • Show All 264 Lines • ▼ Show 20 Lines if (!witness || !witness.passingAttr())
return failure(); return failure();
AssumingOp::inlineRegionIntoParent(op, rewriter); AssumingOp::inlineRegionIntoParent(op, rewriter);
return success(); return success();
} }
}; };
} // namespace } // namespace
void AssumingOp::getCanonicalizationPatterns(OwningRewritePatternList &patterns, void AssumingOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
// If taking a passing witness, inline region. // If taking a passing witness, inline region.
patterns.insert<AssumingWithTrue>(context); patterns.add<AssumingWithTrue>(context);
} }
// See RegionBranchOpInterface in Interfaces/ControlFlowInterfaces.td // See RegionBranchOpInterface in Interfaces/ControlFlowInterfaces.td
void AssumingOp::getSuccessorRegions( void AssumingOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, Optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// AssumingOp has unconditional control flow into the region and back to the // AssumingOp has unconditional control flow into the region and back to the
// parent, so return the correct RegionSuccessor purely based on the index // parent, so return the correct RegionSuccessor purely based on the index
Show All 25 Linesvoid AssumingOp::inlineRegionIntoParent(AssumingOp &op,
rewriter.mergeBlocks(assumingBlock, blockBeforeAssuming); rewriter.mergeBlocks(assumingBlock, blockBeforeAssuming);
rewriter.mergeBlocks(blockAfterAssuming, blockBeforeAssuming); rewriter.mergeBlocks(blockAfterAssuming, blockBeforeAssuming);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AssumingAllOp // AssumingAllOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void AssumingAllOp::getCanonicalizationPatterns( void AssumingAllOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<AssumingAllOneOp>(context); patterns.add<AssumingAllOneOp>(context);
} }
OpFoldResult AssumingAllOp::fold(ArrayRef<Attribute> operands) { OpFoldResult AssumingAllOp::fold(ArrayRef<Attribute> operands) {
// Iterate in reverse to first handle all constant operands. They are // Iterate in reverse to first handle all constant operands. They are
// guaranteed to be the tail of the inputs because this is commutative. // guaranteed to be the tail of the inputs because this is commutative.
for (int idx = operands.size() - 1; idx >= 0; idx--) { for (int idx = operands.size() - 1; idx >= 0; idx--) {
Attribute a = operands[idx]; Attribute a = operands[idx];
// Cannot fold if any inputs are not constant; // Cannot fold if any inputs are not constant;
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Lines if (op.getNumOperands() == 1) {
rewriter.replaceOp(op, op.shapes().front()); rewriter.replaceOp(op, op.shapes().front());
return success(); return success();
} }
return failure(); return failure();
} }
}; };
} // namespace } // namespace
void BroadcastOp::getCanonicalizationPatterns( void BroadcastOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<BroadcastForwardSingleOperandPattern, patterns.add<BroadcastForwardSingleOperandPattern,
RemoveDuplicateOperandsPattern<BroadcastOp>>(context); RemoveDuplicateOperandsPattern<BroadcastOp>>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ConcatOp // ConcatOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
OpFoldResult ConcatOp::fold(ArrayRef<Attribute> operands) { OpFoldResult ConcatOp::fold(ArrayRef<Attribute> operands) {
if (!operands[0] || !operands[1]) if (!operands[0] || !operands[1])
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Linesstatic ParseResult parseConstShapeOp(OpAsmParser &parser,
if (parser.parseColonType(resultTy)) if (parser.parseColonType(resultTy))
return failure(); return failure();
result.types.push_back(resultTy); result.types.push_back(resultTy);
return success(); return success();
} }
OpFoldResult ConstShapeOp::fold(ArrayRef<Attribute>) { return shapeAttr(); } OpFoldResult ConstShapeOp::fold(ArrayRef<Attribute>) { return shapeAttr(); }
void ConstShapeOp::getCanonicalizationPatterns( void ConstShapeOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<TensorCastConstShape>(context); patterns.add<TensorCastConstShape>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CstrBroadcastableOp // CstrBroadcastableOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace { namespace {
// Given an input shape Value, try to obtain the shape's values. // Given an input shape Value, try to obtain the shape's values.
Show All 9 Lines if (auto inputOp = input.getDefiningOp<ShapeOfOp>()) {
return success(); return success();
} else { } else {
return failure(); return failure();
} }
} }
} // namespace } // namespace
void CstrBroadcastableOp::getCanonicalizationPatterns( void CstrBroadcastableOp::getCanonicalizationPatterns(
OwningRewritePatternList &patterns, MLIRContext *context) { RewritePatternSet &patterns, MLIRContext *context) {
// Canonicalization patterns have overlap with the considerations during // Canonicalization patterns have overlap with the considerations during
// folding in case additional shape information is inferred at some point that // folding in case additional shape information is inferred at some point that
// does not result in folding. // does not result in folding.
patterns.insert<CstrBroadcastableEqOps>(context); patterns.add<CstrBroadcastableEqOps>(context);
} }
// Return true if there is exactly one attribute not representing a scalar // Return true if there is exactly one attribute not representing a scalar
// broadcast. // broadcast.
static bool hasAtMostSingleNonScalar(ArrayRef<Attribute> attributes) { static bool hasAtMostSingleNonScalar(ArrayRef<Attribute> attributes) {
bool nonScalarSeen = false; bool nonScalarSeen = false;
for (Attribute a : attributes) { for (Attribute a : attributes) {
if (!a || a.cast<DenseIntElementsAttr>().getNumElements() != 0) { if (!a || a.cast<DenseIntElementsAttr>().getNumElements() != 0) {
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 Lines if (op.getNumOperands() < 2)
return op.emitOpError("required at least 2 input shapes"); return op.emitOpError("required at least 2 input shapes");
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CstrEqOp // CstrEqOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void CstrEqOp::getCanonicalizationPatterns(OwningRewritePatternList &patterns, void CstrEqOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
// If inputs are equal, return passing witness // If inputs are equal, return passing witness
patterns.insert<CstrEqEqOps>(context); patterns.add<CstrEqEqOps>(context);
} }
OpFoldResult CstrEqOp::fold(ArrayRef<Attribute> operands) { OpFoldResult CstrEqOp::fold(ArrayRef<Attribute> operands) {
if (llvm::all_of(operands, if (llvm::all_of(operands,
[&](Attribute a) { return a && a == operands[0]; })) [&](Attribute a) { return a && a == operands[0]; }))
return BoolAttr::get(getContext(), true); return BoolAttr::get(getContext(), true);
// Because a failing witness result here represents an eventual assertion // Because a failing witness result here represents an eventual assertion
▲ Show 20 Lines • Show All 82 Lines • ▼ Show 20 Lines
OpFoldResult IndexToSizeOp::fold(ArrayRef<Attribute> operands) { OpFoldResult IndexToSizeOp::fold(ArrayRef<Attribute> operands) {
// Constant values of both types, `shape.size` and `index`, are represented as // Constant values of both types, `shape.size` and `index`, are represented as
// `IntegerAttr`s which makes constant folding simple. // `IntegerAttr`s which makes constant folding simple.
if (Attribute arg = operands[0]) if (Attribute arg = operands[0])
return arg; return arg;
return {}; return {};
} }
void IndexToSizeOp::getCanonicalizationPatterns( void IndexToSizeOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<SizeToIndexToSizeCanonicalization>(context); patterns.add<SizeToIndexToSizeCanonicalization>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// FromExtentsOp // FromExtentsOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
OpFoldResult FromExtentsOp::fold(ArrayRef<Attribute> operands) { OpFoldResult FromExtentsOp::fold(ArrayRef<Attribute> operands) {
if (llvm::any_of(operands, [](Attribute a) { return !a; })) if (llvm::any_of(operands, [](Attribute a) { return !a; }))
▲ Show 20 Lines • Show All 101 Lines • ▼ Show 20 Lines if (shape.getType().isa<ShapeType>()) {
build(builder, result, builder.getIndexType(), shape, dim); build(builder, result, builder.getIndexType(), shape, dim);
} }
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// IsBroadcastableOp // IsBroadcastableOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void IsBroadcastableOp::getCanonicalizationPatterns( void IsBroadcastableOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<RemoveDuplicateOperandsPattern<IsBroadcastableOp>>(context); patterns.add<RemoveDuplicateOperandsPattern<IsBroadcastableOp>>(context);
} }
OpFoldResult IsBroadcastableOp::fold(ArrayRef<Attribute> operands) { OpFoldResult IsBroadcastableOp::fold(ArrayRef<Attribute> operands) {
// Can always broadcast fewer than two shapes. // Can always broadcast fewer than two shapes.
if (operands.size() < 2) { if (operands.size() < 2) {
return BoolAttr::get(getContext(), true); return BoolAttr::get(getContext(), true);
} }
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(shape::RankOp op,
} else { } else {
return failure(); return failure();
} }
return success(); return success();
} }
}; };
} // namespace } // namespace
void shape::RankOp::getCanonicalizationPatterns( void shape::RankOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<RankShapeOfCanonicalizationPattern>(context); patterns.add<RankShapeOfCanonicalizationPattern>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// NumElementsOp // NumElementsOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
OpFoldResult NumElementsOp::fold(ArrayRef<Attribute> operands) { OpFoldResult NumElementsOp::fold(ArrayRef<Attribute> operands) {
▲ Show 20 Lines • Show All 66 Lines • ▼ Show 20 Lines if (op.getType().isa<ShapedType>())
return failure(); return failure();
rewriter.replaceOpWithNewOp<shape::ShapeOfOp>(op.getOperation(), op.arg()); rewriter.replaceOpWithNewOp<shape::ShapeOfOp>(op.getOperation(), op.arg());
return success(); return success();
} }
}; };
} // namespace } // namespace
void ShapeOfOp::getCanonicalizationPatterns(OwningRewritePatternList &patterns, void ShapeOfOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
patterns.insert<ShapeOfWithTensor>(context); patterns.add<ShapeOfWithTensor>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SizeToIndexOp // SizeToIndexOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
OpFoldResult SizeToIndexOp::fold(ArrayRef<Attribute> operands) { OpFoldResult SizeToIndexOp::fold(ArrayRef<Attribute> operands) {
// Constant values of both types, `shape.size` and `index`, are represented as // Constant values of both types, `shape.size` and `index`, are represented as
// `IntegerAttr`s which makes constant folding simple. // `IntegerAttr`s which makes constant folding simple.
if (Attribute arg = operands[0]) if (Attribute arg = operands[0])
return arg; return arg;
return impl::foldCastOp(*this); return impl::foldCastOp(*this);
} }
void SizeToIndexOp::getCanonicalizationPatterns( void SizeToIndexOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, MLIRContext *context) { MLIRContext *context) {
patterns.insert<IndexToSizeToIndexCanonicalization>(context); patterns.add<IndexToSizeToIndexCanonicalization>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// YieldOp // YieldOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(shape::YieldOp op) { static LogicalResult verify(shape::YieldOp op) {
auto *parentOp = op->getParentOp(); auto *parentOp = op->getParentOp();
▲ Show 20 Lines • Show All 159 Lines • Show Last 20 Lines

mlir/lib/Dialect/Shape/Transforms/Bufferize.cpp

Show All 13 Lines
using namespace mlir; using namespace mlir;
namespace { namespace {
struct ShapeBufferizePass : public ShapeBufferizeBase<ShapeBufferizePass> { struct ShapeBufferizePass : public ShapeBufferizeBase<ShapeBufferizePass> {
void runOnFunction() override { void runOnFunction() override {
MLIRContext &ctx = getContext(); MLIRContext &ctx = getContext();
OwningRewritePatternList patterns(&ctx); RewritePatternSet patterns(&ctx);
BufferizeTypeConverter typeConverter; BufferizeTypeConverter typeConverter;
ConversionTarget target(ctx); ConversionTarget target(ctx);
populateBufferizeMaterializationLegality(target); populateBufferizeMaterializationLegality(target);
populateShapeStructuralTypeConversionsAndLegality(typeConverter, patterns, populateShapeStructuralTypeConversionsAndLegality(typeConverter, patterns,
target); target);
if (failed( if (failed(
Show All 9 Lines

mlir/lib/Dialect/Shape/Transforms/RemoveShapeConstraints.cpp

Show All 40 Lines
/// Removal pass. /// Removal pass.
class RemoveShapeConstraintsPass class RemoveShapeConstraintsPass
: public RemoveShapeConstraintsBase<RemoveShapeConstraintsPass> { : public RemoveShapeConstraintsBase<RemoveShapeConstraintsPass> {
void runOnFunction() override { void runOnFunction() override {
MLIRContext &ctx = getContext(); MLIRContext &ctx = getContext();
OwningRewritePatternList patterns(&ctx); RewritePatternSet patterns(&ctx);
populateRemoveShapeConstraintsPatterns(patterns); populateRemoveShapeConstraintsPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
} }
}; };
} // namespace } // namespace
void mlir::populateRemoveShapeConstraintsPatterns( void mlir::populateRemoveShapeConstraintsPatterns(RewritePatternSet &patterns) {
OwningRewritePatternList &patterns) { patterns.add<RemoveCstrBroadcastableOp, RemoveCstrEqOp>(
patterns.insert<RemoveCstrBroadcastableOp, RemoveCstrEqOp>(
patterns.getContext()); patterns.getContext());
} }
std::unique_ptr<FunctionPass> mlir::createRemoveShapeConstraintsPass() { std::unique_ptr<FunctionPass> mlir::createRemoveShapeConstraintsPass() {
return std::make_unique<RemoveShapeConstraintsPass>(); return std::make_unique<RemoveShapeConstraintsPass>();
} }

mlir/lib/Dialect/Shape/Transforms/ShapeToShapeLowering.cpp

Show First 20 Lines • Show All 55 Lines • ▼ Show 20 Linesstruct ShapeToShapeLowering
: public ShapeToShapeLoweringBase<ShapeToShapeLowering> { : public ShapeToShapeLoweringBase<ShapeToShapeLowering> {
void runOnFunction() override; void runOnFunction() override;
}; };
} // namespace } // namespace
void ShapeToShapeLowering::runOnFunction() { void ShapeToShapeLowering::runOnFunction() {
MLIRContext &ctx = getContext(); MLIRContext &ctx = getContext();
OwningRewritePatternList patterns(&ctx); RewritePatternSet patterns(&ctx);
populateShapeRewritePatterns(patterns); populateShapeRewritePatterns(patterns);
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addLegalDialect<ShapeDialect, StandardOpsDialect>(); target.addLegalDialect<ShapeDialect, StandardOpsDialect>();
target.addIllegalOp<NumElementsOp>(); target.addIllegalOp<NumElementsOp>();
if (failed(mlir::applyPartialConversion(getFunction(), target, if (failed(mlir::applyPartialConversion(getFunction(), target,
std::move(patterns)))) std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
void mlir::populateShapeRewritePatterns(OwningRewritePatternList &patterns) { void mlir::populateShapeRewritePatterns(RewritePatternSet &patterns) {
patterns.insert<NumElementsOpConverter>(patterns.getContext()); patterns.add<NumElementsOpConverter>(patterns.getContext());
} }
std::unique_ptr<Pass> mlir::createShapeToShapeLowering() { std::unique_ptr<Pass> mlir::createShapeToShapeLowering() {
return std::make_unique<ShapeToShapeLowering>(); return std::make_unique<ShapeToShapeLowering>();
} }

mlir/lib/Dialect/Shape/Transforms/StructuralTypeConversions.cpp

Show First 20 Lines • Show All 51 Lines • ▼ Show 20 Lines matchAndRewrite(AssumingYieldOp op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const final { ConversionPatternRewriter &rewriter) const final {
rewriter.replaceOpWithNewOp<AssumingYieldOp>(op, operands); rewriter.replaceOpWithNewOp<AssumingYieldOp>(op, operands);
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateShapeStructuralTypeConversionsAndLegality( void mlir::populateShapeStructuralTypeConversionsAndLegality(
TypeConverter &typeConverter, OwningRewritePatternList &patterns, TypeConverter &typeConverter, RewritePatternSet &patterns,
ConversionTarget &target) { ConversionTarget &target) {
patterns.insert<ConvertAssumingOpTypes, ConvertAssumingYieldOpTypes>( patterns.add<ConvertAssumingOpTypes, ConvertAssumingYieldOpTypes>(
typeConverter, patterns.getContext()); typeConverter, patterns.getContext());
target.addDynamicallyLegalOp<AssumingOp>([&](AssumingOp op) { target.addDynamicallyLegalOp<AssumingOp>([&](AssumingOp op) {
return typeConverter.isLegal(op.getResultTypes()); return typeConverter.isLegal(op.getResultTypes());
}); });
target.addDynamicallyLegalOp<AssumingYieldOp>([&](AssumingYieldOp op) { target.addDynamicallyLegalOp<AssumingYieldOp>([&](AssumingYieldOp op) {
return typeConverter.isLegal(op.getOperandTypes()); return typeConverter.isLegal(op.getOperandTypes());
}); });
} }

mlir/lib/Dialect/StandardOps/IR/Ops.cpp

Show First 20 Lines • Show All 318 Lines • ▼ Show 20 Lines if (matchPattern(op.arg(), m_One())) {
rewriter.eraseOp(op); rewriter.eraseOp(op);
return success(); return success();
} }
return failure(); return failure();
} }
}; };
} // namespace } // namespace
void AssertOp::getCanonicalizationPatterns(OwningRewritePatternList &patterns, void AssertOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
patterns.insert<EraseRedundantAssertions>(context); patterns.add<EraseRedundantAssertions>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AtomicRMWOp // AtomicRMWOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(AtomicRMWOp op) { static LogicalResult verify(AtomicRMWOp op) {
if (op.getMemRefType().getRank() != op.getNumOperands() - 2) if (op.getMemRefType().getRank() != op.getNumOperands() - 2)
▲ Show 20 Lines • Show All 210 Lines • ▼ Show 20 Lines
} // end anonymous namespace. } // end anonymous namespace.
Block *BranchOp::getDest() { return getSuccessor(); } Block *BranchOp::getDest() { return getSuccessor(); }
void BranchOp::setDest(Block *block) { return setSuccessor(block); } void BranchOp::setDest(Block *block) { return setSuccessor(block); }
void BranchOp::eraseOperand(unsigned index) { (*this)->eraseOperand(index); } void BranchOp::eraseOperand(unsigned index) { (*this)->eraseOperand(index); }
void BranchOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void BranchOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyBrToBlockWithSinglePred, SimplifyPassThroughBr>( results.add<SimplifyBrToBlockWithSinglePred, SimplifyPassThroughBr>(context);
context);
} }
Optional<MutableOperandRange> Optional<MutableOperandRange>
BranchOp::getMutableSuccessorOperands(unsigned index) { BranchOp::getMutableSuccessorOperands(unsigned index) {
assert(index == 0 && "invalid successor index"); assert(index == 0 && "invalid successor index");
return destOperandsMutable(); return destOperandsMutable();
} }
▲ Show 20 Lines • Show All 58 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(CallIndirectOp indirectCall,
rewriter.replaceOpWithNewOp<CallOp>(indirectCall, calledFn, rewriter.replaceOpWithNewOp<CallOp>(indirectCall, calledFn,
indirectCall.getResultTypes(), indirectCall.getResultTypes(),
indirectCall.getArgOperands()); indirectCall.getArgOperands());
return success(); return success();
} }
}; };
} // end anonymous namespace. } // end anonymous namespace.
void CallIndirectOp::getCanonicalizationPatterns( void CallIndirectOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyIndirectCallWithKnownCallee>(context); results.add<SimplifyIndirectCallWithKnownCallee>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// General helpers for comparison ops // General helpers for comparison ops
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Return the type of the same shape (scalar, vector or tensor) containing i1. // Return the type of the same shape (scalar, vector or tensor) containing i1.
static Type getI1SameShape(Type type) { static Type getI1SameShape(Type type) {
▲ Show 20 Lines • Show All 315 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(CondBranchOp condbr,
else else
rewriter.replaceOpWithNewOp<BranchOp>(condbr, condbr.falseDest(), rewriter.replaceOpWithNewOp<BranchOp>(condbr, condbr.falseDest(),
condbr.falseDestOperands()); condbr.falseDestOperands());
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void CondBranchOp::getCanonicalizationPatterns( void CondBranchOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<SimplifyConstCondBranchPred, SimplifyPassThroughCondBranch, results.add<SimplifyConstCondBranchPred, SimplifyPassThroughCondBranch,
SimplifyCondBranchIdenticalSuccessors, SimplifyCondBranchIdenticalSuccessors,
SimplifyCondBranchFromCondBranchOnSameCondition>(context); SimplifyCondBranchFromCondBranchOnSameCondition>(context);
} }
Optional<MutableOperandRange> Optional<MutableOperandRange>
CondBranchOp::getMutableSuccessorOperands(unsigned index) { CondBranchOp::getMutableSuccessorOperands(unsigned index) {
assert(index < getNumSuccessors() && "invalid successor index"); assert(index < getNumSuccessors() && "invalid successor index");
return index == trueIndex ? trueDestOperandsMutable() return index == trueIndex ? trueDestOperandsMutable()
: falseDestOperandsMutable(); : falseDestOperandsMutable();
} }
▲ Show 20 Lines • Show All 1,031 Lines • ▼ Show 20 Lines void operator()(PatternRewriter &rewriter, SubTensorOp op,
Value replacement = newOp.getResult(); Value replacement = newOp.getResult();
if (replacement.getType() != op.getType()) if (replacement.getType() != op.getType())
replacement = rewriter.create<tensor::CastOp>(op.getLoc(), op.getType(), replacement = rewriter.create<tensor::CastOp>(op.getLoc(), op.getType(),
replacement); replacement);
rewriter.replaceOp(op, replacement); rewriter.replaceOp(op, replacement);
} }
}; };
void SubTensorOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void SubTensorOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<OpWithOffsetSizesAndStridesConstantArgumentFolder< results.add<OpWithOffsetSizesAndStridesConstantArgumentFolder<
SubTensorOp, SubTensorCanonicalizer>, SubTensorOp, SubTensorCanonicalizer>,
SubTensorOpCastFolder>(context); SubTensorOpCastFolder>(context);
} }
// //
static LogicalResult static LogicalResult
foldIdentityOffsetSizeAndStrideOpInterface(OffsetSizeAndStrideOpInterface op, foldIdentityOffsetSizeAndStrideOpInterface(OffsetSizeAndStrideOpInterface op,
ShapedType shapedType) { ShapedType shapedType) {
OpBuilder b(op.getContext()); OpBuilder b(op.getContext());
for (OpFoldResult ofr : op.getMixedOffsets()) for (OpFoldResult ofr : op.getMixedOffsets())
▲ Show 20 Lines • Show All 150 Lines • ▼ Show 20 Lines if (replacement.getType() != subTensorInsertOp.getType()) {
subTensorInsertOp.getLoc(), subTensorInsertOp.getType(), replacement); subTensorInsertOp.getLoc(), subTensorInsertOp.getType(), replacement);
} }
rewriter.replaceOp(subTensorInsertOp, replacement); rewriter.replaceOp(subTensorInsertOp, replacement);
return success(); return success();
} }
}; };
} // namespace } // namespace
void SubTensorInsertOp::getCanonicalizationPatterns( void SubTensorInsertOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<SubTensorInsertOpConstantArgumentFolder, results.add<SubTensorInsertOpConstantArgumentFolder,
SubTensorInsertOpCastFolder>(context); SubTensorInsertOpCastFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TruncateIOp // TruncateIOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(TruncateIOp op) { static LogicalResult verify(TruncateIOp op) {
auto srcType = getElementTypeOrSelf(op.getOperand().getType()); auto srcType = getElementTypeOrSelf(op.getOperand().getType());
▲ Show 20 Lines • Show All 113 Lines • Show Last 20 Lines

mlir/lib/Dialect/StandardOps/Transforms/Bufferize.cpp

Show First 20 Lines • Show All 49 Lines • ▼ Show 20 Lines matchAndRewrite(SelectOp op, ArrayRef<Value> operands,
rewriter.replaceOpWithNewOp<SelectOp>( rewriter.replaceOpWithNewOp<SelectOp>(
op, adaptor.condition(), adaptor.true_value(), adaptor.false_value()); op, adaptor.condition(), adaptor.true_value(), adaptor.false_value());
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateStdBufferizePatterns(BufferizeTypeConverter &typeConverter, void mlir::populateStdBufferizePatterns(BufferizeTypeConverter &typeConverter,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<BufferizeDimOp, BufferizeSelectOp>(typeConverter, patterns.add<BufferizeDimOp, BufferizeSelectOp>(typeConverter,
patterns.getContext()); patterns.getContext());
} }
namespace { namespace {
struct StdBufferizePass : public StdBufferizeBase<StdBufferizePass> { struct StdBufferizePass : public StdBufferizeBase<StdBufferizePass> {
void runOnFunction() override { void runOnFunction() override {
auto *context = &getContext(); auto *context = &getContext();
BufferizeTypeConverter typeConverter; BufferizeTypeConverter typeConverter;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
ConversionTarget target(*context); ConversionTarget target(*context);
target.addLegalDialect<memref::MemRefDialect>(); target.addLegalDialect<memref::MemRefDialect>();
target.addLegalDialect<StandardOpsDialect>(); target.addLegalDialect<StandardOpsDialect>();
target.addLegalDialect<scf::SCFDialect>(); target.addLegalDialect<scf::SCFDialect>();
populateStdBufferizePatterns(typeConverter, patterns); populateStdBufferizePatterns(typeConverter, patterns);
// We only bufferize the case of tensor selected type and scalar condition, // We only bufferize the case of tensor selected type and scalar condition,
Show All 18 Lines

mlir/lib/Dialect/StandardOps/Transforms/DecomposeCallGraphTypes.cpp

Show First 20 Lines • Show All 182 Lines • ▼ Show 20 Lines matchAndRewrite(CallOp op, ArrayRef<Value> operands,
rewriter.replaceOp(op, replacedValues); rewriter.replaceOp(op, replacedValues);
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateDecomposeCallGraphTypesPatterns( void mlir::populateDecomposeCallGraphTypesPatterns(
MLIRContext *context, TypeConverter &typeConverter, MLIRContext *context, TypeConverter &typeConverter,
ValueDecomposer &decomposer, OwningRewritePatternList &patterns) { ValueDecomposer &decomposer, RewritePatternSet &patterns) {
patterns.insert<DecomposeCallGraphTypesForCallOp, patterns
DecomposeCallGraphTypesForFuncArgs, .add<DecomposeCallGraphTypesForCallOp, DecomposeCallGraphTypesForFuncArgs,
DecomposeCallGraphTypesForReturnOp>(typeConverter, context, DecomposeCallGraphTypesForReturnOp>(typeConverter, context,
decomposer); decomposer);
} }

mlir/lib/Dialect/StandardOps/Transforms/ExpandOps.cpp

Show First 20 Lines • Show All 205 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(SignedFloorDivIOp op,
return success(); return success();
} }
}; };
struct StdExpandOpsPass : public StdExpandOpsBase<StdExpandOpsPass> { struct StdExpandOpsPass : public StdExpandOpsBase<StdExpandOpsPass> {
void runOnFunction() override { void runOnFunction() override {
MLIRContext &ctx = getContext(); MLIRContext &ctx = getContext();
OwningRewritePatternList patterns(&ctx); RewritePatternSet patterns(&ctx);
populateStdExpandOpsPatterns(patterns); populateStdExpandOpsPatterns(patterns);
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addLegalDialect<memref::MemRefDialect, StandardOpsDialect>(); target.addLegalDialect<memref::MemRefDialect, StandardOpsDialect>();
target.addDynamicallyLegalOp<AtomicRMWOp>([](AtomicRMWOp op) { target.addDynamicallyLegalOp<AtomicRMWOp>([](AtomicRMWOp op) {
return op.kind() != AtomicRMWKind::maxf && return op.kind() != AtomicRMWKind::maxf &&
op.kind() != AtomicRMWKind::minf; op.kind() != AtomicRMWKind::minf;
}); });
target.addDynamicallyLegalOp<memref::ReshapeOp>([](memref::ReshapeOp op) { target.addDynamicallyLegalOp<memref::ReshapeOp>([](memref::ReshapeOp op) {
return !op.shape().getType().cast<MemRefType>().hasStaticShape(); return !op.shape().getType().cast<MemRefType>().hasStaticShape();
}); });
target.addIllegalOp<SignedCeilDivIOp>(); target.addIllegalOp<SignedCeilDivIOp>();
target.addIllegalOp<SignedFloorDivIOp>(); target.addIllegalOp<SignedFloorDivIOp>();
if (failed( if (failed(
applyPartialConversion(getFunction(), target, std::move(patterns)))) applyPartialConversion(getFunction(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
}; };
} // namespace } // namespace
void mlir::populateStdExpandOpsPatterns(OwningRewritePatternList &patterns) { void mlir::populateStdExpandOpsPatterns(RewritePatternSet &patterns) {
patterns.insert<AtomicRMWOpConverter, MemRefReshapeOpConverter, patterns.add<AtomicRMWOpConverter, MemRefReshapeOpConverter,
SignedCeilDivIOpConverter, SignedFloorDivIOpConverter>( SignedCeilDivIOpConverter, SignedFloorDivIOpConverter>(
patterns.getContext()); patterns.getContext());
} }
std::unique_ptr<Pass> mlir::createStdExpandOpsPass() { std::unique_ptr<Pass> mlir::createStdExpandOpsPass() {
return std::make_unique<StdExpandOpsPass>(); return std::make_unique<StdExpandOpsPass>();
} }

mlir/lib/Dialect/StandardOps/Transforms/FuncBufferize.cpp

Show All 22 Lines
namespace { namespace {
struct FuncBufferizePass : public FuncBufferizeBase<FuncBufferizePass> { struct FuncBufferizePass : public FuncBufferizeBase<FuncBufferizePass> {
using FuncBufferizeBase<FuncBufferizePass>::FuncBufferizeBase; using FuncBufferizeBase<FuncBufferizePass>::FuncBufferizeBase;
void runOnOperation() override { void runOnOperation() override {
auto module = getOperation(); auto module = getOperation();
auto *context = &getContext(); auto *context = &getContext();
BufferizeTypeConverter typeConverter; BufferizeTypeConverter typeConverter;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
ConversionTarget target(*context); ConversionTarget target(*context);
populateFuncOpTypeConversionPattern(patterns, typeConverter); populateFuncOpTypeConversionPattern(patterns, typeConverter);
target.addDynamicallyLegalOp<FuncOp>([&](FuncOp op) { target.addDynamicallyLegalOp<FuncOp>([&](FuncOp op) {
return typeConverter.isSignatureLegal(op.getType()) && return typeConverter.isSignatureLegal(op.getType()) &&
typeConverter.isLegal(&op.getBody()); typeConverter.isLegal(&op.getBody());
}); });
populateCallOpTypeConversionPattern(patterns, typeConverter); populateCallOpTypeConversionPattern(patterns, typeConverter);
Show All 24 Lines

mlir/lib/Dialect/StandardOps/Transforms/FuncConversions.cpp

Show All 31 Lines matchAndRewrite(CallOp callOp, ArrayRef<Value> operands,
// conversion. // conversion.
rewriter.replaceOpWithNewOp<CallOp>(callOp, callOp.callee(), rewriter.replaceOpWithNewOp<CallOp>(callOp, callOp.callee(),
convertedResults, operands); convertedResults, operands);
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void mlir::populateCallOpTypeConversionPattern( void mlir::populateCallOpTypeConversionPattern(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, TypeConverter &converter) { TypeConverter &converter) {
patterns.insert<CallOpSignatureConversion>(converter, patterns.getContext()); patterns.add<CallOpSignatureConversion>(converter, patterns.getContext());
} }
namespace { namespace {
/// Only needed to support partial conversion of functions where this pattern /// Only needed to support partial conversion of functions where this pattern
/// ensures that the branch operation arguments matches up with the succesor /// ensures that the branch operation arguments matches up with the succesor
/// block arguments. /// block arguments.
class BranchOpInterfaceTypeConversion : public ConversionPattern { class BranchOpInterfaceTypeConversion : public ConversionPattern {
public: public:
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 Lines matchAndRewrite(ReturnOp op, ArrayRef<Value> operands,
rewriter.updateRootInPlace( rewriter.updateRootInPlace(
op, [operands, operation]() { operation->setOperands(operands); }); op, [operands, operation]() { operation->setOperands(operands); });
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void mlir::populateBranchOpInterfaceTypeConversionPattern( void mlir::populateBranchOpInterfaceTypeConversionPattern(
OwningRewritePatternList &patterns, TypeConverter &typeConverter) { RewritePatternSet &patterns, TypeConverter &typeConverter) {
patterns.insert<BranchOpInterfaceTypeConversion>(typeConverter, patterns.add<BranchOpInterfaceTypeConversion>(typeConverter,
patterns.getContext()); patterns.getContext());
} }
bool mlir::isLegalForBranchOpInterfaceTypeConversionPattern( bool mlir::isLegalForBranchOpInterfaceTypeConversionPattern(
Operation *op, TypeConverter &converter) { Operation *op, TypeConverter &converter) {
// All successor operands of branch like operations must be rewritten. // All successor operands of branch like operations must be rewritten.
if (auto branchOp = dyn_cast<BranchOpInterface>(op)) { if (auto branchOp = dyn_cast<BranchOpInterface>(op)) {
for (int p = 0, e = op->getBlock()->getNumSuccessors(); p < e; ++p) { for (int p = 0, e = op->getBlock()->getNumSuccessors(); p < e; ++p) {
auto successorOperands = branchOp.getSuccessorOperands(p); auto successorOperands = branchOp.getSuccessorOperands(p);
if (successorOperands.hasValue() && if (successorOperands.hasValue() &&
!converter.isLegal(successorOperands.getValue().getTypes())) !converter.isLegal(successorOperands.getValue().getTypes()))
return false; return false;
} }
return true; return true;
} }
return false; return false;
} }
void mlir::populateReturnOpTypeConversionPattern( void mlir::populateReturnOpTypeConversionPattern(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, TypeConverter &typeConverter) { TypeConverter &typeConverter) {
patterns.insert<ReturnOpTypeConversion>(typeConverter, patterns.getContext()); patterns.add<ReturnOpTypeConversion>(typeConverter, patterns.getContext());
} }
bool mlir::isLegalForReturnOpTypeConversionPattern(Operation *op, bool mlir::isLegalForReturnOpTypeConversionPattern(Operation *op,
TypeConverter &converter, TypeConverter &converter,
bool returnOpAlwaysLegal) { bool returnOpAlwaysLegal) {
// If this is a `return` and the user pass wants to convert/transform across // If this is a `return` and the user pass wants to convert/transform across
// function boundaries, then `converter` is invoked to check whether the the // function boundaries, then `converter` is invoked to check whether the the
// `return` op is legal. // `return` op is legal.
Show All 24 Lines

mlir/lib/Dialect/StandardOps/Transforms/TensorConstantBufferize.cpp

Show First 20 Lines • Show All 84 Lines • ▼ Show 20 Lines
struct TensorConstantBufferizePass struct TensorConstantBufferizePass
: public TensorConstantBufferizeBase<TensorConstantBufferizePass> { : public TensorConstantBufferizeBase<TensorConstantBufferizePass> {
void runOnOperation() override { void runOnOperation() override {
auto module = getOperation(); auto module = getOperation();
GlobalCreator globals(module); GlobalCreator globals(module);
auto *context = &getContext(); auto *context = &getContext();
BufferizeTypeConverter typeConverter; BufferizeTypeConverter typeConverter;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
ConversionTarget target(*context); ConversionTarget target(*context);
target.addLegalDialect<memref::MemRefDialect>(); target.addLegalDialect<memref::MemRefDialect>();
patterns.insert<BufferizeTensorConstantOp>(globals, typeConverter, context); patterns.add<BufferizeTensorConstantOp>(globals, typeConverter, context);
target.addDynamicallyLegalOp<ConstantOp>( target.addDynamicallyLegalOp<ConstantOp>(
[&](ConstantOp op) { return typeConverter.isLegal(op.getType()); }); [&](ConstantOp op) { return typeConverter.isLegal(op.getType()); });
if (failed(applyPartialConversion(module, target, std::move(patterns)))) if (failed(applyPartialConversion(module, target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
}; };
} // namespace } // namespace
std::unique_ptr<Pass> mlir::createTensorConstantBufferizePass() { std::unique_ptr<Pass> mlir::createTensorConstantBufferizePass() {
return std::make_unique<TensorConstantBufferizePass>(); return std::make_unique<TensorConstantBufferizePass>();
} }

mlir/lib/Dialect/Tensor/IR/TensorOps.cpp

Show First 20 Lines • Show All 171 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(CastOp tensorCast,
rewriter.replaceOpWithNewOp<CastOp>(tensorCast, resultType, rewriter.replaceOpWithNewOp<CastOp>(tensorCast, resultType,
tensorCastOperand.getOperand()); tensorCastOperand.getOperand());
return success(); return success();
} }
}; };
} // namespace } // namespace
void CastOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void CastOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ChainedTensorCast>(context); results.add<ChainedTensorCast>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ExtractOp // ExtractOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(ExtractOp op) { static LogicalResult verify(ExtractOp op) {
// Verify the # indices match if we have a ranked type. // Verify the # indices match if we have a ranked type.
▲ Show 20 Lines • Show All 79 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(tensor::ExtractOp extract,
rewriter.replaceOp(extract, rewriter.replaceOp(extract,
tensorFromElements.getOperand(index.getZExtValue())); tensorFromElements.getOperand(index.getZExtValue()));
return success(); return success();
} }
}; };
} // namespace } // namespace
void FromElementsOp::getCanonicalizationPatterns( void FromElementsOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<ExtractElementFromTensorFromElements>(context); results.add<ExtractElementFromTensorFromElements>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// GenerateOp // GenerateOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(GenerateOp op) { static LogicalResult verify(GenerateOp op) {
// Ensure that the tensor type has as many dynamic dimensions as are specified // Ensure that the tensor type has as many dynamic dimensions as are specified
▲ Show 20 Lines • Show All 141 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(tensor::ExtractOp extract,
rewriter.replaceOpWithNewOp<tensor::ExtractOp>(extract, tensorCast.source(), rewriter.replaceOpWithNewOp<tensor::ExtractOp>(extract, tensorCast.source(),
extract.indices()); extract.indices());
return success(); return success();
} }
}; };
} // namespace } // namespace
void GenerateOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void GenerateOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
// TODO: Move extract patterns to tensor::ExtractOp. // TODO: Move extract patterns to tensor::ExtractOp.
results.insert<ExtractFromTensorGenerate, ExtractFromTensorCast, results.add<ExtractFromTensorGenerate, ExtractFromTensorCast,
StaticTensorGenerate>(context); StaticTensorGenerate>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TableGen'd op method definitions // TableGen'd op method definitions
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
#define GET_OP_CLASSES #define GET_OP_CLASSES
#include "mlir/Dialect/Tensor/IR/TensorOps.cpp.inc" #include "mlir/Dialect/Tensor/IR/TensorOps.cpp.inc"

mlir/lib/Dialect/Tensor/Transforms/Bufferize.cpp

Show First 20 Lines • Show All 132 Lines • ▼ Show 20 Lines matchAndRewrite(tensor::GenerateOp op, ArrayRef<Value> operands,
rewriter.replaceOp(op, {result}); rewriter.replaceOp(op, {result});
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateTensorBufferizePatterns( void mlir::populateTensorBufferizePatterns(
BufferizeTypeConverter &typeConverter, OwningRewritePatternList &patterns) { BufferizeTypeConverter &typeConverter, RewritePatternSet &patterns) {
patterns.insert<BufferizeCastOp, BufferizeExtractOp, BufferizeFromElementsOp, patterns.add<BufferizeCastOp, BufferizeExtractOp, BufferizeFromElementsOp,
BufferizeGenerateOp>(typeConverter, patterns.getContext()); BufferizeGenerateOp>(typeConverter, patterns.getContext());
} }
namespace { namespace {
struct TensorBufferizePass : public TensorBufferizeBase<TensorBufferizePass> { struct TensorBufferizePass : public TensorBufferizeBase<TensorBufferizePass> {
void runOnFunction() override { void runOnFunction() override {
auto *context = &getContext(); auto *context = &getContext();
BufferizeTypeConverter typeConverter; BufferizeTypeConverter typeConverter;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
ConversionTarget target(*context); ConversionTarget target(*context);
populateBufferizeMaterializationLegality(target); populateBufferizeMaterializationLegality(target);
populateTensorBufferizePatterns(typeConverter, patterns); populateTensorBufferizePatterns(typeConverter, patterns);
target.addIllegalOp<tensor::CastOp, tensor::ExtractOp, target.addIllegalOp<tensor::CastOp, tensor::ExtractOp,
tensor::FromElementsOp, tensor::GenerateOp>(); tensor::FromElementsOp, tensor::GenerateOp>();
target.addLegalDialect<memref::MemRefDialect>(); target.addLegalDialect<memref::MemRefDialect>();
Show All 13 Lines

mlir/lib/Dialect/Tosa/Transforms/TosaMakeBroadcastable.cpp

Show First 20 Lines • Show All 245 Lines • ▼ Show 20 Lines
namespace { namespace {
/// Pass that enables broadcast by making all input arrays have the same /// Pass that enables broadcast by making all input arrays have the same
/// number of dimensions. Insert RESHAPE operations to lower rank operand /// number of dimensions. Insert RESHAPE operations to lower rank operand
struct TosaMakeBroadcastable struct TosaMakeBroadcastable
: public TosaMakeBroadcastableBase<TosaMakeBroadcastable> { : public TosaMakeBroadcastableBase<TosaMakeBroadcastable> {
public: public:
void runOnFunction() override { void runOnFunction() override {
auto func = getFunction(); auto func = getFunction();
OwningRewritePatternList patterns(func.getContext()); RewritePatternSet patterns(func.getContext());
MLIRContext *ctx = func.getContext(); MLIRContext *ctx = func.getContext();
// Add the generated patterns to the list. // Add the generated patterns to the list.
patterns.insert<ConvertTosaOp<tosa::AddOp>>(ctx); patterns.add<ConvertTosaOp<tosa::AddOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::SubOp>>(ctx); patterns.add<ConvertTosaOp<tosa::SubOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::MulOp>>(ctx); patterns.add<ConvertTosaOp<tosa::MulOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::MaximumOp>>(ctx); patterns.add<ConvertTosaOp<tosa::MaximumOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::MinimumOp>>(ctx); patterns.add<ConvertTosaOp<tosa::MinimumOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::EqualOp>>(ctx); patterns.add<ConvertTosaOp<tosa::EqualOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::GreaterOp>>(ctx); patterns.add<ConvertTosaOp<tosa::GreaterOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::GreaterEqualOp>>(ctx); patterns.add<ConvertTosaOp<tosa::GreaterEqualOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::LogicalLeftShiftOp>>(ctx); patterns.add<ConvertTosaOp<tosa::LogicalLeftShiftOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::ArithmeticRightShiftOp>>(ctx); patterns.add<ConvertTosaOp<tosa::ArithmeticRightShiftOp>>(ctx);
patterns.insert<ConvertTosaOp<tosa::LogicalRightShiftOp>>(ctx); patterns.add<ConvertTosaOp<tosa::LogicalRightShiftOp>>(ctx);
(void)applyPatternsAndFoldGreedily(func, std::move(patterns)); (void)applyPatternsAndFoldGreedily(func, std::move(patterns));
} }
}; };
} // end anonymous namespace } // end anonymous namespace
std::unique_ptr<Pass> mlir::tosa::createTosaMakeBroadcastablePass() { std::unique_ptr<Pass> mlir::tosa::createTosaMakeBroadcastablePass() {
return std::make_unique<TosaMakeBroadcastable>(); return std::make_unique<TosaMakeBroadcastable>();
} }

mlir/lib/Dialect/Vector/VectorOps.cpp

Show First 20 Lines • Show All 708 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(AddOpType addOp,
Value a = addOp->getOperand(0), b = addOp->getOperand(1); Value a = addOp->getOperand(0), b = addOp->getOperand(1);
vector::ContractionOp contract = canonicalize(a, b); vector::ContractionOp contract = canonicalize(a, b);
contract = contract ? contract : canonicalize(b, a); contract = contract ? contract : canonicalize(b, a);
return success(); return success();
} }
}; };
void ContractionOp::getCanonicalizationPatterns( void ContractionOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results results.add<CanonicalizeContractAdd<AddIOp>, CanonicalizeContractAdd<AddFOp>>(
.insert<CanonicalizeContractAdd<AddIOp>, CanonicalizeContractAdd<AddFOp>>(
context); context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ExtractElementOp // ExtractElementOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void vector::ExtractElementOp::build(OpBuilder &builder, OperationState &result, void vector::ExtractElementOp::build(OpBuilder &builder, OperationState &result,
Value source, Value position) { Value source, Value position) {
▲ Show 20 Lines • Show All 597 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(BroadcastOp broadcastOp,
rewriter.replaceOpWithNewOp<ShapeCastOp>( rewriter.replaceOpWithNewOp<ShapeCastOp>(
broadcastOp, broadcastOp.getVectorType(), broadcastOp.source()); broadcastOp, broadcastOp.getVectorType(), broadcastOp.source());
return success(); return success();
} }
}; };
} // namespace } // namespace
void BroadcastOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void BroadcastOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<BroadcastToShapeCast>(context); results.add<BroadcastToShapeCast>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ShuffleOp // ShuffleOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void ShuffleOp::build(OpBuilder &builder, OperationState &result, Value v1, void ShuffleOp::build(OpBuilder &builder, OperationState &result, Value v1,
Value v2, ArrayRef<int64_t> mask) { Value v2, ArrayRef<int64_t> mask) {
▲ Show 20 Lines • Show All 799 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(ExtractStridedSliceOp op,
rewriter.replaceOpWithNewOp<BroadcastOp>(op, op.getType(), source); rewriter.replaceOpWithNewOp<BroadcastOp>(op, op.getType(), source);
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void ExtractStridedSliceOp::getCanonicalizationPatterns( void ExtractStridedSliceOp::getCanonicalizationPatterns(
OwningRewritePatternList &results, MLIRContext *context) { RewritePatternSet &results, MLIRContext *context) {
// Pattern to rewrite a ExtractStridedSliceOp(ConstantMaskOp) -> // Pattern to rewrite a ExtractStridedSliceOp(ConstantMaskOp) ->
// ConstantMaskOp and ExtractStridedSliceOp(ConstantOp) -> ConstantOp. // ConstantMaskOp and ExtractStridedSliceOp(ConstantOp) -> ConstantOp.
results.insert<StridedSliceConstantMaskFolder, StridedSliceConstantFolder, results.add<StridedSliceConstantMaskFolder, StridedSliceConstantFolder,
StridedSliceBroadcast>(context); StridedSliceBroadcast>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TransferReadOp // TransferReadOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
template <typename EmitFun> template <typename EmitFun>
static LogicalResult verifyPermutationMap(AffineMap permutationMap, static LogicalResult verifyPermutationMap(AffineMap permutationMap,
▲ Show 20 Lines • Show All 607 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(MaskedLoadOp load,
case MaskFormat::Unknown: case MaskFormat::Unknown:
return failure(); return failure();
} }
llvm_unreachable("Unexpected 1DMaskFormat on MaskedLoad"); llvm_unreachable("Unexpected 1DMaskFormat on MaskedLoad");
} }
}; };
} // namespace } // namespace
void MaskedLoadOp::getCanonicalizationPatterns( void MaskedLoadOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<MaskedLoadFolder>(context); results.add<MaskedLoadFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// MaskedStoreOp // MaskedStoreOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(MaskedStoreOp op) { static LogicalResult verify(MaskedStoreOp op) {
VectorType maskVType = op.getMaskVectorType(); VectorType maskVType = op.getMaskVectorType();
Show All 26 Lines LogicalResult matchAndRewrite(MaskedStoreOp store,
case MaskFormat::Unknown: case MaskFormat::Unknown:
return failure(); return failure();
} }
llvm_unreachable("Unexpected 1DMaskFormat on MaskedStore"); llvm_unreachable("Unexpected 1DMaskFormat on MaskedStore");
} }
}; };
} // namespace } // namespace
void MaskedStoreOp::getCanonicalizationPatterns( void MaskedStoreOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<MaskedStoreFolder>(context); results.add<MaskedStoreFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// GatherOp // GatherOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(GatherOp op) { static LogicalResult verify(GatherOp op) {
VectorType indVType = op.getIndexVectorType(); VectorType indVType = op.getIndexVectorType();
Show All 29 Lines LogicalResult matchAndRewrite(GatherOp gather,
case MaskFormat::Unknown: case MaskFormat::Unknown:
return failure(); return failure();
} }
llvm_unreachable("Unexpected 1DMaskFormat on GatherFolder"); llvm_unreachable("Unexpected 1DMaskFormat on GatherFolder");
} }
}; };
} // namespace } // namespace
void GatherOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void GatherOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<GatherFolder>(context); results.add<GatherFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ScatterOp // ScatterOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(ScatterOp op) { static LogicalResult verify(ScatterOp op) {
VectorType indVType = op.getIndexVectorType(); VectorType indVType = op.getIndexVectorType();
Show All 27 Lines LogicalResult matchAndRewrite(ScatterOp scatter,
case MaskFormat::Unknown: case MaskFormat::Unknown:
return failure(); return failure();
} }
llvm_unreachable("Unexpected 1DMaskFormat on ScatterFolder"); llvm_unreachable("Unexpected 1DMaskFormat on ScatterFolder");
} }
}; };
} // namespace } // namespace
void ScatterOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ScatterOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.insert<ScatterFolder>(context); results.add<ScatterFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ExpandLoadOp // ExpandLoadOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(ExpandLoadOp op) { static LogicalResult verify(ExpandLoadOp op) {
VectorType maskVType = op.getMaskVectorType(); VectorType maskVType = op.getMaskVectorType();
Show All 29 Lines LogicalResult matchAndRewrite(ExpandLoadOp expand,
case MaskFormat::Unknown: case MaskFormat::Unknown:
return failure(); return failure();
} }
llvm_unreachable("Unexpected 1DMaskFormat on ExpandLoadFolder"); llvm_unreachable("Unexpected 1DMaskFormat on ExpandLoadFolder");
} }
}; };
} // namespace } // namespace
void ExpandLoadOp::getCanonicalizationPatterns( void ExpandLoadOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<ExpandLoadFolder>(context); results.add<ExpandLoadFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// CompressStoreOp // CompressStoreOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(CompressStoreOp op) { static LogicalResult verify(CompressStoreOp op) {
VectorType maskVType = op.getMaskVectorType(); VectorType maskVType = op.getMaskVectorType();
Show All 27 Lines LogicalResult matchAndRewrite(CompressStoreOp compress,
case MaskFormat::Unknown: case MaskFormat::Unknown:
return failure(); return failure();
} }
llvm_unreachable("Unexpected 1DMaskFormat on CompressStoreFolder"); llvm_unreachable("Unexpected 1DMaskFormat on CompressStoreFolder");
} }
}; };
} // namespace } // namespace
void CompressStoreOp::getCanonicalizationPatterns( void CompressStoreOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<CompressStoreFolder>(context); results.add<CompressStoreFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ShapeCastOp // ShapeCastOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Returns true if each element of 'a' is equal to the product of a contiguous /// Returns true if each element of 'a' is equal to the product of a contiguous
/// sequence of the elements of 'b'. Returns false otherwise. /// sequence of the elements of 'b'. Returns false otherwise.
▲ Show 20 Lines • Show All 117 Lines • ▼ Show 20 Lines auto newAttr = DenseElementsAttr::get(
shapeCastOp.getType().cast<VectorType>(), dense.getSplatValue()); shapeCastOp.getType().cast<VectorType>(), dense.getSplatValue());
rewriter.replaceOpWithNewOp<ConstantOp>(shapeCastOp, newAttr); rewriter.replaceOpWithNewOp<ConstantOp>(shapeCastOp, newAttr);
return success(); return success();
} }
}; };
} // namespace } // namespace
void ShapeCastOp::getCanonicalizationPatterns(OwningRewritePatternList &results, void ShapeCastOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
// Pattern to rewrite a ShapeCastOp(ConstantOp) -> ConstantOp. // Pattern to rewrite a ShapeCastOp(ConstantOp) -> ConstantOp.
results.insert<ShapeCastConstantFolder>(context); results.add<ShapeCastConstantFolder>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// VectorBitCastOp // VectorBitCastOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verify(BitCastOp op) { static LogicalResult verify(BitCastOp op) {
auto sourceVectorType = op.getSourceVectorType(); auto sourceVectorType = op.getSourceVectorType();
▲ Show 20 Lines • Show All 226 Lines • ▼ Show 20 Lines rewriter.replaceOpWithNewOp<vector::TransposeOp>(
vector::getVectorSubscriptAttr(rewriter, permutation)); vector::getVectorSubscriptAttr(rewriter, permutation));
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void vector::TransposeOp::getCanonicalizationPatterns( void vector::TransposeOp::getCanonicalizationPatterns(
OwningRewritePatternList &results, MLIRContext *context) { RewritePatternSet &results, MLIRContext *context) {
results.insert<TransposeFolder>(context); results.add<TransposeFolder>(context);
} }
void vector::TransposeOp::getTransp(SmallVectorImpl<int64_t> &results) { void vector::TransposeOp::getTransp(SmallVectorImpl<int64_t> &results) {
populateFromInt64AttrArray(transp(), results); populateFromInt64AttrArray(transp(), results);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TupleGetOp // TupleGetOp
▲ Show 20 Lines • Show All 117 Lines • ▼ Show 20 Lines rewriter.replaceOpWithNewOp<ConstantMaskOp>(
createMaskOp, createMaskOp.getResult().getType(), createMaskOp, createMaskOp.getResult().getType(),
vector::getVectorSubscriptAttr(rewriter, maskDimSizes)); vector::getVectorSubscriptAttr(rewriter, maskDimSizes));
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void CreateMaskOp::getCanonicalizationPatterns( void CreateMaskOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<CreateMaskFolder>(context); results.add<CreateMaskFolder>(context);
} }
void mlir::vector::populateVectorToVectorCanonicalizationPatterns( void mlir::vector::populateVectorToVectorCanonicalizationPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<CreateMaskFolder, MaskedLoadFolder, MaskedStoreFolder, patterns
GatherFolder, ScatterFolder, ExpandLoadFolder, .add<CreateMaskFolder, MaskedLoadFolder, MaskedStoreFolder, GatherFolder,
CompressStoreFolder, StridedSliceConstantMaskFolder, ScatterFolder, ExpandLoadFolder, CompressStoreFolder,
TransposeFolder>(patterns.getContext()); StridedSliceConstantMaskFolder, TransposeFolder>(
patterns.getContext());
} }
#define GET_OP_CLASSES #define GET_OP_CLASSES
#include "mlir/Dialect/Vector/VectorOps.cpp.inc" #include "mlir/Dialect/Vector/VectorOps.cpp.inc"

mlir/lib/Dialect/Vector/VectorTransforms.cpp

Show First 20 Lines • Show All 3,257 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(vector::BitCastOp bitcastOp,
return success(); return success();
} }
}; };
// TODO: Add pattern to rewrite ExtractSlices(ConstantMaskOp). // TODO: Add pattern to rewrite ExtractSlices(ConstantMaskOp).
// TODO: Add this as DRR pattern. // TODO: Add this as DRR pattern.
void mlir::vector::populateVectorToVectorTransformationPatterns( void mlir::vector::populateVectorToVectorTransformationPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<ShapeCastOpDecomposer, ShapeCastOpFolder, TupleGetFolderOp, patterns.add<ShapeCastOpDecomposer, ShapeCastOpFolder, TupleGetFolderOp,
TransferReadExtractPattern, TransferWriteInsertPattern>( TransferReadExtractPattern, TransferWriteInsertPattern>(
patterns.getContext()); patterns.getContext());
} }
void mlir::vector::populateSplitVectorTransferPatterns( void mlir::vector::populateSplitVectorTransferPatterns(
OwningRewritePatternList &patterns, RewritePatternSet &patterns,
std::function<bool(Operation *)> ignoreFilter) { std::function<bool(Operation *)> ignoreFilter) {
patterns.insert<SplitTransferReadOp, SplitTransferWriteOp>( patterns.add<SplitTransferReadOp, SplitTransferWriteOp>(patterns.getContext(),
patterns.getContext(), ignoreFilter); ignoreFilter);
} }
void mlir::vector::populateCastAwayVectorLeadingOneDimPatterns( void mlir::vector::populateCastAwayVectorLeadingOneDimPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<CastAwayExtractStridedSliceLeadingOneDim, patterns.add<CastAwayExtractStridedSliceLeadingOneDim,
CastAwayInsertStridedSliceLeadingOneDim, CastAwayInsertStridedSliceLeadingOneDim,
CastAwayTransferReadLeadingOneDim, CastAwayTransferReadLeadingOneDim,
CastAwayTransferWriteLeadingOneDim, ShapeCastOpFolder>( CastAwayTransferWriteLeadingOneDim, ShapeCastOpFolder>(
patterns.getContext()); patterns.getContext());
} }
void mlir::vector::populateBubbleVectorBitCastOpPatterns( void mlir::vector::populateBubbleVectorBitCastOpPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<BubbleDownVectorBitCastForExtract, patterns.add<BubbleDownVectorBitCastForExtract,
BubbleDownBitCastForStridedSliceExtract, BubbleDownBitCastForStridedSliceExtract,
BubbleUpBitCastForStridedSliceInsert>(patterns.getContext()); BubbleUpBitCastForStridedSliceInsert>(patterns.getContext());
} }
void mlir::vector::populateVectorSlicesLoweringPatterns( void mlir::vector::populateVectorSlicesLoweringPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<ExtractSlicesOpLowering, InsertSlicesOpLowering>( patterns.add<ExtractSlicesOpLowering, InsertSlicesOpLowering>(
patterns.getContext()); patterns.getContext());
} }
void mlir::vector::populateVectorContractLoweringPatterns( void mlir::vector::populateVectorContractLoweringPatterns(
OwningRewritePatternList &patterns, VectorTransformsOptions parameters) { RewritePatternSet &patterns, VectorTransformsOptions parameters) {
// clang-format off // clang-format off
patterns.insert<BroadcastOpLowering, patterns.add<BroadcastOpLowering,
CreateMaskOpLowering, CreateMaskOpLowering,
ConstantMaskOpLowering, ConstantMaskOpLowering,
OuterProductOpLowering, OuterProductOpLowering,
ShapeCastOp2DDownCastRewritePattern, ShapeCastOp2DDownCastRewritePattern,
ShapeCastOp2DUpCastRewritePattern, ShapeCastOp2DUpCastRewritePattern,
ShapeCastOpRewritePattern>(patterns.getContext()); ShapeCastOpRewritePattern>(patterns.getContext());
patterns.insert<TransposeOpLowering, patterns.add<TransposeOpLowering,
ContractionOpLowering, ContractionOpLowering,
ContractionOpToMatmulOpLowering, ContractionOpToMatmulOpLowering,
ContractionOpToOuterProductOpLowering>(parameters, patterns.getContext()); ContractionOpToOuterProductOpLowering>(parameters, patterns.getContext());
// clang-format on // clang-format on
} }
void mlir::vector::populateVectorTransferLoweringPatterns( void mlir::vector::populateVectorTransferLoweringPatterns(
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<TransferReadToVectorLoadLowering, patterns.add<TransferReadToVectorLoadLowering,
TransferWriteToVectorStoreLowering>(patterns.getContext()); TransferWriteToVectorStoreLowering>(patterns.getContext());
} }

mlir/lib/Rewrite/FrozenRewritePatternList.cpp

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// FrozenRewritePatternList // FrozenRewritePatternList
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
FrozenRewritePatternList::FrozenRewritePatternList() FrozenRewritePatternList::FrozenRewritePatternList()
: impl(std::make_shared<Impl>()) {} : impl(std::make_shared<Impl>()) {}
FrozenRewritePatternList::FrozenRewritePatternList( FrozenRewritePatternList::FrozenRewritePatternList(RewritePatternSet &&patterns)
OwningRewritePatternList &&patterns)
: impl(std::make_shared<Impl>()) { : impl(std::make_shared<Impl>()) {
impl->nativePatterns = std::move(patterns.getNativePatterns()); impl->nativePatterns = std::move(patterns.getNativePatterns());
// Generate the bytecode for the PDL patterns if any were provided. // Generate the bytecode for the PDL patterns if any were provided.
PDLPatternModule &pdlPatterns = patterns.getPDLPatterns(); PDLPatternModule &pdlPatterns = patterns.getPDLPatterns();
ModuleOp pdlModule = pdlPatterns.getModule(); ModuleOp pdlModule = pdlPatterns.getModule();
if (!pdlModule) if (!pdlModule)
return; return;
Show All 11 Lines

mlir/lib/Transforms/Bufferize.cpp

Show First 20 Lines • Show All 78 Lines • ▼ Show 20 Lines matchAndRewrite(memref::BufferCastOp op, ArrayRef<Value> operands,
memref::BufferCastOp::Adaptor adaptor(operands); memref::BufferCastOp::Adaptor adaptor(operands);
rewriter.replaceOp(op, adaptor.tensor()); rewriter.replaceOp(op, adaptor.tensor());
return success(); return success();
} }
}; };
} // namespace } // namespace
void mlir::populateEliminateBufferizeMaterializationsPatterns( void mlir::populateEliminateBufferizeMaterializationsPatterns(
BufferizeTypeConverter &typeConverter, OwningRewritePatternList &patterns) { BufferizeTypeConverter &typeConverter, RewritePatternSet &patterns) {
patterns.insert<BufferizeTensorLoadOp, BufferizeCastOp>( patterns.add<BufferizeTensorLoadOp, BufferizeCastOp>(typeConverter,
typeConverter, patterns.getContext()); patterns.getContext());
} }
namespace { namespace {
struct FinalizingBufferizePass struct FinalizingBufferizePass
: public FinalizingBufferizeBase<FinalizingBufferizePass> { : public FinalizingBufferizeBase<FinalizingBufferizePass> {
using FinalizingBufferizeBase< using FinalizingBufferizeBase<
FinalizingBufferizePass>::FinalizingBufferizeBase; FinalizingBufferizePass>::FinalizingBufferizeBase;
void runOnFunction() override { void runOnFunction() override {
auto func = getFunction(); auto func = getFunction();
auto *context = &getContext(); auto *context = &getContext();
BufferizeTypeConverter typeConverter; BufferizeTypeConverter typeConverter;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
ConversionTarget target(*context); ConversionTarget target(*context);
populateEliminateBufferizeMaterializationsPatterns(typeConverter, patterns); populateEliminateBufferizeMaterializationsPatterns(typeConverter, patterns);
// If all result types are legal, and all block arguments are legal (ensured // If all result types are legal, and all block arguments are legal (ensured
// by func conversion above), then all types in the program are legal. // by func conversion above), then all types in the program are legal.
// //
// We also check that the operand types are legal to avoid creating invalid // We also check that the operand types are legal to avoid creating invalid
Show All 16 Lines

mlir/lib/Transforms/Canonicalizer.cpp

Show All 19 Lines
using namespace mlir; using namespace mlir;
namespace { namespace {
/// Canonicalize operations in nested regions. /// Canonicalize operations in nested regions.
struct Canonicalizer : public CanonicalizerBase<Canonicalizer> { struct Canonicalizer : public CanonicalizerBase<Canonicalizer> {
/// Initialize the canonicalizer by building the set of patterns used during /// Initialize the canonicalizer by building the set of patterns used during
/// execution. /// execution.
LogicalResult initialize(MLIRContext *context) override { LogicalResult initialize(MLIRContext *context) override {
OwningRewritePatternList owningPatterns(context); RewritePatternSet owningPatterns(context);
for (auto *op : context->getRegisteredOperations()) for (auto *op : context->getRegisteredOperations())
op->getCanonicalizationPatterns(owningPatterns, context); op->getCanonicalizationPatterns(owningPatterns, context);
patterns = std::move(owningPatterns); patterns = std::move(owningPatterns);
return success(); return success();
} }
void runOnOperation() override { void runOnOperation() override {
(void)applyPatternsAndFoldGreedily(getOperation()->getRegions(), patterns); (void)applyPatternsAndFoldGreedily(getOperation()->getRegions(), patterns);
} }
Show All 9 Lines

mlir/lib/Transforms/Utils/DialectConversion.cpp

Show First 20 Lines • Show All 2,606 Lines • ▼ Show 20 Lines rewriter.updateRootInPlace(
op, [&] { mlir::impl::setFunctionType(op, newType); }); op, [&] { mlir::impl::setFunctionType(op, newType); });
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void mlir::populateFunctionLikeTypeConversionPattern( void mlir::populateFunctionLikeTypeConversionPattern(
StringRef functionLikeOpName, OwningRewritePatternList &patterns, StringRef functionLikeOpName, RewritePatternSet &patterns,
TypeConverter &converter) { TypeConverter &converter) {
patterns.insert<FunctionLikeSignatureConversion>( patterns.add<FunctionLikeSignatureConversion>(
functionLikeOpName, patterns.getContext(), converter); functionLikeOpName, patterns.getContext(), converter);
} }
void mlir::populateFuncOpTypeConversionPattern( void mlir::populateFuncOpTypeConversionPattern(RewritePatternSet &patterns,
OwningRewritePatternList &patterns, TypeConverter &converter) { TypeConverter &converter) {
populateFunctionLikeTypeConversionPattern<FuncOp>(patterns, converter); populateFunctionLikeTypeConversionPattern<FuncOp>(patterns, converter);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ConversionTarget // ConversionTarget
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Register a legality action for the given operation. /// Register a legality action for the given operation.
▲ Show 20 Lines • Show All 178 Lines • Show Last 20 Lines

mlir/lib/Transforms/Utils/LoopUtils.cpp

Show First 20 Lines • Show All 397 Lines • ▼ Show 20 Lines if (!opGroupQueue.empty()) {
res = generateShiftedLoop(b.getShiftedAffineMap(origLbMap, lbShift), res = generateShiftedLoop(b.getShiftedAffineMap(origLbMap, lbShift),
b.getShiftedAffineMap(origLbMap, d), b.getShiftedAffineMap(origLbMap, d),
opGroupQueue, /*offset=*/0, forOp, b); opGroupQueue, /*offset=*/0, forOp, b);
lbShift = d * step; lbShift = d * step;
} }
if (res) { if (res) {
// Simplify/canonicalize the affine.for. // Simplify/canonicalize the affine.for.
OwningRewritePatternList patterns(res.getContext()); RewritePatternSet patterns(res.getContext());
AffineForOp::getCanonicalizationPatterns(patterns, res.getContext()); AffineForOp::getCanonicalizationPatterns(patterns, res.getContext());
bool erased; bool erased;
(void)applyOpPatternsAndFold(res, std::move(patterns), &erased); (void)applyOpPatternsAndFold(res, std::move(patterns), &erased);
if (!erased && !prologue) if (!erased && !prologue)
prologue = res; prologue = res;
if (!erased) if (!erased)
epilogue = res; epilogue = res;
▲ Show 20 Lines • Show All 2,753 Lines • Show Last 20 Lines

mlir/test/lib/Dialect/Affine/TestAffineDataCopy.cpp

Show First 20 Lines • Show All 104 Lines • ▼ Show 20 Lines nest->walk([&](Operation *op) {
copyOps.push_back(loadOp); copyOps.push_back(loadOp);
else if (auto storeOp = dyn_cast<AffineStoreOp>(op)) else if (auto storeOp = dyn_cast<AffineStoreOp>(op))
copyOps.push_back(storeOp); copyOps.push_back(storeOp);
}); });
// Promoting single iteration loops could lead to simplification of // Promoting single iteration loops could lead to simplification of
// generated load's/store's, and the latter could anyway also be // generated load's/store's, and the latter could anyway also be
// canonicalized. // canonicalized.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
for (auto op : copyOps) { for (auto op : copyOps) {
patterns.clear(); patterns.clear();
if (isa<AffineLoadOp>(op)) { if (isa<AffineLoadOp>(op)) {
AffineLoadOp::getCanonicalizationPatterns(patterns, &getContext()); AffineLoadOp::getCanonicalizationPatterns(patterns, &getContext());
} else { } else {
assert(isa<AffineStoreOp>(op) && "expected affine store op"); assert(isa<AffineStoreOp>(op) && "expected affine store op");
AffineStoreOp::getCanonicalizationPatterns(patterns, &getContext()); AffineStoreOp::getCanonicalizationPatterns(patterns, &getContext());
} }
Show All 10 Lines

mlir/test/lib/Dialect/SPIRV/TestAvailability.cpp

Show First 20 Lines • Show All 133 Lines • ▼ Show 20 Lines auto targetEnv = fn.getOperation()
.cast<spirv::TargetEnvAttr>(); .cast<spirv::TargetEnvAttr>();
if (!targetEnv) { if (!targetEnv) {
fn.emitError("missing 'spv.target_env' attribute"); fn.emitError("missing 'spv.target_env' attribute");
return signalPassFailure(); return signalPassFailure();
} }
auto target = spirv::SPIRVConversionTarget::get(targetEnv); auto target = spirv::SPIRVConversionTarget::get(targetEnv);
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
patterns.insert<ConvertToAtomCmpExchangeWeak, ConvertToBitReverse, patterns.add<ConvertToAtomCmpExchangeWeak, ConvertToBitReverse,
ConvertToGroupNonUniformBallot, ConvertToModule, ConvertToGroupNonUniformBallot, ConvertToModule,
ConvertToSubgroupBallot>(context); ConvertToSubgroupBallot>(context);
if (failed(applyPartialConversion(fn, *target, std::move(patterns)))) if (failed(applyPartialConversion(fn, *target, std::move(patterns))))
return signalPassFailure(); return signalPassFailure();
} }
ConvertToAtomCmpExchangeWeak::ConvertToAtomCmpExchangeWeak(MLIRContext *context) ConvertToAtomCmpExchangeWeak::ConvertToAtomCmpExchangeWeak(MLIRContext *context)
: RewritePattern("test.convert_to_atomic_compare_exchange_weak_op", : RewritePattern("test.convert_to_atomic_compare_exchange_weak_op",
{"spv.AtomicCompareExchangeWeak"}, 1, context) {} {"spv.AtomicCompareExchangeWeak"}, 1, context) {}
▲ Show 20 Lines • Show All 78 Lines • Show Last 20 Lines

mlir/test/lib/Dialect/SPIRV/TestGLSLCanonicalization.cpp

Show All 19 Lines
public: public:
TestGLSLCanonicalizationPass() = default; TestGLSLCanonicalizationPass() = default;
TestGLSLCanonicalizationPass(const TestGLSLCanonicalizationPass &) {} TestGLSLCanonicalizationPass(const TestGLSLCanonicalizationPass &) {}
void runOnOperation() override; void runOnOperation() override;
}; };
} // namespace } // namespace
void TestGLSLCanonicalizationPass::runOnOperation() { void TestGLSLCanonicalizationPass::runOnOperation() {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
spirv::populateSPIRVGLSLCanonicalizationPatterns(patterns); spirv::populateSPIRVGLSLCanonicalizationPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
} }
namespace mlir { namespace mlir {
void registerTestSpirvGLSLCanonicalizationPass() { void registerTestSpirvGLSLCanonicalizationPass() {
PassRegistration<TestGLSLCanonicalizationPass> registration( PassRegistration<TestGLSLCanonicalizationPass> registration(
"test-spirv-glsl-canonicalization", "test-spirv-glsl-canonicalization",
"Tests SPIR-V canonicalization patterns for GLSL extension."); "Tests SPIR-V canonicalization patterns for GLSL extension.");
} }
} // namespace mlir } // namespace mlir

mlir/test/lib/Dialect/Test/TestDialect.cpp

Show First 20 Lines • Show All 229 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(FoldToCallOp op,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
rewriter.replaceOpWithNewOp<CallOp>(op, TypeRange(), op.calleeAttr(), rewriter.replaceOpWithNewOp<CallOp>(op, TypeRange(), op.calleeAttr(),
ValueRange()); ValueRange());
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void FoldToCallOp::getCanonicalizationPatterns( void FoldToCallOp::getCanonicalizationPatterns(RewritePatternSet &results,
OwningRewritePatternList &results, MLIRContext *context) { MLIRContext *context) {
results.insert<FoldToCallOpPattern>(context); results.add<FoldToCallOpPattern>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Test Format* operations // Test Format* operations
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Parsing // Parsing
▲ Show 20 Lines • Show All 361 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(TestOpWithRegionPattern op,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
rewriter.eraseOp(op); rewriter.eraseOp(op);
return success(); return success();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void TestOpWithRegionPattern::getCanonicalizationPatterns( void TestOpWithRegionPattern::getCanonicalizationPatterns(
OwningRewritePatternList &results, MLIRContext *context) { RewritePatternSet &results, MLIRContext *context) {
results.insert<TestRemoveOpWithInnerOps>(context); results.add<TestRemoveOpWithInnerOps>(context);
} }
OpFoldResult TestOpWithRegionFold::fold(ArrayRef<Attribute> operands) { OpFoldResult TestOpWithRegionFold::fold(ArrayRef<Attribute> operands) {
return operand(); return operand();
} }
OpFoldResult TestOpConstant::fold(ArrayRef<Attribute> operands) { OpFoldResult TestOpConstant::fold(ArrayRef<Attribute> operands) {
return getValue(); return getValue();
▲ Show 20 Lines • Show All 275 Lines • Show Last 20 Lines

mlir/test/lib/Dialect/Test/TestPatterns.cpp

Show First 20 Lines • Show All 73 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(Operation *op,
assert(result); assert(result);
rewriter.replaceOp(op, result); rewriter.replaceOp(op, result);
return success(); return success();
} }
}; };
struct TestPatternDriver : public PassWrapper<TestPatternDriver, FunctionPass> { struct TestPatternDriver : public PassWrapper<TestPatternDriver, FunctionPass> {
void runOnFunction() override { void runOnFunction() override {
mlir::OwningRewritePatternList patterns(&getContext()); mlir::RewritePatternSet patterns(&getContext());
populateWithGenerated(patterns); populateWithGenerated(patterns);
// Verify named pattern is generated with expected name. // Verify named pattern is generated with expected name.
patterns.insert<FoldingPattern, TestNamedPatternRule>(&getContext()); patterns.add<FoldingPattern, TestNamedPatternRule>(&getContext());
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
} }
}; };
} // end anonymous namespace } // end anonymous namespace
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ReturnType Driver. // ReturnType Driver.
▲ Show 20 Lines • Show All 457 Lines • ▼ Show 20 Linesstruct TestLegalizePatternDriver
: public PassWrapper<TestLegalizePatternDriver, OperationPass<ModuleOp>> { : public PassWrapper<TestLegalizePatternDriver, OperationPass<ModuleOp>> {
/// The mode of conversion to use with the driver. /// The mode of conversion to use with the driver.
enum class ConversionMode { Analysis, Full, Partial }; enum class ConversionMode { Analysis, Full, Partial };
TestLegalizePatternDriver(ConversionMode mode) : mode(mode) {} TestLegalizePatternDriver(ConversionMode mode) : mode(mode) {}
void runOnOperation() override { void runOnOperation() override {
TestTypeConverter converter; TestTypeConverter converter;
mlir::OwningRewritePatternList patterns(&getContext()); mlir::RewritePatternSet patterns(&getContext());
populateWithGenerated(patterns); populateWithGenerated(patterns);
patterns.insert< patterns
TestRegionRewriteBlockMovement, TestRegionRewriteUndo, TestCreateBlock, .add<TestRegionRewriteBlockMovement, TestRegionRewriteUndo,
TestCreateIllegalBlock, TestUndoBlockArgReplace, TestUndoBlockErase, TestCreateBlock, TestCreateIllegalBlock, TestUndoBlockArgReplace,
TestPassthroughInvalidOp, TestSplitReturnType, TestUndoBlockErase, TestPassthroughInvalidOp, TestSplitReturnType,
TestChangeProducerTypeI32ToF32, TestChangeProducerTypeF32ToF64, TestChangeProducerTypeI32ToF32, TestChangeProducerTypeF32ToF64,
TestChangeProducerTypeF32ToInvalid, TestUpdateConsumerType, TestChangeProducerTypeF32ToInvalid, TestUpdateConsumerType,
TestNonRootReplacement, TestBoundedRecursiveRewrite, TestNonRootReplacement, TestBoundedRecursiveRewrite,
TestNestedOpCreationUndoRewrite>(&getContext()); TestNestedOpCreationUndoRewrite>(&getContext());
patterns.insert<TestDropOpSignatureConversion>(&getContext(), converter); patterns.add<TestDropOpSignatureConversion>(&getContext(), converter);
mlir::populateFuncOpTypeConversionPattern(patterns, converter); mlir::populateFuncOpTypeConversionPattern(patterns, converter);
mlir::populateCallOpTypeConversionPattern(patterns, converter); mlir::populateCallOpTypeConversionPattern(patterns, converter);
// Define the conversion target used for the test. // Define the conversion target used for the test.
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addLegalOp<ModuleOp, ModuleTerminatorOp>(); target.addLegalOp<ModuleOp, ModuleTerminatorOp>();
target.addLegalOp<LegalOpA, LegalOpB, TestCastOp, TestValidOp, target.addLegalOp<LegalOpA, LegalOpB, TestCastOp, TestValidOp,
TerminatorOp>(); TerminatorOp>();
▲ Show 20 Lines • Show All 114 Lines • ▼ Show 20 Lines rewriter.replaceOpWithNewOp<OneVResOneVOperandOp1>(op, op.getResultTypes(),
remappedOperands); remappedOperands);
return success(); return success();
} }
}; };
struct TestRemappedValue struct TestRemappedValue
: public mlir::PassWrapper<TestRemappedValue, FunctionPass> { : public mlir::PassWrapper<TestRemappedValue, FunctionPass> {
void runOnFunction() override { void runOnFunction() override {
mlir::OwningRewritePatternList patterns(&getContext()); mlir::RewritePatternSet patterns(&getContext());
patterns.insert<OneVResOneVOperandOp1Converter>(&getContext()); patterns.add<OneVResOneVOperandOp1Converter>(&getContext());
mlir::ConversionTarget target(getContext()); mlir::ConversionTarget target(getContext());
target.addLegalOp<ModuleOp, ModuleTerminatorOp, FuncOp, TestReturnOp>(); target.addLegalOp<ModuleOp, ModuleTerminatorOp, FuncOp, TestReturnOp>();
// We make OneVResOneVOperandOp1 legal only when it has more that one // We make OneVResOneVOperandOp1 legal only when it has more that one
// operand. This will trigger the conversion that will replace one-operand // operand. This will trigger the conversion that will replace one-operand
// OneVResOneVOperandOp1 with two-operand OneVResOneVOperandOp1. // OneVResOneVOperandOp1 with two-operand OneVResOneVOperandOp1.
target.addDynamicallyLegalOp<OneVResOneVOperandOp1>( target.addDynamicallyLegalOp<OneVResOneVOperandOp1>(
[](Operation *op) -> bool { [](Operation *op) -> bool {
Show All 24 Lines LogicalResult matchAndRewrite(Operation *op,
rewriter.eraseOp(op); rewriter.eraseOp(op);
return success(); return success();
} }
}; };
struct TestUnknownRootOpDriver struct TestUnknownRootOpDriver
: public mlir::PassWrapper<TestUnknownRootOpDriver, FunctionPass> { : public mlir::PassWrapper<TestUnknownRootOpDriver, FunctionPass> {
void runOnFunction() override { void runOnFunction() override {
mlir::OwningRewritePatternList patterns(&getContext()); mlir::RewritePatternSet patterns(&getContext());
patterns.insert<RemoveTestDialectOps>(); patterns.add<RemoveTestDialectOps>();
mlir::ConversionTarget target(getContext()); mlir::ConversionTarget target(getContext());
target.addIllegalDialect<TestDialect>(); target.addIllegalDialect<TestDialect>();
if (failed( if (failed(
applyPartialConversion(getFunction(), target, std::move(patterns)))) applyPartialConversion(getFunction(), target, std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
}; };
▲ Show 20 Lines • Show All 118 Lines • ▼ Show 20 Lines target.addDynamicallyLegalOp<FuncOp>([&](FuncOp op) {
converter.isLegal(&op.getBody()); converter.isLegal(&op.getBody());
}); });
target.addDynamicallyLegalOp<TestCastOp>([&](TestCastOp op) { target.addDynamicallyLegalOp<TestCastOp>([&](TestCastOp op) {
// Allow casts from F64 to F32. // Allow casts from F64 to F32.
return (*op.operand_type_begin()).isF64() && op.getType().isF32(); return (*op.operand_type_begin()).isF64() && op.getType().isF32();
}); });
// Initialize the set of rewrite patterns. // Initialize the set of rewrite patterns.
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
patterns.insert<TestTypeConsumerForward, TestTypeConversionProducer, patterns.add<TestTypeConsumerForward, TestTypeConversionProducer,
TestSignatureConversionUndo>(converter, &getContext()); TestSignatureConversionUndo>(converter, &getContext());
patterns.insert<TestTypeConversionAnotherProducer>(&getContext()); patterns.add<TestTypeConversionAnotherProducer>(&getContext());
mlir::populateFuncOpTypeConversionPattern(patterns, converter); mlir::populateFuncOpTypeConversionPattern(patterns, converter);
if (failed(applyPartialConversion(getOperation(), target, if (failed(applyPartialConversion(getOperation(), target,
std::move(patterns)))) std::move(patterns))))
signalPassFailure(); signalPassFailure();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
▲ Show 20 Lines • Show All 68 Lines • ▼ Show 20 Linesstruct TestMergeSingleBlockOps
} }
}; };
struct TestMergeBlocksPatternDriver struct TestMergeBlocksPatternDriver
: public PassWrapper<TestMergeBlocksPatternDriver, : public PassWrapper<TestMergeBlocksPatternDriver,
OperationPass<ModuleOp>> { OperationPass<ModuleOp>> {
void runOnOperation() override { void runOnOperation() override {
MLIRContext *context = &getContext(); MLIRContext *context = &getContext();
mlir::OwningRewritePatternList patterns(context); mlir::RewritePatternSet patterns(context);
patterns patterns.add<TestMergeBlock, TestUndoBlocksMerge, TestMergeSingleBlockOps>(
.insert<TestMergeBlock, TestUndoBlocksMerge, TestMergeSingleBlockOps>(
context); context);
ConversionTarget target(*context); ConversionTarget target(*context);
target.addLegalOp<FuncOp, ModuleOp, ModuleTerminatorOp, TerminatorOp, target.addLegalOp<FuncOp, ModuleOp, ModuleTerminatorOp, TerminatorOp,
TestBranchOp, TestTypeConsumerOp, TestTypeProducerOp, TestBranchOp, TestTypeConsumerOp, TestTypeProducerOp,
TestReturnOp>(); TestReturnOp>();
target.addIllegalOp<ILLegalOpF>(); target.addIllegalOp<ILLegalOpF>();
/// Expect the op to have a single block after legalization. /// Expect the op to have a single block after legalization.
target.addDynamicallyLegalOp<TestMergeBlocksOp>( target.addDynamicallyLegalOp<TestMergeBlocksOp>(
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Linesstruct TestSelectiveOpReplacementPattern : public OpRewritePattern<TestCastOp> {
} }
}; };
struct TestSelectiveReplacementPatternDriver struct TestSelectiveReplacementPatternDriver
: public PassWrapper<TestSelectiveReplacementPatternDriver, : public PassWrapper<TestSelectiveReplacementPatternDriver,
OperationPass<>> { OperationPass<>> {
void runOnOperation() override { void runOnOperation() override {
MLIRContext *context = &getContext(); MLIRContext *context = &getContext();
mlir::OwningRewritePatternList patterns(context); mlir::RewritePatternSet patterns(context);
patterns.insert<TestSelectiveOpReplacementPattern>(context); patterns.add<TestSelectiveOpReplacementPattern>(context);
(void)applyPatternsAndFoldGreedily(getOperation()->getRegions(), (void)applyPatternsAndFoldGreedily(getOperation()->getRegions(),
std::move(patterns)); std::move(patterns));
} }
}; };
} // namespace } // namespace
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// PassRegistration // PassRegistration
▲ Show 20 Lines • Show All 41 Lines • Show Last 20 Lines

mlir/test/lib/Dialect/Test/TestTraits.cpp

Show All 28 LinesOpFoldResult TestInvolutionTraitSuccesfulOperationFolderOp::fold(
// The success case should cause the trait fold to be supressed. // The success case should cause the trait fold to be supressed.
return argumentOp.getDefiningOp() ? argumentOp : OpFoldResult{}; return argumentOp.getDefiningOp() ? argumentOp : OpFoldResult{};
} }
namespace { namespace {
struct TestTraitFolder : public PassWrapper<TestTraitFolder, FunctionPass> { struct TestTraitFolder : public PassWrapper<TestTraitFolder, FunctionPass> {
void runOnFunction() override { void runOnFunction() override {
(void)applyPatternsAndFoldGreedily(getFunction(), (void)applyPatternsAndFoldGreedily(getFunction(),
OwningRewritePatternList(&getContext())); RewritePatternSet(&getContext()));
} }
}; };
} // end anonymous namespace } // end anonymous namespace
namespace mlir { namespace mlir {
void registerTestTraitsPass() { void registerTestTraitsPass() {
PassRegistration<TestTraitFolder>("test-trait-folder", "Run trait folding"); PassRegistration<TestTraitFolder>("test-trait-folder", "Run trait folding");
} }
} // namespace mlir } // namespace mlir

mlir/test/lib/Dialect/Tosa/TosaTestPasses.cpp

Show First 20 Lines • Show All 178 Lines • ▼ Show 20 Lines
struct TosaTestQuantUtilAPI struct TosaTestQuantUtilAPI
: public PassWrapper<TosaTestQuantUtilAPI, FunctionPass> { : public PassWrapper<TosaTestQuantUtilAPI, FunctionPass> {
void runOnFunction() override; void runOnFunction() override;
}; };
void TosaTestQuantUtilAPI::runOnFunction() { void TosaTestQuantUtilAPI::runOnFunction() {
auto *ctx = &getContext(); auto *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
auto func = getFunction(); auto func = getFunction();
patterns.insert<ConvertTosaNegateOp>(ctx); patterns.add<ConvertTosaNegateOp>(ctx);
patterns.insert<ConvertTosaConv2DOp>(ctx); patterns.add<ConvertTosaConv2DOp>(ctx);
(void)applyPatternsAndFoldGreedily(func, std::move(patterns)); (void)applyPatternsAndFoldGreedily(func, std::move(patterns));
} }
} // anonymous namespace } // anonymous namespace
namespace mlir { namespace mlir {
void registerTosaTestQuantUtilAPIPass() { void registerTosaTestQuantUtilAPIPass() {
PassRegistration<TosaTestQuantUtilAPI>( PassRegistration<TosaTestQuantUtilAPI>(
PASS_NAME, "TOSA Test: Exercise the APIs in QuantUtils.cpp."); PASS_NAME, "TOSA Test: Exercise the APIs in QuantUtils.cpp.");
} }
} // namespace mlir } // namespace mlir

mlir/test/lib/Rewrite/TestPDLByteCode.cpp

Show First 20 Lines • Show All 90 Lines • ▼ Show 20 Lines void runOnOperation() final {
pdlPattern.registerConstraintFunction("multi_entity_var_constraint", pdlPattern.registerConstraintFunction("multi_entity_var_constraint",
customMultiEntityVariadicConstraint); customMultiEntityVariadicConstraint);
pdlPattern.registerRewriteFunction("creator", customCreate); pdlPattern.registerRewriteFunction("creator", customCreate);
pdlPattern.registerRewriteFunction("var_creator", pdlPattern.registerRewriteFunction("var_creator",
customVariadicResultCreate); customVariadicResultCreate);
pdlPattern.registerRewriteFunction("type_creator", customCreateType); pdlPattern.registerRewriteFunction("type_creator", customCreateType);
pdlPattern.registerRewriteFunction("rewriter", customRewriter); pdlPattern.registerRewriteFunction("rewriter", customRewriter);
OwningRewritePatternList patternList(std::move(pdlPattern)); RewritePatternSet patternList(std::move(pdlPattern));
// Invoke the pattern driver with the provided patterns. // Invoke the pattern driver with the provided patterns.
(void)applyPatternsAndFoldGreedily(irModule.getBodyRegion(), (void)applyPatternsAndFoldGreedily(irModule.getBodyRegion(),
std::move(patternList)); std::move(patternList));
} }
}; };
} // end anonymous namespace } // end anonymous namespace
namespace mlir { namespace mlir {
namespace test { namespace test {
void registerTestPDLByteCodePass() { void registerTestPDLByteCodePass() {
PassRegistration<TestPDLByteCodePass>("test-pdl-bytecode-pass", PassRegistration<TestPDLByteCodePass>("test-pdl-bytecode-pass",
"Test PDL ByteCode functionality"); "Test PDL ByteCode functionality");
} }
} // namespace test } // namespace test
} // namespace mlir } // namespace mlir

mlir/test/lib/Transforms/TestConvVectorization.cpp

Show First 20 Lines • Show All 53 Lines • ▼ Show 20 Linesvoid TestConvVectorization::runOnOperation() {
ModuleOp module = getOperation(); ModuleOp module = getOperation();
ConversionTarget target(*context); ConversionTarget target(*context);
target.addLegalDialect<AffineDialect, scf::SCFDialect, StandardOpsDialect, target.addLegalDialect<AffineDialect, scf::SCFDialect, StandardOpsDialect,
VectorDialect>(); VectorDialect>();
target.addLegalOp<ModuleOp, FuncOp, ModuleTerminatorOp, ReturnOp>(); target.addLegalOp<ModuleOp, FuncOp, ModuleTerminatorOp, ReturnOp>();
target.addLegalOp<linalg::FillOp, linalg::YieldOp>(); target.addLegalOp<linalg::FillOp, linalg::YieldOp>();
SmallVector<OwningRewritePatternList, 4> stage1Patterns; SmallVector<RewritePatternSet, 4> stage1Patterns;
linalg::populateConvVectorizationPatterns(context, stage1Patterns, tileSizes); linalg::populateConvVectorizationPatterns(context, stage1Patterns, tileSizes);
SmallVector<FrozenRewritePatternList, 4> frozenStage1Patterns; SmallVector<FrozenRewritePatternList, 4> frozenStage1Patterns;
llvm::move(stage1Patterns, std::back_inserter(frozenStage1Patterns)); llvm::move(stage1Patterns, std::back_inserter(frozenStage1Patterns));
OwningRewritePatternList stage2Patterns = RewritePatternSet stage2Patterns =
linalg::getLinalgTilingCanonicalizationPatterns(context); linalg::getLinalgTilingCanonicalizationPatterns(context);
stage2Patterns.insert<linalg::AffineMinSCFCanonicalizationPattern>(context); stage2Patterns.add<linalg::AffineMinSCFCanonicalizationPattern>(context);
auto stage3Transforms = [](Operation *op) { auto stage3Transforms = [](Operation *op) {
PassManager pm(op->getContext()); PassManager pm(op->getContext());
pm.addPass(createLoopInvariantCodeMotionPass()); pm.addPass(createLoopInvariantCodeMotionPass());
if (failed(pm.run(cast<ModuleOp>(op)))) if (failed(pm.run(cast<ModuleOp>(op))))
llvm_unreachable("Unexpected failure in cleanup pass pipeline."); llvm_unreachable("Unexpected failure in cleanup pass pipeline.");
op->walk([](FuncOp func) { op->walk([](FuncOp func) {
promoteSingleIterationLoops(func); promoteSingleIterationLoops(func);
linalg::hoistRedundantVectorTransfers(func); linalg::hoistRedundantVectorTransfers(func);
}); });
return success(); return success();
}; };
(void)linalg::applyStagedPatterns(module, frozenStage1Patterns, (void)linalg::applyStagedPatterns(module, frozenStage1Patterns,
std::move(stage2Patterns), std::move(stage2Patterns),
stage3Transforms); stage3Transforms);
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Post staged patterns transforms // Post staged patterns transforms
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
VectorTransformsOptions vectorTransformsOptions{ VectorTransformsOptions vectorTransformsOptions{
VectorContractLowering::Dot, VectorTransposeLowering::EltWise}; VectorContractLowering::Dot, VectorTransposeLowering::EltWise};
OwningRewritePatternList vectorTransferPatterns(context); RewritePatternSet vectorTransferPatterns(context);
// Pattern is not applied because rank-reducing vector transfer is not yet // Pattern is not applied because rank-reducing vector transfer is not yet
// supported as can be seen in splitFullAndPartialTransferPrecondition, // supported as can be seen in splitFullAndPartialTransferPrecondition,
// VectorTransforms.cpp // VectorTransforms.cpp
vectorTransferPatterns.insert<VectorTransferFullPartialRewriter>( vectorTransferPatterns.add<VectorTransferFullPartialRewriter>(
context, vectorTransformsOptions); context, vectorTransformsOptions);
(void)applyPatternsAndFoldGreedily(module, std::move(vectorTransferPatterns)); (void)applyPatternsAndFoldGreedily(module, std::move(vectorTransferPatterns));
// Programmatic controlled lowering of linalg.copy and linalg.fill. // Programmatic controlled lowering of linalg.copy and linalg.fill.
PassManager pm(context); PassManager pm(context);
pm.addNestedPass<FuncOp>(createConvertLinalgToLoopsPass()); pm.addNestedPass<FuncOp>(createConvertLinalgToLoopsPass());
if (failed(pm.run(module))) if (failed(pm.run(module)))
llvm_unreachable("Unexpected failure in linalg to loops pass."); llvm_unreachable("Unexpected failure in linalg to loops pass.");
// Programmatic controlled lowering of vector.contract only. // Programmatic controlled lowering of vector.contract only.
OwningRewritePatternList vectorContractLoweringPatterns(context); RewritePatternSet vectorContractLoweringPatterns(context);
populateVectorContractLoweringPatterns(vectorContractLoweringPatterns, populateVectorContractLoweringPatterns(vectorContractLoweringPatterns,
vectorTransformsOptions); vectorTransformsOptions);
(void)applyPatternsAndFoldGreedily(module, (void)applyPatternsAndFoldGreedily(module,
std::move(vectorContractLoweringPatterns)); std::move(vectorContractLoweringPatterns));
// Programmatic controlled lowering of vector.transfer only. // Programmatic controlled lowering of vector.transfer only.
OwningRewritePatternList vectorToLoopsPatterns(context); RewritePatternSet vectorToLoopsPatterns(context);
populateVectorToSCFConversionPatterns(vectorToLoopsPatterns, populateVectorToSCFConversionPatterns(vectorToLoopsPatterns,
VectorTransferToSCFOptions()); VectorTransferToSCFOptions());
(void)applyPatternsAndFoldGreedily(module, std::move(vectorToLoopsPatterns)); (void)applyPatternsAndFoldGreedily(module, std::move(vectorToLoopsPatterns));
// Ensure we drop the marker in the end. // Ensure we drop the marker in the end.
module.walk([](linalg::LinalgOp op) { module.walk([](linalg::LinalgOp op) {
op->removeAttr(linalg::LinalgTransforms::kLinalgTransformMarker); op->removeAttr(linalg::LinalgTransforms::kLinalgTransformMarker);
}); });
Show All 10 Lines

mlir/test/lib/Transforms/TestConvertCallOp.cpp

Show First 20 Lines • Show All 43 Lines • ▼ Show 20 Lines void runOnOperation() override {
// Populate type conversions. // Populate type conversions.
LLVMTypeConverter typeConverter(m.getContext()); LLVMTypeConverter typeConverter(m.getContext());
typeConverter.addConversion([&](test::TestType type) { typeConverter.addConversion([&](test::TestType type) {
return LLVM::LLVMPointerType::get(IntegerType::get(m.getContext(), 8)); return LLVM::LLVMPointerType::get(IntegerType::get(m.getContext(), 8));
}); });
// Populate patterns. // Populate patterns.
OwningRewritePatternList patterns(m.getContext()); RewritePatternSet patterns(m.getContext());
populateStdToLLVMConversionPatterns(typeConverter, patterns); populateStdToLLVMConversionPatterns(typeConverter, patterns);
patterns.insert<TestTypeProducerOpConverter>(typeConverter); patterns.add<TestTypeProducerOpConverter>(typeConverter);
// Set target. // Set target.
ConversionTarget target(getContext()); ConversionTarget target(getContext());
target.addLegalDialect<LLVM::LLVMDialect>(); target.addLegalDialect<LLVM::LLVMDialect>();
target.addIllegalDialect<test::TestDialect>(); target.addIllegalDialect<test::TestDialect>();
target.addIllegalDialect<StandardOpsDialect>(); target.addIllegalDialect<StandardOpsDialect>();
if (failed(applyPartialConversion(m, target, std::move(patterns)))) if (failed(applyPartialConversion(m, target, std::move(patterns))))
Show All 16 Lines

mlir/test/lib/Transforms/TestDecomposeCallGraphTypes.cpp

Show All 27 Lines void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<test::TestDialect>(); registry.insert<test::TestDialect>();
} }
void runOnOperation() override { void runOnOperation() override {
ModuleOp module = getOperation(); ModuleOp module = getOperation();
auto *context = &getContext(); auto *context = &getContext();
TypeConverter typeConverter; TypeConverter typeConverter;
ConversionTarget target(*context); ConversionTarget target(*context);
ValueDecomposer decomposer; ValueDecomposer decomposer;
OwningRewritePatternList patterns(context); RewritePatternSet patterns(context);
target.addLegalDialect<test::TestDialect>(); target.addLegalDialect<test::TestDialect>();
target.addDynamicallyLegalOp<ReturnOp>([&](ReturnOp op) { target.addDynamicallyLegalOp<ReturnOp>([&](ReturnOp op) {
return typeConverter.isLegal(op.getOperandTypes()); return typeConverter.isLegal(op.getOperandTypes());
}); });
target.addDynamicallyLegalOp<CallOp>( target.addDynamicallyLegalOp<CallOp>(
[&](CallOp op) { return typeConverter.isLegal(op); }); [&](CallOp op) { return typeConverter.isLegal(op); });
▲ Show 20 Lines • Show All 51 Lines • Show Last 20 Lines

mlir/test/lib/Transforms/TestExpandTanh.cpp

Show All 18 Lines
namespace { namespace {
struct TestExpandTanhPass struct TestExpandTanhPass
: public PassWrapper<TestExpandTanhPass, FunctionPass> { : public PassWrapper<TestExpandTanhPass, FunctionPass> {
void runOnFunction() override; void runOnFunction() override;
}; };
} // end anonymous namespace } // end anonymous namespace
void TestExpandTanhPass::runOnFunction() { void TestExpandTanhPass::runOnFunction() {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateExpandTanhPattern(patterns); populateExpandTanhPattern(patterns);
(void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
} }
namespace mlir { namespace mlir {
namespace test { namespace test {
void registerTestExpandTanhPass() { void registerTestExpandTanhPass() {
PassRegistration<TestExpandTanhPass> pass("test-expand-tanh", PassRegistration<TestExpandTanhPass> pass("test-expand-tanh",
"Test expanding tanh"); "Test expanding tanh");
} }
} // namespace test } // namespace test
} // namespace mlir } // namespace mlir

mlir/test/lib/Transforms/TestGpuRewrite.cpp

Show All 19 Lines
namespace { namespace {
struct TestGpuRewritePass struct TestGpuRewritePass
: public PassWrapper<TestGpuRewritePass, OperationPass<ModuleOp>> { : public PassWrapper<TestGpuRewritePass, OperationPass<ModuleOp>> {
void getDependentDialects(DialectRegistry &registry) const override { void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<StandardOpsDialect, memref::MemRefDialect>(); registry.insert<StandardOpsDialect, memref::MemRefDialect>();
} }
void runOnOperation() override { void runOnOperation() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateGpuRewritePatterns(patterns); populateGpuRewritePatterns(patterns);
(void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
} }
}; };
} // namespace } // namespace
namespace mlir { namespace mlir {
void registerTestAllReduceLoweringPass() { void registerTestAllReduceLoweringPass() {
PassRegistration<TestGpuRewritePass> pass( PassRegistration<TestGpuRewritePass> pass(
"test-gpu-rewrite", "test-gpu-rewrite",
"Applies all rewrite patterns within the GPU dialect."); "Applies all rewrite patterns within the GPU dialect.");
} }
} // namespace mlir } // namespace mlir

mlir/test/lib/Transforms/TestLinalgFusionTransforms.cpp

Show All 17 Lines
#include "mlir/Transforms/Passes.h" #include "mlir/Transforms/Passes.h"
using namespace mlir; using namespace mlir;
using namespace mlir::linalg; using namespace mlir::linalg;
template <LinalgTilingLoopType LoopType> template <LinalgTilingLoopType LoopType>
static void fillFusionPatterns(MLIRContext *context, static void fillFusionPatterns(MLIRContext *context,
const LinalgDependenceGraph &dependenceGraph, const LinalgDependenceGraph &dependenceGraph,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<LinalgTileAndFusePattern<MatmulOp>, patterns.add<LinalgTileAndFusePattern<MatmulOp>,
LinalgTileAndFusePattern<ConvOp>>( LinalgTileAndFusePattern<ConvOp>>(
context, dependenceGraph, context, dependenceGraph,
LinalgTilingOptions().setTileSizes({32, 64, 16}).setLoopType(LoopType), LinalgTilingOptions().setTileSizes({32, 64, 16}).setLoopType(LoopType),
LinalgFusionOptions().setIndicesToFuse({2}), LinalgFusionOptions().setIndicesToFuse({2}),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("basic_fusion", context), Identifier::get("basic_fusion", context),
Identifier::get("after_basic_fusion", context)), Identifier::get("after_basic_fusion", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Identifier::get("after_basic_fusion_producer", context)), Identifier::get("after_basic_fusion_producer", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Identifier::get("after_basic_fusion_original", context))); Identifier::get("after_basic_fusion_original", context)));
patterns.insert<LinalgTileAndFusePattern<MatmulOp>>( patterns.add<LinalgTileAndFusePattern<MatmulOp>>(
context, dependenceGraph, context, dependenceGraph,
LinalgTilingOptions().setTileSizes({32, 64, 16}).setLoopType(LoopType), LinalgTilingOptions().setTileSizes({32, 64, 16}).setLoopType(LoopType),
LinalgFusionOptions().setIndicesToFuse({0}), LinalgFusionOptions().setIndicesToFuse({0}),
LinalgTransformationFilter(Identifier::get("lhs_fusion", context), LinalgTransformationFilter(Identifier::get("lhs_fusion", context),
Identifier::get("after_lhs_fusion", context)), Identifier::get("after_lhs_fusion", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Identifier::get("after_lhs_fusion_producer", context)), Identifier::get("after_lhs_fusion_producer", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Identifier::get("after_lhs_fusion_original", context))); Identifier::get("after_lhs_fusion_original", context)));
patterns.insert<LinalgTileAndFusePattern<MatmulOp>>( patterns.add<LinalgTileAndFusePattern<MatmulOp>>(
context, dependenceGraph, context, dependenceGraph,
LinalgTilingOptions().setTileSizes({32, 64, 16}).setLoopType(LoopType), LinalgTilingOptions().setTileSizes({32, 64, 16}).setLoopType(LoopType),
LinalgFusionOptions().setIndicesToFuse({1}), LinalgFusionOptions().setIndicesToFuse({1}),
LinalgTransformationFilter(Identifier::get("rhs_fusion", context), LinalgTransformationFilter(Identifier::get("rhs_fusion", context),
Identifier::get("after_rhs_fusion", context)), Identifier::get("after_rhs_fusion", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Identifier::get("after_rhs_fusion_producer", context)), Identifier::get("after_rhs_fusion_producer", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Identifier::get("after_rhs_fusion_original", context))); Identifier::get("after_rhs_fusion_original", context)));
patterns.insert<LinalgTileAndFusePattern<MatmulOp>>( patterns.add<LinalgTileAndFusePattern<MatmulOp>>(
context, dependenceGraph, context, dependenceGraph,
LinalgTilingOptions().setTileSizes({32, 64, 16}).setLoopType(LoopType), LinalgTilingOptions().setTileSizes({32, 64, 16}).setLoopType(LoopType),
LinalgFusionOptions().setIndicesToFuse({0, 2}), LinalgFusionOptions().setIndicesToFuse({0, 2}),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("two_operand_fusion", context), Identifier::get("two_operand_fusion", context),
Identifier::get("after_two_operand_fusion", context)), Identifier::get("after_two_operand_fusion", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Identifier::get("after_two_operand_fusion_producer", context)), Identifier::get("after_two_operand_fusion_producer", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Identifier::get("after_two_operand_fusion_original", context))); Identifier::get("after_two_operand_fusion_original", context)));
patterns.insert<LinalgTileAndFusePattern<GenericOp>>( patterns.add<LinalgTileAndFusePattern<GenericOp>>(
context, dependenceGraph, context, dependenceGraph,
LinalgTilingOptions().setTileSizes({32, 64}).setLoopType(LoopType), LinalgTilingOptions().setTileSizes({32, 64}).setLoopType(LoopType),
LinalgFusionOptions().setIndicesToFuse({0, 1}), LinalgFusionOptions().setIndicesToFuse({0, 1}),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("transpose_fusion", context), Identifier::get("transpose_fusion", context),
Identifier::get("after_transpose_fusion", context)), Identifier::get("after_transpose_fusion", context)),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>(), ArrayRef<Identifier>(),
Show All 13 Linesstruct TestLinalgFusionTransforms
void getDependentDialects(DialectRegistry &registry) const override { void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<AffineDialect, linalg::LinalgDialect, memref::MemRefDialect, registry.insert<AffineDialect, linalg::LinalgDialect, memref::MemRefDialect,
scf::SCFDialect, StandardOpsDialect>(); scf::SCFDialect, StandardOpsDialect>();
} }
void runOnFunction() override { void runOnFunction() override {
MLIRContext *context = &this->getContext(); MLIRContext *context = &this->getContext();
FuncOp funcOp = this->getFunction(); FuncOp funcOp = this->getFunction();
OwningRewritePatternList fusionPatterns(context); RewritePatternSet fusionPatterns(context);
Aliases alias; Aliases alias;
LinalgDependenceGraph dependenceGraph = LinalgDependenceGraph dependenceGraph =
LinalgDependenceGraph::buildDependenceGraph(alias, funcOp); LinalgDependenceGraph::buildDependenceGraph(alias, funcOp);
fillFusionPatterns<LoopType>(context, dependenceGraph, fusionPatterns); fillFusionPatterns<LoopType>(context, dependenceGraph, fusionPatterns);
(void)applyPatternsAndFoldGreedily(funcOp, std::move(fusionPatterns)); (void)applyPatternsAndFoldGreedily(funcOp, std::move(fusionPatterns));
} }
}; };
} // namespace } // namespace
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Linesstatic LogicalResult fuseLinalgOpsGreedily(FuncOp f) {
return changed ? success() : failure(); return changed ? success() : failure();
} }
namespace { namespace {
struct TestLinalgGreedyFusion struct TestLinalgGreedyFusion
: public PassWrapper<TestLinalgGreedyFusion, FunctionPass> { : public PassWrapper<TestLinalgGreedyFusion, FunctionPass> {
void runOnFunction() override { void runOnFunction() override {
MLIRContext *context = &getContext(); MLIRContext *context = &getContext();
OwningRewritePatternList patterns = RewritePatternSet patterns =
linalg::getLinalgTilingCanonicalizationPatterns(context); linalg::getLinalgTilingCanonicalizationPatterns(context);
patterns.insert<AffineMinSCFCanonicalizationPattern>(context); patterns.add<AffineMinSCFCanonicalizationPattern>(context);
FrozenRewritePatternList frozenPatterns(std::move(patterns)); FrozenRewritePatternList frozenPatterns(std::move(patterns));
while (succeeded(fuseLinalgOpsGreedily(getFunction()))) { while (succeeded(fuseLinalgOpsGreedily(getFunction()))) {
(void)applyPatternsAndFoldGreedily(getFunction(), frozenPatterns); (void)applyPatternsAndFoldGreedily(getFunction(), frozenPatterns);
PassManager pm(context); PassManager pm(context);
pm.addPass(createLoopInvariantCodeMotionPass()); pm.addPass(createLoopInvariantCodeMotionPass());
pm.addPass(createCanonicalizerPass()); pm.addPass(createCanonicalizerPass());
pm.addPass(createCSEPass()); pm.addPass(createCSEPass());
LogicalResult res = pm.run(getFunction()->getParentOfType<ModuleOp>()); LogicalResult res = pm.run(getFunction()->getParentOfType<ModuleOp>());
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines

mlir/test/lib/Transforms/TestLinalgTransforms.cpp

Show First 20 Lines • Show All 86 Lines • ▼ Show 20 Linesstruct TestLinalgTransforms
Option<bool> testHoistPadding2Levels{*this, "test-hoist-padding-2-level", Option<bool> testHoistPadding2Levels{*this, "test-hoist-padding-2-level",
llvm::cl::desc("Test hoist padding"), llvm::cl::desc("Test hoist padding"),
llvm::cl::init(false)}; llvm::cl::init(false)};
}; };
} // end anonymous namespace } // end anonymous namespace
static void applyPatterns(FuncOp funcOp) { static void applyPatterns(FuncOp funcOp) {
MLIRContext *ctx = funcOp.getContext(); MLIRContext *ctx = funcOp.getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Linalg tiling patterns. // Linalg tiling patterns.
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, LinalgTilingOptions().setTileSizes({2000, 3000, 4000}), ctx, LinalgTilingOptions().setTileSizes({2000, 3000, 4000}),
LinalgTransformationFilter(Identifier::get("MEM", ctx), LinalgTransformationFilter(Identifier::get("MEM", ctx),
Identifier::get("L3", ctx))); Identifier::get("L3", ctx)));
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, LinalgTilingOptions().setTileSizes({200, 300, 400}), ctx, LinalgTilingOptions().setTileSizes({200, 300, 400}),
LinalgTransformationFilter(Identifier::get("L3", ctx), LinalgTransformationFilter(Identifier::get("L3", ctx),
Identifier::get("L2", ctx))); Identifier::get("L2", ctx)));
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, LinalgTilingOptions().setTileSizes({20, 30, 40}), ctx, LinalgTilingOptions().setTileSizes({20, 30, 40}),
LinalgTransformationFilter(Identifier::get("L2", ctx), LinalgTransformationFilter(Identifier::get("L2", ctx),
Identifier::get("L1", ctx))); Identifier::get("L1", ctx)));
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, LinalgTilingOptions().setTileSizes({2, 3, 4}), ctx, LinalgTilingOptions().setTileSizes({2, 3, 4}),
LinalgTransformationFilter(Identifier::get("L1", ctx), LinalgTransformationFilter(Identifier::get("L1", ctx),
Identifier::get("REG", ctx))); Identifier::get("REG", ctx)));
patterns.insert<LinalgTilingPattern<MatvecOp>>( patterns.add<LinalgTilingPattern<MatvecOp>>(
ctx, ctx,
LinalgTilingOptions().setTileSizes({5, 6}).setLoopType( LinalgTilingOptions().setTileSizes({5, 6}).setLoopType(
LinalgTilingLoopType::ParallelLoops), LinalgTilingLoopType::ParallelLoops),
LinalgTransformationFilter(ArrayRef<Identifier>{}, LinalgTransformationFilter(ArrayRef<Identifier>{},
Identifier::get("L1", ctx))); Identifier::get("L1", ctx)));
patterns.insert<LinalgTilingPattern<DotOp>>( patterns.add<LinalgTilingPattern<DotOp>>(
ctx, LinalgTilingOptions().setTileSizes(8000), ctx, LinalgTilingOptions().setTileSizes(8000),
LinalgTransformationFilter( LinalgTransformationFilter(
ArrayRef<Identifier>{Identifier::get("MEM", ctx), ArrayRef<Identifier>{Identifier::get("MEM", ctx),
Identifier::get("L3", ctx), Identifier::get("L3", ctx),
Identifier::get("L2", ctx)}, Identifier::get("L2", ctx)},
Identifier::get("REG", ctx))); Identifier::get("REG", ctx)));
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Linalg tiling and permutation patterns. // Linalg tiling and permutation patterns.
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, ctx,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({2000, 3000, 4000}) .setTileSizes({2000, 3000, 4000})
.setInterchange({1, 2, 0}), .setInterchange({1, 2, 0}),
LinalgTransformationFilter(Identifier::get("__with_perm__", ctx), LinalgTransformationFilter(Identifier::get("__with_perm__", ctx),
Identifier::get("L2__with_perm__", ctx))); Identifier::get("L2__with_perm__", ctx)));
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, ctx,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({200, 300, 400}) .setTileSizes({200, 300, 400})
.setInterchange({1, 0, 2}), .setInterchange({1, 0, 2}),
LinalgTransformationFilter(Identifier::get("L2__with_perm__", ctx), LinalgTransformationFilter(Identifier::get("L2__with_perm__", ctx),
Identifier::get("L1__with_perm__", ctx))); Identifier::get("L1__with_perm__", ctx)));
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, LinalgTilingOptions().setTileSizes({20, 30, 40}), ctx, LinalgTilingOptions().setTileSizes({20, 30, 40}),
LinalgTransformationFilter(Identifier::get("L1__with_perm__", ctx), LinalgTransformationFilter(Identifier::get("L1__with_perm__", ctx),
Identifier::get("REG__with_perm__", ctx))); Identifier::get("REG__with_perm__", ctx)));
patterns.insert<LinalgTilingPattern<MatvecOp>>( patterns.add<LinalgTilingPattern<MatvecOp>>(
ctx, LinalgTilingOptions().setTileSizes({5, 6}).setInterchange({1, 0}), ctx, LinalgTilingOptions().setTileSizes({5, 6}).setInterchange({1, 0}),
LinalgTransformationFilter(Identifier::get("__with_perm__", ctx), LinalgTransformationFilter(Identifier::get("__with_perm__", ctx),
Identifier::get("L1__with_perm__", ctx))); Identifier::get("L1__with_perm__", ctx)));
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, ctx,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({16, 8, 4}) .setTileSizes({16, 8, 4})
.setInterchange({1, 2, 0}) .setInterchange({1, 2, 0})
.setLoopType(LinalgTilingLoopType::ParallelLoops), .setLoopType(LinalgTilingLoopType::ParallelLoops),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("par__with_perm__", ctx), Identifier::get("par__with_perm__", ctx),
Identifier::get("after_par__with_perm__", ctx))); Identifier::get("after_par__with_perm__", ctx)));
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Linalg to loops patterns. // Linalg to loops patterns.
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
patterns.insert<LinalgLoweringPattern<DotOp>>( patterns.add<LinalgLoweringPattern<DotOp>>(
ctx, ctx,
/*loweringType=*/LinalgLoweringType::Loops, /*loweringType=*/LinalgLoweringType::Loops,
LinalgTransformationFilter(Identifier::get("REG", ctx))); LinalgTransformationFilter(Identifier::get("REG", ctx)));
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Linalg distribution patterns. // Linalg distribution patterns.
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
LinalgLoopDistributionOptions distributionOptions; LinalgLoopDistributionOptions distributionOptions;
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Linalg to vector contraction patterns. // Linalg to vector contraction patterns.
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
patterns.insert<LinalgVectorizationPattern>( patterns.add<LinalgVectorizationPattern>(
LinalgTransformationFilter(Identifier::get("VECTORIZE", ctx)) LinalgTransformationFilter(Identifier::get("VECTORIZE", ctx))
.addOpFilter<MatmulOp, FillOp, CopyOp, GenericOp>()); .addOpFilter<MatmulOp, FillOp, CopyOp, GenericOp>());
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Linalg generic permutation patterns. // Linalg generic permutation patterns.
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
patterns.insert<LinalgInterchangePattern<GenericOp>>( patterns.add<LinalgInterchangePattern<GenericOp>>(
ctx, ctx,
/*interchangeVector=*/ArrayRef<unsigned>{1, 2, 0}, /*interchangeVector=*/ArrayRef<unsigned>{1, 2, 0},
LinalgTransformationFilter(ArrayRef<Identifier>{}, LinalgTransformationFilter(ArrayRef<Identifier>{},
Identifier::get("PERMUTED", ctx))); Identifier::get("PERMUTED", ctx)));
patterns.insert<LinalgInterchangePattern<IndexedGenericOp>>( patterns.add<LinalgInterchangePattern<IndexedGenericOp>>(
ctx, ctx,
/*interchangeVector=*/ArrayRef<unsigned>{1, 2, 0}, /*interchangeVector=*/ArrayRef<unsigned>{1, 2, 0},
LinalgTransformationFilter(ArrayRef<Identifier>{}, LinalgTransformationFilter(ArrayRef<Identifier>{},
Identifier::get("PERMUTED", ctx))); Identifier::get("PERMUTED", ctx)));
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
// Linalg subview operands promotion. // Linalg subview operands promotion.
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
patterns.insert<LinalgPromotionPattern<MatmulOp>>( patterns.add<LinalgPromotionPattern<MatmulOp>>(
ctx, LinalgPromotionOptions().setUseFullTileBuffersByDefault(true), ctx, LinalgPromotionOptions().setUseFullTileBuffersByDefault(true),
LinalgTransformationFilter(Identifier::get("_promote_views_", ctx), LinalgTransformationFilter(Identifier::get("_promote_views_", ctx),
Identifier::get("_views_promoted_", ctx))); Identifier::get("_views_promoted_", ctx)));
patterns.insert<LinalgPromotionPattern<MatmulOp>>( patterns.add<LinalgPromotionPattern<MatmulOp>>(
ctx, ctx,
LinalgPromotionOptions() LinalgPromotionOptions()
.setOperandsToPromote({0}) .setOperandsToPromote({0})
.setUseFullTileBuffersByDefault(true), .setUseFullTileBuffersByDefault(true),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("_promote_first_view_", ctx), Identifier::get("_promote_first_view_", ctx),
Identifier::get("_first_view_promoted_", ctx))); Identifier::get("_first_view_promoted_", ctx)));
patterns.insert<LinalgPromotionPattern<FillOp>>( patterns.add<LinalgPromotionPattern<FillOp>>(
ctx, ctx,
LinalgPromotionOptions() LinalgPromotionOptions()
.setOperandsToPromote({0}) .setOperandsToPromote({0})
.setUseFullTileBuffers({true}) .setUseFullTileBuffers({true})
.setAlignment(32), .setAlignment(32),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("_promote_views_aligned_", ctx), Identifier::get("_promote_views_aligned_", ctx),
Identifier::get("_views_aligned_promoted_", ctx))); Identifier::get("_views_aligned_promoted_", ctx)));
(void)applyPatternsAndFoldGreedily(funcOp, std::move(patterns)); (void)applyPatternsAndFoldGreedily(funcOp, std::move(patterns));
// Drop the marker. // Drop the marker.
funcOp.walk([](LinalgOp op) { funcOp.walk([](LinalgOp op) {
op->removeAttr(LinalgTransforms::kLinalgTransformMarker); op->removeAttr(LinalgTransforms::kLinalgTransformMarker);
}); });
} }
static void fillL1TilingAndMatmulToVectorPatterns( static void fillL1TilingAndMatmulToVectorPatterns(
FuncOp funcOp, StringRef startMarker, FuncOp funcOp, StringRef startMarker,
SmallVectorImpl<OwningRewritePatternList> &patternsVector) { SmallVectorImpl<RewritePatternSet> &patternsVector) {
MLIRContext *ctx = funcOp.getContext(); MLIRContext *ctx = funcOp.getContext();
patternsVector.emplace_back( patternsVector.emplace_back(
ctx, std::make_unique<LinalgTilingPattern<MatmulOp>>( ctx, std::make_unique<LinalgTilingPattern<MatmulOp>>(
ctx, ctx,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({8, 12, 16}) .setTileSizes({8, 12, 16})
.setInterchange({1, 0, 2}), .setInterchange({1, 0, 2}),
LinalgTransformationFilter(Identifier::get(startMarker, ctx), LinalgTransformationFilter(Identifier::get(startMarker, ctx),
Identifier::get("L1", ctx)))); Identifier::get("L1", ctx))));
patternsVector.emplace_back( patternsVector.emplace_back(
ctx, ctx,
std::make_unique<LinalgPromotionPattern<MatmulOp>>( std::make_unique<LinalgPromotionPattern<MatmulOp>>(
ctx, LinalgPromotionOptions().setUseFullTileBuffersByDefault(true), ctx, LinalgPromotionOptions().setUseFullTileBuffersByDefault(true),
LinalgTransformationFilter(Identifier::get("L1", ctx), LinalgTransformationFilter(Identifier::get("L1", ctx),
Identifier::get("VEC", ctx)))); Identifier::get("VEC", ctx))));
patternsVector.emplace_back( patternsVector.emplace_back(
ctx, std::make_unique<LinalgVectorizationPattern>( ctx, std::make_unique<LinalgVectorizationPattern>(
MatmulOp::getOperationName(), ctx, LinalgVectorizationOptions(), MatmulOp::getOperationName(), ctx, LinalgVectorizationOptions(),
LinalgTransformationFilter(Identifier::get("VEC", ctx)))); LinalgTransformationFilter(Identifier::get("VEC", ctx))));
patternsVector.back().insert<LinalgVectorizationPattern>( patternsVector.back().add<LinalgVectorizationPattern>(
LinalgTransformationFilter().addFilter( LinalgTransformationFilter().addFilter(
[](Operation *op) { return success(isa<FillOp, CopyOp>(op)); })); [](Operation *op) { return success(isa<FillOp, CopyOp>(op)); }));
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Test promotion callbacks // Test promotion callbacks
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 27 Lines if (!isOutput)
b.create<FillOp>( b.create<FillOp>(
src.getLoc(), dst, src.getLoc(), dst,
b.create<ConstantOp>(src.getLoc(), FloatAttr::get(floatType, 42.0))); b.create<ConstantOp>(src.getLoc(), FloatAttr::get(floatType, 42.0)));
b.create<CopyOp>(src.getLoc(), src, dst); b.create<CopyOp>(src.getLoc(), src, dst);
return success(); return success();
} }
static void fillPromotionCallBackPatterns(MLIRContext *ctx, static void fillPromotionCallBackPatterns(MLIRContext *ctx,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
ctx, LinalgTilingOptions().setTileSizes({16, 16, 16}), ctx, LinalgTilingOptions().setTileSizes({16, 16, 16}),
LinalgTransformationFilter(Identifier::get("START", ctx), LinalgTransformationFilter(Identifier::get("START", ctx),
Identifier::get("PROMOTE", ctx))); Identifier::get("PROMOTE", ctx)));
patterns.insert<LinalgPromotionPattern<MatmulOp>>( patterns.add<LinalgPromotionPattern<MatmulOp>>(
ctx, ctx,
LinalgPromotionOptions() LinalgPromotionOptions()
.setOperandsToPromote({0, 2}) .setOperandsToPromote({0, 2})
.setUseFullTileBuffers({false, false}) .setUseFullTileBuffers({false, false})
.setAllocationDeallocationFns(allocCallBackFn, deallocCallBackFn) .setAllocationDeallocationFns(allocCallBackFn, deallocCallBackFn)
.setCopyInOutFns( .setCopyInOutFns(
[](OpBuilder &b, Value src, Value dst) -> LogicalResult { [](OpBuilder &b, Value src, Value dst) -> LogicalResult {
return copyCallBackFn(b, src, dst, false); return copyCallBackFn(b, src, dst, false);
Show All 12 LinesgetGpuProcIds(OpBuilder &b, Location loc, ArrayRef<Range> parallelLoopRanges) {
procInfo[0] = {b.create<IdOp>(loc, indexType, b.getStringAttr("y")), procInfo[0] = {b.create<IdOp>(loc, indexType, b.getStringAttr("y")),
b.create<NProcsOp>(loc, indexType, b.getStringAttr("y"))}; b.create<NProcsOp>(loc, indexType, b.getStringAttr("y"))};
procInfo[1] = {b.create<IdOp>(loc, indexType, b.getStringAttr("x")), procInfo[1] = {b.create<IdOp>(loc, indexType, b.getStringAttr("x")),
b.create<NProcsOp>(loc, indexType, b.getStringAttr("x"))}; b.create<NProcsOp>(loc, indexType, b.getStringAttr("x"))};
return procInfo; return procInfo;
} }
static void fillTileAndDistributePatterns(MLIRContext *context, static void fillTileAndDistributePatterns(MLIRContext *context,
OwningRewritePatternList &patterns) { RewritePatternSet &patterns) {
{ {
LinalgLoopDistributionOptions cyclicNprocsEqNiters; LinalgLoopDistributionOptions cyclicNprocsEqNiters;
cyclicNprocsEqNiters.distributionMethod.resize( cyclicNprocsEqNiters.distributionMethod.resize(
2, DistributionMethod::CyclicNumProcsEqNumIters); 2, DistributionMethod::CyclicNumProcsEqNumIters);
cyclicNprocsEqNiters.procInfo = cyclicNprocsEqNiters.procInfo =
getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>; getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>;
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
context, context,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({8, 8, 4}) .setTileSizes({8, 8, 4})
.setLoopType(LinalgTilingLoopType::ParallelLoops) .setLoopType(LinalgTilingLoopType::ParallelLoops)
.setDistributionOptions(cyclicNprocsEqNiters), .setDistributionOptions(cyclicNprocsEqNiters),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("distribute1", context), Identifier::get("distribute1", context),
Identifier::get("after_distribute1", context))); Identifier::get("after_distribute1", context)));
} }
{ {
LinalgLoopDistributionOptions cyclicNprocsGeNiters; LinalgLoopDistributionOptions cyclicNprocsGeNiters;
cyclicNprocsGeNiters.distributionMethod.resize( cyclicNprocsGeNiters.distributionMethod.resize(
2, DistributionMethod::CyclicNumProcsGeNumIters); 2, DistributionMethod::CyclicNumProcsGeNumIters);
cyclicNprocsGeNiters.procInfo = cyclicNprocsGeNiters.procInfo =
getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>; getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>;
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
context, context,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({8, 8, 4}) .setTileSizes({8, 8, 4})
.setLoopType(LinalgTilingLoopType::ParallelLoops) .setLoopType(LinalgTilingLoopType::ParallelLoops)
.setDistributionOptions(cyclicNprocsGeNiters), .setDistributionOptions(cyclicNprocsGeNiters),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("distribute2", context), Identifier::get("distribute2", context),
Identifier::get("after_distribute2", context))); Identifier::get("after_distribute2", context)));
} }
{ {
LinalgLoopDistributionOptions cyclicNprocsDefault; LinalgLoopDistributionOptions cyclicNprocsDefault;
cyclicNprocsDefault.distributionMethod.resize(2, cyclicNprocsDefault.distributionMethod.resize(2,
DistributionMethod::Cyclic); DistributionMethod::Cyclic);
cyclicNprocsDefault.procInfo = cyclicNprocsDefault.procInfo =
getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>; getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>;
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
context, context,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({8, 8, 4}) .setTileSizes({8, 8, 4})
.setLoopType(LinalgTilingLoopType::ParallelLoops) .setLoopType(LinalgTilingLoopType::ParallelLoops)
.setDistributionOptions(cyclicNprocsDefault), .setDistributionOptions(cyclicNprocsDefault),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("distribute3", context), Identifier::get("distribute3", context),
Identifier::get("after_distribute3", context))); Identifier::get("after_distribute3", context)));
} }
{ {
LinalgLoopDistributionOptions cyclicNprocsMixed1; LinalgLoopDistributionOptions cyclicNprocsMixed1;
cyclicNprocsMixed1.distributionMethod = { cyclicNprocsMixed1.distributionMethod = {
DistributionMethod::CyclicNumProcsEqNumIters, DistributionMethod::CyclicNumProcsEqNumIters,
DistributionMethod::CyclicNumProcsGeNumIters}; DistributionMethod::CyclicNumProcsGeNumIters};
cyclicNprocsMixed1.procInfo = getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>; cyclicNprocsMixed1.procInfo = getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>;
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
context, context,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({8, 8, 4}) .setTileSizes({8, 8, 4})
.setLoopType(LinalgTilingLoopType::ParallelLoops) .setLoopType(LinalgTilingLoopType::ParallelLoops)
.setDistributionOptions(cyclicNprocsMixed1), .setDistributionOptions(cyclicNprocsMixed1),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("distribute4", context), Identifier::get("distribute4", context),
Identifier::get("after_distribute4", context))); Identifier::get("after_distribute4", context)));
} }
{ {
LinalgLoopDistributionOptions cyclicNprocsMixed2; LinalgLoopDistributionOptions cyclicNprocsMixed2;
cyclicNprocsMixed2.distributionMethod = { cyclicNprocsMixed2.distributionMethod = {
DistributionMethod::CyclicNumProcsGeNumIters, DistributionMethod::CyclicNumProcsGeNumIters,
DistributionMethod::Cyclic}; DistributionMethod::Cyclic};
cyclicNprocsMixed2.procInfo = getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>; cyclicNprocsMixed2.procInfo = getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>;
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
context, context,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({8, 8, 4}) .setTileSizes({8, 8, 4})
.setLoopType(LinalgTilingLoopType::ParallelLoops) .setLoopType(LinalgTilingLoopType::ParallelLoops)
.setDistributionOptions(cyclicNprocsMixed2), .setDistributionOptions(cyclicNprocsMixed2),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("distribute5", context), Identifier::get("distribute5", context),
Identifier::get("after_distribute5", context))); Identifier::get("after_distribute5", context)));
} }
{ {
LinalgLoopDistributionOptions cyclicNprocsMixed3; LinalgLoopDistributionOptions cyclicNprocsMixed3;
cyclicNprocsMixed3.distributionMethod = { cyclicNprocsMixed3.distributionMethod = {
DistributionMethod::Cyclic, DistributionMethod::Cyclic,
DistributionMethod::CyclicNumProcsEqNumIters}; DistributionMethod::CyclicNumProcsEqNumIters};
cyclicNprocsMixed3.procInfo = getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>; cyclicNprocsMixed3.procInfo = getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>;
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
context, context,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({8, 8, 4}) .setTileSizes({8, 8, 4})
.setLoopType(LinalgTilingLoopType::ParallelLoops) .setLoopType(LinalgTilingLoopType::ParallelLoops)
.setDistributionOptions(cyclicNprocsMixed3), .setDistributionOptions(cyclicNprocsMixed3),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("distribute6", context), Identifier::get("distribute6", context),
Identifier::get("after_distribute6", context))); Identifier::get("after_distribute6", context)));
} }
{ {
LinalgLoopDistributionOptions cyclicNprocsEqNiters; LinalgLoopDistributionOptions cyclicNprocsEqNiters;
cyclicNprocsEqNiters.distributionMethod.resize(2, cyclicNprocsEqNiters.distributionMethod.resize(2,
DistributionMethod::Cyclic); DistributionMethod::Cyclic);
cyclicNprocsEqNiters.procInfo = cyclicNprocsEqNiters.procInfo =
getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>; getGpuProcIds<gpu::BlockIdOp, gpu::GridDimOp>;
patterns.insert<LinalgTilingPattern<MatmulOp>>( patterns.add<LinalgTilingPattern<MatmulOp>>(
context, context,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({8, 8, 4}) .setTileSizes({8, 8, 4})
.setLoopType(LinalgTilingLoopType::Loops) .setLoopType(LinalgTilingLoopType::Loops)
.setDistributionOptions(cyclicNprocsEqNiters), .setDistributionOptions(cyclicNprocsEqNiters),
LinalgTransformationFilter( LinalgTransformationFilter(
Identifier::get("tensors_distribute1", context), Identifier::get("tensors_distribute1", context),
Identifier::get("tensors_after_distribute1", context))); Identifier::get("tensors_after_distribute1", context)));
} }
} }
static void static void
applyMatmulToVectorPatterns(FuncOp funcOp, applyMatmulToVectorPatterns(FuncOp funcOp,
bool testMatmulToVectorPatterns1dTiling, bool testMatmulToVectorPatterns1dTiling,
bool testMatmulToVectorPatterns2dTiling) { bool testMatmulToVectorPatterns2dTiling) {
MLIRContext *ctx = funcOp.getContext(); MLIRContext *ctx = funcOp.getContext();
SmallVector<OwningRewritePatternList, 4> stage1Patterns; SmallVector<RewritePatternSet, 4> stage1Patterns;
if (testMatmulToVectorPatterns1dTiling) { if (testMatmulToVectorPatterns1dTiling) {
fillL1TilingAndMatmulToVectorPatterns(funcOp, Identifier::get("START", ctx), fillL1TilingAndMatmulToVectorPatterns(funcOp, Identifier::get("START", ctx),
stage1Patterns); stage1Patterns);
} else if (testMatmulToVectorPatterns2dTiling) { } else if (testMatmulToVectorPatterns2dTiling) {
stage1Patterns.emplace_back( stage1Patterns.emplace_back(
ctx, std::make_unique<LinalgTilingPattern<MatmulOp>>( ctx, std::make_unique<LinalgTilingPattern<MatmulOp>>(
ctx, ctx,
LinalgTilingOptions() LinalgTilingOptions()
.setTileSizes({768, 264, 768}) .setTileSizes({768, 264, 768})
.setInterchange({1, 2, 0}), .setInterchange({1, 2, 0}),
LinalgTransformationFilter(Identifier::get("START", ctx), LinalgTransformationFilter(Identifier::get("START", ctx),
Identifier::get("L2", ctx)))); Identifier::get("L2", ctx))));
fillL1TilingAndMatmulToVectorPatterns(funcOp, Identifier::get("L2", ctx), fillL1TilingAndMatmulToVectorPatterns(funcOp, Identifier::get("L2", ctx),
stage1Patterns); stage1Patterns);
} }
SmallVector<FrozenRewritePatternList, 4> frozenStage1Patterns; SmallVector<FrozenRewritePatternList, 4> frozenStage1Patterns;
llvm::move(stage1Patterns, std::back_inserter(frozenStage1Patterns)); llvm::move(stage1Patterns, std::back_inserter(frozenStage1Patterns));
FrozenRewritePatternList stage2Patterns = FrozenRewritePatternList stage2Patterns =
getLinalgTilingCanonicalizationPatterns(ctx); getLinalgTilingCanonicalizationPatterns(ctx);
(void)applyStagedPatterns(funcOp, frozenStage1Patterns, (void)applyStagedPatterns(funcOp, frozenStage1Patterns,
std::move(stage2Patterns)); std::move(stage2Patterns));
} }
static void applyVectorTransferForwardingPatterns(FuncOp funcOp) { static void applyVectorTransferForwardingPatterns(FuncOp funcOp) {
OwningRewritePatternList forwardPattern(funcOp.getContext()); RewritePatternSet forwardPattern(funcOp.getContext());
forwardPattern.insert<LinalgCopyVTRForwardingPattern>(funcOp.getContext()); forwardPattern.add<LinalgCopyVTRForwardingPattern>(funcOp.getContext());
forwardPattern.insert<LinalgCopyVTWForwardingPattern>(funcOp.getContext()); forwardPattern.add<LinalgCopyVTWForwardingPattern>(funcOp.getContext());
(void)applyPatternsAndFoldGreedily(funcOp, std::move(forwardPattern)); (void)applyPatternsAndFoldGreedily(funcOp, std::move(forwardPattern));
} }
static void applyLinalgToVectorPatterns(FuncOp funcOp) { static void applyLinalgToVectorPatterns(FuncOp funcOp) {
OwningRewritePatternList patterns(funcOp.getContext()); RewritePatternSet patterns(funcOp.getContext());
patterns.insert<LinalgVectorizationPattern>( patterns.add<LinalgVectorizationPattern>(
LinalgTransformationFilter() LinalgTransformationFilter()
.addOpFilter<ContractionOpInterface, FillOp, CopyOp, GenericOp>()); .addOpFilter<ContractionOpInterface, FillOp, CopyOp, GenericOp>());
patterns.insert<PadTensorOpVectorizationPattern>(funcOp.getContext()); patterns.add<PadTensorOpVectorizationPattern>(funcOp.getContext());
(void)applyPatternsAndFoldGreedily(funcOp, std::move(patterns)); (void)applyPatternsAndFoldGreedily(funcOp, std::move(patterns));
} }
static void applyAffineMinSCFCanonicalizationPatterns(FuncOp funcOp) { static void applyAffineMinSCFCanonicalizationPatterns(FuncOp funcOp) {
OwningRewritePatternList foldPattern(funcOp.getContext()); RewritePatternSet foldPattern(funcOp.getContext());
foldPattern.insert<AffineMinSCFCanonicalizationPattern>(funcOp.getContext()); foldPattern.add<AffineMinSCFCanonicalizationPattern>(funcOp.getContext());
FrozenRewritePatternList frozenPatterns(std::move(foldPattern)); FrozenRewritePatternList frozenPatterns(std::move(foldPattern));
// Explicitly walk and apply the pattern locally to avoid more general folding // Explicitly walk and apply the pattern locally to avoid more general folding
// on the rest of the IR. // on the rest of the IR.
funcOp.walk([&frozenPatterns](AffineMinOp minOp) { funcOp.walk([&frozenPatterns](AffineMinOp minOp) {
(void)applyOpPatternsAndFold(minOp, frozenPatterns); (void)applyOpPatternsAndFold(minOp, frozenPatterns);
}); });
} }
// For now, just assume it is the zero of type. // For now, just assume it is the zero of type.
// In the future, it should be the zero of type + op. // In the future, it should be the zero of type + op.
static Value getNeutralOfLinalgOp(OpBuilder &b, OpOperand &op) { static Value getNeutralOfLinalgOp(OpBuilder &b, OpOperand &op) {
auto t = getElementTypeOrSelf(op.get().getType()); auto t = getElementTypeOrSelf(op.get().getType());
return b.create<ConstantOp>(op.getOwner()->getLoc(), t, b.getZeroAttr(t)); return b.create<ConstantOp>(op.getOwner()->getLoc(), t, b.getZeroAttr(t));
} }
static void applyTileAndPadPattern(FuncOp funcOp) { static void applyTileAndPadPattern(FuncOp funcOp) {
MLIRContext *context = funcOp.getContext(); MLIRContext *context = funcOp.getContext();
OwningRewritePatternList tilingPattern(context); RewritePatternSet tilingPattern(context);
auto linalgTilingOptions = auto linalgTilingOptions =
linalg::LinalgTilingOptions() linalg::LinalgTilingOptions()
.setTileSizes({2, 3, 4}) .setTileSizes({2, 3, 4})
.setPaddingValueComputationFunction(getNeutralOfLinalgOp); .setPaddingValueComputationFunction(getNeutralOfLinalgOp);
tilingPattern.insert<linalg::LinalgTilingPattern<linalg::MatmulI8I8I32Op>>( tilingPattern.add<linalg::LinalgTilingPattern<linalg::MatmulI8I8I32Op>>(
context, linalgTilingOptions, context, linalgTilingOptions,
linalg::LinalgTransformationFilter( linalg::LinalgTransformationFilter(
Identifier::get("tile-and-pad", context))); Identifier::get("tile-and-pad", context)));
(void)applyPatternsAndFoldGreedily(funcOp, std::move(tilingPattern)); (void)applyPatternsAndFoldGreedily(funcOp, std::move(tilingPattern));
} }
/// Apply transformations specified as patterns. /// Apply transformations specified as patterns.
void TestLinalgTransforms::runOnFunction() { void TestLinalgTransforms::runOnFunction() {
auto lambda = [&](void *) { auto lambda = [&](void *) {
getFunction().walk([](LinalgOp op) { getFunction().walk([](LinalgOp op) {
op->removeAttr(LinalgTransforms::kLinalgTransformMarker); op->removeAttr(LinalgTransforms::kLinalgTransformMarker);
}); });
}; };
std::unique_ptr<void, decltype(lambda)> cleanupGuard{(void *)1, lambda}; std::unique_ptr<void, decltype(lambda)> cleanupGuard{(void *)1, lambda};
if (testPromotionOptions) { if (testPromotionOptions) {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
fillPromotionCallBackPatterns(&getContext(), patterns); fillPromotionCallBackPatterns(&getContext(), patterns);
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
return; return;
} }
if (testTileAndDistributionOptions) { if (testTileAndDistributionOptions) {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
fillTileAndDistributePatterns(&getContext(), patterns); fillTileAndDistributePatterns(&getContext(), patterns);
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
return; return;
} }
if (testPatterns) if (testPatterns)
return applyPatterns(getFunction()); return applyPatterns(getFunction());
if (testMatmulToVectorPatterns1dTiling || testMatmulToVectorPatterns2dTiling) if (testMatmulToVectorPatterns1dTiling || testMatmulToVectorPatterns2dTiling)
return applyMatmulToVectorPatterns(getFunction(), return applyMatmulToVectorPatterns(getFunction(),
Show All 26 Lines

mlir/test/lib/Transforms/TestPolynomialApproximation.cpp

Show All 26 Linesstruct TestMathPolynomialApproximationPass
void getDependentDialects(DialectRegistry &registry) const override { void getDependentDialects(DialectRegistry &registry) const override {
registry registry
.insert<vector::VectorDialect, math::MathDialect, LLVM::LLVMDialect>(); .insert<vector::VectorDialect, math::MathDialect, LLVM::LLVMDialect>();
} }
}; };
} // end anonymous namespace } // end anonymous namespace
void TestMathPolynomialApproximationPass::runOnFunction() { void TestMathPolynomialApproximationPass::runOnFunction() {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateMathPolynomialApproximationPatterns(patterns); populateMathPolynomialApproximationPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
} }
namespace mlir { namespace mlir {
namespace test { namespace test {
void registerTestMathPolynomialApproximationPass() { void registerTestMathPolynomialApproximationPass() {
PassRegistration<TestMathPolynomialApproximationPass> pass( PassRegistration<TestMathPolynomialApproximationPass> pass(
"test-math-polynomial-approximation", "test-math-polynomial-approximation",
"Test math polynomial approximations"); "Test math polynomial approximations");
} }
} // namespace test } // namespace test
} // namespace mlir } // namespace mlir

mlir/test/lib/Transforms/TestSparsification.cpp

Show First 20 Lines • Show All 95 Lines • ▼ Show 20 Lines linalg::SparseIntType typeOption(int32_t option) {
case 4: case 4:
return linalg::SparseIntType::kI8; return linalg::SparseIntType::kI8;
} }
} }
/// Runs the test on a function. /// Runs the test on a function.
void runOnOperation() override { void runOnOperation() override {
auto *ctx = &getContext(); auto *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
// Translate strategy flags to strategy options. // Translate strategy flags to strategy options.
linalg::SparsificationOptions options(parallelOption(), vectorOption(), linalg::SparsificationOptions options(parallelOption(), vectorOption(),
vectorLength, typeOption(ptrType), vectorLength, typeOption(ptrType),
typeOption(indType), fastOutput); typeOption(indType), fastOutput);
// Apply rewriting. // Apply rewriting.
linalg::populateSparsificationPatterns(patterns, options); linalg::populateSparsificationPatterns(patterns, options);
vector::populateVectorToVectorCanonicalizationPatterns(patterns); vector::populateVectorToVectorCanonicalizationPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getOperation(), std::move(patterns));
// Lower sparse primitives to calls into runtime support library. // Lower sparse primitives to calls into runtime support library.
if (lower) { if (lower) {
OwningRewritePatternList conversionPatterns(ctx); RewritePatternSet conversionPatterns(ctx);
ConversionTarget target(*ctx); ConversionTarget target(*ctx);
target.addIllegalOp<linalg::SparseTensorFromPointerOp, target.addIllegalOp<linalg::SparseTensorFromPointerOp,
linalg::SparseTensorToPointersMemRefOp, linalg::SparseTensorToPointersMemRefOp,
linalg::SparseTensorToIndicesMemRefOp, linalg::SparseTensorToIndicesMemRefOp,
linalg::SparseTensorToValuesMemRefOp>(); linalg::SparseTensorToValuesMemRefOp>();
target.addLegalOp<CallOp>(); target.addLegalOp<CallOp>();
linalg::populateSparsificationConversionPatterns(conversionPatterns); linalg::populateSparsificationConversionPatterns(conversionPatterns);
if (failed(applyPartialConversion(getOperation(), target, if (failed(applyPartialConversion(getOperation(), target,
Show All 18 Lines

mlir/test/lib/Transforms/TestVectorTransforms.cpp

Show All 31 Lines void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<AffineDialect>(); registry.insert<AffineDialect>();
} }
Option<bool> unroll{*this, "unroll", llvm::cl::desc("Include unrolling"), Option<bool> unroll{*this, "unroll", llvm::cl::desc("Include unrolling"),
llvm::cl::init(false)}; llvm::cl::init(false)};
void runOnFunction() override { void runOnFunction() override {
auto *ctx = &getContext(); auto *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
if (unroll) { if (unroll) {
patterns.insert<UnrollVectorPattern>( patterns.add<UnrollVectorPattern>(
ctx, ctx,
UnrollVectorOptions().setNativeShapeFn(getShape).setFilterConstraint( UnrollVectorOptions().setNativeShapeFn(getShape).setFilterConstraint(
filter)); filter));
} }
populateVectorToVectorCanonicalizationPatterns(patterns); populateVectorToVectorCanonicalizationPatterns(patterns);
populateVectorToVectorTransformationPatterns(patterns); populateVectorToVectorTransformationPatterns(patterns);
populateBubbleVectorBitCastOpPatterns(patterns); populateBubbleVectorBitCastOpPatterns(patterns);
populateCastAwayVectorLeadingOneDimPatterns(patterns); populateCastAwayVectorLeadingOneDimPatterns(patterns);
Show All 14 Linesprivate:
static LogicalResult filter(Operation *op) { static LogicalResult filter(Operation *op) {
return success(isa<AddFOp, SelectOp, CmpFOp, ContractionOp>(op)); return success(isa<AddFOp, SelectOp, CmpFOp, ContractionOp>(op));
} }
}; };
struct TestVectorSlicesConversion struct TestVectorSlicesConversion
: public PassWrapper<TestVectorSlicesConversion, FunctionPass> { : public PassWrapper<TestVectorSlicesConversion, FunctionPass> {
void runOnFunction() override { void runOnFunction() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateVectorSlicesLoweringPatterns(patterns); populateVectorSlicesLoweringPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
} }
}; };
struct TestVectorContractionConversion struct TestVectorContractionConversion
: public PassWrapper<TestVectorContractionConversion, FunctionPass> { : public PassWrapper<TestVectorContractionConversion, FunctionPass> {
TestVectorContractionConversion() = default; TestVectorContractionConversion() = default;
Show All 14 Lines Option<bool> lowerToOuterProduct{
llvm::cl::init(false)}; llvm::cl::init(false)};
Option<bool> lowerToFilterOuterProduct{ Option<bool> lowerToFilterOuterProduct{
*this, "vector-filter-outerproduct", *this, "vector-filter-outerproduct",
llvm::cl::desc("Lower vector.contract to vector.outerproduct but not for " llvm::cl::desc("Lower vector.contract to vector.outerproduct but not for "
"vectors of size 4."), "vectors of size 4."),
llvm::cl::init(false)}; llvm::cl::init(false)};
void runOnFunction() override { void runOnFunction() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
// Test on one pattern in isolation. // Test on one pattern in isolation.
if (lowerToOuterProduct) { if (lowerToOuterProduct) {
VectorContractLowering lowering = VectorContractLowering::OuterProduct; VectorContractLowering lowering = VectorContractLowering::OuterProduct;
VectorTransformsOptions options{lowering}; VectorTransformsOptions options{lowering};
patterns.insert<ContractionOpToOuterProductOpLowering>(options, patterns.add<ContractionOpToOuterProductOpLowering>(options,
&getContext()); &getContext());
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
return; return;
} }
// Test on one pattern in isolation. // Test on one pattern in isolation.
if (lowerToFilterOuterProduct) { if (lowerToFilterOuterProduct) {
VectorContractLowering lowering = VectorContractLowering::OuterProduct; VectorContractLowering lowering = VectorContractLowering::OuterProduct;
VectorTransformsOptions options{lowering}; VectorTransformsOptions options{lowering};
patterns.insert<ContractionOpToOuterProductOpLowering>( patterns.add<ContractionOpToOuterProductOpLowering>(
options, &getContext(), [](vector::ContractionOp op) { options, &getContext(), [](vector::ContractionOp op) {
// Only lowers vector.contract where the lhs as a type vector<MxNx?> // Only lowers vector.contract where the lhs as a type vector<MxNx?>
// where M is not 4. // where M is not 4.
if (op.getRhsType().getShape()[0] == 4) if (op.getRhsType().getShape()[0] == 4)
return failure(); return failure();
return success(); return success();
}); });
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
Show All 15 Lines
}; };
struct TestVectorUnrollingPatterns struct TestVectorUnrollingPatterns
: public PassWrapper<TestVectorUnrollingPatterns, FunctionPass> { : public PassWrapper<TestVectorUnrollingPatterns, FunctionPass> {
TestVectorUnrollingPatterns() = default; TestVectorUnrollingPatterns() = default;
TestVectorUnrollingPatterns(const TestVectorUnrollingPatterns &pass) {} TestVectorUnrollingPatterns(const TestVectorUnrollingPatterns &pass) {}
void runOnFunction() override { void runOnFunction() override {
MLIRContext *ctx = &getContext(); MLIRContext *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
patterns.insert<UnrollVectorPattern>( patterns.add<UnrollVectorPattern>(
ctx, UnrollVectorOptions() ctx, UnrollVectorOptions()
.setNativeShape(ArrayRef<int64_t>{2, 2}) .setNativeShape(ArrayRef<int64_t>{2, 2})
.setFilterConstraint([](Operation *op) { .setFilterConstraint([](Operation *op) {
return success(isa<AddFOp, vector::FMAOp>(op)); return success(isa<AddFOp, vector::FMAOp>(op));
})); }));
if (unrollBasedOnType) { if (unrollBasedOnType) {
UnrollVectorOptions::NativeShapeFnType nativeShapeFn = UnrollVectorOptions::NativeShapeFnType nativeShapeFn =
[](Operation *op) -> Optional<SmallVector<int64_t, 4>> { [](Operation *op) -> Optional<SmallVector<int64_t, 4>> {
vector::ContractionOp contractOp = cast<vector::ContractionOp>(op); vector::ContractionOp contractOp = cast<vector::ContractionOp>(op);
SmallVector<int64_t, 4> nativeShape = {4, 4, 2}; SmallVector<int64_t, 4> nativeShape = {4, 4, 2};
if (auto floatType = contractOp.getLhsType() if (auto floatType = contractOp.getLhsType()
.getElementType() .getElementType()
.dyn_cast<FloatType>()) { .dyn_cast<FloatType>()) {
if (floatType.getWidth() == 16) { if (floatType.getWidth() == 16) {
nativeShape[2] = 4; nativeShape[2] = 4;
} }
} }
return nativeShape; return nativeShape;
}; };
patterns.insert<UnrollVectorPattern>( patterns.add<UnrollVectorPattern>(
ctx, UnrollVectorOptions() ctx, UnrollVectorOptions()
.setNativeShapeFn(nativeShapeFn) .setNativeShapeFn(nativeShapeFn)
.setFilterConstraint([](Operation *op) { .setFilterConstraint([](Operation *op) {
return success(isa<ContractionOp>(op)); return success(isa<ContractionOp>(op));
})); }));
} else { } else {
patterns.insert<UnrollVectorPattern>( patterns.add<UnrollVectorPattern>(
ctx, UnrollVectorOptions() ctx, UnrollVectorOptions()
.setNativeShape(ArrayRef<int64_t>{2, 2, 2}) .setNativeShape(ArrayRef<int64_t>{2, 2, 2})
.setFilterConstraint([](Operation *op) { .setFilterConstraint([](Operation *op) {
return success(isa<ContractionOp>(op)); return success(isa<ContractionOp>(op));
})); }));
} }
populateVectorToVectorCanonicalizationPatterns(patterns); populateVectorToVectorCanonicalizationPatterns(patterns);
populateVectorToVectorTransformationPatterns(patterns); populateVectorToVectorTransformationPatterns(patterns);
Show All 15 Lines void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<AffineDialect>(); registry.insert<AffineDialect>();
} }
ListOption<int32_t> multiplicity{ ListOption<int32_t> multiplicity{
*this, "distribution-multiplicity", llvm::cl::MiscFlags::CommaSeparated, *this, "distribution-multiplicity", llvm::cl::MiscFlags::CommaSeparated,
llvm::cl::desc("Set the multiplicity used for distributing vector")}; llvm::cl::desc("Set the multiplicity used for distributing vector")};
void runOnFunction() override { void runOnFunction() override {
MLIRContext *ctx = &getContext(); MLIRContext *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
FuncOp func = getFunction(); FuncOp func = getFunction();
func.walk([&](AddFOp op) { func.walk([&](AddFOp op) {
OpBuilder builder(op); OpBuilder builder(op);
if (auto vecType = op.getType().dyn_cast<VectorType>()) { if (auto vecType = op.getType().dyn_cast<VectorType>()) {
SmallVector<int64_t, 2> mul; SmallVector<int64_t, 2> mul;
SmallVector<AffineExpr, 2> perm; SmallVector<AffineExpr, 2> perm;
SmallVector<Value, 2> ids; SmallVector<Value, 2> ids;
unsigned count = 0; unsigned count = 0;
Show All 13 Lines func.walk([&](AddFOp op) {
builder, op.getOperation(), ids, mul, map); builder, op.getOperation(), ids, mul, map);
if (ops.hasValue()) { if (ops.hasValue()) {
SmallPtrSet<Operation *, 1> extractOp({ops->extract, ops->insert}); SmallPtrSet<Operation *, 1> extractOp({ops->extract, ops->insert});
op.getResult().replaceAllUsesExcept(ops->insert.getResult(), op.getResult().replaceAllUsesExcept(ops->insert.getResult(),
extractOp); extractOp);
} }
} }
}); });
patterns.insert<PointwiseExtractPattern>(ctx); patterns.add<PointwiseExtractPattern>(ctx);
populateVectorToVectorTransformationPatterns(patterns); populateVectorToVectorTransformationPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
} }
}; };
struct TestVectorToLoopPatterns struct TestVectorToLoopPatterns
: public PassWrapper<TestVectorToLoopPatterns, FunctionPass> { : public PassWrapper<TestVectorToLoopPatterns, FunctionPass> {
TestVectorToLoopPatterns() = default; TestVectorToLoopPatterns() = default;
TestVectorToLoopPatterns(const TestVectorToLoopPatterns &pass) {} TestVectorToLoopPatterns(const TestVectorToLoopPatterns &pass) {}
void getDependentDialects(DialectRegistry &registry) const override { void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<VectorDialect>(); registry.insert<VectorDialect>();
registry.insert<AffineDialect>(); registry.insert<AffineDialect>();
} }
Option<int32_t> multiplicity{ Option<int32_t> multiplicity{
*this, "distribution-multiplicity", *this, "distribution-multiplicity",
llvm::cl::desc("Set the multiplicity used for distributing vector"), llvm::cl::desc("Set the multiplicity used for distributing vector"),
llvm::cl::init(32)}; llvm::cl::init(32)};
void runOnFunction() override { void runOnFunction() override {
MLIRContext *ctx = &getContext(); MLIRContext *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
FuncOp func = getFunction(); FuncOp func = getFunction();
func.walk([&](AddFOp op) { func.walk([&](AddFOp op) {
// Check that the operation type can be broken down into a loop. // Check that the operation type can be broken down into a loop.
VectorType type = op.getType().dyn_cast<VectorType>(); VectorType type = op.getType().dyn_cast<VectorType>();
if (!type || type.getRank() != 1 || if (!type || type.getRank() != 1 ||
type.getNumElements() % multiplicity != 0) type.getNumElements() % multiplicity != 0)
return mlir::WalkResult::advance(); return mlir::WalkResult::advance();
auto filterAlloc = [](Operation *op) { auto filterAlloc = [](Operation *op) {
Show All 23 Lines func.walk([&](AddFOp op) {
builder, op.getOperation(), {forOp.getInductionVar()}, {multiplicity}, builder, op.getOperation(), {forOp.getInductionVar()}, {multiplicity},
map); map);
if (ops.hasValue()) { if (ops.hasValue()) {
SmallPtrSet<Operation *, 1> extractOp({ops->extract, ops->insert}); SmallPtrSet<Operation *, 1> extractOp({ops->extract, ops->insert});
op.getResult().replaceAllUsesExcept(ops->insert.getResult(), extractOp); op.getResult().replaceAllUsesExcept(ops->insert.getResult(), extractOp);
} }
return mlir::WalkResult::interrupt(); return mlir::WalkResult::interrupt();
}); });
patterns.insert<PointwiseExtractPattern>(ctx); patterns.add<PointwiseExtractPattern>(ctx);
populateVectorToVectorTransformationPatterns(patterns); populateVectorToVectorTransformationPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
} }
}; };
struct TestVectorTransferUnrollingPatterns struct TestVectorTransferUnrollingPatterns
: public PassWrapper<TestVectorTransferUnrollingPatterns, FunctionPass> { : public PassWrapper<TestVectorTransferUnrollingPatterns, FunctionPass> {
void getDependentDialects(DialectRegistry &registry) const override { void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<AffineDialect>(); registry.insert<AffineDialect>();
} }
void runOnFunction() override { void runOnFunction() override {
MLIRContext *ctx = &getContext(); MLIRContext *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
patterns.insert<UnrollVectorPattern>( patterns.add<UnrollVectorPattern>(
ctx, ctx,
UnrollVectorOptions() UnrollVectorOptions()
.setNativeShape(ArrayRef<int64_t>{2, 2}) .setNativeShape(ArrayRef<int64_t>{2, 2})
.setFilterConstraint([](Operation *op) { .setFilterConstraint([](Operation *op) {
return success( return success(
isa<vector::TransferReadOp, vector::TransferWriteOp>(op)); isa<vector::TransferReadOp, vector::TransferWriteOp>(op));
})); }));
populateVectorToVectorCanonicalizationPatterns(patterns); populateVectorToVectorCanonicalizationPatterns(patterns);
Show All 16 Linesstruct TestVectorTransferFullPartialSplitPatterns
Option<bool> useLinalgOps{ Option<bool> useLinalgOps{
*this, "use-linalg-copy", *this, "use-linalg-copy",
llvm::cl::desc("Split using a unmasked vector.transfer + linalg.fill + " llvm::cl::desc("Split using a unmasked vector.transfer + linalg.fill + "
"linalg.copy operations."), "linalg.copy operations."),
llvm::cl::init(false)}; llvm::cl::init(false)};
void runOnFunction() override { void runOnFunction() override {
MLIRContext *ctx = &getContext(); MLIRContext *ctx = &getContext();
OwningRewritePatternList patterns(ctx); RewritePatternSet patterns(ctx);
VectorTransformsOptions options; VectorTransformsOptions options;
if (useLinalgOps) if (useLinalgOps)
options.setVectorTransferSplit(VectorTransferSplit::LinalgCopy); options.setVectorTransferSplit(VectorTransferSplit::LinalgCopy);
else else
options.setVectorTransferSplit(VectorTransferSplit::VectorTransfer); options.setVectorTransferSplit(VectorTransferSplit::VectorTransfer);
patterns.insert<VectorTransferFullPartialRewriter>(ctx, options); patterns.add<VectorTransferFullPartialRewriter>(ctx, options);
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
} }
}; };
struct TestVectorTransferOpt struct TestVectorTransferOpt
: public PassWrapper<TestVectorTransferOpt, FunctionPass> { : public PassWrapper<TestVectorTransferOpt, FunctionPass> {
void runOnFunction() override { transferOpflowOpt(getFunction()); } void runOnFunction() override { transferOpflowOpt(getFunction()); }
}; };
struct TestVectorTransferLoweringPatterns struct TestVectorTransferLoweringPatterns
: public PassWrapper<TestVectorTransferLoweringPatterns, FunctionPass> { : public PassWrapper<TestVectorTransferLoweringPatterns, FunctionPass> {
void getDependentDialects(DialectRegistry &registry) const override { void getDependentDialects(DialectRegistry &registry) const override {
registry.insert<memref::MemRefDialect>(); registry.insert<memref::MemRefDialect>();
} }
void runOnFunction() override { void runOnFunction() override {
OwningRewritePatternList patterns(&getContext()); RewritePatternSet patterns(&getContext());
populateVectorTransferLoweringPatterns(patterns); populateVectorTransferLoweringPatterns(patterns);
(void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns)); (void)applyPatternsAndFoldGreedily(getFunction(), std::move(patterns));
} }
}; };
} // end anonymous namespace } // end anonymous namespace
namespace mlir { namespace mlir {
▲ Show 20 Lines • Show All 46 Lines • Show Last 20 Lines

mlir/test/mlir-tblgen/op-decl-and-defs.td

Show First 20 Lines • Show All 90 Lines • ▼ Show 20 Lines
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, Value val); // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, Value val);
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, int integer = 0); // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, int integer = 0);
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type r, ::mlir::TypeRange s, ::mlir::Value a, ::mlir::ValueRange b, ::mlir::IntegerAttr attr1, /*optional*/::mlir::FloatAttr attr2, unsigned someRegionsCount) // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type r, ::mlir::TypeRange s, ::mlir::Value a, ::mlir::ValueRange b, ::mlir::IntegerAttr attr1, /*optional*/::mlir::FloatAttr attr2, unsigned someRegionsCount)
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type r, ::mlir::TypeRange s, ::mlir::Value a, ::mlir::ValueRange b, uint32_t attr1, /*optional*/::mlir::FloatAttr attr2, unsigned someRegionsCount) // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type r, ::mlir::TypeRange s, ::mlir::Value a, ::mlir::ValueRange b, uint32_t attr1, /*optional*/::mlir::FloatAttr attr2, unsigned someRegionsCount)
// CHECK: static void build(::mlir::OpBuilder &, ::mlir::OperationState &odsState, ::mlir::TypeRange resultTypes, ::mlir::ValueRange operands, ::llvm::ArrayRef<::mlir::NamedAttribute> attributes, unsigned numRegions) // CHECK: static void build(::mlir::OpBuilder &, ::mlir::OperationState &odsState, ::mlir::TypeRange resultTypes, ::mlir::ValueRange operands, ::llvm::ArrayRef<::mlir::NamedAttribute> attributes, unsigned numRegions)
// CHECK: static ::mlir::ParseResult parse(::mlir::OpAsmParser &parser, ::mlir::OperationState &result); // CHECK: static ::mlir::ParseResult parse(::mlir::OpAsmParser &parser, ::mlir::OperationState &result);
// CHECK: void print(::mlir::OpAsmPrinter &p); // CHECK: void print(::mlir::OpAsmPrinter &p);
// CHECK: ::mlir::LogicalResult verify(); // CHECK: ::mlir::LogicalResult verify();
// CHECK: static void getCanonicalizationPatterns(::mlir::OwningRewritePatternList &results, ::mlir::MLIRContext *context); // CHECK: static void getCanonicalizationPatterns(::mlir::RewritePatternSet &results, ::mlir::MLIRContext *context);
// CHECK: ::mlir::LogicalResult fold(::llvm::ArrayRef<::mlir::Attribute> operands, ::llvm::SmallVectorImpl<::mlir::OpFoldResult> &results); // CHECK: ::mlir::LogicalResult fold(::llvm::ArrayRef<::mlir::Attribute> operands, ::llvm::SmallVectorImpl<::mlir::OpFoldResult> &results);
// CHECK: // Display a graph for debugging purposes. // CHECK: // Display a graph for debugging purposes.
// CHECK: void displayGraph(); // CHECK: void displayGraph();
// CHECK: }; // CHECK: };
// DEFS-LABEL: NS::AOp definitions // DEFS-LABEL: NS::AOp definitions
// DEFS: AOpAdaptor::AOpAdaptor(::mlir::ValueRange values, ::mlir::DictionaryAttr attrs, ::mlir::RegionRange regions) : odsOperands(values), odsAttrs(attrs), odsRegions(regions) // DEFS: AOpAdaptor::AOpAdaptor(::mlir::ValueRange values, ::mlir::DictionaryAttr attrs, ::mlir::RegionRange regions) : odsOperands(values), odsAttrs(attrs), odsRegions(regions)
▲ Show 20 Lines • Show All 217 Lines • Show Last 20 Lines

mlir/tools/mlir-linalg-ods-gen/mlir-linalg-ods-gen.cpp

Show First 20 Lines • Show All 1,512 Lines • ▼ Show 20 LinesLogicalResult TCParser::parseExpression(TensorUse currentDefinition,
// Check if it has a reduction set, discover dimensions eagerly. // Check if it has a reduction set, discover dimensions eagerly.
if (parser.curToken.is(Token::Kind::lt)) { if (parser.curToken.is(Token::Kind::lt)) {
auto iters = parseAffineExprs(EagerDiscoveryMode::Dimensions, state.dims, auto iters = parseAffineExprs(EagerDiscoveryMode::Dimensions, state.dims,
Token::Kind::lt, Token::Kind::gt); Token::Kind::lt, Token::Kind::gt);
for (auto iter : iters) for (auto iter : iters)
reductionDims.push_back(iter.cast<AffineDimExpr>().getPosition()); reductionDims.push_back(iter.cast<AffineDimExpr>().getPosition());
} }
auto parseExpr = [&]() -> LogicalResult { auto parseExpr = [&]() -> LogicalResult {
std::unique_ptr<Expression> e; std::unique_ptr<Expression> e;
if (failed(parseExpression(currentDefinition, e, state))) if (failed(parseExpression(currentDefinition, e, state)))
return failure(); return failure();
expressions.push_back(std::move(e)); expressions.push_back(std::move(e));
return success(); return success();
}; };
if (failed(parser.parseToken(Token::Kind::l_paren, "expected '('")) || if (failed(parser.parseToken(Token::Kind::l_paren, "expected '('")) ||
▲ Show 20 Lines • Show All 539 Lines • ▼ Show 20 Linesvoid TCParser::printReferenceIterators(llvm::raw_ostream &os,
os << llvm::formatv(referenceReferenceIteratorsFmt, cppOpName, iteratorsStr); os << llvm::formatv(referenceReferenceIteratorsFmt, cppOpName, iteratorsStr);
} }
void TCParser::printCanonicalizersAndFolders(llvm::raw_ostream &os, void TCParser::printCanonicalizersAndFolders(llvm::raw_ostream &os,
StringRef cppOpName) { StringRef cppOpName) {
const char *canonicalizersAndFoldersFmt = R"FMT( const char *canonicalizersAndFoldersFmt = R"FMT(
void {0}::getCanonicalizationPatterns( void {0}::getCanonicalizationPatterns(
OwningRewritePatternList &results, RewritePatternSet &results,
MLIRContext *context) {{ MLIRContext *context) {{
results.insert<EraseDeadLinalgOp>(); results.add<EraseDeadLinalgOp>();
results.insert<FoldTensorCastOp>(); results.add<FoldTensorCastOp>();
} }
LogicalResult {0}::fold(ArrayRef<Attribute>, LogicalResult {0}::fold(ArrayRef<Attribute>,
SmallVectorImpl<OpFoldResult> &) {{ SmallVectorImpl<OpFoldResult> &) {{
return foldMemRefCast(*this); return foldMemRefCast(*this);
} }
void {0}::getEffects(SmallVectorImpl< void {0}::getEffects(SmallVectorImpl<
SideEffects::EffectInstance<MemoryEffects::Effect> >&effects) {{ SideEffects::EffectInstance<MemoryEffects::Effect> >&effects) {{
getGenericEffectsImpl(effects, getGenericEffectsImpl(effects,
▲ Show 20 Lines • Show All 363 Lines • Show Last 20 Lines

mlir/tools/mlir-linalg-ods-gen/mlir-linalg-ods-yaml-gen.cpp

Show First 20 Lines • Show All 513 Lines • ▼ Show 20 Lines
} }
)FMT"; )FMT";
// Implementations of getCanonicalizationPatterns, fold and getEffects. // Implementations of getCanonicalizationPatterns, fold and getEffects.
// Parameters: // Parameters:
// {0}: Class name // {0}: Class name
const char structuredOpCanonicalizersAndFoldersFormat[] = R"FMT( const char structuredOpCanonicalizersAndFoldersFormat[] = R"FMT(
void {0}::getCanonicalizationPatterns( void {0}::getCanonicalizationPatterns(
OwningRewritePatternList &results, RewritePatternSet &results,
MLIRContext *context) {{ MLIRContext *context) {{
results.insert<EraseDeadLinalgOp>(); results.add<EraseDeadLinalgOp>();
results.insert<FoldTensorCastOp>(); results.add<FoldTensorCastOp>();
} }
LogicalResult {0}::fold(ArrayRef<Attribute>, LogicalResult {0}::fold(ArrayRef<Attribute>,
SmallVectorImpl<OpFoldResult> &) {{ SmallVectorImpl<OpFoldResult> &) {{
return foldMemRefCast(*this); return foldMemRefCast(*this);
} }
void {0}::getEffects(SmallVectorImpl< void {0}::getEffects(SmallVectorImpl<
SideEffects::EffectInstance<MemoryEffects::Effect> >&effects) {{ SideEffects::EffectInstance<MemoryEffects::Effect> >&effects) {{
getGenericEffectsImpl(effects, getGenericEffectsImpl(effects,
▲ Show 20 Lines • Show All 419 Lines • Show Last 20 Lines

mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp

Show First 20 Lines • Show All 1,672 Lines • ▼ Show 20 Linesvoid OpEmitter::genCodeForAddingArgAndRegionForBuilder(OpMethodBody &body,
} }
} }
void OpEmitter::genCanonicalizerDecls() { void OpEmitter::genCanonicalizerDecls() {
if (!def.getValueAsBit("hasCanonicalizer")) if (!def.getValueAsBit("hasCanonicalizer"))
return; return;
SmallVector<OpMethodParameter, 2> paramList; SmallVector<OpMethodParameter, 2> paramList;
paramList.emplace_back("::mlir::OwningRewritePatternList &", "results"); paramList.emplace_back("::mlir::RewritePatternSet &", "results");
paramList.emplace_back("::mlir::MLIRContext *", "context"); paramList.emplace_back("::mlir::MLIRContext *", "context");
opClass.addMethodAndPrune("void", "getCanonicalizationPatterns", opClass.addMethodAndPrune("void", "getCanonicalizationPatterns",
OpMethod::MP_StaticDeclaration, OpMethod::MP_StaticDeclaration,
std::move(paramList)); std::move(paramList));
} }
void OpEmitter::genFolderDecls() { void OpEmitter::genFolderDecls() {
bool hasSingleResult = bool hasSingleResult =
▲ Show 20 Lines • Show All 824 Lines • Show Last 20 Lines

mlir/tools/mlir-tblgen/RewriterGen.cpp

Show First 20 Lines • Show All 1,285 Lines • ▼ Show 20 Lines for (Record *p : patterns) {
PatternEmitter(p, &recordOpMap, os).emit(name); PatternEmitter(p, &recordOpMap, os).emit(name);
LLVM_DEBUG(llvm::dbgs() LLVM_DEBUG(llvm::dbgs()
<< "=== done generating pattern '" << name << "' ===\n"); << "=== done generating pattern '" << name << "' ===\n");
rewriterNames.push_back(std::move(name)); rewriterNames.push_back(std::move(name));
} }
// Emit function to add the generated matchers to the pattern list. // Emit function to add the generated matchers to the pattern list.
os << "void LLVM_ATTRIBUTE_UNUSED populateWithGenerated(" os << "void LLVM_ATTRIBUTE_UNUSED populateWithGenerated("
"::mlir::OwningRewritePatternList &patterns) {\n"; "::mlir::RewritePatternSet &patterns) {\n";
for (const auto &name : rewriterNames) { for (const auto &name : rewriterNames) {
os << " patterns.insert<" << name << ">(patterns.getContext());\n"; os << " patterns.add<" << name << ">(patterns.getContext());\n";
} }
os << "}\n"; os << "}\n";
} }
static mlir::GenRegistration static mlir::GenRegistration
genRewriters("gen-rewriters", "Generate pattern rewriters", genRewriters("gen-rewriters", "Generate pattern rewriters",
[](const RecordKeeper &records, raw_ostream &os) { [](const RecordKeeper &records, raw_ostream &os) {
emitRewriters(records, os); emitRewriters(records, os);
return false; return false;
}); });

mlir/unittests/Rewrite/PatternBenefit.cpp

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Lines matchAndRewrite(Operation * /*op*/,
*called = true; *called = true;
return failure(); return failure();
} }
private: private:
bool *called; bool *called;
}; };
OwningRewritePatternList patterns(&context); RewritePatternSet patterns(&context);
bool called1 = false; bool called1 = false;
bool called2 = false; bool called2 = false;
patterns.insert<Pattern1>(&context, &called1); patterns.add<Pattern1>(&context, &called1);
patterns.insert<Pattern2>(&called2); patterns.add<Pattern2>(&called2);
FrozenRewritePatternList frozenPatterns(std::move(patterns)); FrozenRewritePatternList frozenPatterns(std::move(patterns));
PatternApplicator pa(frozenPatterns); PatternApplicator pa(frozenPatterns);
pa.applyDefaultCostModel(); pa.applyDefaultCostModel();
class MyPatternRewriter : public PatternRewriter { class MyPatternRewriter : public PatternRewriter {
public: public:
MyPatternRewriter(MLIRContext *ctx) : PatternRewriter(ctx) {} MyPatternRewriter(MLIRContext *ctx) : PatternRewriter(ctx) {}
Show All 9 Lines