This is an archive of the discontinued LLVM Phabricator instance.

Differential D138934

mlir/tblgen: use std::optional in generation
ClosedPublic

Authored by artagnon on Nov 29 2022, 11:11 AM.

Details

Reviewers
kazu
antiagainst
rriddle
aartbik
jpienaar
ftynse
aaron.ballman
bondhugula
ThomasRaoux
nicolasvasilache
herhut
dcaballe
hanchung
Commits
rG22426110c5ef: mlir/tblgen: use std::optional in generation
Summary

This is part of an effort to migrate from llvm::Optional to
std::optional. This patch changes the way mlir-tblgen generates .inc
files, and modifies tests and documentation appropriately. It is a "no
compromises" patch, and doesn't leave the user with an unpleasant mix of
llvm::Optional and std::optional.

A non-trivial change has been made to ControlFlowInterfaces to split one
constructor into two, relating to a build failure on Windows.

See also: https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

Signed-off-by: Ramkumar Ramachandra <r@artagnon.com>

Diff Detail

Event Timeline

artagnon created this revision.Nov 29 2022, 11:11 AM
Herald added a reviewer: rriddle. · View Herald TranscriptNov 29 2022, 11:11 AM
Herald added a reviewer: aartbik. · View Herald Transcript
Herald added a reviewer: jpienaar. · View Herald Transcript
Herald added a reviewer: ftynse. · View Herald Transcript
Herald added a reviewer: aaron.ballman. · View Herald Transcript
Herald added a reviewer: aartbik. · View Herald Transcript
Herald added a reviewer: bondhugula. · View Herald Transcript
Herald added a reviewer: ThomasRaoux. · View Herald Transcript
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: jsetoain, hanchung, Moerafaat and 37 others. · View Herald Transcript
artagnon requested review of this revision.Nov 29 2022, 11:11 AM
Herald added a reviewer: nicolasvasilache. · View Herald TranscriptNov 29 2022, 11:11 AM
Herald added a reviewer: herhut. · View Herald Transcript
Herald added a reviewer: dcaballe. · View Herald Transcript
Herald added projects: Restricted Project, Restricted Project. · View Herald Transcript
Herald added subscribers: llvm-commits, stephenneuendorffer, nicolasvasilache, jholewinski. · View Herald Transcript
kazu added a comment.Nov 29 2022, 12:01 PM

@artagnon, thanks for your help. I have a couple of concerns:

  • You seem to touch formatting in a few places. Could you keep unrelated parts intact? Alternatively, you can format the entire file in a separate patch so that your modification will stand out by itself. You can use git clang-format HEAD~ to format the parts you touch.
  • Somewhat related to formatting, your patch touches the documentation a lot. If it's not related to the std::optional migration, please upload a separate patch.
  • llvm::None is interchangeable with std::nullopt. If I were you, I might consider replacing llvm::None with std::nullopt first. This way, even if this patch causes a regression and get reverted, the patch to migrate to std::nullopt should stay, resulting in a net progress.
Harbormaster completed remote builds in B200068: Diff 478652.Nov 29 2022, 1:00 PM
artagnon added a parent revision: D138984: mlir: replace llvm::None with std::nullopt (NFC).Nov 30 2022, 12:09 AM
artagnon updated this revision to Diff 478828.Nov 30 2022, 12:09 AM

Rebased on top of D138984.

Harbormaster completed remote builds in B200191: Diff 478828.Nov 30 2022, 12:58 AM
artagnon updated this revision to Diff 479002.Nov 30 2022, 9:58 AM

Address some build failures in flang. Unfortunately, it's very
difficult to build flang locally, as it takes a really long time.
I'm relying on the CI to tell us about additional failures.

Herald added a project: Restricted Project. · View Herald TranscriptNov 30 2022, 9:58 AM
artagnon mentioned this in D138984: mlir: replace llvm::None with std::nullopt (NFC).Nov 30 2022, 10:45 AM
Harbormaster completed remote builds in B200312: Diff 479002.Nov 30 2022, 10:47 AM
Herald added a subscriber: jdoerfert. · View Herald TranscriptNov 30 2022, 10:47 AM
artagnon updated this revision to Diff 479191.Dec 1 2022, 1:24 AM

Fix all flang compile errors.

Herald added a reviewer: hanchung. · View Herald TranscriptDec 1 2022, 1:24 AM
Harbormaster completed remote builds in B200448: Diff 479191.Dec 1 2022, 2:51 AM
kazu mentioned this in D139240: [llvm] Add support for hashing std::optional.Dec 2 2022, 5:37 PM
kazu mentioned this in rGf64d4a26cee0: [llvm] Add support for hashing std::optional.Dec 2 2022, 6:53 PM
jpienaar added inline comments.Dec 5 2022, 3:21 PM
mlir/lib/Dialect/Shape/IR/Shape.cpp
1342

Why couldn't this be std::optional<int64_t> ?

artagnon added a comment.Dec 6 2022, 1:19 PM

The Windows build is failing, and I don't know why. In light of @kazu's comments on D138984, and his recent commits, I'm tempted to abandon this change, and let him take over the work.

mlir/lib/Dialect/Shape/IR/Shape.cpp
1342

It could very well be. I suppose it's a matter of style: personally, I prefer auto to make the code less brittle to changes.

clementval added a subscriber: clementval.Dec 7 2022, 6:55 AM
clementval added inline comments.
mlir/lib/Dialect/Shape/IR/Shape.cpp
1342

std::optional<int64_t> seems better here (https://llvm.org/docs/CodingStandards.html#use-auto-type-deduction-to-make-code-more-readable)

artagnon edited the summary of this revision. (Show Details)Dec 13 2022, 6:38 AM
artagnon removed a parent revision: D138984: mlir: replace llvm::None with std::nullopt (NFC).
artagnon updated this revision to Diff 482453.Dec 13 2022, 6:40 AM

Rebase onto main, address review comments, and attempt to fix build on
Windows.

artagnon marked 2 inline comments as done.Dec 13 2022, 6:41 AM
artagnon added inline comments.
mlir/lib/Dialect/Shape/IR/Shape.cpp
1342

Fixed now.

Harbormaster completed remote builds in B202826: Diff 482453.Dec 13 2022, 7:17 AM
artagnon edited the summary of this revision. (Show Details)Dec 14 2022, 3:24 AM
artagnon updated this revision to Diff 482787.Dec 14 2022, 3:26 AM
artagnon marked an inline comment as done.

Attempt to fix failing builds on Debian/Windows.

Harbormaster completed remote builds in B203075: Diff 482787.Dec 14 2022, 4:00 AM
aartbik added a comment.Dec 14 2022, 2:14 PM

LGTM for the sparse part, since changes are mechanical in nature

jpienaar accepted this revision.Dec 16 2022, 1:24 PM

This looks direct enough and replaces deprecated struct, so LGTM.

This revision is now accepted and ready to land.Dec 16 2022, 1:24 PM
jpienaar added a comment.Dec 16 2022, 1:25 PM

(@kazu seems to be doing related cleanups, so I'd leave last check to him)

artagnon updated this revision to Diff 483734.Dec 17 2022, 12:51 AM

Rebased onto main, with changes. Posting here for pre-merge CI checks.

Harbormaster completed remote builds in B203751: Diff 483734.Dec 17 2022, 1:47 AM
Closed by commit rG22426110c5ef: mlir/tblgen: use std::optional in generation (authored by artagnon). · Explain WhyDec 17 2022, 2:14 AM
This revision was automatically updated to reflect the committed changes.
artagnon added a commit: rG22426110c5ef: mlir/tblgen: use std::optional in generation.

Revision Contents

PathSize
flang/
include/
flang/
Optimizer/
Dialect/
18 lines
4 lines
HLFIR/
2 lines
lib/
Frontend/
2 lines
Lower/
2 lines
Optimizer/
CodeGen/
20 lines
Dialect/
42 lines
2 lines
HLFIR/
IR/
2 lines
Transforms/
15 lines
2 lines
mlir/
docs/
DefiningDialects/
24 lines
Dialects/
Linalg/
4 lines
include/
mlir/
Analysis/
DataFlow/
2 lines
Conversion/
MemRefToLLVM/
2 lines
Dialect/
Affine/
IR/
2 lines
Arith/
IR/
6 lines
GPU/
TransformOps/
2 lines
LLVMIR/
14 lines
Linalg/
IR/
2 lines
TransformOps/
2 lines
Transforms/
37 lines
MemRef/
IR/
2 lines
PDLInterp/
IR/
4 lines
SCF/
IR/
10 lines
SPIRV/
IR/
3 lines
8 lines
Shape/
IR/
4 lines
SparseTensor/
IR/
4 lines
Tensor/
IR/
4 lines
Utils/
2 lines
Vector/
IR/
2 lines
10 lines
IR/
6 lines
2 lines
6 lines
2 lines
6 lines
3 lines
4 lines
2 lines
Interfaces/
23 lines
14 lines
13 lines
6 lines
8 lines
2 lines
4 lines
lib/
Analysis/
AliasAnalysis/
13 lines
DataFlow/
10 lines
6 lines
AsmParser/
2 lines
CAPI/
Interfaces/
2 lines
Conversion/
AffineToStandard/
4 lines
MemRefToLLVM/
6 lines
NVGPUToNVVM/
18 lines
PDLToPDLInterp/
2 lines
16 lines
23 lines
Dialect/
Affine/
IR/
33 lines
Arith/
IR/
15 lines
Async/
IR/
7 lines
Bufferization/
IR/
2 lines
Transforms/
2 lines
ControlFlow/
IR/
5 lines
EmitC/
IR/
4 lines
GPU/
IR/
3 lines
TransformOps/
42 lines
LLVMIR/
IR/
10 lines
20 lines
Linalg/
TransformOps/
2 lines
Transforms/
18 lines
18 lines
4 lines
7 lines
Utils/
2 lines
MemRef/
IR/
27 lines
Transforms/
9 lines
4 lines
NVGPU/
Transforms/
2 lines
OpenMP/
IR/
20 lines
PDL/
IR/
8 lines
SCF/
IR/
36 lines
Transforms/
6 lines
14 lines
Utils/
4 lines
SPIRV/
IR/
36 lines
22 lines
Transforms/
4 lines
4 lines
Shape/
IR/
34 lines
SparseTensor/
IR/
15 lines
Transforms/
9 lines
2 lines
Tensor/
IR/
36 lines
2 lines
Transforms/
2 lines
Tosa/
IR/
62 lines
Transforms/
2 lines
Transform/
IR/
16 lines
Utils/
9 lines
2 lines
Vector/
IR/
31 lines
Transforms/
45 lines
IR/
4 lines
8 lines
2 lines
Interfaces/
36 lines
11 lines
Rewrite/
6 lines
Target/
LLVMIR/
6 lines
Dialect/
OpenMP/
13 lines
8 lines
Tools/
PDLL/
CodeGen/
2 lines
test/
lib/
Dialect/
Test/
38 lines
2 lines
14 lines
mlir-tblgen/
8 lines
10 lines
4 lines
python/
2 lines
tools/
mlir-tblgen/
2 lines
24 lines
6 lines
4 lines
6 lines
unittests/
Interfaces/
8 lines
TableGen/
4 lines

Diff 483736

flang/include/flang/Optimizer/Dialect/FIROps.td

Show First 20 Lines • Show All 477 Lines • ▼ Show 20 Lines string extraSwitchClassDeclaration = [{
} }
mlir::Value getSelector(mlir::ValueRange operands) { mlir::Value getSelector(mlir::ValueRange operands) {
return operands.front(); return operands.front();
} }
// The number of blocks that may be branched to // The number of blocks that may be branched to
unsigned getNumDest() { return (*this)->getNumSuccessors(); } unsigned getNumDest() { return (*this)->getNumSuccessors(); }
llvm::Optional<mlir::OperandRange> getCompareOperands(unsigned cond); std::optional<mlir::OperandRange> getCompareOperands(unsigned cond);
llvm::Optional<llvm::ArrayRef<mlir::Value>> getCompareOperands( std::optional<llvm::ArrayRef<mlir::Value>> getCompareOperands(
llvm::ArrayRef<mlir::Value> operands, unsigned cond); llvm::ArrayRef<mlir::Value> operands, unsigned cond);
llvm::Optional<mlir::ValueRange> getCompareOperands( std::optional<mlir::ValueRange> getCompareOperands(
mlir::ValueRange operands, unsigned cond); mlir::ValueRange operands, unsigned cond);
llvm::Optional<llvm::ArrayRef<mlir::Value>> getSuccessorOperands( std::optional<llvm::ArrayRef<mlir::Value>> getSuccessorOperands(
llvm::ArrayRef<mlir::Value> operands, unsigned cond); llvm::ArrayRef<mlir::Value> operands, unsigned cond);
llvm::Optional<mlir::ValueRange> getSuccessorOperands( std::optional<mlir::ValueRange> getSuccessorOperands(
mlir::ValueRange operands, unsigned cond); mlir::ValueRange operands, unsigned cond);
// Helper function to deal with Optional operand forms // Helper function to deal with Optional operand forms
void printSuccessorAtIndex(mlir::OpAsmPrinter &p, unsigned i) { void printSuccessorAtIndex(mlir::OpAsmPrinter &p, unsigned i) {
auto *succ = getSuccessor(i); auto *succ = getSuccessor(i);
auto ops = getSuccessorOperands(i); auto ops = getSuccessorOperands(i);
p.printSuccessorAndUseList(succ, ops.getForwardedOperands()); p.printSuccessorAndUseList(succ, ops.getForwardedOperands());
} }
▲ Show 20 Lines • Show All 1,918 Lines • ▼ Show 20 Linesdef fir_StringLitOp : fir_Op<"string_lit", [NoMemoryEffect]> {
let results = (outs fir_CharacterType); let results = (outs fir_CharacterType);
let hasCustomAssemblyFormat = 1; let hasCustomAssemblyFormat = 1;
let hasVerifier = 1; let hasVerifier = 1;
let builders = [ let builders = [
OpBuilder<(ins "fir::CharacterType":$inType, OpBuilder<(ins "fir::CharacterType":$inType,
"llvm::StringRef":$value, "llvm::StringRef":$value,
CArg<"llvm::Optional<int64_t>", "{}">:$len)>, CArg<"std::optional<int64_t>", "{}">:$len)>,
OpBuilder<(ins "fir::CharacterType":$inType, OpBuilder<(ins "fir::CharacterType":$inType,
"llvm::ArrayRef<char>":$xlist, "llvm::ArrayRef<char>":$xlist,
CArg<"llvm::Optional<int64_t>", "{}">:$len)>, CArg<"std::optional<int64_t>", "{}">:$len)>,
OpBuilder<(ins "fir::CharacterType":$inType, OpBuilder<(ins "fir::CharacterType":$inType,
"llvm::ArrayRef<char16_t>":$xlist, "llvm::ArrayRef<char16_t>":$xlist,
CArg<"llvm::Optional<int64_t>", "{}">:$len)>, CArg<"std::optional<int64_t>", "{}">:$len)>,
OpBuilder<(ins "fir::CharacterType":$inType, OpBuilder<(ins "fir::CharacterType":$inType,
"llvm::ArrayRef<char32_t>":$xlist, "llvm::ArrayRef<char32_t>":$xlist,
CArg<"llvm::Optional<int64_t>", "{}">:$len)>]; CArg<"std::optional<int64_t>", "{}">:$len)>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static constexpr const char *size() { return "size"; } static constexpr const char *size() { return "size"; }
static constexpr const char *value() { return "value"; } static constexpr const char *value() { return "value"; }
static constexpr const char *xlist() { return "xlist"; } static constexpr const char *xlist() { return "xlist"; }
// Get the LEN attribute of this character constant // Get the LEN attribute of this character constant
mlir::Attribute getSize() { return (*this)->getAttr(size()); } mlir::Attribute getSize() { return (*this)->getAttr(size()); }
▲ Show 20 Lines • Show All 515 Lines • Show Last 20 Lines

flang/include/flang/Optimizer/Dialect/FortranVariableInterface.td

Show All 30 Lines InterfaceMethod<
/*methodBody=*/[{}], /*methodBody=*/[{}],
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
ConcreteOp op = mlir::cast<ConcreteOp>(this->getOperation()); ConcreteOp op = mlir::cast<ConcreteOp>(this->getOperation());
return op.getResult(); return op.getResult();
}] }]
>, >,
InterfaceMethod< InterfaceMethod<
/*desc=*/"Get Fortran attributes", /*desc=*/"Get Fortran attributes",
/*retTy=*/"llvm::Optional<fir::FortranVariableFlagsEnum>", /*retTy=*/"std::optional<fir::FortranVariableFlagsEnum>",
/*methodName=*/"getFortranAttrs", /*methodName=*/"getFortranAttrs",
/*args=*/(ins), /*args=*/(ins),
/*methodBody=*/[{}], /*methodBody=*/[{}],
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
ConcreteOp op = mlir::cast<ConcreteOp>(this->getOperation()); ConcreteOp op = mlir::cast<ConcreteOp>(this->getOperation());
return op.getFortran_attrs(); return op.getFortran_attrs();
}] }]
>, >,
Show All 38 Lines let extraClassDeclaration = [{
} }
/// Is the variable an array? /// Is the variable an array?
bool isArray() { bool isArray() {
return getElementOrSequenceType().isa<fir::SequenceType>(); return getElementOrSequenceType().isa<fir::SequenceType>();
} }
/// Return the rank of the entity if it is known at compile time. /// Return the rank of the entity if it is known at compile time.
llvm::Optional<unsigned> getRank() { std::optional<unsigned> getRank() {
if (auto sequenceType = if (auto sequenceType =
getElementOrSequenceType().dyn_cast<fir::SequenceType>()) { getElementOrSequenceType().dyn_cast<fir::SequenceType>()) {
if (sequenceType.hasUnknownShape()) if (sequenceType.hasUnknownShape())
return {}; return {};
return sequenceType.getDimension(); return sequenceType.getDimension();
} }
return 0; return 0;
} }
▲ Show 20 Lines • Show All 64 Lines • Show Last 20 Lines

flang/include/flang/Optimizer/HLFIR/HLFIROps.td

Show First 20 Lines • Show All 196 Lines • ▼ Show 20 Lines let extraClassDeclaration = [{
using Subscripts = llvm::SmallVector<Subscript, 8>; using Subscripts = llvm::SmallVector<Subscript, 8>;
}]; }];
let builders = [ let builders = [
OpBuilder<(ins "mlir::Type":$result_type, "mlir::Value":$memref, OpBuilder<(ins "mlir::Type":$result_type, "mlir::Value":$memref,
"llvm::StringRef":$component, "mlir::Value":$component_shape, "llvm::StringRef":$component, "mlir::Value":$component_shape,
"llvm::ArrayRef<std::variant<mlir::Value, std::tuple<mlir::Value, mlir::Value, mlir::Value>>>":$subscripts, "llvm::ArrayRef<std::variant<mlir::Value, std::tuple<mlir::Value, mlir::Value, mlir::Value>>>":$subscripts,
CArg<"mlir::ValueRange", "{}">:$substring, CArg<"mlir::ValueRange", "{}">:$substring,
CArg<"llvm::Optional<bool>", "{}">:$complex_part, CArg<"std::optional<bool>", "{}">:$complex_part,
CArg<"mlir::Value", "{}">:$shape, CArg<"mlir::ValueRange", "{}">:$typeparams, CArg<"mlir::Value", "{}">:$shape, CArg<"mlir::ValueRange", "{}">:$typeparams,
CArg<"fir::FortranVariableFlagsAttr", "{}">:$fortran_attrs)>, CArg<"fir::FortranVariableFlagsAttr", "{}">:$fortran_attrs)>,
OpBuilder<(ins "mlir::Type":$result_type, "mlir::Value":$memref, OpBuilder<(ins "mlir::Type":$result_type, "mlir::Value":$memref,
"mlir::ValueRange":$indices, "mlir::ValueRange":$indices,
CArg<"mlir::ValueRange", "{}">:$typeparams, CArg<"mlir::ValueRange", "{}">:$typeparams,
CArg<"fir::FortranVariableFlagsAttr", "{}">:$fortran_attrs)> CArg<"fir::FortranVariableFlagsAttr", "{}">:$fortran_attrs)>
]; ];
▲ Show 20 Lines • Show All 227 Lines • Show Last 20 Lines

flang/lib/Frontend/FrontendActions.cpp

Show First 20 Lines • Show All 546 Lines • ▼ Show 20 Linesvoid CodeGenAction::generateLLVMIR() {
// run the pass manager // run the pass manager
if (!mlir::succeeded(pm.run(*mlirModule))) { if (!mlir::succeeded(pm.run(*mlirModule))) {
unsigned diagID = ci.getDiagnostics().getCustomDiagID( unsigned diagID = ci.getDiagnostics().getCustomDiagID(
clang::DiagnosticsEngine::Error, "Lowering to LLVM IR failed"); clang::DiagnosticsEngine::Error, "Lowering to LLVM IR failed");
ci.getDiagnostics().Report(diagID); ci.getDiagnostics().Report(diagID);
} }
// Translate to LLVM IR // Translate to LLVM IR
llvm::Optional<llvm::StringRef> moduleName = mlirModule->getName(); std::optional<llvm::StringRef> moduleName = mlirModule->getName();
llvmModule = mlir::translateModuleToLLVMIR( llvmModule = mlir::translateModuleToLLVMIR(
*mlirModule, *llvmCtx, moduleName ? *moduleName : "FIRModule"); *mlirModule, *llvmCtx, moduleName ? *moduleName : "FIRModule");
// Set PIC/PIE level LLVM module flags. // Set PIC/PIE level LLVM module flags.
if (opts.PICLevel > 0) { if (opts.PICLevel > 0) {
llvmModule->setPICLevel(static_cast<llvm::PICLevel::Level>(opts.PICLevel)); llvmModule->setPICLevel(static_cast<llvm::PICLevel::Level>(opts.PICLevel));
if (opts.IsPIE) if (opts.IsPIE)
llvmModule->setPIELevel( llvmModule->setPIELevel(
▲ Show 20 Lines • Show All 291 Lines • Show Last 20 Lines

flang/lib/Lower/ConvertExprToHLFIR.cpp

Show First 20 Lines • Show All 93 Lines • ▼ Show 20 Lines if (charType && charType.hasDynamicLen())
resultType = resultType =
fir::BoxCharType::get(charType.getContext(), charType.getFKind()); fir::BoxCharType::get(charType.getContext(), charType.getFKind());
else if (resultValueType.isa<fir::SequenceType>() || else if (resultValueType.isa<fir::SequenceType>() ||
fir::hasDynamicSize(resultValueType)) fir::hasDynamicSize(resultValueType))
resultType = fir::BoxType::get(resultValueType); resultType = fir::BoxType::get(resultValueType);
else else
resultType = fir::ReferenceType::get(resultValueType); resultType = fir::ReferenceType::get(resultValueType);
llvm::Optional<bool> complexPart; std::optional<bool> complexPart;
llvm::SmallVector<mlir::Value> substring; llvm::SmallVector<mlir::Value> substring;
auto designate = getBuilder().create<hlfir::DesignateOp>( auto designate = getBuilder().create<hlfir::DesignateOp>(
getLoc(), resultType, partInfo.base.getBase(), "", getLoc(), resultType, partInfo.base.getBase(), "",
/*componentShape=*/mlir::Value{}, partInfo.subscripts, substring, /*componentShape=*/mlir::Value{}, partInfo.subscripts, substring,
complexPart, partInfo.resultShape, partInfo.typeParams); complexPart, partInfo.resultShape, partInfo.typeParams);
return mlir::cast<fir::FortranVariableOpInterface>( return mlir::cast<fir::FortranVariableOpInterface>(
designate.getOperation()); designate.getOperation());
} }
▲ Show 20 Lines • Show All 792 Lines • Show Last 20 Lines

flang/lib/Optimizer/CodeGen/CodeGen.cpp

Show First 20 Lines • Show All 1,612 Lines • ▼ Show 20 Linesstruct EmboxCommonConversion : public FIROpConversion<OP> {
// The indices from the "outer" dimensions (every dimension after the first // The indices from the "outer" dimensions (every dimension after the first
// one (inlcuded) that is not a compile time constant) must have been // one (inlcuded) that is not a compile time constant) must have been
// multiplied with the related extents and added together into \p outerOffset. // multiplied with the related extents and added together into \p outerOffset.
mlir::Value mlir::Value
genBoxOffsetGep(mlir::ConversionPatternRewriter &rewriter, mlir::Location loc, genBoxOffsetGep(mlir::ConversionPatternRewriter &rewriter, mlir::Location loc,
mlir::Value base, mlir::Value outerOffset, mlir::Value base, mlir::Value outerOffset,
mlir::ValueRange cstInteriorIndices, mlir::ValueRange cstInteriorIndices,
mlir::ValueRange componentIndices, mlir::ValueRange componentIndices,
llvm::Optional<mlir::Value> substringOffset) const { std::optional<mlir::Value> substringOffset) const {
llvm::SmallVector<mlir::LLVM::GEPArg> gepArgs{outerOffset}; llvm::SmallVector<mlir::LLVM::GEPArg> gepArgs{outerOffset};
mlir::Type resultTy = mlir::Type resultTy =
base.getType().cast<mlir::LLVM::LLVMPointerType>().getElementType(); base.getType().cast<mlir::LLVM::LLVMPointerType>().getElementType();
// Fortran is column major, llvm GEP is row major: reverse the indices here. // Fortran is column major, llvm GEP is row major: reverse the indices here.
for (mlir::Value interiorIndex : llvm::reverse(cstInteriorIndices)) { for (mlir::Value interiorIndex : llvm::reverse(cstInteriorIndices)) {
auto arrayTy = resultTy.dyn_cast<mlir::LLVM::LLVMArrayType>(); auto arrayTy = resultTy.dyn_cast<mlir::LLVM::LLVMArrayType>();
if (!arrayTy) if (!arrayTy)
fir::emitFatalError( fir::emitFatalError(
▲ Show 20 Lines • Show All 272 Lines • ▼ Show 20 Lines for (unsigned di = 0, descIdx = 0; di < rank; ++di) {
if (hasShift) if (hasShift)
++shiftOffset; ++shiftOffset;
if (hasSlice) if (hasSlice)
sliceOffset += 3; sliceOffset += 3;
} }
if (hasSlice || hasSubcomp || hasSubstr) { if (hasSlice || hasSubcomp || hasSubstr) {
// Shift the base address. // Shift the base address.
llvm::SmallVector<mlir::Value> fieldIndices; llvm::SmallVector<mlir::Value> fieldIndices;
llvm::Optional<mlir::Value> substringOffset; std::optional<mlir::Value> substringOffset;
if (hasSubcomp) if (hasSubcomp)
getSubcomponentIndices(xbox, xbox.getMemref(), operands, fieldIndices); getSubcomponentIndices(xbox, xbox.getMemref(), operands, fieldIndices);
if (hasSubstr) if (hasSubstr)
substringOffset = operands[xbox.substrOffset()]; substringOffset = operands[xbox.substrOffset()];
base = genBoxOffsetGep(rewriter, loc, base, ptrOffset, cstInteriorIndices, base = genBoxOffsetGep(rewriter, loc, base, ptrOffset, cstInteriorIndices,
fieldIndices, substringOffset); fieldIndices, substringOffset);
} }
dest = insertBaseAddress(rewriter, loc, dest, base); dest = insertBaseAddress(rewriter, loc, dest, base);
▲ Show 20 Lines • Show All 123 Lines • ▼ Show 20 Lines sliceBox(fir::cg::XReboxOp rebox, mlir::Value dest, mlir::Value base,
if (!rebox.getSubcomponent().empty() || !rebox.getSubstr().empty()) { if (!rebox.getSubcomponent().empty() || !rebox.getSubstr().empty()) {
// Cast to inputEleTy* so that a GEP can be used. // Cast to inputEleTy* so that a GEP can be used.
mlir::Type inputEleTy = getInputEleTy(rebox); mlir::Type inputEleTy = getInputEleTy(rebox);
auto llvmElePtrTy = auto llvmElePtrTy =
mlir::LLVM::LLVMPointerType::get(convertType(inputEleTy)); mlir::LLVM::LLVMPointerType::get(convertType(inputEleTy));
base = rewriter.create<mlir::LLVM::BitcastOp>(loc, llvmElePtrTy, base); base = rewriter.create<mlir::LLVM::BitcastOp>(loc, llvmElePtrTy, base);
llvm::SmallVector<mlir::Value> fieldIndices; llvm::SmallVector<mlir::Value> fieldIndices;
llvm::Optional<mlir::Value> substringOffset; std::optional<mlir::Value> substringOffset;
if (!rebox.getSubcomponent().empty()) if (!rebox.getSubcomponent().empty())
getSubcomponentIndices(rebox, rebox.getBox(), operands, fieldIndices); getSubcomponentIndices(rebox, rebox.getBox(), operands, fieldIndices);
if (!rebox.getSubstr().empty()) if (!rebox.getSubstr().empty())
substringOffset = operands[rebox.substrOffset()]; substringOffset = operands[rebox.substrOffset()];
base = genBoxOffsetGep(rewriter, loc, base, zero, base = genBoxOffsetGep(rewriter, loc, base, zero,
/*cstInteriorIndices=*/std::nullopt, fieldIndices, /*cstInteriorIndices=*/std::nullopt, fieldIndices,
substringOffset); substringOffset);
} }
▲ Show 20 Lines • Show All 661 Lines • ▼ Show 20 Lines if (fir::hasDynamicSize(fir::unwrapSequenceType(cpnTy)))
return mlir::emitError( return mlir::emitError(
loc, "fir.coordinate_of with a dynamic element size is unsupported"); loc, "fir.coordinate_of with a dynamic element size is unsupported");
if (hasKnownShape || columnIsDeferred) { if (hasKnownShape || columnIsDeferred) {
llvm::SmallVector<mlir::LLVM::GEPArg> offs; llvm::SmallVector<mlir::LLVM::GEPArg> offs;
if (hasKnownShape && hasSubdimension) { if (hasKnownShape && hasSubdimension) {
offs.push_back(0); offs.push_back(0);
} }
llvm::Optional<int> dims; std::optional<int> dims;
llvm::SmallVector<mlir::Value> arrIdx; llvm::SmallVector<mlir::Value> arrIdx;
for (std::size_t i = 1, sz = operands.size(); i < sz; ++i) { for (std::size_t i = 1, sz = operands.size(); i < sz; ++i) {
mlir::Value nxtOpnd = operands[i]; mlir::Value nxtOpnd = operands[i];
if (!cpnTy) if (!cpnTy)
return mlir::emitError(loc, "invalid coordinate/check failed"); return mlir::emitError(loc, "invalid coordinate/check failed");
// check if the i-th coordinate relates to an array // check if the i-th coordinate relates to an array
▲ Show 20 Lines • Show All 188 Lines • ▼ Show 20 Lines for (unsigned i = 0; i < indexes.size(); i += 2) {
return false; return false;
} }
return true; return true;
} }
// TODO: String comparaison should be avoided. Replace linkName with an // TODO: String comparaison should be avoided. Replace linkName with an
// enumeration. // enumeration.
mlir::LLVM::Linkage mlir::LLVM::Linkage
convertLinkage(llvm::Optional<llvm::StringRef> optLinkage) const { convertLinkage(std::optional<llvm::StringRef> optLinkage) const {
if (optLinkage) { if (optLinkage) {
auto name = *optLinkage; auto name = *optLinkage;
if (name == "internal") if (name == "internal")
return mlir::LLVM::Linkage::Internal; return mlir::LLVM::Linkage::Internal;
if (name == "linkonce") if (name == "linkonce")
return mlir::LLVM::Linkage::Linkonce; return mlir::LLVM::Linkage::Linkonce;
if (name == "linkonce_odr") if (name == "linkonce_odr")
return mlir::LLVM::Linkage::LinkonceODR; return mlir::LLVM::Linkage::LinkonceODR;
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Linesstruct NoReassocOpConversion : public FIROpConversion<fir::NoReassocOp> {
matchAndRewrite(fir::NoReassocOp noreassoc, OpAdaptor adaptor, matchAndRewrite(fir::NoReassocOp noreassoc, OpAdaptor adaptor,
mlir::ConversionPatternRewriter &rewriter) const override { mlir::ConversionPatternRewriter &rewriter) const override {
rewriter.replaceOp(noreassoc, adaptor.getOperands()[0]); rewriter.replaceOp(noreassoc, adaptor.getOperands()[0]);
return mlir::success(); return mlir::success();
} }
}; };
static void genCondBrOp(mlir::Location loc, mlir::Value cmp, mlir::Block *dest, static void genCondBrOp(mlir::Location loc, mlir::Value cmp, mlir::Block *dest,
llvm::Optional<mlir::ValueRange> destOps, std::optional<mlir::ValueRange> destOps,
mlir::ConversionPatternRewriter &rewriter, mlir::ConversionPatternRewriter &rewriter,
mlir::Block *newBlock) { mlir::Block *newBlock) {
if (destOps) if (destOps)
rewriter.create<mlir::LLVM::CondBrOp>(loc, cmp, dest, *destOps, newBlock, rewriter.create<mlir::LLVM::CondBrOp>(loc, cmp, dest, *destOps, newBlock,
mlir::ValueRange()); mlir::ValueRange());
else else
rewriter.create<mlir::LLVM::CondBrOp>(loc, cmp, dest, newBlock); rewriter.create<mlir::LLVM::CondBrOp>(loc, cmp, dest, newBlock);
} }
template <typename A, typename B> template <typename A, typename B>
static void genBrOp(A caseOp, mlir::Block *dest, llvm::Optional<B> destOps, static void genBrOp(A caseOp, mlir::Block *dest, std::optional<B> destOps,
mlir::ConversionPatternRewriter &rewriter) { mlir::ConversionPatternRewriter &rewriter) {
if (destOps) if (destOps)
rewriter.replaceOpWithNewOp<mlir::LLVM::BrOp>(caseOp, *destOps, dest); rewriter.replaceOpWithNewOp<mlir::LLVM::BrOp>(caseOp, *destOps, dest);
else else
rewriter.replaceOpWithNewOp<mlir::LLVM::BrOp>(caseOp, std::nullopt, dest); rewriter.replaceOpWithNewOp<mlir::LLVM::BrOp>(caseOp, std::nullopt, dest);
} }
static void genCaseLadderStep(mlir::Location loc, mlir::Value cmp, static void genCaseLadderStep(mlir::Location loc, mlir::Value cmp,
mlir::Block *dest, mlir::Block *dest,
llvm::Optional<mlir::ValueRange> destOps, std::optional<mlir::ValueRange> destOps,
mlir::ConversionPatternRewriter &rewriter) { mlir::ConversionPatternRewriter &rewriter) {
auto *thisBlock = rewriter.getInsertionBlock(); auto *thisBlock = rewriter.getInsertionBlock();
auto *newBlock = createBlock(rewriter, dest); auto *newBlock = createBlock(rewriter, dest);
rewriter.setInsertionPointToEnd(thisBlock); rewriter.setInsertionPointToEnd(thisBlock);
genCondBrOp(loc, cmp, dest, destOps, rewriter, newBlock); genCondBrOp(loc, cmp, dest, destOps, rewriter, newBlock);
rewriter.setInsertionPointToEnd(newBlock); rewriter.setInsertionPointToEnd(newBlock);
} }
Show All 29 Lines matchAndRewrite(fir::SelectCaseOp caseOp, OpAdaptor adaptor,
if (ty.isa<fir::CharacterType>()) { if (ty.isa<fir::CharacterType>()) {
TODO(caseOp.getLoc(), "fir.select_case codegen with character type"); TODO(caseOp.getLoc(), "fir.select_case codegen with character type");
return mlir::failure(); return mlir::failure();
} }
mlir::Value selector = caseOp.getSelector(adaptor.getOperands()); mlir::Value selector = caseOp.getSelector(adaptor.getOperands());
auto loc = caseOp.getLoc(); auto loc = caseOp.getLoc();
for (unsigned t = 0; t != conds; ++t) { for (unsigned t = 0; t != conds; ++t) {
mlir::Block *dest = caseOp.getSuccessor(t); mlir::Block *dest = caseOp.getSuccessor(t);
llvm::Optional<mlir::ValueRange> destOps = std::optional<mlir::ValueRange> destOps =
caseOp.getSuccessorOperands(adaptor.getOperands(), t); caseOp.getSuccessorOperands(adaptor.getOperands(), t);
llvm::Optional<mlir::ValueRange> cmpOps = std::optional<mlir::ValueRange> cmpOps =
*caseOp.getCompareOperands(adaptor.getOperands(), t); *caseOp.getCompareOperands(adaptor.getOperands(), t);
mlir::Value caseArg = *(cmpOps.value().begin()); mlir::Value caseArg = *(cmpOps.value().begin());
mlir::Attribute attr = cases[t]; mlir::Attribute attr = cases[t];
if (attr.isa<fir::PointIntervalAttr>()) { if (attr.isa<fir::PointIntervalAttr>()) {
auto cmp = rewriter.create<mlir::LLVM::ICmpOp>( auto cmp = rewriter.create<mlir::LLVM::ICmpOp>(
loc, mlir::LLVM::ICmpPredicate::eq, selector, caseArg); loc, mlir::LLVM::ICmpPredicate::eq, selector, caseArg);
genCaseLadderStep(loc, cmp, dest, destOps, rewriter); genCaseLadderStep(loc, cmp, dest, destOps, rewriter);
continue; continue;
▲ Show 20 Lines • Show All 702 Lines • Show Last 20 Lines

flang/lib/Optimizer/Dialect/FIROps.cpp

Show First 20 Lines • Show All 2,523 Lines • ▼ Show 20 LinesgetMutableSuccessorOperands(unsigned pos, mlir::MutableOperandRange operands,
mlir::NamedAttribute targetOffsetAttr = mlir::NamedAttribute targetOffsetAttr =
*owner->getAttrDictionary().getNamed(offsetAttr); *owner->getAttrDictionary().getNamed(offsetAttr);
return getSubOperands( return getSubOperands(
pos, operands, pos, operands,
targetOffsetAttr.getValue().cast<mlir::DenseI32ArrayAttr>(), targetOffsetAttr.getValue().cast<mlir::DenseI32ArrayAttr>(),
mlir::MutableOperandRange::OperandSegment(pos, targetOffsetAttr)); mlir::MutableOperandRange::OperandSegment(pos, targetOffsetAttr));
} }
llvm::Optional<mlir::OperandRange> fir::SelectOp::getCompareOperands(unsigned) { std::optional<mlir::OperandRange> fir::SelectOp::getCompareOperands(unsigned) {
return {}; return {};
} }
llvm::Optional<llvm::ArrayRef<mlir::Value>> std::optional<llvm::ArrayRef<mlir::Value>>
fir::SelectOp::getCompareOperands(llvm::ArrayRef<mlir::Value>, unsigned) { fir::SelectOp::getCompareOperands(llvm::ArrayRef<mlir::Value>, unsigned) {
return {}; return {};
} }
mlir::SuccessorOperands fir::SelectOp::getSuccessorOperands(unsigned oper) { mlir::SuccessorOperands fir::SelectOp::getSuccessorOperands(unsigned oper) {
return mlir::SuccessorOperands(::getMutableSuccessorOperands( return mlir::SuccessorOperands(::getMutableSuccessorOperands(
oper, getTargetArgsMutable(), getTargetOffsetAttr())); oper, getTargetArgsMutable(), getTargetOffsetAttr()));
} }
llvm::Optional<llvm::ArrayRef<mlir::Value>> std::optional<llvm::ArrayRef<mlir::Value>>
fir::SelectOp::getSuccessorOperands(llvm::ArrayRef<mlir::Value> operands, fir::SelectOp::getSuccessorOperands(llvm::ArrayRef<mlir::Value> operands,
unsigned oper) { unsigned oper) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(oper, getSubOperands(2, operands, segments), a)}; return {getSubOperands(oper, getSubOperands(2, operands, segments), a)};
} }
llvm::Optional<mlir::ValueRange> std::optional<mlir::ValueRange>
fir::SelectOp::getSuccessorOperands(mlir::ValueRange operands, unsigned oper) { fir::SelectOp::getSuccessorOperands(mlir::ValueRange operands, unsigned oper) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(oper, getSubOperands(2, operands, segments), a)}; return {getSubOperands(oper, getSubOperands(2, operands, segments), a)};
} }
unsigned fir::SelectOp::targetOffsetSize() { unsigned fir::SelectOp::targetOffsetSize() {
return (*this) return (*this)
->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()) ->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr())
.size(); .size();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SelectCaseOp // SelectCaseOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
llvm::Optional<mlir::OperandRange> std::optional<mlir::OperandRange>
fir::SelectCaseOp::getCompareOperands(unsigned cond) { fir::SelectCaseOp::getCompareOperands(unsigned cond) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getCompareOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getCompareOffsetAttr());
return {getSubOperands(cond, getCompareArgs(), a)}; return {getSubOperands(cond, getCompareArgs(), a)};
} }
llvm::Optional<llvm::ArrayRef<mlir::Value>> std::optional<llvm::ArrayRef<mlir::Value>>
fir::SelectCaseOp::getCompareOperands(llvm::ArrayRef<mlir::Value> operands, fir::SelectCaseOp::getCompareOperands(llvm::ArrayRef<mlir::Value> operands,
unsigned cond) { unsigned cond) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getCompareOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getCompareOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(cond, getSubOperands(1, operands, segments), a)}; return {getSubOperands(cond, getSubOperands(1, operands, segments), a)};
} }
llvm::Optional<mlir::ValueRange> std::optional<mlir::ValueRange>
fir::SelectCaseOp::getCompareOperands(mlir::ValueRange operands, fir::SelectCaseOp::getCompareOperands(mlir::ValueRange operands,
unsigned cond) { unsigned cond) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getCompareOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getCompareOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(cond, getSubOperands(1, operands, segments), a)}; return {getSubOperands(cond, getSubOperands(1, operands, segments), a)};
} }
mlir::SuccessorOperands fir::SelectCaseOp::getSuccessorOperands(unsigned oper) { mlir::SuccessorOperands fir::SelectCaseOp::getSuccessorOperands(unsigned oper) {
return mlir::SuccessorOperands(::getMutableSuccessorOperands( return mlir::SuccessorOperands(::getMutableSuccessorOperands(
oper, getTargetArgsMutable(), getTargetOffsetAttr())); oper, getTargetArgsMutable(), getTargetOffsetAttr()));
} }
llvm::Optional<llvm::ArrayRef<mlir::Value>> std::optional<llvm::ArrayRef<mlir::Value>>
fir::SelectCaseOp::getSuccessorOperands(llvm::ArrayRef<mlir::Value> operands, fir::SelectCaseOp::getSuccessorOperands(llvm::ArrayRef<mlir::Value> operands,
unsigned oper) { unsigned oper) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(oper, getSubOperands(2, operands, segments), a)}; return {getSubOperands(oper, getSubOperands(2, operands, segments), a)};
} }
llvm::Optional<mlir::ValueRange> std::optional<mlir::ValueRange>
fir::SelectCaseOp::getSuccessorOperands(mlir::ValueRange operands, fir::SelectCaseOp::getSuccessorOperands(mlir::ValueRange operands,
unsigned oper) { unsigned oper) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(oper, getSubOperands(2, operands, segments), a)}; return {getSubOperands(oper, getSubOperands(2, operands, segments), a)};
} }
▲ Show 20 Lines • Show All 233 Lines • ▼ Show 20 Linesmlir::ParseResult fir::SelectRankOp::parse(mlir::OpAsmParser &parser,
return parseIntegralSwitchTerminator(parser, result, getCasesAttr(), return parseIntegralSwitchTerminator(parser, result, getCasesAttr(),
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
} }
void fir::SelectRankOp::print(mlir::OpAsmPrinter &p) { void fir::SelectRankOp::print(mlir::OpAsmPrinter &p) {
printIntegralSwitchTerminator(*this, p); printIntegralSwitchTerminator(*this, p);
} }
llvm::Optional<mlir::OperandRange> std::optional<mlir::OperandRange>
fir::SelectRankOp::getCompareOperands(unsigned) { fir::SelectRankOp::getCompareOperands(unsigned) {
return {}; return {};
} }
llvm::Optional<llvm::ArrayRef<mlir::Value>> std::optional<llvm::ArrayRef<mlir::Value>>
fir::SelectRankOp::getCompareOperands(llvm::ArrayRef<mlir::Value>, unsigned) { fir::SelectRankOp::getCompareOperands(llvm::ArrayRef<mlir::Value>, unsigned) {
return {}; return {};
} }
mlir::SuccessorOperands fir::SelectRankOp::getSuccessorOperands(unsigned oper) { mlir::SuccessorOperands fir::SelectRankOp::getSuccessorOperands(unsigned oper) {
return mlir::SuccessorOperands(::getMutableSuccessorOperands( return mlir::SuccessorOperands(::getMutableSuccessorOperands(
oper, getTargetArgsMutable(), getTargetOffsetAttr())); oper, getTargetArgsMutable(), getTargetOffsetAttr()));
} }
llvm::Optional<llvm::ArrayRef<mlir::Value>> std::optional<llvm::ArrayRef<mlir::Value>>
fir::SelectRankOp::getSuccessorOperands(llvm::ArrayRef<mlir::Value> operands, fir::SelectRankOp::getSuccessorOperands(llvm::ArrayRef<mlir::Value> operands,
unsigned oper) { unsigned oper) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(oper, getSubOperands(2, operands, segments), a)}; return {getSubOperands(oper, getSubOperands(2, operands, segments), a)};
} }
llvm::Optional<mlir::ValueRange> std::optional<mlir::ValueRange>
fir::SelectRankOp::getSuccessorOperands(mlir::ValueRange operands, fir::SelectRankOp::getSuccessorOperands(mlir::ValueRange operands,
unsigned oper) { unsigned oper) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(oper, getSubOperands(2, operands, segments), a)}; return {getSubOperands(oper, getSubOperands(2, operands, segments), a)};
} }
unsigned fir::SelectRankOp::targetOffsetSize() { unsigned fir::SelectRankOp::targetOffsetSize() {
return (*this) return (*this)
->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()) ->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr())
.size(); .size();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SelectTypeOp // SelectTypeOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
llvm::Optional<mlir::OperandRange> std::optional<mlir::OperandRange>
fir::SelectTypeOp::getCompareOperands(unsigned) { fir::SelectTypeOp::getCompareOperands(unsigned) {
return {}; return {};
} }
llvm::Optional<llvm::ArrayRef<mlir::Value>> std::optional<llvm::ArrayRef<mlir::Value>>
fir::SelectTypeOp::getCompareOperands(llvm::ArrayRef<mlir::Value>, unsigned) { fir::SelectTypeOp::getCompareOperands(llvm::ArrayRef<mlir::Value>, unsigned) {
return {}; return {};
} }
mlir::SuccessorOperands fir::SelectTypeOp::getSuccessorOperands(unsigned oper) { mlir::SuccessorOperands fir::SelectTypeOp::getSuccessorOperands(unsigned oper) {
return mlir::SuccessorOperands(::getMutableSuccessorOperands( return mlir::SuccessorOperands(::getMutableSuccessorOperands(
oper, getTargetArgsMutable(), getTargetOffsetAttr())); oper, getTargetArgsMutable(), getTargetOffsetAttr()));
} }
llvm::Optional<llvm::ArrayRef<mlir::Value>> std::optional<llvm::ArrayRef<mlir::Value>>
fir::SelectTypeOp::getSuccessorOperands(llvm::ArrayRef<mlir::Value> operands, fir::SelectTypeOp::getSuccessorOperands(llvm::ArrayRef<mlir::Value> operands,
unsigned oper) { unsigned oper) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(oper, getSubOperands(2, operands, segments), a)}; return {getSubOperands(oper, getSubOperands(2, operands, segments), a)};
} }
llvm::Optional<mlir::ValueRange> std::optional<mlir::ValueRange>
fir::SelectTypeOp::getSuccessorOperands(mlir::ValueRange operands, fir::SelectTypeOp::getSuccessorOperands(mlir::ValueRange operands,
unsigned oper) { unsigned oper) {
auto a = auto a =
(*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr()); (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(getTargetOffsetAttr());
auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>( auto segments = (*this)->getAttrOfType<mlir::DenseI32ArrayAttr>(
getOperandSegmentSizeAttr()); getOperandSegmentSizeAttr());
return {getSubOperands(oper, getSubOperands(2, operands, segments), a)}; return {getSubOperands(oper, getSubOperands(2, operands, segments), a)};
} }
▲ Show 20 Lines • Show All 274 Lines • ▼ Show 20 LinesmkNamedIntegerAttr(mlir::OpBuilder &builder, llvm::StringRef name, int64_t v) {
assert(v > 0); assert(v > 0);
return builder.getNamedAttr( return builder.getNamedAttr(
name, builder.getIntegerAttr(builder.getIntegerType(64), v)); name, builder.getIntegerAttr(builder.getIntegerType(64), v));
} }
void fir::StringLitOp::build(mlir::OpBuilder &builder, void fir::StringLitOp::build(mlir::OpBuilder &builder,
mlir::OperationState &result, mlir::OperationState &result,
fir::CharacterType inType, llvm::StringRef val, fir::CharacterType inType, llvm::StringRef val,
llvm::Optional<int64_t> len) { std::optional<int64_t> len) {
auto valAttr = builder.getNamedAttr(value(), builder.getStringAttr(val)); auto valAttr = builder.getNamedAttr(value(), builder.getStringAttr(val));
int64_t length = len ? *len : inType.getLen(); int64_t length = len ? *len : inType.getLen();
auto lenAttr = mkNamedIntegerAttr(builder, size(), length); auto lenAttr = mkNamedIntegerAttr(builder, size(), length);
result.addAttributes({valAttr, lenAttr}); result.addAttributes({valAttr, lenAttr});
result.addTypes(inType); result.addTypes(inType);
} }
template <typename C> template <typename C>
static mlir::ArrayAttr convertToArrayAttr(mlir::OpBuilder &builder, static mlir::ArrayAttr convertToArrayAttr(mlir::OpBuilder &builder,
llvm::ArrayRef<C> xlist) { llvm::ArrayRef<C> xlist) {
llvm::SmallVector<mlir::Attribute> attrs; llvm::SmallVector<mlir::Attribute> attrs;
auto ty = builder.getIntegerType(8 * sizeof(C)); auto ty = builder.getIntegerType(8 * sizeof(C));
for (auto ch : xlist) for (auto ch : xlist)
attrs.push_back(builder.getIntegerAttr(ty, ch)); attrs.push_back(builder.getIntegerAttr(ty, ch));
return builder.getArrayAttr(attrs); return builder.getArrayAttr(attrs);
} }
void fir::StringLitOp::build(mlir::OpBuilder &builder, void fir::StringLitOp::build(mlir::OpBuilder &builder,
mlir::OperationState &result, mlir::OperationState &result,
fir::CharacterType inType, fir::CharacterType inType,
llvm::ArrayRef<char> vlist, llvm::ArrayRef<char> vlist,
llvm::Optional<std::int64_t> len) { std::optional<std::int64_t> len) {
auto valAttr = auto valAttr =
builder.getNamedAttr(xlist(), convertToArrayAttr(builder, vlist)); builder.getNamedAttr(xlist(), convertToArrayAttr(builder, vlist));
std::int64_t length = len ? *len : inType.getLen(); std::int64_t length = len ? *len : inType.getLen();
auto lenAttr = mkNamedIntegerAttr(builder, size(), length); auto lenAttr = mkNamedIntegerAttr(builder, size(), length);
result.addAttributes({valAttr, lenAttr}); result.addAttributes({valAttr, lenAttr});
result.addTypes(inType); result.addTypes(inType);
} }
void fir::StringLitOp::build(mlir::OpBuilder &builder, void fir::StringLitOp::build(mlir::OpBuilder &builder,
mlir::OperationState &result, mlir::OperationState &result,
fir::CharacterType inType, fir::CharacterType inType,
llvm::ArrayRef<char16_t> vlist, llvm::ArrayRef<char16_t> vlist,
llvm::Optional<std::int64_t> len) { std::optional<std::int64_t> len) {
auto valAttr = auto valAttr =
builder.getNamedAttr(xlist(), convertToArrayAttr(builder, vlist)); builder.getNamedAttr(xlist(), convertToArrayAttr(builder, vlist));
std::int64_t length = len ? *len : inType.getLen(); std::int64_t length = len ? *len : inType.getLen();
auto lenAttr = mkNamedIntegerAttr(builder, size(), length); auto lenAttr = mkNamedIntegerAttr(builder, size(), length);
result.addAttributes({valAttr, lenAttr}); result.addAttributes({valAttr, lenAttr});
result.addTypes(inType); result.addTypes(inType);
} }
void fir::StringLitOp::build(mlir::OpBuilder &builder, void fir::StringLitOp::build(mlir::OpBuilder &builder,
mlir::OperationState &result, mlir::OperationState &result,
fir::CharacterType inType, fir::CharacterType inType,
llvm::ArrayRef<char32_t> vlist, llvm::ArrayRef<char32_t> vlist,
llvm::Optional<std::int64_t> len) { std::optional<std::int64_t> len) {
auto valAttr = auto valAttr =
builder.getNamedAttr(xlist(), convertToArrayAttr(builder, vlist)); builder.getNamedAttr(xlist(), convertToArrayAttr(builder, vlist));
std::int64_t length = len ? *len : inType.getLen(); std::int64_t length = len ? *len : inType.getLen();
auto lenAttr = mkNamedIntegerAttr(builder, size(), length); auto lenAttr = mkNamedIntegerAttr(builder, size(), length);
result.addAttributes({valAttr, lenAttr}); result.addAttributes({valAttr, lenAttr});
result.addTypes(inType); result.addTypes(inType);
} }
▲ Show 20 Lines • Show All 340 Lines • Show Last 20 Lines

flang/lib/Optimizer/Dialect/FortranVariableInterface.cpp

Show First 20 Lines • Show All 51 Lines • ▼ Show 20 Lines if (mlir::Value shape = getShape()) {
shapeRank = shapeShiftType.getRank(); shapeRank = shapeShiftType.getRank();
} else { } else {
if (!sourceIsBoxValue) if (!sourceIsBoxValue)
emitOpError("of array entity with a raw address base must have a " emitOpError("of array entity with a raw address base must have a "
"shape operand that is a shape or shapeshift"); "shape operand that is a shape or shapeshift");
shapeRank = shape.getType().cast<fir::ShiftType>().getRank(); shapeRank = shape.getType().cast<fir::ShiftType>().getRank();
} }
llvm::Optional<unsigned> rank = getRank(); std::optional<unsigned> rank = getRank();
if (!rank || *rank != shapeRank) if (!rank || *rank != shapeRank)
return emitOpError("has conflicting shape and base operand ranks"); return emitOpError("has conflicting shape and base operand ranks");
} else if (!sourceIsBox) { } else if (!sourceIsBox) {
emitOpError("of array entity with a raw address base must have a shape " emitOpError("of array entity with a raw address base must have a shape "
"operand that is a shape or shapeshift"); "operand that is a shape or shapeshift");
} }
} }
return mlir::success(); return mlir::success();
} }

flang/lib/Optimizer/HLFIR/IR/HLFIROps.cpp

Show First 20 Lines • Show All 81 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DesignateOp // DesignateOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void hlfir::DesignateOp::build( void hlfir::DesignateOp::build(
mlir::OpBuilder &builder, mlir::OperationState &result, mlir::OpBuilder &builder, mlir::OperationState &result,
mlir::Type result_type, mlir::Value memref, llvm::StringRef component, mlir::Type result_type, mlir::Value memref, llvm::StringRef component,
mlir::Value component_shape, llvm::ArrayRef<Subscript> subscripts, mlir::Value component_shape, llvm::ArrayRef<Subscript> subscripts,
mlir::ValueRange substring, llvm::Optional<bool> complex_part, mlir::ValueRange substring, std::optional<bool> complex_part,
mlir::Value shape, mlir::ValueRange typeparams, mlir::Value shape, mlir::ValueRange typeparams,
fir::FortranVariableFlagsAttr fortran_attrs) { fir::FortranVariableFlagsAttr fortran_attrs) {
auto componentAttr = auto componentAttr =
component.empty() ? mlir::StringAttr{} : builder.getStringAttr(component); component.empty() ? mlir::StringAttr{} : builder.getStringAttr(component);
llvm::SmallVector<mlir::Value> indices; llvm::SmallVector<mlir::Value> indices;
llvm::SmallVector<bool> isTriplet; llvm::SmallVector<bool> isTriplet;
for (auto subscript : subscripts) { for (auto subscript : subscripts) {
if (auto *triplet = std::get_if<Triplet>(&subscript)) { if (auto *triplet = std::get_if<Triplet>(&subscript)) {
▲ Show 20 Lines • Show All 390 Lines • Show Last 20 Lines

flang/lib/Optimizer/Transforms/ControlFlowConverter.cpp

Show First 20 Lines • Show All 391 Lines • ▼ Show 20 Lines matchAndRewrite(fir::SelectTypeOp selectType, OpAdaptor adaptor,
} }
orderedTypeGuards.append(orderedClassIsGuards); orderedTypeGuards.append(orderedClassIsGuards);
orderedTypeGuards.push_back(defaultGuard); orderedTypeGuards.push_back(defaultGuard);
assert(orderedTypeGuards.size() == typeGuardNum && assert(orderedTypeGuards.size() == typeGuardNum &&
"ordered type guard size doesn't match number of type guards"); "ordered type guard size doesn't match number of type guards");
for (unsigned idx : orderedTypeGuards) { for (unsigned idx : orderedTypeGuards) {
auto *dest = selectType.getSuccessor(idx); auto *dest = selectType.getSuccessor(idx);
llvm::Optional<mlir::ValueRange> destOps = std::optional<mlir::ValueRange> destOps =
selectType.getSuccessorOperands(operands, idx); selectType.getSuccessorOperands(operands, idx);
if (typeGuards[idx].dyn_cast<mlir::UnitAttr>()) if (typeGuards[idx].dyn_cast<mlir::UnitAttr>())
rewriter.replaceOpWithNewOp<mlir::cf::BranchOp>(selectType, dest); rewriter.replaceOpWithNewOp<mlir::cf::BranchOp>(selectType, dest);
else if (mlir::failed(genTypeLadderStep(loc, selector, typeGuards[idx], else if (mlir::failed(genTypeLadderStep(loc, selector, typeGuards[idx],
dest, destOps, mod, rewriter, dest, destOps, mod, rewriter,
kindMap))) kindMap)))
return mlir::failure(); return mlir::failure();
} }
▲ Show 20 Lines • Show All 56 Lines • ▼ Show 20 Lines if (fir::isa_complex(ty)) {
kindMap.getRealBitsize(cmplxTy.getFKind())); kindMap.getRealBitsize(cmplxTy.getFKind()));
} }
if (auto charTy = ty.dyn_cast<fir::CharacterType>()) if (auto charTy = ty.dyn_cast<fir::CharacterType>())
return fir::characterBitsToTypeCode( return fir::characterBitsToTypeCode(
kindMap.getCharacterBitsize(charTy.getFKind())); kindMap.getCharacterBitsize(charTy.getFKind()));
return 0; return 0;
} }
mlir::LogicalResult mlir::LogicalResult genTypeLadderStep(mlir::Location loc,
genTypeLadderStep(mlir::Location loc, mlir::Value selector, mlir::Value selector,
mlir::Attribute attr, mlir::Block *dest, mlir::Attribute attr, mlir::Block *dest,
llvm::Optional<mlir::ValueRange> destOps, std::optional<mlir::ValueRange> destOps,
mlir::ModuleOp mod, mlir::PatternRewriter &rewriter, mlir::ModuleOp mod,
mlir::PatternRewriter &rewriter,
fir::KindMapping &kindMap) const { fir::KindMapping &kindMap) const {
mlir::Value cmp; mlir::Value cmp;
// TYPE IS type guard comparison are all done inlined. // TYPE IS type guard comparison are all done inlined.
if (auto a = attr.dyn_cast<fir::ExactTypeAttr>()) { if (auto a = attr.dyn_cast<fir::ExactTypeAttr>()) {
if (fir::isa_trivial(a.getType()) || if (fir::isa_trivial(a.getType()) ||
a.getType().isa<fir::CharacterType>()) { a.getType().isa<fir::CharacterType>()) {
// For type guard statement with Intrinsic type spec the type code of // For type guard statement with Intrinsic type spec the type code of
// the descriptor is compared. // the descriptor is compared.
int code = getTypeCode(a.getType(), kindMap); int code = getTypeCode(a.getType(), kindMap);
▲ Show 20 Lines • Show All 115 Lines • Show Last 20 Lines

flang/lib/Optimizer/Transforms/MemoryAllocation.cpp

Show First 20 Lines • Show All 110 Lines • ▼ Show 20 Lines AllocaOpConversion(mlir::MLIRContext *ctx,
: OpRewritePattern(ctx), returnOps(rets) {} : OpRewritePattern(ctx), returnOps(rets) {}
mlir::LogicalResult mlir::LogicalResult
matchAndRewrite(fir::AllocaOp alloca, matchAndRewrite(fir::AllocaOp alloca,
mlir::PatternRewriter &rewriter) const override { mlir::PatternRewriter &rewriter) const override {
auto loc = alloca.getLoc(); auto loc = alloca.getLoc();
mlir::Type varTy = alloca.getInType(); mlir::Type varTy = alloca.getInType();
auto unpackName = auto unpackName =
[](llvm::Optional<llvm::StringRef> opt) -> llvm::StringRef { [](std::optional<llvm::StringRef> opt) -> llvm::StringRef {
if (opt) if (opt)
return *opt; return *opt;
return {}; return {};
}; };
auto uniqName = unpackName(alloca.getUniqName()); auto uniqName = unpackName(alloca.getUniqName());
auto bindcName = unpackName(alloca.getBindcName()); auto bindcName = unpackName(alloca.getBindcName());
auto heap = rewriter.create<fir::AllocMemOp>( auto heap = rewriter.create<fir::AllocMemOp>(
loc, varTy, uniqName, bindcName, alloca.getTypeparams(), loc, varTy, uniqName, bindcName, alloca.getTypeparams(),
▲ Show 20 Lines • Show All 97 Lines • Show Last 20 Lines

mlir/docs/DefiningDialects/Operations.md

Show First 20 Lines • Show All 1,335 Lines • ▼ Show 20 Lines
namespace Outer { namespace Outer {
namespace Inner { namespace Inner {
// An example int enum // An example int enum
enum class MyIntEnum : uint32_t { enum class MyIntEnum : uint32_t {
Case15 = 15, Case15 = 15,
Case20 = 20, Case20 = 20,
}; };
llvm::Optional<MyIntEnum> symbolizeMyIntEnum(uint32_t); std::optional<MyIntEnum> symbolizeMyIntEnum(uint32_t);
llvm::StringRef ConvertToString(MyIntEnum); llvm::StringRef ConvertToString(MyIntEnum);
llvm::Optional<MyIntEnum> ConvertToEnum(llvm::StringRef); std::optional<MyIntEnum> ConvertToEnum(llvm::StringRef);
inline constexpr unsigned getMaxEnumValForMyIntEnum() { inline constexpr unsigned getMaxEnumValForMyIntEnum() {
return 20; return 20;
} }
} // namespace Inner } // namespace Inner
} // namespace Outer } // namespace Outer
namespace llvm { namespace llvm {
Show All 27 Lines
llvm::StringRef ConvertToString(MyIntEnum val) { llvm::StringRef ConvertToString(MyIntEnum val) {
switch (val) { switch (val) {
case MyIntEnum::Case15: return "Case15"; case MyIntEnum::Case15: return "Case15";
case MyIntEnum::Case20: return "Case20"; case MyIntEnum::Case20: return "Case20";
} }
return ""; return "";
} }
llvm::Optional<MyIntEnum> ConvertToEnum(llvm::StringRef str) { std::optional<MyIntEnum> ConvertToEnum(llvm::StringRef str) {
return llvm::StringSwitch<llvm::Optional<MyIntEnum>>(str) return llvm::StringSwitch<std::optional<MyIntEnum>>(str)
.Case("Case15", MyIntEnum::Case15) .Case("Case15", MyIntEnum::Case15)
.Case("Case20", MyIntEnum::Case20) .Case("Case20", MyIntEnum::Case20)
.Default(std::nullopt); .Default(std::nullopt);
} }
llvm::Optional<MyIntEnum> symbolizeMyIntEnum(uint32_t value) { std::optional<MyIntEnum> symbolizeMyIntEnum(uint32_t value) {
switch (value) { switch (value) {
case 15: return MyIntEnum::Case15; case 15: return MyIntEnum::Case15;
case 20: return MyIntEnum::Case20; case 20: return MyIntEnum::Case20;
default: return std::nullopt; default: return std::nullopt;
} }
} }
} // namespace Inner } // namespace Inner
Show All 20 Lines
enum class MyBitEnum : uint32_t { enum class MyBitEnum : uint32_t {
None = 0, None = 0,
Bit0 = 1, Bit0 = 1,
Bit1 = 2, Bit1 = 2,
Bit2 = 4, Bit2 = 4,
Bit3 = 8, Bit3 = 8,
}; };
llvm::Optional<MyBitEnum> symbolizeMyBitEnum(uint32_t); std::optional<MyBitEnum> symbolizeMyBitEnum(uint32_t);
std::string stringifyMyBitEnum(MyBitEnum); std::string stringifyMyBitEnum(MyBitEnum);
llvm::Optional<MyBitEnum> symbolizeMyBitEnum(llvm::StringRef); std::optional<MyBitEnum> symbolizeMyBitEnum(llvm::StringRef);
inline constexpr MyBitEnum operator|(MyBitEnum a, MyBitEnum b) { inline constexpr MyBitEnum operator|(MyBitEnum a, MyBitEnum b) {
return static_cast<MyBitEnum>(static_cast<uint32_t>(a) | static_cast<uint32_t>(b)); return static_cast<MyBitEnum>(static_cast<uint32_t>(a) | static_cast<uint32_t>(b));
} }
inline constexpr MyBitEnum operator&(MyBitEnum a, MyBitEnum b) { inline constexpr MyBitEnum operator&(MyBitEnum a, MyBitEnum b) {
return static_cast<MyBitEnum>(static_cast<uint32_t>(a) & static_cast<uint32_t>(b)); return static_cast<MyBitEnum>(static_cast<uint32_t>(a) & static_cast<uint32_t>(b));
} }
inline constexpr MyBitEnum operator^(MyBitEnum a, MyBitEnum b) { inline constexpr MyBitEnum operator^(MyBitEnum a, MyBitEnum b) {
Show All 13 Linesinline constexpr MyBitEnum bitEnumClear(MyBitEnum bits, MyBitEnum bit) {
return bits & ~bit; return bits & ~bit;
} }
inline std::string stringifyEnum(MyBitEnum enumValue) { inline std::string stringifyEnum(MyBitEnum enumValue) {
return stringifyMyBitEnum(enumValue); return stringifyMyBitEnum(enumValue);
} }
template <typename EnumType> template <typename EnumType>
::llvm::Optional<EnumType> symbolizeEnum(::llvm::StringRef); ::std::optional<EnumType> symbolizeEnum(::llvm::StringRef);
template <> template <>
inline ::llvm::Optional<MyBitEnum> symbolizeEnum<MyBitEnum>(::llvm::StringRef str) { inline ::std::optional<MyBitEnum> symbolizeEnum<MyBitEnum>(::llvm::StringRef str) {
return symbolizeMyBitEnum(str); return symbolizeMyBitEnum(str);
} }
namespace llvm { namespace llvm {
template<> struct DenseMapInfo<::MyBitEnum> { template<> struct DenseMapInfo<::MyBitEnum> {
using StorageInfo = llvm::DenseMapInfo<uint32_t>; using StorageInfo = llvm::DenseMapInfo<uint32_t>;
static inline ::MyBitEnum getEmptyKey() { static inline ::MyBitEnum getEmptyKey() {
Show All 24 Linesstd::string stringifyMyBitEnum(MyBitEnum symbol) {
if (1u == (1u & val)) { strs.push_back("tagged"); } if (1u == (1u & val)) { strs.push_back("tagged"); }
if (2u == (2u & val)) { strs.push_back("Bit1"); } if (2u == (2u & val)) { strs.push_back("Bit1"); }
if (4u == (4u & val)) { strs.push_back("Bit2"); } if (4u == (4u & val)) { strs.push_back("Bit2"); }
if (8u == (8u & val)) { strs.push_back("Bit3"); } if (8u == (8u & val)) { strs.push_back("Bit3"); }
return llvm::join(strs, "|"); return llvm::join(strs, "|");
} }
llvm::Optional<MyBitEnum> symbolizeMyBitEnum(llvm::StringRef str) { std::optional<MyBitEnum> symbolizeMyBitEnum(llvm::StringRef str) {
// Special case for all bits unset. // Special case for all bits unset.
if (str == "None") return MyBitEnum::None; if (str == "None") return MyBitEnum::None;
llvm::SmallVector<llvm::StringRef, 2> symbols; llvm::SmallVector<llvm::StringRef, 2> symbols;
str.split(symbols, "|"); str.split(symbols, "|");
uint32_t val = 0; uint32_t val = 0;
for (auto symbol : symbols) { for (auto symbol : symbols) {
auto bit = llvm::StringSwitch<llvm::Optional<uint32_t>>(symbol) auto bit = llvm::StringSwitch<std::optional<uint32_t>>(symbol)
.Case("tagged", 1) .Case("tagged", 1)
.Case("Bit1", 2) .Case("Bit1", 2)
.Case("Bit2", 4) .Case("Bit2", 4)
.Case("Bit3", 8) .Case("Bit3", 8)
.Default(std::nullopt); .Default(std::nullopt);
if (bit) { val |= *bit; } else { return std::nullopt; } if (bit) { val |= *bit; } else { return std::nullopt; }
} }
return static_cast<MyBitEnum>(val); return static_cast<MyBitEnum>(val);
} }
llvm::Optional<MyBitEnum> symbolizeMyBitEnum(uint32_t value) { std::optional<MyBitEnum> symbolizeMyBitEnum(uint32_t value) {
// Special case for all bits unset. // Special case for all bits unset.
if (value == 0) return MyBitEnum::None; if (value == 0) return MyBitEnum::None;
if (value & ~static_cast<uint32_t>(15u)) return std::nullopt; if (value & ~static_cast<uint32_t>(15u)) return std::nullopt;
return static_cast<MyBitEnum>(value); return static_cast<MyBitEnum>(value);
} }
``` ```
▲ Show 20 Lines • Show All 116 Lines • Show Last 20 Lines

mlir/docs/Dialects/Linalg/_index.md

Show First 20 Lines • Show All 625 Lines • ▼ Show 20 Linesdef batchmatmulOp : LinalgNamedStructured_Op<"batchmatmul", [
NInputs<2>, NInputs<2>,
NOutputs<1>, NOutputs<1>,
NamedStructuredOpTrait]> { ... } NamedStructuredOpTrait]> { ... }
``` ```
When `mlir-linalg-ods-gen -gen-impl=1` is called, the following C++ is produced: When `mlir-linalg-ods-gen -gen-impl=1` is called, the following C++ is produced:
``` ```
llvm::Optional<SmallVector<StringRef, 8>> batchmatmul::referenceIterators() { std::optional<SmallVector<StringRef, 8>> batchmatmul::referenceIterators() {
return SmallVector<StringRef, 8>{ return SmallVector<StringRef, 8>{
getParallelIteratorTypeName(), getParallelIteratorTypeName(),
getParallelIteratorTypeName(), getParallelIteratorTypeName(),
getParallelIteratorTypeName(), getParallelIteratorTypeName(),
getReductionIteratorTypeName() }; getReductionIteratorTypeName() };
} }
llvm::Optional<SmallVector<AffineMap, 8>> batchmatmul::referenceIndexingMaps() { std::optional<SmallVector<AffineMap, 8>> batchmatmul::referenceIndexingMaps() {
MLIRContext *context = getContext(); MLIRContext *context = getContext();
AffineExpr d0, d1, d2, d3; AffineExpr d0, d1, d2, d3;
bindDims(context, d0, d1, d2, d3); bindDims(context, d0, d1, d2, d3);
return SmallVector<AffineMap, 8>{ return SmallVector<AffineMap, 8>{
AffineMap::get(4, 0, {d0, d1, d3}), AffineMap::get(4, 0, {d0, d1, d3}),
AffineMap::get(4, 0, {d3, d2}), AffineMap::get(4, 0, {d3, d2}),
AffineMap::get(4, 0, {d0, d1, d2}) }; AffineMap::get(4, 0, {d0, d1, d2}) };
} }
▲ Show 20 Lines • Show All 48 Lines • Show Last 20 Lines

mlir/include/mlir/Analysis/DataFlow/SparseAnalysis.h

Show First 20 Lines • Show All 237 Lines • ▼ Show 20 Linesprivate:
/// determined from block predecessors. /// determined from block predecessors.
void visitBlock(Block *block); void visitBlock(Block *block);
/// Visit a program point `point` with predecessors within a region branch /// Visit a program point `point` with predecessors within a region branch
/// operation `branch`, which can either be the entry block of one of the /// operation `branch`, which can either be the entry block of one of the
/// regions or the parent operation itself, and set either the argument or /// regions or the parent operation itself, and set either the argument or
/// parent result lattices. /// parent result lattices.
void visitRegionSuccessors(ProgramPoint point, RegionBranchOpInterface branch, void visitRegionSuccessors(ProgramPoint point, RegionBranchOpInterface branch,
Optional<unsigned> successorIndex, std::optional<unsigned> successorIndex,
ArrayRef<AbstractSparseLattice *> lattices); ArrayRef<AbstractSparseLattice *> lattices);
}; };
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SparseDataFlowAnalysis // SparseDataFlowAnalysis
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// A sparse (forward) data-flow analysis for propagating SSA value lattices /// A sparse (forward) data-flow analysis for propagating SSA value lattices
▲ Show 20 Lines • Show All 215 Lines • Show Last 20 Lines

mlir/include/mlir/Conversion/MemRefToLLVM/AllocLikeConversion.h

Show First 20 Lines • Show All 55 Lines • ▼ Show 20 Linesprotected:
/// the memory allocation op. /// the memory allocation op.
/// ///
/// Aligned_alloc requires the allocation size to be a power of two, and the /// Aligned_alloc requires the allocation size to be a power of two, and the
/// allocation size to be a multiple of the alignment. /// allocation size to be a multiple of the alignment.
template <typename OpType> template <typename OpType>
int64_t alignedAllocationGetAlignment(ConversionPatternRewriter &rewriter, int64_t alignedAllocationGetAlignment(ConversionPatternRewriter &rewriter,
Location loc, OpType op, Location loc, OpType op,
const DataLayout *defaultLayout) const { const DataLayout *defaultLayout) const {
if (Optional<uint64_t> alignment = op.getAlignment()) if (std::optional<uint64_t> alignment = op.getAlignment())
return *alignment; return *alignment;
// Whenever we don't have alignment set, we will use an alignment // Whenever we don't have alignment set, we will use an alignment
// consistent with the element type; since the allocation size has to be a // consistent with the element type; since the allocation size has to be a
// power of two, we will bump to the next power of two if it isn't. // power of two, we will bump to the next power of two if it isn't.
unsigned eltSizeBytes = unsigned eltSizeBytes =
getMemRefEltSizeInBytes(op.getType(), op, defaultLayout); getMemRefEltSizeInBytes(op.getType(), op, defaultLayout);
return std::max(kMinAlignedAllocAlignment, return std::max(kMinAlignedAllocAlignment,
▲ Show 20 Lines • Show All 68 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Affine/IR/AffineOps.td

Show First 20 Lines • Show All 702 Lines • ▼ Show 20 Lines OpBuilder<(ins "TypeRange":$resultTypes,
"ArrayRef<int64_t>":$steps)> "ArrayRef<int64_t>":$steps)>
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Get the number of dimensions. /// Get the number of dimensions.
unsigned getNumDims(); unsigned getNumDims();
/// Get ranges as constants, may fail in dynamic case. /// Get ranges as constants, may fail in dynamic case.
Optional<SmallVector<int64_t, 8>> getConstantRanges(); std::optional<SmallVector<int64_t, 8>> getConstantRanges();
Block *getBody(); Block *getBody();
OpBuilder getBodyBuilder(); OpBuilder getBodyBuilder();
MutableArrayRef<BlockArgument> getIVs() { MutableArrayRef<BlockArgument> getIVs() {
return getBody()->getArguments(); return getBody()->getArguments();
} }
/// Returns elements of the loop lower bound. /// Returns elements of the loop lower bound.
▲ Show 20 Lines • Show All 390 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Arith/IR/ArithOps.td

Show First 20 Lines • Show All 262 Lines • ▼ Show 20 Lines OpBuilder<(ins "Value":$lhs, "Value":$rhs), [{
lhs, rhs); lhs, rhs);
}]> }]>
]; ];
let hasFolder = 1; let hasFolder = 1;
let hasCanonicalizer = 1; let hasCanonicalizer = 1;
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Optional<SmallVector<int64_t, 4>> getShapeForUnroll(); std::optional<SmallVector<int64_t, 4>> getShapeForUnroll();
}]; }];
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SubIOp // SubIOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
def Arith_SubIOp : Arith_TotalIntBinaryOp<"subi"> { def Arith_SubIOp : Arith_TotalIntBinaryOp<"subi"> {
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Linesdef Arith_MulSIExtendedOp : Arith_Op<"mulsi_extended", [Pure, Commutative,
let results = (outs SignlessIntegerLike:$low, SignlessIntegerLike:$high); let results = (outs SignlessIntegerLike:$low, SignlessIntegerLike:$high);
let assemblyFormat = "$lhs `,` $rhs attr-dict `:` type($lhs)"; let assemblyFormat = "$lhs `,` $rhs attr-dict `:` type($lhs)";
let hasFolder = 1; let hasFolder = 1;
let hasCanonicalizer = 1; let hasCanonicalizer = 1;
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Optional<SmallVector<int64_t, 4>> getShapeForUnroll(); std::optional<SmallVector<int64_t, 4>> getShapeForUnroll();
}]; }];
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// MulUIExtendedOp // MulUIExtendedOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
def Arith_MulUIExtendedOp : Arith_Op<"mului_extended", [Pure, Commutative, def Arith_MulUIExtendedOp : Arith_Op<"mului_extended", [Pure, Commutative,
Show All 26 Linesdef Arith_MulUIExtendedOp : Arith_Op<"mului_extended", [Pure, Commutative,
let results = (outs SignlessIntegerLike:$low, SignlessIntegerLike:$high); let results = (outs SignlessIntegerLike:$low, SignlessIntegerLike:$high);
let assemblyFormat = "$lhs `,` $rhs attr-dict `:` type($lhs)"; let assemblyFormat = "$lhs `,` $rhs attr-dict `:` type($lhs)";
let hasFolder = 1; let hasFolder = 1;
let hasCanonicalizer = 1; let hasCanonicalizer = 1;
let extraClassDeclaration = [{ let extraClassDeclaration = [{
Optional<SmallVector<int64_t, 4>> getShapeForUnroll(); std::optional<SmallVector<int64_t, 4>> getShapeForUnroll();
}]; }];
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DivUIOp // DivUIOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
def Arith_DivUIOp : Arith_IntBinaryOp<"divui", [ConditionallySpeculatable]> { def Arith_DivUIOp : Arith_IntBinaryOp<"divui", [ConditionallySpeculatable]> {
▲ Show 20 Lines • Show All 1,041 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/GPU/TransformOps/GPUTransformOps.h

Show All 37 Lines
/// `scf.foreach_thread` are rewritten to gpu.thread_id according to the /// `scf.foreach_thread` are rewritten to gpu.thread_id according to the
/// thread_dim_apping attribute. Sibling `scf.foreach_thread` are supported in /// thread_dim_apping attribute. Sibling `scf.foreach_thread` are supported in
/// which case, the union of the number of threads is computed and may result in /// which case, the union of the number of threads is computed and may result in
/// predication. Dynamic, `scf.foreach_thread` trip counts are currently not /// predication. Dynamic, `scf.foreach_thread` trip counts are currently not
/// supported. Dynamic block dim sizes are currently not supported. /// supported. Dynamic block dim sizes are currently not supported.
DiagnosedSilenceableFailure mapNestedForeachToThreadsImpl( DiagnosedSilenceableFailure mapNestedForeachToThreadsImpl(
RewriterBase &rewriter, Operation *target, RewriterBase &rewriter, Operation *target,
const SmallVectorImpl<int64_t> &blockDim, bool syncAfterDistribute, const SmallVectorImpl<int64_t> &blockDim, bool syncAfterDistribute,
llvm::Optional<TransformOpInterface> transformOp, std::optional<TransformOpInterface> transformOp,
const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes); const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes);
/// Maps the top level `scf.foreach_thread` op to GPU Thread Blocks. Mapping is /// Maps the top level `scf.foreach_thread` op to GPU Thread Blocks. Mapping is
/// one-to-one and the induction variables of `scf.foreach_thread` are rewritten /// one-to-one and the induction variables of `scf.foreach_thread` are rewritten
/// to gpu.block_id according to the thread_dim_apping attribute. Dynamic, /// to gpu.block_id according to the thread_dim_apping attribute. Dynamic,
/// `scf.foreach_thread` trip counts are currently not supported. Dynamic block /// `scf.foreach_thread` trip counts are currently not supported. Dynamic block
/// dim sizes are currently not supported. /// dim sizes are currently not supported.
DiagnosedSilenceableFailure mapForeachToBlocksImpl( DiagnosedSilenceableFailure mapForeachToBlocksImpl(
Show All 22 Lines

mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td

Show First 20 Lines • Show All 984 Lines • ▼ Show 20 Lines let arguments = (ins NVVM_MMAShapeAttr:$shape,
OptionalAttr<MMATypesAttr>:$multiplicandBPtxType, OptionalAttr<MMATypesAttr>:$multiplicandBPtxType,
Variadic<LLVM_Type>:$operandA, Variadic<LLVM_Type>:$operandA,
Variadic<LLVM_Type>:$operandB, Variadic<LLVM_Type>:$operandB,
Variadic<LLVM_Type>:$operandC); Variadic<LLVM_Type>:$operandC);
let extraClassDeclaration = !strconcat([{ let extraClassDeclaration = !strconcat([{
static llvm::Intrinsic::ID getIntrinsicID( static llvm::Intrinsic::ID getIntrinsicID(
int64_t m, int64_t n, uint64_t k, int64_t m, int64_t n, uint64_t k,
llvm::Optional<MMAB1Op> b1Op, std::optional<MMAB1Op> b1Op,
llvm::Optional<MMAIntOverflow> sat, std::optional<MMAIntOverflow> sat,
mlir::NVVM::MMALayout layoutAEnum, mlir::NVVM::MMALayout layoutBEnum, mlir::NVVM::MMALayout layoutAEnum, mlir::NVVM::MMALayout layoutBEnum,
mlir::NVVM::MMATypes eltypeAEnum, mlir::NVVM::MMATypes eltypeBEnum, mlir::NVVM::MMATypes eltypeAEnum, mlir::NVVM::MMATypes eltypeBEnum,
mlir::NVVM::MMATypes eltypeCEnum, mlir::NVVM::MMATypes eltypeDEnum) { mlir::NVVM::MMATypes eltypeCEnum, mlir::NVVM::MMATypes eltypeDEnum) {
llvm::StringRef layoutA = stringifyEnum(layoutAEnum); llvm::StringRef layoutA = stringifyEnum(layoutAEnum);
llvm::StringRef layoutB = stringifyEnum(layoutBEnum); llvm::StringRef layoutB = stringifyEnum(layoutBEnum);
llvm::StringRef eltypeA = stringifyEnum(eltypeAEnum); llvm::StringRef eltypeA = stringifyEnum(eltypeAEnum);
llvm::StringRef eltypeB = stringifyEnum(eltypeBEnum); llvm::StringRef eltypeB = stringifyEnum(eltypeBEnum);
llvm::StringRef eltypeC = stringifyEnum(eltypeCEnum); llvm::StringRef eltypeC = stringifyEnum(eltypeCEnum);
llvm::StringRef eltypeD = stringifyEnum(eltypeDEnum); llvm::StringRef eltypeD = stringifyEnum(eltypeDEnum);
}], }],
MMA_SYNC_INTR<>.id, [{ MMA_SYNC_INTR<>.id, [{
return 0; return 0;
} }
static Optional<mlir::NVVM::MMATypes> inferOperandMMAType(Type operandElType, static std::optional<mlir::NVVM::MMATypes> inferOperandMMAType(Type operandElType,
bool isAccumulator); bool isAccumulator);
MMATypes accumPtxType(); MMATypes accumPtxType();
MMATypes resultPtxType(); MMATypes resultPtxType();
}]); }]);
let builders = [ let builders = [
OpBuilder<(ins "Type":$resultType, "ValueRange":$operandA, OpBuilder<(ins "Type":$resultType, "ValueRange":$operandA,
"ValueRange":$operandB, "ValueRange":$operandC, "ValueRange":$operandB, "ValueRange":$operandC,
"ArrayRef<int64_t>":$shape, "Optional<MMAB1Op>":$b1Op, "ArrayRef<int64_t>":$shape, "std::optional<MMAB1Op>":$b1Op,
"Optional<MMAIntOverflow>":$intOverflow, "std::optional<MMAIntOverflow>":$intOverflow,
"Optional<std::array<MMATypes, 2>>":$multiplicandPtxTypes, "std::optional<std::array<MMATypes, 2>>":$multiplicandPtxTypes,
"Optional<std::array<MMALayout, 2>>":$multiplicandLayouts)> "std::optional<std::array<MMALayout, 2>>":$multiplicandLayouts)>
]; ];
string llvmBuilder = [{ string llvmBuilder = [{
auto operands = moduleTranslation.lookupValues(opInst.getOperands()); auto operands = moduleTranslation.lookupValues(opInst.getOperands());
auto intId = mlir::NVVM::MmaOp::getIntrinsicID( auto intId = mlir::NVVM::MmaOp::getIntrinsicID(
$shape.getM(), $shape.getN(), $shape.getK(), $shape.getM(), $shape.getN(), $shape.getK(),
$b1Op, $intOverflowBehavior, $b1Op, $intOverflowBehavior,
$layoutA, $layoutB, $layoutA, $layoutB,
Show All 14 Lines

mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td

Show First 20 Lines • Show All 41 Lines • ▼ Show 20 Lines code structuredOpsBaseDecls = [{
LogicalResult reifyResultShapes(OpBuilder &b, LogicalResult reifyResultShapes(OpBuilder &b,
ReifiedRankedShapedTypeDims &reifiedReturnShapes) { ReifiedRankedShapedTypeDims &reifiedReturnShapes) {
return llvm::cast<LinalgOp>(getOperation()).reifyResultShapes(b, return llvm::cast<LinalgOp>(getOperation()).reifyResultShapes(b,
reifiedReturnShapes); reifiedReturnShapes);
} }
void getSuccessorRegions( void getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// Op has a region, but conceptually the control flow does not enter the // Op has a region, but conceptually the control flow does not enter the
// region. // region.
} }
}]; }];
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 471 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Linalg/TransformOps/LinalgTransformOps.h

Show First 20 Lines • Show All 42 Lines • ▼ Show 20 Lines
namespace transform { namespace transform {
/// Implementation of tiling operations using `scf.foreach_thread`. /// Implementation of tiling operations using `scf.foreach_thread`.
DiagnosedSilenceableFailure tileToForeachThreadOpImpl( DiagnosedSilenceableFailure tileToForeachThreadOpImpl(
RewriterBase &rewriter, transform::TransformState &state, RewriterBase &rewriter, transform::TransformState &state,
TransformOpInterface transformOp, ArrayRef<Operation *> targets, TransformOpInterface transformOp, ArrayRef<Operation *> targets,
ArrayRef<OpFoldResult> mixedNumThreads, ArrayRef<OpFoldResult> mixedNumThreads,
ArrayRef<OpFoldResult> mixedTileSizes, Optional<ArrayAttr> mapping, ArrayRef<OpFoldResult> mixedTileSizes, std::optional<ArrayAttr> mapping,
SmallVector<Operation *> &tileOps, SmallVector<Operation *> &tiledOps); SmallVector<Operation *> &tileOps, SmallVector<Operation *> &tiledOps);
} // namespace transform } // namespace transform
namespace linalg { namespace linalg {
void registerTransformDialectExtension(DialectRegistry &registry); void registerTransformDialectExtension(DialectRegistry &registry);
} // namespace linalg } // namespace linalg
} // namespace mlir } // namespace mlir
#endif // MLIR_DIALECT_LINALG_TRANSFORMOPS_LINALGTRANSFORMOPS_H #endif // MLIR_DIALECT_LINALG_TRANSFORMOPS_LINALGTRANSFORMOPS_H

mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h

Show First 20 Lines • Show All 238 Lines • ▼ Show 20 Lines
FailureOr<GenericOp> generalizeNamedOp(RewriterBase &rewriter, FailureOr<GenericOp> generalizeNamedOp(RewriterBase &rewriter,
LinalgOp namedOp); LinalgOp namedOp);
/// Callback function type used to perform the allocation for the promoted /// Callback function type used to perform the allocation for the promoted
/// `subView`. In `boundingSubViewsize` a best attempt is made to find the /// `subView`. In `boundingSubViewsize` a best attempt is made to find the
/// smallest constant value for the size of the buffer needed for each /// smallest constant value for the size of the buffer needed for each
/// dimension. If that is not possible, contains the dynamic size of the /// dimension. If that is not possible, contains the dynamic size of the
/// subview. The call back should return the buffer to use. /// subview. The call back should return the buffer to use.
using AllocBufferCallbackFn = std::function<Optional<Value>( using AllocBufferCallbackFn = std::function<std::optional<Value>(
OpBuilder &b, memref::SubViewOp subView, OpBuilder &b, memref::SubViewOp subView,
ArrayRef<Value> boundingSubViewSize, DataLayout &layout)>; ArrayRef<Value> boundingSubViewSize, DataLayout &layout)>;
/// Callback function type used to deallocate the buffers used to hold the /// Callback function type used to deallocate the buffers used to hold the
/// promoted subview. /// promoted subview.
using DeallocBufferCallbackFn = using DeallocBufferCallbackFn =
std::function<LogicalResult(OpBuilder &b, Value buffer)>; std::function<LogicalResult(OpBuilder &b, Value buffer)>;
/// Callback function type used to insert copy from original subview to /// Callback function type used to insert copy from original subview to
/// subview of the promoted region for the read operands/subview of promoted /// subview of the promoted region for the read operands/subview of promoted
/// region to original subview for the results. The copy has to happen from /// region to original subview for the results. The copy has to happen from
/// `src` to `dst`. /// `src` to `dst`.
using CopyCallbackFn = using CopyCallbackFn =
std::function<LogicalResult(OpBuilder &b, Value src, Value dst)>; std::function<LogicalResult(OpBuilder &b, Value src, Value dst)>;
struct LinalgPromotionOptions { struct LinalgPromotionOptions {
/// Indices of subViews to promote. If `None`, try to promote all operands. /// Indices of subViews to promote. If `None`, try to promote all operands.
Optional<DenseSet<unsigned>> operandsToPromote = std::nullopt; std::optional<DenseSet<unsigned>> operandsToPromote = std::nullopt;
LinalgPromotionOptions &setOperandsToPromote(ArrayRef<int64_t> operands) { LinalgPromotionOptions &setOperandsToPromote(ArrayRef<int64_t> operands) {
operandsToPromote = DenseSet<unsigned>(); operandsToPromote = DenseSet<unsigned>();
operandsToPromote->insert(operands.begin(), operands.end()); operandsToPromote->insert(operands.begin(), operands.end());
return *this; return *this;
} }
/// If ith element of `useFullTiles` is true the full view should be used /// If ith element of `useFullTiles` is true the full view should be used
/// for the promoted buffer of the ith operand in `operandsToPromote`. /// for the promoted buffer of the ith operand in `operandsToPromote`.
/// Otherwise the partial view will be used. The decision is defaulted to /// Otherwise the partial view will be used. The decision is defaulted to
/// `useFullTileBuffersDefault` when `useFullTileBuffers` is None and for /// `useFullTileBuffersDefault` when `useFullTileBuffers` is None and for
/// operands missing from `useFullTileBuffers`. /// operands missing from `useFullTileBuffers`.
Optional<llvm::SmallBitVector> useFullTileBuffers = std::nullopt; std::optional<llvm::SmallBitVector> useFullTileBuffers = std::nullopt;
LinalgPromotionOptions &setUseFullTileBuffers(ArrayRef<bool> useFullTiles) { LinalgPromotionOptions &setUseFullTileBuffers(ArrayRef<bool> useFullTiles) {
unsigned size = useFullTiles.size(); unsigned size = useFullTiles.size();
llvm::SmallBitVector tmp(size, false); llvm::SmallBitVector tmp(size, false);
for (unsigned i = 0; i < size; ++i) for (unsigned i = 0; i < size; ++i)
tmp[i] = useFullTiles[i]; tmp[i] = useFullTiles[i];
useFullTileBuffers = tmp; useFullTileBuffers = tmp;
return *this; return *this;
} }
/// If true all operands unspecified by `useFullTileBuffers` will use the /// If true all operands unspecified by `useFullTileBuffers` will use the
/// full view, otherwise the partial view. /// full view, otherwise the partial view.
bool useFullTileBuffersDefault = false; bool useFullTileBuffersDefault = false;
LinalgPromotionOptions &setUseFullTileBuffersByDefault(bool use) { LinalgPromotionOptions &setUseFullTileBuffersByDefault(bool use) {
useFullTileBuffersDefault = use; useFullTileBuffersDefault = use;
return *this; return *this;
} }
/// Alignment of promoted buffer. If `None` do not specify alignment. /// Alignment of promoted buffer. If `None` do not specify alignment.
Optional<unsigned> alignment = std::nullopt; std::optional<unsigned> alignment = std::nullopt;
LinalgPromotionOptions &setAlignment(unsigned align) { LinalgPromotionOptions &setAlignment(unsigned align) {
alignment = align; alignment = align;
return *this; return *this;
} }
/// Use alloca with the default allocation scheme. /// Use alloca with the default allocation scheme.
bool useAlloca = false; bool useAlloca = false;
LinalgPromotionOptions &setUseAlloca(bool use) { LinalgPromotionOptions &setUseAlloca(bool use) {
useAlloca = use; useAlloca = use;
return *this; return *this;
} }
/// Callback function to do the allocation of the promoted buffer. If None, /// Callback function to do the allocation of the promoted buffer. If None,
/// then the default allocation scheme of allocating a memref<?xi8> buffer /// then the default allocation scheme of allocating a memref<?xi8> buffer
/// followed by a view operation is used. /// followed by a view operation is used.
Optional<AllocBufferCallbackFn> allocationFn = std::nullopt; std::optional<AllocBufferCallbackFn> allocationFn = std::nullopt;
Optional<DeallocBufferCallbackFn> deallocationFn = std::nullopt; std::optional<DeallocBufferCallbackFn> deallocationFn = std::nullopt;
LinalgPromotionOptions & LinalgPromotionOptions &
setAllocationDeallocationFns(AllocBufferCallbackFn const &allocFn, setAllocationDeallocationFns(AllocBufferCallbackFn const &allocFn,
DeallocBufferCallbackFn const &deallocFn) { DeallocBufferCallbackFn const &deallocFn) {
allocationFn = allocFn; allocationFn = allocFn;
deallocationFn = deallocFn; deallocationFn = deallocFn;
return *this; return *this;
} }
/// Callback function to do the copy of data to and from the promoted /// Callback function to do the copy of data to and from the promoted
/// subview. If None then a memref.copy is used. /// subview. If None then a memref.copy is used.
Optional<CopyCallbackFn> copyInFn = std::nullopt; std::optional<CopyCallbackFn> copyInFn = std::nullopt;
Optional<CopyCallbackFn> copyOutFn = std::nullopt; std::optional<CopyCallbackFn> copyOutFn = std::nullopt;
LinalgPromotionOptions &setCopyInOutFns(CopyCallbackFn const &copyIn, LinalgPromotionOptions &setCopyInOutFns(CopyCallbackFn const &copyIn,
CopyCallbackFn const &copyOut) { CopyCallbackFn const &copyOut) {
copyInFn = copyIn; copyInFn = copyIn;
copyOutFn = copyOut; copyOutFn = copyOut;
return *this; return *this;
} }
}; };
▲ Show 20 Lines • Show All 136 Lines • ▼ Show 20 Lines
/// (i.e. that only tiles parallel dimensions, e.g. in the Linalg case). /// (i.e. that only tiles parallel dimensions, e.g. in the Linalg case).
struct ForeachThreadTilingResult { struct ForeachThreadTilingResult {
Operation *tileOp; Operation *tileOp;
Operation *tiledOp; Operation *tiledOp;
}; };
FailureOr<ForeachThreadTilingResult> FailureOr<ForeachThreadTilingResult>
tileToForeachThreadOp(RewriterBase &builder, TilingInterface op, tileToForeachThreadOp(RewriterBase &builder, TilingInterface op,
ArrayRef<OpFoldResult> numThreads, ArrayRef<OpFoldResult> numThreads,
Optional<ArrayAttr> mapping); std::optional<ArrayAttr> mapping);
/// Same as `tileToForeachThreadOp`, but calculate the number of threads /// Same as `tileToForeachThreadOp`, but calculate the number of threads
/// required using the given tileSizes. /// required using the given tileSizes.
FailureOr<ForeachThreadTilingResult> FailureOr<ForeachThreadTilingResult>
tileToForeachThreadOpUsingTileSizes(RewriterBase &builder, TilingInterface op, tileToForeachThreadOpUsingTileSizes(RewriterBase &builder, TilingInterface op,
ArrayRef<OpFoldResult> tileSizes, ArrayRef<OpFoldResult> tileSizes,
Optional<ArrayAttr> mapping); std::optional<ArrayAttr> mapping);
/// Transformation information returned after reduction tiling. /// Transformation information returned after reduction tiling.
struct ForeachThreadReductionTilingResult { struct ForeachThreadReductionTilingResult {
/// The partial reduction tiled op generated. /// The partial reduction tiled op generated.
Operation *parallelTiledOp; Operation *parallelTiledOp;
/// The final reduction operation merging all the partial reductions. /// The final reduction operation merging all the partial reductions.
Operation *mergeOp; Operation *mergeOp;
/// The op initializing the tensor used for partial reductions. /// The op initializing the tensor used for partial reductions.
Show All 21 Lines
/// %3 = tensor.extract_slice %in : tensor<7x9xf32> -> tensor<7x?xf32> /// %3 = tensor.extract_slice %in : tensor<7x9xf32> -> tensor<7x?xf32>
/// %4 = linalg.generic %2, %3 ["parallel", "reduction"] /// %4 = linalg.generic %2, %3 ["parallel", "reduction"]
/// : tensor<7x?xf32> -> tensor<7xf32> /// : tensor<7x?xf32> -> tensor<7xf32>
/// %5 = tensor.insert_slice %3, %arg0[0, %iv] : tensor<7x4xf32> /// %5 = tensor.insert_slice %3, %arg0[0, %iv] : tensor<7x4xf32>
/// } /// }
/// %6 = linalg.generic %1 ["parallel", "reduction"] /// %6 = linalg.generic %1 ["parallel", "reduction"]
/// : tensor<7x4xf32> -> tensor<7xf32> /// : tensor<7x4xf32> -> tensor<7xf32>
/// ``` /// ```
FailureOr<ForeachThreadReductionTilingResult> FailureOr<ForeachThreadReductionTilingResult> tileReductionUsingForeachThread(
tileReductionUsingForeachThread(RewriterBase &b, PartialReductionOpInterface op, RewriterBase &b, PartialReductionOpInterface op,
ArrayRef<OpFoldResult> numThreads, ArrayRef<OpFoldResult> numThreads, ArrayRef<OpFoldResult> tileSizes = {},
ArrayRef<OpFoldResult> tileSizes = {}, std::optional<ArrayAttr> mapping = std::nullopt);
Optional<ArrayAttr> mapping = std::nullopt);
/// All indices returned by IndexOp should be invariant with respect to /// All indices returned by IndexOp should be invariant with respect to
/// tiling. Therefore, if an operation is tiled, we have to transform the /// tiling. Therefore, if an operation is tiled, we have to transform the
/// indices accordingly, i.e. offset them by the values of the corresponding /// indices accordingly, i.e. offset them by the values of the corresponding
/// induction variables that are captured implicitly in the body of the op. /// induction variables that are captured implicitly in the body of the op.
/// ///
/// Example. `linalg.generic` before tiling: /// Example. `linalg.generic` before tiling:
/// ///
▲ Show 20 Lines • Show All 88 Lines • ▼ Show 20 Linesstruct LinalgTilingAndFusionOptions {
LinalgTilingAndFusionOptions &setTileSizes(ArrayRef<int64_t> ts) { LinalgTilingAndFusionOptions &setTileSizes(ArrayRef<int64_t> ts) {
tileSizes.assign(ts.begin(), ts.end()); tileSizes.assign(ts.begin(), ts.end());
return *this; return *this;
} }
/// Tile interchange used to permute the tile loops. /// Tile interchange used to permute the tile loops.
SmallVector<int64_t> tileInterchange; SmallVector<int64_t> tileInterchange;
/// When specified, specifies distribution of generated tile loops to /// When specified, specifies distribution of generated tile loops to
/// processors. /// processors.
Optional<LinalgLoopDistributionOptions> tileDistribution = std::nullopt; std::optional<LinalgLoopDistributionOptions> tileDistribution = std::nullopt;
LinalgTilingAndFusionOptions & LinalgTilingAndFusionOptions &
setDistributionOptions(LinalgLoopDistributionOptions distributionOptions) { setDistributionOptions(LinalgLoopDistributionOptions distributionOptions) {
tileDistribution = std::move(distributionOptions); tileDistribution = std::move(distributionOptions);
return *this; return *this;
} }
}; };
struct LinalgTilingOptions { struct LinalgTilingOptions {
Show All 36 Linesstruct LinalgTilingOptions {
LinalgTilingOptions &setLoopType(LinalgTilingLoopType lt) { LinalgTilingOptions &setLoopType(LinalgTilingLoopType lt) {
loopType = lt; loopType = lt;
return *this; return *this;
} }
/// When specified, specifies distribution of generated tile loops to /// When specified, specifies distribution of generated tile loops to
/// processors. /// processors.
Optional<LinalgLoopDistributionOptions> distribution = std::nullopt; std::optional<LinalgLoopDistributionOptions> distribution = std::nullopt;
LinalgTilingOptions & LinalgTilingOptions &
setDistributionOptions(LinalgLoopDistributionOptions distributionOptions) { setDistributionOptions(LinalgLoopDistributionOptions distributionOptions) {
distribution = std::move(distributionOptions); distribution = std::move(distributionOptions);
return *this; return *this;
} }
/// Specification markers of how to distribute the `linalg.tiled_loop`. /// Specification markers of how to distribute the `linalg.tiled_loop`.
▲ Show 20 Lines • Show All 113 Lines • ▼ Show 20 Lines
struct CopyVectorizationPattern : public OpRewritePattern<memref::CopyOp> { struct CopyVectorizationPattern : public OpRewritePattern<memref::CopyOp> {
using OpRewritePattern<memref::CopyOp>::OpRewritePattern; using OpRewritePattern<memref::CopyOp>::OpRewritePattern;
LogicalResult matchAndRewrite(memref::CopyOp copyOp, LogicalResult matchAndRewrite(memref::CopyOp copyOp,
PatternRewriter &rewriter) const override; PatternRewriter &rewriter) const override;
}; };
/// Return vector::CombiningKind for the given op. /// Return vector::CombiningKind for the given op.
llvm::Optional<vector::CombiningKind> getCombinerOpKind(Operation *combinerOp); std::optional<vector::CombiningKind> getCombinerOpKind(Operation *combinerOp);
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Transformations exposed as rewrite patterns. // Transformations exposed as rewrite patterns.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Linalg generalization patterns /// Linalg generalization patterns
/// Populates `patterns` with patterns to convert spec-generated named ops to /// Populates `patterns` with patterns to convert spec-generated named ops to
▲ Show 20 Lines • Show All 143 Lines • ▼ Show 20 Linesstruct ExtractSliceOfPadTensorSwapPattern
/// ///
/// The function will be given the slice op and should return: /// The function will be given the slice op and should return:
/// - std::nullopt: to fail the match and not apply the pattern; /// - std::nullopt: to fail the match and not apply the pattern;
/// - true: to apply the pattern with zero slice guard; /// - true: to apply the pattern with zero slice guard;
/// - false: to apply the pattern without zero slice guard. /// - false: to apply the pattern without zero slice guard.
/// ///
/// See the documentation for tensor::bubbleUpPadSlice regarding zero slice /// See the documentation for tensor::bubbleUpPadSlice regarding zero slice
/// guard. /// guard.
using ControlFn = std::function<llvm::Optional<bool>(tensor::ExtractSliceOp)>; using ControlFn = std::function<std::optional<bool>(tensor::ExtractSliceOp)>;
ExtractSliceOfPadTensorSwapPattern(MLIRContext *context, ExtractSliceOfPadTensorSwapPattern(MLIRContext *context,
ControlFn controlFn = nullptr, ControlFn controlFn = nullptr,
PatternBenefit benefit = 1) PatternBenefit benefit = 1)
: OpRewritePattern(context, benefit), controlFn(std::move(controlFn)) {} : OpRewritePattern(context, benefit), controlFn(std::move(controlFn)) {}
LogicalResult matchAndRewrite(tensor::ExtractSliceOp sliceOp, LogicalResult matchAndRewrite(tensor::ExtractSliceOp sliceOp,
PatternRewriter &rewriter) const override; PatternRewriter &rewriter) const override;
▲ Show 20 Lines • Show All 142 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/MemRef/IR/MemRefOps.td

Show First 20 Lines • Show All 580 Lines • ▼ Show 20 Linesdef MemRef_DimOp : MemRef_Op<"dim", [
}]; }];
let builders = [ let builders = [
OpBuilder<(ins "Value":$source, "int64_t":$index)>, OpBuilder<(ins "Value":$source, "int64_t":$index)>,
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Helper function to get the index as a simple integer if it is constant. /// Helper function to get the index as a simple integer if it is constant.
Optional<int64_t> getConstantIndex(); std::optional<int64_t> getConstantIndex();
/// Interface method of ShapedDimOpInterface: Return the source memref. /// Interface method of ShapedDimOpInterface: Return the source memref.
Value getShapedValue() { return getSource(); } Value getShapedValue() { return getSource(); }
/// Interface method of ShapedDimOpInterface: Return the dimension. /// Interface method of ShapedDimOpInterface: Return the dimension.
OpFoldResult getDimension() { return getIndex(); } OpFoldResult getDimension() { return getIndex(); }
/// Interface method for ConditionallySpeculatable. /// Interface method for ConditionallySpeculatable.
▲ Show 20 Lines • Show All 1,566 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/PDLInterp/IR/PDLInterpOps.td

Show First 20 Lines • Show All 810 Lines • ▼ Show 20 Linesdef PDLInterp_GetOperandsOp : PDLInterp_Op<"get_operands", [Pure]> {
let arguments = (ins let arguments = (ins
PDL_Operation:$inputOp, PDL_Operation:$inputOp,
OptionalAttr<ConfinedAttr<I32Attr, [IntNonNegative]>>:$index OptionalAttr<ConfinedAttr<I32Attr, [IntNonNegative]>>:$index
); );
let results = (outs PDL_InstOrRangeOf<PDL_Value>:$value); let results = (outs PDL_InstOrRangeOf<PDL_Value>:$value);
let assemblyFormat = "($index^)? `of` $inputOp `:` type($value) attr-dict"; let assemblyFormat = "($index^)? `of` $inputOp `:` type($value) attr-dict";
let builders = [ let builders = [
OpBuilder<(ins "Type":$resultType, "Value":$inputOp, OpBuilder<(ins "Type":$resultType, "Value":$inputOp,
"Optional<unsigned>":$index), [{ "std::optional<unsigned>":$index), [{
build($_builder, $_state, resultType, inputOp, build($_builder, $_state, resultType, inputOp,
index ? $_builder.getI32IntegerAttr(*index) : IntegerAttr()); index ? $_builder.getI32IntegerAttr(*index) : IntegerAttr());
}]>, }]>,
]; ];
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// pdl_interp::GetResultOp // pdl_interp::GetResultOp
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Linesdef PDLInterp_GetResultsOp : PDLInterp_Op<"get_results", [Pure]> {
let arguments = (ins let arguments = (ins
PDL_Operation:$inputOp, PDL_Operation:$inputOp,
OptionalAttr<ConfinedAttr<I32Attr, [IntNonNegative]>>:$index OptionalAttr<ConfinedAttr<I32Attr, [IntNonNegative]>>:$index
); );
let results = (outs PDL_InstOrRangeOf<PDL_Value>:$value); let results = (outs PDL_InstOrRangeOf<PDL_Value>:$value);
let assemblyFormat = "($index^)? `of` $inputOp `:` type($value) attr-dict"; let assemblyFormat = "($index^)? `of` $inputOp `:` type($value) attr-dict";
let builders = [ let builders = [
OpBuilder<(ins "Type":$resultType, "Value":$inputOp, OpBuilder<(ins "Type":$resultType, "Value":$inputOp,
"Optional<unsigned>":$index), [{ "std::optional<unsigned>":$index), [{
build($_builder, $_state, resultType, inputOp, build($_builder, $_state, resultType, inputOp,
index ? $_builder.getI32IntegerAttr(*index) : IntegerAttr()); index ? $_builder.getI32IntegerAttr(*index) : IntegerAttr());
}]>, }]>,
OpBuilder<(ins "Value":$inputOp), [{ OpBuilder<(ins "Value":$inputOp), [{
build($_builder, $_state, build($_builder, $_state,
pdl::RangeType::get($_builder.getType<pdl::ValueType>()), inputOp, pdl::RangeType::get($_builder.getType<pdl::ValueType>()), inputOp,
IntegerAttr()); IntegerAttr());
}]>, }]>,
▲ Show 20 Lines • Show All 423 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/SCF/IR/SCFOps.td

Show First 20 Lines • Show All 268 Lines • ▼ Show 20 Lines bool hasIterOperands() {
return getOperation()->getNumOperands() > getNumControlOperands(); return getOperation()->getNumOperands() > getNumControlOperands();
} }
/// Get Number of loop-carried values /// Get Number of loop-carried values
unsigned getNumIterOperands() { unsigned getNumIterOperands() {
return getOperation()->getNumOperands() - getNumControlOperands(); return getOperation()->getNumOperands() - getNumControlOperands();
} }
/// Get the iter arg number for an operand. If it isnt an iter arg /// Get the iter arg number for an operand. If it isnt an iter arg
/// operand return std::nullopt. /// operand return std::nullopt.
Optional<unsigned> getIterArgNumberForOpOperand(OpOperand &opOperand) { std::optional<unsigned> getIterArgNumberForOpOperand(OpOperand &opOperand) {
if (opOperand.getOwner() != getOperation()) if (opOperand.getOwner() != getOperation())
return std::nullopt; return std::nullopt;
unsigned operandNumber = opOperand.getOperandNumber(); unsigned operandNumber = opOperand.getOperandNumber();
if (operandNumber < getNumControlOperands()) if (operandNumber < getNumControlOperands())
return std::nullopt; return std::nullopt;
return operandNumber - getNumControlOperands(); return operandNumber - getNumControlOperands();
} }
Show All 40 Lines OpOperand &getOpOperandForResult(OpResult opResult) {
return getOperation()->getOpOperand( return getOperation()->getOpOperand(
getNumControlOperands() + opResult.getResultNumber()); getNumControlOperands() + opResult.getResultNumber());
} }
/// Return operands used when entering the region at 'index'. These operands /// Return operands used when entering the region at 'index'. These operands
/// correspond to the loop iterator operands, i.e., those exclusing the /// correspond to the loop iterator operands, i.e., those exclusing the
/// induction variable. LoopOp only has one region, so 0 is the only valid /// induction variable. LoopOp only has one region, so 0 is the only valid
/// value for `index`. /// value for `index`.
OperandRange getSuccessorEntryOperands(Optional<unsigned> index); OperandRange getSuccessorEntryOperands(std::optional<unsigned> index);
/// Returns the step as an `APInt` if it is constant. /// Returns the step as an `APInt` if it is constant.
Optional<APInt> getConstantStep(); std::optional<APInt> getConstantStep();
/// Interface method for ConditionallySpeculatable. /// Interface method for ConditionallySpeculatable.
Speculation::Speculatability getSpeculatability(); Speculation::Speculatability getSpeculatability();
}]; }];
let hasCanonicalizer = 1; let hasCanonicalizer = 1;
let hasCustomAssemblyFormat = 1; let hasCustomAssemblyFormat = 1;
let hasVerifier = 1; let hasVerifier = 1;
▲ Show 20 Lines • Show All 145 Lines • ▼ Show 20 Linesdef ForeachThreadOp : SCF_Op<"foreach_thread", [
let hasCustomAssemblyFormat = 1; let hasCustomAssemblyFormat = 1;
let hasVerifier = 1; let hasVerifier = 1;
// The default builder does not add the proper body BBargs, roll our own. // The default builder does not add the proper body BBargs, roll our own.
let skipDefaultBuilders = 1; let skipDefaultBuilders = 1;
let builders = [ let builders = [
// Bodyless builder, outputs must be specified. // Bodyless builder, outputs must be specified.
OpBuilder<(ins "ValueRange":$outputs, "ValueRange":$num_threads, OpBuilder<(ins "ValueRange":$outputs, "ValueRange":$num_threads,
"Optional<ArrayAttr>":$mapping)>, "std::optional<ArrayAttr>":$mapping)>,
// Builder that takes a bodyBuilder lambda. // Builder that takes a bodyBuilder lambda.
OpBuilder<(ins "ValueRange":$outputs, "ValueRange":$num_threads, OpBuilder<(ins "ValueRange":$outputs, "ValueRange":$num_threads,
"ArrayRef<Attribute>":$mapping, "ArrayRef<Attribute>":$mapping,
"function_ref<void(OpBuilder &, Location, ValueRange)>":$bodyBuilder)> "function_ref<void(OpBuilder &, Location, ValueRange)>":$bodyBuilder)>
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
int64_t getRank() { return getNumThreads().size(); } int64_t getRank() { return getNumThreads().size(); }
▲ Show 20 Lines • Show All 490 Lines • ▼ Show 20 Lines OpBuilder<(ins "TypeRange":$resultTypes, "ValueRange":$operands,
"function_ref<void(OpBuilder &, Location, ValueRange)>":$beforeBuilder, "function_ref<void(OpBuilder &, Location, ValueRange)>":$beforeBuilder,
"function_ref<void(OpBuilder &, Location, ValueRange)>":$afterBuilder)> "function_ref<void(OpBuilder &, Location, ValueRange)>":$afterBuilder)>
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
using BodyBuilderFn = using BodyBuilderFn =
function_ref<void(OpBuilder &, Location, ValueRange)>; function_ref<void(OpBuilder &, Location, ValueRange)>;
OperandRange getSuccessorEntryOperands(Optional<unsigned> index); OperandRange getSuccessorEntryOperands(std::optional<unsigned> index);
ConditionOp getConditionOp(); ConditionOp getConditionOp();
YieldOp getYieldOp(); YieldOp getYieldOp();
Block::BlockArgListType getBeforeArguments(); Block::BlockArgListType getBeforeArguments();
Block::BlockArgListType getAfterArguments(); Block::BlockArgListType getAfterArguments();
}]; }];
let hasCanonicalizer = 1; let hasCanonicalizer = 1;
let hasCustomAssemblyFormat = 1; let hasCustomAssemblyFormat = 1;
▲ Show 20 Lines • Show All 95 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/SPIRV/IR/ParserUtils.h

Show All 25 Lines
static ParseResult static ParseResult
parseEnumKeywordAttr(EnumClass &value, ParserType &parser, parseEnumKeywordAttr(EnumClass &value, ParserType &parser,
StringRef attrName = spirv::attributeName<EnumClass>()) { StringRef attrName = spirv::attributeName<EnumClass>()) {
StringRef keyword; StringRef keyword;
SmallVector<NamedAttribute, 1> attr; SmallVector<NamedAttribute, 1> attr;
auto loc = parser.getCurrentLocation(); auto loc = parser.getCurrentLocation();
if (parser.parseKeyword(&keyword)) if (parser.parseKeyword(&keyword))
return failure(); return failure();
if (Optional<EnumClass> attr = spirv::symbolizeEnum<EnumClass>(keyword)) { if (std::optional<EnumClass> attr =
spirv::symbolizeEnum<EnumClass>(keyword)) {
value = *attr; value = *attr;
return success(); return success();
} }
return parser.emitError(loc, "invalid ") return parser.emitError(loc, "invalid ")
<< attrName << " attribute specification: " << keyword; << attrName << " attribute specification: " << keyword;
} }
} // namespace mlir } // namespace mlir
#endif // MLIR_DIALECT_SPIRV_IR_PARSERUTILS_H_ #endif // MLIR_DIALECT_SPIRV_IR_PARSERUTILS_H_

mlir/include/mlir/Dialect/SPIRV/IR/SPIRVStructureOps.td

Show First 20 Lines • Show All 486 Lines • ▼ Show 20 Lines let arguments = (ins
OptionalAttr<StrAttr>:$sym_name OptionalAttr<StrAttr>:$sym_name
); );
let results = (outs); let results = (outs);
let regions = (region AnyRegion); let regions = (region AnyRegion);
let builders = [ let builders = [
OpBuilder<(ins CArg<"Optional<StringRef>", "std::nullopt">:$name)>, OpBuilder<(ins CArg<"std::optional<StringRef>", "std::nullopt">:$name)>,
OpBuilder<(ins "spirv::AddressingModel":$addressing_model, OpBuilder<(ins "spirv::AddressingModel":$addressing_model,
"spirv::MemoryModel":$memory_model, "spirv::MemoryModel":$memory_model,
CArg<"Optional<spirv::VerCapExtAttr>", "std::nullopt">:$vce_triple, CArg<"std::optional<spirv::VerCapExtAttr>", "std::nullopt">:$vce_triple,
CArg<"Optional<StringRef>", "std::nullopt">:$name)> CArg<"std::optional<StringRef>", "std::nullopt">:$name)>
]; ];
// We need to ensure the block inside the region is properly terminated; // We need to ensure the block inside the region is properly terminated;
// the auto-generated builders do not guarantee that. // the auto-generated builders do not guarantee that.
let skipDefaultBuilders = 1; let skipDefaultBuilders = 1;
let hasOpcode = 0; let hasOpcode = 0;
let autogenSerialization = 0; let autogenSerialization = 0;
let extraClassDeclaration = [{ let extraClassDeclaration = [{
bool isOptionalSymbol() { return true; } bool isOptionalSymbol() { return true; }
Optional<StringRef> getName() { return getSymName(); } std::optional<StringRef> getName() { return getSymName(); }
static StringRef getVCETripleAttrName() { return "vce_triple"; } static StringRef getVCETripleAttrName() { return "vce_triple"; }
}]; }];
let hasVerifier = 0; let hasVerifier = 0;
let hasRegionVerifier = 1; let hasRegionVerifier = 1;
} }
▲ Show 20 Lines • Show All 282 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Shape/IR/ShapeOps.td

Show First 20 Lines • Show All 346 Lines • ▼ Show 20 Linesdef Shape_DimOp : Shape_Op<"dim",
let builders = [ let builders = [
// Builder that allows passing a constant dimension as a simple integer. // Builder that allows passing a constant dimension as a simple integer.
OpBuilder<(ins "Value":$value, "int64_t":$index)> OpBuilder<(ins "Value":$value, "int64_t":$index)>
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Get the `index` value as integer if it is constant. /// Get the `index` value as integer if it is constant.
Optional<int64_t> getConstantIndex(); std::optional<int64_t> getConstantIndex();
/// Returns when two result types are compatible for this op; method used by /// Returns when two result types are compatible for this op; method used by
/// InferTypeOpInterface /// InferTypeOpInterface
static bool isCompatibleReturnTypes(TypeRange l, TypeRange r); static bool isCompatibleReturnTypes(TypeRange l, TypeRange r);
}]; }];
let hasFolder = 1; let hasFolder = 1;
let hasVerifier = 1; let hasVerifier = 1;
Show All 14 Linesdef Shape_GetExtentOp : Shape_Op<"get_extent",
let builders = [ let builders = [
// Builder that allows passing a constant dimension as a simple integer. // Builder that allows passing a constant dimension as a simple integer.
OpBuilder<(ins "Value":$shape, "int64_t":$dim)> OpBuilder<(ins "Value":$shape, "int64_t":$dim)>
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Get the `dim` value as integer if it is constant. /// Get the `dim` value as integer if it is constant.
Optional<int64_t> getConstantDim(); std::optional<int64_t> getConstantDim();
/// Returns when two result types are compatible for this op; method used by /// Returns when two result types are compatible for this op; method used by
/// InferTypeOpInterface /// InferTypeOpInterface
static bool isCompatibleReturnTypes(TypeRange l, TypeRange r); static bool isCompatibleReturnTypes(TypeRange l, TypeRange r);
}]; }];
let hasFolder = 1; let hasFolder = 1;
let hasVerifier = 1; let hasVerifier = 1;
} }
▲ Show 20 Lines • Show All 790 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorTypes.td

Show First 20 Lines • Show All 61 Lines • ▼ Show 20 Lines let builders = [
TypeBuilderWithInferredContext<(ins "Value":$tensor), [{ TypeBuilderWithInferredContext<(ins "Value":$tensor), [{
return get(tensor.getType()); return get(tensor.getType());
}]> }]>
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
// Get the integer type used to store memory and dimension sizes. // Get the integer type used to store memory and dimension sizes.
IntegerType getSizesType() const; IntegerType getSizesType() const;
Type getFieldType(StorageSpecifierKind kind, Optional<unsigned> dim) const; Type getFieldType(StorageSpecifierKind kind, std::optional<unsigned> dim) const;
Type getFieldType(StorageSpecifierKind kind, Optional<APInt> dim) const; Type getFieldType(StorageSpecifierKind kind, std::optional<APInt> dim) const;
}]; }];
let assemblyFormat="`<` qualified($encoding) `>`"; let assemblyFormat="`<` qualified($encoding) `>`";
} }
def IsSparseTensorStorageSpecifierTypePred def IsSparseTensorStorageSpecifierTypePred
: CPred<"$_self.isa<::mlir::sparse_tensor::StorageSpecifierType>()">; : CPred<"$_self.isa<::mlir::sparse_tensor::StorageSpecifierType>()">;
def SparseTensorStorageSpecifier def SparseTensorStorageSpecifier
: Type<CPred<"$_self.isa<::mlir::sparse_tensor::StorageSpecifierType>()">, "metadata", : Type<CPred<"$_self.isa<::mlir::sparse_tensor::StorageSpecifierType>()">, "metadata",
"::mlir::sparse_tensor::StorageSpecifierType">; "::mlir::sparse_tensor::StorageSpecifierType">;
#endif // SPARSETENSOR_TYPES #endif // SPARSETENSOR_TYPES

mlir/include/mlir/Dialect/Tensor/IR/TensorOps.td

Show First 20 Lines • Show All 123 Lines • ▼ Show 20 Linesdef Tensor_DimOp : Tensor_Op<"dim", [
}]; }];
let builders = [ let builders = [
OpBuilder<(ins "Value":$source, "int64_t":$index)> OpBuilder<(ins "Value":$source, "int64_t":$index)>
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Helper function to get the index as a simple integer if it is constant. /// Helper function to get the index as a simple integer if it is constant.
Optional<int64_t> getConstantIndex(); std::optional<int64_t> getConstantIndex();
/// Interface method of ShapedDimOpInterface: Return the source tensor. /// Interface method of ShapedDimOpInterface: Return the source tensor.
Value getShapedValue() { return getSource(); } Value getShapedValue() { return getSource(); }
/// Interface method of ShapedDimOpInterface: Return the dimension. /// Interface method of ShapedDimOpInterface: Return the dimension.
OpFoldResult getDimension() { return getIndex(); } OpFoldResult getDimension() { return getIndex(); }
/// Interface method for ConditionallySpeculatable. /// Interface method for ConditionallySpeculatable.
▲ Show 20 Lines • Show All 234 Lines • ▼ Show 20 Lines let extraClassDeclaration = extraBaseClassDeclaration # [{
/// The result of an extract_slice is always a tensor. /// The result of an extract_slice is always a tensor.
RankedTensorType getType() { RankedTensorType getType() {
return getResult().getType().cast<RankedTensorType>(); return getResult().getType().cast<RankedTensorType>();
} }
/// Compute the rank-reduction mask that can be applied to map the source /// Compute the rank-reduction mask that can be applied to map the source
/// tensor type to the result tensor type by dropping unit dims. /// tensor type to the result tensor type by dropping unit dims.
llvm::Optional<llvm::SmallDenseSet<unsigned>> std::optional<llvm::SmallDenseSet<unsigned>>
computeRankReductionMask() { computeRankReductionMask() {
return ::mlir::computeRankReductionMask(getSourceType().getShape(), return ::mlir::computeRankReductionMask(getSourceType().getShape(),
getType().getShape()); getType().getShape());
}; };
/// An extract_slice result type can be inferred, when it is not /// An extract_slice result type can be inferred, when it is not
/// rank-reduced, from the source type and the static representation of /// rank-reduced, from the source type and the static representation of
/// offsets, sizes and strides. Special sentinels encode the dynamic case. /// offsets, sizes and strides. Special sentinels encode the dynamic case.
▲ Show 20 Lines • Show All 1,481 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Utils/StaticValueUtils.h

Show First 20 Lines • Show All 61 Lines • ▼ Show 20 Lines
/// Given an array of values, try to extract a constant Attribute from each /// Given an array of values, try to extract a constant Attribute from each
/// value. If this fails, return the original value. /// value. If this fails, return the original value.
SmallVector<OpFoldResult> getAsOpFoldResult(ValueRange values); SmallVector<OpFoldResult> getAsOpFoldResult(ValueRange values);
/// Convert `arrayAttr` to a vector of OpFoldResult. /// Convert `arrayAttr` to a vector of OpFoldResult.
SmallVector<OpFoldResult> getAsOpFoldResult(ArrayAttr arrayAttr); SmallVector<OpFoldResult> getAsOpFoldResult(ArrayAttr arrayAttr);
/// If ofr is a constant integer or an IntegerAttr, return the integer. /// If ofr is a constant integer or an IntegerAttr, return the integer.
Optional<int64_t> getConstantIntValue(OpFoldResult ofr); std::optional<int64_t> getConstantIntValue(OpFoldResult ofr);
/// Return true if `ofr` is constant integer equal to `value`. /// Return true if `ofr` is constant integer equal to `value`.
bool isConstantIntValue(OpFoldResult ofr, int64_t value); bool isConstantIntValue(OpFoldResult ofr, int64_t value);
/// Return true if ofr1 and ofr2 are the same integer constant attribute values /// Return true if ofr1 and ofr2 are the same integer constant attribute values
/// or the same SSA value. /// or the same SSA value.
/// Ignore integer bitwitdh and type mismatch that come from the fact there is /// Ignore integer bitwitdh and type mismatch that come from the fact there is
/// no IndexAttr and that IndexType have no bitwidth. /// no IndexAttr and that IndexType have no bitwidth.
Show All 24 Lines

mlir/include/mlir/Dialect/Vector/IR/VectorOps.h

Show First 20 Lines • Show All 112 Lines • ▼ Show 20 Lines
/// Collect a set of transfer read/write lowering patterns. /// Collect a set of transfer read/write lowering patterns.
/// ///
/// These patterns lower transfer ops to simpler ops like `vector.load`, /// These patterns lower transfer ops to simpler ops like `vector.load`,
/// `vector.store` and `vector.broadcast`. Only transfers with a transfer rank /// `vector.store` and `vector.broadcast`. Only transfers with a transfer rank
/// of a most `maxTransferRank` are lowered. This is useful when combined with /// of a most `maxTransferRank` are lowered. This is useful when combined with
/// VectorToSCF, which reduces the rank of vector transfer ops. /// VectorToSCF, which reduces the rank of vector transfer ops.
void populateVectorTransferLoweringPatterns( void populateVectorTransferLoweringPatterns(
RewritePatternSet &patterns, RewritePatternSet &patterns,
llvm::Optional<unsigned> maxTransferRank = std::nullopt, std::optional<unsigned> maxTransferRank = std::nullopt,
PatternBenefit benefit = 1); PatternBenefit benefit = 1);
/// These patterns materialize masks for various vector ops such as transfers. /// These patterns materialize masks for various vector ops such as transfers.
void populateVectorMaskMaterializationPatterns(RewritePatternSet &patterns, void populateVectorMaskMaterializationPatterns(RewritePatternSet &patterns,
bool force32BitVectorIndices, bool force32BitVectorIndices,
PatternBenefit benefit = 1); PatternBenefit benefit = 1);
/// Collects patterns to progressively lower vector.broadcast ops on high-D /// Collects patterns to progressively lower vector.broadcast ops on high-D
▲ Show 20 Lines • Show All 79 Lines • Show Last 20 Lines

mlir/include/mlir/Dialect/Vector/IR/VectorOps.td

Show First 20 Lines • Show All 1,364 Lines • ▼ Show 20 Lines OpBuilder<(ins "VectorType":$vectorType,
"ValueRange":$indices, "ValueRange":$indices,
"AffineMapAttr":$permutationMapAttr, "AffineMapAttr":$permutationMapAttr,
"ArrayAttr":$inBoundsAttr)>, "ArrayAttr":$inBoundsAttr)>,
/// 2. Builder that sets padding to zero and an empty mask (variant without attrs). /// 2. Builder that sets padding to zero and an empty mask (variant without attrs).
OpBuilder<(ins "VectorType":$vectorType, OpBuilder<(ins "VectorType":$vectorType,
"Value":$source, "Value":$source,
"ValueRange":$indices, "ValueRange":$indices,
"AffineMap":$permutationMap, "AffineMap":$permutationMap,
CArg<"Optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>, CArg<"std::optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>,
/// 3. Builder that sets permutation map to 'getMinorIdentityMap'. /// 3. Builder that sets permutation map to 'getMinorIdentityMap'.
OpBuilder<(ins "VectorType":$vectorType, OpBuilder<(ins "VectorType":$vectorType,
"Value":$source, "Value":$source,
"ValueRange":$indices, "ValueRange":$indices,
"Value":$padding, "Value":$padding,
CArg<"Optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>, CArg<"std::optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>,
/// 4. Builder that sets padding to zero and permutation map to /// 4. Builder that sets padding to zero and permutation map to
/// 'getMinorIdentityMap'. /// 'getMinorIdentityMap'.
OpBuilder<(ins "VectorType":$vectorType, OpBuilder<(ins "VectorType":$vectorType,
"Value":$source, "Value":$source,
"ValueRange":$indices, "ValueRange":$indices,
CArg<"Optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>, CArg<"std::optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>,
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
// MaskableOpInterface methods. // MaskableOpInterface methods.
bool supportsPassthru() { return true; } bool supportsPassthru() { return true; }
}]; }];
let hasCanonicalizer = 1; let hasCanonicalizer = 1;
▲ Show 20 Lines • Show All 122 Lines • ▼ Show 20 Lines OpBuilder<(ins "Value":$vector,
"ValueRange":$indices, "ValueRange":$indices,
"AffineMapAttr":$permutationMapAttr, "AffineMapAttr":$permutationMapAttr,
"ArrayAttr":$inBoundsAttr)>, "ArrayAttr":$inBoundsAttr)>,
/// 3. Builder with type inference that sets an empty mask (variant without attrs). /// 3. Builder with type inference that sets an empty mask (variant without attrs).
OpBuilder<(ins "Value":$vector, OpBuilder<(ins "Value":$vector,
"Value":$dest, "Value":$dest,
"ValueRange":$indices, "ValueRange":$indices,
"AffineMap":$permutationMap, "AffineMap":$permutationMap,
CArg<"Optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>, CArg<"std::optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>,
/// 4. Builder with type inference that sets an empty mask and sets permutation /// 4. Builder with type inference that sets an empty mask and sets permutation
/// map to 'getMinorIdentityMap'. /// map to 'getMinorIdentityMap'.
OpBuilder<(ins "Value":$vector, OpBuilder<(ins "Value":$vector,
"Value":$dest, "Value":$dest,
"ValueRange":$indices, "ValueRange":$indices,
CArg<"Optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>, CArg<"std::optional<ArrayRef<bool>>", "::std::nullopt">:$inBounds)>,
]; ];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// This method is added to maintain uniformity with load/store /// This method is added to maintain uniformity with load/store
/// ops of other dialects. /// ops of other dialects.
Value getValue() { return getVector(); } Value getValue() { return getVector(); }
std::pair<int64_t, int64_t> getDpsInitsPositionRange() { std::pair<int64_t, int64_t> getDpsInitsPositionRange() {
▲ Show 20 Lines • Show All 1,299 Lines • Show Last 20 Lines

mlir/include/mlir/IR/BuiltinOps.td

Show First 20 Lines • Show All 52 Lines • ▼ Show 20 Lines let description = [{
``` ```
}]; }];
let arguments = (ins OptionalAttr<SymbolNameAttr>:$sym_name, let arguments = (ins OptionalAttr<SymbolNameAttr>:$sym_name,
OptionalAttr<StrAttr>:$sym_visibility); OptionalAttr<StrAttr>:$sym_visibility);
let regions = (region SizedRegion<1>:$bodyRegion); let regions = (region SizedRegion<1>:$bodyRegion);
let assemblyFormat = "($sym_name^)? attr-dict-with-keyword $bodyRegion"; let assemblyFormat = "($sym_name^)? attr-dict-with-keyword $bodyRegion";
let builders = [OpBuilder<(ins CArg<"Optional<StringRef>", "{}">:$name)>]; let builders = [OpBuilder<(ins CArg<"std::optional<StringRef>", "{}">:$name)>];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Construct a module from the given location with an optional name. /// Construct a module from the given location with an optional name.
static ModuleOp create(Location loc, Optional<StringRef> name = std::nullopt); static ModuleOp create(Location loc, std::optional<StringRef> name = std::nullopt);
/// Return the name of this module if present. /// Return the name of this module if present.
Optional<StringRef> getName() { return getSymName(); } std::optional<StringRef> getName() { return getSymName(); }
//===------------------------------------------------------------------===// //===------------------------------------------------------------------===//
// SymbolOpInterface Methods // SymbolOpInterface Methods
//===------------------------------------------------------------------===// //===------------------------------------------------------------------===//
/// A ModuleOp may optionally define a symbol. /// A ModuleOp may optionally define a symbol.
bool isOptionalSymbol() { return true; } bool isOptionalSymbol() { return true; }
▲ Show 20 Lines • Show All 71 Lines • Show Last 20 Lines

mlir/include/mlir/IR/BuiltinTypeInterfaces.td

Show First 20 Lines • Show All 57 Lines • ▼ Show 20 Lines let description = [{
size of the dimension is dynamic, or not statically known). size of the dimension is dynamic, or not statically known).
}]; }];
let methods = [ let methods = [
InterfaceMethod<[{ InterfaceMethod<[{
Returns a clone of this type with the given shape and element Returns a clone of this type with the given shape and element
type. If a shape is not provided, the current shape of the type is used. type. If a shape is not provided, the current shape of the type is used.
}], }],
"::mlir::ShapedType", "cloneWith", (ins "::mlir::ShapedType", "cloneWith", (ins
"::llvm::Optional<::llvm::ArrayRef<int64_t>>":$shape, "::std::optional<::llvm::ArrayRef<int64_t>>":$shape,
"::mlir::Type":$elementType "::mlir::Type":$elementType
)>, )>,
InterfaceMethod<[{ InterfaceMethod<[{
Returns the element type of this shaped type. Returns the element type of this shaped type.
}], }],
"::mlir::Type", "getElementType">, "::mlir::Type", "getElementType">,
▲ Show 20 Lines • Show All 115 Lines • Show Last 20 Lines

mlir/include/mlir/IR/BuiltinTypes.h

Show First 20 Lines • Show All 84 Lines • ▼ Show 20 Linespublic:
/// Returns if this type is ranked, i.e. it has a known number of dimensions. /// Returns if this type is ranked, i.e. it has a known number of dimensions.
bool hasRank() const; bool hasRank() const;
/// Returns the shape of this tensor type. /// Returns the shape of this tensor type.
ArrayRef<int64_t> getShape() const; ArrayRef<int64_t> getShape() const;
/// Clone this type with the given shape and element type. If the /// Clone this type with the given shape and element type. If the
/// provided shape is `None`, the current shape of the type is used. /// provided shape is `None`, the current shape of the type is used.
TensorType cloneWith(Optional<ArrayRef<int64_t>> shape, TensorType cloneWith(std::optional<ArrayRef<int64_t>> shape,
Type elementType) const; Type elementType) const;
/// Return true if the specified element type is ok in a tensor. /// Return true if the specified element type is ok in a tensor.
static bool isValidElementType(Type type); static bool isValidElementType(Type type);
/// Methods for support type inquiry through isa, cast, and dyn_cast. /// Methods for support type inquiry through isa, cast, and dyn_cast.
static bool classof(Type type); static bool classof(Type type);
Show All 19 Linespublic:
/// Returns if this type is ranked, i.e. it has a known number of dimensions. /// Returns if this type is ranked, i.e. it has a known number of dimensions.
bool hasRank() const; bool hasRank() const;
/// Returns the shape of this memref type. /// Returns the shape of this memref type.
ArrayRef<int64_t> getShape() const; ArrayRef<int64_t> getShape() const;
/// Clone this type with the given shape and element type. If the /// Clone this type with the given shape and element type. If the
/// provided shape is `None`, the current shape of the type is used. /// provided shape is `None`, the current shape of the type is used.
BaseMemRefType cloneWith(Optional<ArrayRef<int64_t>> shape, BaseMemRefType cloneWith(std::optional<ArrayRef<int64_t>> shape,
Type elementType) const; Type elementType) const;
/// Return true if the specified element type is ok in a memref. /// Return true if the specified element type is ok in a memref.
static bool isValidElementType(Type type); static bool isValidElementType(Type type);
/// Methods for support type inquiry through isa, cast, and dyn_cast. /// Methods for support type inquiry through isa, cast, and dyn_cast.
static bool classof(Type type); static bool classof(Type type);
▲ Show 20 Lines • Show All 194 Lines • ▼ Show 20 Lines
/// Given an `originalShape` and a `reducedShape` assumed to be a subset of /// Given an `originalShape` and a `reducedShape` assumed to be a subset of
/// `originalShape` with some `1` entries erased, return the set of indices /// `originalShape` with some `1` entries erased, return the set of indices
/// that specifies which of the entries of `originalShape` are dropped to obtain /// that specifies which of the entries of `originalShape` are dropped to obtain
/// `reducedShape`. The returned mask can be applied as a projection to /// `reducedShape`. The returned mask can be applied as a projection to
/// `originalShape` to obtain the `reducedShape`. This mask is useful to track /// `originalShape` to obtain the `reducedShape`. This mask is useful to track
/// which dimensions must be kept when e.g. compute MemRef strides under /// which dimensions must be kept when e.g. compute MemRef strides under
/// rank-reducing operations. Return std::nullopt if reducedShape cannot be /// rank-reducing operations. Return std::nullopt if reducedShape cannot be
/// obtained by dropping only `1` entries in `originalShape`. /// obtained by dropping only `1` entries in `originalShape`.
llvm::Optional<llvm::SmallDenseSet<unsigned>> std::optional<llvm::SmallDenseSet<unsigned>>
computeRankReductionMask(ArrayRef<int64_t> originalShape, computeRankReductionMask(ArrayRef<int64_t> originalShape,
ArrayRef<int64_t> reducedShape); ArrayRef<int64_t> reducedShape);
/// Enum that captures information related to verifier error conditions on /// Enum that captures information related to verifier error conditions on
/// slice insert/extract type of ops. /// slice insert/extract type of ops.
enum class SliceVerificationResult { enum class SliceVerificationResult {
Success, Success,
RankTooLarge, RankTooLarge,
▲ Show 20 Lines • Show All 129 Lines • Show Last 20 Lines

mlir/include/mlir/IR/BuiltinTypes.td

Show First 20 Lines • Show All 1,025 Lines • ▼ Show 20 Lines let extraClassDeclaration = [{
/// Return null if the scaled element type cannot be represented. /// Return null if the scaled element type cannot be represented.
VectorType scaleElementBitwidth(unsigned scale); VectorType scaleElementBitwidth(unsigned scale);
/// Returns if this type is ranked (always true). /// Returns if this type is ranked (always true).
bool hasRank() const { return true; } bool hasRank() const { return true; }
/// Clone this vector type with the given shape and element type. If the /// Clone this vector type with the given shape and element type. If the
/// provided shape is `None`, the current shape of the type is used. /// provided shape is `None`, the current shape of the type is used.
VectorType cloneWith(Optional<ArrayRef<int64_t>> shape, VectorType cloneWith(std::optional<ArrayRef<int64_t>> shape,
Type elementType) const; Type elementType) const;
}]; }];
let skipDefaultBuilders = 1; let skipDefaultBuilders = 1;
let genVerifyDecl = 1; let genVerifyDecl = 1;
} }
#endif // BUILTIN_TYPES #endif // BUILTIN_TYPES

mlir/include/mlir/IR/Diagnostics.h

Show First 20 Lines • Show All 481 Lines • ▼ Show 20 Lines
InFlightDiagnostic emitRemark(Location loc, const Twine &message); InFlightDiagnostic emitRemark(Location loc, const Twine &message);
/// Overloads of the above emission functions that take an optionally null /// Overloads of the above emission functions that take an optionally null
/// location. If the location is null, no diagnostic is emitted and a failure is /// location. If the location is null, no diagnostic is emitted and a failure is
/// returned. Given that the provided location may be null, these methods take /// returned. Given that the provided location may be null, these methods take
/// the diagnostic arguments directly instead of relying on the returned /// the diagnostic arguments directly instead of relying on the returned
/// InFlightDiagnostic. /// InFlightDiagnostic.
template <typename... Args> template <typename... Args>
LogicalResult emitOptionalError(Optional<Location> loc, Args &&...args) { LogicalResult emitOptionalError(std::optional<Location> loc, Args &&...args) {
if (loc) if (loc)
return emitError(*loc).append(std::forward<Args>(args)...); return emitError(*loc).append(std::forward<Args>(args)...);
return failure(); return failure();
} }
template <typename... Args> template <typename... Args>
LogicalResult emitOptionalWarning(Optional<Location> loc, Args &&...args) { LogicalResult emitOptionalWarning(std::optional<Location> loc, Args &&...args) {
if (loc) if (loc)
return emitWarning(*loc).append(std::forward<Args>(args)...); return emitWarning(*loc).append(std::forward<Args>(args)...);
return failure(); return failure();
} }
template <typename... Args> template <typename... Args>
LogicalResult emitOptionalRemark(Optional<Location> loc, Args &&...args) { LogicalResult emitOptionalRemark(std::optional<Location> loc, Args &&...args) {
if (loc) if (loc)
return emitRemark(*loc).append(std::forward<Args>(args)...); return emitRemark(*loc).append(std::forward<Args>(args)...);
return failure(); return failure();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ScopedDiagnosticHandler // ScopedDiagnosticHandler
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 169 Lines • Show Last 20 Lines

mlir/include/mlir/IR/Dialect.h

Show First 20 Lines • Show All 109 Lines • ▼ Show 20 Linespublic:
virtual void printType(Type, DialectAsmPrinter &) const { virtual void printType(Type, DialectAsmPrinter &) const {
llvm_unreachable("dialect has no registered type printing hook"); llvm_unreachable("dialect has no registered type printing hook");
} }
/// Return the hook to parse an operation registered to this dialect, if any. /// Return the hook to parse an operation registered to this dialect, if any.
/// By default this will lookup for registered operations and return the /// By default this will lookup for registered operations and return the
/// `parse()` method registered on the RegisteredOperationName. Dialects can /// `parse()` method registered on the RegisteredOperationName. Dialects can
/// override this behavior and handle unregistered operations as well. /// override this behavior and handle unregistered operations as well.
virtual Optional<ParseOpHook> getParseOperationHook(StringRef opName) const; virtual std::optional<ParseOpHook>
getParseOperationHook(StringRef opName) const;
/// Print an operation registered to this dialect. /// Print an operation registered to this dialect.
/// This hook is invoked for registered operation which don't override the /// This hook is invoked for registered operation which don't override the
/// `print()` method to define their own custom assembly. /// `print()` method to define their own custom assembly.
virtual llvm::unique_function<void(Operation *, OpAsmPrinter &printer)> virtual llvm::unique_function<void(Operation *, OpAsmPrinter &printer)>
getOperationPrinter(Operation *op) const; getOperationPrinter(Operation *op) const;
//===--------------------------------------------------------------------===// //===--------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 224 Lines • Show Last 20 Lines

mlir/include/mlir/IR/EnumAttr.td

Show First 20 Lines • Show All 124 Lines • ▼ Show 20 Linesclass EnumAttrInfo<
// The underlying type for the C++ enum class. An empty string mean the // The underlying type for the C++ enum class. An empty string mean the
// underlying type is not explicitly specified. // underlying type is not explicitly specified.
string underlyingType = ""; string underlyingType = "";
// The name of the utility function that converts a value of the underlying // The name of the utility function that converts a value of the underlying
// type to the corresponding symbol. It will have the following signature: // type to the corresponding symbol. It will have the following signature:
// //
// ```c++ // ```c++
// llvm::Optional<<qualified-enum-class-name>> <fn-name>(<underlying-type>); // std::optional<<qualified-enum-class-name>> <fn-name>(<underlying-type>);
// ``` // ```
string underlyingToSymbolFnName = "symbolize" # name; string underlyingToSymbolFnName = "symbolize" # name;
// The name of the utility function that converts a string to the // The name of the utility function that converts a string to the
// corresponding symbol. It will have the following signature: // corresponding symbol. It will have the following signature:
// //
// ```c++ // ```c++
// llvm::Optional<<qualified-enum-class-name>> <fn-name>(llvm::StringRef); // std::optional<<qualified-enum-class-name>> <fn-name>(llvm::StringRef);
// ``` // ```
string stringToSymbolFnName = "symbolize" # name; string stringToSymbolFnName = "symbolize" # name;
// The name of the utility function that converts a symbol to the // The name of the utility function that converts a symbol to the
// corresponding string. It will have the following signature: // corresponding string. It will have the following signature:
// //
// ```c++ // ```c++
// <return-type> <fn-name>(<qualified-enum-class-name>); // <return-type> <fn-name>(<qualified-enum-class-name>);
▲ Show 20 Lines • Show All 263 Lines • Show Last 20 Lines

mlir/include/mlir/IR/OpBase.td

Show First 20 Lines • Show All 1,027 Lines • ▼ Show 20 Lines
} }
// Decorates an attribute as optional. The return type of the generated // Decorates an attribute as optional. The return type of the generated
// attribute accessor method will be Optional<>. // attribute accessor method will be Optional<>.
class OptionalAttr<Attr attr> : Attr<attr.predicate, attr.summary> { class OptionalAttr<Attr attr> : Attr<attr.predicate, attr.summary> {
// Rewrite the attribute to be optional. // Rewrite the attribute to be optional.
// Note: this has to be kept up to date with Attr above. // Note: this has to be kept up to date with Attr above.
let storageType = attr.storageType; let storageType = attr.storageType;
let returnType = "::llvm::Optional<" # attr.returnType #">"; let returnType = "::std::optional<" # attr.returnType #">";
let convertFromStorage = "$_self ? " # returnType # "(" # let convertFromStorage = "$_self ? " # returnType # "(" #
attr.convertFromStorage # ") : (::std::nullopt)"; attr.convertFromStorage # ") : (::std::nullopt)";
let valueType = attr.valueType; let valueType = attr.valueType;
let isOptional = 1; let isOptional = 1;
let baseAttr = attr; let baseAttr = attr;
} }
▲ Show 20 Lines • Show All 1,446 Lines • Show Last 20 Lines

mlir/include/mlir/Interfaces/ControlFlowInterfaces.h

Show First 20 Lines • Show All 122 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// BranchOpInterface // BranchOpInterface
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace detail { namespace detail {
/// Return the `BlockArgument` corresponding to operand `operandIndex` in some /// Return the `BlockArgument` corresponding to operand `operandIndex` in some
/// successor if `operandIndex` is within the range of `operands`, or /// successor if `operandIndex` is within the range of `operands`, or
/// std::nullopt if `operandIndex` isn't a successor operand index. /// std::nullopt if `operandIndex` isn't a successor operand index.
Optional<BlockArgument> std::optional<BlockArgument>
getBranchSuccessorArgument(const SuccessorOperands &operands, getBranchSuccessorArgument(const SuccessorOperands &operands,
unsigned operandIndex, Block *successor); unsigned operandIndex, Block *successor);
/// Verify that the given operands match those of the given successor block. /// Verify that the given operands match those of the given successor block.
LogicalResult verifyBranchSuccessorOperands(Operation *op, unsigned succNo, LogicalResult verifyBranchSuccessorOperands(Operation *op, unsigned succNo,
const SuccessorOperands &operands); const SuccessorOperands &operands);
} // namespace detail } // namespace detail
Show All 19 Lines
/// terminator operations in the blocks within this region. /// terminator operations in the blocks within this region.
class RegionSuccessor { class RegionSuccessor {
public: public:
/// Initialize a successor that branches to another region of the parent /// Initialize a successor that branches to another region of the parent
/// operation. /// operation.
RegionSuccessor(Region *region, Block::BlockArgListType regionInputs = {}) RegionSuccessor(Region *region, Block::BlockArgListType regionInputs = {})
: region(region), inputs(regionInputs) {} : region(region), inputs(regionInputs) {}
/// Initialize a successor that branches back to/out of the parent operation. /// Initialize a successor that branches back to/out of the parent operation.
RegionSuccessor(Optional<Operation::result_range> results = {}) RegionSuccessor(Operation::result_range results)
: inputs(results ? ValueRange(*results) : ValueRange()) {} : inputs(ValueRange(results)) {}
/// Constructor with no arguments.
RegionSuccessor() : inputs(ValueRange()) {}
/// Return the given region successor. Returns nullptr if the successor is the /// Return the given region successor. Returns nullptr if the successor is the
/// parent operation. /// parent operation.
Region *getSuccessor() const { return region; } Region *getSuccessor() const { return region; }
/// Return true if the successor is the parent operation. /// Return true if the successor is the parent operation.
bool isParent() const { return region == nullptr; } bool isParent() const { return region == nullptr; }
/// Return the inputs to the successor that are remapped by the exit values of /// Return the inputs to the successor that are remapped by the exit values of
/// the current region. /// the current region.
ValueRange getSuccessorInputs() const { return inputs; } ValueRange getSuccessorInputs() const { return inputs; }
private: private:
Region *region{nullptr}; Region *region{nullptr};
ValueRange inputs; ValueRange inputs;
}; };
/// This class represents upper and lower bounds on the number of times a region /// This class represents upper and lower bounds on the number of times a region
/// of a `RegionBranchOpInterface` can be invoked. The lower bound is at least /// of a `RegionBranchOpInterface` can be invoked. The lower bound is at least
/// zero, but the upper bound may not be known. /// zero, but the upper bound may not be known.
class InvocationBounds { class InvocationBounds {
public: public:
/// Create invocation bounds. The lower bound must be at least 0 and only the /// Create invocation bounds. The lower bound must be at least 0 and only the
/// upper bound can be unknown. /// upper bound can be unknown.
InvocationBounds(unsigned lb, Optional<unsigned> ub) : lower(lb), upper(ub) { InvocationBounds(unsigned lb, std::optional<unsigned> ub)
: lower(lb), upper(ub) {
assert((!ub || ub >= lb) && "upper bound cannot be less than lower bound"); assert((!ub || ub >= lb) && "upper bound cannot be less than lower bound");
} }
/// Return the lower bound. /// Return the lower bound.
unsigned getLowerBound() const { return lower; } unsigned getLowerBound() const { return lower; }
/// Return the upper bound. /// Return the upper bound.
Optional<unsigned> getUpperBound() const { return upper; } std::optional<unsigned> getUpperBound() const { return upper; }
/// Returns the unknown invocation bounds, i.e., there is no information on /// Returns the unknown invocation bounds, i.e., there is no information on
/// how many times a region may be invoked. /// how many times a region may be invoked.
static InvocationBounds getUnknown() { return {0, std::nullopt}; } static InvocationBounds getUnknown() { return {0, std::nullopt}; }
private: private:
/// The minimum number of times the successor region will be invoked. /// The minimum number of times the successor region will be invoked.
unsigned lower; unsigned lower;
/// The maximum number of times the successor region will be invoked or /// The maximum number of times the successor region will be invoked or
/// `std::nullopt` if an upper bound is not known. /// `std::nullopt` if an upper bound is not known.
Optional<unsigned> upper; std::optional<unsigned> upper;
}; };
/// Return `true` if `a` and `b` are in mutually exclusive regions as per /// Return `true` if `a` and `b` are in mutually exclusive regions as per
/// RegionBranchOpInterface. /// RegionBranchOpInterface.
bool insideMutuallyExclusiveRegions(Operation *a, Operation *b); bool insideMutuallyExclusiveRegions(Operation *a, Operation *b);
/// Return the first enclosing region of the given op that may be executed /// Return the first enclosing region of the given op that may be executed
/// repetitively as per RegionBranchOpInterface or `nullptr` if no such region /// repetitively as per RegionBranchOpInterface or `nullptr` if no such region
Show All 15 Lines
/// Returns the mutable operands that are passed to the region with the given /// Returns the mutable operands that are passed to the region with the given
/// `regionIndex`. If the operation does not implement the /// `regionIndex`. If the operation does not implement the
/// `RegionBranchTerminatorOpInterface` and is not marked as `ReturnLike`, the /// `RegionBranchTerminatorOpInterface` and is not marked as `ReturnLike`, the
/// result will be `std::nullopt`. In all other cases, the resulting /// result will be `std::nullopt`. In all other cases, the resulting
/// `OperandRange` represents all operands that are passed to the specified /// `OperandRange` represents all operands that are passed to the specified
/// successor region. If `regionIndex` is `std::nullopt`, all operands that are /// successor region. If `regionIndex` is `std::nullopt`, all operands that are
/// passed to the parent operation will be returned. /// passed to the parent operation will be returned.
Optional<MutableOperandRange> std::optional<MutableOperandRange>
getMutableRegionBranchSuccessorOperands(Operation *operation, getMutableRegionBranchSuccessorOperands(Operation *operation,
Optional<unsigned> regionIndex); std::optional<unsigned> regionIndex);
/// Returns the read only operands that are passed to the region with the given /// Returns the read only operands that are passed to the region with the given
/// `regionIndex`. See `getMutableRegionBranchSuccessorOperands` for more /// `regionIndex`. See `getMutableRegionBranchSuccessorOperands` for more
/// information. /// information.
Optional<OperandRange> std::optional<OperandRange>
getRegionBranchSuccessorOperands(Operation *operation, getRegionBranchSuccessorOperands(Operation *operation,
Optional<unsigned> regionIndex); std::optional<unsigned> regionIndex);
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ControlFlow Traits // ControlFlow Traits
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace OpTrait { namespace OpTrait {
/// This trait indicates that a terminator operation is "return-like". This /// This trait indicates that a terminator operation is "return-like". This
/// means that it exits its current region and forwards its operands as "exit" /// means that it exits its current region and forwards its operands as "exit"
Show All 26 Lines

mlir/include/mlir/Interfaces/ControlFlowInterfaces.td

Show First 20 Lines • Show All 64 Lines • ▼ Show 20 Lines InterfaceMethod<[{
"::mlir::SuccessorOperands", "getSuccessorOperands", "::mlir::SuccessorOperands", "getSuccessorOperands",
(ins "unsigned":$index) (ins "unsigned":$index)
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
Returns the `BlockArgument` corresponding to operand `operandIndex` in Returns the `BlockArgument` corresponding to operand `operandIndex` in
some successor, or None if `operandIndex` isn't a successor operand some successor, or None if `operandIndex` isn't a successor operand
index. index.
}], }],
"::llvm::Optional<::mlir::BlockArgument>", "getSuccessorBlockArgument", "::std::optional<::mlir::BlockArgument>", "getSuccessorBlockArgument",
(ins "unsigned":$operandIndex), [{ (ins "unsigned":$operandIndex), [{
::mlir::Operation *opaqueOp = $_op; ::mlir::Operation *opaqueOp = $_op;
for (unsigned i = 0, e = opaqueOp->getNumSuccessors(); i != e; ++i) { for (unsigned i = 0, e = opaqueOp->getNumSuccessors(); i != e; ++i) {
if (::llvm::Optional<::mlir::BlockArgument> arg = ::mlir::detail::getBranchSuccessorArgument( if (::std::optional<::mlir::BlockArgument> arg = ::mlir::detail::getBranchSuccessorArgument(
$_op.getSuccessorOperands(i), operandIndex, $_op.getSuccessorOperands(i), operandIndex,
opaqueOp->getSuccessor(i))) opaqueOp->getSuccessor(i)))
return arg; return arg;
} }
return ::std::nullopt; return ::std::nullopt;
}] }]
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Lines InterfaceMethod<[{
Returns the operands of this operation used as the entry arguments when Returns the operands of this operation used as the entry arguments when
entering the region at `index`, which was specified as a successor of entering the region at `index`, which was specified as a successor of
this operation by `getSuccessorRegions`, or the operands forwarded to this operation by `getSuccessorRegions`, or the operands forwarded to
the operation's results when it branches back to itself. These operands the operation's results when it branches back to itself. These operands
should correspond 1-1 with the successor inputs specified in should correspond 1-1 with the successor inputs specified in
`getSuccessorRegions`. `getSuccessorRegions`.
}], }],
"::mlir::OperandRange", "getSuccessorEntryOperands", "::mlir::OperandRange", "getSuccessorEntryOperands",
(ins "::llvm::Optional<unsigned>":$index), [{}], (ins "::std::optional<unsigned>":$index), [{}],
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
auto operandEnd = this->getOperation()->operand_end(); auto operandEnd = this->getOperation()->operand_end();
return ::mlir::OperandRange(operandEnd, operandEnd); return ::mlir::OperandRange(operandEnd, operandEnd);
}] }]
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
Returns the viable successors of a region at `index`, or the possible Returns the viable successors of a region at `index`, or the possible
successors when branching from the parent op if `index` is None. These successors when branching from the parent op if `index` is None. These
are the regions that may be selected during the flow of control. If are the regions that may be selected during the flow of control. If
`index` is None, `operands` is a set of optional attributes that `index` is None, `operands` is a set of optional attributes that
either correspond to a constant value for each operand of this either correspond to a constant value for each operand of this
operation, or null if that operand is not a constant. If `index` is operation, or null if that operand is not a constant. If `index` is
valid, `operands` corresponds to the entry values of the region at valid, `operands` corresponds to the entry values of the region at
`index`. Only a region, i.e. a valid `index`, may use the parent `index`. Only a region, i.e. a valid `index`, may use the parent
operation as a successor. This method allows for describing which operation as a successor. This method allows for describing which
regions may be executed when entering an operation, and which regions regions may be executed when entering an operation, and which regions
are executed after having executed another region of the parent op. The are executed after having executed another region of the parent op. The
successor region must be non-empty. successor region must be non-empty.
}], }],
"void", "getSuccessorRegions", "void", "getSuccessorRegions",
(ins "::llvm::Optional<unsigned>":$index, (ins "::std::optional<unsigned>":$index,
"::llvm::ArrayRef<::mlir::Attribute>":$operands, "::llvm::ArrayRef<::mlir::Attribute>":$operands,
"::llvm::SmallVectorImpl<::mlir::RegionSuccessor> &":$regions) "::llvm::SmallVectorImpl<::mlir::RegionSuccessor> &":$regions)
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
Populates `invocationBounds` with the minimum and maximum number of Populates `invocationBounds` with the minimum and maximum number of
times this operation will invoke the attached regions (assuming the times this operation will invoke the attached regions (assuming the
regions yield normally, i.e. do not abort or invoke an infinite loop). regions yield normally, i.e. do not abort or invoke an infinite loop).
The minimum number of invocations is at least 0. If the maximum number The minimum number of invocations is at least 0. If the maximum number
Show All 30 Lines let verify = [{
static_assert(!ConcreteOp::template hasTrait<OpTrait::ZeroRegions>(), static_assert(!ConcreteOp::template hasTrait<OpTrait::ZeroRegions>(),
"expected operation to have non-zero regions"); "expected operation to have non-zero regions");
return detail::verifyTypesAlongControlFlowEdges($_op); return detail::verifyTypesAlongControlFlowEdges($_op);
}]; }];
let verifyWithRegions = 1; let verifyWithRegions = 1;
let extraClassDeclaration = [{ let extraClassDeclaration = [{
/// Convenience helper in case none of the operands is known. /// Convenience helper in case none of the operands is known.
void getSuccessorRegions(Optional<unsigned> index, void getSuccessorRegions(std::optional<unsigned> index,
SmallVectorImpl<RegionSuccessor> &regions); SmallVectorImpl<RegionSuccessor> &regions);
/// Return `true` if control flow originating from the given region may /// Return `true` if control flow originating from the given region may
/// eventually branch back to the same region. (Maybe after passing through /// eventually branch back to the same region. (Maybe after passing through
/// other regions.) /// other regions.)
bool isRepetitiveRegion(unsigned index); bool isRepetitiveRegion(unsigned index);
}]; }];
} }
Show All 14 Linesdef RegionBranchTerminatorOpInterface :
let methods = [ let methods = [
InterfaceMethod<[{ InterfaceMethod<[{
Returns a mutable range of operands that are semantically "returned" by Returns a mutable range of operands that are semantically "returned" by
passing them to the region successor given by `index`. If `index` is passing them to the region successor given by `index`. If `index` is
None, this function returns the operands that are passed as a result to None, this function returns the operands that are passed as a result to
the parent operation. the parent operation.
}], }],
"::mlir::MutableOperandRange", "getMutableSuccessorOperands", "::mlir::MutableOperandRange", "getMutableSuccessorOperands",
(ins "::llvm::Optional<unsigned>":$index) (ins "::std::optional<unsigned>":$index)
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
Returns a range of operands that are semantically "returned" by passing Returns a range of operands that are semantically "returned" by passing
them to the region successor given by `index`. If `index` is None, this them to the region successor given by `index`. If `index` is None, this
function returns the operands that are passed as a result to the parent function returns the operands that are passed as a result to the parent
operation. operation.
}], }],
"::mlir::OperandRange", "getSuccessorOperands", "::mlir::OperandRange", "getSuccessorOperands",
(ins "::llvm::Optional<unsigned>":$index), [{}], (ins "::std::optional<unsigned>":$index), [{}],
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
return $_op.getMutableSuccessorOperands(index); return $_op.getMutableSuccessorOperands(index);
}] }]
> >
]; ];
let verify = [{ let verify = [{
static_assert(ConcreteOp::template hasTrait<OpTrait::IsTerminator>(), static_assert(ConcreteOp::template hasTrait<OpTrait::IsTerminator>(),
Show All 17 Lines

mlir/include/mlir/Interfaces/InferTypeOpInterface.h

Show First 20 Lines • Show All 229 Lines • ▼ Show 20 Lines
namespace detail { namespace detail {
// Helper function to infer return tensor returns types given element and // Helper function to infer return tensor returns types given element and
// shape inference function. // shape inference function.
// //
// TODO: Consider generating typedefs for trait member functions if this usage // TODO: Consider generating typedefs for trait member functions if this usage
// becomes more common. // becomes more common.
LogicalResult inferReturnTensorTypes( LogicalResult inferReturnTensorTypes(
function_ref<LogicalResult( function_ref<
MLIRContext *, Optional<Location> location, ValueShapeRange operands, LogicalResult(MLIRContext *, std::optional<Location> location,
DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes,
RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &retComponents)> SmallVectorImpl<ShapedTypeComponents> &retComponents)>
componentTypeFn, componentTypeFn,
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes); SmallVectorImpl<Type> &inferredReturnTypes);
/// Verifies that the inferred result types match the actual result types for /// Verifies that the inferred result types match the actual result types for
/// the op. Precondition: op implements InferTypeOpInterface. /// the op. Precondition: op implements InferTypeOpInterface.
LogicalResult verifyInferredResultTypes(Operation *op); LogicalResult verifyInferredResultTypes(Operation *op);
} // namespace detail } // namespace detail
Show All 15 Lines
/// Less strict is possible (e.g., implements inferReturnTypeComponents and /// Less strict is possible (e.g., implements inferReturnTypeComponents and
/// these always populates all element types and shapes or fails, but this\ /// these always populates all element types and shapes or fails, but this\
/// trait is currently only used where the interfaces are, so keep it /// trait is currently only used where the interfaces are, so keep it
/// restricted for now). /// restricted for now).
template <typename ConcreteType> template <typename ConcreteType>
class InferTensorType : public TraitBase<ConcreteType, InferTensorType> { class InferTensorType : public TraitBase<ConcreteType, InferTensorType> {
public: public:
static LogicalResult static LogicalResult
inferReturnTypes(MLIRContext *context, Optional<Location> location, inferReturnTypes(MLIRContext *context, std::optional<Location> location,
ValueRange operands, DictionaryAttr attributes, ValueRange operands, DictionaryAttr attributes,
RegionRange regions, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
static_assert( static_assert(
ConcreteType::template hasTrait<InferShapedTypeOpInterface::Trait>(), ConcreteType::template hasTrait<InferShapedTypeOpInterface::Trait>(),
"requires InferShapedTypeOpInterface to ensure succesful invocation"); "requires InferShapedTypeOpInterface to ensure succesful invocation");
static_assert( static_assert(
ConcreteType::template hasTrait<InferTypeOpInterface::Trait>(), ConcreteType::template hasTrait<InferTypeOpInterface::Trait>(),
Show All 11 Lines

mlir/include/mlir/Interfaces/InferTypeOpInterface.td

Show All 35 Lines StaticInterfaceMethod<
which an Operation would be created (e.g., as used in Operation::create) which an Operation would be created (e.g., as used in Operation::create)
and the regions of the op. Be aware that this method is supposed to be and the regions of the op. Be aware that this method is supposed to be
called with valid arguments, e.g., operands are verified, or it may result called with valid arguments, e.g., operands are verified, or it may result
in an undefined behavior. in an undefined behavior.
}], }],
/*retTy=*/"::mlir::LogicalResult", /*retTy=*/"::mlir::LogicalResult",
/*methodName=*/"inferReturnTypes", /*methodName=*/"inferReturnTypes",
/*args=*/(ins "::mlir::MLIRContext *":$context, /*args=*/(ins "::mlir::MLIRContext *":$context,
"::llvm::Optional<::mlir::Location>":$location, "::std::optional<::mlir::Location>":$location,
"::mlir::ValueRange":$operands, "::mlir::ValueRange":$operands,
"::mlir::DictionaryAttr":$attributes, "::mlir::DictionaryAttr":$attributes,
"::mlir::RegionRange":$regions, "::mlir::RegionRange":$regions,
"::llvm::SmallVectorImpl<::mlir::Type>&":$inferredReturnTypes) "::llvm::SmallVectorImpl<::mlir::Type>&":$inferredReturnTypes)
>, >,
StaticInterfaceMethod< StaticInterfaceMethod<
/*desc=*/[{Refine the return types that an op would generate. /*desc=*/[{Refine the return types that an op would generate.
Show All 14 Lines StaticInterfaceMethod<
that should just be returned but not verified. that should just be returned but not verified.
Be aware that this method is supposed to be called with valid arguments, Be aware that this method is supposed to be called with valid arguments,
e.g., operands are verified, or it may result in an undefined behavior. e.g., operands are verified, or it may result in an undefined behavior.
}], }],
/*retTy=*/"::mlir::LogicalResult", /*retTy=*/"::mlir::LogicalResult",
/*methodName=*/"refineReturnTypes", /*methodName=*/"refineReturnTypes",
/*args=*/(ins "::mlir::MLIRContext *":$context, /*args=*/(ins "::mlir::MLIRContext *":$context,
"::llvm::Optional<::mlir::Location>":$location, "::std::optional<::mlir::Location>":$location,
"::mlir::ValueRange":$operands, "::mlir::ValueRange":$operands,
"::mlir::DictionaryAttr":$attributes, "::mlir::DictionaryAttr":$attributes,
"::mlir::RegionRange":$regions, "::mlir::RegionRange":$regions,
"::llvm::SmallVectorImpl<::mlir::Type>&":$returnTypes), "::llvm::SmallVectorImpl<::mlir::Type>&":$returnTypes),
/*methodBody=*/[{}], /*methodBody=*/[{}],
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
llvm::SmallVector<Type, 4> inferredReturnTypes; llvm::SmallVector<Type, 4> inferredReturnTypes;
if (failed(ConcreteOp::inferReturnTypes(context, location, operands, if (failed(ConcreteOp::inferReturnTypes(context, location, operands,
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Lines StaticInterfaceMethod<
Unknown (e.g., unranked) shape and nullptrs for element type and attribute Unknown (e.g., unranked) shape and nullptrs for element type and attribute
may be returned by this function while returning success. E.g., partial may be returned by this function while returning success. E.g., partial
population of components is not error condition. population of components is not error condition.
}], }],
/*retTy=*/"::mlir::LogicalResult", /*retTy=*/"::mlir::LogicalResult",
/*methodName=*/"inferReturnTypeComponents", /*methodName=*/"inferReturnTypeComponents",
/*args=*/(ins "::mlir::MLIRContext*":$context, /*args=*/(ins "::mlir::MLIRContext*":$context,
"::llvm::Optional<::mlir::Location>":$location, "::std::optional<::mlir::Location>":$location,
"::mlir::ValueShapeRange":$operands, "::mlir::ValueShapeRange":$operands,
"::mlir::DictionaryAttr":$attributes, "::mlir::DictionaryAttr":$attributes,
"::mlir::RegionRange":$regions, "::mlir::RegionRange":$regions,
"::llvm::SmallVectorImpl<::mlir::ShapedTypeComponents>&": "::llvm::SmallVectorImpl<::mlir::ShapedTypeComponents>&":
$inferredReturnShapes), $inferredReturnShapes),
/*methodBody=*/[{}], /*methodBody=*/[{}],
/*defaultImplementation=*/[{ return ::mlir::failure(); }] /*defaultImplementation=*/[{ return ::mlir::failure(); }]
>, >,
▲ Show 20 Lines • Show All 81 Lines • Show Last 20 Lines

mlir/include/mlir/Interfaces/LoopLikeInterface.td

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Lines InterfaceMethod<[{
(ins "::mlir::Operation *":$op), [{}], [{ (ins "::mlir::Operation *":$op), [{}], [{
op->moveBefore($_op); op->moveBefore($_op);
}] }]
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
If there is a single induction variable return it, otherwise return If there is a single induction variable return it, otherwise return
std::nullopt. std::nullopt.
}], }],
/*retTy=*/"::llvm::Optional<::mlir::Value>", /*retTy=*/"::std::optional<::mlir::Value>",
/*methodName=*/"getSingleInductionVar", /*methodName=*/"getSingleInductionVar",
/*args=*/(ins), /*args=*/(ins),
/*methodBody=*/"", /*methodBody=*/"",
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
return std::nullopt; return std::nullopt;
}] }]
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
Return the single lower bound value or attribute if it exists, otherwise Return the single lower bound value or attribute if it exists, otherwise
return std::nullopt. return std::nullopt.
}], }],
/*retTy=*/"::llvm::Optional<::mlir::OpFoldResult>", /*retTy=*/"::std::optional<::mlir::OpFoldResult>",
/*methodName=*/"getSingleLowerBound", /*methodName=*/"getSingleLowerBound",
/*args=*/(ins), /*args=*/(ins),
/*methodBody=*/"", /*methodBody=*/"",
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
return std::nullopt; return std::nullopt;
}] }]
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
Return the single step value or attribute if it exists, otherwise Return the single step value or attribute if it exists, otherwise
return std::nullopt. return std::nullopt.
}], }],
/*retTy=*/"::llvm::Optional<::mlir::OpFoldResult>", /*retTy=*/"::std::optional<::mlir::OpFoldResult>",
/*methodName=*/"getSingleStep", /*methodName=*/"getSingleStep",
/*args=*/(ins), /*args=*/(ins),
/*methodBody=*/"", /*methodBody=*/"",
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
return std::nullopt; return std::nullopt;
}] }]
>, >,
InterfaceMethod<[{ InterfaceMethod<[{
Return the single upper bound value or attribute if it exists, otherwise Return the single upper bound value or attribute if it exists, otherwise
return std::nullopt. return std::nullopt.
}], }],
/*retTy=*/"::llvm::Optional<::mlir::OpFoldResult>", /*retTy=*/"::std::optional<::mlir::OpFoldResult>",
/*methodName=*/"getSingleUpperBound", /*methodName=*/"getSingleUpperBound",
/*args=*/(ins), /*args=*/(ins),
/*methodBody=*/"", /*methodBody=*/"",
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
return std::nullopt; return std::nullopt;
}] }]
>, >,
]; ];
} }
#endif // MLIR_INTERFACES_LOOPLIKEINTERFACE #endif // MLIR_INTERFACES_LOOPLIKEINTERFACE

mlir/include/mlir/Interfaces/SideEffectInterfaceBase.td

Show First 20 Lines • Show All 100 Lines • ▼ Show 20 Lines void getEffectsOnValue(::mlir::Value value,
}] # baseEffect # [{>> & effects) { }] # baseEffect # [{>> & effects) {
getEffects(effects); getEffects(effects);
llvm::erase_if(effects, [&](auto &it) { return it.getValue() != value; }); llvm::erase_if(effects, [&](auto &it) { return it.getValue() != value; });
} }
/// Return the effect of the given type `Effect` that is applied to the /// Return the effect of the given type `Effect` that is applied to the
/// given value, or std::nullopt if no effect exists. /// given value, or std::nullopt if no effect exists.
template <typename Effect> template <typename Effect>
::llvm::Optional<::mlir::SideEffects::EffectInstance<}] # baseEffect # [{>> ::std::optional<::mlir::SideEffects::EffectInstance<}] # baseEffect # [{>>
getEffectOnValue(::mlir::Value value) { getEffectOnValue(::mlir::Value value) {
llvm::SmallVector<::mlir::SideEffects::EffectInstance< llvm::SmallVector<::mlir::SideEffects::EffectInstance<
}] # baseEffect # [{>, 4> effects; }] # baseEffect # [{>, 4> effects;
getEffects(effects); getEffects(effects);
auto it = llvm::find_if(effects, [&](auto &it) { auto it = llvm::find_if(effects, [&](auto &it) {
return isa<Effect>(it.getEffect()) && it.getValue() == value; return isa<Effect>(it.getEffect()) && it.getValue() == value;
}); });
if (it == effects.end()) if (it == effects.end())
▲ Show 20 Lines • Show All 69 Lines • Show Last 20 Lines

mlir/include/mlir/Interfaces/VectorInterfaces.td

Show All 22 Linesdef VectorUnrollOpInterface : OpInterface<"VectorUnrollOpInterface"> {
let methods = [ let methods = [
InterfaceMethod< InterfaceMethod<
/*desc=*/[{ /*desc=*/[{
Return the shape ratio of unrolling to the target vector shape Return the shape ratio of unrolling to the target vector shape
`targetShape`. Return `None` if the op cannot be unrolled to the target `targetShape`. Return `None` if the op cannot be unrolled to the target
vector shape. vector shape.
}], }],
/*retTy=*/"::llvm::Optional<::llvm::SmallVector<int64_t, 4>>", /*retTy=*/"::std::optional<::llvm::SmallVector<int64_t, 4>>",
/*methodName=*/"getShapeForUnroll", /*methodName=*/"getShapeForUnroll",
/*args=*/(ins), /*args=*/(ins),
/*methodBody=*/"", /*methodBody=*/"",
/*defaultImplementation=*/[{ /*defaultImplementation=*/[{
assert($_op->getNumResults() == 1); assert($_op->getNumResults() == 1);
auto vt = $_op.getResult().getType(). auto vt = $_op.getResult().getType().
template dyn_cast<::mlir::VectorType>(); template dyn_cast<::mlir::VectorType>();
if (!vt) if (!vt)
▲ Show 20 Lines • Show All 98 Lines • ▼ Show 20 Lines InterfaceMethod<
if ($_op.isBroadcastDim(i)) if ($_op.isBroadcastDim(i))
return true; return true;
} }
return false; return false;
}] }]
>, >,
InterfaceMethod< InterfaceMethod<
/*desc=*/"Return the `in_bounds` boolean ArrayAttr.", /*desc=*/"Return the `in_bounds` boolean ArrayAttr.",
/*retTy=*/"::llvm::Optional<::mlir::ArrayAttr>", /*retTy=*/"::std::optional<::mlir::ArrayAttr>",
/*methodName=*/"in_bounds", /*methodName=*/"in_bounds",
/*args=*/(ins), /*args=*/(ins),
/*methodBody=*/"return $_op.getInBounds();" /*methodBody=*/"return $_op.getInBounds();"
/*defaultImplementation=*/ /*defaultImplementation=*/
>, >,
InterfaceMethod< InterfaceMethod<
/*desc=*/"Return the ShapedType.", /*desc=*/"Return the ShapedType.",
/*retTy=*/"::mlir::ShapedType", /*retTy=*/"::mlir::ShapedType",
▲ Show 20 Lines • Show All 134 Lines • Show Last 20 Lines

mlir/lib/Analysis/AliasAnalysis/LocalAliasAnalysis.cpp

Show All 39 Linesstatic void collectUnderlyingAddressValues(RegionBranchOpInterface branch,
unsigned maxDepth, unsigned maxDepth,
DenseSet<Value> &visited, DenseSet<Value> &visited,
SmallVectorImpl<Value> &output) { SmallVectorImpl<Value> &output) {
// Given the index of a region of the branch (`predIndex`), or std::nullopt to // Given the index of a region of the branch (`predIndex`), or std::nullopt to
// represent the parent operation, try to return the index into the outputs of // represent the parent operation, try to return the index into the outputs of
// this region predecessor that correspond to the input values of `region`. If // this region predecessor that correspond to the input values of `region`. If
// an index could not be found, std::nullopt is returned instead. // an index could not be found, std::nullopt is returned instead.
auto getOperandIndexIfPred = auto getOperandIndexIfPred =
[&](Optional<unsigned> predIndex) -> Optional<unsigned> { [&](std::optional<unsigned> predIndex) -> std::optional<unsigned> {
SmallVector<RegionSuccessor, 2> successors; SmallVector<RegionSuccessor, 2> successors;
branch.getSuccessorRegions(predIndex, successors); branch.getSuccessorRegions(predIndex, successors);
for (RegionSuccessor &successor : successors) { for (RegionSuccessor &successor : successors) {
if (successor.getSuccessor() != region) if (successor.getSuccessor() != region)
continue; continue;
// Check that the successor inputs map to the given input value. // Check that the successor inputs map to the given input value.
ValueRange inputs = successor.getSuccessorInputs(); ValueRange inputs = successor.getSuccessorInputs();
if (inputs.empty()) { if (inputs.empty()) {
Show All 13 Lines for (RegionSuccessor &successor : successors) {
break; break;
} }
return inputIndex - firstInputIndex; return inputIndex - firstInputIndex;
} }
return std::nullopt; return std::nullopt;
}; };
// Check branches from the parent operation. // Check branches from the parent operation.
Optional<unsigned> regionIndex; std::optional<unsigned> regionIndex;
if (region) { if (region) {
// Determine the actual region number from the passed region. // Determine the actual region number from the passed region.
regionIndex = region->getRegionNumber(); regionIndex = region->getRegionNumber();
} }
if (Optional<unsigned> operandIndex = if (std::optional<unsigned> operandIndex =
getOperandIndexIfPred(/*predIndex=*/std::nullopt)) { getOperandIndexIfPred(/*predIndex=*/std::nullopt)) {
collectUnderlyingAddressValues( collectUnderlyingAddressValues(
branch.getSuccessorEntryOperands(regionIndex)[*operandIndex], maxDepth, branch.getSuccessorEntryOperands(regionIndex)[*operandIndex], maxDepth,
visited, output); visited, output);
} }
// Check branches from each child region. // Check branches from each child region.
Operation *op = branch.getOperation(); Operation *op = branch.getOperation();
for (int i = 0, e = op->getNumRegions(); i != e; ++i) { for (int i = 0, e = op->getNumRegions(); i != e; ++i) {
if (Optional<unsigned> operandIndex = getOperandIndexIfPred(i)) { if (std::optional<unsigned> operandIndex = getOperandIndexIfPred(i)) {
for (Block &block : op->getRegion(i)) { for (Block &block : op->getRegion(i)) {
Operation *term = block.getTerminator(); Operation *term = block.getTerminator();
// Try to determine possible region-branch successor operands for the // Try to determine possible region-branch successor operands for the
// current region. // current region.
auto successorOperands = auto successorOperands =
getRegionBranchSuccessorOperands(term, regionIndex); getRegionBranchSuccessorOperands(term, regionIndex);
if (successorOperands) { if (successorOperands) {
collectUnderlyingAddressValues((*successorOperands)[*operandIndex], collectUnderlyingAddressValues((*successorOperands)[*operandIndex],
▲ Show 20 Lines • Show All 105 Lines • ▼ Show 20 Lines
/// Given a value, try to get an allocation effect attached to it. If /// Given a value, try to get an allocation effect attached to it. If
/// successful, `allocEffect` is populated with the effect. If an effect was /// successful, `allocEffect` is populated with the effect. If an effect was
/// found, `allocScopeOp` is also specified if a parent operation of `value` /// found, `allocScopeOp` is also specified if a parent operation of `value`
/// could be identified that bounds the scope of the allocated value; i.e. if /// could be identified that bounds the scope of the allocated value; i.e. if
/// non-null it specifies the parent operation that the allocation does not /// non-null it specifies the parent operation that the allocation does not
/// escape. If no scope is found, `allocScopeOp` is set to nullptr. /// escape. If no scope is found, `allocScopeOp` is set to nullptr.
static LogicalResult static LogicalResult
getAllocEffectFor(Value value, Optional<MemoryEffects::EffectInstance> &effect, getAllocEffectFor(Value value,
std::optional<MemoryEffects::EffectInstance> &effect,
Operation *&allocScopeOp) { Operation *&allocScopeOp) {
// Try to get a memory effect interface for the parent operation. // Try to get a memory effect interface for the parent operation.
Operation *op; Operation *op;
if (BlockArgument arg = value.dyn_cast<BlockArgument>()) if (BlockArgument arg = value.dyn_cast<BlockArgument>())
op = arg.getOwner()->getParentOp(); op = arg.getOwner()->getParentOp();
else else
op = value.cast<OpResult>().getOwner(); op = value.cast<OpResult>().getOwner();
MemoryEffectOpInterface interface = dyn_cast<MemoryEffectOpInterface>(op); MemoryEffectOpInterface interface = dyn_cast<MemoryEffectOpInterface>(op);
Show All 21 LinesgetAllocEffectFor(Value value,
return success(); return success();
} }
/// Given the two values, return their aliasing behavior. /// Given the two values, return their aliasing behavior.
static AliasResult aliasImpl(Value lhs, Value rhs) { static AliasResult aliasImpl(Value lhs, Value rhs) {
if (lhs == rhs) if (lhs == rhs)
return AliasResult::MustAlias; return AliasResult::MustAlias;
Operation *lhsAllocScope = nullptr, *rhsAllocScope = nullptr; Operation *lhsAllocScope = nullptr, *rhsAllocScope = nullptr;
Optional<MemoryEffects::EffectInstance> lhsAlloc, rhsAlloc; std::optional<MemoryEffects::EffectInstance> lhsAlloc, rhsAlloc;
// Handle the case where lhs is a constant. // Handle the case where lhs is a constant.
Attribute lhsAttr, rhsAttr; Attribute lhsAttr, rhsAttr;
if (matchPattern(lhs, m_Constant(&lhsAttr))) { if (matchPattern(lhs, m_Constant(&lhsAttr))) {
// TODO: This is overly conservative. Two matching constants don't // TODO: This is overly conservative. Two matching constants don't
// necessarily map to the same address. For example, if the two values // necessarily map to the same address. For example, if the two values
// correspond to different symbols that both represent a definition. // correspond to different symbols that both represent a definition.
if (matchPattern(rhs, m_Constant(&rhsAttr))) if (matchPattern(rhs, m_Constant(&rhsAttr)))
▲ Show 20 Lines • Show All 137 Lines • Show Last 20 Lines

mlir/lib/Analysis/DataFlow/IntegerRangeAnalysis.cpp

Show First 20 Lines • Show All 159 Lines • ▼ Show 20 Lines if (auto inferrable = dyn_cast<InferIntRangeInterface>(op)) {
inferrable.inferResultRanges(argRanges, joinCallback); inferrable.inferResultRanges(argRanges, joinCallback);
return; return;
} }
/// Given the results of getConstant{Lower,Upper}Bound() or getConstantStep() /// Given the results of getConstant{Lower,Upper}Bound() or getConstantStep()
/// on a LoopLikeInterface return the lower/upper bound for that result if /// on a LoopLikeInterface return the lower/upper bound for that result if
/// possible. /// possible.
auto getLoopBoundFromFold = [&](Optional<OpFoldResult> loopBound, auto getLoopBoundFromFold = [&](std::optional<OpFoldResult> loopBound,
Type boundType, bool getUpper) { Type boundType, bool getUpper) {
unsigned int width = ConstantIntRanges::getStorageBitwidth(boundType); unsigned int width = ConstantIntRanges::getStorageBitwidth(boundType);
if (loopBound.has_value()) { if (loopBound.has_value()) {
if (loopBound->is<Attribute>()) { if (loopBound->is<Attribute>()) {
if (auto bound = if (auto bound =
loopBound->get<Attribute>().dyn_cast_or_null<IntegerAttr>()) loopBound->get<Attribute>().dyn_cast_or_null<IntegerAttr>())
return bound.getValue(); return bound.getValue();
} else if (auto value = loopBound->dyn_cast<Value>()) { } else if (auto value = loopBound->dyn_cast<Value>()) {
const IntegerValueRangeLattice *lattice = const IntegerValueRangeLattice *lattice =
getLatticeElementFor(op, value); getLatticeElementFor(op, value);
if (lattice != nullptr) if (lattice != nullptr)
return getUpper ? lattice->getValue().getValue().smax() return getUpper ? lattice->getValue().getValue().smax()
: lattice->getValue().getValue().smin(); : lattice->getValue().getValue().smin();
} }
} }
// Given the results of getConstant{Lower,Upper}Bound() // Given the results of getConstant{Lower,Upper}Bound()
// or getConstantStep() on a LoopLikeInterface return the lower/upper // or getConstantStep() on a LoopLikeInterface return the lower/upper
// bound // bound
return getUpper ? APInt::getSignedMaxValue(width) return getUpper ? APInt::getSignedMaxValue(width)
: APInt::getSignedMinValue(width); : APInt::getSignedMinValue(width);
}; };
// Infer bounds for loop arguments that have static bounds // Infer bounds for loop arguments that have static bounds
if (auto loop = dyn_cast<LoopLikeOpInterface>(op)) { if (auto loop = dyn_cast<LoopLikeOpInterface>(op)) {
Optional<Value> iv = loop.getSingleInductionVar(); std::optional<Value> iv = loop.getSingleInductionVar();
if (!iv) { if (!iv) {
return SparseDataFlowAnalysis ::visitNonControlFlowArguments( return SparseDataFlowAnalysis ::visitNonControlFlowArguments(
op, successor, argLattices, firstIndex); op, successor, argLattices, firstIndex);
} }
Optional<OpFoldResult> lowerBound = loop.getSingleLowerBound(); std::optional<OpFoldResult> lowerBound = loop.getSingleLowerBound();
Optional<OpFoldResult> upperBound = loop.getSingleUpperBound(); std::optional<OpFoldResult> upperBound = loop.getSingleUpperBound();
Optional<OpFoldResult> step = loop.getSingleStep(); std::optional<OpFoldResult> step = loop.getSingleStep();
APInt min = getLoopBoundFromFold(lowerBound, iv->getType(), APInt min = getLoopBoundFromFold(lowerBound, iv->getType(),
/*getUpper=*/false); /*getUpper=*/false);
APInt max = getLoopBoundFromFold(upperBound, iv->getType(), APInt max = getLoopBoundFromFold(upperBound, iv->getType(),
/*getUpper=*/true); /*getUpper=*/true);
// Assume positivity for uniscoverable steps by way of getUpper = true. // Assume positivity for uniscoverable steps by way of getUpper = true.
APInt stepVal = APInt stepVal =
getLoopBoundFromFold(step, iv->getType(), /*getUpper=*/true); getLoopBoundFromFold(step, iv->getType(), /*getUpper=*/true);
Show All 17 Lines

mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp

Show First 20 Lines • Show All 202 Lines • ▼ Show 20 Lines for (Block::pred_iterator it = block->pred_begin(), e = block->pred_end();
} else { } else {
return setAllToEntryStates(argLattices); return setAllToEntryStates(argLattices);
} }
} }
} }
void AbstractSparseDataFlowAnalysis::visitRegionSuccessors( void AbstractSparseDataFlowAnalysis::visitRegionSuccessors(
ProgramPoint point, RegionBranchOpInterface branch, ProgramPoint point, RegionBranchOpInterface branch,
Optional<unsigned> successorIndex, std::optional<unsigned> successorIndex,
ArrayRef<AbstractSparseLattice *> lattices) { ArrayRef<AbstractSparseLattice *> lattices) {
const auto *predecessors = getOrCreateFor<PredecessorState>(point, point); const auto *predecessors = getOrCreateFor<PredecessorState>(point, point);
assert(predecessors->allPredecessorsKnown() && assert(predecessors->allPredecessorsKnown() &&
"unexpected unresolved region successors"); "unexpected unresolved region successors");
for (Operation *op : predecessors->getKnownPredecessors()) { for (Operation *op : predecessors->getKnownPredecessors()) {
// Get the incoming successor operands. // Get the incoming successor operands.
Optional<OperandRange> operands; std::optional<OperandRange> operands;
// Check if the predecessor is the parent op. // Check if the predecessor is the parent op.
if (op == branch) { if (op == branch) {
operands = branch.getSuccessorEntryOperands(successorIndex); operands = branch.getSuccessorEntryOperands(successorIndex);
// Otherwise, try to deduce the operands from a region return-like op. // Otherwise, try to deduce the operands from a region return-like op.
} else { } else {
if (isRegionReturnLike(op)) if (isRegionReturnLike(op))
operands = getRegionBranchSuccessorOperands(op, successorIndex); operands = getRegionBranchSuccessorOperands(op, successorIndex);
▲ Show 20 Lines • Show All 157 Lines • ▼ Show 20 Lines if (auto branch = dyn_cast<BranchOpInterface>(op)) {
for (auto [index, block] : llvm::enumerate(op->getSuccessors())) { for (auto [index, block] : llvm::enumerate(op->getSuccessors())) {
SuccessorOperands successorOperands = branch.getSuccessorOperands(index); SuccessorOperands successorOperands = branch.getSuccessorOperands(index);
OperandRange forwarded = successorOperands.getForwardedOperands(); OperandRange forwarded = successorOperands.getForwardedOperands();
if (forwarded.size()) { if (forwarded.size()) {
MutableArrayRef<OpOperand> operands = op->getOpOperands().slice( MutableArrayRef<OpOperand> operands = op->getOpOperands().slice(
forwarded.getBeginOperandIndex(), forwarded.size()); forwarded.getBeginOperandIndex(), forwarded.size());
for (OpOperand &operand : operands) { for (OpOperand &operand : operands) {
unaccounted.reset(operand.getOperandNumber()); unaccounted.reset(operand.getOperandNumber());
if (Optional<BlockArgument> blockArg = if (std::optional<BlockArgument> blockArg =
detail::getBranchSuccessorArgument( detail::getBranchSuccessorArgument(
successorOperands, operand.getOperandNumber(), block)) { successorOperands, operand.getOperandNumber(), block)) {
meet(getLatticeElement(operand.get()), meet(getLatticeElement(operand.get()),
*getLatticeElementFor(op, *blockArg)); *getLatticeElementFor(op, *blockArg));
} }
} }
} }
} }
▲ Show 20 Lines • Show All 150 Lines • Show Last 20 Lines

mlir/lib/AsmParser/Parser.cpp

Show First 20 Lines • Show All 1,865 Lines • ▼ Show 20 LinesOperationParser::parseCustomOperation(ArrayRef<ResultRecord> resultIDs) {
StringRef defaultDialect = ""; StringRef defaultDialect = "";
if (auto opInfo = opNameInfo->getRegisteredInfo()) { if (auto opInfo = opNameInfo->getRegisteredInfo()) {
parseAssemblyFn = opInfo->getParseAssemblyFn(); parseAssemblyFn = opInfo->getParseAssemblyFn();
isIsolatedFromAbove = opInfo->hasTrait<OpTrait::IsIsolatedFromAbove>(); isIsolatedFromAbove = opInfo->hasTrait<OpTrait::IsIsolatedFromAbove>();
auto *iface = opInfo->getInterface<OpAsmOpInterface>(); auto *iface = opInfo->getInterface<OpAsmOpInterface>();
if (iface && !iface->getDefaultDialect().empty()) if (iface && !iface->getDefaultDialect().empty())
defaultDialect = iface->getDefaultDialect(); defaultDialect = iface->getDefaultDialect();
} else { } else {
Optional<Dialect::ParseOpHook> dialectHook; std::optional<Dialect::ParseOpHook> dialectHook;
Dialect *dialect = opNameInfo->getDialect(); Dialect *dialect = opNameInfo->getDialect();
if (!dialect) { if (!dialect) {
InFlightDiagnostic diag = InFlightDiagnostic diag =
emitError(opLoc) << "Dialect `" << opNameInfo->getDialectNamespace() emitError(opLoc) << "Dialect `" << opNameInfo->getDialectNamespace()
<< "' not found for custom op '" << originalOpName << "' not found for custom op '" << originalOpName
<< "' "; << "' ";
if (originalOpName != opName) if (originalOpName != opName)
diag << " (tried '" << opName << "' as well)"; diag << " (tried '" << opName << "' as well)";
▲ Show 20 Lines • Show All 779 Lines • Show Last 20 Lines

mlir/lib/CAPI/Interfaces/Interfaces.cpp

Show All 40 LinesMlirLogicalResult mlirInferTypeOpInterfaceInferReturnTypes(
intptr_t nRegions, MlirRegion *regions, MlirTypesCallback callback, intptr_t nRegions, MlirRegion *regions, MlirTypesCallback callback,
void *userData) { void *userData) {
StringRef name(opName.data, opName.length); StringRef name(opName.data, opName.length);
Optional<RegisteredOperationName> info = Optional<RegisteredOperationName> info =
RegisteredOperationName::lookup(name, unwrap(context)); RegisteredOperationName::lookup(name, unwrap(context));
if (!info) if (!info)
return mlirLogicalResultFailure(); return mlirLogicalResultFailure();
llvm::Optional<Location> maybeLocation; std::optional<Location> maybeLocation;
if (!mlirLocationIsNull(location)) if (!mlirLocationIsNull(location))
maybeLocation = unwrap(location); maybeLocation = unwrap(location);
SmallVector<Value> unwrappedOperands; SmallVector<Value> unwrappedOperands;
(void)unwrapList(nOperands, operands, unwrappedOperands); (void)unwrapList(nOperands, operands, unwrappedOperands);
DictionaryAttr attributeDict; DictionaryAttr attributeDict;
if (!mlirAttributeIsNull(attributes)) if (!mlirAttributeIsNull(attributes))
attributeDict = unwrap(attributes).cast<DictionaryAttr>(); attributeDict = unwrap(attributes).cast<DictionaryAttr>();
Show All 26 Lines

mlir/lib/Conversion/AffineToStandard/AffineToStandard.cpp

Show First 20 Lines • Show All 217 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(AffineParallelOp op,
// scf.parallel handles the reduction operation differently unlike // scf.parallel handles the reduction operation differently unlike
// affine.parallel. // affine.parallel.
ArrayRef<Attribute> reductions = op.getReductions().getValue(); ArrayRef<Attribute> reductions = op.getReductions().getValue();
for (auto pair : llvm::zip(reductions, op.getResultTypes())) { for (auto pair : llvm::zip(reductions, op.getResultTypes())) {
// For each of the reduction operations get the identity values for // For each of the reduction operations get the identity values for
// initialization of the result values. // initialization of the result values.
Attribute reduction = std::get<0>(pair); Attribute reduction = std::get<0>(pair);
Type resultType = std::get<1>(pair); Type resultType = std::get<1>(pair);
Optional<arith::AtomicRMWKind> reductionOp = std::optional<arith::AtomicRMWKind> reductionOp =
arith::symbolizeAtomicRMWKind( arith::symbolizeAtomicRMWKind(
static_cast<uint64_t>(reduction.cast<IntegerAttr>().getInt())); static_cast<uint64_t>(reduction.cast<IntegerAttr>().getInt()));
assert(reductionOp && "Reduction operation cannot be of None Type"); assert(reductionOp && "Reduction operation cannot be of None Type");
arith::AtomicRMWKind reductionOpValue = *reductionOp; arith::AtomicRMWKind reductionOpValue = *reductionOp;
identityVals.push_back( identityVals.push_back(
arith::getIdentityValue(reductionOpValue, resultType, rewriter, loc)); arith::getIdentityValue(reductionOpValue, resultType, rewriter, loc));
} }
parOp = rewriter.create<scf::ParallelOp>( parOp = rewriter.create<scf::ParallelOp>(
loc, lowerBoundTuple, upperBoundTuple, steps, identityVals, loc, lowerBoundTuple, upperBoundTuple, steps, identityVals,
/*bodyBuilderFn=*/nullptr); /*bodyBuilderFn=*/nullptr);
// Copy the body of the affine.parallel op. // Copy the body of the affine.parallel op.
rewriter.eraseBlock(parOp.getBody()); rewriter.eraseBlock(parOp.getBody());
rewriter.inlineRegionBefore(op.getRegion(), parOp.getRegion(), rewriter.inlineRegionBefore(op.getRegion(), parOp.getRegion(),
parOp.getRegion().end()); parOp.getRegion().end());
assert(reductions.size() == affineParOpTerminator->getNumOperands() && assert(reductions.size() == affineParOpTerminator->getNumOperands() &&
"Unequal number of reductions and operands."); "Unequal number of reductions and operands.");
for (unsigned i = 0, end = reductions.size(); i < end; i++) { for (unsigned i = 0, end = reductions.size(); i < end; i++) {
// For each of the reduction operations get the respective mlir::Value. // For each of the reduction operations get the respective mlir::Value.
Optional<arith::AtomicRMWKind> reductionOp = std::optional<arith::AtomicRMWKind> reductionOp =
arith::symbolizeAtomicRMWKind( arith::symbolizeAtomicRMWKind(
reductions[i].cast<IntegerAttr>().getInt()); reductions[i].cast<IntegerAttr>().getInt());
assert(reductionOp && "Reduction Operation cannot be of None Type"); assert(reductionOp && "Reduction Operation cannot be of None Type");
arith::AtomicRMWKind reductionOpValue = *reductionOp; arith::AtomicRMWKind reductionOpValue = *reductionOp;
rewriter.setInsertionPoint(&parOp.getBody()->back()); rewriter.setInsertionPoint(&parOp.getBody()->back());
auto reduceOp = rewriter.create<scf::ReduceOp>( auto reduceOp = rewriter.create<scf::ReduceOp>(
loc, affineParOpTerminator->getOperand(i)); loc, affineParOpTerminator->getOperand(i));
rewriter.setInsertionPointToEnd(&reduceOp.getReductionOperator().front()); rewriter.setInsertionPointToEnd(&reduceOp.getReductionOperator().front());
▲ Show 20 Lines • Show All 316 Lines • Show Last 20 Lines

mlir/lib/Conversion/MemRefToLLVM/MemRefToLLVM.cpp

Show First 20 Lines • Show All 436 Lines • ▼ Show 20 Lines Value extractSizeOfUnrankedMemRef(Type operandType, memref::DimOp dimOp,
// `dimOp.index() + 1` index argument. // `dimOp.index() + 1` index argument.
Value idxPlusOne = rewriter.create<LLVM::AddOp>( Value idxPlusOne = rewriter.create<LLVM::AddOp>(
loc, createIndexConstant(rewriter, loc, 1), adaptor.getIndex()); loc, createIndexConstant(rewriter, loc, 1), adaptor.getIndex());
Value sizePtr = Value sizePtr =
rewriter.create<LLVM::GEPOp>(loc, indexPtrTy, offsetPtr, idxPlusOne); rewriter.create<LLVM::GEPOp>(loc, indexPtrTy, offsetPtr, idxPlusOne);
return rewriter.create<LLVM::LoadOp>(loc, sizePtr); return rewriter.create<LLVM::LoadOp>(loc, sizePtr);
} }
Optional<int64_t> getConstantDimIndex(memref::DimOp dimOp) const { std::optional<int64_t> getConstantDimIndex(memref::DimOp dimOp) const {
if (Optional<int64_t> idx = dimOp.getConstantIndex()) if (auto idx = dimOp.getConstantIndex())
return idx; return idx;
if (auto constantOp = dimOp.getIndex().getDefiningOp<LLVM::ConstantOp>()) if (auto constantOp = dimOp.getIndex().getDefiningOp<LLVM::ConstantOp>())
return constantOp.getValue() return constantOp.getValue()
.cast<IntegerAttr>() .cast<IntegerAttr>()
.getValue() .getValue()
.getSExtValue(); .getSExtValue();
return std::nullopt; return std::nullopt;
} }
Value extractSizeOfRankedMemRef(Type operandType, memref::DimOp dimOp, Value extractSizeOfRankedMemRef(Type operandType, memref::DimOp dimOp,
OpAdaptor adaptor, OpAdaptor adaptor,
ConversionPatternRewriter &rewriter) const { ConversionPatternRewriter &rewriter) const {
Location loc = dimOp.getLoc(); Location loc = dimOp.getLoc();
// Take advantage if index is constant. // Take advantage if index is constant.
MemRefType memRefType = operandType.cast<MemRefType>(); MemRefType memRefType = operandType.cast<MemRefType>();
if (Optional<int64_t> index = getConstantDimIndex(dimOp)) { if (std::optional<int64_t> index = getConstantDimIndex(dimOp)) {
int64_t i = *index; int64_t i = *index;
if (memRefType.isDynamicDim(i)) { if (memRefType.isDynamicDim(i)) {
// extract dynamic size from the memref descriptor. // extract dynamic size from the memref descriptor.
MemRefDescriptor descriptor(adaptor.getSource()); MemRefDescriptor descriptor(adaptor.getSource());
return descriptor.size(rewriter, loc, i); return descriptor.size(rewriter, loc, i);
} }
// Use constant for static size. // Use constant for static size.
int64_t dimSize = memRefType.getDimSize(i); int64_t dimSize = memRefType.getDimSize(i);
▲ Show 20 Lines • Show All 1,594 Lines • Show Last 20 Lines

mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp

Show First 20 Lines • Show All 292 Lines • ▼ Show 20 Lines if (aType.getElementType().isF32() && !tf32Enabled)
return failure(); return failure();
FailureOr<NVVM::MMATypes> ptxTypeA = getNvvmMmaType(aType); FailureOr<NVVM::MMATypes> ptxTypeA = getNvvmMmaType(aType);
if (failed(ptxTypeA)) if (failed(ptxTypeA))
return op->emitOpError("failed to deduce operand PTX types"); return op->emitOpError("failed to deduce operand PTX types");
FailureOr<NVVM::MMATypes> ptxTypeB = getNvvmMmaType(bType); FailureOr<NVVM::MMATypes> ptxTypeB = getNvvmMmaType(bType);
if (failed(ptxTypeB)) if (failed(ptxTypeB))
return op->emitOpError("failed to deduce operand PTX types"); return op->emitOpError("failed to deduce operand PTX types");
Optional<NVVM::MMATypes> ptxTypeC = NVVM::MmaOp::inferOperandMMAType( std::optional<NVVM::MMATypes> ptxTypeC =
cType.getElementType(), /*isAccumulator=*/true); NVVM::MmaOp::inferOperandMMAType(cType.getElementType(),
/*isAccumulator=*/true);
if (!ptxTypeC) if (!ptxTypeC)
return op->emitError( return op->emitError(
"could not infer the PTX type for the accumulator/result"); "could not infer the PTX type for the accumulator/result");
// TODO: add an attribute to the op to customize this behavior. // TODO: add an attribute to the op to customize this behavior.
Optional<NVVM::MMAIntOverflow> overflow(std::nullopt); std::optional<NVVM::MMAIntOverflow> overflow(std::nullopt);
if (aType.getElementType().isa<IntegerType>()) if (aType.getElementType().isa<IntegerType>())
overflow = NVVM::MMAIntOverflow::satfinite; overflow = NVVM::MMAIntOverflow::satfinite;
SmallVector<Value> matA = SmallVector<Value> matA =
unpackOperandVector(rewriter, loc, adaptor.getMatrixA(), *ptxTypeA); unpackOperandVector(rewriter, loc, adaptor.getMatrixA(), *ptxTypeA);
SmallVector<Value> matB = SmallVector<Value> matB =
unpackOperandVector(rewriter, loc, adaptor.getMatrixB(), *ptxTypeB); unpackOperandVector(rewriter, loc, adaptor.getMatrixB(), *ptxTypeB);
SmallVector<Value> matC = SmallVector<Value> matC =
▲ Show 20 Lines • Show All 91 Lines • ▼ Show 20 Lines
/// the give parameters. Note that this function doesn't do any validation, /// the give parameters. Note that this function doesn't do any validation,
/// it's expected that the provided parameters correspond to a valid /// it's expected that the provided parameters correspond to a valid
/// instruction. /// instruction.
static std::string static std::string
buildMmaSparseAsmString(const std::array<int64_t, 3> &shape, unsigned matASize, buildMmaSparseAsmString(const std::array<int64_t, 3> &shape, unsigned matASize,
unsigned matBSize, unsigned matCSize, unsigned matBSize, unsigned matCSize,
NVVM::MMATypes ptxTypeA, NVVM::MMATypes ptxTypeB, NVVM::MMATypes ptxTypeA, NVVM::MMATypes ptxTypeB,
NVVM::MMATypes ptxTypeC, NVVM::MMATypes ptxTypeD, NVVM::MMATypes ptxTypeC, NVVM::MMATypes ptxTypeD,
Optional<NVVM::MMAIntOverflow> overflow) { std::optional<NVVM::MMAIntOverflow> overflow) {
auto ptxTypeStr = [](NVVM::MMATypes ptxType) { auto ptxTypeStr = [](NVVM::MMATypes ptxType) {
return NVVM::stringifyMMATypes(ptxType); return NVVM::stringifyMMATypes(ptxType);
}; };
std::string asmStr; std::string asmStr;
llvm::raw_string_ostream ss(asmStr); llvm::raw_string_ostream ss(asmStr);
ss << "mma.sp.sync.aligned.m" << shape[0] << "n" << shape[1] << "k" ss << "mma.sp.sync.aligned.m" << shape[0] << "n" << shape[1] << "k"
<< shape[2] << ".row.col."; << shape[2] << ".row.col.";
Show All 19 LinesbuildMmaSparseAsmString(const std::array<int64_t, 3> &shape, unsigned matASize,
return asmStr; return asmStr;
} }
/// Builds an inline assembly operation corresponding to the specified MMA /// Builds an inline assembly operation corresponding to the specified MMA
/// sparse sync operation. /// sparse sync operation.
static FailureOr<LLVM::InlineAsmOp> emitMmaSparseSyncOpAsm( static FailureOr<LLVM::InlineAsmOp> emitMmaSparseSyncOpAsm(
Location loc, NVVM::MMATypes ptxTypeA, NVVM::MMATypes ptxTypeB, Location loc, NVVM::MMATypes ptxTypeA, NVVM::MMATypes ptxTypeB,
NVVM::MMATypes ptxTypeC, NVVM::MMATypes ptxTypeD, NVVM::MMATypes ptxTypeC, NVVM::MMATypes ptxTypeD,
Optional<NVVM::MMAIntOverflow> overflow, ArrayRef<Value> unpackedAData, std::optional<NVVM::MMAIntOverflow> overflow, ArrayRef<Value> unpackedAData,
ArrayRef<Value> unpackedB, ArrayRef<Value> unpackedC, Value indexData, ArrayRef<Value> unpackedB, ArrayRef<Value> unpackedC, Value indexData,
int64_t metadataSelector, const std::array<int64_t, 3> &shape, int64_t metadataSelector, const std::array<int64_t, 3> &shape,
Type intrinsicResultType, ConversionPatternRewriter &rewriter) { Type intrinsicResultType, ConversionPatternRewriter &rewriter) {
auto asmDialectAttr = LLVM::AsmDialectAttr::get(rewriter.getContext(), auto asmDialectAttr = LLVM::AsmDialectAttr::get(rewriter.getContext(),
LLVM::AsmDialect::AD_ATT); LLVM::AsmDialect::AD_ATT);
std::string asmStr = buildMmaSparseAsmString( std::string asmStr = buildMmaSparseAsmString(
shape, unpackedAData.size(), unpackedB.size(), unpackedC.size(), ptxTypeA, shape, unpackedAData.size(), unpackedB.size(), unpackedC.size(), ptxTypeA,
Show All 39 Lines matchAndRewrite(nvgpu::MmaSparseSyncOp op, OpAdaptor adaptor,
VectorType cType = op.getMatrixC().getType(); VectorType cType = op.getMatrixC().getType();
FailureOr<NVVM::MMATypes> ptxTypeA = getNvvmMmaType(aType); FailureOr<NVVM::MMATypes> ptxTypeA = getNvvmMmaType(aType);
if (failed(ptxTypeA)) if (failed(ptxTypeA))
return op->emitOpError("failed to deduce operand PTX types"); return op->emitOpError("failed to deduce operand PTX types");
FailureOr<NVVM::MMATypes> ptxTypeB = getNvvmMmaType(bType); FailureOr<NVVM::MMATypes> ptxTypeB = getNvvmMmaType(bType);
if (failed(ptxTypeB)) if (failed(ptxTypeB))
return op->emitOpError("failed to deduce operand PTX types"); return op->emitOpError("failed to deduce operand PTX types");
Optional<NVVM::MMATypes> ptxTypeC = NVVM::MmaOp::inferOperandMMAType( std::optional<NVVM::MMATypes> ptxTypeC =
cType.getElementType(), /*isAccumulator=*/true); NVVM::MmaOp::inferOperandMMAType(cType.getElementType(),
/*isAccumulator=*/true);
if (!ptxTypeC) if (!ptxTypeC)
return op->emitError( return op->emitError(
"could not infer the PTX type for the accumulator/result"); "could not infer the PTX type for the accumulator/result");
// Same as `mma.sync`, F32 works only with TensorFloat32 (TF32). // Same as `mma.sync`, F32 works only with TensorFloat32 (TF32).
bool tf32Enabled = op->hasAttr(op.getTf32EnabledAttrName()); bool tf32Enabled = op->hasAttr(op.getTf32EnabledAttrName());
if (aType.getElementType().isF32() && !tf32Enabled) if (aType.getElementType().isF32() && !tf32Enabled)
return failure(); return failure();
// TODO: add an attribute to the op to customize this behavior. // TODO: add an attribute to the op to customize this behavior.
Optional<NVVM::MMAIntOverflow> overflow(std::nullopt); std::optional<NVVM::MMAIntOverflow> overflow(std::nullopt);
if (aType.getElementType().isa<IntegerType>()) if (aType.getElementType().isa<IntegerType>())
overflow = NVVM::MMAIntOverflow::satfinite; overflow = NVVM::MMAIntOverflow::satfinite;
SmallVector<Value> matA = SmallVector<Value> matA =
unpackOperandVector(rewriter, loc, adaptor.getMatrixA(), *ptxTypeA); unpackOperandVector(rewriter, loc, adaptor.getMatrixA(), *ptxTypeA);
SmallVector<Value> matB = SmallVector<Value> matB =
unpackOperandVector(rewriter, loc, adaptor.getMatrixB(), *ptxTypeB); unpackOperandVector(rewriter, loc, adaptor.getMatrixB(), *ptxTypeB);
SmallVector<Value> matC = SmallVector<Value> matC =
▲ Show 20 Lines • Show All 143 Lines • Show Last 20 Lines

mlir/lib/Conversion/PDLToPDLInterp/PDLToPDLInterp.cpp

Show First 20 Lines • Show All 589 Lines • ▼ Show 20 Lines for (auto op :
generatedOps.push_back(*op.getOpName()); generatedOps.push_back(*op.getOpName());
ArrayAttr generatedOpsAttr; ArrayAttr generatedOpsAttr;
if (!generatedOps.empty()) if (!generatedOps.empty())
generatedOpsAttr = builder.getStrArrayAttr(generatedOps); generatedOpsAttr = builder.getStrArrayAttr(generatedOps);
// Grab the root kind if present. // Grab the root kind if present.
StringAttr rootKindAttr; StringAttr rootKindAttr;
if (pdl::OperationOp rootOp = root.getDefiningOp<pdl::OperationOp>()) if (pdl::OperationOp rootOp = root.getDefiningOp<pdl::OperationOp>())
if (Optional<StringRef> rootKind = rootOp.getOpName()) if (std::optional<StringRef> rootKind = rootOp.getOpName())
rootKindAttr = builder.getStringAttr(*rootKind); rootKindAttr = builder.getStringAttr(*rootKind);
builder.setInsertionPointToEnd(currentBlock); builder.setInsertionPointToEnd(currentBlock);
auto matchOp = builder.create<pdl_interp::RecordMatchOp>( auto matchOp = builder.create<pdl_interp::RecordMatchOp>(
pattern.getLoc(), mappedMatchValues, locOps.getArrayRef(), pattern.getLoc(), mappedMatchValues, locOps.getArrayRef(),
rewriterFuncRef, rootKindAttr, generatedOpsAttr, pattern.getBenefitAttr(), rewriterFuncRef, rootKindAttr, generatedOpsAttr, pattern.getBenefitAttr(),
failureBlockStack.back()); failureBlockStack.back());
▲ Show 20 Lines • Show All 394 Lines • Show Last 20 Lines

mlir/lib/Conversion/PDLToPDLInterp/Predicate.h

Show First 20 Lines • Show All 216 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// OperandGroupPosition // OperandGroupPosition
/// A position describing an operand group of an operation. /// A position describing an operand group of an operation.
struct OperandGroupPosition struct OperandGroupPosition
: public PredicateBase< : public PredicateBase<
OperandGroupPosition, Position, OperandGroupPosition, Position,
std::tuple<OperationPosition *, Optional<unsigned>, bool>, std::tuple<OperationPosition *, std::optional<unsigned>, bool>,
Predicates::OperandGroupPos> { Predicates::OperandGroupPos> {
explicit OperandGroupPosition(const KeyTy &key); explicit OperandGroupPosition(const KeyTy &key);
/// Returns a hash suitable for the given keytype. /// Returns a hash suitable for the given keytype.
static llvm::hash_code hashKey(const KeyTy &key) { static llvm::hash_code hashKey(const KeyTy &key) {
return llvm::hash_value(key); return llvm::hash_value(key);
} }
/// Returns the group number of this position. If std::nullopt, this group /// Returns the group number of this position. If std::nullopt, this group
/// refers to all operands. /// refers to all operands.
Optional<unsigned> getOperandGroupNumber() const { return std::get<1>(key); } std::optional<unsigned> getOperandGroupNumber() const {
return std::get<1>(key);
}
/// Returns if the operand group has unknown size. If false, the operand group /// Returns if the operand group has unknown size. If false, the operand group
/// has at max one element. /// has at max one element.
bool isVariadic() const { return std::get<2>(key); } bool isVariadic() const { return std::get<2>(key); }
}; };
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// OperationPosition // OperationPosition
▲ Show 20 Lines • Show All 48 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ResultGroupPosition // ResultGroupPosition
/// A position describing a result group of an operation. /// A position describing a result group of an operation.
struct ResultGroupPosition struct ResultGroupPosition
: public PredicateBase< : public PredicateBase<
ResultGroupPosition, Position, ResultGroupPosition, Position,
std::tuple<OperationPosition *, Optional<unsigned>, bool>, std::tuple<OperationPosition *, std::optional<unsigned>, bool>,
Predicates::ResultGroupPos> { Predicates::ResultGroupPos> {
explicit ResultGroupPosition(const KeyTy &key) : Base(key) { explicit ResultGroupPosition(const KeyTy &key) : Base(key) {
parent = std::get<0>(key); parent = std::get<0>(key);
} }
/// Returns a hash suitable for the given keytype. /// Returns a hash suitable for the given keytype.
static llvm::hash_code hashKey(const KeyTy &key) { static llvm::hash_code hashKey(const KeyTy &key) {
return llvm::hash_value(key); return llvm::hash_value(key);
} }
/// Returns the group number of this position. If std::nullopt, this group /// Returns the group number of this position. If std::nullopt, this group
/// refers to all results. /// refers to all results.
Optional<unsigned> getResultGroupNumber() const { return std::get<1>(key); } std::optional<unsigned> getResultGroupNumber() const {
return std::get<1>(key);
}
/// Returns if the result group has unknown size. If false, the result group /// Returns if the result group has unknown size. If false, the result group
/// has at max one element. /// has at max one element.
bool isVariadic() const { return std::get<2>(key); } bool isVariadic() const { return std::get<2>(key); }
}; };
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TypePosition // TypePosition
▲ Show 20 Lines • Show All 267 Lines • ▼ Show 20 Linespublic:
} }
/// Returns an operand position for an operand of the given operation. /// Returns an operand position for an operand of the given operation.
Position *getOperand(OperationPosition *p, unsigned operand) { Position *getOperand(OperationPosition *p, unsigned operand) {
return OperandPosition::get(uniquer, p, operand); return OperandPosition::get(uniquer, p, operand);
} }
/// Returns a position for a group of operands of the given operation. /// Returns a position for a group of operands of the given operation.
Position *getOperandGroup(OperationPosition *p, Optional<unsigned> group, Position *getOperandGroup(OperationPosition *p, std::optional<unsigned> group,
bool isVariadic) { bool isVariadic) {
return OperandGroupPosition::get(uniquer, p, group, isVariadic); return OperandGroupPosition::get(uniquer, p, group, isVariadic);
} }
Position *getAllOperands(OperationPosition *p) { Position *getAllOperands(OperationPosition *p) {
return getOperandGroup(p, /*group=*/std::nullopt, /*isVariadic=*/true); return getOperandGroup(p, /*group=*/std::nullopt, /*isVariadic=*/true);
} }
/// Returns a result position for a result of the given operation. /// Returns a result position for a result of the given operation.
Position *getResult(OperationPosition *p, unsigned result) { Position *getResult(OperationPosition *p, unsigned result) {
return ResultPosition::get(uniquer, p, result); return ResultPosition::get(uniquer, p, result);
} }
/// Returns a position for a group of results of the given operation. /// Returns a position for a group of results of the given operation.
Position *getResultGroup(OperationPosition *p, Optional<unsigned> group, Position *getResultGroup(OperationPosition *p, std::optional<unsigned> group,
bool isVariadic) { bool isVariadic) {
return ResultGroupPosition::get(uniquer, p, group, isVariadic); return ResultGroupPosition::get(uniquer, p, group, isVariadic);
} }
Position *getAllResults(OperationPosition *p) { Position *getAllResults(OperationPosition *p) {
return getResultGroup(p, /*group=*/std::nullopt, /*isVariadic=*/true); return getResultGroup(p, /*group=*/std::nullopt, /*isVariadic=*/true);
} }
/// Returns a type position for the given entity. /// Returns a type position for the given entity.
▲ Show 20 Lines • Show All 99 Lines • Show Last 20 Lines

mlir/lib/Conversion/PDLToPDLInterp/PredicateTree.cpp

Show First 20 Lines • Show All 75 Lines • ▼ Show 20 Lines TypeSwitch<Operation *>(val.getDefiningOp())
cast<OperandGroupPosition>(pos)->getOperandGroupNumber()) cast<OperandGroupPosition>(pos)->getOperandGroupNumber())
predList.emplace_back(pos, builder.getIsNotNull()); predList.emplace_back(pos, builder.getIsNotNull());
if (Value type = op.getValueType()) if (Value type = op.getValueType())
getTreePredicates(predList, type, builder, inputs, getTreePredicates(predList, type, builder, inputs,
builder.getType(pos)); builder.getType(pos));
}) })
.Case<pdl::ResultOp, pdl::ResultsOp>([&](auto op) { .Case<pdl::ResultOp, pdl::ResultsOp>([&](auto op) {
Optional<unsigned> index = op.getIndex(); std::optional<unsigned> index = op.getIndex();
// Prevent traversal into a null value if the result has a proper index. // Prevent traversal into a null value if the result has a proper index.
if (index) if (index)
predList.emplace_back(pos, builder.getIsNotNull()); predList.emplace_back(pos, builder.getIsNotNull());
// Get the parent operation of this operand. // Get the parent operation of this operand.
OperationPosition *parentPos = builder.getOperandDefiningOp(pos); OperationPosition *parentPos = builder.getOperandDefiningOp(pos);
predList.emplace_back(parentPos, builder.getIsNotNull()); predList.emplace_back(parentPos, builder.getIsNotNull());
// Ensure that the operands match the corresponding results of the // Ensure that the operands match the corresponding results of the
// parent operation. // parent operation.
Position *resultPos = nullptr; Position *resultPos = nullptr;
if (std::is_same<pdl::ResultOp, decltype(op)>::value) if (std::is_same<pdl::ResultOp, decltype(op)>::value)
resultPos = builder.getResult(parentPos, *index); resultPos = builder.getResult(parentPos, *index);
else else
resultPos = builder.getResultGroup(parentPos, index, isVariadic); resultPos = builder.getResultGroup(parentPos, index, isVariadic);
predList.emplace_back(resultPos, builder.getEqualTo(pos)); predList.emplace_back(resultPos, builder.getEqualTo(pos));
// Collect the predicates of the parent operation. // Collect the predicates of the parent operation.
getTreePredicates(predList, op.getParent(), builder, inputs, getTreePredicates(predList, op.getParent(), builder, inputs,
(Position *)parentPos); (Position *)parentPos);
}); });
} }
static void getTreePredicates(std::vector<PositionalPredicate> &predList, static void
Value val, PredicateBuilder &builder, getTreePredicates(std::vector<PositionalPredicate> &predList, Value val,
DenseMap<Value, Position *> &inputs, PredicateBuilder &builder,
OperationPosition *pos, DenseMap<Value, Position *> &inputs, OperationPosition *pos,
Optional<unsigned> ignoreOperand = std::nullopt) { std::optional<unsigned> ignoreOperand = std::nullopt) {
assert(val.getType().isa<pdl::OperationType>() && "expected operation"); assert(val.getType().isa<pdl::OperationType>() && "expected operation");
pdl::OperationOp op = cast<pdl::OperationOp>(val.getDefiningOp()); pdl::OperationOp op = cast<pdl::OperationOp>(val.getDefiningOp());
OperationPosition *opPos = cast<OperationPosition>(pos); OperationPosition *opPos = cast<OperationPosition>(pos);
// Ensure getDefiningOp returns a non-null operation. // Ensure getDefiningOp returns a non-null operation.
if (!opPos->isRoot()) if (!opPos->isRoot())
predList.emplace_back(pos, builder.getIsNotNull()); predList.emplace_back(pos, builder.getIsNotNull());
// Check that this is the correct root operation. // Check that this is the correct root operation.
if (Optional<StringRef> opName = op.getOpName()) if (std::optional<StringRef> opName = op.getOpName())
predList.emplace_back(pos, builder.getOperationName(*opName)); predList.emplace_back(pos, builder.getOperationName(*opName));
// Check that the operation has the proper number of operands. If there are // Check that the operation has the proper number of operands. If there are
// any variable length operands, we check a minimum instead of an exact count. // any variable length operands, we check a minimum instead of an exact count.
OperandRange operands = op.getOperandValues(); OperandRange operands = op.getOperandValues();
unsigned minOperands = getNumNonRangeValues(operands); unsigned minOperands = getNumNonRangeValues(operands);
if (minOperands != operands.size()) { if (minOperands != operands.size()) {
if (minOperands) if (minOperands)
▲ Show 20 Lines • Show All 165 Lines • ▼ Show 20 Linesstatic void getResultPredicates(pdl::ResultsOp op,
DenseMap<Value, Position *> &inputs) { DenseMap<Value, Position *> &inputs) {
Position *&resultPos = inputs[op]; Position *&resultPos = inputs[op];
if (resultPos) if (resultPos)
return; return;
// Ensure that the result isn't null if the result has an index. // Ensure that the result isn't null if the result has an index.
auto *parentPos = cast<OperationPosition>(inputs.lookup(op.getParent())); auto *parentPos = cast<OperationPosition>(inputs.lookup(op.getParent()));
bool isVariadic = op.getType().isa<pdl::RangeType>(); bool isVariadic = op.getType().isa<pdl::RangeType>();
Optional<unsigned> index = op.getIndex(); std::optional<unsigned> index = op.getIndex();
resultPos = builder.getResultGroup(parentPos, index, isVariadic); resultPos = builder.getResultGroup(parentPos, index, isVariadic);
if (index) if (index)
predList.emplace_back(resultPos, builder.getIsNotNull()); predList.emplace_back(resultPos, builder.getIsNotNull());
} }
static void getTypePredicates(Value typeValue, static void getTypePredicates(Value typeValue,
function_ref<Attribute()> typeAttrFn, function_ref<Attribute()> typeAttrFn,
PredicateBuilder &builder, PredicateBuilder &builder,
Show All 37 Linesstatic void getNonTreePredicates(pdl::PatternOp pattern,
} }
} }
namespace { namespace {
/// An op accepting a value at an optional index. /// An op accepting a value at an optional index.
struct OpIndex { struct OpIndex {
Value parent; Value parent;
Optional<unsigned> index; std::optional<unsigned> index;
}; };
/// The parent and operand index of each operation for each root, stored /// The parent and operand index of each operation for each root, stored
/// as a nested map [root][operation]. /// as a nested map [root][operation].
using ParentMaps = DenseMap<Value, DenseMap<Value, OpIndex>>; using ParentMaps = DenseMap<Value, DenseMap<Value, OpIndex>>;
} // namespace } // namespace
Show All 35 Linesstatic void buildCostGraph(ArrayRef<Value> roots, RootOrderingGraph &graph,
ParentMaps &parentMaps) { ParentMaps &parentMaps) {
// The entry of a queue. The entry consists of the following items: // The entry of a queue. The entry consists of the following items:
// * the value in the DAG underneath the root; // * the value in the DAG underneath the root;
// * the parent of the value; // * the parent of the value;
// * the operand index of the value in its parent; // * the operand index of the value in its parent;
// * the depth of the visited value. // * the depth of the visited value.
struct Entry { struct Entry {
Entry(Value value, Value parent, Optional<unsigned> index, unsigned depth) Entry(Value value, Value parent, std::optional<unsigned> index,
unsigned depth)
: value(value), parent(parent), index(index), depth(depth) {} : value(value), parent(parent), index(index), depth(depth) {}
Value value; Value value;
Value parent; Value parent;
Optional<unsigned> index; std::optional<unsigned> index;
unsigned depth; unsigned depth;
}; };
// A root of a value and its depth (distance from root to the value). // A root of a value and its depth (distance from root to the value).
struct RootDepth { struct RootDepth {
Value root; Value root;
unsigned depth = 0; unsigned depth = 0;
}; };
▲ Show 20 Lines • Show All 579 Lines • Show Last 20 Lines

mlir/lib/Dialect/Affine/IR/AffineOps.cpp

Show First 20 Lines • Show All 329 Lines • ▼ Show 20 Linesstatic bool isDimOpValidSymbol(ShapedDimOpInterface dimOp, Region *region) {
// Conservatively handle remaining BlockArguments as non-valid symbols. // Conservatively handle remaining BlockArguments as non-valid symbols.
// E.g. scf.for iterArgs. // E.g. scf.for iterArgs.
if (dimOp.getShapedValue().template isa<BlockArgument>()) if (dimOp.getShapedValue().template isa<BlockArgument>())
return false; return false;
// The dim op is also okay if its operand memref is a view/subview whose // The dim op is also okay if its operand memref is a view/subview whose
// corresponding size is a valid symbol. // corresponding size is a valid symbol.
Optional<int64_t> index = getConstantIntValue(dimOp.getDimension()); std::optional<int64_t> index = getConstantIntValue(dimOp.getDimension());
assert(index.has_value() && assert(index.has_value() &&
"expect only `dim` operations with a constant index"); "expect only `dim` operations with a constant index");
int64_t i = index.value(); int64_t i = index.value();
return TypeSwitch<Operation *, bool>(dimOp.getShapedValue().getDefiningOp()) return TypeSwitch<Operation *, bool>(dimOp.getShapedValue().getDefiningOp())
.Case<memref::ViewOp, memref::SubViewOp, memref::AllocOp>( .Case<memref::ViewOp, memref::SubViewOp, memref::AllocOp>(
[&](auto op) { return isMemRefSizeValidSymbol(op, i, region); }) [&](auto op) { return isMemRefSizeValidSymbol(op, i, region); })
.Default([](Operation *) { return false; }); .Default([](Operation *) { return false; });
} }
▲ Show 20 Lines • Show All 571 Lines • ▼ Show 20 Lines
/// The operation is also expected to always produce a single result. Return an /// The operation is also expected to always produce a single result. Return an
/// `OpFoldResult` containing the Attribute representing the folded constant if /// `OpFoldResult` containing the Attribute representing the folded constant if
/// complete folding was possible and a Value produced by the created operation /// complete folding was possible and a Value produced by the created operation
/// otherwise. /// otherwise.
template <typename OpTy, typename... Args> template <typename OpTy, typename... Args>
static std::enable_if_t<OpTy::template hasTrait<OpTrait::OneResult>(), static std::enable_if_t<OpTy::template hasTrait<OpTrait::OneResult>(),
OpFoldResult> OpFoldResult>
createOrFold(OpBuilder &b, Location loc, ValueRange operands, createOrFold(OpBuilder &b, Location loc, ValueRange operands,
Args &&... leadingArguments) { Args &&...leadingArguments) {
// Identify the constant operands and extract their values as attributes. // Identify the constant operands and extract their values as attributes.
// Note that we cannot use the original values directly because the list of // Note that we cannot use the original values directly because the list of
// operands may have changed due to canonicalization and composition. // operands may have changed due to canonicalization and composition.
SmallVector<Attribute> constantOperands; SmallVector<Attribute> constantOperands;
constantOperands.reserve(operands.size()); constantOperands.reserve(operands.size());
for (Value operand : operands) { for (Value operand : operands) {
IntegerAttr attr; IntegerAttr attr;
if (matchPattern(operand, m_Constant(&attr))) if (matchPattern(operand, m_Constant(&attr)))
▲ Show 20 Lines • Show All 1,069 Lines • ▼ Show 20 Lines if (lbMap != prevLbMap)
forOp.setLowerBound(lbOperands, lbMap); forOp.setLowerBound(lbOperands, lbMap);
if (ubMap != prevUbMap) if (ubMap != prevUbMap)
forOp.setUpperBound(ubOperands, ubMap); forOp.setUpperBound(ubOperands, ubMap);
return success(); return success();
} }
namespace { namespace {
/// Returns constant trip count in trivial cases. /// Returns constant trip count in trivial cases.
static Optional<uint64_t> getTrivialConstantTripCount(AffineForOp forOp) { static std::optional<uint64_t> getTrivialConstantTripCount(AffineForOp forOp) {
int64_t step = forOp.getStep(); int64_t step = forOp.getStep();
if (!forOp.hasConstantBounds() || step <= 0) if (!forOp.hasConstantBounds() || step <= 0)
return std::nullopt; return std::nullopt;
int64_t lb = forOp.getConstantLowerBound(); int64_t lb = forOp.getConstantLowerBound();
int64_t ub = forOp.getConstantUpperBound(); int64_t ub = forOp.getConstantUpperBound();
return ub - lb <= 0 ? 0 : (ub - lb + step - 1) / step; return ub - lb <= 0 ? 0 : (ub - lb + step - 1) / step;
} }
/// This is a pattern to fold trivially empty loop bodies. /// This is a pattern to fold trivially empty loop bodies.
/// TODO: This should be moved into the folding hook. /// TODO: This should be moved into the folding hook.
struct AffineForEmptyLoopFolder : public OpRewritePattern<AffineForOp> { struct AffineForEmptyLoopFolder : public OpRewritePattern<AffineForOp> {
using OpRewritePattern<AffineForOp>::OpRewritePattern; using OpRewritePattern<AffineForOp>::OpRewritePattern;
LogicalResult matchAndRewrite(AffineForOp forOp, LogicalResult matchAndRewrite(AffineForOp forOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
// Check that the body only contains a yield. // Check that the body only contains a yield.
if (!llvm::hasSingleElement(*forOp.getBody())) if (!llvm::hasSingleElement(*forOp.getBody()))
return failure(); return failure();
if (forOp.getNumResults() == 0) if (forOp.getNumResults() == 0)
return success(); return success();
Optional<uint64_t> tripCount = getTrivialConstantTripCount(forOp); std::optional<uint64_t> tripCount = getTrivialConstantTripCount(forOp);
if (tripCount && *tripCount == 0) { if (tripCount && *tripCount == 0) {
// The initial values of the iteration arguments would be the op's // The initial values of the iteration arguments would be the op's
// results. // results.
rewriter.replaceOp(forOp, forOp.getIterOperands()); rewriter.replaceOp(forOp, forOp.getIterOperands());
return success(); return success();
} }
SmallVector<Value, 4> replacements; SmallVector<Value, 4> replacements;
auto yieldOp = cast<AffineYieldOp>(forOp.getBody()->getTerminator()); auto yieldOp = cast<AffineYieldOp>(forOp.getBody()->getTerminator());
Show All 35 Linesvoid AffineForOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.add<AffineForEmptyLoopFolder>(context); results.add<AffineForEmptyLoopFolder>(context);
} }
/// Return operands used when entering the region at 'index'. These operands /// Return operands used when entering the region at 'index'. These operands
/// correspond to the loop iterator operands, i.e., those excluding the /// correspond to the loop iterator operands, i.e., those excluding the
/// induction variable. AffineForOp only has one region, so zero is the only /// induction variable. AffineForOp only has one region, so zero is the only
/// valid value for `index`. /// valid value for `index`.
OperandRange AffineForOp::getSuccessorEntryOperands(Optional<unsigned> index) { OperandRange
AffineForOp::getSuccessorEntryOperands(std::optional<unsigned> index) {
assert((!index || *index == 0) && "invalid region index"); assert((!index || *index == 0) && "invalid region index");
// The initial operands map to the loop arguments after the induction // The initial operands map to the loop arguments after the induction
// variable or are forwarded to the results when the trip count is zero. // variable or are forwarded to the results when the trip count is zero.
return getIterOperands(); return getIterOperands();
} }
/// Given the region at `index`, or the parent operation if `index` is None, /// Given the region at `index`, or the parent operation if `index` is None,
/// return the successor regions. These are the regions that may be selected /// return the successor regions. These are the regions that may be selected
/// during the flow of control. `operands` is a set of optional attributes that /// during the flow of control. `operands` is a set of optional attributes that
/// correspond to a constant value for each operand, or null if that operand is /// correspond to a constant value for each operand, or null if that operand is
/// not a constant. /// not a constant.
void AffineForOp::getSuccessorRegions( void AffineForOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
assert((!index.has_value() || index.value() == 0) && "expected loop region"); assert((!index.has_value() || index.value() == 0) && "expected loop region");
// The loop may typically branch back to its body or to the parent operation. // The loop may typically branch back to its body or to the parent operation.
// If the predecessor is the parent op and the trip count is known to be at // If the predecessor is the parent op and the trip count is known to be at
// least one, branch into the body using the iterator arguments. And in cases // least one, branch into the body using the iterator arguments. And in cases
// we know the trip count is zero, it can only branch back to its parent. // we know the trip count is zero, it can only branch back to its parent.
Optional<uint64_t> tripCount = getTrivialConstantTripCount(*this); std::optional<uint64_t> tripCount = getTrivialConstantTripCount(*this);
if (!index.has_value() && tripCount.has_value()) { if (!index.has_value() && tripCount.has_value()) {
if (tripCount.value() > 0) { if (tripCount.value() > 0) {
regions.push_back(RegionSuccessor(&getLoopBody(), getRegionIterArgs())); regions.push_back(RegionSuccessor(&getLoopBody(), getRegionIterArgs()));
return; return;
} }
if (tripCount.value() == 0) { if (tripCount.value() == 0) {
regions.push_back(RegionSuccessor(getResults())); regions.push_back(RegionSuccessor(getResults()));
return; return;
Show All 10 Linesvoid AffineForOp::getSuccessorRegions(
// In all other cases, the loop may branch back to itself or the parent // In all other cases, the loop may branch back to itself or the parent
// operation. // operation.
regions.push_back(RegionSuccessor(&getLoopBody(), getRegionIterArgs())); regions.push_back(RegionSuccessor(&getLoopBody(), getRegionIterArgs()));
regions.push_back(RegionSuccessor(getResults())); regions.push_back(RegionSuccessor(getResults()));
} }
/// Returns true if the affine.for has zero iterations in trivial cases. /// Returns true if the affine.for has zero iterations in trivial cases.
static bool hasTrivialZeroTripCount(AffineForOp op) { static bool hasTrivialZeroTripCount(AffineForOp op) {
Optional<uint64_t> tripCount = getTrivialConstantTripCount(op); std::optional<uint64_t> tripCount = getTrivialConstantTripCount(op);
return tripCount && *tripCount == 0; return tripCount && *tripCount == 0;
} }
LogicalResult AffineForOp::fold(ArrayRef<Attribute> operands, LogicalResult AffineForOp::fold(ArrayRef<Attribute> operands,
SmallVectorImpl<OpFoldResult> &results) { SmallVectorImpl<OpFoldResult> &results) {
bool folded = succeeded(foldLoopBounds(*this)); bool folded = succeeded(foldLoopBounds(*this));
folded |= succeeded(canonicalizeLoopBounds(*this)); folded |= succeeded(canonicalizeLoopBounds(*this));
if (hasTrivialZeroTripCount(*this)) { if (hasTrivialZeroTripCount(*this)) {
▲ Show 20 Lines • Show All 115 Lines • ▼ Show 20 Lines for (unsigned i = 0, e = lbMap.getNumInputs(); i < e; i++) {
if (getOperand(i) != getOperand(numOperands + i)) if (getOperand(i) != getOperand(numOperands + i))
return false; return false;
} }
return true; return true;
} }
Region &AffineForOp::getLoopBody() { return getRegion(); } Region &AffineForOp::getLoopBody() { return getRegion(); }
Optional<Value> AffineForOp::getSingleInductionVar() { std::optional<Value> AffineForOp::getSingleInductionVar() {
return getInductionVar(); return getInductionVar();
} }
Optional<OpFoldResult> AffineForOp::getSingleLowerBound() { std::optional<OpFoldResult> AffineForOp::getSingleLowerBound() {
if (!hasConstantLowerBound()) if (!hasConstantLowerBound())
return std::nullopt; return std::nullopt;
OpBuilder b(getContext()); OpBuilder b(getContext());
return OpFoldResult(b.getI64IntegerAttr(getConstantLowerBound())); return OpFoldResult(b.getI64IntegerAttr(getConstantLowerBound()));
} }
Optional<OpFoldResult> AffineForOp::getSingleStep() { std::optional<OpFoldResult> AffineForOp::getSingleStep() {
OpBuilder b(getContext()); OpBuilder b(getContext());
return OpFoldResult(b.getI64IntegerAttr(getStep())); return OpFoldResult(b.getI64IntegerAttr(getStep()));
} }
Optional<OpFoldResult> AffineForOp::getSingleUpperBound() { std::optional<OpFoldResult> AffineForOp::getSingleUpperBound() {
if (!hasConstantUpperBound()) if (!hasConstantUpperBound())
return std::nullopt; return std::nullopt;
OpBuilder b(getContext()); OpBuilder b(getContext());
return OpFoldResult(b.getI64IntegerAttr(getConstantUpperBound())); return OpFoldResult(b.getI64IntegerAttr(getConstantUpperBound()));
} }
Speculation::Speculatability AffineForOp::getSpeculatability() { Speculation::Speculatability AffineForOp::getSpeculatability() {
// `affine.for (I = Start; I < End; I += 1)` terminates for all values of // `affine.for (I = Start; I < End; I += 1)` terminates for all values of
▲ Show 20 Lines • Show All 246 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(AffineIfOp op,
return success(); return success();
} }
}; };
} // namespace } // namespace
/// AffineIfOp has two regions -- `then` and `else`. The flow of data should be /// AffineIfOp has two regions -- `then` and `else`. The flow of data should be
/// as follows: AffineIfOp -> `then`/`else` -> AffineIfOp /// as follows: AffineIfOp -> `then`/`else` -> AffineIfOp
void AffineIfOp::getSuccessorRegions( void AffineIfOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// If the predecessor is an AffineIfOp, then branching into both `then` and // If the predecessor is an AffineIfOp, then branching into both `then` and
// `else` region is valid. // `else` region is valid.
if (!index.has_value()) { if (!index.has_value()) {
regions.reserve(2); regions.reserve(2);
regions.push_back( regions.push_back(
RegionSuccessor(&getThenRegion(), getThenRegion().getArguments())); RegionSuccessor(&getThenRegion(), getThenRegion().getArguments()));
// Don't consider the else region if it is empty. // Don't consider the else region if it is empty.
▲ Show 20 Lines • Show All 1,009 Lines • ▼ Show 20 Lines
AffineValueMap AffineParallelOp::getLowerBoundsValueMap() { AffineValueMap AffineParallelOp::getLowerBoundsValueMap() {
return AffineValueMap(getLowerBoundsMap(), getLowerBoundsOperands()); return AffineValueMap(getLowerBoundsMap(), getLowerBoundsOperands());
} }
AffineValueMap AffineParallelOp::getUpperBoundsValueMap() { AffineValueMap AffineParallelOp::getUpperBoundsValueMap() {
return AffineValueMap(getUpperBoundsMap(), getUpperBoundsOperands()); return AffineValueMap(getUpperBoundsMap(), getUpperBoundsOperands());
} }
Optional<SmallVector<int64_t, 8>> AffineParallelOp::getConstantRanges() { std::optional<SmallVector<int64_t, 8>> AffineParallelOp::getConstantRanges() {
if (hasMinMaxBounds()) if (hasMinMaxBounds())
return std::nullopt; return std::nullopt;
// Try to convert all the ranges to constant expressions. // Try to convert all the ranges to constant expressions.
SmallVector<int64_t, 8> out; SmallVector<int64_t, 8> out;
AffineValueMap rangesValueMap; AffineValueMap rangesValueMap;
AffineValueMap::difference(getUpperBoundsValueMap(), getLowerBoundsValueMap(), AffineValueMap::difference(getUpperBoundsValueMap(), getLowerBoundsValueMap(),
&rangesValueMap); &rangesValueMap);
▲ Show 20 Lines • Show All 401 Lines • ▼ Show 20 Lines auto parseAttributes = [&]() -> ParseResult {
// verify it is a member of the enum and convert to it's integer // verify it is a member of the enum and convert to it's integer
// representation. // representation.
StringAttr attrVal; StringAttr attrVal;
NamedAttrList attrStorage; NamedAttrList attrStorage;
auto loc = parser.getCurrentLocation(); auto loc = parser.getCurrentLocation();
if (parser.parseAttribute(attrVal, builder.getNoneType(), "reduce", if (parser.parseAttribute(attrVal, builder.getNoneType(), "reduce",
attrStorage)) attrStorage))
return failure(); return failure();
llvm::Optional<arith::AtomicRMWKind> reduction = std::optional<arith::AtomicRMWKind> reduction =
arith::symbolizeAtomicRMWKind(attrVal.getValue()); arith::symbolizeAtomicRMWKind(attrVal.getValue());
if (!reduction) if (!reduction)
return parser.emitError(loc, "invalid reduction value: ") << attrVal; return parser.emitError(loc, "invalid reduction value: ") << attrVal;
reductions.push_back( reductions.push_back(
builder.getI64IntegerAttr(static_cast<int64_t>(reduction.value()))); builder.getI64IntegerAttr(static_cast<int64_t>(reduction.value())));
// While we keep getting commas, keep parsing. // While we keep getting commas, keep parsing.
return success(); return success();
}; };
▲ Show 20 Lines • Show All 229 Lines • ▼ Show 20 Lines
void AffineDelinearizeIndexOp::build(OpBuilder &builder, OperationState &result, void AffineDelinearizeIndexOp::build(OpBuilder &builder, OperationState &result,
Value linearIndex, Value linearIndex,
ArrayRef<OpFoldResult> basis) { ArrayRef<OpFoldResult> basis) {
result.addTypes(SmallVector<Type>(basis.size(), builder.getIndexType())); result.addTypes(SmallVector<Type>(basis.size(), builder.getIndexType()));
result.addOperands(linearIndex); result.addOperands(linearIndex);
SmallVector<Value> basisValues = SmallVector<Value> basisValues =
llvm::to_vector(llvm::map_range(basis, [&](OpFoldResult ofr) -> Value { llvm::to_vector(llvm::map_range(basis, [&](OpFoldResult ofr) -> Value {
Optional<int64_t> staticDim = getConstantIntValue(ofr); std::optional<int64_t> staticDim = getConstantIntValue(ofr);
if (staticDim.has_value()) if (staticDim.has_value())
return builder.create<arith::ConstantIndexOp>(result.location, return builder.create<arith::ConstantIndexOp>(result.location,
*staticDim); *staticDim);
return ofr.dyn_cast<Value>(); return ofr.dyn_cast<Value>();
})); }));
result.addOperands(basisValues); result.addOperands(basisValues);
} }
Show All 14 Lines

mlir/lib/Dialect/Arith/IR/ArithOps.cpp

Show First 20 Lines • Show All 240 Lines • ▼ Show 20 Linesvoid arith::AddIOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
patterns.add<AddIAddConstant, AddISubConstantRHS, AddISubConstantLHS>( patterns.add<AddIAddConstant, AddISubConstantRHS, AddISubConstantLHS>(
context); context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AddUIExtendedOp // AddUIExtendedOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Optional<SmallVector<int64_t, 4>> arith::AddUIExtendedOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>>
arith::AddUIExtendedOp::getShapeForUnroll() {
if (auto vt = getType(0).dyn_cast<VectorType>()) if (auto vt = getType(0).dyn_cast<VectorType>())
return llvm::to_vector<4>(vt.getShape()); return llvm::to_vector<4>(vt.getShape());
return std::nullopt; return std::nullopt;
} }
// Returns the overflow bit, assuming that `sum` is the result of unsigned // Returns the overflow bit, assuming that `sum` is the result of unsigned
// addition of `operand` and another number. // addition of `operand` and another number.
static APInt calculateUnsignedOverflow(const APInt &sum, const APInt &operand) { static APInt calculateUnsignedOverflow(const APInt &sum, const APInt &operand) {
▲ Show 20 Lines • Show All 115 Lines • ▼ Show 20 LinesOpFoldResult arith::MulIOp::fold(ArrayRef<Attribute> operands) {
return constFoldBinaryOp<IntegerAttr>( return constFoldBinaryOp<IntegerAttr>(
operands, [](const APInt &a, const APInt &b) { return a * b; }); operands, [](const APInt &a, const APInt &b) { return a * b; });
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// MulSIExtendedOp // MulSIExtendedOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Optional<SmallVector<int64_t, 4>> arith::MulSIExtendedOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>>
arith::MulSIExtendedOp::getShapeForUnroll() {
if (auto vt = getType(0).dyn_cast<VectorType>()) if (auto vt = getType(0).dyn_cast<VectorType>())
return llvm::to_vector<4>(vt.getShape()); return llvm::to_vector<4>(vt.getShape());
return std::nullopt; return std::nullopt;
} }
LogicalResult LogicalResult
arith::MulSIExtendedOp::fold(ArrayRef<Attribute> operands, arith::MulSIExtendedOp::fold(ArrayRef<Attribute> operands,
SmallVectorImpl<OpFoldResult> &results) { SmallVectorImpl<OpFoldResult> &results) {
Show All 29 Linesvoid arith::MulSIExtendedOp::getCanonicalizationPatterns(
RewritePatternSet &patterns, MLIRContext *context) { RewritePatternSet &patterns, MLIRContext *context) {
patterns.add<MulSIExtendedToMulI, MulSIExtendedRHSOne>(context); patterns.add<MulSIExtendedToMulI, MulSIExtendedRHSOne>(context);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// MulUIExtendedOp // MulUIExtendedOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Optional<SmallVector<int64_t, 4>> arith::MulUIExtendedOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>>
arith::MulUIExtendedOp::getShapeForUnroll() {
if (auto vt = getType(0).dyn_cast<VectorType>()) if (auto vt = getType(0).dyn_cast<VectorType>())
return llvm::to_vector<4>(vt.getShape()); return llvm::to_vector<4>(vt.getShape());
return std::nullopt; return std::nullopt;
} }
LogicalResult LogicalResult
arith::MulUIExtendedOp::fold(ArrayRef<Attribute> operands, arith::MulUIExtendedOp::fold(ArrayRef<Attribute> operands,
SmallVectorImpl<OpFoldResult> &results) { SmallVectorImpl<OpFoldResult> &results) {
▲ Show 20 Lines • Show All 1,198 Lines • ▼ Show 20 Lines
static Attribute getBoolAttribute(Type type, MLIRContext *ctx, bool value) { static Attribute getBoolAttribute(Type type, MLIRContext *ctx, bool value) {
auto boolAttr = BoolAttr::get(ctx, value); auto boolAttr = BoolAttr::get(ctx, value);
ShapedType shapedType = type.dyn_cast_or_null<ShapedType>(); ShapedType shapedType = type.dyn_cast_or_null<ShapedType>();
if (!shapedType) if (!shapedType)
return boolAttr; return boolAttr;
return DenseElementsAttr::get(shapedType, boolAttr); return DenseElementsAttr::get(shapedType, boolAttr);
} }
static Optional<int64_t> getIntegerWidth(Type t) { static std::optional<int64_t> getIntegerWidth(Type t) {
if (auto intType = t.dyn_cast<IntegerType>()) { if (auto intType = t.dyn_cast<IntegerType>()) {
return intType.getWidth(); return intType.getWidth();
} }
if (auto vectorIntType = t.dyn_cast<VectorType>()) { if (auto vectorIntType = t.dyn_cast<VectorType>()) {
return vectorIntType.getElementType().cast<IntegerType>().getWidth(); return vectorIntType.getElementType().cast<IntegerType>().getWidth();
} }
return std::nullopt; return std::nullopt;
} }
OpFoldResult arith::CmpIOp::fold(ArrayRef<Attribute> operands) { OpFoldResult arith::CmpIOp::fold(ArrayRef<Attribute> operands) {
assert(operands.size() == 2 && "cmpi takes two operands"); assert(operands.size() == 2 && "cmpi takes two operands");
// cmpi(pred, x, x) // cmpi(pred, x, x)
if (getLhs() == getRhs()) { if (getLhs() == getRhs()) {
auto val = applyCmpPredicateToEqualOperands(getPredicate()); auto val = applyCmpPredicateToEqualOperands(getPredicate());
return getBoolAttribute(getType(), getContext(), val); return getBoolAttribute(getType(), getContext(), val);
} }
if (matchPattern(getRhs(), m_Zero())) { if (matchPattern(getRhs(), m_Zero())) {
if (auto extOp = getLhs().getDefiningOp<ExtSIOp>()) { if (auto extOp = getLhs().getDefiningOp<ExtSIOp>()) {
// extsi(%x : i1 -> iN) != 0 -> %x // extsi(%x : i1 -> iN) != 0 -> %x
Optional<int64_t> integerWidth = std::optional<int64_t> integerWidth =
getIntegerWidth(extOp.getOperand().getType()); getIntegerWidth(extOp.getOperand().getType());
if (integerWidth && integerWidth.value() == 1 && if (integerWidth && integerWidth.value() == 1 &&
getPredicate() == arith::CmpIPredicate::ne) getPredicate() == arith::CmpIPredicate::ne)
return extOp.getOperand(); return extOp.getOperand();
} }
if (auto extOp = getLhs().getDefiningOp<ExtUIOp>()) { if (auto extOp = getLhs().getDefiningOp<ExtUIOp>()) {
// extui(%x : i1 -> iN) != 0 -> %x // extui(%x : i1 -> iN) != 0 -> %x
Optional<int64_t> integerWidth = std::optional<int64_t> integerWidth =
getIntegerWidth(extOp.getOperand().getType()); getIntegerWidth(extOp.getOperand().getType());
if (integerWidth && integerWidth.value() == 1 && if (integerWidth && integerWidth.value() == 1 &&
getPredicate() == arith::CmpIPredicate::ne) getPredicate() == arith::CmpIPredicate::ne)
return extOp.getOperand(); return extOp.getOperand();
} }
} }
// Move constant to the right side. // Move constant to the right side.
▲ Show 20 Lines • Show All 719 Lines • Show Last 20 Lines

mlir/lib/Dialect/Async/IR/Async.cpp

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 LinesLogicalResult YieldOp::verify() {
if (getOperandTypes() != types) if (getOperandTypes() != types)
return emitOpError("operand types do not match the types returned from " return emitOpError("operand types do not match the types returned from "
"the parent ExecuteOp"); "the parent ExecuteOp");
return success(); return success();
} }
MutableOperandRange MutableOperandRange
YieldOp::getMutableSuccessorOperands(Optional<unsigned> index) { YieldOp::getMutableSuccessorOperands(std::optional<unsigned> index) {
return getOperandsMutable(); return getOperandsMutable();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// ExecuteOp /// ExecuteOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
constexpr char kOperandSegmentSizesAttr[] = "operand_segment_sizes"; constexpr char kOperandSegmentSizesAttr[] = "operand_segment_sizes";
OperandRange ExecuteOp::getSuccessorEntryOperands(Optional<unsigned> index) { OperandRange
ExecuteOp::getSuccessorEntryOperands(std::optional<unsigned> index) {
assert(index && *index == 0 && "invalid region index"); assert(index && *index == 0 && "invalid region index");
return getBodyOperands(); return getBodyOperands();
} }
bool ExecuteOp::areTypesCompatible(Type lhs, Type rhs) { bool ExecuteOp::areTypesCompatible(Type lhs, Type rhs) {
const auto getValueOrTokenType = [](Type type) { const auto getValueOrTokenType = [](Type type) {
if (auto value = type.dyn_cast<ValueType>()) if (auto value = type.dyn_cast<ValueType>())
return value.getValueType(); return value.getValueType();
return type; return type;
}; };
return getValueOrTokenType(lhs) == getValueOrTokenType(rhs); return getValueOrTokenType(lhs) == getValueOrTokenType(rhs);
} }
void ExecuteOp::getSuccessorRegions(Optional<unsigned> index, void ExecuteOp::getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute>, ArrayRef<Attribute>,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// The `body` region branch back to the parent operation. // The `body` region branch back to the parent operation.
if (index) { if (index) {
assert(*index == 0 && "invalid region index"); assert(*index == 0 && "invalid region index");
regions.push_back(RegionSuccessor(getBodyResults())); regions.push_back(RegionSuccessor(getBodyResults()));
return; return;
} }
▲ Show 20 Lines • Show All 397 Lines • Show Last 20 Lines

mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Linesstruct ReplaceStaticShapeDims : OpRewritePattern<AllocTensorOp> {
} }
}; };
struct FoldDimOfAllocTensorOp : public OpRewritePattern<tensor::DimOp> { struct FoldDimOfAllocTensorOp : public OpRewritePattern<tensor::DimOp> {
using OpRewritePattern<tensor::DimOp>::OpRewritePattern; using OpRewritePattern<tensor::DimOp>::OpRewritePattern;
LogicalResult matchAndRewrite(tensor::DimOp dimOp, LogicalResult matchAndRewrite(tensor::DimOp dimOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
Optional<int64_t> maybeConstantIndex = dimOp.getConstantIndex(); std::optional<int64_t> maybeConstantIndex = dimOp.getConstantIndex();
auto allocTensorOp = dimOp.getSource().getDefiningOp<AllocTensorOp>(); auto allocTensorOp = dimOp.getSource().getDefiningOp<AllocTensorOp>();
if (!allocTensorOp || !maybeConstantIndex) if (!allocTensorOp || !maybeConstantIndex)
return failure(); return failure();
if (!allocTensorOp.getType().isDynamicDim(*maybeConstantIndex)) if (!allocTensorOp.getType().isDynamicDim(*maybeConstantIndex))
return failure(); return failure();
rewriter.replaceOp( rewriter.replaceOp(
dimOp, allocTensorOp.getDynamicSize(rewriter, *maybeConstantIndex)); dimOp, allocTensorOp.getDynamicSize(rewriter, *maybeConstantIndex));
return success(); return success();
▲ Show 20 Lines • Show All 348 Lines • Show Last 20 Lines

mlir/lib/Dialect/Bufferization/Transforms/BufferViewFlowAnalysis.cpp

Show First 20 Lines • Show All 98 Lines • ▼ Show 20 Lines op->walk([&](RegionBranchOpInterface regionInterface) {
for (Region &region : regionInterface->getRegions()) { for (Region &region : regionInterface->getRegions()) {
// Iterate over all successor region entries that are reachable from the // Iterate over all successor region entries that are reachable from the
// current region. // current region.
SmallVector<RegionSuccessor, 2> successorRegions; SmallVector<RegionSuccessor, 2> successorRegions;
regionInterface.getSuccessorRegions(region.getRegionNumber(), regionInterface.getSuccessorRegions(region.getRegionNumber(),
successorRegions); successorRegions);
for (RegionSuccessor &successorRegion : successorRegions) { for (RegionSuccessor &successorRegion : successorRegions) {
// Determine the current region index (if any). // Determine the current region index (if any).
Optional<unsigned> regionIndex; std::optional<unsigned> regionIndex;
Region *regionSuccessor = successorRegion.getSuccessor(); Region *regionSuccessor = successorRegion.getSuccessor();
if (regionSuccessor) if (regionSuccessor)
regionIndex = regionSuccessor->getRegionNumber(); regionIndex = regionSuccessor->getRegionNumber();
// Iterate over all immediate terminator operations and wire the // Iterate over all immediate terminator operations and wire the
// successor inputs with the successor operands of each terminator. // successor inputs with the successor operands of each terminator.
for (Block &block : region) { for (Block &block : region) {
auto successorOperands = getRegionBranchSuccessorOperands( auto successorOperands = getRegionBranchSuccessorOperands(
block.getTerminator(), regionIndex); block.getTerminator(), regionIndex);
Show All 15 Lines

mlir/lib/Dialect/ControlFlow/IR/ControlFlowOps.cpp

Show First 20 Lines • Show All 589 Lines • ▼ Show 20 Lines
SuccessorOperands SwitchOp::getSuccessorOperands(unsigned index) { SuccessorOperands SwitchOp::getSuccessorOperands(unsigned index) {
assert(index < getNumSuccessors() && "invalid successor index"); assert(index < getNumSuccessors() && "invalid successor index");
return SuccessorOperands(index == 0 ? getDefaultOperandsMutable() return SuccessorOperands(index == 0 ? getDefaultOperandsMutable()
: getCaseOperandsMutable(index - 1)); : getCaseOperandsMutable(index - 1));
} }
Block *SwitchOp::getSuccessorForOperands(ArrayRef<Attribute> operands) { Block *SwitchOp::getSuccessorForOperands(ArrayRef<Attribute> operands) {
Optional<DenseIntElementsAttr> caseValues = getCaseValues(); std::optional<DenseIntElementsAttr> caseValues = getCaseValues();
if (!caseValues) if (!caseValues)
return getDefaultDestination(); return getDefaultDestination();
SuccessorRange caseDests = getCaseDestinations(); SuccessorRange caseDests = getCaseDestinations();
if (auto value = operands.front().dyn_cast_or_null<IntegerAttr>()) { if (auto value = operands.front().dyn_cast_or_null<IntegerAttr>()) {
for (const auto &it : llvm::enumerate(caseValues->getValues<APInt>())) for (const auto &it : llvm::enumerate(caseValues->getValues<APInt>()))
if (it.value() == value.getValue()) if (it.value() == value.getValue())
▲ Show 20 Lines • Show All 193 Lines • ▼ Show 20 LinessimplifySwitchFromSwitchOnSameCondition(SwitchOp op,
if (!predSwitch || op.getFlag() != predSwitch.getFlag() || if (!predSwitch || op.getFlag() != predSwitch.getFlag() ||
predSwitch.getDefaultDestination() == currentBlock) predSwitch.getDefaultDestination() == currentBlock)
return failure(); return failure();
// Fold this switch to an unconditional branch. // Fold this switch to an unconditional branch.
SuccessorRange predDests = predSwitch.getCaseDestinations(); SuccessorRange predDests = predSwitch.getCaseDestinations();
auto it = llvm::find(predDests, currentBlock); auto it = llvm::find(predDests, currentBlock);
if (it != predDests.end()) { if (it != predDests.end()) {
Optional<DenseIntElementsAttr> predCaseValues = predSwitch.getCaseValues(); std::optional<DenseIntElementsAttr> predCaseValues =
predSwitch.getCaseValues();
foldSwitch(op, rewriter, foldSwitch(op, rewriter,
predCaseValues->getValues<APInt>()[it - predDests.begin()]); predCaseValues->getValues<APInt>()[it - predDests.begin()]);
} else { } else {
rewriter.replaceOpWithNewOp<BranchOp>(op, op.getDefaultDestination(), rewriter.replaceOpWithNewOp<BranchOp>(op, op.getDefaultDestination(),
op.getDefaultOperands()); op.getDefaultOperands());
} }
return success(); return success();
} }
▲ Show 20 Lines • Show All 88 Lines • Show Last 20 Lines

mlir/lib/Dialect/EmitC/IR/EmitC.cpp

Show First 20 Lines • Show All 78 Lines • ▼ Show 20 Lines
// CallOp // CallOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
LogicalResult emitc::CallOp::verify() { LogicalResult emitc::CallOp::verify() {
// Callee must not be empty. // Callee must not be empty.
if (getCallee().empty()) if (getCallee().empty())
return emitOpError("callee must not be empty"); return emitOpError("callee must not be empty");
if (Optional<ArrayAttr> argsAttr = getArgs()) { if (std::optional<ArrayAttr> argsAttr = getArgs()) {
for (Attribute arg : *argsAttr) { for (Attribute arg : *argsAttr) {
auto intAttr = arg.dyn_cast<IntegerAttr>(); auto intAttr = arg.dyn_cast<IntegerAttr>();
if (intAttr && intAttr.getType().isa<IndexType>()) { if (intAttr && intAttr.getType().isa<IndexType>()) {
int64_t index = intAttr.getInt(); int64_t index = intAttr.getInt();
// Args with elements of type index must be in range // Args with elements of type index must be in range
// [0..operands.size). // [0..operands.size).
if ((index < 0) || (index >= static_cast<int64_t>(getNumOperands()))) if ((index < 0) || (index >= static_cast<int64_t>(getNumOperands())))
return emitOpError("index argument is out of range"); return emitOpError("index argument is out of range");
// Args with elements of type ArrayAttr must have a type. // Args with elements of type ArrayAttr must have a type.
} else if (arg.isa<ArrayAttr>() /*&& arg.getType().isa<NoneType>()*/) { } else if (arg.isa<ArrayAttr>() /*&& arg.getType().isa<NoneType>()*/) {
// FIXME: Array attributes never have types // FIXME: Array attributes never have types
return emitOpError("array argument has no type"); return emitOpError("array argument has no type");
} }
} }
} }
if (Optional<ArrayAttr> templateArgsAttr = getTemplateArgs()) { if (std::optional<ArrayAttr> templateArgsAttr = getTemplateArgs()) {
for (Attribute tArg : *templateArgsAttr) { for (Attribute tArg : *templateArgsAttr) {
if (!tArg.isa<TypeAttr, IntegerAttr, FloatAttr, emitc::OpaqueAttr>()) if (!tArg.isa<TypeAttr, IntegerAttr, FloatAttr, emitc::OpaqueAttr>())
return emitOpError("template argument has invalid type"); return emitOpError("template argument has invalid type");
} }
} }
return success(); return success();
} }
▲ Show 20 Lines • Show All 145 Lines • Show Last 20 Lines

mlir/lib/Dialect/GPU/IR/GPUDialect.cpp

Show First 20 Lines • Show All 361 Lines • ▼ Show 20 LinesLogicalResult gpu::AllReduceOp::verifyRegions() {
return success(); return success();
} }
// TODO: Support optional custom attributes (without dialect prefix). // TODO: Support optional custom attributes (without dialect prefix).
static ParseResult parseAllReduceOperation(AsmParser &parser, static ParseResult parseAllReduceOperation(AsmParser &parser,
AllReduceOperationAttr &attr) { AllReduceOperationAttr &attr) {
StringRef enumStr; StringRef enumStr;
if (!parser.parseOptionalKeyword(&enumStr)) { if (!parser.parseOptionalKeyword(&enumStr)) {
Optional<AllReduceOperation> op = gpu::symbolizeAllReduceOperation(enumStr); std::optional<AllReduceOperation> op =
gpu::symbolizeAllReduceOperation(enumStr);
if (!op) if (!op)
return parser.emitError(parser.getCurrentLocation(), "invalid op kind"); return parser.emitError(parser.getCurrentLocation(), "invalid op kind");
attr = AllReduceOperationAttr::get(parser.getContext(), *op); attr = AllReduceOperationAttr::get(parser.getContext(), *op);
} }
return success(); return success();
} }
static void printAllReduceOperation(AsmPrinter &printer, Operation *op, static void printAllReduceOperation(AsmPrinter &printer, Operation *op,
▲ Show 20 Lines • Show All 1,034 Lines • Show Last 20 Lines

mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp

Show All 20 Lines
using namespace mlir; using namespace mlir;
using namespace mlir::gpu; using namespace mlir::gpu;
using namespace mlir::transform; using namespace mlir::transform;
/// Check if given mapping attributes are one of the desired attributes /// Check if given mapping attributes are one of the desired attributes
static DiagnosedSilenceableFailure static DiagnosedSilenceableFailure
checkAttributeType(ArrayRef<DeviceMappingAttrInterface> threadMappingAttributes, checkAttributeType(ArrayRef<DeviceMappingAttrInterface> threadMappingAttributes,
const Optional<ArrayAttr> &foreachMapping, const std::optional<ArrayAttr> &foreachMapping,
Optional<TransformOpInterface> transformOp) { std::optional<TransformOpInterface> transformOp) {
if (!foreachMapping.has_value()) if (!foreachMapping.has_value())
return transformOp->emitSilenceableError() << "mapping must be present"; return transformOp->emitSilenceableError() << "mapping must be present";
DenseSet<Attribute> seen; DenseSet<Attribute> seen;
for (Attribute map : foreachMapping->getValue()) { for (Attribute map : foreachMapping->getValue()) {
if (!llvm::is_contained(threadMappingAttributes, map)) { if (!llvm::is_contained(threadMappingAttributes, map)) {
return transformOp->emitDefiniteFailure() return transformOp->emitDefiniteFailure()
<< "mapping must be one of " << threadMappingAttributes; << "mapping must be one of " << threadMappingAttributes;
} }
if (llvm::is_contained(seen, map)) { if (llvm::is_contained(seen, map)) {
return transformOp->emitDefiniteFailure() return transformOp->emitDefiniteFailure()
<< map << map
<< " is duplicated, cannot map different " << " is duplicated, cannot map different "
"loops to the same processor"; "loops to the same processor";
} }
seen.insert(map); seen.insert(map);
} }
return DiagnosedSilenceableFailure::success(); return DiagnosedSilenceableFailure::success();
} }
/// Determines if the size of the kernel configuration is supported by the GPU /// Determines if the size of the kernel configuration is supported by the GPU
/// architecture being used. It presently makes use of CUDA limitations, however /// architecture being used. It presently makes use of CUDA limitations, however
/// that aspect may be enhanced for other GPUs. /// that aspect may be enhanced for other GPUs.
static DiagnosedSilenceableFailure static DiagnosedSilenceableFailure checkGpuLimits(
checkGpuLimits(TransformOpInterface transformOp, Optional<int64_t> gridDimX, TransformOpInterface transformOp, std::optional<int64_t> gridDimX,
Optional<int64_t> gridDimY, Optional<int64_t> gridDimZ, std::optional<int64_t> gridDimY, std::optional<int64_t> gridDimZ,
Optional<int64_t> blockDimX, Optional<int64_t> blockDimY, std::optional<int64_t> blockDimX, std::optional<int64_t> blockDimY,
Optional<int64_t> blockDimZ) { std::optional<int64_t> blockDimZ) {
static constexpr int maxTotalBlockdim = 1024; static constexpr int maxTotalBlockdim = 1024;
static constexpr int maxBlockdimx = 1024; static constexpr int maxBlockdimx = 1024;
static constexpr int maxBlockdimy = 1024; static constexpr int maxBlockdimy = 1024;
static constexpr int maxBlockdimz = 64; static constexpr int maxBlockdimz = 64;
static constexpr int maxTotalGriddim = 2147483647; static constexpr int maxTotalGriddim = 2147483647;
static constexpr int maxGriddimx = 2147483647; static constexpr int maxGriddimx = 2147483647;
static constexpr int maxGriddimy = 65535; static constexpr int maxGriddimy = 65535;
Show All 19 Linesstatic DiagnosedSilenceableFailure checkGpuLimits(
return DiagnosedSilenceableFailure::success(); return DiagnosedSilenceableFailure::success();
} }
/// Creates an empty-body gpu::LaunchOp using the provided kernel settings and /// Creates an empty-body gpu::LaunchOp using the provided kernel settings and
/// put a terminator within. /// put a terminator within.
static DiagnosedSilenceableFailure static DiagnosedSilenceableFailure
createGpuLaunch(RewriterBase &rewriter, Location loc, createGpuLaunch(RewriterBase &rewriter, Location loc,
TransformOpInterface transformOp, LaunchOp &launchOp, TransformOpInterface transformOp, LaunchOp &launchOp,
Optional<int64_t> gridDimX = std::nullopt, std::optional<int64_t> gridDimX = std::nullopt,
Optional<int64_t> gridDimY = std::nullopt, std::optional<int64_t> gridDimY = std::nullopt,
Optional<int64_t> gridDimZ = std::nullopt, std::optional<int64_t> gridDimZ = std::nullopt,
Optional<int64_t> blockDimX = std::nullopt, std::optional<int64_t> blockDimX = std::nullopt,
Optional<int64_t> blockDimY = std::nullopt, std::optional<int64_t> blockDimY = std::nullopt,
Optional<int64_t> blockDimZ = std::nullopt) { std::optional<int64_t> blockDimZ = std::nullopt) {
DiagnosedSilenceableFailure diag = DiagnosedSilenceableFailure diag =
checkGpuLimits(transformOp, gridDimX, gridDimY, gridDimZ, blockDimX, checkGpuLimits(transformOp, gridDimX, gridDimY, gridDimZ, blockDimX,
blockDimY, blockDimZ); blockDimY, blockDimZ);
if (!diag.succeeded()) if (!diag.succeeded())
return diag; return diag;
auto createConst = [&](int dim) { auto createConst = [&](int dim) {
return rewriter.create<arith::ConstantIndexOp>(loc, dim); return rewriter.create<arith::ConstantIndexOp>(loc, dim);
Show All 12 LinescreateGpuLaunch(RewriterBase &rewriter, Location loc,
rewriter.create<TerminatorOp>(loc); rewriter.create<TerminatorOp>(loc);
return DiagnosedSilenceableFailure::success(); return DiagnosedSilenceableFailure::success();
} }
/// Alter kernel configuration of the given kernel. /// Alter kernel configuration of the given kernel.
static DiagnosedSilenceableFailure static DiagnosedSilenceableFailure
alterGpuLaunch(TrivialPatternRewriter &rewriter, LaunchOp gpuLaunch, alterGpuLaunch(TrivialPatternRewriter &rewriter, LaunchOp gpuLaunch,
TransformOpInterface transformOp, TransformOpInterface transformOp,
Optional<int64_t> gridDimX = std::nullopt, std::optional<int64_t> gridDimX = std::nullopt,
Optional<int64_t> gridDimY = std::nullopt, std::optional<int64_t> gridDimY = std::nullopt,
Optional<int64_t> gridDimZ = std::nullopt, std::optional<int64_t> gridDimZ = std::nullopt,
Optional<int64_t> blockDimX = std::nullopt, std::optional<int64_t> blockDimX = std::nullopt,
Optional<int64_t> blockDimY = std::nullopt, std::optional<int64_t> blockDimY = std::nullopt,
Optional<int64_t> blockDimZ = std::nullopt) { std::optional<int64_t> blockDimZ = std::nullopt) {
DiagnosedSilenceableFailure diag = DiagnosedSilenceableFailure diag =
checkGpuLimits(transformOp, gridDimX, gridDimY, gridDimZ, blockDimX, checkGpuLimits(transformOp, gridDimX, gridDimY, gridDimZ, blockDimX,
blockDimY, blockDimZ); blockDimY, blockDimZ);
if (!diag.succeeded()) if (!diag.succeeded())
return diag; return diag;
KernelDim3 currentBlockdim = gpuLaunch.getBlockSizeOperandValues(); KernelDim3 currentBlockdim = gpuLaunch.getBlockSizeOperandValues();
OpBuilder::InsertionGuard guard(rewriter); OpBuilder::InsertionGuard guard(rewriter);
▲ Show 20 Lines • Show All 222 Lines • ▼ Show 20 Lines
/// `scf.foreach_thread` are rewritten to gpu.thread_id according to the /// `scf.foreach_thread` are rewritten to gpu.thread_id according to the
/// thread_dim_apping attribute. Sibling `scf.foreach_thread` are supported in /// thread_dim_apping attribute. Sibling `scf.foreach_thread` are supported in
/// which case, the union of the number of threads is computed and may result /// which case, the union of the number of threads is computed and may result
/// in predication. Dynamic, `scf.foreach_thread` trip counts are currently /// in predication. Dynamic, `scf.foreach_thread` trip counts are currently
/// not supported. Dynamic block dim sizes are currently not supported. /// not supported. Dynamic block dim sizes are currently not supported.
static DiagnosedSilenceableFailure rewriteOneForeachThreadToGpuThreads( static DiagnosedSilenceableFailure rewriteOneForeachThreadToGpuThreads(
RewriterBase &rewriter, scf::ForeachThreadOp foreachThreadOp, RewriterBase &rewriter, scf::ForeachThreadOp foreachThreadOp,
const SmallVectorImpl<int64_t> &globalBlockDims, bool syncAfterDistribute, const SmallVectorImpl<int64_t> &globalBlockDims, bool syncAfterDistribute,
llvm::Optional<TransformOpInterface> transformOp, std::optional<TransformOpInterface> transformOp,
const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes) { const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes) {
// Step 0. Target-specific verifications. There is no good place to anchor // Step 0. Target-specific verifications. There is no good place to anchor
// those right now: the ForeachThreadOp is target-independent and the // those right now: the ForeachThreadOp is target-independent and the
// transform op does not apply to individual ForeachThreadOp. // transform op does not apply to individual ForeachThreadOp.
auto failureHelper = auto failureHelper =
[&](const Twine &message) -> DiagnosedSilenceableFailure { [&](const Twine &message) -> DiagnosedSilenceableFailure {
if (transformOp.has_value()) { if (transformOp.has_value()) {
return transformOp->emitSilenceableError() << message; return transformOp->emitSilenceableError() << message;
▲ Show 20 Lines • Show All 115 Lines • ▼ Show 20 Linesstatic DiagnosedSilenceableFailure rewriteOneForeachThreadToGpuThreads(
rewriter.eraseOp(foreachThreadOp); rewriter.eraseOp(foreachThreadOp);
return DiagnosedSilenceableFailure::success(); return DiagnosedSilenceableFailure::success();
} }
DiagnosedSilenceableFailure mlir::transform::gpu::mapNestedForeachToThreadsImpl( DiagnosedSilenceableFailure mlir::transform::gpu::mapNestedForeachToThreadsImpl(
RewriterBase &rewriter, Operation *target, RewriterBase &rewriter, Operation *target,
const SmallVectorImpl<int64_t> &blockDim, bool syncAfterDistribute, const SmallVectorImpl<int64_t> &blockDim, bool syncAfterDistribute,
llvm::Optional<TransformOpInterface> transformOp, std::optional<TransformOpInterface> transformOp,
const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes) { const ArrayRef<DeviceMappingAttrInterface> &threadMappingAttributes) {
DiagnosedSilenceableFailure diag = DiagnosedSilenceableFailure::success(); DiagnosedSilenceableFailure diag = DiagnosedSilenceableFailure::success();
target->walk([&](scf::ForeachThreadOp foreachThreadOp) { target->walk([&](scf::ForeachThreadOp foreachThreadOp) {
diag = checkAttributeType(threadMappingAttributes, diag = checkAttributeType(threadMappingAttributes,
foreachThreadOp.getMapping(), transformOp); foreachThreadOp.getMapping(), transformOp);
if (diag.succeeded()) { if (diag.succeeded()) {
rewriter.setInsertionPoint(foreachThreadOp); rewriter.setInsertionPoint(foreachThreadOp);
diag = rewriteOneForeachThreadToGpuThreads( diag = rewriteOneForeachThreadToGpuThreads(
▲ Show 20 Lines • Show All 85 Lines • Show Last 20 Lines

mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp

Show First 20 Lines • Show All 141 Lines • ▼ Show 20 Lines if (parser.getCurrentLocation(&predicateLoc) ||
parser.getCurrentLocation(&trailingTypeLoc) || parser.parseType(type) || parser.getCurrentLocation(&trailingTypeLoc) || parser.parseType(type) ||
parser.resolveOperand(lhs, type, result.operands) || parser.resolveOperand(lhs, type, result.operands) ||
parser.resolveOperand(rhs, type, result.operands)) parser.resolveOperand(rhs, type, result.operands))
return failure(); return failure();
// Replace the string attribute `predicate` with an integer attribute. // Replace the string attribute `predicate` with an integer attribute.
int64_t predicateValue = 0; int64_t predicateValue = 0;
if (std::is_same<CmpPredicateType, ICmpPredicate>()) { if (std::is_same<CmpPredicateType, ICmpPredicate>()) {
Optional<ICmpPredicate> predicate = std::optional<ICmpPredicate> predicate =
symbolizeICmpPredicate(predicateAttr.getValue()); symbolizeICmpPredicate(predicateAttr.getValue());
if (!predicate) if (!predicate)
return parser.emitError(predicateLoc) return parser.emitError(predicateLoc)
<< "'" << predicateAttr.getValue() << "'" << predicateAttr.getValue()
<< "' is an incorrect value of the 'predicate' attribute"; << "' is an incorrect value of the 'predicate' attribute";
predicateValue = static_cast<int64_t>(*predicate); predicateValue = static_cast<int64_t>(*predicate);
} else { } else {
Optional<FCmpPredicate> predicate = std::optional<FCmpPredicate> predicate =
symbolizeFCmpPredicate(predicateAttr.getValue()); symbolizeFCmpPredicate(predicateAttr.getValue());
if (!predicate) if (!predicate)
return parser.emitError(predicateLoc) return parser.emitError(predicateLoc)
<< "'" << predicateAttr.getValue() << "'" << predicateAttr.getValue()
<< "' is an incorrect value of the 'predicate' attribute"; << "' is an incorrect value of the 'predicate' attribute";
predicateValue = static_cast<int64_t>(*predicate); predicateValue = static_cast<int64_t>(*predicate);
} }
▲ Show 20 Lines • Show All 81 Lines • ▼ Show 20 LinesParseResult AllocaOp::parse(OpAsmParser &parser, OperationState &result) {
result.addTypes({funcType.getResult(0)}); result.addTypes({funcType.getResult(0)});
return success(); return success();
} }
/// Checks that the elemental type is present in either the pointer type or /// Checks that the elemental type is present in either the pointer type or
/// the attribute, but not both. /// the attribute, but not both.
static LogicalResult verifyOpaquePtr(Operation *op, LLVMPointerType ptrType, static LogicalResult verifyOpaquePtr(Operation *op, LLVMPointerType ptrType,
Optional<Type> ptrElementType) { std::optional<Type> ptrElementType) {
if (ptrType.isOpaque() && !ptrElementType.has_value()) { if (ptrType.isOpaque() && !ptrElementType.has_value()) {
return op->emitOpError() << "expected '" << kElemTypeAttrName return op->emitOpError() << "expected '" << kElemTypeAttrName
<< "' attribute if opaque pointer type is used"; << "' attribute if opaque pointer type is used";
} }
if (!ptrType.isOpaque() && ptrElementType.has_value()) { if (!ptrType.isOpaque() && ptrElementType.has_value()) {
return op->emitOpError() return op->emitOpError()
<< "unexpected '" << kElemTypeAttrName << "unexpected '" << kElemTypeAttrName
<< "' attribute when non-opaque pointer type is used"; << "' attribute when non-opaque pointer type is used";
▲ Show 20 Lines • Show All 395 Lines • ▼ Show 20 LinesLogicalResult LLVM::GEPOp::verify() {
if (llvm::Error err = if (llvm::Error err =
verifyStructIndices(getSourceElementType(), getIndices())) verifyStructIndices(getSourceElementType(), getIndices()))
return emitOpError() << llvm::toString(std::move(err)); return emitOpError() << llvm::toString(std::move(err));
return success(); return success();
} }
Type LLVM::GEPOp::getSourceElementType() { Type LLVM::GEPOp::getSourceElementType() {
if (Optional<Type> elemType = getElemType()) if (std::optional<Type> elemType = getElemType())
return *elemType; return *elemType;
return extractVectorElementType(getBase().getType()) return extractVectorElementType(getBase().getType())
.cast<LLVMPointerType>() .cast<LLVMPointerType>()
.getElementType(); .getElementType();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 1,171 Lines • ▼ Show 20 LinesLogicalResult GlobalOp::verify() {
if (getLinkage() == Linkage::Appending) { if (getLinkage() == Linkage::Appending) {
if (!getType().isa<LLVMArrayType>()) { if (!getType().isa<LLVMArrayType>()) {
return emitOpError() << "expected array type for '" return emitOpError() << "expected array type for '"
<< stringifyLinkage(Linkage::Appending) << stringifyLinkage(Linkage::Appending)
<< "' linkage"; << "' linkage";
} }
} }
Optional<uint64_t> alignAttr = getAlignment(); std::optional<uint64_t> alignAttr = getAlignment();
if (alignAttr.has_value()) { if (alignAttr.has_value()) {
uint64_t value = alignAttr.value(); uint64_t value = alignAttr.value();
if (!llvm::isPowerOf2_64(value)) if (!llvm::isPowerOf2_64(value))
return emitError() << "alignment attribute is not a power of 2"; return emitError() << "alignment attribute is not a power of 2";
} }
return success(); return success();
} }
▲ Show 20 Lines • Show All 1,054 Lines • Show Last 20 Lines

mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp

Show First 20 Lines • Show All 71 Lines • ▼ Show 20 LinesLogicalResult CpAsyncOp::verify() {
if (getBypassL1() && getSize() != 16) if (getBypassL1() && getSize() != 16)
return emitError("bypass l1 is only support for 16 bytes copy."); return emitError("bypass l1 is only support for 16 bytes copy.");
return success(); return success();
} }
// Given the element type of an operand and whether or not it is an accumulator, // Given the element type of an operand and whether or not it is an accumulator,
// this function returns the PTX type (`NVVM::MMATypes`) that corresponds to the // this function returns the PTX type (`NVVM::MMATypes`) that corresponds to the
// operand's element type. // operand's element type.
Optional<mlir::NVVM::MMATypes> MmaOp::inferOperandMMAType(Type operandElType, std::optional<mlir::NVVM::MMATypes>
bool isAccumulator) { MmaOp::inferOperandMMAType(Type operandElType, bool isAccumulator) {
auto half2Type = auto half2Type =
LLVM::getFixedVectorType(Float16Type::get(operandElType.getContext()), 2); LLVM::getFixedVectorType(Float16Type::get(operandElType.getContext()), 2);
if (operandElType.isF64()) if (operandElType.isF64())
return NVVM::MMATypes::f64; return NVVM::MMATypes::f64;
if (operandElType.isF16() || operandElType == half2Type) if (operandElType.isF16() || operandElType == half2Type)
return NVVM::MMATypes::f16; return NVVM::MMATypes::f16;
if (operandElType.isF32() && isAccumulator) if (operandElType.isF32() && isAccumulator)
return NVVM::MMATypes::f32; return NVVM::MMATypes::f32;
Show All 23 Lines
} }
static bool isIntegerPtxType(MMATypes type) { static bool isIntegerPtxType(MMATypes type) {
return isInt4PtxType(type) || isInt8PtxType(type) || type == MMATypes::b1 || return isInt4PtxType(type) || isInt8PtxType(type) || type == MMATypes::b1 ||
type == MMATypes::s32; type == MMATypes::s32;
} }
MMATypes MmaOp::accumPtxType() { MMATypes MmaOp::accumPtxType() {
Optional<mlir::NVVM::MMATypes> val = inferOperandMMAType( std::optional<mlir::NVVM::MMATypes> val = inferOperandMMAType(
getODSOperands(2).getTypes().front(), /*isAccum=*/true); getODSOperands(2).getTypes().front(), /*isAccum=*/true);
assert(val.has_value() && "accumulator PTX type should always be inferrable"); assert(val.has_value() && "accumulator PTX type should always be inferrable");
return val.value(); return val.value();
} }
MMATypes MmaOp::resultPtxType() { MMATypes MmaOp::resultPtxType() {
Optional<mlir::NVVM::MMATypes> val = std::optional<mlir::NVVM::MMATypes> val =
inferOperandMMAType(getResult().getType(), /*isAccum=*/true); inferOperandMMAType(getResult().getType(), /*isAccum=*/true);
assert(val.has_value() && "result PTX type should always be inferrable"); assert(val.has_value() && "result PTX type should always be inferrable");
return val.value(); return val.value();
} }
void MmaOp::print(OpAsmPrinter &p) { void MmaOp::print(OpAsmPrinter &p) {
SmallVector<Type, 4> regTypes; SmallVector<Type, 4> regTypes;
struct OperandFragment { struct OperandFragment {
Show All 17 Lines for (unsigned fragIdx = 0; fragIdx < frags.size(); fragIdx++) {
for (auto operandIdx = varOperandSpec.first; for (auto operandIdx = varOperandSpec.first;
operandIdx < varOperandSpec.first + varOperandSpec.second; operandIdx < varOperandSpec.first + varOperandSpec.second;
operandIdx++) { operandIdx++) {
frag.regs.push_back(this->getOperand(operandIdx)); frag.regs.push_back(this->getOperand(operandIdx));
if (operandIdx == 0) { if (operandIdx == 0) {
regTypes.push_back(this->getOperand(operandIdx).getType()); regTypes.push_back(this->getOperand(operandIdx).getType());
} }
} }
Optional<MMATypes> inferredType = std::optional<MMATypes> inferredType =
inferOperandMMAType(regTypes.back(), /*isAccum=*/fragIdx >= 2); inferOperandMMAType(regTypes.back(), /*isAccum=*/fragIdx >= 2);
if (inferredType) if (inferredType)
ignoreAttrNames.push_back(frag.ptxTypeAttr); ignoreAttrNames.push_back(frag.ptxTypeAttr);
} }
auto printMmaOperand = [&](const OperandFragment &frag) -> void { auto printMmaOperand = [&](const OperandFragment &frag) -> void {
p << " " << frag.operandName; p << " " << frag.operandName;
p << "["; p << "[";
Show All 15 Lines llvm::interleaveComma(SmallVector<Type, 3>{frags[0].regs[0].getType(),
frags[2].regs[0].getType()}, frags[2].regs[0].getType()},
p); p);
p << ")"; p << ")";
p.printArrowTypeList(TypeRange{this->getRes().getType()}); p.printArrowTypeList(TypeRange{this->getRes().getType()});
} }
void MmaOp::build(OpBuilder &builder, OperationState &result, Type resultType, void MmaOp::build(OpBuilder &builder, OperationState &result, Type resultType,
ValueRange operandA, ValueRange operandB, ValueRange operandC, ValueRange operandA, ValueRange operandB, ValueRange operandC,
ArrayRef<int64_t> shape, Optional<MMAB1Op> b1Op, ArrayRef<int64_t> shape, std::optional<MMAB1Op> b1Op,
Optional<MMAIntOverflow> intOverflow, std::optional<MMAIntOverflow> intOverflow,
Optional<std::array<MMATypes, 2>> multiplicandPtxTypes, std::optional<std::array<MMATypes, 2>> multiplicandPtxTypes,
Optional<std::array<MMALayout, 2>> multiplicandLayouts) { std::optional<std::array<MMALayout, 2>> multiplicandLayouts) {
assert(shape.size() == 3 && "expected shape to have size 3 (m, n, k)"); assert(shape.size() == 3 && "expected shape to have size 3 (m, n, k)");
MLIRContext *ctx = builder.getContext(); MLIRContext *ctx = builder.getContext();
result.addAttribute( result.addAttribute(
"shape", builder.getAttr<MMAShapeAttr>(shape[0], shape[1], shape[2])); "shape", builder.getAttr<MMAShapeAttr>(shape[0], shape[1], shape[2]));
result.addOperands(operandA); result.addOperands(operandA);
result.addOperands(operandB); result.addOperands(operandB);
Show All 36 Lines
} }
// <operation> := // <operation> :=
// A `[` $operandA `]` B `[` $operandB `]` C `[` $operandC `]` // A `[` $operandA `]` B `[` $operandB `]` C `[` $operandC `]`
// attr-dict : (type($operandA[0]), type($operandB[0]), type($operandC[0])) // attr-dict : (type($operandA[0]), type($operandB[0]), type($operandC[0]))
// `->` type($res) // `->` type($res)
ParseResult MmaOp::parse(OpAsmParser &parser, OperationState &result) { ParseResult MmaOp::parse(OpAsmParser &parser, OperationState &result) {
struct OperandFragment { struct OperandFragment {
Optional<MMATypes> elemtype; std::optional<MMATypes> elemtype;
SmallVector<OpAsmParser::UnresolvedOperand, 4> regs; SmallVector<OpAsmParser::UnresolvedOperand, 4> regs;
SmallVector<Type> regTypes; SmallVector<Type> regTypes;
}; };
Builder &builder = parser.getBuilder(); Builder &builder = parser.getBuilder();
std::array<OperandFragment, 4> frags; std::array<OperandFragment, 4> frags;
NamedAttrList namedAttributes; NamedAttrList namedAttributes;
▲ Show 20 Lines • Show All 458 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp

Show First 20 Lines • Show All 1,563 Lines • ▼ Show 20 Linesstatic DiagnosedSilenceableFailure unpackPDLOperations(
return DiagnosedSilenceableFailure::success(); return DiagnosedSilenceableFailure::success();
} }
DiagnosedSilenceableFailure transform::tileToForeachThreadOpImpl( DiagnosedSilenceableFailure transform::tileToForeachThreadOpImpl(
RewriterBase &rewriter, transform::TransformState &state, RewriterBase &rewriter, transform::TransformState &state,
TransformOpInterface transformOp, ArrayRef<Operation *> targets, TransformOpInterface transformOp, ArrayRef<Operation *> targets,
ArrayRef<OpFoldResult> mixedNumThreads, ArrayRef<OpFoldResult> mixedNumThreads,
ArrayRef<OpFoldResult> mixedTileSizes, Optional<ArrayAttr> mapping, ArrayRef<OpFoldResult> mixedTileSizes, std::optional<ArrayAttr> mapping,
SmallVector<Operation *> &tileOps, SmallVector<Operation *> &tiledOps) { SmallVector<Operation *> &tileOps, SmallVector<Operation *> &tiledOps) {
if (targets.empty()) if (targets.empty())
return DiagnosedSilenceableFailure::success(); return DiagnosedSilenceableFailure::success();
// getMixedNumThreads are OpFoldResults[index attributes or PDL operation]. // getMixedNumThreads are OpFoldResults[index attributes or PDL operation].
// Convert to OpFoldResults[index attributes or payload op]. // Convert to OpFoldResults[index attributes or payload op].
SmallVector<OpFoldResult> numThreads; SmallVector<OpFoldResult> numThreads;
DiagnosedSilenceableFailure status = DiagnosedSilenceableFailure status =
▲ Show 20 Lines • Show All 438 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Transforms/Promotion.cpp

Show All 38 Lines
/// Alloc a new buffer of `size` * `width` i8; where `width` is given by the /// Alloc a new buffer of `size` * `width` i8; where `width` is given by the
/// data `layout` for `elementType`. /// data `layout` for `elementType`.
/// Use AllocOp or AllocaOp depending on `options`. /// Use AllocOp or AllocaOp depending on `options`.
/// Take an optional alignment. /// Take an optional alignment.
static Value allocBuffer(ImplicitLocOpBuilder &b, static Value allocBuffer(ImplicitLocOpBuilder &b,
const LinalgPromotionOptions &options, const LinalgPromotionOptions &options,
Type elementType, Value allocSize, DataLayout &layout, Type elementType, Value allocSize, DataLayout &layout,
Optional<unsigned> alignment = std::nullopt) { std::optional<unsigned> alignment = std::nullopt) {
auto width = layout.getTypeSize(elementType); auto width = layout.getTypeSize(elementType);
IntegerAttr alignmentAttr; IntegerAttr alignmentAttr;
if (alignment.has_value()) if (alignment.has_value())
alignmentAttr = b.getI64IntegerAttr(alignment.value()); alignmentAttr = b.getI64IntegerAttr(alignment.value());
// Static buffer. // Static buffer.
if (auto cst = allocSize.getDefiningOp<arith::ConstantIndexOp>()) { if (auto cst = allocSize.getDefiningOp<arith::ConstantIndexOp>()) {
Show All 16 Lines if (options.useAlloca)
return b.create<memref::AllocaOp>(dynamicBufferType, mul, alignmentAttr); return b.create<memref::AllocaOp>(dynamicBufferType, mul, alignmentAttr);
return b.create<memref::AllocOp>(dynamicBufferType, mul, alignmentAttr); return b.create<memref::AllocOp>(dynamicBufferType, mul, alignmentAttr);
} }
/// Default allocation callback function. This allocates a promoted buffer when /// Default allocation callback function. This allocates a promoted buffer when
/// no call back to do so is provided. The default is to allocate a /// no call back to do so is provided. The default is to allocate a
/// memref<..xi8> and return a view to get a memref type of shape /// memref<..xi8> and return a view to get a memref type of shape
/// boundingSubViewSize. /// boundingSubViewSize.
static Optional<Value> static std::optional<Value> defaultAllocBufferCallBack(
defaultAllocBufferCallBack(const LinalgPromotionOptions &options, const LinalgPromotionOptions &options, OpBuilder &builder,
OpBuilder &builder, memref::SubViewOp subView, memref::SubViewOp subView, ArrayRef<Value> boundingSubViewSize,
ArrayRef<Value> boundingSubViewSize, std::optional<unsigned> alignment, DataLayout &layout) {
Optional<unsigned> alignment, DataLayout &layout) {
ShapedType viewType = subView.getType(); ShapedType viewType = subView.getType();
ImplicitLocOpBuilder b(subView.getLoc(), builder); ImplicitLocOpBuilder b(subView.getLoc(), builder);
auto zero = b.createOrFold<arith::ConstantIndexOp>(0); auto zero = b.createOrFold<arith::ConstantIndexOp>(0);
auto one = b.createOrFold<arith::ConstantIndexOp>(1); auto one = b.createOrFold<arith::ConstantIndexOp>(1);
Value allocSize = one; Value allocSize = one;
for (const auto &size : llvm::enumerate(boundingSubViewSize)) for (const auto &size : llvm::enumerate(boundingSubViewSize))
allocSize = b.createOrFold<arith::MulIOp>(allocSize, size.value()); allocSize = b.createOrFold<arith::MulIOp>(allocSize, size.value());
Show All 38 Linesstruct LinalgOpInstancePromotionOptions {
/// Callback functions for allocation and deallocation of promoted buffers, as /// Callback functions for allocation and deallocation of promoted buffers, as
/// well as to copy the data into and out of these buffers. /// well as to copy the data into and out of these buffers.
AllocBufferCallbackFn allocationFn; AllocBufferCallbackFn allocationFn;
DeallocBufferCallbackFn deallocationFn; DeallocBufferCallbackFn deallocationFn;
CopyCallbackFn copyInFn; CopyCallbackFn copyInFn;
CopyCallbackFn copyOutFn; CopyCallbackFn copyOutFn;
/// Alignment of promoted buffer. /// Alignment of promoted buffer.
Optional<unsigned> alignment; std::optional<unsigned> alignment;
}; };
} // namespace } // namespace
LinalgOpInstancePromotionOptions::LinalgOpInstancePromotionOptions( LinalgOpInstancePromotionOptions::LinalgOpInstancePromotionOptions(
LinalgOp linalgOp, const LinalgPromotionOptions &options) LinalgOp linalgOp, const LinalgPromotionOptions &options)
: subViews(), alignment(options.alignment) { : subViews(), alignment(options.alignment) {
assert(linalgOp.hasBufferSemantics() && "revisit usage of shaped operand"); assert(linalgOp.hasBufferSemantics() && "revisit usage of shaped operand");
auto vUseFullTileBuffers = auto vUseFullTileBuffers =
Show All 13 Lines for (OpOperand &opOperand : linalgOp->getOpOperands()) {
} }
} }
if (options.allocationFn) { if (options.allocationFn) {
allocationFn = *options.allocationFn; allocationFn = *options.allocationFn;
} else { } else {
allocationFn = [&](OpBuilder &b, memref::SubViewOp subViewOp, allocationFn = [&](OpBuilder &b, memref::SubViewOp subViewOp,
ArrayRef<Value> boundingSubViewSize, ArrayRef<Value> boundingSubViewSize,
DataLayout &layout) -> Optional<Value> { DataLayout &layout) -> std::optional<Value> {
return defaultAllocBufferCallBack(options, b, subViewOp, return defaultAllocBufferCallBack(options, b, subViewOp,
boundingSubViewSize, alignment, layout); boundingSubViewSize, alignment, layout);
}; };
} }
if (options.deallocationFn) { if (options.deallocationFn) {
deallocationFn = *options.deallocationFn; deallocationFn = *options.deallocationFn;
} else { } else {
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 Lines for (const auto &en : llvm::enumerate(subView.getOrCreateRanges(b, loc))) {
LLVM_DEBUG(llvm::dbgs() << "Extracted tightest: " << size << "\n"); LLVM_DEBUG(llvm::dbgs() << "Extracted tightest: " << size << "\n");
fullSizes.push_back(size); fullSizes.push_back(size);
partialSizes.push_back( partialSizes.push_back(
b.createOrFold<memref::DimOp>(loc, subView, resultDimIdx++)); b.createOrFold<memref::DimOp>(loc, subView, resultDimIdx++));
} }
SmallVector<int64_t, 4> dynSizes(fullSizes.size(), ShapedType::kDynamic); SmallVector<int64_t, 4> dynSizes(fullSizes.size(), ShapedType::kDynamic);
// If a callback is not specified, then use the default implementation for // If a callback is not specified, then use the default implementation for
// allocating the promoted buffer. // allocating the promoted buffer.
Optional<Value> fullLocalView = allocationFn(b, subView, fullSizes, layout); std::optional<Value> fullLocalView =
allocationFn(b, subView, fullSizes, layout);
if (!fullLocalView) if (!fullLocalView)
return failure(); return failure();
SmallVector<OpFoldResult, 4> zeros(fullSizes.size(), b.getIndexAttr(0)); SmallVector<OpFoldResult, 4> zeros(fullSizes.size(), b.getIndexAttr(0));
SmallVector<OpFoldResult, 4> ones(fullSizes.size(), b.getIndexAttr(1)); SmallVector<OpFoldResult, 4> ones(fullSizes.size(), b.getIndexAttr(1));
auto partialLocalView = b.createOrFold<memref::SubViewOp>( auto partialLocalView = b.createOrFold<memref::SubViewOp>(
loc, *fullLocalView, zeros, partialSizes, ones); loc, *fullLocalView, zeros, partialSizes, ones);
return PromotionInfo{*fullLocalView, partialLocalView}; return PromotionInfo{*fullLocalView, partialLocalView};
} }
▲ Show 20 Lines • Show All 142 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Transforms/Tiling.cpp

Show First 20 Lines • Show All 185 Lines • ▼ Show 20 Linesmlir::linalg::computeMultiTileSizes(OpBuilder &builder, LinalgOp op,
return spec; return spec;
} }
/// Returns true if the maximum tile offset `tileSize * numThreads-1` is less /// Returns true if the maximum tile offset `tileSize * numThreads-1` is less
/// than `iterationSize`. /// than `iterationSize`.
static bool canOmitTileOffsetInBoundsCheck(OpFoldResult tileSize, static bool canOmitTileOffsetInBoundsCheck(OpFoldResult tileSize,
OpFoldResult numThreads, OpFoldResult numThreads,
OpFoldResult iterationSize) { OpFoldResult iterationSize) {
Optional<int64_t> tileSizeConst = getConstantIntValue(tileSize); std::optional<int64_t> tileSizeConst = getConstantIntValue(tileSize);
Optional<int64_t> numThreadsConst = getConstantIntValue(numThreads); std::optional<int64_t> numThreadsConst = getConstantIntValue(numThreads);
Optional<int64_t> iterSizeConst = getConstantIntValue(iterationSize); std::optional<int64_t> iterSizeConst = getConstantIntValue(iterationSize);
if (!tileSizeConst || !numThreadsConst || !iterSizeConst) if (!tileSizeConst || !numThreadsConst || !iterSizeConst)
return false; return false;
return *tileSizeConst * (*numThreadsConst - 1) < *iterSizeConst; return *tileSizeConst * (*numThreadsConst - 1) < *iterSizeConst;
} }
/// Build an `affine_max` of all the `vals`. /// Build an `affine_max` of all the `vals`.
static OpFoldResult buildMax(OpBuilder &b, Location loc, static OpFoldResult buildMax(OpBuilder &b, Location loc,
ArrayRef<OpFoldResult> vals) { ArrayRef<OpFoldResult> vals) {
Show All 11 Lines
} }
/// Fill out the `tiledOffsets` and `tiledSizes` to be used to tile to a given /// Fill out the `tiledOffsets` and `tiledSizes` to be used to tile to a given
/// number of threads. /// number of threads.
static void calculateTileOffsetsAndSizes( static void calculateTileOffsetsAndSizes(
RewriterBase &b, Location loc, scf::ForeachThreadOp foreachThreadOp, RewriterBase &b, Location loc, scf::ForeachThreadOp foreachThreadOp,
ArrayRef<OpFoldResult> numThreads, SmallVector<Range> loopRanges, ArrayRef<OpFoldResult> numThreads, SmallVector<Range> loopRanges,
bool omitTileOffsetBoundsCheck, bool omitTileOffsetBoundsCheck,
Optional<ArrayRef<OpFoldResult>> nominalTileSizes, std::optional<ArrayRef<OpFoldResult>> nominalTileSizes,
SmallVector<OpFoldResult> &tiledOffsets, SmallVector<OpFoldResult> &tiledOffsets,
SmallVector<OpFoldResult> &tiledSizes) { SmallVector<OpFoldResult> &tiledSizes) {
OpBuilder::InsertionGuard g(b); OpBuilder::InsertionGuard g(b);
b.setInsertionPointToStart(foreachThreadOp.getBody(0)); b.setInsertionPointToStart(foreachThreadOp.getBody(0));
ValueRange threadIds = foreachThreadOp.getThreadIndices(); ValueRange threadIds = foreachThreadOp.getThreadIndices();
SmallVector<OpFoldResult> nonZeroNumThreads = SmallVector<OpFoldResult> nonZeroNumThreads =
llvm::to_vector(llvm::make_filter_range(numThreads, [](OpFoldResult ofr) { llvm::to_vector(llvm::make_filter_range(numThreads, [](OpFoldResult ofr) {
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Lines
/// that the dimension is not tiled, and can be thought of as tiling by the full /// that the dimension is not tiled, and can be thought of as tiling by the full
/// size of data. /// size of data.
/// It is the user's responsibility to ensure that `numThreads` is a valid /// It is the user's responsibility to ensure that `numThreads` is a valid
/// tiling specification (i.e. that only tiles parallel dimensions, e.g. in the /// tiling specification (i.e. that only tiles parallel dimensions, e.g. in the
/// Linalg case). If `omitTileOffsetBoundsCheck` is true, then the function will /// Linalg case). If `omitTileOffsetBoundsCheck` is true, then the function will
/// assume that `tileSize[i] * (numThread[i] -1) <= dimSize[i]` holds. /// assume that `tileSize[i] * (numThread[i] -1) <= dimSize[i]` holds.
static FailureOr<ForeachThreadTilingResult> tileToForeachThreadOpImpl( static FailureOr<ForeachThreadTilingResult> tileToForeachThreadOpImpl(
RewriterBase &b, TilingInterface op, ArrayRef<OpFoldResult> numThreads, RewriterBase &b, TilingInterface op, ArrayRef<OpFoldResult> numThreads,
Optional<ArrayRef<OpFoldResult>> nominalTileSizes, std::optional<ArrayRef<OpFoldResult>> nominalTileSizes,
Optional<ArrayAttr> mapping, bool omitTileOffsetBoundsCheck) { std::optional<ArrayAttr> mapping, bool omitTileOffsetBoundsCheck) {
Location loc = op->getLoc(); Location loc = op->getLoc();
OpBuilder::InsertionGuard g(b); OpBuilder::InsertionGuard g(b);
SmallVector<Range> loopRanges = op.getIterationDomain(b); SmallVector<Range> loopRanges = op.getIterationDomain(b);
if (loopRanges.empty()) if (loopRanges.empty())
return op->emitOpError("expected non-empty loop ranges"); return op->emitOpError("expected non-empty loop ranges");
auto hasStrideOne = [](Range r) { return !isConstantIntValue(r.stride, 1); }; auto hasStrideOne = [](Range r) { return !isConstantIntValue(r.stride, 1); };
if (llvm::any_of(loopRanges, hasStrideOne)) if (llvm::any_of(loopRanges, hasStrideOne))
▲ Show 20 Lines • Show All 79 Lines • ▼ Show 20 Lines b.create<tensor::ParallelInsertSliceOp>(loc, std::get<1>(it),
resultSizes, strides); resultSizes, strides);
} }
return ForeachThreadTilingResult{foreachThreadOp, tiledOp}; return ForeachThreadTilingResult{foreachThreadOp, tiledOp};
} }
FailureOr<ForeachThreadTilingResult> FailureOr<ForeachThreadTilingResult>
linalg::tileToForeachThreadOp(RewriterBase &b, TilingInterface op, linalg::tileToForeachThreadOp(RewriterBase &b, TilingInterface op,
ArrayRef<OpFoldResult> numThreads, ArrayRef<OpFoldResult> numThreads,
Optional<ArrayAttr> mapping) { std::optional<ArrayAttr> mapping) {
return tileToForeachThreadOpImpl(b, op, numThreads, return tileToForeachThreadOpImpl(b, op, numThreads,
/*nominalTileSizes=*/std::nullopt, mapping, /*nominalTileSizes=*/std::nullopt, mapping,
/*omitTileOffsetBoundsCheck=*/false); /*omitTileOffsetBoundsCheck=*/false);
} }
FailureOr<ForeachThreadTilingResult> FailureOr<ForeachThreadTilingResult>
linalg::tileToForeachThreadOpUsingTileSizes(RewriterBase &b, TilingInterface op, linalg::tileToForeachThreadOpUsingTileSizes(RewriterBase &b, TilingInterface op,
ArrayRef<OpFoldResult> tileSizes, ArrayRef<OpFoldResult> tileSizes,
Optional<ArrayAttr> mapping) { std::optional<ArrayAttr> mapping) {
SmallVector<Range> loopRanges = op.getIterationDomain(b); SmallVector<Range> loopRanges = op.getIterationDomain(b);
unsigned nLoops = loopRanges.size(); unsigned nLoops = loopRanges.size();
SmallVector<OpFoldResult> numThreads; SmallVector<OpFoldResult> numThreads;
numThreads.reserve(nLoops); numThreads.reserve(nLoops);
AffineExpr s0, s1; AffineExpr s0, s1;
bindSymbols(b.getContext(), s0, s1); bindSymbols(b.getContext(), s0, s1);
AffineExpr divExpr = s0.ceilDiv(s1); AffineExpr divExpr = s0.ceilDiv(s1);
for (const auto &it : llvm::zip(tileSizes, loopRanges)) { for (const auto &it : llvm::zip(tileSizes, loopRanges)) {
▲ Show 20 Lines • Show All 161 Lines • ▼ Show 20 Lines return TiledLinalgOp{
res, loops, outermostLoop ? outermostLoop->getResults() : tensorResults}; res, loops, outermostLoop ? outermostLoop->getResults() : tensorResults};
} }
FailureOr<linalg::ForeachThreadReductionTilingResult> FailureOr<linalg::ForeachThreadReductionTilingResult>
linalg::tileReductionUsingForeachThread(RewriterBase &b, linalg::tileReductionUsingForeachThread(RewriterBase &b,
PartialReductionOpInterface op, PartialReductionOpInterface op,
ArrayRef<OpFoldResult> numThreads, ArrayRef<OpFoldResult> numThreads,
ArrayRef<OpFoldResult> tileSizes, ArrayRef<OpFoldResult> tileSizes,
Optional<ArrayAttr> mapping) { std::optional<ArrayAttr> mapping) {
Location loc = op.getLoc(); Location loc = op.getLoc();
OpBuilder::InsertionGuard g(b); OpBuilder::InsertionGuard g(b);
// Ops implementing PartialReductionOpInterface are expected to implement // Ops implementing PartialReductionOpInterface are expected to implement
// TilingInterface. // TilingInterface.
// TODO: proper core mechanism to tie interfaces together. // TODO: proper core mechanism to tie interfaces together.
auto tilingInterfaceOp = cast<TilingInterface>(op.getOperation()); auto tilingInterfaceOp = cast<TilingInterface>(op.getOperation());
▲ Show 20 Lines • Show All 375 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp

Show First 20 Lines • Show All 454 Lines • ▼ Show 20 Lines if (!sliceOp.hasUnitStride())
return failure(); return failure();
auto padOp = sliceOp.getSource().getDefiningOp<tensor::PadOp>(); auto padOp = sliceOp.getSource().getDefiningOp<tensor::PadOp>();
if (!padOp) if (!padOp)
return failure(); return failure();
bool zeroSliceGuard = true; bool zeroSliceGuard = true;
if (controlFn) { if (controlFn) {
if (Optional<bool> control = controlFn(sliceOp)) if (std::optional<bool> control = controlFn(sliceOp))
zeroSliceGuard = *control; zeroSliceGuard = *control;
else else
return failure(); return failure();
} }
Operation *tiledPadOp = Operation *tiledPadOp =
tensor::bubbleUpPadSlice(rewriter, padOp, sliceOp.getMixedOffsets(), tensor::bubbleUpPadSlice(rewriter, padOp, sliceOp.getMixedOffsets(),
sliceOp.getMixedSizes(), zeroSliceGuard); sliceOp.getMixedSizes(), zeroSliceGuard);
Show All 24 Linesstatic Value getPackOpSourceOrPaddedSource(OpBuilder &builder,
for (int64_t dim = 0; dim < inputRank; ++dim) { for (int64_t dim = 0; dim < inputRank; ++dim) {
int64_t size = inputType.getDimSize(dim); int64_t size = inputType.getDimSize(dim);
if (!tileAndPosMapping.count(dim)) { if (!tileAndPosMapping.count(dim)) {
paddedShape.push_back(size); paddedShape.push_back(size);
continue; continue;
} }
// The size is less than or equal to tileSize because outer dims are all 1s. // The size is less than or equal to tileSize because outer dims are all 1s.
Optional<int64_t> tileSize = std::optional<int64_t> tileSize =
getConstantIntValue(tileAndPosMapping.lookup(dim)); getConstantIntValue(tileAndPosMapping.lookup(dim));
assert(tileSize.has_value() && "dynamic inner tile size is not supported"); assert(tileSize.has_value() && "dynamic inner tile size is not supported");
paddedShape.push_back(tileSize.value()); paddedShape.push_back(tileSize.value());
} }
auto resultType = auto resultType =
RankedTensorType::get(paddedShape, inputType.getElementType()); RankedTensorType::get(paddedShape, inputType.getElementType());
return tensor::createPadHighOp(resultType, input, packOp.getPaddingValue(), return tensor::createPadHighOp(resultType, input, packOp.getPaddingValue(),
/*nofold=*/false, loc, builder); /*nofold=*/false, loc, builder);
▲ Show 20 Lines • Show All 256 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp

Show First 20 Lines • Show All 367 Lines • ▼ Show 20 Lines
struct VectorizationResult { struct VectorizationResult {
/// Return status from vectorizing the current op. /// Return status from vectorizing the current op.
enum VectorizationStatus status = VectorizationStatus::Failure; enum VectorizationStatus status = VectorizationStatus::Failure;
/// New vectorized operation to replace the current op. /// New vectorized operation to replace the current op.
/// Replacement behavior is specified by `status`. /// Replacement behavior is specified by `status`.
Operation *newOp; Operation *newOp;
}; };
llvm::Optional<vector::CombiningKind> std::optional<vector::CombiningKind>
mlir::linalg::getCombinerOpKind(Operation *combinerOp) { mlir::linalg::getCombinerOpKind(Operation *combinerOp) {
using ::mlir::vector::CombiningKind; using ::mlir::vector::CombiningKind;
if (!combinerOp) if (!combinerOp)
return std::nullopt; return std::nullopt;
return llvm::TypeSwitch<Operation *, llvm::Optional<CombiningKind>>( return llvm::TypeSwitch<Operation *, std::optional<CombiningKind>>(combinerOp)
combinerOp)
.Case<arith::AddIOp, arith::AddFOp>( .Case<arith::AddIOp, arith::AddFOp>(
[&](auto op) { return CombiningKind::ADD; }) [&](auto op) { return CombiningKind::ADD; })
.Case<arith::AndIOp>([&](auto op) { return CombiningKind::AND; }) .Case<arith::AndIOp>([&](auto op) { return CombiningKind::AND; })
.Case<arith::MaxSIOp>([&](auto op) { return CombiningKind::MAXSI; }) .Case<arith::MaxSIOp>([&](auto op) { return CombiningKind::MAXSI; })
.Case<arith::MaxFOp>([&](auto op) { return CombiningKind::MAXF; }) .Case<arith::MaxFOp>([&](auto op) { return CombiningKind::MAXF; })
.Case<arith::MinSIOp>([&](auto op) { return CombiningKind::MINSI; }) .Case<arith::MinSIOp>([&](auto op) { return CombiningKind::MINSI; })
.Case<arith::MinFOp>([&](auto op) { return CombiningKind::MINF; }) .Case<arith::MinFOp>([&](auto op) { return CombiningKind::MINF; })
.Case<arith::MulIOp, arith::MulFOp>( .Case<arith::MulIOp, arith::MulFOp>(
▲ Show 20 Lines • Show All 1,450 Lines • ▼ Show 20 Lines if (lhsShapedType.getRank() != 3 ||
(rhsShapedType.getRank() != 2 && rhsShapedType.getRank() != 3) || (rhsShapedType.getRank() != 2 && rhsShapedType.getRank() != 3) ||
resShapedType.getRank() != 3) resShapedType.getRank() != 3)
return; return;
// Check for reduction `add` preceded by `mul`. // Check for reduction `add` preceded by `mul`.
Operation *reduceOp = matchLinalgReduction(linalgOp.getDpsInitOperand(0)); Operation *reduceOp = matchLinalgReduction(linalgOp.getDpsInitOperand(0));
if (!reduceOp) if (!reduceOp)
return; return;
llvm::Optional<vector::CombiningKind> maybeKind; std::optional<vector::CombiningKind> maybeKind;
maybeKind = getCombinerOpKind(reduceOp); maybeKind = getCombinerOpKind(reduceOp);
if (!maybeKind || *maybeKind != vector::CombiningKind::ADD) if (!maybeKind || *maybeKind != vector::CombiningKind::ADD)
return; return;
// Check for single `mul` predecessor. The `mul` operands must be block // Check for single `mul` predecessor. The `mul` operands must be block
// arguments or extension of block arguments. // arguments or extension of block arguments.
Operation *mulOp = nullptr; Operation *mulOp = nullptr;
for (Value operand : reduceOp->getOperands()) { for (Value operand : reduceOp->getOperands()) {
if (operand.isa<BlockArgument>()) if (operand.isa<BlockArgument>())
▲ Show 20 Lines • Show All 471 Lines • Show Last 20 Lines

mlir/lib/Dialect/Linalg/Utils/Utils.cpp

Show First 20 Lines • Show All 809 Lines • ▼ Show 20 Lines for (unsigned r = 0; r < rank; ++r) {
} }
// The size of the subview / extract_slice should be trimmed to avoid // The size of the subview / extract_slice should be trimmed to avoid
// out-of-bounds accesses, unless: // out-of-bounds accesses, unless:
// a. We statically know the subshape size divides the shape size evenly. // a. We statically know the subshape size divides the shape size evenly.
// b. The subshape size is 1. According to the way the loops are set up, // b. The subshape size is 1. According to the way the loops are set up,
// tensors with "0" dimensions would never be constructed. // tensors with "0" dimensions would never be constructed.
int64_t shapeSize = shape[r]; int64_t shapeSize = shape[r];
Optional<int64_t> sizeCst = getConstantIntValue(size); std::optional<int64_t> sizeCst = getConstantIntValue(size);
auto hasTileSizeOne = sizeCst && *sizeCst == 1; auto hasTileSizeOne = sizeCst && *sizeCst == 1;
auto dividesEvenly = sizeCst && !ShapedType::isDynamic(shapeSize) && auto dividesEvenly = sizeCst && !ShapedType::isDynamic(shapeSize) &&
((shapeSize % *sizeCst) == 0); ((shapeSize % *sizeCst) == 0);
if (!hasTileSizeOne && !dividesEvenly) { if (!hasTileSizeOne && !dividesEvenly) {
LLVM_DEBUG(llvm::dbgs() << "makeTiledShape: shapeSize=" << shapeSize LLVM_DEBUG(llvm::dbgs() << "makeTiledShape: shapeSize=" << shapeSize
<< ", size: " << size << ", size: " << size
<< ": make sure in bound with affine.min\n"); << ": make sure in bound with affine.min\n");
▲ Show 20 Lines • Show All 269 Lines • Show Last 20 Lines

mlir/lib/Dialect/MemRef/IR/MemRefOps.cpp

Show First 20 Lines • Show All 178 Lines • ▼ Show 20 Lines if (ofr.is<Attribute>()) {
// others use getIndexAttr). // others use getIndexAttr).
// The workaround here is to stick to the IndexAttr type for all the // The workaround here is to stick to the IndexAttr type for all the
// values, hence we recreate the attribute even when it is already static // values, hence we recreate the attribute even when it is already static
// to make sure the type is consistent. // to make sure the type is consistent.
ofr = builder.getIndexAttr( ofr = builder.getIndexAttr(
ofr.get<Attribute>().cast<IntegerAttr>().getInt()); ofr.get<Attribute>().cast<IntegerAttr>().getInt());
continue; continue;
} }
Optional<int64_t> maybeConstant = getConstantIntValue(ofr.get<Value>()); std::optional<int64_t> maybeConstant =
getConstantIntValue(ofr.get<Value>());
if (maybeConstant) if (maybeConstant)
ofr = builder.getIndexAttr(*maybeConstant); ofr = builder.getIndexAttr(*maybeConstant);
} }
} }
/// Wrapper around `getShape` that conforms to the function signature /// Wrapper around `getShape` that conforms to the function signature
/// expected for `getAttributes` in `constifyIndexValues`. /// expected for `getAttributes` in `constifyIndexValues`.
static SmallVector<int64_t> getConstantSizes(MemRefType memRefTy) { static SmallVector<int64_t> getConstantSizes(MemRefType memRefTy) {
▲ Show 20 Lines • Show All 257 Lines • ▼ Show 20 LinesParseResult AllocaScopeOp::parse(OpAsmParser &parser, OperationState &result) {
// Parse the optional attribute list. // Parse the optional attribute list.
if (parser.parseOptionalAttrDict(result.attributes)) if (parser.parseOptionalAttrDict(result.attributes))
return failure(); return failure();
return success(); return success();
} }
void AllocaScopeOp::getSuccessorRegions( void AllocaScopeOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
if (index) { if (index) {
regions.push_back(RegionSuccessor(getResults())); regions.push_back(RegionSuccessor(getResults()));
return; return;
} }
regions.push_back(RegionSuccessor(&getBodyRegion())); regions.push_back(RegionSuccessor(&getBodyRegion()));
} }
▲ Show 20 Lines • Show All 447 Lines • ▼ Show 20 Lines
void DimOp::build(OpBuilder &builder, OperationState &result, Value source, void DimOp::build(OpBuilder &builder, OperationState &result, Value source,
int64_t index) { int64_t index) {
auto loc = result.location; auto loc = result.location;
Value indexValue = builder.create<arith::ConstantIndexOp>(loc, index); Value indexValue = builder.create<arith::ConstantIndexOp>(loc, index);
build(builder, result, source, indexValue); build(builder, result, source, indexValue);
} }
Optional<int64_t> DimOp::getConstantIndex() { std::optional<int64_t> DimOp::getConstantIndex() {
return getConstantIntValue(getIndex()); return getConstantIntValue(getIndex());
} }
Speculation::Speculatability DimOp::getSpeculatability() { Speculation::Speculatability DimOp::getSpeculatability() {
auto constantIndex = getConstantIndex(); auto constantIndex = getConstantIndex();
if (!constantIndex) if (!constantIndex)
return Speculation::NotSpeculatable; return Speculation::NotSpeculatable;
auto rankedSourceType = dyn_cast<MemRefType>(getSource().getType()); auto rankedSourceType = dyn_cast<MemRefType>(getSource().getType());
if (!rankedSourceType) if (!rankedSourceType)
return Speculation::NotSpeculatable; return Speculation::NotSpeculatable;
// The verifier rejects operations that violate this assertion. // The verifier rejects operations that violate this assertion.
assert(constantIndex < rankedSourceType.getRank()); assert(constantIndex < rankedSourceType.getRank());
return Speculation::Speculatable; return Speculation::Speculatable;
} }
LogicalResult DimOp::verify() { LogicalResult DimOp::verify() {
// Assume unknown index to be in range. // Assume unknown index to be in range.
Optional<int64_t> index = getConstantIndex(); std::optional<int64_t> index = getConstantIndex();
if (!index) if (!index)
return success(); return success();
// Check that constant index is not knowingly out of range. // Check that constant index is not knowingly out of range.
auto type = getSource().getType(); auto type = getSource().getType();
if (auto memrefType = type.dyn_cast<MemRefType>()) { if (auto memrefType = type.dyn_cast<MemRefType>()) {
if (*index >= memrefType.getRank()) if (*index >= memrefType.getRank())
return emitOpError("index is out of range"); return emitOpError("index is out of range");
Show All 18 Lines
/// Given the `originalType` and a `candidateReducedType` whose shape is assumed /// Given the `originalType` and a `candidateReducedType` whose shape is assumed
/// to be a subset of `originalType` with some `1` entries erased, return the /// to be a subset of `originalType` with some `1` entries erased, return the
/// set of indices that specifies which of the entries of `originalShape` are /// set of indices that specifies which of the entries of `originalShape` are
/// dropped to obtain `reducedShape`. /// dropped to obtain `reducedShape`.
/// This accounts for cases where there are multiple unit-dims, but only a /// This accounts for cases where there are multiple unit-dims, but only a
/// subset of those are dropped. For MemRefTypes these can be disambiguated /// subset of those are dropped. For MemRefTypes these can be disambiguated
/// using the strides. If a dimension is dropped the stride must be dropped too. /// using the strides. If a dimension is dropped the stride must be dropped too.
static llvm::Optional<llvm::SmallBitVector> static std::optional<llvm::SmallBitVector>
computeMemRefRankReductionMask(MemRefType originalType, MemRefType reducedType, computeMemRefRankReductionMask(MemRefType originalType, MemRefType reducedType,
ArrayRef<OpFoldResult> sizes) { ArrayRef<OpFoldResult> sizes) {
llvm::SmallBitVector unusedDims(originalType.getRank()); llvm::SmallBitVector unusedDims(originalType.getRank());
if (originalType.getRank() == reducedType.getRank()) if (originalType.getRank() == reducedType.getRank())
return unusedDims; return unusedDims;
for (const auto &dim : llvm::enumerate(sizes)) for (const auto &dim : llvm::enumerate(sizes))
if (auto attr = dim.value().dyn_cast<Attribute>()) if (auto attr = dim.value().dyn_cast<Attribute>())
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Lines if ((int64_t)unusedDims.count() + reducedType.getRank() !=
originalType.getRank()) originalType.getRank())
return std::nullopt; return std::nullopt;
return unusedDims; return unusedDims;
} }
llvm::SmallBitVector SubViewOp::getDroppedDims() { llvm::SmallBitVector SubViewOp::getDroppedDims() {
MemRefType sourceType = getSourceType(); MemRefType sourceType = getSourceType();
MemRefType resultType = getType(); MemRefType resultType = getType();
llvm::Optional<llvm::SmallBitVector> unusedDims = std::optional<llvm::SmallBitVector> unusedDims =
computeMemRefRankReductionMask(sourceType, resultType, getMixedSizes()); computeMemRefRankReductionMask(sourceType, resultType, getMixedSizes());
assert(unusedDims && "unable to find unused dims of subview"); assert(unusedDims && "unable to find unused dims of subview");
return *unusedDims; return *unusedDims;
} }
OpFoldResult DimOp::fold(ArrayRef<Attribute> operands) { OpFoldResult DimOp::fold(ArrayRef<Attribute> operands) {
// All forms of folding require a known index. // All forms of folding require a known index.
auto index = operands[1].dyn_cast_or_null<IntegerAttr>(); auto index = operands[1].dyn_cast_or_null<IntegerAttr>();
▲ Show 20 Lines • Show All 298 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ExtractStridedMetadataOp // ExtractStridedMetadataOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// The number and type of the results are inferred from the /// The number and type of the results are inferred from the
/// shape of the source. /// shape of the source.
LogicalResult ExtractStridedMetadataOp::inferReturnTypes( LogicalResult ExtractStridedMetadataOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
ExtractStridedMetadataOpAdaptor extractAdaptor(operands, attributes, regions); ExtractStridedMetadataOpAdaptor extractAdaptor(operands, attributes, regions);
auto sourceType = extractAdaptor.getSource().getType().dyn_cast<MemRefType>(); auto sourceType = extractAdaptor.getSource().getType().dyn_cast<MemRefType>();
if (!sourceType) if (!sourceType)
return failure(); return failure();
unsigned sourceRank = sourceType.getRank(); unsigned sourceRank = sourceType.getRank();
▲ Show 20 Lines • Show All 244 Lines • ▼ Show 20 Lines if (auto elementsAttr = initValue.dyn_cast<ElementsAttr>()) {
Type initType = elementsAttr.getType(); Type initType = elementsAttr.getType();
Type tensorType = getTensorTypeFromMemRefType(memrefType); Type tensorType = getTensorTypeFromMemRefType(memrefType);
if (initType != tensorType) if (initType != tensorType)
return emitOpError("initial value expected to be of type ") return emitOpError("initial value expected to be of type ")
<< tensorType << ", but was of type " << initType; << tensorType << ", but was of type " << initType;
} }
} }
if (Optional<uint64_t> alignAttr = getAlignment()) { if (std::optional<uint64_t> alignAttr = getAlignment()) {
uint64_t alignment = *alignAttr; uint64_t alignment = *alignAttr;
if (!llvm::isPowerOf2_64(alignment)) if (!llvm::isPowerOf2_64(alignment))
return emitError() << "alignment attribute value " << alignment return emitError() << "alignment attribute value " << alignment
<< " is not a power of 2"; << " is not a power of 2";
} }
// TODO: verify visibility for declarations. // TODO: verify visibility for declarations.
▲ Show 20 Lines • Show All 968 Lines • ▼ Show 20 Lines auto inferredType =
inferResultType(sourceRankedTensorType, offsets, sizes, strides) inferResultType(sourceRankedTensorType, offsets, sizes, strides)
.cast<MemRefType>(); .cast<MemRefType>();
assert(inferredType.getRank() >= static_cast<int64_t>(resultShape.size()) && assert(inferredType.getRank() >= static_cast<int64_t>(resultShape.size()) &&
"expected "); "expected ");
if (inferredType.getRank() == static_cast<int64_t>(resultShape.size())) if (inferredType.getRank() == static_cast<int64_t>(resultShape.size()))
return inferredType; return inferredType;
// Compute which dimensions are dropped. // Compute which dimensions are dropped.
Optional<llvm::SmallDenseSet<unsigned>> dimsToProject = std::optional<llvm::SmallDenseSet<unsigned>> dimsToProject =
computeRankReductionMask(inferredType.getShape(), resultShape); computeRankReductionMask(inferredType.getShape(), resultShape);
assert(dimsToProject.has_value() && "invalid rank reduction"); assert(dimsToProject.has_value() && "invalid rank reduction");
// Compute the layout and result type. // Compute the layout and result type.
auto inferredLayout = inferredType.getLayout().cast<StridedLayoutAttr>(); auto inferredLayout = inferredType.getLayout().cast<StridedLayoutAttr>();
SmallVector<int64_t> rankReducedStrides; SmallVector<int64_t> rankReducedStrides;
rankReducedStrides.reserve(resultShape.size()); rankReducedStrides.reserve(resultShape.size());
for (auto [idx, value] : llvm::enumerate(inferredLayout.getStrides())) { for (auto [idx, value] : llvm::enumerate(inferredLayout.getStrides())) {
▲ Show 20 Lines • Show All 260 Lines • ▼ Show 20 Lines
/// `currentResultType`. /// `currentResultType`.
static MemRefType getCanonicalSubViewResultType( static MemRefType getCanonicalSubViewResultType(
MemRefType currentResultType, MemRefType currentSourceType, MemRefType currentResultType, MemRefType currentSourceType,
MemRefType sourceType, ArrayRef<OpFoldResult> mixedOffsets, MemRefType sourceType, ArrayRef<OpFoldResult> mixedOffsets,
ArrayRef<OpFoldResult> mixedSizes, ArrayRef<OpFoldResult> mixedStrides) { ArrayRef<OpFoldResult> mixedSizes, ArrayRef<OpFoldResult> mixedStrides) {
auto nonRankReducedType = SubViewOp::inferResultType(sourceType, mixedOffsets, auto nonRankReducedType = SubViewOp::inferResultType(sourceType, mixedOffsets,
mixedSizes, mixedStrides) mixedSizes, mixedStrides)
.cast<MemRefType>(); .cast<MemRefType>();
llvm::Optional<llvm::SmallBitVector> unusedDims = std::optional<llvm::SmallBitVector> unusedDims =
computeMemRefRankReductionMask(currentSourceType, currentResultType, computeMemRefRankReductionMask(currentSourceType, currentResultType,
mixedSizes); mixedSizes);
// Return nullptr as failure mode. // Return nullptr as failure mode.
if (!unusedDims) if (!unusedDims)
return nullptr; return nullptr;
auto layout = nonRankReducedType.getLayout().cast<StridedLayoutAttr>(); auto layout = nonRankReducedType.getLayout().cast<StridedLayoutAttr>();
SmallVector<int64_t> shape, strides; SmallVector<int64_t> shape, strides;
▲ Show 20 Lines • Show All 66 Lines • ▼ Show 20 Lines if (subViewOp.getSourceType().getRank() != subViewOp.getType().getRank())
return false; return false;
auto mixedOffsets = subViewOp.getMixedOffsets(); auto mixedOffsets = subViewOp.getMixedOffsets();
auto mixedSizes = subViewOp.getMixedSizes(); auto mixedSizes = subViewOp.getMixedSizes();
auto mixedStrides = subViewOp.getMixedStrides(); auto mixedStrides = subViewOp.getMixedStrides();
// Check offsets are zero. // Check offsets are zero.
if (llvm::any_of(mixedOffsets, [](OpFoldResult ofr) { if (llvm::any_of(mixedOffsets, [](OpFoldResult ofr) {
Optional<int64_t> intValue = getConstantIntValue(ofr); std::optional<int64_t> intValue = getConstantIntValue(ofr);
return !intValue || intValue.value() != 0; return !intValue || intValue.value() != 0;
})) }))
return false; return false;
// Check strides are one. // Check strides are one.
if (llvm::any_of(mixedStrides, [](OpFoldResult ofr) { if (llvm::any_of(mixedStrides, [](OpFoldResult ofr) {
Optional<int64_t> intValue = getConstantIntValue(ofr); std::optional<int64_t> intValue = getConstantIntValue(ofr);
return !intValue || intValue.value() != 1; return !intValue || intValue.value() != 1;
})) }))
return false; return false;
// Check all size values are static and matches the (static) source shape. // Check all size values are static and matches the (static) source shape.
ArrayRef<int64_t> sourceShape = subViewOp.getSourceType().getShape(); ArrayRef<int64_t> sourceShape = subViewOp.getSourceType().getShape();
for (const auto &size : llvm::enumerate(mixedSizes)) { for (const auto &size : llvm::enumerate(mixedSizes)) {
Optional<int64_t> intValue = getConstantIntValue(size.value()); std::optional<int64_t> intValue = getConstantIntValue(size.value());
if (!intValue || *intValue != sourceShape[size.index()]) if (!intValue || *intValue != sourceShape[size.index()])
return false; return false;
} }
// All conditions met. The `SubViewOp` is foldable as a no-op. // All conditions met. The `SubViewOp` is foldable as a no-op.
return true; return true;
} }
namespace { namespace {
▲ Show 20 Lines • Show All 411 Lines • Show Last 20 Lines

mlir/lib/Dialect/MemRef/Transforms/MultiBuffer.cpp

Show First 20 Lines • Show All 92 Lines • ▼ Show 20 Lines if (llvm::any_of(allocOp->getUsers(), [&](Operation *otherUser) {
return !dom.dominates(user, otherUser); return !dom.dominates(user, otherUser);
})) }))
continue; continue;
candidateLoop = parentLoop; candidateLoop = parentLoop;
break; break;
} }
if (!candidateLoop) if (!candidateLoop)
return failure(); return failure();
llvm::Optional<Value> inductionVar = candidateLoop.getSingleInductionVar(); std::optional<Value> inductionVar = candidateLoop.getSingleInductionVar();
llvm::Optional<OpFoldResult> lowerBound = candidateLoop.getSingleLowerBound(); std::optional<OpFoldResult> lowerBound = candidateLoop.getSingleLowerBound();
llvm::Optional<OpFoldResult> singleStep = candidateLoop.getSingleStep(); std::optional<OpFoldResult> singleStep = candidateLoop.getSingleStep();
if (!inductionVar || !lowerBound || !singleStep) if (!inductionVar || !lowerBound || !singleStep)
return failure(); return failure();
if (!dom.dominates(allocOp.getOperation(), candidateLoop)) if (!dom.dominates(allocOp.getOperation(), candidateLoop))
return failure(); return failure();
OpBuilder builder(candidateLoop); OpBuilder builder(candidateLoop);
SmallVector<int64_t, 4> newShape(1, multiplier); SmallVector<int64_t, 4> newShape(1, multiplier);
ArrayRef<int64_t> oldShape = allocOp.getType().getShape(); ArrayRef<int64_t> oldShape = allocOp.getType().getShape();
newShape.append(oldShape.begin(), oldShape.end()); newShape.append(oldShape.begin(), oldShape.end());
auto newMemref = MemRefType::get(newShape, allocOp.getType().getElementType(), auto newMemref = MemRefType::get(newShape, allocOp.getType().getElementType(),
MemRefLayoutAttrInterface(), MemRefLayoutAttrInterface(),
allocOp.getType().getMemorySpace()); allocOp.getType().getMemorySpace());
builder.setInsertionPoint(allocOp); builder.setInsertionPoint(allocOp);
Location loc = allocOp->getLoc(); Location loc = allocOp->getLoc();
auto newAlloc = builder.create<memref::AllocOp>(loc, newMemref, ValueRange{}, auto newAlloc = builder.create<memref::AllocOp>(loc, newMemref, ValueRange{},
allocOp->getAttrs()); allocOp->getAttrs());
builder.setInsertionPoint(&candidateLoop.getLoopBody().front(), builder.setInsertionPoint(&candidateLoop.getLoopBody().front(),
candidateLoop.getLoopBody().front().begin()); candidateLoop.getLoopBody().front().begin());
SmallVector<Value> operands = {*inductionVar}; SmallVector<Value> operands = {*inductionVar};
AffineExpr induc = getAffineDimExpr(0, allocOp.getContext()); AffineExpr induc = getAffineDimExpr(0, allocOp.getContext());
unsigned dimCount = 1; unsigned dimCount = 1;
auto getAffineExpr = [&](OpFoldResult e) -> AffineExpr { auto getAffineExpr = [&](OpFoldResult e) -> AffineExpr {
if (Optional<int64_t> constValue = getConstantIntValue(e)) { if (std::optional<int64_t> constValue = getConstantIntValue(e)) {
return getAffineConstantExpr(*constValue, allocOp.getContext()); return getAffineConstantExpr(*constValue, allocOp.getContext());
} }
auto value = getOrCreateValue(e, builder, candidateLoop->getLoc()); auto value = getOrCreateValue(e, builder, candidateLoop->getLoc());
operands.push_back(value); operands.push_back(value);
return getAffineDimExpr(dimCount++, allocOp.getContext()); return getAffineDimExpr(dimCount++, allocOp.getContext());
}; };
auto init = getAffineExpr(*lowerBound); auto init = getAffineExpr(*lowerBound);
auto step = getAffineExpr(*singleStep); auto step = getAffineExpr(*singleStep);
AffineExpr expr = ((induc - init).floorDiv(step)) % multiplier; AffineExpr expr = ((induc - init).floorDiv(step)) % multiplier;
auto map = AffineMap::get(dimCount, 0, expr); auto map = AffineMap::get(dimCount, 0, expr);
Value bufferIndex = builder.create<AffineApplyOp>(loc, map, operands); Value bufferIndex = builder.create<AffineApplyOp>(loc, map, operands);
SmallVector<OpFoldResult> offsets, sizes, strides; SmallVector<OpFoldResult> offsets, sizes, strides;
Show All 16 Lines

mlir/lib/Dialect/MemRef/Transforms/ResolveShapedTypeResultDims.cpp

Show All 40 Lines LogicalResult matchAndRewrite(OpTy dimOp,
OpResult dimValue = dimOp.getSource().template dyn_cast<OpResult>(); OpResult dimValue = dimOp.getSource().template dyn_cast<OpResult>();
if (!dimValue) if (!dimValue)
return failure(); return failure();
auto shapedTypeOp = auto shapedTypeOp =
dyn_cast<InferShapedTypeOpInterface>(dimValue.getOwner()); dyn_cast<InferShapedTypeOpInterface>(dimValue.getOwner());
if (!shapedTypeOp) if (!shapedTypeOp)
return failure(); return failure();
Optional<int64_t> dimIndex = dimOp.getConstantIndex(); std::optional<int64_t> dimIndex = dimOp.getConstantIndex();
if (!dimIndex) if (!dimIndex)
return failure(); return failure();
SmallVector<Value> reifiedResultShapes; SmallVector<Value> reifiedResultShapes;
if (failed(shapedTypeOp.reifyReturnTypeShapes( if (failed(shapedTypeOp.reifyReturnTypeShapes(
rewriter, shapedTypeOp->getOperands(), reifiedResultShapes))) rewriter, shapedTypeOp->getOperands(), reifiedResultShapes)))
return failure(); return failure();
Show All 25 Lines LogicalResult matchAndRewrite(OpTy dimOp,
OpResult dimValue = dimOp.getSource().template dyn_cast<OpResult>(); OpResult dimValue = dimOp.getSource().template dyn_cast<OpResult>();
if (!dimValue) if (!dimValue)
return failure(); return failure();
auto rankedShapeTypeOp = auto rankedShapeTypeOp =
dyn_cast<ReifyRankedShapedTypeOpInterface>(dimValue.getOwner()); dyn_cast<ReifyRankedShapedTypeOpInterface>(dimValue.getOwner());
if (!rankedShapeTypeOp) if (!rankedShapeTypeOp)
return failure(); return failure();
Optional<int64_t> dimIndex = dimOp.getConstantIndex(); std::optional<int64_t> dimIndex = dimOp.getConstantIndex();
if (!dimIndex) if (!dimIndex)
return failure(); return failure();
SmallVector<SmallVector<Value>> reifiedResultShapes; SmallVector<SmallVector<Value>> reifiedResultShapes;
if (failed( if (failed(
rankedShapeTypeOp.reifyResultShapes(rewriter, reifiedResultShapes))) rankedShapeTypeOp.reifyResultShapes(rewriter, reifiedResultShapes)))
return failure(); return failure();
▲ Show 20 Lines • Show All 73 Lines • Show Last 20 Lines

mlir/lib/Dialect/NVGPU/Transforms/OptimizeSharedMemory.cpp

Show First 20 Lines • Show All 143 Lines • ▼ Show 20 Lines
static LogicalResult static LogicalResult
getShmReadAndWriteOps(Operation *parentOp, Value shmMemRef, getShmReadAndWriteOps(Operation *parentOp, Value shmMemRef,
SmallVector<Operation *, 16> &readOps, SmallVector<Operation *, 16> &readOps,
SmallVector<Operation *, 16> &writeOps) { SmallVector<Operation *, 16> &writeOps) {
parentOp->walk([&](Operation *op) { parentOp->walk([&](Operation *op) {
MemoryEffectOpInterface iface = dyn_cast<MemoryEffectOpInterface>(op); MemoryEffectOpInterface iface = dyn_cast<MemoryEffectOpInterface>(op);
if (!iface) if (!iface)
return; return;
Optional<MemoryEffects::EffectInstance> effect = std::optional<MemoryEffects::EffectInstance> effect =
iface.getEffectOnValue<MemoryEffects::Read>(shmMemRef); iface.getEffectOnValue<MemoryEffects::Read>(shmMemRef);
if (effect) { if (effect) {
readOps.push_back(op); readOps.push_back(op);
return; return;
} }
effect = iface.getEffectOnValue<MemoryEffects::Write>(shmMemRef); effect = iface.getEffectOnValue<MemoryEffects::Write>(shmMemRef);
if (effect) if (effect)
writeOps.push_back(op); writeOps.push_back(op);
▲ Show 20 Lines • Show All 120 Lines • Show Last 20 Lines

mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp

Show First 20 Lines • Show All 111 Lines • ▼ Show 20 Lines
template <typename ClauseAttr> template <typename ClauseAttr>
static ParseResult parseClauseAttr(AsmParser &parser, ClauseAttr &attr) { static ParseResult parseClauseAttr(AsmParser &parser, ClauseAttr &attr) {
using ClauseT = decltype(std::declval<ClauseAttr>().getValue()); using ClauseT = decltype(std::declval<ClauseAttr>().getValue());
StringRef enumStr; StringRef enumStr;
SMLoc loc = parser.getCurrentLocation(); SMLoc loc = parser.getCurrentLocation();
if (parser.parseKeyword(&enumStr)) if (parser.parseKeyword(&enumStr))
return failure(); return failure();
if (Optional<ClauseT> enumValue = symbolizeEnum<ClauseT>(enumStr)) { if (std::optional<ClauseT> enumValue = symbolizeEnum<ClauseT>(enumStr)) {
attr = ClauseAttr::get(parser.getContext(), *enumValue); attr = ClauseAttr::get(parser.getContext(), *enumValue);
return success(); return success();
} }
return parser.emitError(loc, "invalid clause value: '") << enumStr << "'"; return parser.emitError(loc, "invalid clause value: '") << enumStr << "'";
} }
template <typename ClauseAttr> template <typename ClauseAttr>
void printClauseAttr(OpAsmPrinter &p, Operation *op, ClauseAttr attr) { void printClauseAttr(OpAsmPrinter &p, Operation *op, ClauseAttr attr) {
Show All 39 Lines if (linearStepVars.size() > i)
p << " = " << linearStepVars[i]; p << " = " << linearStepVars[i];
p << " : " << linearVars[i].getType() << separator; p << " : " << linearVars[i].getType() << separator;
} }
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Parser, verifier and printer for Aligned Clause // Parser, verifier and printer for Aligned Clause
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult verifyAlignedClause(Operation *op, static LogicalResult
Optional<ArrayAttr> alignmentValues, verifyAlignedClause(Operation *op, std::optional<ArrayAttr> alignmentValues,
OperandRange alignedVariables) { OperandRange alignedVariables) {
// Check if number of alignment values equals to number of aligned variables // Check if number of alignment values equals to number of aligned variables
if (!alignedVariables.empty()) { if (!alignedVariables.empty()) {
if (!alignmentValues || alignmentValues->size() != alignedVariables.size()) if (!alignmentValues || alignmentValues->size() != alignedVariables.size())
return op->emitOpError() return op->emitOpError()
<< "expected as many alignment values as aligned variables"; << "expected as many alignment values as aligned variables";
} else { } else {
if (alignmentValues) if (alignmentValues)
return op->emitOpError() << "unexpected alignment values attribute"; return op->emitOpError() << "unexpected alignment values attribute";
▲ Show 20 Lines • Show All 44 Lines • ▼ Show 20 Linesstatic ParseResult parseAlignedClause(
alignmentValues = ArrayAttr::get(parser.getContext(), alignments); alignmentValues = ArrayAttr::get(parser.getContext(), alignments);
return success(); return success();
} }
/// Print Aligned Clause /// Print Aligned Clause
static void printAlignedClause(OpAsmPrinter &p, Operation *op, static void printAlignedClause(OpAsmPrinter &p, Operation *op,
ValueRange alignedVars, ValueRange alignedVars,
TypeRange alignedVarTypes, TypeRange alignedVarTypes,
Optional<ArrayAttr> alignmentValues) { std::optional<ArrayAttr> alignmentValues) {
for (unsigned i = 0; i < alignedVars.size(); ++i) { for (unsigned i = 0; i < alignedVars.size(); ++i) {
if (i != 0) if (i != 0)
p << ", "; p << ", ";
p << alignedVars[i] << " : " << alignedVars[i].getType(); p << alignedVars[i] << " : " << alignedVars[i].getType();
p << " -> " << (*alignmentValues)[i]; p << " -> " << (*alignmentValues)[i];
} }
} }
Show All 40 Lines
/// sched-with-chunk ::= sched-with-chunk-types (`=` ssa-id-and-type)? /// sched-with-chunk ::= sched-with-chunk-types (`=` ssa-id-and-type)?
/// sched-with-chunk-types ::= `static` | `dynamic` | `guided` /// sched-with-chunk-types ::= `static` | `dynamic` | `guided`
/// sched-wo-chunk ::= `auto` | `runtime` /// sched-wo-chunk ::= `auto` | `runtime`
/// sched-modifier ::= sched-mod-val | sched-mod-val `,` sched-mod-val /// sched-modifier ::= sched-mod-val | sched-mod-val `,` sched-mod-val
/// sched-mod-val ::= `monotonic` | `nonmonotonic` | `simd` | `none` /// sched-mod-val ::= `monotonic` | `nonmonotonic` | `simd` | `none`
static ParseResult parseScheduleClause( static ParseResult parseScheduleClause(
OpAsmParser &parser, ClauseScheduleKindAttr &scheduleAttr, OpAsmParser &parser, ClauseScheduleKindAttr &scheduleAttr,
ScheduleModifierAttr &scheduleModifier, UnitAttr &simdModifier, ScheduleModifierAttr &scheduleModifier, UnitAttr &simdModifier,
Optional<OpAsmParser::UnresolvedOperand> &chunkSize, Type &chunkType) { std::optional<OpAsmParser::UnresolvedOperand> &chunkSize, Type &chunkType) {
StringRef keyword; StringRef keyword;
if (parser.parseKeyword(&keyword)) if (parser.parseKeyword(&keyword))
return failure(); return failure();
llvm::Optional<mlir::omp::ClauseScheduleKind> schedule = std::optional<mlir::omp::ClauseScheduleKind> schedule =
symbolizeClauseScheduleKind(keyword); symbolizeClauseScheduleKind(keyword);
if (!schedule) if (!schedule)
return parser.emitError(parser.getNameLoc()) << " expected schedule kind"; return parser.emitError(parser.getNameLoc()) << " expected schedule kind";
scheduleAttr = ClauseScheduleKindAttr::get(parser.getContext(), *schedule); scheduleAttr = ClauseScheduleKindAttr::get(parser.getContext(), *schedule);
switch (*schedule) { switch (*schedule) {
case ClauseScheduleKind::Static: case ClauseScheduleKind::Static:
case ClauseScheduleKind::Dynamic: case ClauseScheduleKind::Dynamic:
Show All 20 Lines while (succeeded(parser.parseOptionalComma())) {
modifiers.push_back(mod); modifiers.push_back(mod);
} }
if (verifyScheduleModifiers(parser, modifiers)) if (verifyScheduleModifiers(parser, modifiers))
return failure(); return failure();
if (!modifiers.empty()) { if (!modifiers.empty()) {
SMLoc loc = parser.getCurrentLocation(); SMLoc loc = parser.getCurrentLocation();
if (Optional<ScheduleModifier> mod = if (std::optional<ScheduleModifier> mod =
symbolizeScheduleModifier(modifiers[0])) { symbolizeScheduleModifier(modifiers[0])) {
scheduleModifier = ScheduleModifierAttr::get(parser.getContext(), *mod); scheduleModifier = ScheduleModifierAttr::get(parser.getContext(), *mod);
} else { } else {
return parser.emitError(loc, "invalid schedule modifier"); return parser.emitError(loc, "invalid schedule modifier");
} }
// Only SIMD attribute is allowed here! // Only SIMD attribute is allowed here!
if (modifiers.size() > 1) { if (modifiers.size() > 1) {
assert(symbolizeScheduleModifier(modifiers[1]) == ScheduleModifier::simd); assert(symbolizeScheduleModifier(modifiers[1]) == ScheduleModifier::simd);
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 LinesparseReductionVarList(OpAsmParser &parser,
redcuctionSymbols = ArrayAttr::get(parser.getContext(), reductions); redcuctionSymbols = ArrayAttr::get(parser.getContext(), reductions);
return success(); return success();
} }
/// Print Reduction clause /// Print Reduction clause
static void printReductionVarList(OpAsmPrinter &p, Operation *op, static void printReductionVarList(OpAsmPrinter &p, Operation *op,
OperandRange reductionVars, OperandRange reductionVars,
TypeRange reductionTypes, TypeRange reductionTypes,
Optional<ArrayAttr> reductions) { std::optional<ArrayAttr> reductions) {
for (unsigned i = 0, e = reductions->size(); i < e; ++i) { for (unsigned i = 0, e = reductions->size(); i < e; ++i) {
if (i != 0) if (i != 0)
p << ", "; p << ", ";
p << (*reductions)[i] << " -> " << reductionVars[i] << " : " p << (*reductions)[i] << " -> " << reductionVars[i] << " : "
<< reductionVars[i].getType(); << reductionVars[i].getType();
} }
} }
/// Verifies Reduction Clause /// Verifies Reduction Clause
static LogicalResult verifyReductionVarList(Operation *op, static LogicalResult verifyReductionVarList(Operation *op,
Optional<ArrayAttr> reductions, std::optional<ArrayAttr> reductions,
OperandRange reductionVars) { OperandRange reductionVars) {
if (!reductionVars.empty()) { if (!reductionVars.empty()) {
if (!reductions || reductions->size() != reductionVars.size()) if (!reductions || reductions->size() != reductionVars.size())
return op->emitOpError() return op->emitOpError()
<< "expected as many reduction symbol references " << "expected as many reduction symbol references "
"as reduction variables"; "as reduction variables";
} else { } else {
if (reductions) if (reductions)
▲ Show 20 Lines • Show All 728 Lines • Show Last 20 Lines

mlir/lib/Dialect/PDL/IR/PDL.cpp

Show First 20 Lines • Show All 106 Lines • ▼ Show 20 Lines
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// pdl::AttributeOp // pdl::AttributeOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
LogicalResult AttributeOp::verify() { LogicalResult AttributeOp::verify() {
Value attrType = getValueType(); Value attrType = getValueType();
Optional<Attribute> attrValue = getValue(); std::optional<Attribute> attrValue = getValue();
if (!attrValue) { if (!attrValue) {
if (isa<RewriteOp>((*this)->getParentOp())) if (isa<RewriteOp>((*this)->getParentOp()))
return emitOpError( return emitOpError(
"expected constant value when specified within a `pdl.rewrite`"); "expected constant value when specified within a `pdl.rewrite`");
return verifyHasBindingUse(*this); return verifyHasBindingUse(*this);
} }
if (attrType) if (attrType)
▲ Show 20 Lines • Show All 74 Lines • ▼ Show 20 Linesstatic LogicalResult verifyResultTypesAreInferrable(OperationOp op,
// types. // types.
if (llvm::any_of(op.getOp().getUses(), canInferTypeFromUse)) if (llvm::any_of(op.getOp().getUses(), canInferTypeFromUse))
return success(); return success();
// Handle the case where the operation has no explicit result types. // Handle the case where the operation has no explicit result types.
if (resultTypes.empty()) { if (resultTypes.empty()) {
// If we don't know the concrete operation, don't attempt any verification. // If we don't know the concrete operation, don't attempt any verification.
// We can't make assumptions if we don't know the concrete operation. // We can't make assumptions if we don't know the concrete operation.
Optional<StringRef> rawOpName = op.getOpName(); std::optional<StringRef> rawOpName = op.getOpName();
if (!rawOpName) if (!rawOpName)
return success(); return success();
Optional<RegisteredOperationName> opName = Optional<RegisteredOperationName> opName =
RegisteredOperationName::lookup(*rawOpName, op.getContext()); RegisteredOperationName::lookup(*rawOpName, op.getContext());
if (!opName) if (!opName)
return success(); return success();
// If no explicit result types were provided, check to see if the operation // If no explicit result types were provided, check to see if the operation
▲ Show 20 Lines • Show All 70 Lines • ▼ Show 20 Lines if (isWithinRewrite && !mightHaveTypeInference()) {
if (failed(verifyResultTypesAreInferrable(*this, getTypeValues()))) if (failed(verifyResultTypesAreInferrable(*this, getTypeValues())))
return failure(); return failure();
} }
return verifyHasBindingUse(*this); return verifyHasBindingUse(*this);
} }
bool OperationOp::hasTypeInference() { bool OperationOp::hasTypeInference() {
if (Optional<StringRef> rawOpName = getOpName()) { if (std::optional<StringRef> rawOpName = getOpName()) {
OperationName opName(*rawOpName, getContext()); OperationName opName(*rawOpName, getContext());
return opName.hasInterface<InferTypeOpInterface>(); return opName.hasInterface<InferTypeOpInterface>();
} }
return false; return false;
} }
bool OperationOp::mightHaveTypeInference() { bool OperationOp::mightHaveTypeInference() {
if (Optional<StringRef> rawOpName = getOpName()) { if (std::optional<StringRef> rawOpName = getOpName()) {
OperationName opName(*rawOpName, getContext()); OperationName opName(*rawOpName, getContext());
return opName.mightHaveInterface<InferTypeOpInterface>(); return opName.mightHaveInterface<InferTypeOpInterface>();
} }
return false; return false;
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// pdl::PatternOp // pdl::PatternOp
▲ Show 20 Lines • Show All 226 Lines • Show Last 20 Lines

mlir/lib/Dialect/SCF/IR/SCF.cpp

Show First 20 Lines • Show All 242 Lines • ▼ Show 20 Lines
} }
/// Given the region at `index`, or the parent operation if `index` is None, /// Given the region at `index`, or the parent operation if `index` is None,
/// return the successor regions. These are the regions that may be selected /// return the successor regions. These are the regions that may be selected
/// during the flow of control. `operands` is a set of optional attributes that /// during the flow of control. `operands` is a set of optional attributes that
/// correspond to a constant value for each operand, or null if that operand is /// correspond to a constant value for each operand, or null if that operand is
/// not a constant. /// not a constant.
void ExecuteRegionOp::getSuccessorRegions( void ExecuteRegionOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// If the predecessor is the ExecuteRegionOp, branch into the body. // If the predecessor is the ExecuteRegionOp, branch into the body.
if (!index) { if (!index) {
regions.push_back(RegionSuccessor(&getRegion())); regions.push_back(RegionSuccessor(&getRegion()));
return; return;
} }
// Otherwise, the region branches back to the parent operation. // Otherwise, the region branches back to the parent operation.
regions.push_back(RegionSuccessor(getResults())); regions.push_back(RegionSuccessor(getResults()));
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ConditionOp // ConditionOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
MutableOperandRange MutableOperandRange
ConditionOp::getMutableSuccessorOperands(Optional<unsigned> index) { ConditionOp::getMutableSuccessorOperands(std::optional<unsigned> index) {
// Pass all operands except the condition to the successor region. // Pass all operands except the condition to the successor region.
return getArgsMutable(); return getArgsMutable();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ForOp // ForOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 70 Lines • ▼ Show 20 Lines if (std::get<1>(e).getType() != std::get<2>(e).getType())
return emitOpError() << "types mismatch between " << i return emitOpError() << "types mismatch between " << i
<< "th iter region arg and defined value"; << "th iter region arg and defined value";
i++; i++;
} }
return success(); return success();
} }
Optional<Value> ForOp::getSingleInductionVar() { return getInductionVar(); } std::optional<Value> ForOp::getSingleInductionVar() {
return getInductionVar();
}
Optional<OpFoldResult> ForOp::getSingleLowerBound() { std::optional<OpFoldResult> ForOp::getSingleLowerBound() {
return OpFoldResult(getLowerBound()); return OpFoldResult(getLowerBound());
} }
Optional<OpFoldResult> ForOp::getSingleStep() { std::optional<OpFoldResult> ForOp::getSingleStep() {
return OpFoldResult(getStep()); return OpFoldResult(getStep());
} }
Optional<OpFoldResult> ForOp::getSingleUpperBound() { std::optional<OpFoldResult> ForOp::getSingleUpperBound() {
return OpFoldResult(getUpperBound()); return OpFoldResult(getUpperBound());
} }
/// Prints the initialization list in the form of /// Prints the initialization list in the form of
/// <prefix>(%inner = %outer, %inner2 = %outer2, <...>) /// <prefix>(%inner = %outer, %inner2 = %outer2, <...>)
/// where 'inner' values are assumed to be region arguments and 'outer' values /// where 'inner' values are assumed to be region arguments and 'outer' values
/// are regular SSA values. /// are regular SSA values.
static void printInitializationList(OpAsmPrinter &p, static void printInitializationList(OpAsmPrinter &p,
▲ Show 20 Lines • Show All 97 Lines • ▼ Show 20 LinesForOp mlir::scf::getForInductionVarOwner(Value val) {
auto *containingOp = ivArg.getOwner()->getParentOp(); auto *containingOp = ivArg.getOwner()->getParentOp();
return dyn_cast_or_null<ForOp>(containingOp); return dyn_cast_or_null<ForOp>(containingOp);
} }
/// Return operands used when entering the region at 'index'. These operands /// Return operands used when entering the region at 'index'. These operands
/// correspond to the loop iterator operands, i.e., those excluding the /// correspond to the loop iterator operands, i.e., those excluding the
/// induction variable. LoopOp only has one region, so 0 is the only valid value /// induction variable. LoopOp only has one region, so 0 is the only valid value
/// for `index`. /// for `index`.
OperandRange ForOp::getSuccessorEntryOperands(Optional<unsigned> index) { OperandRange ForOp::getSuccessorEntryOperands(std::optional<unsigned> index) {
assert(index && *index == 0 && "invalid region index"); assert(index && *index == 0 && "invalid region index");
// The initial operands map to the loop arguments after the induction // The initial operands map to the loop arguments after the induction
// variable. // variable.
return getInitArgs(); return getInitArgs();
} }
/// Given the region at `index`, or the parent operation if `index` is None, /// Given the region at `index`, or the parent operation if `index` is None,
/// return the successor regions. These are the regions that may be selected /// return the successor regions. These are the regions that may be selected
/// during the flow of control. `operands` is a set of optional attributes that /// during the flow of control. `operands` is a set of optional attributes that
/// correspond to a constant value for each operand, or null if that operand is /// correspond to a constant value for each operand, or null if that operand is
/// not a constant. /// not a constant.
void ForOp::getSuccessorRegions(Optional<unsigned> index, void ForOp::getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute> operands, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// If the predecessor is the ForOp, branch into the body using the iterator // If the predecessor is the ForOp, branch into the body using the iterator
// arguments. // arguments.
if (!index) { if (!index) {
regions.push_back(RegionSuccessor(&getLoopBody(), getRegionIterArgs())); regions.push_back(RegionSuccessor(&getLoopBody(), getRegionIterArgs()));
return; return;
} }
▲ Show 20 Lines • Show All 215 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(scf::ForOp forOp,
rewriter.replaceOp(forOp, newResultValues); rewriter.replaceOp(forOp, newResultValues);
return success(); return success();
} }
}; };
/// Util function that tries to compute a constant diff between u and l. /// Util function that tries to compute a constant diff between u and l.
/// Returns std::nullopt when the difference between two AffineValueMap is /// Returns std::nullopt when the difference between two AffineValueMap is
/// dynamic. /// dynamic.
static Optional<int64_t> computeConstDiff(Value l, Value u) { static std::optional<int64_t> computeConstDiff(Value l, Value u) {
IntegerAttr clb, cub; IntegerAttr clb, cub;
if (matchPattern(l, m_Constant(&clb)) && matchPattern(u, m_Constant(&cub))) { if (matchPattern(l, m_Constant(&clb)) && matchPattern(u, m_Constant(&cub))) {
llvm::APInt lbValue = clb.getValue(); llvm::APInt lbValue = clb.getValue();
llvm::APInt ubValue = cub.getValue(); llvm::APInt ubValue = cub.getValue();
return (ubValue - lbValue).getSExtValue(); return (ubValue - lbValue).getSExtValue();
} }
// Else a simple pattern match for x + c or c + x // Else a simple pattern match for x + c or c + x
Show All 16 Lines LogicalResult matchAndRewrite(ForOp op,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
// If the upper bound is the same as the lower bound, the loop does not // If the upper bound is the same as the lower bound, the loop does not
// iterate, just remove it. // iterate, just remove it.
if (op.getLowerBound() == op.getUpperBound()) { if (op.getLowerBound() == op.getUpperBound()) {
rewriter.replaceOp(op, op.getIterOperands()); rewriter.replaceOp(op, op.getIterOperands());
return success(); return success();
} }
Optional<int64_t> diff = std::optional<int64_t> diff =
computeConstDiff(op.getLowerBound(), op.getUpperBound()); computeConstDiff(op.getLowerBound(), op.getUpperBound());
if (!diff) if (!diff)
return failure(); return failure();
// If the loop is known to have 0 iterations, remove it. // If the loop is known to have 0 iterations, remove it.
if (*diff <= 0) { if (*diff <= 0) {
rewriter.replaceOp(op, op.getIterOperands()); rewriter.replaceOp(op, op.getIterOperands());
return success(); return success();
} }
llvm::Optional<llvm::APInt> maybeStepValue = op.getConstantStep(); std::optional<llvm::APInt> maybeStepValue = op.getConstantStep();
if (!maybeStepValue) if (!maybeStepValue)
return failure(); return failure();
// If the loop is known to have 1 iteration, inline its body and remove the // If the loop is known to have 1 iteration, inline its body and remove the
// loop. // loop.
llvm::APInt stepValue = *maybeStepValue; llvm::APInt stepValue = *maybeStepValue;
if (stepValue.sge(*diff)) { if (stepValue.sge(*diff)) {
SmallVector<Value, 4> blockArgs; SmallVector<Value, 4> blockArgs;
▲ Show 20 Lines • Show All 286 Lines • ▼ Show 20 Lines
} // namespace } // namespace
void ForOp::getCanonicalizationPatterns(RewritePatternSet &results, void ForOp::getCanonicalizationPatterns(RewritePatternSet &results,
MLIRContext *context) { MLIRContext *context) {
results.add<ForOpIterArgsFolder, SimplifyTrivialLoops, results.add<ForOpIterArgsFolder, SimplifyTrivialLoops,
LastTensorLoadCanonicalization, ForOpTensorCastFolder>(context); LastTensorLoadCanonicalization, ForOpTensorCastFolder>(context);
} }
Optional<APInt> ForOp::getConstantStep() { std::optional<APInt> ForOp::getConstantStep() {
IntegerAttr step; IntegerAttr step;
if (matchPattern(getStep(), m_Constant(&step))) if (matchPattern(getStep(), m_Constant(&step)))
return step.getValue(); return step.getValue();
return {}; return {};
} }
Speculation::Speculatability ForOp::getSpeculatability() { Speculation::Speculatability ForOp::getSpeculatability() {
// `scf.for (I = Start; I < End; I += 1)` terminates for all values of Start // `scf.for (I = Start; I < End; I += 1)` terminates for all values of Start
▲ Show 20 Lines • Show All 127 Lines • ▼ Show 20 Lines result.addAttribute("operand_segment_sizes",
static_cast<int32_t>(outOperands.size())})); static_cast<int32_t>(outOperands.size())}));
return success(); return success();
} }
// Bodyless builder, outputs must be specified. // Bodyless builder, outputs must be specified.
void ForeachThreadOp::build(mlir::OpBuilder &builder, void ForeachThreadOp::build(mlir::OpBuilder &builder,
mlir::OperationState &result, ValueRange outputs, mlir::OperationState &result, ValueRange outputs,
ValueRange numThreads, ValueRange numThreads,
Optional<ArrayAttr> mapping) { std::optional<ArrayAttr> mapping) {
result.addOperands(numThreads); result.addOperands(numThreads);
result.addOperands(outputs); result.addOperands(outputs);
if (mapping.has_value()) { if (mapping.has_value()) {
result.addAttribute(ForeachThreadOp::getMappingAttrName(result.name), result.addAttribute(ForeachThreadOp::getMappingAttrName(result.name),
mapping.value()); mapping.value());
} }
result.addAttribute( result.addAttribute(
▲ Show 20 Lines • Show All 336 Lines • ▼ Show 20 Linesvoid IfOp::print(OpAsmPrinter &p) {
p.printOptionalAttrDict((*this)->getAttrs()); p.printOptionalAttrDict((*this)->getAttrs());
} }
/// Given the region at `index`, or the parent operation if `index` is None, /// Given the region at `index`, or the parent operation if `index` is None,
/// return the successor regions. These are the regions that may be selected /// return the successor regions. These are the regions that may be selected
/// during the flow of control. `operands` is a set of optional attributes that /// during the flow of control. `operands` is a set of optional attributes that
/// correspond to a constant value for each operand, or null if that operand is /// correspond to a constant value for each operand, or null if that operand is
/// not a constant. /// not a constant.
void IfOp::getSuccessorRegions(Optional<unsigned> index, void IfOp::getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute> operands, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// The `then` and the `else` region branch back to the parent operation. // The `then` and the `else` region branch back to the parent operation.
if (index) { if (index) {
regions.push_back(RegionSuccessor(getResults())); regions.push_back(RegionSuccessor(getResults()));
return; return;
} }
▲ Show 20 Lines • Show All 1,141 Lines • ▼ Show 20 Linesvoid WhileOp::build(::mlir::OpBuilder &odsBuilder,
beforeBuilder(odsBuilder, odsState.location, beforeBlock->getArguments()); beforeBuilder(odsBuilder, odsState.location, beforeBlock->getArguments());
Region *afterRegion = odsState.addRegion(); Region *afterRegion = odsState.addRegion();
Block *afterBlock = odsBuilder.createBlock(afterRegion, /*insertPt=*/{}, Block *afterBlock = odsBuilder.createBlock(afterRegion, /*insertPt=*/{},
resultTypes, blockArgLocs); resultTypes, blockArgLocs);
afterBuilder(odsBuilder, odsState.location, afterBlock->getArguments()); afterBuilder(odsBuilder, odsState.location, afterBlock->getArguments());
} }
OperandRange WhileOp::getSuccessorEntryOperands(Optional<unsigned> index) { OperandRange WhileOp::getSuccessorEntryOperands(std::optional<unsigned> index) {
assert(index && *index == 0 && assert(index && *index == 0 &&
"WhileOp is expected to branch only to the first region"); "WhileOp is expected to branch only to the first region");
return getInits(); return getInits();
} }
ConditionOp WhileOp::getConditionOp() { ConditionOp WhileOp::getConditionOp() {
return cast<ConditionOp>(getBefore().front().getTerminator()); return cast<ConditionOp>(getBefore().front().getTerminator());
} }
YieldOp WhileOp::getYieldOp() { YieldOp WhileOp::getYieldOp() {
return cast<YieldOp>(getAfter().front().getTerminator()); return cast<YieldOp>(getAfter().front().getTerminator());
} }
Block::BlockArgListType WhileOp::getBeforeArguments() { Block::BlockArgListType WhileOp::getBeforeArguments() {
return getBefore().front().getArguments(); return getBefore().front().getArguments();
} }
Block::BlockArgListType WhileOp::getAfterArguments() { Block::BlockArgListType WhileOp::getAfterArguments() {
return getAfter().front().getArguments(); return getAfter().front().getArguments();
} }
void WhileOp::getSuccessorRegions(Optional<unsigned> index, void WhileOp::getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute> operands, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// The parent op always branches to the condition region. // The parent op always branches to the condition region.
if (!index) { if (!index) {
regions.emplace_back(&getBefore(), getBefore().getArguments()); regions.emplace_back(&getBefore(), getBefore().getArguments());
return; return;
} }
▲ Show 20 Lines • Show All 761 Lines • ▼ Show 20 Lines
} }
Block &scf::IndexSwitchOp::getCaseBlock(unsigned idx) { Block &scf::IndexSwitchOp::getCaseBlock(unsigned idx) {
assert(idx < getNumCases() && "case index out-of-bounds"); assert(idx < getNumCases() && "case index out-of-bounds");
return getCaseRegions()[idx].front(); return getCaseRegions()[idx].front();
} }
void IndexSwitchOp::getSuccessorRegions( void IndexSwitchOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &successors) { SmallVectorImpl<RegionSuccessor> &successors) {
// All regions branch back to the parent op. // All regions branch back to the parent op.
if (index) { if (index) {
successors.emplace_back(getResults()); successors.emplace_back(getResults());
return; return;
} }
// If a constant was not provided, all regions are possible successors. // If a constant was not provided, all regions are possible successors.
▲ Show 20 Lines • Show All 42 Lines • Show Last 20 Lines

mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp

Show First 20 Lines • Show All 447 Lines • ▼ Show 20 Lines
#endif // NDEBUG #endif // NDEBUG
return getMemRefTypeWithFullyDynamicLayout( return getMemRefTypeWithFullyDynamicLayout(
iterArg.getType().cast<RankedTensorType>(), iterArg.getType().cast<RankedTensorType>(),
yieldedRanked.getMemorySpace()); yieldedRanked.getMemorySpace());
} }
/// Return `true` if the given loop may have 0 iterations. /// Return `true` if the given loop may have 0 iterations.
bool mayHaveZeroIterations(scf::ForOp forOp) { bool mayHaveZeroIterations(scf::ForOp forOp) {
Optional<int64_t> lb = getConstantIntValue(forOp.getLowerBound()); std::optional<int64_t> lb = getConstantIntValue(forOp.getLowerBound());
Optional<int64_t> ub = getConstantIntValue(forOp.getUpperBound()); std::optional<int64_t> ub = getConstantIntValue(forOp.getUpperBound());
if (!lb.has_value() || !ub.has_value()) if (!lb.has_value() || !ub.has_value())
return true; return true;
return *ub <= *lb; return *ub <= *lb;
} }
/// Bufferization of scf.for. Replace with a new scf.for that operates on /// Bufferization of scf.for. Replace with a new scf.for that operates on
/// memrefs. /// memrefs.
struct ForOpInterface struct ForOpInterface
▲ Show 20 Lines • Show All 584 Lines • ▼ Show 20 Lines LogicalResult bufferize(Operation *op, RewriterBase &rewriter,
return success(); return success();
} }
}; };
/// Return `true` if the given loop may have 0 iterations. /// Return `true` if the given loop may have 0 iterations.
bool mayHaveZeroIterations(scf::ForeachThreadOp foreachThreadOp) { bool mayHaveZeroIterations(scf::ForeachThreadOp foreachThreadOp) {
int64_t p = 1; int64_t p = 1;
for (Value v : foreachThreadOp.getNumThreads()) { for (Value v : foreachThreadOp.getNumThreads()) {
if (Optional<int64_t> c = getConstantIntValue(v)) { if (std::optional<int64_t> c = getConstantIntValue(v)) {
p *= *c; p *= *c;
} else { } else {
return true; return true;
} }
} }
return p == 0; return p == 0;
} }
▲ Show 20 Lines • Show All 152 Lines • Show Last 20 Lines

mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp

Show First 20 Lines • Show All 60 Lines • ▼ Show 20 Lines
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// tileUsingSCFForOp implementation. // tileUsingSCFForOp implementation.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Check if `stride` evenly divides the trip count `size - offset`. // Check if `stride` evenly divides the trip count `size - offset`.
static bool tileDividesIterationDomain(Range loopRange) { static bool tileDividesIterationDomain(Range loopRange) {
Optional<int64_t> offsetAsInt = getConstantIntValue(loopRange.offset); std::optional<int64_t> offsetAsInt = getConstantIntValue(loopRange.offset);
if (!offsetAsInt) if (!offsetAsInt)
return false; return false;
Optional<int64_t> sizeAsInt = getConstantIntValue(loopRange.size); std::optional<int64_t> sizeAsInt = getConstantIntValue(loopRange.size);
if (!sizeAsInt) if (!sizeAsInt)
return false; return false;
Optional<int64_t> strideAsInt = getConstantIntValue(loopRange.stride); std::optional<int64_t> strideAsInt = getConstantIntValue(loopRange.stride);
if (!strideAsInt) if (!strideAsInt)
return false; return false;
return ((sizeAsInt.value() - offsetAsInt.value()) % strideAsInt.value() == 0); return ((sizeAsInt.value() - offsetAsInt.value()) % strideAsInt.value() == 0);
} }
/// Returns the bounded tile size given the current `iv`, `loopRange` and /// Returns the bounded tile size given the current `iv`, `loopRange` and
/// `tileSize`, i.e., `min(tileSize, range.end() - iv)`. /// `tileSize`, i.e., `min(tileSize, range.end() - iv)`.
static OpFoldResult getBoundedTileSize(OpBuilder &b, Location loc, static OpFoldResult getBoundedTileSize(OpBuilder &b, Location loc,
Range loopRange, Value iv, Range loopRange, Value iv,
Value tileSize) { Value tileSize) {
Optional<int64_t> ts = getConstantIntValue(tileSize); std::optional<int64_t> ts = getConstantIntValue(tileSize);
if (ts && ts.value() == 1) if (ts && ts.value() == 1)
return getAsOpFoldResult(tileSize); return getAsOpFoldResult(tileSize);
if (tileDividesIterationDomain( if (tileDividesIterationDomain(
Range{loopRange.offset, loopRange.size, tileSize})) Range{loopRange.offset, loopRange.size, tileSize}))
return tileSize; return tileSize;
// The tile size to use (to avoid out of bounds access) is minimum of // The tile size to use (to avoid out of bounds access) is minimum of
▲ Show 20 Lines • Show All 384 Lines • ▼ Show 20 Lines
// tileConsumerAndFuseProducerGreedilyUsingSCFForOp implementation. // tileConsumerAndFuseProducerGreedilyUsingSCFForOp implementation.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Return the untiled producer whose slice is used in a tiled consumer. The /// Return the untiled producer whose slice is used in a tiled consumer. The
/// method traverses the tile loop nest (`loops`) if needed, and returns the /// method traverses the tile loop nest (`loops`) if needed, and returns the
/// `iter_args` of the outer most that is encountered. Traversing the iter_args /// `iter_args` of the outer most that is encountered. Traversing the iter_args
/// indicates that this is a destination operand of the consumer. If there was /// indicates that this is a destination operand of the consumer. If there was
/// no loop traversal needed, the second value of the returned tuple is empty. /// no loop traversal needed, the second value of the returned tuple is empty.
static std::tuple<OpResult, Optional<OpOperand *>> static std::tuple<OpResult, std::optional<OpOperand *>>
getUntiledProducerFromSliceSource(OpOperand *source, getUntiledProducerFromSliceSource(OpOperand *source,
ArrayRef<scf::ForOp> loops) { ArrayRef<scf::ForOp> loops) {
Optional<OpOperand *> destinationIterArg; std::optional<OpOperand *> destinationIterArg;
auto loopIt = loops.rbegin(); auto loopIt = loops.rbegin();
while (auto iterArg = source->get().dyn_cast<BlockArgument>()) { while (auto iterArg = source->get().dyn_cast<BlockArgument>()) {
scf::ForOp loop = *loopIt; scf::ForOp loop = *loopIt;
if (iterArg.getOwner()->getParentOp() != loop) if (iterArg.getOwner()->getParentOp() != loop)
break; break;
source = &loop.getOpOperandForRegionIterArg(iterArg); source = &loop.getOpOperandForRegionIterArg(iterArg);
loopIt++; loopIt++;
} }
▲ Show 20 Lines • Show All 129 Lines • ▼ Show 20 Lines while (!candidates.empty()) {
// %4 = linalg.fill .. outs(%3 : ) // %4 = linalg.fill .. outs(%3 : )
// .. = linalg.matmul .. outs(%4 : ) // .. = linalg.matmul .. outs(%4 : )
// } // }
// } // }
// ``` // ```
// TODO: This can be modeled better if the `DestinationStyleOpInterface`. // TODO: This can be modeled better if the `DestinationStyleOpInterface`.
// Update to use that when it does become available. // Update to use that when it does become available.
scf::ForOp outerMostLoop = tileAndFuseResult.loops.front(); scf::ForOp outerMostLoop = tileAndFuseResult.loops.front();
Optional<unsigned> iterArgNumber; std::optional<unsigned> iterArgNumber;
if (destinationIterArg) { if (destinationIterArg) {
iterArgNumber = outerMostLoop.getIterArgNumberForOpOperand( iterArgNumber = outerMostLoop.getIterArgNumberForOpOperand(
*destinationIterArg.value()); *destinationIterArg.value());
} }
if (iterArgNumber) { if (iterArgNumber) {
int64_t resultNumber = fusableProducer.getResultNumber(); int64_t resultNumber = fusableProducer.getResultNumber();
if (auto dstOp = dyn_cast<DestinationStyleOpInterface>( if (auto dstOp = dyn_cast<DestinationStyleOpInterface>(
fusableProducer.getOwner())) { fusableProducer.getOwner())) {
▲ Show 20 Lines • Show All 51 Lines • Show Last 20 Lines

mlir/lib/Dialect/SCF/Utils/AffineCanonicalizationUtils.cpp

Show First 20 Lines • Show All 212 Lines • ▼ Show 20 LinesaddLoopRangeConstraints(FlatAffineValueConstraints &constraints, Value iv,
auto lbv = lb.dyn_cast<Value>(); auto lbv = lb.dyn_cast<Value>();
unsigned symLb = lbv ? constraints.appendSymbolVar(lbv) unsigned symLb = lbv ? constraints.appendSymbolVar(lbv)
: constraints.appendSymbolVar(/*num=*/1); : constraints.appendSymbolVar(/*num=*/1);
auto ubv = ub.dyn_cast<Value>(); auto ubv = ub.dyn_cast<Value>();
unsigned symUb = ubv ? constraints.appendSymbolVar(ubv) unsigned symUb = ubv ? constraints.appendSymbolVar(ubv)
: constraints.appendSymbolVar(/*num=*/1); : constraints.appendSymbolVar(/*num=*/1);
// If loop lower/upper bounds are constant: Add EQ constraint. // If loop lower/upper bounds are constant: Add EQ constraint.
Optional<int64_t> lbInt = getConstantIntValue(lb); std::optional<int64_t> lbInt = getConstantIntValue(lb);
Optional<int64_t> ubInt = getConstantIntValue(ub); std::optional<int64_t> ubInt = getConstantIntValue(ub);
if (lbInt) if (lbInt)
constraints.addBound(IntegerPolyhedron::EQ, symLb, *lbInt); constraints.addBound(IntegerPolyhedron::EQ, symLb, *lbInt);
if (ubInt) if (ubInt)
constraints.addBound(IntegerPolyhedron::EQ, symUb, *ubInt); constraints.addBound(IntegerPolyhedron::EQ, symUb, *ubInt);
// Lower bound: iv >= lb (equiv.: iv - lb >= 0) // Lower bound: iv >= lb (equiv.: iv - lb >= 0)
SmallVector<int64_t> ineqLb(constraints.getNumCols(), 0); SmallVector<int64_t> ineqLb(constraints.getNumCols(), 0);
ineqLb[dimIv] = 1; ineqLb[dimIv] = 1;
▲ Show 20 Lines • Show All 124 Lines • Show Last 20 Lines

mlir/lib/Dialect/SPIRV/IR/SPIRVDialect.cpp

Show First 20 Lines • Show All 136 Lines • ▼ Show 20 Lines
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Type Parsing // Type Parsing
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Forward declarations. // Forward declarations.
template <typename ValTy> template <typename ValTy>
static Optional<ValTy> parseAndVerify(SPIRVDialect const &dialect, static std::optional<ValTy> parseAndVerify(SPIRVDialect const &dialect,
DialectAsmParser &parser); DialectAsmParser &parser);
template <> template <>
Optional<Type> parseAndVerify<Type>(SPIRVDialect const &dialect, std::optional<Type> parseAndVerify<Type>(SPIRVDialect const &dialect,
DialectAsmParser &parser); DialectAsmParser &parser);
template <> template <>
Optional<unsigned> parseAndVerify<unsigned>(SPIRVDialect const &dialect, std::optional<unsigned> parseAndVerify<unsigned>(SPIRVDialect const &dialect,
DialectAsmParser &parser); DialectAsmParser &parser);
static Type parseAndVerifyType(SPIRVDialect const &dialect, static Type parseAndVerifyType(SPIRVDialect const &dialect,
DialectAsmParser &parser) { DialectAsmParser &parser) {
Type type; Type type;
SMLoc typeLoc = parser.getCurrentLocation(); SMLoc typeLoc = parser.getCurrentLocation();
if (parser.parseType(type)) if (parser.parseType(type))
return Type(); return Type();
▲ Show 20 Lines • Show All 97 Lines • ▼ Show 20 Lines if (failed(parser.parseOptionalComma())) {
stride = 0; stride = 0;
return success(); return success();
} }
if (parser.parseKeyword("stride") || parser.parseEqual()) if (parser.parseKeyword("stride") || parser.parseEqual())
return failure(); return failure();
SMLoc strideLoc = parser.getCurrentLocation(); SMLoc strideLoc = parser.getCurrentLocation();
Optional<unsigned> optStride = parseAndVerify<unsigned>(dialect, parser); std::optional<unsigned> optStride = parseAndVerify<unsigned>(dialect, parser);
if (!optStride) if (!optStride)
return failure(); return failure();
if (!(stride = *optStride)) { if (!(stride = *optStride)) {
parser.emitError(strideLoc, "ArrayStride must be greater than zero"); parser.emitError(strideLoc, "ArrayStride must be greater than zero");
return failure(); return failure();
} }
return success(); return success();
▲ Show 20 Lines • Show All 193 Lines • ▼ Show 20 Lines if (parser.parseGreater())
return Type(); return Type();
return MatrixType::get(columnType, columnCount); return MatrixType::get(columnType, columnCount);
} }
// Specialize this function to parse each of the parameters that define an // Specialize this function to parse each of the parameters that define an
// ImageType. By default it assumes this is an enum type. // ImageType. By default it assumes this is an enum type.
template <typename ValTy> template <typename ValTy>
static Optional<ValTy> parseAndVerify(SPIRVDialect const &dialect, static std::optional<ValTy> parseAndVerify(SPIRVDialect const &dialect,
DialectAsmParser &parser) { DialectAsmParser &parser) {
StringRef enumSpec; StringRef enumSpec;
SMLoc enumLoc = parser.getCurrentLocation(); SMLoc enumLoc = parser.getCurrentLocation();
if (parser.parseKeyword(&enumSpec)) { if (parser.parseKeyword(&enumSpec)) {
return std::nullopt; return std::nullopt;
} }
auto val = spirv::symbolizeEnum<ValTy>(enumSpec); auto val = spirv::symbolizeEnum<ValTy>(enumSpec);
if (!val) if (!val)
parser.emitError(enumLoc, "unknown attribute: '") << enumSpec << "'"; parser.emitError(enumLoc, "unknown attribute: '") << enumSpec << "'";
return val; return val;
} }
template <> template <>
Optional<Type> parseAndVerify<Type>(SPIRVDialect const &dialect, std::optional<Type> parseAndVerify<Type>(SPIRVDialect const &dialect,
DialectAsmParser &parser) { DialectAsmParser &parser) {
// TODO: Further verify that the element type can be sampled // TODO: Further verify that the element type can be sampled
auto ty = parseAndVerifyType(dialect, parser); auto ty = parseAndVerifyType(dialect, parser);
if (!ty) if (!ty)
return std::nullopt; return std::nullopt;
return ty; return ty;
} }
template <typename IntTy> template <typename IntTy>
static Optional<IntTy> parseAndVerifyInteger(SPIRVDialect const &dialect, static std::optional<IntTy> parseAndVerifyInteger(SPIRVDialect const &dialect,
DialectAsmParser &parser) { DialectAsmParser &parser) {
IntTy offsetVal = std::numeric_limits<IntTy>::max(); IntTy offsetVal = std::numeric_limits<IntTy>::max();
if (parser.parseInteger(offsetVal)) if (parser.parseInteger(offsetVal))
return std::nullopt; return std::nullopt;
return offsetVal; return offsetVal;
} }
template <> template <>
Optional<unsigned> parseAndVerify<unsigned>(SPIRVDialect const &dialect, std::optional<unsigned> parseAndVerify<unsigned>(SPIRVDialect const &dialect,
DialectAsmParser &parser) { DialectAsmParser &parser) {
return parseAndVerifyInteger<unsigned>(dialect, parser); return parseAndVerifyInteger<unsigned>(dialect, parser);
} }
namespace { namespace {
// Functor object to parse a comma separated list of specs. The function // Functor object to parse a comma separated list of specs. The function
// parseAndVerify does the actual parsing and verification of individual // parseAndVerify does the actual parsing and verification of individual
// elements. This is a functor since parsing the last element of the list // elements. This is a functor since parsing the last element of the list
// (termination condition) needs partial specialization. // (termination condition) needs partial specialization.
template <typename ParseType, typename... Args> template <typename ParseType, typename... Args>
struct ParseCommaSeparatedList { struct ParseCommaSeparatedList {
Optional<std::tuple<ParseType, Args...>> std::optional<std::tuple<ParseType, Args...>>
operator()(SPIRVDialect const &dialect, DialectAsmParser &parser) const { operator()(SPIRVDialect const &dialect, DialectAsmParser &parser) const {
auto parseVal = parseAndVerify<ParseType>(dialect, parser); auto parseVal = parseAndVerify<ParseType>(dialect, parser);
if (!parseVal) if (!parseVal)
return std::nullopt; return std::nullopt;
auto numArgs = std::tuple_size<std::tuple<Args...>>::value; auto numArgs = std::tuple_size<std::tuple<Args...>>::value;
if (numArgs != 0 && failed(parser.parseComma())) if (numArgs != 0 && failed(parser.parseComma()))
return std::nullopt; return std::nullopt;
auto remainingValues = ParseCommaSeparatedList<Args...>{}(dialect, parser); auto remainingValues = ParseCommaSeparatedList<Args...>{}(dialect, parser);
if (!remainingValues) if (!remainingValues)
return std::nullopt; return std::nullopt;
return std::tuple_cat(std::tuple<ParseType>(parseVal.value()), return std::tuple_cat(std::tuple<ParseType>(parseVal.value()),
remainingValues.value()); remainingValues.value());
} }
}; };
// Partial specialization of the function to parse a comma separated list of // Partial specialization of the function to parse a comma separated list of
// specs to parse the last element of the list. // specs to parse the last element of the list.
template <typename ParseType> template <typename ParseType>
struct ParseCommaSeparatedList<ParseType> { struct ParseCommaSeparatedList<ParseType> {
Optional<std::tuple<ParseType>> operator()(SPIRVDialect const &dialect, std::optional<std::tuple<ParseType>>
DialectAsmParser &parser) const { operator()(SPIRVDialect const &dialect, DialectAsmParser &parser) const {
if (auto value = parseAndVerify<ParseType>(dialect, parser)) if (auto value = parseAndVerify<ParseType>(dialect, parser))
return std::tuple<ParseType>(*value); return std::tuple<ParseType>(*value);
return std::nullopt; return std::nullopt;
} }
}; };
} // namespace } // namespace
// dim ::= `1D` | `2D` | `3D` | `Cube` | <and other SPIR-V Dim specifiers...> // dim ::= `1D` | `2D` | `3D` | `Cube` | <and other SPIR-V Dim specifiers...>
▲ Show 20 Lines • Show All 441 Lines • Show Last 20 Lines

mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp

Show First 20 Lines • Show All 307 Lines • ▼ Show 20 Linesstatic ParseResult parseSourceMemoryAccessAttributes(OpAsmParser &parser,
} }
return parser.parseRSquare(); return parser.parseRSquare();
} }
template <typename MemoryOpTy> template <typename MemoryOpTy>
static void printMemoryAccessAttribute( static void printMemoryAccessAttribute(
MemoryOpTy memoryOp, OpAsmPrinter &printer, MemoryOpTy memoryOp, OpAsmPrinter &printer,
SmallVectorImpl<StringRef> &elidedAttrs, SmallVectorImpl<StringRef> &elidedAttrs,
Optional<spirv::MemoryAccess> memoryAccessAtrrValue = std::nullopt, std::optional<spirv::MemoryAccess> memoryAccessAtrrValue = std::nullopt,
Optional<uint32_t> alignmentAttrValue = std::nullopt) { std::optional<uint32_t> alignmentAttrValue = std::nullopt) {
// Print optional memory access attribute. // Print optional memory access attribute.
if (auto memAccess = (memoryAccessAtrrValue ? memoryAccessAtrrValue if (auto memAccess = (memoryAccessAtrrValue ? memoryAccessAtrrValue
: memoryOp.getMemoryAccess())) { : memoryOp.getMemoryAccess())) {
elidedAttrs.push_back(kMemoryAccessAttrName); elidedAttrs.push_back(kMemoryAccessAttrName);
printer << " [\"" << stringifyMemoryAccess(*memAccess) << "\""; printer << " [\"" << stringifyMemoryAccess(*memAccess) << "\"";
if (spirv::bitEnumContainsAll(*memAccess, spirv::MemoryAccess::Aligned)) { if (spirv::bitEnumContainsAll(*memAccess, spirv::MemoryAccess::Aligned)) {
Show All 12 Lines
// TODO Make sure to merge this and the previous function into one template // TODO Make sure to merge this and the previous function into one template
// parameterized by memory access attribute name and alignment. Doing so now // parameterized by memory access attribute name and alignment. Doing so now
// results in VS2017 in producing an internal error (at the call site) that's // results in VS2017 in producing an internal error (at the call site) that's
// not detailed enough to understand what is happening. // not detailed enough to understand what is happening.
template <typename MemoryOpTy> template <typename MemoryOpTy>
static void printSourceMemoryAccessAttribute( static void printSourceMemoryAccessAttribute(
MemoryOpTy memoryOp, OpAsmPrinter &printer, MemoryOpTy memoryOp, OpAsmPrinter &printer,
SmallVectorImpl<StringRef> &elidedAttrs, SmallVectorImpl<StringRef> &elidedAttrs,
Optional<spirv::MemoryAccess> memoryAccessAtrrValue = std::nullopt, std::optional<spirv::MemoryAccess> memoryAccessAtrrValue = std::nullopt,
Optional<uint32_t> alignmentAttrValue = std::nullopt) { std::optional<uint32_t> alignmentAttrValue = std::nullopt) {
printer << ", "; printer << ", ";
// Print optional memory access attribute. // Print optional memory access attribute.
if (auto memAccess = (memoryAccessAtrrValue ? memoryAccessAtrrValue if (auto memAccess = (memoryAccessAtrrValue ? memoryAccessAtrrValue
: memoryOp.getMemoryAccess())) { : memoryOp.getMemoryAccess())) {
elidedAttrs.push_back(kSourceMemoryAccessAttrName); elidedAttrs.push_back(kSourceMemoryAccessAttrName);
▲ Show 20 Lines • Show All 551 Lines • ▼ Show 20 Linesstatic ParseResult parseGroupNonUniformArithmeticOp(OpAsmParser &parser,
OpAsmParser::UnresolvedOperand valueInfo; OpAsmParser::UnresolvedOperand valueInfo;
if (parseEnumStrAttr<spirv::ScopeAttr>(executionScope, parser, state, if (parseEnumStrAttr<spirv::ScopeAttr>(executionScope, parser, state,
kExecutionScopeAttrName) || kExecutionScopeAttrName) ||
parseEnumStrAttr<spirv::GroupOperationAttr>(groupOperation, parser, state, parseEnumStrAttr<spirv::GroupOperationAttr>(groupOperation, parser, state,
kGroupOperationAttrName) || kGroupOperationAttrName) ||
parser.parseOperand(valueInfo)) parser.parseOperand(valueInfo))
return failure(); return failure();
Optional<OpAsmParser::UnresolvedOperand> clusterSizeInfo; std::optional<OpAsmParser::UnresolvedOperand> clusterSizeInfo;
if (succeeded(parser.parseOptionalKeyword(kClusterSize))) { if (succeeded(parser.parseOptionalKeyword(kClusterSize))) {
clusterSizeInfo = OpAsmParser::UnresolvedOperand(); clusterSizeInfo = OpAsmParser::UnresolvedOperand();
if (parser.parseLParen() || parser.parseOperand(*clusterSizeInfo) || if (parser.parseLParen() || parser.parseOperand(*clusterSizeInfo) ||
parser.parseRParen()) parser.parseRParen())
return failure(); return failure();
} }
Type resultType; Type resultType;
▲ Show 20 Lines • Show All 2,419 Lines • ▼ Show 20 LinesLogicalResult spirv::MergeOp::verify() {
return success(); return success();
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spirv.module // spirv.module
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void spirv::ModuleOp::build(OpBuilder &builder, OperationState &state, void spirv::ModuleOp::build(OpBuilder &builder, OperationState &state,
Optional<StringRef> name) { std::optional<StringRef> name) {
OpBuilder::InsertionGuard guard(builder); OpBuilder::InsertionGuard guard(builder);
builder.createBlock(state.addRegion()); builder.createBlock(state.addRegion());
if (name) { if (name) {
state.attributes.append(mlir::SymbolTable::getSymbolAttrName(), state.attributes.append(mlir::SymbolTable::getSymbolAttrName(),
builder.getStringAttr(*name)); builder.getStringAttr(*name));
} }
} }
void spirv::ModuleOp::build(OpBuilder &builder, OperationState &state, void spirv::ModuleOp::build(OpBuilder &builder, OperationState &state,
spirv::AddressingModel addressingModel, spirv::AddressingModel addressingModel,
spirv::MemoryModel memoryModel, spirv::MemoryModel memoryModel,
Optional<VerCapExtAttr> vceTriple, std::optional<VerCapExtAttr> vceTriple,
Optional<StringRef> name) { std::optional<StringRef> name) {
state.addAttribute( state.addAttribute(
"addressing_model", "addressing_model",
builder.getAttr<spirv::AddressingModelAttr>(addressingModel)); builder.getAttr<spirv::AddressingModelAttr>(addressingModel));
state.addAttribute("memory_model", state.addAttribute("memory_model",
builder.getAttr<spirv::MemoryModelAttr>(memoryModel)); builder.getAttr<spirv::MemoryModelAttr>(memoryModel));
OpBuilder::InsertionGuard guard(builder); OpBuilder::InsertionGuard guard(builder);
builder.createBlock(state.addRegion()); builder.createBlock(state.addRegion());
if (vceTriple) if (vceTriple)
Show All 36 LinesParseResult spirv::ModuleOp::parse(OpAsmParser &parser,
// Make sure we have at least one block. // Make sure we have at least one block.
if (body->empty()) if (body->empty())
body->push_back(new Block()); body->push_back(new Block());
return success(); return success();
} }
void spirv::ModuleOp::print(OpAsmPrinter &printer) { void spirv::ModuleOp::print(OpAsmPrinter &printer) {
if (Optional<StringRef> name = getName()) { if (std::optional<StringRef> name = getName()) {
printer << ' '; printer << ' ';
printer.printSymbolName(*name); printer.printSymbolName(*name);
} }
SmallVector<StringRef, 2> elidedAttrs; SmallVector<StringRef, 2> elidedAttrs;
printer << " " << spirv::stringifyAddressingModel(getAddressingModel()) << " " printer << " " << spirv::stringifyAddressingModel(getAddressingModel()) << " "
<< spirv::stringifyMemoryModel(getMemoryModel()); << spirv::stringifyMemoryModel(getMemoryModel());
auto addressingModelAttrName = spirv::attributeName<spirv::AddressingModel>(); auto addressingModelAttrName = spirv::attributeName<spirv::AddressingModel>();
auto memoryModelAttrName = spirv::attributeName<spirv::MemoryModel>(); auto memoryModelAttrName = spirv::attributeName<spirv::MemoryModel>();
elidedAttrs.assign({addressingModelAttrName, memoryModelAttrName, elidedAttrs.assign({addressingModelAttrName, memoryModelAttrName,
mlir::SymbolTable::getSymbolAttrName()}); mlir::SymbolTable::getSymbolAttrName()});
if (Optional<spirv::VerCapExtAttr> triple = getVceTriple()) { if (std::optional<spirv::VerCapExtAttr> triple = getVceTriple()) {
printer << " requires " << *triple; printer << " requires " << *triple;
elidedAttrs.push_back(spirv::ModuleOp::getVCETripleAttrName()); elidedAttrs.push_back(spirv::ModuleOp::getVCETripleAttrName());
} }
printer.printOptionalAttrDictWithKeyword((*this)->getAttrs(), elidedAttrs); printer.printOptionalAttrDictWithKeyword((*this)->getAttrs(), elidedAttrs);
printer << ' '; printer << ' ';
printer.printRegion(getRegion()); printer.printRegion(getRegion());
} }
▲ Show 20 Lines • Show All 361 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// spirv.Variable // spirv.Variable
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
ParseResult spirv::VariableOp::parse(OpAsmParser &parser, ParseResult spirv::VariableOp::parse(OpAsmParser &parser,
OperationState &result) { OperationState &result) {
// Parse optional initializer // Parse optional initializer
Optional<OpAsmParser::UnresolvedOperand> initInfo; std::optional<OpAsmParser::UnresolvedOperand> initInfo;
if (succeeded(parser.parseOptionalKeyword("init"))) { if (succeeded(parser.parseOptionalKeyword("init"))) {
initInfo = OpAsmParser::UnresolvedOperand(); initInfo = OpAsmParser::UnresolvedOperand();
if (parser.parseLParen() || parser.parseOperand(*initInfo) || if (parser.parseLParen() || parser.parseOperand(*initInfo) ||
parser.parseRParen()) parser.parseRParen())
return failure(); return failure();
} }
if (parseVariableDecorations(parser, result)) { if (parseVariableDecorations(parser, result)) {
▲ Show 20 Lines • Show All 1,125 Lines • Show Last 20 Lines

mlir/lib/Dialect/SPIRV/Transforms/LowerABIAttributesPass.cpp

Show First 20 Lines • Show All 142 Lines • ▼ Show 20 Lines if (failed(executionModel))
return funcOp.emitRemark("lower entry point failure: could not select " return funcOp.emitRemark("lower entry point failure: could not select "
"execution model based on 'spirv.target_env'"); "execution model based on 'spirv.target_env'");
builder.create<spirv::EntryPointOp>(funcOp.getLoc(), executionModel.value(), builder.create<spirv::EntryPointOp>(funcOp.getLoc(), executionModel.value(),
funcOp, interfaceVars); funcOp, interfaceVars);
// Specifies the spirv.ExecutionModeOp. // Specifies the spirv.ExecutionModeOp.
if (DenseI32ArrayAttr workgroupSizeAttr = entryPointAttr.getWorkgroupSize()) { if (DenseI32ArrayAttr workgroupSizeAttr = entryPointAttr.getWorkgroupSize()) {
Optional<ArrayRef<spirv::Capability>> caps = std::optional<ArrayRef<spirv::Capability>> caps =
spirv::getCapabilities(spirv::ExecutionMode::LocalSize); spirv::getCapabilities(spirv::ExecutionMode::LocalSize);
if (!caps || targetEnv.allows(*caps)) { if (!caps || targetEnv.allows(*caps)) {
builder.create<spirv::ExecutionModeOp>(funcOp.getLoc(), funcOp, builder.create<spirv::ExecutionModeOp>(funcOp.getLoc(), funcOp,
spirv::ExecutionMode::LocalSize, spirv::ExecutionMode::LocalSize,
workgroupSizeAttr.asArrayRef()); workgroupSizeAttr.asArrayRef());
// Erase workgroup size. // Erase workgroup size.
entryPointAttr = spirv::EntryPointABIAttr::get( entryPointAttr = spirv::EntryPointABIAttr::get(
entryPointAttr.getContext(), DenseI32ArrayAttr(), entryPointAttr.getContext(), DenseI32ArrayAttr(),
entryPointAttr.getSubgroupSize()); entryPointAttr.getSubgroupSize());
} }
} }
if (Optional<int> subgroupSize = entryPointAttr.getSubgroupSize()) { if (Optional<int> subgroupSize = entryPointAttr.getSubgroupSize()) {
Optional<ArrayRef<spirv::Capability>> caps = std::optional<ArrayRef<spirv::Capability>> caps =
spirv::getCapabilities(spirv::ExecutionMode::SubgroupSize); spirv::getCapabilities(spirv::ExecutionMode::SubgroupSize);
if (!caps || targetEnv.allows(*caps)) { if (!caps || targetEnv.allows(*caps)) {
builder.create<spirv::ExecutionModeOp>(funcOp.getLoc(), funcOp, builder.create<spirv::ExecutionModeOp>(funcOp.getLoc(), funcOp,
spirv::ExecutionMode::SubgroupSize, spirv::ExecutionMode::SubgroupSize,
*subgroupSize); *subgroupSize);
// Erase subgroup size. // Erase subgroup size.
entryPointAttr = spirv::EntryPointABIAttr::get( entryPointAttr = spirv::EntryPointABIAttr::get(
entryPointAttr.getContext(), entryPointAttr.getWorkgroupSize(), entryPointAttr.getContext(), entryPointAttr.getWorkgroupSize(),
▲ Show 20 Lines • Show All 157 Lines • Show Last 20 Lines

mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Lines
using AliasedResourceMap = using AliasedResourceMap =
DenseMap<Descriptor, SmallVector<spirv::GlobalVariableOp>>; DenseMap<Descriptor, SmallVector<spirv::GlobalVariableOp>>;
/// Collects all aliased resources in the given SPIR-V `moduleOp`. /// Collects all aliased resources in the given SPIR-V `moduleOp`.
static AliasedResourceMap collectAliasedResources(spirv::ModuleOp moduleOp) { static AliasedResourceMap collectAliasedResources(spirv::ModuleOp moduleOp) {
AliasedResourceMap aliasedResources; AliasedResourceMap aliasedResources;
moduleOp->walk([&aliasedResources](spirv::GlobalVariableOp varOp) { moduleOp->walk([&aliasedResources](spirv::GlobalVariableOp varOp) {
if (varOp->getAttrOfType<UnitAttr>("aliased")) { if (varOp->getAttrOfType<UnitAttr>("aliased")) {
Optional<uint32_t> set = varOp.getDescriptorSet(); std::optional<uint32_t> set = varOp.getDescriptorSet();
Optional<uint32_t> binding = varOp.getBinding(); std::optional<uint32_t> binding = varOp.getBinding();
if (set && binding) if (set && binding)
aliasedResources[{*set, *binding}].push_back(varOp); aliasedResources[{*set, *binding}].push_back(varOp);
} }
}); });
return aliasedResources; return aliasedResources;
} }
/// Returns the element type if the given `type` is a runtime array resource: /// Returns the element type if the given `type` is a runtime array resource:
▲ Show 20 Lines • Show All 534 Lines • Show Last 20 Lines

mlir/lib/Dialect/Shape/IR/Shape.cpp

Show First 20 Lines • Show All 329 Lines • ▼ Show 20 Lines
void AssumingOp::getCanonicalizationPatterns(RewritePatternSet &patterns, void AssumingOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
patterns.add<AssumingOpRemoveUnusedResults, AssumingWithTrue>(context); patterns.add<AssumingOpRemoveUnusedResults, AssumingWithTrue>(context);
} }
// See RegionBranchOpInterface in Interfaces/ControlFlowInterfaces.td // See RegionBranchOpInterface in Interfaces/ControlFlowInterfaces.td
void AssumingOp::getSuccessorRegions( void AssumingOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// AssumingOp has unconditional control flow into the region and back to the // AssumingOp has unconditional control flow into the region and back to the
// parent, so return the correct RegionSuccessor purely based on the index // parent, so return the correct RegionSuccessor purely based on the index
// being None or 0. // being None or 0.
if (index) { if (index) {
regions.push_back(RegionSuccessor(getResults())); regions.push_back(RegionSuccessor(getResults()));
return; return;
} }
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Linesvoid AssumingOp::build(
result.addTypes(assumingTypes); result.addTypes(assumingTypes);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AddOp // AddOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
LogicalResult mlir::shape::AddOp::inferReturnTypes( LogicalResult mlir::shape::AddOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType().isa<SizeType>() || if (operands[0].getType().isa<SizeType>() ||
operands[1].getType().isa<SizeType>()) operands[1].getType().isa<SizeType>())
inferredReturnTypes.assign({SizeType::get(context)}); inferredReturnTypes.assign({SizeType::get(context)});
else else
inferredReturnTypes.assign({IndexType::get(context)}); inferredReturnTypes.assign({IndexType::get(context)});
return success(); return success();
▲ Show 20 Lines • Show All 500 Lines • ▼ Show 20 Lines
OpFoldResult ConstShapeOp::fold(ArrayRef<Attribute>) { return getShapeAttr(); } OpFoldResult ConstShapeOp::fold(ArrayRef<Attribute>) { return getShapeAttr(); }
void ConstShapeOp::getCanonicalizationPatterns(RewritePatternSet &patterns, void ConstShapeOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
patterns.add<TensorCastConstShape>(context); patterns.add<TensorCastConstShape>(context);
} }
LogicalResult mlir::shape::ConstShapeOp::inferReturnTypes( LogicalResult mlir::shape::ConstShapeOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
Builder b(context); Builder b(context);
auto shape = attributes.getAs<DenseIntElementsAttr>("shape"); auto shape = attributes.getAs<DenseIntElementsAttr>("shape");
if (!shape) if (!shape)
return emitOptionalError(location, "missing shape attribute"); return emitOptionalError(location, "missing shape attribute");
inferredReturnTypes.assign({RankedTensorType::get( inferredReturnTypes.assign({RankedTensorType::get(
{static_cast<int64_t>(shape.size())}, b.getIndexType())}); {static_cast<int64_t>(shape.size())}, b.getIndexType())});
▲ Show 20 Lines • Show All 140 Lines • ▼ Show 20 Lines
OpFoldResult CstrRequireOp::fold(ArrayRef<Attribute> operands) { OpFoldResult CstrRequireOp::fold(ArrayRef<Attribute> operands) {
return operands[0]; return operands[0];
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// DimOp // DimOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Optional<int64_t> DimOp::getConstantIndex() { std::optional<int64_t> DimOp::getConstantIndex() {
if (auto constSizeOp = getIndex().getDefiningOp<ConstSizeOp>()) if (auto constSizeOp = getIndex().getDefiningOp<ConstSizeOp>())
return constSizeOp.getValue().getLimitedValue(); return constSizeOp.getValue().getLimitedValue();
if (auto constantOp = getIndex().getDefiningOp<arith::ConstantOp>()) if (auto constantOp = getIndex().getDefiningOp<arith::ConstantOp>())
return constantOp.getValue().cast<IntegerAttr>().getInt(); return constantOp.getValue().cast<IntegerAttr>().getInt();
return std::nullopt; return std::nullopt;
} }
OpFoldResult DimOp::fold(ArrayRef<Attribute> operands) { OpFoldResult DimOp::fold(ArrayRef<Attribute> operands) {
Type valType = getValue().getType(); Type valType = getValue().getType();
auto valShapedType = valType.dyn_cast<ShapedType>(); auto valShapedType = valType.dyn_cast<ShapedType>();
if (!valShapedType || !valShapedType.hasRank()) if (!valShapedType || !valShapedType.hasRank())
return nullptr; return nullptr;
Optional<int64_t> index = getConstantIndex(); std::optional<int64_t> index = getConstantIndex();
if (!index.has_value()) if (!index.has_value())
return nullptr; return nullptr;
if (index.value() >= valShapedType.getRank()) if (index.value() >= valShapedType.getRank())
return nullptr; return nullptr;
auto extent = valShapedType.getDimSize(*index); auto extent = valShapedType.getDimSize(*index);
if (ShapedType::isDynamic(extent)) if (ShapedType::isDynamic(extent))
return nullptr; return nullptr;
return IntegerAttr::get(IndexType::get(getContext()), extent); return IntegerAttr::get(IndexType::get(getContext()), extent);
} }
LogicalResult mlir::shape::DimOp::inferReturnTypes( LogicalResult mlir::shape::DimOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
DimOpAdaptor dimOp(operands); DimOpAdaptor dimOp(operands);
inferredReturnTypes.assign({dimOp.getIndex().getType()}); inferredReturnTypes.assign({dimOp.getIndex().getType()});
return success(); return success();
} }
bool mlir::shape::DimOp::isCompatibleReturnTypes(TypeRange l, TypeRange r) { bool mlir::shape::DimOp::isCompatibleReturnTypes(TypeRange l, TypeRange r) {
Show All 31 Lines if (quotient.isNegative() && !remainder.isNullValue()) {
quotient -= 1; quotient -= 1;
} }
Type indexTy = IndexType::get(getContext()); Type indexTy = IndexType::get(getContext());
return IntegerAttr::get(indexTy, quotient); return IntegerAttr::get(indexTy, quotient);
} }
LogicalResult mlir::shape::DivOp::inferReturnTypes( LogicalResult mlir::shape::DivOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType().isa<SizeType>() || if (operands[0].getType().isa<SizeType>() ||
operands[1].getType().isa<SizeType>()) operands[1].getType().isa<SizeType>())
inferredReturnTypes.assign({SizeType::get(context)}); inferredReturnTypes.assign({SizeType::get(context)});
else else
inferredReturnTypes.assign({IndexType::get(context)}); inferredReturnTypes.assign({IndexType::get(context)});
return success(); return success();
▲ Show 20 Lines • Show All 169 Lines • ▼ Show 20 Lines function_interface_impl::printFunctionOp(
p, *this, /*isVariadic=*/false, getFunctionTypeAttrName(), p, *this, /*isVariadic=*/false, getFunctionTypeAttrName(),
getArgAttrsAttrName(), getResAttrsAttrName()); getArgAttrsAttrName(), getResAttrsAttrName());
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// GetExtentOp // GetExtentOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Optional<int64_t> GetExtentOp::getConstantDim() { std::optional<int64_t> GetExtentOp::getConstantDim() {
if (auto constSizeOp = getDim().getDefiningOp<ConstSizeOp>()) if (auto constSizeOp = getDim().getDefiningOp<ConstSizeOp>())
return constSizeOp.getValue().getLimitedValue(); return constSizeOp.getValue().getLimitedValue();
if (auto constantOp = getDim().getDefiningOp<arith::ConstantOp>()) if (auto constantOp = getDim().getDefiningOp<arith::ConstantOp>())
return constantOp.getValue().cast<IntegerAttr>().getInt(); return constantOp.getValue().cast<IntegerAttr>().getInt();
return std::nullopt; return std::nullopt;
} }
OpFoldResult GetExtentOp::fold(ArrayRef<Attribute> operands) { OpFoldResult GetExtentOp::fold(ArrayRef<Attribute> operands) {
auto elements = operands[0].dyn_cast_or_null<DenseIntElementsAttr>(); auto elements = operands[0].dyn_cast_or_null<DenseIntElementsAttr>();
if (!elements) if (!elements)
return nullptr; return nullptr;
Optional<int64_t> dim = getConstantDim(); std::optional<int64_t> dim = getConstantDim();
jpienaarUnsubmitted
Done
ReplyInline Actions

Why couldn't this be std::optional<int64_t> ?

jpienaar: Why couldn't this be std::optional<int64_t> ?
artagnonAuthorUnsubmitted
Done
ReplyInline Actions

It could very well be. I suppose it's a matter of style: personally, I prefer auto to make the code less brittle to changes.

artagnon: It could very well be. I suppose it's a matter of style: personally, I prefer `auto` to make…
clementvalUnsubmitted
Done
ReplyInline Actions

std::optional<int64_t> seems better here (https://llvm.org/docs/CodingStandards.html#use-auto-type-deduction-to-make-code-more-readable)

clementval: `std::optional<int64_t>` seems better here (https://llvm.org/docs/CodingStandards.html#use-auto…
artagnonAuthorUnsubmitted
Done
ReplyInline Actions

Fixed now.

artagnon: Fixed now.
if (!dim.has_value()) if (!dim.has_value())
return nullptr; return nullptr;
if (dim.value() >= elements.getNumElements()) if (dim.value() >= elements.getNumElements())
return nullptr; return nullptr;
return elements.getValues<Attribute>()[(uint64_t)dim.value()]; return elements.getValues<Attribute>()[(uint64_t)dim.value()];
} }
void GetExtentOp::build(OpBuilder &builder, OperationState &result, Value shape, void GetExtentOp::build(OpBuilder &builder, OperationState &result, Value shape,
int64_t dim) { int64_t dim) {
auto loc = result.location; auto loc = result.location;
auto dimAttr = builder.getIndexAttr(dim); auto dimAttr = builder.getIndexAttr(dim);
if (shape.getType().isa<ShapeType>()) { if (shape.getType().isa<ShapeType>()) {
Value dim = builder.create<ConstSizeOp>(loc, dimAttr); Value dim = builder.create<ConstSizeOp>(loc, dimAttr);
build(builder, result, builder.getType<SizeType>(), shape, dim); build(builder, result, builder.getType<SizeType>(), shape, dim);
} else { } else {
Value dim = Value dim =
builder.create<arith::ConstantOp>(loc, builder.getIndexType(), dimAttr); builder.create<arith::ConstantOp>(loc, builder.getIndexType(), dimAttr);
build(builder, result, builder.getIndexType(), shape, dim); build(builder, result, builder.getIndexType(), shape, dim);
} }
} }
LogicalResult mlir::shape::GetExtentOp::inferReturnTypes( LogicalResult mlir::shape::GetExtentOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
inferredReturnTypes.assign({IndexType::get(context)}); inferredReturnTypes.assign({IndexType::get(context)});
return success(); return success();
} }
bool mlir::shape::GetExtentOp::isCompatibleReturnTypes(TypeRange l, bool mlir::shape::GetExtentOp::isCompatibleReturnTypes(TypeRange l,
TypeRange r) { TypeRange r) {
Show All 21 LinesOpFoldResult IsBroadcastableOp::fold(ArrayRef<Attribute> operands) {
return nullptr; return nullptr;
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// MeetOp // MeetOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
LogicalResult mlir::shape::MeetOp::inferReturnTypes( LogicalResult mlir::shape::MeetOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands.empty()) if (operands.empty())
return failure(); return failure();
auto isShapeType = [](Type arg) { auto isShapeType = [](Type arg) {
if (arg.isa<ShapeType>()) if (arg.isa<ShapeType>())
return true; return true;
▲ Show 20 Lines • Show All 119 Lines • ▼ Show 20 Lines
} // namespace } // namespace
void shape::RankOp::getCanonicalizationPatterns(RewritePatternSet &patterns, void shape::RankOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
patterns.add<RankShapeOfCanonicalizationPattern>(context); patterns.add<RankShapeOfCanonicalizationPattern>(context);
} }
LogicalResult mlir::shape::RankOp::inferReturnTypes( LogicalResult mlir::shape::RankOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType().isa<ShapeType>()) if (operands[0].getType().isa<ShapeType>())
inferredReturnTypes.assign({SizeType::get(context)}); inferredReturnTypes.assign({SizeType::get(context)});
else else
inferredReturnTypes.assign({IndexType::get(context)}); inferredReturnTypes.assign({IndexType::get(context)});
return success(); return success();
} }
Show All 19 LinesOpFoldResult NumElementsOp::fold(ArrayRef<Attribute> operands) {
APInt product(64, 1); APInt product(64, 1);
for (auto value : shape.cast<DenseIntElementsAttr>()) for (auto value : shape.cast<DenseIntElementsAttr>())
product *= value; product *= value;
Builder builder(getContext()); Builder builder(getContext());
return builder.getIndexAttr(product.getLimitedValue()); return builder.getIndexAttr(product.getLimitedValue());
} }
LogicalResult mlir::shape::NumElementsOp::inferReturnTypes( LogicalResult mlir::shape::NumElementsOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType().isa<ShapeType>()) if (operands[0].getType().isa<ShapeType>())
inferredReturnTypes.assign({SizeType::get(context)}); inferredReturnTypes.assign({SizeType::get(context)});
else else
inferredReturnTypes.assign({IndexType::get(context)}); inferredReturnTypes.assign({IndexType::get(context)});
return success(); return success();
} }
Show All 15 Lines
OpFoldResult MaxOp::fold(llvm::ArrayRef<mlir::Attribute> operands) { OpFoldResult MaxOp::fold(llvm::ArrayRef<mlir::Attribute> operands) {
// If operands are equal, just propagate one. // If operands are equal, just propagate one.
if (getLhs() == getRhs()) if (getLhs() == getRhs())
return getLhs(); return getLhs();
return nullptr; return nullptr;
} }
LogicalResult mlir::shape::MaxOp::inferReturnTypes( LogicalResult mlir::shape::MaxOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType() == operands[1].getType()) if (operands[0].getType() == operands[1].getType())
inferredReturnTypes.assign({operands[0].getType()}); inferredReturnTypes.assign({operands[0].getType()});
else else
inferredReturnTypes.assign({SizeType::get(context)}); inferredReturnTypes.assign({SizeType::get(context)});
return success(); return success();
} }
Show All 15 Lines
OpFoldResult MinOp::fold(llvm::ArrayRef<mlir::Attribute> operands) { OpFoldResult MinOp::fold(llvm::ArrayRef<mlir::Attribute> operands) {
// If operands are equal, just propagate one. // If operands are equal, just propagate one.
if (getLhs() == getRhs()) if (getLhs() == getRhs())
return getLhs(); return getLhs();
return nullptr; return nullptr;
} }
LogicalResult mlir::shape::MinOp::inferReturnTypes( LogicalResult mlir::shape::MinOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType() == operands[1].getType()) if (operands[0].getType() == operands[1].getType())
inferredReturnTypes.assign({operands[0].getType()}); inferredReturnTypes.assign({operands[0].getType()});
else else
inferredReturnTypes.assign({SizeType::get(context)}); inferredReturnTypes.assign({SizeType::get(context)});
return success(); return success();
} }
Show All 20 LinesOpFoldResult MulOp::fold(ArrayRef<Attribute> operands) {
if (!rhs) if (!rhs)
return nullptr; return nullptr;
APInt folded = lhs.getValue() * rhs.getValue(); APInt folded = lhs.getValue() * rhs.getValue();
Type indexTy = IndexType::get(getContext()); Type indexTy = IndexType::get(getContext());
return IntegerAttr::get(indexTy, folded); return IntegerAttr::get(indexTy, folded);
} }
LogicalResult mlir::shape::MulOp::inferReturnTypes( LogicalResult mlir::shape::MulOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType().isa<SizeType>() || if (operands[0].getType().isa<SizeType>() ||
operands[1].getType().isa<SizeType>()) operands[1].getType().isa<SizeType>())
inferredReturnTypes.assign({SizeType::get(context)}); inferredReturnTypes.assign({SizeType::get(context)});
else else
inferredReturnTypes.assign({IndexType::get(context)}); inferredReturnTypes.assign({IndexType::get(context)});
return success(); return success();
▲ Show 20 Lines • Show All 70 Lines • ▼ Show 20 Lines
void ShapeOfOp::getCanonicalizationPatterns(RewritePatternSet &patterns, void ShapeOfOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
MLIRContext *context) { MLIRContext *context) {
patterns.add<ShapeOfCastExtentTensor, ShapeOfWithTensor, patterns.add<ShapeOfCastExtentTensor, ShapeOfWithTensor,
ExtractFromShapeOfExtentTensor>(context); ExtractFromShapeOfExtentTensor>(context);
} }
LogicalResult mlir::shape::ShapeOfOp::inferReturnTypes( LogicalResult mlir::shape::ShapeOfOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType().isa<ValueShapeType>()) if (operands[0].getType().isa<ValueShapeType>())
inferredReturnTypes.assign({ShapeType::get(context)}); inferredReturnTypes.assign({ShapeType::get(context)});
else { else {
auto shapedTy = operands[0].getType().cast<ShapedType>(); auto shapedTy = operands[0].getType().cast<ShapedType>();
int64_t rank = int64_t rank =
shapedTy.hasRank() ? shapedTy.getRank() : ShapedType::kDynamic; shapedTy.hasRank() ? shapedTy.getRank() : ShapedType::kDynamic;
▲ Show 20 Lines • Show All 239 Lines • Show Last 20 Lines

mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp

Show First 20 Lines • Show All 327 Lines • ▼ Show 20 Lines unsigned idxBitWidth =
getEncoding().getIndexBitWidth() ? getEncoding().getIndexBitWidth() : 64u; getEncoding().getIndexBitWidth() ? getEncoding().getIndexBitWidth() : 64u;
unsigned ptrBitWidth = unsigned ptrBitWidth =
getEncoding().getIndexBitWidth() ? getEncoding().getIndexBitWidth() : 64u; getEncoding().getIndexBitWidth() ? getEncoding().getIndexBitWidth() : 64u;
return IntegerType::get(getContext(), std::max(idxBitWidth, ptrBitWidth)); return IntegerType::get(getContext(), std::max(idxBitWidth, ptrBitWidth));
} }
Type StorageSpecifierType::getFieldType(StorageSpecifierKind kind, Type StorageSpecifierType::getFieldType(StorageSpecifierKind kind,
Optional<unsigned> dim) const { std::optional<unsigned> dim) const {
if (kind != StorageSpecifierKind::ValMemSize) if (kind != StorageSpecifierKind::ValMemSize)
assert(dim); assert(dim);
// Right now, we store every sizes metadata using the same size type. // Right now, we store every sizes metadata using the same size type.
// TODO: the field size type can be defined dimensional wise after sparse // TODO: the field size type can be defined dimensional wise after sparse
// tensor encoding supports per dimension index/pointer bitwidth. // tensor encoding supports per dimension index/pointer bitwidth.
return getSizesType(); return getSizesType();
} }
Type StorageSpecifierType::getFieldType(StorageSpecifierKind kind, Type StorageSpecifierType::getFieldType(StorageSpecifierKind kind,
Optional<APInt> dim) const { std::optional<APInt> dim) const {
Optional<unsigned> intDim = std::nullopt; std::optional<unsigned> intDim = std::nullopt;
if (dim) if (dim)
intDim = dim.value().getZExtValue(); intDim = dim.value().getZExtValue();
return getFieldType(kind, intDim); return getFieldType(kind, intDim);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SparseTensorDialect Operations. // SparseTensorDialect Operations.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
static LogicalResult isInBounds(uint64_t dim, Value tensor) { static LogicalResult isInBounds(uint64_t dim, Value tensor) {
uint64_t rank = tensor.getType().cast<RankedTensorType>().getRank(); uint64_t rank = tensor.getType().cast<RankedTensorType>().getRank();
if (dim >= rank) if (dim >= rank)
return failure(); return failure();
return success(); // in bounds return success(); // in bounds
} }
static LogicalResult isMatchingWidth(Value result, unsigned width) { static LogicalResult isMatchingWidth(Value result, unsigned width) {
Type etp = result.getType().cast<MemRefType>().getElementType(); Type etp = result.getType().cast<MemRefType>().getElementType();
if ((width == 0 && etp.isIndex()) || (width > 0 && etp.isInteger(width))) if ((width == 0 && etp.isIndex()) || (width > 0 && etp.isInteger(width)))
return success(); return success();
return failure(); return failure();
} }
static LogicalResult static LogicalResult verifySparsifierGetterSetter(
verifySparsifierGetterSetter(StorageSpecifierKind mdKind, Optional<APInt> dim, StorageSpecifierKind mdKind, std::optional<APInt> dim,
TypedValue<StorageSpecifierType> md, TypedValue<StorageSpecifierType> md, Operation *op) {
Operation *op) {
if (mdKind == StorageSpecifierKind::ValMemSize && dim) { if (mdKind == StorageSpecifierKind::ValMemSize && dim) {
return op->emitError( return op->emitError(
"redundant dimension argument for querying value memory size"); "redundant dimension argument for querying value memory size");
} }
auto enc = md.getType().getEncoding(); auto enc = md.getType().getEncoding();
ArrayRef<DimLevelType> dlts = enc.getDimLevelType(); ArrayRef<DimLevelType> dlts = enc.getDimLevelType();
unsigned rank = dlts.size(); unsigned rank = dlts.size();
▲ Show 20 Lines • Show All 93 Lines • ▼ Show 20 Lines
template <typename SpecifierOp> template <typename SpecifierOp>
static SetStorageSpecifierOp getSpecifierSetDef(SpecifierOp op) { static SetStorageSpecifierOp getSpecifierSetDef(SpecifierOp op) {
return op.getSpecifier().template getDefiningOp<SetStorageSpecifierOp>(); return op.getSpecifier().template getDefiningOp<SetStorageSpecifierOp>();
} }
OpFoldResult GetStorageSpecifierOp::fold(ArrayRef<Attribute> operands) { OpFoldResult GetStorageSpecifierOp::fold(ArrayRef<Attribute> operands) {
StorageSpecifierKind kind = getSpecifierKind(); StorageSpecifierKind kind = getSpecifierKind();
Optional<APInt> dim = getDim(); std::optional<APInt> dim = getDim();
for (auto op = getSpecifierSetDef(*this); op; op = getSpecifierSetDef(op)) for (auto op = getSpecifierSetDef(*this); op; op = getSpecifierSetDef(op))
if (kind == op.getSpecifierKind() && dim == op.getDim()) if (kind == op.getSpecifierKind() && dim == op.getDim())
return op.getValue(); return op.getValue();
return {}; return {};
} }
LogicalResult SetStorageSpecifierOp::verify() { LogicalResult SetStorageSpecifierOp::verify() {
if (failed(verifySparsifierGetterSetter(getSpecifierKind(), getDim(), if (failed(verifySparsifierGetterSetter(getSpecifierKind(), getDim(),
▲ Show 20 Lines • Show All 416 Lines • Show Last 20 Lines

mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorCodegen.cpp

Show First 20 Lines • Show All 127 Lines • ▼ Show 20 Lines for (unsigned i = 0, e = fields.size(); i < e; i++)
fields[i] = forOp.getRegionIterArg(i); fields[i] = forOp.getRegionIterArg(i);
builder.setInsertionPointToStart(forOp.getBody()); builder.setInsertionPointToStart(forOp.getBody());
return forOp; return forOp;
} }
/// Gets the dimension size for the given sparse tensor at the given /// Gets the dimension size for the given sparse tensor at the given
/// original dimension 'dim'. Returns std::nullopt if no sparse encoding is /// original dimension 'dim'. Returns std::nullopt if no sparse encoding is
/// attached to the given tensor type. /// attached to the given tensor type.
static Optional<Value> sizeFromTensorAtDim(OpBuilder &builder, Location loc, static std::optional<Value> sizeFromTensorAtDim(OpBuilder &builder,
Location loc,
SparseTensorDescriptor desc, SparseTensorDescriptor desc,
unsigned dim) { unsigned dim) {
RankedTensorType rtp = desc.getTensorType(); RankedTensorType rtp = desc.getTensorType();
// Access into static dimension can query original type directly. // Access into static dimension can query original type directly.
// Note that this is typically already done by DimOp's folding. // Note that this is typically already done by DimOp's folding.
auto shape = rtp.getShape(); auto shape = rtp.getShape();
if (!ShapedType::isDynamic(shape[dim])) if (!ShapedType::isDynamic(shape[dim]))
return constantIndex(builder, loc, shape[dim]); return constantIndex(builder, loc, shape[dim]);
// Any other query can consult the dimSizes array at field DimSizesIdx, // Any other query can consult the dimSizes array at field DimSizesIdx,
▲ Show 20 Lines • Show All 529 Lines • ▼ Show 20 Lines
/// Sparse codegen rule for dimension accesses. /// Sparse codegen rule for dimension accesses.
class SparseDimOpConverter : public OpConversionPattern<tensor::DimOp> { class SparseDimOpConverter : public OpConversionPattern<tensor::DimOp> {
public: public:
using OpConversionPattern::OpConversionPattern; using OpConversionPattern::OpConversionPattern;
LogicalResult LogicalResult
matchAndRewrite(tensor::DimOp op, OpAdaptor adaptor, matchAndRewrite(tensor::DimOp op, OpAdaptor adaptor,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
Optional<int64_t> index = op.getConstantIndex(); std::optional<int64_t> index = op.getConstantIndex();
if (!index || !getSparseTensorEncoding(adaptor.getSource().getType())) if (!index || !getSparseTensorEncoding(adaptor.getSource().getType()))
return failure(); return failure();
auto desc = getDescriptorFromTensorTuple(adaptor.getSource()); auto desc = getDescriptorFromTensorTuple(adaptor.getSource());
auto sz = sizeFromTensorAtDim(rewriter, op.getLoc(), desc, *index); auto sz = sizeFromTensorAtDim(rewriter, op.getLoc(), desc, *index);
if (!sz) if (!sz)
return failure(); return failure();
▲ Show 20 Lines • Show All 377 Lines • Show Last 20 Lines

mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp

Show First 20 Lines • Show All 700 Lines • ▼ Show 20 Linespublic:
matchAndRewrite(tensor::DimOp op, OpAdaptor adaptor, matchAndRewrite(tensor::DimOp op, OpAdaptor adaptor,
ConversionPatternRewriter &rewriter) const override { ConversionPatternRewriter &rewriter) const override {
auto stp = op.getSource().getType().cast<ShapedType>(); auto stp = op.getSource().getType().cast<ShapedType>();
// Only rewrite sparse DimOp. // Only rewrite sparse DimOp.
auto enc = getSparseTensorEncoding(stp); auto enc = getSparseTensorEncoding(stp);
if (!enc) if (!enc)
return failure(); return failure();
// Only rewrite DimOp with constant index. // Only rewrite DimOp with constant index.
Optional<int64_t> dim = op.getConstantIndex(); std::optional<int64_t> dim = op.getConstantIndex();
if (!dim) if (!dim)
return failure(); return failure();
// Generate the call. // Generate the call.
Value src = adaptor.getOperands()[0]; Value src = adaptor.getOperands()[0];
rewriter.replaceOp( rewriter.replaceOp(
op, createOrFoldDimCall(rewriter, op->getLoc(), enc, stp, src, *dim)); op, createOrFoldDimCall(rewriter, op->getLoc(), enc, stp, src, *dim));
return success(); return success();
} }
▲ Show 20 Lines • Show All 822 Lines • Show Last 20 Lines

mlir/lib/Dialect/Tensor/IR/TensorOps.cpp

Show First 20 Lines • Show All 374 Lines • ▼ Show 20 Lines
void DimOp::build(OpBuilder &builder, OperationState &result, Value source, void DimOp::build(OpBuilder &builder, OperationState &result, Value source,
int64_t index) { int64_t index) {
auto loc = result.location; auto loc = result.location;
Value indexValue = builder.create<arith::ConstantIndexOp>(loc, index); Value indexValue = builder.create<arith::ConstantIndexOp>(loc, index);
build(builder, result, source, indexValue); build(builder, result, source, indexValue);
} }
Optional<int64_t> DimOp::getConstantIndex() { std::optional<int64_t> DimOp::getConstantIndex() {
return getConstantIntValue(getIndex()); return getConstantIntValue(getIndex());
} }
Speculation::Speculatability DimOp::getSpeculatability() { Speculation::Speculatability DimOp::getSpeculatability() {
auto constantIndex = getConstantIndex(); auto constantIndex = getConstantIndex();
if (!constantIndex) if (!constantIndex)
return Speculation::NotSpeculatable; return Speculation::NotSpeculatable;
auto rankedSourceType = dyn_cast<RankedTensorType>(getSource().getType()); auto rankedSourceType = dyn_cast<RankedTensorType>(getSource().getType());
if (!rankedSourceType) if (!rankedSourceType)
return Speculation::NotSpeculatable; return Speculation::NotSpeculatable;
// The verifier rejects operations that violate this assertion. // The verifier rejects operations that violate this assertion.
assert(constantIndex < rankedSourceType.getRank()); assert(constantIndex < rankedSourceType.getRank());
return Speculation::Speculatable; return Speculation::Speculatable;
} }
LogicalResult DimOp::verify() { LogicalResult DimOp::verify() {
// Assume unknown index to be in range. // Assume unknown index to be in range.
Optional<int64_t> index = getConstantIndex(); std::optional<int64_t> index = getConstantIndex();
if (!index) if (!index)
return success(); return success();
// Check that constant index is not knowingly out of range. // Check that constant index is not knowingly out of range.
auto type = getSource().getType(); auto type = getSource().getType();
if (auto tensorType = type.dyn_cast<RankedTensorType>()) { if (auto tensorType = type.dyn_cast<RankedTensorType>()) {
if (*index >= tensorType.getRank()) if (*index >= tensorType.getRank())
return emitOpError("index is out of range"); return emitOpError("index is out of range");
▲ Show 20 Lines • Show All 181 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(EmptyOp op,
SmallVector<Value> dynamicSizes; SmallVector<Value> dynamicSizes;
// Compute new static and dynamic sizes. // Compute new static and dynamic sizes.
unsigned ctr = 0; unsigned ctr = 0;
bool changedType = false; bool changedType = false;
for (int64_t i = 0; i < op.getType().getRank(); ++i) { for (int64_t i = 0; i < op.getType().getRank(); ++i) {
if (op.getType().isDynamicDim(i)) { if (op.getType().isDynamicDim(i)) {
Value dynamicSize = op.getDynamicSizes()[ctr++]; Value dynamicSize = op.getDynamicSizes()[ctr++];
Optional<int64_t> cst = getConstantIntValue(dynamicSize); std::optional<int64_t> cst = getConstantIntValue(dynamicSize);
if (cst.has_value()) { if (cst.has_value()) {
staticShape[i] = *cst; staticShape[i] = *cst;
changedType = true; changedType = true;
} else { } else {
dynamicSizes.push_back(dynamicSize); dynamicSizes.push_back(dynamicSize);
} }
} }
} }
Show All 11 Linesstruct ReplaceEmptyTensorStaticShapeDims : OpRewritePattern<EmptyOp> {
} }
}; };
struct FoldEmptyTensorWithDimOp : public OpRewritePattern<DimOp> { struct FoldEmptyTensorWithDimOp : public OpRewritePattern<DimOp> {
using OpRewritePattern<DimOp>::OpRewritePattern; using OpRewritePattern<DimOp>::OpRewritePattern;
LogicalResult matchAndRewrite(tensor::DimOp dimOp, LogicalResult matchAndRewrite(tensor::DimOp dimOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
Optional<int64_t> maybeConstantIndex = dimOp.getConstantIndex(); std::optional<int64_t> maybeConstantIndex = dimOp.getConstantIndex();
auto emptyTensorOp = dimOp.getSource().getDefiningOp<EmptyOp>(); auto emptyTensorOp = dimOp.getSource().getDefiningOp<EmptyOp>();
if (!emptyTensorOp || !maybeConstantIndex) if (!emptyTensorOp || !maybeConstantIndex)
return failure(); return failure();
if (!emptyTensorOp.getType().isDynamicDim(*maybeConstantIndex)) if (!emptyTensorOp.getType().isDynamicDim(*maybeConstantIndex))
return failure(); return failure();
rewriter.replaceOp(dimOp, rewriter.replaceOp(dimOp,
emptyTensorOp.getDynamicSize(*maybeConstantIndex)); emptyTensorOp.getDynamicSize(*maybeConstantIndex));
return success(); return success();
▲ Show 20 Lines • Show All 802 Lines • ▼ Show 20 Linesstruct FoldDimOfExpandShape : public OpRewritePattern<DimOp> {
LogicalResult matchAndRewrite(DimOp dimOp, LogicalResult matchAndRewrite(DimOp dimOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
auto expandShapeOp = dimOp.getSource().getDefiningOp<ExpandShapeOp>(); auto expandShapeOp = dimOp.getSource().getDefiningOp<ExpandShapeOp>();
if (!expandShapeOp) if (!expandShapeOp)
return failure(); return failure();
// Only constant dimension values are supported. // Only constant dimension values are supported.
Optional<int64_t> dim = dimOp.getConstantIndex(); std::optional<int64_t> dim = dimOp.getConstantIndex();
if (!dim.has_value()) if (!dim.has_value())
return failure(); return failure();
// Skip static dims. These are folded to constant ops. // Skip static dims. These are folded to constant ops.
TensorType resultType = expandShapeOp.getResultType(); TensorType resultType = expandShapeOp.getResultType();
if (!resultType.isDynamicDim(*dim)) if (!resultType.isDynamicDim(*dim))
return failure(); return failure();
Show All 27 Linesstruct FoldDimOfCollapseShape : public OpRewritePattern<DimOp> {
LogicalResult matchAndRewrite(DimOp dimOp, LogicalResult matchAndRewrite(DimOp dimOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
auto collapseShapeOp = dimOp.getSource().getDefiningOp<CollapseShapeOp>(); auto collapseShapeOp = dimOp.getSource().getDefiningOp<CollapseShapeOp>();
if (!collapseShapeOp) if (!collapseShapeOp)
return failure(); return failure();
// Only constant dimension values are supported. // Only constant dimension values are supported.
Optional<int64_t> dim = dimOp.getConstantIndex(); std::optional<int64_t> dim = dimOp.getConstantIndex();
if (!dim.has_value()) if (!dim.has_value())
return failure(); return failure();
// Skip static dims. These are folded to constant ops. // Skip static dims. These are folded to constant ops.
TensorType resultType = collapseShapeOp.getResultType(); TensorType resultType = collapseShapeOp.getResultType();
if (!resultType.isDynamicDim(*dim)) if (!resultType.isDynamicDim(*dim))
return failure(); return failure();
▲ Show 20 Lines • Show All 226 Lines • ▼ Show 20 Lines
} }
llvm::SmallBitVector ExtractSliceOp::getDroppedDims() { llvm::SmallBitVector ExtractSliceOp::getDroppedDims() {
ArrayRef<int64_t> resultShape = getType().getShape(); ArrayRef<int64_t> resultShape = getType().getShape();
SmallVector<OpFoldResult> mixedSizes = getMixedSizes(); SmallVector<OpFoldResult> mixedSizes = getMixedSizes();
llvm::SmallBitVector droppedDims(mixedSizes.size()); llvm::SmallBitVector droppedDims(mixedSizes.size());
unsigned shapePos = 0; unsigned shapePos = 0;
for (const auto &size : enumerate(mixedSizes)) { for (const auto &size : enumerate(mixedSizes)) {
Optional<int64_t> sizeVal = getConstantIntValue(size.value()); std::optional<int64_t> sizeVal = getConstantIntValue(size.value());
// If the size is not 1, or if the current matched dimension of the result // If the size is not 1, or if the current matched dimension of the result
// is the same static shape as the size value (which is 1), then the // is the same static shape as the size value (which is 1), then the
// dimension is preserved. // dimension is preserved.
if (!sizeVal || *sizeVal != 1 || if (!sizeVal || *sizeVal != 1 ||
(shapePos < resultShape.size() && resultShape[shapePos] == 1)) { (shapePos < resultShape.size() && resultShape[shapePos] == 1)) {
shapePos++; shapePos++;
continue; continue;
} }
▲ Show 20 Lines • Show All 529 Lines • ▼ Show 20 Linesstruct InsertSliceOpCastFolder final : public OpRewritePattern<InsertOpTy> {
LogicalResult matchAndRewrite(InsertOpTy insertSliceOp, LogicalResult matchAndRewrite(InsertOpTy insertSliceOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
if (llvm::any_of(insertSliceOp.getOperands(), [](Value operand) { if (llvm::any_of(insertSliceOp.getOperands(), [](Value operand) {
return matchPattern(operand, matchConstantIndex()); return matchPattern(operand, matchConstantIndex());
})) }))
return failure(); return failure();
auto getSourceOfCastOp = [](Value v) -> Optional<Value> { auto getSourceOfCastOp = [](Value v) -> std::optional<Value> {
auto castOp = v.getDefiningOp<tensor::CastOp>(); auto castOp = v.getDefiningOp<tensor::CastOp>();
if (!castOp || !canFoldIntoConsumerOp(castOp)) if (!castOp || !canFoldIntoConsumerOp(castOp))
return std::nullopt; return std::nullopt;
return castOp.getSource(); return castOp.getSource();
}; };
Optional<Value> sourceCastSource = std::optional<Value> sourceCastSource =
getSourceOfCastOp(insertSliceOp.getSource()); getSourceOfCastOp(insertSliceOp.getSource());
Optional<Value> destCastSource = getSourceOfCastOp(insertSliceOp.getDest()); std::optional<Value> destCastSource =
getSourceOfCastOp(insertSliceOp.getDest());
if (!sourceCastSource && !destCastSource) if (!sourceCastSource && !destCastSource)
return failure(); return failure();
auto src = auto src =
(sourceCastSource ? *sourceCastSource : insertSliceOp.getSource()); (sourceCastSource ? *sourceCastSource : insertSliceOp.getSource());
auto dst = (destCastSource ? *destCastSource : insertSliceOp.getDest()); auto dst = (destCastSource ? *destCastSource : insertSliceOp.getDest());
auto srcType = src.getType().template cast<ShapedType>(); auto srcType = src.getType().template cast<ShapedType>();
auto dstType = dst.getType().template cast<ShapedType>(); auto dstType = dst.getType().template cast<ShapedType>();
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Linesstruct InsertSliceOpSourceCastInserter final
LogicalResult matchAndRewrite(InsertOpTy insertSliceOp, LogicalResult matchAndRewrite(InsertOpTy insertSliceOp,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
RankedTensorType srcType = insertSliceOp.getSourceType(); RankedTensorType srcType = insertSliceOp.getSourceType();
if (srcType.getRank() != insertSliceOp.getDestType().getRank()) if (srcType.getRank() != insertSliceOp.getDestType().getRank())
return failure(); return failure();
SmallVector<int64_t> newSrcShape(srcType.getShape().begin(), SmallVector<int64_t> newSrcShape(srcType.getShape().begin(),
srcType.getShape().end()); srcType.getShape().end());
for (int64_t i = 0; i < srcType.getRank(); ++i) { for (int64_t i = 0; i < srcType.getRank(); ++i) {
if (Optional<int64_t> constInt = if (std::optional<int64_t> constInt =
getConstantIntValue(insertSliceOp.getMixedSizes()[i])) getConstantIntValue(insertSliceOp.getMixedSizes()[i]))
newSrcShape[i] = *constInt; newSrcShape[i] = *constInt;
} }
RankedTensorType newSrcType = RankedTensorType newSrcType =
RankedTensorType::get(newSrcShape, srcType.getElementType()); RankedTensorType::get(newSrcShape, srcType.getElementType());
if (srcType == newSrcType || if (srcType == newSrcType ||
!preservesStaticInformation(srcType, newSrcType) || !preservesStaticInformation(srcType, newSrcType) ||
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Linesvoid PadOp::getAsmResultNames(function_ref<void(Value, StringRef)> setNameFn) {
setNameFn(getResult(), "padded"); setNameFn(getResult(), "padded");
} }
// TODO: Replace custom<InferType> directive with AllTypesMatch as soon as it // TODO: Replace custom<InferType> directive with AllTypesMatch as soon as it
// supports optional types. // supports optional types.
void printInferType(OpAsmPrinter &printer, Operation *op, Value optOperand, void printInferType(OpAsmPrinter &printer, Operation *op, Value optOperand,
Type typeToInfer, Type typeToInferFrom) {} Type typeToInfer, Type typeToInferFrom) {}
ParseResult parseInferType(OpAsmParser &parser, ParseResult
Optional<OpAsmParser::UnresolvedOperand> optOperand, parseInferType(OpAsmParser &parser,
std::optional<OpAsmParser::UnresolvedOperand> optOperand,
Type &typeToInfer, Type typeToInferFrom) { Type &typeToInfer, Type typeToInferFrom) {
if (optOperand) if (optOperand)
typeToInfer = typeToInferFrom; typeToInfer = typeToInferFrom;
return success(); return success();
} }
LogicalResult PadOp::verify() { LogicalResult PadOp::verify() {
auto sourceType = getSource().getType().cast<RankedTensorType>(); auto sourceType = getSource().getType().cast<RankedTensorType>();
auto resultType = getResult().getType().cast<RankedTensorType>(); auto resultType = getResult().getType().cast<RankedTensorType>();
▲ Show 20 Lines • Show All 713 Lines • ▼ Show 20 Lines if (!areAllInBound(expectedPackedType.getShape(), packedType.getShape())) {
return op->emitError("the shape of output is not large enough to hold the " return op->emitError("the shape of output is not large enough to hold the "
"packed data. Expected at least ") "packed data. Expected at least ")
<< expectedPackedType << ", got " << packedType; << expectedPackedType << ", got " << packedType;
} }
if (!llvm::all_of( if (!llvm::all_of(
llvm::zip(packedType.getShape().take_back(mixedTiles.size()), llvm::zip(packedType.getShape().take_back(mixedTiles.size()),
mixedTiles), mixedTiles),
[](std::tuple<int64_t, OpFoldResult> it) { [](std::tuple<int64_t, OpFoldResult> it) {
Optional<int64_t> constTileSize = std::optional<int64_t> constTileSize =
getConstantIntValue(std::get<1>(it)); getConstantIntValue(std::get<1>(it));
int64_t shape = std::get<0>(it); int64_t shape = std::get<0>(it);
if (!constTileSize) { if (!constTileSize) {
// If specified tile size is dynamic, output shape should // If specified tile size is dynamic, output shape should
// be dynamic too. // be dynamic too.
return ShapedType::isDynamic(shape); return ShapedType::isDynamic(shape);
} else { } else {
if (ShapedType::isDynamic(shape)) { if (ShapedType::isDynamic(shape)) {
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 LinesareNotFullTiles(ArrayRef<int64_t> inputShape,
DenseMap<int64_t, OpFoldResult> const &dimAndTileMapping) { DenseMap<int64_t, OpFoldResult> const &dimAndTileMapping) {
int64_t rank = inputShape.size(); int64_t rank = inputShape.size();
for (int64_t dim = 0; dim < rank; dim++) { for (int64_t dim = 0; dim < rank; dim++) {
if (ShapedType::isDynamic(inputShape[dim])) if (ShapedType::isDynamic(inputShape[dim]))
continue; continue;
auto it = dimAndTileMapping.find(dim); auto it = dimAndTileMapping.find(dim);
if (it == dimAndTileMapping.end()) if (it == dimAndTileMapping.end())
continue; continue;
Optional<int64_t> constantTile = getConstantIntValue(it->second); std::optional<int64_t> constantTile = getConstantIntValue(it->second);
if (!constantTile) if (!constantTile)
continue; continue;
if (inputShape[dim] % (*constantTile) != 0) if (inputShape[dim] % (*constantTile) != 0)
return true; return true;
} }
return false; return false;
} }
▲ Show 20 Lines • Show All 84 Lines • ▼ Show 20 Linesbool areTilesAndTiledDimsAllConstant(OpTy op) {
static_assert(llvm::is_one_of<OpTy, PackOp, UnPackOp>::value, static_assert(llvm::is_one_of<OpTy, PackOp, UnPackOp>::value,
"applies to only pack or unpack operations"); "applies to only pack or unpack operations");
ShapedType packedType = (std::is_same<OpTy, PackOp>::value) ShapedType packedType = (std::is_same<OpTy, PackOp>::value)
? op.getDestType() ? op.getDestType()
: op.getSourceType(); : op.getSourceType();
SmallVector<OpFoldResult> mixedTiles = op.getMixedTiles(); SmallVector<OpFoldResult> mixedTiles = op.getMixedTiles();
for (auto [dimDest, tile] : llvm::zip( for (auto [dimDest, tile] : llvm::zip(
packedType.getShape().take_back(mixedTiles.size()), mixedTiles)) { packedType.getShape().take_back(mixedTiles.size()), mixedTiles)) {
Optional<int64_t> constTileSize = getConstantIntValue(tile); std::optional<int64_t> constTileSize = getConstantIntValue(tile);
if (!constTileSize || ShapedType::isDynamic(dimDest)) if (!constTileSize || ShapedType::isDynamic(dimDest))
return false; return false;
} }
return true; return true;
} }
Speculation::Speculatability PackOp::getSpeculatability() { Speculation::Speculatability PackOp::getSpeculatability() {
if (auto paddingValue = getPaddingValue()) if (auto paddingValue = getPaddingValue())
▲ Show 20 Lines • Show All 170 Lines • Show Last 20 Lines

mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp

Show First 20 Lines • Show All 259 Lines • ▼ Show 20 Linesstatic UnpackTileDimInfo getUnpackTileDimInfo(OpBuilder &b, UnPackOp unpackOp,
bindDims(b.getContext(), dim0, dim1); bindDims(b.getContext(), dim0, dim1);
bindSymbols(b.getContext(), sym0); bindSymbols(b.getContext(), sym0);
OpFoldResult innerTileSize = dimAndTileMapping[tileDim]; OpFoldResult innerTileSize = dimAndTileMapping[tileDim];
info.isAlignedToInnerTileSize = false; info.isAlignedToInnerTileSize = false;
FailureOr<int64_t> cstSize = linalg::getConstantUpperBoundForIndex( FailureOr<int64_t> cstSize = linalg::getConstantUpperBoundForIndex(
getValueOrCreateConstantIndexOp(b, loc, tileSize)); getValueOrCreateConstantIndexOp(b, loc, tileSize));
Optional<int64_t> cstInnerSize = getConstantIntValue(innerTileSize); std::optional<int64_t> cstInnerSize = getConstantIntValue(innerTileSize);
if (!failed(cstSize) && cstInnerSize) { if (!failed(cstSize) && cstInnerSize) {
if (cstSize.value() % cstInnerSize.value() == 0) if (cstSize.value() % cstInnerSize.value() == 0)
info.isAlignedToInnerTileSize = true; info.isAlignedToInnerTileSize = true;
// If the tiling size equals to the inner tiling size, the outer dims are // If the tiling size equals to the inner tiling size, the outer dims are
// always 1. // always 1.
if (cstInnerSize.value() == cstSize.value()) { if (cstInnerSize.value() == cstSize.value()) {
auto lhs = AV(dim0).bind(tileOffset); auto lhs = AV(dim0).bind(tileOffset);
▲ Show 20 Lines • Show All 371 Lines • Show Last 20 Lines

mlir/lib/Dialect/Tensor/Transforms/SplitPaddingPatterns.cpp

Show All 20 Lines
#include "llvm/Support/Debug.h" #include "llvm/Support/Debug.h"
#define DEBUG_TYPE "mlir-tensor-split-padding" #define DEBUG_TYPE "mlir-tensor-split-padding"
using namespace mlir; using namespace mlir;
/// Returns true if the the given `attrOrValue` is a constant zero. /// Returns true if the the given `attrOrValue` is a constant zero.
static bool isZero(OpFoldResult attrOrValue) { static bool isZero(OpFoldResult attrOrValue) {
if (Optional<int64_t> val = getConstantIntValue(attrOrValue)) if (std::optional<int64_t> val = getConstantIntValue(attrOrValue))
return *val == 0; return *val == 0;
return false; return false;
} }
/// Gets the given `attrOrValue` as a Value by creating constant ops for /// Gets the given `attrOrValue` as a Value by creating constant ops for
/// attributes. /// attributes.
static Value getAsValue(OpFoldResult attrOrValue, OpBuilder &builder, static Value getAsValue(OpFoldResult attrOrValue, OpBuilder &builder,
Location loc) { Location loc) {
▲ Show 20 Lines • Show All 58 Lines • Show Last 20 Lines

mlir/lib/Dialect/Tosa/IR/TosaOps.cpp

Show First 20 Lines • Show All 367 Lines • ▼ Show 20 Lines for (size_t i = 0, e = shape.getRank(); i < e; ++i) {
outShape[i + rankDiff] = resolvedDim; outShape[i + rankDiff] = resolvedDim;
} }
} }
return success(); return success();
} }
LogicalResult tosa::ArgMaxOp::inferReturnTypeComponents( LogicalResult tosa::ArgMaxOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ShapeAdaptor inputShape = operands.getShape(0); ShapeAdaptor inputShape = operands.getShape(0);
IntegerAttr axis = attributes.get("axis").cast<IntegerAttr>(); IntegerAttr axis = attributes.get("axis").cast<IntegerAttr>();
int32_t axisVal = axis.getValue().getSExtValue(); int32_t axisVal = axis.getValue().getSExtValue();
if (!inputShape.hasRank()) { if (!inputShape.hasRank()) {
inferredReturnShapes.push_back(ShapedTypeComponents()); inferredReturnShapes.push_back(ShapedTypeComponents());
return success(); return success();
} }
SmallVector<int64_t> outShape; SmallVector<int64_t> outShape;
outShape.reserve(inputShape.getRank() - 1); outShape.reserve(inputShape.getRank() - 1);
for (int i = 0, s = inputShape.getRank(); i < s; i++) { for (int i = 0, s = inputShape.getRank(); i < s; i++) {
if (i == axisVal) if (i == axisVal)
continue; continue;
outShape.push_back(inputShape.getDimSize(i)); outShape.push_back(inputShape.getDimSize(i));
} }
inferredReturnShapes.push_back(ShapedTypeComponents(outShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outShape));
return success(); return success();
} }
LogicalResult tosa::ConcatOp::inferReturnTypeComponents( LogicalResult tosa::ConcatOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
// Infer all dimension sizes by reducing based on inputs. // Infer all dimension sizes by reducing based on inputs.
int32_t axis = int32_t axis =
attributes.get("axis").cast<IntegerAttr>().getValue().getSExtValue(); attributes.get("axis").cast<IntegerAttr>().getValue().getSExtValue();
llvm::SmallVector<int64_t> outputShape; llvm::SmallVector<int64_t> outputShape;
bool hasRankedInput = false; bool hasRankedInput = false;
for (auto operand : operands) { for (auto operand : operands) {
Show All 40 LinesLogicalResult tosa::ConcatOp::inferReturnTypeComponents(
outputShape[axis] = concatDimSize; outputShape[axis] = concatDimSize;
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult tosa::EqualOp::inferReturnTypeComponents( LogicalResult tosa::EqualOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
llvm::SmallVector<int64_t> outShape; llvm::SmallVector<int64_t> outShape;
if (resolveBroadcastShape(operands, outShape).failed()) { if (resolveBroadcastShape(operands, outShape).failed()) {
inferredReturnShapes.push_back(ShapedTypeComponents()); inferredReturnShapes.push_back(ShapedTypeComponents());
return success(); return success();
} }
inferredReturnShapes.push_back( inferredReturnShapes.push_back(
ShapedTypeComponents(outShape, IntegerType::get(context, /*width=*/1))); ShapedTypeComponents(outShape, IntegerType::get(context, /*width=*/1)));
return success(); return success();
} }
bool tosa::EqualOp::isCompatibleReturnTypes(TypeRange l, TypeRange r) { bool tosa::EqualOp::isCompatibleReturnTypes(TypeRange l, TypeRange r) {
if (l.size() != r.size() || l.size() != 1) if (l.size() != r.size() || l.size() != 1)
return false; return false;
return succeeded(verifyCompatibleShape(l[0], r[0])); return succeeded(verifyCompatibleShape(l[0], r[0]));
} }
LogicalResult tosa::FullyConnectedOp::inferReturnTypeComponents( LogicalResult tosa::FullyConnectedOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ShapeAdaptor inputShape = operands.getShape(0); ShapeAdaptor inputShape = operands.getShape(0);
ShapeAdaptor weightShape = operands.getShape(1); ShapeAdaptor weightShape = operands.getShape(1);
ShapeAdaptor biasShape = operands.getShape(2); ShapeAdaptor biasShape = operands.getShape(2);
// All shapes are dynamic. // All shapes are dynamic.
SmallVector<int64_t> outShape; SmallVector<int64_t> outShape;
outShape.resize(2, ShapedType::kDynamic); outShape.resize(2, ShapedType::kDynamic);
if (inputShape.hasRank()) { if (inputShape.hasRank()) {
outShape[0] = inputShape.getDimSize(0); outShape[0] = inputShape.getDimSize(0);
} }
if (weightShape.hasRank()) { if (weightShape.hasRank()) {
outShape[1] = weightShape.getDimSize(0); outShape[1] = weightShape.getDimSize(0);
} }
if (biasShape.hasRank()) { if (biasShape.hasRank()) {
outShape[1] = outShape[1] == ShapedType::kDynamic outShape[1] = outShape[1] == ShapedType::kDynamic ? biasShape.getDimSize(0)
? biasShape.getDimSize(0)
: outShape[1]; : outShape[1];
} }
inferredReturnShapes.push_back(ShapedTypeComponents(outShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outShape));
return success(); return success();
} }
LogicalResult FullyConnectedOp::verify() { return verifyConvOp(*this); } LogicalResult FullyConnectedOp::verify() { return verifyConvOp(*this); }
LogicalResult tosa::MatMulOp::inferReturnTypeComponents( LogicalResult tosa::MatMulOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ShapeAdaptor lhsShape = operands.getShape(0); ShapeAdaptor lhsShape = operands.getShape(0);
ShapeAdaptor rhsShape = operands.getShape(1); ShapeAdaptor rhsShape = operands.getShape(1);
// All shapes are dynamic. // All shapes are dynamic.
SmallVector<int64_t> outShape; SmallVector<int64_t> outShape;
outShape.resize(3, ShapedType::kDynamic); outShape.resize(3, ShapedType::kDynamic);
if (lhsShape.hasRank()) { if (lhsShape.hasRank()) {
outShape[0] = lhsShape.getDimSize(0); outShape[0] = lhsShape.getDimSize(0);
outShape[1] = lhsShape.getDimSize(1); outShape[1] = lhsShape.getDimSize(1);
} }
if (rhsShape.hasRank()) { if (rhsShape.hasRank()) {
outShape[0] = outShape[0] == ShapedType::kDynamic outShape[0] = outShape[0] == ShapedType::kDynamic ? rhsShape.getDimSize(0)
? rhsShape.getDimSize(0)
: outShape[0]; : outShape[0];
outShape[2] = rhsShape.getDimSize(2); outShape[2] = rhsShape.getDimSize(2);
} }
inferredReturnShapes.push_back(ShapedTypeComponents(outShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outShape));
return success(); return success();
} }
LogicalResult tosa::PadOp::inferReturnTypeComponents( LogicalResult tosa::PadOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ShapeAdaptor inputShape = operands.getShape(0); ShapeAdaptor inputShape = operands.getShape(0);
ShapeAdaptor paddingShape = operands.getShape(1); ShapeAdaptor paddingShape = operands.getShape(1);
SmallVector<int64_t> outputShape; SmallVector<int64_t> outputShape;
// If both inputs have unknown shape, we cannot determine the shape of the // If both inputs have unknown shape, we cannot determine the shape of the
// output. // output.
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Lines
static SmallVector<int64_t> convertToMlirShape(ArrayRef<int64_t> shape) { static SmallVector<int64_t> convertToMlirShape(ArrayRef<int64_t> shape) {
return to_vector(llvm::map_range(shape, [](int64_t dim) { return to_vector(llvm::map_range(shape, [](int64_t dim) {
return dim == -1 ? ShapedType::kDynamic : dim; return dim == -1 ? ShapedType::kDynamic : dim;
})); }));
} }
LogicalResult tosa::SliceOp::inferReturnTypeComponents( LogicalResult tosa::SliceOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ArrayAttr sizes = SliceOpAdaptor(operands, attributes).getSize(); ArrayAttr sizes = SliceOpAdaptor(operands, attributes).getSize();
SmallVector<int64_t> outputShape; SmallVector<int64_t> outputShape;
outputShape.reserve(sizes.size()); outputShape.reserve(sizes.size());
for (auto val : sizes) { for (auto val : sizes) {
outputShape.push_back(val.cast<IntegerAttr>().getValue().getSExtValue()); outputShape.push_back(val.cast<IntegerAttr>().getValue().getSExtValue());
} }
inferredReturnShapes.push_back(ShapedTypeComponents( inferredReturnShapes.push_back(
convertToMlirShape(outputShape))); ShapedTypeComponents(convertToMlirShape(outputShape)));
return success(); return success();
} }
LogicalResult tosa::TableOp::inferReturnTypeComponents( LogicalResult tosa::TableOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ShapeAdaptor inputShape = operands.getShape(0); ShapeAdaptor inputShape = operands.getShape(0);
if (!inputShape.hasRank()) { if (!inputShape.hasRank()) {
inferredReturnShapes.push_back(ShapedTypeComponents()); inferredReturnShapes.push_back(ShapedTypeComponents());
return success(); return success();
} }
inferredReturnShapes.resize(1); inferredReturnShapes.resize(1);
inputShape.getDims(inferredReturnShapes[0]); inputShape.getDims(inferredReturnShapes[0]);
return success(); return success();
} }
LogicalResult tosa::TileOp::inferReturnTypeComponents( LogicalResult tosa::TileOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
TileOpAdaptor adaptor(operands, attributes); TileOpAdaptor adaptor(operands, attributes);
ArrayAttr multiples = adaptor.getMultiples(); ArrayAttr multiples = adaptor.getMultiples();
ShapeAdaptor inputShape = operands.getShape(0); ShapeAdaptor inputShape = operands.getShape(0);
SmallVector<int64_t> outputShape; SmallVector<int64_t> outputShape;
if (!inputShape.hasRank()) { if (!inputShape.hasRank()) {
outputShape.resize(multiples.size(), ShapedType::kDynamic); outputShape.resize(multiples.size(), ShapedType::kDynamic);
Show All 17 Lines for (int i = 0, s = inputShape.getRank(); i < s; i++) {
outputShape.push_back(dim); outputShape.push_back(dim);
} }
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult tosa::ReshapeOp::inferReturnTypeComponents( LogicalResult tosa::ReshapeOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ReshapeOpAdaptor adaptor(operands, attributes); ReshapeOpAdaptor adaptor(operands, attributes);
ShapeAdaptor inputShape = operands.getShape(0); ShapeAdaptor inputShape = operands.getShape(0);
ArrayAttr newShape = adaptor.getNewShape(); ArrayAttr newShape = adaptor.getNewShape();
llvm::SmallVector<int64_t> newShapeValue; llvm::SmallVector<int64_t> newShapeValue;
getI64Values(newShape, newShapeValue); getI64Values(newShape, newShapeValue);
Show All 23 Lines if (ShapedType::isDynamic(val))
val = numElements / staticMul; val = numElements / staticMul;
} }
inferredReturnShapes.push_back(ShapedTypeComponents(newShapeValue)); inferredReturnShapes.push_back(ShapedTypeComponents(newShapeValue));
return success(); return success();
} }
LogicalResult tosa::TransposeOp::inferReturnTypeComponents( LogicalResult tosa::TransposeOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ShapeAdaptor inputShape = operands.getShape(0); ShapeAdaptor inputShape = operands.getShape(0);
ShapeAdaptor permsShape = operands.getShape(1); ShapeAdaptor permsShape = operands.getShape(1);
// If input rank and permutation length is unknown, the output rank is // If input rank and permutation length is unknown, the output rank is
// unknown. // unknown.
if (!inputShape.hasRank() || !permsShape.hasRank() || if (!inputShape.hasRank() || !permsShape.hasRank() ||
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Lines if (ShapeAdaptor permShape = operands.getValueAsShape(1)) {
} }
} }
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult tosa::GatherOp::inferReturnTypeComponents( LogicalResult tosa::GatherOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
llvm::SmallVector<int64_t> outputShape; llvm::SmallVector<int64_t> outputShape;
outputShape.resize(3, ShapedType::kDynamic); outputShape.resize(3, ShapedType::kDynamic);
ShapeAdaptor valuesShape = operands.getShape(0); ShapeAdaptor valuesShape = operands.getShape(0);
if (valuesShape.hasRank()) { if (valuesShape.hasRank()) {
outputShape[0] = valuesShape.getDimSize(0); outputShape[0] = valuesShape.getDimSize(0);
outputShape[2] = valuesShape.getDimSize(2); outputShape[2] = valuesShape.getDimSize(2);
} }
ShapeAdaptor indicesShape = operands.getShape(1); ShapeAdaptor indicesShape = operands.getShape(1);
if (indicesShape.hasRank()) { if (indicesShape.hasRank()) {
if (outputShape[0] == ShapedType::kDynamic) if (outputShape[0] == ShapedType::kDynamic)
outputShape[0] = indicesShape.getDimSize(0); outputShape[0] = indicesShape.getDimSize(0);
if (outputShape[1] == ShapedType::kDynamic) if (outputShape[1] == ShapedType::kDynamic)
outputShape[1] = indicesShape.getDimSize(1); outputShape[1] = indicesShape.getDimSize(1);
} }
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult tosa::ResizeOp::inferReturnTypeComponents( LogicalResult tosa::ResizeOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
ResizeOpAdaptor adaptor(operands, attributes); ResizeOpAdaptor adaptor(operands, attributes);
llvm::SmallVector<int64_t, 4> outputShape; llvm::SmallVector<int64_t, 4> outputShape;
outputShape.resize(4, ShapedType::kDynamic); outputShape.resize(4, ShapedType::kDynamic);
ShapeAdaptor inputShape = operands.getShape(adaptor.getInput()); ShapeAdaptor inputShape = operands.getShape(adaptor.getInput());
if (!inputShape.hasRank()) if (!inputShape.hasRank())
Show All 26 Lines outputShape[2] =
scaleInt[3]) + scaleInt[3]) +
1; 1;
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult tosa::ScatterOp::inferReturnTypeComponents( LogicalResult tosa::ScatterOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
llvm::SmallVector<int64_t> outputShape; llvm::SmallVector<int64_t> outputShape;
outputShape.resize(3, ShapedType::kDynamic); outputShape.resize(3, ShapedType::kDynamic);
ShapeAdaptor valuesInShape = operands.getShape(0); ShapeAdaptor valuesInShape = operands.getShape(0);
if (valuesInShape.hasRank()) { if (valuesInShape.hasRank()) {
outputShape[0] = valuesInShape.getDimSize(0); outputShape[0] = valuesInShape.getDimSize(0);
Show All 32 Linesstatic LogicalResult ReduceInferReturnTypes(
int64_t axisVal = axis.getValue().getSExtValue(); int64_t axisVal = axis.getValue().getSExtValue();
outputShape[axisVal] = 1; outputShape[axisVal] = 1;
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
#define REDUCE_SHAPE_INFER(OP) \ #define REDUCE_SHAPE_INFER(OP) \
LogicalResult OP::inferReturnTypeComponents( \ LogicalResult OP::inferReturnTypeComponents( \
MLIRContext *context, ::llvm::Optional<Location> location, \ MLIRContext *context, ::std::optional<Location> location, \
ValueShapeRange operands, DictionaryAttr attributes, \ ValueShapeRange operands, DictionaryAttr attributes, \
RegionRange regions, \ RegionRange regions, \
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { \ SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { \
return ReduceInferReturnTypes(operands.getShape(0), \ return ReduceInferReturnTypes(operands.getShape(0), \
attributes.get("axis").cast<IntegerAttr>(), \ attributes.get("axis").cast<IntegerAttr>(), \
inferredReturnShapes); \ inferredReturnShapes); \
} }
Show All 14 Linesstatic LogicalResult NAryInferReturnTypes(
} else { } else {
inferredReturnShapes.push_back(ShapedTypeComponents(outShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outShape));
} }
return success(); return success();
} }
#define NARY_SHAPE_INFER(OP) \ #define NARY_SHAPE_INFER(OP) \
LogicalResult OP::inferReturnTypeComponents( \ LogicalResult OP::inferReturnTypeComponents( \
MLIRContext *context, ::llvm::Optional<Location> location, \ MLIRContext *context, ::std::optional<Location> location, \
ValueShapeRange operands, DictionaryAttr attributes, \ ValueShapeRange operands, DictionaryAttr attributes, \
RegionRange regions, \ RegionRange regions, \
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { \ SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { \
return NAryInferReturnTypes(operands, inferredReturnShapes); \ return NAryInferReturnTypes(operands, inferredReturnShapes); \
} }
NARY_SHAPE_INFER(tosa::AbsOp) NARY_SHAPE_INFER(tosa::AbsOp)
NARY_SHAPE_INFER(tosa::AddOp) NARY_SHAPE_INFER(tosa::AddOp)
▲ Show 20 Lines • Show All 72 Lines • ▼ Show 20 Lines if (!ShapedType::isDynamic(width)) {
outputShape[2] = padded / stride[1] + 1; outputShape[2] = padded / stride[1] + 1;
} }
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult Conv2DOp::inferReturnTypeComponents( LogicalResult Conv2DOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
llvm::SmallVector<int64_t> outputShape(4, ShapedType::kDynamic); llvm::SmallVector<int64_t> outputShape(4, ShapedType::kDynamic);
Conv2DOp::Adaptor adaptor(operands.getValues(), attributes); Conv2DOp::Adaptor adaptor(operands.getValues(), attributes);
int64_t inputWidth = ShapedType::kDynamic; int64_t inputWidth = ShapedType::kDynamic;
int64_t inputHeight = ShapedType::kDynamic; int64_t inputHeight = ShapedType::kDynamic;
int64_t weightWidth = ShapedType::kDynamic; int64_t weightWidth = ShapedType::kDynamic;
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 LinesLogicalResult Conv2DOp::inferReturnTypeComponents(
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult Conv2DOp::verify() { return verifyConvOp(*this); } LogicalResult Conv2DOp::verify() { return verifyConvOp(*this); }
LogicalResult Conv3DOp::inferReturnTypeComponents( LogicalResult Conv3DOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
llvm::SmallVector<int64_t> outputShape(5, ShapedType::kDynamic); llvm::SmallVector<int64_t> outputShape(5, ShapedType::kDynamic);
Conv3DOp::Adaptor adaptor(operands.getValues(), attributes); Conv3DOp::Adaptor adaptor(operands.getValues(), attributes);
int64_t inputWidth = ShapedType::kDynamic; int64_t inputWidth = ShapedType::kDynamic;
int64_t inputHeight = ShapedType::kDynamic; int64_t inputHeight = ShapedType::kDynamic;
int64_t inputDepth = ShapedType::kDynamic; int64_t inputDepth = ShapedType::kDynamic;
▲ Show 20 Lines • Show All 60 Lines • ▼ Show 20 LinesLogicalResult Conv3DOp::inferReturnTypeComponents(
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult Conv3DOp::verify() { return verifyConvOp(*this); } LogicalResult Conv3DOp::verify() { return verifyConvOp(*this); }
LogicalResult AvgPool2dOp::inferReturnTypeComponents( LogicalResult AvgPool2dOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
return poolingInferReturnTypes(operands, attributes, inferredReturnShapes); return poolingInferReturnTypes(operands, attributes, inferredReturnShapes);
} }
LogicalResult MaxPool2dOp::inferReturnTypeComponents( LogicalResult MaxPool2dOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
return poolingInferReturnTypes(operands, attributes, inferredReturnShapes); return poolingInferReturnTypes(operands, attributes, inferredReturnShapes);
} }
LogicalResult DepthwiseConv2DOp::inferReturnTypeComponents( LogicalResult DepthwiseConv2DOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
llvm::SmallVector<int64_t> outputShape(4, ShapedType::kDynamic); llvm::SmallVector<int64_t> outputShape(4, ShapedType::kDynamic);
DepthwiseConv2DOp::Adaptor adaptor(operands.getValues(), attributes); DepthwiseConv2DOp::Adaptor adaptor(operands.getValues(), attributes);
int64_t inputWidth = ShapedType::kDynamic; int64_t inputWidth = ShapedType::kDynamic;
int64_t inputHeight = ShapedType::kDynamic; int64_t inputHeight = ShapedType::kDynamic;
int64_t inputChannels = ShapedType::kDynamic; int64_t inputChannels = ShapedType::kDynamic;
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 LinesLogicalResult DepthwiseConv2DOp::inferReturnTypeComponents(
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult DepthwiseConv2DOp::verify() { return verifyConvOp(*this); } LogicalResult DepthwiseConv2DOp::verify() { return verifyConvOp(*this); }
LogicalResult TransposeConv2DOp::inferReturnTypeComponents( LogicalResult TransposeConv2DOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
TransposeConv2DOp::Adaptor adaptor(operands.getValues(), attributes); TransposeConv2DOp::Adaptor adaptor(operands.getValues(), attributes);
llvm::SmallVector<int64_t> outputShape; llvm::SmallVector<int64_t> outputShape;
getI64Values(adaptor.getOutShape(), outputShape); getI64Values(adaptor.getOutShape(), outputShape);
outputShape = convertToMlirShape(outputShape); outputShape = convertToMlirShape(outputShape);
int64_t inputWidth = ShapedType::kDynamic; int64_t inputWidth = ShapedType::kDynamic;
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines outputShape[2] =
ShapedType::isDynamic(outputShape[2]) ? calculateSize : outputShape[2]; ShapedType::isDynamic(outputShape[2]) ? calculateSize : outputShape[2];
} }
inferredReturnShapes.push_back(ShapedTypeComponents(outputShape)); inferredReturnShapes.push_back(ShapedTypeComponents(outputShape));
return success(); return success();
} }
LogicalResult IfOp::inferReturnTypeComponents( LogicalResult IfOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
llvm::SmallVector<tosa::YieldOp> yieldOps; llvm::SmallVector<tosa::YieldOp> yieldOps;
for (Region *region : regions) { for (Region *region : regions) {
for (auto &block : *region) for (auto &block : *region)
if (auto returnOp = dyn_cast<tosa::YieldOp>(block.getTerminator())) if (auto returnOp = dyn_cast<tosa::YieldOp>(block.getTerminator()))
yieldOps.push_back(returnOp); yieldOps.push_back(returnOp);
} }
Show All 27 LinesLogicalResult IfOp::inferReturnTypeComponents(
for (const ValueKnowledge &result : resultKnowledge) { for (const ValueKnowledge &result : resultKnowledge) {
inferredReturnShapes.push_back(result.getShapedTypeComponents()); inferredReturnShapes.push_back(result.getShapedTypeComponents());
} }
return success(); return success();
} }
LogicalResult WhileOp::inferReturnTypeComponents( LogicalResult WhileOp::inferReturnTypeComponents(
MLIRContext *context, ::llvm::Optional<Location> location, MLIRContext *context, ::std::optional<Location> location,
ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
llvm::SmallVector<tosa::YieldOp> yieldOps; llvm::SmallVector<tosa::YieldOp> yieldOps;
for (auto &block : *regions[1]) for (auto &block : *regions[1])
if (auto returnOp = dyn_cast<tosa::YieldOp>(block.getTerminator())) if (auto returnOp = dyn_cast<tosa::YieldOp>(block.getTerminator()))
yieldOps.push_back(returnOp); yieldOps.push_back(returnOp);
// TOSA's while must have a tosa.yield as its terminator. If not found this // TOSA's while must have a tosa.yield as its terminator. If not found this
▲ Show 20 Lines • Show All 46 Lines • Show Last 20 Lines

mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp

Show All 40 Lines
struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> { struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
public: public:
explicit TosaValidation() = default; explicit TosaValidation() = default;
private: private:
void runOnOperation() override; void runOnOperation() override;
llvm::Optional<TosaProfileEnum> profileType; std::optional<TosaProfileEnum> profileType;
}; };
void TosaValidation::runOnOperation() { void TosaValidation::runOnOperation() {
profileType = symbolizeEnum<TosaProfileEnum>(profileName); profileType = symbolizeEnum<TosaProfileEnum>(profileName);
getOperation().walk([&](Operation *op) { getOperation().walk([&](Operation *op) {
for (Value operand : op->getOperands()) { for (Value operand : op->getOperands()) {
if ((profileType == TosaProfileEnum::BaseInference) && if ((profileType == TosaProfileEnum::BaseInference) &&
Show All 11 Lines

mlir/lib/Dialect/Transform/IR/TransformOps.cpp

Show First 20 Lines • Show All 111 Lines • ▼ Show 20 LinesLogicalResult PatternApplicatorExtension::findAllMatches(
return success(); return success();
} }
} // namespace } // namespace
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AlternativesOp // AlternativesOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
OperandRange OperandRange transform::AlternativesOp::getSuccessorEntryOperands(
transform::AlternativesOp::getSuccessorEntryOperands(Optional<unsigned> index) { std::optional<unsigned> index) {
if (index && getOperation()->getNumOperands() == 1) if (index && getOperation()->getNumOperands() == 1)
return getOperation()->getOperands(); return getOperation()->getOperands();
return OperandRange(getOperation()->operand_end(), return OperandRange(getOperation()->operand_end(),
getOperation()->operand_end()); getOperation()->operand_end());
} }
void transform::AlternativesOp::getSuccessorRegions( void transform::AlternativesOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
for (Region &alternative : llvm::drop_begin( for (Region &alternative : llvm::drop_begin(
getAlternatives(), index.has_value() ? *index + 1 : 0)) { getAlternatives(), index.has_value() ? *index + 1 : 0)) {
regions.emplace_back(&alternative, !getOperands().empty() regions.emplace_back(&alternative, !getOperands().empty()
? alternative.getArguments() ? alternative.getArguments()
: Block::BlockArgListType()); : Block::BlockArgListType());
} }
if (index.has_value()) if (index.has_value())
▲ Show 20 Lines • Show All 195 Lines • ▼ Show 20 Lines if (any_of(getBody().front().without_terminator(), [&](Operation &op) {
onlyReadsHandle(getTarget(), effects); onlyReadsHandle(getTarget(), effects);
} }
for (Value result : getResults()) for (Value result : getResults())
producesHandle(result, effects); producesHandle(result, effects);
} }
void transform::ForeachOp::getSuccessorRegions( void transform::ForeachOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
Region *bodyRegion = &getBody(); Region *bodyRegion = &getBody();
if (!index) { if (!index) {
regions.emplace_back(bodyRegion, bodyRegion->getArguments()); regions.emplace_back(bodyRegion, bodyRegion->getArguments());
return; return;
} }
// Branch back to the region or the parent. // Branch back to the region or the parent.
assert(*index == 0 && "unexpected region index"); assert(*index == 0 && "unexpected region index");
regions.emplace_back(bodyRegion, bodyRegion->getArguments()); regions.emplace_back(bodyRegion, bodyRegion->getArguments());
regions.emplace_back(); regions.emplace_back();
} }
OperandRange OperandRange
transform::ForeachOp::getSuccessorEntryOperands(Optional<unsigned> index) { transform::ForeachOp::getSuccessorEntryOperands(std::optional<unsigned> index) {
// The iteration variable op handle is mapped to a subset (one op to be // The iteration variable op handle is mapped to a subset (one op to be
// precise) of the payload ops of the ForeachOp operand. // precise) of the payload ops of the ForeachOp operand.
assert(index && *index == 0 && "unexpected region index"); assert(index && *index == 0 && "unexpected region index");
return getOperation()->getOperands(); return getOperation()->getOperands();
} }
transform::YieldOp transform::ForeachOp::getYieldOp() { transform::YieldOp transform::ForeachOp::getYieldOp() {
return cast<transform::YieldOp>(getBody().front().getTerminator()); return cast<transform::YieldOp>(getBody().front().getTerminator());
▲ Show 20 Lines • Show All 367 Lines • ▼ Show 20 Lines for (Operation &op : *getBodyBlock()) {
SmallVector<MemoryEffects::EffectInstance, 2> nestedEffects; SmallVector<MemoryEffects::EffectInstance, 2> nestedEffects;
iface.getEffectsOnValue(getBodyBlock()->getArgument(0), nestedEffects); iface.getEffectsOnValue(getBodyBlock()->getArgument(0), nestedEffects);
for (const auto &effect : nestedEffects) for (const auto &effect : nestedEffects)
effects.emplace_back(effect.getEffect(), getRoot(), effect.getResource()); effects.emplace_back(effect.getEffect(), getRoot(), effect.getResource());
} }
} }
OperandRange OperandRange transform::SequenceOp::getSuccessorEntryOperands(
transform::SequenceOp::getSuccessorEntryOperands(Optional<unsigned> index) { std::optional<unsigned> index) {
assert(index && *index == 0 && "unexpected region index"); assert(index && *index == 0 && "unexpected region index");
if (getOperation()->getNumOperands() == 1) if (getOperation()->getNumOperands() == 1)
return getOperation()->getOperands(); return getOperation()->getOperands();
return OperandRange(getOperation()->operand_end(), return OperandRange(getOperation()->operand_end(),
getOperation()->operand_end()); getOperation()->operand_end());
} }
void transform::SequenceOp::getSuccessorRegions( void transform::SequenceOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
if (!index) { if (!index) {
Region *bodyRegion = &getBody(); Region *bodyRegion = &getBody();
regions.emplace_back(bodyRegion, !operands.empty() regions.emplace_back(bodyRegion, !operands.empty()
? bodyRegion->getArguments() ? bodyRegion->getArguments()
: Block::BlockArgListType()); : Block::BlockArgListType());
return; return;
} }
▲ Show 20 Lines • Show All 153 Lines • Show Last 20 Lines

mlir/lib/Dialect/Utils/ReshapeOpsUtils.cpp

Show First 20 Lines • Show All 51 Lines • ▼ Show 20 Lines while (sourceShape[sourceDim] != ShapedType::kDynamic &&
prodOfCollapsedDims *= sourceShape[sourceDim]; prodOfCollapsedDims *= sourceShape[sourceDim];
currIndices.push_back(sourceDim++); currIndices.push_back(sourceDim++);
} }
// If the current expanded dimension is dynamic, then the collapsed // If the current expanded dimension is dynamic, then the collapsed
// dimensions should also be dynamic and product of all previous unprocessed // dimensions should also be dynamic and product of all previous unprocessed
// dimensions of the expanded shape should be 1. // dimensions of the expanded shape should be 1.
if (sourceShape[sourceDim] == ShapedType::kDynamic && if (sourceShape[sourceDim] == ShapedType::kDynamic &&
(currTargetShape != ShapedType::kDynamic || (currTargetShape != ShapedType::kDynamic || prodOfCollapsedDims != 1))
prodOfCollapsedDims != 1))
return std::nullopt; return std::nullopt;
// If the collapsed dim is dynamic, the current expanded dim should also // If the collapsed dim is dynamic, the current expanded dim should also
// be dynamic. // be dynamic.
if (currTargetShape == ShapedType::kDynamic && if (currTargetShape == ShapedType::kDynamic &&
sourceShape[sourceDim] != ShapedType::kDynamic) sourceShape[sourceDim] != ShapedType::kDynamic)
return std::nullopt; return std::nullopt;
▲ Show 20 Lines • Show All 154 Lines • ▼ Show 20 Lines
} }
LogicalResult mlir::reshapeLikeShapesAreCompatible( LogicalResult mlir::reshapeLikeShapesAreCompatible(
function_ref<LogicalResult(const Twine &)> emitError, function_ref<LogicalResult(const Twine &)> emitError,
ArrayRef<int64_t> collapsedShape, ArrayRef<int64_t> expandedShape, ArrayRef<int64_t> collapsedShape, ArrayRef<int64_t> expandedShape,
ArrayRef<ReassociationIndices> reassociationMaps, bool isExpandingReshape) { ArrayRef<ReassociationIndices> reassociationMaps, bool isExpandingReshape) {
unsigned expandedDimStart = 0; unsigned expandedDimStart = 0;
for (const auto &map : llvm::enumerate(reassociationMaps)) { for (const auto &map : llvm::enumerate(reassociationMaps)) {
Optional<int64_t> dynamicShape; std::optional<int64_t> dynamicShape;
int64_t linearizedStaticShape = 1; int64_t linearizedStaticShape = 1;
for (const auto &dim : llvm::enumerate( for (const auto &dim : llvm::enumerate(
expandedShape.slice(expandedDimStart, map.value().size()))) { expandedShape.slice(expandedDimStart, map.value().size()))) {
if (ShapedType::isDynamic(dim.value())) { if (ShapedType::isDynamic(dim.value())) {
if (isExpandingReshape && dynamicShape) { if (isExpandingReshape && dynamicShape) {
return emitError("invalid to have a single dimension (" + return emitError("invalid to have a single dimension (" +
Twine(map.index()) + Twine(map.index()) +
") expanded into multiple dynamic dims (" + ") expanded into multiple dynamic dims (" +
Show All 33 Lines
llvm::SmallBitVector llvm::SmallBitVector
mlir::getSlicedDimensions(ArrayRef<OpFoldResult> sliceInputShape, mlir::getSlicedDimensions(ArrayRef<OpFoldResult> sliceInputShape,
ArrayRef<Range> sliceParams) { ArrayRef<Range> sliceParams) {
assert(sliceParams.size() == sliceInputShape.size() && assert(sliceParams.size() == sliceInputShape.size() &&
"only supports non rank-reducing case"); "only supports non rank-reducing case");
llvm::SmallBitVector mask(sliceInputShape.size()); llvm::SmallBitVector mask(sliceInputShape.size());
unsigned idx = 0; unsigned idx = 0;
for (const auto &[offset, size, stride] : sliceParams) { for (const auto &[offset, size, stride] : sliceParams) {
Optional<int64_t> offsetConst = getConstantIntValue(offset); std::optional<int64_t> offsetConst = getConstantIntValue(offset);
Optional<int64_t> strideConst = getConstantIntValue(stride); std::optional<int64_t> strideConst = getConstantIntValue(stride);
mask[idx] = !isEqualConstantIntOrValue(size, sliceInputShape[idx]) || mask[idx] = !isEqualConstantIntOrValue(size, sliceInputShape[idx]) ||
(!strideConst || *strideConst != 1) || (!strideConst || *strideConst != 1) ||
(!offsetConst || *offsetConst != 0); (!offsetConst || *offsetConst != 0);
idx++; idx++;
} }
return mask; return mask;
} }
▲ Show 20 Lines • Show All 160 Lines • Show Last 20 Lines

mlir/lib/Dialect/Utils/StaticValueUtils.cpp

Show First 20 Lines • Show All 85 Lines • ▼ Show 20 LinesSmallVector<OpFoldResult> getAsOpFoldResult(ArrayAttr arrayAttr) {
SmallVector<OpFoldResult> res; SmallVector<OpFoldResult> res;
res.reserve(arrayAttr.size()); res.reserve(arrayAttr.size());
for (Attribute a : arrayAttr) for (Attribute a : arrayAttr)
res.push_back(a); res.push_back(a);
return res; return res;
} }
/// If ofr is a constant integer or an IntegerAttr, return the integer. /// If ofr is a constant integer or an IntegerAttr, return the integer.
Optional<int64_t> getConstantIntValue(OpFoldResult ofr) { std::optional<int64_t> getConstantIntValue(OpFoldResult ofr) {
// Case 1: Check for Constant integer. // Case 1: Check for Constant integer.
if (auto val = ofr.dyn_cast<Value>()) { if (auto val = ofr.dyn_cast<Value>()) {
APSInt intVal; APSInt intVal;
if (matchPattern(val, m_ConstantInt(&intVal))) if (matchPattern(val, m_ConstantInt(&intVal)))
return intVal.getSExtValue(); return intVal.getSExtValue();
return std::nullopt; return std::nullopt;
} }
// Case 2: Check for IntegerAttr. // Case 2: Check for IntegerAttr.
▲ Show 20 Lines • Show All 73 Lines • Show Last 20 Lines

mlir/lib/Dialect/Vector/IR/VectorOps.cpp

Show First 20 Lines • Show All 309 Lines • ▼ Show 20 Lines
OpFoldResult MultiDimReductionOp::fold(ArrayRef<Attribute> operands) { OpFoldResult MultiDimReductionOp::fold(ArrayRef<Attribute> operands) {
// Single parallel dim, this is a noop. // Single parallel dim, this is a noop.
if (getSourceVectorType().getRank() == 1 && !isReducedDim(0)) if (getSourceVectorType().getRank() == 1 && !isReducedDim(0))
return getSource(); return getSource();
return {}; return {};
} }
Optional<SmallVector<int64_t, 4>> MultiDimReductionOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>>
MultiDimReductionOp::getShapeForUnroll() {
return llvm::to_vector<4>(getSourceVectorType().getShape()); return llvm::to_vector<4>(getSourceVectorType().getShape());
} }
LogicalResult MultiDimReductionOp::verify() { LogicalResult MultiDimReductionOp::verify() {
SmallVector<int64_t> targetShape; SmallVector<int64_t> targetShape;
Type inferredReturnType; Type inferredReturnType;
for (auto it : llvm::enumerate(getSourceVectorType().getShape())) for (auto it : llvm::enumerate(getSourceVectorType().getShape()))
if (!llvm::any_of(getReductionDims().getValue(), [&](Attribute attr) { if (!llvm::any_of(getReductionDims().getValue(), [&](Attribute attr) {
▲ Show 20 Lines • Show All 168 Lines • ▼ Show 20 LinesValue mlir::vector::getVectorReductionOp(arith::AtomicRMWKind op,
// TODO: Add remaining reduction operations. // TODO: Add remaining reduction operations.
default: default:
(void)emitOptionalError(loc, "Reduction operation type not supported"); (void)emitOptionalError(loc, "Reduction operation type not supported");
break; break;
} }
return nullptr; return nullptr;
} }
Optional<SmallVector<int64_t, 4>> ReductionOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>> ReductionOp::getShapeForUnroll() {
return llvm::to_vector<4>(getVectorType().getShape()); return llvm::to_vector<4>(getVectorType().getShape());
} }
namespace { namespace {
struct ElideSingleElementReduction : public OpRewritePattern<ReductionOp> { struct ElideSingleElementReduction : public OpRewritePattern<ReductionOp> {
using OpRewritePattern::OpRewritePattern; using OpRewritePattern::OpRewritePattern;
LogicalResult matchAndRewrite(ReductionOp reductionOp, LogicalResult matchAndRewrite(ReductionOp reductionOp,
▲ Show 20 Lines • Show All 422 Lines • ▼ Show 20 Lines
} }
std::vector<std::pair<int64_t, int64_t>> ContractionOp::getBatchDimMap() { std::vector<std::pair<int64_t, int64_t>> ContractionOp::getBatchDimMap() {
SmallVector<AffineMap, 4> indexingMaps(getIndexingMapsArray()); SmallVector<AffineMap, 4> indexingMaps(getIndexingMapsArray());
return getDimMap(indexingMaps, getIteratorTypes(), IteratorType::parallel, return getDimMap(indexingMaps, getIteratorTypes(), IteratorType::parallel,
getContext()); getContext());
} }
Optional<SmallVector<int64_t, 4>> ContractionOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>> ContractionOp::getShapeForUnroll() {
SmallVector<int64_t, 4> shape; SmallVector<int64_t, 4> shape;
getIterationBounds(shape); getIterationBounds(shape);
return shape; return shape;
} }
/// Return a fused vector::ContractionOp which represents a patterns such as: /// Return a fused vector::ContractionOp which represents a patterns such as:
/// ///
/// ```mlir /// ```mlir
▲ Show 20 Lines • Show All 121 Lines • ▼ Show 20 Linesvoid vector::ExtractOp::build(OpBuilder &builder, OperationState &result,
SmallVector<int64_t, 4> positionConstants = SmallVector<int64_t, 4> positionConstants =
llvm::to_vector<4>(llvm::map_range(position, [](Value pos) { llvm::to_vector<4>(llvm::map_range(position, [](Value pos) {
return pos.getDefiningOp<arith::ConstantIndexOp>().value(); return pos.getDefiningOp<arith::ConstantIndexOp>().value();
})); }));
build(builder, result, source, positionConstants); build(builder, result, source, positionConstants);
} }
LogicalResult LogicalResult
ExtractOp::inferReturnTypes(MLIRContext *, Optional<Location>, ExtractOp::inferReturnTypes(MLIRContext *, std::optional<Location>,
ValueRange operands, DictionaryAttr attributes, ValueRange operands, DictionaryAttr attributes,
RegionRange, RegionRange,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
ExtractOp::Adaptor op(operands, attributes); ExtractOp::Adaptor op(operands, attributes);
auto vectorType = op.getVector().getType().cast<VectorType>(); auto vectorType = op.getVector().getType().cast<VectorType>();
if (static_cast<int64_t>(op.getPosition().size()) == vectorType.getRank()) { if (static_cast<int64_t>(op.getPosition().size()) == vectorType.getRank()) {
inferredReturnTypes.push_back(vectorType.getElementType()); inferredReturnTypes.push_back(vectorType.getElementType());
} else { } else {
▲ Show 20 Lines • Show All 627 Lines • ▼ Show 20 Linesstatic void populateFromInt64AttrArray(ArrayAttr arrayAttr,
for (auto attr : arrayAttr) for (auto attr : arrayAttr)
results.push_back(attr.cast<IntegerAttr>().getInt()); results.push_back(attr.cast<IntegerAttr>().getInt());
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// FmaOp // FmaOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Optional<SmallVector<int64_t, 4>> FMAOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>> FMAOp::getShapeForUnroll() {
return llvm::to_vector<4>(getVectorType().getShape()); return llvm::to_vector<4>(getVectorType().getShape());
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// BroadcastOp // BroadcastOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Return the dimensions of the result vector that were formerly ones in the /// Return the dimensions of the result vector that were formerly ones in the
▲ Show 20 Lines • Show All 263 Lines • ▼ Show 20 Lines for (const auto &en : llvm::enumerate(maskAttr)) {
auto attr = en.value().dyn_cast<IntegerAttr>(); auto attr = en.value().dyn_cast<IntegerAttr>();
if (!attr || attr.getInt() < 0 || attr.getInt() >= indexSize) if (!attr || attr.getInt() < 0 || attr.getInt() >= indexSize)
return emitOpError("mask index #") << (en.index() + 1) << " out of range"; return emitOpError("mask index #") << (en.index() + 1) << " out of range";
} }
return success(); return success();
} }
LogicalResult LogicalResult
ShuffleOp::inferReturnTypes(MLIRContext *, Optional<Location>, ShuffleOp::inferReturnTypes(MLIRContext *, std::optional<Location>,
ValueRange operands, DictionaryAttr attributes, ValueRange operands, DictionaryAttr attributes,
RegionRange, RegionRange,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
ShuffleOp::Adaptor op(operands, attributes); ShuffleOp::Adaptor op(operands, attributes);
auto v1Type = op.getV1().getType().cast<VectorType>(); auto v1Type = op.getV1().getType().cast<VectorType>();
auto v1Rank = v1Type.getRank(); auto v1Rank = v1Type.getRank();
// Construct resulting type: leading dimension matches mask // Construct resulting type: leading dimension matches mask
// length, all trailing dimensions match the operands. // length, all trailing dimensions match the operands.
▲ Show 20 Lines • Show All 1,160 Lines • ▼ Show 20 Linesvoid TransferReadOp::build(OpBuilder &builder, OperationState &result,
build(builder, result, vectorType, source, indices, permutationMapAttr, build(builder, result, vectorType, source, indices, permutationMapAttr,
padding, /*mask=*/Value(), inBoundsAttr); padding, /*mask=*/Value(), inBoundsAttr);
} }
/// 2. Builder that sets padding to zero an empty mask (variant without attrs). /// 2. Builder that sets padding to zero an empty mask (variant without attrs).
void TransferReadOp::build(OpBuilder &builder, OperationState &result, void TransferReadOp::build(OpBuilder &builder, OperationState &result,
VectorType vectorType, Value source, VectorType vectorType, Value source,
ValueRange indices, AffineMap permutationMap, ValueRange indices, AffineMap permutationMap,
Optional<ArrayRef<bool>> inBounds) { std::optional<ArrayRef<bool>> inBounds) {
auto permutationMapAttr = AffineMapAttr::get(permutationMap); auto permutationMapAttr = AffineMapAttr::get(permutationMap);
auto inBoundsAttr = (inBounds && !inBounds.value().empty()) auto inBoundsAttr = (inBounds && !inBounds.value().empty())
? builder.getBoolArrayAttr(inBounds.value()) ? builder.getBoolArrayAttr(inBounds.value())
: ArrayAttr(); : ArrayAttr();
build(builder, result, vectorType, source, indices, permutationMapAttr, build(builder, result, vectorType, source, indices, permutationMapAttr,
inBoundsAttr); inBoundsAttr);
} }
/// 3. Builder that sets permutation map to 'getMinorIdentityMap'. /// 3. Builder that sets permutation map to 'getMinorIdentityMap'.
void TransferReadOp::build(OpBuilder &builder, OperationState &result, void TransferReadOp::build(OpBuilder &builder, OperationState &result,
VectorType vectorType, Value source, VectorType vectorType, Value source,
ValueRange indices, Value padding, ValueRange indices, Value padding,
Optional<ArrayRef<bool>> inBounds) { std::optional<ArrayRef<bool>> inBounds) {
AffineMap permutationMap = getTransferMinorIdentityMap( AffineMap permutationMap = getTransferMinorIdentityMap(
source.getType().cast<ShapedType>(), vectorType); source.getType().cast<ShapedType>(), vectorType);
auto permutationMapAttr = AffineMapAttr::get(permutationMap); auto permutationMapAttr = AffineMapAttr::get(permutationMap);
auto inBoundsAttr = (inBounds && !inBounds.value().empty()) auto inBoundsAttr = (inBounds && !inBounds.value().empty())
? builder.getBoolArrayAttr(inBounds.value()) ? builder.getBoolArrayAttr(inBounds.value())
: ArrayAttr(); : ArrayAttr();
build(builder, result, vectorType, source, indices, permutationMapAttr, build(builder, result, vectorType, source, indices, permutationMapAttr,
padding, padding,
/*mask=*/Value(), inBoundsAttr); /*mask=*/Value(), inBoundsAttr);
} }
/// 4. Builder that sets padding to zero and permutation map to /// 4. Builder that sets padding to zero and permutation map to
/// 'getMinorIdentityMap'. /// 'getMinorIdentityMap'.
void TransferReadOp::build(OpBuilder &builder, OperationState &result, void TransferReadOp::build(OpBuilder &builder, OperationState &result,
VectorType vectorType, Value source, VectorType vectorType, Value source,
ValueRange indices, ValueRange indices,
Optional<ArrayRef<bool>> inBounds) { std::optional<ArrayRef<bool>> inBounds) {
Type elemType = source.getType().cast<ShapedType>().getElementType(); Type elemType = source.getType().cast<ShapedType>().getElementType();
Value padding = builder.create<arith::ConstantOp>( Value padding = builder.create<arith::ConstantOp>(
result.location, elemType, builder.getZeroAttr(elemType)); result.location, elemType, builder.getZeroAttr(elemType));
build(builder, result, vectorType, source, indices, padding, inBounds); build(builder, result, vectorType, source, indices, padding, inBounds);
} }
template <typename EmitFun> template <typename EmitFun>
static LogicalResult verifyPermutationMap(AffineMap permutationMap, static LogicalResult verifyPermutationMap(AffineMap permutationMap,
▲ Show 20 Lines • Show All 348 Lines • ▼ Show 20 Lines if (succeeded(foldTransferInBoundsAttribute(*this)))
return getResult(); return getResult();
if (succeeded(memref::foldMemRefCast(*this))) if (succeeded(memref::foldMemRefCast(*this)))
return getResult(); return getResult();
if (succeeded(tensor::foldTensorCast(*this))) if (succeeded(tensor::foldTensorCast(*this)))
return getResult(); return getResult();
return OpFoldResult(); return OpFoldResult();
} }
Optional<SmallVector<int64_t, 4>> TransferReadOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>> TransferReadOp::getShapeForUnroll() {
return llvm::to_vector<4>(getVectorType().getShape()); return llvm::to_vector<4>(getVectorType().getShape());
} }
void TransferReadOp::getEffects( void TransferReadOp::getEffects(
SmallVectorImpl<SideEffects::EffectInstance<MemoryEffects::Effect>> SmallVectorImpl<SideEffects::EffectInstance<MemoryEffects::Effect>>
&effects) { &effects) {
if (getShapedType().isa<MemRefType>()) if (getShapedType().isa<MemRefType>())
effects.emplace_back(MemoryEffects::Read::get(), getSource(), effects.emplace_back(MemoryEffects::Read::get(), getSource(),
▲ Show 20 Lines • Show All 210 Lines • ▼ Show 20 Lines build(builder, result, vector, dest, indices, permutationMapAttr,
/*mask=*/Value(), inBoundsAttr); /*mask=*/Value(), inBoundsAttr);
} }
/// 3. Builder with type inference that sets an empty mask (variant without /// 3. Builder with type inference that sets an empty mask (variant without
/// attrs) /// attrs)
void TransferWriteOp::build(OpBuilder &builder, OperationState &result, void TransferWriteOp::build(OpBuilder &builder, OperationState &result,
Value vector, Value dest, ValueRange indices, Value vector, Value dest, ValueRange indices,
AffineMap permutationMap, AffineMap permutationMap,
Optional<ArrayRef<bool>> inBounds) { std::optional<ArrayRef<bool>> inBounds) {
auto permutationMapAttr = AffineMapAttr::get(permutationMap); auto permutationMapAttr = AffineMapAttr::get(permutationMap);
auto inBoundsAttr = (inBounds && !inBounds.value().empty()) auto inBoundsAttr = (inBounds && !inBounds.value().empty())
? builder.getBoolArrayAttr(inBounds.value()) ? builder.getBoolArrayAttr(inBounds.value())
: ArrayAttr(); : ArrayAttr();
build(builder, result, vector, dest, indices, permutationMapAttr, build(builder, result, vector, dest, indices, permutationMapAttr,
/*mask=*/Value(), inBoundsAttr); /*mask=*/Value(), inBoundsAttr);
} }
/// 4. Builder with type inference that sets an empty mask and sets permutation /// 4. Builder with type inference that sets an empty mask and sets permutation
/// map to 'getMinorIdentityMap'. /// map to 'getMinorIdentityMap'.
void TransferWriteOp::build(OpBuilder &builder, OperationState &result, void TransferWriteOp::build(OpBuilder &builder, OperationState &result,
Value vector, Value dest, ValueRange indices, Value vector, Value dest, ValueRange indices,
Optional<ArrayRef<bool>> inBounds) { std::optional<ArrayRef<bool>> inBounds) {
auto vectorType = vector.getType().cast<VectorType>(); auto vectorType = vector.getType().cast<VectorType>();
AffineMap permutationMap = getTransferMinorIdentityMap( AffineMap permutationMap = getTransferMinorIdentityMap(
dest.getType().cast<ShapedType>(), vectorType); dest.getType().cast<ShapedType>(), vectorType);
build(builder, result, vector, dest, indices, permutationMap, inBounds); build(builder, result, vector, dest, indices, permutationMap, inBounds);
} }
/// Infers the mask type for a transfer write given its vector type and /// Infers the mask type for a transfer write given its vector type and
/// permutation map. The mask in a transfer read operation applies to the /// permutation map. The mask in a transfer read operation applies to the
▲ Show 20 Lines • Show All 216 Lines • ▼ Show 20 Lines if (succeeded(foldReadInitWrite(*this, operands, results)))
return success(); return success();
if (succeeded(foldWAR(*this, results))) if (succeeded(foldWAR(*this, results)))
return success(); return success();
if (succeeded(foldTransferInBoundsAttribute(*this))) if (succeeded(foldTransferInBoundsAttribute(*this)))
return success(); return success();
return memref::foldMemRefCast(*this); return memref::foldMemRefCast(*this);
} }
Optional<SmallVector<int64_t, 4>> TransferWriteOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>> TransferWriteOp::getShapeForUnroll() {
return llvm::to_vector<4>(getVectorType().getShape()); return llvm::to_vector<4>(getVectorType().getShape());
} }
void TransferWriteOp::getEffects( void TransferWriteOp::getEffects(
SmallVectorImpl<SideEffects::EffectInstance<MemoryEffects::Effect>> SmallVectorImpl<SideEffects::EffectInstance<MemoryEffects::Effect>>
&effects) { &effects) {
if (getShapedType().isa<MemRefType>()) if (getShapedType().isa<MemRefType>())
effects.emplace_back(MemoryEffects::Write::get(), getSource(), effects.emplace_back(MemoryEffects::Write::get(), getSource(),
▲ Show 20 Lines • Show All 974 Lines • ▼ Show 20 Lines if (seen[i])
return emitOpError("duplicate position index: ") << i; return emitOpError("duplicate position index: ") << i;
seen[i] = true; seen[i] = true;
if (resultType.getDimSize(ta.index()) != vectorType.getDimSize(i)) if (resultType.getDimSize(ta.index()) != vectorType.getDimSize(i))
return emitOpError("dimension size mismatch at: ") << i; return emitOpError("dimension size mismatch at: ") << i;
} }
return success(); return success();
} }
Optional<SmallVector<int64_t, 4>> TransposeOp::getShapeForUnroll() { std::optional<SmallVector<int64_t, 4>> TransposeOp::getShapeForUnroll() {
return llvm::to_vector<4>(getResultType().getShape()); return llvm::to_vector<4>(getResultType().getShape());
} }
namespace { namespace {
// Rewrites two back-to-back TransposeOp operations into a single TransposeOp. // Rewrites two back-to-back TransposeOp operations into a single TransposeOp.
class TransposeFolder final : public OpRewritePattern<vector::TransposeOp> { class TransposeFolder final : public OpRewritePattern<vector::TransposeOp> {
public: public:
▲ Show 20 Lines • Show All 526 Lines • ▼ Show 20 LinesParseResult WarpExecuteOnLane0Op::parse(OpAsmParser &parser,
// Parse the optional attribute list. // Parse the optional attribute list.
if (parser.parseOptionalAttrDict(result.attributes)) if (parser.parseOptionalAttrDict(result.attributes))
return failure(); return failure();
return success(); return success();
} }
void WarpExecuteOnLane0Op::getSuccessorRegions( void WarpExecuteOnLane0Op::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
if (index) { if (index) {
regions.push_back(RegionSuccessor(getResults())); regions.push_back(RegionSuccessor(getResults()));
return; return;
} }
// The warp region is always executed // The warp region is always executed
regions.push_back(RegionSuccessor(&getWarpRegion())); regions.push_back(RegionSuccessor(&getWarpRegion()));
▲ Show 20 Lines • Show All 149 Lines • Show Last 20 Lines

mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp

Show All 38 Lines
#include "llvm/Support/raw_ostream.h" #include "llvm/Support/raw_ostream.h"
#define DEBUG_TYPE "vector-to-vector" #define DEBUG_TYPE "vector-to-vector"
using namespace mlir; using namespace mlir;
using namespace mlir::vector; using namespace mlir::vector;
// Helper to find an index in an affine map. // Helper to find an index in an affine map.
static Optional<int64_t> getResultIndex(AffineMap map, int64_t index) { static std::optional<int64_t> getResultIndex(AffineMap map, int64_t index) {
for (int64_t i = 0, e = map.getNumResults(); i < e; ++i) { for (int64_t i = 0, e = map.getNumResults(); i < e; ++i) {
int64_t idx = map.getDimPosition(i); int64_t idx = map.getDimPosition(i);
if (idx == index) if (idx == index)
return i; return i;
} }
return std::nullopt; return std::nullopt;
} }
▲ Show 20 Lines • Show All 86 Lines • ▼ Show 20 Lines
template <typename IntType> template <typename IntType>
static SmallVector<IntType> extractVector(ArrayAttr arrayAttr) { static SmallVector<IntType> extractVector(ArrayAttr arrayAttr) {
return llvm::to_vector<4>(llvm::map_range( return llvm::to_vector<4>(llvm::map_range(
arrayAttr.getAsRange<IntegerAttr>(), arrayAttr.getAsRange<IntegerAttr>(),
[](IntegerAttr attr) { return static_cast<IntType>(attr.getInt()); })); [](IntegerAttr attr) { return static_cast<IntType>(attr.getInt()); }));
} }
/// Helper to create arithmetic operation associated with a kind of contraction. /// Helper to create arithmetic operation associated with a kind of contraction.
static Optional<Value> createContractArithOp(Location loc, Value x, Value y, static std::optional<Value> createContractArithOp(Location loc, Value x,
Value acc, Value y, Value acc,
vector::CombiningKind kind, vector::CombiningKind kind,
PatternRewriter &rewriter, PatternRewriter &rewriter,
bool isInt) { bool isInt) {
using vector::CombiningKind; using vector::CombiningKind;
Value mul; Value mul;
if (isInt) { if (isInt) {
if (kind == CombiningKind::MINF || kind == CombiningKind::MAXF) if (kind == CombiningKind::MINF || kind == CombiningKind::MAXF)
// Only valid for floating point types. // Only valid for floating point types.
return std::nullopt; return std::nullopt;
mul = rewriter.create<arith::MulIOp>(loc, x, y); mul = rewriter.create<arith::MulIOp>(loc, x, y);
} else { } else {
// Float case. // Float case.
if (kind == CombiningKind::AND || kind == CombiningKind::MINUI || if (kind == CombiningKind::AND || kind == CombiningKind::MINUI ||
kind == CombiningKind::MINSI || kind == CombiningKind::MAXUI || kind == CombiningKind::MINSI || kind == CombiningKind::MAXUI ||
kind == CombiningKind::MAXSI || kind == CombiningKind::OR || kind == CombiningKind::MAXSI || kind == CombiningKind::OR ||
kind == CombiningKind::XOR) kind == CombiningKind::XOR)
// Only valid for integer types. // Only valid for integer types.
return std::nullopt; return std::nullopt;
// Special case for fused multiply-add. // Special case for fused multiply-add.
if (acc && acc.getType().isa<VectorType>() && kind == CombiningKind::ADD) { if (acc && acc.getType().isa<VectorType>() && kind == CombiningKind::ADD) {
return Optional<Value>(rewriter.create<vector::FMAOp>(loc, x, y, acc)); return std::optional<Value>(
rewriter.create<vector::FMAOp>(loc, x, y, acc));
} }
mul = rewriter.create<arith::MulFOp>(loc, x, y); mul = rewriter.create<arith::MulFOp>(loc, x, y);
} }
if (!acc) if (!acc)
return Optional<Value>(mul); return std::optional<Value>(mul);
return makeArithReduction(rewriter, loc, kind, mul, acc); return makeArithReduction(rewriter, loc, kind, mul, acc);
} }
/// Return the positions of the reductions in the given map. /// Return the positions of the reductions in the given map.
static SmallVector<int64_t> getReductionIndex(AffineMap map, static SmallVector<int64_t> getReductionIndex(AffineMap map,
ArrayAttr iteratorTypes) { ArrayAttr iteratorTypes) {
SmallVector<int64_t> dimsIdx; SmallVector<int64_t> dimsIdx;
for (unsigned i = 0, e = map.getNumResults(); i < e; i++) { for (unsigned i = 0, e = map.getNumResults(); i < e; i++) {
if (isReductionIterator(iteratorTypes[map.getDimPosition(i)])) if (isReductionIterator(iteratorTypes[map.getDimPosition(i)]))
dimsIdx.push_back(i); dimsIdx.push_back(i);
} }
return dimsIdx; return dimsIdx;
} }
/// Look for a given dimension in an affine map and return its position. Return /// Look for a given dimension in an affine map and return its position. Return
/// std::nullopt if the dimension is not in the map results. /// std::nullopt if the dimension is not in the map results.
static llvm::Optional<unsigned> getDimPosition(AffineMap map, unsigned dim) { static std::optional<unsigned> getDimPosition(AffineMap map, unsigned dim) {
for (unsigned i = 0, e = map.getNumResults(); i < e; i++) { for (unsigned i = 0, e = map.getNumResults(); i < e; i++) {
if (map.getDimPosition(i) == dim) if (map.getDimPosition(i) == dim)
return i; return i;
} }
return std::nullopt; return std::nullopt;
} }
namespace { namespace {
▲ Show 20 Lines • Show All 344 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(vector::OuterProductOp op,
Type eltType = resType.getElementType(); Type eltType = resType.getElementType();
bool isInt = eltType.isa<IntegerType, IndexType>(); bool isInt = eltType.isa<IntegerType, IndexType>();
Value acc = (op.getAcc().empty()) ? nullptr : op.getAcc()[0]; Value acc = (op.getAcc().empty()) ? nullptr : op.getAcc()[0];
vector::CombiningKind kind = op.getKind(); vector::CombiningKind kind = op.getKind();
if (!rhsType) { if (!rhsType) {
// Special case: AXPY operation. // Special case: AXPY operation.
Value b = rewriter.create<vector::BroadcastOp>(loc, lhsType, op.getRhs()); Value b = rewriter.create<vector::BroadcastOp>(loc, lhsType, op.getRhs());
Optional<Value> mult = createContractArithOp(loc, op.getLhs(), b, acc, std::optional<Value> mult = createContractArithOp(
kind, rewriter, isInt); loc, op.getLhs(), b, acc, kind, rewriter, isInt);
if (!mult.has_value()) if (!mult.has_value())
return failure(); return failure();
rewriter.replaceOp(op, mult.value()); rewriter.replaceOp(op, mult.value());
return success(); return success();
} }
Value result = rewriter.create<arith::ConstantOp>( Value result = rewriter.create<arith::ConstantOp>(
loc, resType, rewriter.getZeroAttr(resType)); loc, resType, rewriter.getZeroAttr(resType));
for (int64_t d = 0, e = resType.getDimSize(0); d < e; ++d) { for (int64_t d = 0, e = resType.getDimSize(0); d < e; ++d) {
auto pos = rewriter.getI64ArrayAttr(d); auto pos = rewriter.getI64ArrayAttr(d);
Value x = Value x =
rewriter.create<vector::ExtractOp>(loc, eltType, op.getLhs(), pos); rewriter.create<vector::ExtractOp>(loc, eltType, op.getLhs(), pos);
Value a = rewriter.create<vector::BroadcastOp>(loc, rhsType, x); Value a = rewriter.create<vector::BroadcastOp>(loc, rhsType, x);
Value r = nullptr; Value r = nullptr;
if (acc) if (acc)
r = rewriter.create<vector::ExtractOp>(loc, rhsType, acc, pos); r = rewriter.create<vector::ExtractOp>(loc, rhsType, acc, pos);
Optional<Value> m = std::optional<Value> m =
createContractArithOp(loc, a, op.getRhs(), r, kind, rewriter, isInt); createContractArithOp(loc, a, op.getRhs(), r, kind, rewriter, isInt);
if (!m.has_value()) if (!m.has_value())
return failure(); return failure();
result = rewriter.create<vector::InsertOp>(loc, resType, m.value(), result = rewriter.create<vector::InsertOp>(loc, resType, m.value(),
result, pos); result, pos);
} }
rewriter.replaceOp(op, result); rewriter.replaceOp(op, result);
return success(); return success();
▲ Show 20 Lines • Show All 58 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
SmallVector<int64_t> rhsTranspose; SmallVector<int64_t> rhsTranspose;
for (int64_t dim : lhsReductionDims) for (int64_t dim : lhsReductionDims)
lhsTranspose.push_back(numLhsDimToBroadcast + dim); lhsTranspose.push_back(numLhsDimToBroadcast + dim);
for (int64_t dim : rhsReductionDims) for (int64_t dim : rhsReductionDims)
rhsTranspose.push_back(numRhsDimToBroadcast + dim); rhsTranspose.push_back(numRhsDimToBroadcast + dim);
// Loop through the parallel dimensions to calculate the dimensions to // Loop through the parallel dimensions to calculate the dimensions to
// broadcast and to permute in order to extract only parallel dimensions. // broadcast and to permute in order to extract only parallel dimensions.
for (unsigned i = 0; i < numParallelDims; i++) { for (unsigned i = 0; i < numParallelDims; i++) {
llvm::Optional<unsigned> lhsDim = std::optional<unsigned> lhsDim =
getDimPosition(lhsMap, accMap.getDimPosition(i)); getDimPosition(lhsMap, accMap.getDimPosition(i));
if (lhsDim) { if (lhsDim) {
lhsTranspose.push_back(numLhsDimToBroadcast + *lhsDim); lhsTranspose.push_back(numLhsDimToBroadcast + *lhsDim);
} else { } else {
// If the parallel dimension doesn't exist we will have to broadcast it. // If the parallel dimension doesn't exist we will have to broadcast it.
lhsDims.push_back( lhsDims.push_back(
contractOp.getResultType().cast<VectorType>().getDimSize(i)); contractOp.getResultType().cast<VectorType>().getDimSize(i));
lhsTranspose.push_back(lhsDims.size() - 1); lhsTranspose.push_back(lhsDims.size() - 1);
} }
llvm::Optional<unsigned> rhsDim = std::optional<unsigned> rhsDim =
getDimPosition(rhsMap, accMap.getDimPosition(i)); getDimPosition(rhsMap, accMap.getDimPosition(i));
if (rhsDim) { if (rhsDim) {
rhsTranspose.push_back(numRhsDimToBroadcast + *rhsDim); rhsTranspose.push_back(numRhsDimToBroadcast + *rhsDim);
} else { } else {
// If the parallel dimension doesn't exist we will have to broadcast it. // If the parallel dimension doesn't exist we will have to broadcast it.
rhsDims.push_back( rhsDims.push_back(
contractOp.getResultType().cast<VectorType>().getDimSize(i)); contractOp.getResultType().cast<VectorType>().getDimSize(i));
rhsTranspose.push_back(rhsDims.size() - 1); rhsTranspose.push_back(rhsDims.size() - 1);
Show All 18 Lines LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
newLhs = rewriter.create<vector::TransposeOp>(loc, newLhs, lhsTranspose); newLhs = rewriter.create<vector::TransposeOp>(loc, newLhs, lhsTranspose);
newRhs = rewriter.create<vector::TransposeOp>(loc, newRhs, rhsTranspose); newRhs = rewriter.create<vector::TransposeOp>(loc, newRhs, rhsTranspose);
SmallVector<int64_t> lhsOffsets(lhsReductionDims.size(), 0); SmallVector<int64_t> lhsOffsets(lhsReductionDims.size(), 0);
SmallVector<int64_t> rhsOffsets(rhsReductionDims.size(), 0); SmallVector<int64_t> rhsOffsets(rhsReductionDims.size(), 0);
newLhs = rewriter.create<vector::ExtractOp>( newLhs = rewriter.create<vector::ExtractOp>(
loc, newLhs, rewriter.getI64ArrayAttr(lhsOffsets)); loc, newLhs, rewriter.getI64ArrayAttr(lhsOffsets));
newRhs = rewriter.create<vector::ExtractOp>( newRhs = rewriter.create<vector::ExtractOp>(
loc, newRhs, rewriter.getI64ArrayAttr(rhsOffsets)); loc, newRhs, rewriter.getI64ArrayAttr(rhsOffsets));
Optional<Value> result = std::optional<Value> result =
createContractArithOp(loc, newLhs, newRhs, contractOp.getAcc(), createContractArithOp(loc, newLhs, newRhs, contractOp.getAcc(),
contractOp.getKind(), rewriter, isInt); contractOp.getKind(), rewriter, isInt);
rewriter.replaceOp(contractOp, {*result}); rewriter.replaceOp(contractOp, {*result});
return success(); return success();
} }
private: private:
/// Options to control the vector patterns. /// Options to control the vector patterns.
▲ Show 20 Lines • Show All 1,303 Lines • ▼ Show 20 LinesContractionOpLowering::lowerReduction(vector::ContractionOp op,
Type resType = op.getResultType(); Type resType = op.getResultType();
if (resType.isa<VectorType>()) if (resType.isa<VectorType>())
return rewriter.notifyMatchFailure(op, return rewriter.notifyMatchFailure(op,
"did not expect a VectorType result"); "did not expect a VectorType result");
bool isInt = resType.isa<IntegerType>(); bool isInt = resType.isa<IntegerType>();
// Use iterator index 0. // Use iterator index 0.
int64_t iterIndex = 0; int64_t iterIndex = 0;
SmallVector<AffineMap> iMap = op.getIndexingMapsArray(); SmallVector<AffineMap> iMap = op.getIndexingMapsArray();
Optional<int64_t> lookupLhs = getResultIndex(iMap[0], iterIndex); std::optional<int64_t> lookupLhs = getResultIndex(iMap[0], iterIndex);
Optional<int64_t> lookupRhs = getResultIndex(iMap[1], iterIndex); std::optional<int64_t> lookupRhs = getResultIndex(iMap[1], iterIndex);
if (!lookupLhs.has_value()) if (!lookupLhs.has_value())
return rewriter.notifyMatchFailure(op, [&](Diagnostic &diag) { return rewriter.notifyMatchFailure(op, [&](Diagnostic &diag) {
diag << "expected iterIndex=" << iterIndex << "to map to a LHS dimension"; diag << "expected iterIndex=" << iterIndex << "to map to a LHS dimension";
}); });
if (!lookupRhs.has_value()) if (!lookupRhs.has_value())
return rewriter.notifyMatchFailure(op, [&](Diagnostic &diag) { return rewriter.notifyMatchFailure(op, [&](Diagnostic &diag) {
diag << "expected iterIndex=" << iterIndex << "to map to a RHS dimension"; diag << "expected iterIndex=" << iterIndex << "to map to a RHS dimension";
}); });
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines
/// - Stride of most minor memref dimension must be 1. /// - Stride of most minor memref dimension must be 1.
/// - Out-of-bounds masking is not required. /// - Out-of-bounds masking is not required.
/// - If the memref's element type is a vector type then it coincides with the /// - If the memref's element type is a vector type then it coincides with the
/// result type. /// result type.
/// - The permutation map doesn't perform permutation (broadcasting is allowed). /// - The permutation map doesn't perform permutation (broadcasting is allowed).
struct TransferReadToVectorLoadLowering struct TransferReadToVectorLoadLowering
: public OpRewritePattern<vector::TransferReadOp> { : public OpRewritePattern<vector::TransferReadOp> {
TransferReadToVectorLoadLowering(MLIRContext *context, TransferReadToVectorLoadLowering(MLIRContext *context,
llvm::Optional<unsigned> maxRank, std::optional<unsigned> maxRank,
PatternBenefit benefit = 1) PatternBenefit benefit = 1)
: OpRewritePattern<vector::TransferReadOp>(context, benefit), : OpRewritePattern<vector::TransferReadOp>(context, benefit),
maxTransferRank(maxRank) {} maxTransferRank(maxRank) {}
LogicalResult matchAndRewrite(vector::TransferReadOp read, LogicalResult matchAndRewrite(vector::TransferReadOp read,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
if (maxTransferRank && read.getVectorType().getRank() > *maxTransferRank) if (maxTransferRank && read.getVectorType().getRank() > *maxTransferRank)
return failure(); return failure();
▲ Show 20 Lines • Show All 59 Lines • ▼ Show 20 Lines if (!broadcastedDims.empty()) {
read, read.getVectorType(), loadOp->getResult(0)); read, read.getVectorType(), loadOp->getResult(0));
} else { } else {
rewriter.replaceOp(read, loadOp->getResult(0)); rewriter.replaceOp(read, loadOp->getResult(0));
} }
return success(); return success();
} }
llvm::Optional<unsigned> maxTransferRank; std::optional<unsigned> maxTransferRank;
}; };
/// Replace a 0-d vector.load with a memref.load + vector.broadcast. /// Replace a 0-d vector.load with a memref.load + vector.broadcast.
// TODO: we shouldn't cross the vector/scalar domains just for this // TODO: we shouldn't cross the vector/scalar domains just for this
// but atm we lack the infra to avoid it. Possible solutions include: // but atm we lack the infra to avoid it. Possible solutions include:
// - go directly to LLVM + bitcast // - go directly to LLVM + bitcast
// - introduce a bitcast op and likely a new pointer dialect // - introduce a bitcast op and likely a new pointer dialect
// - let memref.load/store additionally support the 0-d vector case // - let memref.load/store additionally support the 0-d vector case
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Lines
/// - Out-of-bounds masking is not required. /// - Out-of-bounds masking is not required.
/// - If the memref's element type is a vector type then it coincides with the /// - If the memref's element type is a vector type then it coincides with the
/// type of the written value. /// type of the written value.
/// - The permutation map is the minor identity map (neither permutation nor /// - The permutation map is the minor identity map (neither permutation nor
/// broadcasting is allowed). /// broadcasting is allowed).
struct TransferWriteToVectorStoreLowering struct TransferWriteToVectorStoreLowering
: public OpRewritePattern<vector::TransferWriteOp> { : public OpRewritePattern<vector::TransferWriteOp> {
TransferWriteToVectorStoreLowering(MLIRContext *context, TransferWriteToVectorStoreLowering(MLIRContext *context,
llvm::Optional<unsigned> maxRank, std::optional<unsigned> maxRank,
PatternBenefit benefit = 1) PatternBenefit benefit = 1)
: OpRewritePattern<vector::TransferWriteOp>(context, benefit), : OpRewritePattern<vector::TransferWriteOp>(context, benefit),
maxTransferRank(maxRank) {} maxTransferRank(maxRank) {}
LogicalResult matchAndRewrite(vector::TransferWriteOp write, LogicalResult matchAndRewrite(vector::TransferWriteOp write,
PatternRewriter &rewriter) const override { PatternRewriter &rewriter) const override {
if (maxTransferRank && write.getVectorType().getRank() > *maxTransferRank) if (maxTransferRank && write.getVectorType().getRank() > *maxTransferRank)
return rewriter.notifyMatchFailure(write.getLoc(), [=](Diagnostic &diag) { return rewriter.notifyMatchFailure(write.getLoc(), [=](Diagnostic &diag) {
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 Lines if (write.getMask()) {
write.getVector()); write.getVector());
} else { } else {
rewriter.replaceOpWithNewOp<vector::StoreOp>( rewriter.replaceOpWithNewOp<vector::StoreOp>(
write, write.getVector(), write.getSource(), write.getIndices()); write, write.getVector(), write.getSource(), write.getIndices());
} }
return success(); return success();
} }
llvm::Optional<unsigned> maxTransferRank; std::optional<unsigned> maxTransferRank;
}; };
// Returns the values in `arrayAttr` as an integer vector. // Returns the values in `arrayAttr` as an integer vector.
static SmallVector<int64_t> getIntValueVector(ArrayAttr arrayAttr) { static SmallVector<int64_t> getIntValueVector(ArrayAttr arrayAttr) {
return llvm::to_vector<4>( return llvm::to_vector<4>(
llvm::map_range(arrayAttr.getAsRange<IntegerAttr>(), llvm::map_range(arrayAttr.getAsRange<IntegerAttr>(),
[](IntegerAttr attr) { return attr.getInt(); })); [](IntegerAttr attr) { return attr.getInt(); }));
} }
▲ Show 20 Lines • Show All 729 Lines • ▼ Show 20 Lines
void mlir::vector:: void mlir::vector::
populateVectorTransferCollapseInnerMostContiguousDimsPatterns( populateVectorTransferCollapseInnerMostContiguousDimsPatterns(
RewritePatternSet &patterns, PatternBenefit benefit) { RewritePatternSet &patterns, PatternBenefit benefit) {
patterns.add<DropInnerMostUnitDims>(patterns.getContext(), benefit); patterns.add<DropInnerMostUnitDims>(patterns.getContext(), benefit);
} }
void mlir::vector::populateVectorTransferLoweringPatterns( void mlir::vector::populateVectorTransferLoweringPatterns(
RewritePatternSet &patterns, llvm::Optional<unsigned> maxTransferRank, RewritePatternSet &patterns, std::optional<unsigned> maxTransferRank,
PatternBenefit benefit) { PatternBenefit benefit) {
patterns.add<TransferReadToVectorLoadLowering, patterns.add<TransferReadToVectorLoadLowering,
TransferWriteToVectorStoreLowering>(patterns.getContext(), TransferWriteToVectorStoreLowering>(patterns.getContext(),
maxTransferRank, benefit); maxTransferRank, benefit);
patterns patterns
.add<VectorLoadToMemrefLoadLowering, VectorStoreToMemrefStoreLowering>( .add<VectorLoadToMemrefLoadLowering, VectorStoreToMemrefStoreLowering>(
patterns.getContext(), benefit); patterns.getContext(), benefit);
} }
void mlir::vector::populateVectorScanLoweringPatterns( void mlir::vector::populateVectorScanLoweringPatterns(
RewritePatternSet &patterns, PatternBenefit benefit) { RewritePatternSet &patterns, PatternBenefit benefit) {
patterns.add<ScanToArithOps>(patterns.getContext(), benefit); patterns.add<ScanToArithOps>(patterns.getContext(), benefit);
} }

mlir/lib/IR/BuiltinDialect.cpp

Show First 20 Lines • Show All 120 Lines • ▼ Show 20 Lines#include "mlir/IR/BuiltinOps.cpp.inc"
builtin_dialect_detail::addBytecodeInterface(this); builtin_dialect_detail::addBytecodeInterface(this);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// ModuleOp // ModuleOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void ModuleOp::build(OpBuilder &builder, OperationState &state, void ModuleOp::build(OpBuilder &builder, OperationState &state,
Optional<StringRef> name) { std::optional<StringRef> name) {
state.addRegion()->emplaceBlock(); state.addRegion()->emplaceBlock();
if (name) { if (name) {
state.attributes.push_back(builder.getNamedAttr( state.attributes.push_back(builder.getNamedAttr(
mlir::SymbolTable::getSymbolAttrName(), builder.getStringAttr(*name))); mlir::SymbolTable::getSymbolAttrName(), builder.getStringAttr(*name)));
} }
} }
/// Construct a module from the given context. /// Construct a module from the given context.
ModuleOp ModuleOp::create(Location loc, Optional<StringRef> name) { ModuleOp ModuleOp::create(Location loc, std::optional<StringRef> name) {
OpBuilder builder(loc->getContext()); OpBuilder builder(loc->getContext());
return builder.create<ModuleOp>(loc, name); return builder.create<ModuleOp>(loc, name);
} }
DataLayoutSpecInterface ModuleOp::getDataLayoutSpec() { DataLayoutSpecInterface ModuleOp::getDataLayoutSpec() {
// Take the first and only (if present) attribute that implements the // Take the first and only (if present) attribute that implements the
// interface. This needs a linear search, but is called only once per data // interface. This needs a linear search, but is called only once per data
// layout object construction that is used for repeated queries. // layout object construction that is used for repeated queries.
▲ Show 20 Lines • Show All 85 Lines • Show Last 20 Lines

mlir/lib/IR/BuiltinTypes.cpp

Show First 20 Lines • Show All 240 Lines • ▼ Show 20 Lines if (auto et = getElementType().dyn_cast<IntegerType>())
if (auto scaledEt = et.scaleElementBitwidth(scale)) if (auto scaledEt = et.scaleElementBitwidth(scale))
return VectorType::get(getShape(), scaledEt, getNumScalableDims()); return VectorType::get(getShape(), scaledEt, getNumScalableDims());
if (auto et = getElementType().dyn_cast<FloatType>()) if (auto et = getElementType().dyn_cast<FloatType>())
if (auto scaledEt = et.scaleElementBitwidth(scale)) if (auto scaledEt = et.scaleElementBitwidth(scale))
return VectorType::get(getShape(), scaledEt, getNumScalableDims()); return VectorType::get(getShape(), scaledEt, getNumScalableDims());
return VectorType(); return VectorType();
} }
VectorType VectorType::cloneWith(Optional<ArrayRef<int64_t>> shape, VectorType VectorType::cloneWith(std::optional<ArrayRef<int64_t>> shape,
Type elementType) const { Type elementType) const {
return VectorType::get(shape.value_or(getShape()), elementType, return VectorType::get(shape.value_or(getShape()), elementType,
getNumScalableDims()); getNumScalableDims());
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// TensorType // TensorType
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Type TensorType::getElementType() const { Type TensorType::getElementType() const {
return llvm::TypeSwitch<TensorType, Type>(*this) return llvm::TypeSwitch<TensorType, Type>(*this)
.Case<RankedTensorType, UnrankedTensorType>( .Case<RankedTensorType, UnrankedTensorType>(
[](auto type) { return type.getElementType(); }); [](auto type) { return type.getElementType(); });
} }
bool TensorType::hasRank() const { return !isa<UnrankedTensorType>(); } bool TensorType::hasRank() const { return !isa<UnrankedTensorType>(); }
ArrayRef<int64_t> TensorType::getShape() const { ArrayRef<int64_t> TensorType::getShape() const {
return cast<RankedTensorType>().getShape(); return cast<RankedTensorType>().getShape();
} }
TensorType TensorType::cloneWith(Optional<ArrayRef<int64_t>> shape, TensorType TensorType::cloneWith(std::optional<ArrayRef<int64_t>> shape,
Type elementType) const { Type elementType) const {
if (auto unrankedTy = dyn_cast<UnrankedTensorType>()) { if (auto unrankedTy = dyn_cast<UnrankedTensorType>()) {
if (shape) if (shape)
return RankedTensorType::get(*shape, elementType); return RankedTensorType::get(*shape, elementType);
return UnrankedTensorType::get(elementType); return UnrankedTensorType::get(elementType);
} }
auto rankedTy = cast<RankedTensorType>(); auto rankedTy = cast<RankedTensorType>();
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines
} }
bool BaseMemRefType::hasRank() const { return !isa<UnrankedMemRefType>(); } bool BaseMemRefType::hasRank() const { return !isa<UnrankedMemRefType>(); }
ArrayRef<int64_t> BaseMemRefType::getShape() const { ArrayRef<int64_t> BaseMemRefType::getShape() const {
return cast<MemRefType>().getShape(); return cast<MemRefType>().getShape();
} }
BaseMemRefType BaseMemRefType::cloneWith(Optional<ArrayRef<int64_t>> shape, BaseMemRefType BaseMemRefType::cloneWith(std::optional<ArrayRef<int64_t>> shape,
Type elementType) const { Type elementType) const {
if (auto unrankedTy = dyn_cast<UnrankedMemRefType>()) { if (auto unrankedTy = dyn_cast<UnrankedMemRefType>()) {
if (!shape) if (!shape)
return UnrankedMemRefType::get(elementType, getMemorySpace()); return UnrankedMemRefType::get(elementType, getMemorySpace());
MemRefType::Builder builder(*shape, elementType); MemRefType::Builder builder(*shape, elementType);
builder.setMemorySpace(getMemorySpace()); builder.setMemorySpace(getMemorySpace());
return builder; return builder;
} }
Show All 24 Lines
/// Given an `originalShape` and a `reducedShape` assumed to be a subset of /// Given an `originalShape` and a `reducedShape` assumed to be a subset of
/// `originalShape` with some `1` entries erased, return the set of indices /// `originalShape` with some `1` entries erased, return the set of indices
/// that specifies which of the entries of `originalShape` are dropped to obtain /// that specifies which of the entries of `originalShape` are dropped to obtain
/// `reducedShape`. The returned mask can be applied as a projection to /// `reducedShape`. The returned mask can be applied as a projection to
/// `originalShape` to obtain the `reducedShape`. This mask is useful to track /// `originalShape` to obtain the `reducedShape`. This mask is useful to track
/// which dimensions must be kept when e.g. compute MemRef strides under /// which dimensions must be kept when e.g. compute MemRef strides under
/// rank-reducing operations. Return std::nullopt if reducedShape cannot be /// rank-reducing operations. Return std::nullopt if reducedShape cannot be
/// obtained by dropping only `1` entries in `originalShape`. /// obtained by dropping only `1` entries in `originalShape`.
llvm::Optional<llvm::SmallDenseSet<unsigned>> std::optional<llvm::SmallDenseSet<unsigned>>
mlir::computeRankReductionMask(ArrayRef<int64_t> originalShape, mlir::computeRankReductionMask(ArrayRef<int64_t> originalShape,
ArrayRef<int64_t> reducedShape) { ArrayRef<int64_t> reducedShape) {
size_t originalRank = originalShape.size(), reducedRank = reducedShape.size(); size_t originalRank = originalShape.size(), reducedRank = reducedShape.size();
llvm::SmallDenseSet<unsigned> unusedDims; llvm::SmallDenseSet<unsigned> unusedDims;
unsigned reducedIdx = 0; unsigned reducedIdx = 0;
for (unsigned originalIdx = 0; originalIdx < originalRank; ++originalIdx) { for (unsigned originalIdx = 0; originalIdx < originalRank; ++originalIdx) {
// Greedily insert `originalIdx` if match. // Greedily insert `originalIdx` if match.
if (reducedIdx < reducedRank && if (reducedIdx < reducedRank &&
▲ Show 20 Lines • Show All 536 Lines • Show Last 20 Lines

mlir/lib/IR/Dialect.cpp

Show First 20 Lines • Show All 69 Lines • ▼ Show 20 Lines if (allowsUnknownTypes()) {
return OpaqueType::get(ns, parser.getFullSymbolSpec()); return OpaqueType::get(ns, parser.getFullSymbolSpec());
} }
parser.emitError(parser.getNameLoc()) parser.emitError(parser.getNameLoc())
<< "dialect '" << getNamespace() << "' provides no type parsing hook"; << "dialect '" << getNamespace() << "' provides no type parsing hook";
return Type(); return Type();
} }
Optional<Dialect::ParseOpHook> std::optional<Dialect::ParseOpHook>
Dialect::getParseOperationHook(StringRef opName) const { Dialect::getParseOperationHook(StringRef opName) const {
return std::nullopt; return std::nullopt;
} }
llvm::unique_function<void(Operation *, OpAsmPrinter &printer)> llvm::unique_function<void(Operation *, OpAsmPrinter &printer)>
Dialect::getOperationPrinter(Operation *op) const { Dialect::getOperationPrinter(Operation *op) const {
assert(op->getDialect() == this && assert(op->getDialect() == this &&
"Dialect hook invoked on non-dialect owned operation"); "Dialect hook invoked on non-dialect owned operation");
▲ Show 20 Lines • Show All 176 Lines • Show Last 20 Lines

mlir/lib/Interfaces/ControlFlowInterfaces.cpp

Show All 30 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// BranchOpInterface // BranchOpInterface
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Returns the `BlockArgument` corresponding to operand `operandIndex` in some /// Returns the `BlockArgument` corresponding to operand `operandIndex` in some
/// successor if 'operandIndex' is within the range of 'operands', or /// successor if 'operandIndex' is within the range of 'operands', or
/// std::nullopt if `operandIndex` isn't a successor operand index. /// std::nullopt if `operandIndex` isn't a successor operand index.
Optional<BlockArgument> std::optional<BlockArgument>
detail::getBranchSuccessorArgument(const SuccessorOperands &operands, detail::getBranchSuccessorArgument(const SuccessorOperands &operands,
unsigned operandIndex, Block *successor) { unsigned operandIndex, Block *successor) {
OperandRange forwardedOperands = operands.getForwardedOperands(); OperandRange forwardedOperands = operands.getForwardedOperands();
// Check that the operands are valid. // Check that the operands are valid.
if (forwardedOperands.empty()) if (forwardedOperands.empty())
return std::nullopt; return std::nullopt;
// Check to ensure that this operand is within the range. // Check to ensure that this operand is within the range.
Show All 37 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
/// Verify that types match along all region control flow edges originating from /// Verify that types match along all region control flow edges originating from
/// `sourceNo` (region # if source is a region, std::nullopt if source is parent /// `sourceNo` (region # if source is a region, std::nullopt if source is parent
/// op). `getInputsTypesForRegion` is a function that returns the types of the /// op). `getInputsTypesForRegion` is a function that returns the types of the
/// inputs that flow from `sourceIndex' to the given region, or std::nullopt if /// inputs that flow from `sourceIndex' to the given region, or std::nullopt if
/// the exact type match verification is not necessary (e.g., if the Op verifies /// the exact type match verification is not necessary (e.g., if the Op verifies
/// the match itself). /// the match itself).
static LogicalResult static LogicalResult verifyTypesAlongAllEdges(
verifyTypesAlongAllEdges(Operation *op, Optional<unsigned> sourceNo, Operation *op, std::optional<unsigned> sourceNo,
function_ref<Optional<TypeRange>(Optional<unsigned>)> function_ref<std::optional<TypeRange>(std::optional<unsigned>)>
getInputsTypesForRegion) { getInputsTypesForRegion) {
auto regionInterface = cast<RegionBranchOpInterface>(op); auto regionInterface = cast<RegionBranchOpInterface>(op);
SmallVector<RegionSuccessor, 2> successors; SmallVector<RegionSuccessor, 2> successors;
regionInterface.getSuccessorRegions(sourceNo, successors); regionInterface.getSuccessorRegions(sourceNo, successors);
for (RegionSuccessor &succ : successors) { for (RegionSuccessor &succ : successors) {
Optional<unsigned> succRegionNo; std::optional<unsigned> succRegionNo;
if (!succ.isParent()) if (!succ.isParent())
succRegionNo = succ.getSuccessor()->getRegionNumber(); succRegionNo = succ.getSuccessor()->getRegionNumber();
auto printEdgeName = [&](InFlightDiagnostic &diag) -> InFlightDiagnostic & { auto printEdgeName = [&](InFlightDiagnostic &diag) -> InFlightDiagnostic & {
diag << "from "; diag << "from ";
if (sourceNo) if (sourceNo)
diag << "Region #" << sourceNo.value(); diag << "Region #" << sourceNo.value();
else else
diag << "parent operands"; diag << "parent operands";
diag << " to "; diag << " to ";
if (succRegionNo) if (succRegionNo)
diag << "Region #" << succRegionNo.value(); diag << "Region #" << succRegionNo.value();
else else
diag << "parent results"; diag << "parent results";
return diag; return diag;
}; };
Optional<TypeRange> sourceTypes = getInputsTypesForRegion(succRegionNo); std::optional<TypeRange> sourceTypes =
getInputsTypesForRegion(succRegionNo);
if (!sourceTypes.has_value()) if (!sourceTypes.has_value())
continue; continue;
TypeRange succInputsTypes = succ.getSuccessorInputs().getTypes(); TypeRange succInputsTypes = succ.getSuccessorInputs().getTypes();
if (sourceTypes->size() != succInputsTypes.size()) { if (sourceTypes->size() != succInputsTypes.size()) {
InFlightDiagnostic diag = op->emitOpError(" region control flow edge "); InFlightDiagnostic diag = op->emitOpError(" region control flow edge ");
return printEdgeName(diag) << ": source has " << sourceTypes->size() return printEdgeName(diag) << ": source has " << sourceTypes->size()
<< " operands, but target successor needs " << " operands, but target successor needs "
Show All 15 Linesstatic LogicalResult verifyTypesAlongAllEdges(
} }
return success(); return success();
} }
/// Verify that types match along control flow edges described the given op. /// Verify that types match along control flow edges described the given op.
LogicalResult detail::verifyTypesAlongControlFlowEdges(Operation *op) { LogicalResult detail::verifyTypesAlongControlFlowEdges(Operation *op) {
auto regionInterface = cast<RegionBranchOpInterface>(op); auto regionInterface = cast<RegionBranchOpInterface>(op);
auto inputTypesFromParent = [&](Optional<unsigned> regionNo) -> TypeRange { auto inputTypesFromParent =
[&](std::optional<unsigned> regionNo) -> TypeRange {
return regionInterface.getSuccessorEntryOperands(regionNo).getTypes(); return regionInterface.getSuccessorEntryOperands(regionNo).getTypes();
}; };
// Verify types along control flow edges originating from the parent. // Verify types along control flow edges originating from the parent.
if (failed(verifyTypesAlongAllEdges(op, std::nullopt, inputTypesFromParent))) if (failed(verifyTypesAlongAllEdges(op, std::nullopt, inputTypesFromParent)))
return failure(); return failure();
auto areTypesCompatible = [&](TypeRange lhs, TypeRange rhs) { auto areTypesCompatible = [&](TypeRange lhs, TypeRange rhs) {
Show All 11 LinesLogicalResult detail::verifyTypesAlongControlFlowEdges(Operation *op) {
// Verify types along control flow edges originating from each region. // Verify types along control flow edges originating from each region.
for (unsigned regionNo : llvm::seq(0U, op->getNumRegions())) { for (unsigned regionNo : llvm::seq(0U, op->getNumRegions())) {
Region &region = op->getRegion(regionNo); Region &region = op->getRegion(regionNo);
// Since there can be multiple `ReturnLike` terminators or others // Since there can be multiple `ReturnLike` terminators or others
// implementing the `RegionBranchTerminatorOpInterface`, all should have the // implementing the `RegionBranchTerminatorOpInterface`, all should have the
// same operand types when passing them to the same region. // same operand types when passing them to the same region.
Optional<OperandRange> regionReturnOperands; std::optional<OperandRange> regionReturnOperands;
for (Block &block : region) { for (Block &block : region) {
Operation *terminator = block.getTerminator(); Operation *terminator = block.getTerminator();
auto terminatorOperands = auto terminatorOperands =
getRegionBranchSuccessorOperands(terminator, regionNo); getRegionBranchSuccessorOperands(terminator, regionNo);
if (!terminatorOperands) if (!terminatorOperands)
continue; continue;
if (!regionReturnOperands) { if (!regionReturnOperands) {
regionReturnOperands = terminatorOperands; regionReturnOperands = terminatorOperands;
continue; continue;
} }
// Found more than one ReturnLike terminator. Make sure the operand types // Found more than one ReturnLike terminator. Make sure the operand types
// match with the first one. // match with the first one.
if (!areTypesCompatible(regionReturnOperands->getTypes(), if (!areTypesCompatible(regionReturnOperands->getTypes(),
terminatorOperands->getTypes())) terminatorOperands->getTypes()))
return op->emitOpError("Region #") return op->emitOpError("Region #")
<< regionNo << regionNo
<< " operands mismatch between return-like terminators"; << " operands mismatch between return-like terminators";
} }
auto inputTypesFromRegion = auto inputTypesFromRegion =
[&](Optional<unsigned> regionNo) -> Optional<TypeRange> { [&](std::optional<unsigned> regionNo) -> std::optional<TypeRange> {
// If there is no return-like terminator, the op itself should verify // If there is no return-like terminator, the op itself should verify
// type consistency. // type consistency.
if (!regionReturnOperands) if (!regionReturnOperands)
return std::nullopt; return std::nullopt;
// All successors get the same set of operand types. // All successors get the same set of operand types.
return TypeRange(regionReturnOperands->getTypes()); return TypeRange(regionReturnOperands->getTypes());
}; };
▲ Show 20 Lines • Show All 88 Lines • ▼ Show 20 Lines
} }
bool RegionBranchOpInterface::isRepetitiveRegion(unsigned index) { bool RegionBranchOpInterface::isRepetitiveRegion(unsigned index) {
Region *region = &getOperation()->getRegion(index); Region *region = &getOperation()->getRegion(index);
return isRegionReachable(region, region); return isRegionReachable(region, region);
} }
void RegionBranchOpInterface::getSuccessorRegions( void RegionBranchOpInterface::getSuccessorRegions(
Optional<unsigned> index, SmallVectorImpl<RegionSuccessor> &regions) { std::optional<unsigned> index, SmallVectorImpl<RegionSuccessor> &regions) {
unsigned numInputs = 0; unsigned numInputs = 0;
if (index) { if (index) {
// If the predecessor is a region, get the number of operands from an // If the predecessor is a region, get the number of operands from an
// exiting terminator in the region. // exiting terminator in the region.
for (Block &block : getOperation()->getRegion(*index)) { for (Block &block : getOperation()->getRegion(*index)) {
Operation *terminator = block.getTerminator(); Operation *terminator = block.getTerminator();
if (getRegionBranchSuccessorOperands(terminator, *index)) { if (getRegionBranchSuccessorOperands(terminator, *index)) {
numInputs = terminator->getNumOperands(); numInputs = terminator->getNumOperands();
▲ Show 20 Lines • Show All 43 Lines • ▼ Show 20 Lines
/// Returns the mutable operands that are passed to the region with the given /// Returns the mutable operands that are passed to the region with the given
/// `regionIndex`. If the operation does not implement the /// `regionIndex`. If the operation does not implement the
/// `RegionBranchTerminatorOpInterface` and is not marked as `ReturnLike`, the /// `RegionBranchTerminatorOpInterface` and is not marked as `ReturnLike`, the
/// result will be `std::nullopt`. In all other cases, the resulting /// result will be `std::nullopt`. In all other cases, the resulting
/// `OperandRange` represents all operands that are passed to the specified /// `OperandRange` represents all operands that are passed to the specified
/// successor region. If `regionIndex` is `std::nullopt`, all operands that are /// successor region. If `regionIndex` is `std::nullopt`, all operands that are
/// passed to the parent operation will be returned. /// passed to the parent operation will be returned.
Optional<MutableOperandRange> std::optional<MutableOperandRange>
mlir::getMutableRegionBranchSuccessorOperands(Operation *operation, mlir::getMutableRegionBranchSuccessorOperands(
Optional<unsigned> regionIndex) { Operation *operation, std::optional<unsigned> regionIndex) {
// Try to query a RegionBranchTerminatorOpInterface to determine // Try to query a RegionBranchTerminatorOpInterface to determine
// all successor operands that will be passed to the successor // all successor operands that will be passed to the successor
// input arguments. // input arguments.
if (auto regionTerminatorInterface = if (auto regionTerminatorInterface =
dyn_cast<RegionBranchTerminatorOpInterface>(operation)) dyn_cast<RegionBranchTerminatorOpInterface>(operation))
return regionTerminatorInterface.getMutableSuccessorOperands(regionIndex); return regionTerminatorInterface.getMutableSuccessorOperands(regionIndex);
// TODO: The ReturnLike trait should imply a default implementation of the // TODO: The ReturnLike trait should imply a default implementation of the
// RegionBranchTerminatorOpInterface. This would make this code significantly // RegionBranchTerminatorOpInterface. This would make this code significantly
// easier. Furthermore, this may even make this function obsolete. // easier. Furthermore, this may even make this function obsolete.
if (operation->hasTrait<OpTrait::ReturnLike>()) if (operation->hasTrait<OpTrait::ReturnLike>())
return MutableOperandRange(operation); return MutableOperandRange(operation);
return std::nullopt; return std::nullopt;
} }
/// Returns the read only operands that are passed to the region with the given /// Returns the read only operands that are passed to the region with the given
/// `regionIndex`. See `getMutableRegionBranchSuccessorOperands` for more /// `regionIndex`. See `getMutableRegionBranchSuccessorOperands` for more
/// information. /// information.
Optional<OperandRange> std::optional<OperandRange>
mlir::getRegionBranchSuccessorOperands(Operation *operation, mlir::getRegionBranchSuccessorOperands(Operation *operation,
Optional<unsigned> regionIndex) { std::optional<unsigned> regionIndex) {
auto range = getMutableRegionBranchSuccessorOperands(operation, regionIndex); auto range = getMutableRegionBranchSuccessorOperands(operation, regionIndex);
return range ? Optional<OperandRange>(*range) : std::nullopt; return range ? std::optional<OperandRange>(*range) : std::nullopt;
} }

mlir/lib/Interfaces/InferTypeOpInterface.cpp

Show First 20 Lines • Show All 168 Lines • ▼ Show 20 Lines
ShapeAdaptor ValueShapeRange::getShape(int index) const { ShapeAdaptor ValueShapeRange::getShape(int index) const {
if (index < 0 || static_cast<size_t>(index) >= size()) if (index < 0 || static_cast<size_t>(index) >= size())
return nullptr; return nullptr;
return getShape(operator[](index)); return getShape(operator[](index));
} }
LogicalResult mlir::detail::inferReturnTensorTypes( LogicalResult mlir::detail::inferReturnTensorTypes(
function_ref<LogicalResult( function_ref<
MLIRContext *, Optional<Location> location, ValueShapeRange operands, LogicalResult(MLIRContext *, std::optional<Location> location,
DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes,
RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &retComponents)> SmallVectorImpl<ShapedTypeComponents> &retComponents)>
componentTypeFn, componentTypeFn,
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
SmallVector<ShapedTypeComponents, 2> retComponents; SmallVector<ShapedTypeComponents, 2> retComponents;
if (failed(componentTypeFn(context, location, operands, attributes, regions, if (failed(componentTypeFn(context, location, operands, attributes, regions,
retComponents))) retComponents)))
return failure(); return failure();
for (const auto &shapeAndType : retComponents) { for (const auto &shapeAndType : retComponents) {
Type element_ty = shapeAndType.getElementType(); Type element_ty = shapeAndType.getElementType();
Show All 21 Lines

mlir/lib/Rewrite/ByteCode.cpp

Show All 40 LinesPDLByteCodePattern PDLByteCodePattern::create(pdl_interp::RecordMatchOp matchOp,
// Collect the set of generated operations. // Collect the set of generated operations.
SmallVector<StringRef, 8> generatedOps; SmallVector<StringRef, 8> generatedOps;
if (ArrayAttr generatedOpsAttr = matchOp.getGeneratedOpsAttr()) if (ArrayAttr generatedOpsAttr = matchOp.getGeneratedOpsAttr())
generatedOps = generatedOps =
llvm::to_vector<8>(generatedOpsAttr.getAsValueRange<StringAttr>()); llvm::to_vector<8>(generatedOpsAttr.getAsValueRange<StringAttr>());
// Check to see if this is pattern matches a specific operation type. // Check to see if this is pattern matches a specific operation type.
if (Optional<StringRef> rootKind = matchOp.getRootKind()) if (std::optional<StringRef> rootKind = matchOp.getRootKind())
return PDLByteCodePattern(rewriterAddr, configSet, *rootKind, benefit, ctx, return PDLByteCodePattern(rewriterAddr, configSet, *rootKind, benefit, ctx,
generatedOps); generatedOps);
return PDLByteCodePattern(rewriterAddr, configSet, MatchAnyOpTypeTag(), return PDLByteCodePattern(rewriterAddr, configSet, MatchAnyOpTypeTag(),
benefit, ctx, generatedOps); benefit, ctx, generatedOps);
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// PDLByteCodeMutableState // PDLByteCodeMutableState
▲ Show 20 Lines • Show All 877 Lines • ▼ Show 20 Linesvoid Generator::generate(pdl_interp::GetOperandOp op, ByteCodeWriter &writer) {
if (index < 4) if (index < 4)
writer.append(static_cast<OpCode>(OpCode::GetOperand0 + index)); writer.append(static_cast<OpCode>(OpCode::GetOperand0 + index));
else else
writer.append(OpCode::GetOperandN, index); writer.append(OpCode::GetOperandN, index);
writer.append(op.getInputOp(), op.getValue()); writer.append(op.getInputOp(), op.getValue());
} }
void Generator::generate(pdl_interp::GetOperandsOp op, ByteCodeWriter &writer) { void Generator::generate(pdl_interp::GetOperandsOp op, ByteCodeWriter &writer) {
Value result = op.getValue(); Value result = op.getValue();
Optional<uint32_t> index = op.getIndex(); std::optional<uint32_t> index = op.getIndex();
writer.append(OpCode::GetOperands, writer.append(OpCode::GetOperands,
index.value_or(std::numeric_limits<uint32_t>::max()), index.value_or(std::numeric_limits<uint32_t>::max()),
op.getInputOp()); op.getInputOp());
if (result.getType().isa<pdl::RangeType>()) if (result.getType().isa<pdl::RangeType>())
writer.append(getRangeStorageIndex(result)); writer.append(getRangeStorageIndex(result));
else else
writer.append(std::numeric_limits<ByteCodeField>::max()); writer.append(std::numeric_limits<ByteCodeField>::max());
writer.append(result); writer.append(result);
} }
void Generator::generate(pdl_interp::GetResultOp op, ByteCodeWriter &writer) { void Generator::generate(pdl_interp::GetResultOp op, ByteCodeWriter &writer) {
uint32_t index = op.getIndex(); uint32_t index = op.getIndex();
if (index < 4) if (index < 4)
writer.append(static_cast<OpCode>(OpCode::GetResult0 + index)); writer.append(static_cast<OpCode>(OpCode::GetResult0 + index));
else else
writer.append(OpCode::GetResultN, index); writer.append(OpCode::GetResultN, index);
writer.append(op.getInputOp(), op.getValue()); writer.append(op.getInputOp(), op.getValue());
} }
void Generator::generate(pdl_interp::GetResultsOp op, ByteCodeWriter &writer) { void Generator::generate(pdl_interp::GetResultsOp op, ByteCodeWriter &writer) {
Value result = op.getValue(); Value result = op.getValue();
Optional<uint32_t> index = op.getIndex(); std::optional<uint32_t> index = op.getIndex();
writer.append(OpCode::GetResults, writer.append(OpCode::GetResults,
index.value_or(std::numeric_limits<uint32_t>::max()), index.value_or(std::numeric_limits<uint32_t>::max()),
op.getInputOp()); op.getInputOp());
if (result.getType().isa<pdl::RangeType>()) if (result.getType().isa<pdl::RangeType>())
writer.append(getRangeStorageIndex(result)); writer.append(getRangeStorageIndex(result));
else else
writer.append(std::numeric_limits<ByteCodeField>::max()); writer.append(std::numeric_limits<ByteCodeField>::max());
writer.append(result); writer.append(result);
▲ Show 20 Lines • Show All 1,356 Lines • Show Last 20 Lines

mlir/lib/Target/LLVMIR/DebugImporter.cpp

Show All 36 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
DIBasicTypeAttr DebugImporter::translateImpl(llvm::DIBasicType *node) { DIBasicTypeAttr DebugImporter::translateImpl(llvm::DIBasicType *node) {
return DIBasicTypeAttr::get(context, node->getTag(), node->getName(), return DIBasicTypeAttr::get(context, node->getTag(), node->getName(),
node->getSizeInBits(), node->getEncoding()); node->getSizeInBits(), node->getEncoding());
} }
DICompileUnitAttr DebugImporter::translateImpl(llvm::DICompileUnit *node) { DICompileUnitAttr DebugImporter::translateImpl(llvm::DICompileUnit *node) {
Optional<DIEmissionKind> emissionKind = std::optional<DIEmissionKind> emissionKind =
symbolizeDIEmissionKind(node->getEmissionKind()); symbolizeDIEmissionKind(node->getEmissionKind());
return DICompileUnitAttr::get(context, node->getSourceLanguage(), return DICompileUnitAttr::get(context, node->getSourceLanguage(),
translate(node->getFile()), translate(node->getFile()),
StringAttr::get(context, node->getProducer()), StringAttr::get(context, node->getProducer()),
node->isOptimized(), emissionKind.value()); node->isOptimized(), emissionKind.value());
} }
DICompositeTypeAttr DebugImporter::translateImpl(llvm::DICompositeType *node) { DICompositeTypeAttr DebugImporter::translateImpl(llvm::DICompositeType *node) {
Optional<DIFlags> flags = symbolizeDIFlags(node->getFlags()); std::optional<DIFlags> flags = symbolizeDIFlags(node->getFlags());
SmallVector<DINodeAttr> elements; SmallVector<DINodeAttr> elements;
for (llvm::DINode *element : node->getElements()) { for (llvm::DINode *element : node->getElements()) {
assert(element && "expected a non-null element type"); assert(element && "expected a non-null element type");
elements.push_back(translate(element)); elements.push_back(translate(element));
} }
return DICompositeTypeAttr::get( return DICompositeTypeAttr::get(
context, node->getTag(), StringAttr::get(context, node->getName()), context, node->getTag(), StringAttr::get(context, node->getName()),
translate(node->getFile()), node->getLine(), translate(node->getScope()), translate(node->getFile()), node->getLine(), translate(node->getScope()),
Show All 34 Lines return DILocalVariableAttr::get(context, translate(node->getScope()),
translate(node->getType())); translate(node->getType()));
} }
DIScopeAttr DebugImporter::translateImpl(llvm::DIScope *node) { DIScopeAttr DebugImporter::translateImpl(llvm::DIScope *node) {
return cast<DIScopeAttr>(translate(static_cast<llvm::DINode *>(node))); return cast<DIScopeAttr>(translate(static_cast<llvm::DINode *>(node)));
} }
DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) { DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
Optional<DISubprogramFlags> subprogramFlags = std::optional<DISubprogramFlags> subprogramFlags =
symbolizeDISubprogramFlags(node->getSubprogram()->getSPFlags()); symbolizeDISubprogramFlags(node->getSubprogram()->getSPFlags());
return DISubprogramAttr::get( return DISubprogramAttr::get(
context, translate(node->getUnit()), translate(node->getScope()), context, translate(node->getUnit()), translate(node->getScope()),
StringAttr::get(context, node->getName()), StringAttr::get(context, node->getName()),
node->getRawLinkageName() node->getRawLinkageName()
? StringAttr::get(context, node->getLinkageName()) ? StringAttr::get(context, node->getLinkageName())
: nullptr, : nullptr,
translate(node->getFile()), node->getLine(), node->getScopeLine(), translate(node->getFile()), node->getLine(), node->getScopeLine(),
▲ Show 20 Lines • Show All 96 Lines • Show Last 20 Lines

mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp

Show All 21 Lines
#include "llvm/Frontend/OpenMP/OMPIRBuilder.h" #include "llvm/Frontend/OpenMP/OMPIRBuilder.h"
#include "llvm/IR/DebugInfoMetadata.h" #include "llvm/IR/DebugInfoMetadata.h"
#include "llvm/IR/IRBuilder.h" #include "llvm/IR/IRBuilder.h"
using namespace mlir; using namespace mlir;
namespace { namespace {
static llvm::omp::ScheduleKind static llvm::omp::ScheduleKind
convertToScheduleKind(Optional<omp::ClauseScheduleKind> schedKind) { convertToScheduleKind(std::optional<omp::ClauseScheduleKind> schedKind) {
if (!schedKind.has_value()) if (!schedKind.has_value())
return llvm::omp::OMP_SCHEDULE_Default; return llvm::omp::OMP_SCHEDULE_Default;
switch (schedKind.value()) { switch (schedKind.value()) {
case omp::ClauseScheduleKind::Static: case omp::ClauseScheduleKind::Static:
return llvm::omp::OMP_SCHEDULE_Static; return llvm::omp::OMP_SCHEDULE_Static;
case omp::ClauseScheduleKind::Dynamic: case omp::ClauseScheduleKind::Dynamic:
return llvm::omp::OMP_SCHEDULE_Dynamic; return llvm::omp::OMP_SCHEDULE_Dynamic;
case omp::ClauseScheduleKind::Guided: case omp::ClauseScheduleKind::Guided:
▲ Show 20 Lines • Show All 354 Lines • ▼ Show 20 Linesstatic omp::ReductionDeclareOp findReductionDecl(omp::WsLoopOp container,
return SymbolTable::lookupNearestSymbolFrom<omp::ReductionDeclareOp>( return SymbolTable::lookupNearestSymbolFrom<omp::ReductionDeclareOp>(
container, reductionSymbol); container, reductionSymbol);
} }
/// Populates `reductions` with reduction declarations used in the given loop. /// Populates `reductions` with reduction declarations used in the given loop.
static void static void
collectReductionDecls(omp::WsLoopOp loop, collectReductionDecls(omp::WsLoopOp loop,
SmallVectorImpl<omp::ReductionDeclareOp> &reductions) { SmallVectorImpl<omp::ReductionDeclareOp> &reductions) {
Optional<ArrayAttr> attr = loop.getReductions(); std::optional<ArrayAttr> attr = loop.getReductions();
if (!attr) if (!attr)
return; return;
reductions.reserve(reductions.size() + loop.getNumReductionVars()); reductions.reserve(reductions.size() + loop.getNumReductionVars());
for (auto symbolRef : attr->getAsRange<SymbolRefAttr>()) { for (auto symbolRef : attr->getAsRange<SymbolRefAttr>()) {
reductions.push_back( reductions.push_back(
SymbolTable::lookupNearestSymbolFrom<omp::ReductionDeclareOp>( SymbolTable::lookupNearestSymbolFrom<omp::ReductionDeclareOp>(
loop, symbolRef)); loop, symbolRef));
▲ Show 20 Lines • Show All 440 Lines • ▼ Show 20 LinesconvertOmpWsLoop(Operation &opInst, llvm::IRBuilderBase &builder,
llvm::IRBuilderBase::InsertPoint afterIP = loopInfos.front()->getAfterIP(); llvm::IRBuilderBase::InsertPoint afterIP = loopInfos.front()->getAfterIP();
llvm::CanonicalLoopInfo *loopInfo = llvm::CanonicalLoopInfo *loopInfo =
ompBuilder->collapseLoops(ompLoc.DL, loopInfos, {}); ompBuilder->collapseLoops(ompLoc.DL, loopInfos, {});
allocaIP = findAllocaInsertPoint(builder, moduleTranslation); allocaIP = findAllocaInsertPoint(builder, moduleTranslation);
// TODO: Handle doacross loops when the ordered clause has a parameter. // TODO: Handle doacross loops when the ordered clause has a parameter.
bool isOrdered = loop.getOrderedVal().has_value(); bool isOrdered = loop.getOrderedVal().has_value();
Optional<omp::ScheduleModifier> scheduleModifier = loop.getScheduleModifier(); std::optional<omp::ScheduleModifier> scheduleModifier =
loop.getScheduleModifier();
bool isSimd = loop.getSimdModifier(); bool isSimd = loop.getSimdModifier();
ompBuilder->applyWorkshareLoop( ompBuilder->applyWorkshareLoop(
ompLoc.DL, loopInfo, allocaIP, !loop.getNowait(), ompLoc.DL, loopInfo, allocaIP, !loop.getNowait(),
convertToScheduleKind(schedule), chunk, isSimd, convertToScheduleKind(schedule), chunk, isSimd,
scheduleModifier == omp::ScheduleModifier::monotonic, scheduleModifier == omp::ScheduleModifier::monotonic,
scheduleModifier == omp::ScheduleModifier::nonmonotonic, isOrdered); scheduleModifier == omp::ScheduleModifier::nonmonotonic, isOrdered);
▲ Show 20 Lines • Show All 117 Lines • ▼ Show 20 LinesconvertOmpSimdLoop(Operation &opInst, llvm::IRBuilderBase &builder,
} }
// Collapse loops. // Collapse loops.
llvm::IRBuilderBase::InsertPoint afterIP = loopInfos.front()->getAfterIP(); llvm::IRBuilderBase::InsertPoint afterIP = loopInfos.front()->getAfterIP();
llvm::CanonicalLoopInfo *loopInfo = llvm::CanonicalLoopInfo *loopInfo =
ompBuilder->collapseLoops(ompLoc.DL, loopInfos, {}); ompBuilder->collapseLoops(ompLoc.DL, loopInfos, {});
llvm::ConstantInt *simdlen = nullptr; llvm::ConstantInt *simdlen = nullptr;
if (llvm::Optional<uint64_t> simdlenVar = loop.getSimdlen()) if (std::optional<uint64_t> simdlenVar = loop.getSimdlen())
simdlen = builder.getInt64(simdlenVar.value()); simdlen = builder.getInt64(simdlenVar.value());
llvm::ConstantInt *safelen = nullptr; llvm::ConstantInt *safelen = nullptr;
if (llvm::Optional<uint64_t> safelenVar = loop.getSafelen()) if (std::optional<uint64_t> safelenVar = loop.getSafelen())
safelen = builder.getInt64(safelenVar.value()); safelen = builder.getInt64(safelenVar.value());
llvm::MapVector<llvm::Value *, llvm::Value *> alignedVars; llvm::MapVector<llvm::Value *, llvm::Value *> alignedVars;
ompBuilder->applySimd( ompBuilder->applySimd(
loopInfo, alignedVars, loopInfo, alignedVars,
loop.getIfExpr() ? moduleTranslation.lookupValue(loop.getIfExpr()) loop.getIfExpr() ? moduleTranslation.lookupValue(loop.getIfExpr())
: nullptr, : nullptr,
llvm::omp::OrderKind::OMP_ORDER_unknown, simdlen, safelen); llvm::omp::OrderKind::OMP_ORDER_unknown, simdlen, safelen);
builder.restoreIP(afterIP); builder.restoreIP(afterIP);
return success(); return success();
} }
/// Convert an Atomic Ordering attribute to llvm::AtomicOrdering. /// Convert an Atomic Ordering attribute to llvm::AtomicOrdering.
llvm::AtomicOrdering llvm::AtomicOrdering
convertAtomicOrdering(Optional<omp::ClauseMemoryOrderKind> ao) { convertAtomicOrdering(std::optional<omp::ClauseMemoryOrderKind> ao) {
if (!ao) if (!ao)
return llvm::AtomicOrdering::Monotonic; // Default Memory Ordering return llvm::AtomicOrdering::Monotonic; // Default Memory Ordering
switch (*ao) { switch (*ao) {
case omp::ClauseMemoryOrderKind::Seq_cst: case omp::ClauseMemoryOrderKind::Seq_cst:
return llvm::AtomicOrdering::SequentiallyConsistent; return llvm::AtomicOrdering::SequentiallyConsistent;
case omp::ClauseMemoryOrderKind::Acq_rel: case omp::ClauseMemoryOrderKind::Acq_rel:
return llvm::AtomicOrdering::AcquireRelease; return llvm::AtomicOrdering::AcquireRelease;
▲ Show 20 Lines • Show All 451 Lines • Show Last 20 Lines

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

Show First 20 Lines • Show All 681 Lines • ▼ Show 20 Lines for (auto op : getModuleBody(mlirModule).getOps<LLVM::GlobalOp>()) {
if (op.getUnnamedAddr().has_value()) if (op.getUnnamedAddr().has_value())
var->setUnnamedAddr(convertUnnamedAddrToLLVM(*op.getUnnamedAddr())); var->setUnnamedAddr(convertUnnamedAddrToLLVM(*op.getUnnamedAddr()));
if (op.getSection().has_value()) if (op.getSection().has_value())
var->setSection(*op.getSection()); var->setSection(*op.getSection());
addRuntimePreemptionSpecifier(op.getDsoLocal(), var); addRuntimePreemptionSpecifier(op.getDsoLocal(), var);
Optional<uint64_t> alignment = op.getAlignment(); std::optional<uint64_t> alignment = op.getAlignment();
if (alignment.has_value()) if (alignment.has_value())
var->setAlignment(llvm::MaybeAlign(alignment.value())); var->setAlignment(llvm::MaybeAlign(alignment.value()));
globalsMapping.try_emplace(op, var); globalsMapping.try_emplace(op, var);
} }
// Convert global variable bodies. This is done after all global variables // Convert global variable bodies. This is done after all global variables
// have been created in LLVM IR because a global body may refer to another // have been created in LLVM IR because a global body may refer to another
▲ Show 20 Lines • Show All 79 Lines • ▼ Show 20 Lines
/// Attaches the attributes listed in the given array attribute to `llvmFunc`. /// Attaches the attributes listed in the given array attribute to `llvmFunc`.
/// Reports error to `loc` if any and returns immediately. Expects `attributes` /// Reports error to `loc` if any and returns immediately. Expects `attributes`
/// to be an array attribute containing either string attributes, treated as /// to be an array attribute containing either string attributes, treated as
/// value-less LLVM attributes, or array attributes containing two string /// value-less LLVM attributes, or array attributes containing two string
/// attributes, with the first string being the name of the corresponding LLVM /// attributes, with the first string being the name of the corresponding LLVM
/// attribute and the second string beings its value. Note that even integer /// attribute and the second string beings its value. Note that even integer
/// attributes are expected to have their values expressed as strings. /// attributes are expected to have their values expressed as strings.
static LogicalResult static LogicalResult
forwardPassthroughAttributes(Location loc, Optional<ArrayAttr> attributes, forwardPassthroughAttributes(Location loc, std::optional<ArrayAttr> attributes,
llvm::Function *llvmFunc) { llvm::Function *llvmFunc) {
if (!attributes) if (!attributes)
return success(); return success();
for (Attribute attr : *attributes) { for (Attribute attr : *attributes) {
if (auto stringAttr = attr.dyn_cast<StringAttr>()) { if (auto stringAttr = attr.dyn_cast<StringAttr>()) {
if (failed( if (failed(
checkedAddLLVMFnAttribute(loc, llvmFunc, stringAttr.getValue()))) checkedAddLLVMFnAttribute(loc, llvmFunc, stringAttr.getValue())))
▲ Show 20 Lines • Show All 311 Lines • ▼ Show 20 Lines
LogicalResult ModuleTranslation::createAliasScopeMetadata() { LogicalResult ModuleTranslation::createAliasScopeMetadata() {
mlirModule->walk([&](LLVM::MetadataOp metadatas) { mlirModule->walk([&](LLVM::MetadataOp metadatas) {
// Create the domains first, so they can be reference below in the scopes. // Create the domains first, so they can be reference below in the scopes.
DenseMap<Operation *, llvm::MDNode *> aliasScopeDomainMetadataMapping; DenseMap<Operation *, llvm::MDNode *> aliasScopeDomainMetadataMapping;
metadatas.walk([&](LLVM::AliasScopeDomainMetadataOp op) { metadatas.walk([&](LLVM::AliasScopeDomainMetadataOp op) {
llvm::LLVMContext &ctx = llvmModule->getContext(); llvm::LLVMContext &ctx = llvmModule->getContext();
llvm::SmallVector<llvm::Metadata *, 2> operands; llvm::SmallVector<llvm::Metadata *, 2> operands;
operands.push_back({}); // Placeholder for self-reference operands.push_back({}); // Placeholder for self-reference
if (Optional<StringRef> description = op.getDescription()) if (std::optional<StringRef> description = op.getDescription())
operands.push_back(llvm::MDString::get(ctx, *description)); operands.push_back(llvm::MDString::get(ctx, *description));
llvm::MDNode *domain = llvm::MDNode::get(ctx, operands); llvm::MDNode *domain = llvm::MDNode::get(ctx, operands);
domain->replaceOperandWith(0, domain); // Self-reference for uniqueness domain->replaceOperandWith(0, domain); // Self-reference for uniqueness
aliasScopeDomainMetadataMapping.insert({op, domain}); aliasScopeDomainMetadataMapping.insert({op, domain});
}); });
// Now create the scopes, referencing the domains created above. // Now create the scopes, referencing the domains created above.
metadatas.walk([&](LLVM::AliasScopeMetadataOp op) { metadatas.walk([&](LLVM::AliasScopeMetadataOp op) {
llvm::LLVMContext &ctx = llvmModule->getContext(); llvm::LLVMContext &ctx = llvmModule->getContext();
assert(isa<LLVM::MetadataOp>(op->getParentOp())); assert(isa<LLVM::MetadataOp>(op->getParentOp()));
auto metadataOp = dyn_cast<LLVM::MetadataOp>(op->getParentOp()); auto metadataOp = dyn_cast<LLVM::MetadataOp>(op->getParentOp());
Operation *domainOp = Operation *domainOp =
SymbolTable::lookupNearestSymbolFrom(metadataOp, op.getDomainAttr()); SymbolTable::lookupNearestSymbolFrom(metadataOp, op.getDomainAttr());
llvm::MDNode *domain = aliasScopeDomainMetadataMapping.lookup(domainOp); llvm::MDNode *domain = aliasScopeDomainMetadataMapping.lookup(domainOp);
assert(domain && "Scope's domain should already be valid"); assert(domain && "Scope's domain should already be valid");
llvm::SmallVector<llvm::Metadata *, 3> operands; llvm::SmallVector<llvm::Metadata *, 3> operands;
operands.push_back({}); // Placeholder for self-reference operands.push_back({}); // Placeholder for self-reference
operands.push_back(domain); operands.push_back(domain);
if (Optional<StringRef> description = op.getDescription()) if (std::optional<StringRef> description = op.getDescription())
operands.push_back(llvm::MDString::get(ctx, *description)); operands.push_back(llvm::MDString::get(ctx, *description));
llvm::MDNode *scope = llvm::MDNode::get(ctx, operands); llvm::MDNode *scope = llvm::MDNode::get(ctx, operands);
scope->replaceOperandWith(0, scope); // Self-reference for uniqueness scope->replaceOperandWith(0, scope); // Self-reference for uniqueness
aliasScopeMetadataMapping.insert({op, scope}); aliasScopeMetadataMapping.insert({op, scope});
}); });
}); });
return success(); return success();
} }
▲ Show 20 Lines • Show All 144 Lines • Show Last 20 Lines

mlir/lib/Tools/PDLL/CodeGen/CPPGen.cpp

Show First 20 Lines • Show All 73 Lines • ▼ Show 20 Linesvoid CodeGen::generate(const ast::Module &astModule, ModuleOp module) {
// Generate code for any native functions within the module. // Generate code for any native functions within the module.
generateConstraintAndRewrites(astModule, module, nativeFunctions); generateConstraintAndRewrites(astModule, module, nativeFunctions);
os << "namespace {\n"; os << "namespace {\n";
std::string basePatternName = "GeneratedPDLLPattern"; std::string basePatternName = "GeneratedPDLLPattern";
int patternIndex = 0; int patternIndex = 0;
for (pdl::PatternOp pattern : module.getOps<pdl::PatternOp>()) { for (pdl::PatternOp pattern : module.getOps<pdl::PatternOp>()) {
// If the pattern has a name, use that. Otherwise, generate a unique name. // If the pattern has a name, use that. Otherwise, generate a unique name.
if (Optional<StringRef> patternName = pattern.getSymName()) { if (std::optional<StringRef> patternName = pattern.getSymName()) {
patternNames.insert(patternName->str()); patternNames.insert(patternName->str());
} else { } else {
std::string name; std::string name;
do { do {
name = (basePatternName + Twine(patternIndex++)).str(); name = (basePatternName + Twine(patternIndex++)).str();
} while (!patternNames.insert(name)); } while (!patternNames.insert(name));
} }
▲ Show 20 Lines • Show All 166 Lines • Show Last 20 Lines

mlir/test/lib/Dialect/Test/TestDialect.cpp

Show First 20 Lines • Show All 78 Lines • ▼ Show 20 Linesstruct TestOpAsmInterface : public OpAsmDialectInterface {
AliasResult getAlias(Attribute attr, raw_ostream &os) const final { AliasResult getAlias(Attribute attr, raw_ostream &os) const final {
StringAttr strAttr = attr.dyn_cast<StringAttr>(); StringAttr strAttr = attr.dyn_cast<StringAttr>();
if (!strAttr) if (!strAttr)
return AliasResult::NoAlias; return AliasResult::NoAlias;
// Check the contents of the string attribute to see what the test alias // Check the contents of the string attribute to see what the test alias
// should be named. // should be named.
Optional<StringRef> aliasName = std::optional<StringRef> aliasName =
StringSwitch<Optional<StringRef>>(strAttr.getValue()) StringSwitch<std::optional<StringRef>>(strAttr.getValue())
.Case("alias_test:dot_in_name", StringRef("test.alias")) .Case("alias_test:dot_in_name", StringRef("test.alias"))
.Case("alias_test:trailing_digit", StringRef("test_alias0")) .Case("alias_test:trailing_digit", StringRef("test_alias0"))
.Case("alias_test:prefixed_digit", StringRef("0_test_alias")) .Case("alias_test:prefixed_digit", StringRef("0_test_alias"))
.Case("alias_test:sanitize_conflict_a", .Case("alias_test:sanitize_conflict_a",
StringRef("test_alias_conflict0")) StringRef("test_alias_conflict0"))
.Case("alias_test:sanitize_conflict_b", .Case("alias_test:sanitize_conflict_b",
StringRef("test_alias_conflict0_")) StringRef("test_alias_conflict0_"))
.Case("alias_test:tensor_encoding", StringRef("test_encoding")) .Case("alias_test:tensor_encoding", StringRef("test_encoding"))
▲ Show 20 Lines • Show All 281 Lines • ▼ Show 20 Lines
} }
Operation *TestDialect::materializeConstant(OpBuilder &builder, Attribute value, Operation *TestDialect::materializeConstant(OpBuilder &builder, Attribute value,
Type type, Location loc) { Type type, Location loc) {
return builder.create<TestOpConstant>(loc, type, value); return builder.create<TestOpConstant>(loc, type, value);
} }
::mlir::LogicalResult FormatInferType2Op::inferReturnTypes( ::mlir::LogicalResult FormatInferType2Op::inferReturnTypes(
::mlir::MLIRContext *context, ::llvm::Optional<::mlir::Location> location, ::mlir::MLIRContext *context, ::std::optional<::mlir::Location> location,
::mlir::ValueRange operands, ::mlir::DictionaryAttr attributes, ::mlir::ValueRange operands, ::mlir::DictionaryAttr attributes,
::mlir::RegionRange regions, ::mlir::RegionRange regions,
::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) { ::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) {
inferredReturnTypes.assign({::mlir::IntegerType::get(context, 16)}); inferredReturnTypes.assign({::mlir::IntegerType::get(context, 16)});
return ::mlir::success(); return ::mlir::success();
} }
void *TestDialect::getRegisteredInterfaceForOp(TypeID typeID, void *TestDialect::getRegisteredInterfaceForOp(TypeID typeID,
Show All 24 Lines
TestDialect::verifyRegionResultAttribute(Operation *op, unsigned regionIndex, TestDialect::verifyRegionResultAttribute(Operation *op, unsigned regionIndex,
unsigned resultIndex, unsigned resultIndex,
NamedAttribute namedAttr) { NamedAttribute namedAttr) {
if (namedAttr.getName() == "test.invalid_attr") if (namedAttr.getName() == "test.invalid_attr")
return op->emitError() << "invalid to use 'test.invalid_attr'"; return op->emitError() << "invalid to use 'test.invalid_attr'";
return success(); return success();
} }
Optional<Dialect::ParseOpHook> std::optional<Dialect::ParseOpHook>
TestDialect::getParseOperationHook(StringRef opName) const { TestDialect::getParseOperationHook(StringRef opName) const {
if (opName == "test.dialect_custom_printer") { if (opName == "test.dialect_custom_printer") {
return ParseOpHook{[](OpAsmParser &parser, OperationState &state) { return ParseOpHook{[](OpAsmParser &parser, OperationState &state) {
return parser.parseKeyword("custom_format"); return parser.parseKeyword("custom_format");
}}; }};
} }
if (opName == "test.dialect_custom_format_fallback") { if (opName == "test.dialect_custom_format_fallback") {
return ParseOpHook{[](OpAsmParser &parser, OperationState &state) { return ParseOpHook{[](OpAsmParser &parser, OperationState &state) {
▲ Show 20 Lines • Show All 128 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Test Format* operations // Test Format* operations
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Parsing // Parsing
static ParseResult parseCustomOptionalOperand( static ParseResult parseCustomOptionalOperand(
OpAsmParser &parser, Optional<OpAsmParser::UnresolvedOperand> &optOperand) { OpAsmParser &parser,
std::optional<OpAsmParser::UnresolvedOperand> &optOperand) {
if (succeeded(parser.parseOptionalLParen())) { if (succeeded(parser.parseOptionalLParen())) {
optOperand.emplace(); optOperand.emplace();
if (parser.parseOperand(*optOperand) || parser.parseRParen()) if (parser.parseOperand(*optOperand) || parser.parseRParen())
return failure(); return failure();
} }
return success(); return success();
} }
static ParseResult parseCustomDirectiveOperands( static ParseResult parseCustomDirectiveOperands(
OpAsmParser &parser, OpAsmParser::UnresolvedOperand &operand, OpAsmParser &parser, OpAsmParser::UnresolvedOperand &operand,
Optional<OpAsmParser::UnresolvedOperand> &optOperand, std::optional<OpAsmParser::UnresolvedOperand> &optOperand,
SmallVectorImpl<OpAsmParser::UnresolvedOperand> &varOperands) { SmallVectorImpl<OpAsmParser::UnresolvedOperand> &varOperands) {
if (parser.parseOperand(operand)) if (parser.parseOperand(operand))
return failure(); return failure();
if (succeeded(parser.parseOptionalComma())) { if (succeeded(parser.parseOptionalComma())) {
optOperand.emplace(); optOperand.emplace();
if (parser.parseOperand(*optOperand)) if (parser.parseOperand(*optOperand))
return failure(); return failure();
} }
Show All 36 LinesparseCustomDirectiveWithTypeRefs(OpAsmParser &parser, Type operandType,
if (operandType != operandType2 || optOperandType != optOperandType2 || if (operandType != operandType2 || optOperandType != optOperandType2 ||
varOperandTypes != varOperandTypes2) varOperandTypes != varOperandTypes2)
return failure(); return failure();
return success(); return success();
} }
static ParseResult parseCustomDirectiveOperandsAndTypes( static ParseResult parseCustomDirectiveOperandsAndTypes(
OpAsmParser &parser, OpAsmParser::UnresolvedOperand &operand, OpAsmParser &parser, OpAsmParser::UnresolvedOperand &operand,
Optional<OpAsmParser::UnresolvedOperand> &optOperand, std::optional<OpAsmParser::UnresolvedOperand> &optOperand,
SmallVectorImpl<OpAsmParser::UnresolvedOperand> &varOperands, SmallVectorImpl<OpAsmParser::UnresolvedOperand> &varOperands,
Type &operandType, Type &optOperandType, Type &operandType, Type &optOperandType,
SmallVectorImpl<Type> &varOperandTypes) { SmallVectorImpl<Type> &varOperandTypes) {
if (parseCustomDirectiveOperands(parser, operand, optOperand, varOperands) || if (parseCustomDirectiveOperands(parser, operand, optOperand, varOperands) ||
parseCustomDirectiveResults(parser, operandType, optOperandType, parseCustomDirectiveResults(parser, operandType, optOperandType,
varOperandTypes)) varOperandTypes))
return failure(); return failure();
return success(); return success();
Show All 39 Linesstatic ParseResult parseCustomDirectiveSpacing(OpAsmParser &parser,
mlir::StringAttr &attr) { mlir::StringAttr &attr) {
return parser.parseAttribute(attr); return parser.parseAttribute(attr);
} }
static ParseResult parseCustomDirectiveAttrDict(OpAsmParser &parser, static ParseResult parseCustomDirectiveAttrDict(OpAsmParser &parser,
NamedAttrList &attrs) { NamedAttrList &attrs) {
return parser.parseOptionalAttrDict(attrs); return parser.parseOptionalAttrDict(attrs);
} }
static ParseResult parseCustomDirectiveOptionalOperandRef( static ParseResult parseCustomDirectiveOptionalOperandRef(
OpAsmParser &parser, Optional<OpAsmParser::UnresolvedOperand> &optOperand) { OpAsmParser &parser,
std::optional<OpAsmParser::UnresolvedOperand> &optOperand) {
int64_t operandCount = 0; int64_t operandCount = 0;
if (parser.parseInteger(operandCount)) if (parser.parseInteger(operandCount))
return failure(); return failure();
bool expectedOptionalOperand = operandCount == 0; bool expectedOptionalOperand = operandCount == 0;
return success(expectedOptionalOperand != optOperand.has_value()); return success(expectedOptionalOperand != optOperand.has_value());
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 419 Lines • ▼ Show 20 LinesOpFoldResult TestOpInPlaceFold::fold(ArrayRef<Attribute> operands) {
return {}; return {};
} }
OpFoldResult TestPassthroughFold::fold(ArrayRef<Attribute> operands) { OpFoldResult TestPassthroughFold::fold(ArrayRef<Attribute> operands) {
return getOperand(); return getOperand();
} }
LogicalResult OpWithInferTypeInterfaceOp::inferReturnTypes( LogicalResult OpWithInferTypeInterfaceOp::inferReturnTypes(
MLIRContext *, Optional<Location> location, ValueRange operands, MLIRContext *, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &inferredReturnTypes) { SmallVectorImpl<Type> &inferredReturnTypes) {
if (operands[0].getType() != operands[1].getType()) { if (operands[0].getType() != operands[1].getType()) {
return emitOptionalError(location, "operand type mismatch ", return emitOptionalError(location, "operand type mismatch ",
operands[0].getType(), " vs ", operands[0].getType(), " vs ",
operands[1].getType()); operands[1].getType());
} }
inferredReturnTypes.assign({operands[0].getType()}); inferredReturnTypes.assign({operands[0].getType()});
return success(); return success();
} }
// TODO: We should be able to only define either inferReturnType or // TODO: We should be able to only define either inferReturnType or
// refineReturnType, currently only refineReturnType can be omitted. // refineReturnType, currently only refineReturnType can be omitted.
LogicalResult OpWithRefineTypeInterfaceOp::inferReturnTypes( LogicalResult OpWithRefineTypeInterfaceOp::inferReturnTypes(
MLIRContext *context, Optional<Location> location, ValueRange operands, MLIRContext *context, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &returnTypes) { SmallVectorImpl<Type> &returnTypes) {
returnTypes.clear(); returnTypes.clear();
return OpWithRefineTypeInterfaceOp::refineReturnTypes( return OpWithRefineTypeInterfaceOp::refineReturnTypes(
context, location, operands, attributes, regions, returnTypes); context, location, operands, attributes, regions, returnTypes);
} }
LogicalResult OpWithRefineTypeInterfaceOp::refineReturnTypes( LogicalResult OpWithRefineTypeInterfaceOp::refineReturnTypes(
MLIRContext *, Optional<Location> location, ValueRange operands, MLIRContext *, std::optional<Location> location, ValueRange operands,
DictionaryAttr attributes, RegionRange regions, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<Type> &returnTypes) { SmallVectorImpl<Type> &returnTypes) {
if (operands[0].getType() != operands[1].getType()) { if (operands[0].getType() != operands[1].getType()) {
return emitOptionalError(location, "operand type mismatch ", return emitOptionalError(location, "operand type mismatch ",
operands[0].getType(), " vs ", operands[0].getType(), " vs ",
operands[1].getType()); operands[1].getType());
} }
// TODO: Add helper to make this more concise to write. // TODO: Add helper to make this more concise to write.
if (returnTypes.empty()) if (returnTypes.empty())
returnTypes.resize(1, nullptr); returnTypes.resize(1, nullptr);
if (returnTypes[0] && returnTypes[0] != operands[0].getType()) if (returnTypes[0] && returnTypes[0] != operands[0].getType())
return emitOptionalError(location, return emitOptionalError(location,
"required first operand and result to match"); "required first operand and result to match");
returnTypes[0] = operands[0].getType(); returnTypes[0] = operands[0].getType();
return success(); return success();
} }
LogicalResult OpWithShapedTypeInferTypeInterfaceOp::inferReturnTypeComponents( LogicalResult OpWithShapedTypeInferTypeInterfaceOp::inferReturnTypeComponents(
MLIRContext *context, Optional<Location> location, ValueShapeRange operands, MLIRContext *context, std::optional<Location> location,
DictionaryAttr attributes, RegionRange regions, ValueShapeRange operands, DictionaryAttr attributes, RegionRange regions,
SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) { SmallVectorImpl<ShapedTypeComponents> &inferredReturnShapes) {
// Create return type consisting of the last element of the first operand. // Create return type consisting of the last element of the first operand.
auto operandType = operands.front().getType(); auto operandType = operands.front().getType();
auto sval = operandType.dyn_cast<ShapedType>(); auto sval = operandType.dyn_cast<ShapedType>();
if (!sval) { if (!sval) {
return emitOptionalError(location, "only shaped type operands allowed"); return emitOptionalError(location, "only shaped type operands allowed");
} }
int64_t dim = int64_t dim = sval.hasRank() ? sval.getShape().front() : ShapedType::kDynamic;
sval.hasRank() ? sval.getShape().front() : ShapedType::kDynamic;
auto type = IntegerType::get(context, 17); auto type = IntegerType::get(context, 17);
Attribute encoding; Attribute encoding;
if (auto ranked_ty = sval.dyn_cast<RankedTensorType>()) if (auto ranked_ty = sval.dyn_cast<RankedTensorType>())
encoding = ranked_ty.getEncoding(); encoding = ranked_ty.getEncoding();
inferredReturnShapes.push_back(ShapedTypeComponents({dim}, type, encoding)); inferredReturnShapes.push_back(ShapedTypeComponents({dim}, type, encoding));
return success(); return success();
} }
▲ Show 20 Lines • Show All 256 Lines • ▼ Show 20 Lines if (parser.parseKeyword("then") || parser.parseRegion(*thenRegion, {}, {}) ||
parser.parseKeyword("else") || parser.parseRegion(*elseRegion, {}, {}) || parser.parseKeyword("else") || parser.parseRegion(*elseRegion, {}, {}) ||
parser.parseKeyword("join") || parser.parseRegion(*joinRegion, {}, {})) parser.parseKeyword("join") || parser.parseRegion(*joinRegion, {}, {}))
return failure(); return failure();
return parser.resolveOperands(operandInfos, operandTypes, return parser.resolveOperands(operandInfos, operandTypes,
parser.getCurrentLocation(), result.operands); parser.getCurrentLocation(), result.operands);
} }
OperandRange RegionIfOp::getSuccessorEntryOperands(Optional<unsigned> index) { OperandRange
RegionIfOp::getSuccessorEntryOperands(std::optional<unsigned> index) {
assert(index && *index < 2 && "invalid region index"); assert(index && *index < 2 && "invalid region index");
return getOperands(); return getOperands();
} }
void RegionIfOp::getSuccessorRegions( void RegionIfOp::getSuccessorRegions(
Optional<unsigned> index, ArrayRef<Attribute> operands, std::optional<unsigned> index, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// We always branch to the join region. // We always branch to the join region.
if (index.has_value()) { if (index.has_value()) {
if (index.value() < 2) if (index.value() < 2)
regions.push_back(RegionSuccessor(&getJoinRegion(), getJoinArgs())); regions.push_back(RegionSuccessor(&getJoinRegion(), getJoinArgs()));
else else
regions.push_back(RegionSuccessor(getResults())); regions.push_back(RegionSuccessor(getResults()));
return; return;
Show All 10 Linesvoid RegionIfOp::getRegionInvocationBounds(
// Each region is invoked at most once. // Each region is invoked at most once.
invocationBounds.assign(/*NumElts=*/3, /*Elt=*/{0, 1}); invocationBounds.assign(/*NumElts=*/3, /*Elt=*/{0, 1});
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AnyCondOp // AnyCondOp
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
void AnyCondOp::getSuccessorRegions(Optional<unsigned> index, void AnyCondOp::getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute> operands, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
// The parent op branches into the only region, and the region branches back // The parent op branches into the only region, and the region branches back
// to the parent op. // to the parent op.
if (!index) if (!index)
regions.emplace_back(&getRegion()); regions.emplace_back(&getRegion());
else else
regions.emplace_back(getResults()); regions.emplace_back(getResults());
▲ Show 20 Lines • Show All 126 Lines • Show Last 20 Lines

mlir/test/lib/Dialect/Test/TestDialect.td

Show All 25 Linesdef Test_Dialect : Dialect {
let isExtensible = 1; let isExtensible = 1;
let dependentDialects = ["::mlir::DLTIDialect"]; let dependentDialects = ["::mlir::DLTIDialect"];
let extraClassDeclaration = [{ let extraClassDeclaration = [{
void registerAttributes(); void registerAttributes();
void registerTypes(); void registerTypes();
// Provides a custom printing/parsing for some operations. // Provides a custom printing/parsing for some operations.
::llvm::Optional<ParseOpHook> ::std::optional<ParseOpHook>
getParseOperationHook(::llvm::StringRef opName) const override; getParseOperationHook(::llvm::StringRef opName) const override;
::llvm::unique_function<void(::mlir::Operation *, ::llvm::unique_function<void(::mlir::Operation *,
::mlir::OpAsmPrinter &printer)> ::mlir::OpAsmPrinter &printer)>
getOperationPrinter(::mlir::Operation *op) const override; getOperationPrinter(::mlir::Operation *op) const override;
::mlir::Type parseType(::mlir::DialectAsmParser &parser) const override; ::mlir::Type parseType(::mlir::DialectAsmParser &parser) const override;
void printType(::mlir::Type type, void printType(::mlir::Type type,
::mlir::DialectAsmPrinter &printer) const override; ::mlir::DialectAsmPrinter &printer) const override;
Show All 13 Lines

mlir/test/lib/Dialect/Test/TestOps.td

Show First 20 Lines • Show All 415 Lines • ▼ Show 20 Lines
def VariadicRegionInferredTypesOp : TEST_Op<"variadic_region_inferred", def VariadicRegionInferredTypesOp : TEST_Op<"variadic_region_inferred",
[InferTypeOpInterface]> { [InferTypeOpInterface]> {
let regions = (region VariadicRegion<AnyRegion>:$bodies); let regions = (region VariadicRegion<AnyRegion>:$bodies);
let results = (outs Variadic<AnyType>); let results = (outs Variadic<AnyType>);
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static mlir::LogicalResult inferReturnTypes(mlir::MLIRContext *context, static mlir::LogicalResult inferReturnTypes(mlir::MLIRContext *context,
llvm::Optional<::mlir::Location> location, mlir::ValueRange operands, std::optional<::mlir::Location> location, mlir::ValueRange operands,
mlir::DictionaryAttr attributes, mlir::RegionRange regions, mlir::DictionaryAttr attributes, mlir::RegionRange regions,
llvm::SmallVectorImpl<mlir::Type> &inferredReturnTypes) { llvm::SmallVectorImpl<mlir::Type> &inferredReturnTypes) {
inferredReturnTypes.assign({mlir::IntegerType::get(context, 16)}); inferredReturnTypes.assign({mlir::IntegerType::get(context, 16)});
return mlir::success(); return mlir::success();
} }
}]; }];
} }
▲ Show 20 Lines • Show All 1,966 Lines • ▼ Show 20 Lines
// InferTypeOpInterface type inference in assembly format // InferTypeOpInterface type inference in assembly format
def FormatInferTypeOp : TEST_Op<"format_infer_type", [InferTypeOpInterface]> { def FormatInferTypeOp : TEST_Op<"format_infer_type", [InferTypeOpInterface]> {
let results = (outs AnyType); let results = (outs AnyType);
let assemblyFormat = "attr-dict"; let assemblyFormat = "attr-dict";
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *context, static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *context,
::llvm::Optional<::mlir::Location> location, ::mlir::ValueRange operands, ::std::optional<::mlir::Location> location, ::mlir::ValueRange operands,
::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions, ::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions,
::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) { ::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) {
inferredReturnTypes.assign({::mlir::IntegerType::get(context, 16)}); inferredReturnTypes.assign({::mlir::IntegerType::get(context, 16)});
return ::mlir::success(); return ::mlir::success();
} }
}]; }];
} }
// Check that formatget supports DeclareOpInterfaceMethods. // Check that formatget supports DeclareOpInterfaceMethods.
def FormatInferType2Op : TEST_Op<"format_infer_type2", [DeclareOpInterfaceMethods<InferTypeOpInterface>]> { def FormatInferType2Op : TEST_Op<"format_infer_type2", [DeclareOpInterfaceMethods<InferTypeOpInterface>]> {
let results = (outs AnyType); let results = (outs AnyType);
let assemblyFormat = "attr-dict"; let assemblyFormat = "attr-dict";
} }
// Base class for testing mixing allOperandTypes, allOperands, and // Base class for testing mixing allOperandTypes, allOperands, and
// inferResultTypes. // inferResultTypes.
class FormatInferAllTypesBaseOp<string mnemonic, list<Trait> traits = []> class FormatInferAllTypesBaseOp<string mnemonic, list<Trait> traits = []>
: TEST_Op<mnemonic, [InferTypeOpInterface] # traits> { : TEST_Op<mnemonic, [InferTypeOpInterface] # traits> {
let arguments = (ins Variadic<AnyType>:$args); let arguments = (ins Variadic<AnyType>:$args);
let results = (outs Variadic<AnyType>:$outs); let results = (outs Variadic<AnyType>:$outs);
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *context, static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *context,
::llvm::Optional<::mlir::Location> location, ::mlir::ValueRange operands, ::std::optional<::mlir::Location> location, ::mlir::ValueRange operands,
::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions, ::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions,
::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) { ::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) {
::mlir::TypeRange operandTypes = operands.getTypes(); ::mlir::TypeRange operandTypes = operands.getTypes();
inferredReturnTypes.assign(operandTypes.begin(), operandTypes.end()); inferredReturnTypes.assign(operandTypes.begin(), operandTypes.end());
return ::mlir::success(); return ::mlir::success();
} }
}]; }];
} }
Show All 30 Lines
// Test inferReturnTypes coupled with regions. // Test inferReturnTypes coupled with regions.
def FormatInferTypeRegionsOp def FormatInferTypeRegionsOp
: TEST_Op<"format_infer_type_regions", [InferTypeOpInterface]> { : TEST_Op<"format_infer_type_regions", [InferTypeOpInterface]> {
let results = (outs Variadic<AnyType>:$outs); let results = (outs Variadic<AnyType>:$outs);
let regions = (region AnyRegion:$region); let regions = (region AnyRegion:$region);
let assemblyFormat = "$region attr-dict"; let assemblyFormat = "$region attr-dict";
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *context, static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *context,
::llvm::Optional<::mlir::Location> location, ::mlir::ValueRange operands, ::std::optional<::mlir::Location> location, ::mlir::ValueRange operands,
::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions, ::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions,
::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) { ::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) {
if (regions.empty()) if (regions.empty())
return ::mlir::failure(); return ::mlir::failure();
auto types = regions.front()->getArgumentTypes(); auto types = regions.front()->getArgumentTypes();
inferredReturnTypes.assign(types.begin(), types.end()); inferredReturnTypes.assign(types.begin(), types.end());
return ::mlir::success(); return ::mlir::success();
} }
}]; }];
} }
// Test inferReturnTypes coupled with variadic operands (operand_segment_sizes). // Test inferReturnTypes coupled with variadic operands (operand_segment_sizes).
def FormatInferTypeVariadicOperandsOp def FormatInferTypeVariadicOperandsOp
: TEST_Op<"format_infer_type_variadic_operands", : TEST_Op<"format_infer_type_variadic_operands",
[InferTypeOpInterface, AttrSizedOperandSegments]> { [InferTypeOpInterface, AttrSizedOperandSegments]> {
let arguments = (ins Variadic<I32>:$a, Variadic<I64>:$b); let arguments = (ins Variadic<I32>:$a, Variadic<I64>:$b);
let results = (outs Variadic<AnyType>:$outs); let results = (outs Variadic<AnyType>:$outs);
let assemblyFormat = "`(` $a `:` type($a) `)` `(` $b `:` type($b) `)` attr-dict"; let assemblyFormat = "`(` $a `:` type($a) `)` `(` $b `:` type($b) `)` attr-dict";
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *context, static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *context,
::llvm::Optional<::mlir::Location> location, ::mlir::ValueRange operands, ::std::optional<::mlir::Location> location, ::mlir::ValueRange operands,
::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions, ::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions,
::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) { ::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) {
FormatInferTypeVariadicOperandsOpAdaptor adaptor(operands, attributes); FormatInferTypeVariadicOperandsOpAdaptor adaptor(operands, attributes);
auto aTypes = adaptor.getA().getTypes(); auto aTypes = adaptor.getA().getTypes();
auto bTypes = adaptor.getB().getTypes(); auto bTypes = adaptor.getB().getTypes();
inferredReturnTypes.append(aTypes.begin(), aTypes.end()); inferredReturnTypes.append(aTypes.begin(), aTypes.end());
inferredReturnTypes.append(bTypes.begin(), bTypes.end()); inferredReturnTypes.append(bTypes.begin(), bTypes.end());
return ::mlir::success(); return ::mlir::success();
▲ Show 20 Lines • Show All 128 Lines • ▼ Show 20 Lines let extraClassDeclaration = [{
} }
::mlir::Block::BlockArgListType getElseArgs() { ::mlir::Block::BlockArgListType getElseArgs() {
return getBody(1)->getArguments(); return getBody(1)->getArguments();
} }
::mlir::Block::BlockArgListType getJoinArgs() { ::mlir::Block::BlockArgListType getJoinArgs() {
return getBody(2)->getArguments(); return getBody(2)->getArguments();
} }
::mlir::OperandRange getSuccessorEntryOperands( ::mlir::OperandRange getSuccessorEntryOperands(
::llvm::Optional<unsigned> index); ::std::optional<unsigned> index);
}]; }];
let hasCustomAssemblyFormat = 1; let hasCustomAssemblyFormat = 1;
} }
def AnyCondOp : TEST_Op<"any_cond", def AnyCondOp : TEST_Op<"any_cond",
[DeclareOpInterfaceMethods<RegionBranchOpInterface, [DeclareOpInterfaceMethods<RegionBranchOpInterface,
["getRegionInvocationBounds"]>, ["getRegionInvocationBounds"]>,
RecursiveMemoryEffects]> { RecursiveMemoryEffects]> {
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 Lines
class TableGenBuildInferReturnTypeBaseOp<string mnemonic, class TableGenBuildInferReturnTypeBaseOp<string mnemonic,
list<Trait> traits = []> list<Trait> traits = []>
: TEST_Op<mnemonic, [InferTypeOpInterface] # traits> { : TEST_Op<mnemonic, [InferTypeOpInterface] # traits> {
let arguments = (ins Variadic<AnyType>:$inputs); let arguments = (ins Variadic<AnyType>:$inputs);
let results = (outs AnyType:$result); let results = (outs AnyType:$result);
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *, static ::mlir::LogicalResult inferReturnTypes(::mlir::MLIRContext *,
::llvm::Optional<::mlir::Location> location, ::mlir::ValueRange operands, ::std::optional<::mlir::Location> location, ::mlir::ValueRange operands,
::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions, ::mlir::DictionaryAttr attributes, ::mlir::RegionRange regions,
::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) { ::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) {
inferredReturnTypes.assign({operands[0].getType()}); inferredReturnTypes.assign({operands[0].getType()});
return ::mlir::success(); return ::mlir::success();
} }
}]; }];
} }
▲ Show 20 Lines • Show All 411 Lines • Show Last 20 Lines

mlir/test/mlir-tblgen/enums-gen.td

Show All 21 Lines
// DECL-LABEL: enum class MyBitEnum : uint32_t // DECL-LABEL: enum class MyBitEnum : uint32_t
// DECL: None = 0, // DECL: None = 0,
// DECL: Bit0 = 1, // DECL: Bit0 = 1,
// DECL: Bit1 = 2, // DECL: Bit1 = 2,
// DECL: Bit2 = 4, // DECL: Bit2 = 4,
// DECL: Bit3 = 8, // DECL: Bit3 = 8,
// DECL: } // DECL: }
// DECL: ::llvm::Optional<MyBitEnum> symbolizeMyBitEnum(uint32_t); // DECL: ::std::optional<MyBitEnum> symbolizeMyBitEnum(uint32_t);
// DECL: std::string stringifyMyBitEnum(MyBitEnum); // DECL: std::string stringifyMyBitEnum(MyBitEnum);
// DECL: ::llvm::Optional<MyBitEnum> symbolizeMyBitEnum(::llvm::StringRef); // DECL: ::std::optional<MyBitEnum> symbolizeMyBitEnum(::llvm::StringRef);
// DECL: struct FieldParser<::MyBitEnum, ::MyBitEnum> { // DECL: struct FieldParser<::MyBitEnum, ::MyBitEnum> {
// DECL: template <typename ParserT> // DECL: template <typename ParserT>
// DECL: static FailureOr<::MyBitEnum> parse(ParserT &parser) { // DECL: static FailureOr<::MyBitEnum> parse(ParserT &parser) {
// DECL: // Parse the keyword/string containing the enum. // DECL: // Parse the keyword/string containing the enum.
// DECL: std::string enumKeyword; // DECL: std::string enumKeyword;
// DECL: auto loc = parser.getCurrentLocation(); // DECL: auto loc = parser.getCurrentLocation();
// DECL: if (failed(parser.parseOptionalKeywordOrString(&enumKeyword))) // DECL: if (failed(parser.parseOptionalKeywordOrString(&enumKeyword)))
// DECL: return parser.emitError(loc, "expected keyword for An example bit enum"); // DECL: return parser.emitError(loc, "expected keyword for An example bit enum");
// DECL: // Symbolize the keyword. // DECL: // Symbolize the keyword.
// DECL: if (::llvm::Optional<::MyBitEnum> attr = ::symbolizeEnum<::MyBitEnum>(enumKeyword)) // DECL: if (::std::optional<::MyBitEnum> attr = ::symbolizeEnum<::MyBitEnum>(enumKeyword))
// DECL: return *attr; // DECL: return *attr;
// DECL: return parser.emitError(loc, "invalid An example bit enum specification: ") << enumKeyword; // DECL: return parser.emitError(loc, "invalid An example bit enum specification: ") << enumKeyword;
// DECL: } // DECL: }
// DECL: inline ::llvm::raw_ostream &operator<<(::llvm::raw_ostream &p, ::MyBitEnum value) { // DECL: inline ::llvm::raw_ostream &operator<<(::llvm::raw_ostream &p, ::MyBitEnum value) {
// DECL: auto valueStr = stringifyEnum(value); // DECL: auto valueStr = stringifyEnum(value);
// DECL: switch (value) { // DECL: switch (value) {
// DECL: case ::MyBitEnum::BitGroup: // DECL: case ::MyBitEnum::BitGroup:
Show All 9 Lines
// DEF-LABEL: std::string stringifyMyBitEnum // DEF-LABEL: std::string stringifyMyBitEnum
// DEF: auto val = static_cast<uint32_t> // DEF: auto val = static_cast<uint32_t>
// DEF: if (val == 0) return "none"; // DEF: if (val == 0) return "none";
// DEF: if (1u == (1u & val)) // DEF: if (1u == (1u & val))
// DEF-NEXT: push_back("tagged") // DEF-NEXT: push_back("tagged")
// DEF: if (2u == (2u & val)) // DEF: if (2u == (2u & val))
// DEF-NEXT: push_back("Bit1") // DEF-NEXT: push_back("Bit1")
// DEF-LABEL: ::llvm::Optional<MyBitEnum> symbolizeMyBitEnum(::llvm::StringRef str) // DEF-LABEL: ::std::optional<MyBitEnum> symbolizeMyBitEnum(::llvm::StringRef str)
// DEF: if (str == "none") return MyBitEnum::None; // DEF: if (str == "none") return MyBitEnum::None;
// DEF: .Case("tagged", 1) // DEF: .Case("tagged", 1)
// DEF: .Case("Bit1", 2) // DEF: .Case("Bit1", 2)
// Test enum printer generation for non non-keyword enums. // Test enum printer generation for non non-keyword enums.
def NonKeywordBit: I32BitEnumAttrCaseBit<"Bit0", 0, "tag-ged">; def NonKeywordBit: I32BitEnumAttrCaseBit<"Bit0", 0, "tag-ged">;
def MyMixedNonKeywordBitEnum: I32BitEnumAttr<"MyMixedNonKeywordBitEnum", "An example bit enum", [ def MyMixedNonKeywordBitEnum: I32BitEnumAttr<"MyMixedNonKeywordBitEnum", "An example bit enum", [
Show All 26 Lines

mlir/test/mlir-tblgen/op-attribute.td

Show First 20 Lines • Show All 104 Lines • ▼ Show 20 Lines
// DEF: some-attr-kind AOp::getBAttrAttr() // DEF: some-attr-kind AOp::getBAttrAttr()
// DEF-NEXT: ::mlir::impl::getAttrFromSortedRange((*this)->getAttrs().begin() + 1, (*this)->getAttrs().end() - 0, getBAttrAttrName()).dyn_cast_or_null<some-attr-kind>() // DEF-NEXT: ::mlir::impl::getAttrFromSortedRange((*this)->getAttrs().begin() + 1, (*this)->getAttrs().end() - 0, getBAttrAttrName()).dyn_cast_or_null<some-attr-kind>()
// DEF: some-return-type AOp::getBAttr() { // DEF: some-return-type AOp::getBAttr() {
// DEF-NEXT: auto attr = getBAttrAttr(); // DEF-NEXT: auto attr = getBAttrAttr();
// DEF-NEXT: return attr.some-convert-from-storage(); // DEF-NEXT: return attr.some-convert-from-storage();
// DEF: some-attr-kind AOp::getCAttrAttr() // DEF: some-attr-kind AOp::getCAttrAttr()
// DEF-NEXT: ::mlir::impl::getAttrFromSortedRange((*this)->getAttrs().begin() + 1, (*this)->getAttrs().end() - 0, getCAttrAttrName()).dyn_cast_or_null<some-attr-kind>() // DEF-NEXT: ::mlir::impl::getAttrFromSortedRange((*this)->getAttrs().begin() + 1, (*this)->getAttrs().end() - 0, getCAttrAttrName()).dyn_cast_or_null<some-attr-kind>()
// DEF: ::llvm::Optional<some-return-type> AOp::getCAttr() { // DEF: ::std::optional<some-return-type> AOp::getCAttr() {
// DEF-NEXT: auto attr = getCAttrAttr() // DEF-NEXT: auto attr = getCAttrAttr()
// DEF-NEXT: return attr ? ::llvm::Optional<some-return-type>(attr.some-convert-from-storage()) : (::std::nullopt); // DEF-NEXT: return attr ? ::std::optional<some-return-type>(attr.some-convert-from-storage()) : (::std::nullopt);
// DEF: some-attr-kind AOp::getDAttrAttr() // DEF: some-attr-kind AOp::getDAttrAttr()
// DEF-NEXT: ::mlir::impl::getAttrFromSortedRange((*this)->getAttrs().begin() + 1, (*this)->getAttrs().end() - 0, getDAttrAttrName()).dyn_cast_or_null<some-attr-kind>() // DEF-NEXT: ::mlir::impl::getAttrFromSortedRange((*this)->getAttrs().begin() + 1, (*this)->getAttrs().end() - 0, getDAttrAttrName()).dyn_cast_or_null<some-attr-kind>()
// DEF: some-return-type AOp::getDAttr() { // DEF: some-return-type AOp::getDAttr() {
// DEF-NEXT: auto attr = getDAttrAttr(); // DEF-NEXT: auto attr = getDAttrAttr();
// DEF-NEXT: if (!attr) // DEF-NEXT: if (!attr)
// DEF-NEXT: return some-const-builder-call(::mlir::Builder((*this)->getContext()), 4.2).some-convert-from-storage(); // DEF-NEXT: return some-const-builder-call(::mlir::Builder((*this)->getContext()), 4.2).some-convert-from-storage();
// DEF-NEXT: return attr.some-convert-from-storage(); // DEF-NEXT: return attr.some-convert-from-storage();
// Test setter methods // Test setter methods
// --- // ---
// DEF: void AOp::setAAttrAttr(some-attr-kind attr) { // DEF: void AOp::setAAttrAttr(some-attr-kind attr) {
// DEF-NEXT: (*this)->setAttr(getAAttrAttrName(), attr); // DEF-NEXT: (*this)->setAttr(getAAttrAttrName(), attr);
// DEF: void AOp::setAAttr(some-return-type attrValue) { // DEF: void AOp::setAAttr(some-return-type attrValue) {
// DEF-NEXT: (*this)->setAttr(getAAttrAttrName(), some-const-builder-call(::mlir::Builder((*this)->getContext()), attrValue)); // DEF-NEXT: (*this)->setAttr(getAAttrAttrName(), some-const-builder-call(::mlir::Builder((*this)->getContext()), attrValue));
// DEF: void AOp::setBAttrAttr(some-attr-kind attr) { // DEF: void AOp::setBAttrAttr(some-attr-kind attr) {
// DEF-NEXT: (*this)->setAttr(getBAttrAttrName(), attr); // DEF-NEXT: (*this)->setAttr(getBAttrAttrName(), attr);
// DEF: void AOp::setBAttr(some-return-type attrValue) { // DEF: void AOp::setBAttr(some-return-type attrValue) {
// DEF-NEXT: (*this)->setAttr(getBAttrAttrName(), some-const-builder-call(::mlir::Builder((*this)->getContext()), attrValue)); // DEF-NEXT: (*this)->setAttr(getBAttrAttrName(), some-const-builder-call(::mlir::Builder((*this)->getContext()), attrValue));
// DEF: void AOp::setCAttrAttr(some-attr-kind attr) { // DEF: void AOp::setCAttrAttr(some-attr-kind attr) {
// DEF-NEXT: (*this)->setAttr(getCAttrAttrName(), attr); // DEF-NEXT: (*this)->setAttr(getCAttrAttrName(), attr);
// DEF: void AOp::setCAttr(::llvm::Optional<some-return-type> attrValue) { // DEF: void AOp::setCAttr(::std::optional<some-return-type> attrValue) {
// DEF-NEXT: if (attrValue) // DEF-NEXT: if (attrValue)
// DEF-NEXT: return (*this)->setAttr(getCAttrAttrName(), some-const-builder-call(::mlir::Builder((*this)->getContext()), *attrValue)); // DEF-NEXT: return (*this)->setAttr(getCAttrAttrName(), some-const-builder-call(::mlir::Builder((*this)->getContext()), *attrValue));
// DEF-NEXT: (*this)->removeAttr(getCAttrAttrName()); // DEF-NEXT: (*this)->removeAttr(getCAttrAttrName());
// Test remove methods // Test remove methods
// --- // ---
// DEF: ::mlir::Attribute AOp::removeCAttrAttr() { // DEF: ::mlir::Attribute AOp::removeCAttrAttr() {
▲ Show 20 Lines • Show All 95 Lines • ▼ Show 20 Lines
// DEF: some-return-type AgetOp::getBAttr() { // DEF: some-return-type AgetOp::getBAttr() {
// DEF-NEXT: auto attr = getBAttrAttr(); // DEF-NEXT: auto attr = getBAttrAttr();
// DEF-NEXT: if (!attr) // DEF-NEXT: if (!attr)
// DEF-NEXT: return some-const-builder-call(::mlir::Builder((*this)->getContext()), 4.2).some-convert-from-storage(); // DEF-NEXT: return some-const-builder-call(::mlir::Builder((*this)->getContext()), 4.2).some-convert-from-storage();
// DEF-NEXT: return attr.some-convert-from-storage(); // DEF-NEXT: return attr.some-convert-from-storage();
// DEF: some-attr-kind AgetOp::getCAttrAttr() // DEF: some-attr-kind AgetOp::getCAttrAttr()
// DEF-NEXT: return ::mlir::impl::getAttrFromSortedRange({{.*}}).dyn_cast_or_null<some-attr-kind>() // DEF-NEXT: return ::mlir::impl::getAttrFromSortedRange({{.*}}).dyn_cast_or_null<some-attr-kind>()
// DEF: ::llvm::Optional<some-return-type> AgetOp::getCAttr() { // DEF: ::std::optional<some-return-type> AgetOp::getCAttr() {
// DEF-NEXT: auto attr = getCAttrAttr() // DEF-NEXT: auto attr = getCAttrAttr()
// DEF-NEXT: return attr ? ::llvm::Optional<some-return-type>(attr.some-convert-from-storage()) : (::std::nullopt); // DEF-NEXT: return attr ? ::std::optional<some-return-type>(attr.some-convert-from-storage()) : (::std::nullopt);
// Test setter methods // Test setter methods
// --- // ---
// DEF: void AgetOp::setAAttrAttr(some-attr-kind attr) { // DEF: void AgetOp::setAAttrAttr(some-attr-kind attr) {
// DEF-NEXT: (*this)->setAttr(getAAttrAttrName(), attr); // DEF-NEXT: (*this)->setAttr(getAAttrAttrName(), attr);
// DEF: void AgetOp::setBAttrAttr(some-attr-kind attr) { // DEF: void AgetOp::setBAttrAttr(some-attr-kind attr) {
// DEF-NEXT: (*this)->setAttr(getBAttrAttrName(), attr); // DEF-NEXT: (*this)->setAttr(getBAttrAttrName(), attr);
▲ Show 20 Lines • Show All 252 Lines • Show Last 20 Lines

mlir/test/mlir-tblgen/op-decl-and-defs.td

Show First 20 Lines • Show All 55 Lines • ▼ Show 20 Lines
// CHECK: public: // CHECK: public:
// CHECK: AOpAdaptor(::mlir::ValueRange values // CHECK: AOpAdaptor(::mlir::ValueRange values
// CHECK: ::mlir::ValueRange getODSOperands(unsigned index); // CHECK: ::mlir::ValueRange getODSOperands(unsigned index);
// CHECK: ::mlir::Value getA(); // CHECK: ::mlir::Value getA();
// CHECK: ::mlir::ValueRange getB(); // CHECK: ::mlir::ValueRange getB();
// CHECK: ::mlir::IntegerAttr getAttr1Attr(); // CHECK: ::mlir::IntegerAttr getAttr1Attr();
// CHECK: uint32_t getAttr1(); // CHECK: uint32_t getAttr1();
// CHECK: ::mlir::FloatAttr getSomeAttr2Attr(); // CHECK: ::mlir::FloatAttr getSomeAttr2Attr();
// CHECK: ::llvm::Optional< ::llvm::APFloat > getSomeAttr2(); // CHECK: ::std::optional< ::llvm::APFloat > getSomeAttr2();
// CHECK: ::mlir::Region &getSomeRegion(); // CHECK: ::mlir::Region &getSomeRegion();
// CHECK: ::mlir::RegionRange getSomeRegions(); // CHECK: ::mlir::RegionRange getSomeRegions();
// CHECK: private: // CHECK: private:
// CHECK: ::mlir::ValueRange odsOperands; // CHECK: ::mlir::ValueRange odsOperands;
// CHECK: }; // CHECK: };
// CHECK: class AOp : public ::mlir::Op<AOp, ::mlir::OpTrait::AtLeastNRegions<1>::Impl, ::mlir::OpTrait::AtLeastNResults<1>::Impl, ::mlir::OpTrait::ZeroSuccessors, ::mlir::OpTrait::AtLeastNOperands<1>::Impl, ::mlir::OpTrait::OpInvariants, ::mlir::OpTrait::IsIsolatedFromAbove // CHECK: class AOp : public ::mlir::Op<AOp, ::mlir::OpTrait::AtLeastNRegions<1>::Impl, ::mlir::OpTrait::AtLeastNResults<1>::Impl, ::mlir::OpTrait::ZeroSuccessors, ::mlir::OpTrait::AtLeastNOperands<1>::Impl, ::mlir::OpTrait::OpInvariants, ::mlir::OpTrait::IsIsolatedFromAbove
// CHECK-NOT: ::mlir::OpTrait::IsIsolatedFromAbove // CHECK-NOT: ::mlir::OpTrait::IsIsolatedFromAbove
Show All 10 Lines
// CHECK: ::mlir::MutableOperandRange getBMutable(); // CHECK: ::mlir::MutableOperandRange getBMutable();
// CHECK: ::mlir::Operation::result_range getODSResults(unsigned index); // CHECK: ::mlir::Operation::result_range getODSResults(unsigned index);
// CHECK: ::mlir::TypedValue<::mlir::IntegerType> getR(); // CHECK: ::mlir::TypedValue<::mlir::IntegerType> getR();
// CHECK: ::mlir::Region &getSomeRegion(); // CHECK: ::mlir::Region &getSomeRegion();
// CHECK: ::mlir::MutableArrayRef<::mlir::Region> getSomeRegions(); // CHECK: ::mlir::MutableArrayRef<::mlir::Region> getSomeRegions();
// CHECK: ::mlir::IntegerAttr getAttr1Attr() // CHECK: ::mlir::IntegerAttr getAttr1Attr()
// CHECK: uint32_t getAttr1(); // CHECK: uint32_t getAttr1();
// CHECK: ::mlir::FloatAttr getSomeAttr2Attr() // CHECK: ::mlir::FloatAttr getSomeAttr2Attr()
// CHECK: ::llvm::Optional< ::llvm::APFloat > getSomeAttr2(); // CHECK: ::std::optional< ::llvm::APFloat > getSomeAttr2();
// CHECK: ::mlir::Attribute removeSomeAttr2Attr(); // CHECK: ::mlir::Attribute removeSomeAttr2Attr();
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, Value val); // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, Value val);
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, int integer = 0); // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, int integer = 0);
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type r, ::mlir::TypeRange s, ::mlir::Value a, ::mlir::ValueRange b, ::mlir::IntegerAttr attr1, /*optional*/::mlir::FloatAttr some_attr2, unsigned someRegionsCount) // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type r, ::mlir::TypeRange s, ::mlir::Value a, ::mlir::ValueRange b, ::mlir::IntegerAttr attr1, /*optional*/::mlir::FloatAttr some_attr2, unsigned someRegionsCount)
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::TypeRange resultTypes, ::mlir::Value a, ::mlir::ValueRange b, ::mlir::IntegerAttr attr1, /*optional*/::mlir::FloatAttr some_attr2, unsigned someRegionsCount); // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::TypeRange resultTypes, ::mlir::Value a, ::mlir::ValueRange b, ::mlir::IntegerAttr attr1, /*optional*/::mlir::FloatAttr some_attr2, unsigned someRegionsCount);
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type r, ::mlir::TypeRange s, ::mlir::Value a, ::mlir::ValueRange b, uint32_t attr1, /*optional*/::mlir::FloatAttr some_attr2, unsigned someRegionsCount) // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type r, ::mlir::TypeRange s, ::mlir::Value a, ::mlir::ValueRange b, uint32_t attr1, /*optional*/::mlir::FloatAttr some_attr2, unsigned someRegionsCount)
// CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::TypeRange resultTypes, ::mlir::Value a, ::mlir::ValueRange b, uint32_t attr1, /*optional*/::mlir::FloatAttr some_attr2, unsigned someRegionsCount); // CHECK: static void build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::TypeRange resultTypes, ::mlir::Value a, ::mlir::ValueRange b, uint32_t attr1, /*optional*/::mlir::FloatAttr some_attr2, unsigned someRegionsCount);
// CHECK: static void build(::mlir::OpBuilder &, ::mlir::OperationState &odsState, ::mlir::TypeRange resultTypes, ::mlir::ValueRange operands, ::llvm::ArrayRef<::mlir::NamedAttribute> attributes, unsigned numRegions) // CHECK: static void build(::mlir::OpBuilder &, ::mlir::OperationState &odsState, ::mlir::TypeRange resultTypes, ::mlir::ValueRange operands, ::llvm::ArrayRef<::mlir::NamedAttribute> attributes, unsigned numRegions)
▲ Show 20 Lines • Show All 253 Lines • Show Last 20 Lines

mlir/test/python/python_test_ops.td

Show First 20 Lines • Show All 65 Lines • ▼ Show 20 Lines
} }
def InferResultsOp : TestOp<"infer_results_op", [InferTypeOpInterface]> { def InferResultsOp : TestOp<"infer_results_op", [InferTypeOpInterface]> {
let arguments = (ins); let arguments = (ins);
let results = (outs AnyInteger:$single, AnyInteger:$doubled); let results = (outs AnyInteger:$single, AnyInteger:$doubled);
let extraClassDeclaration = [{ let extraClassDeclaration = [{
static ::mlir::LogicalResult inferReturnTypes( static ::mlir::LogicalResult inferReturnTypes(
::mlir::MLIRContext *context, ::llvm::Optional<::mlir::Location> location, ::mlir::MLIRContext *context, ::std::optional<::mlir::Location> location,
::mlir::ValueRange operands, ::mlir::DictionaryAttr attributes, ::mlir::ValueRange operands, ::mlir::DictionaryAttr attributes,
::mlir::RegionRange regions, ::mlir::RegionRange regions,
::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) { ::llvm::SmallVectorImpl<::mlir::Type> &inferredReturnTypes) {
::mlir::Builder b(context); ::mlir::Builder b(context);
inferredReturnTypes.push_back(b.getI32Type()); inferredReturnTypes.push_back(b.getI32Type());
inferredReturnTypes.push_back(b.getI64Type()); inferredReturnTypes.push_back(b.getI64Type());
return ::mlir::success(); return ::mlir::success();
} }
Show All 33 Lines

mlir/tools/mlir-tblgen/DialectGen.cpp

Show First 20 Lines • Show All 248 Lines • ▼ Show 20 Lines
/// The code block to generate a dialect constructor definition. /// The code block to generate a dialect constructor definition.
/// ///
/// {0}: The name of the dialect class. /// {0}: The name of the dialect class.
/// {1}: initialization code that is emitted in the ctor body before calling /// {1}: initialization code that is emitted in the ctor body before calling
/// initialize(). /// initialize().
/// {2}: The dialect parent class. /// {2}: The dialect parent class.
static const char *const dialectConstructorStr = R"( static const char *const dialectConstructorStr = R"(
{0}::{0}(::mlir::MLIRContext *context) {0}::{0}(::mlir::MLIRContext *context)
: ::mlir::{2}(getDialectNamespace(), context, ::mlir::TypeID::get<{0}>()) {{ : ::mlir::{2}(getDialectNamespace(), context, ::mlir::TypeID::get<{0}>()) {{
{1} {1}
initialize(); initialize();
} }
)"; )";
/// The code block to generate a default desturctor definition. /// The code block to generate a default desturctor definition.
/// ///
▲ Show 20 Lines • Show All 66 Lines • Show Last 20 Lines

mlir/tools/mlir-tblgen/EnumsGen.cpp

Show First 20 Lines • Show All 91 Lines • ▼ Show 20 Linesstruct FieldParser<{0}, {0}> {{
static FailureOr<{0}> parse(ParserT &parser) {{ static FailureOr<{0}> parse(ParserT &parser) {{
// Parse the keyword/string containing the enum. // Parse the keyword/string containing the enum.
std::string enumKeyword; std::string enumKeyword;
auto loc = parser.getCurrentLocation(); auto loc = parser.getCurrentLocation();
if (failed(parser.parseOptionalKeywordOrString(&enumKeyword))) if (failed(parser.parseOptionalKeywordOrString(&enumKeyword)))
return parser.emitError(loc, "expected keyword for {2}"); return parser.emitError(loc, "expected keyword for {2}");
// Symbolize the keyword. // Symbolize the keyword.
if (::llvm::Optional<{0}> attr = {1}::symbolizeEnum<{0}>(enumKeyword)) if (::std::optional<{0}> attr = {1}::symbolizeEnum<{0}>(enumKeyword))
return *attr; return *attr;
return parser.emitError(loc, "invalid {2} specification: ") << enumKeyword; return parser.emitError(loc, "invalid {2} specification: ") << enumKeyword;
} }
}; };
} // namespace mlir } // namespace mlir
namespace llvm { namespace llvm {
inline ::llvm::raw_ostream &operator<<(::llvm::raw_ostream &p, {0} value) {{ inline ::llvm::raw_ostream &operator<<(::llvm::raw_ostream &p, {0} value) {{
▲ Show 20 Lines • Show All 113 Lines • ▼ Show 20 Linesstatic void emitMaxValueFn(const Record &enumDef, raw_ostream &os) {
// Emit the function to return the max enum value // Emit the function to return the max enum value
os << formatv("inline constexpr unsigned {0}() {{\n", maxEnumValFnName); os << formatv("inline constexpr unsigned {0}() {{\n", maxEnumValFnName);
os << formatv(" return {0};\n", maxEnumVal); os << formatv(" return {0};\n", maxEnumVal);
os << "}\n\n"; os << "}\n\n";
} }
// Returns the EnumAttrCase whose value is zero if exists; returns std::nullopt // Returns the EnumAttrCase whose value is zero if exists; returns std::nullopt
// otherwise. // otherwise.
static llvm::Optional<EnumAttrCase> static std::optional<EnumAttrCase>
getAllBitsUnsetCase(llvm::ArrayRef<EnumAttrCase> cases) { getAllBitsUnsetCase(llvm::ArrayRef<EnumAttrCase> cases) {
for (auto attrCase : cases) { for (auto attrCase : cases) {
if (attrCase.getValue() == 0) if (attrCase.getValue() == 0)
return attrCase; return attrCase;
} }
return std::nullopt; return std::nullopt;
} }
▲ Show 20 Lines • Show All 137 Lines • ▼ Show 20 Lines
} }
static void emitStrToSymFnForIntEnum(const Record &enumDef, raw_ostream &os) { static void emitStrToSymFnForIntEnum(const Record &enumDef, raw_ostream &os) {
EnumAttr enumAttr(enumDef); EnumAttr enumAttr(enumDef);
StringRef enumName = enumAttr.getEnumClassName(); StringRef enumName = enumAttr.getEnumClassName();
StringRef strToSymFnName = enumAttr.getStringToSymbolFnName(); StringRef strToSymFnName = enumAttr.getStringToSymbolFnName();
auto enumerants = enumAttr.getAllCases(); auto enumerants = enumAttr.getAllCases();
os << formatv("::llvm::Optional<{0}> {1}(::llvm::StringRef str) {{\n", os << formatv("::std::optional<{0}> {1}(::llvm::StringRef str) {{\n",
enumName, strToSymFnName); enumName, strToSymFnName);
os << formatv(" return ::llvm::StringSwitch<::llvm::Optional<{0}>>(str)\n", os << formatv(" return ::llvm::StringSwitch<::std::optional<{0}>>(str)\n",
enumName); enumName);
for (const auto &enumerant : enumerants) { for (const auto &enumerant : enumerants) {
auto symbol = enumerant.getSymbol(); auto symbol = enumerant.getSymbol();
auto str = enumerant.getStr(); auto str = enumerant.getStr();
os << formatv(" .Case(\"{1}\", {0}::{2})\n", enumName, str, os << formatv(" .Case(\"{1}\", {0}::{2})\n", enumName, str,
makeIdentifier(symbol)); makeIdentifier(symbol));
} }
os << " .Default(::std::nullopt);\n"; os << " .Default(::std::nullopt);\n";
os << "}\n"; os << "}\n";
} }
static void emitStrToSymFnForBitEnum(const Record &enumDef, raw_ostream &os) { static void emitStrToSymFnForBitEnum(const Record &enumDef, raw_ostream &os) {
EnumAttr enumAttr(enumDef); EnumAttr enumAttr(enumDef);
StringRef enumName = enumAttr.getEnumClassName(); StringRef enumName = enumAttr.getEnumClassName();
std::string underlyingType = std::string(enumAttr.getUnderlyingType()); std::string underlyingType = std::string(enumAttr.getUnderlyingType());
StringRef strToSymFnName = enumAttr.getStringToSymbolFnName(); StringRef strToSymFnName = enumAttr.getStringToSymbolFnName();
StringRef separator = enumDef.getValueAsString("separator"); StringRef separator = enumDef.getValueAsString("separator");
StringRef separatorTrimmed = separator.trim(); StringRef separatorTrimmed = separator.trim();
auto enumerants = enumAttr.getAllCases(); auto enumerants = enumAttr.getAllCases();
auto allBitsUnsetCase = getAllBitsUnsetCase(enumerants); auto allBitsUnsetCase = getAllBitsUnsetCase(enumerants);
os << formatv("::llvm::Optional<{0}> {1}(::llvm::StringRef str) {{\n", os << formatv("::std::optional<{0}> {1}(::llvm::StringRef str) {{\n",
enumName, strToSymFnName); enumName, strToSymFnName);
if (allBitsUnsetCase) { if (allBitsUnsetCase) {
os << " // Special case for all bits unset.\n"; os << " // Special case for all bits unset.\n";
StringRef caseSymbol = allBitsUnsetCase->getSymbol(); StringRef caseSymbol = allBitsUnsetCase->getSymbol();
os << formatv(" if (str == \"{1}\") return {0}::{2};\n\n", enumName, os << formatv(" if (str == \"{1}\") return {0}::{2};\n\n", enumName,
allBitsUnsetCase->getStr(), makeIdentifier(caseSymbol)); allBitsUnsetCase->getStr(), makeIdentifier(caseSymbol));
} }
// Split the string to get symbols for all the bits. // Split the string to get symbols for all the bits.
os << " ::llvm::SmallVector<::llvm::StringRef, 2> symbols;\n"; os << " ::llvm::SmallVector<::llvm::StringRef, 2> symbols;\n";
// Remove whitespace from the separator string when parsing. // Remove whitespace from the separator string when parsing.
os << formatv(" str.split(symbols, \"{0}\");\n\n", separatorTrimmed); os << formatv(" str.split(symbols, \"{0}\");\n\n", separatorTrimmed);
os << formatv(" {0} val = 0;\n", underlyingType); os << formatv(" {0} val = 0;\n", underlyingType);
os << " for (auto symbol : symbols) {\n"; os << " for (auto symbol : symbols) {\n";
// Convert each symbol to the bit ordinal and set the corresponding bit. // Convert each symbol to the bit ordinal and set the corresponding bit.
os << formatv(" auto bit = " os << formatv(" auto bit = "
"llvm::StringSwitch<::llvm::Optional<{0}>>(symbol.trim())\n", "llvm::StringSwitch<::std::optional<{0}>>(symbol.trim())\n",
underlyingType); underlyingType);
for (const auto &enumerant : enumerants) { for (const auto &enumerant : enumerants) {
// Skip the special enumerant for None. // Skip the special enumerant for None.
if (auto val = enumerant.getValue()) if (auto val = enumerant.getValue())
os.indent(6) << formatv(".Case(\"{0}\", {1})\n", enumerant.getStr(), val); os.indent(6) << formatv(".Case(\"{0}\", {1})\n", enumerant.getStr(), val);
} }
os.indent(6) << ".Default(::std::nullopt);\n"; os.indent(6) << ".Default(::std::nullopt);\n";
Show All 14 Linesstatic void emitUnderlyingToSymFnForIntEnum(const Record &enumDef,
// Avoid generating the underlying value to symbol conversion function if // Avoid generating the underlying value to symbol conversion function if
// there is an enumerant without explicit value. // there is an enumerant without explicit value.
if (llvm::any_of(enumerants, [](EnumAttrCase enumerant) { if (llvm::any_of(enumerants, [](EnumAttrCase enumerant) {
return enumerant.getValue() < 0; return enumerant.getValue() < 0;
})) }))
return; return;
os << formatv("::llvm::Optional<{0}> {1}({2} value) {{\n", enumName, os << formatv("::std::optional<{0}> {1}({2} value) {{\n", enumName,
underlyingToSymFnName, underlyingToSymFnName,
underlyingType.empty() ? std::string("unsigned") underlyingType.empty() ? std::string("unsigned")
: underlyingType) : underlyingType)
<< " switch (value) {\n"; << " switch (value) {\n";
for (const auto &enumerant : enumerants) { for (const auto &enumerant : enumerants) {
auto symbol = enumerant.getSymbol(); auto symbol = enumerant.getSymbol();
auto value = enumerant.getValue(); auto value = enumerant.getValue();
os << formatv(" case {0}: return {1}::{2};\n", value, enumName, os << formatv(" case {0}: return {1}::{2};\n", value, enumName,
▲ Show 20 Lines • Show All 66 Lines • ▼ Show 20 Linesstatic void emitUnderlyingToSymFnForBitEnum(const Record &enumDef,
raw_ostream &os) { raw_ostream &os) {
EnumAttr enumAttr(enumDef); EnumAttr enumAttr(enumDef);
StringRef enumName = enumAttr.getEnumClassName(); StringRef enumName = enumAttr.getEnumClassName();
std::string underlyingType = std::string(enumAttr.getUnderlyingType()); std::string underlyingType = std::string(enumAttr.getUnderlyingType());
StringRef underlyingToSymFnName = enumAttr.getUnderlyingToSymbolFnName(); StringRef underlyingToSymFnName = enumAttr.getUnderlyingToSymbolFnName();
auto enumerants = enumAttr.getAllCases(); auto enumerants = enumAttr.getAllCases();
auto allBitsUnsetCase = getAllBitsUnsetCase(enumerants); auto allBitsUnsetCase = getAllBitsUnsetCase(enumerants);
os << formatv("::llvm::Optional<{0}> {1}({2} value) {{\n", enumName, os << formatv("::std::optional<{0}> {1}({2} value) {{\n", enumName,
underlyingToSymFnName, underlyingType); underlyingToSymFnName, underlyingType);
if (allBitsUnsetCase) { if (allBitsUnsetCase) {
os << " // Special case for all bits unset.\n"; os << " // Special case for all bits unset.\n";
os << formatv(" if (value == 0) return {0}::{1};\n\n", enumName, os << formatv(" if (value == 0) return {0}::{1};\n\n", enumName,
makeIdentifier(allBitsUnsetCase->getSymbol())); makeIdentifier(allBitsUnsetCase->getSymbol()));
} }
int64_t validBits = enumDef.getValueAsInt("validBits"); int64_t validBits = enumDef.getValueAsInt("validBits");
os << formatv(" if (value & ~static_cast<{0}>({1}u)) return std::nullopt;\n", os << formatv(" if (value & ~static_cast<{0}>({1}u)) return std::nullopt;\n",
Show All 23 Linesstatic void emitEnumDecl(const Record &enumDef, raw_ostream &os) {
// Emit the enum class definition // Emit the enum class definition
emitEnumClass(enumDef, enumName, underlyingType, description, enumerants, os); emitEnumClass(enumDef, enumName, underlyingType, description, enumerants, os);
// Emit conversion function declarations // Emit conversion function declarations
if (llvm::all_of(enumerants, [](EnumAttrCase enumerant) { if (llvm::all_of(enumerants, [](EnumAttrCase enumerant) {
return enumerant.getValue() >= 0; return enumerant.getValue() >= 0;
})) { })) {
os << formatv( os << formatv(
"::llvm::Optional<{0}> {1}({2});\n", enumName, underlyingToSymFnName, "::std::optional<{0}> {1}({2});\n", enumName, underlyingToSymFnName,
underlyingType.empty() ? std::string("unsigned") : underlyingType); underlyingType.empty() ? std::string("unsigned") : underlyingType);
} }
os << formatv("{2} {1}({0});\n", enumName, symToStrFnName, symToStrFnRetType); os << formatv("{2} {1}({0});\n", enumName, symToStrFnName, symToStrFnRetType);
os << formatv("::llvm::Optional<{0}> {1}(::llvm::StringRef);\n", enumName, os << formatv("::std::optional<{0}> {1}(::llvm::StringRef);\n", enumName,
strToSymFnName); strToSymFnName);
if (enumAttr.isBitEnum()) { if (enumAttr.isBitEnum()) {
emitOperators(enumDef, os); emitOperators(enumDef, os);
} else { } else {
emitMaxValueFn(enumDef, os); emitMaxValueFn(enumDef, os);
} }
// Generate a generic `stringifyEnum` function that forwards to the method // Generate a generic `stringifyEnum` function that forwards to the method
// specified by the user. // specified by the user.
const char *const stringifyEnumStr = R"( const char *const stringifyEnumStr = R"(
inline {0} stringifyEnum({1} enumValue) {{ inline {0} stringifyEnum({1} enumValue) {{
return {2}(enumValue); return {2}(enumValue);
} }
)"; )";
os << formatv(stringifyEnumStr, symToStrFnRetType, enumName, symToStrFnName); os << formatv(stringifyEnumStr, symToStrFnRetType, enumName, symToStrFnName);
// Generate a generic `symbolizeEnum` function that forwards to the method // Generate a generic `symbolizeEnum` function that forwards to the method
// specified by the user. // specified by the user.
const char *const symbolizeEnumStr = R"( const char *const symbolizeEnumStr = R"(
template <typename EnumType> template <typename EnumType>
::llvm::Optional<EnumType> symbolizeEnum(::llvm::StringRef); ::std::optional<EnumType> symbolizeEnum(::llvm::StringRef);
template <> template <>
inline ::llvm::Optional<{0}> symbolizeEnum<{0}>(::llvm::StringRef str) { inline ::std::optional<{0}> symbolizeEnum<{0}>(::llvm::StringRef str) {
return {1}(str); return {1}(str);
} }
)"; )";
os << formatv(symbolizeEnumStr, enumName, strToSymFnName); os << formatv(symbolizeEnumStr, enumName, strToSymFnName);
const char *const attrClassDecl = R"( const char *const attrClassDecl = R"(
class {1} : public ::mlir::{2} { class {1} : public ::mlir::{2} {
public: public:
▲ Show 20 Lines • Show All 86 Lines • Show Last 20 Lines

mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp

Show First 20 Lines • Show All 1,123 Lines • ▼ Show 20 Lines auto emitAttrWithReturnType = [&](StringRef setterName, StringRef getterName,
// Build the method using the correct parameter type depending on // Build the method using the correct parameter type depending on
// optionality. // optionality.
Method *method = nullptr; Method *method = nullptr;
if (isUnitAttr) if (isUnitAttr)
method = createMethod("bool"); method = createMethod("bool");
else if (isOptional) else if (isOptional)
method = method =
createMethod("::llvm::Optional<" + baseAttr.getReturnType() + ">"); createMethod("::std::optional<" + baseAttr.getReturnType() + ">");
else else
method = createMethod(attr.getReturnType()); method = createMethod(attr.getReturnType());
if (!method) if (!method)
return; return;
// If the value isn't optional, just set it directly. // If the value isn't optional, just set it directly.
if (!isOptional) { if (!isOptional) {
method->body() << formatv( method->body() << formatv(
" (*this)->setAttr({0}AttrName(), {1});", getterName, " (*this)->setAttr({0}AttrName(), {1});", getterName,
constBuildAttrFromParam(attr, fctx, "attrValue")); constBuildAttrFromParam(attr, fctx, "attrValue"));
return; return;
} }
// Otherwise, we only set if the provided value is valid. If it isn't, we // Otherwise, we only set if the provided value is valid. If it isn't, we
// remove the attribute. // remove the attribute.
// TODO: Handle unit attr parameters specially, given that it is treated as // TODO: Handle unit attr parameters specially, given that it is treated as
// optional but not in the same way as the others (i.e. it uses bool over // optional but not in the same way as the others (i.e. it uses bool over
// llvm::Optional<>). // std::optional<>).
StringRef paramStr = isUnitAttr ? "attrValue" : "*attrValue"; StringRef paramStr = isUnitAttr ? "attrValue" : "*attrValue";
const char *optionalCodeBody = R"( const char *optionalCodeBody = R"(
if (attrValue) if (attrValue)
return (*this)->setAttr({0}AttrName(), {1}); return (*this)->setAttr({0}AttrName(), {1});
(*this)->removeAttr({0}AttrName());)"; (*this)->removeAttr({0}AttrName());)";
method->body() << formatv( method->body() << formatv(
optionalCodeBody, getterName, optionalCodeBody, getterName,
constBuildAttrFromParam(baseAttr, fctx, paramStr)); constBuildAttrFromParam(baseAttr, fctx, paramStr));
▲ Show 20 Lines • Show All 1,784 Lines • ▼ Show 20 LinesOpOperandAdaptorEmitter::OpOperandAdaptorEmitter(
const Operator &op, const Operator &op,
const StaticVerifierFunctionEmitter &staticVerifierEmitter) const StaticVerifierFunctionEmitter &staticVerifierEmitter)
: op(op), adaptor(op.getAdaptorName()), : op(op), adaptor(op.getAdaptorName()),
staticVerifierEmitter(staticVerifierEmitter), staticVerifierEmitter(staticVerifierEmitter),
emitHelper(op, /*emitForOp=*/false) { emitHelper(op, /*emitForOp=*/false) {
adaptor.addField("::mlir::ValueRange", "odsOperands"); adaptor.addField("::mlir::ValueRange", "odsOperands");
adaptor.addField("::mlir::DictionaryAttr", "odsAttrs"); adaptor.addField("::mlir::DictionaryAttr", "odsAttrs");
adaptor.addField("::mlir::RegionRange", "odsRegions"); adaptor.addField("::mlir::RegionRange", "odsRegions");
adaptor.addField("::llvm::Optional<::mlir::OperationName>", "odsOpName"); adaptor.addField("::std::optional<::mlir::OperationName>", "odsOpName");
const auto *attrSizedOperands = const auto *attrSizedOperands =
op.getTrait("::m::OpTrait::AttrSizedOperandSegments"); op.getTrait("::m::OpTrait::AttrSizedOperandSegments");
{ {
SmallVector<MethodParameter> paramList; SmallVector<MethodParameter> paramList;
paramList.emplace_back("::mlir::ValueRange", "values"); paramList.emplace_back("::mlir::ValueRange", "values");
paramList.emplace_back("::mlir::DictionaryAttr", "attrs", paramList.emplace_back("::mlir::DictionaryAttr", "attrs",
attrSizedOperands ? "" : "nullptr"); attrSizedOperands ? "" : "nullptr");
▲ Show 20 Lines • Show All 237 Lines • Show Last 20 Lines

mlir/tools/mlir-tblgen/OpFormatGen.cpp

Show First 20 Lines • Show All 938 Lines • ▼ Show 20 Linesstatic void genCustomDirectiveParser(CustomDirective *dir, MethodBody &body) {
// * Set the location of operand variables. // * Set the location of operand variables.
for (FormatElement *param : dir->getArguments()) { for (FormatElement *param : dir->getArguments()) {
if (auto *operand = dyn_cast<OperandVariable>(param)) { if (auto *operand = dyn_cast<OperandVariable>(param)) {
auto *var = operand->getVar(); auto *var = operand->getVar();
body << " " << var->name body << " " << var->name
<< "OperandsLoc = parser.getCurrentLocation();\n"; << "OperandsLoc = parser.getCurrentLocation();\n";
if (var->isOptional()) { if (var->isOptional()) {
body << llvm::formatv( body << llvm::formatv(
" ::llvm::Optional<::mlir::OpAsmParser::UnresolvedOperand> " " ::std::optional<::mlir::OpAsmParser::UnresolvedOperand> "
"{0}Operand;\n", "{0}Operand;\n",
var->name); var->name);
} else if (var->isVariadicOfVariadic()) { } else if (var->isVariadicOfVariadic()) {
body << llvm::formatv(" " body << llvm::formatv(" "
"::llvm::SmallVector<::llvm::SmallVector<::mlir::" "::llvm::SmallVector<::llvm::SmallVector<::mlir::"
"OpAsmParser::UnresolvedOperand>> " "OpAsmParser::UnresolvedOperand>> "
"{0}OperandGroups;\n", "{0}OperandGroups;\n",
var->name); var->name);
Show All 12 Lines for (FormatElement *param : dir->getArguments()) {
} else if (auto *dir = dyn_cast<RefDirective>(param)) { } else if (auto *dir = dyn_cast<RefDirective>(param)) {
FormatElement *input = dir->getArg(); FormatElement *input = dir->getArg();
if (auto *operand = dyn_cast<OperandVariable>(input)) { if (auto *operand = dyn_cast<OperandVariable>(input)) {
if (!operand->getVar()->isOptional()) if (!operand->getVar()->isOptional())
continue; continue;
body << llvm::formatv( body << llvm::formatv(
" {0} {1}Operand = {1}Operands.empty() ? {0}() : " " {0} {1}Operand = {1}Operands.empty() ? {0}() : "
"{1}Operands[0];\n", "{1}Operands[0];\n",
"::llvm::Optional<::mlir::OpAsmParser::UnresolvedOperand>", "::std::optional<::mlir::OpAsmParser::UnresolvedOperand>",
operand->getVar()->name); operand->getVar()->name);
} else if (auto *type = dyn_cast<TypeDirective>(input)) { } else if (auto *type = dyn_cast<TypeDirective>(input)) {
ArgumentLengthKind lengthKind; ArgumentLengthKind lengthKind;
StringRef listName = getTypeListName(type->getArg(), lengthKind); StringRef listName = getTypeListName(type->getArg(), lengthKind);
if (lengthKind == ArgumentLengthKind::Optional) { if (lengthKind == ArgumentLengthKind::Optional) {
body << llvm::formatv(" ::mlir::Type {0}Type = {0}Types.empty() ? " body << llvm::formatv(" ::mlir::Type {0}Type = {0}Types.empty() ? "
"::mlir::Type() : {0}Types[0];\n", "::mlir::Type() : {0}Types[0];\n",
▲ Show 20 Lines • Show All 2,327 Lines • Show Last 20 Lines

mlir/tools/mlir-tblgen/SPIRVUtilsGen.cpp

Show First 20 Lines • Show All 340 Lines • ▼ Show 20 Linesstatic void emitAvailabilityQueryForIntEnum(const Record &enumDef,
// availability classes. // availability classes.
for (const EnumAttrCase &enumerant : enumerants) for (const EnumAttrCase &enumerant : enumerants)
for (const Availability &avail : getAvailabilities(enumerant.getDef())) for (const Availability &avail : getAvailabilities(enumerant.getDef()))
classCaseMap[avail.getClass()].push_back({enumerant, avail}); classCaseMap[avail.getClass()].push_back({enumerant, avail});
for (const auto &classCasePair : classCaseMap) { for (const auto &classCasePair : classCaseMap) {
Availability avail = classCasePair.getValue().front().second; Availability avail = classCasePair.getValue().front().second;
os << formatv("llvm::Optional<{0}> {1}({2} value) {{\n", os << formatv("std::optional<{0}> {1}({2} value) {{\n",
avail.getMergeInstanceType(), avail.getQueryFnName(), avail.getMergeInstanceType(), avail.getQueryFnName(),
enumName); enumName);
os << " switch (value) {\n"; os << " switch (value) {\n";
for (const auto &caseSpecPair : classCasePair.getValue()) { for (const auto &caseSpecPair : classCasePair.getValue()) {
EnumAttrCase enumerant = caseSpecPair.first; EnumAttrCase enumerant = caseSpecPair.first;
Availability avail = caseSpecPair.second; Availability avail = caseSpecPair.second;
os << formatv(" case {0}::{1}: { {2} return {3}({4}); }\n", enumName, os << formatv(" case {0}::{1}: { {2} return {3}({4}); }\n", enumName,
Show All 25 Linesstatic void emitAvailabilityQueryForBitEnum(const Record &enumDef,
// availability classes. // availability classes.
for (const EnumAttrCase &enumerant : enumerants) for (const EnumAttrCase &enumerant : enumerants)
for (const Availability &avail : getAvailabilities(enumerant.getDef())) for (const Availability &avail : getAvailabilities(enumerant.getDef()))
classCaseMap[avail.getClass()].push_back({enumerant, avail}); classCaseMap[avail.getClass()].push_back({enumerant, avail});
for (const auto &classCasePair : classCaseMap) { for (const auto &classCasePair : classCaseMap) {
Availability avail = classCasePair.getValue().front().second; Availability avail = classCasePair.getValue().front().second;
os << formatv("llvm::Optional<{0}> {1}({2} value) {{\n", os << formatv("std::optional<{0}> {1}({2} value) {{\n",
avail.getMergeInstanceType(), avail.getQueryFnName(), avail.getMergeInstanceType(), avail.getQueryFnName(),
enumName); enumName);
os << formatv( os << formatv(
" assert(::llvm::countPopulation(static_cast<{0}>(value)) <= 1" " assert(::llvm::countPopulation(static_cast<{0}>(value)) <= 1"
" && \"cannot have more than one bit set\");\n", " && \"cannot have more than one bit set\");\n",
underlyingType); underlyingType);
Show All 28 Linesstatic void emitEnumDecl(const Record &enumDef, raw_ostream &os) {
// Place all availability specifications to their corresponding // Place all availability specifications to their corresponding
// availability classes. // availability classes.
for (const EnumAttrCase &enumerant : enumerants) for (const EnumAttrCase &enumerant : enumerants)
for (const Availability &avail : getAvailabilities(enumerant.getDef())) { for (const Availability &avail : getAvailabilities(enumerant.getDef())) {
StringRef className = avail.getClass(); StringRef className = avail.getClass();
if (handledClasses.count(className)) if (handledClasses.count(className))
continue; continue;
os << formatv("llvm::Optional<{0}> {1}({2} value);\n", os << formatv("std::optional<{0}> {1}({2} value);\n",
avail.getMergeInstanceType(), avail.getQueryFnName(), avail.getMergeInstanceType(), avail.getQueryFnName(),
enumName); enumName);
handledClasses.insert(className); handledClasses.insert(className);
} }
for (auto ns : llvm::reverse(namespaces)) for (auto ns : llvm::reverse(namespaces))
os << "} // namespace " << ns << "\n"; os << "} // namespace " << ns << "\n";
} }
▲ Show 20 Lines • Show All 1,017 Lines • Show Last 20 Lines

mlir/unittests/Interfaces/ControlFlowInterfacesTest.cpp

Show All 31 Linesstruct MutuallyExclusiveRegionsOp
using Op::Op; using Op::Op;
static ArrayRef<StringRef> getAttributeNames() { return {}; } static ArrayRef<StringRef> getAttributeNames() { return {}; }
static StringRef getOperationName() { static StringRef getOperationName() {
return "cftest.mutually_exclusive_regions_op"; return "cftest.mutually_exclusive_regions_op";
} }
// Regions have no successors. // Regions have no successors.
void getSuccessorRegions(Optional<unsigned> index, void getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute> operands, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) {} SmallVectorImpl<RegionSuccessor> &regions) {}
}; };
/// All regions of this op call each other in a large circle. /// All regions of this op call each other in a large circle.
struct LoopRegionsOp struct LoopRegionsOp
: public Op<LoopRegionsOp, RegionBranchOpInterface::Trait> { : public Op<LoopRegionsOp, RegionBranchOpInterface::Trait> {
using Op::Op; using Op::Op;
static const unsigned kNumRegions = 3; static const unsigned kNumRegions = 3;
static ArrayRef<StringRef> getAttributeNames() { return {}; } static ArrayRef<StringRef> getAttributeNames() { return {}; }
static StringRef getOperationName() { return "cftest.loop_regions_op"; } static StringRef getOperationName() { return "cftest.loop_regions_op"; }
void getSuccessorRegions(Optional<unsigned> index, void getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute> operands, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
if (index) { if (index) {
if (*index == 1) if (*index == 1)
// This region also branches back to the parent. // This region also branches back to the parent.
regions.push_back(RegionSuccessor()); regions.push_back(RegionSuccessor());
regions.push_back( regions.push_back(
RegionSuccessor(&getOperation()->getRegion(*index % kNumRegions))); RegionSuccessor(&getOperation()->getRegion(*index % kNumRegions)));
} }
} }
}; };
/// Each region branches back it itself or the parent. /// Each region branches back it itself or the parent.
struct DoubleLoopRegionsOp struct DoubleLoopRegionsOp
: public Op<DoubleLoopRegionsOp, RegionBranchOpInterface::Trait> { : public Op<DoubleLoopRegionsOp, RegionBranchOpInterface::Trait> {
using Op::Op; using Op::Op;
static ArrayRef<StringRef> getAttributeNames() { return {}; } static ArrayRef<StringRef> getAttributeNames() { return {}; }
static StringRef getOperationName() { static StringRef getOperationName() {
return "cftest.double_loop_regions_op"; return "cftest.double_loop_regions_op";
} }
void getSuccessorRegions(Optional<unsigned> index, void getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute> operands, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
if (index.has_value()) { if (index.has_value()) {
regions.push_back(RegionSuccessor()); regions.push_back(RegionSuccessor());
regions.push_back(RegionSuccessor(&getOperation()->getRegion(*index))); regions.push_back(RegionSuccessor(&getOperation()->getRegion(*index)));
} }
} }
}; };
/// Regions are executed sequentially. /// Regions are executed sequentially.
struct SequentialRegionsOp struct SequentialRegionsOp
: public Op<SequentialRegionsOp, RegionBranchOpInterface::Trait> { : public Op<SequentialRegionsOp, RegionBranchOpInterface::Trait> {
using Op::Op; using Op::Op;
static ArrayRef<StringRef> getAttributeNames() { return {}; } static ArrayRef<StringRef> getAttributeNames() { return {}; }
static StringRef getOperationName() { return "cftest.sequential_regions_op"; } static StringRef getOperationName() { return "cftest.sequential_regions_op"; }
// Region 0 has Region 1 as a successor. // Region 0 has Region 1 as a successor.
void getSuccessorRegions(Optional<unsigned> index, void getSuccessorRegions(std::optional<unsigned> index,
ArrayRef<Attribute> operands, ArrayRef<Attribute> operands,
SmallVectorImpl<RegionSuccessor> &regions) { SmallVectorImpl<RegionSuccessor> &regions) {
if (index == 0u) { if (index == 0u) {
Operation *thisOp = this->getOperation(); Operation *thisOp = this->getOperation();
regions.push_back(RegionSuccessor(&thisOp->getRegion(1))); regions.push_back(RegionSuccessor(&thisOp->getRegion(1)));
} }
} }
}; };
▲ Show 20 Lines • Show All 153 Lines • Show Last 20 Lines

mlir/unittests/TableGen/EnumsGenTest.cpp

Show First 20 Lines • Show All 48 Lines • ▼ Show 20 Lines
} }
TEST(EnumsGenTest, GeneratedSymbolToStringFn) { TEST(EnumsGenTest, GeneratedSymbolToStringFn) {
EXPECT_EQ(ConvertToString(FooEnum::CaseA), "CaseA"); EXPECT_EQ(ConvertToString(FooEnum::CaseA), "CaseA");
EXPECT_EQ(ConvertToString(FooEnum::CaseB), "CaseB"); EXPECT_EQ(ConvertToString(FooEnum::CaseB), "CaseB");
} }
TEST(EnumsGenTest, GeneratedStringToSymbolFn) { TEST(EnumsGenTest, GeneratedStringToSymbolFn) {
EXPECT_EQ(llvm::Optional<FooEnum>(FooEnum::CaseA), ConvertToEnum("CaseA")); EXPECT_EQ(std::optional<FooEnum>(FooEnum::CaseA), ConvertToEnum("CaseA"));
EXPECT_EQ(llvm::Optional<FooEnum>(FooEnum::CaseB), ConvertToEnum("CaseB")); EXPECT_EQ(std::optional<FooEnum>(FooEnum::CaseB), ConvertToEnum("CaseB"));
EXPECT_EQ(std::nullopt, ConvertToEnum("X")); EXPECT_EQ(std::nullopt, ConvertToEnum("X"));
} }
TEST(EnumsGenTest, GeneratedUnderlyingType) { TEST(EnumsGenTest, GeneratedUnderlyingType) {
bool v = std::is_same<uint32_t, std::underlying_type<I32Enum>::type>::value; bool v = std::is_same<uint32_t, std::underlying_type<I32Enum>::type>::value;
EXPECT_TRUE(v); EXPECT_TRUE(v);
} }
▲ Show 20 Lines • Show All 129 Lines • Show Last 20 Lines