This is an archive of the discontinued LLVM Phabricator instance.

Differential D78647

[MLIR] Lower GenericAtomicRMWOp to llvm.cmpxchg.
ClosedPublic

Authored by pifon2a on Apr 22 2020, 8:26 AM.

Details

Reviewers
herhut
flaub
ftynse
Commits
rG21caba599e6c: [MLIR] Lower GenericAtomicRMWOp to llvm.cmpxchg.
Summary

Lowering is pretty much a copy of AtomicRMWOp -> llvm.cmpxchg
pattern.

Diff Detail

Event Timeline

pifon2a created this revision.Apr 22 2020, 8:26 AM
Herald added a project: Restricted Project. · View Herald TranscriptApr 22 2020, 8:26 AM
Herald added subscribers: llvm-commits, frgossen, grosul1 and 13 others. · View Herald Transcript
pifon2a added a comment.Apr 22 2020, 8:29 AM

What should we do with the lowering for AtomicRMWOp "minf", "maxf"? Should we lower it to GenericAtomicRMWOp and then to LLVM? Or maybe we should not use AtomicRMWOp for these cases at all? It can always be replaced by GenericAtomicRMWOp.

Harbormaster failed remote builds in B54259: Diff 259299!Apr 22 2020, 9:12 AM
flaub added a comment.Apr 22 2020, 11:25 AM

OK, I could see always lowering enums that would lower to cmpxchg via the generic op. But at the moment we have a higher layer that uses the enumerated version for min/max, so I'd prefer to don't remove that. It's much easier to pattern match (even declaratively) the enumerated version than the one with a region.

flaub accepted this revision.Apr 22 2020, 11:27 AM
This revision is now accepted and ready to land.Apr 22 2020, 11:27 AM
pifon2a added a comment.Apr 23 2020, 12:17 AM

@flaub Thanks, lowering of AtomicRMWOp -> GenericAtomicRMWOp then it is.

Herald added a subscriber: Kayjukh. · View Herald TranscriptApr 23 2020, 12:17 AM
Closed by commit rG21caba599e6c: [MLIR] Lower GenericAtomicRMWOp to llvm.cmpxchg. (authored by pifon2a). · Explain WhyApr 23 2020, 1:02 AM
This revision was automatically updated to reflect the committed changes.
ftynse added a subscriber: ftynse.Apr 23 2020, 1:16 AM
ftynse added inline comments.
mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp
2796

This is not allowed in conversion patterns (it is not undoable). I've recently introduced a functionality to create block in a rewriter-compatible way, rewriter.createBlock(initBlock->getParent(), std::next(Region::iterator(initBlock)), valueType); should work.

pifon2a marked an inline comment as done.Apr 23 2020, 2:04 AM
pifon2a added inline comments.
mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp
2796

Thanks a lot, Alex. Good to know that it is not allowed. I ll send a patch to fix this.

Herald added a reviewer: ftynse. · View Herald TranscriptApr 23 2020, 2:04 AM

Revision Contents

PathSize
mlir/
include/
mlir/
Dialect/
StandardOps/
IR/
3 lines
lib/
Conversion/
StandardToLLVM/
100 lines
test/
Conversion/
StandardToLLVM/
24 lines

Diff 259493

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

Show First 20 Lines • Show All 533 Lines • ▼ Show 20 Lines OpBuilder getBodyBuilder() {
assert(!body().empty() && "Unexpected empty 'body' region."); assert(!body().empty() && "Unexpected empty 'body' region.");
Block &block = body().front(); Block &block = body().front();
return OpBuilder(&block, block.end()); return OpBuilder(&block, block.end());
} }
// The value stored in memref[ivs]. // The value stored in memref[ivs].
Value getCurrentValue() { Value getCurrentValue() {
return body().front().getArgument(0); return body().front().getArgument(0);
} }
MemRefType getMemRefType() {
return memref().getType().cast<MemRefType>();
}
}]; }];
} }
def AtomicYieldOp : Std_Op<"atomic_yield", [ def AtomicYieldOp : Std_Op<"atomic_yield", [
HasParent<"GenericAtomicRMWOp">, HasParent<"GenericAtomicRMWOp">,
NoSideEffect, NoSideEffect,
Terminator Terminator
]> { ]> {
▲ Show 20 Lines • Show All 2,376 Lines • Show Last 20 Lines

mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp

Show All 11 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
#include "../PassDetail.h" #include "../PassDetail.h"
#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h" #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVMPass.h" #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVMPass.h"
#include "mlir/Dialect/LLVMIR/LLVMDialect.h" #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
#include "mlir/Dialect/StandardOps/IR/Ops.h" #include "mlir/Dialect/StandardOps/IR/Ops.h"
#include "mlir/IR/Attributes.h" #include "mlir/IR/Attributes.h"
#include "mlir/IR/BlockAndValueMapping.h"
#include "mlir/IR/Builders.h" #include "mlir/IR/Builders.h"
#include "mlir/IR/MLIRContext.h" #include "mlir/IR/MLIRContext.h"
#include "mlir/IR/Module.h" #include "mlir/IR/Module.h"
#include "mlir/IR/PatternMatch.h" #include "mlir/IR/PatternMatch.h"
#include "mlir/IR/TypeUtilities.h" #include "mlir/IR/TypeUtilities.h"
#include "mlir/Support/LogicalResult.h" #include "mlir/Support/LogicalResult.h"
#include "mlir/Transforms/DialectConversion.h" #include "mlir/Transforms/DialectConversion.h"
#include "mlir/Transforms/Passes.h" #include "mlir/Transforms/Passes.h"
▲ Show 20 Lines • Show All 2,713 Lines • ▼ Show 20 Lines matchAndRewrite(Operation *op, ArrayRef<Value> operands,
// The 'result' of the atomic_rmw op is the newly loaded value. // The 'result' of the atomic_rmw op is the newly loaded value.
rewriter.replaceOp(op, {newLoaded}); rewriter.replaceOp(op, {newLoaded});
return success(); return success();
} }
}; };
/// Wrap a llvm.cmpxchg operation in a while loop so that the operation can be
/// retried until it succeeds in atomically storing a new value into memory.
///
/// +---------------------------------+
/// | <code before the AtomicRMWOp> |
/// | <compute initial %loaded> |
/// | br loop(%loaded) |
/// +---------------------------------+
/// |
/// -------| |
/// | v v
/// | +--------------------------------+
/// | | loop(%loaded): |
/// | | <body contents> |
/// | | %pair = cmpxchg |
/// | | %ok = %pair[0] |
/// | | %new = %pair[1] |
/// | | cond_br %ok, end, loop(%new) |
/// | +--------------------------------+
/// | | |
/// |----------- |
/// v
/// +--------------------------------+
/// | end: |
/// | <code after the AtomicRMWOp> |
/// +--------------------------------+
///
struct GenericAtomicRMWOpLowering
: public LoadStoreOpLowering<GenericAtomicRMWOp> {
using Base::Base;
LogicalResult
matchAndRewrite(Operation *op, ArrayRef<Value> operands,
ConversionPatternRewriter &rewriter) const override {
auto atomicOp = cast<GenericAtomicRMWOp>(op);
auto loc = op->getLoc();
OperandAdaptor<GenericAtomicRMWOp> adaptor(operands);
LLVM::LLVMType valueType =
typeConverter.convertType(atomicOp.getResult().getType())
.cast<LLVM::LLVMType>();
// Split the block into initial, loop, and ending parts.
auto *initBlock = rewriter.getInsertionBlock();
auto initPosition = rewriter.getInsertionPoint();
auto *loopBlock = rewriter.splitBlock(initBlock, initPosition);
auto loopArgument = loopBlock->addArgument(valueType);
ftynseUnsubmitted
Not Done
ReplyInline Actions

This is not allowed in conversion patterns (it is not undoable). I've recently introduced a functionality to create block in a rewriter-compatible way, rewriter.createBlock(initBlock->getParent(), std::next(Region::iterator(initBlock)), valueType); should work.

ftynse: This is not allowed in conversion patterns (it is not undoable). I've recently introduced a…
pifon2aAuthorUnsubmitted
Done
ReplyInline Actions

Thanks a lot, Alex. Good to know that it is not allowed. I ll send a patch to fix this.

pifon2a: Thanks a lot, Alex. Good to know that it is not allowed. I ll send a patch to fix this.
auto loopPosition = rewriter.getInsertionPoint();
auto *endBlock = rewriter.splitBlock(loopBlock, loopPosition);
// Compute the loaded value and branch to the loop block.
rewriter.setInsertionPointToEnd(initBlock);
auto memRefType = atomicOp.memref().getType().cast<MemRefType>();
auto dataPtr = getDataPtr(loc, memRefType, adaptor.memref(),
adaptor.indices(), rewriter, getModule());
Value init = rewriter.create<LLVM::LoadOp>(loc, dataPtr);
rewriter.create<LLVM::BrOp>(loc, init, loopBlock);
// Prepare the body of the loop block.
rewriter.setInsertionPointToStart(loopBlock);
auto boolType = LLVM::LLVMType::getInt1Ty(&getDialect());
// Clone the GenericAtomicRMWOp region and extract the result.
BlockAndValueMapping mapping;
mapping.map(atomicOp.getCurrentValue(), loopArgument);
Block &entryBlock = atomicOp.body().front();
for (auto &nestedOp : entryBlock.without_terminator()) {
Operation *clone = rewriter.clone(nestedOp, mapping);
mapping.map(nestedOp.getResults(), clone->getResults());
}
Value result = mapping.lookup(entryBlock.getTerminator()->getOperand(0));
// Prepare the epilog of the loop block.
rewriter.setInsertionPointToEnd(loopBlock);
// Append the cmpxchg op to the end of the loop block.
auto successOrdering = LLVM::AtomicOrdering::acq_rel;
auto failureOrdering = LLVM::AtomicOrdering::monotonic;
auto pairType = LLVM::LLVMType::getStructTy(valueType, boolType);
auto cmpxchg = rewriter.create<LLVM::AtomicCmpXchgOp>(
loc, pairType, dataPtr, loopArgument, result, successOrdering,
failureOrdering);
// Extract the %new_loaded and %ok values from the pair.
Value newLoaded = rewriter.create<LLVM::ExtractValueOp>(
loc, valueType, cmpxchg, rewriter.getI64ArrayAttr({0}));
Value ok = rewriter.create<LLVM::ExtractValueOp>(
loc, boolType, cmpxchg, rewriter.getI64ArrayAttr({1}));
// Conditionally branch to the end or back to the loop depending on %ok.
rewriter.create<LLVM::CondBrOp>(loc, ok, endBlock, ArrayRef<Value>(),
loopBlock, newLoaded);
// The 'result' of the atomic_rmw op is the newly loaded value.
rewriter.replaceOp(op, {newLoaded});
return success();
}
};
} // namespace } // namespace
/// Collect a set of patterns to convert from the Standard dialect to LLVM. /// Collect a set of patterns to convert from the Standard dialect to LLVM.
void mlir::populateStdToLLVMNonMemoryConversionPatterns( void mlir::populateStdToLLVMNonMemoryConversionPatterns(
LLVMTypeConverter &converter, OwningRewritePatternList &patterns) { LLVMTypeConverter &converter, OwningRewritePatternList &patterns) {
// FIXME: this should be tablegen'ed // FIXME: this should be tablegen'ed
// clang-format off // clang-format off
patterns.insert< patterns.insert<
Show All 13 Lines patterns.insert<
CondBranchOpLowering, CondBranchOpLowering,
CopySignOpLowering, CopySignOpLowering,
CosOpLowering, CosOpLowering,
ConstLLVMOpLowering, ConstLLVMOpLowering,
DialectCastOpLowering, DialectCastOpLowering,
DivFOpLowering, DivFOpLowering,
ExpOpLowering, ExpOpLowering,
Exp2OpLowering, Exp2OpLowering,
GenericAtomicRMWOpLowering,
LogOpLowering, LogOpLowering,
Log10OpLowering, Log10OpLowering,
Log2OpLowering, Log2OpLowering,
FPExtLowering, FPExtLowering,
FPTruncLowering, FPTruncLowering,
IndexCastOpLowering, IndexCastOpLowering,
MulFOpLowering, MulFOpLowering,
MulIOpLowering, MulIOpLowering,
▲ Show 20 Lines • Show All 197 Lines • Show Last 20 Lines

mlir/test/Conversion/StandardToLLVM/convert-to-llvmir.mlir

Show First 20 Lines • Show All 1,023 Lines • ▼ Show 20 Linesfunc @cmpxchg(%F : memref<10xf32>, %fval : f32, %i : index) -> f32 {
// CHECK-NEXT: llvm.cond_br %[[ok]], ^bb2, ^bb1(%[[new]] : !llvm.float) // CHECK-NEXT: llvm.cond_br %[[ok]], ^bb2, ^bb1(%[[new]] : !llvm.float)
// CHECK-NEXT: ^bb2: // CHECK-NEXT: ^bb2:
return %x : f32 return %x : f32
// CHECK-NEXT: llvm.return %[[new]] // CHECK-NEXT: llvm.return %[[new]]
} }
// ----- // -----
// CHECK-LABEL: func @generic_atomic_rmw
// CHECK32-LABEL: func @generic_atomic_rmw
func @generic_atomic_rmw(%I : memref<10xf32>, %i : index) -> f32 {
%x = generic_atomic_rmw %I[%i] : memref<10xf32> {
^bb0(%old_value : f32):
%c1 = constant 1.0 : f32
atomic_yield %c1 : f32
}
// CHECK: [[init:%.*]] = llvm.load %{{.*}} : !llvm<"float*">
// CHECK-NEXT: llvm.br ^bb1([[init]] : !llvm.float)
// CHECK-NEXT: ^bb1([[loaded:%.*]]: !llvm.float):
// CHECK-NEXT: [[c1:%.*]] = llvm.mlir.constant(1.000000e+00 : f32)
// CHECK-NEXT: [[pair:%.*]] = llvm.cmpxchg %{{.*}}, [[loaded]], [[c1]]
// CHECK-SAME: acq_rel monotonic : !llvm.float
// CHECK-NEXT: [[new:%.*]] = llvm.extractvalue [[pair]][0]
// CHECK-NEXT: [[ok:%.*]] = llvm.extractvalue [[pair]][1]
// CHECK-NEXT: llvm.cond_br [[ok]], ^bb2, ^bb1([[new]] : !llvm.float)
// CHECK-NEXT: ^bb2:
return %x : f32
// CHECK-NEXT: llvm.return [[new]]
}
// -----
// CHECK-LABEL: func @assume_alignment // CHECK-LABEL: func @assume_alignment
func @assume_alignment(%0 : memref<4x4xf16>) { func @assume_alignment(%0 : memref<4x4xf16>) {
// CHECK: %[[PTR:.*]] = llvm.extractvalue %[[MEMREF:.*]][1] : !llvm<"{ half*, half*, i64, [2 x i64], [2 x i64] }"> // CHECK: %[[PTR:.*]] = llvm.extractvalue %[[MEMREF:.*]][1] : !llvm<"{ half*, half*, i64, [2 x i64], [2 x i64] }">
// CHECK-NEXT: %[[ZERO:.*]] = llvm.mlir.constant(0 : index) : !llvm.i64 // CHECK-NEXT: %[[ZERO:.*]] = llvm.mlir.constant(0 : index) : !llvm.i64
// CHECK-NEXT: %[[MASK:.*]] = llvm.mlir.constant(15 : index) : !llvm.i64 // CHECK-NEXT: %[[MASK:.*]] = llvm.mlir.constant(15 : index) : !llvm.i64
// CHECK-NEXT: %[[INT:.*]] = llvm.ptrtoint %[[PTR]] : !llvm<"half*"> to !llvm.i64 // CHECK-NEXT: %[[INT:.*]] = llvm.ptrtoint %[[PTR]] : !llvm<"half*"> to !llvm.i64
// CHECK-NEXT: %[[MASKED_PTR:.*]] = llvm.and %[[INT]], %[[MASK:.*]] : !llvm.i64 // CHECK-NEXT: %[[MASKED_PTR:.*]] = llvm.and %[[INT]], %[[MASK:.*]] : !llvm.i64
// CHECK-NEXT: %[[CONDITION:.*]] = llvm.icmp "eq" %[[MASKED_PTR]], %[[ZERO]] : !llvm.i64 // CHECK-NEXT: %[[CONDITION:.*]] = llvm.icmp "eq" %[[MASKED_PTR]], %[[ZERO]] : !llvm.i64
Show All 40 Lines