Index: llvm/trunk/docs/Atomics.rst =================================================================== --- llvm/trunk/docs/Atomics.rst +++ llvm/trunk/docs/Atomics.rst @@ -413,19 +413,28 @@ this is not correct in the IR sense of volatile, but CodeGen handles anything marked volatile very conservatively. This should get fixed at some point. -Common architectures have some way of representing at least a pointer-sized -lock-free ``cmpxchg``; such an operation can be used to implement all the other -atomic operations which can be represented in IR up to that size. Backends are -expected to implement all those operations, but not operations which cannot be -implemented in a lock-free manner. It is expected that backends will give an -error when given an operation which cannot be implemented. (The LLVM code -generator is not very helpful here at the moment, but hopefully that will -change.) +One very important property of the atomic operations is that if your backend +supports any inline lock-free atomic operations of a given size, you should +support *ALL* operations of that size in a lock-free manner. + +When the target implements atomic ``cmpxchg`` or LL/SC instructions (as most do) +this is trivial: all the other operations can be implemented on top of those +primitives. However, on many older CPUs (e.g. ARMv5, SparcV8, Intel 80386) there +are atomic load and store instructions, but no ``cmpxchg`` or LL/SC. As it is +invalid to implement ``atomic load`` using the native instruction, but +``cmpxchg`` using a library call to a function that uses a mutex, ``atomic +load`` must *also* expand to a library call on such architectures, so that it +can remain atomic with regards to a simultaneous ``cmpxchg``, by using the same +mutex. + +AtomicExpandPass can help with that: it will expand all atomic operations to the +proper ``__atomic_*`` libcalls for any size above the maximum set by +``setMaxAtomicSizeInBitsSupported`` (which defaults to 0). On x86, all atomic loads generate a ``MOV``. SequentiallyConsistent stores generate an ``XCHG``, other stores generate a ``MOV``. SequentiallyConsistent fences generate an ``MFENCE``, other fences do not cause any code to be -generated. cmpxchg uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg`` +generated. ``cmpxchg`` uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg`` uses ``XCHG``, ``atomicrmw add`` and ``atomicrmw sub`` use ``XADD``, and all other ``atomicrmw`` operations generate a loop with ``LOCK CMPXCHG``. Depending on the users of the result, some ``atomicrmw`` operations can be translated into @@ -446,10 +455,151 @@ ``emitStoreConditional()`` * large loads/stores -> ll-sc/cmpxchg by overriding ``shouldExpandAtomicStoreInIR()``/``shouldExpandAtomicLoadInIR()`` -* strong atomic accesses -> monotonic accesses + fences - by using ``setInsertFencesForAtomic()`` and overriding ``emitLeadingFence()`` - and ``emitTrailingFence()`` +* strong atomic accesses -> monotonic accesses + fences by overriding + ``shouldInsertFencesForAtomic()``, ``emitLeadingFence()``, and + ``emitTrailingFence()`` * atomic rmw -> loop with cmpxchg or load-linked/store-conditional by overriding ``expandAtomicRMWInIR()`` +* expansion to __atomic_* libcalls for unsupported sizes. For an example of all of these, look at the ARM backend. + +Libcalls: __atomic_* +==================== + +There are two kinds of atomic library calls that are generated by LLVM. Please +note that both sets of library functions somewhat confusingly share the names of +builtin functions defined by clang. Despite this, the library functions are +not directly related to the builtins: it is *not* the case that ``__atomic_*`` +builtins lower to ``__atomic_*`` library calls and ``__sync_*`` builtins lower +to ``__sync_*`` library calls. + +The first set of library functions are named ``__atomic_*``. This set has been +"standardized" by GCC, and is described below. (See also `GCC's documentation +`_) + +LLVM's AtomicExpandPass will translate atomic operations on data sizes above +``MaxAtomicSizeInBitsSupported`` into calls to these functions. + +There are four generic functions, which can be called with data of any size or +alignment:: + + void __atomic_load(size_t size, void *ptr, void *ret, int ordering) + void __atomic_store(size_t size, void *ptr, void *val, int ordering) + void __atomic_exchange(size_t size, void *ptr, void *val, void *ret, int ordering) + bool __atomic_compare_exchange(size_t size, void *ptr, void *expected, void *desired, int success_order, int failure_order) + +There are also size-specialized versions of the above functions, which can only +be used with *naturally-aligned* pointers of the appropriate size. In the +signatures below, "N" is one of 1, 2, 4, 8, and 16, and "iN" is the appropriate +integer type of that size; if no such integer type exists, the specialization +cannot be used:: + + iN __atomic_load_N(iN *ptr, iN val, int ordering) + void __atomic_store_N(iN *ptr, iN val, int ordering) + iN __atomic_exchange_N(iN *ptr, iN val, int ordering) + bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired, int success_order, int failure_order) + +Finally there are some read-modify-write functions, which are only available in +the size-specific variants (any other sizes use a ``__atomic_compare_exchange`` +loop):: + + iN __atomic_fetch_add_N(iN *ptr, iN val, int ordering) + iN __atomic_fetch_sub_N(iN *ptr, iN val, int ordering) + iN __atomic_fetch_and_N(iN *ptr, iN val, int ordering) + iN __atomic_fetch_or_N(iN *ptr, iN val, int ordering) + iN __atomic_fetch_xor_N(iN *ptr, iN val, int ordering) + iN __atomic_fetch_nand_N(iN *ptr, iN val, int ordering) + +This set of library functions have some interesting implementation requirements +to take note of: + +- They support all sizes and alignments -- including those which cannot be + implemented natively on any existing hardware. Therefore, they will certainly + use mutexes in for some sizes/alignments. + +- As a consequence, they cannot be shipped in a statically linked + compiler-support library, as they have state which must be shared amongst all + DSOs loaded in the program. They must be provided in a shared library used by + all objects. + +- The set of atomic sizes supported lock-free must be a superset of the sizes + any compiler can emit. That is: if a new compiler introduces support for + inline-lock-free atomics of size N, the ``__atomic_*`` functions must also have a + lock-free implementation for size N. This is a requirement so that code + produced by an old compiler (which will have called the ``__atomic_*`` function) + interoperates with code produced by the new compiler (which will use native + the atomic instruction). + +Note that it's possible to write an entirely target-independent implementation +of these library functions by using the compiler atomic builtins themselves to +implement the operations on naturally-aligned pointers of supported sizes, and a +generic mutex implementation otherwise. + +Libcalls: __sync_* +================== + +Some targets or OS/target combinations can support lock-free atomics, but for +various reasons, it is not practical to emit the instructions inline. + +There's two typical examples of this. + +Some CPUs support multiple instruction sets which can be swiched back and forth +on function-call boundaries. For example, MIPS supports the MIPS16 ISA, which +has a smaller instruction encoding than the usual MIPS32 ISA. ARM, similarly, +has the Thumb ISA. In MIPS16 and earlier versions of Thumb, the atomic +instructions are not encodable. However, those instructions are available via a +function call to a function with the longer encoding. + +Additionally, a few OS/target pairs provide kernel-supported lock-free +atomics. ARM/Linux is an example of this: the kernel `provides +`_ a +function which on older CPUs contains a "magically-restartable" atomic sequence +(which looks atomic so long as there's only one CPU), and contains actual atomic +instructions on newer multicore models. This sort of functionality can typically +be provided on any architecture, if all CPUs which are missing atomic +compare-and-swap support are uniprocessor (no SMP). This is almost always the +case. The only common architecture without that property is SPARC -- SPARCV8 SMP +systems were common, yet it doesn't support any sort of compare-and-swap +operation. + +In either of these cases, the Target in LLVM can claim support for atomics of an +appropriate size, and then implement some subset of the operations via libcalls +to a ``__sync_*`` function. Such functions *must* not use locks in their +implementation, because unlike the ``__atomic_*`` routines used by +AtomicExpandPass, these may be mixed-and-matched with native instructions by the +target lowering. + +Further, these routines do not need to be shared, as they are stateless. So, +there is no issue with having multiple copies included in one binary. Thus, +typically these routines are implemented by the statically-linked compiler +runtime support library. + +LLVM will emit a call to an appropriate ``__sync_*`` routine if the target +ISelLowering code has set the corresponding ``ATOMIC_CMPXCHG``, ``ATOMIC_SWAP``, +or ``ATOMIC_LOAD_*`` operation to "Expand", and if it has opted-into the +availablity of those library functions via a call to ``initSyncLibcalls()``. + +The full set of functions that may be called by LLVM is (for ``N`` being 1, 2, +4, 8, or 16):: + + iN __sync_val_compare_and_swap_N(iN *ptr, iN expected, iN desired) + iN __sync_lock_test_and_set_N(iN *ptr, iN val) + iN __sync_fetch_and_add_N(iN *ptr, iN val) + iN __sync_fetch_and_sub_N(iN *ptr, iN val) + iN __sync_fetch_and_and_N(iN *ptr, iN val) + iN __sync_fetch_and_or_N(iN *ptr, iN val) + iN __sync_fetch_and_xor_N(iN *ptr, iN val) + iN __sync_fetch_and_nand_N(iN *ptr, iN val) + iN __sync_fetch_and_max_N(iN *ptr, iN val) + iN __sync_fetch_and_umax_N(iN *ptr, iN val) + iN __sync_fetch_and_min_N(iN *ptr, iN val) + iN __sync_fetch_and_umin_N(iN *ptr, iN val) + +This list doesn't include any function for atomic load or store; all known +architectures support atomic loads and stores directly (possibly by emitting a +fence on either side of a normal load or store.) + +There's also, somewhat separately, the possibility to lower ``ATOMIC_FENCE`` to +``__sync_synchronize()``. This may happen or not happen independent of all the +above, controlled purely by ``setOperationAction(ISD::ATOMIC_FENCE, ...)``. Index: llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h =================================================================== --- llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h +++ llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h @@ -336,7 +336,11 @@ // EXCEPTION HANDLING UNWIND_RESUME, - // Family ATOMICs + // Note: there's two sets of atomics libcalls; see + // for more info on the + // difference between them. + + // Atomic '__sync_*' libcalls. SYNC_VAL_COMPARE_AND_SWAP_1, SYNC_VAL_COMPARE_AND_SWAP_2, SYNC_VAL_COMPARE_AND_SWAP_4, @@ -398,6 +402,73 @@ SYNC_FETCH_AND_UMIN_8, SYNC_FETCH_AND_UMIN_16, + // Atomic '__atomic_*' libcalls. + ATOMIC_LOAD, + ATOMIC_LOAD_1, + ATOMIC_LOAD_2, + ATOMIC_LOAD_4, + ATOMIC_LOAD_8, + ATOMIC_LOAD_16, + + ATOMIC_STORE, + ATOMIC_STORE_1, + ATOMIC_STORE_2, + ATOMIC_STORE_4, + ATOMIC_STORE_8, + ATOMIC_STORE_16, + + ATOMIC_EXCHANGE, + ATOMIC_EXCHANGE_1, + ATOMIC_EXCHANGE_2, + ATOMIC_EXCHANGE_4, + ATOMIC_EXCHANGE_8, + ATOMIC_EXCHANGE_16, + + ATOMIC_COMPARE_EXCHANGE, + ATOMIC_COMPARE_EXCHANGE_1, + ATOMIC_COMPARE_EXCHANGE_2, + ATOMIC_COMPARE_EXCHANGE_4, + ATOMIC_COMPARE_EXCHANGE_8, + ATOMIC_COMPARE_EXCHANGE_16, + + ATOMIC_FETCH_ADD_1, + ATOMIC_FETCH_ADD_2, + ATOMIC_FETCH_ADD_4, + ATOMIC_FETCH_ADD_8, + ATOMIC_FETCH_ADD_16, + + ATOMIC_FETCH_SUB_1, + ATOMIC_FETCH_SUB_2, + ATOMIC_FETCH_SUB_4, + ATOMIC_FETCH_SUB_8, + ATOMIC_FETCH_SUB_16, + + ATOMIC_FETCH_AND_1, + ATOMIC_FETCH_AND_2, + ATOMIC_FETCH_AND_4, + ATOMIC_FETCH_AND_8, + ATOMIC_FETCH_AND_16, + + ATOMIC_FETCH_OR_1, + ATOMIC_FETCH_OR_2, + ATOMIC_FETCH_OR_4, + ATOMIC_FETCH_OR_8, + ATOMIC_FETCH_OR_16, + + ATOMIC_FETCH_XOR_1, + ATOMIC_FETCH_XOR_2, + ATOMIC_FETCH_XOR_4, + ATOMIC_FETCH_XOR_8, + ATOMIC_FETCH_XOR_16, + + ATOMIC_FETCH_NAND_1, + ATOMIC_FETCH_NAND_2, + ATOMIC_FETCH_NAND_4, + ATOMIC_FETCH_NAND_8, + ATOMIC_FETCH_NAND_16, + + ATOMIC_IS_LOCK_FREE, + // Stack Protector Fail. STACKPROTECTOR_CHECK_FAIL, Index: llvm/trunk/include/llvm/Target/TargetLowering.h =================================================================== --- llvm/trunk/include/llvm/Target/TargetLowering.h +++ llvm/trunk/include/llvm/Target/TargetLowering.h @@ -1059,6 +1059,14 @@ /// \name Helpers for atomic expansion. /// @{ + /// Returns the maximum atomic operation size (in bits) supported by + /// the backend. Atomic operations greater than this size (as well + /// as ones that are not naturally aligned), will be expanded by + /// AtomicExpandPass into an __atomic_* library call. + unsigned getMaxAtomicSizeInBitsSupported() const { + return MaxAtomicSizeInBitsSupported; + } + /// Whether AtomicExpandPass should automatically insert fences and reduce /// ordering for this atomic. This should be true for most architectures with /// weak memory ordering. Defaults to false. @@ -1454,6 +1462,14 @@ MinStackArgumentAlignment = Align; } + /// Set the maximum atomic operation size supported by the + /// backend. Atomic operations greater than this size (as well as + /// ones that are not naturally aligned), will be expanded by + /// AtomicExpandPass into an __atomic_* library call. + void setMaxAtomicSizeInBitsSupported(unsigned SizeInBits) { + MaxAtomicSizeInBitsSupported = SizeInBits; + } + public: //===--------------------------------------------------------------------===// // Addressing mode description hooks (used by LSR etc). @@ -1863,6 +1879,9 @@ /// The preferred loop alignment. unsigned PrefLoopAlignment; + /// Size in bits of the maximum atomics size the backend supports. + /// Accesses larger than this will be expanded by AtomicExpandPass. + unsigned MaxAtomicSizeInBitsSupported; /// If set to a physical register, this specifies the register that /// llvm.savestack/llvm.restorestack should save and restore. Index: llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp =================================================================== --- llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp +++ llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp @@ -8,10 +8,10 @@ //===----------------------------------------------------------------------===// // // This file contains a pass (at IR level) to replace atomic instructions with -// target specific instruction which implement the same semantics in a way -// which better fits the target backend. This can include the use of either -// (intrinsic-based) load-linked/store-conditional loops, AtomicCmpXchg, or -// type coercions. +// __atomic_* library calls, or target specific instruction which implement the +// same semantics in a way which better fits the target backend. This can +// include the use of (intrinsic-based) load-linked/store-conditional loops, +// AtomicCmpXchg, or type coercions. // //===----------------------------------------------------------------------===// @@ -64,19 +64,95 @@ bool expandAtomicCmpXchg(AtomicCmpXchgInst *CI); bool isIdempotentRMW(AtomicRMWInst *AI); bool simplifyIdempotentRMW(AtomicRMWInst *AI); + + bool expandAtomicOpToLibcall(Instruction *I, unsigned Size, unsigned Align, + Value *PointerOperand, Value *ValueOperand, + Value *CASExpected, AtomicOrdering Ordering, + AtomicOrdering Ordering2, + ArrayRef Libcalls); + void expandAtomicLoadToLibcall(LoadInst *LI); + void expandAtomicStoreToLibcall(StoreInst *LI); + void expandAtomicRMWToLibcall(AtomicRMWInst *I); + void expandAtomicCASToLibcall(AtomicCmpXchgInst *I); }; } char AtomicExpand::ID = 0; char &llvm::AtomicExpandID = AtomicExpand::ID; -INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand", - "Expand Atomic calls in terms of either load-linked & store-conditional or cmpxchg", - false, false) +INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand", "Expand Atomic instructions", + false, false) FunctionPass *llvm::createAtomicExpandPass(const TargetMachine *TM) { return new AtomicExpand(TM); } +namespace { +// Helper functions to retrieve the size of atomic instructions. +unsigned getAtomicOpSize(LoadInst *LI) { + const DataLayout &DL = LI->getModule()->getDataLayout(); + return DL.getTypeStoreSize(LI->getType()); +} + +unsigned getAtomicOpSize(StoreInst *SI) { + const DataLayout &DL = SI->getModule()->getDataLayout(); + return DL.getTypeStoreSize(SI->getValueOperand()->getType()); +} + +unsigned getAtomicOpSize(AtomicRMWInst *RMWI) { + const DataLayout &DL = RMWI->getModule()->getDataLayout(); + return DL.getTypeStoreSize(RMWI->getValOperand()->getType()); +} + +unsigned getAtomicOpSize(AtomicCmpXchgInst *CASI) { + const DataLayout &DL = CASI->getModule()->getDataLayout(); + return DL.getTypeStoreSize(CASI->getCompareOperand()->getType()); +} + +// Helper functions to retrieve the alignment of atomic instructions. +unsigned getAtomicOpAlign(LoadInst *LI) { + unsigned Align = LI->getAlignment(); + // In the future, if this IR restriction is relaxed, we should + // return DataLayout::getABITypeAlignment when there's no align + // value. + assert(Align != 0 && "An atomic LoadInst always has an explicit alignment"); + return Align; +} + +unsigned getAtomicOpAlign(StoreInst *SI) { + unsigned Align = SI->getAlignment(); + // In the future, if this IR restriction is relaxed, we should + // return DataLayout::getABITypeAlignment when there's no align + // value. + assert(Align != 0 && "An atomic StoreInst always has an explicit alignment"); + return Align; +} + +unsigned getAtomicOpAlign(AtomicRMWInst *RMWI) { + // TODO(PR27168): This instruction has no alignment attribute, but unlike the + // default alignment for load/store, the default here is to assume + // it has NATURAL alignment, not DataLayout-specified alignment. + const DataLayout &DL = RMWI->getModule()->getDataLayout(); + return DL.getTypeStoreSize(RMWI->getValOperand()->getType()); +} + +unsigned getAtomicOpAlign(AtomicCmpXchgInst *CASI) { + // TODO(PR27168): same comment as above. + const DataLayout &DL = CASI->getModule()->getDataLayout(); + return DL.getTypeStoreSize(CASI->getCompareOperand()->getType()); +} + +// Determine if a particular atomic operation has a supported size, +// and is of appropriate alignment, to be passed through for target +// lowering. (Versus turning into a __atomic libcall) +template +bool atomicSizeSupported(const TargetLowering *TLI, Inst *I) { + unsigned Size = getAtomicOpSize(I); + unsigned Align = getAtomicOpAlign(I); + return Align >= Size && Size <= TLI->getMaxAtomicSizeInBitsSupported() / 8; +} + +} // end anonymous namespace + bool AtomicExpand::runOnFunction(Function &F) { if (!TM || !TM->getSubtargetImpl(F)->enableAtomicExpand()) return false; @@ -100,6 +176,33 @@ auto CASI = dyn_cast(I); assert((LI || SI || RMWI || CASI) && "Unknown atomic instruction"); + // If the Size/Alignment is not supported, replace with a libcall. + if (LI) { + if (!atomicSizeSupported(TLI, LI)) { + expandAtomicLoadToLibcall(LI); + MadeChange = true; + continue; + } + } else if (SI) { + if (!atomicSizeSupported(TLI, SI)) { + expandAtomicStoreToLibcall(SI); + MadeChange = true; + continue; + } + } else if (RMWI) { + if (!atomicSizeSupported(TLI, RMWI)) { + expandAtomicRMWToLibcall(RMWI); + MadeChange = true; + continue; + } + } else if (CASI) { + if (!atomicSizeSupported(TLI, CASI)) { + expandAtomicCASToLibcall(CASI); + MadeChange = true; + continue; + } + } + if (TLI->shouldInsertFencesForAtomic(I)) { auto FenceOrdering = AtomicOrdering::Monotonic; bool IsStore, IsLoad; @@ -144,7 +247,7 @@ assert(LI->getType()->isIntegerTy() && "invariant broken"); MadeChange = true; } - + MadeChange |= tryExpandAtomicLoad(LI); } else if (SI) { if (SI->getValueOperand()->getType()->isFloatingPointTy()) { @@ -833,3 +936,381 @@ return true; } + +// This converts from LLVM's internal AtomicOrdering enum to the +// memory_order_* value required by the __atomic_* libcalls. +static int libcallAtomicModel(AtomicOrdering AO) { + enum { + AO_ABI_memory_order_relaxed = 0, + AO_ABI_memory_order_consume = 1, + AO_ABI_memory_order_acquire = 2, + AO_ABI_memory_order_release = 3, + AO_ABI_memory_order_acq_rel = 4, + AO_ABI_memory_order_seq_cst = 5 + }; + + switch (AO) { + case AtomicOrdering::NotAtomic: + llvm_unreachable("Expected atomic memory order."); + case AtomicOrdering::Unordered: + case AtomicOrdering::Monotonic: + return AO_ABI_memory_order_relaxed; + // Not implemented yet in llvm: + // case AtomicOrdering::Consume: + // return AO_ABI_memory_order_consume; + case AtomicOrdering::Acquire: + return AO_ABI_memory_order_acquire; + case AtomicOrdering::Release: + return AO_ABI_memory_order_release; + case AtomicOrdering::AcquireRelease: + return AO_ABI_memory_order_acq_rel; + case AtomicOrdering::SequentiallyConsistent: + return AO_ABI_memory_order_seq_cst; + } + llvm_unreachable("Unknown atomic memory order."); +} + +// In order to use one of the sized library calls such as +// __atomic_fetch_add_4, the alignment must be sufficient, the size +// must be one of the potentially-specialized sizes, and the value +// type must actually exist in C on the target (otherwise, the +// function wouldn't actually be defined.) +static bool canUseSizedAtomicCall(unsigned Size, unsigned Align, + const DataLayout &DL) { + // TODO: "LargestSize" is an approximation for "largest type that + // you can express in C". It seems to be the case that int128 is + // supported on all 64-bit platforms, otherwise only up to 64-bit + // integers are supported. If we get this wrong, then we'll try to + // call a sized libcall that doesn't actually exist. There should + // really be some more reliable way in LLVM of determining integer + // sizes which are valid in the target's C ABI... + unsigned LargestSize = DL.getLargestLegalIntTypeSize() >= 64 ? 16 : 8; + return Align >= Size && + (Size == 1 || Size == 2 || Size == 4 || Size == 8 || Size == 16) && + Size <= LargestSize; +} + +void AtomicExpand::expandAtomicLoadToLibcall(LoadInst *I) { + static const RTLIB::Libcall Libcalls[6] = { + RTLIB::ATOMIC_LOAD, RTLIB::ATOMIC_LOAD_1, RTLIB::ATOMIC_LOAD_2, + RTLIB::ATOMIC_LOAD_4, RTLIB::ATOMIC_LOAD_8, RTLIB::ATOMIC_LOAD_16}; + unsigned Size = getAtomicOpSize(I); + unsigned Align = getAtomicOpAlign(I); + + bool expanded = expandAtomicOpToLibcall( + I, Size, Align, I->getPointerOperand(), nullptr, nullptr, + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls); + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Load"); +} + +void AtomicExpand::expandAtomicStoreToLibcall(StoreInst *I) { + static const RTLIB::Libcall Libcalls[6] = { + RTLIB::ATOMIC_STORE, RTLIB::ATOMIC_STORE_1, RTLIB::ATOMIC_STORE_2, + RTLIB::ATOMIC_STORE_4, RTLIB::ATOMIC_STORE_8, RTLIB::ATOMIC_STORE_16}; + unsigned Size = getAtomicOpSize(I); + unsigned Align = getAtomicOpAlign(I); + + bool expanded = expandAtomicOpToLibcall( + I, Size, Align, I->getPointerOperand(), I->getValueOperand(), nullptr, + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls); + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Store"); +} + +void AtomicExpand::expandAtomicCASToLibcall(AtomicCmpXchgInst *I) { + static const RTLIB::Libcall Libcalls[6] = { + RTLIB::ATOMIC_COMPARE_EXCHANGE, RTLIB::ATOMIC_COMPARE_EXCHANGE_1, + RTLIB::ATOMIC_COMPARE_EXCHANGE_2, RTLIB::ATOMIC_COMPARE_EXCHANGE_4, + RTLIB::ATOMIC_COMPARE_EXCHANGE_8, RTLIB::ATOMIC_COMPARE_EXCHANGE_16}; + unsigned Size = getAtomicOpSize(I); + unsigned Align = getAtomicOpAlign(I); + + bool expanded = expandAtomicOpToLibcall( + I, Size, Align, I->getPointerOperand(), I->getNewValOperand(), + I->getCompareOperand(), I->getSuccessOrdering(), I->getFailureOrdering(), + Libcalls); + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor CAS"); +} + +static ArrayRef GetRMWLibcall(AtomicRMWInst::BinOp Op) { + static const RTLIB::Libcall LibcallsXchg[6] = { + RTLIB::ATOMIC_EXCHANGE, RTLIB::ATOMIC_EXCHANGE_1, + RTLIB::ATOMIC_EXCHANGE_2, RTLIB::ATOMIC_EXCHANGE_4, + RTLIB::ATOMIC_EXCHANGE_8, RTLIB::ATOMIC_EXCHANGE_16}; + static const RTLIB::Libcall LibcallsAdd[6] = { + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_ADD_1, + RTLIB::ATOMIC_FETCH_ADD_2, RTLIB::ATOMIC_FETCH_ADD_4, + RTLIB::ATOMIC_FETCH_ADD_8, RTLIB::ATOMIC_FETCH_ADD_16}; + static const RTLIB::Libcall LibcallsSub[6] = { + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_SUB_1, + RTLIB::ATOMIC_FETCH_SUB_2, RTLIB::ATOMIC_FETCH_SUB_4, + RTLIB::ATOMIC_FETCH_SUB_8, RTLIB::ATOMIC_FETCH_SUB_16}; + static const RTLIB::Libcall LibcallsAnd[6] = { + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_AND_1, + RTLIB::ATOMIC_FETCH_AND_2, RTLIB::ATOMIC_FETCH_AND_4, + RTLIB::ATOMIC_FETCH_AND_8, RTLIB::ATOMIC_FETCH_AND_16}; + static const RTLIB::Libcall LibcallsOr[6] = { + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_OR_1, + RTLIB::ATOMIC_FETCH_OR_2, RTLIB::ATOMIC_FETCH_OR_4, + RTLIB::ATOMIC_FETCH_OR_8, RTLIB::ATOMIC_FETCH_OR_16}; + static const RTLIB::Libcall LibcallsXor[6] = { + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_XOR_1, + RTLIB::ATOMIC_FETCH_XOR_2, RTLIB::ATOMIC_FETCH_XOR_4, + RTLIB::ATOMIC_FETCH_XOR_8, RTLIB::ATOMIC_FETCH_XOR_16}; + static const RTLIB::Libcall LibcallsNand[6] = { + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_NAND_1, + RTLIB::ATOMIC_FETCH_NAND_2, RTLIB::ATOMIC_FETCH_NAND_4, + RTLIB::ATOMIC_FETCH_NAND_8, RTLIB::ATOMIC_FETCH_NAND_16}; + + switch (Op) { + case AtomicRMWInst::BAD_BINOP: + llvm_unreachable("Should not have BAD_BINOP."); + case AtomicRMWInst::Xchg: + return LibcallsXchg; + case AtomicRMWInst::Add: + return LibcallsAdd; + case AtomicRMWInst::Sub: + return LibcallsSub; + case AtomicRMWInst::And: + return LibcallsAnd; + case AtomicRMWInst::Or: + return LibcallsOr; + case AtomicRMWInst::Xor: + return LibcallsXor; + case AtomicRMWInst::Nand: + return LibcallsNand; + case AtomicRMWInst::Max: + case AtomicRMWInst::Min: + case AtomicRMWInst::UMax: + case AtomicRMWInst::UMin: + // No atomic libcalls are available for max/min/umax/umin. + return {}; + } + llvm_unreachable("Unexpected AtomicRMW operation."); +} + +void AtomicExpand::expandAtomicRMWToLibcall(AtomicRMWInst *I) { + ArrayRef Libcalls = GetRMWLibcall(I->getOperation()); + + unsigned Size = getAtomicOpSize(I); + unsigned Align = getAtomicOpAlign(I); + + bool Success = false; + if (!Libcalls.empty()) + Success = expandAtomicOpToLibcall( + I, Size, Align, I->getPointerOperand(), I->getValOperand(), nullptr, + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls); + + // The expansion failed: either there were no libcalls at all for + // the operation (min/max), or there were only size-specialized + // libcalls (add/sub/etc) and we needed a generic. So, expand to a + // CAS libcall, via a CAS loop, instead. + if (!Success) { + expandAtomicRMWToCmpXchg(I, [this](IRBuilder<> &Builder, Value *Addr, + Value *Loaded, Value *NewVal, + AtomicOrdering MemOpOrder, + Value *&Success, Value *&NewLoaded) { + // Create the CAS instruction normally... + AtomicCmpXchgInst *Pair = Builder.CreateAtomicCmpXchg( + Addr, Loaded, NewVal, MemOpOrder, + AtomicCmpXchgInst::getStrongestFailureOrdering(MemOpOrder)); + Success = Builder.CreateExtractValue(Pair, 1, "success"); + NewLoaded = Builder.CreateExtractValue(Pair, 0, "newloaded"); + + // ...and then expand the CAS into a libcall. + expandAtomicCASToLibcall(Pair); + }); + } +} + +// A helper routine for the above expandAtomic*ToLibcall functions. +// +// 'Libcalls' contains an array of enum values for the particular +// ATOMIC libcalls to be emitted. All of the other arguments besides +// 'I' are extracted from the Instruction subclass by the +// caller. Depending on the particular call, some will be null. +bool AtomicExpand::expandAtomicOpToLibcall( + Instruction *I, unsigned Size, unsigned Align, Value *PointerOperand, + Value *ValueOperand, Value *CASExpected, AtomicOrdering Ordering, + AtomicOrdering Ordering2, ArrayRef Libcalls) { + assert(Libcalls.size() == 6); + + LLVMContext &Ctx = I->getContext(); + Module *M = I->getModule(); + const DataLayout &DL = M->getDataLayout(); + IRBuilder<> Builder(I); + IRBuilder<> AllocaBuilder(&I->getFunction()->getEntryBlock().front()); + + bool UseSizedLibcall = canUseSizedAtomicCall(Size, Align, DL); + Type *SizedIntTy = Type::getIntNTy(Ctx, Size * 8); + + unsigned AllocaAlignment = DL.getPrefTypeAlignment(SizedIntTy); + + // TODO: the "order" argument type is "int", not int32. So + // getInt32Ty may be wrong if the arch uses e.g. 16-bit ints. + ConstantInt *SizeVal64 = ConstantInt::get(Type::getInt64Ty(Ctx), Size); + Constant *OrderingVal = + ConstantInt::get(Type::getInt32Ty(Ctx), libcallAtomicModel(Ordering)); + Constant *Ordering2Val = CASExpected + ? ConstantInt::get(Type::getInt32Ty(Ctx), + libcallAtomicModel(Ordering2)) + : nullptr; + bool HasResult = I->getType() != Type::getVoidTy(Ctx); + + RTLIB::Libcall RTLibType; + if (UseSizedLibcall) { + switch (Size) { + case 1: RTLibType = Libcalls[1]; break; + case 2: RTLibType = Libcalls[2]; break; + case 4: RTLibType = Libcalls[3]; break; + case 8: RTLibType = Libcalls[4]; break; + case 16: RTLibType = Libcalls[5]; break; + } + } else if (Libcalls[0] != RTLIB::UNKNOWN_LIBCALL) { + RTLibType = Libcalls[0]; + } else { + // Can't use sized function, and there's no generic for this + // operation, so give up. + return false; + } + + // Build up the function call. There's two kinds. First, the sized + // variants. These calls are going to be one of the following (with + // N=1,2,4,8,16): + // iN __atomic_load_N(iN *ptr, int ordering) + // void __atomic_store_N(iN *ptr, iN val, int ordering) + // iN __atomic_{exchange|fetch_*}_N(iN *ptr, iN val, int ordering) + // bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired, + // int success_order, int failure_order) + // + // Note that these functions can be used for non-integer atomic + // operations, the values just need to be bitcast to integers on the + // way in and out. + // + // And, then, the generic variants. They look like the following: + // void __atomic_load(size_t size, void *ptr, void *ret, int ordering) + // void __atomic_store(size_t size, void *ptr, void *val, int ordering) + // void __atomic_exchange(size_t size, void *ptr, void *val, void *ret, + // int ordering) + // bool __atomic_compare_exchange(size_t size, void *ptr, void *expected, + // void *desired, int success_order, + // int failure_order) + // + // The different signatures are built up depending on the + // 'UseSizedLibcall', 'CASExpected', 'ValueOperand', and 'HasResult' + // variables. + + AllocaInst *AllocaCASExpected = nullptr; + Value *AllocaCASExpected_i8 = nullptr; + AllocaInst *AllocaValue = nullptr; + Value *AllocaValue_i8 = nullptr; + AllocaInst *AllocaResult = nullptr; + Value *AllocaResult_i8 = nullptr; + + Type *ResultTy; + SmallVector Args; + AttributeSet Attr; + + // 'size' argument. + if (!UseSizedLibcall) { + // Note, getIntPtrType is assumed equivalent to size_t. + Args.push_back(ConstantInt::get(DL.getIntPtrType(Ctx), Size)); + } + + // 'ptr' argument. + Value *PtrVal = + Builder.CreateBitCast(PointerOperand, Type::getInt8PtrTy(Ctx)); + Args.push_back(PtrVal); + + // 'expected' argument, if present. + if (CASExpected) { + AllocaCASExpected = AllocaBuilder.CreateAlloca(CASExpected->getType()); + AllocaCASExpected->setAlignment(AllocaAlignment); + AllocaCASExpected_i8 = + Builder.CreateBitCast(AllocaCASExpected, Type::getInt8PtrTy(Ctx)); + Builder.CreateLifetimeStart(AllocaCASExpected_i8, SizeVal64); + Builder.CreateAlignedStore(CASExpected, AllocaCASExpected, AllocaAlignment); + Args.push_back(AllocaCASExpected_i8); + } + + // 'val' argument ('desired' for cas), if present. + if (ValueOperand) { + if (UseSizedLibcall) { + Value *IntValue = + Builder.CreateBitOrPointerCast(ValueOperand, SizedIntTy); + Args.push_back(IntValue); + } else { + AllocaValue = AllocaBuilder.CreateAlloca(ValueOperand->getType()); + AllocaValue->setAlignment(AllocaAlignment); + AllocaValue_i8 = + Builder.CreateBitCast(AllocaValue, Type::getInt8PtrTy(Ctx)); + Builder.CreateLifetimeStart(AllocaValue_i8, SizeVal64); + Builder.CreateAlignedStore(ValueOperand, AllocaValue, AllocaAlignment); + Args.push_back(AllocaValue_i8); + } + } + + // 'ret' argument. + if (!CASExpected && HasResult && !UseSizedLibcall) { + AllocaResult = AllocaBuilder.CreateAlloca(I->getType()); + AllocaResult->setAlignment(AllocaAlignment); + AllocaResult_i8 = + Builder.CreateBitCast(AllocaResult, Type::getInt8PtrTy(Ctx)); + Builder.CreateLifetimeStart(AllocaResult_i8, SizeVal64); + Args.push_back(AllocaResult_i8); + } + + // 'ordering' ('success_order' for cas) argument. + Args.push_back(OrderingVal); + + // 'failure_order' argument, if present. + if (Ordering2Val) + Args.push_back(Ordering2Val); + + // Now, the return type. + if (CASExpected) { + ResultTy = Type::getInt1Ty(Ctx); + Attr = Attr.addAttribute(Ctx, AttributeSet::ReturnIndex, Attribute::ZExt); + } else if (HasResult && UseSizedLibcall) + ResultTy = SizedIntTy; + else + ResultTy = Type::getVoidTy(Ctx); + + // Done with setting up arguments and return types, create the call: + SmallVector ArgTys; + for (Value *Arg : Args) + ArgTys.push_back(Arg->getType()); + FunctionType *FnType = FunctionType::get(ResultTy, ArgTys, false); + Constant *LibcallFn = + M->getOrInsertFunction(TLI->getLibcallName(RTLibType), FnType, Attr); + CallInst *Call = Builder.CreateCall(LibcallFn, Args); + Call->setAttributes(Attr); + Value *Result = Call; + + // And then, extract the results... + if (ValueOperand && !UseSizedLibcall) + Builder.CreateLifetimeEnd(AllocaValue_i8, SizeVal64); + + if (CASExpected) { + // The final result from the CAS is {load of 'expected' alloca, bool result + // from call} + Type *FinalResultTy = I->getType(); + Value *V = UndefValue::get(FinalResultTy); + Value *ExpectedOut = + Builder.CreateAlignedLoad(AllocaCASExpected, AllocaAlignment); + Builder.CreateLifetimeEnd(AllocaCASExpected_i8, SizeVal64); + V = Builder.CreateInsertValue(V, ExpectedOut, 0); + V = Builder.CreateInsertValue(V, Result, 1); + I->replaceAllUsesWith(V); + } else if (HasResult) { + Value *V; + if (UseSizedLibcall) + V = Builder.CreateBitOrPointerCast(Result, I->getType()); + else { + V = Builder.CreateAlignedLoad(AllocaResult, AllocaAlignment); + Builder.CreateLifetimeEnd(AllocaResult_i8, SizeVal64); + } + I->replaceAllUsesWith(V); + } + I->eraseFromParent(); + return true; +} Index: llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp =================================================================== --- llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp +++ llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp @@ -405,7 +405,66 @@ Names[RTLIB::SYNC_FETCH_AND_UMIN_4] = "__sync_fetch_and_umin_4"; Names[RTLIB::SYNC_FETCH_AND_UMIN_8] = "__sync_fetch_and_umin_8"; Names[RTLIB::SYNC_FETCH_AND_UMIN_16] = "__sync_fetch_and_umin_16"; - + + Names[RTLIB::ATOMIC_LOAD] = "__atomic_load"; + Names[RTLIB::ATOMIC_LOAD_1] = "__atomic_load_1"; + Names[RTLIB::ATOMIC_LOAD_2] = "__atomic_load_2"; + Names[RTLIB::ATOMIC_LOAD_4] = "__atomic_load_4"; + Names[RTLIB::ATOMIC_LOAD_8] = "__atomic_load_8"; + Names[RTLIB::ATOMIC_LOAD_16] = "__atomic_load_16"; + + Names[RTLIB::ATOMIC_STORE] = "__atomic_store"; + Names[RTLIB::ATOMIC_STORE_1] = "__atomic_store_1"; + Names[RTLIB::ATOMIC_STORE_2] = "__atomic_store_2"; + Names[RTLIB::ATOMIC_STORE_4] = "__atomic_store_4"; + Names[RTLIB::ATOMIC_STORE_8] = "__atomic_store_8"; + Names[RTLIB::ATOMIC_STORE_16] = "__atomic_store_16"; + + Names[RTLIB::ATOMIC_EXCHANGE] = "__atomic_exchange"; + Names[RTLIB::ATOMIC_EXCHANGE_1] = "__atomic_exchange_1"; + Names[RTLIB::ATOMIC_EXCHANGE_2] = "__atomic_exchange_2"; + Names[RTLIB::ATOMIC_EXCHANGE_4] = "__atomic_exchange_4"; + Names[RTLIB::ATOMIC_EXCHANGE_8] = "__atomic_exchange_8"; + Names[RTLIB::ATOMIC_EXCHANGE_16] = "__atomic_exchange_16"; + + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE] = "__atomic_compare_exchange"; + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_1] = "__atomic_compare_exchange_1"; + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_2] = "__atomic_compare_exchange_2"; + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_4] = "__atomic_compare_exchange_4"; + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_8] = "__atomic_compare_exchange_8"; + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_16] = "__atomic_compare_exchange_16"; + + Names[RTLIB::ATOMIC_FETCH_ADD_1] = "__atomic_fetch_add_1"; + Names[RTLIB::ATOMIC_FETCH_ADD_2] = "__atomic_fetch_add_2"; + Names[RTLIB::ATOMIC_FETCH_ADD_4] = "__atomic_fetch_add_4"; + Names[RTLIB::ATOMIC_FETCH_ADD_8] = "__atomic_fetch_add_8"; + Names[RTLIB::ATOMIC_FETCH_ADD_16] = "__atomic_fetch_add_16"; + Names[RTLIB::ATOMIC_FETCH_SUB_1] = "__atomic_fetch_sub_1"; + Names[RTLIB::ATOMIC_FETCH_SUB_2] = "__atomic_fetch_sub_2"; + Names[RTLIB::ATOMIC_FETCH_SUB_4] = "__atomic_fetch_sub_4"; + Names[RTLIB::ATOMIC_FETCH_SUB_8] = "__atomic_fetch_sub_8"; + Names[RTLIB::ATOMIC_FETCH_SUB_16] = "__atomic_fetch_sub_16"; + Names[RTLIB::ATOMIC_FETCH_AND_1] = "__atomic_fetch_and_1"; + Names[RTLIB::ATOMIC_FETCH_AND_2] = "__atomic_fetch_and_2"; + Names[RTLIB::ATOMIC_FETCH_AND_4] = "__atomic_fetch_and_4"; + Names[RTLIB::ATOMIC_FETCH_AND_8] = "__atomic_fetch_and_8"; + Names[RTLIB::ATOMIC_FETCH_AND_16] = "__atomic_fetch_and_16"; + Names[RTLIB::ATOMIC_FETCH_OR_1] = "__atomic_fetch_or_1"; + Names[RTLIB::ATOMIC_FETCH_OR_2] = "__atomic_fetch_or_2"; + Names[RTLIB::ATOMIC_FETCH_OR_4] = "__atomic_fetch_or_4"; + Names[RTLIB::ATOMIC_FETCH_OR_8] = "__atomic_fetch_or_8"; + Names[RTLIB::ATOMIC_FETCH_OR_16] = "__atomic_fetch_or_16"; + Names[RTLIB::ATOMIC_FETCH_XOR_1] = "__atomic_fetch_xor_1"; + Names[RTLIB::ATOMIC_FETCH_XOR_2] = "__atomic_fetch_xor_2"; + Names[RTLIB::ATOMIC_FETCH_XOR_4] = "__atomic_fetch_xor_4"; + Names[RTLIB::ATOMIC_FETCH_XOR_8] = "__atomic_fetch_xor_8"; + Names[RTLIB::ATOMIC_FETCH_XOR_16] = "__atomic_fetch_xor_16"; + Names[RTLIB::ATOMIC_FETCH_NAND_1] = "__atomic_fetch_nand_1"; + Names[RTLIB::ATOMIC_FETCH_NAND_2] = "__atomic_fetch_nand_2"; + Names[RTLIB::ATOMIC_FETCH_NAND_4] = "__atomic_fetch_nand_4"; + Names[RTLIB::ATOMIC_FETCH_NAND_8] = "__atomic_fetch_nand_8"; + Names[RTLIB::ATOMIC_FETCH_NAND_16] = "__atomic_fetch_nand_16"; + if (TT.getEnvironment() == Triple::GNU) { Names[RTLIB::SINCOS_F32] = "sincosf"; Names[RTLIB::SINCOS_F64] = "sincos"; @@ -777,6 +836,9 @@ GatherAllAliasesMaxDepth = 6; MinStackArgumentAlignment = 1; MinimumJumpTableEntries = 4; + // TODO: the default will be switched to 0 in the next commit, along + // with the Target-specific changes necessary. + MaxAtomicSizeInBitsSupported = 1024; InitLibcallNames(LibcallRoutineNames, TM.getTargetTriple()); InitCmpLibcallCCs(CmpLibcallCCs); Index: llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp =================================================================== --- llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp +++ llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp @@ -1611,6 +1611,13 @@ } // ATOMICs. + // Atomics are only supported on Sparcv9. (32bit atomics are also + // supported by the Leon sparcv8 variant, but we don't support that + // yet.) + if (Subtarget->isV9()) + setMaxAtomicSizeInBitsSupported(64); + else + setMaxAtomicSizeInBitsSupported(0); setOperationAction(ISD::ATOMIC_SWAP, MVT::i32, Legal); setOperationAction(ISD::ATOMIC_CMP_SWAP, MVT::i32, Index: llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll =================================================================== --- llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll +++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll @@ -0,0 +1,257 @@ +; RUN: opt -S %s -atomic-expand | FileCheck %s + +;;; NOTE: this test is actually target-independent -- any target which +;;; doesn't support inline atomics can be used. (E.g. X86 i386 would +;;; work, if LLVM is properly taught about what it's missing vs i586.) + +;target datalayout = "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128" +;target triple = "i386-unknown-unknown" +target datalayout = "e-m:e-p:32:32-i64:64-f128:64-n32-S64" +target triple = "sparc-unknown-unknown" + +;; First, check the sized calls. Except for cmpxchg, these are fairly +;; straightforward. + +; CHECK-LABEL: @test_load_i16( +; CHECK: %1 = bitcast i16* %arg to i8* +; CHECK: %2 = call i16 @__atomic_load_2(i8* %1, i32 5) +; CHECK: ret i16 %2 +define i16 @test_load_i16(i16* %arg) { + %ret = load atomic i16, i16* %arg seq_cst, align 4 + ret i16 %ret +} + +; CHECK-LABEL: @test_store_i16( +; CHECK: %1 = bitcast i16* %arg to i8* +; CHECK: call void @__atomic_store_2(i8* %1, i16 %val, i32 5) +; CHECK: ret void +define void @test_store_i16(i16* %arg, i16 %val) { + store atomic i16 %val, i16* %arg seq_cst, align 4 + ret void +} + +; CHECK-LABEL: @test_exchange_i16( +; CHECK: %1 = bitcast i16* %arg to i8* +; CHECK: %2 = call i16 @__atomic_exchange_2(i8* %1, i16 %val, i32 5) +; CHECK: ret i16 %2 +define i16 @test_exchange_i16(i16* %arg, i16 %val) { + %ret = atomicrmw xchg i16* %arg, i16 %val seq_cst + ret i16 %ret +} + +; CHECK-LABEL: @test_cmpxchg_i16( +; CHECK: %1 = bitcast i16* %arg to i8* +; CHECK: %2 = alloca i16, align 2 +; CHECK: %3 = bitcast i16* %2 to i8* +; CHECK: call void @llvm.lifetime.start(i64 2, i8* %3) +; CHECK: store i16 %old, i16* %2, align 2 +; CHECK: %4 = call zeroext i1 @__atomic_compare_exchange_2(i8* %1, i8* %3, i16 %new, i32 5, i32 0) +; CHECK: %5 = load i16, i16* %2, align 2 +; CHECK: call void @llvm.lifetime.end(i64 2, i8* %3) +; CHECK: %6 = insertvalue { i16, i1 } undef, i16 %5, 0 +; CHECK: %7 = insertvalue { i16, i1 } %6, i1 %4, 1 +; CHECK: %ret = extractvalue { i16, i1 } %7, 0 +; CHECK: ret i16 %ret +define i16 @test_cmpxchg_i16(i16* %arg, i16 %old, i16 %new) { + %ret_succ = cmpxchg i16* %arg, i16 %old, i16 %new seq_cst monotonic + %ret = extractvalue { i16, i1 } %ret_succ, 0 + ret i16 %ret +} + +; CHECK-LABEL: @test_add_i16( +; CHECK: %1 = bitcast i16* %arg to i8* +; CHECK: %2 = call i16 @__atomic_fetch_add_2(i8* %1, i16 %val, i32 5) +; CHECK: ret i16 %2 +define i16 @test_add_i16(i16* %arg, i16 %val) { + %ret = atomicrmw add i16* %arg, i16 %val seq_cst + ret i16 %ret +} + + +;; Now, check the output for the unsized libcalls. i128 is used for +;; these tests because the "16" suffixed functions aren't available on +;; 32-bit i386. + +; CHECK-LABEL: @test_load_i128( +; CHECK: %1 = bitcast i128* %arg to i8* +; CHECK: %2 = alloca i128, align 8 +; CHECK: %3 = bitcast i128* %2 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3) +; CHECK: call void @__atomic_load(i32 16, i8* %1, i8* %3, i32 5) +; CHECK: %4 = load i128, i128* %2, align 8 +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3) +; CHECK: ret i128 %4 +define i128 @test_load_i128(i128* %arg) { + %ret = load atomic i128, i128* %arg seq_cst, align 16 + ret i128 %ret +} + +; CHECK-LABEL @test_store_i128( +; CHECK: %1 = bitcast i128* %arg to i8* +; CHECK: %2 = alloca i128, align 8 +; CHECK: %3 = bitcast i128* %2 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3) +; CHECK: store i128 %val, i128* %2, align 8 +; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5) +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3) +; CHECK: ret void +define void @test_store_i128(i128* %arg, i128 %val) { + store atomic i128 %val, i128* %arg seq_cst, align 16 + ret void +} + +; CHECK-LABEL: @test_exchange_i128( +; CHECK: %1 = bitcast i128* %arg to i8* +; CHECK: %2 = alloca i128, align 8 +; CHECK: %3 = bitcast i128* %2 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3) +; CHECK: store i128 %val, i128* %2, align 8 +; CHECK: %4 = alloca i128, align 8 +; CHECK: %5 = bitcast i128* %4 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5) +; CHECK: call void @__atomic_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5) +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3) +; CHECK: %6 = load i128, i128* %4, align 8 +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5) +; CHECK: ret i128 %6 +define i128 @test_exchange_i128(i128* %arg, i128 %val) { + %ret = atomicrmw xchg i128* %arg, i128 %val seq_cst + ret i128 %ret +} + +; CHECK-LABEL: @test_cmpxchg_i128( +; CHECK: %1 = bitcast i128* %arg to i8* +; CHECK: %2 = alloca i128, align 8 +; CHECK: %3 = bitcast i128* %2 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3) +; CHECK: store i128 %old, i128* %2, align 8 +; CHECK: %4 = alloca i128, align 8 +; CHECK: %5 = bitcast i128* %4 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5) +; CHECK: store i128 %new, i128* %4, align 8 +; CHECK: %6 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5, i32 0) +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5) +; CHECK: %7 = load i128, i128* %2, align 8 +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3) +; CHECK: %8 = insertvalue { i128, i1 } undef, i128 %7, 0 +; CHECK: %9 = insertvalue { i128, i1 } %8, i1 %6, 1 +; CHECK: %ret = extractvalue { i128, i1 } %9, 0 +; CHECK: ret i128 %ret +define i128 @test_cmpxchg_i128(i128* %arg, i128 %old, i128 %new) { + %ret_succ = cmpxchg i128* %arg, i128 %old, i128 %new seq_cst monotonic + %ret = extractvalue { i128, i1 } %ret_succ, 0 + ret i128 %ret +} + +; This one is a verbose expansion, as there is no generic +; __atomic_fetch_add function, so it needs to expand to a cmpxchg +; loop, which then itself expands into a libcall. + +; CHECK-LABEL: @test_add_i128( +; CHECK: %1 = alloca i128, align 8 +; CHECK: %2 = alloca i128, align 8 +; CHECK: %3 = load i128, i128* %arg, align 16 +; CHECK: br label %atomicrmw.start +; CHECK:atomicrmw.start: +; CHECK: %loaded = phi i128 [ %3, %0 ], [ %newloaded, %atomicrmw.start ] +; CHECK: %new = add i128 %loaded, %val +; CHECK: %4 = bitcast i128* %arg to i8* +; CHECK: %5 = bitcast i128* %1 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5) +; CHECK: store i128 %loaded, i128* %1, align 8 +; CHECK: %6 = bitcast i128* %2 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %6) +; CHECK: store i128 %new, i128* %2, align 8 +; CHECK: %7 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %4, i8* %5, i8* %6, i32 5, i32 5) +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %6) +; CHECK: %8 = load i128, i128* %1, align 8 +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5) +; CHECK: %9 = insertvalue { i128, i1 } undef, i128 %8, 0 +; CHECK: %10 = insertvalue { i128, i1 } %9, i1 %7, 1 +; CHECK: %success = extractvalue { i128, i1 } %10, 1 +; CHECK: %newloaded = extractvalue { i128, i1 } %10, 0 +; CHECK: br i1 %success, label %atomicrmw.end, label %atomicrmw.start +; CHECK:atomicrmw.end: +; CHECK: ret i128 %newloaded +define i128 @test_add_i128(i128* %arg, i128 %val) { + %ret = atomicrmw add i128* %arg, i128 %val seq_cst + ret i128 %ret +} + +;; Ensure that non-integer types get bitcast correctly on the way in and out of a libcall: + +; CHECK-LABEL: @test_load_double( +; CHECK: %1 = bitcast double* %arg to i8* +; CHECK: %2 = call i64 @__atomic_load_8(i8* %1, i32 5) +; CHECK: %3 = bitcast i64 %2 to double +; CHECK: ret double %3 +define double @test_load_double(double* %arg, double %val) { + %1 = load atomic double, double* %arg seq_cst, align 16 + ret double %1 +} + +; CHECK-LABEL: @test_store_double( +; CHECK: %1 = bitcast double* %arg to i8* +; CHECK: %2 = bitcast double %val to i64 +; CHECK: call void @__atomic_store_8(i8* %1, i64 %2, i32 5) +; CHECK: ret void +define void @test_store_double(double* %arg, double %val) { + store atomic double %val, double* %arg seq_cst, align 16 + ret void +} + +; CHECK-LABEL: @test_cmpxchg_ptr( +; CHECK: %1 = bitcast i16** %arg to i8* +; CHECK: %2 = alloca i16*, align 4 +; CHECK: %3 = bitcast i16** %2 to i8* +; CHECK: call void @llvm.lifetime.start(i64 4, i8* %3) +; CHECK: store i16* %old, i16** %2, align 4 +; CHECK: %4 = ptrtoint i16* %new to i32 +; CHECK: %5 = call zeroext i1 @__atomic_compare_exchange_4(i8* %1, i8* %3, i32 %4, i32 5, i32 2) +; CHECK: %6 = load i16*, i16** %2, align 4 +; CHECK: call void @llvm.lifetime.end(i64 4, i8* %3) +; CHECK: %7 = insertvalue { i16*, i1 } undef, i16* %6, 0 +; CHECK: %8 = insertvalue { i16*, i1 } %7, i1 %5, 1 +; CHECK: %ret = extractvalue { i16*, i1 } %8, 0 +; CHECK: ret i16* %ret +; CHECK: } +define i16* @test_cmpxchg_ptr(i16** %arg, i16* %old, i16* %new) { + %ret_succ = cmpxchg i16** %arg, i16* %old, i16* %new seq_cst acquire + %ret = extractvalue { i16*, i1 } %ret_succ, 0 + ret i16* %ret +} + +;; ...and for a non-integer type of large size too. + +; CHECK-LABEL: @test_store_fp128 +; CHECK: %1 = bitcast fp128* %arg to i8* +; CHECK: %2 = alloca fp128, align 8 +; CHECK: %3 = bitcast fp128* %2 to i8* +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3) +; CHECK: store fp128 %val, fp128* %2, align 8 +; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5) +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3) +; CHECK: ret void +define void @test_store_fp128(fp128* %arg, fp128 %val) { + store atomic fp128 %val, fp128* %arg seq_cst, align 16 + ret void +} + +;; Unaligned loads and stores should be expanded to the generic +;; libcall, just like large loads/stores, and not a specialized one. +;; NOTE: atomicrmw and cmpxchg don't yet support an align attribute; +;; when such support is added, they should also be tested here. + +; CHECK-LABEL: @test_unaligned_load_i16( +; CHECK: __atomic_load( +define i16 @test_unaligned_load_i16(i16* %arg) { + %ret = load atomic i16, i16* %arg seq_cst, align 1 + ret i16 %ret +} + +; CHECK-LABEL: @test_unaligned_store_i16( +; CHECK: __atomic_store( +define void @test_unaligned_store_i16(i16* %arg, i16 %val) { + store atomic i16 %val, i16* %arg seq_cst, align 1 + ret void +} Index: llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg =================================================================== --- llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg +++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg @@ -0,0 +1,2 @@ +if not 'Sparc' in config.root.targets: + config.unsupported = True