Index: docs/LangRef.rst =================================================================== --- docs/LangRef.rst +++ docs/LangRef.rst @@ -2160,10 +2160,24 @@ .. _singlethread: -If an atomic operation is marked ``singlethread``, it only *synchronizes -with* or participates in modification and seq\_cst total orderings with -other operations running in the same thread (for example, in signal -handlers). +If an atomic operation is marked ``singlethread``, it only *synchronizes with*, +and only participates in the seq\_cst total orderings of, other operations +running in the same thread (for example, in signal handlers). + +.. _syncscope: + +If an atomic operation is marked ``syncscope()``, then it +*synchronizes with*, and participates in the seq\_cst total +orderings of, other atomic operations marked ``syncscope()`` +for the same value of ````. It is target defined how it +interacts with atomic operations marked ``singlethread``, +marked ``syncscope()`` with a different value of ````, +or not marked ``singlethread`` or ``syncscope()``. + +Otherwise, an atomic operation that is not marked ``singlethread`` +or ``syncscope()`` *synchronizes with*, and participates in +the global seq\_cst total orderings of, other operations that are +not marked ``singlethread`` or ``syncscope()``. .. _fastmath: @@ -7020,7 +7034,7 @@ :: = load [volatile] , * [, align ][, !nontemporal !][, !invariant.load !][, !invariant.group !][, !nonnull !][, !dereferenceable !][, !dereferenceable_or_null !][, !align !] - = load atomic [volatile] , * [singlethread] , align [, !invariant.group !] + = load atomic [volatile] , * [singlethread|syncscope()] , align [, !invariant.group !] ! = !{ i32 1 } ! = !{i64 } ! = !{ i64 } @@ -7041,14 +7055,14 @@ :ref:`volatile operations `. If the ``load`` is marked as ``atomic``, it takes an extra :ref:`ordering -` and optional ``singlethread`` argument. The ``release`` and -``acq_rel`` orderings are not valid on ``load`` instructions. Atomic loads -produce :ref:`defined ` results when they may see multiple atomic -stores. The type of the pointee must be an integer, pointer, or floating-point -type whose bit width is a power of two greater than or equal to eight and less -than or equal to a target-specific size limit. ``align`` must be explicitly -specified on atomic loads, and the load has undefined behavior if the alignment -is not set to a value which is at least the size in bytes of the +` and optional ``singlethread`` or ``syncscope()`` argument. The +``release`` and ``acq_rel`` orderings are not valid on ``load`` instructions. +Atomic loads produce :ref:`defined ` results when they may see +multiple atomic stores. The type of the pointee must be an integer, pointer, or +floating-point type whose bit width is a power of two greater than or equal to +eight and less than or equal to a target-specific size limit. ``align`` must be +explicitly specified on atomic loads, and the load has undefined behavior if the +alignment is not set to a value which is at least the size in bytes of the pointee. ``!nontemporal`` does not have any defined semantics for atomic loads. The optional constant ``align`` argument specifies the alignment of the @@ -7149,7 +7163,7 @@ :: store [volatile] , * [, align ][, !nontemporal !][, !invariant.group !] ; yields void - store atomic [volatile] , * [singlethread] , align [, !invariant.group !] ; yields void + store atomic [volatile] , * [singlethread|syncscope()] , align [, !invariant.group !] ; yields void Overview: """"""""" @@ -7169,14 +7183,14 @@ structural type `) can be stored. If the ``store`` is marked as ``atomic``, it takes an extra :ref:`ordering -` and optional ``singlethread`` argument. The ``acquire`` and -``acq_rel`` orderings aren't valid on ``store`` instructions. Atomic loads -produce :ref:`defined ` results when they may see multiple atomic -stores. The type of the pointee must be an integer, pointer, or floating-point -type whose bit width is a power of two greater than or equal to eight and less -than or equal to a target-specific size limit. ``align`` must be explicitly -specified on atomic stores, and the store has undefined behavior if the -alignment is not set to a value which is at least the size in bytes of the +` and optional ``singlethread`` or ``syncscope()`` argument. The +``acquire`` and ``acq_rel`` orderings aren't valid on ``store`` instructions. +Atomic loads produce :ref:`defined ` results when they may see +multiple atomic stores. The type of the pointee must be an integer, pointer, or +floating-point type whose bit width is a power of two greater than or equal to +eight and less than or equal to a target-specific size limit. ``align`` must be +explicitly specified on atomic stores, and the store has undefined behavior if +the alignment is not set to a value which is at least the size in bytes of the pointee. ``!nontemporal`` does not have any defined semantics for atomic stores. The optional constant ``align`` argument specifies the alignment of the @@ -7237,7 +7251,7 @@ :: - fence [singlethread] ; yields void + fence [singlethread|syncscope()] ; yields void Overview: """"""""" @@ -7271,9 +7285,8 @@ ``acquire`` and ``release`` semantics specified above, participates in the global program order of other ``seq_cst`` operations and/or fences. -The optional ":ref:`singlethread `" argument specifies -that the fence only synchronizes with other fences in the same thread. -(This is useful for interacting with signal handlers.) +A ``fence`` instruction can also take an optional +":ref:`singlethread `" or ":ref:`syncscope `" argument. Example: """""""" @@ -7282,6 +7295,7 @@ fence acquire ; yields void fence singlethread seq_cst ; yields void + fence syncscope(2) seq_cst ; yields void .. _i_cmpxchg: @@ -7293,7 +7307,7 @@ :: - cmpxchg [weak] [volatile] * , , [singlethread] ; yields { ty, i1 } + cmpxchg [weak] [volatile] * , , [singlethread|syncscope()] ; yields { ty, i1 } Overview: """"""""" @@ -7322,10 +7336,8 @@ stronger than that on success, and the failure ordering cannot be either ``release`` or ``acq_rel``. -The optional "``singlethread``" argument declares that the ``cmpxchg`` -is only atomic with respect to code (usually signal handlers) running in -the same thread as the ``cmpxchg``. Otherwise the cmpxchg is atomic with -respect to all other code in the system. +A ``cmpxchg`` instruction can also take an optional +":ref:`singlethread `" or ":ref:`syncscope `" argument. The pointer passed into cmpxchg must have alignment greater than or equal to the size in memory of the operand. @@ -7379,7 +7391,7 @@ :: - atomicrmw [volatile] * , [singlethread] ; yields ty + atomicrmw [volatile] * , [singlethread|syncscope()] ; yields ty Overview: """"""""" @@ -7413,6 +7425,9 @@ order of execution of this ``atomicrmw`` with other :ref:`volatile operations `. +A ``atomicrmw`` instruction can also take an optional +":ref:`singlethread `" or ":ref:`syncscope `" argument. + Semantics: """""""""" Index: include/llvm/Bitcode/LLVMBitCodes.h =================================================================== --- include/llvm/Bitcode/LLVMBitCodes.h +++ include/llvm/Bitcode/LLVMBitCodes.h @@ -370,9 +370,15 @@ }; /// Encoded SynchronizationScope values. -enum AtomicSynchScopeCodes { +enum AtomicSynchScopeCodes : unsigned { + /// Encoded value for SingleThread synchronization scope. SYNCHSCOPE_SINGLETHREAD = 0, - SYNCHSCOPE_CROSSTHREAD = 1 + + /// Encoded value for CrossThread synchronization scope. + SYNCHSCOPE_CROSSTHREAD = 1, + + /// First encoded value for target specific synchronization scope. + SYNCHSCOPE_FIRSTTARGETSPECIFIC = 2 }; /// Markers and flags for call instruction. Index: include/llvm/CodeGen/SelectionDAGNodes.h =================================================================== --- include/llvm/CodeGen/SelectionDAGNodes.h +++ include/llvm/CodeGen/SelectionDAGNodes.h @@ -424,10 +424,9 @@ uint16_t IsNonTemporal : 1; uint16_t IsDereferenceable : 1; uint16_t IsInvariant : 1; - uint16_t SynchScope : 1; // enum SynchronizationScope uint16_t Ordering : 4; // enum AtomicOrdering }; - enum { NumMemSDNodeBits = NumSDNodeBits + 9 }; + enum { NumMemSDNodeBits = NumSDNodeBits + 8 }; class LSBaseSDNodeBitfields { friend class LSBaseSDNode; @@ -1079,6 +1078,10 @@ /// Memory reference information. MachineMemOperand *MMO; + /// The synchronization scope of this memory operation. Not quite enough room + /// in SubclassData for everything, so synch scope gets its own field. + SynchronizationScope SynchScope; + public: MemSDNode(unsigned Opc, unsigned Order, const DebugLoc &dl, SDVTList VTs, EVT MemoryVT, MachineMemOperand *MMO); @@ -1108,11 +1111,14 @@ bool isDereferenceable() const { return MemSDNodeBits.IsDereferenceable; } bool isInvariant() const { return MemSDNodeBits.IsInvariant; } + /// Returns the atomic ordering requirements for this memory operation. AtomicOrdering getOrdering() const { return static_cast(MemSDNodeBits.Ordering); } + + /// Returns the synchronization scope for this memory operation. SynchronizationScope getSynchScope() const { - return static_cast(MemSDNodeBits.SynchScope); + return SynchScope; } // Returns the offset from the location of the access. @@ -1186,8 +1192,9 @@ /// This is an SDNode representing atomic operations. class AtomicSDNode : public MemSDNode { - /// For cmpxchg instructions, the ordering requirements when a store does not - /// occur. + /// For cmpxchg atomic operations, the atomic ordering requirements when store + /// does not occur. Not quite enough room in SubclassData for everything, so + /// failure ordering gets its own field. AtomicOrdering FailureOrdering; void InitAtomic(AtomicOrdering SuccessOrdering, @@ -1195,9 +1202,8 @@ SynchronizationScope SynchScope) { MemSDNodeBits.Ordering = static_cast(SuccessOrdering); assert(getOrdering() == SuccessOrdering && "Value truncated"); - MemSDNodeBits.SynchScope = static_cast(SynchScope); - assert(getSynchScope() == SynchScope && "Value truncated"); this->FailureOrdering = FailureOrdering; + this->SynchScope = SynchScope; } public: @@ -1212,12 +1218,14 @@ const SDValue &getBasePtr() const { return getOperand(1); } const SDValue &getVal() const { return getOperand(2); } + /// For cmpxchg atomic operations, returns the atomic ordering requirements + /// when store occurs. AtomicOrdering getSuccessOrdering() const { return getOrdering(); } - // Not quite enough room in SubclassData for everything, so failure gets its - // own field. + /// For cmpxchg atomic operations, returns the atomic ordering requirements + /// when store does not occur. AtomicOrdering getFailureOrdering() const { return FailureOrdering; } Index: include/llvm/IR/Instructions.h =================================================================== --- include/llvm/IR/Instructions.h +++ include/llvm/IR/Instructions.h @@ -37,9 +37,16 @@ class DataLayout; class LLVMContext; -enum SynchronizationScope { +/// Predefined synchronization scopes. +enum SynchronizationScope : unsigned { + /// Synchronized with respect to signal handlers executing in the same thread. SingleThread = 0, - CrossThread = 1 + + /// Synchronized with respect to all concurrently executing threads. + CrossThread = 1, + + /// First target specific synchronization scope. + SynchronizationScopeFirstTargetSpecific = 2 }; //===----------------------------------------------------------------------===// @@ -219,30 +226,30 @@ void setAlignment(unsigned Align); - /// Returns the ordering effect of this fence. + /// Returns the ordering constraint of this load instruction. AtomicOrdering getOrdering() const { return AtomicOrdering((getSubclassDataFromInstruction() >> 7) & 7); } - /// Set the ordering constraint on this load. May not be Release or - /// AcquireRelease. + /// Sets the ordering constraint on this load instruction. May not be Release + /// or AcquireRelease. void setOrdering(AtomicOrdering Ordering) { setInstructionSubclassData((getSubclassDataFromInstruction() & ~(7 << 7)) | ((unsigned)Ordering << 7)); } + /// Returns the synchronization scope of this load instruction. SynchronizationScope getSynchScope() const { - return SynchronizationScope((getSubclassDataFromInstruction() >> 6) & 1); + return SynchScope; } - /// Specify whether this load is ordered with respect to all - /// concurrently executing threads, or only with respect to signal handlers - /// executing in the same thread. - void setSynchScope(SynchronizationScope xthread) { - setInstructionSubclassData((getSubclassDataFromInstruction() & ~(1 << 6)) | - (xthread << 6)); + /// Sets the synchronization scope on this load instruction. + void setSynchScope(SynchronizationScope SynchScope) { + this->SynchScope = SynchScope; } + /// Sets the ordering constraint and synchronization scope on this load + /// instruction. void setAtomic(AtomicOrdering Ordering, SynchronizationScope SynchScope = CrossThread) { setOrdering(Ordering); @@ -279,6 +286,11 @@ void setInstructionSubclassData(unsigned short D) { Instruction::setInstructionSubclassData(D); } + + /// The synchronization scope of this load instruction. Not quite enough room + /// in SubClassData for everything, so synchronization scope gets its own + /// field. + SynchronizationScope SynchScope; }; //===----------------------------------------------------------------------===// @@ -337,30 +349,30 @@ void setAlignment(unsigned Align); - /// Returns the ordering effect of this store. + /// Returns the ordering constraint of this store instruction. AtomicOrdering getOrdering() const { return AtomicOrdering((getSubclassDataFromInstruction() >> 7) & 7); } - /// Set the ordering constraint on this store. May not be Acquire or - /// AcquireRelease. + /// Sets the ordering constraint on this store instruction. May not be Acquire + /// or AcquireRelease. void setOrdering(AtomicOrdering Ordering) { setInstructionSubclassData((getSubclassDataFromInstruction() & ~(7 << 7)) | ((unsigned)Ordering << 7)); } + /// Returns the synchronization scope of this store instruction. SynchronizationScope getSynchScope() const { - return SynchronizationScope((getSubclassDataFromInstruction() >> 6) & 1); + return SynchScope; } - /// Specify whether this store instruction is ordered with respect to all - /// concurrently executing threads, or only with respect to signal handlers - /// executing in the same thread. - void setSynchScope(SynchronizationScope xthread) { - setInstructionSubclassData((getSubclassDataFromInstruction() & ~(1 << 6)) | - (xthread << 6)); + /// Sets the synchronization scope on this store instruction. + void setSynchScope(SynchronizationScope SynchScope) { + this->SynchScope = SynchScope; } + /// Sets the ordering constraint and synchronization scope on this store + /// instruction. void setAtomic(AtomicOrdering Ordering, SynchronizationScope SynchScope = CrossThread) { setOrdering(Ordering); @@ -400,6 +412,11 @@ void setInstructionSubclassData(unsigned short D) { Instruction::setInstructionSubclassData(D); } + + /// The synchronization scope of this store instruction. Not quite enough room + /// in SubClassData for everything, so synchronization scope gets its own + /// field. + SynchronizationScope SynchScope; }; template <> @@ -437,28 +454,26 @@ SynchronizationScope SynchScope, BasicBlock *InsertAtEnd); - /// Returns the ordering effect of this fence. + /// Returns the ordering constraint of this fence instruction. AtomicOrdering getOrdering() const { return AtomicOrdering(getSubclassDataFromInstruction() >> 1); } - /// Set the ordering constraint on this fence. May only be Acquire, Release, - /// AcquireRelease, or SequentiallyConsistent. + /// Sets the ordering constraint on this fence instruction. May only be + /// Acquire, Release, AcquireRelease, or SequentiallyConsistent. void setOrdering(AtomicOrdering Ordering) { setInstructionSubclassData((getSubclassDataFromInstruction() & 1) | ((unsigned)Ordering << 1)); } + /// Returns the synchronization scope of this fence instruction. SynchronizationScope getSynchScope() const { - return SynchronizationScope(getSubclassDataFromInstruction() & 1); + return SynchScope; } - /// Specify whether this fence orders other operations with respect to all - /// concurrently executing threads, or only with respect to signal handlers - /// executing in the same thread. - void setSynchScope(SynchronizationScope xthread) { - setInstructionSubclassData((getSubclassDataFromInstruction() & ~1) | - xthread); + /// Sets the synchronization scope on this fence instruction. + void setSynchScope(SynchronizationScope SynchScope) { + this->SynchScope = SynchScope; } // Methods for support type inquiry through isa, cast, and dyn_cast: @@ -475,6 +490,11 @@ void setInstructionSubclassData(unsigned short D) { Instruction::setInstructionSubclassData(D); } + + /// The synchronization scope of this fence instruction. Not quite enough room + /// in SubClassData for everything, so synchronization scope gets its own + /// field. + SynchronizationScope SynchScope; }; //===----------------------------------------------------------------------===// @@ -539,7 +559,14 @@ /// Transparently provide more efficient getOperand methods. DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value); - /// Set the ordering constraint on this cmpxchg. + /// Returns the ordering constraint of this cmpxchg instruction when store + /// occurs. + AtomicOrdering getSuccessOrdering() const { + return AtomicOrdering((getSubclassDataFromInstruction() >> 2) & 7); + } + + /// Sets the ordering constraint on this cmpxchg instruction when store + /// occurs. void setSuccessOrdering(AtomicOrdering Ordering) { assert(Ordering != AtomicOrdering::NotAtomic && "CmpXchg instructions can only be atomic."); @@ -547,6 +574,14 @@ ((unsigned)Ordering << 2)); } + /// Returns the ordering constraint of this cmpxchg instruction when store + /// does not occur. + AtomicOrdering getFailureOrdering() const { + return AtomicOrdering((getSubclassDataFromInstruction() >> 5) & 7); + } + + /// Sets the ordering constraint on this cmpxchg instruction when store + /// does not occur. void setFailureOrdering(AtomicOrdering Ordering) { assert(Ordering != AtomicOrdering::NotAtomic && "CmpXchg instructions can only be atomic."); @@ -554,28 +589,14 @@ ((unsigned)Ordering << 5)); } - /// Specify whether this cmpxchg is atomic and orders other operations with - /// respect to all concurrently executing threads, or only with respect to - /// signal handlers executing in the same thread. - void setSynchScope(SynchronizationScope SynchScope) { - setInstructionSubclassData((getSubclassDataFromInstruction() & ~2) | - (SynchScope << 1)); - } - - /// Returns the ordering constraint on this cmpxchg. - AtomicOrdering getSuccessOrdering() const { - return AtomicOrdering((getSubclassDataFromInstruction() >> 2) & 7); - } - - /// Returns the ordering constraint on this cmpxchg. - AtomicOrdering getFailureOrdering() const { - return AtomicOrdering((getSubclassDataFromInstruction() >> 5) & 7); + /// Returns the synchronization scope of this cmpxchg instruction. + SynchronizationScope getSynchScope() const { + return SynchScope; } - /// Returns whether this cmpxchg is atomic between threads or only within a - /// single thread. - SynchronizationScope getSynchScope() const { - return SynchronizationScope((getSubclassDataFromInstruction() & 2) >> 1); + /// Sets the synchronization scope on this cmpxchg instruction. + void setSynchScope(SynchronizationScope SynchScope) { + this->SynchScope = SynchScope; } Value *getPointerOperand() { return getOperand(0); } @@ -630,6 +651,11 @@ void setInstructionSubclassData(unsigned short D) { Instruction::setInstructionSubclassData(D); } + + /// The synchronization scope of this cmpxchg instruction. Not quite enough + /// room in SubClassData for everything, so synchronization scope gets its own + /// field. + SynchronizationScope SynchScope; }; template <> @@ -726,7 +752,12 @@ /// Transparently provide more efficient getOperand methods. DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value); - /// Set the ordering constraint on this RMW. + /// Returns the ordering constraint of this RMW instruction. + AtomicOrdering getOrdering() const { + return AtomicOrdering((getSubclassDataFromInstruction() >> 2) & 7); + } + + /// Sets the ordering constraint on this RMW instruction. void setOrdering(AtomicOrdering Ordering) { assert(Ordering != AtomicOrdering::NotAtomic && "atomicrmw instructions can only be atomic."); @@ -734,25 +765,16 @@ ((unsigned)Ordering << 2)); } - /// Specify whether this RMW orders other operations with respect to all - /// concurrently executing threads, or only with respect to signal handlers - /// executing in the same thread. - void setSynchScope(SynchronizationScope SynchScope) { - setInstructionSubclassData((getSubclassDataFromInstruction() & ~2) | - (SynchScope << 1)); - } - - /// Returns the ordering constraint on this RMW. - AtomicOrdering getOrdering() const { - return AtomicOrdering((getSubclassDataFromInstruction() >> 2) & 7); - } - - /// Returns whether this RMW is atomic between threads or only within a - /// single thread. + /// Returns the synchronization scope of this RMW instruction. SynchronizationScope getSynchScope() const { - return SynchronizationScope((getSubclassDataFromInstruction() & 2) >> 1); + return SynchScope; } + /// Sets the synchronization scope on this RMW instruction. + void setSynchScope(SynchronizationScope SynchScope) { + this->SynchScope = SynchScope; + } + Value *getPointerOperand() { return getOperand(0); } const Value *getPointerOperand() const { return getOperand(0); } static unsigned getPointerOperandIndex() { return 0U; } @@ -781,6 +803,11 @@ void setInstructionSubclassData(unsigned short D) { Instruction::setInstructionSubclassData(D); } + + /// The synchronization scope of this RMW instruction. Not quite enough room + /// in SubClassData for everything, so synchronization scope gets its own + /// field. + SynchronizationScope SynchScope; }; template <> Index: lib/AsmParser/LLLexer.cpp =================================================================== --- lib/AsmParser/LLLexer.cpp +++ lib/AsmParser/LLLexer.cpp @@ -541,6 +541,7 @@ KEYWORD(acq_rel); KEYWORD(seq_cst); KEYWORD(singlethread); + KEYWORD(syncscope); KEYWORD(nnan); KEYWORD(ninf); Index: lib/AsmParser/LLParser.h =================================================================== --- lib/AsmParser/LLParser.h +++ lib/AsmParser/LLParser.h @@ -239,6 +239,7 @@ bool ParseOptionalDerefAttrBytes(lltok::Kind AttrKind, uint64_t &Bytes); bool ParseScopeAndOrdering(bool isAtomic, SynchronizationScope &Scope, AtomicOrdering &Ordering); + bool ParseScope(SynchronizationScope &Scope); bool ParseOrdering(AtomicOrdering &Ordering); bool ParseOptionalStackAlignment(unsigned &Alignment); bool ParseOptionalCommaAlign(unsigned &Alignment, bool &AteExtraComma); Index: lib/AsmParser/LLParser.cpp =================================================================== --- lib/AsmParser/LLParser.cpp +++ lib/AsmParser/LLParser.cpp @@ -1887,8 +1887,10 @@ } /// ParseScopeAndOrdering -/// if isAtomic: ::= 'singlethread'? AtomicOrdering -/// else: ::= +/// if isAtomic: +/// ::= 'singlethread' or 'syncscope' '(' uint32 ')'? AtomicOrdering +/// else +/// ::= /// /// This sets Scope and Ordering to the parsed values. bool LLParser::ParseScopeAndOrdering(bool isAtomic, SynchronizationScope &Scope, @@ -1896,11 +1898,41 @@ if (!isAtomic) return false; + return ParseScope(Scope) || ParseOrdering(Ordering); +} + +/// ParseScope +/// ::= /* empty */ +/// ::= 'singlethread' +/// ::= 'syncscope' '(' uint32 ')' +/// +/// This sets Scope to the parsed value. +bool LLParser::ParseScope(SynchronizationScope &Scope) { + if (EatIfPresent(lltok::kw_syncscope)) { + auto StartParen = Lex.getLoc(); + if (!EatIfPresent(lltok::lparen)) + return Error(StartParen, "expected '(' in syncscope"); + + unsigned ScopeU32 = 0; + auto ScopeU32At = Lex.getLoc(); + if (ParseUInt32(ScopeU32)) + return true; + if (ScopeU32 < SynchronizationScopeFirstTargetSpecific) + return Error(ScopeU32At, "invalid target specific synchronization scope"); + + auto EndParen = Lex.getLoc(); + if (!EatIfPresent(lltok::rparen)) + return Error(EndParen, "expected ')' in syncscope"); + + Scope = SynchronizationScope(ScopeU32); + return false; + } + Scope = CrossThread; if (EatIfPresent(lltok::kw_singlethread)) Scope = SingleThread; - return ParseOrdering(Ordering); + return false; } /// ParseOrdering Index: lib/AsmParser/LLToken.h =================================================================== --- lib/AsmParser/LLToken.h +++ lib/AsmParser/LLToken.h @@ -94,6 +94,8 @@ kw_acq_rel, kw_seq_cst, kw_singlethread, + kw_syncscope, + kw_nnan, kw_ninf, kw_nsz, Index: lib/Bitcode/Reader/BitcodeReader.cpp =================================================================== --- lib/Bitcode/Reader/BitcodeReader.cpp +++ lib/Bitcode/Reader/BitcodeReader.cpp @@ -911,9 +911,13 @@ } static SynchronizationScope getDecodedSynchScope(unsigned Val) { + if (Val >= bitc::SYNCHSCOPE_FIRSTTARGETSPECIFIC) + return SynchronizationScope(SynchronizationScopeFirstTargetSpecific + + (Val - bitc::SYNCHSCOPE_FIRSTTARGETSPECIFIC)); + switch (Val) { + default: llvm_unreachable("Invalid syncscope"); case bitc::SYNCHSCOPE_SINGLETHREAD: return SingleThread; - default: // Map unknown scopes to cross-thread. case bitc::SYNCHSCOPE_CROSSTHREAD: return CrossThread; } } Index: lib/Bitcode/Writer/BitcodeWriter.cpp =================================================================== --- lib/Bitcode/Writer/BitcodeWriter.cpp +++ lib/Bitcode/Writer/BitcodeWriter.cpp @@ -590,11 +590,15 @@ } static unsigned getEncodedSynchScope(SynchronizationScope SynchScope) { + if (SynchScope >= SynchronizationScopeFirstTargetSpecific) + return unsigned(bitc::SYNCHSCOPE_FIRSTTARGETSPECIFIC + + (SynchScope - SynchronizationScopeFirstTargetSpecific)); + switch (SynchScope) { + default: llvm_unreachable("Invalid syncscope"); case SingleThread: return bitc::SYNCHSCOPE_SINGLETHREAD; case CrossThread: return bitc::SYNCHSCOPE_CROSSTHREAD; } - llvm_unreachable("Invalid synch scope"); } void ModuleBitcodeWriter::writeStringRecord(unsigned Code, StringRef Str, Index: lib/CodeGen/SelectionDAG/SelectionDAG.cpp =================================================================== --- lib/CodeGen/SelectionDAG/SelectionDAG.cpp +++ lib/CodeGen/SelectionDAG/SelectionDAG.cpp @@ -6710,7 +6710,8 @@ MemSDNode::MemSDNode(unsigned Opc, unsigned Order, const DebugLoc &dl, SDVTList VTs, EVT memvt, MachineMemOperand *mmo) - : SDNode(Opc, Order, dl, VTs), MemoryVT(memvt), MMO(mmo) { + : SDNode(Opc, Order, dl, VTs), MemoryVT(memvt), MMO(mmo), + SynchScope(CrossThread) { MemSDNodeBits.IsVolatile = MMO->isVolatile(); MemSDNodeBits.IsNonTemporal = MMO->isNonTemporal(); MemSDNodeBits.IsDereferenceable = MMO->isDereferenceable(); Index: lib/IR/AsmWriter.cpp =================================================================== --- lib/IR/AsmWriter.cpp +++ lib/IR/AsmWriter.cpp @@ -2120,9 +2120,18 @@ if (Ordering == AtomicOrdering::NotAtomic) return; - switch (SynchScope) { - case SingleThread: Out << " singlethread"; break; - case CrossThread: break; + if (SynchScope >= SynchronizationScopeFirstTargetSpecific) { + Out << " syncscope(" << unsigned(SynchScope) << ')'; + } else { + switch (SynchScope) { + case SingleThread: + Out << " singlethread"; + break; + case CrossThread: + break; + default: + llvm_unreachable("Invalid syncscope"); + } } Out << " " << toIRString(Ordering); @@ -2134,9 +2143,18 @@ assert(SuccessOrdering != AtomicOrdering::NotAtomic && FailureOrdering != AtomicOrdering::NotAtomic); - switch (SynchScope) { - case SingleThread: Out << " singlethread"; break; - case CrossThread: break; + if (SynchScope >= SynchronizationScopeFirstTargetSpecific) { + Out << " syncscope(" << unsigned(SynchScope) << ')'; + } else { + switch (SynchScope) { + case SingleThread: + Out << " singlethread"; + break; + case CrossThread: + break; + default: + llvm_unreachable("Invalid syncscope"); + } } Out << " " << toIRString(SuccessOrdering); Index: lib/Target/AMDGPU/AMDGPU.h =================================================================== --- lib/Target/AMDGPU/AMDGPU.h +++ lib/Target/AMDGPU/AMDGPU.h @@ -11,6 +11,8 @@ #ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPU_H #define LLVM_LIB_TARGET_AMDGPU_AMDGPU_H +#include "llvm/IR/Instructions.h" + namespace llvm { class AMDGPUTargetMachine; @@ -162,4 +164,38 @@ } // namespace AMDGPUAS +/// AMDGPU-specific synchronization scopes. +enum class AMDGPUSynchronizationScope : unsigned { + /// Synchronized with respect to the entire system, which includes all + /// work-items on all agents executing kernel dispatches for the same + /// application process, together with all agents executing the same + /// application process as the executing work-item. Only supported for the + /// global segment. + System = llvm::CrossThread, + + /// Synchronized with respect to signal handlers executing in the same + /// work-item. + SignalHandler = llvm::SingleThread, + + /// Synchronized with respect to the agent, which includes all work-items on + /// the same agent executing kernel dispatches for the same application + /// process as the executing work-item. Only supported for the global segment. + Agent = llvm::SynchronizationScopeFirstTargetSpecific, + + /// Synchronized with respect to the work-group, which includes all work-items + /// in the same work-group as the executing work-item. + WorkGroup, + + /// Synchronized with respect to the wavefront, which includes all work-items + /// in the same wavefront as the executing work-item. + Wavefront, + + /// Synchronized with respect to image fence instruction executing in the same + /// work-item. + Image, + + /// Unknown synchronization scope. + Unknown +}; + #endif Index: lib/Target/SystemZ/SystemZISelLowering.cpp =================================================================== --- lib/Target/SystemZ/SystemZISelLowering.cpp +++ lib/Target/SystemZ/SystemZISelLowering.cpp @@ -3178,7 +3178,7 @@ // The only fence that needs an instruction is a sequentially-consistent // cross-thread fence. if (FenceOrdering == AtomicOrdering::SequentiallyConsistent && - FenceScope == CrossThread) { + FenceScope != SingleThread) { return SDValue(DAG.getMachineNode(SystemZ::Serialize, DL, MVT::Other, Op.getOperand(0)), 0); Index: lib/Target/X86/X86ISelLowering.cpp =================================================================== --- lib/Target/X86/X86ISelLowering.cpp +++ lib/Target/X86/X86ISelLowering.cpp @@ -20994,7 +20994,7 @@ // The only fence that needs an instruction is a sequentially-consistent // cross-thread fence. if (FenceOrdering == AtomicOrdering::SequentiallyConsistent && - FenceScope == CrossThread) { + FenceScope != SingleThread) { if (Subtarget.hasMFence()) return DAG.getNode(X86ISD::MFENCE, dl, MVT::Other, Op.getOperand(0)); Index: lib/Transforms/Instrumentation/ThreadSanitizer.cpp =================================================================== --- lib/Transforms/Instrumentation/ThreadSanitizer.cpp +++ lib/Transforms/Instrumentation/ThreadSanitizer.cpp @@ -366,9 +366,9 @@ static bool isAtomic(Instruction *I) { if (LoadInst *LI = dyn_cast(I)) - return LI->isAtomic() && LI->getSynchScope() == CrossThread; + return LI->isAtomic() && LI->getSynchScope() != SingleThread; if (StoreInst *SI = dyn_cast(I)) - return SI->isAtomic() && SI->getSynchScope() == CrossThread; + return SI->isAtomic() && SI->getSynchScope() != SingleThread; if (isa(I)) return true; if (isa(I)) Index: test/Assembler/atomic.ll =================================================================== --- test/Assembler/atomic.ll +++ test/Assembler/atomic.ll @@ -7,10 +7,14 @@ load atomic i32, i32* %x unordered, align 4 ; CHECK: load atomic volatile i32, i32* %x singlethread acquire, align 4 load atomic volatile i32, i32* %x singlethread acquire, align 4 + ; CHECK: load atomic volatile i32, i32* %x syncscope(2) acquire, align 4 + load atomic volatile i32, i32* %x syncscope(2) acquire, align 4 ; CHECK: store atomic i32 3, i32* %x release, align 4 store atomic i32 3, i32* %x release, align 4 ; CHECK: store atomic volatile i32 3, i32* %x singlethread monotonic, align 4 store atomic volatile i32 3, i32* %x singlethread monotonic, align 4 + ; CHECK: store atomic volatile i32 3, i32* %x syncscope(3) monotonic, align 4 + store atomic volatile i32 3, i32* %x syncscope(3) monotonic, align 4 ; CHECK: cmpxchg i32* %x, i32 1, i32 0 singlethread monotonic monotonic cmpxchg i32* %x, i32 1, i32 0 singlethread monotonic monotonic ; CHECK: cmpxchg volatile i32* %x, i32 0, i32 1 acq_rel acquire @@ -19,13 +23,19 @@ cmpxchg i32* %x, i32 42, i32 0 acq_rel monotonic ; CHECK: cmpxchg weak i32* %x, i32 13, i32 0 seq_cst monotonic cmpxchg weak i32* %x, i32 13, i32 0 seq_cst monotonic + ; CHECK: cmpxchg weak i32* %x, i32 13, i32 0 syncscope(4) seq_cst monotonic + cmpxchg weak i32* %x, i32 13, i32 0 syncscope(4) seq_cst monotonic ; CHECK: atomicrmw add i32* %x, i32 10 seq_cst atomicrmw add i32* %x, i32 10 seq_cst ; CHECK: atomicrmw volatile xchg i32* %x, i32 10 monotonic atomicrmw volatile xchg i32* %x, i32 10 monotonic + ; CHECK: atomicrmw volatile xchg i32* %x, i32 10 syncscope(5) monotonic + atomicrmw volatile xchg i32* %x, i32 10 syncscope(5) monotonic ; CHECK: fence singlethread release fence singlethread release ; CHECK: fence seq_cst fence seq_cst + ; CHECK: fence syncscope(6) seq_cst + fence syncscope(6) seq_cst ret void }