MS ABI: Implement /volatile:ms

[majnemer]

Summary

The /volatile:ms semantics turn volatile loads and stores into atomic
acquire and release operations. This distinction is important because
volatile memory operations do not form a happens-before relationship
with non-atomic memory. This means that a volatile store is not
sufficient for implementing a mutex unlock routine.

[ABataev]

lib/CodeGen/CGStmtOpenMP.cpp

836 ↗

(On Diff #19811)

I'd say that if IsSeqCst is false, Atomic Ordering must be Monotonic rather than Unordered.

1. According to OpenMP 4.0 "non-sequentially consistent atomic construct has the same semantics as a

memory_order_relaxed atomic operation in C++11/C11.".

2. According to http://llvm.org/docs/Atomics.html "Monotonic... This corresponds to the C++11/C11 memory_order_relaxed"

[timurrrr]

Dima might want to take a look.

[dvyukov]

lib/CodeGen/CGAtomic.cpp

1011 ↗

(On Diff #19811)

I don't get the !AI.shouldUseLibcall() part.
If the atomic operation would use a libcall, then you don't turn volatile access into atomic? Why? How is it related to libcalls?

1017 ↗

(On Diff #19811)

Same here. How presence of builtin atomic affects this?

1018 ↗

(On Diff #19811)

In volatileIsAtomic you check that the value is volatile, but here you don't. Please make then consistent. Either check in both or don't check in both.

lib/CodeGen/CGExpr.cpp

1141 ↗

(On Diff #19811)

Won't this be overly fragile?
MS semantics effectively turn volatile variables into atomic variables. So the current isAtomicType becomes effectively useless, and whoever uses isAtomicType must also remember to add volatileIsAtomic. And there is nothing that reminds of that. Ty->isAtomicType() looks pretty much like what you need.
It would be less fragile if you fix the isAtomic predicate itself.
But I know very little about llvm internals, maybe it is not possible, or a bad idea for some other reason. I don't know.

lib/CodeGen/CodeGenFunction.h

2159 ↗

(On Diff #19811)

Here you use IsSeqCst, but in EmitAtomicStore you use AO.
Please make them consistent.

[majnemer]

lib/CodeGen/CGAtomic.cpp

1011 ↗

(On Diff #19811)

shouldUseLibcall means that than atomic operation is not suitably aligned or sized to be lowered to a load atomic or store atomic instruction. These conditions are identical to the conditions that MSVC uses to say that a volatile load or store will also have acquire or release semantics.

1017 ↗

(On Diff #19811)

As I mentioned earlier, not all volatile loads or stores will turn into acquire or release loads and stores.

1018 ↗

(On Diff #19811)

I can't check it here because the type being passed in might not be qualified with volatile, it might be a record with a volatile field. I'd rather not make everyone who calls the other overload to have to manually check LV.isVolatile() || hasVolatileMember(LV.getType()).

I'll just rename one of these to make it clear what it does.

lib/CodeGen/CGExpr.cpp

1141 ↗

(On Diff #19811)

I don't think that is semantically correct. isAtomicType() is used for types which are "qualified" with _Atomic.

Moreover, we would still need to be able to determine the different from a real _Atomic qualified type and a MS volatile type in order to determine the memory operation's order.

lib/CodeGen/CGStmtOpenMP.cpp

836 ↗

(On Diff #19811)

Good catch, will do.

lib/CodeGen/CodeGenFunction.h

2159 ↗

(On Diff #19811)

Sure, will do.

[dvyukov]

lib/CodeGen/CGAtomic.cpp

1011 ↗

(On Diff #19811)

Doesn't shouldUseLibcall also mean that the value is too fat for backend codegen, so just call a library functions (e.g. a 64-bit store on a 32-bit platform)? That's what I figure out from the name. Perhaps deserves a comment.

[majnemer]

• Address review feedback.
• Cleanup argument parsing

[rnk]

Looks good to me. Feel free to gather other feedback.

lib/CodeGen/CGAtomic.cpp

1017 ↗

(On Diff #19848)

Early return false if !CGOpts.MSVolatile before checking struct types for volatile members (not O(1))?

1023 ↗

(On Diff #19848)

Make the leading letter lowercase? "LValueIsSuitable..." starts with an initialism, so it stays upper.

lib/CodeGen/CodeGenFunction.h

2152–2153 ↗

(On Diff #19848)

Big enough to make not inline?

2171 ↗

(On Diff #19848)

ditto

[hans]

I just looked at the flag stuff since others seem to have looked at the rest.

include/clang/Driver/CLCompatOptions.td

206 ↗

(On Diff #19848)

nit: I'd skip "C++": just "standard semantics" is shorter (and more accurate in the C case I guess)

208 ↗

(On Diff #19848)

nit: there's no "will" in the :iso flag description. I'd leave it out here too. Would it be less accurate to just say that loads and stores are atomic?

test/Driver/cl-options.c

126 ↗

(On Diff #19848)

maybe add a test without /volatile and check that -fms-volatile doesn't get passed by default?

[majnemer]

include/clang/Driver/CLCompatOptions.td

206 ↗

(On Diff #19848)

Done.

208 ↗

(On Diff #19848)

Done.

lib/CodeGen/CGAtomic.cpp

1017 ↗

(On Diff #19848)

Sema has already figured it out so it is cheap to query.

1023 ↗

(On Diff #19848)

Done.

lib/CodeGen/CodeGenFunction.h

2152–2153 ↗

(On Diff #19848)

Done.

2171 ↗

(On Diff #19848)

Done.

test/Driver/cl-options.c

126 ↗

(On Diff #19848)

-fms-volatile is passed by default depending on whether or not you are targeting x86.

cfe/trunk/include/clang/Driver/CLCompatOptions.td

 
 
 // Non-aliases:
 
 def _SLASH_arch : CLCompileJoined<"arch:">,
   HelpText<"Set architecture for code generation">;
 
 def _SLASH_M_Group : OptionGroup<"</M group>">, Group<cl_compile_Group>;
+def _SLASH_volatile_Group : OptionGroup<"</volatile group>">, Group<cl_compile_Group>;
 
 def _SLASH_EH : CLJoined<"EH">, HelpText<"Exception handling model">;
 def _SLASH_EP : CLFlag<"EP">,
   HelpText<"Disable linemarker output and preprocess to stdout">;
 def _SLASH_FA : CLFlag<"FA">,
   HelpText<"Output assembly code file during compilation">;
 def _SLASH_Fa : CLJoined<"Fa">,
   HelpText<"Output assembly code to this file during compilation">,
def _SLASH_o : CLJoinedOrSeparate<"o">,
   MetaVarName<"<file or directory>">;
 def _SLASH_P : CLFlag<"P">, HelpText<"Preprocess to file">;
 def _SLASH_Tc : CLCompileJoinedOrSeparate<"Tc">,
   HelpText<"Specify a C source file">, MetaVarName<"<filename>">;
 def _SLASH_TC : CLCompileFlag<"TC">, HelpText<"Treat all source files as C">;
 def _SLASH_Tp : CLCompileJoinedOrSeparate<"Tp">,
   HelpText<"Specify a C++ source file">, MetaVarName<"<filename>">;
 def _SLASH_TP : CLCompileFlag<"TP">, HelpText<"Treat all source files as C++">;
-
+def _SLASH_volatile_iso : Option<["/", "-"], "volatile:iso", KIND_FLAG>, Group<_SLASH_volatile_Group>,
+  Flags<[CLOption, DriverOption]>, HelpText<"Volatile loads and stores have standard semantics">;
+def _SLASH_volatile_ms  : Option<["/", "-"], "volatile:ms", KIND_FLAG>, Group<_SLASH_volatile_Group>,
+  Flags<[CLOption, DriverOption]>, HelpText<"Volatile loads and stores have acquire and release semantics">;
 
 // Ignored:
 
 def _SLASH_analyze_ : CLIgnoredFlag<"analyze-">;
 def _SLASH_cgthreads : CLIgnoredJoined<"cgthreads">;
 def _SLASH_d2Zi_PLUS : CLIgnoredFlag<"d2Zi+">;
 def _SLASH_errorReport : CLIgnoredJoined<"errorReport">;
 def _SLASH_FS : CLIgnoredFlag<"FS">, HelpText<"Force synchronous PDB writes">;
 def _SLASH_Gd : CLIgnoredFlag<"Gd">;
 def _SLASH_GF : CLIgnoredFlag<"GF">;
 def _SLASH_GS_ : CLIgnoredFlag<"GS-">;
 def _SLASH_kernel_ : CLIgnoredFlag<"kernel-">;
 def _SLASH_nologo : CLIgnoredFlag<"nologo">;
 def _SLASH_Ob1 : CLIgnoredFlag<"Ob1">;
 def _SLASH_Ob2 : CLIgnoredFlag<"Ob2">;
 def _SLASH_RTC : CLIgnoredJoined<"RTC">;
 def _SLASH_sdl : CLIgnoredFlag<"sdl">;
 def _SLASH_sdl_ : CLIgnoredFlag<"sdl-">;
-def _SLASH_volatile_iso : CLIgnoredFlag<"volatile:iso">;
 def _SLASH_w : CLIgnoredJoined<"w">;
 def _SLASH_Zc_auto : CLIgnoredFlag<"Zc:auto">;
 def _SLASH_Zc_forScope : CLIgnoredFlag<"Zc:forScope">;
 def _SLASH_Zc_inline : CLIgnoredFlag<"Zc:inline">;
 def _SLASH_Zc_rvalueCast : CLIgnoredFlag<"Zc:rvalueCast">;
 def _SLASH_Zc_wchar_t : CLIgnoredFlag<"Zc:wchar_t">;
 def _SLASH_Zm : CLIgnoredJoined<"Zm">;
 def _SLASH_Zo : CLIgnoredFlag<"Zo">;

 def _SLASH_openmp : CLFlag<"openmp">;
 def _SLASH_Qfast_transcendentals : CLFlag<"Qfast_transcendentals">;
 def _SLASH_QIfist : CLFlag<"QIfist">;
 def _SLASH_Qimprecise_fwaits : CLFlag<"Qimprecise_fwaits">;
 def _SLASH_Qpar : CLFlag<"Qpar">;
 def _SLASH_Qvec_report : CLJoined<"Qvec-report">;
 def _SLASH_u : CLFlag<"u">;
 def _SLASH_V : CLFlag<"V">;
-def _SLASH_volatile_ms : CLFlag<"volatile:ms">;
 def _SLASH_WL : CLFlag<"WL">;
 def _SLASH_Wp64 : CLFlag<"Wp64">;
 def _SLASH_X : CLFlag<"X">;
 def _SLASH_Yc : CLJoined<"Yc">;
 def _SLASH_Y_ : CLFlag<"Y-">;
 def _SLASH_Yd : CLFlag<"Yd">;
 def _SLASH_Yl : CLJoined<"Yl">;
 def _SLASH_Yu : CLJoined<"Yu">;
 def _SLASH_Za : CLFlag<"Za">;
 def _SLASH_Zc : CLJoined<"Zc:">;
 def _SLASH_Ze : CLFlag<"Ze">;
 def _SLASH_Zg : CLFlag<"Zg">;
 def _SLASH_ZI : CLFlag<"ZI">;
 def _SLASH_Zl : CLFlag<"Zl">;
 def _SLASH_ZW : CLJoined<"ZW">;

cfe/trunk/include/clang/Driver/Options.td

 def fmacro_backtrace_limit_EQ : Joined<["-"], "fmacro-backtrace-limit=">,
                                 Group<f_Group>;
 def fmerge_all_constants : Flag<["-"], "fmerge-all-constants">, Group<f_Group>;
 def fmessage_length_EQ : Joined<["-"], "fmessage-length=">, Group<f_Group>;
 def fms_extensions : Flag<["-"], "fms-extensions">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Accept some non-standard constructs supported by the Microsoft compiler">;
 def fms_compatibility : Flag<["-"], "fms-compatibility">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Enable full Microsoft Visual C++ compatibility">;
+def fms_volatile : Joined<["-"], "fms-volatile">, Group<f_Group>, Flags<[CC1Option]>;
 def fmsc_version : Joined<["-"], "fmsc-version=">, Group<f_Group>, Flags<[DriverOption, CoreOption]>,
   HelpText<"Microsoft compiler version number to report in _MSC_VER (0 = don't define it (default))">;
 def fms_compatibility_version
     : Joined<["-"], "fms-compatibility-version=">,
       Group<f_Group>,
       Flags<[ CC1Option, CoreOption ]>,
       HelpText<"Dot-separated value representing the Microsoft compiler "
                "version number to report in _MSC_VER (0 = don't define it "

cfe/trunk/include/clang/Frontend/CodeGenOptions.def

 CODEGENOPT(FunctionSections  , 1, 0) ///< Set when -ffunction-sections is enabled.
 CODEGENOPT(InstrumentFunctions , 1, 0) ///< Set when -finstrument-functions is
                                        ///< enabled.
 CODEGENOPT(InstrumentForProfiling , 1, 0) ///< Set when -pg is enabled.
 CODEGENOPT(LessPreciseFPMAD  , 1, 0) ///< Enable less precise MAD instructions to
                                      ///< be generated.
 CODEGENOPT(MergeAllConstants , 1, 1) ///< Merge identical constants.
 CODEGENOPT(MergeFunctions    , 1, 0) ///< Set when -fmerge-functions is enabled.
+CODEGENOPT(MSVolatile        , 1, 0) ///< Set when /volatile:ms is enabled.
 CODEGENOPT(NoCommon          , 1, 0) ///< Set when -fno-common or C++ is enabled.
 CODEGENOPT(NoDwarfDirectoryAsm , 1, 0) ///< Set when -fno-dwarf-directory-asm is
                                        ///< enabled.
 CODEGENOPT(NoExecStack       , 1, 0) ///< Set when -Wa,--noexecstack is enabled.
 CODEGENOPT(FatalWarnings     , 1, 0) ///< Set when -Wa,--fatal-warnings is
                                      ///< enabled.
 CODEGENOPT(EnableSegmentedStacks , 1, 0) ///< Set when -fsplit-stack is enabled.
 CODEGENOPT(NoGlobalMerge     , 1, 0) ///< Set when -mno-global-merge is enabled.

cfe/trunk/lib/CodeGen/CGAtomic.cpp

RValue AtomicInfo::convertIntToValue(llvm::Value *IntVal,
   // Slam the integer into the temporary.
   llvm::Value *CastTemp = emitCastToAtomicIntPointer(Temp);
   CGF.Builder.CreateAlignedStore(IntVal, CastTemp, TempAlignment.getQuantity())
       ->setVolatile(TempIsVolatile);
 
   return convertTempToRValue(Temp, ResultSlot, Loc);
 }
 
+/// An LValue is a candidate for having its loads and stores be made atomic if
+/// we are operating under /volatile:ms *and* the LValue itself is volatile and
+/// performing such an operation can be performed without a libcall.
+bool CodeGenFunction::LValueIsSuitableForInlineAtomic(LValue LV) {
+  AtomicInfo AI(*this, LV);
+  bool IsVolatile = LV.isVolatile() || hasVolatileMember(LV.getType());
+  // An atomic is inline if we don't need to use a libcall.
+  bool AtomicIsInline = !AI.shouldUseLibcall();
+  return CGM.getCodeGenOpts().MSVolatile && IsVolatile && AtomicIsInline;
+}
+
+/// An type is a candidate for having its loads and stores be made atomic if
+/// we are operating under /volatile:ms *and* we know the access is volatile and
+/// performing such an operation can be performed without a libcall.
+bool CodeGenFunction::typeIsSuitableForInlineAtomic(QualType Ty,
+                                                    bool IsVolatile) const {
+  // An atomic is inline if we don't need to use a libcall (e.g. it is builtin).
+  bool AtomicIsInline = getContext().getTargetInfo().hasBuiltinAtomic(
+      getContext().getTypeSize(Ty), getContext().getTypeAlign(Ty));
+  return CGM.getCodeGenOpts().MSVolatile && IsVolatile && AtomicIsInline;
+}
+
+RValue CodeGenFunction::EmitAtomicLoad(LValue LV, SourceLocation SL,
+                                       AggValueSlot Slot) {
+  llvm::AtomicOrdering AO;
+  bool IsVolatile = LV.isVolatileQualified();
+  if (LV.getType()->isAtomicType()) {
+    AO = llvm::SequentiallyConsistent;
+  } else {
+    AO = llvm::Acquire;
+    IsVolatile = true;
+  }
+  return EmitAtomicLoad(LV, SL, AO, IsVolatile, Slot);
+}
+
 /// Emit a load from an l-value of atomic type.  Note that the r-value
 /// we produce is an r-value of the atomic *value* type.
 RValue CodeGenFunction::EmitAtomicLoad(LValue src, SourceLocation loc,
+                                       llvm::AtomicOrdering AO, bool IsVolatile,
                                        AggValueSlot resultSlot) {
   AtomicInfo atomics(*this, src);
   LValue LVal = atomics.getAtomicLValue();
   llvm::Value *SrcAddr = nullptr;
   llvm::AllocaInst *NonSimpleTempAlloca = nullptr;
   if (LVal.isSimple())
     SrcAddr = LVal.getAddress();
   else {
if (atomics.shouldUseLibcall()) {
 
     // Produce the r-value.
     return atomics.convertTempToRValue(tempAddr, resultSlot, loc);
   }
 
   // Okay, we're doing this natively.
   llvm::Value *addr = atomics.emitCastToAtomicIntPointer(SrcAddr);
   llvm::LoadInst *load = Builder.CreateLoad(addr, "atomic-load");
-  load->setAtomic(llvm::SequentiallyConsistent);
+  load->setAtomic(AO);
 
   // Other decoration.
   load->setAlignment(src.getAlignment().getQuantity());
-  if (src.isVolatileQualified())
+  if (IsVolatile)
     load->setVolatile(true);
   if (src.getTBAAInfo())
     CGM.DecorateInstruction(load, src.getTBAAInfo());
 
   // If we're ignoring an aggregate return, don't do anything.
   if (atomics.getEvaluationKind() == TEK_Aggregate && resultSlot.isIgnored())
     return RValue::getAggregate(nullptr, false);
 
llvm::Value *AtomicInfo::convertRValueToInt(RValue RVal) const {
   llvm::Value *Addr = materializeRValue(RVal);
 
   // Cast the temporary to the atomic int type and pull a value out.
   Addr = emitCastToAtomicIntPointer(Addr);
   return CGF.Builder.CreateAlignedLoad(Addr,
                                        getAtomicAlignment().getQuantity());
 }
 
+void CodeGenFunction::EmitAtomicStore(RValue rvalue, LValue lvalue,
+                                      bool isInit) {
+  bool IsVolatile = lvalue.isVolatileQualified();
+  llvm::AtomicOrdering AO;
+  if (lvalue.getType()->isAtomicType()) {
+    AO = llvm::SequentiallyConsistent;
+  } else {
+    AO = llvm::Release;
+    IsVolatile = true;
+  }
+  return EmitAtomicStore(rvalue, lvalue, AO, IsVolatile, isInit);
+}
+
 /// Emit a store to an l-value of atomic type.
 ///
 /// Note that the r-value is expected to be an r-value *of the atomic
 /// type*; this means that for aggregate r-values, it should include
 /// storage for any padding that was necessary.
-void CodeGenFunction::EmitAtomicStore(RValue rvalue, LValue dest, bool isInit) {
+void CodeGenFunction::EmitAtomicStore(RValue rvalue, LValue dest,
+                                      llvm::AtomicOrdering AO, bool IsVolatile,
+                                      bool isInit) {
   // If this is an aggregate r-value, it should agree in type except
   // maybe for address-space qualification.
   assert(!rvalue.isAggregate() ||
          rvalue.getAggregateAddr()->getType()->getPointerElementType()
            == dest.getAddress()->getType()->getPointerElementType());
 
   AtomicInfo atomics(*this, dest);
 
void CodeGenFunction::EmitAtomicStore(RValue rvalue, LValue dest,
   // Okay, we're doing this natively.
   llvm::Value *intValue = atomics.convertRValueToInt(rvalue);
 
   // Do the atomic store.
   llvm::Value *addr = atomics.emitCastToAtomicIntPointer(dest.getAddress());
   llvm::StoreInst *store = Builder.CreateStore(intValue, addr);
 
   // Initializations don't need to be atomic.
-  if (!isInit) store->setAtomic(llvm::SequentiallyConsistent);
+  if (!isInit) store->setAtomic(AO);
 
   // Other decoration.
   store->setAlignment(dest.getAlignment().getQuantity());
-  if (dest.isVolatileQualified())
+  if (IsVolatile)
     store->setVolatile(true);
   if (dest.getTBAAInfo())
     CGM.DecorateInstruction(store, dest.getTBAAInfo());
 }
 
 /// Emit a compare-and-exchange op for atomic type.
 ///
 std::pair<RValue, RValue> CodeGenFunction::EmitAtomicCompareExchange(

cfe/trunk/lib/CodeGen/CGExpr.cpp

if (VTy->getNumElements() == 3) {
       V = Builder.CreateShuffleVector(LoadVal,
                                       llvm::UndefValue::get(vec4Ty),
                                       MaskV, "extractVec");
       return EmitFromMemory(V, Ty);
     }
   }
 
   // Atomic operations have to be done on integral types.
-  if (Ty->isAtomicType()) {
+  if (Ty->isAtomicType() || typeIsSuitableForInlineAtomic(Ty, Volatile)) {
     LValue lvalue = LValue::MakeAddr(Addr, Ty,
                                      CharUnits::fromQuantity(Alignment),
                                      getContext(), TBAAInfo);
     return EmitAtomicLoad(lvalue, Loc).getScalarVal();
   }
 
   llvm::LoadInst *Load = Builder.CreateLoad(Addr);
   if (Volatile)
if (DstPtr->getElementType() != SrcTy) {
       llvm::Type *MemTy =
       llvm::PointerType::get(SrcTy, DstPtr->getAddressSpace());
       Addr = Builder.CreateBitCast(Addr, MemTy, "storetmp");
     }
   }
 
   Value = EmitToMemory(Value, Ty);
 
-  if (Ty->isAtomicType()) {
+  if (Ty->isAtomicType() ||
+      (!isInit && typeIsSuitableForInlineAtomic(Ty, Volatile))) {
     EmitAtomicStore(RValue::get(Value),
                     LValue::MakeAddr(Addr, Ty,
                                      CharUnits::fromQuantity(Alignment),
                                      getContext(), TBAAInfo),
                     isInit);
     return;
   }
 

cfe/trunk/lib/CodeGen/CGExprAgg.cpp

 
 /// EmitAggLoadOfLValue - Given an expression with aggregate type that
 /// represents a value lvalue, this method emits the address of the lvalue,
 /// then loads the result into DestPtr.
 void AggExprEmitter::EmitAggLoadOfLValue(const Expr *E) {
   LValue LV = CGF.EmitLValue(E);
 
   // If the type of the l-value is atomic, then do an atomic load.
-  if (LV.getType()->isAtomicType()) {
+  if (LV.getType()->isAtomicType() || CGF.LValueIsSuitableForInlineAtomic(LV)) {
     CGF.EmitAtomicLoad(LV, E->getExprLoc(), Dest);
     return;
   }
 
   EmitFinalDestCopy(E->getType(), LV);
 }
 
 /// \brief True if the given aggregate type requires special GC API calls.
if (isBlockVarRef(E->getLHS()) &&
     // Ensure that we have a destination, and evaluate the RHS into that.
     EnsureDest(E->getRHS()->getType());
     Visit(E->getRHS());
 
     // Now emit the LHS and copy into it.
     LValue LHS = CGF.EmitCheckedLValue(E->getLHS(), CodeGenFunction::TCK_Store);
 
     // That copy is an atomic copy if the LHS is atomic.
-    if (LHS.getType()->isAtomicType()) {
+    if (LHS.getType()->isAtomicType() ||
+        CGF.LValueIsSuitableForInlineAtomic(LHS)) {
       CGF.EmitAtomicStore(Dest.asRValue(), LHS, /*isInit*/ false);
       return;
     }
 
     EmitCopy(E->getLHS()->getType(),
              AggValueSlot::forLValue(LHS, AggValueSlot::IsDestructed,
                                      needsGC(E->getLHS()->getType()),
                                      AggValueSlot::IsAliased),
              Dest);
     return;
   }
   
   LValue LHS = CGF.EmitLValue(E->getLHS());
 
   // If we have an atomic type, evaluate into the destination and then
   // do an atomic copy.
-  if (LHS.getType()->isAtomicType()) {
+  if (LHS.getType()->isAtomicType() ||
+      CGF.LValueIsSuitableForInlineAtomic(LHS)) {
     EnsureDest(E->getRHS()->getType());
     Visit(E->getRHS());
     CGF.EmitAtomicStore(Dest.asRValue(), LHS, /*isInit*/ false);
     return;
   }
 
   // Codegen the RHS so that it stores directly into the LHS.
   AggValueSlot LHSSlot =

cfe/trunk/lib/CodeGen/CGExprComplex.cpp

ComplexPairTy ComplexExprEmitter::EmitLoadOfLValue(LValue lvalue,
   }
   return ComplexPairTy(Real, Imag);
 }
 
 /// EmitStoreOfComplex - Store the specified real/imag parts into the
 /// specified value pointer.
 void ComplexExprEmitter::EmitStoreOfComplex(ComplexPairTy Val, LValue lvalue,
                                             bool isInit) {
-  if (lvalue.getType()->isAtomicType())
+  if (lvalue.getType()->isAtomicType() ||
+      (!isInit && CGF.LValueIsSuitableForInlineAtomic(lvalue)))
     return CGF.EmitAtomicStore(RValue::getComplex(Val), lvalue, isInit);
 
   llvm::Value *Ptr = lvalue.getAddress();
   llvm::Value *RealPtr = Builder.CreateStructGEP(Ptr, 0, "real");
   llvm::Value *ImagPtr = Builder.CreateStructGEP(Ptr, 1, "imag");
   unsigned AlignR = lvalue.getAlignment().getQuantity();
   ASTContext &C = CGF.getContext();
   QualType ComplexTy = lvalue.getType();

cfe/trunk/lib/CodeGen/CGStmtOpenMP.cpp

 static void EmitOMPAtomicReadExpr(CodeGenFunction &CGF, bool IsSeqCst,
                                   const Expr *X, const Expr *V,
                                   SourceLocation Loc) {
   // v = x;
   assert(V->isLValue() && "V of 'omp atomic read' is not lvalue");
   assert(X->isLValue() && "X of 'omp atomic read' is not lvalue");
   LValue XLValue = CGF.EmitLValue(X);
   LValue VLValue = CGF.EmitLValue(V);
-  RValue Res = XLValue.isGlobalReg() ? CGF.EmitLoadOfLValue(XLValue, Loc)
-                                     : CGF.EmitAtomicLoad(XLValue, Loc);
+  RValue Res = XLValue.isGlobalReg()
+                   ? CGF.EmitLoadOfLValue(XLValue, Loc)
+                   : CGF.EmitAtomicLoad(XLValue, Loc,
+                                        IsSeqCst ? llvm::SequentiallyConsistent
+                                                 : llvm::Monotonic);
   // OpenMP, 2.12.6, atomic Construct
   // Any atomic construct with a seq_cst clause forces the atomically
   // performed operation to include an implicit flush operation without a
   // list.
   if (IsSeqCst)
     CGF.CGM.getOpenMPRuntime().EmitOMPFlush(CGF, llvm::None, Loc);
   switch (CGF.getEvaluationKind(V->getType())) {
   case TEK_Scalar:

cfe/trunk/lib/CodeGen/CodeGenFunction.h

public:
   /// that the address will be used to access the object.
   LValue EmitCheckedLValue(const Expr *E, TypeCheckKind TCK);
 
   RValue convertTempToRValue(llvm::Value *addr, QualType type,
                              SourceLocation Loc);
 
   void EmitAtomicInit(Expr *E, LValue lvalue);
 
+  bool LValueIsSuitableForInlineAtomic(LValue Src);
+  bool typeIsSuitableForInlineAtomic(QualType Ty, bool IsVolatile) const;
+
+  RValue EmitAtomicLoad(LValue LV, SourceLocation SL,
+                        AggValueSlot Slot = AggValueSlot::ignored());
+
   RValue EmitAtomicLoad(LValue lvalue, SourceLocation loc,
+                        llvm::AtomicOrdering AO, bool IsVolatile = false,
                         AggValueSlot slot = AggValueSlot::ignored());
 
   void EmitAtomicStore(RValue rvalue, LValue lvalue, bool isInit);
 
+  void EmitAtomicStore(RValue rvalue, LValue lvalue, llvm::AtomicOrdering AO,
+                       bool IsVolatile, bool isInit);
+
   std::pair<RValue, RValue> EmitAtomicCompareExchange(
       LValue Obj, RValue Expected, RValue Desired, SourceLocation Loc,
       llvm::AtomicOrdering Success = llvm::SequentiallyConsistent,
       llvm::AtomicOrdering Failure = llvm::SequentiallyConsistent,
       bool IsWeak = false, AggValueSlot Slot = AggValueSlot::ignored());
 
   /// EmitToMemory - Change a scalar value from its value
   /// representation to its in-memory representation.

cfe/trunk/lib/Driver/Tools.cpp

void Clang::AddClangCLArgs(const ArgList &Args, ArgStringList &CmdArgs) const {
   }
 
   // /EP should expand to -E -P.
   if (Args.hasArg(options::OPT__SLASH_EP)) {
     CmdArgs.push_back("-E");
     CmdArgs.push_back("-P");
   }
 
+  unsigned VolatileOptionID;
+  if (getToolChain().getTriple().getArch() == llvm::Triple::x86_64 ||
+      getToolChain().getTriple().getArch() == llvm::Triple::x86)
+    VolatileOptionID = options::OPT__SLASH_volatile_ms;
+  else
+    VolatileOptionID = options::OPT__SLASH_volatile_iso;
+
+  if (Arg *A = Args.getLastArg(options::OPT__SLASH_volatile_Group))
+    VolatileOptionID = A->getOption().getID();
+
+  if (VolatileOptionID == options::OPT__SLASH_volatile_ms)
+    CmdArgs.push_back("-fms-volatile");
+
   Arg *MostGeneralArg = Args.getLastArg(options::OPT__SLASH_vmg);
   Arg *BestCaseArg = Args.getLastArg(options::OPT__SLASH_vmb);
   if (MostGeneralArg && BestCaseArg)
     D.Diag(clang::diag::err_drv_argument_not_allowed_with)
         << MostGeneralArg->getAsString(Args) << BestCaseArg->getAsString(Args);
 
   if (MostGeneralArg) {
     Arg *SingleArg = Args.getLastArg(options::OPT__SLASH_vms);

cfe/trunk/lib/Frontend/CompilerInvocation.cpp

static bool ParseCodeGenArgs(CodeGenOptions &Opts, ArgList &Args, InputKind IK,
   Opts.UseInitArray = Args.hasArg(OPT_fuse_init_array);
 
   Opts.FunctionSections = Args.hasFlag(OPT_ffunction_sections,
                                        OPT_fno_function_sections, false);
   Opts.DataSections = Args.hasFlag(OPT_fdata_sections,
                                    OPT_fno_data_sections, false);
   Opts.MergeFunctions = Args.hasArg(OPT_fmerge_functions);
 
+  Opts.MSVolatile = Args.hasArg(OPT_fms_volatile);
+
   Opts.VectorizeBB = Args.hasArg(OPT_vectorize_slp_aggressive);
   Opts.VectorizeLoop = Args.hasArg(OPT_vectorize_loops);
   Opts.VectorizeSLP = Args.hasArg(OPT_vectorize_slp);
 
   Opts.MainFileName = Args.getLastArgValue(OPT_main_file_name);
   Opts.VerifyModule = !Args.hasArg(OPT_disable_llvm_verifier);
 
   Opts.DisableGCov = Args.hasArg(OPT_test_coverage);

cfe/trunk/test/CodeGen/ms-volatile.c

+// RUN: %clang_cc1 -triple i386-pc-win32 -emit-llvm -fms-volatile -o - < %s | FileCheck %s
+struct foo {
+  volatile int x;
+};
+struct bar {
+  int x;
+};
+typedef _Complex float __declspec(align(8)) baz;
+
+void test1(struct foo *p, struct foo *q) {
+  *p = *q;
+  // CHECK-LABEL: @test1
+  // CHECK: load atomic volatile {{.*}} acquire
+  // CHECK: store atomic volatile {{.*}}, {{.*}} release
+}
+void test2(volatile int *p, volatile int *q) {
+  *p = *q;
+  // CHECK-LABEL: @test2
+  // CHECK: load atomic volatile {{.*}} acquire
+  // CHECK: store atomic volatile {{.*}}, {{.*}} release
+}
+void test3(struct foo *p, struct foo *q) {
+  p->x = q->x;
+  // CHECK-LABEL: @test3
+  // CHECK: load atomic volatile {{.*}} acquire
+  // CHECK: store atomic volatile {{.*}}, {{.*}} release
+}
+void test4(volatile struct foo *p, volatile struct foo *q) {
+  p->x = q->x;
+  // CHECK-LABEL: @test4
+  // CHECK: load atomic volatile {{.*}} acquire
+  // CHECK: store atomic volatile {{.*}}, {{.*}} release
+}
+void test5(volatile struct foo *p, volatile struct foo *q) {
+  *p = *q;
+  // CHECK-LABEL: @test5
+  // CHECK: load atomic volatile {{.*}} acquire
+  // CHECK: store atomic volatile {{.*}}, {{.*}} release
+}
+void test6(struct bar *p, struct bar *q) {
+  *p = *q;
+  // CHECK-LABEL: @test6
+  // CHECK-NOT: load atomic volatile {{.*}}
+  // CHECK-NOT: store atomic volatile {{.*}}, {{.*}}
+}
+void test7(volatile struct bar *p, volatile struct bar *q) {
+  *p = *q;
+  // CHECK-LABEL: @test7
+  // CHECK: load atomic volatile {{.*}} acquire
+  // CHECK: store atomic volatile {{.*}}, {{.*}} release
+}
+void test8(volatile double *p, volatile double *q) {
+  *p = *q;
+  // CHECK-LABEL: @test8
+  // CHECK: load atomic volatile {{.*}} acquire
+  // CHECK: store atomic volatile {{.*}}, {{.*}} release
+}
+void test9(volatile baz *p, baz *q) {
+  *p = *q;
+  // CHECK-LABEL: @test9
+  // CHECK: store atomic volatile {{.*}}, {{.*}} release
+}

cfe/trunk/test/Driver/cl-options.c

 // VMV: "-fms-memptr-rep=virtual"
 
 // RUN: %clang_cl /vmg /vmb -### -- %s 2>&1 | FileCheck -check-prefix=VMB %s
 // VMB: '/vmg' not allowed with '/vmb'
 
 // RUN: %clang_cl /vmg /vmm /vms -### -- %s 2>&1 | FileCheck -check-prefix=VMX %s
 // VMX: '/vms' not allowed with '/vmm'
 
+// RUN: %clang_cl /volatile:iso -### -- %s 2>&1 | FileCheck -check-prefix=VOLATILE-ISO %s
+// VOLATILE-ISO-NOT: "-fms-volatile"
+
+// RUN: %clang_cl /volatile:ms -### -- %s 2>&1 | FileCheck -check-prefix=VOLATILE-MS %s
+// VOLATILE-MS: "-fms-volatile"
+
 // RUN: %clang_cl /W0 -### -- %s 2>&1 | FileCheck -check-prefix=W0 %s
 // W0: -w
 
 // RUN: %clang_cl /W1 -### -- %s 2>&1 | FileCheck -check-prefix=W1 %s
 // RUN: %clang_cl /W2 -### -- %s 2>&1 | FileCheck -check-prefix=W1 %s
 // RUN: %clang_cl /W3 -### -- %s 2>&1 | FileCheck -check-prefix=W1 %s
 // RUN: %clang_cl /W4 -### -- %s 2>&1 | FileCheck -check-prefix=W1 %s
 // RUN: %clang_cl /Wall -### -- %s 2>&1 | FileCheck -check-prefix=W1 %s

cfe/trunk/test/OpenMP/atomic_read_codegen.c

 // CHECK: store float
 #pragma omp atomic read
   cfv = usx;
 // CHECK: load atomic i64*
 // CHECK: store double
 // CHECK: store double
 #pragma omp atomic read
   cdv = llx;
-// CHECK: [[I128VAL:%.+]] = load atomic i128* bitcast (<4 x i32>* @{{.+}} to i128*) seq_cst
+// CHECK: [[I128VAL:%.+]] = load atomic i128* bitcast (<4 x i32>* @{{.+}} to i128*) monotonic
 // CHECK: [[I128PTR:%.+]] = bitcast <4 x i32>* [[LDTEMP:%.+]] to i128*
 // CHECK: store i128 [[I128VAL]], i128* [[I128PTR]]
 // CHECK: [[LD:%.+]] = load <4 x i32>* [[LDTEMP]]
 // CHECK: extractelement <4 x i32> [[LD]]
 // CHECK: store i8
 #pragma omp atomic read
   bv = int4x[0];
-// CHECK: [[LD:%.+]] = load atomic i32* bitcast (i8* getelementptr (i8* bitcast (%{{.+}}* @{{.+}} to i8*), i64 4) to i32*) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i32* bitcast (i8* getelementptr (i8* bitcast (%{{.+}}* @{{.+}} to i8*), i64 4) to i32*) monotonic
 // CHECK: store i32 [[LD]], i32* [[LDTEMP:%.+]]
 // CHECK: [[LD:%.+]] = load i32* [[LDTEMP]]
 // CHECK: [[SHL:%.+]] = shl i32 [[LD]], 1
 // CHECK: ashr i32 [[SHL]], 1
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx.a;
 // CHECK: [[LDTEMP_VOID_PTR:%.+]] = bitcast i32* [[LDTEMP:%.+]] to i8*
 // CHECK: call void @__atomic_load(i64 4, i8* getelementptr (i8* bitcast (%struct.BitFields_packed* @bfx_packed to i8*), i64 4), i8* [[LDTEMP_VOID_PTR]], i32 5)
 // CHECK: [[LD:%.+]] = load i32* [[LDTEMP]]
 // CHECK: [[SHL:%.+]] = shl i32 [[LD]], 1
 // CHECK: ashr i32 [[SHL]], 1
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx_packed.a;
-// CHECK: [[LD:%.+]] = load atomic i32* getelementptr inbounds (%struct.BitFields2* @bfx2, i32 0, i32 0) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i32* getelementptr inbounds (%struct.BitFields2* @bfx2, i32 0, i32 0) monotonic
 // CHECK: store i32 [[LD]], i32* [[LDTEMP:%.+]]
 // CHECK: [[LD:%.+]] = load i32* [[LDTEMP]]
 // CHECK: ashr i32 [[LD]], 31
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx2.a;
-// CHECK: [[LD:%.+]] = load atomic i8* getelementptr (i8* bitcast (%struct.BitFields2_packed* @bfx2_packed to i8*), i64 3) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i8* getelementptr (i8* bitcast (%struct.BitFields2_packed* @bfx2_packed to i8*), i64 3) monotonic
 // CHECK: store i8 [[LD]], i8* [[LDTEMP:%.+]]
 // CHECK: [[LD:%.+]] = load i8* [[LDTEMP]]
 // CHECK: ashr i8 [[LD]], 7
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx2_packed.a;
-// CHECK: [[LD:%.+]] = load atomic i32* getelementptr inbounds (%struct.BitFields3* @bfx3, i32 0, i32 0) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i32* getelementptr inbounds (%struct.BitFields3* @bfx3, i32 0, i32 0) monotonic
 // CHECK: store i32 [[LD]], i32* [[LDTEMP:%.+]]
 // CHECK: [[LD:%.+]] = load i32* [[LDTEMP]]
 // CHECK: [[SHL:%.+]] = shl i32 [[LD]], 7
 // CHECK: ashr i32 [[SHL]], 18
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx3.a;
 // CHECK: [[LDTEMP_VOID_PTR:%.+]] = bitcast i24* [[LDTEMP:%.+]] to i8*
 // CHECK: call void @__atomic_load(i64 3, i8* getelementptr (i8* bitcast (%struct.BitFields3_packed* @bfx3_packed to i8*), i64 1), i8* [[LDTEMP_VOID_PTR]], i32 5)
 // CHECK: [[LD:%.+]] = load i24* [[LDTEMP]]
 // CHECK: [[SHL:%.+]] = shl i24 [[LD]], 7
 // CHECK: [[ASHR:%.+]] = ashr i24 [[SHL]], 10
 // CHECK: sext i24 [[ASHR]] to i32
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx3_packed.a;
-// CHECK: [[LD:%.+]] = load atomic i64* bitcast (%struct.BitFields4* @bfx4 to i64*) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i64* bitcast (%struct.BitFields4* @bfx4 to i64*) monotonic
 // CHECK: store i64 [[LD]], i64* [[LDTEMP:%.+]]
 // CHECK: [[LD:%.+]] = load i64* [[LDTEMP]]
 // CHECK: [[SHL:%.+]] = shl i64 [[LD]], 47
 // CHECK: [[ASHR:%.+]] = ashr i64 [[SHL]], 63
 // CHECK: trunc i64 [[ASHR]] to i32
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx4.a;
-// CHECK: [[LD:%.+]] = load atomic i8* getelementptr inbounds (%struct.BitFields4_packed* @bfx4_packed, i32 0, i32 0, i64 2) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i8* getelementptr inbounds (%struct.BitFields4_packed* @bfx4_packed, i32 0, i32 0, i64 2) monotonic
 // CHECK: store i8 [[LD]], i8* [[LDTEMP:%.+]]
 // CHECK: [[LD:%.+]] = load i8* [[LDTEMP]]
 // CHECK: [[SHL:%.+]] = shl i8 [[LD]], 7
 // CHECK: [[ASHR:%.+]] = ashr i8 [[SHL]], 7
 // CHECK: sext i8 [[ASHR]] to i32
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx4_packed.a;
-// CHECK: [[LD:%.+]] = load atomic i64* bitcast (%struct.BitFields4* @bfx4 to i64*) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i64* bitcast (%struct.BitFields4* @bfx4 to i64*) monotonic
 // CHECK: store i64 [[LD]], i64* [[LDTEMP:%.+]]
 // CHECK: [[LD:%.+]] = load i64* [[LDTEMP]]
 // CHECK: [[SHL:%.+]] = shl i64 [[LD]], 40
 // CHECK: [[ASHR:%.+]] = ashr i64 [[SHL]], 57
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx4.b;
-// CHECK: [[LD:%.+]] = load atomic i8* getelementptr inbounds (%struct.BitFields4_packed* @bfx4_packed, i32 0, i32 0, i64 2) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i8* getelementptr inbounds (%struct.BitFields4_packed* @bfx4_packed, i32 0, i32 0, i64 2) monotonic
 // CHECK: store i8 [[LD]], i8* [[LDTEMP:%.+]]
 // CHECK: [[LD:%.+]] = load i8* [[LDTEMP]]
 // CHECK: [[ASHR:%.+]] = ashr i8 [[LD]], 1
 // CHECK: sext i8 [[ASHR]] to i64
 // CHECK: store x86_fp80
 #pragma omp atomic read
   ldv = bfx4_packed.b;
-// CHECK: [[LD:%.+]] = load atomic i32* bitcast (<2 x float>* @{{.+}} to i32*) seq_cst
+// CHECK: [[LD:%.+]] = load atomic i32* bitcast (<2 x float>* @{{.+}} to i32*) monotonic
 // CHECK: [[BITCAST:%.+]] = bitcast <2 x float>* [[LDTEMP:%.+]] to i32*
 // CHECK: store i32 [[LD]], i32* [[BITCAST]]
 // CHECK: [[LD:%.+]] = load <2 x float>* [[LDTEMP]]
 // CHECK: extractelement <2 x float> [[LD]]
 // CHECK: store i64
 #pragma omp atomic read
   ulv = float2x.x;
 // CHECK: call{{.*}} i{{[0-9]+}} @llvm.read_register
 // CHECK: call{{.*}} @__kmpc_flush(
 // CHECK: store double
 #pragma omp atomic read seq_cst
   dv = rix;
   return 0;
 }
 
 #endif