Bugfix for Codegen of atomic load/store ops.

[ABataev]

Summary

Currently clang fires assertions on x86-64 on atomic load/store operations for long double operands. Patch fixes codegen for such operations.

[rjmccall]

Thanks for taking this on. Please test all the basic atomic operations, not just loads and stores, and ensure that the extra bits take on some sensible behavior.

test/CodeGen/x86_64-atomic-long_double.c

7 ↗

(On Diff #16857)

You need to be more specific in this test. You should be able to match the alloca.

14 ↗

(On Diff #16857)

This isn't good enough; x86_fp80's store size doesn't fill its rounded-up size. So you also need to ensure that the extra bits have some consistent value — i.e. zero. In other words, there needs to be a memset of the alloca to zero before you can store the long double into it.

[ABataev]

John, thanks for the review! Also I need to add a test for 32 bit mode (long double has size 96 bits there) and check that in this mode we create a libcall.

test/CodeGen/x86_64-atomic-long_double.c

7 ↗

(On Diff #16857)

Ok, I'll improve test

14 ↗

(On Diff #16857)

Ok, I'll add additional checks

[ABataev]

Rework after review.
Improved test and fixed bugs in codegen.

[rjmccall]

Hmm. Using the optimized libcalls is a great idea, but it should be a separate patch. You also need to mention that you'd like to do this on cfe-dev first, since it does introduce new runtime dependencies, even if they're a documented part of the existing atomics runtime.

lib/CodeGen/CGAtomic.cpp

1061

Consider making this a method on the AtomicInfo.

1244

Inst->setVolatile(Obj.isVolatileQualified());
Inst->setWeak(IsWeak);

1252

This part should also be a method on the AtomicInfo, or at least its own helper function. convertIntToRValue?

We probably have the same code in the atomic-load path.

[ABataev]

John, Ok, I'll split this patch in 2 parts.

lib/CodeGen/CGAtomic.cpp

1061

Ok, done.

1244

Done

1252

I reworked it.

[ABataev]

Update after review.

[rjmccall]

Thanks for splitting the patch. Generally looks good. One comment:

lib/CodeGen/CGAtomic.cpp

985

Is there a good reason that this has to be conditional rather than just being a single convertIntToRValue method? It's okay to pass a ReturnValueSlot in.

[ABataev]

John, here is my answer

lib/CodeGen/CGAtomic.cpp

985

If some value cannot be bitcasted from Int to ValueType (from Int128 to fp80, for example), we have to create a temp. But we have to create different temps for AtomicLoad() and for AtomicCompareExchange() (here we must create a structure of 2 elements). And these temps must be created only if we were unable to directly cast from int to value type.

[rjmccall]

lib/CodeGen/CGAtomic.cpp

985

Wait, what? Why is EmitAtomicCompareExchange returning a first-class aggregate? Just have it return a std::pair of values. That will also let you return an RValue for the current value, which is probably more generally useful to its callers.

[ABataev]

lib/CodeGen/CGAtomic.cpp

985

Ok, will do

[ABataev]

Update after review

[rjmccall]

Looks great, thanks.

include/clang/Basic/TargetInfo.h

public:
   unsigned getLargeArrayAlign() const { return LargeArrayAlign; }
 
   /// \brief Return the maximum width lock-free atomic operation which will
   /// ever be supported for the given target
   unsigned getMaxAtomicPromoteWidth() const { return MaxAtomicPromoteWidth; }
   /// \brief Return the maximum width lock-free atomic operation which can be
   /// inlined given the supported features of the given target.
   unsigned getMaxAtomicInlineWidth() const { return MaxAtomicInlineWidth; }
+  /// \brief Returns true if the given target supports lock-free atomic
+  /// operations at the specified width and alignment.
+  virtual bool hasBuiltinAtomic(uint64_t AtomicSizeInBits,
+                                uint64_t AlignmentInBits) const {
+    return AtomicSizeInBits <= AlignmentInBits &&
+           AtomicSizeInBits <= getMaxAtomicInlineWidth() &&
+           (AtomicSizeInBits <= getCharWidth() ||
+            llvm::isPowerOf2_64(AtomicSizeInBits / getCharWidth()));
+  }
 
   /// \brief Return the maximum vector alignment supported for the given target.
   unsigned getMaxVectorAlign() const { return MaxVectorAlign; }
 
   /// \brief Return the size of intmax_t and uintmax_t for this target, in bits.
   unsigned getIntMaxTWidth() const {
     return getTypeWidth(IntMaxType);
   }

lib/CodeGen/CGAtomic.cpp

AtomicInfo(CodeGenFunction &CGF, LValue &lvalue) : CGF(CGF) {
       assert(ValueSizeInBits <= AtomicSizeInBits);
       assert(ValueAlignInBits <= AtomicAlignInBits);
 
       AtomicAlign = C.toCharUnitsFromBits(AtomicAlignInBits);
       ValueAlign = C.toCharUnitsFromBits(ValueAlignInBits);
       if (lvalue.getAlignment().isZero())
         lvalue.setAlignment(AtomicAlign);
 
-      UseLibcall =
-        (AtomicSizeInBits > uint64_t(C.toBits(lvalue.getAlignment())) ||
-         AtomicSizeInBits > C.getTargetInfo().getMaxAtomicInlineWidth());
+      UseLibcall = !C.getTargetInfo().hasBuiltinAtomic(
+          AtomicSizeInBits, C.toBits(lvalue.getAlignment()));
     }
 
     QualType getAtomicType() const { return AtomicTy; }
     QualType getValueType() const { return ValueTy; }
     CharUnits getAtomicAlignment() const { return AtomicAlign; }
     CharUnits getValueAlignment() const { return ValueAlign; }
     uint64_t getAtomicSizeInBits() const { return AtomicSizeInBits; }
-    uint64_t getValueSizeInBits() const { return AtomicSizeInBits; }
+    uint64_t getValueSizeInBits() const { return ValueSizeInBits; }
     TypeEvaluationKind getEvaluationKind() const { return EvaluationKind; }
     bool shouldUseLibcall() const { return UseLibcall; }
 
     /// Is the atomic size larger than the underlying value type?
     ///
     /// Note that the absence of padding does not mean that atomic
     /// objects are completely interchangeable with non-atomic
     /// objects: we might have promoted the alignment of a type
public:
     /// atomic operations.
     llvm::Value *emitCastToAtomicIntPointer(llvm::Value *addr) const;
 
     /// Turn an atomic-layout object into an r-value.
     RValue convertTempToRValue(llvm::Value *addr,
                                AggValueSlot resultSlot,
                                SourceLocation loc) const;
 
+    /// \brief Converts a rvalue to integer value.
+    llvm::Value *convertRValueToInt(RValue RVal) const;
+
+    llvm::Value *tryConvertIntToValue(llvm::Value *IntVal,
+                                      bool PerformSimpleCasts = true) const;
+
     /// Copy an atomic r-value into atomic-layout memory.
     void emitCopyIntoMemory(RValue rvalue, LValue lvalue) const;
 
     /// Project an l-value down to the value field.
     LValue projectValue(LValue lvalue) const {
       llvm::Value *addr = lvalue.getAddress();
       if (hasPadding())
         addr = CGF.Builder.CreateStructGEP(addr, 0);
RValue AtomicInfo::convertTempToRValue(llvm::Value *addr,
   if (hasPadding())
     addr = CGF.Builder.CreateStructGEP(addr, 0);
 
   // Otherwise, just convert the temporary to an r-value using the
   // normal conversion routine.
   return CGF.convertTempToRValue(addr, getValueType(), loc);
 }
 
+llvm::Value *AtomicInfo::tryConvertIntToValue(llvm::Value *IntVal,
+                                              bool PerformSimpleCasts) const {
+  // Try not to in some easy cases.
+  assert(IntVal->getType()->isIntegerTy() && "Expected integer value");
+  if (getEvaluationKind() == TEK_Scalar && !hasPadding()) {
+    auto *ValTy = CGF.ConvertTypeForMem(ValueTy);
+    if (ValTy->isIntegerTy()) {
+      assert(IntVal->getType() == ValTy && "Different integer types.");
+      return CGF.EmitFromMemory(IntVal, ValueTy);
+    }
+    if (PerformSimpleCasts) {
+      if (ValTy->isPointerTy())
+        return CGF.Builder.CreateIntToPtr(IntVal, ValTy);
+      else if (llvm::CastInst::isBitCastable(IntVal->getType(), ValTy))
+        return CGF.Builder.CreateBitCast(IntVal, ValTy);
+    }
+  }
+
+  return nullptr;
+}
+
 /// Emit a load from an l-value of atomic type.  Note that the r-value
 /// we produce is an r-value of the atomic *value* type.
 RValue CodeGenFunction::EmitAtomicLoad(LValue src, SourceLocation loc,
                                        AggValueSlot resultSlot) {
   AtomicInfo atomics(*this, src);
 
   // Check whether we should use a library call.
   if (atomics.shouldUseLibcall()) {
RValue CodeGenFunction::EmitAtomicLoad(LValue src, SourceLocation loc,
 
   // Other decoration.
   load->setAlignment(src.getAlignment().getQuantity());
   if (src.isVolatileQualified())
     load->setVolatile(true);
   if (src.getTBAAInfo())
     CGM.DecorateInstruction(load, src.getTBAAInfo());
 
-  // Okay, turn that back into the original value type.
-  QualType valueType = atomics.getValueType();
-  llvm::Value *result = load;
-
   // If we're ignoring an aggregate return, don't do anything.
   if (atomics.getEvaluationKind() == TEK_Aggregate && resultSlot.isIgnored())
     return RValue::getAggregate(nullptr, false);
 
-  // The easiest way to do this this is to go through memory, but we
-  // try not to in some easy cases.
-  if (atomics.getEvaluationKind() == TEK_Scalar && !atomics.hasPadding()) {
-    llvm::Type *resultTy = CGM.getTypes().ConvertTypeForMem(valueType);
-    if (isa<llvm::IntegerType>(resultTy)) {
-      assert(result->getType() == resultTy);
-      result = EmitFromMemory(result, valueType);
-    } else if (isa<llvm::PointerType>(resultTy)) {
-      result = Builder.CreateIntToPtr(result, resultTy);
-    } else {
-      result = Builder.CreateBitCast(result, resultTy);
-    }
-    return RValue::get(result);
-  }
+  // Okay, turn that back into the original value type.
+  if (auto *Result = atomics.tryConvertIntToValue(load))
+    return RValue::get(Result);
 
   // Create a temporary.  This needs to be big enough to hold the
   // atomic integer.
   llvm::Value *temp;
   bool tempIsVolatile = false;
   CharUnits tempAlignment;
   if (atomics.getEvaluationKind() == TEK_Aggregate) {
     assert(!resultSlot.isIgnored());
     temp = resultSlot.getAddr();
     tempAlignment = atomics.getValueAlignment();
     tempIsVolatile = resultSlot.isVolatile();
   } else {
     temp = CreateMemTemp(atomics.getAtomicType(), "atomic-load-temp");
     tempAlignment = atomics.getAtomicAlignment();
   }
 
   // Slam the integer into the temporary.
   llvm::Value *castTemp = atomics.emitCastToAtomicIntPointer(temp);
-  Builder.CreateAlignedStore(result, castTemp, tempAlignment.getQuantity())
+  Builder.CreateAlignedStore(load, castTemp, tempAlignment.getQuantity())
       ->setVolatile(tempIsVolatile);
 
   return atomics.convertTempToRValue(temp, resultSlot, loc);
 }
 
 
 
 /// Copy an r-value into memory as part of storing to an atomic type.
 /// This needs to create a bit-pattern suitable for atomic operations.
llvm::Value *AtomicInfo::materializeRValue(RValue rvalue) const {
 
   // Otherwise, make a temporary and materialize into it.
   llvm::Value *temp = CGF.CreateMemTemp(getAtomicType(), "atomic-store-temp");
   LValue tempLV = CGF.MakeAddrLValue(temp, getAtomicType(), getAtomicAlignment());
   emitCopyIntoMemory(rvalue, tempLV);
   return temp;
 }
 
+llvm::Value *AtomicInfo::convertRValueToInt(RValue RVal) const {
+  // If we've got a scalar value of the right size, try to avoid going
+  // through memory.
+  if (RVal.isScalar() && !hasPadding()) {
+    llvm::Value *Value = RVal.getScalarVal();
+    if (isa<llvm::IntegerType>(Value->getType()))
+      return Value;
+    else {
+      llvm::IntegerType *InputIntTy =
+          llvm::IntegerType::get(CGF.getLLVMContext(), getValueSizeInBits());
+      if (isa<llvm::PointerType>(Value->getType()))
+        return CGF.Builder.CreatePtrToInt(Value, InputIntTy);
+      else if (llvm::BitCastInst::isBitCastable(Value->getType(), InputIntTy))
+        return CGF.Builder.CreateBitCast(Value, InputIntTy);
+    }
+  }
+  // Otherwise, we need to go through memory.
+  // Put the r-value in memory.
+  llvm::Value *Addr = materializeRValue(RVal);
+
+  // Cast the temporary to the atomic int type and pull a value out.
+  Addr = emitCastToAtomicIntPointer(Addr);
+  return CGF.Builder.CreateAlignedLoad(Addr,
+                                       getAtomicAlignment().getQuantity());
+}
+
 /// Emit a store to an l-value of atomic type.
 ///
 /// Note that the r-value is expected to be an r-value *of the atomic
 /// type*; this means that for aggregate r-values, it should include
 /// storage for any padding that was necessary.
 void CodeGenFunction::EmitAtomicStore(RValue rvalue, LValue dest, bool isInit) {
   // If this is an aggregate r-value, it should agree in type except
   // maybe for address-space qualification.
if (atomics.shouldUseLibcall()) {
     args.add(RValue::get(llvm::ConstantInt::get(
                  IntTy, AtomicExpr::AO_ABI_memory_order_seq_cst)),
              getContext().IntTy);
     emitAtomicLibcall(*this, "__atomic_store", getContext().VoidTy, args);
     return;
   }
 
   // Okay, we're doing this natively.
-  llvm::Value *intValue;
-
-  // If we've got a scalar value of the right size, try to avoid going
-  // through memory.
-  if (rvalue.isScalar() && !atomics.hasPadding()) {
-    llvm::Value *value = rvalue.getScalarVal();
-    if (isa<llvm::IntegerType>(value->getType())) {
-      intValue = value;
-    } else {
-      llvm::IntegerType *inputIntTy =
-        llvm::IntegerType::get(getLLVMContext(), atomics.getValueSizeInBits());
-      if (isa<llvm::PointerType>(value->getType())) {
-        intValue = Builder.CreatePtrToInt(value, inputIntTy);
-      } else {
-        intValue = Builder.CreateBitCast(value, inputIntTy);
-      }
-    }
-
-  // Otherwise, we need to go through memory.
-  } else {
-    // Put the r-value in memory.
-    llvm::Value *addr = atomics.materializeRValue(rvalue);
-
-    // Cast the temporary to the atomic int type and pull a value out.
-    addr = atomics.emitCastToAtomicIntPointer(addr);
-    intValue = Builder.CreateAlignedLoad(addr,
-                                 atomics.getAtomicAlignment().getQuantity());
-  }
+  llvm::Value *intValue = atomics.convertRValueToInt(rvalue);
 
   // Do the atomic store.
   llvm::Value *addr = atomics.emitCastToAtomicIntPointer(dest.getAddress());
   llvm::StoreInst *store = Builder.CreateStore(intValue, addr);
 
   // Initializations don't need to be atomic.
   if (!isInit) store->setAtomic(llvm::SequentiallyConsistent);
 
   // Other decoration.
   store->setAlignment(dest.getAlignment().getQuantity());
   if (dest.isVolatileQualified())
     store->setVolatile(true);
   if (dest.getTBAAInfo())
     CGM.DecorateInstruction(store, dest.getTBAAInfo());
 }
 
+/// Emit a compare-and-exchange op for atomic type.
+///
+llvm::Value *CodeGenFunction::EmitAtomicCompareExchange(
+    LValue Obj, RValue Expected, RValue Desired, SourceLocation Loc,
+    llvm::AtomicOrdering Success, llvm::AtomicOrdering Failure, bool IsWeak) {
+  // If this is an aggregate r-value, it should agree in type except
+  // maybe for address-space qualification.
+  assert(!Expected.isAggregate() ||
+         Expected.getAggregateAddr()->getType()->getPointerElementType() ==
+             Obj.getAddress()->getType()->getPointerElementType());
+  assert(!Desired.isAggregate() ||
+         Desired.getAggregateAddr()->getType()->getPointerElementType() ==
+             Obj.getAddress()->getType()->getPointerElementType());
+  AtomicInfo Atomics(*this, Obj);
+
+  if (Failure >= Success)
+    // Don't assert on undefined behavior.
+    Failure = llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(Success);
+
+  QualType ValueType = Atomics.getValueType();
+  if (Obj.isVolatileQualified())
+    ValueType = getContext().getVolatileType(ValueType);
+  auto Alignment = Atomics.getValueAlignment();
+  auto *ResType = llvm::StructType::get(ConvertTypeForMem(ValueType),
+                                        Builder.getInt1Ty(), nullptr);
+  // Check whether we should use a library call.
+  if (Atomics.shouldUseLibcall()) {
+    auto *ExpectedAddr = Atomics.materializeRValue(Expected);
+    // Produce a source address.
+    auto *DesiredAddr = Atomics.materializeRValue(Desired);
+    // bool __atomic_compare_exchange (size_t size, void *obj, void *expected,
+    // void *desired, int success, int failure);
+    CallArgList Args;
+    Args.add(RValue::get(Atomics.getAtomicSizeValue()),
+             getContext().getSizeType());
+    Args.add(RValue::get(EmitCastToVoidPtr(Obj.getAddress())),
+             getContext().VoidPtrTy);
+    Args.add(RValue::get(EmitCastToVoidPtr(ExpectedAddr)),
+             getContext().VoidPtrTy);
+    Args.add(RValue::get(EmitCastToVoidPtr(DesiredAddr)),
+             getContext().VoidPtrTy);
+    Args.add(RValue::get(llvm::ConstantInt::get(IntTy, Success)),
+             getContext().IntTy);
+    Args.add(RValue::get(llvm::ConstantInt::get(IntTy, Failure)),
+             getContext().IntTy);
+    auto ResRVal = emitAtomicLibcall(*this, "__atomic_compare_exchange",
+                                     getContext().BoolTy, Args);
+    auto *Alloc = CreateTempAlloca(ResType, "atomic-cmpxchg-temp");
+    Alloc->setAlignment(Alignment.getQuantity());
+    EmitStoreThroughLValue(
+        EmitLoadOfLValue(LValue::MakeAddr(ExpectedAddr, Atomics.getAtomicType(),
+                                          Atomics.getAtomicAlignment(),
+                                          getContext()),
+                         Loc),
+        LValue::MakeAddr(Builder.CreateStructGEP(Alloc, /*Idx=*/0), ValueType,
+                         Alignment, getContext()));
+    Builder.CreateAlignedStore(ResRVal.getScalarVal(),
+                               Builder.CreateStructGEP(Alloc, /*Idx=*/1),
+                               getContext().getTypeAlign(getContext().BoolTy))
+        ->setVolatile(Obj.isVolatileQualified());
+    auto *Res = Builder.CreateAlignedLoad(Alloc, Alignment.getQuantity());
+    Res->setVolatile(Obj.isVolatileQualified());
+    return Res;
+  }
+
+  // If we've got a scalar value of the right size, try to avoid going
+  // through memory.
+  auto *ExpectedIntVal = Atomics.convertRValueToInt(Expected);
+  auto *DesiredIntVal = Atomics.convertRValueToInt(Desired);
+
+  // Do the atomic store.
+  auto *Addr = Atomics.emitCastToAtomicIntPointer(Obj.getAddress());
+  auto *Inst = Builder.CreateAtomicCmpXchg(Addr, ExpectedIntVal, DesiredIntVal,
+                                          Success, Failure);
+  // Other decoration.
+  Inst->setVolatile(Obj.isVolatileQualified());
+  Inst->setWeak(IsWeak);
+
+  // Okay, turn that back into the original value type.
+  auto *Result = Builder.CreateExtractValue(Inst, /*Idxs=*/0);
+
+  // The easiest way to do this this is to go through memory, but we
+  // try not to in some easy cases.
+  if (Atomics.tryConvertIntToValue(Result, /*PerformSimpleCasts=*/false))
+    return Inst;
+
+  // Create a temporary.  This needs to be big enough to hold the
+  // atomic integer.
+  auto *Alloc = CreateTempAlloca(ResType, "atomic-cmpxchg-temp");
+  Alloc->setAlignment(Alignment.getQuantity());
+
+  // Slam the integer into the temporary.
+  auto *CastTemp = Atomics.emitCastToAtomicIntPointer(
+      Builder.CreateStructGEP(Alloc, /*Idx=*/0));
+  Builder.CreateAlignedStore(Result, CastTemp, Alignment.getQuantity())
+      ->setVolatile(Obj.isVolatileQualified());
+  Builder.CreateAlignedStore(Builder.CreateExtractValue(Inst, /*Idxs=*/1),
+                             Builder.CreateStructGEP(Alloc, /*Idx=*/1),
+                             getContext().getTypeAlign(getContext().BoolTy))
+      ->setVolatile(Obj.isVolatileQualified());
+
+  auto *Res = Builder.CreateAlignedLoad(Alloc, Alignment.getQuantity());
+  Res->setVolatile(Obj.isVolatileQualified());
+  return Res;
+}
+
 void CodeGenFunction::EmitAtomicInit(Expr *init, LValue dest) {
   AtomicInfo atomics(*this, dest);
 
   switch (atomics.getEvaluationKind()) {
   case TEK_Scalar: {
     llvm::Value *value = EmitScalarExpr(init);
     atomics.emitCopyIntoMemory(RValue::get(value), dest);
     return;
   }
 

lib/CodeGen/CGExprScalar.cpp

// Objective-C pointer types.
     else
       value = Builder.CreateInBoundsGEP(value, sizeValue, "incdec.objptr");
     value = Builder.CreateBitCast(value, input->getType());
   }
 
   if (atomicPHI) {
     llvm::BasicBlock *opBB = Builder.GetInsertBlock();
     llvm::BasicBlock *contBB = CGF.createBasicBlock("atomic_cont", CGF.CurFn);
-    llvm::Value *pair = Builder.CreateAtomicCmpXchg(
-        LV.getAddress(), atomicPHI, CGF.EmitToMemory(value, type),
-        llvm::SequentiallyConsistent, llvm::SequentiallyConsistent);
+    llvm::Value *pair = CGF.EmitAtomicCompareExchange(
+                                LV, RValue::get(atomicPHI),
+                                RValue::get(CGF.EmitToMemory(value, type)),
+                                E->getExprLoc());
     llvm::Value *old = Builder.CreateExtractValue(pair, 0);
     llvm::Value *success = Builder.CreateExtractValue(pair, 1);
     atomicPHI->addIncoming(old, opBB);
     Builder.CreateCondBr(success, contBB, opBB);
     Builder.SetInsertPoint(contBB);
     return isPre ? value : input;
   }
 
LValue ScalarExprEmitter::EmitCompoundAssignLValue(
   Result = (this->*Func)(OpInfo);
 
   // Convert the result back to the LHS type.
   Result = EmitScalarConversion(Result, E->getComputationResultType(), LHSTy);
 
   if (atomicPHI) {
     llvm::BasicBlock *opBB = Builder.GetInsertBlock();
     llvm::BasicBlock *contBB = CGF.createBasicBlock("atomic_cont", CGF.CurFn);
-    llvm::Value *pair = Builder.CreateAtomicCmpXchg(
-        LHSLV.getAddress(), atomicPHI, CGF.EmitToMemory(Result, LHSTy),
-        llvm::SequentiallyConsistent, llvm::SequentiallyConsistent);
+    llvm::Value *pair = CGF.EmitAtomicCompareExchange(
+                                LHSLV, RValue::get(atomicPHI),
+                                RValue::get(CGF.EmitToMemory(Result, LHSTy)),
+                                E->getExprLoc());
     llvm::Value *old = Builder.CreateExtractValue(pair, 0);
     llvm::Value *success = Builder.CreateExtractValue(pair, 1);
     atomicPHI->addIncoming(old, opBB);
     Builder.CreateCondBr(success, contBB, opBB);
     Builder.SetInsertPoint(contBB);
     return LHSLV;
   }
 

lib/CodeGen/CodeGenFunction.h

#endif
 
   void EmitAtomicInit(Expr *E, LValue lvalue);
 
   RValue EmitAtomicLoad(LValue lvalue, SourceLocation loc,
                         AggValueSlot slot = AggValueSlot::ignored());
 
   void EmitAtomicStore(RValue rvalue, LValue lvalue, bool isInit);
 
+  llvm::Value *EmitAtomicCompareExchange(
+      LValue Obj, RValue Expected, RValue Desired, SourceLocation Loc,
+      llvm::AtomicOrdering Success = llvm::SequentiallyConsistent,
+      llvm::AtomicOrdering Failure = llvm::SequentiallyConsistent,
+      bool IsWeak = false);
+
   /// EmitToMemory - Change a scalar value from its value
   /// representation to its in-memory representation.
   llvm::Value *EmitToMemory(llvm::Value *Value, QualType Ty);
 
   /// EmitFromMemory - Change a scalar value from its memory
   /// representation to its value representation.
   llvm::Value *EmitFromMemory(llvm::Value *Value, QualType Ty);
 

test/CodeGen/c11atomics.c

void testinc(void)
   ++l;
   // CHECK: atomicrmw add i16* @s, i16 1 seq_cst
   // CHECK: add i16
   ++s;
 }
 // CHECK: testdec
 void testdec(void)
 {
-  // CHECK: cmpxchg i8* @b
+  // CHECK: call arm_aapcscc zeroext i1 @__atomic_compare_exchange(i32 1, i8* @b
   b--;
   // CHECK: atomicrmw sub i32* @i, i32 1 seq_cst
   i--;
   // CHECK: atomicrmw sub i64* @l, i64 1 seq_cst
   l--;
   // CHECK: atomicrmw sub i16* @s, i16 1 seq_cst
   s--;
-  // CHECK: cmpxchg i8* @b
+  // CHECK: call arm_aapcscc zeroext i1 @__atomic_compare_exchange(i32 1, i8* @b
   --b;
   // CHECK: atomicrmw sub i32* @i, i32 1 seq_cst
   // CHECK: sub i32
   --i;
   // CHECK: atomicrmw sub i64* @l, i64 1 seq_cst
   // CHECK: sub i64
   --l;
   // CHECK: atomicrmw sub i16* @s, i16 1 seq_cst
   // CHECK: sub i16
   --s;
 }
 // CHECK: testaddeq
 void testaddeq(void)
 {
-  // CHECK: cmpxchg i8* @b
+  // CHECK: call arm_aapcscc zeroext i1 @__atomic_compare_exchange(i32 1, i8* @b
   // CHECK: atomicrmw add i32* @i, i32 42 seq_cst
   // CHECK: atomicrmw add i64* @l, i64 42 seq_cst
   // CHECK: atomicrmw add i16* @s, i16 42 seq_cst
   b += 42;
   i += 42;
   l += 42;
   s += 42;
 }
 // CHECK: testsubeq
 void testsubeq(void)
 {
-  // CHECK: cmpxchg i8* @b
+  // CHECK: call arm_aapcscc zeroext i1 @__atomic_compare_exchange(i32 1, i8* @b
   // CHECK: atomicrmw sub i32* @i, i32 42 seq_cst
   // CHECK: atomicrmw sub i64* @l, i64 42 seq_cst
   // CHECK: atomicrmw sub i16* @s, i16 42 seq_cst
   b -= 42;
   i -= 42;
   l -= 42;
   s -= 42;
 }
 // CHECK: testxoreq
 void testxoreq(void)
 {
-  // CHECK: cmpxchg i8* @b
+  // CHECK: call arm_aapcscc zeroext i1 @__atomic_compare_exchange(i32 1, i8* @b
   // CHECK: atomicrmw xor i32* @i, i32 42 seq_cst
   // CHECK: atomicrmw xor i64* @l, i64 42 seq_cst
   // CHECK: atomicrmw xor i16* @s, i16 42 seq_cst
   b ^= 42;
   i ^= 42;
   l ^= 42;
   s ^= 42;
 }
 // CHECK: testoreq
 void testoreq(void)
 {
-  // CHECK: cmpxchg i8* @b
+  // CHECK: call arm_aapcscc zeroext i1 @__atomic_compare_exchange(i32 1, i8* @b
   // CHECK: atomicrmw or i32* @i, i32 42 seq_cst
   // CHECK: atomicrmw or i64* @l, i64 42 seq_cst
   // CHECK: atomicrmw or i16* @s, i16 42 seq_cst
   b |= 42;
   i |= 42;
   l |= 42;
   s |= 42;
 }
 // CHECK: testandeq
 void testandeq(void)
 {
-  // CHECK: cmpxchg i8* @b
+  // CHECK: call arm_aapcscc zeroext i1 @__atomic_compare_exchange(i32 1, i8* @b
   // CHECK: atomicrmw and i32* @i, i32 42 seq_cst
   // CHECK: atomicrmw and i64* @l, i64 42 seq_cst
   // CHECK: atomicrmw and i16* @s, i16 42 seq_cst
   b &= 42;
   i &= 42;
   l &= 42;
   s &= 42;
 }

test/CodeGen/x86-atomic-long_double.c

+// RUN: %clang_cc1 -triple x86_64-linux-gnu -target-cpu core2 %s -S -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 -triple i686-linux-gnu -target-cpu core2 %s -S -emit-llvm -o - | FileCheck -check-prefix=CHECK32 %s
+
+long double testinc(_Atomic long double *addr) {
+  // CHECK-LABEL: @testinc
+  // CHECK: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 8
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 8
+  // CHECK: [[INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VALUE:%.+]] = load atomic i128* [[INT_ADDR]] seq_cst, align 16
+  // CHECK: [[INT_LOAD_ADDR:%.+]] = bitcast x86_fp80* [[LD_ADDR:%.+]] to i128*
+  // CHECK: store i128 [[INT_VALUE]], i128* [[INT_LOAD_ADDR]], align 16
+  // CHECK: [[LD_VALUE:%.+]] = load x86_fp80* [[LD_ADDR]], align 16
+  // CHECK: br label %[[ATOMIC_OP:.+]]
+  // CHECK: [[ATOMIC_OP]]
+  // CHECK: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK: [[INC_VALUE:%.+]] = fadd x86_fp80 [[OLD_VALUE]],
+  // CHECK: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 16
+  // CHECK: [[OLD_INT_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i128*
+  // CHECK: [[OLD_INT:%.+]] = load i128* [[OLD_INT_ADDR]], align 16
+  // CHECK: [[NEW_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[NEW_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[INC_VALUE]], x86_fp80* [[NEW_VALUE_ADDR]], align 16
+  // CHECK: [[NEW_INT_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR]] to i128*
+  // CHECK: [[NEW_INT:%.+]] = load i128* [[NEW_INT_ADDR]], align 16
+  // CHECK: [[OBJ_INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[RES:%.+]] = cmpxchg i128* [[OBJ_INT_ADDR]], i128 [[OLD_INT]], i128 [[NEW_INT]] seq_cst seq_cst
+  // CHECK: [[OLD_VALUE:%.+]] = extractvalue { i128, i1 } [[RES]], 0
+  // CHECK: [[OLD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK: [[OLD_VALUE_RES_INT_PTR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_RES_PTR]] to i128*
+  // CHECK: store i128 [[OLD_VALUE]], i128* [[OLD_VALUE_RES_INT_PTR]], align 16
+  // CHECK: [[FAIL_SUCCESS:%.+]] = extractvalue { i128, i1 } [[RES]], 1
+  // CHECK: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK: store i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK: [[RES:%.+]] = load { x86_fp80, i1 }* [[RES_PTR]], align 16
+  // CHECK: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK: [[ATOMIC_CONT]]
+  // CHECK: ret x86_fp80 [[INC_VALUE]]
+  // CHECK32-LABEL: @testinc
+  // CHECK32: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 4
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 4
+  // CHECK32: [[VOID_PTR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[TEMP_LD_PTR:%.+]] = bitcast x86_fp80* [[TEMP_LD_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_PTR]], i8* [[TEMP_LD_PTR]], i32 5)
+  // CHECK32: [[LD_VALUE:%.+]] = load x86_fp80* [[TEMP_LD_ADDR]], align 4
+  // CHECK32: br label %[[ATOMIC_OP:.+]]
+  // CHECK32: [[ATOMIC_OP]]
+  // CHECK32: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK32: [[INC_VALUE:%.+]] = fadd x86_fp80 [[OLD_VALUE]],
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 4
+  // CHECK32: [[DESIRED_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[DESIRED_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[INC_VALUE]], x86_fp80* [[DESIRED_VALUE_ADDR]], align 4
+  // CHECK32: [[OBJ:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[EXPECTED:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[DESIRED:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR]] to i8*
+  // CHECK32: [[FAIL_SUCCESS:%.+]] = call zeroext i1 @__atomic_compare_exchange(i32 12, i8* [[OBJ]], i8* [[EXPECTED]], i8* [[DESIRED]], i32 7, i32 7)
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[OLD_VALUE_VOID_GET_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_GET_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[OLD_VALUE_VOID_ADDR]], i8* [[OLD_VALUE_VOID_GET_ADDR]], i32 5)
+  // CHECK32: [[OLD_VALUE:%.+]] = load x86_fp80* [[OLD_VALUE_GET_ADDR]], align 4
+  // CHECK32: [[LD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[LD_VALUE_RES_PTR]], align 4
+  // CHECK32: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK32: store i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK32: [[RES:%.+]] = load { x86_fp80, i1 }* [[RES_PTR]], align 4
+  // CHECK32: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK32: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK32: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK32: [[ATOMIC_CONT]]
+  // CHECK32: ret x86_fp80 [[INC_VALUE]]
+
+  return ++*addr;
+}
+
+long double testdec(_Atomic long double *addr) {
+  // CHECK-LABEL: @testdec
+  // CHECK: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 8
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 8
+  // CHECK: [[INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VALUE:%.+]] = load atomic i128* [[INT_ADDR]] seq_cst, align 16
+  // CHECK: [[INT_LOAD_ADDR:%.+]] = bitcast x86_fp80* [[LD_ADDR:%.+]] to i128*
+  // CHECK: store i128 [[INT_VALUE]], i128* [[INT_LOAD_ADDR]], align 16
+  // CHECK: [[ORIG_LD_VALUE:%.+]] = load x86_fp80* [[LD_ADDR]], align 16
+  // CHECK: br label %[[ATOMIC_OP:.+]]
+  // CHECK: [[ATOMIC_OP]]
+  // CHECK: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[ORIG_LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK: [[DEC_VALUE:%.+]] = fadd x86_fp80 [[OLD_VALUE]],
+  // CHECK: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 16
+  // CHECK: [[OLD_INT_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i128*
+  // CHECK: [[OLD_INT:%.+]] = load i128* [[OLD_INT_ADDR]], align 16
+  // CHECK: [[NEW_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[NEW_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[DEC_VALUE]], x86_fp80* [[NEW_VALUE_ADDR]], align 16
+  // CHECK: [[NEW_INT_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR]] to i128*
+  // CHECK: [[NEW_INT:%.+]] = load i128* [[NEW_INT_ADDR]], align 16
+  // CHECK: [[OBJ_INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[RES:%.+]] = cmpxchg i128* [[OBJ_INT_ADDR]], i128 [[OLD_INT]], i128 [[NEW_INT]] seq_cst seq_cst
+  // CHECK: [[OLD_VALUE:%.+]] = extractvalue { i128, i1 } [[RES]], 0
+  // CHECK: [[OLD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK: [[OLD_VALUE_RES_INT_PTR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_RES_PTR]] to i128*
+  // CHECK: store i128 [[OLD_VALUE]], i128* [[OLD_VALUE_RES_INT_PTR]], align 16
+  // CHECK: [[FAIL_SUCCESS:%.+]] = extractvalue { i128, i1 } [[RES]], 1
+  // CHECK: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK: store i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK: [[RES:%.+]] = load { x86_fp80, i1 }* [[RES_PTR]], align 16
+  // CHECK: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK: [[ATOMIC_CONT]]
+  // CHECK: ret x86_fp80 [[ORIG_LD_VALUE]]
+  // CHECK32-LABEL: @testdec
+  // CHECK32: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 4
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 4
+  // CHECK32: [[VOID_PTR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[TEMP_LD_PTR:%.+]] = bitcast x86_fp80* [[TEMP_LD_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_PTR]], i8* [[TEMP_LD_PTR]], i32 5)
+  // CHECK32: [[ORIG_LD_VALUE:%.+]] = load x86_fp80* [[TEMP_LD_ADDR]], align 4
+  // CHECK32: br label %[[ATOMIC_OP:.+]]
+  // CHECK32: [[ATOMIC_OP]]
+  // CHECK32: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[ORIG_LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK32: [[DEC_VALUE:%.+]] = fadd x86_fp80 [[OLD_VALUE]],
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 4
+  // CHECK32: [[DESIRED_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[DESIRED_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[DEC_VALUE]], x86_fp80* [[DESIRED_VALUE_ADDR]], align 4
+  // CHECK32: [[OBJ:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[EXPECTED:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[DESIRED:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR]] to i8*
+  // CHECK32: [[FAIL_SUCCESS:%.+]] = call zeroext i1 @__atomic_compare_exchange(i32 12, i8* [[OBJ]], i8* [[EXPECTED]], i8* [[DESIRED]], i32 7, i32 7)
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[OLD_VALUE_VOID_GET_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_GET_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[OLD_VALUE_VOID_ADDR]], i8* [[OLD_VALUE_VOID_GET_ADDR]], i32 5)
+  // CHECK32: [[OLD_VALUE:%.+]] = load x86_fp80* [[OLD_VALUE_GET_ADDR]], align 4
+  // CHECK32: [[LD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[LD_VALUE_RES_PTR]], align 4
+  // CHECK32: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK32: store i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK32: [[RES:%.+]] = load { x86_fp80, i1 }* [[RES_PTR]], align 4
+  // CHECK32: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK32: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK32: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK32: [[ATOMIC_CONT]]
+  // CHECK32: ret x86_fp80 [[ORIG_LD_VALUE]]
+
+  return (*addr)--;
+}
+
+long double testcompassign(_Atomic long double *addr) {
+  *addr -= 25;
+  // CHECK-LABEL: @testcompassign
+  // CHECK: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 8
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 8
+  // CHECK: [[INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VALUE:%.+]] = load atomic i128* [[INT_ADDR]] seq_cst, align 16
+  // CHECK: [[INT_LOAD_ADDR:%.+]] = bitcast x86_fp80* [[LD_ADDR:%.+]] to i128*
+  // CHECK: store i128 [[INT_VALUE]], i128* [[INT_LOAD_ADDR]], align 16
+  // CHECK: [[LD_VALUE:%.+]] = load x86_fp80* [[LD_ADDR]], align 16
+  // CHECK: br label %[[ATOMIC_OP:.+]]
+  // CHECK: [[ATOMIC_OP]]
+  // CHECK: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK: [[SUB_VALUE:%.+]] = fsub x86_fp80 [[OLD_VALUE]],
+  // CHECK: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 16
+  // CHECK: [[OLD_INT_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i128*
+  // CHECK: [[OLD_INT:%.+]] = load i128* [[OLD_INT_ADDR]], align 16
+  // CHECK: [[NEW_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[NEW_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[SUB_VALUE]], x86_fp80* [[NEW_VALUE_ADDR]], align 16
+  // CHECK: [[NEW_INT_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR]] to i128*
+  // CHECK: [[NEW_INT:%.+]] = load i128* [[NEW_INT_ADDR]], align 16
+  // CHECK: [[OBJ_INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[RES:%.+]] = cmpxchg i128* [[OBJ_INT_ADDR]], i128 [[OLD_INT]], i128 [[NEW_INT]] seq_cst seq_cst
+  // CHECK: [[OLD_VALUE:%.+]] = extractvalue { i128, i1 } [[RES]], 0
+  // CHECK: [[OLD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK: [[OLD_VALUE_RES_INT_PTR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_RES_PTR]] to i128*
+  // CHECK: store i128 [[OLD_VALUE]], i128* [[OLD_VALUE_RES_INT_PTR]], align 16
+  // CHECK: [[FAIL_SUCCESS:%.+]] = extractvalue { i128, i1 } [[RES]], 1
+  // CHECK: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK: store i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK: [[RES:%.+]] = load { x86_fp80, i1 }* [[RES_PTR]], align 16
+  // CHECK: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK: [[ATOMIC_CONT]]
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** %{{.+}}, align 8
+  // CHECK: [[ADDR_INT:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VAL:%.+]] = load atomic i128* [[ADDR_INT]] seq_cst, align 16
+  // CHECK: [[INT_LD_TEMP:%.+]] = bitcast x86_fp80* [[LD_TEMP:%.+]] to i128*
+  // CHECK: store i128 [[INT_VAL]], i128* [[INT_LD_TEMP:%.+]], align 16
+  // CHECK: [[RET_VAL:%.+]] = load x86_fp80* [[LD_TEMP]], align 16
+  // CHECK: ret x86_fp80 [[RET_VAL]]
+  // CHECK32-LABEL: @testcompassign
+  // CHECK32: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 4
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 4
+  // CHECK32: [[VOID_PTR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[TEMP_LD_PTR:%.+]] = bitcast x86_fp80* [[TEMP_LD_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_PTR]], i8* [[TEMP_LD_PTR]], i32 5)
+  // CHECK32: [[LD_VALUE:%.+]] = load x86_fp80* [[TEMP_LD_ADDR]], align 4
+  // CHECK32: br label %[[ATOMIC_OP:.+]]
+  // CHECK32: [[ATOMIC_OP]]
+  // CHECK32: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK32: [[INC_VALUE:%.+]] = fsub x86_fp80 [[OLD_VALUE]],
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 4
+  // CHECK32: [[DESIRED_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[DESIRED_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[INC_VALUE]], x86_fp80* [[DESIRED_VALUE_ADDR]], align 4
+  // CHECK32: [[OBJ:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[EXPECTED:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[DESIRED:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR]] to i8*
+  // CHECK32: [[FAIL_SUCCESS:%.+]] = call zeroext i1 @__atomic_compare_exchange(i32 12, i8* [[OBJ]], i8* [[EXPECTED]], i8* [[DESIRED]], i32 7, i32 7)
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[OLD_VALUE_VOID_GET_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_GET_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[OLD_VALUE_VOID_ADDR]], i8* [[OLD_VALUE_VOID_GET_ADDR]], i32 5)
+  // CHECK32: [[OLD_VALUE:%.+]] = load x86_fp80* [[OLD_VALUE_GET_ADDR]], align 4
+  // CHECK32: [[LD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[LD_VALUE_RES_PTR]], align 4
+  // CHECK32: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK32: store i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK32: [[RES:%.+]] = load { x86_fp80, i1 }* [[RES_PTR]], align 4
+  // CHECK32: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK32: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK32: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK32: [[ATOMIC_CONT]]
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** %{{.+}}, align 4
+  // CHECK32: [[VOID_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[VOID_GET_ADDR:%.+]] = bitcast x86_fp80* [[GET_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_ADDR]], i8* [[VOID_GET_ADDR]], i32 5)
+  // CHECK32: [[RET_VAL:%.+]] = load x86_fp80* [[GET_ADDR]], align 4
+  // CHECK32: ret x86_fp80 [[RET_VAL]]
+  return *addr;
+}
+
+long double testassign(_Atomic long double *addr) {
+  // CHECK-LABEL: @testassign
+  // CHECK: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 8
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 8
+  // CHECK: [[STORE_TEMP_VOID_PTR:%.+]] = bitcast x86_fp80* [[STORE_TEMP_PTR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[STORE_TEMP_VOID_PTR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 {{.+}}, x86_fp80* [[STORE_TEMP_PTR]], align 16
+  // CHECK: [[STORE_TEMP_INT_PTR:%.+]] = bitcast x86_fp80* [[STORE_TEMP_PTR]] to i128*
+  // CHECK: [[STORE_TEMP_INT:%.+]] = load i128* [[STORE_TEMP_INT_PTR]], align 16
+  // CHECK: [[ADDR_INT:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: store atomic i128 [[STORE_TEMP_INT]], i128* [[ADDR_INT]] seq_cst, align 16
+  // CHECK32-LABEL: @testassign
+  // CHECK32: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 4
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 4
+  // CHECK32: [[STORE_TEMP_VOID_PTR:%.+]] = bitcast x86_fp80* [[STORE_TEMP_PTR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[STORE_TEMP_VOID_PTR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 {{.+}}, x86_fp80* [[STORE_TEMP_PTR]], align 4
+  // CHECK32: [[ADDR_VOID:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[STORE_TEMP_VOID_PTR:%.+]] = bitcast x86_fp80* [[STORE_TEMP_PTR]] to i8*
+  // CHECK32: call void @__atomic_store(i32 12, i8* [[ADDR_VOID]], i8* [[STORE_TEMP_VOID_PTR]], i32 5)
+  *addr = 115;
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** %{{.+}}, align 8
+  // CHECK: [[ADDR_INT:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VAL:%.+]] = load atomic i128* [[ADDR_INT]] seq_cst, align 16
+  // CHECK: [[INT_LD_TEMP:%.+]] = bitcast x86_fp80* [[LD_TEMP:%.+]] to i128*
+  // CHECK: store i128 [[INT_VAL]], i128* [[INT_LD_TEMP:%.+]], align 16
+  // CHECK: [[RET_VAL:%.+]] = load x86_fp80* [[LD_TEMP]], align 16
+  // CHECK: ret x86_fp80 [[RET_VAL]]
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** %{{.+}}, align 4
+  // CHECK32: [[VOID_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[VOID_LD_TEMP:%.+]] = bitcast x86_fp80* [[LD_TEMP:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_ADDR]], i8* [[VOID_LD_TEMP]], i32 5)
+  // CHECK32: [[RET_VAL:%.+]] = load x86_fp80* [[LD_TEMP]], align 4
+  // CHECK32: ret x86_fp80 [[RET_VAL]]
+
+  return *addr;
+}
+
+long double test_volatile_inc(volatile _Atomic long double *addr) {
+  // CHECK-LABEL: @test_volatile_inc
+  // CHECK: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 8
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 8
+  // CHECK: [[INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VALUE:%.+]] = load atomic volatile i128* [[INT_ADDR]] seq_cst, align 16
+  // CHECK: [[INT_LOAD_ADDR:%.+]] = bitcast x86_fp80* [[LD_ADDR:%.+]] to i128*
+  // CHECK: store i128 [[INT_VALUE]], i128* [[INT_LOAD_ADDR]], align 16
+  // CHECK: [[LD_VALUE:%.+]] = load x86_fp80* [[LD_ADDR]], align 16
+  // CHECK: br label %[[ATOMIC_OP:.+]]
+  // CHECK: [[ATOMIC_OP]]
+  // CHECK: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK: [[INC_VALUE:%.+]] = fadd x86_fp80 [[OLD_VALUE]],
+  // CHECK: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 16
+  // CHECK: [[OLD_INT_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i128*
+  // CHECK: [[OLD_INT:%.+]] = load i128* [[OLD_INT_ADDR]], align 16
+  // CHECK: [[NEW_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[NEW_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[INC_VALUE]], x86_fp80* [[NEW_VALUE_ADDR]], align 16
+  // CHECK: [[NEW_INT_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR]] to i128*
+  // CHECK: [[NEW_INT:%.+]] = load i128* [[NEW_INT_ADDR]], align 16
+  // CHECK: [[OBJ_INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[RES:%.+]] = cmpxchg volatile i128* [[OBJ_INT_ADDR]], i128 [[OLD_INT]], i128 [[NEW_INT]] seq_cst seq_cst
+  // CHECK: [[OLD_VALUE:%.+]] = extractvalue { i128, i1 } [[RES]], 0
+  // CHECK: [[OLD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK: [[OLD_VALUE_RES_INT_PTR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_RES_PTR]] to i128*
+  // CHECK: store volatile i128 [[OLD_VALUE]], i128* [[OLD_VALUE_RES_INT_PTR]], align 16
+  // CHECK: [[FAIL_SUCCESS:%.+]] = extractvalue { i128, i1 } [[RES]], 1
+  // CHECK: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK: store volatile i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK: [[RES:%.+]] = load volatile { x86_fp80, i1 }* [[RES_PTR]], align 16
+  // CHECK: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK: [[ATOMIC_CONT]]
+  // CHECK: ret x86_fp80 [[INC_VALUE]]
+  // CHECK32-LABEL: @test_volatile_inc
+  // CHECK32: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 4
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 4
+  // CHECK32: [[VOID_PTR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[TEMP_LD_PTR:%.+]] = bitcast x86_fp80* [[TEMP_LD_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_PTR]], i8* [[TEMP_LD_PTR]], i32 5)
+  // CHECK32: [[LD_VALUE:%.+]] = load x86_fp80* [[TEMP_LD_ADDR]], align 4
+  // CHECK32: br label %[[ATOMIC_OP:.+]]
+  // CHECK32: [[ATOMIC_OP]]
+  // CHECK32: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK32: [[INC_VALUE:%.+]] = fadd x86_fp80 [[OLD_VALUE]],
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 4
+  // CHECK32: [[DESIRED_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[DESIRED_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[INC_VALUE]], x86_fp80* [[DESIRED_VALUE_ADDR]], align 4
+  // CHECK32: [[OBJ:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[EXPECTED:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[DESIRED:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR]] to i8*
+  // CHECK32: [[FAIL_SUCCESS:%.+]] = call zeroext i1 @__atomic_compare_exchange(i32 12, i8* [[OBJ]], i8* [[EXPECTED]], i8* [[DESIRED]], i32 7, i32 7)
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[OLD_VALUE_VOID_GET_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_GET_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[OLD_VALUE_VOID_ADDR]], i8* [[OLD_VALUE_VOID_GET_ADDR]], i32 5)
+  // CHECK32: [[OLD_VALUE:%.+]] = load x86_fp80* [[OLD_VALUE_GET_ADDR]], align 4
+  // CHECK32: [[LD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK32: store volatile x86_fp80 [[OLD_VALUE]], x86_fp80* [[LD_VALUE_RES_PTR]], align 4
+  // CHECK32: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK32: store volatile i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK32: [[RES:%.+]] = load volatile { x86_fp80, i1 }* [[RES_PTR]], align 4
+  // CHECK32: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK32: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK32: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK32: [[ATOMIC_CONT]]
+  // CHECK32: ret x86_fp80 [[INC_VALUE]]
+  return ++*addr;
+}
+
+long double test_volatile_dec(volatile _Atomic long double *addr) {
+  // CHECK-LABEL: @test_volatile_dec
+  // CHECK: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 8
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 8
+  // CHECK: [[INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VALUE:%.+]] = load atomic volatile i128* [[INT_ADDR]] seq_cst, align 16
+  // CHECK: [[INT_LOAD_ADDR:%.+]] = bitcast x86_fp80* [[LD_ADDR:%.+]] to i128*
+  // CHECK: store i128 [[INT_VALUE]], i128* [[INT_LOAD_ADDR]], align 16
+  // CHECK: [[ORIG_LD_VALUE:%.+]] = load x86_fp80* [[LD_ADDR]], align 16
+  // CHECK: br label %[[ATOMIC_OP:.+]]
+  // CHECK: [[ATOMIC_OP]]
+  // CHECK: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[ORIG_LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK: [[DEC_VALUE:%.+]] = fadd x86_fp80 [[OLD_VALUE]],
+  // CHECK: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 16
+  // CHECK: [[OLD_INT_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i128*
+  // CHECK: [[OLD_INT:%.+]] = load i128* [[OLD_INT_ADDR]], align 16
+  // CHECK: [[NEW_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[NEW_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[DEC_VALUE]], x86_fp80* [[NEW_VALUE_ADDR]], align 16
+  // CHECK: [[NEW_INT_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR]] to i128*
+  // CHECK: [[NEW_INT:%.+]] = load i128* [[NEW_INT_ADDR]], align 16
+  // CHECK: [[OBJ_INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[RES:%.+]] = cmpxchg volatile i128* [[OBJ_INT_ADDR]], i128 [[OLD_INT]], i128 [[NEW_INT]] seq_cst seq_cst
+  // CHECK: [[OLD_VALUE:%.+]] = extractvalue { i128, i1 } [[RES]], 0
+  // CHECK: [[OLD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK: [[OLD_VALUE_RES_INT_PTR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_RES_PTR]] to i128*
+  // CHECK: store volatile i128 [[OLD_VALUE]], i128* [[OLD_VALUE_RES_INT_PTR]], align 16
+  // CHECK: [[FAIL_SUCCESS:%.+]] = extractvalue { i128, i1 } [[RES]], 1
+  // CHECK: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK: store volatile i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK: [[RES:%.+]] = load volatile { x86_fp80, i1 }* [[RES_PTR]], align 16
+  // CHECK: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK: [[ATOMIC_CONT]]
+  // CHECK: ret x86_fp80 [[ORIG_LD_VALUE]]
+  // CHECK32-LABEL: @test_volatile_dec
+  // CHECK32: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 4
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 4
+  // CHECK32: [[VOID_PTR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[TEMP_LD_PTR:%.+]] = bitcast x86_fp80* [[TEMP_LD_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_PTR]], i8* [[TEMP_LD_PTR]], i32 5)
+  // CHECK32: [[ORIG_LD_VALUE:%.+]] = load x86_fp80* [[TEMP_LD_ADDR]], align 4
+  // CHECK32: br label %[[ATOMIC_OP:.+]]
+  // CHECK32: [[ATOMIC_OP]]
+  // CHECK32: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[ORIG_LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK32: [[DEC_VALUE:%.+]] = fadd x86_fp80 [[OLD_VALUE]],
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 4
+  // CHECK32: [[DESIRED_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[DESIRED_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[DEC_VALUE]], x86_fp80* [[DESIRED_VALUE_ADDR]], align 4
+  // CHECK32: [[OBJ:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[EXPECTED:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[DESIRED:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR]] to i8*
+  // CHECK32: [[FAIL_SUCCESS:%.+]] = call zeroext i1 @__atomic_compare_exchange(i32 12, i8* [[OBJ]], i8* [[EXPECTED]], i8* [[DESIRED]], i32 7, i32 7)
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[OLD_VALUE_VOID_GET_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_GET_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[OLD_VALUE_VOID_ADDR]], i8* [[OLD_VALUE_VOID_GET_ADDR]], i32 5)
+  // CHECK32: [[OLD_VALUE:%.+]] = load x86_fp80* [[OLD_VALUE_GET_ADDR]], align 4
+  // CHECK32: [[LD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK32: store volatile x86_fp80 [[OLD_VALUE]], x86_fp80* [[LD_VALUE_RES_PTR]], align 4
+  // CHECK32: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK32: store volatile i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK32: [[RES:%.+]] = load volatile { x86_fp80, i1 }* [[RES_PTR]], align 4
+  // CHECK32: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK32: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK32: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK32: [[ATOMIC_CONT]]
+  // CHECK32: ret x86_fp80 [[ORIG_LD_VALUE]]
+  return (*addr)--;
+}
+
+long double test_volatile_compassign(volatile _Atomic long double *addr) {
+  *addr -= 25;
+  // CHECK-LABEL: @test_volatile_compassign
+  // CHECK: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 8
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 8
+  // CHECK: [[INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VALUE:%.+]] = load atomic volatile i128* [[INT_ADDR]] seq_cst, align 16
+  // CHECK: [[INT_LOAD_ADDR:%.+]] = bitcast x86_fp80* [[LD_ADDR:%.+]] to i128*
+  // CHECK: store i128 [[INT_VALUE]], i128* [[INT_LOAD_ADDR]], align 16
+  // CHECK: [[LD_VALUE:%.+]] = load x86_fp80* [[LD_ADDR]], align 16
+  // CHECK: br label %[[ATOMIC_OP:.+]]
+  // CHECK: [[ATOMIC_OP]]
+  // CHECK: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK: [[SUB_VALUE:%.+]] = fsub x86_fp80 [[OLD_VALUE]],
+  // CHECK: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 16
+  // CHECK: [[OLD_INT_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i128*
+  // CHECK: [[OLD_INT:%.+]] = load i128* [[OLD_INT_ADDR]], align 16
+  // CHECK: [[NEW_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[NEW_VALUE_VOID_ADDR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 [[SUB_VALUE]], x86_fp80* [[NEW_VALUE_ADDR]], align 16
+  // CHECK: [[NEW_INT_ADDR:%.+]] = bitcast x86_fp80* [[NEW_VALUE_ADDR]] to i128*
+  // CHECK: [[NEW_INT:%.+]] = load i128* [[NEW_INT_ADDR]], align 16
+  // CHECK: [[OBJ_INT_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[RES:%.+]] = cmpxchg volatile i128* [[OBJ_INT_ADDR]], i128 [[OLD_INT]], i128 [[NEW_INT]] seq_cst seq_cst
+  // CHECK: [[OLD_VALUE:%.+]] = extractvalue { i128, i1 } [[RES]], 0
+  // CHECK: [[OLD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK: [[OLD_VALUE_RES_INT_PTR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_RES_PTR]] to i128*
+  // CHECK: store volatile i128 [[OLD_VALUE]], i128* [[OLD_VALUE_RES_INT_PTR]], align 16
+  // CHECK: [[FAIL_SUCCESS:%.+]] = extractvalue { i128, i1 } [[RES]], 1
+  // CHECK: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK: store volatile i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK: [[RES:%.+]] = load volatile { x86_fp80, i1 }* [[RES_PTR]], align 16
+  // CHECK: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK: [[ATOMIC_CONT]]
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** %{{.+}}, align 8
+  // CHECK: [[ADDR_INT:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VAL:%.+]] = load atomic volatile i128* [[ADDR_INT]] seq_cst, align 16
+  // CHECK: [[INT_LD_TEMP:%.+]] = bitcast x86_fp80* [[LD_TEMP:%.+]] to i128*
+  // CHECK: store i128 [[INT_VAL]], i128* [[INT_LD_TEMP:%.+]], align 16
+  // CHECK: [[RET_VAL:%.+]] = load x86_fp80* [[LD_TEMP]], align 16
+  // CHECK32-LABEL: @test_volatile_compassign
+  // CHECK32: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 4
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 4
+  // CHECK32: [[VOID_PTR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[TEMP_LD_PTR:%.+]] = bitcast x86_fp80* [[TEMP_LD_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_PTR]], i8* [[TEMP_LD_PTR]], i32 5)
+  // CHECK32: [[LD_VALUE:%.+]] = load x86_fp80* [[TEMP_LD_ADDR]], align 4
+  // CHECK32: br label %[[ATOMIC_OP:.+]]
+  // CHECK32: [[ATOMIC_OP]]
+  // CHECK32: [[OLD_VALUE:%.+]] = phi x86_fp80 [ [[LD_VALUE]], %{{.+}} ], [ [[LD_VALUE:%.+]], %[[ATOMIC_OP]] ]
+  // CHECK32: [[INC_VALUE:%.+]] = fsub x86_fp80 [[OLD_VALUE]],
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[OLD_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[OLD_VALUE]], x86_fp80* [[OLD_VALUE_ADDR]], align 4
+  // CHECK32: [[DESIRED_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[DESIRED_VALUE_VOID_ADDR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 [[INC_VALUE]], x86_fp80* [[DESIRED_VALUE_ADDR]], align 4
+  // CHECK32: [[OBJ:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[EXPECTED:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[DESIRED:%.+]] = bitcast x86_fp80* [[DESIRED_VALUE_ADDR]] to i8*
+  // CHECK32: [[FAIL_SUCCESS:%.+]] = call zeroext i1 @__atomic_compare_exchange(i32 12, i8* [[OBJ]], i8* [[EXPECTED]], i8* [[DESIRED]], i32 7, i32 7)
+  // CHECK32: [[OLD_VALUE_VOID_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_ADDR]] to i8*
+  // CHECK32: [[OLD_VALUE_VOID_GET_ADDR:%.+]] = bitcast x86_fp80* [[OLD_VALUE_GET_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[OLD_VALUE_VOID_ADDR]], i8* [[OLD_VALUE_VOID_GET_ADDR]], i32 5)
+  // CHECK32: [[OLD_VALUE:%.+]] = load x86_fp80* [[OLD_VALUE_GET_ADDR]], align 4
+  // CHECK32: [[LD_VALUE_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR:%.+]], i32 0, i32 0
+  // CHECK32: store volatile x86_fp80 [[OLD_VALUE]], x86_fp80* [[LD_VALUE_RES_PTR]], align 4
+  // CHECK32: [[FAIL_SUCCESS_RES_PTR:%.+]] = getelementptr inbounds { x86_fp80, i1 }* [[RES_PTR]], i32 0, i32 1
+  // CHECK32: store volatile i1 [[FAIL_SUCCESS]], i1* [[FAIL_SUCCESS_RES_PTR]], align 8
+  // CHECK32: [[RES:%.+]] = load volatile { x86_fp80, i1 }* [[RES_PTR]], align 4
+  // CHECK32: [[LD_VALUE]] = extractvalue { x86_fp80, i1 } [[RES]], 0
+  // CHECK32: [[FAIL_SUCCESS_RES:%.+]] = extractvalue { x86_fp80, i1 } [[RES]], 1
+  // CHECK32: br i1 [[FAIL_SUCCESS_RES]], label %[[ATOMIC_CONT:.+]], label %[[ATOMIC_OP]]
+  // CHECK32: [[ATOMIC_CONT]]
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** %{{.+}}, align 4
+  // CHECK32: [[VOID_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[VOID_GET_ADDR:%.+]] = bitcast x86_fp80* [[GET_ADDR:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_ADDR]], i8* [[VOID_GET_ADDR]], i32 5)
+  // CHECK32: [[RET_VAL:%.+]] = load x86_fp80* [[GET_ADDR]], align 4
+  // CHECK32: ret x86_fp80 [[RET_VAL]]
+  return *addr;
+}
+
+long double test_volatile_assign(volatile _Atomic long double *addr) {
+  // CHECK-LABEL: @test_volatile_assign
+  // CHECK: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 8
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 8
+  // CHECK: [[STORE_TEMP_VOID_PTR:%.+]] = bitcast x86_fp80* [[STORE_TEMP_PTR:%.+]] to i8*
+  // CHECK: call void @llvm.memset.p0i8.i64(i8* [[STORE_TEMP_VOID_PTR]], i8 0, i64 16, i32 16, i1 false)
+  // CHECK: store x86_fp80 {{.+}}, x86_fp80* [[STORE_TEMP_PTR]], align 16
+  // CHECK: [[STORE_TEMP_INT_PTR:%.+]] = bitcast x86_fp80* [[STORE_TEMP_PTR]] to i128*
+  // CHECK: [[STORE_TEMP_INT:%.+]] = load i128* [[STORE_TEMP_INT_PTR]], align 16
+  // CHECK: [[ADDR_INT:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: store atomic volatile i128 [[STORE_TEMP_INT]], i128* [[ADDR_INT]] seq_cst, align 16
+  // CHECK32-LABEL: @test_volatile_assign
+  // CHECK32: store x86_fp80* %{{.+}}, x86_fp80** [[ADDR_ADDR:%.+]], align 4
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** [[ADDR_ADDR]], align 4
+  // CHECK32: [[STORE_TEMP_VOID_PTR:%.+]] = bitcast x86_fp80* [[STORE_TEMP_PTR:%.+]] to i8*
+  // CHECK32: call void @llvm.memset.p0i8.i64(i8* [[STORE_TEMP_VOID_PTR]], i8 0, i64 12, i32 4, i1 false)
+  // CHECK32: store x86_fp80 {{.+}}, x86_fp80* [[STORE_TEMP_PTR]], align 4
+  // CHECK32: [[ADDR_VOID:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[STORE_TEMP_VOID_PTR:%.+]] = bitcast x86_fp80* [[STORE_TEMP_PTR]] to i8*
+  // CHECK32: call void @__atomic_store(i32 12, i8* [[ADDR_VOID]], i8* [[STORE_TEMP_VOID_PTR]], i32 5)
+  *addr = 115;
+  // CHECK: [[ADDR:%.+]] = load x86_fp80** %{{.+}}, align 8
+  // CHECK: [[ADDR_INT:%.+]] = bitcast x86_fp80* [[ADDR]] to i128*
+  // CHECK: [[INT_VAL:%.+]] = load atomic volatile i128* [[ADDR_INT]] seq_cst, align 16
+  // CHECK: [[INT_LD_TEMP:%.+]] = bitcast x86_fp80* [[LD_TEMP:%.+]] to i128*
+  // CHECK: store i128 [[INT_VAL]], i128* [[INT_LD_TEMP:%.+]], align 16
+  // CHECK: [[RET_VAL:%.+]] = load x86_fp80* [[LD_TEMP]], align 16
+  // CHECK: ret x86_fp80 [[RET_VAL]]
+  // CHECK32: [[ADDR:%.+]] = load x86_fp80** %{{.+}}, align 4
+  // CHECK32: [[VOID_ADDR:%.+]] = bitcast x86_fp80* [[ADDR]] to i8*
+  // CHECK32: [[VOID_LD_TEMP:%.+]] = bitcast x86_fp80* [[LD_TEMP:%.+]] to i8*
+  // CHECK32: call void @__atomic_load(i32 12, i8* [[VOID_ADDR]], i8* [[VOID_LD_TEMP]], i32 5)
+  // CHECK32: [[RET_VAL:%.+]] = load x86_fp80* [[LD_TEMP]], align 4
+  // CHECK32: ret x86_fp80 [[RET_VAL]]
+
+  return *addr;
+}