[PowerPC] Support 16-byte lock free atomics on pwr8 and up

[lkail]

Summary

Make 16-byte atomic type aligned to 16-byte on PPC64, thus consistent with GCC. Also enable inlining 16-byte atomics on non-AIX targets on PPC64.

[jsji]

clang/test/CodeGen/PowerPC/atomic-alignment.c

7

How about AIX pwr8?

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

126

So we enable it by default , *EVEN* on AIX?

[MaskRay]

clang/lib/Basic/Targets/PPC.h

449

Many isOSLinux() usage may be really isOSBinFormatELF() (to help FreeBSD, etc). Can you check whether that's the case?

[lkail]

@dim @pkubaj We are modifying layout of 16-byte atomic type on Linux to be consistent with GCC, I don't know if it also applies to freebsd as @MaskRay pointed out.
To be more specific, on Linux, for arch level supporting 16-byte lock free atomics, 16-byte atomic type should be properly aligned.

#include <stdatomic.h>

int printf(const char *, ...);

typedef struct A {
  char x[16];
} A;

typedef struct B {
  char q;
  _Atomic(A) a;
} B;

int main(void) {
  _Atomic(A) *p = 0;
  printf("aligned: %d\n", __builtin_offsetof(B, a) % 16 == 0);
#if __clang__
  printf("lock free (size built-in): %d\n", __c11_atomic_is_lock_free(sizeof(*p)));
#endif
  printf("lock free (type query using pointer): %d\n", atomic_is_lock_free(p));
}

Current clang gives (-mcpu=power8)

aligned: 0
lock free (size built-in): 1
lock free (type query using pointer): 1

GCC gives

aligned: 1
lock free (type query using pointer): 1

This patch also modifies the query of __atomic_always_lock_free(16, 0) to return true for supported arch level.

[lkail]

Removed alter of -ppc-quadword-atomcis in backend to decouple from frontend.

[efriedma]

clang/lib/Basic/Targets/PPC.h

451

MaxAtomicPromoteWidth should not depend on whether quadword-atomics is present, only the target OS. It determines the layout of _Atomic(__int128_t).

(MaxAtomicInlineWidth is allowed to adjust as necessary.)

clang/lib/Driver/ToolChains/Clang.cpp

2140 ↗

(On Diff #418142)

Please avoid using "-mllvm" options like this; among other issues, it doesn't work with LTO. If we need to indicate this, please use a function attribute in the IR.

(Yes, there are other cases where we translate flags to -mllvm options, but it's not something you should copy.)

[lkail]

Thanks @efriedma for pointing out. Updated guard code. Removed -mllvm while adjusting backend to reflect ABI change.

[efriedma]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

1329

You probably want an "else" here to call setMaxAtomicSizeInBitsSupported(64) or something like that. But you can leave that to a followup patch if you want.

[hubert.reinterpretcast]

clang/test/CodeGen/PowerPC/atomic-alignment.c

45

Although it is a change that affects layout compatibility with the short history of 16-byte atomic types compiled with LLVM-based compilers on AIX, having this 16-byte atomic type aligned also on AIX for PPC64 would be reasonable. The consequences of not doing so is that operations on some types would be lock-free only on ELF-based platforms.

[hubert.reinterpretcast]

clang/test/CodeGen/PowerPC/atomic-alignment.c

34

Just noting that GCC increases the alignment even for ppc32:

typedef struct A8 { char x[8]; } A8;
typedef struct A16 { char x[16]; } A16;
extern char q8[_Alignof(_Atomic(A8))], q8[8]; // okay for GCC targeting ppc32
extern char q16[_Alignof(_Atomic(A16))], q16[16]; // okay for GCC targeting ppc32

Apparently, the change for i686 in GCC occurred with version 11.
https://godbolt.org/z/fTTGoqWW1

[hubert.reinterpretcast]

clang/lib/Basic/Targets/PPC.h

448

I believe this should be presented more as this override being implemented currently only for ELF targets. The current presentation seems to imply more design intent for non-ELF targets than there is consensus for.

For example:

if (!getTriple().isOSBinFormatELF())
  return PPCTargetInfo::setMaxAtomicWidth();

clang/test/Sema/atomic-ops.c

13

This will need a separate patch to cover ppc32 (likely with AIX).

[lkail]

Make 16-byte atomic type aligned to 16-byte on PPC64.

[lkail]

clang/lib/Basic/Targets/PPC.h

448

It looks a chance to adjust according to arch level to me(Considering we finally will make xcoff and elf targets consistent here). If you have strong opinion on this, I'll make a change here.

clang/test/CodeGen/PowerPC/atomic-alignment.c

34

I'll post another patch to address ppc32 issue.

[hubert.reinterpretcast]

clang/lib/Basic/Targets/PPC.h

448

This function currently modifies two different things. One that should be arch-level-sensitive (and, iiuc, is already so) and one that should not be arch-level-sensitive.

So, the isOSBinFormatELF checks are here mainly to scope the patch (temporarily).

I only suggested to use the call to the base class as a shorthand for "nothing special that this function is doing". The associated comment should read something like "Restrict 16-byte atomic changes to ELF targets for now. TODO: Consider other targets as well."

At the very least (for this patch), I do not believe the isOSBinFormatELF should be checked twice in terms of control flow.

[hubert.reinterpretcast]

clang/lib/Basic/Targets/PPC.h

448

The new version of the change already addresses this; thanks.

[hubert.reinterpretcast]

clang/lib/Basic/Targets/PPC.h

444

Clarify that the statement is not for PPC64 in general.

446

Clarify that the support up to 8 bytes is for "baseline" PPC64 (i.e., non-baseline implementations may support lock-free inline 16-byte atomic operations).

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

18044

Can we have a comment here to explain the isOSBinFormatELF check?

[hubert.reinterpretcast]

I am not sure that the choice of isOSBinFormatELF to (afaik) primarily scope this change from affecting AIX (where we know the library calls are not implemented to be lock-free yet) is better than alternative where the condition is for little-endian mode or specifically for not AIX.

[lkail]

In D122377#3428533, @hubert.reinterpretcast wrote:

I am not sure that the choice of isOSBinFormatELF to (afaik) primarily scope this change from affecting AIX (where we know the library calls are not implemented to be lock-free yet) is better than alternative where the condition is for little-endian mode or specifically for not AIX.

I agree with we should exclude AIX explicitly(use !isOSAIX() rather than isOSBinFormatELF()) for current situation where AIX's support is on progress as we have known. What is unknown at present is the impact on freebsd.

[lkail]

Address comments.

[adalava]

clang/lib/Basic/Targets/PPC.h

440

I don't understand what DataLayout does mean, but I'm wondering why FreeBSD doesn't append this as well. Is it expected to be AIX() and Linux only?

[hubert.reinterpretcast]

clang/test/CodeGen/PowerPC/atomic-alignment.c

14

I was recently informed that, instead of using expected-no-diagnostics and -verify, using -Werror achieves the same with less processing cost. Sorry for the churn.

Noting also that expected-no-diagnostics only needs to be added once the the file.

clang/test/CodeGen/PowerPC/quadword-atomics.c

6

Can we add an AIX run line for pwr7? It should match the PPC64 check lines.

43–45

The pointer cast is avoidable; let's avoid it.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

18042

Typo fix; grammar fix.

llvm/test/CodeGen/PowerPC/atomics-i128.ll

17

I think this case deserves an extra comment to explain the -ppc-quadword-atomics.

74

What library is this expected to provide this symbol?

[lkail]

clang/lib/Basic/Targets/PPC.h

440

As maskray has pointed out, usually using isOSBinFormatELF rather than isOSLinux can cover freebsd as well. This snippet of code might be beyond this patch, we might post another one to address this issue.

[lkail]

llvm/test/CodeGen/PowerPC/atomics-i128.ll

74

As far as I know, should be none on AIX. This issue is fixed in https://reviews.llvm.org/D122868.

[lkail]

Address comments.

[hubert.reinterpretcast]

LGTM with minor comments.

clang/test/CodeGen/PowerPC/atomic-alignment.c

1–12

Use -Werror in place of -verify with no diagnostics.

clang/test/CodeGen/PowerPC/quadword-atomics.c

2–7

Same comment as for the other file.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

18042

Minor nit: Use semicolon.

llvm/test/CodeGen/PowerPC/atomics-i128.ll

127

I have verified that the registers in the pairs are used correctly for this case. I've skimmed the other cases for the instructions applied to the loaded value (or, for non-inline cases, the functions called).

I did not check that the set-up for the calls, etc.

I also haven't looked into the memory barrier instruction usage (but that should be common with other widths).

clang/lib/Basic/Targets/PPC.h

class LLVM_LIBRARY_VISIBILITY PPCTargetInfo : public TargetInfo {
   bool PairedVectorMemops = false;
   bool HasP10Vector = false;
   bool HasPCRelativeMemops = false;
   bool HasPrefixInstrs = false;
   bool IsISA2_06 = false;
   bool IsISA2_07 = false;
   bool IsISA3_0 = false;
   bool IsISA3_1 = false;
+  bool HasQuadwordAtomics = false;
 
 protected:
   std::string ABI;
 
 public:
   PPCTargetInfo(const llvm::Triple &Triple, const TargetOptions &)
       : TargetInfo(Triple) {
     SuitableAlign = 128;
PPC64TargetInfo(const llvm::Triple &Triple, const TargetOptions &Opts)
     }
 
     if (Triple.isOSFreeBSD() || Triple.isOSOpenBSD() || Triple.isMusl()) {
       LongDoubleWidth = LongDoubleAlign = 64;
       LongDoubleFormat = &llvm::APFloat::IEEEdouble();
     }
 
     if (Triple.isOSAIX() || Triple.isOSLinux())
       DataLayout += "-S128-v256:256:256-v512:512:512";
     resetDataLayout(DataLayout);
 
     // PPC64 supports atomics up to 8 bytes.
     MaxAtomicPromoteWidth = MaxAtomicInlineWidth = 64;
   }
 
+  void setMaxAtomicWidth() override {
+    // For layout on ELF targets, we support up to 16 bytes.
+    if (getTriple().isOSBinFormatELF())
+      MaxAtomicPromoteWidth = 128;
+
+    // For power8 and up, backend is able to inline 16-byte atomic lock free
+    // code.
+    if (getTriple().isOSBinFormatELF() && hasFeature("quadword-atomics"))
+      MaxAtomicInlineWidth = 128;
+  }
+
   BuiltinVaListKind getBuiltinVaListKind() const override {
     return TargetInfo::CharPtrBuiltinVaList;
   }
 
   // PPC64 Linux-specific ABI options.
   bool setABI(const std::string &Name) override {
     if (Name == "elfv1" || Name == "elfv2") {
       ABI = Name;

clang/lib/Basic/Targets/PPC.cpp

for (const auto &Feature : Features) {
     } else if (Feature == "+isa-v206-instructions") {
       IsISA2_06 = true;
     } else if (Feature == "+isa-v207-instructions") {
       IsISA2_07 = true;
     } else if (Feature == "+isa-v30-instructions") {
       IsISA3_0 = true;
     } else if (Feature == "+isa-v31-instructions") {
       IsISA3_1 = true;
+    } else if (Feature == "+quadword-atomics") {
+      HasQuadwordAtomics = true;
     }
     // TODO: Finish this list and add an assert that we've handled them
     // all.
   }
 
   return true;
 }
 
Features["isa-v207-instructions"] = llvm::StringSwitch<bool>(CPU)
                                           .Case("ppc64le", true)
                                           .Case("pwr9", true)
                                           .Case("pwr8", true)
                                           .Default(false);
 
   Features["isa-v30-instructions"] =
       llvm::StringSwitch<bool>(CPU).Case("pwr9", true).Default(false);
 
+  Features["quadword-atomics"] =
+      getTriple().isArch64Bit() && llvm::StringSwitch<bool>(CPU)
+                                       .Case("pwr9", true)
+                                       .Case("pwr8", true)
+                                       .Default(false);
+
   // Power10 includes all the same features as Power9 plus any features specific
   // to the Power10 core.
   if (CPU == "pwr10" || CPU == "power10") {
     initFeatureMap(Features, Diags, "pwr9", FeaturesVec);
     addP10SpecificFeatures(Features);
   }
 
   // Future CPU should include all of the features of Power 10 as well as any
return llvm::StringSwitch<bool>(Feature)
       .Case("spe", HasSPE)
       .Case("mma", HasMMA)
       .Case("rop-protect", HasROPProtect)
       .Case("privileged", HasPrivileged)
       .Case("isa-v206-instructions", IsISA2_06)
       .Case("isa-v207-instructions", IsISA2_07)
       .Case("isa-v30-instructions", IsISA3_0)
       .Case("isa-v31-instructions", IsISA3_1)
+      .Case("quadword-atomics", HasQuadwordAtomics)
       .Default(false);
 }
 
 void PPCTargetInfo::setFeatureEnabled(llvm::StringMap<bool> &Features,
                                       StringRef Name, bool Enabled) const {
   if (Enabled) {
     if (Name == "efpu2")
       Features["spe"] = true;

clang/test/CodeGen/PowerPC/atomic-alignment.c

 // RUN: %clang_cc1 -verify -triple powerpc-unknown-unknown -emit-llvm -o - %s | \
 // RUN:   FileCheck %s --check-prefixes=PPC,PPC32
 // RUN: %clang_cc1 -verify -triple powerpc64le-unknown-linux -emit-llvm -o - %s | \
 // RUN:   FileCheck %s --check-prefixes=PPC,PPC64
+// RUN: %clang_cc1 -verify -triple powerpc64le-unknown-linux -emit-llvm -o - %s \
+// RUN:   -target-cpu pwr8 | FileCheck %s --check-prefixes=PPC,PPC64
 // RUN: %clang_cc1 -verify -triple powerpc64-unknown-aix -emit-llvm -o - %s | \
-// RUN:   FileCheck %s --check-prefixes=PPC,PPC64
+// RUN:   FileCheck %s --check-prefixes=PPC,AIX64
+// RUN: %clang_cc1 -verify -triple powerpc64-unknown-aix -emit-llvm -o - %s \
+// RUN:   -target-cpu pwr8 | FileCheck %s --check-prefixes=PPC,AIX64
+
 
 // PPC: @c = global i8 0, align 1{{$}}
 _Atomic(char) c; // expected-no-diagnostics
 
 // PPC: @s = global i16 0, align 2{{$}}
 _Atomic(short) s; // expected-no-diagnostics
 
 // PPC: @i = global i32 0, align 4{{$}}
 _Atomic(int) i; // expected-no-diagnostics
 
 // PPC32: @l = global i32 0, align 4{{$}}
 // PPC64: @l = global i64 0, align 8{{$}}
+// AIX64: @l = global i64 0, align 8{{$}}
 _Atomic(long) l; // expected-no-diagnostics
 
 // PPC: @ll = global i64 0, align 8{{$}}
 _Atomic(long long) ll; // expected-no-diagnostics
 
 typedef struct {
   char x[8];
 } O;
 
 // PPC32: @o = global %struct.O zeroinitializer, align 1{{$}}
 // PPC64: @o = global %struct.O zeroinitializer, align 8{{$}}
+// AIX64: @o = global %struct.O zeroinitializer, align 8{{$}}
 _Atomic(O) o; // expected-no-diagnostics
 
 typedef struct {
   char x[16];
 } Q;
 
-// PPC: @q = global %struct.Q zeroinitializer, align 1{{$}}
+// PPC32: @q = global %struct.Q zeroinitializer, align 1{{$}}
+// PPC64: @q = global %struct.Q zeroinitializer, align 16{{$}}
+// AIX64: @q = global %struct.Q zeroinitializer, align 1{{$}}
 _Atomic(Q) q; // expected-no-diagnostics

clang/test/CodeGen/PowerPC/quadword-atomics.c

+// RUN: %clang_cc1 -verify -Wno-atomic-alignment -triple powerpc64le-linux-gnu \
+// RUN:   -target-cpu pwr8 -emit-llvm -o - %s | FileCheck %s --check-prefix=PPC64-PWR8
+// RUN: %clang_cc1 -verify -Wno-atomic-alignment -triple powerpc64le-linux-gnu \
+// RUN:   -emit-llvm -o - %s | FileCheck %s --check-prefix=PPC64
+
+typedef struct {
+  char x[16];
+} Q;
+
+typedef _Atomic(Q) AtomicQ;
+
+typedef __int128_t int128_t;
+
+// PPC64-PWR8-LABEL: @test_load(
+// PPC64-PWR8:    [[TMP3:%.*]] = load atomic i128, i128* [[TMP1:%.*]] acquire, align 16
+//
+// PPC64-LABEL: @test_load(
+// PPC64:    call void @__atomic_load(i64 noundef 16, i8* noundef [[TMP3:%.*]], i8* noundef [[TMP4:%.*]], i32 noundef signext 2)
+//
+Q test_load(AtomicQ *ptr) {
+  // expected-no-diagnostics
+  return __c11_atomic_load(ptr, __ATOMIC_ACQUIRE);
+}
+
+// PPC64-PWR8-LABEL: @test_store(
+// PPC64-PWR8:    store atomic i128 [[TMP6:%.*]], i128* [[TMP4:%.*]] release, align 16
+//
+// PPC64-LABEL: @test_store(
+// PPC64:    call void @__atomic_store(i64 noundef 16, i8* noundef [[TMP6:%.*]], i8* noundef [[TMP7:%.*]], i32 noundef signext 3)
+//
+void test_store(Q val, AtomicQ *ptr) {
+  // expected-no-diagnostics
+  __c11_atomic_store(ptr, val, __ATOMIC_RELEASE);
+}
+
+// PPC64-PWR8-LABEL: @test_add(
+// PPC64-PWR8:    [[TMP3:%.*]] = atomicrmw add i128* [[TMP0:%.*]], i128 [[TMP2:%.*]] monotonic, align 16
+//
+// PPC64-LABEL: @test_add(
+// PPC64:    [[CALL:%.*]] = call i128 @__atomic_fetch_add_16(i8* noundef [[TMP2:%.*]], i128 noundef [[TMP3:%.*]], i32 noundef signext 0)
+//
+void test_add(int128_t *ptr, int128_t x) {
+  // expected-no-diagnostics
+  __c11_atomic_fetch_add((_Atomic(int128_t) *)ptr, x, __ATOMIC_RELAXED);
+}
+
+// PPC64-PWR8-LABEL: @test_xchg(
+// PPC64-PWR8:    [[TMP8:%.*]] = atomicrmw xchg i128* [[TMP4:%.*]], i128 [[TMP7:%.*]] seq_cst, align 16
+//
+// PPC64-LABEL: @test_xchg(
+// PPC64:    call void @__atomic_exchange(i64 noundef 16, i8* noundef [[TMP7:%.*]], i8* noundef [[TMP8:%.*]], i8* noundef [[TMP9:%.*]], i32 noundef signext 5)
+//
+Q test_xchg(AtomicQ *ptr, Q new) {
+  // expected-no-diagnostics
+  return __c11_atomic_exchange(ptr, new, __ATOMIC_SEQ_CST);
+}
+
+// PPC64-PWR8-LABEL: @test_cmpxchg(
+// PPC64-PWR8:    [[TMP10:%.*]] = cmpxchg i128* [[TMP5:%.*]], i128 [[TMP8:%.*]], i128 [[TMP9:%.*]] seq_cst monotonic, align 16
+//
+// PPC64-LABEL: @test_cmpxchg(
+// PPC64:    [[CALL:%.*]] = call zeroext i1 @__atomic_compare_exchange(i64 noundef 16, i8* noundef [[TMP8:%.*]], i8* noundef [[TMP9:%.*]], i8* noundef [[TMP10:%.*]], i32 noundef signext 5, i32 noundef signext 0)
+//
+int test_cmpxchg(AtomicQ *ptr, Q *cmp, Q new) {
+  // expected-no-diagnostics
+  return __c11_atomic_compare_exchange_strong(ptr, cmp, new, __ATOMIC_SEQ_CST, __ATOMIC_RELAXED);
+}
+
+// PPC64-PWR8-LABEL: @test_cmpxchg_weak(
+// PPC64-PWR8:    [[TMP10:%.*]] = cmpxchg weak i128* [[TMP5:%.*]], i128 [[TMP8:%.*]], i128 [[TMP9:%.*]] seq_cst monotonic, align 16
+//
+// PPC64-LABEL: @test_cmpxchg_weak(
+// PPC64:    [[CALL:%.*]] = call zeroext i1 @__atomic_compare_exchange(i64 noundef 16, i8* noundef [[TMP8:%.*]], i8* noundef [[TMP9:%.*]], i8* noundef [[TMP10:%.*]], i32 noundef signext 5, i32 noundef signext 0)
+//
+int test_cmpxchg_weak(AtomicQ *ptr, Q *cmp, Q new) {
+  // expected-no-diagnostics
+  return __c11_atomic_compare_exchange_weak(ptr, cmp, new, __ATOMIC_SEQ_CST, __ATOMIC_RELAXED);
+}
+
+// PPC64-PWR8-LABEL: @is_lock_free(
+// PPC64-PWR8:    ret i32 1
+//
+// PPC64-LABEL: @is_lock_free(
+// PPC64:    [[CALL:%.*]] = call zeroext i1 @__atomic_is_lock_free(i64 noundef 16, i8* noundef null)
+//
+int is_lock_free() {
+  AtomicQ q;
+ // expected-no-diagnostics
+  return __c11_atomic_is_lock_free(sizeof(q));
+}

clang/test/Sema/atomic-ops.c

 // RUN: %clang_cc1 %s -verify=expected,fp80,noi128 -fgnuc-version=4.2.1 -ffreestanding \
 // RUN:   -fsyntax-only -triple=i686-linux-gnu -std=c11
 // RUN: %clang_cc1 %s -verify=expected,noi128 -fgnuc-version=4.2.1 -ffreestanding \
 // RUN:   -fsyntax-only -triple=i686-linux-android -std=c11
 // RUN: %clang_cc1 %s -verify -fgnuc-version=4.2.1 -ffreestanding \
 // RUN:   -fsyntax-only -triple=powerpc64-linux-gnu -std=c11
 // RUN: %clang_cc1 %s -verify -fgnuc-version=4.2.1 -ffreestanding \
 // RUN:   -fsyntax-only -triple=powerpc64-linux-gnu -std=c11 \
 // RUN:   -target-cpu pwr7
 // RUN: %clang_cc1 %s -verify -fgnuc-version=4.2.1 -ffreestanding \
 // RUN:   -fsyntax-only -triple=powerpc64le-linux-gnu -std=c11 \
-// RUN:   -target-cpu pwr8
+// RUN:   -target-cpu pwr8 -DPPC64_PWR8
 
 // Basic parsing/Sema tests for __c11_atomic_*
 
 #include <stdatomic.h>
 
 struct S { char c[3]; };
 
 _Static_assert(__GCC_ATOMIC_BOOL_LOCK_FREE == 2, "");
 _Static_assert(__GCC_ATOMIC_BOOL_LOCK_FREE == __CLANG_ATOMIC_BOOL_LOCK_FREE, "");

 _Static_assert(__GCC_ATOMIC_POINTER_LOCK_FREE == 2, "");
 _Static_assert(__GCC_ATOMIC_POINTER_LOCK_FREE == __CLANG_ATOMIC_POINTER_LOCK_FREE, "");
 
 _Static_assert(__c11_atomic_is_lock_free(1), "");
 _Static_assert(__c11_atomic_is_lock_free(2), "");
 _Static_assert(__c11_atomic_is_lock_free(3), ""); // expected-error {{not an integral constant expression}}
 _Static_assert(__c11_atomic_is_lock_free(4), "");
 _Static_assert(__c11_atomic_is_lock_free(8), "");
+#ifndef PPC64_PWR8
 _Static_assert(__c11_atomic_is_lock_free(16), ""); // expected-error {{not an integral constant expression}}
+#else
+_Static_assert(__c11_atomic_is_lock_free(16), ""); // expected-no-error
+#endif
 _Static_assert(__c11_atomic_is_lock_free(17), ""); // expected-error {{not an integral constant expression}}
 
 _Static_assert(__atomic_is_lock_free(1, 0), "");
 _Static_assert(__atomic_is_lock_free(2, 0), "");
 _Static_assert(__atomic_is_lock_free(3, 0), ""); // expected-error {{not an integral constant expression}}
 _Static_assert(__atomic_is_lock_free(4, 0), "");
 _Static_assert(__atomic_is_lock_free(8, 0), "");
+#ifndef PPC64_PWR8
 _Static_assert(__atomic_is_lock_free(16, 0), ""); // expected-error {{not an integral constant expression}}
+#else
+_Static_assert(__atomic_is_lock_free(16, 0), ""); // expected-no-error
+#endif
 _Static_assert(__atomic_is_lock_free(17, 0), ""); // expected-error {{not an integral constant expression}}
 
 _Static_assert(atomic_is_lock_free((atomic_char*)0), "");
 _Static_assert(atomic_is_lock_free((atomic_short*)0), "");
 _Static_assert(atomic_is_lock_free((atomic_int*)0), "");
 _Static_assert(atomic_is_lock_free((atomic_long*)0), "");
+#ifndef PPC64_PWR8
 // noi128-error@+1 {{__int128 is not supported on this target}}
 _Static_assert(atomic_is_lock_free((_Atomic(__int128)*)0), ""); // expected-error {{not an integral constant expression}}
+#else
+_Static_assert(atomic_is_lock_free((_Atomic(__int128)*)0), ""); // expected-no-error
+#endif
 _Static_assert(atomic_is_lock_free(0 + (atomic_char*)0), "");
 
 char i8;
 short i16;
 int i32;
 int __attribute__((vector_size(8))) i64;
 struct Incomplete *incomplete; // expected-note {{forward declaration of 'struct Incomplete'}}
 
 _Static_assert(__atomic_is_lock_free(1, &i8), "");
 _Static_assert(__atomic_is_lock_free(1, &i64), "");
 _Static_assert(__atomic_is_lock_free(2, &i8), ""); // expected-error {{not an integral constant expression}}
 _Static_assert(__atomic_is_lock_free(2, &i16), "");
 _Static_assert(__atomic_is_lock_free(2, &i64), "");
 _Static_assert(__atomic_is_lock_free(4, &i16), ""); // expected-error {{not an integral constant expression}}
 _Static_assert(__atomic_is_lock_free(4, &i32), "");
 _Static_assert(__atomic_is_lock_free(4, &i64), "");
 _Static_assert(__atomic_is_lock_free(8, &i32), ""); // expected-error {{not an integral constant expression}}
 _Static_assert(__atomic_is_lock_free(8, &i64), "");
 
 _Static_assert(__atomic_always_lock_free(1, 0), "");
 _Static_assert(__atomic_always_lock_free(2, 0), "");
 _Static_assert(!__atomic_always_lock_free(3, 0), "");
 _Static_assert(__atomic_always_lock_free(4, 0), "");
 _Static_assert(__atomic_always_lock_free(8, 0), "");
+#ifndef PPC64_PWR8
 _Static_assert(!__atomic_always_lock_free(16, 0), "");
+#else
+_Static_assert(__atomic_always_lock_free(16, 0), "");
+#endif
 _Static_assert(!__atomic_always_lock_free(17, 0), "");
 
 _Static_assert(__atomic_always_lock_free(1, incomplete), "");
 _Static_assert(!__atomic_always_lock_free(2, incomplete), "");
 _Static_assert(!__atomic_always_lock_free(4, incomplete), "");
 
 _Static_assert(__atomic_always_lock_free(1, &i8), "");
 _Static_assert(__atomic_always_lock_free(1, &i64), "");

llvm/lib/Target/PowerPC/PPCISelLowering.h

bool shouldInsertFencesForAtomic(const Instruction *I) const override {
       return true;
     }
 
     Instruction *emitLeadingFence(IRBuilderBase &Builder, Instruction *Inst,
                                   AtomicOrdering Ord) const override;
     Instruction *emitTrailingFence(IRBuilderBase &Builder, Instruction *Inst,
                                    AtomicOrdering Ord) const override;
 
+    bool shouldInlineQuadwordAtomics() const;
+
     TargetLowering::AtomicExpansionKind
     shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
 
     TargetLowering::AtomicExpansionKind
     shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const override;
 
     Value *emitMaskedAtomicRMWIntrinsic(IRBuilderBase &Builder,
                                         AtomicRMWInst *AI, Value *AlignedAddr,

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

 static cl::opt<bool> DisableInnermostLoopAlign32("disable-ppc-innermost-loop-align32",
 cl::desc("don't always align innermost loop to 32 bytes on ppc"), cl::Hidden);
 
 static cl::opt<bool> UseAbsoluteJumpTables("ppc-use-absolute-jumptables",
 cl::desc("use absolute jump tables on ppc"), cl::Hidden);
 
 static cl::opt<bool> EnableQuadwordAtomics(
     "ppc-quadword-atomics",
     cl::desc("enable quadword lock-free atomic operations"), cl::init(false),
     cl::Hidden);
 
 static cl::opt<bool>
     DisablePerfectShuffle("ppc-disable-perfect-shuffle",
                           cl::desc("disable vector permute decomposition"),
                           cl::init(true), cl::Hidden);
 
 STATISTIC(NumTailCalls, "Number of tail calls");

 
   setOperationAction(ISD::READCYCLECOUNTER, MVT::i64, isPPC64 ? Legal : Custom);
 
   if (!isPPC64) {
     setOperationAction(ISD::ATOMIC_LOAD,  MVT::i64, Expand);
     setOperationAction(ISD::ATOMIC_STORE, MVT::i64, Expand);
   }
 
-  if (EnableQuadwordAtomics && Subtarget.hasQuadwordAtomics()) {
+  if (shouldInlineQuadwordAtomics()) {
     setMaxAtomicSizeInBitsSupported(128);
     setOperationAction(ISD::ATOMIC_LOAD, MVT::i128, Custom);
     setOperationAction(ISD::ATOMIC_STORE, MVT::i128, Custom);
     setOperationAction(ISD::INTRINSIC_VOID, MVT::i128, Custom);
   }
 
   setBooleanContents(ZeroOrOneBooleanContent);
 
   if (Subtarget.hasAltivec()) {
     // Altivec instructions set fields to all zeros or all ones.
     setBooleanVectorContents(ZeroOrNegativeOneBooleanContent);
   }
 
CCAssignFn *PPCTargetLowering::ccAssignFnForCall(CallingConv::ID CC,
   switch (CC) {
   case CallingConv::Cold:
     return (Return ? RetCC_PPC_Cold : CC_PPC64_ELF_FIS);
   default:
     return CC_PPC64_ELF_FIS;
   }
 }
 
+bool PPCTargetLowering::shouldInlineQuadwordAtomics() const {
+  return (EnableQuadwordAtomics ||
+          Subtarget.getTargetTriple().isOSBinFormatELF()) &&
+         Subtarget.hasQuadwordAtomics();
+}
+
 TargetLowering::AtomicExpansionKind
 PPCTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
   unsigned Size = AI->getType()->getPrimitiveSizeInBits();
-  if (EnableQuadwordAtomics && Subtarget.hasQuadwordAtomics() && Size == 128)
+  if (shouldInlineQuadwordAtomics() && Size == 128)
     return AtomicExpansionKind::MaskedIntrinsic;
   return TargetLowering::shouldExpandAtomicRMWInIR(AI);
 }
 
 TargetLowering::AtomicExpansionKind
 PPCTargetLowering::shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *AI) const {
   unsigned Size = AI->getNewValOperand()->getType()->getPrimitiveSizeInBits();
-  if (EnableQuadwordAtomics && Subtarget.hasQuadwordAtomics() && Size == 128)
+  if (shouldInlineQuadwordAtomics() && Size == 128)
     return AtomicExpansionKind::MaskedIntrinsic;
   return TargetLowering::shouldExpandAtomicCmpXchgInIR(AI);
 }
 
 static Intrinsic::ID
 getIntrinsicForAtomicRMWBinOp128(AtomicRMWInst::BinOp BinOp) {
   switch (BinOp) {
   default:
getIntrinsicForAtomicRMWBinOp128(AtomicRMWInst::BinOp BinOp) {
   case AtomicRMWInst::Nand:
     return Intrinsic::ppc_atomicrmw_nand_i128;
   }
 }
 
 Value *PPCTargetLowering::emitMaskedAtomicRMWIntrinsic(
     IRBuilderBase &Builder, AtomicRMWInst *AI, Value *AlignedAddr, Value *Incr,
     Value *Mask, Value *ShiftAmt, AtomicOrdering Ord) const {
-  assert(EnableQuadwordAtomics && Subtarget.hasQuadwordAtomics() &&
-         "Only support quadword now");
+  assert(shouldInlineQuadwordAtomics() && "Only support quadword now");
   Module *M = Builder.GetInsertBlock()->getParent()->getParent();
   Type *ValTy = Incr->getType();
   assert(ValTy->getPrimitiveSizeInBits() == 128);
   Function *RMW = Intrinsic::getDeclaration(
       M, getIntrinsicForAtomicRMWBinOp128(AI->getOperation()));
   Type *Int64Ty = Type::getInt64Ty(M->getContext());
   Value *IncrLo = Builder.CreateTrunc(Incr, Int64Ty, "incr_lo");
   Value *IncrHi =
       Builder.CreateTrunc(Builder.CreateLShr(Incr, 64), Int64Ty, "incr_hi");
   Value *Addr =
       Builder.CreateBitCast(AlignedAddr, Type::getInt8PtrTy(M->getContext()));
   Value *LoHi = Builder.CreateCall(RMW, {Addr, IncrLo, IncrHi});
   Value *Lo = Builder.CreateExtractValue(LoHi, 0, "lo");
   Value *Hi = Builder.CreateExtractValue(LoHi, 1, "hi");
   Lo = Builder.CreateZExt(Lo, ValTy, "lo64");
   Hi = Builder.CreateZExt(Hi, ValTy, "hi64");
   return Builder.CreateOr(
       Lo, Builder.CreateShl(Hi, ConstantInt::get(ValTy, 64)), "val64");
 }
 
 Value *PPCTargetLowering::emitMaskedAtomicCmpXchgIntrinsic(
     IRBuilderBase &Builder, AtomicCmpXchgInst *CI, Value *AlignedAddr,
     Value *CmpVal, Value *NewVal, Value *Mask, AtomicOrdering Ord) const {
-  assert(EnableQuadwordAtomics && Subtarget.hasQuadwordAtomics() &&
-         "Only support quadword now");
+  assert(shouldInlineQuadwordAtomics() && "Only support quadword now");
   Module *M = Builder.GetInsertBlock()->getParent()->getParent();
   Type *ValTy = CmpVal->getType();
   assert(ValTy->getPrimitiveSizeInBits() == 128);
   Function *IntCmpXchg =
       Intrinsic::getDeclaration(M, Intrinsic::ppc_cmpxchg_i128);
   Type *Int64Ty = Type::getInt64Ty(M->getContext());
   Value *CmpLo = Builder.CreateTrunc(CmpVal, Int64Ty, "cmp_lo");
   Value *CmpHi =

llvm/test/CodeGen/PowerPC/atomics-i128.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-unknown -mcpu=pwr8 \
 ; RUN:   -ppc-asm-full-reg-names -ppc-quadword-atomics \
 ; RUN:   -ppc-track-subreg-liveness < %s | FileCheck %s
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-unknown -mcpu=pwr7 \
 ; RUN:   -ppc-asm-full-reg-names -ppc-quadword-atomics \
 ; RUN:   -ppc-track-subreg-liveness < %s | FileCheck --check-prefix=PWR7 %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 \
+; RUN:   -ppc-asm-full-reg-names -ppc-track-subreg-liveness < %s | FileCheck \
+; RUN:   --check-prefix=LE-PWR8 %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-freebsd -mcpu=pwr8 \
+; RUN:   -ppc-asm-full-reg-names -ppc-track-subreg-liveness < %s | FileCheck \
+; RUN:   --check-prefix=LE-PWR8 %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-aix -mcpu=pwr8 \
+; RUN:   -ppc-asm-full-reg-names -ppc-track-subreg-liveness < %s | FileCheck \
+; RUN:   --check-prefix=AIX64-PWR8 %s
 
 
 define i128 @swap(i128* %a, i128 %x) {
 ; CHECK-LABEL: swap:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    sync
 ; CHECK-NEXT:  .LBB0_1: # %entry
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    lqarx r6, 0, r3

 ; PWR7-NEXT:    sync
 ; PWR7-NEXT:    bl __sync_lock_test_and_set_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: swap:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    sync
+; LE-PWR8-NEXT:  .LBB0_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r6, 0, r3
+; LE-PWR8-NEXT:    mr r9, r4
+; LE-PWR8-NEXT:    mr r8, r5
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB0_1
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r7
+; LE-PWR8-NEXT:    mr r4, r6
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: swap:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    sync
+; AIX64-PWR8-NEXT:    bl .__sync_lock_test_and_set_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = atomicrmw xchg i128* %a, i128 %x seq_cst, align 16
   ret i128 %0
 }
 
 define i128 @add(i128* %a, i128 %x) {
 ; CHECK-LABEL: add:
 ; CHECK:       # %bb.0: # %entry

 ; PWR7-NEXT:    sync
 ; PWR7-NEXT:    bl __sync_fetch_and_add_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: add:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    sync
+; LE-PWR8-NEXT:  .LBB1_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r6, 0, r3
+; LE-PWR8-NEXT:    addc r9, r4, r7
+; LE-PWR8-NEXT:    adde r8, r5, r6
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB1_1
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r7
+; LE-PWR8-NEXT:    mr r4, r6
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: add:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    sync
+; AIX64-PWR8-NEXT:    bl .__sync_fetch_and_add_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = atomicrmw add i128* %a, i128 %x seq_cst, align 16
   ret i128 %0
 }
 
 define i128 @sub(i128* %a, i128 %x) {
 ; CHECK-LABEL: sub:
 ; CHECK:       # %bb.0: # %entry

 ; PWR7-NEXT:    sync
 ; PWR7-NEXT:    bl __sync_fetch_and_sub_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: sub:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    sync
+; LE-PWR8-NEXT:  .LBB2_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r6, 0, r3
+; LE-PWR8-NEXT:    subc r9, r7, r4
+; LE-PWR8-NEXT:    subfe r8, r5, r6
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB2_1
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r7
+; LE-PWR8-NEXT:    mr r4, r6
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: sub:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    sync
+; AIX64-PWR8-NEXT:    bl .__sync_fetch_and_sub_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = atomicrmw sub i128* %a, i128 %x seq_cst, align 16
   ret i128 %0
 }
 
 define i128 @and(i128* %a, i128 %x) {
 ; CHECK-LABEL: and:
 ; CHECK:       # %bb.0: # %entry

 ; PWR7-NEXT:    sync
 ; PWR7-NEXT:    bl __sync_fetch_and_and_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: and:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    sync
+; LE-PWR8-NEXT:  .LBB3_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r6, 0, r3
+; LE-PWR8-NEXT:    and r9, r4, r7
+; LE-PWR8-NEXT:    and r8, r5, r6
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB3_1
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r7
+; LE-PWR8-NEXT:    mr r4, r6
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: and:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    sync
+; AIX64-PWR8-NEXT:    bl .__sync_fetch_and_and_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = atomicrmw and i128* %a, i128 %x seq_cst, align 16
   ret i128 %0
 }
 
 define i128 @or(i128* %a, i128 %x) {
 ; CHECK-LABEL: or:
 ; CHECK:       # %bb.0: # %entry

 ; PWR7-NEXT:    sync
 ; PWR7-NEXT:    bl __sync_fetch_and_or_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: or:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    sync
+; LE-PWR8-NEXT:  .LBB4_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r6, 0, r3
+; LE-PWR8-NEXT:    or r9, r4, r7
+; LE-PWR8-NEXT:    or r8, r5, r6
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB4_1
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r7
+; LE-PWR8-NEXT:    mr r4, r6
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: or:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    sync
+; AIX64-PWR8-NEXT:    bl .__sync_fetch_and_or_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = atomicrmw or i128* %a, i128 %x seq_cst, align 16
   ret i128 %0
 }
 
 define i128 @xor(i128* %a, i128 %x) {
 ; CHECK-LABEL: xor:
 ; CHECK:       # %bb.0: # %entry

 ; PWR7-NEXT:    sync
 ; PWR7-NEXT:    bl __sync_fetch_and_xor_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: xor:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    sync
+; LE-PWR8-NEXT:  .LBB5_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r6, 0, r3
+; LE-PWR8-NEXT:    xor r9, r4, r7
+; LE-PWR8-NEXT:    xor r8, r5, r6
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB5_1
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r7
+; LE-PWR8-NEXT:    mr r4, r6
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: xor:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    sync
+; AIX64-PWR8-NEXT:    bl .__sync_fetch_and_xor_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = atomicrmw xor i128* %a, i128 %x seq_cst, align 16
   ret i128 %0
 }
 
 define i128 @nand(i128* %a, i128 %x) {
 ; CHECK-LABEL: nand:
 ; CHECK:       # %bb.0: # %entry

 ; PWR7-NEXT:    sync
 ; PWR7-NEXT:    bl __sync_fetch_and_nand_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: nand:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    sync
+; LE-PWR8-NEXT:  .LBB6_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r6, 0, r3
+; LE-PWR8-NEXT:    nand r9, r4, r7
+; LE-PWR8-NEXT:    nand r8, r5, r6
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB6_1
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r7
+; LE-PWR8-NEXT:    mr r4, r6
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: nand:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    sync
+; AIX64-PWR8-NEXT:    bl .__sync_fetch_and_nand_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = atomicrmw nand i128* %a, i128 %x seq_cst, align 16
   ret i128 %0
 }
 
 ;; CmpXchg
 define i128 @cas_weak_acquire_acquire(i128* %a, i128 %cmp, i128 %new) {
 ; CHECK-LABEL: cas_weak_acquire_acquire:

 ; PWR7-NEXT:    .cfi_offset lr, 16
 ; PWR7-NEXT:    bl __sync_val_compare_and_swap_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: cas_weak_acquire_acquire:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:  .LBB7_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r8, 0, r3
+; LE-PWR8-NEXT:    xor r11, r9, r4
+; LE-PWR8-NEXT:    xor r10, r8, r5
+; LE-PWR8-NEXT:    or. r11, r11, r10
+; LE-PWR8-NEXT:    bne cr0, .LBB7_3
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    mr r11, r6
+; LE-PWR8-NEXT:    mr r10, r7
+; LE-PWR8-NEXT:    stqcx. r10, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB7_1
+; LE-PWR8-NEXT:    b .LBB7_4
+; LE-PWR8-NEXT:  .LBB7_3: # %entry
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:  .LBB7_4: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r9
+; LE-PWR8-NEXT:    mr r4, r8
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: cas_weak_acquire_acquire:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    bl .__sync_val_compare_and_swap_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = cmpxchg weak i128* %a, i128 %cmp, i128 %new acquire acquire
   %1 = extractvalue { i128, i1 } %0, 0
   ret i128 %1
 }
 
 define i128 @cas_weak_release_monotonic(i128* %a, i128 %cmp, i128 %new) {
 ; CHECK-LABEL: cas_weak_release_monotonic:

 ; PWR7-NEXT:    .cfi_offset lr, 16
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    bl __sync_val_compare_and_swap_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: cas_weak_release_monotonic:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:  .LBB8_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r8, 0, r3
+; LE-PWR8-NEXT:    xor r11, r9, r4
+; LE-PWR8-NEXT:    xor r10, r8, r5
+; LE-PWR8-NEXT:    or. r11, r11, r10
+; LE-PWR8-NEXT:    bne cr0, .LBB8_3
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    mr r11, r6
+; LE-PWR8-NEXT:    mr r10, r7
+; LE-PWR8-NEXT:    stqcx. r10, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB8_1
+; LE-PWR8-NEXT:    b .LBB8_4
+; LE-PWR8-NEXT:  .LBB8_3: # %entry
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:  .LBB8_4: # %entry
+; LE-PWR8-NEXT:    mr r3, r9
+; LE-PWR8-NEXT:    mr r4, r8
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: cas_weak_release_monotonic:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    bl .__sync_val_compare_and_swap_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = cmpxchg weak i128* %a, i128 %cmp, i128 %new release monotonic
   %1 = extractvalue { i128, i1 } %0, 0
   ret i128 %1
 }
 
 define i128 @cas_sc_sc(i128* %a, i128 %cmp, i128 %new) {
 ; CHECK-LABEL: cas_sc_sc:

 ; PWR7-NEXT:    sync
 ; PWR7-NEXT:    bl __sync_val_compare_and_swap_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: cas_sc_sc:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    sync
+; LE-PWR8-NEXT:  .LBB9_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r8, 0, r3
+; LE-PWR8-NEXT:    xor r11, r9, r4
+; LE-PWR8-NEXT:    xor r10, r8, r5
+; LE-PWR8-NEXT:    or. r11, r11, r10
+; LE-PWR8-NEXT:    bne cr0, .LBB9_3
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    mr r11, r6
+; LE-PWR8-NEXT:    mr r10, r7
+; LE-PWR8-NEXT:    stqcx. r10, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB9_1
+; LE-PWR8-NEXT:    b .LBB9_4
+; LE-PWR8-NEXT:  .LBB9_3: # %entry
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:  .LBB9_4: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r9
+; LE-PWR8-NEXT:    mr r4, r8
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: cas_sc_sc:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    sync
+; AIX64-PWR8-NEXT:    bl .__sync_val_compare_and_swap_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = cmpxchg i128* %a, i128 %cmp, i128 %new seq_cst seq_cst
   %1 = extractvalue { i128, i1 } %0, 0
   ret i128 %1
 }
 
 define i128 @cas_acqrel_acquire(i128* %a, i128 %cmp, i128 %new) {
 ; CHECK-LABEL: cas_acqrel_acquire:

 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    bl __sync_val_compare_and_swap_16
 ; PWR7-NEXT:    nop
 ; PWR7-NEXT:    lwsync
 ; PWR7-NEXT:    addi r1, r1, 112
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: cas_acqrel_acquire:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:  .LBB10_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r8, 0, r3
+; LE-PWR8-NEXT:    xor r11, r9, r4
+; LE-PWR8-NEXT:    xor r10, r8, r5
+; LE-PWR8-NEXT:    or. r11, r11, r10
+; LE-PWR8-NEXT:    bne cr0, .LBB10_3
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    mr r11, r6
+; LE-PWR8-NEXT:    mr r10, r7
+; LE-PWR8-NEXT:    stqcx. r10, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB10_1
+; LE-PWR8-NEXT:    b .LBB10_4
+; LE-PWR8-NEXT:  .LBB10_3: # %entry
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:  .LBB10_4: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    mr r3, r9
+; LE-PWR8-NEXT:    mr r4, r8
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: cas_acqrel_acquire:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -112(r1)
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    bl .__sync_val_compare_and_swap_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    addi r1, r1, 112
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = cmpxchg i128* %a, i128 %cmp, i128 %new acq_rel acquire
   %1 = extractvalue { i128, i1 } %0, 0
   ret i128 %1
 }
 
 define i1 @cas_acqrel_acquire_check_succ(i128* %a, i128 %cmp, i128 %new) {
 ; CHECK-LABEL: cas_acqrel_acquire_check_succ:

 ; PWR7-NEXT:    ld r30, 128(r1) # 8-byte Folded Reload
 ; PWR7-NEXT:    ld r29, 120(r1) # 8-byte Folded Reload
 ; PWR7-NEXT:    cntlzd r3, r3
 ; PWR7-NEXT:    rldicl r3, r3, 58, 63
 ; PWR7-NEXT:    addi r1, r1, 144
 ; PWR7-NEXT:    ld r0, 16(r1)
 ; PWR7-NEXT:    mtlr r0
 ; PWR7-NEXT:    blr
+;
+; LE-PWR8-LABEL: cas_acqrel_acquire_check_succ:
+; LE-PWR8:       # %bb.0: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:  .LBB11_1: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    lqarx r8, 0, r3
+; LE-PWR8-NEXT:    xor r11, r9, r4
+; LE-PWR8-NEXT:    xor r10, r8, r5
+; LE-PWR8-NEXT:    or. r11, r11, r10
+; LE-PWR8-NEXT:    bne cr0, .LBB11_3
+; LE-PWR8-NEXT:  # %bb.2: # %entry
+; LE-PWR8-NEXT:    #
+; LE-PWR8-NEXT:    mr r11, r6
+; LE-PWR8-NEXT:    mr r10, r7
+; LE-PWR8-NEXT:    stqcx. r10, 0, r3
+; LE-PWR8-NEXT:    bne cr0, .LBB11_1
+; LE-PWR8-NEXT:    b .LBB11_4
+; LE-PWR8-NEXT:  .LBB11_3: # %entry
+; LE-PWR8-NEXT:    stqcx. r8, 0, r3
+; LE-PWR8-NEXT:  .LBB11_4: # %entry
+; LE-PWR8-NEXT:    lwsync
+; LE-PWR8-NEXT:    xor r3, r5, r8
+; LE-PWR8-NEXT:    xor r4, r4, r9
+; LE-PWR8-NEXT:    or r3, r4, r3
+; LE-PWR8-NEXT:    cntlzd r3, r3
+; LE-PWR8-NEXT:    rldicl r3, r3, 58, 63
+; LE-PWR8-NEXT:    blr
+;
+; AIX64-PWR8-LABEL: cas_acqrel_acquire_check_succ:
+; AIX64-PWR8:       # %bb.0: # %entry
+; AIX64-PWR8-NEXT:    mflr r0
+; AIX64-PWR8-NEXT:    std r0, 16(r1)
+; AIX64-PWR8-NEXT:    stdu r1, -128(r1)
+; AIX64-PWR8-NEXT:    std r30, 112(r1) # 8-byte Folded Spill
+; AIX64-PWR8-NEXT:    std r31, 120(r1) # 8-byte Folded Spill
+; AIX64-PWR8-NEXT:    mr r31, r5
+; AIX64-PWR8-NEXT:    mr r30, r4
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    bl .__sync_val_compare_and_swap_16[PR]
+; AIX64-PWR8-NEXT:    nop
+; AIX64-PWR8-NEXT:    xor r3, r3, r30
+; AIX64-PWR8-NEXT:    xor r4, r4, r31
+; AIX64-PWR8-NEXT:    lwsync
+; AIX64-PWR8-NEXT:    or r3, r4, r3
+; AIX64-PWR8-NEXT:    ld r31, 120(r1) # 8-byte Folded Reload
+; AIX64-PWR8-NEXT:    ld r30, 112(r1) # 8-byte Folded Reload
+; AIX64-PWR8-NEXT:    cntlzd r3, r3
+; AIX64-PWR8-NEXT:    rldicl r3, r3, 58, 63
+; AIX64-PWR8-NEXT:    addi r1, r1, 128
+; AIX64-PWR8-NEXT:    ld r0, 16(r1)
+; AIX64-PWR8-NEXT:    mtlr r0
+; AIX64-PWR8-NEXT:    blr
 entry:
   %0 = cmpxchg i128* %a, i128 %cmp, i128 %new acq_rel acquire
   %1 = extractvalue { i128, i1 } %0, 1
   ret i1 %1
 }