Add support for part-word atomics for PPC

[nemanjai]

Summary

This patch does two things:

1. Adds support for part-word atomic instructions for Power7 (ISA 2.06) and Power8 (ISA 2.07)
2. Adds the "lock" versions of all of the loads. These are the loads with the Exclusive Access hint bit set to 1

Atomic instructions in the IR will now use the part-word versions on platforms that support them (for types i8 and i16). The lock versions are only available through asm for now.

[nemanjai]

It would appear that in applying the patch to my repository from an anonymous one has somehow lost some of my changes. Moved the position of the stwcx definition and added the check prefix for all the Power7 and up checks in the IR test case.

[hfinkel]

lib/Target/PowerPC/PPCSubtarget.h

117

You need to initialize this in PPCSubtarget.cpp with the others.

[nemanjai]

lib/Target/PowerPC/PPCSubtarget.h

117

Ah yes, thank you. Doing so certainly helps with readability and consistency. Sorry I missed that. Added to the next patch that I'll upload here.

[nemanjai]

Added the initialization for the bool variable to address Hal's comment.

[nemanjai]

One thing to note is that no Clang option is added for this (i.e. -mpartword-atomics). As a follow-on task for this, the quadword version will be implemented which does have a corresponding option in GCC and I plan to implement the same option in Clang. I don't think that the inconsistency is an issue since part word atomics can be implemented without the part word atomic instructions whereas I imagine that the quad word version would require locks if there is no instruction for it in the ISA. Furthermore, the quadword version is new for ISA 2.07 whereas the part word versions existed in ISA 2.06.

[hfinkel]

lib/Target/PowerPC/PPCInstrInfo.td

1469

This pattern is identical to the pattern for LBARX above; can they both really be selected?

(and the same as the LWARX pattern below)

Maybe you need PPClarx to carry a type node operand, like the extload nodes?

1485

Duplicate pattern here too.

[nemanjai]

lib/Target/PowerPC/PPCInstrInfo.td

1469

Please correct any of the below if my understanding of how this works is incorrect (I do not yet fully understand how we go from IR to SDAGs to actual instructions).

This is a good point, there would be no way to disambiguate which instruction to select if a selection DAG matched this pattern. However, I don't see anything in the PPC target implementation that would produce such a DAG node, so I am not sure how useful having this pattern is at all. As far as I can see, these instructions are selected only through the Pseudo instructions and are all emitted through

PPCTargetLowering::EmitInstrWithCustomInserter

And with my limited understanding of how things work, it would make sense that this can only be emitted through a custom code sequence since a load with reservation makes little sense without a subsequent store conditional (and likely the associated conditional branch). On the other hand, something in the PPC back end may decide to produce the PPClarx SD node so it would be good if we can match a likely pattern.
I understand what you mean about the SD Node carrying the type (much like all the versions of the [sign|zero] extend loads that you mention).
What I propose is the following (please comment on validity of the proposal):

• Provide PPClarx8, PPClarx16, PPClarx32 and PPClarx64 SD nodes (PPClarx128 to follow with implementation of quadword version)
• Provide PPCstcx8, PPCstcx16, PPCstcx32 and PPCstcx64 SD nodes (PPCstcx128 to follow)
• Modify the patterns to use the respective version
• Remove the existing PPClarx and PPCstcx SD nodes

In a subsequent patch, provide test coverage for all of the atomic instructions in the IR (currently we seem to only have testing for some of them and only for i32 and i64 types). Maybe even add another test case that contains the IR from existing tests of all of the gcc __sync_* builtins to ensure we produce expected code for all of those.

[hfinkel]

lib/Target/PowerPC/PPCInstrInfo.td

1469

This is a good point, there would be no way to disambiguate which instruction to select if a selection DAG matched this pattern.

Right, and we should not add useless patterns, SDAG nodes, etc. Relative to LLVM itself, the PPC backend is pretty old, and has accumulated some dead constructs no one has yet cleaned up. It seems that you have stumbled upon one...

PPCISD::LARX seems to be dead, as is PPCISD::STCX. We should remove them, and any associated patterns and nodes. The one thing to be careful about here is to make sure that any instruction properties (mayLoad, etc.) that are being implied by the default patterns may need to be explicitly added once the patterns are removed. The definitive way to do this is to check that lib/Target/PowerPC/PPCGenInstrInfo.inc in your build directory has no relevant changes from removing the patterns.

[nemanjai]

Removed the SD nodes and patterns for the atomic instructions. These instructions are never emitted directly from a Selection DAG since there is a very close relationship between the load and the store. As a result, they are only emitted from the atomic Pseudo instructions using a custom inserter. It is therefore not necessary to maintain patterns and SD nodes for the new or existing ones.

[hfinkel]

LGTM (aside from some minor formatting nits).

lib/Target/PowerPC/PPC.td

122

Indentation is off here.

lib/Target/PowerPC/PPCInstr64Bit.td

239

Indentation is off here and below (the " should be under the 3).

lib/Target/PowerPC/PPCInstrInfo.td

1464

Indentation looks off-by-one here too.

[nemanjai]

Committed revision 231843.

lib/Target/PowerPC/PPC.td

def FeatureP8Altivec : SubtargetFeature<"power8-altivec", "HasP8Altivec", "true",
                                         "Enable POWER8 Altivec instructions",
                                         [FeatureAltivec]>;
 def FeatureP8Crypto : SubtargetFeature<"crypto", "HasP8Crypto", "true",
                                         "Enable POWER8 Crypto instructions",
                                         [FeatureP8Altivec]>;
 def FeatureP8Vector  : SubtargetFeature<"power8-vector", "HasP8Vector", "true",
                                         "Enable POWER8 vector instructions",
                                         [FeatureVSX, FeatureP8Altivec]>;
-
+def FeaturePartwordAtomic : SubtargetFeature<"partword-atomics",
+                                               "HasPartwordAtomics", "true",
+                                               "Enable l[bh]arx and st[bh]cx.">;
 def FeatureInvariantFunctionDescriptors :
   SubtargetFeature<"invariant-function-descriptors",
                    "HasInvariantFunctionDescriptors", "true",
                    "Assume function descriptors are invariant">;
 
 def DeprecatedMFTB   : SubtargetFeature<"", "DeprecatedMFTB", "true",
                                         "Treat mftb as deprecated">;
 def DeprecatedDST    : SubtargetFeature<"", "DeprecatedDST", "true",
def ProcessorFeatures {
     list<SubtargetFeature> Power8FeatureList =
         [DirectivePwr8, FeatureAltivec, FeatureP8Altivec, FeatureVSX, 
         FeatureP8Vector, FeatureMFOCRF, FeatureFCPSGN, FeatureFSqrt, 
         FeatureFRE, FeatureFRES, FeatureFRSQRTE, FeatureFRSQRTES,
         FeatureRecipPrec, FeatureSTFIWX, FeatureLFIWAX,
         FeatureFPRND, FeatureFPCVT, FeatureISEL,
         FeaturePOPCNTD, FeatureCMPB, FeatureLDBRX, FeatureP8Crypto,
         Feature64Bit /*, Feature64BitRegs */, FeatureICBT,
-        DeprecatedMFTB, DeprecatedDST];
+        FeaturePartwordAtomic, DeprecatedMFTB, DeprecatedDST];
 }
 
 def : ProcessorModel<"970", G5Model,
                   [Directive970, FeatureAltivec,
                    FeatureMFOCRF, FeatureFSqrt,
                    FeatureFRES, FeatureFRSQRTE, FeatureSTFIWX,
                    Feature64Bit /*, Feature64BitRegs */]>;
 def : ProcessorModel<"g5", G5Model,
def : ProcessorModel<"pwr6x", G5Model,
                    DeprecatedMFTB, DeprecatedDST]>;
 def : ProcessorModel<"pwr7", P7Model,
                   [DirectivePwr7, FeatureAltivec, FeatureVSX,
                    FeatureMFOCRF, FeatureFCPSGN, FeatureFSqrt, FeatureFRE,
                    FeatureFRES, FeatureFRSQRTE, FeatureFRSQRTES,
                    FeatureRecipPrec, FeatureSTFIWX, FeatureLFIWAX,
                    FeatureFPRND, FeatureFPCVT, FeatureISEL,
                    FeaturePOPCNTD, FeatureCMPB, FeatureLDBRX,
-                   Feature64Bit /*, Feature64BitRegs */,
+                   Feature64Bit /*, Feature64BitRegs */, FeaturePartwordAtomic,
                    DeprecatedMFTB, DeprecatedDST]>;
 def : ProcessorModel<"pwr8", P8Model, ProcessorFeatures.Power8FeatureList>;
 def : Processor<"ppc", G3Itineraries, [Directive32]>;
 def : ProcessorModel<"ppc64", G5Model,
                   [Directive64, FeatureAltivec,
                    FeatureMFOCRF, FeatureFSqrt, FeatureFRES,
                    FeatureFRSQRTE, FeatureSTFIWX,
                    Feature64Bit /*, Feature64BitRegs */]>;

lib/Target/PowerPC/PPCISelLowering.h

enum NodeType {
 
       /// F8RC = MFFS - This moves the FPSCR (not modeled) into the register.
       MFFS,
 
       /// LARX = This corresponds to PPC l{w|d}arx instrcution: load and
       /// reserve indexed. This is used to implement atomic operations.
       LARX,
 
+      /// LARXL = the lock version of LARX: this is the same instruction but
+      /// with EH=1 to provide a hint that it is used to load a lock
+      LARXL,
+
       /// STCX = This corresponds to PPC stcx. instrcution: store conditional
       /// indexed. This is used to implement atomic operations.
       STCX,
 
       /// TC_RETURN - A tail call return.
       ///   operand #0 chain
       ///   operand #1 callee (register or absolute)
       ///   operand #2 stack adjustment
Instruction* emitLeadingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
                                   bool IsStore, bool IsLoad) const override;
     Instruction* emitTrailingFence(IRBuilder<> &Builder, AtomicOrdering Ord,
                                    bool IsStore, bool IsLoad) const override;
 
     MachineBasicBlock *
       EmitInstrWithCustomInserter(MachineInstr *MI,
                                   MachineBasicBlock *MBB) const override;
     MachineBasicBlock *EmitAtomicBinary(MachineInstr *MI,
-                                        MachineBasicBlock *MBB, bool is64Bit,
+                                        MachineBasicBlock *MBB,
+                                        unsigned AtomicSize,
                                         unsigned BinOpcode) const;
     MachineBasicBlock *EmitPartwordAtomicBinary(MachineInstr *MI,
                                                 MachineBasicBlock *MBB,
                                             bool is8bit, unsigned Opcode) const;
 
     MachineBasicBlock *emitEHSjLjSetJmp(MachineInstr *MI,
                                         MachineBasicBlock *MBB) const;
 

lib/Target/PowerPC/PPCISelLowering.cpp

Instruction* PPCTargetLowering::emitTrailingFence(IRBuilder<> &Builder,
   // http://www.rdrop.com/users/paulmck/scalability/paper/N2745r.2011.03.04a.html
   // and http://www.cl.cam.ac.uk/~pes20/cppppc/ for justification.
   else
     return nullptr;
 }
 
 MachineBasicBlock *
 PPCTargetLowering::EmitAtomicBinary(MachineInstr *MI, MachineBasicBlock *BB,
-                                    bool is64bit, unsigned BinOpcode) const {
+                                    unsigned AtomicSize,
+                                    unsigned BinOpcode) const {
   // This also handles ATOMIC_SWAP, indicated by BinOpcode==0.
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
 
+  auto LoadMnemonic = PPC::LDARX;
+  auto StoreMnemonic = PPC::STDCX;
+  switch (AtomicSize) {
+  default:
+    llvm_unreachable("Unexpected size of atomic entity");
+  case 1:
+    LoadMnemonic = PPC::LBARX;
+    StoreMnemonic = PPC::STBCX;
+    assert(Subtarget.hasPartwordAtomics() && "Call this only with size >=4");
+    break;
+  case 2:
+    LoadMnemonic = PPC::LHARX;
+    StoreMnemonic = PPC::STHCX;
+    assert(Subtarget.hasPartwordAtomics() && "Call this only with size >=4");
+    break;
+  case 4:
+    LoadMnemonic = PPC::LWARX;
+    StoreMnemonic = PPC::STWCX;
+    break;
+  case 8:
+    LoadMnemonic = PPC::LDARX;
+    StoreMnemonic = PPC::STDCX;
+    break;
+  }
+
   const BasicBlock *LLVM_BB = BB->getBasicBlock();
   MachineFunction *F = BB->getParent();
   MachineFunction::iterator It = BB;
   ++It;
 
   unsigned dest = MI->getOperand(0).getReg();
   unsigned ptrA = MI->getOperand(1).getReg();
   unsigned ptrB = MI->getOperand(2).getReg();
   unsigned incr = MI->getOperand(3).getReg();
   DebugLoc dl = MI->getDebugLoc();
 
   MachineBasicBlock *loopMBB = F->CreateMachineBasicBlock(LLVM_BB);
   MachineBasicBlock *exitMBB = F->CreateMachineBasicBlock(LLVM_BB);
   F->insert(It, loopMBB);
   F->insert(It, exitMBB);
   exitMBB->splice(exitMBB->begin(), BB,
                   std::next(MachineBasicBlock::iterator(MI)), BB->end());
   exitMBB->transferSuccessorsAndUpdatePHIs(BB);
 
   MachineRegisterInfo &RegInfo = F->getRegInfo();
   unsigned TmpReg = (!BinOpcode) ? incr :
-    RegInfo.createVirtualRegister( is64bit ? &PPC::G8RCRegClass
+    RegInfo.createVirtualRegister( AtomicSize == 8 ? &PPC::G8RCRegClass
                                            : &PPC::GPRCRegClass);
 
   //  thisMBB:
   //   ...
   //   fallthrough --> loopMBB
   BB->addSuccessor(loopMBB);
 
   //  loopMBB:
   //   l[wd]arx dest, ptr
   //   add r0, dest, incr
   //   st[wd]cx. r0, ptr
   //   bne- loopMBB
   //   fallthrough --> exitMBB
   BB = loopMBB;
-  BuildMI(BB, dl, TII->get(is64bit ? PPC::LDARX : PPC::LWARX), dest)
+  BuildMI(BB, dl, TII->get(LoadMnemonic), dest)
     .addReg(ptrA).addReg(ptrB);
   if (BinOpcode)
     BuildMI(BB, dl, TII->get(BinOpcode), TmpReg).addReg(incr).addReg(dest);
-  BuildMI(BB, dl, TII->get(is64bit ? PPC::STDCX : PPC::STWCX))
+  BuildMI(BB, dl, TII->get(StoreMnemonic))
     .addReg(TmpReg).addReg(ptrA).addReg(ptrB);
   BuildMI(BB, dl, TII->get(PPC::BCC))
     .addImm(PPC::PRED_NE).addReg(PPC::CR0).addMBB(loopMBB);
   BB->addSuccessor(loopMBB);
   BB->addSuccessor(exitMBB);
 
   //  exitMBB:
   //   ...
   BB = exitMBB;
   return BB;
 }
 
 MachineBasicBlock *
 PPCTargetLowering::EmitPartwordAtomicBinary(MachineInstr *MI,
                                             MachineBasicBlock *BB,
                                             bool is8bit,    // operation
                                             unsigned BinOpcode) const {
+  // If we support part-word atomic mnemonics, just use them
+  if (Subtarget.hasPartwordAtomics())
+    return EmitAtomicBinary(MI, BB, is8bit ? 1 : 2, BinOpcode);
+
   // This also handles ATOMIC_SWAP, indicated by BinOpcode==0.
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   // In 64 bit mode we have to use 64 bits for addresses, even though the
   // lwarx/stwcx are 32 bits.  With the 32-bit atomics we can use address
   // registers without caring whether they're 32 or 64, but here we're
   // doing actual arithmetic on the addresses.
   bool is64bit = Subtarget.isPPC64();
   unsigned ZeroReg = is64bit ? PPC::ZERO8 : PPC::ZERO;
if (Subtarget.hasISEL() && (MI->getOpcode() == PPC::SELECT_CC_I4 ||
     BB->addSuccessor(readMBB);
     BB->addSuccessor(sinkMBB);
   }
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_ADD_I8)
     BB = EmitPartwordAtomicBinary(MI, BB, true, PPC::ADD4);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_ADD_I16)
     BB = EmitPartwordAtomicBinary(MI, BB, false, PPC::ADD4);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_ADD_I32)
-    BB = EmitAtomicBinary(MI, BB, false, PPC::ADD4);
+    BB = EmitAtomicBinary(MI, BB, 4, PPC::ADD4);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_ADD_I64)
-    BB = EmitAtomicBinary(MI, BB, true, PPC::ADD8);
+    BB = EmitAtomicBinary(MI, BB, 8, PPC::ADD8);
 
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_AND_I8)
     BB = EmitPartwordAtomicBinary(MI, BB, true, PPC::AND);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_AND_I16)
     BB = EmitPartwordAtomicBinary(MI, BB, false, PPC::AND);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_AND_I32)
-    BB = EmitAtomicBinary(MI, BB, false, PPC::AND);
+    BB = EmitAtomicBinary(MI, BB, 4, PPC::AND);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_AND_I64)
-    BB = EmitAtomicBinary(MI, BB, true, PPC::AND8);
+    BB = EmitAtomicBinary(MI, BB, 8, PPC::AND8);
 
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_OR_I8)
     BB = EmitPartwordAtomicBinary(MI, BB, true, PPC::OR);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_OR_I16)
     BB = EmitPartwordAtomicBinary(MI, BB, false, PPC::OR);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_OR_I32)
-    BB = EmitAtomicBinary(MI, BB, false, PPC::OR);
+    BB = EmitAtomicBinary(MI, BB, 4, PPC::OR);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_OR_I64)
-    BB = EmitAtomicBinary(MI, BB, true, PPC::OR8);
+    BB = EmitAtomicBinary(MI, BB, 8, PPC::OR8);
 
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_XOR_I8)
     BB = EmitPartwordAtomicBinary(MI, BB, true, PPC::XOR);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_XOR_I16)
     BB = EmitPartwordAtomicBinary(MI, BB, false, PPC::XOR);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_XOR_I32)
-    BB = EmitAtomicBinary(MI, BB, false, PPC::XOR);
+    BB = EmitAtomicBinary(MI, BB, 4, PPC::XOR);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_XOR_I64)
-    BB = EmitAtomicBinary(MI, BB, true, PPC::XOR8);
+    BB = EmitAtomicBinary(MI, BB, 8, PPC::XOR8);
 
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_NAND_I8)
     BB = EmitPartwordAtomicBinary(MI, BB, true, PPC::NAND);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_NAND_I16)
     BB = EmitPartwordAtomicBinary(MI, BB, false, PPC::NAND);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_NAND_I32)
-    BB = EmitAtomicBinary(MI, BB, false, PPC::NAND);
+    BB = EmitAtomicBinary(MI, BB, 4, PPC::NAND);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_NAND_I64)
-    BB = EmitAtomicBinary(MI, BB, true, PPC::NAND8);
+    BB = EmitAtomicBinary(MI, BB, 8, PPC::NAND8);
 
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_SUB_I8)
     BB = EmitPartwordAtomicBinary(MI, BB, true, PPC::SUBF);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_SUB_I16)
     BB = EmitPartwordAtomicBinary(MI, BB, false, PPC::SUBF);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_SUB_I32)
-    BB = EmitAtomicBinary(MI, BB, false, PPC::SUBF);
+    BB = EmitAtomicBinary(MI, BB, 4, PPC::SUBF);
   else if (MI->getOpcode() == PPC::ATOMIC_LOAD_SUB_I64)
-    BB = EmitAtomicBinary(MI, BB, true, PPC::SUBF8);
+    BB = EmitAtomicBinary(MI, BB, 8, PPC::SUBF8);
 
   else if (MI->getOpcode() == PPC::ATOMIC_SWAP_I8)
     BB = EmitPartwordAtomicBinary(MI, BB, true, 0);
   else if (MI->getOpcode() == PPC::ATOMIC_SWAP_I16)
     BB = EmitPartwordAtomicBinary(MI, BB, false, 0);
   else if (MI->getOpcode() == PPC::ATOMIC_SWAP_I32)
-    BB = EmitAtomicBinary(MI, BB, false, 0);
+    BB = EmitAtomicBinary(MI, BB, 4, 0);
   else if (MI->getOpcode() == PPC::ATOMIC_SWAP_I64)
-    BB = EmitAtomicBinary(MI, BB, true, 0);
+    BB = EmitAtomicBinary(MI, BB, 8, 0);
 
   else if (MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I32 ||
-           MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I64) {
+           MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I64 ||
+           (Subtarget.hasPartwordAtomics() &&
+            MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I8) ||
+           (Subtarget.hasPartwordAtomics() &&
+            MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I16)) {
     bool is64bit = MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I64;
 
+    auto LoadMnemonic = PPC::LDARX;
+    auto StoreMnemonic = PPC::STDCX;
+    switch(MI->getOpcode()) {
+    default:
+      llvm_unreachable("Compare and swap of unknown size");
+    case PPC::ATOMIC_CMP_SWAP_I8:
+      LoadMnemonic = PPC::LBARX;
+      StoreMnemonic = PPC::STBCX;
+      assert(Subtarget.hasPartwordAtomics() && "No support partword atomics.");
+      break;
+    case PPC::ATOMIC_CMP_SWAP_I16:
+      LoadMnemonic = PPC::LHARX;
+      StoreMnemonic = PPC::STHCX;
+      assert(Subtarget.hasPartwordAtomics() && "No support partword atomics.");
+      break;
+    case PPC::ATOMIC_CMP_SWAP_I32:
+      LoadMnemonic = PPC::LWARX;
+      StoreMnemonic = PPC::STWCX;
+      break;
+    case PPC::ATOMIC_CMP_SWAP_I64:
+      LoadMnemonic = PPC::LDARX;
+      StoreMnemonic = PPC::STDCX;
+      break;
+    }
     unsigned dest   = MI->getOperand(0).getReg();
     unsigned ptrA   = MI->getOperand(1).getReg();
     unsigned ptrB   = MI->getOperand(2).getReg();
     unsigned oldval = MI->getOperand(3).getReg();
     unsigned newval = MI->getOperand(4).getReg();
     DebugLoc dl     = MI->getDebugLoc();
 
     MachineBasicBlock *loop1MBB = F->CreateMachineBasicBlock(LLVM_BB);
else if (MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I32 ||
     exitMBB->transferSuccessorsAndUpdatePHIs(BB);
 
     //  thisMBB:
     //   ...
     //   fallthrough --> loopMBB
     BB->addSuccessor(loop1MBB);
 
     // loop1MBB:
-    //   l[wd]arx dest, ptr
+    //   l[bhwd]arx dest, ptr
     //   cmp[wd] dest, oldval
     //   bne- midMBB
     // loop2MBB:
-    //   st[wd]cx. newval, ptr
+    //   st[bhwd]cx. newval, ptr
     //   bne- loopMBB
     //   b exitBB
     // midMBB:
-    //   st[wd]cx. dest, ptr
+    //   st[bhwd]cx. dest, ptr
     // exitBB:
     BB = loop1MBB;
-    BuildMI(BB, dl, TII->get(is64bit ? PPC::LDARX : PPC::LWARX), dest)
+    BuildMI(BB, dl, TII->get(LoadMnemonic), dest)
       .addReg(ptrA).addReg(ptrB);
     BuildMI(BB, dl, TII->get(is64bit ? PPC::CMPD : PPC::CMPW), PPC::CR0)
       .addReg(oldval).addReg(dest);
     BuildMI(BB, dl, TII->get(PPC::BCC))
       .addImm(PPC::PRED_NE).addReg(PPC::CR0).addMBB(midMBB);
     BB->addSuccessor(loop2MBB);
     BB->addSuccessor(midMBB);
 
     BB = loop2MBB;
-    BuildMI(BB, dl, TII->get(is64bit ? PPC::STDCX : PPC::STWCX))
+    BuildMI(BB, dl, TII->get(StoreMnemonic))
       .addReg(newval).addReg(ptrA).addReg(ptrB);
     BuildMI(BB, dl, TII->get(PPC::BCC))
       .addImm(PPC::PRED_NE).addReg(PPC::CR0).addMBB(loop1MBB);
     BuildMI(BB, dl, TII->get(PPC::B)).addMBB(exitMBB);
     BB->addSuccessor(loop1MBB);
     BB->addSuccessor(exitMBB);
 
     BB = midMBB;
-    BuildMI(BB, dl, TII->get(is64bit ? PPC::STDCX : PPC::STWCX))
+    BuildMI(BB, dl, TII->get(StoreMnemonic))
       .addReg(dest).addReg(ptrA).addReg(ptrB);
     BB->addSuccessor(exitMBB);
 
     //  exitMBB:
     //   ...
     BB = exitMBB;
   } else if (MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I8 ||
              MI->getOpcode() == PPC::ATOMIC_CMP_SWAP_I16) {

lib/Target/PowerPC/PPCInstr64Bit.td

let Defs = [CR0] in {
     def ATOMIC_SWAP_I64 : Pseudo<
       (outs g8rc:$dst), (ins memrr:$ptr, g8rc:$new), "#ATOMIC_SWAP_I64",
       [(set i64:$dst, (atomic_swap_64 xoaddr:$ptr, i64:$new))]>;
   }
 }
 
 // Instructions to support atomic operations
 def LDARX : XForm_1<31,  84, (outs g8rc:$rD), (ins memrr:$ptr),
                    "ldarx $rD, $ptr", IIC_LdStLDARX,
                    [(set i64:$rD, (PPClarx xoaddr:$ptr))]>;
 
+// Instruction to support lock versions of atomics
+// (EH=1 - see Power ISA 2.07 Book II 4.4.2)
+def LDARXL : XForm_1<31,  84, (outs g8rc:$rD), (ins memrr:$ptr),
+                   "ldarx $rD, $ptr, 1", IIC_LdStLDARX,
+                   [(set i64:$rD, (PPClarxl xoaddr:$ptr))]>, isDOT;
+
 let Defs = [CR0] in
 def STDCX : XForm_1<31, 214, (outs), (ins g8rc:$rS, memrr:$dst),
                    "stdcx. $rS, $dst", IIC_LdStSTDCX,
                    [(PPCstcx i64:$rS, xoaddr:$dst)]>,
                    isDOT;
 
 let Interpretation64Bit = 1, isCodeGenOnly = 1 in {
 let isCall = 1, isTerminator = 1, isReturn = 1, isBarrier = 1, Uses = [RM] in

lib/Target/PowerPC/PPCInstrInfo.td

def PPCcr6unset : SDNode<"PPCISD::CR6UNSET", SDTNone,
                          [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue]>;
 
 // Instructions to support atomic operations
 def PPClarx      : SDNode<"PPCISD::LARX", SDT_PPClarx,
                           [SDNPHasChain, SDNPMayLoad]>;
 def PPCstcx      : SDNode<"PPCISD::STCX", SDT_PPCstcx,
                           [SDNPHasChain, SDNPMayStore]>;
 
+// Instructions to support atomic lock operations
+def PPClarxl      : SDNode<"PPCISD::LARXL", SDT_PPClarx,
+                          [SDNPHasChain, SDNPMayLoad]>;
+
 // Instructions to support dynamic alloca.
 def SDTDynOp  : SDTypeProfile<1, 2, []>;
 def PPCdynalloc   : SDNode<"PPCISD::DYNALLOC", SDTDynOp, [SDNPHasChain]>;
 
 //===----------------------------------------------------------------------===//
 // PowerPC specific transformation functions and pattern fragments.
 //
 

 def IsNotBookE  : Predicate<"!PPCSubTarget->isBookE()">;
 def HasOnlyMSYNC : Predicate<"PPCSubTarget->hasOnlyMSYNC()">;
 def HasSYNC   : Predicate<"!PPCSubTarget->hasOnlyMSYNC()">;
 def IsPPC4xx  : Predicate<"PPCSubTarget->isPPC4xx()">;
 def IsPPC6xx  : Predicate<"PPCSubTarget->isPPC6xx()">;
 def IsE500  : Predicate<"PPCSubTarget->isE500()">;
 def HasSPE  : Predicate<"PPCSubTarget->HasSPE()">;
 def HasICBT : Predicate<"PPCSubTarget->hasICBT()">;
-
+def HasPartwordAtomics : Predicate<"PPCSubTarget->hasPartwordAtomics()">;
 def NoNaNsFPMath : Predicate<"TM.Options.NoNaNsFPMath">;
 def NaNsFPMath   : Predicate<"!TM.Options.NoNaNsFPMath">;
 
 //===----------------------------------------------------------------------===//
 // PowerPC Multiclass Definitions.
 
 multiclass XForm_6r<bits<6> opcode, bits<10> xo, dag OOL, dag IOL,
                     string asmbase, string asmstr, InstrItinClass itin,
def ATOMIC_SWAP_I16 : Pseudo<
       [(set i32:$dst, (atomic_swap_16 xoaddr:$ptr, i32:$new))]>;
     def ATOMIC_SWAP_I32 : Pseudo<
       (outs gprc:$dst), (ins memrr:$ptr, gprc:$new), "#ATOMIC_SWAP_I32",
       [(set i32:$dst, (atomic_swap_32 xoaddr:$ptr, i32:$new))]>;
   }
 }
 
 // Instructions to support atomic operations
+def LBARX : XForm_1<31,  52, (outs gprc:$rD), (ins memrr:$src),
+                   "lbarx $rD, $src", IIC_LdStLWARX,
+                   [(set i32:$rD, (PPClarx xoaddr:$src))]>,
+                   Requires<[HasPartwordAtomics]>;
+
+def LHARX : XForm_1<31,  116, (outs gprc:$rD), (ins memrr:$src),
+                   "lharx $rD, $src", IIC_LdStLWARX,
+                   [(set i32:$rD, (PPClarx xoaddr:$src))]>,
+                   Requires<[HasPartwordAtomics]>;
+
 def LWARX : XForm_1<31,  20, (outs gprc:$rD), (ins memrr:$src),
                    "lwarx $rD, $src", IIC_LdStLWARX,
                    [(set i32:$rD, (PPClarx xoaddr:$src))]>;
 
-let Defs = [CR0] in
+// Instructions to support lock versions of atomics
+// (EH=1 - see Power ISA 2.07 Book II 4.4.2)
+def LBARXL : XForm_1<31,  52, (outs gprc:$rD), (ins memrr:$src),
+                   "lbarx $rD, $src, 1", IIC_LdStLWARX,
+                   [(set i32:$rD, (PPClarxl xoaddr:$src))]>, isDOT,
+                   Requires<[HasPartwordAtomics]>;
+
+def LHARXL : XForm_1<31,  116, (outs gprc:$rD), (ins memrr:$src),
+                   "lharx $rD, $src, 1", IIC_LdStLWARX,
+                   [(set i32:$rD, (PPClarxl xoaddr:$src))]>, isDOT,
+                   Requires<[HasPartwordAtomics]>;
+
+def LWARXL : XForm_1<31,  20, (outs gprc:$rD), (ins memrr:$src),
+                   "lwarx $rD, $src, 1", IIC_LdStLWARX,
+                   [(set i32:$rD, (PPClarxl xoaddr:$src))]>, isDOT;
+
+let Defs = [CR0] in {
+def STBCX : XForm_1<31, 694, (outs), (ins gprc:$rS, memrr:$dst),
+                   "stbcx. $rS, $dst", IIC_LdStSTWCX,
+                   [(PPCstcx i32:$rS, xoaddr:$dst)]>,
+                   isDOT, Requires<[HasPartwordAtomics]>;
+
+def STHCX : XForm_1<31, 726, (outs), (ins gprc:$rS, memrr:$dst),
+                   "sthcx. $rS, $dst", IIC_LdStSTWCX,
+                   [(PPCstcx i32:$rS, xoaddr:$dst)]>,
+                   isDOT, Requires<[HasPartwordAtomics]>;
+
 def STWCX : XForm_1<31, 150, (outs), (ins gprc:$rS, memrr:$dst),
                    "stwcx. $rS, $dst", IIC_LdStSTWCX,
                    [(PPCstcx i32:$rS, xoaddr:$dst)]>,
                    isDOT;
+}
 
 let isTerminator = 1, isBarrier = 1, hasCtrlDep = 1 in
 def TRAP  : XForm_24<31, 4, (outs), (ins), "trap", IIC_LdStLoad, [(trap)]>;
 
 def TWI : DForm_base<3, (outs), (ins u5imm:$to, gprc:$rA, s16imm:$imm),
                      "twi $to, $rA, $imm", IIC_IntTrapW, []>;
 def TW : XForm_1<31, 4, (outs), (ins u5imm:$to, gprc:$rA, gprc:$rB),
                  "tw $to, $rA, $rB", IIC_IntTrapW, []>;

lib/Target/PowerPC/PPCSubtarget.h

protected:
   bool IsPPC4xx;
   bool IsPPC6xx;
   bool DeprecatedMFTB;
   bool DeprecatedDST;
   bool HasLazyResolverStubs;
   bool IsLittleEndian;
   bool HasICBT;
   bool HasInvariantFunctionDescriptors;
+  bool HasPartwordAtomics;
 
   /// When targeting QPX running a stock PPC64 Linux kernel where the stack
   /// alignment has not been changed, we need to keep the 16-byte alignment
   /// of the stack.
   bool IsQPXStackUnaligned;
 
   const PPCTargetMachine &TM;
   PPCFrameLowering FrameLowering;
public:
   bool isPPC6xx() const { return IsPPC6xx; }
   bool isE500() const { return IsE500; }
   bool isDeprecatedMFTB() const { return DeprecatedMFTB; }
   bool isDeprecatedDST() const { return DeprecatedDST; }
   bool hasICBT() const { return HasICBT; }
   bool hasInvariantFunctionDescriptors() const {
     return HasInvariantFunctionDescriptors;
   }
+  bool hasPartwordAtomics() const { return HasPartwordAtomics; }
 
   bool isQPXStackUnaligned() const { return IsQPXStackUnaligned; }
   unsigned getPlatformStackAlignment() const {
     if ((hasQPX() || isBGQ()) && !isQPXStackUnaligned())
       return 32;
 
     return 16;
   }

lib/Target/PowerPC/PPCSubtarget.cpp

void PPCSubtarget::initializeEnvironment() {
   IsPPC4xx = false;
   IsPPC6xx = false;
   IsE500 = false;
   DeprecatedMFTB = false;
   DeprecatedDST = false;
   HasLazyResolverStubs = false;
   HasICBT = false;
   HasInvariantFunctionDescriptors = false;
+  HasPartwordAtomics = false;
   IsQPXStackUnaligned = false;
 }
 
 void PPCSubtarget::initSubtargetFeatures(StringRef CPU, StringRef FS) {
   // Determine default and user specified characteristics
   std::string CPUName = CPU;
   if (CPUName.empty()) {
     // If cross-compiling with -march=ppc64le without -mcpu

test/CodeGen/PowerPC/atomic-2.ll

 ; RUN: llc < %s -march=ppc64 | FileCheck %s
+; RUN: llc < %s -march=ppc64 -mcpu=pwr7 | FileCheck %s -check-prefix=CHECK-P7U
+; RUN: llc < %s -march=ppc64 -mcpu=pwr8 | FileCheck %s -check-prefix=CHECK-P7U
 
 define i64 @exchange_and_add(i64* %mem, i64 %val) nounwind {
 ; CHECK-LABEL: exchange_and_add:
 ; CHECK: ldarx
   %tmp = atomicrmw add i64* %mem, i64 %val monotonic
 ; CHECK: stdcx.
   ret i64 %tmp
 }
 
+define i8 @exchange_and_add8(i8* %mem, i8 %val) nounwind {
+; CHECK-LABEL: exchange_and_add8:
+; CHECK-P7U: lbarx
+  %tmp = atomicrmw add i8* %mem, i8 %val monotonic
+; CHECK-P7U: stbcx.
+  ret i8 %tmp
+}
+
+define i16 @exchange_and_add16(i16* %mem, i16 %val) nounwind {
+; CHECK-LABEL: exchange_and_add16:
+; CHECK-P7U: lharx
+  %tmp = atomicrmw add i16* %mem, i16 %val monotonic
+; CHECK-P7U: sthcx.
+  ret i16 %tmp
+}
+
 define i64 @exchange_and_cmp(i64* %mem) nounwind {
 ; CHECK-LABEL: exchange_and_cmp:
 ; CHECK: ldarx
   %tmppair = cmpxchg i64* %mem, i64 0, i64 1 monotonic monotonic
   %tmp = extractvalue { i64, i1 } %tmppair, 0
 ; CHECK: stdcx.
 ; CHECK: stdcx.
   ret i64 %tmp
 }
 
+define i8 @exchange_and_cmp8(i8* %mem) nounwind {
+; CHECK-LABEL: exchange_and_cmp8:
+; CHECK-P7U: lbarx
+  %tmppair = cmpxchg i8* %mem, i8 0, i8 1 monotonic monotonic
+  %tmp = extractvalue { i8, i1 } %tmppair, 0
+; CHECK-P7U: stbcx.
+; CHECK-P7U: stbcx.
+  ret i8 %tmp
+}
+
+define i16 @exchange_and_cmp16(i16* %mem) nounwind {
+; CHECK-LABEL: exchange_and_cmp16:
+; CHECK-P7U: lharx
+  %tmppair = cmpxchg i16* %mem, i16 0, i16 1 monotonic monotonic
+  %tmp = extractvalue { i16, i1 } %tmppair, 0
+; CHECK-P7U: sthcx.
+; CHECK-P7U: sthcx.
+  ret i16 %tmp
+}
+
 define i64 @exchange(i64* %mem, i64 %val) nounwind {
 ; CHECK-LABEL: exchange:
 ; CHECK: ldarx
   %tmp = atomicrmw xchg i64* %mem, i64 1 monotonic
 ; CHECK: stdcx.
   ret i64 %tmp
 }
 
+define i8 @exchange8(i8* %mem, i8 %val) nounwind {
+; CHECK-LABEL: exchange8:
+; CHECK-P7U: lbarx
+  %tmp = atomicrmw xchg i8* %mem, i8 1 monotonic
+; CHECK-P7U: stbcx.
+  ret i8 %tmp
+}
+
+define i16 @exchange16(i16* %mem, i16 %val) nounwind {
+; CHECK-LABEL: exchange16:
+; CHECK-P7U: lharx
+  %tmp = atomicrmw xchg i16* %mem, i16 1 monotonic
+; CHECK-P7U: sthcx.
+  ret i16 %tmp
+}
+
 define void @atomic_store(i64* %mem, i64 %val) nounwind {
 entry:
 ; CHECK: @atomic_store
   store atomic i64 %val, i64* %mem release, align 64
 ; CHECK: sync 1
 ; CHECK-NOT: stdcx
 ; CHECK: std
   ret void

test/MC/Disassembler/PowerPC/ppc64-encoding-bookII.txt

 0x7c 0x40 0x00 0x7c
 
 # CHECK: mbar 1
 0x7c 0x20 0x06 0xac
 
 # CHECK: dcbf 2, 3                       
 0x7c 0x02 0x18 0xac
 
-# CHECK: lwarx 2, 3, 4                   
+# CHECK: lbarx 2, 3, 4
+0x7c 0x43 0x20 0x68
+
+# CHECK: lharx 2, 3, 4
+0x7c 0x43 0x20 0xe8
+
+# CHECK: lwarx 2, 3, 4
 0x7c 0x43 0x20 0x28
 
-# CHECK: ldarx 2, 3, 4                   
+# CHECK: ldarx 2, 3, 4
 0x7c 0x43 0x20 0xa8
 
+# CHECK: lbarx 2, 3, 4, 1
+0x7c 0x43 0x20 0x69
+
+# CHECK: lharx 2, 3, 4, 1
+0x7c 0x43 0x20 0xe9
+
+# CHECK: lwarx 2, 3, 4, 1
+0x7c 0x43 0x20 0x29
+
+# CHECK: ldarx 2, 3, 4, 1
+0x7c 0x43 0x20 0xa9
+
 # CHECK: sync 0                          
 0x7c 0x00 0x04 0xac
 
 # CHECK: sync 0                          
 0x7c 0x00 0x04 0xac
 
 # CHECK: sync 1                          
 0x7c 0x20 0x04 0xac

test/MC/PowerPC/ppc64-encoding-bookII.s

 # FIXME:    dcbf 2, 3, 1
 
 # Synchronization instructions
 
 # CHECK-BE: isync                           # encoding: [0x4c,0x00,0x01,0x2c]
 # CHECK-LE: isync                           # encoding: [0x2c,0x01,0x00,0x4c]
             isync
 
-# FIXME:    lbarx 2, 3, 4, 1
-# FIXME:    lharx 2, 3, 4, 1
-# FIXME:    lwarx 2, 3, 4, 1
-# FIXME:    ldarx 2, 3, 4, 1
-
 # FIXME:    stbcx. 2, 3, 4
 # FIXME:    sthcx. 2, 3, 4
 # CHECK-BE: stwcx. 2, 3, 4                  # encoding: [0x7c,0x43,0x21,0x2d]
 # CHECK-LE: stwcx. 2, 3, 4                  # encoding: [0x2d,0x21,0x43,0x7c]
             stwcx. 2, 3, 4
 # CHECK-BE: stdcx. 2, 3, 4                  # encoding: [0x7c,0x43,0x21,0xad]
 # CHECK-LE: stdcx. 2, 3, 4                  # encoding: [0xad,0x21,0x43,0x7c]
             stdcx. 2, 3, 4

 
 # Extended mnemonics
 
 # CHECK-BE: dcbf 2, 3                       # encoding: [0x7c,0x02,0x18,0xac]
 # CHECK-LE: dcbf 2, 3                       # encoding: [0xac,0x18,0x02,0x7c]
             dcbf 2, 3
 # FIXME:    dcbfl 2, 3
 
-# FIXME:    lbarx 2, 3, 4
-# FIXME:    lharx 2, 3, 4
+# CHECK-BE: lbarx 2, 3, 4                   # encoding: [0x7c,0x43,0x20,0x68]
+# CHECK-LE: lbarx 2, 3, 4                   # encoding: [0x68,0x20,0x43,0x7c]
+            lbarx 2, 3, 4
+
+# CHECK-BE: lharx 2, 3, 4                   # encoding: [0x7c,0x43,0x20,0xe8]
+# CHECK-LE: lharx 2, 3, 4                   # encoding: [0xe8,0x20,0x43,0x7c]
+            lharx 2, 3, 4
+
 # CHECK-BE: lwarx 2, 3, 4                   # encoding: [0x7c,0x43,0x20,0x28]
 # CHECK-LE: lwarx 2, 3, 4                   # encoding: [0x28,0x20,0x43,0x7c]
             lwarx 2, 3, 4
+
 # CHECK-BE: ldarx 2, 3, 4                   # encoding: [0x7c,0x43,0x20,0xa8]
 # CHECK-LE: ldarx 2, 3, 4                   # encoding: [0xa8,0x20,0x43,0x7c]
             ldarx 2, 3, 4
 
+# CHECK-BE: lbarx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x20,0x69]
+# CHECK-LE: lbarx 2, 3, 4, 1                # encoding: [0x69,0x20,0x43,0x7c]
+            lbarx 2, 3, 4, 1
+
+# CHECK-BE: lharx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x20,0xe9]
+# CHECK-LE: lharx 2, 3, 4, 1                # encoding: [0xe9,0x20,0x43,0x7c]
+            lharx 2, 3, 4, 1
+
+# CHECK-BE: lwarx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x20,0x29]
+# CHECK-LE: lwarx 2, 3, 4, 1                # encoding: [0x29,0x20,0x43,0x7c]
+            lwarx 2, 3, 4, 1
+
+# CHECK-BE: ldarx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x20,0xa9]
+# CHECK-LE: ldarx 2, 3, 4, 1                # encoding: [0xa9,0x20,0x43,0x7c]
+            ldarx 2, 3, 4, 1
+
 # CHECK-BE: sync 0                          # encoding: [0x7c,0x00,0x04,0xac]
 # CHECK-LE: sync 0                          # encoding: [0xac,0x04,0x00,0x7c]
             sync
 # CHECK-BE: sync 0                          # encoding: [0x7c,0x00,0x04,0xac]
 # CHECK-LE: sync 0                          # encoding: [0xac,0x04,0x00,0x7c]
             msync
 # CHECK-BE: sync 1                          # encoding: [0x7c,0x20,0x04,0xac]
 # CHECK-LE: sync 1                          # encoding: [0xac,0x04,0x20,0x7c]