LSE Atomics reorg - Part I

[steleman]

Summary

This is the first part of adding memory synchronization semantics to LSE Atomics.

This patch does not change the functionality of the existing LSE Atomics.
It is just the first step necessary for adding memory synchronization semantics.

The memory semantics feature will be added in a subsequent patch.

In this patch, several corrections were added to the existing LSE Atomics implementation, based on the ARM Errata D11904 from 05/12/2017.

[mcrosier]

If at all possible, I would like to see D35309 committed before/with this patch.

[steleman]

In D35319#807933, @mcrosier wrote:

If at all possible, I would like to see D35309 committed before/with this patch.

One of the things I am doing in this - D35319 - changeset is correcting the naming convention of the LSE instructions mnemonics, which currently does not observe the
established AArch64 naming convention for instruction mnemonics.

Namely: 'B', 'H', 'W' and 'X' suffixes, and not 'b', 'h', 's' and 'd'.

[t.p.northover]

This diff covers lots of different areas:

1. DeadRegisterDefinitions: seems like a reasonable improvement,
2. The instruction definitions: horrible on the surface, but a massive bug might justify them. It's completely unclear why they're necessary (especially as this patch contains no tests).
3. The scheduling: whatever, assuming you have inside knowledge of ThunderX.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

This loop seems too generic: we know what instructions we're looking at, and we know that the entire purpose is to replace a register with XZR/WZR -- the surrounding code isn't going to do that if it's not in that class.

lib/Target/AArch64/AArch64InstrFormats.td

9501–9503

Why are you replacing an InstAlias with a real instruction? That's architecturally incorrect and only really done in LLVM when there's no other choice (weird encoding issues and so on in ARM).

9581–9582

What does this have to do with atomics?

[steleman]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

I do not understand what you are saying here.

The whole purpose of this function is to prevent the use of WZR/XZR, and not replace some other register with XZR/WZR, which would be wrong, as per ARM Errata.

[steleman]

In D35319#807963, @t.p.northover wrote:

This diff covers lots of different areas:

3. The scheduling: whatever, assuming you have inside knowledge of ThunderX.

What does whatever mean, in this context?

[t.p.northover]

What does whatever mean, in this context?

The code looks vaguely plausible for defining how these are scheduled but I have no idea whether it's actually a performance improvement because I don't have access to any ThunderX internal documentation that describes its microarchitecture.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

Yes, but all of these instructions do have some definition that would otherwise be eligible. And if they didn't then the function wouldn't be necessary in the first place because the main loop checks.

[mcrosier]

In D35319#807979, @steleman wrote:

In D35319#807963, @t.p.northover wrote:

This diff covers lots of different areas:

3. The scheduling: whatever, assuming you have inside knowledge of ThunderX.

What does whatever mean, in this context?

The scheduling changes for ThunderX can't really be reviewed by the community (as we don't have the domain specific knowledge), so Tim is basically saying do whatever you see fit.

[christof]

Few inline comments on the dead register pass. I've not looked in detail to the other changes done in this patch.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

124

I don't think CAS and CASP lose acquire semantics when they target the zero register. Am I wrong?

141

Any particular reason why not to look only at the target (Wt) operand? Only the zero register as target operand makes the instructions ignore the acquire behaviour.

[steleman]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

The main loop checks after the ShouldSkip function is called, and not before it.

Therefore, the ineligible instructions have not already been excluded by the main loop, because the main loop hasn't executed yet.

[t.p.northover]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

124

I think christof is right too. LD* and SWP are the ones affected.

[t.p.northover]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

Why does when it's checked matter? Also, even without that the loop is still redundant on the grounds that all of these instructions *do* have a dodgy class.

154

No need to mention that, especially not with a wrong-endian date. It's part of the ARMv8.1a spec.

[mssimpso]

Hi Stefan,

I went ahead and committed D35309, so you will likely need to rebase the changes to AArch64InstrAtomics.td and AArch64DeadRegisterDefinitionsPass.cpp. I would also recommend separating out the ThunderX scheduling changes into their own patch, at least when committing.

[steleman]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

Also, even without that the loop is still redundant on the grounds that all of these instructions *do* have a dodgy class.

I don't understand what this means.

ShouldSkip needs to iterate over all the definitions of the MCInstrDesc, as to obtain the TargetRegisterClass:

const MCInstrDesc &Desc = MI.getDesc();
unsigned ND = Desc.getNumDefs();

// [ ... ]

const TargetRegisterClass *RC = TII->getRegClass(Desc, I, TRI, MF);
if (RC == nullptr)
  continue;

if (RC->contains(AArch64::WZR) || RC->contains(AArch64::XZR))
  return true;

// [ ... ]

Does it not?

[steleman]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141

No reason - just that it was easier to write this way.

But looking only at the Wt operand is a very good suggestion, so I will change it thusly.

[steleman]

In D35319#808044, @mssimpso wrote:

Hi Stefan,

I went ahead and committed D35309, so you will likely need to rebase the changes to AArch64InstrAtomics.td and AArch64DeadRegisterDefinitionsPass.cpp. I would also recommend separating out the ThunderX scheduling changes into their own patch, at least when committing.

OK, that's fine. I'll rebase.

[t.p.northover]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

I don't understand what this means.

There is no instance in which this loop (once reached) will not return true, and that is guaranteed not just by dynamic register classes but by the structure of the instructions being considered (i.e. the MCInstrDescs are generated at compile time).

And just to keep banging on about the other point so it's not forgotten: even if it did return false because no such operand was present, the later code wouldn't try to replace the register anyway because it also checks whether XZR/WZR is valid.

The loop does nothing useful, and if it did that should be tested (since it's an addition rather than pure refactoring).

[steleman]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

124

Yes, after re-reading the Errata three times, CAS/CASP shouldn't be excluded.

[steleman]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

There is no instance in which this loop (once reached) will not return true

OK, now I am really lost.

Are you saying that this (inner) loop:

for (unsigned I = 0; I < ND; ++I) {
// [ ... ]
}

will always return true?

[christof]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

Yes. If you run through all the operands of the instructions listed earlier, at some point it will hit the operand for Wt. That operand has a known register class which includes the zero register.

[steleman]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

Ahh, I just realized my mistake. So the TargetRegisterClass will always contain WZR/XZR because it contains all the registers used by the AArch64 Target, and not just the registers being used by this particular instruction.

What I am really looking for is the set of registers being used by this particular instruction.

[christof]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

The TargetRegisterClass of an instruction operand lists all permissible registers for that operand.

Why do you want to know the register that is currently in use? This pass is trying to change the target operand register into WZR/XZR. The blacklist you create here should not care which registers are currently in use.

It might be worth to note that these 8.1 instructions will never have a target operand of WZR/XZR as the compiler will not generate such instruction. Maybe you planned to allow them in these pass? I think that is a bit premature and complicates this black-list.

[steleman]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

Maybe you planned to allow them in these pass? I think that is a bit premature and complicates this black-list.

I was trying to expand the blacklist to include all the <I>A and <I>AL instructions, and ended up over-complicating it unnecessarily.

So, I will re-submit and it will be much simpler.

It might be worth to note that these 8.1 instructions will never have a target operand of WZR/XZR as the compiler will not generate such instruction.

So, then, is this blacklist even necessary?

[t.p.northover]

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

141–142

So, then, is this blacklist even necessary?

I think he meant intentionally from a TableGen pattern or something. This blacklist is what prevents the compiler from generating them unintentionally. Without it when the compiler sees something like

%vreg0<def, dead> = LDADDAL %vreg1, %vreg2

it'll notice that vreg0 is dead, and that LDADDAL can potentially use XZR there and make the switch.

[steleman]

Updated and corrected AArch64DeadRegisterDefinitions::ShouldSkip.

[t.p.northover]

OK, that pass looks fine to me now. And we're still going to trust you know what you're doing with the scheduling. Which leaves the churn in the TableGen files...

As far as I can see the only actual change is to the internal names of these instructions, which could be accomplished in a much less intrusive way. You're also creating real ST* instructions, but that seems unnecessary to me (and actively bad style).

[steleman]

In D35319#807963, @t.p.northover wrote:

This diff covers lots of different areas:

2. The instruction definitions: horrible on the surface, but a massive bug might justify them. It's completely unclear why they're necessary (especially as this patch contains no tests).

Now on to why the changes to the instruction definitions:

I implemented the memory ordering semantics for all the LSE Atomics with Intrinsics. As in:

{..}include/llvm/Intrinsics/IntrinsicsAArch64.td:

// Atomic LD<OP> Intrinsics.
def int_aarch64_ldadd_32 :
  Intrinsic<[llvm_i32_ty], [llvm_i32_ty, llvm_anyptr_ty]>;
def int_aarch64_ldadd_64 :
  Intrinsic<[llvm_i64_ty], [llvm_i64_ty, llvm_anyptr_ty]>;
def int_aarch64_ldadda_32 :
  Intrinsic<[llvm_i32_ty], [llvm_i32_ty, llvm_anyptr_ty]>;
def int_aarch64_ldadda_64 :
  Intrinsic<[llvm_i64_ty], [llvm_i64_ty, llvm_anyptr_ty]>;
def int_aarch64_ldaddl_32 :
  Intrinsic<[llvm_i32_ty], [llvm_i32_ty, llvm_anyptr_ty]>;
def int_aarch64_ldaddl_64 :
  Intrinsic<[llvm_i64_ty], [llvm_i64_ty, llvm_anyptr_ty]>;
def int_aarch64_ldaddal_32 :
  Intrinsic<[llvm_i32_ty], [llvm_i32_ty, llvm_anyptr_ty]>;
def int_aarch64_ldaddal_64 :
  Intrinsic<[llvm_i64_ty], [llvm_i64_ty, llvm_anyptr_ty]>;

[ etc etc etc ]

The instruction definition themselves (in AArch64InstrInfo.td) are changed to:

let AddedComplexity = 5, Predicates = [HasLSE] in {
 def LDADDB    : BaseLDOP<0b00, 0, 0, 0b000, "add", "", "b",
                          int_aarch64_ldadd_32, GPR32>;
 def LDADDH    : BaseLDOP<0b01, 0, 0, 0b000, "add", "", "h",
                          int_aarch64_ldadd_32, GPR32>;
 def LDADDW    : BaseLDOP<0b10, 0, 0, 0b000, "add", "", "",
                          int_aarch64_ldadd_32, GPR32>;
 def LDADDX    : BaseLDOP<0b11, 0, 0, 0b000, "add", "", "",
                          int_aarch64_ldadd_64, GPR64>;

[ etc etc etc ]

And in AArch64InstrFormats.td:

class BaseLDOP<bits<2> sz, bits<1> acq, bits<1> rel, bits<3> opc,
               string op, string order, string size,
               Intrinsic OpNode, RegisterClass RC>
  : BaseLDOPEncoding<(outs RC:$Rt),
                     (ins RC:$Rs, GPR64sp:$Rn),
                     "ld" # op # order # size,
                     "\t$Rs, $Rt, [$Rn]", "",
                     []>,
                     Sched<[WriteAtomic, WriteLD, WriteST]> {
  let Sz = sz;
  let Acq = acq;
  let Rel = rel;
  let Opc = opc;
}

In AArch64ISelLowering.cpp, each instruction lowering function will discover the correct
memory ordering model from the Intrinsic Opcode and the AtomicOrdering provided by the AtomicSDNode.

Some ISD NodeTypes can be lowered by the same function -- in this case the lowering function acts as a pure pass-through to a specific LSE Opcode. Some others need special treatment (for example ISD::ATOMIC_LOAD_SUB becomes ISD::ATOMIC_LOAD_ADD).

In AArch64ISelDAGToDAG.cpp, the correct instruction selection - with the correct memory ordering and register size - will be done by each instruction selection function, based on the corresponding Intrinsics Opcode. Just like in the instruction lowering case, several different instructions can be handled by the same instruction selection function.

This is the reason why the instruction definitions were expanded to explicitly describe each and every single instruction - as opposed to using the original multiclass design: a different Intrinsic need to be passed to the Instruction Definition depending on register size, and memory ordering. I do not think it is possible to accomplish this particular design with a multiclass.

[t.p.northover]

I implemented the memory ordering semantics for all the LSE Atomics with Intrinsics.

It's unnecessary to add intrinsics to handle this, let alone get C++ code involved. We've got exactly the ISD::AtomicRMW node we want and it already has all the information we need attached.

If you add a fragment like this to include/llvm/Target/TargetSelectionDAG.td:

multiclass binary_atomic_op_ord {
  def #NAME#_monotonic : PatFrag<(ops node:$ptr, node:$val),
        (!cast<SDNode>(#NAME) node:$ptr, node:$val), [{
      return cast<AtomicSDNode>(N)->getOrdering() == AtomicOrdering::Monotonic;
    }]>;
   [...]
}

and adapt the definition of binary_atomic_op to use it you can refer directly to things like atomic_load_add_8_acquire from TableGen. A few more multiclasses to reduce copy/paste in AArch64InstrAtomics.td along the lines of:

multiclass LDOPregister_patterns_ord_dag<string inst, string suffix, string op,
                                     string size, dag SrcRHS, dag DstRHS> {
  def : Pat<(!cast<SDNode>(op#"_"#size#"_monotonic") GPR64sp:$Rn, SrcRHS),
            (!cast<Instruction>(inst#suffix) DstRHS, GPR64sp:$Rn)>;
  [...]
}

multiclass LDOPregister_patterns<string inst, string op> {
  defm : LDOPregister_patterns_ord<inst, "d", op, "64", (i64 GPR64:$Rm)>; // 64-bit
  [...]
}

defm : LDOPregister_patterns<"LDADD", "atomic_load_add">;
[...]

and the job's a goodun.

This is the reason why the instruction definitions were expanded to explicitly describe each and every single instruction - as opposed to using the original multiclass design: a different Intrinsic need to be passed to the Instruction Definition depending on register size, and memory ordering. I do not think it is possible to accomplish this particular design with a multiclass.

Even if we wanted the intrinsic-based design (I definitely don't) it would be better to put separate patterns elsewhere than to contort the existing instruction definitions to accommodate it. And splitting the ST* instructions is still a bad idea either way.

[steleman]

In D35319#808812, @t.p.northover wrote:

It's unnecessary to add intrinsics to handle this, let alone get C++ code involved. We've got exactly the ISD::AtomicRMW node we want and it already has all the information we need attached.

OK I will look at your suggestions and see what I come up with.

[steleman]

Progress update: updated with the latest changes.

I still need to add tests for all the new atomic ops.

[t.p.northover]

Thanks very much for updating the patch. It's starting to look a lot more like how I'd expect these to be implemented.

I think you've slightly missed the benefit I was suggesting multiclasses would give so there's still more duplication than is needed. Hopefully my explanations below make sense but I can try to clarify if not.

include/llvm/Target/TargetSelectionDAG.td

1118 ↗

(On Diff #108326)

You can put these into the "multiclass binary_atomic_op"

multiclass binary_atomic_op<...> {
  defm NAME#_8 : binary_atomic_op_ord;
  [...]
}

That'll automatically instantiate the _8 ordered variants when the plain _8 is created. Obviously you need similar _16, _32, ...

1168–1171 ↗

(On Diff #108326)

Here onwards is unnecessary I believe. You're actually creating duplicate (and unused) nodes like atomic_load_add_monotonic_seq_cst, ...

The whole point of the multiclass is that you instantiate multiple variants at the same time. The multiclass takes the base name (e.g.atomic_load_add_8) and tacks on its named suffix (e.g. _monotonic) and hopefully implements the correct checks to make sure it matches the right node.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

59

Tiny nit: functions should start with a lower-case letter.

lib/Target/AArch64/AArch64InstrAtomics.td

410

You can eliminate lots of copy/paste with multiclasses here too.

// Differing SrcRHS and DstRHS allow you to cover CLR & SUB by giving a more
// complex DAG for DstRHS.
let Predicates = [HasLSE] in
multiclass LDOPregister_patterns_ord_dag<string inst, string suffix, string op,
                                         string size, dag SrcRHS, dag DstRHS> {
  def : Pat<(!cast<SDNode>(op#"_"#size#"_monotonic") GPR64sp:$Rn, SrcRHS),
            (!cast<Instruction>(inst#suffix) DstRHS, GPR64sp:$Rn)>;
  [... variants for acquire, release, acq_rel and seq_cst ...]
}

// Simple case for non-CLR, non-SUB instructions where it's just one result inst.
multiclass LDOPregister_patterns_ord<string inst, string suffix, string op,
                                         string size, dag RHS> {
  defm : LDOPregister_patterns_ord_dag<inst, suffix, op, size, RHS, RHS>;
}

multiclass LDOPregister_patterns<string inst, string op> {
  defm : LDOPregister_patterns_ord<inst, "d", op, "64", (i64 GPR64:$Rm)>;
  defm : LDOPregister_patterns_ord<inst, "s", op, "32", (i32 GPR32:$Rm)>;
  defm : LDOPregister_patterns_ord<inst, "h", op, "16", (i32 GPR32:$Rm)>;
  defm : LDOPregister_patterns_ord<inst, "b", op, "8",  (i32 GPR32:$Rm)>;
}

defm : LDOPregister_patterns<"LDADD", "atomic_load_add">;
[... all other ops except SUB/CLR ...]

// Then slightly more complex version of LDOPregister_patterns to handle CLR/SUB and a pair
// of defms for them. About 8 lines of code.

lib/Target/AArch64/AArch64InstrFormats.td

9401

I'm fine with this change, but it should probably be committed before as a separate "NFC" rename. In fact, feel free to do that any time you like.

9517

These STOP patterns appear to be unused.

lib/Target/AArch64/AArch64SchedThunderX2T99.td

318–319

Separate commit for the scheduling changes please. But as far as I'm concerned you can go ahead whenever you want, as with the renaming.

[steleman]

lib/Target/AArch64/AArch64InstrFormats.td

9401

Won't this complicate things, though?

If I commit this change as a separate - and "before" change, I'll still have to change all the instruction mnemonics in the existing Pats, and in the AArch64DeadRegister blacklist.

It seems like a lot of code churn.

9517

Yes, they are unused for now. But I plan on adding the ST<OP> instructions as well.

They need ISD NodeTypes, though, as ISD::ATOMIC_STORE_ADD, ISD::ATOMIC_STORE_SUB,
etc, aren't defined in include/llvm/CodeGen/ISDOpcodes.h. Which is why I'm saving
this change for a subsequent changeset.

[t.p.northover]

lib/Target/AArch64/AArch64InstrFormats.td

9401

OK, I won't insist.

9517

Then that would be a separate patch.

Fair warning: I strongly suspect it's unnecessary and should be automatically handled by the DeadDefinitions pass. ATOMIC_STORE_ADD is nothing but a normal ATOMIC_LOAD_ADD where the definition is dead, which is exactly what that pass is supposed to handle.

[steleman]

It does not appear that the multiclass design you are advocating here does what we'd expect it to do.

For one of the simple LD<OP> atomics - namely LDADD - I have placed a test program with corresponding output here:

1. lseadd.c - https://drive.google.com/open?id=0B9ulXWgBcdxxcDNmNU1yVnQ5aXM
2. lseadd.sh - https://drive.google.com/open?id=0B9ulXWgBcdxxTHI2MXd6dTZ0aVk
3. lseadd-clang-O2.S - https://drive.google.com/open?id=0B9ulXWgBcdxxUW0yVEJCeE9mUTQ
4. lseadd-clang-O2.ll - https://drive.google.com/open?id=0B9ulXWgBcdxxOTZRNjl6b1drVGM

The contents of these files is pretty self-explanatory.

Description of the problem: regardless of the atomic ordering model chosen in the test program -- in this case it's __ATOMIC_SEQ_CST -- TableGen will always match to ldadd, which is incorrect, instead of ldaddal, which is what would be expected in this case. This is shown in the lseadd-clang-O2.S assembler file indicated above.

However, the LLVM IR output matches the atomic ordering correctly.

The problem disappears and the matching is performed correctly if the Pats are explicitly written in the AArch64InstrAtomics.td file.

[steleman]

I have uploaded the LLVM patch that exhibits this problem here:

https://drive.google.com/open?id=0B9ulXWgBcdxxYnN3LXNHZndaWHM

[steleman]

Updated changeset to use LDOPregister_patterns.
Implements all the LD<OP> LSE Atomics with the exception of NAND.
Updated test in atomic-ops.lse to cover all memory ordering models.

[t.p.northover]

Thanks for going through so many revisions on this. I think it looks good now.

[christof]

Thanks for the hard work. It looks good. Quite an exhaustive list of tests as well.

[steleman]

In D35319#831965, @christof wrote:

Thanks for the hard work. It looks good. Quite an exhaustive list of tests as well.

Thank you.

I wanted to make certain all the memory models work correctly, so that's the reason
for the very large number of tests. :-)

[steleman]

Restored definition for defm atomic_load_nand in TargetSelectionDAG.td,
which I had removed by mistake.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

   void processMachineBasicBlock(MachineBasicBlock &MBB);
 public:
   static char ID; // Pass identification, replacement for typeid.
   AArch64DeadRegisterDefinitions() : MachineFunctionPass(ID) {
     initializeAArch64DeadRegisterDefinitionsPass(
         *PassRegistry::getPassRegistry());
   }
 
   bool runOnMachineFunction(MachineFunction &F) override;
 
   StringRef getPassName() const override { return AARCH64_DEAD_REG_DEF_NAME; }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
     MachineFunctionPass::getAnalysisUsage(AU);
   }
+
+  bool ShouldSkip(const MachineInstr &MI, const MachineFunction &MF) const;
 };
 char AArch64DeadRegisterDefinitions::ID = 0;
 } // end anonymous namespace
 
 INITIALIZE_PASS(AArch64DeadRegisterDefinitions, "aarch64-dead-defs",
                 AARCH64_DEAD_REG_DEF_NAME, false, false)
 
 static bool usesFrameIndex(const MachineInstr &MI) {
   for (const MachineOperand &MO : MI.uses())
     if (MO.isFI())
       return true;
   return false;
 }
 
+bool
+AArch64DeadRegisterDefinitions::ShouldSkip(const MachineInstr &MI,
+                                           const MachineFunction &MF) const {
+  if (!MF.getSubtarget<AArch64Subtarget>().hasLSE())
+    return false;
+
+#define CASE_AARCH64_ATOMIC_(PREFIX) \
+  case AArch64::PREFIX##X: \
+  case AArch64::PREFIX##W: \
+  case AArch64::PREFIX##H: \
+  case AArch64::PREFIX##B
+
+  // ARM Errata D11904 05/12/2017:
+  // The Acquire semantics are present when (A == 1 && Rt != 11111).
+  // The Release semantics are present when (R == 1).
+  for (const MachineMemOperand *MMO : MI.memoperands()) {
+    if (MMO->isAtomic()) {
+      unsigned Opcode = MI.getOpcode();
+      switch (Opcode) {
+      default:
+        return false;
+        break;
+
+      CASE_AARCH64_ATOMIC_(LDADDA):
+      CASE_AARCH64_ATOMIC_(LDADDAL):
+
+      CASE_AARCH64_ATOMIC_(LDCLRA):
+      CASE_AARCH64_ATOMIC_(LDCLRAL):
+
+      CASE_AARCH64_ATOMIC_(LDEORA):
+      CASE_AARCH64_ATOMIC_(LDEORAL):
+
+      CASE_AARCH64_ATOMIC_(LDSETA):
+      CASE_AARCH64_ATOMIC_(LDSETAL):
+
+      CASE_AARCH64_ATOMIC_(LDSMAXA):
+      CASE_AARCH64_ATOMIC_(LDSMAXAL):
+
+      CASE_AARCH64_ATOMIC_(LDSMINA):
+      CASE_AARCH64_ATOMIC_(LDSMINAL):
+
+      CASE_AARCH64_ATOMIC_(LDUMAXA):
+      CASE_AARCH64_ATOMIC_(LDUMAXAL):
+
+      CASE_AARCH64_ATOMIC_(LDUMINA):
+      CASE_AARCH64_ATOMIC_(LDUMINAL):
+
+      CASE_AARCH64_ATOMIC_(SWPA):
+      CASE_AARCH64_ATOMIC_(SWPAL):
+        return true;
+        break;
+      }
+    }
+  }
+
+#undef CASE_AARCH64_ATOMIC_
+
+  return false;
+}
+
+
+
 void AArch64DeadRegisterDefinitions::processMachineBasicBlock(
     MachineBasicBlock &MBB) {
   const MachineFunction &MF = *MBB.getParent();
   for (MachineInstr &MI : MBB) {
     if (usesFrameIndex(MI)) {
       // We need to skip this instruction because while it appears to have a
       // dead def it uses a frame index which might expand into a multi
       // instruction sequence during EPI.
       DEBUG(dbgs() << "    Ignoring, operand is frame index\n");
       continue;
     }
     if (MI.definesRegister(AArch64::XZR) || MI.definesRegister(AArch64::WZR)) {
       // It is not allowed to write to the same register (not even the zero
       // register) twice in a single instruction.
       DEBUG(dbgs() << "    Ignoring, XZR or WZR already used by the instruction\n");
       continue;
     }
-    if (MF.getSubtarget<AArch64Subtarget>().hasLSE()) {
-      // XZ/WZ for LSE can only be used when acquire semantics are not used,
-      // LDOPAL WZ is an invalid opcode.
-      switch (MI.getOpcode()) {
-      case AArch64::CASALb:
-      case AArch64::CASALh:
-      case AArch64::CASALs:
-      case AArch64::CASALd:
-      case AArch64::SWPALb:
-      case AArch64::SWPALh:
-      case AArch64::SWPALs:
-      case AArch64::SWPALd:
-      case AArch64::LDADDALb:
-      case AArch64::LDADDALh:
-      case AArch64::LDADDALs:
-      case AArch64::LDADDALd:
-      case AArch64::LDEORALb:
-      case AArch64::LDEORALh:
-      case AArch64::LDEORALs:
-      case AArch64::LDEORALd:
-      case AArch64::LDSETALb:
-      case AArch64::LDSETALh:
-      case AArch64::LDSETALs:
-      case AArch64::LDSETALd:
-      case AArch64::LDSMINALb:
-      case AArch64::LDSMINALh:
-      case AArch64::LDSMINALs:
-      case AArch64::LDSMINALd:
-      case AArch64::LDSMAXALb:
-      case AArch64::LDSMAXALh:
-      case AArch64::LDSMAXALs:
-      case AArch64::LDSMAXALd:
-      case AArch64::LDUMINALb:
-      case AArch64::LDUMINALh:
-      case AArch64::LDUMINALs:
-      case AArch64::LDUMINALd:
-      case AArch64::LDUMAXALb:
-      case AArch64::LDUMAXALh:
-      case AArch64::LDUMAXALs:
-      case AArch64::LDUMAXALd:
-        continue;
-      default:
-        break;
-      }
+
+    if (ShouldSkip(MI, MF)) {
+      DEBUG(dbgs() << "    Ignoring, Atomic instruction with acquire semantics using WZR/XZR\n");
+      continue;
     }
+
     const MCInstrDesc &Desc = MI.getDesc();
     for (int I = 0, E = Desc.getNumDefs(); I != E; ++I) {
       MachineOperand &MO = MI.getOperand(I);
       if (!MO.isReg() || !MO.isDef())
         continue;
       // We should not have any relevant physreg defs that are replacable by
       // zero before register allocation. So we just check for dead vreg defs.
       unsigned Reg = MO.getReg();
       if (!TargetRegisterInfo::isVirtualRegister(Reg) ||
           (!MO.isDead() && !MRI->use_nodbg_empty(Reg)))
         continue;
       assert(!MO.isImplicit() && "Unexpected implicit def!");
       DEBUG(dbgs() << "  Dead def operand #" << I << " in:\n    ";
             MI.print(dbgs()));
       // Be careful not to change the register if it's a tied operand.
       if (MI.isRegTiedToUseOperand(I)) {

lib/Target/AArch64/AArch64InstrAtomics.td

                          (ins GPR64:$addr, GPR32:$desired, GPR32:$new), []>,
                   Sched<[WriteAtomic]>;
 
 def CMP_SWAP_64 : Pseudo<(outs GPR64:$Rd, GPR32:$scratch),
                          (ins GPR64:$addr, GPR64:$desired, GPR64:$new), []>,
                   Sched<[WriteAtomic]>;
 }
 
 let Constraints = "@earlyclobber $RdLo,@earlyclobber $RdHi,@earlyclobber $scratch",
     mayLoad = 1, mayStore = 1 in
 def CMP_SWAP_128 : Pseudo<(outs GPR64:$RdLo, GPR64:$RdHi, GPR32:$scratch),
                           (ins GPR64:$addr, GPR64:$desiredLo, GPR64:$desiredHi,
                                GPR64:$newLo, GPR64:$newHi), []>,
                    Sched<[WriteAtomic]>;
 
 // v8.1 Atomic instructions:
-def : Pat<(atomic_load_add_8 GPR64:$Rn, GPR32:$Rs), (LDADDALb GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_add_16 GPR64:$Rn, GPR32:$Rs), (LDADDALh GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_add_32 GPR64:$Rn, GPR32:$Rs), (LDADDALs GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_add_64 GPR64:$Rn, GPR64:$Rs), (LDADDALd GPR64:$Rs, GPR64sp:$Rn)>;
-
-def : Pat<(atomic_load_or_8 GPR64:$Rn, GPR32:$Rs), (LDSETALb GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_or_16 GPR64:$Rn, GPR32:$Rs), (LDSETALh GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_or_32 GPR64:$Rn, GPR32:$Rs), (LDSETALs GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_or_64 GPR64:$Rn, GPR64:$Rs), (LDSETALd GPR64:$Rs, GPR64sp:$Rn)>;
-
-def : Pat<(atomic_load_xor_8 GPR64:$Rn, GPR32:$Rs), (LDEORALb GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_xor_16 GPR64:$Rn, GPR32:$Rs), (LDEORALh GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_xor_32 GPR64:$Rn, GPR32:$Rs), (LDEORALs GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_xor_64 GPR64:$Rn, GPR64:$Rs), (LDEORALd GPR64:$Rs, GPR64sp:$Rn)>;
-
-def : Pat<(atomic_load_max_8 GPR64:$Rn, GPR32:$Rs), (LDSMAXALb GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_max_16 GPR64:$Rn, GPR32:$Rs), (LDSMAXALh GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_max_32 GPR64:$Rn, GPR32:$Rs), (LDSMAXALs GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_max_64 GPR64:$Rn, GPR64:$Rs), (LDSMAXALd GPR64:$Rs, GPR64sp:$Rn)>;
-
-def : Pat<(atomic_load_umax_8 GPR64:$Rn, GPR32:$Rs), (LDUMAXALb GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_umax_16 GPR64:$Rn, GPR32:$Rs), (LDUMAXALh GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_umax_32 GPR64:$Rn, GPR32:$Rs), (LDUMAXALs GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_umax_64 GPR64:$Rn, GPR64:$Rs), (LDUMAXALd GPR64:$Rs, GPR64sp:$Rn)>;
-
-def : Pat<(atomic_load_min_8 GPR64:$Rn, GPR32:$Rs), (LDSMINALb GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_min_16 GPR64:$Rn, GPR32:$Rs), (LDSMINALh GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_min_32 GPR64:$Rn, GPR32:$Rs), (LDSMINALs GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_min_64 GPR64:$Rn, GPR64:$Rs), (LDSMINALd GPR64:$Rs, GPR64sp:$Rn)>;
-
-def : Pat<(atomic_load_umin_8 GPR64:$Rn, GPR32:$Rs), (LDUMINALb GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_umin_16 GPR64:$Rn, GPR32:$Rs), (LDUMINALh GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_umin_32 GPR64:$Rn, GPR32:$Rs), (LDUMINALs GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_load_umin_64 GPR64:$Rn, GPR64:$Rs), (LDUMINALd GPR64:$Rs, GPR64sp:$Rn)>;
-
-def : Pat<(atomic_cmp_swap_8 GPR64:$Rn, GPR32:$Rold, GPR32:$Rnew), (CASALb GPR32:$Rold, GPR32:$Rnew, GPR64sp:$Rn)>;
-def : Pat<(atomic_cmp_swap_16 GPR64:$Rn, GPR32:$Rold, GPR32:$Rnew), (CASALh GPR32:$Rold, GPR32:$Rnew, GPR64sp:$Rn)>;
-def : Pat<(atomic_cmp_swap_32 GPR64:$Rn, GPR32:$Rold, GPR32:$Rnew), (CASALs GPR32:$Rold, GPR32:$Rnew, GPR64sp:$Rn)>;
-def : Pat<(atomic_cmp_swap_64 GPR64:$Rn, GPR64:$Rold, GPR64:$Rnew), (CASALd GPR64:$Rold, GPR64:$Rnew, GPR64sp:$Rn)>;
-
-def : Pat<(atomic_swap_8 GPR64:$Rn, GPR32:$Rs), (SWPALb GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_swap_16 GPR64:$Rn, GPR32:$Rs), (SWPALh GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_swap_32 GPR64:$Rn, GPR32:$Rs), (SWPALs GPR32:$Rs, GPR64sp:$Rn)>;
-def : Pat<(atomic_swap_64 GPR64:$Rn, GPR64:$Rs), (SWPALd GPR64:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_add_8 GPR64:$Rn, GPR32:$Rs), (LDADDALB GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_add_16 GPR64:$Rn, GPR32:$Rs), (LDADDALH GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_add_32 GPR64:$Rn, GPR32:$Rs), (LDADDALW GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_add_64 GPR64:$Rn, GPR64:$Rs), (LDADDALX GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_or_8 GPR64:$Rn, GPR32:$Rs), (LDSETALB GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_or_16 GPR64:$Rn, GPR32:$Rs), (LDSETALH GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_or_32 GPR64:$Rn, GPR32:$Rs), (LDSETALW GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_or_64 GPR64:$Rn, GPR64:$Rs), (LDSETALX GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_xor_8 GPR64:$Rn, GPR32:$Rs), (LDEORALB GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_xor_16 GPR64:$Rn, GPR32:$Rs), (LDEORALH GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_xor_32 GPR64:$Rn, GPR32:$Rs), (LDEORALW GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_xor_64 GPR64:$Rn, GPR64:$Rs), (LDEORALX GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_max_8 GPR64:$Rn, GPR32:$Rs), (LDSMAXALB GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_max_16 GPR64:$Rn, GPR32:$Rs), (LDSMAXALH GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_max_32 GPR64:$Rn, GPR32:$Rs), (LDSMAXALW GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_max_64 GPR64:$Rn, GPR64:$Rs), (LDSMAXALX GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_umax_8 GPR64:$Rn, GPR32:$Rs), (LDUMAXALB GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umax_16 GPR64:$Rn, GPR32:$Rs), (LDUMAXALH GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umax_32 GPR64:$Rn, GPR32:$Rs), (LDUMAXALW GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umax_64 GPR64:$Rn, GPR64:$Rs), (LDUMAXALX GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_min_8 GPR64:$Rn, GPR32:$Rs), (LDSMINALB GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_min_16 GPR64:$Rn, GPR32:$Rs), (LDSMINALH GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_min_32 GPR64:$Rn, GPR32:$Rs), (LDSMINALW GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_min_64 GPR64:$Rn, GPR64:$Rs), (LDSMINALX GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_umin_8 GPR64:$Rn, GPR32:$Rs), (LDUMINALB GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umin_16 GPR64:$Rn, GPR32:$Rs), (LDUMINALH GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umin_32 GPR64:$Rn, GPR32:$Rs), (LDUMINALW GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umin_64 GPR64:$Rn, GPR64:$Rs), (LDUMINALX GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_cmp_swap_8 GPR64:$Rn, GPR32:$Rold, GPR32:$Rnew), (CASALB GPR32:$Rold, GPR32:$Rnew, GPR64sp:$Rn)>;
+def : Pat<(atomic_cmp_swap_16 GPR64:$Rn, GPR32:$Rold, GPR32:$Rnew), (CASALH GPR32:$Rold, GPR32:$Rnew, GPR64sp:$Rn)>;
+def : Pat<(atomic_cmp_swap_32 GPR64:$Rn, GPR32:$Rold, GPR32:$Rnew), (CASALW GPR32:$Rold, GPR32:$Rnew, GPR64sp:$Rn)>;
+def : Pat<(atomic_cmp_swap_64 GPR64:$Rn, GPR64:$Rold, GPR64:$Rnew), (CASALX GPR64:$Rold, GPR64:$Rnew, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_swap_8 GPR64:$Rn, GPR32:$Rs), (SWPALB GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_swap_16 GPR64:$Rn, GPR32:$Rs), (SWPALH GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_swap_32 GPR64:$Rn, GPR32:$Rs), (SWPALW GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_swap_64 GPR64:$Rn, GPR64:$Rs), (SWPALX GPR64:$Rs, GPR64sp:$Rn)>;

lib/Target/AArch64/AArch64InstrFormats.td

 // LD   SZ   |111000|A |R |1 |Rs   |0 |OPC|00|Rn |Rt
 // ST   SZ   |111000|A |R |1 |Rs   |0 |OPC|00|Rn |11111
 
 // Instruction syntax:
 //
 // CAS{<order>}[<size>] <Ws>, <Wt>, [<Xn|SP>]
 // CAS{<order>} <Xs>, <Xt>, [<Xn|SP>]
 // CASP{<order>} <Ws>, <W(s+1)>, <Wt>, <W(t+1)>, [<Xn|SP>]
 // CASP{<order>} <Xs>, <X(s+1)>, <Xt>, <X(t+1)>, [<Xn|SP>]
 // SWP{<order>}[<size>] <Ws>, <Wt>, [<Xn|SP>]
 // SWP{<order>} <Xs>, <Xt>, [<Xn|SP>]
 // LD<OP>{<order>}[<size>] <Ws>, <Wt>, [<Xn|SP>]
 // LD<OP>{<order>} <Xs>, <Xt>, [<Xn|SP>]
 // ST<OP>{<order>}[<size>] <Ws>, [<Xn|SP>]
 // ST<OP>{<order>} <Xs>, [<Xn|SP>]
 
-let Predicates = [HasLSE], mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
+let Predicates = [HasLSE],
+  mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
 class BaseCASEncoding<dag oops, dag iops, string asm, string operands,
                       string cstr, list<dag> pattern>
-      : I<oops, iops, asm, operands, cstr, pattern> {
   bits<2> Sz;
   bit NP;
   bit Acq;
   bit Rel;
   bits<5> Rs;
   bits<5> Rn;
   bits<5> Rt;
   let Inst{31-30} = Sz;
   let Inst{29-24} = 0b001000;
   let Inst{23} = NP;
   let Inst{22} = Acq;
   let Inst{21} = 0b1;
   let Inst{20-16} = Rs;
   let Inst{15} = Rel;
   let Inst{14-10} = 0b11111;
   let Inst{9-5} = Rn;
   let Inst{4-0} = Rt;
   let Predicates = [HasLSE];
 }
 
-class BaseCAS<string order, string size, RegisterClass RC>
-      : BaseCASEncoding<(outs RC:$out),(ins RC:$Rs, RC:$Rt, GPR64sp:$Rn),
-                        "cas" # order # size, "\t$Rs, $Rt, [$Rn]",
-                        "$out = $Rs",[]>,
-        Sched<[WriteAtomic]> {
+let Predicates = [HasLSE], mayLoad = 1, mayStore = 1,
+  hasSideEffects = 1 in
+class BaseCAS<bits<2> sz, bits<1> acq, bits<1> rel, string order,
+              string size, RegisterClass RC>
+  : BaseCASEncoding<(outs RC:$out), (ins RC:$Rs, RC:$Rt, GPR64sp:$Rn),
+                    "cas" # order # size, "\t$Rs, $Rt, [$Rn]", "$out = $Rs",
+                    []>, Sched<[WriteAtomic, WriteLD, WriteST]> {
+  let Sz = sz;
+  let Acq = acq;
+  let Rel = rel;
   let NP = 1;
 }
 
-multiclass CompareAndSwap<bits<1> Acq, bits<1> Rel, string order> {
-  let Sz = 0b00, Acq = Acq, Rel = Rel in def b : BaseCAS<order, "b", GPR32>;
-  let Sz = 0b01, Acq = Acq, Rel = Rel in def h : BaseCAS<order, "h", GPR32>;
-  let Sz = 0b10, Acq = Acq, Rel = Rel in def s : BaseCAS<order, "", GPR32>;
-  let Sz = 0b11, Acq = Acq, Rel = Rel in def d : BaseCAS<order, "", GPR64>;
-}
-
-class BaseCASP<string order, string size, RegisterOperand RC>
-      : BaseCASEncoding<(outs RC:$out),(ins RC:$Rs, RC:$Rt, GPR64sp:$Rn),
-                        "casp" # order # size, "\t$Rs, $Rt, [$Rn]",
-                        "$out = $Rs",[]>,
-        Sched<[WriteAtomic]> {
+let Predicates = [HasLSE], mayLoad = 1, mayStore = 1,
+  hasSideEffects = 1 in
+class BaseCASP<bits<2> sz, bits<1> acq, bits<1> rel, string order,
+               string size, RegisterOperand RC>
+  : BaseCASEncoding<(outs RC:$out),
+                    (ins RC:$Rs, RC:$Rt, GPR64sp:$Rn),
+                     "casp" # order # size, "\t$Rs, $Rt, [$Rn]",
+                     "$out = $Rs",
+                     []>, Sched<[WriteAtomic, WriteLD, WriteST]> {
+  let Sz = sz;
+  let Acq = acq;
+  let Rel = rel;
   let NP = 0;
 }
 
-multiclass CompareAndSwapPair<bits<1> Acq, bits<1> Rel, string order> {
-  let Sz = 0b00, Acq = Acq, Rel = Rel in 
-    def s : BaseCASP<order, "", WSeqPairClassOperand>;
-  let Sz = 0b01, Acq = Acq, Rel = Rel in 
-    def d : BaseCASP<order, "", XSeqPairClassOperand>;
-}
-
-let Predicates = [HasLSE] in
-class BaseSWP<string order, string size, RegisterClass RC>
-      : I<(outs RC:$Rt),(ins RC:$Rs, GPR64sp:$Rn), "swp" # order # size,
-          "\t$Rs, $Rt, [$Rn]","",[]>,
-        Sched<[WriteAtomic]> {
+let Predicates = [HasLSE],
+  mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
+class BaseSWPEncoding<dag oops, dag iops, string asm, string operands,
+                      string cstr, list<dag> pattern>
+  : I<oops, iops, asm, operands, cstr, pattern> {
   bits<2> Sz;
   bit Acq;
   bit Rel;
   bits<5> Rs;
   bits<3> opc = 0b000;
   bits<5> Rn;
   bits<5> Rt;
   let Inst{31-30} = Sz;
   let Inst{29-24} = 0b111000;
   let Inst{23} = Acq;
   let Inst{22} = Rel;
   let Inst{21} = 0b1;
   let Inst{20-16} = Rs;
   let Inst{15} = 0b1;
   let Inst{14-12} = opc;
   let Inst{11-10} = 0b00;
   let Inst{9-5} = Rn;
   let Inst{4-0} = Rt;
   let Predicates = [HasLSE];
 }
 
-multiclass Swap<bits<1> Acq, bits<1> Rel, string order> {
-  let Sz = 0b00, Acq = Acq, Rel = Rel in def b : BaseSWP<order, "b", GPR32>;
-  let Sz = 0b01, Acq = Acq, Rel = Rel in def h : BaseSWP<order, "h", GPR32>;
-  let Sz = 0b10, Acq = Acq, Rel = Rel in def s : BaseSWP<order, "", GPR32>;
-  let Sz = 0b11, Acq = Acq, Rel = Rel in def d : BaseSWP<order, "", GPR64>;
+let Predicates = [HasLSE], mayStore = 1, hasSideEffects = 1 in
+class BaseSWP<bits<2> sz, bit acq, bit rel, string order,
+              string size, RegisterClass RC>
+  : BaseSWPEncoding<(outs RC:$Rt),
+                    (ins RC:$Rs, GPR64sp:$Rn),
+                    "swp" # order # size,
+                    "\t$Rs, $Rt, [$Rn]", "",
+                    []>, Sched<[WriteAtomic, WriteLD, WriteST]> {
+  let Sz = sz;
+  let Acq = acq;
+  let Rel = rel;
+  let Constraints = "@earlyclobber $Rt";
 }
 
 let Predicates = [HasLSE], mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
-class BaseLDOPregister<string op, string order, string size, RegisterClass RC>
-      : I<(outs RC:$Rt),(ins RC:$Rs, GPR64sp:$Rn), "ld" # op # order # size,
-          "\t$Rs, $Rt, [$Rn]","",[]>,
-        Sched<[WriteAtomic]> {
+class BaseLDOPEncoding<dag oops, dag iops, string asm, string operands,
+                       string cstr, list<dag> pattern>
+  : I<oops, iops, asm, operands, cstr, pattern> {
   bits<2> Sz;
   bit Acq;
   bit Rel;
   bits<5> Rs;
-  bits<3> opc;
+  bits<3> Opc;
   bits<5> Rn;
   bits<5> Rt;
   let Inst{31-30} = Sz;
   let Inst{29-24} = 0b111000;
   let Inst{23} = Acq;
   let Inst{22} = Rel;
   let Inst{21} = 0b1;
   let Inst{20-16} = Rs;
   let Inst{15} = 0b0;
-  let Inst{14-12} = opc;
+  let Inst{14-12} = Opc;
   let Inst{11-10} = 0b00;
   let Inst{9-5} = Rn;
   let Inst{4-0} = Rt;
   let Predicates = [HasLSE];
 }
 
-multiclass LDOPregister<bits<3> opc, string op, bits<1> Acq, bits<1> Rel, 
-                        string order> {
-  let Sz = 0b00, Acq = Acq, Rel = Rel, opc = opc in 
-    def b : BaseLDOPregister<op, order, "b", GPR32>;
-  let Sz = 0b01, Acq = Acq, Rel = Rel, opc = opc in 
-    def h : BaseLDOPregister<op, order, "h", GPR32>;
-  let Sz = 0b10, Acq = Acq, Rel = Rel, opc = opc in 
-    def s : BaseLDOPregister<op, order, "", GPR32>;
-  let Sz = 0b11, Acq = Acq, Rel = Rel, opc = opc in 
-    def d : BaseLDOPregister<op, order, "", GPR64>;
-}
-
-let Predicates = [HasLSE] in
-class BaseSTOPregister<string asm, RegisterClass OP, Register Reg,
-                        Instruction inst> :
-      InstAlias<asm # "\t$Rs, [$Rn]", (inst Reg, OP:$Rs, GPR64sp:$Rn)>;
-
-multiclass STOPregister<string asm, string instr> {
-  def : BaseSTOPregister<asm # "lb", GPR32, WZR, 
-                    !cast<Instruction>(instr # "Lb")>;
-  def : BaseSTOPregister<asm # "lh", GPR32, WZR, 
-                    !cast<Instruction>(instr # "Lh")>;
-  def : BaseSTOPregister<asm # "l",  GPR32, WZR, 
-                    !cast<Instruction>(instr # "Ls")>;
-  def : BaseSTOPregister<asm # "l",  GPR64, XZR, 
-                    !cast<Instruction>(instr # "Ld")>;
-  def : BaseSTOPregister<asm # "b",  GPR32, WZR, 
-                    !cast<Instruction>(instr # "b")>;
-  def : BaseSTOPregister<asm # "h",  GPR32, WZR, 
-                    !cast<Instruction>(instr # "h")>;
-  def : BaseSTOPregister<asm,        GPR32, WZR, 
-                    !cast<Instruction>(instr # "s")>;
-  def : BaseSTOPregister<asm,        GPR64, XZR, 
-                    !cast<Instruction>(instr # "d")>;
+let Predicates = [HasLSE], mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
+class BaseLDOP<bits<2> sz, bit acq, bit rel, bits<3> opc,
+               string op, string order, string size,
+               RegisterClass RC>
+  : BaseLDOPEncoding<(outs RC:$Rt),
+                     (ins RC:$Rs, GPR64sp:$Rn),
+                     "ld" # op # order # size,
+                     "\t$Rs, $Rt, [$Rn]", "",
+                     []>, Sched<[WriteAtomic, WriteLD, WriteST]> {
+  let Sz = sz;
+  let Acq = acq;
+  let Rel = rel;
+  let Opc = opc;
 }
 
+let Predicates = [HasLSE], mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
+class BaseSTOPEncoding<dag oops, dag iops, string asm, string operands,
+                       string cstr, list<dag> pattern>
+  : I<oops, iops, asm, operands, cstr, pattern> {
+  bits<2> Sz;
+  bit Acq;
+  bit Rel;
+  bits<5> Rs;
+  bits<3> Opc;
+  bits<5> Rn;
+  bits<5> Rt;
+  let Inst{31-30} = Sz;
+  let Inst{29-24} = 0b111000;
+  let Inst{23} = Acq;
+  let Inst{22} = Rel;
+  let Inst{21} = 0b1;
+  let Inst{20-16} = Rs;
+  let Inst{15} = 0b0;
+  let Inst{14-12} = Opc;
+  let Inst{11-10} = 0b00;
+  let Inst{9-5} = Rn;
+  let Inst{4-0} = Rt;
+  let Predicates = [HasLSE];
+}
+
+let Predicates = [HasLSE], mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
+class BaseSTOP<bits<2> sz, bits<1> acq, bits<1> rel, bits<3> opc,
+               bits<5> rt, string op, string order, string size,
+               RegisterClass RC>
+  : BaseSTOPEncoding<(outs),
+                     (ins RC:$Rs, GPR64sp:$Rn),
+                     "st" # op # order # size,
+                     "\t$Rs, [$Rn]", "",
+                     []>, Sched<[WriteAtomic, WriteLD, WriteST]> {
+  let Sz = sz;
+  let Acq = acq;
+  let Rel = rel;
+  let Opc = opc;
+  let Rt = rt;
+}
+
+let Predicates = [HasLSE], mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
+class BaseSTLOEncoding<dag oops, dag iops, string asm, string operands,
+                       string cstr, list<dag> pattern>
+  : I<oops, iops, asm, operands, cstr, pattern> {
+  bits<2> Sz;
+  bit Lo;
+  bit Oo;
+  bits<5> Rs;
+  bits<5> Rn;
+  bits<5> Rt;
+  bits<5> Rt2;
+  let Inst{31-30} = Sz;
+  let Inst{29-24} = 0b001000;
+  let Inst{23} = 0b1;
+  let Inst{22} = Lo;
+  let Inst{21} = 0b0;
+  let Inst{20-16} = Rs;
+  let Inst{15} = Oo;
+  let Inst{14-10} = Rt2;
+  let Inst{9-5} = Rn;
+  let Inst{4-0} = Rt;
+  let Predicates = [HasLSE];
+}
+
+let Predicates = [HasLSE], mayLoad = 1, mayStore = 1, hasSideEffects = 1 in
+class BaseSTLO<bits<2> sz, bits<1> lo, bits<1> oo, bits<5> rs,
+               bits<5> rt2, string order,
+               RegisterClass RC>
+  : BaseSTLOEncoding<(outs), (ins RC:$Rs, GPR64sp:$Rn),
+                      "stllr" # order,
+                      "\t$Rs [$Rn]", "",
+                      []>, Sched<[WriteAtomic, WriteLD, WriteST]> {
+  let Sz = sz;
+  let Lo = lo;
+  let Oo = oo;
+  let Rs = rs;
+  let Rt2 = rt2;
+}
+
+class BaseNOPEncoding<dag oops, dag iops, string asm, string operands,
+                      string cstr, list<dag> pattern>
+  : I<oops, iops, asm, operands, cstr, pattern> {
+  let Inst{31-22} = 0b1101010100;
+  let Inst{21} =  0b0;
+  let Inst{20-19} = 0b00;
+  let Inst{18-16} = 0b011;
+  let Inst{15-12} = 0b0010;
+  let Inst{11-8} = 0b0000;
+  let Inst{7-5} = 0b000;
+  let Inst{4-0} = 0b11111;
+}
+
+class BaseNOP : BaseNOPEncoding<(outs), (ins), "nop", "", "", []>,
+                                Sched<[]> { }
+
 //----------------------------------------------------------------------------
 // Allow the size specifier tokens to be upper case, not just lower.
 def : TokenAlias<".8B", ".8b">;
 def : TokenAlias<".4H", ".4h">;
 def : TokenAlias<".2S", ".2s">;
 def : TokenAlias<".1D", ".1d">;
 def : TokenAlias<".16B", ".16b">;
 def : TokenAlias<".8H", ".8h">;
 def : TokenAlias<".4S", ".4s">;
 def : TokenAlias<".2D", ".2d">;
 def : TokenAlias<".1Q", ".1q">;
 def : TokenAlias<".2H", ".2h">;
 def : TokenAlias<".B", ".b">;
 def : TokenAlias<".H", ".h">;
 def : TokenAlias<".S", ".s">;
 def : TokenAlias<".D", ".d">;

lib/Target/AArch64/AArch64InstrInfo.td

 def AArch64smull    : SDNode<"AArch64ISD::SMULL", SDT_AArch64mull>;
 def AArch64umull    : SDNode<"AArch64ISD::UMULL", SDT_AArch64mull>;
 
 def AArch64frecpe   : SDNode<"AArch64ISD::FRECPE", SDTFPUnaryOp>;
 def AArch64frecps   : SDNode<"AArch64ISD::FRECPS", SDTFPBinOp>;
 def AArch64frsqrte  : SDNode<"AArch64ISD::FRSQRTE", SDTFPUnaryOp>;
 def AArch64frsqrts  : SDNode<"AArch64ISD::FRSQRTS", SDTFPBinOp>;
 
 def AArch64saddv    : SDNode<"AArch64ISD::SADDV", SDT_AArch64UnaryVec>;
 def AArch64uaddv    : SDNode<"AArch64ISD::UADDV", SDT_AArch64UnaryVec>;
 def AArch64sminv    : SDNode<"AArch64ISD::SMINV", SDT_AArch64UnaryVec>;
 def AArch64uminv    : SDNode<"AArch64ISD::UMINV", SDT_AArch64UnaryVec>;
 def AArch64smaxv    : SDNode<"AArch64ISD::SMAXV", SDT_AArch64UnaryVec>;
 def AArch64umaxv    : SDNode<"AArch64ISD::UMAXV", SDT_AArch64UnaryVec>;
 
 //===----------------------------------------------------------------------===//
-
 //===----------------------------------------------------------------------===//
 
 // AArch64 Instruction Predicate Definitions.
 def IsDarwin  : Predicate<"Subtarget->isTargetDarwin()">;
 def IsNotDarwin: Predicate<"!Subtarget->isTargetDarwin()">;
-
 // We could compute these on a per-module basis but doing so requires accessing
 // the Function object through the <Target>Subtarget and objections were raised
 // to that (see post-commit review comments for r301750).
 let RecomputePerFunction = 1 in {
   def ForCodeSize   : Predicate<"MF->getFunction()->optForSize()">;
   def NotForCodeSize   : Predicate<"!MF->getFunction()->optForSize()">;
 }
 

 def CRC32Wrr : BaseCRC32<0, 0b10, 0, GPR32, int_aarch64_crc32w, "crc32w">;
 def CRC32Xrr : BaseCRC32<1, 0b11, 0, GPR64, int_aarch64_crc32x, "crc32x">;
 
 def CRC32CBrr : BaseCRC32<0, 0b00, 1, GPR32, int_aarch64_crc32cb, "crc32cb">;
 def CRC32CHrr : BaseCRC32<0, 0b01, 1, GPR32, int_aarch64_crc32ch, "crc32ch">;
 def CRC32CWrr : BaseCRC32<0, 0b10, 1, GPR32, int_aarch64_crc32cw, "crc32cw">;
 def CRC32CXrr : BaseCRC32<1, 0b11, 1, GPR64, int_aarch64_crc32cx, "crc32cx">;
 
+// NOP
+def NOP : BaseNOP;
+
 // v8.1 atomic CAS
-defm CAS   : CompareAndSwap<0, 0, "">;
-defm CASA  : CompareAndSwap<1, 0, "a">;
-defm CASL  : CompareAndSwap<0, 1, "l">;
-defm CASAL : CompareAndSwap<1, 1, "al">;
+let AddedComplexity = 5, Predicates = [HasLSE] in {
+  def CASB    : BaseCAS<0b00, 0, 0, "", "b", GPR32>;
+  def CASH    : BaseCAS<0b01, 0, 0, "", "h", GPR32>;
+  def CASW    : BaseCAS<0b10, 0, 0, "", "", GPR32>;
+  def CASX    : BaseCAS<0b11, 0, 0, "", "", GPR64>;
+
+  def CASAB   : BaseCAS<0b00, 1, 0, "a", "b", GPR32>;
+  def CASAH   : BaseCAS<0b01, 1, 0, "a", "h", GPR32>;
+  def CASAW   : BaseCAS<0b10, 1, 0, "a", "", GPR32>;
+  def CASAX   : BaseCAS<0b11, 1, 0, "a", "", GPR64>;
+
+  def CASLB   : BaseCAS<0b00, 0, 1, "l", "b", GPR32>;
+  def CASLH   : BaseCAS<0b01, 0, 1, "l", "h", GPR32>;
+  def CASLW   : BaseCAS<0b10, 0, 1, "l", "", GPR32>;
+  def CASLX   : BaseCAS<0b11, 0, 1, "l", "", GPR64>;
+
+  def CASALB  : BaseCAS<0b00, 1, 1, "al", "b", GPR32>;
+  def CASALH  : BaseCAS<0b01, 1, 1, "al", "h", GPR32>;
+  def CASALW  : BaseCAS<0b10, 1, 1, "al", "", GPR32>;
+  def CASALX  : BaseCAS<0b11, 1, 1, "al", "", GPR64>;
+}
 
 // v8.1 atomic CASP
-defm CASP   : CompareAndSwapPair<0, 0, "">;
-defm CASPA  : CompareAndSwapPair<1, 0, "a">;
-defm CASPL  : CompareAndSwapPair<0, 1, "l">;
-defm CASPAL : CompareAndSwapPair<1, 1, "al">;
+let AddedComplexity = 5, Predicates = [HasLSE] in {
+  def CASPW   : BaseCASP<0b00, 0, 0, "", "", WSeqPairClassOperand>;
+  def CASPAW  : BaseCASP<0b00, 1, 0, "a", "", WSeqPairClassOperand>;
+  def CASPLW  : BaseCASP<0b00, 0, 1, "l", "", WSeqPairClassOperand>;
+  def CASPALW : BaseCASP<0b00, 1, 1, "al", "", WSeqPairClassOperand>;
+
+  def CASPX   : BaseCASP<0b01, 0, 0, "", "", XSeqPairClassOperand>;
+  def CASPAX  : BaseCASP<0b01, 1, 0, "a", "", XSeqPairClassOperand>;
+  def CASPLX  : BaseCASP<0b01, 0, 1, "l", "", XSeqPairClassOperand>;
+  def CASPALX : BaseCASP<0b01, 1, 1, "al", "", XSeqPairClassOperand>;
+ }
+
 
 // v8.1 atomic SWP
-defm SWP   : Swap<0, 0, "">;
-defm SWPA  : Swap<1, 0, "a">;
-defm SWPL  : Swap<0, 1, "l">;
-defm SWPAL : Swap<1, 1, "al">;
-
-// v8.1 atomic LD<OP>(register). Performs load and then ST<OP>(register)
-defm LDADD   : LDOPregister<0b000, "add", 0, 0, "">;
-defm LDADDA  : LDOPregister<0b000, "add", 1, 0, "a">;
-defm LDADDL  : LDOPregister<0b000, "add", 0, 1, "l">;
-defm LDADDAL : LDOPregister<0b000, "add", 1, 1, "al">;
-
-defm LDCLR   : LDOPregister<0b001, "clr", 0, 0, "">;
-defm LDCLRA  : LDOPregister<0b001, "clr", 1, 0, "a">;
-defm LDCLRL  : LDOPregister<0b001, "clr", 0, 1, "l">;
-defm LDCLRAL : LDOPregister<0b001, "clr", 1, 1, "al">;
-
-defm LDEOR   : LDOPregister<0b010, "eor", 0, 0, "">;
-defm LDEORA  : LDOPregister<0b010, "eor", 1, 0, "a">;
-defm LDEORL  : LDOPregister<0b010, "eor", 0, 1, "l">;
-defm LDEORAL : LDOPregister<0b010, "eor", 1, 1, "al">;
-
-defm LDSET   : LDOPregister<0b011, "set", 0, 0, "">;
-defm LDSETA  : LDOPregister<0b011, "set", 1, 0, "a">;
-defm LDSETL  : LDOPregister<0b011, "set", 0, 1, "l">;
-defm LDSETAL : LDOPregister<0b011, "set", 1, 1, "al">;
-
-defm LDSMAX   : LDOPregister<0b100, "smax", 0, 0, "">;
-defm LDSMAXA  : LDOPregister<0b100, "smax", 1, 0, "a">;
-defm LDSMAXL  : LDOPregister<0b100, "smax", 0, 1, "l">;
-defm LDSMAXAL : LDOPregister<0b100, "smax", 1, 1, "al">;
-
-defm LDSMIN   : LDOPregister<0b101, "smin", 0, 0, "">;
-defm LDSMINA  : LDOPregister<0b101, "smin", 1, 0, "a">;
-defm LDSMINL  : LDOPregister<0b101, "smin", 0, 1, "l">;
-defm LDSMINAL : LDOPregister<0b101, "smin", 1, 1, "al">;
-
-defm LDUMAX   : LDOPregister<0b110, "umax", 0, 0, "">;
-defm LDUMAXA  : LDOPregister<0b110, "umax", 1, 0, "a">;
-defm LDUMAXL  : LDOPregister<0b110, "umax", 0, 1, "l">;
-defm LDUMAXAL : LDOPregister<0b110, "umax", 1, 1, "al">;
-
-defm LDUMIN   : LDOPregister<0b111, "umin", 0, 0, "">;
-defm LDUMINA  : LDOPregister<0b111, "umin", 1, 0, "a">;
-defm LDUMINL  : LDOPregister<0b111, "umin", 0, 1, "l">;
-defm LDUMINAL : LDOPregister<0b111, "umin", 1, 1, "al">;
-
-// v8.1 atomic ST<OP>(register) as aliases to "LD<OP>(register) when Rt=xZR"
-defm : STOPregister<"stadd","LDADD">; // STADDx
-defm : STOPregister<"stclr","LDCLR">; // STCLRx
-defm : STOPregister<"steor","LDEOR">; // STEORx
-defm : STOPregister<"stset","LDSET">; // STSETx
-defm : STOPregister<"stsmax","LDSMAX">;// STSMAXx
-defm : STOPregister<"stsmin","LDSMIN">;// STSMINx
-defm : STOPregister<"stumax","LDUMAX">;// STUMAXx
-defm : STOPregister<"stumin","LDUMIN">;// STUMINx
+let AddedComplexity = 5, Predicates = [HasLSE] in {
+  def SWPB    : BaseSWP<0b00, 0, 0, "", "b", GPR32>;
+  def SWPH    : BaseSWP<0b01, 0, 0, "", "h", GPR32>;
+  def SWPW    : BaseSWP<0b10, 0, 0, "", "", GPR32>;
+  def SWPX    : BaseSWP<0b11, 0, 0, "", "", GPR64>;
+  def SWPAB   : BaseSWP<0b00, 1, 0, "a", "b", GPR32>;
+  def SWPAH   : BaseSWP<0b01, 1, 0, "a", "h", GPR32>;
+  def SWPAW   : BaseSWP<0b10, 1, 0, "a", "", GPR32>;
+  def SWPAX   : BaseSWP<0b11, 1, 0, "a", "", GPR64>;
+  def SWPLB   : BaseSWP<0b00, 0, 1, "l", "b", GPR32>;
+  def SWPLH   : BaseSWP<0b01, 0, 1, "l", "h", GPR32>;
+  def SWPLW   : BaseSWP<0b10, 0, 1, "l", "", GPR32>;
+  def SWPLX   : BaseSWP<0b11, 0, 1, "l", "", GPR64>;
+  def SWPALB  : BaseSWP<0b00, 1, 1, "al", "b", GPR32>;
+  def SWPALH  : BaseSWP<0b01, 1, 1, "al", "h", GPR32>;
+  def SWPALW  : BaseSWP<0b10, 1, 1, "al", "", GPR32>;
+  def SWPALX  : BaseSWP<0b11, 1, 1, "al", "", GPR64>;
+}
+
+let AddedComplexity = 5, Predicates = [HasLSE] in {
+ def LDADDB    : BaseLDOP<0b00, 0, 0, 0b000, "add", "", "b",
+                          GPR32>;
+ def LDADDH    : BaseLDOP<0b01, 0, 0, 0b000, "add", "", "h",
+                          GPR32>;
+ def LDADDW    : BaseLDOP<0b10, 0, 0, 0b000, "add", "", "",
+                          GPR32>;
+ def LDADDX    : BaseLDOP<0b11, 0, 0, 0b000, "add", "", "",
+                          GPR64>;
+
+ def LDADDAB   : BaseLDOP<0b00, 1, 0, 0b000, "add", "a", "b",
+                          GPR32>;
+ def LDADDAH   : BaseLDOP<0b01, 1, 0, 0b000, "add", "a", "h",
+                          GPR32>;
+ def LDADDAW   : BaseLDOP<0b10, 1, 0, 0b000, "add", "a", "",
+                          GPR32>;
+ def LDADDAX   : BaseLDOP<0b11, 1, 0, 0b000, "add", "a", "",
+                          GPR64>;
+
+ def LDADDLB   : BaseLDOP<0b00, 0, 1, 0b000, "add", "l", "b",
+                          GPR32>;
+ def LDADDLH   : BaseLDOP<0b01, 0, 1, 0b000, "add", "l", "h",
+                          GPR32>;
+ def LDADDLW   : BaseLDOP<0b10, 0, 1, 0b000, "add", "l", "",
+                          GPR32>;
+ def LDADDLX   : BaseLDOP<0b11, 0, 1, 0b000, "add", "l", "",
+                          GPR64>;
+
+ def LDADDALB  : BaseLDOP<0b00, 1, 1, 0b000, "add", "al", "b",
+                         GPR32>;
+ def LDADDALH  : BaseLDOP<0b01, 1, 1, 0b000, "add", "al", "h",
+                          GPR32>;
+ def LDADDALW  : BaseLDOP<0b10, 1, 1, 0b000, "add", "al", "",
+                          GPR32>;
+ def LDADDALX  : BaseLDOP<0b11, 1, 1, 0b000, "add", "al", "",
+                          GPR64>;
+
+ def LDCLRB    : BaseLDOP<0b00, 0, 0, 0b001, "clr", "", "b",
+                          GPR32>;
+ def LDCLRH    : BaseLDOP<0b01, 0, 0, 0b001, "clr", "", "h",
+                          GPR32>;
+ def LDCLRW    : BaseLDOP<0b10, 0, 0, 0b001, "clr", "", "",
+                          GPR32>;
+ def LDCLRX    : BaseLDOP<0b11, 0, 0, 0b001, "clr", "", "",
+                          GPR64>;
+
+ def LDCLRAB   : BaseLDOP<0b00, 1, 0, 0b001, "clr", "a", "b",
+                          GPR32>;
+ def LDCLRAH   : BaseLDOP<0b01, 1, 0, 0b001, "clr", "a", "h",
+                          GPR32>;
+ def LDCLRAW   : BaseLDOP<0b10, 1, 0, 0b001, "clr", "a", "",
+                          GPR32>;
+ def LDCLRAX   : BaseLDOP<0b11, 1, 0, 0b001, "clr", "a", "",
+                          GPR64>;
+
+ def LDCLRLB   : BaseLDOP<0b00, 0, 1, 0b001, "clr", "l", "b",
+                          GPR32>;
+ def LDCLRLH   : BaseLDOP<0b01, 0, 1, 0b001, "clr", "l", "h",
+                          GPR32>;
+ def LDCLRLW   : BaseLDOP<0b10, 0, 1, 0b001, "clr", "l", "",
+                          GPR32>;
+ def LDCLRLX   : BaseLDOP<0b11, 0, 1, 0b001, "clr", "l", "",
+                          GPR64>;
+
+ def LDCLRALB  : BaseLDOP<0b00, 1, 1, 0b001, "clr", "al", "b",
+                          GPR32>;
+ def LDCLRALH  : BaseLDOP<0b01, 1, 1, 0b001, "clr", "al", "h",
+                          GPR32>;
+ def LDCLRALW  : BaseLDOP<0b10, 1, 1, 0b001, "clr", "al", "",
+                          GPR32>;
+ def LDCLRALX  : BaseLDOP<0b11, 1, 1, 0b001, "clr", "al", "",
+                          GPR64>;
+
+ def LDEORB    : BaseLDOP<0b00, 0, 0, 0b010, "eor", "", "b",
+                          GPR32>;
+ def LDEORH    : BaseLDOP<0b01, 0, 0, 0b010, "eor", "", "h",
+                          GPR32>;
+ def LDEORW    : BaseLDOP<0b10, 0, 0, 0b010, "eor", "", "",
+                          GPR32>;
+ def LDEORX    : BaseLDOP<0b11, 0, 0, 0b010, "eor", "", "",
+                          GPR64>;
+
+ def LDEORAB   : BaseLDOP<0b00, 1, 0, 0b010, "eor", "a", "b",
+                          GPR32>;
+ def LDEORAH   : BaseLDOP<0b01, 1, 0, 0b010, "eor", "a", "h",
+                          GPR32>;
+ def LDEORAW   : BaseLDOP<0b10, 1, 0, 0b010, "eor", "a", "",
+                          GPR32>;
+ def LDEORAX   : BaseLDOP<0b11, 1, 0, 0b010, "eor", "a", "",
+                          GPR64>;
+
+ def LDEORLB   : BaseLDOP<0b00, 0, 1, 0b010, "eor", "l", "b",
+                          GPR32>;
+ def LDEORLH   : BaseLDOP<0b01, 0, 1, 0b010, "eor", "l", "h",
+                          GPR32>;
+ def LDEORLW   : BaseLDOP<0b10, 0, 1, 0b010, "eor", "l", "",
+                          GPR32>;
+ def LDEORLX   : BaseLDOP<0b11, 0, 1, 0b010, "eor", "l", "",
+                          GPR64>;
+
+ def LDEORALB  : BaseLDOP<0b00, 1, 1, 0b010, "eor", "al", "b",
+                          GPR32>;
+ def LDEORALH  : BaseLDOP<0b01, 1, 1, 0b010, "eor", "al", "h",
+                          GPR32>;
+ def LDEORALW  : BaseLDOP<0b10, 1, 1, 0b010, "eor", "al", "",
+                          GPR32>;
+ def LDEORALX  : BaseLDOP<0b11, 1, 1, 0b010, "eor", "al", "",
+                          GPR64>;
+
+ def LDSETB    : BaseLDOP<0b00, 0, 0, 0b011, "set", "", "b",
+                          GPR32>;
+ def LDSETH    : BaseLDOP<0b01, 0, 0, 0b011, "set", "", "h",
+                          GPR32>;
+ def LDSETW    : BaseLDOP<0b10, 0, 0, 0b011, "set", "", "",
+                          GPR32>;
+ def LDSETX    : BaseLDOP<0b11, 0, 0, 0b011, "set", "", "",
+                          GPR64>;
+
+ def LDSETAB   : BaseLDOP<0b00, 1, 0, 0b011, "set", "a", "b",
+                          GPR32>;
+ def LDSETAH   : BaseLDOP<0b01, 1, 0, 0b011, "set", "a", "h",
+                          GPR32>;
+ def LDSETAW   : BaseLDOP<0b10, 1, 0, 0b011, "set", "a", "",
+                          GPR32>;
+ def LDSETAX   : BaseLDOP<0b11, 1, 0, 0b011, "set", "a", "",
+                          GPR64>;
+
+ def LDSETLB   : BaseLDOP<0b00, 0, 1, 0b011, "set", "l", "b",
+                          GPR32>;
+ def LDSETLH   : BaseLDOP<0b01, 0, 1, 0b011, "set", "l", "h",
+                          GPR32>;
+ def LDSETLW   : BaseLDOP<0b10, 0, 1, 0b011, "set", "l", "",
+                          GPR32>;
+ def LDSETLX   : BaseLDOP<0b11, 0, 1, 0b011, "set", "l", "",
+                          GPR64>;
+
+ def LDSETALB  : BaseLDOP<0b00, 1, 1, 0b011, "set", "al", "b",
+                          GPR32>;
+ def LDSETALH  : BaseLDOP<0b01, 1, 1, 0b011, "set", "al", "h",
+                          GPR32>;
+ def LDSETALW  : BaseLDOP<0b10, 1, 1, 0b011, "set", "al", "",
+                          GPR32>;
+ def LDSETALX  : BaseLDOP<0b11, 1, 1, 0b011, "set", "al", "",
+                          GPR64>;
+
+ def LDSMAXB   : BaseLDOP<0b00, 0, 0, 0b100, "smax", "", "b",
+                          GPR32>;
+ def LDSMAXH   : BaseLDOP<0b01, 0, 0, 0b100, "smax", "", "h",
+                          GPR32>;
+ def LDSMAXW   : BaseLDOP<0b10, 0, 0, 0b100, "smax", "", "",
+                          GPR32>;
+ def LDSMAXX   : BaseLDOP<0b11, 0, 0, 0b100, "smax", "", "",
+                          GPR64>;
+
+ def LDSMAXAB  : BaseLDOP<0b00, 1, 0, 0b100, "smax", "a", "b",
+                          GPR32>;
+ def LDSMAXAH  : BaseLDOP<0b01, 1, 0, 0b100, "smax", "a", "h",
+                          GPR32>;
+ def LDSMAXAW  : BaseLDOP<0b10, 1, 0, 0b100, "smax", "a", "",
+                          GPR32>;
+ def LDSMAXAX  : BaseLDOP<0b11, 1, 0, 0b100, "smax", "a", "",
+                          GPR64>;
+
+ def LDSMAXLB  : BaseLDOP<0b00, 0, 1, 0b100, "smax", "l", "b",
+                          GPR32>;
+ def LDSMAXLH  : BaseLDOP<0b01, 0, 1, 0b100, "smax", "l", "h",
+                          GPR32>;
+ def LDSMAXLW  : BaseLDOP<0b10, 0, 1, 0b100, "smax", "l", "",
+                          GPR32>;
+ def LDSMAXLX  : BaseLDOP<0b11, 0, 1, 0b100, "smax", "l", "",
+                          GPR64>;
+
+ def LDSMAXALB : BaseLDOP<0b00, 1, 1, 0b100, "smax", "al", "b",
+                          GPR32>;
+ def LDSMAXALH : BaseLDOP<0b01, 1, 1, 0b100, "smax", "al", "h",
+                          GPR32>;
+ def LDSMAXALW : BaseLDOP<0b10, 1, 1, 0b100, "smax", "al", "",
+                          GPR32>;
+ def LDSMAXALX : BaseLDOP<0b11, 1, 1, 0b100, "smax", "al", "",
+                          GPR64>;
+
+ def LDSMINB   : BaseLDOP<0b00, 0, 0, 0b101, "smin", "", "b",
+                          GPR32>;
+ def LDSMINH   : BaseLDOP<0b01, 0, 0, 0b101, "smin", "", "h",
+                          GPR32>;
+ def LDSMINW   : BaseLDOP<0b10, 0, 0, 0b101, "smin", "", "",
+                          GPR32>;
+ def LDSMINX   : BaseLDOP<0b11, 0, 0, 0b101, "smin", "", "",
+                          GPR64>;
+
+ def LDSMINAB  : BaseLDOP<0b00, 1, 0, 0b101, "smin", "a", "b",
+                          GPR32>;
+ def LDSMINAH  : BaseLDOP<0b01, 1, 0, 0b101, "smin", "a", "h",
+                          GPR32>;
+ def LDSMINAW  : BaseLDOP<0b10, 1, 0, 0b101, "smin", "a", "",
+                          GPR32>;
+ def LDSMINAX  : BaseLDOP<0b11, 1, 0, 0b101, "smin", "a", "",
+                          GPR64>;
+
+ def LDSMINLB  : BaseLDOP<0b00, 0, 1, 0b101, "smin", "l", "b",
+                          GPR32>;
+ def LDSMINLH  : BaseLDOP<0b01, 0, 1, 0b101, "smin", "l", "h",
+                          GPR32>;
+ def LDSMINLW  : BaseLDOP<0b10, 0, 1, 0b101, "smin", "l", "",
+                          GPR32>;
+ def LDSMINLX  : BaseLDOP<0b11, 0, 1, 0b101, "smin", "l", "",
+                          GPR64>;
+
+ def LDSMINALB : BaseLDOP<0b00, 1, 1, 0b101, "smin", "al", "b",
+                          GPR32>;
+ def LDSMINALH : BaseLDOP<0b01, 1, 1, 0b101, "smin", "al", "h",
+                          GPR32>;
+ def LDSMINALW : BaseLDOP<0b10, 1, 1, 0b101, "smin", "al", "",
+                          GPR32>;
+ def LDSMINALX : BaseLDOP<0b11, 1, 1, 0b101, "smin", "al", "",
+                          GPR64>;
+
+ def LDUMAXB   : BaseLDOP<0b00, 0, 0, 0b110, "umax", "", "b",
+                          GPR32>;
+ def LDUMAXH   : BaseLDOP<0b01, 0, 0, 0b110, "umax", "", "h",
+                          GPR32>;
+ def LDUMAXW   : BaseLDOP<0b10, 0, 0, 0b110, "umax", "", "",
+                          GPR32>;
+ def LDUMAXX   : BaseLDOP<0b11, 0, 0, 0b110, "umax", "", "",
+                          GPR64>;
+
+ def LDUMAXAB  : BaseLDOP<0b00, 1, 0, 0b110, "umax", "a", "b",
+                          GPR32>;
+ def LDUMAXAH  : BaseLDOP<0b01, 1, 0, 0b110, "umax", "a", "h",
+                          GPR32>;
+ def LDUMAXAW  : BaseLDOP<0b10, 1, 0, 0b110, "umax", "a", "",
+                          GPR32>;
+ def LDUMAXAX  : BaseLDOP<0b11, 1, 0, 0b110, "umax", "a", "",
+                          GPR64>;
+
+ def LDUMAXLB  : BaseLDOP<0b00, 0, 1, 0b110, "umax", "l", "b",
+                          GPR32>;
+ def LDUMAXLH  : BaseLDOP<0b01, 0, 1, 0b110, "umax", "l", "h",
+                          GPR32>;
+ def LDUMAXLW  : BaseLDOP<0b10, 0, 1, 0b110, "umax", "l", "",
+                          GPR32>;
+ def LDUMAXLX  : BaseLDOP<0b11, 0, 1, 0b110, "umax", "l", "",
+                          GPR64>;
+
+ def LDUMAXALB : BaseLDOP<0b00, 1, 1, 0b110, "umax", "al", "b",
+                          GPR32>;
+ def LDUMAXALH : BaseLDOP<0b01, 1, 1, 0b110, "umax", "al", "h",
+                          GPR32>;
+ def LDUMAXALW : BaseLDOP<0b10, 1, 1, 0b110, "umax", "al", "",
+                          GPR32>;
+ def LDUMAXALX : BaseLDOP<0b11, 1, 1, 0b110, "umax", "al", "",
+                          GPR64>;
+
+ def LDUMINB   : BaseLDOP<0b00, 0, 0, 0b111, "umin", "", "b",
+                          GPR32>;
+ def LDUMINH   : BaseLDOP<0b01, 0, 0, 0b111, "umin", "", "h",
+                          GPR32>;
+ def LDUMINW   : BaseLDOP<0b10, 0, 0, 0b111, "umin", "", "",
+                          GPR32>;
+ def LDUMINX   : BaseLDOP<0b11, 0, 0, 0b111, "umin", "", "",
+                          GPR64>;
+
+ def LDUMINAB  : BaseLDOP<0b00, 1, 0, 0b111, "umin", "a", "b",
+                          GPR32>;
+ def LDUMINAH  : BaseLDOP<0b01, 1, 0, 0b111, "umin", "a", "h",
+                          GPR32>;
+ def LDUMINAW  : BaseLDOP<0b10, 1, 0, 0b111, "umin", "a", "",
+                          GPR32>;
+ def LDUMINAX  : BaseLDOP<0b11, 1, 0, 0b111, "umin", "a", "",
+                          GPR64>;
+
+ def LDUMINLB  : BaseLDOP<0b00, 0, 1, 0b111, "umin", "l", "b",
+                          GPR32>;
+ def LDUMINLH  : BaseLDOP<0b01, 0, 1, 0b111, "umin", "l", "h",
+                          GPR32>;
+ def LDUMINLW  : BaseLDOP<0b10, 0, 1, 0b111, "umin", "l", "",
+                          GPR32>;
+ def LDUMINLX  : BaseLDOP<0b11, 0, 1, 0b111, "umin", "l", "",
+                          GPR64>;
+
+ def LDUMINALB : BaseLDOP<0b00, 1, 1, 0b111, "umin", "al", "b",
+                          GPR32>;
+ def LDUMINALH : BaseLDOP<0b01, 1, 1, 0b111, "umin", "al", "h",
+                          GPR32>;
+ def LDUMINALW : BaseLDOP<0b10, 1, 1, 0b111, "umin", "al", "",
+                          GPR32>;
+ def LDUMINALX : BaseLDOP<0b11, 1, 1, 0b111, "umin", "al", "",
+                          GPR64>;
+}
+
+// v8.1 atomic ST<OP>(register)
+let AddedComplexity = 5, Predicates = [HasLSE] in {
+ def STADDB    : BaseSTOP<0b00, 0, 0, 0b000, 0b11111, "add", "", "b",
+                         GPR32>;
+ def STADDH    : BaseSTOP<0b01, 0, 0, 0b000, 0b11111, "add", "", "h",
+                         GPR32>;
+ def STADDW    : BaseSTOP<0b10, 0, 0, 0b000, 0b11111, "add", "", "",
+                         GPR32>;
+ def STADDX    : BaseSTOP<0b11, 0, 0, 0b000, 0b11111, "add", "", "",
+                         GPR64>;
+
+ def STADDLB   : BaseSTOP<0b00, 0, 1, 0b000, 0b11111, "add", "l", "b",
+                          GPR32>;
+ def STADDLH   : BaseSTOP<0b01, 0, 1, 0b000, 0b11111, "add", "l", "h",
+                          GPR32>;
+ def STADDLW   : BaseSTOP<0b10, 0, 1, 0b000, 0b11111,  "add", "l", "",
+                          GPR32>;
+ def STADDLX   : BaseSTOP<0b11, 0, 1, 0b000, 0b11111, "add", "l", "",
+                          GPR64>;
+
+ def STCLRB    : BaseSTOP<0b00, 0, 0, 0b001, 0b11111, "clr", "", "b",
+                          GPR32>;
+ def STCLRH    : BaseSTOP<0b01, 0, 0, 0b001, 0b11111, "clr", "", "h",
+                          GPR32>;
+ def STCLRW    : BaseSTOP<0b10, 0, 0, 0b001, 0b11111, "clr", "", "",
+                          GPR32>;
+ def STCLRX    : BaseSTOP<0b11, 0, 0, 0b001, 0b11111, "clr", "", "",
+                          GPR64>;
+
+ def STCLRLB   : BaseSTOP<0b00, 0, 1, 0b001, 0b11111, "clr", "l", "b",
+                          GPR32>;
+ def STCLRLH   : BaseSTOP<0b01, 0, 1, 0b001, 0b11111, "clr", "l", "h",
+                          GPR32>;
+ def STCLRLW   : BaseSTOP<0b10, 0, 1, 0b001, 0b11111, "clr", "l", "",
+                          GPR32>;
+ def STCLRLX   : BaseSTOP<0b11, 0, 1, 0b001, 0b11111, "clr", "l", "",
+                          GPR64>;
+
+ def STEORB    : BaseSTOP<0b00, 0, 0, 0b010, 0b11111, "eor", "", "b",
+                          GPR32>;
+ def STEORH    : BaseSTOP<0b01, 0, 0, 0b010, 0b11111, "eor", "", "h",
+                          GPR32>;
+ def STEORW    : BaseSTOP<0b10, 0, 0, 0b010, 0b11111, "eor", "", "",
+                          GPR32>;
+ def STEORX    : BaseSTOP<0b11, 0, 0, 0b010, 0b11111, "eor", "", "",
+                          GPR64>;
+
+ def STEORLB   : BaseSTOP<0b00, 0, 1, 0b010, 0b11111, "eor", "l", "b",
+                          GPR32>;
+ def STEORLH   : BaseSTOP<0b01, 0, 1, 0b010, 0b11111, "eor", "l", "h",
+                          GPR32>;
+ def STEORLW   : BaseSTOP<0b10, 0, 1, 0b010, 0b11111, "eor", "l", "",
+                          GPR32>;
+ def STEORLX   : BaseSTOP<0b11, 0, 1, 0b010, 0b11111, "eor", "l", "",
+                          GPR64>;
+
+ def STSETB    : BaseSTOP<0b00, 0, 0, 0b011, 0b11111, "set", "", "b",
+                         GPR32>;
+ def STSETH    : BaseSTOP<0b01, 0, 0, 0b011, 0b11111, "set", "", "h",
+                         GPR32>;
+ def STSETW    : BaseSTOP<0b10, 0, 0, 0b011, 0b11111, "set", "", "",
+                         GPR32>;
+ def STSETX    : BaseSTOP<0b11, 0, 0, 0b011, 0b11111, "set", "", "",
+                         GPR64>;
+
+ def STSETLB   : BaseSTOP<0b00, 0, 1, 0b011, 0b11111, "set", "l", "b",
+                          GPR32>;
+ def STSETLH   : BaseSTOP<0b01, 0, 1, 0b011, 0b11111, "set", "l", "h",
+                          GPR32>;
+ def STSETLW   : BaseSTOP<0b10, 0, 1, 0b011, 0b11111, "set", "l", "",
+                          GPR32>;
+ def STSETLX   : BaseSTOP<0b11, 0, 1, 0b011, 0b11111, "set", "l", "",
+                          GPR64>;
+
+ def STSMAXB   : BaseSTOP<0b00, 0, 0, 0b100, 0b11111, "smax", "", "b",
+                          GPR32>;
+ def STSMAXH   : BaseSTOP<0b01, 0, 0, 0b100, 0b11111, "smax", "", "h",
+                          GPR32>;
+ def STSMAXW   : BaseSTOP<0b10, 0, 0, 0b100, 0b11111, "smax", "", "",
+                          GPR32>;
+ def STSMAXX   : BaseSTOP<0b11, 0, 0, 0b100, 0b11111, "smax", "", "",
+                          GPR64>;
+
+ def STSMAXLB  : BaseSTOP<0b00, 0, 1, 0b100, 0b11111, "smax", "l", "b",
+                          GPR32>;
+ def STSMAXLH  : BaseSTOP<0b01, 0, 1, 0b100, 0b11111, "smax", "l", "h",
+                          GPR32>;
+ def STSMAXLW  : BaseSTOP<0b10, 0, 1, 0b100, 0b11111, "smax", "l", "",
+                          GPR32>;
+ def STSMAXLX  : BaseSTOP<0b11, 0, 1, 0b100, 0b11111, "smax", "l", "",
+                          GPR64>;
+
+ def STSMINB   : BaseSTOP<0b00, 0, 0, 0b101, 0b11111, "smin", "", "b",
+                          GPR32>;
+ def STSMINH   : BaseSTOP<0b01, 0, 0, 0b101, 0b11111, "smin", "", "h",
+                          GPR32>;
+ def STSMINW   : BaseSTOP<0b10, 0, 0, 0b101, 0b11111, "smin", "", "",
+                          GPR32>;
+ def STSMINX   : BaseSTOP<0b11, 0, 0, 0b101, 0b11111, "smin", "", "",
+                          GPR64>;
+
+ def STSMINLB  : BaseSTOP<0b00, 0, 1, 0b101, 0b11111, "smin", "l", "b",
+                          GPR32>;
+ def STSMINLH  : BaseSTOP<0b01, 0, 1, 0b101, 0b11111, "smin", "l", "h",
+                          GPR32>;
+ def STSMINLW  : BaseSTOP<0b10, 0, 1, 0b101, 0b11111, "smin", "l", "",
+                          GPR32>;
+ def STSMINLX  : BaseSTOP<0b11, 0, 1, 0b101, 0b11111, "smin", "l", "",
+                          GPR64>;
+
+ def STUMAXB   : BaseSTOP<0b00, 0, 0, 0b110, 0b11111, "umax", "", "b",
+                          GPR32>;
+ def STUMAXH   : BaseSTOP<0b01, 0, 0, 0b110, 0b11111, "umax", "", "h",
+                          GPR32>;
+ def STUMAXW   : BaseSTOP<0b10, 0, 0, 0b110, 0b11111, "umax", "", "",
+                          GPR32>;
+ def STUMAXX   : BaseSTOP<0b11, 0, 0, 0b110, 0b11111, "umax", "", "",
+                          GPR64>;
+
+ def STUMAXLB  : BaseSTOP<0b00, 0, 1, 0b110, 0b11111, "umax", "l", "b",
+                          GPR32>;
+ def STUMAXLH  : BaseSTOP<0b01, 0, 1, 0b110, 0b11111, "umax", "l", "h",
+                          GPR32>;
+ def STUMAXLW  : BaseSTOP<0b10, 0, 1, 0b110, 0b11111, "umax", "l", "",
+                          GPR32>;
+ def STUMAXLX  : BaseSTOP<0b11, 0, 1, 0b110, 0b11111, "umax", "l", "",
+                          GPR64>;
+
+ def STUMINB   : BaseSTOP<0b00, 0, 0, 0b111, 0b11111, "umin", "", "b",
+                          GPR32>;
+ def STUMINH   : BaseSTOP<0b01, 0, 0, 0b111, 0b11111, "umin", "", "h",
+                          GPR32>;
+ def STUMINW   : BaseSTOP<0b10, 0, 0, 0b111, 0b11111, "umin", "", "",
+                          GPR32>;
+ def STUMINX   : BaseSTOP<0b11, 0, 0, 0b111, 0b11111, "umin", "", "",
+                          GPR64>;
+
+ def STUMINLB  : BaseSTOP<0b00, 0, 1, 0b111, 0b11111, "umin", "l", "b",
+                          GPR32>;
+ def STUMINLH  : BaseSTOP<0b01, 0, 1, 0b111, 0b11111, "umin", "l", "h",
+                          GPR32>;
+ def STUMINLW  : BaseSTOP<0b10, 0, 1, 0b111, 0b11111, "umin", "l", "",
+                          GPR32>;
+ def STUMINLX  : BaseSTOP<0b11, 0, 1, 0b111, 0b11111, "umin", "l", "",
+                          GPR64>;
+}
 
 //===----------------------------------------------------------------------===//
 // Logical instructions.
 //===----------------------------------------------------------------------===//
 
 // (immediate)
 defm ANDS : LogicalImmS<0b11, "ands", AArch64and_flag, "bics">;
 defm AND  : LogicalImm<0b00, "and", and, "bic">;
 defm EOR  : LogicalImm<0b10, "eor", xor, "eon">;
 defm ORR  : LogicalImm<0b01, "orr", or, "orn">;
 
 // FIXME: these aliases *are* canonical sometimes (when movz can't be
 // used). Actually, it seems to be working right now, but putting logical_immXX
 // here is a bit dodgy on the AsmParser side too.
 def : InstAlias<"mov $dst, $imm", (ORRWri GPR32sp:$dst, WZR,
                                           logical_imm32:$imm), 0>;

lib/Target/AArch64/AArch64RegisterInfo.td

 defm VecListTwo   : VectorList<2, DD,    QQ>;
 defm VecListThree : VectorList<3, DDD,   QQQ>;
 defm VecListFour  : VectorList<4, DDDD,  QQQQ>;
 
 
 // Register operand versions of the scalar FP registers.
 def FPR16Op : RegisterOperand<FPR16, "printOperand">;
 def FPR32Op : RegisterOperand<FPR32, "printOperand">;
 def FPR64Op : RegisterOperand<FPR64, "printOperand">;
 def FPR128Op : RegisterOperand<FPR128, "printOperand">;
 
 
 //===----------------------------------------------------------------------===//
 // ARMv8.1a atomic CASP register operands
 
 
-def WSeqPairs : RegisterTuples<[sube32, subo32], 
+def WSeqPairs : RegisterTuples<[sube32, subo32],
                                [(rotl GPR32, 0), (rotl GPR32, 1)]>;
-def XSeqPairs : RegisterTuples<[sube64, subo64], 
+def XSeqPairs : RegisterTuples<[sube64, subo64],
                                [(rotl GPR64, 0), (rotl GPR64, 1)]>;
 
-def WSeqPairsClass   : RegisterClass<"AArch64", [untyped], 32, 
-                                     (add WSeqPairs)>{
+def WSeqPairsClass   : RegisterClass<"AArch64", [untyped], 32,
+                                     (add WSeqPairs)> {
   let Size = 64;
 }
-def XSeqPairsClass   : RegisterClass<"AArch64", [untyped], 64, 
-                                     (add XSeqPairs)>{
+
+def XSeqPairsClass   : RegisterClass<"AArch64", [untyped], 64,
+                                     (add XSeqPairs)> {
   let Size = 128;
 }
 
 
 let RenderMethod = "addRegOperands", ParserMethod="tryParseGPRSeqPair" in {
   def WSeqPairsAsmOperandClass : AsmOperandClass { let Name = "WSeqPair"; }
   def XSeqPairsAsmOperandClass : AsmOperandClass { let Name = "XSeqPair"; }
 }
 
 def WSeqPairClassOperand :
     RegisterOperand<WSeqPairsClass, "printGPRSeqPairsClassOperand<32>"> {
   let ParserMatchClass = WSeqPairsAsmOperandClass;
 }
 def XSeqPairClassOperand :
     RegisterOperand<XSeqPairsClass, "printGPRSeqPairsClassOperand<64>"> {
   let ParserMatchClass = XSeqPairsAsmOperandClass;

lib/Target/AArch64/AArch64SchedThunderX2T99.td

   let Latency = 6;
   let NumMicroOps = 3;
 }
 
 // 7 cycles on LS0 or LS1 and F0 or F1.
 def THX2T99Write_7Cyc_LS01_F01 : SchedWriteRes<[THX2T99LS01, THX2T99F01]> {
   let Latency = 7;
   let NumMicroOps = 3;
 }
 
 // 8 cycles on LS0 or LS1 and F0 or F1.
 def THX2T99Write_8Cyc_LS01_F01 : SchedWriteRes<[THX2T99LS01, THX2T99F01]> {
   let Latency = 8;
   let NumMicroOps = 3;
 }
 
+// 8 cycles on LS0 or LS1 and I0, I1, or I2.
+def THX2T99Write_8Cyc_I012 : SchedWriteRes<[THX2T99LS01, THX2T99I012]> {
+  let Latency = 8;
+  let NumMicroOps = 4;
+}
+
+// 12 cycles on LS0 or LS1 and I0, I1, or I2.
+def THX2T99Write_12Cyc_I012 : SchedWriteRes<[THX2T99LS01, THX2T99I012]> {
+  let Latency = 12;
+  let NumMicroOps = 6;
+}
+
+// 16 cycles on LS0 or LS1 and I0, I1, or I2.
+def THX2T99Write_16Cyc_I012 : SchedWriteRes<[THX2T99LS01, THX2T99I012]> {
+  let Latency = 16;
+  let NumMicroOps = 8;
+}
+
+// 24 cycles on LS0 or LS1 and I0, I1, or I2.
+def THX2T99Write_24Cyc_I012 : SchedWriteRes<[THX2T99LS01, THX2T99I012]> {
+  let Latency = 24;
+  let NumMicroOps = 12;
+}
+
+// 32 cycles on LS0 or LS1 and I0, I1, or I2.
+def THX2T99Write_32Cyc_I012 : SchedWriteRes<[THX2T99LS01, THX2T99I012]> {
+  let Latency = 32;
+  let NumMicroOps = 16;
+}
+
 // Define commonly used read types.
 
 // No forwarding is provided for these types.
 def : ReadAdvance<ReadI,       0>;
 def : ReadAdvance<ReadISReg,   0>;
 def : ReadAdvance<ReadIEReg,   0>;
 def : ReadAdvance<ReadIM,      0>;
 def : ReadAdvance<ReadIMA,     0>;

 
 // ASIMD store, 4 element, one lane, B/H
 // ASIMD store, 4 element, one lane, S
 // ASIMD store, 4 element, one lane, D
 def : InstRW<[THX2T99Write_1Cyc_LS01_F01], (instregex "^ST4i(8|16|32|64)$")>;
 def : InstRW<[THX2T99Write_1Cyc_LS01_F01, WriteAdr],
             (instregex "^ST4i(8|16|32|64)_POST$")>;
 
+// V8.1a Atomics (LSE)
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs CASB, CASH, CASW, CASX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs CASAB, CASAH, CASAW, CASAX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs CASLB, CASLH, CASLW, CASLX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs CASALB, CASALH, CASALW, CASALX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDLARB, LDLARH, LDLARW, LDLARX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs LDADDB, LDADDH, LDADDW, LDADDX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDADDAB, LDADDAH, LDADDAW, LDADDAX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDADDLB, LDADDLH, LDADDLW, LDADDLX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs LDADDALB, LDADDALH, LDADDALW, LDADDALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs LDCLRB, LDCLRH, LDCLRW, LDCLRX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDCLRAB, LDCLRAH, LDCLRAW, LDCLRAX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDCLRLB, LDCLRLH, LDCLRLW, LDCLRLX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs LDCLRALB, LDCLRALH, LDCLRALW, LDCLRALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs LDEORB, LDEORH, LDEORW, LDEORX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDEORAB, LDEORAH, LDEORAW, LDEORAX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDEORLB, LDEORLH, LDEORLW, LDEORLX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs LDEORALB, LDEORALH, LDEORALW, LDEORALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs LDSETB, LDSETH, LDSETW, LDSETX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDSETAB, LDSETAH, LDSETAW, LDSETAX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs LDSETLB, LDSETLH, LDSETLW, LDSETLX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs LDSETALB, LDSETALH, LDSETALW, LDSETALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs LDSMAXB, LDSMAXH, LDSMAXW, LDSMAXX,
+             LDSMAXAB, LDSMAXAH, LDSMAXAW, LDSMAXAX,
+             LDSMAXLB, LDSMAXLH, LDSMAXLW, LDSMAXLX,
+             LDSMAXALB, LDSMAXALH, LDSMAXALW, LDSMAXALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs LDSMINB, LDSMINH, LDSMINW, LDSMINX,
+             LDSMINAB, LDSMINAH, LDSMINAW, LDSMINAX,
+             LDSMINLB, LDSMINLH, LDSMINLW, LDSMINLX,
+             LDSMINALB, LDSMINALH, LDSMINALW, LDSMINALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs LDUMAXB, LDUMAXH, LDUMAXW, LDUMAXX,
+             LDUMAXAB, LDUMAXAH, LDUMAXAW, LDUMAXAX,
+             LDUMAXLB, LDUMAXLH, LDUMAXLW, LDUMAXLX,
+             LDUMAXALB, LDUMAXALH, LDUMAXALW, LDUMAXALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs LDUMINB, LDUMINH, LDUMINW, LDUMINX,
+             LDUMINAB, LDUMINAH, LDUMINAW, LDUMINAX,
+             LDUMINLB, LDUMINLH, LDUMINLW, LDUMINLX,
+             LDUMINALB, LDUMINALH, LDUMINALW, LDUMINALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs SWPB, SWPH, SWPW, SWPX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs SWPAB, SWPAH, SWPAW, SWPAX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs SWPLB, SWPLH, SWPLW, SWPLX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs SWPALB, SWPALH, SWPALW, SWPALX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs STLLRB, STLLRH, STLLRW, STLLRX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs STADDB, STADDH, STADDW, STADDX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs STADDLB, STADDLH, STADDLW, STADDLX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs STCLRB, STCLRH, STCLRW, STCLRX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs STCLRLB, STCLRLH, STCLRLW, STCLRLX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs STEORB, STEORH, STEORW, STEORX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs STEORLB, STEORLH, STEORLW, STEORLX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs STSETB, STSETH, STSETW, STSETX)>;
+
+def : InstRW<[THX2T99Write_16Cyc_I012, WriteAtomic],
+            (instrs STSETLB, STSETLH, STSETLW, STSETLX)>;
+
+def : InstRW<[THX2T99Write_8Cyc_I012, WriteAtomic],
+            (instrs STSMAXB, STSMAXH, STSMAXW, STSMAXX,
+             STSMINB, STSMINH, STSMINW, STSMINX,
+             STUMAXB, STUMAXH, STUMAXW, STUMAXX,
+             STUMINB, STUMINH, STUMINW, STUMINX)>;
+
+def : InstRW<[THX2T99Write_12Cyc_I012, WriteAtomic],
+            (instrs STSMAXLB, STSMAXLH, STSMAXLW, STSMAXLX,
+             STSMINLB, STSMINLH, STSMINLW, STSMINLX,
+             STUMAXLB, STUMAXLH, STUMAXLW, STUMAXLX,
+             STUMINLB, STUMINLH, STUMINLW, STUMINLX)>;
+
 } // SchedModel = ThunderX2T99Model