ARMv8.1 support for LLVM AArch64

[ajasty-cavium]

Summary

Proposed patch for ARMv8.1 Large System Extensions support.

Currently missing support for CASP, NAND (not supported by LSE instructions), and subword SUB/AND.

Requesting comments, looking into fixing subword SUB/AND, and writing tests (currently tested by atomic-ops.ll with -mcpu=thunderx2t99). Also, the ATOMIC_LOAD_CLR was added to generic IR, I suspect I should move it to AArch64ISD, feedback requested.

ATOMIC_LOAD_ADD with discarded return should probably be performed via a separate IR (ATOMIC_ADD, etc). This is straightforward to implement and should increase performance, RFC there too.

Finally, weaker memory ordering should be implemented, but even LDX/STX seem to use full acquire/relax even when __ATOMIC_RELAXED is specified. This will likely be looked into in a later patch.

[rengolin]

Hi, can you please re-share the patch with more context?

git diff -U9999

would do.

Thanks!

lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

2664

Wouldn't this be better inside SelectCMP_SWAP?

[t.p.northover]

One other thing you have to be careful of: if an instruction loads into xzr or wzr then it's actually an "STwhatever" operation and defined not to have acquire semantics regardless of its mnemonic. So you probably have to modify AArch64DeadRegisterDefinitionsPass.cpp to ignore these instructions.

Also, you should add regression tests under tests/CodeGen/AArch64 to make sure the instructions you're expecting get emitted.

include/llvm/CodeGen/ISDOpcodes.h

763

This doesn't need to be a global ISD opcode since only AArch64 ever uses it. It can be added to the list in AArch64ISelLowering.h instead.

Actually, you don't even need any special ISD handling for SUB or CLR. You can make an SDNodeXForm that inserts the appropriate - or ~ during selection. Something like

def i32not : Operand<i64>, SDNodeXForm<imm, [{
  return CurDAG->getTargetConstant(~N->getZExtValue(), SDLoc(N), MVT::i32);
]}>;

If you use that in the destination part of the Pat it'll all work automatically.

lib/Target/AArch64/AArch64ISelLowering.cpp

453

Indentation and commented code here.

10620–10621

How does this work with your special ISD handling above? It seems like it means it would never trigger for bytes and halfs, but there are valid instructions there too.

lib/Target/AArch64/AArch64InstrAtomics.td

410

It looks like this only ever emits the operations with the full barrier, it would be good to support the more relaxed access modes (acquire/release/relaxed) too.

multiclasses ought to be able to reduce a lot of the repetition, but what you'd fundamentally want to do is use PatFrags to cast the node to an AtomicSDNode and check its ordering. Maybe add an extra level in TargetSelectionDAG.td:

multiclass binary_atomic_op_ord {
  def #NAME#_monotonic : PatFrag<(ops node:$ptr, node:$val),
        (!cast<SDNode>(#NAME) node:$ptr, node:$val), [{
      return cast<AtomicSDNode>(N)->getOrdering() == AtomicOrdering::Monotonic;
    }]>;
   ...
}

then add a defm NAME#_8 : binary_atomic_op_ord; (and other sizes) to the binary_atomic_op multiclass.

After that you'd be able to check atomic_load_add_8_acquire and so on in these patterns.

[ajasty-cavium]

Fixed to full diff.

[ajasty-cavium]

In D33586#765646, @t.p.northover wrote:

One other thing you have to be careful of: if an instruction loads into xzr or wzr then it's actually an "STwhatever" operation and defined not to have acquire semantics regardless of its mnemonic. So you probably have to modify AArch64DeadRegisterDefinitionsPass.cpp to ignore these instructions.

Yeah, this was a major concern of mine, technically it's an invalid mnemonic with ladaddal x0, xz, p. One solution was to custom lower all the atomic_load_X where operand 0 wasn't used, and create new AArch64ISD's for them. I wasn't thrilled about this for the first pass (because it seems to work anyway), but for the final it's something to consider.

Also, you should add regression tests under tests/CodeGen/AArch64 to make sure the instructions you're expecting get emitted.

Adapting CodeGen/AArch64/atomic-ops.ll, it's partly in place, will add to updated patch. Any other tests to look at for atomics? Also, I'm using -mcpu=thunderx2t99 for lse as -mcpu=generic+lse doesn't work, is there another preferred mechanism?

Thank you for all the great feedback btw!

lib/Target/AArch64/AArch64ISelLowering.cpp

10620–10621

Currently bytes and halfs cause issues because I didn't get the bitcast right, for the moment everything that isn't cleanly covered falls back to llsc.

lib/Target/AArch64/AArch64InstrAtomics.td

410

Agree completely on relaxed accesses, let me experiment with your PatFrag.

[christof]

In D33586#765995, @ajasty-cavium wrote:

In D33586#765646, @t.p.northover wrote:

One other thing you have to be careful of: if an instruction loads into xzr or wzr then it's actually an "STwhatever" operation and defined not to have acquire semantics regardless of its mnemonic. So you probably have to modify AArch64DeadRegisterDefinitionsPass.cpp to ignore these instructions.

Yeah, this was a major concern of mine, technically it's an invalid mnemonic with ladaddal x0, xz, p. One solution was to custom lower all the atomic_load_X where operand 0 wasn't used, and create new AArch64ISD's for them. I wasn't thrilled about this for the first pass (because it seems to work anyway), but for the final it's something to consider.

I don't think you can say that it works that easily. The instruction will not implement the acquire semantics, so the memory model gets broken. Not an easy thing to track down once that happens. I think ignoring these instructions in AArch64DeadRegisterDefinitionsPass.cpp as suggested is a good approach.

? Also, I'm using -mcpu=thunderx2t99 for lse as -mcpu=generic+lse doesn't work, is there another preferred mechanism?

The target -march=8.1-a should be enough. I gave you patch a spin. The options -march=8.1-a -O1 I can see CASAL being used, but it is not used when passing -O0. I think this might be related to that comment in shouldExpandAtomicCmpXchgInIR().

[ajasty-cavium]

In D33586#768830, @christof wrote:

In D33586#765995, @ajasty-cavium wrote:

In D33586#765646, @t.p.northover wrote:

One other thing you have to be careful of: if an instruction loads into xzr or wzr then it's actually an "STwhatever" operation and defined not to have acquire semantics regardless of its mnemonic. So you probably have to modify AArch64DeadRegisterDefinitionsPass.cpp to ignore these instructions.

Yeah, this was a major concern of mine, technically it's an invalid mnemonic with ladaddal x0, xz, p. One solution was to custom lower all the atomic_load_X where operand 0 wasn't used, and create new AArch64ISD's for them. I wasn't thrilled about this for the first pass (because it seems to work anyway), but for the final it's something to consider.

I don't think you can say that it works that easily. The instruction will not implement the acquire semantics, so the memory model gets broken. Not an easy thing to track down once that happens. I think ignoring these instructions in AArch64DeadRegisterDefinitionsPass.cpp as suggested is a good approach.

Understood, this gives us correctness, will also have to ensure deadregister catches ordering and allows wzr for release or weaker.

? Also, I'm using -mcpu=thunderx2t99 for lse as -mcpu=generic+lse doesn't work, is there another preferred mechanism?

The target -march=8.1-a should be enough. I gave you patch a spin. The options -march=8.1-a -O1 I can see CASAL being used, but it is not used when passing -O0. I think this might be related to that comment in shouldExpandAtomicCmpXchgInIR().

I need to debug this, for -O0 there shouldn't be an issue with the split, the reason for not splitting in fastreg is to prevent a spill between the ldx/stx.

For my testing I see CASAL with '-O0 -march=v8.1-a', can you send me your -emit-llvm?

include/llvm/CodeGen/ISDOpcodes.h

763

One thing about moving this to AArch64ISD namespace, I'm writing my own getNode support for ATOMIC_LOAD_CLR specifically, is there a less intrusive mechanism you can think of?

[t.p.northover]

Understood, this gives us correctness, will also have to ensure deadregister catches ordering and allows wzr for release or weaker.

They're different instructions so that shouldn't be too difficult. FWIW I don't think any kind of AArch64ISD node is necessary or useful for this problem.

include/llvm/CodeGen/ISDOpcodes.h

763

It doesn't seem like a terrible idea to relax the assert in SelectionDAG::getAtomic so that it allows target-specific nodes (by checking Opcode against FIRST_TARGET_MEMORY_OPCODE).

[christof]

In D33586#769261, @ajasty-cavium wrote:

Understood, this gives us correctness, will also have to ensure deadregister catches ordering and allows wzr for release or weaker.

After further digging, I believe that WZR can't be used for any of the LD<OP> instructions. If you do, you are actually encoding ST<OP>. The issue is that a memory barrier that synchronises on loads will synchronise on LD<OP>, but not on ST<OP>. Hence, this transformation might break the memory ordering specified by the program.

[t.p.northover]

After further digging, I believe that WZR can't be used for any of the LD<OP> instructions. If you do, you are actually encoding ST<OP>.

That's pretty much true. "ST<OP> ..." is an alias for "LD<OP> [wx]zr" so you're allowed to write both and they mean the same thing, with one caveat: if the instruction has acquire semantics there is no ST<OP> alias for it. You can only write "ldaddal wzr, w1, [x0]" that way, not as "staddal w1, [x1]" for example.

The issue is that a memory barrier that synchronises on loads will synchronise on LD<OP>, but not on ST<OP>. Hence, this transformation might break the memory ordering specified by the program.

Right. And the dead definitions pass is the only thing likely to insert an XZR behind your back since the register is reserved. (Obviously an explicit pattern could do it too, but you'd only write that where it's valid).

[christof]

In D33586#770042, @t.p.northover wrote:

After further digging, I believe that WZR can't be used for any of the LD<OP> instructions. If you do, you are actually encoding ST<OP>.

That's pretty much true. "ST<OP> ..." is an alias for "LD<OP> [wx]zr" so you're allowed to write both and they mean the same thing, with one caveat: if the instruction has acquire semantics there is no ST<OP> alias for it. You can only write "ldaddal wzr, w1, [x0]" that way, not as "staddal w1, [x1]" for example.

I expect you mean ldaddal w1, wzr, [x0]. Those instructions have their target register as second operand. I dislike that instruction: There is no acquire semantics if the target register is wzr, even though the mnemonic makes it look like there is. It might be better if we diagnose such an instruction instead of using them. Also, I would not be surprised if that instruction behaves exactly the same as staddl w1 [x0].

The issue is that a memory barrier that synchronises on loads will synchronise on LD<OP>, but not on ST<OP>. Hence, this transformation might break the memory ordering specified by the program.

Right. And the dead definitions pass is the only thing likely to insert an XZR behind your back since the register is reserved. (Obviously an explicit pattern could do it too, but you'd only write that where it's valid).

That is certainly what it looks like to me. I suggest to err on the safe side and not use the WZR/XZR as target for any of these LD<OP> instructions.

[t.p.northover]

That is certainly what it looks like to me. I suggest to err on the safe side and not use the WZR/XZR as target for any of these LD<OP> instructions.

I think that's a bit strong, we know exactly what circumstances you can use ZR safely, and about the only way we can ban it in DeadRegisters is with an explicit list of instructions anyway, so just leave the ones where it's valid out of that list.

[t.p.northover]

Oh, and remember this same issue applies to SWP (but not CAS).

[ajasty-cavium]

Updated with dead-register checking, added test.

Temporarily removed SUB/AND, will commit as separate patch.

[ajasty-cavium]

Updated with full context.

[christof]

Seems like a good start for 8.1-A atomics to me. There are some things that might be improved upon, like supporting weaker orderings. Do you plan on doing that work as well?
Just a few questions inline.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

88 ↗

(On Diff #101591)

I think this will also catch a load-exclusive. As far as I know those are ok with XZR/WZR as target register. The only problematic once are the atomicrmw operations. These are SWP and the LD<OP> where <OP> is ADD, CLR, EOR, SET, SMAX, SMIN, UMAX, and UMIN.

But then again, it at least is safe to overestimate.

lib/Target/AArch64/AArch64ISelLowering.cpp

10586

Any reason for expanding cmpxchg again? It was not mentioned in your comment and the test you added will probably fail on it.

test/CodeGen/AArch64/atomic-ops-lse.ll

1–2 ↗

(On Diff #101591)

Why not -mattr=+v8.1a instead of -mcpu=thunderx2t99 ?

[ajasty-cavium]

In D33586#775316, @christof wrote:

Seems like a good start for 8.1-A atomics to me. There are some things that might be improved upon, like supporting weaker orderings. Do you plan on doing that work as well?

Absolutely, and SUB/AND support. Also, for AtomicOrdering without acquire/seq-cst I'll re-enable DeadRegister-s and xform to ST(OP).

Would prefer to land this patch before adding the weaker ordering, but ST(OP) is dependent on weaker ordering. In any case this should perform equal to, or better than current LDX/STX on hardware that supports it.

[t.p.northover]

In any case this should perform equal to, or better than current LDX/STX on hardware that supports it.

For seq_cst and acq_rel accesses, yes. Switching from ldxr/stxr to instructions with full barriers is rather more dubious though. As long as you're planning to support them soon though, I've got no real objections.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

88 ↗

(On Diff #101591)

It'll also catch plain "load atomic" operations. I think a blacklist of instructions is probably best.

[christof]

In D33586#775434, @ajasty-cavium wrote:

Would prefer to land this patch before adding the weaker ordering, but ST(OP) is dependent on weaker ordering. In any case this should perform equal to, or better than current LDX/STX on hardware that supports it.

For that to happen, you need to get test/CodeGen/AArch64/atomic-ops-lse.ll and shouldExpandAtomicCmpXchgInIR() to agree with each other. As it stands, the test will fail because it expect CAS while the code is now expanding it. See my previous comment.

Also, I think it is best to follow Tim's suggestion and make AArch64DeadRegisterDefinitions.cpp match only the AArch::LD<OP>[A|L|AL] instructions instead of the blanket isAtomic(). The danger here is that somebody will see a regression when this is committed. Typically this means that the patch that causes it is reverted if not fixed within the day. Better not to introduce that regression.

[ajasty-cavium]

Fixed dead-register blacklist, and disabled expandInIR for cmp-swap. Passes make check and individual test. Changed test invocation to -mattr=+lse.

[christof]

Looks good to me.
If you could still add a comment in the source on why you blacklist the instruction in the DeadRegisterDefinitionPass that would be appreciated.
Thanks for the nice patch. Looking forward to the patterns for the more relaxed memory models.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

89 ↗

(On Diff #102203)

If you can leave a comment here explaining the reason for this black listing, that would be great for future eyes. Something among the lines of:
// The atomic instructions part of LSE have different memory ordering semantics when targeting the zero register.

[christof]

Sorry, I was too quick. That blacklist is not working properly. Also, it would be good to add some test that show the blacklisting in operation. I just tried an example:

define void @test_atomic_load_add_i32(i32 %offset) nounwind {
     %old = atomicrmw add i32* @var32, i32 %offset seq_cst
   ret void
}

Which should not use WZR, but still uses it with the latest patch.

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

99 ↗

(On Diff #102203)

Hold on. These are Selection DAG instructions. But this pass actually runs just before register allocation and after all DAG lowering is done, does it not?
So you'll need to use AArch64:: LDADDALs and such here, or you will never match these instructions.

[christof]

Found something else.

lib/Target/AArch64/AArch64InstrAtomics.td

445

The CAS and CASP instructions have a different operand order. Correct is:
CASALb $Rs, $Rt, $Rn
This also applies for the patterns below.

test/CodeGen/AArch64/atomic-ops-lse.ll

471 ↗

(On Diff #102203)

I was wondering if you can check the operand order here. I think that since OLD and NEW are directly coming from the function parameters, you should be able to match on w0 and w1 directly. Meaning this could be:

casal w0, w1, [x[[ADDR]]]

That catches any unintended swapping of operands. This holds for all the cmpxchg tests.

[ajasty-cavium]

Fixed CAS, renamed operands to "old/new" for clarity. Also updated test to use x0/x1 to confirm proper operand order.

Also reimplemented blacklist using all aarch64 opcodes.

[christof]

In D33586#783960, @ajasty-cavium wrote:

Fixed CAS, renamed operands to "old/new" for clarity. Also updated test to use x0/x1 to confirm proper operand order.

Great!

Also reimplemented blacklist using all aarch64 opcodes.

Did you test that this blacklist now indeed works as expected? If you have such a test, can you add it. Thanks.

[ajasty-cavium]

Also reimplemented blacklist using all aarch64 opcodes.

Did you test that this blacklist now indeed works as expected? If you have such a test, can you add it. Thanks.

Tested using my development code scrap, will have to make many more test cases when ST(OP)L and barriers are enabled:

78:   f8e8012a        ldaddal x8, x10, [x9]
7c:   f8e82128        ldeoral x8, x8, [x9]

__atomic_fetch_add(&i, l, __ATOMIC_RELAXED);
      j = __sync_fetch_and_xor(&i, l);

lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp

88 ↗

(On Diff #101591)

Did not think that LDX would be issued with a XZR, though now I can see it as a SEQ_CST atomic store. Good catch.

88 ↗

(On Diff #101591)

Point taken, will look at the blacklist.

lib/Target/AArch64/AArch64ISelLowering.cpp

10586

Reversed this.

lib/Target/AArch64/AArch64InstrAtomics.td

445

My first development test case used __atomic_compare_exchange_n without checking the behavior (it could see false positives from the builtin as llvm cmps expected and desired). Reworked my test to actually run the cas in 4 cases to confirm returned value (not just success bool) as appropriate, seems to handle cases properly.

Also renamed operands for clarity.

test/CodeGen/AArch64/atomic-ops-lse.ll

1–2 ↗

(On Diff #101591)

Actually +v8.1a does not give you LSE, but I found -mattr=+lse does. Will add to update.

[ajasty-cavium]

Added additional test cases for dead-register.

[christof]

OK, it looks good to me now. Thanks.

[ajasty-cavium]

Thank you Chris, I do not have commit rights, should I forward to the mailing list?

[christof]

In D33586#786349, @ajasty-cavium wrote:

Thank you Chris, I do not have commit rights, should I forward to the mailing list?

I can commit it on your behalf if you want me to.

[ajasty-cavium]

In D33586#786361, @christof wrote:

In D33586#786349, @ajasty-cavium wrote:

Thank you Chris, I do not have commit rights, should I forward to the mailing list?

I can commit it on your behalf if you want me to.

Would greatly appreciate it!

[christof]

In D33586#786362, @ajasty-cavium wrote:

I can commit it on your behalf if you want me to.

Would greatly appreciate it!

You want me to use the summary as commit message?

[ajasty-cavium]

In D33586#786369, @christof wrote:

In D33586#786362, @ajasty-cavium wrote:

I can commit it on your behalf if you want me to.

Would greatly appreciate it!

You want me to use the summary as commit message?

commit fca5da5b3fa75a7aff1ac622d3a1af44deb0d369
Author: Ananth Jasty <[e-mail]>
Date: Wed Jun 21 01:55:06 2017 -0700

[AARCH64][LSE] Preliminary support for ARMv8.1 LSE Atomics.

Implemented support to AArch64 codegen for ARMv8.1 Large System Extensions atomic instructions. Where supported, these instructions can provide atomic operations with higher performance.

Currently supported operations include: fetch_add, fetch_or, fetch_xor, fetch_smin, fetch_min/max (signed and unsigned), swap, and compare_exchange.

This implementation implies sequential-consistency ordering, more relaxed ordering is under development.

Subtarget->hasLSE is currently supported for Cavium ThunderX2T99.

Differential Revision: https://reviews.llvm.org/D33586

[tschuett]

Is it possible that the tests were excluded from the commit?

[christof]

In D33586#786691, @tschuett wrote:

Is it possible that the tests were excluded from the commit?

Ugh, yes. It seems like I missed adding the new file after applying this patch. I'll commit it now. Thanks for flagging it.

include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     ATOMIC_LOAD_AND,
     ATOMIC_LOAD_OR,
     ATOMIC_LOAD_XOR,
     ATOMIC_LOAD_NAND,
     ATOMIC_LOAD_MIN,
     ATOMIC_LOAD_MAX,
     ATOMIC_LOAD_UMIN,
     ATOMIC_LOAD_UMAX,
+    ATOMIC_LOAD_CLR,
 
     // Masked load and store - consecutive vector load and store operations
     // with additional mask operand that prevents memory accesses to the
     // masked-off lanes.
     MLOAD, MSTORE,
 
     // Masked gather and scatter - load and store operations for a vector of
     // random addresses with additional mask operand that prevents memory

include/llvm/CodeGen/SelectionDAGNodes.h

return N->getOpcode() == ISD::LOAD                ||
            N->getOpcode() == ISD::ATOMIC_LOAD_AND     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_OR      ||
            N->getOpcode() == ISD::ATOMIC_LOAD_XOR     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_NAND    ||
            N->getOpcode() == ISD::ATOMIC_LOAD_MIN     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_MAX     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_UMIN    ||
            N->getOpcode() == ISD::ATOMIC_LOAD_UMAX    ||
+           N->getOpcode() == ISD::ATOMIC_LOAD_CLR     ||
            N->getOpcode() == ISD::ATOMIC_LOAD         ||
            N->getOpcode() == ISD::ATOMIC_STORE        ||
            N->getOpcode() == ISD::MLOAD               ||
            N->getOpcode() == ISD::MSTORE              ||
            N->getOpcode() == ISD::MGATHER             ||
            N->getOpcode() == ISD::MSCATTER            ||
            N->isMemIntrinsic()                        ||
            N->isTargetMemoryOpcode();
return N->getOpcode() == ISD::ATOMIC_CMP_SWAP     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_AND     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_OR      ||
            N->getOpcode() == ISD::ATOMIC_LOAD_XOR     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_NAND    ||
            N->getOpcode() == ISD::ATOMIC_LOAD_MIN     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_MAX     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_UMIN    ||
            N->getOpcode() == ISD::ATOMIC_LOAD_UMAX    ||
+           N->getOpcode() == ISD::ATOMIC_LOAD_CLR     ||
            N->getOpcode() == ISD::ATOMIC_LOAD         ||
            N->getOpcode() == ISD::ATOMIC_STORE;
   }
 };
 
 /// This SDNode is used for target intrinsics that touch
 /// memory and need an associated MachineMemOperand. Its opcode may be
 /// INTRINSIC_VOID, INTRINSIC_W_CHAIN, PREFETCH, or a target-specific opcode

include/llvm/Target/TargetSelectionDAG.td

 def atomic_load_umin : SDNode<"ISD::ATOMIC_LOAD_UMIN", SDTAtomic2,
                     [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
 def atomic_load_umax : SDNode<"ISD::ATOMIC_LOAD_UMAX", SDTAtomic2,
                     [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
 def atomic_load      : SDNode<"ISD::ATOMIC_LOAD", SDTAtomicLoad,
                     [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
 def atomic_store     : SDNode<"ISD::ATOMIC_STORE", SDTAtomicStore,
                     [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
+def atomic_load_clr : SDNode<"ISD::ATOMIC_LOAD_CLR" , SDTAtomic2,
+                    [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
 
 def masked_store : SDNode<"ISD::MSTORE",  SDTMaskedStore,
                        [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
 def masked_load  : SDNode<"ISD::MLOAD",  SDTMaskedLoad,
                        [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
 def masked_scatter : SDNode<"ISD::MSCATTER",  SDTMaskedScatter,
                        [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
 def masked_gather  : SDNode<"ISD::MGATHER",  SDTMaskedGather,

 defm atomic_load_and  : binary_atomic_op<atomic_load_and>;
 defm atomic_load_or   : binary_atomic_op<atomic_load_or>;
 defm atomic_load_xor  : binary_atomic_op<atomic_load_xor>;
 defm atomic_load_nand : binary_atomic_op<atomic_load_nand>;
 defm atomic_load_min  : binary_atomic_op<atomic_load_min>;
 defm atomic_load_max  : binary_atomic_op<atomic_load_max>;
 defm atomic_load_umin : binary_atomic_op<atomic_load_umin>;
 defm atomic_load_umax : binary_atomic_op<atomic_load_umax>;
+defm atomic_load_clr  : binary_atomic_op<atomic_load_clr>;
 defm atomic_store     : binary_atomic_op<atomic_store>;
 
 def atomic_load_8 :
   PatFrag<(ops node:$ptr),
           (atomic_load node:$ptr), [{
   return cast<AtomicSDNode>(N)->getMemoryVT() == MVT::i8;
 }]>;
 def atomic_load_16 :

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

static void AddNodeIDCustom(FoldingSetNodeID &ID, const SDNode *N) {
   case ISD::ATOMIC_CMP_SWAP_WITH_SUCCESS:
   case ISD::ATOMIC_SWAP:
   case ISD::ATOMIC_LOAD_ADD:
   case ISD::ATOMIC_LOAD_SUB:
   case ISD::ATOMIC_LOAD_AND:
   case ISD::ATOMIC_LOAD_OR:
   case ISD::ATOMIC_LOAD_XOR:
   case ISD::ATOMIC_LOAD_NAND:
+  case ISD::ATOMIC_LOAD_CLR:
   case ISD::ATOMIC_LOAD_MIN:
   case ISD::ATOMIC_LOAD_MAX:
   case ISD::ATOMIC_LOAD_UMIN:
   case ISD::ATOMIC_LOAD_UMAX:
   case ISD::ATOMIC_LOAD:
   case ISD::ATOMIC_STORE: {
     const AtomicSDNode *AT = cast<AtomicSDNode>(N);
     ID.AddInteger(AT->getMemoryVT().getRawBits());
SDValue SelectionDAG::getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT,
                                 SDValue Chain, SDValue Ptr, SDValue Val,
                                 MachineMemOperand *MMO) {
   assert((Opcode == ISD::ATOMIC_LOAD_ADD ||
           Opcode == ISD::ATOMIC_LOAD_SUB ||
           Opcode == ISD::ATOMIC_LOAD_AND ||
           Opcode == ISD::ATOMIC_LOAD_OR ||
           Opcode == ISD::ATOMIC_LOAD_XOR ||
           Opcode == ISD::ATOMIC_LOAD_NAND ||
+          Opcode == ISD::ATOMIC_LOAD_CLR ||
           Opcode == ISD::ATOMIC_LOAD_MIN ||
           Opcode == ISD::ATOMIC_LOAD_MAX ||
           Opcode == ISD::ATOMIC_LOAD_UMIN ||
           Opcode == ISD::ATOMIC_LOAD_UMAX ||
           Opcode == ISD::ATOMIC_SWAP ||
           Opcode == ISD::ATOMIC_STORE) &&
          "Invalid Atomic Op");
 

lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

void AArch64DAGToDAGISel::Select(SDNode *Node) {
   // Few custom selection stuff.
   EVT VT = Node->getValueType(0);
 
   switch (Node->getOpcode()) {
   default:
     break;
 
   case ISD::ATOMIC_CMP_SWAP:
+    // Leave IR for LSE if subtarget supports it.
+    if (Subtarget->hasLSE()) break;
     SelectCMP_SWAP(Node);
     return;
 
   case ISD::READ_REGISTER:
     if (tryReadRegister(Node))
       return;
     break;
 

lib/Target/AArch64/AArch64ISelLowering.h

private:
   SDValue LowerFP_EXTEND(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFP_ROUND(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFP_TO_INT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINT_TO_FP(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorAND(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerCONCAT_VECTORS(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFSINCOS(SDValue Op, SelectionDAG &DAG) const;
+  SDValue LowerATOMIC_LOAD_COMBINED(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVECREDUCE(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue BuildSDIVPow2(SDNode *N, const APInt &Divisor, SelectionDAG &DAG,
                         std::vector<SDNode *> *Created) const override;
   SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
                           int &ExtraSteps, bool &UseOneConst,
                           bool Reciprocal) const override;
   SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,

lib/Target/AArch64/AArch64ISelLowering.cpp

for (MVT Ty : {MVT::f32, MVT::f64}) {
     setOperationAction(ISD::FMAXNUM, Ty, Legal);
     setOperationAction(ISD::FMINNAN, Ty, Legal);
     setOperationAction(ISD::FMAXNAN, Ty, Legal);
   }
 
   setOperationAction(ISD::PREFETCH, MVT::Other, Custom);
 
   setOperationAction(ISD::ATOMIC_CMP_SWAP, MVT::i128, Custom);
+  if (Subtarget->hasLSE()){
+  // LSE does not support these operations natively, handle them
+  // by breaking them in to constituent ops.
+  //setOperationAction(ISD::ATOMIC_LOAD_SUB, MVT::i8, Custom);
+  //setOperationAction(ISD::ATOMIC_LOAD_SUB, MVT::i16, Custom);
+  setOperationAction(ISD::ATOMIC_LOAD_SUB, MVT::i32, Custom);
+  setOperationAction(ISD::ATOMIC_LOAD_SUB, MVT::i64, Custom);
+  //setOperationAction(ISD::ATOMIC_LOAD_AND, MVT::i8, Custom);
+  //setOperationAction(ISD::ATOMIC_LOAD_AND, MVT::i16, Custom);
+  setOperationAction(ISD::ATOMIC_LOAD_AND, MVT::i32, Custom);
+  setOperationAction(ISD::ATOMIC_LOAD_AND, MVT::i64, Custom);
+  }
 
   // Lower READCYCLECOUNTER using an mrs from PMCCNTR_EL0.
   // This requires the Performance Monitors extension.
   if (Subtarget->hasPerfMon())
     setOperationAction(ISD::READCYCLECOUNTER, MVT::i64, Legal);
 
   if (Subtarget->isTargetMachO()) {
     // For iOS, we don't want to the normal expansion of a libcall to
case Intrinsic::aarch64_neon_smin:
     return DAG.getNode(ISD::SMIN, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   case Intrinsic::aarch64_neon_umin:
     return DAG.getNode(ISD::UMIN, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   }
 }
 
+SDValue AArch64TargetLowering::LowerATOMIC_LOAD_COMBINED(SDValue Op,
+                SelectionDAG &DAG) const {
+  AtomicSDNode *AN = cast<AtomicSDNode>(Op.getNode());
+
+  SDLoc DL(Op);
+  MVT VT = Op->getSimpleValueType(0);
+  SDValue Chain = Op->getOperand(0);
+  SDValue LHS = Op->getOperand(1);
+  SDValue RHS = Op->getOperand(2);
+
+  if (Op.getOpcode() == ISD::ATOMIC_LOAD_SUB) {
+    // AtomicLoadSub breaks into negate and atomic-add.
+    RHS = DAG.getNode(ISD::SUB, DL, VT, DAG.getConstant(0, DL, VT), RHS);
+    return DAG.getAtomic(ISD::ATOMIC_LOAD_ADD, DL, VT, Chain, LHS, RHS, AN->getMemOperand());
+  }
+  if (Op.getOpcode() == ISD::ATOMIC_LOAD_AND) {
+    // AtomicLoadAnd breaks into invert and atomic-bit-clear.
+    RHS = DAG.getNOT(DL, RHS, VT);
+    return DAG.getAtomic(ISD::ATOMIC_LOAD_CLR, DL, VT, Chain, LHS, RHS, AN->getMemOperand());
+  }
+  llvm_unreachable("Attempt to custom lower unknown atomic op.");
+}
+
 SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
                                               SelectionDAG &DAG) const {
   switch (Op.getOpcode()) {
   default:
     llvm_unreachable("unimplemented operand");
     return SDValue();
   case ISD::BITCAST:
     return LowerBITCAST(Op, DAG);
SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
   case ISD::FP_TO_UINT:
     return LowerFP_TO_INT(Op, DAG);
   case ISD::FSINCOS:
     return LowerFSINCOS(Op, DAG);
   case ISD::MUL:
     return LowerMUL(Op, DAG);
   case ISD::INTRINSIC_WO_CHAIN:
     return LowerINTRINSIC_WO_CHAIN(Op, DAG);
+  case ISD::ATOMIC_LOAD_SUB:
+  case ISD::ATOMIC_LOAD_AND:
+    return LowerATOMIC_LOAD_COMBINED(Op, DAG);
   case ISD::VECREDUCE_ADD:
   case ISD::VECREDUCE_SMAX:
   case ISD::VECREDUCE_SMIN:
   case ISD::VECREDUCE_UMAX:
   case ISD::VECREDUCE_UMIN:
   case ISD::VECREDUCE_FMAX:
   case ISD::VECREDUCE_FMIN:
     return LowerVECREDUCE(Op, DAG);
static void ReplaceCMP_SWAP_128Results(SDNode *N,
   Results.push_back(SDValue(CmpSwap, 1));
   Results.push_back(SDValue(CmpSwap, 3));
 }
 
 void AArch64TargetLowering::ReplaceNodeResults(
     SDNode *N, SmallVectorImpl<SDValue> &Results, SelectionDAG &DAG) const {
   switch (N->getOpcode()) {
   default:
+    DEBUG(dbgs() << "Invalid Opcode in replace:" << N->getOperationName());
+    N->dumprFull();
     llvm_unreachable("Don't know how to custom expand this");
   case ISD::BITCAST:
     ReplaceBITCASTResults(N, Results, DAG);
     return;
   case ISD::VECREDUCE_ADD:
   case ISD::VECREDUCE_SMAX:
   case ISD::VECREDUCE_SMIN:
   case ISD::VECREDUCE_UMAX:
void AArch64TargetLowering::ReplaceNodeResults(
   case ISD::FP_TO_UINT:
   case ISD::FP_TO_SINT:
     assert(N->getValueType(0) == MVT::i128 && "unexpected illegal conversion");
     // Let normal code take care of it by not adding anything to Results.
     return;
   case ISD::ATOMIC_CMP_SWAP:
     ReplaceCMP_SWAP_128Results(N, Results, DAG);
     return;
+  case ISD::ATOMIC_LOAD_AND:
+    ReplaceReductionResults(N, Results, DAG, ISD::ATOMIC_LOAD_CLR, ISD::ATOMIC_LOAD_AND);
+    return;
+  case ISD::ATOMIC_LOAD_SUB:
+    ReplaceReductionResults(N, Results, DAG, ISD::ATOMIC_LOAD_SUB, ISD::ATOMIC_LOAD_ADD);
+    return;
+
   }
 }
 
 bool AArch64TargetLowering::useLoadStackGuardNode() const {
   if (Subtarget->isTargetAndroid() || Subtarget->isTargetFuchsia())
     return TargetLowering::useLoadStackGuardNode();
   return true;
 }
 
 unsigned AArch64TargetLowering::combineRepeatedFPDivisors() const {
   // Combine multiple FDIVs with the same divisor into multiple FMULs by the
   // reciprocal if there are three or more FDIVs.
   return 3;
 }
 
 TargetLoweringBase::LegalizeTypeAction
 AArch64TargetLowering::getPreferredVectorAction(EVT VT) const {
   MVT SVT = VT.getSimpleVT();
   // During type legalization, we prefer to widen v1i8, v1i16, v1i32  to v8i8,
   // v4i16, v2i32 instead of to promote.
   if (SVT == MVT::v1i8 || SVT == MVT::v1i16 || SVT == MVT::v1i32
       || SVT == MVT::v1f32)
     return TypeWidenVector;
 
   return TargetLoweringBase::getPreferredVectorAction(VT);
 }
 
AArch64TargetLowering::shouldExpandAtomicLoadInIR(LoadInst *LI) const {
   unsigned Size = LI->getType()->getPrimitiveSizeInBits();
   return Size == 128 ? AtomicExpansionKind::LLSC : AtomicExpansionKind::None;
 }
 
 // For the real atomic operations, we have ldxr/stxr up to 128 bits,
 TargetLowering::AtomicExpansionKind
 AArch64TargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
   unsigned Size = AI->getType()->getPrimitiveSizeInBits();
-  return Size <= 128 ? AtomicExpansionKind::LLSC : AtomicExpansionKind::None;
+  if (Size > 128) return AtomicExpansionKind::None;
+  // Nand not supported in LSE.
+  if (AI->getOperation() == AtomicRMWInst::Nand) return AtomicExpansionKind::LLSC;
+  // Need to fix bitcast.
+  if (((AI->getOperation() == AtomicRMWInst::And) && (Size < 32)) ||
+      ((AI->getOperation() == AtomicRMWInst::Sub) && (Size < 32))) return AtomicExpansionKind::LLSC;
+  // Leave 128 bits to LLSC.
+  return (Subtarget->hasLSE() && Size < 128) ? AtomicExpansionKind::None : AtomicExpansionKind::LLSC;
 }
 
 bool AArch64TargetLowering::shouldExpandAtomicCmpXchgInIR(
     AtomicCmpXchgInst *AI) const {
   // At -O0, fast-regalloc cannot cope with the live vregs necessary to
   // implement cmpxchg without spilling. If the address being exchanged is also
   // on the stack and close enough to the spill slot, this can lead to a
   // situation where the monitor always gets cleared and the atomic operation
   // can never succeed. So at -O0 we need a late-expanded pseudo-inst instead.
+  if (Subtarget->hasLSE()) return false;
   return getTargetMachine().getOptLevel() != 0;
 }
 
 Value *AArch64TargetLowering::emitLoadLinked(IRBuilder<> &Builder, Value *Addr,
                                              AtomicOrdering Ord) const {
   Module *M = Builder.GetInsertBlock()->getParent()->getParent();
   Type *ValTy = cast<PointerType>(Addr->getType())->getElementType();
   bool IsAcquire = isAcquireOrStronger(Ord);

lib/Target/AArch64/AArch64InstrAtomics.td

 }
 
 let Constraints = "@earlyclobber $RdLo,@earlyclobber $RdHi,@earlyclobber $scratch",
     mayLoad = 1, mayStore = 1 in
 def CMP_SWAP_128 : Pseudo<(outs GPR64:$RdLo, GPR64:$RdHi, GPR32:$scratch),
                           (ins GPR64:$addr, GPR64:$desiredLo, GPR64:$desiredHi,
                                GPR64:$newLo, GPR64:$newHi), []>,
                    Sched<[WriteAtomic]>;
+
+// v8.1 Atomic instructions:
+def : Pat<(atomic_load_clr_8 GPR64:$Rn, GPR32:$Rs), (LDCLRALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_clr_16 GPR64:$Rn, GPR32:$Rs), (LDCLRALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_clr_32 GPR64:$Rn, GPR32:$Rs), (LDCLRALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_clr_64 GPR64:$Rn, GPR64:$Rs), (LDCLRALd GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_add_8 GPR64:$Rn, GPR32:$Rs), (LDADDALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_add_16 GPR64:$Rn, GPR32:$Rs), (LDADDALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_add_32 GPR64:$Rn, GPR32:$Rs), (LDADDALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_add_64 GPR64:$Rn, GPR64:$Rs), (LDADDALd GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_or_8 GPR64:$Rn, GPR32:$Rs), (LDSETALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_or_16 GPR64:$Rn, GPR32:$Rs), (LDSETALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_or_32 GPR64:$Rn, GPR32:$Rs), (LDSETALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_or_64 GPR64:$Rn, GPR64:$Rs), (LDSETALd GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_xor_8 GPR64:$Rn, GPR32:$Rs), (LDEORALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_xor_16 GPR64:$Rn, GPR32:$Rs), (LDEORALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_xor_32 GPR64:$Rn, GPR32:$Rs), (LDEORALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_xor_64 GPR64:$Rn, GPR64:$Rs), (LDEORALd GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_max_8 GPR64:$Rn, GPR32:$Rs), (LDSMAXALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_max_16 GPR64:$Rn, GPR32:$Rs), (LDSMAXALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_max_32 GPR64:$Rn, GPR32:$Rs), (LDSMAXALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_max_64 GPR64:$Rn, GPR64:$Rs), (LDSMAXALd GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_umax_8 GPR64:$Rn, GPR32:$Rs), (LDUMAXALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umax_16 GPR64:$Rn, GPR32:$Rs), (LDUMAXALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umax_32 GPR64:$Rn, GPR32:$Rs), (LDUMAXALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umax_64 GPR64:$Rn, GPR64:$Rs), (LDUMAXALd GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_min_8 GPR64:$Rn, GPR32:$Rs), (LDSMINALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_min_16 GPR64:$Rn, GPR32:$Rs), (LDSMINALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_min_32 GPR64:$Rn, GPR32:$Rs), (LDSMINALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_min_64 GPR64:$Rn, GPR64:$Rs), (LDSMINALd GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_load_umin_8 GPR64:$Rn, GPR32:$Rs), (LDUMINALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umin_16 GPR64:$Rn, GPR32:$Rs), (LDUMINALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umin_32 GPR64:$Rn, GPR32:$Rs), (LDUMINALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_load_umin_64 GPR64:$Rn, GPR64:$Rs), (LDUMINALd GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_cmp_swap_8 GPR64:$Rn, GPR32:$Rs, GPR32:$Rt), (CASALb GPR32:$Rt, GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_cmp_swap_16 GPR64:$Rn, GPR32:$Rs, GPR32:$Rt), (CASALh GPR32:$Rt, GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_cmp_swap_32 GPR64:$Rn, GPR32:$Rs, GPR32:$Rt), (CASALs GPR32:$Rt, GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_cmp_swap_64 GPR64:$Rn, GPR64:$Rs, GPR64:$Rt), (CASALd GPR64:$Rt, GPR64:$Rs, GPR64sp:$Rn)>;
+
+def : Pat<(atomic_swap_8 GPR64:$Rn, GPR32:$Rs), (SWPALb GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_swap_16 GPR64:$Rn, GPR32:$Rs), (SWPALh GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_swap_32 GPR64:$Rn, GPR32:$Rs), (SWPALs GPR32:$Rs, GPR64sp:$Rn)>;
+def : Pat<(atomic_swap_64 GPR64:$Rn, GPR64:$Rs), (SWPALd GPR64:$Rs, GPR64sp:$Rn)>;