[AArch64] Tie source and destination operands for AESMC/AESIMC.

[fhahn]

Summary

Most CPUs implementing AES fusion require instruction pairs of the form

AESE Vn, _
AESMC Vn, Vn

and

AESD Vn, _
AESIMC Vn, Vn

The constraint is added to AES(I)MC instructions which use the result of
an AES(E|D) instruction by using AES(I)MCTrr pseudo instructions, which
constraint source and destination registers to be the same.

A nice side effect of this change is that now all possible pairs are
scheduled back-to-back on the exynos-m1 for the misched-fusion-aes.ll
test case.

I had to update aes_load_store. The version I added initially was very
reduced and with the new constraint, AESE/AESMC could not be scheduled
back-to-back. I updated the test to be more realistic and still expose
the same scheduling problem as the initial test case.

[evandro]

I thought that the data dependency was enough for fusion in general:

AESE Vi, Vj
AESMC Vk, Vi

And:

AESD Vi, Vj
AESIMC Vk, Vi

[fhahn]

In D35299#806967, @evandro wrote:

I thought that the data dependency was enough for fusion in general:

It is for the MacroFusion pass and they will be scheduled back-to-back, regardless of the register constraint.

AESE Vi, Vj
AESMC Vk, Vi

And:

AESD Vi, Vj
AESIMC Vk, Vi

But unless Vk == Vi for AESMC/AESIMC, the most CPUs won't be able to fuse the pair and won't execute it more efficiently.

[evandro]

In D35299#807127, @fhahn wrote:

But unless Vk == Vi for AESMC/AESIMC, the most CPUs won't be able to fuse the pair and won't execute it more efficiently.

Perhaps most CPUs require this, but certainly not all. Thus, would there be a way to make this constraint specific to such CPUs?

[fhahn]

In D35299#807213, @evandro wrote:

In D35299#807127, @fhahn wrote:

But unless Vk == Vi for AESMC/AESIMC, the most CPUs won't be able to fuse the pair and won't execute it more efficiently.

Perhaps most CPUs require this, but certainly not all. Thus, would there be a way to make this constraint specific to such CPUs?

One solution I could think of would be to add pseudo AESMC/AESIMC instructions and expand them to either AESMC/AESIMC with constraint or without constraint, depending on the target-features. There probably is a more elegant solution I am not aware of though, so I would appreciate any input :)

[fhahn]

I've introduced AES(I)MCTrr pseudo instructions which have the "src = dst" constraint and patterns to use them for AES instruction pairs, if the result of the AESE/D in the pair has a single user.

I think this change keeps the codegen impact on CPUs that do not require the constraint to fuse AES instructions as small as possible, as the constraint should reduce register pressure and I don't expect AESMC instructions with the same source/destination register to be slower than if the registers are different.

@evandro does this address your concern? In my opinion adding a separate attribute seems not worth it.

[fhahn]

Ping. Do you think the route with pseudo instructions is a workable approach?

[evandro]

I kind of prefer the previous patch for its being less intrusive, but I'm afraid that it would create problems for the assembler too, wouldn't it?

[fhahn]

The previous patch had a smaller diff and was simpler, because it added the constraint to all AESMC/AESIMC instructions. The latest version of the patch only applies the constraint to AESMC/AESIMC instructions that are part of a pair and for targets with FeatureFuseAES, so the impact on the generated code is smaller. I think applying the constraint only to instruction pairs is nice, but I would be fine with adding the constraint to all AESMC/AESIMC instructions too.

[t.p.northover]

I'm afraid that it would create problems for the assembler too, wouldn't it?

Yeah, the original causes failures in test/MC. I wonder why that wasn't spotted earlier?

Anyway, I'm not sure the "single-use" heuristic is actually what we should care about. As far as I understand AES these instructions are only ever going to be seen with single-uses in reality so optimizing for other cases is probably premature.

For out-of-order non-fused implementations the register shouldn't matter as far as I can see, which leaves in-order unfused implementations. I've got much less experience there, but don't they tend to have early forwarding to dependent instructions anyway?

So perhaps a hybrid approach where the Pattern always picks these pseudo-instructions would be best?

[fhahn]

In D35299#816711, @t.p.northover wrote:

I'm afraid that it would create problems for the assembler too, wouldn't it?

Yeah, the original causes failures in test/MC. I wonder why that wasn't spotted earlier?

I'm not entirely sure what you mean here. I've updated one of the AArch64 MC tests to build with +fuse-aes, to make sure the constraint does not mess up MC codegen.

Anyway, I'm not sure the "single-use" heuristic is actually what we should care about. As far as I understand AES these instructions are only ever going to be seen with single-uses in reality so optimizing for other cases is probably premature.

For out-of-order non-fused implementations the register shouldn't matter as far as I can see, which leaves in-order unfused implementations. I've got much less experience there, but don't they tend to have early forwarding to dependent instructions anyway?

So perhaps a hybrid approach where the Pattern always picks these pseudo-instructions would be best?

Yes, the single-use case is what we should see in practice. I've removed the single use pattern from the patch.

[t.p.northover]

I'm not entirely sure what you mean here.

I downloaded and built the very first patch, and the obvious test/MC/AArch64 tests failed because they expected "aesmc v0.16b, v1.16b" to be valid syntax. It's not an issue with the newest patch but it confirms evandro's suspicion about the earlier one.

Yes, the single-use case is what we should see in practice. I've removed the single use pattern from the patch.

Thanks. I don't know about @evandro, but I think this is looking pretty reasonable now. I certainly can't think of a fundamentally better approach.

[fhahn]

ping

[evandro]

I agree, this is a sensible approach.

However, the pseudo indicator in the middle of the instruction (AES.*T.*) looks too much like an instruction. Can this be made more explicit that it's a pseudo, perhaps as P_AES.* or some such?

[t.p.northover]

Not convinced by a Hungarian tagging like that. It's not what happens with existing pseudo-instructions. If anything they have a semi-descriptive name: "MOVaddr", "MOVaddrJT", "CMP_SWAP_8", ...

[fhahn]

How about something like AESIMCrrTied?

[evandro]

In D35299#824492, @fhahn wrote:

How about something like AESIMCrrTied?

LGTM

[t.p.northover]

Yep, I'd be happy with that too. Commit away!

[fhahn]

Renamed pseudo instructions, will commit soon

lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
   case AArch64::CMP_SWAP_64:
     return expandCMP_SWAP(MBB, MBBI,
                           AArch64::LDAXRX, AArch64::STLXRX, AArch64::SUBSXrs,
                           AArch64_AM::getShifterImm(AArch64_AM::LSL, 0),
                           AArch64::XZR, NextMBBI);
   case AArch64::CMP_SWAP_128:
     return expandCMP_SWAP_128(MBB, MBBI, NextMBBI);
 
+  case AArch64::AESMCrrTied:
+  case AArch64::AESIMCrrTied: {
+    MachineInstrBuilder MIB =
+    BuildMI(MBB, MBBI, MI.getDebugLoc(),
+            TII->get(Opcode == AArch64::AESMCrrTied ? AArch64::AESMCrr :
+                                                      AArch64::AESIMCrr))
+      .add(MI.getOperand(0))
+      .add(MI.getOperand(1));
+    transferImpOps(MI, MIB, MIB);
+    MI.eraseFromParent();
+    return true;
+   }
   }
   return false;
 }
 
 /// \brief Iterate over the instructions in basic block MBB and expand any
 /// pseudo instructions.  Return true if anything was modified.
 bool AArch64ExpandPseudo::expandMBB(MachineBasicBlock &MBB) {
   bool Modified = false;

lib/Target/AArch64/AArch64InstrInfo.td

def HasRAS           : Predicate<"Subtarget->hasRAS()">,
                                  AssemblerPredicate<"FeatureRAS", "ras">;
 def HasRDM           : Predicate<"Subtarget->hasRDM()">,
                                  AssemblerPredicate<"FeatureRDM", "rdm">;
 def HasPerfMon       : Predicate<"Subtarget->hasPerfMon()">;
 def HasFullFP16      : Predicate<"Subtarget->hasFullFP16()">,
                                  AssemblerPredicate<"FeatureFullFP16", "fullfp16">;
 def HasSPE           : Predicate<"Subtarget->hasSPE()">,
                                  AssemblerPredicate<"FeatureSPE", "spe">;
+def HasFuseAES       : Predicate<"Subtarget->hasFuseAES()">,
+                                 AssemblerPredicate<"FeatureFuseAES",
+                                 "fuse-aes">;
 def HasSVE           : Predicate<"Subtarget->hasSVE()">,
                                  AssemblerPredicate<"FeatureSVE", "sve">;
 
 def IsLE             : Predicate<"Subtarget->isLittleEndian()">;
 def IsBE             : Predicate<"!Subtarget->isLittleEndian()">;
 def UseAlternateSExtLoadCVTF32
     : Predicate<"Subtarget->useAlternateSExtLoadCVTF32Pattern()">;
 

 // Crypto extensions
 //----------------------------------------------------------------------------
 
 def AESErr   : AESTiedInst<0b0100, "aese",   int_aarch64_crypto_aese>;
 def AESDrr   : AESTiedInst<0b0101, "aesd",   int_aarch64_crypto_aesd>;
 def AESMCrr  : AESInst<    0b0110, "aesmc",  int_aarch64_crypto_aesmc>;
 def AESIMCrr : AESInst<    0b0111, "aesimc", int_aarch64_crypto_aesimc>;
 
+// Pseudo instructions for AESMCrr/AESIMCrr with a register constraint required
+// for AES fusion on some CPUs.
+let hasSideEffects = 0, mayStore = 0, mayLoad = 0 in {
+def AESMCrrTied: Pseudo<(outs V128:$Rd), (ins V128:$Rn), [], "$Rn = $Rd">,
+                        Sched<[WriteV]>;
+def AESIMCrrTied: Pseudo<(outs V128:$Rd), (ins V128:$Rn), [], "$Rn = $Rd">,
+                         Sched<[WriteV]>;
+}
+
+// Only use constrained versions of AES(I)MC instructions if they are paired with
+// AESE/AESD.
+def : Pat<(v16i8 (int_aarch64_crypto_aesmc
+            (v16i8 (int_aarch64_crypto_aese (v16i8 V128:$src1),
+                                            (v16i8 V128:$src2))))),
+          (v16i8 (AESMCrrTied (v16i8 (AESErr (v16i8 V128:$src1),
+                                             (v16i8 V128:$src2)))))>,
+          Requires<[HasFuseAES]>;
+
+def : Pat<(v16i8 (int_aarch64_crypto_aesimc
+            (v16i8 (int_aarch64_crypto_aesd (v16i8 V128:$src1),
+                                            (v16i8 V128:$src2))))),
+          (v16i8 (AESIMCrrTied (v16i8 (AESDrr (v16i8 V128:$src1),
+                                              (v16i8 V128:$src2)))))>,
+          Requires<[HasFuseAES]>;
+
 def SHA1Crrr     : SHATiedInstQSV<0b000, "sha1c",   int_aarch64_crypto_sha1c>;
 def SHA1Prrr     : SHATiedInstQSV<0b001, "sha1p",   int_aarch64_crypto_sha1p>;
 def SHA1Mrrr     : SHATiedInstQSV<0b010, "sha1m",   int_aarch64_crypto_sha1m>;
 def SHA1SU0rrr   : SHATiedInstVVV<0b011, "sha1su0", int_aarch64_crypto_sha1su0>;
 def SHA256Hrrr   : SHATiedInstQQV<0b100, "sha256h", int_aarch64_crypto_sha256h>;
 def SHA256H2rrr  : SHATiedInstQQV<0b101, "sha256h2",int_aarch64_crypto_sha256h2>;
 def SHA256SU1rrr :SHATiedInstVVV<0b110, "sha256su1",int_aarch64_crypto_sha256su1>;
 

lib/Target/AArch64/AArch64MacroFusion.cpp

if (SecondOpcode == AArch64::CBNZW || SecondOpcode == AArch64::CBNZX ||
       case AArch64::INSTRUCTION_LIST_END:
         return true;
       }
 
   if (ST.hasFuseAES())
     // Fuse AES crypto operations.
     switch(SecondOpcode) {
     // AES encode.
-    case AArch64::AESMCrr :
+    case AArch64::AESMCrr:
+    case AArch64::AESMCrrTied:
       return FirstOpcode == AArch64::AESErr ||
              FirstOpcode == AArch64::INSTRUCTION_LIST_END;
     // AES decode.
     case AArch64::AESIMCrr:
+    case AArch64::AESIMCrrTied:
       return FirstOpcode == AArch64::AESDrr ||
              FirstOpcode == AArch64::INSTRUCTION_LIST_END;
     }
 
   if (ST.hasFuseLiterals())
     // Fuse literal generation operations.
     switch (SecondOpcode) {
     // PC relative address.

test/CodeGen/AArch64/misched-fusion-aes.ll

-; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=+fuse-aes,+crypto | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSEALLPAIRS
-; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=generic -mattr=+crypto | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSEALLPAIRS
-; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a53 | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSEALLPAIRS
-; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a57 | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSEALLPAIRS
-; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a72 | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSEALLPAIRS
-; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a73 | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSEALLPAIRS
-; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m1  | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKM1
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=+fuse-aes,+crypto | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=generic -mattr=+crypto | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a53 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a57 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a72 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a73 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m1  | FileCheck %s
 
 declare <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d, <16 x i8> %k)
 declare <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %d)
 declare <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d, <16 x i8> %k)
 declare <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %d)
 
 define void @aesea(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   %d0 = load <16 x i8>, <16 x i8>* %a0
define void @aesea(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   store <16 x i8> %h1, <16 x i8>* %c1
   %c2 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 2
   store <16 x i8> %h2, <16 x i8>* %c2
   %c3 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 3
   store <16 x i8> %h3, <16 x i8>* %c3
   ret void
 
 ; CHECK-LABEL: aesea:
-; CHECKFUSEALLPAIRS: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesmc {{v[0-7].16b}}, [[VA]]
-; CHECKFUSEALLPAIRS: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesmc {{v[0-7].16b}}, [[VB]]
-; CHECKFUSEALLPAIRS: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesmc {{v[0-7].16b}}, [[VC]]
-; CHECKFUSEALLPAIRS: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesmc {{v[0-7].16b}}, [[VD]]
-; CHECKFUSEALLPAIRS: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesmc {{v[0-7].16b}}, [[VE]]
-; CHECKFUSEALLPAIRS: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesmc {{v[0-7].16b}}, [[VF]]
-; CHECKFUSEALLPAIRS: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesmc {{v[0-7].16b}}, [[VG]]
-; CHECKFUSEALLPAIRS: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesmc {{v[0-7].16b}}, [[VH]]
-; CHECKFUSEALLPAIRS-NOT: aesmc
-
-; CHECKM1: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VA]]
-; CHECKM1: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VB]]
-; CHECKM1: aese {{v[0-7].16b}}, {{v[0-7].16b}}
-; CHECKM1: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VC]]
-; CHECKM1: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VD]]
-; CHECKM1: aesmc {{v[0-7].16b}}, [[VH]]
-; CHECKM1: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VE]]
-; CHECKM1: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VF]]
-; CHECKM1: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VG]]
+; CHECK: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesmc [[VA]], [[VA]]
+; CHECK: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesmc [[VB]], [[VB]]
+; CHECK: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesmc [[VC]], [[VC]]
+; CHECK: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesmc [[VD]], [[VD]]
+; CHECK: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesmc [[VE]], [[VE]]
+; CHECK: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesmc [[VF]], [[VF]]
+; CHECK: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesmc [[VG]], [[VG]]
+; CHECK: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesmc [[VH]], [[VH]]
+; CHECK-NOT: aesmc
 }
 
 define void @aesda(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   %d0 = load <16 x i8>, <16 x i8>* %a0
   %a1 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 1
   %d1 = load <16 x i8>, <16 x i8>* %a1
   %a2 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 2
   %d2 = load <16 x i8>, <16 x i8>* %a2
define void @aesda(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   store <16 x i8> %h1, <16 x i8>* %c1
   %c2 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 2
   store <16 x i8> %h2, <16 x i8>* %c2
   %c3 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 3
   store <16 x i8> %h3, <16 x i8>* %c3
   ret void
 
 ; CHECK-LABEL: aesda:
-; CHECKFUSEALLPAIRS: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesimc {{v[0-7].16b}}, [[VA]]
-; CHECKFUSEALLPAIRS: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesimc {{v[0-7].16b}}, [[VB]]
-; CHECKFUSEALLPAIRS: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesimc {{v[0-7].16b}}, [[VC]]
-; CHECKFUSEALLPAIRS: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesimc {{v[0-7].16b}}, [[VD]]
-; CHECKFUSEALLPAIRS: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesimc {{v[0-7].16b}}, [[VE]]
-; CHECKFUSEALLPAIRS: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesimc {{v[0-7].16b}}, [[VF]]
-; CHECKFUSEALLPAIRS: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesimc {{v[0-7].16b}}, [[VG]]
-; CHECKFUSEALLPAIRS: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKFUSEALLPAIRS-NEXT: aesimc {{v[0-7].16b}}, [[VH]]
-; CHECKFUSEALLPAIRS-NOT: aesimc
-
-; CHECKM1: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VA]]
-; CHECKM1: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VB]]
-; CHECKM1: aesd {{v[0-7].16b}}, {{v[0-7].16b}}
-; CHECKM1: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VC]]
-; CHECKM1: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VD]]
-; CHECKM1: aesimc {{v[0-7].16b}}, [[VH]]
-; CHECKM1: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VE]]
-; CHECKM1: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VF]]
-; CHECKM1: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VG]]
+; CHECK: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesimc [[VA]], [[VA]]
+; CHECK: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesimc [[VB]], [[VB]]
+; CHECK: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesimc [[VC]], [[VC]]
+; CHECK: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesimc [[VD]], [[VD]]
+; CHECK: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesimc [[VE]], [[VE]]
+; CHECK: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesimc [[VF]], [[VF]]
+; CHECK: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesimc [[VG]], [[VG]]
+; CHECK: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECK-NEXT: aesimc [[VH]], [[VH]]
+; CHECK-NOT: aesimc
 }
 
 define void @aes_load_store(<16 x i8> *%p1, <16 x i8> *%p2 , <16 x i8> *%p3) {
 entry:
   %x1 = alloca <16 x i8>, align 16
   %x2 = alloca <16 x i8>, align 16
   %x3 = alloca <16 x i8>, align 16
   %x4 = alloca <16 x i8>, align 16
   %x5 = alloca <16 x i8>, align 16
   %in1 = load <16 x i8>, <16 x i8>* %p1, align 16
   store <16 x i8> %in1, <16 x i8>* %x1, align 16
   %aese1 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %in1, <16 x i8> %in1) #2
-  store <16 x i8> %aese1, <16 x i8>* %x2, align 16
   %in2 = load <16 x i8>, <16 x i8>* %p2, align 16
   %aesmc1= call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %aese1) #2
-  store <16 x i8> %aesmc1, <16 x i8>* %x3, align 16
   %aese2 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %in1, <16 x i8> %in2) #2
-  store <16 x i8> %aese2, <16 x i8>* %x4, align 16
+  store <16 x i8> %aesmc1, <16 x i8>* %x3, align 16
+  %in3 = load <16 x i8>, <16 x i8>* %p3, align 16
   %aesmc2= call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %aese2) #2
-  store <16 x i8> %aesmc2, <16 x i8>* %x5, align 16
+  %aese3 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %aesmc2, <16 x i8> %in3) #2
+  store <16 x i8> %aese3, <16 x i8>* %x5, align 16
   ret void
 
 ; CHECK-LABEL: aes_load_store:
 ; CHECK: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
-; CHECK-NEXT: aesmc {{v[0-7].16b}}, [[VA]]
+; CHECK-NEXT: aesmc [[VA]], [[VA]]
 ; CHECK: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
-; CHECK-NEXT: aesmc {{v[0-7].16b}}, [[VB]]
+; CHECK-NEXT: aesmc [[VB]], [[VB]]
 ; CHECK-NOT: aesmc
 }

test/MC/AArch64/arm64-crypto.s

 ; RUN: llvm-mc -triple arm64-apple-darwin -mattr=crypto -show-encoding -output-asm-variant=1 < %s | FileCheck %s
+; RUN: llvm-mc -triple arm64-apple-darwin -mattr='+crypto,+fuse-aes' -show-encoding -output-asm-variant=1 < %s | FileCheck %s
 
 foo:
   aese.16b v0, v1
   aesd.16b v0, v1
   aesmc.16b v0, v1
   aesimc.16b v0, v1
 
   sha1c.4s q0, s1, v2