[AArch64] Add new subtarget feature to fuse AES crypto operations

[evandro]

Summary

This feature enables the fusion of such operations on Cortex A57, as recommended in its Software Optimisation Guide, section 4.13, and on Exynos M1.

On A57, it improves the results of a proprietary benchmark by about 20%.

[MatzeB]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1984–1988

Have you actually tested this properly? I am pretty sure this function is only called for terminator instructions. I am currently working on bigger rewrites of the macrofusion code to allow fusion inside of basic blocks, I don't think it is possible today.

[evandro]

@MatzeB, please see the parent patch D28489.

[MatzeB]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1984–1988

Just saw that you linked a dependent patch about that.

[evandro]

The MacroFusion pass is currently being added before the RA runs. However, since the AArch64ExpandPseudo pass is run after the RA (in AArch64PassConfig::addPreSched2()), I wonder if it'd make more sense to run the MISched after the RA as well, and not before as it is now. Thoughts?

[MatzeB]

In D28491#643256, @evandro wrote:

The MacroFusion pass is currently being added before the RA runs. However, since the AArch64ExpandPseudo pass is run after the RA (in AArch64PassConfig::addPreSched2()), I wonder if it'd make more sense to run the MISched after the RA as well, and not before as it is now. Thoughts?

There are a number of benefits when running the scheduler before register allocation (for example we can still reduce register pressure). We already have the PostMachineScheduler for scheduling again after regalloc (it's based on the same MISched framework but added considerably later in the pipeline; see also TargetSubtargetInfo::enablePostRAScheduler()).

[evandro]

In D28491#643259, @MatzeB wrote:

In D28491#643256, @evandro wrote:

The MacroFusion pass is currently being added before the RA runs. However, since the AArch64ExpandPseudo pass is run after the RA (in AArch64PassConfig::addPreSched2()), I wonder if it'd make more sense to run the MISched after the RA as well, and not before as it is now. Thoughts?

There are a number of benefits when running the scheduler before register allocation (for example we can still reduce register pressure). We already have the PostMachineScheduler for scheduling again after regalloc (it's based on the same MISched framework but added considerably later in the pipeline; see also TargetSubtargetInfo::enablePostRAScheduler()).

I'm asking this because, looking further at other pairs of instrs that A57 fuses, such as ADRP/ADD, they only appear in the instr stream after pseudo expansion.

[MatzeB]

In D28491#643283, @evandro wrote:

In D28491#643259, @MatzeB wrote:

In D28491#643256, @evandro wrote:

The MacroFusion pass is currently being added before the RA runs. However, since the AArch64ExpandPseudo pass is run after the RA (in AArch64PassConfig::addPreSched2()), I wonder if it'd make more sense to run the MISched after the RA as well, and not before as it is now. Thoughts?

There are a number of benefits when running the scheduler before register allocation (for example we can still reduce register pressure). We already have the PostMachineScheduler for scheduling again after regalloc (it's based on the same MISched framework but added considerably later in the pipeline; see also TargetSubtargetInfo::enablePostRAScheduler()).

I'm asking this because, looking further at other pairs of instrs that A57 fuses, such as ADRP/ADD, they only appear in the instr stream after pseudo expansion.

Well if there is no reason to ever break the instructions apart, then using a Pseudo instruction and expanding that later may be the easier solution, is that the case for the AES instructions?

[evandro]

In D28491#643290, @MatzeB wrote:

In D28491#643283, @evandro wrote:

I'm asking this because, looking further at other pairs of instrs that A57 fuses, such as ADRP/ADD, they only appear in the instr stream after pseudo expansion.

Well if there is no reason to ever break the instructions apart, then using a Pseudo instruction and expanding that later may be the easier solution, is that the case for the AES instructions?

No, since they are pretty opaque. But the pseudo MOVaddr is expanded into the pair ADRP/ADD only after the RA. On A57, it's important to schedule them back to back, e.g., by running the MISched after the RA instead of before.

[evandro]

No, since they are pretty opaque. But the pseudo MOVaddr is expanded into the pair ADRP/ADD only after the RA. On A57, it's important to schedule them back to back, e.g., by running the MISched after the RA instead of before.

Or rather, I wonder why pseudo expansion is happening this late, when they are very simple instrs in AArch64. Methinks that expanding them sooner would expose them to more optimizations, yes?

[MatzeB]

Look good overal

llvm/lib/Target/AArch64/AArch64.td

183

The features seem to be sorted alphabetically (same with the Exynos entry).

llvm/test/CodeGen/AArch64/misched-fusion.ll

13–19

Why is this test affected here? I see no AES instructions.

[evandro]

llvm/test/CodeGen/AArch64/misched-fusion.ll

13–19

Indeed, this test shouldn't be part of this patch.

[MatzeB]

LGTM

[evandro]

Final patch after approval.

llvm/lib/Target/AArch64/AArch64.td

 def FeatureArithmeticBccFusion : SubtargetFeature<
     "arith-bcc-fusion", "HasArithmeticBccFusion", "true",
     "CPU fuses arithmetic+bcc operations">;
 
 def FeatureArithmeticCbzFusion : SubtargetFeature<
     "arith-cbz-fusion", "HasArithmeticCbzFusion", "true",
     "CPU fuses arithmetic + cbz/cbnz operations">;
 
+def FeatureFuseAES : SubtargetFeature<
+    "fuse-aes", "HasFuseAES", "true",
+    "CPU fuses AES crypto operations">;
+
 def FeatureDisableLatencySchedHeuristic : SubtargetFeature<
     "disable-latency-sched-heuristic", "DisableLatencySchedHeuristic", "true",
     "Disable latency scheduling heuristic">;
 
 def FeatureUseRSqrt : SubtargetFeature<
     "use-reciprocal-square-root", "UseRSqrt", "true",
     "Use the reciprocal square root approximation">;
 
def ProcA53     : SubtargetFeature<"a53", "ARMProcFamily", "CortexA53",
                                    FeatureNEON,
                                    FeaturePerfMon,
                                    FeaturePostRAScheduler,
                                    FeatureUseAA
                                    ]>;
 
 def ProcA57     : SubtargetFeature<"a57", "ARMProcFamily", "CortexA57",
                                    "Cortex-A57 ARM processors", [
+                                   FeatureFuseAES,
                                    FeatureBalanceFPOps,
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureCustomCheapAsMoveHandling,
                                    FeatureFPARMv8,
                                    FeatureNEON,
                                    FeaturePerfMon,
                                    FeaturePostRAScheduler,
def ProcCyclone : SubtargetFeature<"cyclone", "ARMProcFamily", "Cyclone",
                                    FeaturePerfMon,
                                    FeatureSlowMisaligned128Store,
                                    FeatureZCRegMove,
                                    FeatureZCZeroing
                                    ]>;
 
 def ProcExynosM1 : SubtargetFeature<"exynosm1", "ARMProcFamily", "ExynosM1",
                                     "Samsung Exynos-M1 processors",
-                                    [FeatureAvoidQuadLdStPairs,
+                                    [FeatureFuseAES,
+                                     FeatureAvoidQuadLdStPairs,
                                      FeatureCRC,
                                      FeatureCrypto,
                                      FeatureCustomCheapAsMoveHandling,
                                      FeatureFPARMv8,
                                      FeatureNEON,
                                      FeaturePerfMon,
                                      FeaturePostRAScheduler,
                                      FeatureSlowMisaligned128Store,

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

bool AArch64InstrInfo::shouldClusterMemOps(MachineInstr &FirstLdSt,
 
   // The caller should already have ordered First/SecondLdSt by offset.
   assert(Offset1 <= Offset2 && "Caller should have ordered offsets.");
   return Offset1 + 1 == Offset2;
 }
 
 bool AArch64InstrInfo::shouldScheduleAdjacent(
     const MachineInstr &First, const MachineInstr &Second) const {
-  if (Subtarget.hasArithmeticBccFusion()) {
-    // Fuse CMN, CMP, TST followed by Bcc.
-    unsigned SecondOpcode = Second.getOpcode();
-    if (SecondOpcode == AArch64::Bcc) {
-      switch (First.getOpcode()) {
+  unsigned FirstOpcode = First.getOpcode();
+
+  if (Subtarget.hasArithmeticBccFusion())
+    // Fuse CMN, CMP, TST followed by Bcc.
+    if (SecondOpcode == AArch64::Bcc)
+      switch (FirstOpcode) {
       default:
         return false;
       case AArch64::ADDSWri:
       case AArch64::ADDSWrr:
       case AArch64::ADDSXri:
       case AArch64::ADDSXrr:
       case AArch64::ANDSWri:
       case AArch64::ANDSWrr:
if (SecondOpcode == AArch64::Bcc)
       case AArch64::ANDSXrs:
       case AArch64::SUBSWrs:
       case AArch64::SUBSXrs:
       case AArch64::BICSWrs:
       case AArch64::BICSXrs:
         // Shift value can be 0 making these behave like the "rr" variant...
         return !hasShiftedReg(Second);
       }
-    }
-  }
-  if (Subtarget.hasArithmeticCbzFusion()) {
+
+  if (Subtarget.hasArithmeticCbzFusion())
     // Fuse ALU operations followed by CBZ/CBNZ.
-    unsigned SecondOpcode = Second.getOpcode();
     if (SecondOpcode == AArch64::CBNZW || SecondOpcode == AArch64::CBNZX ||
-        SecondOpcode == AArch64::CBZW || SecondOpcode == AArch64::CBZX) {
-      switch (First.getOpcode()) {
+        SecondOpcode == AArch64::CBZW || SecondOpcode == AArch64::CBZX)
+      switch (FirstOpcode) {
       default:
         return false;
       case AArch64::ADDWri:
       case AArch64::ADDWrr:
       case AArch64::ADDXri:
       case AArch64::ADDXrr:
       case AArch64::ANDWri:
       case AArch64::ANDWrr:
if (SecondOpcode == AArch64::CBNZW || SecondOpcode == AArch64::CBNZX ||
       case AArch64::ANDXrs:
       case AArch64::SUBWrs:
       case AArch64::SUBXrs:
       case AArch64::BICWrs:
       case AArch64::BICXrs:
         // Shift value can be 0 making these behave like the "rr" variant...
         return !hasShiftedReg(Second);
       }
-    }
-  }
+
+  if (Subtarget.hasFuseAES())
+    // Fuse AES crypto operations:
+    // - AES encode
+    // - AES decode
+    if ((FirstOpcode == AArch64::AESErr && SecondOpcode == AArch64::AESMCrr) ||
+        (FirstOpcode == AArch64::AESDrr && SecondOpcode == AArch64::AESIMCrr))
+      return true;
+
   return false;
 }
 
 MachineInstr *AArch64InstrInfo::emitFrameIndexDebugValue(
     MachineFunction &MF, int FrameIx, uint64_t Offset, const MDNode *Var,
     const MDNode *Expr, const DebugLoc &DL) const {
   MachineInstrBuilder MIB = BuildMI(MF, DL, get(AArch64::DBG_VALUE))
                                 .addFrameIndex(FrameIx)

llvm/lib/Target/AArch64/AArch64SchedA57.td

 def : InstRW<[A57Write_1cyc_1I],    (instrs EXTRWrri)>;
 def : InstRW<[A57Write_3cyc_1I_1M], (instrs EXTRXrri)>;
 def : InstRW<[A57Write_2cyc_1M],    (instregex "BFM")>;
 
 
 // Cryptography Extensions
 // -----------------------------------------------------------------------------
 
-def : InstRW<[A57Write_3cyc_1W], (instregex "^AES")>;
+def A57ReadAES  : SchedReadAdvance<3, [A57Write_3cyc_1W]>;
+def : InstRW<[A57Write_3cyc_1W], (instregex "^AES[DE]")>;
+def : InstRW<[A57Write_3cyc_1W, A57ReadAES], (instregex "^AESI?MC")>;
 def : InstRW<[A57Write_6cyc_2V], (instregex "^SHA1SU0")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^SHA1(H|SU1)")>;
 def : InstRW<[A57Write_6cyc_2W], (instregex "^SHA1[CMP]")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^SHA256SU0")>;
 def : InstRW<[A57Write_6cyc_2W], (instregex "^SHA256(H|H2|SU1)")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^CRC32")>;
 
 

llvm/lib/Target/AArch64/AArch64SchedM1.td

 
 // ASIMD load instructions.
 
 // ASIMD store instructions.
 
 // Cryptography instructions.
 def M1WriteAES : SchedWriteRes<[M1UnitNCRYPT]> { let Latency = 1; }
 def M1ReadAES  : SchedReadAdvance<1, [M1WriteAES]>;
-def : InstRW<[M1WriteAES, M1ReadAES], (instregex "^AES")>;
+def : InstRW<[M1WriteAES], (instregex "^AES[DE]")>;
+def : InstRW<[M1WriteAES, M1ReadAES], (instregex "^AESI?MC")>;
 
 def : InstRW<[M1WriteNCRYPT1], (instregex "^PMUL")>;
 def : InstRW<[M1WriteNCRYPT1], (instregex "^SHA1(H|SU)")>;
 def : InstRW<[M1WriteNCRYPT5], (instregex "^SHA1[CMP]")>;
 def : InstRW<[M1WriteNCRYPT1], (instregex "^SHA256SU0")>;
 def : InstRW<[M1WriteNCRYPT5], (instregex "^SHA256(H|SU1)")>;
 
 // CRC instructions.
 def : InstRW<[M1WriteC2], (instregex "^CRC32")>;
 
 } // SchedModel = ExynosM1Model

llvm/lib/Target/AArch64/AArch64Subtarget.h

protected:
   bool BalanceFPOps = false;
   bool CustomAsCheapAsMove = false;
   bool UsePostRAScheduler = false;
   bool Misaligned128StoreIsSlow = false;
   bool AvoidQuadLdStPairs = false;
   bool UseAlternateSExtLoadCVTF32Pattern = false;
   bool HasArithmeticBccFusion = false;
   bool HasArithmeticCbzFusion = false;
+  bool HasFuseAES = false;
   bool DisableLatencySchedHeuristic = false;
   bool UseRSqrt = false;
   uint8_t MaxInterleaveFactor = 2;
   uint8_t VectorInsertExtractBaseCost = 3;
   uint16_t CacheLineSize = 0;
   uint16_t PrefetchDistance = 0;
   uint16_t MinPrefetchStride = 1;
   unsigned MaxPrefetchIterationsAhead = UINT_MAX;
public:
   bool hasCustomCheapAsMoveHandling() const { return CustomAsCheapAsMove; }
   bool isMisaligned128StoreSlow() const { return Misaligned128StoreIsSlow; }
   bool avoidQuadLdStPairs() const { return AvoidQuadLdStPairs; }
   bool useAlternateSExtLoadCVTF32Pattern() const {
     return UseAlternateSExtLoadCVTF32Pattern;
   }
   bool hasArithmeticBccFusion() const { return HasArithmeticBccFusion; }
   bool hasArithmeticCbzFusion() const { return HasArithmeticCbzFusion; }
+  bool hasFuseAES() const { return HasFuseAES; }
   bool useRSqrt() const { return UseRSqrt; }
   unsigned getMaxInterleaveFactor() const { return MaxInterleaveFactor; }
   unsigned getVectorInsertExtractBaseCost() const {
     return VectorInsertExtractBaseCost;
   }
   unsigned getCacheLineSize() const { return CacheLineSize; }
   unsigned getPrefetchDistance() const { return PrefetchDistance; }
   unsigned getMinPrefetchStride() const { return MinPrefetchStride; }

llvm/test/CodeGen/AArch64/misched-fusion-aes.ll

+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a57 | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKA57
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m1  | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKM1
+
+declare <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d, <16 x i8> %k)
+declare <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %d)
+declare <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d, <16 x i8> %k)
+declare <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %d)
+
+define void @aesea(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
+  %d0 = load <16 x i8>, <16 x i8>* %a0
+  %a1 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 1
+  %d1 = load <16 x i8>, <16 x i8>* %a1
+  %a2 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 2
+  %d2 = load <16 x i8>, <16 x i8>* %a2
+  %a3 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 3
+  %d3 = load <16 x i8>, <16 x i8>* %a3
+  %k0 = load <16 x i8>, <16 x i8>* %b0
+  %e00 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d0, <16 x i8> %k0)
+  %f00 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e00)
+  %e01 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d1, <16 x i8> %k0)
+  %f01 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e01)
+  %e02 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d2, <16 x i8> %k0)
+  %f02 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e02)
+  %e03 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d3, <16 x i8> %k0)
+  %f03 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e03)
+  %b1 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 1
+  %k1 = load <16 x i8>, <16 x i8>* %b1
+  %e10 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f00, <16 x i8> %k1)
+  %f10 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e00)
+  %e11 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f01, <16 x i8> %k1)
+  %f11 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e01)
+  %e12 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f02, <16 x i8> %k1)
+  %f12 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e02)
+  %e13 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f03, <16 x i8> %k1)
+  %f13 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e03)
+  %b2 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 2
+  %k2 = load <16 x i8>, <16 x i8>* %b2
+  %e20 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f10, <16 x i8> %k2)
+  %f20 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e10)
+  %e21 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f11, <16 x i8> %k2)
+  %f21 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e11)
+  %e22 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f12, <16 x i8> %k2)
+  %f22 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e12)
+  %e23 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f13, <16 x i8> %k2)
+  %f23 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e13)
+  %b3 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 3
+  %k3 = load <16 x i8>, <16 x i8>* %b3
+  %e30 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f20, <16 x i8> %k3)
+  %f30 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e20)
+  %e31 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f21, <16 x i8> %k3)
+  %f31 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e21)
+  %e32 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f22, <16 x i8> %k3)
+  %f32 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e22)
+  %e33 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f23, <16 x i8> %k3)
+  %f33 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e23)
+  %g0 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f30, <16 x i8> %d)
+  %h0 = xor <16 x i8> %g0, %e
+  %g1 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f31, <16 x i8> %d)
+  %h1 = xor <16 x i8> %g1, %e
+  %g2 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f32, <16 x i8> %d)
+  %h2 = xor <16 x i8> %g2, %e
+  %g3 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f33, <16 x i8> %d)
+  %h3 = xor <16 x i8> %g3, %e
+  store <16 x i8> %h0, <16 x i8>* %c0
+  %c1 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 1
+  store <16 x i8> %h1, <16 x i8>* %c1
+  %c2 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 2
+  store <16 x i8> %h2, <16 x i8>* %c2
+  %c3 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 3
+  store <16 x i8> %h3, <16 x i8>* %c3
+  ret void
+
+; CHECK-LABEL: aesea:
+; CHECKA57: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VC]]
+; CHECKA57: aesmc {{v[0-7].16b}}, [[VA]]
+; CHECKA57: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VD]]
+; CHECKA57: aesmc {{v[0-7].16b}}, [[VB]]
+; CHECKA57: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VE]]
+; CHECKA57: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VF]]
+; CHECKA57: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VG]]
+; CHECKA57: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VH]]
+; CHECKM1: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1: aesmc {{v[0-7].16b}}, [[VA]]
+; CHECKM1: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VB]]
+; CHECKM1: aese {{v[0-7].16b}}, {{v[0-7].16b}}
+; CHECKM1: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VC]]
+; CHECKM1: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1: aesmc {{v[0-7].16b}}, [[VD]]
+; CHECKM1: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VE]]
+; CHECKM1: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VF]]
+; CHECKM1: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VG]]
+; CHECKM1: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VH]]
+}
+
+define void @aesda(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
+  %d0 = load <16 x i8>, <16 x i8>* %a0
+  %a1 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 1
+  %d1 = load <16 x i8>, <16 x i8>* %a1
+  %a2 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 2
+  %d2 = load <16 x i8>, <16 x i8>* %a2
+  %a3 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 3
+  %d3 = load <16 x i8>, <16 x i8>* %a3
+  %k0 = load <16 x i8>, <16 x i8>* %b0
+  %e00 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d0, <16 x i8> %k0)
+  %f00 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e00)
+  %e01 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d1, <16 x i8> %k0)
+  %f01 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e01)
+  %e02 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d2, <16 x i8> %k0)
+  %f02 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e02)
+  %e03 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d3, <16 x i8> %k0)
+  %f03 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e03)
+  %b1 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 1
+  %k1 = load <16 x i8>, <16 x i8>* %b1
+  %e10 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f00, <16 x i8> %k1)
+  %f10 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e00)
+  %e11 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f01, <16 x i8> %k1)
+  %f11 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e01)
+  %e12 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f02, <16 x i8> %k1)
+  %f12 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e02)
+  %e13 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f03, <16 x i8> %k1)
+  %f13 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e03)
+  %b2 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 2
+  %k2 = load <16 x i8>, <16 x i8>* %b2
+  %e20 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f10, <16 x i8> %k2)
+  %f20 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e10)
+  %e21 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f11, <16 x i8> %k2)
+  %f21 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e11)
+  %e22 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f12, <16 x i8> %k2)
+  %f22 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e12)
+  %e23 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f13, <16 x i8> %k2)
+  %f23 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e13)
+  %b3 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 3
+  %k3 = load <16 x i8>, <16 x i8>* %b3
+  %e30 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f20, <16 x i8> %k3)
+  %f30 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e20)
+  %e31 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f21, <16 x i8> %k3)
+  %f31 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e21)
+  %e32 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f22, <16 x i8> %k3)
+  %f32 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e22)
+  %e33 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f23, <16 x i8> %k3)
+  %f33 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e23)
+  %g0 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f30, <16 x i8> %d)
+  %h0 = xor <16 x i8> %g0, %e
+  %g1 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f31, <16 x i8> %d)
+  %h1 = xor <16 x i8> %g1, %e
+  %g2 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f32, <16 x i8> %d)
+  %h2 = xor <16 x i8> %g2, %e
+  %g3 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f33, <16 x i8> %d)
+  %h3 = xor <16 x i8> %g3, %e
+  store <16 x i8> %h0, <16 x i8>* %c0
+  %c1 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 1
+  store <16 x i8> %h1, <16 x i8>* %c1
+  %c2 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 2
+  store <16 x i8> %h2, <16 x i8>* %c2
+  %c3 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 3
+  store <16 x i8> %h3, <16 x i8>* %c3
+  ret void
+
+; CHECK-LABEL: aesda:
+; CHECKA57: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VC]]
+; CHECKA57: aesimc {{v[0-7].16b}}, [[VA]]
+; CHECKA57: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VD]]
+; CHECKA57: aesimc {{v[0-7].16b}}, [[VB]]
+; CHECKA57: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VE]]
+; CHECKA57: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VF]]
+; CHECKA57: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VG]]
+; CHECKA57: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VH]]
+; CHECKM1: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1: aesimc {{v[0-7].16b}}, [[VA]]
+; CHECKM1: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VB]]
+; CHECKM1: aesd {{v[0-7].16b}}, {{v[0-7].16b}}
+; CHECKM1: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VC]]
+; CHECKM1: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1: aesimc {{v[0-7].16b}}, [[VD]]
+; CHECKM1: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VE]]
+; CHECKM1: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VF]]
+; CHECKM1: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VG]]
+; CHECKM1: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VH]]
+}

llvm/test/CodeGen/AArch64/misched-fusion.ll

 ; RUN: llc -o - %s -mattr=+arith-cbz-fusion | FileCheck %s
 ; RUN: llc -o - %s -mcpu=cyclone | FileCheck %s
 
-target triple = "arm64-apple-ios"
+target triple = "aarch64-unknown"
 
 declare void @foobar(i32 %v0, i32 %v1)
 
 ; Make sure sub is scheduled in front of cbnz
 ; CHECK-LABEL: test_sub_cbz:
-; CHECK: add w[[ADDRES:[0-9]+]], w1, #7
 ; CHECK: sub w[[SUBRES:[0-9]+]], w0, #13
-; CHECK-NEXT: cbnz w[[SUBRES]], [[SKIPBLOCK:LBB[0-9_]+]]
-; CHECK: mov [[REGTY:[x,w]]]0, [[REGTY]][[ADDRES]]
-; CHECK: mov [[REGTY]]1, [[REGTY]][[SUBRES]]
-; CHECK: bl _foobar
-; CHECK: [[SKIPBLOCK]]:
-; CHECK: mov [[REGTY]]0, [[REGTY]][[SUBRES]]
-; CHECK: mov [[REGTY]]1, [[REGTY]][[ADDRES]]
-; CHECK: bl _foobar
+; CHECK-NEXT: cbnz w[[SUBRES]], {{.?LBB[0-9_]+}}
+
 define void @test_sub_cbz(i32 %a0, i32 %a1) {
 entry:
   ; except for the fusion opportunity the sub/add should be equal so the
   ; scheduler would leave them in source order if it weren't for the scheduling
   %v0 = sub i32 %a0, 13
   %cond = icmp eq i32 %v0, 0
   %v1 = add i32 %a1, 7
   br i1 %cond, label %if, label %exit