[AArch64] Add new subtarget feature to fuse AES crypto operations

[evandro]

Summary

This feature enables the fusion of such operations on Cortex A57, as recommended in its Software Optimisation Guide, section 4.13, and on Exynos M1.

On A57, it improves the results of a proprietary benchmark by about 20%.

[MatzeB]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

2059–2063

Have you actually tested this properly? I am pretty sure this function is only called for terminator instructions. I am currently working on bigger rewrites of the macrofusion code to allow fusion inside of basic blocks, I don't think it is possible today.

[evandro]

@MatzeB, please see the parent patch D28489.

[MatzeB]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

2059–2063

Just saw that you linked a dependent patch about that.

[evandro]

The MacroFusion pass is currently being added before the RA runs. However, since the AArch64ExpandPseudo pass is run after the RA (in AArch64PassConfig::addPreSched2()), I wonder if it'd make more sense to run the MISched after the RA as well, and not before as it is now. Thoughts?

[MatzeB]

In D28491#643256, @evandro wrote:

The MacroFusion pass is currently being added before the RA runs. However, since the AArch64ExpandPseudo pass is run after the RA (in AArch64PassConfig::addPreSched2()), I wonder if it'd make more sense to run the MISched after the RA as well, and not before as it is now. Thoughts?

There are a number of benefits when running the scheduler before register allocation (for example we can still reduce register pressure). We already have the PostMachineScheduler for scheduling again after regalloc (it's based on the same MISched framework but added considerably later in the pipeline; see also TargetSubtargetInfo::enablePostRAScheduler()).

[evandro]

In D28491#643259, @MatzeB wrote:

In D28491#643256, @evandro wrote:

The MacroFusion pass is currently being added before the RA runs. However, since the AArch64ExpandPseudo pass is run after the RA (in AArch64PassConfig::addPreSched2()), I wonder if it'd make more sense to run the MISched after the RA as well, and not before as it is now. Thoughts?

There are a number of benefits when running the scheduler before register allocation (for example we can still reduce register pressure). We already have the PostMachineScheduler for scheduling again after regalloc (it's based on the same MISched framework but added considerably later in the pipeline; see also TargetSubtargetInfo::enablePostRAScheduler()).

I'm asking this because, looking further at other pairs of instrs that A57 fuses, such as ADRP/ADD, they only appear in the instr stream after pseudo expansion.

[MatzeB]

In D28491#643283, @evandro wrote:

In D28491#643259, @MatzeB wrote:

In D28491#643256, @evandro wrote:

The MacroFusion pass is currently being added before the RA runs. However, since the AArch64ExpandPseudo pass is run after the RA (in AArch64PassConfig::addPreSched2()), I wonder if it'd make more sense to run the MISched after the RA as well, and not before as it is now. Thoughts?

There are a number of benefits when running the scheduler before register allocation (for example we can still reduce register pressure). We already have the PostMachineScheduler for scheduling again after regalloc (it's based on the same MISched framework but added considerably later in the pipeline; see also TargetSubtargetInfo::enablePostRAScheduler()).

I'm asking this because, looking further at other pairs of instrs that A57 fuses, such as ADRP/ADD, they only appear in the instr stream after pseudo expansion.

Well if there is no reason to ever break the instructions apart, then using a Pseudo instruction and expanding that later may be the easier solution, is that the case for the AES instructions?

[evandro]

In D28491#643290, @MatzeB wrote:

In D28491#643283, @evandro wrote:

I'm asking this because, looking further at other pairs of instrs that A57 fuses, such as ADRP/ADD, they only appear in the instr stream after pseudo expansion.

Well if there is no reason to ever break the instructions apart, then using a Pseudo instruction and expanding that later may be the easier solution, is that the case for the AES instructions?

No, since they are pretty opaque. But the pseudo MOVaddr is expanded into the pair ADRP/ADD only after the RA. On A57, it's important to schedule them back to back, e.g., by running the MISched after the RA instead of before.

[evandro]

No, since they are pretty opaque. But the pseudo MOVaddr is expanded into the pair ADRP/ADD only after the RA. On A57, it's important to schedule them back to back, e.g., by running the MISched after the RA instead of before.

Or rather, I wonder why pseudo expansion is happening this late, when they are very simple instrs in AArch64. Methinks that expanding them sooner would expose them to more optimizations, yes?

[MatzeB]

Look good overal

llvm/lib/Target/AArch64/AArch64.td

183

The features seem to be sorted alphabetically (same with the Exynos entry).

llvm/test/CodeGen/AArch64/misched-fusion.ll

13–19 ↗

(On Diff #86106)

Why is this test affected here? I see no AES instructions.

[evandro]

llvm/test/CodeGen/AArch64/misched-fusion.ll

13–19 ↗

(On Diff #86106)

Indeed, this test shouldn't be part of this patch.

[MatzeB]

LGTM

[evandro]

Final patch after approval.

llvm/lib/Target/AArch64/AArch64.td

 def FeatureArithmeticBccFusion : SubtargetFeature<
     "arith-bcc-fusion", "HasArithmeticBccFusion", "true",
     "CPU fuses arithmetic+bcc operations">;
 
 def FeatureArithmeticCbzFusion : SubtargetFeature<
     "arith-cbz-fusion", "HasArithmeticCbzFusion", "true",
     "CPU fuses arithmetic + cbz/cbnz operations">;
 
+def FeatureFuseAES : SubtargetFeature<
+    "fuse-aes", "HasFuseAES", "true",
+    "CPU fuses AES crypto operations">;
+
 def FeatureDisableLatencySchedHeuristic : SubtargetFeature<
     "disable-latency-sched-heuristic", "DisableLatencySchedHeuristic", "true",
     "Disable latency scheduling heuristic">;
 
 def FeatureUseRSqrt : SubtargetFeature<
     "use-reciprocal-square-root", "UseRSqrt", "true",
     "Use the reciprocal square root approximation">;
 
def ProcA53     : SubtargetFeature<"a53", "ARMProcFamily", "CortexA53",
                                    FeatureNEON,
                                    FeaturePerfMon,
                                    FeaturePostRAScheduler,
                                    FeatureUseAA
                                    ]>;
 
 def ProcA57     : SubtargetFeature<"a57", "ARMProcFamily", "CortexA57",
                                    "Cortex-A57 ARM processors", [
                                    FeatureBalanceFPOps,
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureCustomCheapAsMoveHandling,
                                    FeatureFPARMv8,
+                                   FeatureFuseAES,
                                    FeatureNEON,
                                    FeaturePerfMon,
                                    FeaturePostRAScheduler,
                                    FeaturePredictableSelectIsExpensive
                                    ]>;
 
 def ProcA72     : SubtargetFeature<"a72", "ARMProcFamily", "CortexA72",
                                    "Cortex-A72 ARM processors", [

 
 def ProcExynosM1 : SubtargetFeature<"exynosm1", "ARMProcFamily", "ExynosM1",
                                     "Samsung Exynos-M1 processors",
                                     [FeatureAvoidQuadLdStPairs,
                                      FeatureCRC,
                                      FeatureCrypto,
                                      FeatureCustomCheapAsMoveHandling,
                                      FeatureFPARMv8,
+                                     FeatureFuseAES,
                                      FeatureNEON,
                                      FeaturePerfMon,
                                      FeaturePostRAScheduler,
                                      FeatureSlowMisaligned128Store,
                                      FeatureUseRSqrt,
                                      FeatureZCZeroing]>;
 
 def ProcExynosM2 : SubtargetFeature<"exynosm2", "ARMProcFamily", "ExynosM1",

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

if (SecondOpcode == AArch64::CBNZW || SecondOpcode == AArch64::CBNZX ||
       case AArch64::BICWrs:
       case AArch64::BICXrs:
         // Shift value can be 0 making these behave like the "rr" variant...
         return !TII.hasShiftedReg(*First);
       case AArch64::INSTRUCTION_LIST_END:
         return true;
       }
 
+  if (ST.hasFuseAES())
+    // Fuse AES crypto operations.
+    switch(FirstOpcode) {
+    // AES encode.
+    case AArch64::AESErr:
+      return SecondOpcode == AArch64::AESMCrr ||
+             SecondOpcode == AArch64::INSTRUCTION_LIST_END;
+    // AES decode.
+    case AArch64::AESDrr:
+      return SecondOpcode == AArch64::AESIMCrr ||
+             SecondOpcode == AArch64::INSTRUCTION_LIST_END;
+    }
+
   return false;
 }
 
 /// \brief Implement the fusion of instruction pairs in the scheduling
 /// \param DAG, anchored at the instruction in \param ASU. \param Preds
 /// indicates if its dependencies in \param APreds are predecessors instead of
 /// successors.
 static bool scheduleAdjacentImpl(const AArch64InstrInfo &TII,
for (SDep &BDep : APreds) {
     // Adjust the latency between the 2nd instr and its successors/predecessors.
     auto &BSuccs = Preds ? BSU->Succs : BSU->Preds;
     for (SDep &Dep : BSuccs)
       if (Dep.getSUnit() == ASU)
         Dep.setLatency(0);
 
     return true;
   }
 
   return false;
 }
 
 /// \brief Callback from DAG postProcessing to create cluster edges to encourage
 /// fused operations.
 void
 AArch64InstrInfo::scheduleAdjacent(ScheduleDAGInstrs *DAGInstrs) const {
   ScheduleDAGMI *DAG = static_cast<ScheduleDAGMI*>(DAGInstrs);
 
   // For each of the SUnits in the scheduling block, try to fuse the instruction
   // in it with one in its successors.
   for (SUnit &ASU : DAG->SUnits)

llvm/lib/Target/AArch64/AArch64SchedA57.td

 def : InstRW<[A57Write_1cyc_1I],    (instrs EXTRWrri)>;
 def : InstRW<[A57Write_3cyc_1I_1M], (instrs EXTRXrri)>;
 def : InstRW<[A57Write_2cyc_1M],    (instregex "BFM")>;
 
 
 // Cryptography Extensions
 // -----------------------------------------------------------------------------
 
-def : InstRW<[A57Write_3cyc_1W], (instregex "^AES")>;
+def A57ReadAES  : SchedReadAdvance<3, [A57Write_3cyc_1W]>;
+def : InstRW<[A57Write_3cyc_1W], (instregex "^AES[DE]")>;
+def : InstRW<[A57Write_3cyc_1W, A57ReadAES], (instregex "^AESI?MC")>;
 def : InstRW<[A57Write_6cyc_2V], (instregex "^SHA1SU0")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^SHA1(H|SU1)")>;
 def : InstRW<[A57Write_6cyc_2W], (instregex "^SHA1[CMP]")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^SHA256SU0")>;
 def : InstRW<[A57Write_6cyc_2W], (instregex "^SHA256(H|H2|SU1)")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^CRC32")>;
 
 

llvm/lib/Target/AArch64/AArch64SchedM1.td

 
 // ASIMD load instructions.
 
 // ASIMD store instructions.
 
 // Cryptography instructions.
 def M1WriteAES : SchedWriteRes<[M1UnitNCRYPT]> { let Latency = 1; }
 def M1ReadAES  : SchedReadAdvance<1, [M1WriteAES]>;
-def : InstRW<[M1WriteAES, M1ReadAES], (instregex "^AES")>;
+def : InstRW<[M1WriteAES], (instregex "^AES[DE]")>;
+def : InstRW<[M1WriteAES, M1ReadAES], (instregex "^AESI?MC")>;
 
 def : InstRW<[M1WriteNCRYPT1], (instregex "^PMUL")>;
 def : InstRW<[M1WriteNCRYPT1], (instregex "^SHA1(H|SU)")>;
 def : InstRW<[M1WriteNCRYPT5], (instregex "^SHA1[CMP]")>;
 def : InstRW<[M1WriteNCRYPT1], (instregex "^SHA256SU0")>;
 def : InstRW<[M1WriteNCRYPT5], (instregex "^SHA256(H|SU1)")>;
 
 // CRC instructions.
 def : InstRW<[M1WriteC2], (instregex "^CRC32")>;
 
 } // SchedModel = ExynosM1Model

llvm/lib/Target/AArch64/AArch64Subtarget.h

protected:
   bool BalanceFPOps = false;
   bool CustomAsCheapAsMove = false;
   bool UsePostRAScheduler = false;
   bool Misaligned128StoreIsSlow = false;
   bool AvoidQuadLdStPairs = false;
   bool UseAlternateSExtLoadCVTF32Pattern = false;
   bool HasArithmeticBccFusion = false;
   bool HasArithmeticCbzFusion = false;
+  bool HasFuseAES = false;
   bool DisableLatencySchedHeuristic = false;
   bool UseRSqrt = false;
   uint8_t MaxInterleaveFactor = 2;
   uint8_t VectorInsertExtractBaseCost = 3;
   uint16_t CacheLineSize = 0;
   uint16_t PrefetchDistance = 0;
   uint16_t MinPrefetchStride = 1;
   unsigned MaxPrefetchIterationsAhead = UINT_MAX;
public:
   bool hasCustomCheapAsMoveHandling() const { return CustomAsCheapAsMove; }
   bool isMisaligned128StoreSlow() const { return Misaligned128StoreIsSlow; }
   bool avoidQuadLdStPairs() const { return AvoidQuadLdStPairs; }
   bool useAlternateSExtLoadCVTF32Pattern() const {
     return UseAlternateSExtLoadCVTF32Pattern;
   }
   bool hasArithmeticBccFusion() const { return HasArithmeticBccFusion; }
   bool hasArithmeticCbzFusion() const { return HasArithmeticCbzFusion; }
+  bool hasFuseAES() const { return HasFuseAES; }
   bool useRSqrt() const { return UseRSqrt; }
   unsigned getMaxInterleaveFactor() const { return MaxInterleaveFactor; }
   unsigned getVectorInsertExtractBaseCost() const {
     return VectorInsertExtractBaseCost;
   }
   unsigned getCacheLineSize() const { return CacheLineSize; }
   unsigned getPrefetchDistance() const { return PrefetchDistance; }
   unsigned getMinPrefetchStride() const { return MinPrefetchStride; }

llvm/test/CodeGen/AArch64/misched-fusion-aes.ll

+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a57 | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKA57
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m1  | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKM1
+
+declare <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d, <16 x i8> %k)
+declare <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %d)
+declare <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d, <16 x i8> %k)
+declare <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %d)
+
+define void @aesea(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
+  %d0 = load <16 x i8>, <16 x i8>* %a0
+  %a1 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 1
+  %d1 = load <16 x i8>, <16 x i8>* %a1
+  %a2 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 2
+  %d2 = load <16 x i8>, <16 x i8>* %a2
+  %a3 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 3
+  %d3 = load <16 x i8>, <16 x i8>* %a3
+  %k0 = load <16 x i8>, <16 x i8>* %b0
+  %e00 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d0, <16 x i8> %k0)
+  %f00 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e00)
+  %e01 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d1, <16 x i8> %k0)
+  %f01 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e01)
+  %e02 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d2, <16 x i8> %k0)
+  %f02 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e02)
+  %e03 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d3, <16 x i8> %k0)
+  %f03 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e03)
+  %b1 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 1
+  %k1 = load <16 x i8>, <16 x i8>* %b1
+  %e10 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f00, <16 x i8> %k1)
+  %f10 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e00)
+  %e11 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f01, <16 x i8> %k1)
+  %f11 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e01)
+  %e12 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f02, <16 x i8> %k1)
+  %f12 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e02)
+  %e13 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f03, <16 x i8> %k1)
+  %f13 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e03)
+  %b2 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 2
+  %k2 = load <16 x i8>, <16 x i8>* %b2
+  %e20 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f10, <16 x i8> %k2)
+  %f20 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e10)
+  %e21 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f11, <16 x i8> %k2)
+  %f21 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e11)
+  %e22 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f12, <16 x i8> %k2)
+  %f22 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e12)
+  %e23 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f13, <16 x i8> %k2)
+  %f23 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e13)
+  %b3 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 3
+  %k3 = load <16 x i8>, <16 x i8>* %b3
+  %e30 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f20, <16 x i8> %k3)
+  %f30 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e20)
+  %e31 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f21, <16 x i8> %k3)
+  %f31 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e21)
+  %e32 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f22, <16 x i8> %k3)
+  %f32 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e22)
+  %e33 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f23, <16 x i8> %k3)
+  %f33 = call <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %e23)
+  %g0 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f30, <16 x i8> %d)
+  %h0 = xor <16 x i8> %g0, %e
+  %g1 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f31, <16 x i8> %d)
+  %h1 = xor <16 x i8> %g1, %e
+  %g2 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f32, <16 x i8> %d)
+  %h2 = xor <16 x i8> %g2, %e
+  %g3 = call <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %f33, <16 x i8> %d)
+  %h3 = xor <16 x i8> %g3, %e
+  store <16 x i8> %h0, <16 x i8>* %c0
+  %c1 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 1
+  store <16 x i8> %h1, <16 x i8>* %c1
+  %c2 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 2
+  store <16 x i8> %h2, <16 x i8>* %c2
+  %c3 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 3
+  store <16 x i8> %h3, <16 x i8>* %c3
+  ret void
+
+; CHECK-LABEL: aesea:
+; CHECKA57: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VC]]
+; CHECKA57: aesmc {{v[0-7].16b}}, [[VA]]
+; CHECKA57: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VD]]
+; CHECKA57: aesmc {{v[0-7].16b}}, [[VB]]
+; CHECKA57: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VE]]
+; CHECKA57: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VF]]
+; CHECKA57: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VG]]
+; CHECKA57: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesmc {{v[0-7].16b}}, [[VH]]
+; CHECKM1: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1: aesmc {{v[0-7].16b}}, [[VA]]
+; CHECKM1: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VB]]
+; CHECKM1: aese {{v[0-7].16b}}, {{v[0-7].16b}}
+; CHECKM1: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VC]]
+; CHECKM1: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1: aesmc {{v[0-7].16b}}, [[VD]]
+; CHECKM1: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VE]]
+; CHECKM1: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VF]]
+; CHECKM1: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VG]]
+; CHECKM1: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VH]]
+}
+
+define void @aesda(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
+  %d0 = load <16 x i8>, <16 x i8>* %a0
+  %a1 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 1
+  %d1 = load <16 x i8>, <16 x i8>* %a1
+  %a2 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 2
+  %d2 = load <16 x i8>, <16 x i8>* %a2
+  %a3 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 3
+  %d3 = load <16 x i8>, <16 x i8>* %a3
+  %k0 = load <16 x i8>, <16 x i8>* %b0
+  %e00 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d0, <16 x i8> %k0)
+  %f00 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e00)
+  %e01 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d1, <16 x i8> %k0)
+  %f01 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e01)
+  %e02 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d2, <16 x i8> %k0)
+  %f02 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e02)
+  %e03 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d3, <16 x i8> %k0)
+  %f03 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e03)
+  %b1 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 1
+  %k1 = load <16 x i8>, <16 x i8>* %b1
+  %e10 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f00, <16 x i8> %k1)
+  %f10 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e00)
+  %e11 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f01, <16 x i8> %k1)
+  %f11 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e01)
+  %e12 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f02, <16 x i8> %k1)
+  %f12 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e02)
+  %e13 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f03, <16 x i8> %k1)
+  %f13 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e03)
+  %b2 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 2
+  %k2 = load <16 x i8>, <16 x i8>* %b2
+  %e20 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f10, <16 x i8> %k2)
+  %f20 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e10)
+  %e21 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f11, <16 x i8> %k2)
+  %f21 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e11)
+  %e22 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f12, <16 x i8> %k2)
+  %f22 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e12)
+  %e23 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f13, <16 x i8> %k2)
+  %f23 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e13)
+  %b3 = getelementptr inbounds <16 x i8>, <16 x i8>* %b0, i64 3
+  %k3 = load <16 x i8>, <16 x i8>* %b3
+  %e30 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f20, <16 x i8> %k3)
+  %f30 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e20)
+  %e31 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f21, <16 x i8> %k3)
+  %f31 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e21)
+  %e32 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f22, <16 x i8> %k3)
+  %f32 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e22)
+  %e33 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f23, <16 x i8> %k3)
+  %f33 = call <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %e23)
+  %g0 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f30, <16 x i8> %d)
+  %h0 = xor <16 x i8> %g0, %e
+  %g1 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f31, <16 x i8> %d)
+  %h1 = xor <16 x i8> %g1, %e
+  %g2 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f32, <16 x i8> %d)
+  %h2 = xor <16 x i8> %g2, %e
+  %g3 = call <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %f33, <16 x i8> %d)
+  %h3 = xor <16 x i8> %g3, %e
+  store <16 x i8> %h0, <16 x i8>* %c0
+  %c1 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 1
+  store <16 x i8> %h1, <16 x i8>* %c1
+  %c2 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 2
+  store <16 x i8> %h2, <16 x i8>* %c2
+  %c3 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 3
+  store <16 x i8> %h3, <16 x i8>* %c3
+  ret void
+
+; CHECK-LABEL: aesda:
+; CHECKA57: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VC]]
+; CHECKA57: aesimc {{v[0-7].16b}}, [[VA]]
+; CHECKA57: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VD]]
+; CHECKA57: aesimc {{v[0-7].16b}}, [[VB]]
+; CHECKA57: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VE]]
+; CHECKA57: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VF]]
+; CHECKA57: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VG]]
+; CHECKA57: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKA57-NEXT: aesimc {{v[0-7].16b}}, [[VH]]
+; CHECKM1: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1: aesimc {{v[0-7].16b}}, [[VA]]
+; CHECKM1: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VB]]
+; CHECKM1: aesd {{v[0-7].16b}}, {{v[0-7].16b}}
+; CHECKM1: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VC]]
+; CHECKM1: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1: aesimc {{v[0-7].16b}}, [[VD]]
+; CHECKM1: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VE]]
+; CHECKM1: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VF]]
+; CHECKM1: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VG]]
+; CHECKM1: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VH]]
+}