[AArch64] Move AES instruction fusion support

[evandro]

Summary

Move the support to fuse such instructions to the generic mechanism in the machine model used by the scheduler, as proposed in the parent revisions.

[fhahn]

llvm/lib/Target/AArch64/AArch64SchedM1.td

637

This is an unrelated whitespace change?

llvm/test/CodeGen/AArch64/misched-fusion-aes.ll

100

Do you know why the generated code for exynos-m1 changed? Maybe it uses a different number of instructions to look ahead? I think we should be careful not to introduce any regressions before committing to this new approach.

I could run a few benchmarks on some Cortex-A cores to check for performance regressions, if you are happy with the state of the set of patches.

[evandro]

Ping 🔔

[kristof.beyls]

In D35260#893312, @evandro wrote:

Ping 🔔

Hi Evandro,

It seems that your patch hasn't been updated to ToT?
Specifically, I was looking for how this impacts -mcpu=generic, but it seems your patch doesn't have the changes from D33836.
I wonder if this would regress the fusion for -mcpu=generic?

Thanks,

Kristof

llvm/lib/Target/AArch64/AArch64.td

 def FeatureArithmeticBccFusion : SubtargetFeature<
     "arith-bcc-fusion", "HasArithmeticBccFusion", "true",
     "CPU fuses arithmetic+bcc operations">;
 
 def FeatureArithmeticCbzFusion : SubtargetFeature<
     "arith-cbz-fusion", "HasArithmeticCbzFusion", "true",
     "CPU fuses arithmetic + cbz/cbnz operations">;
 
-def FeatureFuseAES : SubtargetFeature<
-    "fuse-aes", "HasFuseAES", "true",
-    "CPU fuses AES crypto operations">;
-
 def FeatureFuseLiterals : SubtargetFeature<
     "fuse-literals", "HasFuseLiterals", "true",
     "CPU fuses literal generation operations">;
 
 def FeatureDisableLatencySchedHeuristic : SubtargetFeature<
     "disable-latency-sched-heuristic", "DisableLatencySchedHeuristic", "true",
     "Disable latency scheduling heuristic">;
 

 
 def ProcA53     : SubtargetFeature<"a53", "ARMProcFamily", "CortexA53",
                                    "Cortex-A53 ARM processors", [
                                    FeatureBalanceFPOps,
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureCustomCheapAsMoveHandling,
                                    FeatureFPARMv8,
-                                   FeatureFuseAES,
                                    FeatureNEON,
                                    FeaturePerfMon,
                                    FeaturePostRAScheduler,
                                    FeatureUseAA
                                    ]>;
 
 def ProcA57     : SubtargetFeature<"a57", "ARMProcFamily", "CortexA57",
                                    "Cortex-A57 ARM processors", [
                                    FeatureBalanceFPOps,
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureCustomCheapAsMoveHandling,
                                    FeatureFPARMv8,
-                                   FeatureFuseAES,
                                    FeatureFuseLiterals,
                                    FeatureNEON,
                                    FeaturePerfMon,
                                    FeaturePostRAScheduler,
                                    FeaturePredictableSelectIsExpensive
                                    ]>;
 
 def ProcA72     : SubtargetFeature<"a72", "ARMProcFamily", "CortexA72",
                                    "Cortex-A72 ARM processors", [
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureFPARMv8,
-                                   FeatureFuseAES,
                                    FeatureNEON,
-                                   FeaturePerfMon
+                                   FeaturePerfMon,
+                                   FeaturePostRAScheduler
                                    ]>;
 
 def ProcA73     : SubtargetFeature<"a73", "ARMProcFamily", "CortexA73",
                                    "Cortex-A73 ARM processors", [
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureFPARMv8,
-                                   FeatureFuseAES,
                                    FeatureNEON,
-                                   FeaturePerfMon
+                                   FeaturePerfMon,
+                                   FeaturePostRAScheduler
                                    ]>;
 
 def ProcCyclone : SubtargetFeature<"cyclone", "ARMProcFamily", "Cyclone",
                                    "Cyclone", [
                                    FeatureAlternateSExtLoadCVTF32Pattern,
                                    FeatureCrypto,
                                    FeatureDisableLatencySchedHeuristic,
                                    FeatureFPARMv8,
                                    FeatureArithmeticBccFusion,
                                    FeatureArithmeticCbzFusion,
                                    FeatureNEON,
                                    FeaturePerfMon,
                                    FeatureSlowMisaligned128Store,
                                    FeatureZCRegMove,
                                    FeatureZCZeroing
                                    ]>;
 
 def ProcExynosM1 : SubtargetFeature<"exynosm1", "ARMProcFamily", "ExynosM1",
                                     "Samsung Exynos-M1 processors",
                                     [FeatureSlowPaired128,
                                      FeatureCRC,
                                      FeatureCrypto,
                                      FeatureCustomCheapAsMoveHandling,
                                      FeatureFPARMv8,
-                                     FeatureFuseAES,
                                      FeatureNEON,
                                      FeaturePerfMon,
                                      FeaturePostRAScheduler,
                                      FeatureSlowMisaligned128Store,
                                      FeatureUseRSqrt,
                                      FeatureZCZeroing]>;
 
 def ProcExynosM2 : SubtargetFeature<"exynosm2", "ARMProcFamily", "ExynosM1",

llvm/lib/Target/AArch64/AArch64MacroFusion.cpp

if (SecondOpcode == AArch64::CBNZW || SecondOpcode == AArch64::CBNZX ||
       case AArch64::BICWrs:
       case AArch64::BICXrs:
         // Shift value can be 0 making these behave like the "rr" variant...
         return !II.hasShiftedReg(*FirstMI);
       case AArch64::INSTRUCTION_LIST_END:
         return true;
       }
 
-  if (ST.hasFuseAES())
-    // Fuse AES crypto operations.
-    switch(FirstOpcode) {
-    // AES encode.
-    case AArch64::AESErr:
-      return SecondOpcode == AArch64::AESMCrr ||
-             SecondOpcode == AArch64::INSTRUCTION_LIST_END;
-    // AES decode.
-    case AArch64::AESDrr:
-      return SecondOpcode == AArch64::AESIMCrr ||
-             SecondOpcode == AArch64::INSTRUCTION_LIST_END;
-    }
-
   if (ST.hasFuseLiterals())
     // Fuse literal generation operations.
     switch (FirstOpcode) {
     // PC relative address.
     case AArch64::ADRP:
       return SecondOpcode == AArch64::ADDXri ||
              SecondOpcode == AArch64::INSTRUCTION_LIST_END;
     // 32 bit immediate.

llvm/lib/Target/AArch64/AArch64SchedA53.td

 def : InstRW<[A53WriteFMAC], (instregex "^FML(A|S).*")>;
 def : InstRW<[A53WriteFDivSP], (instrs FDIVSrr)>;
 def : InstRW<[A53WriteFDivDP], (instrs FDIVDrr)>;
 def : InstRW<[A53WriteFDivSP], (instregex "^FDIVv.*32$")>;
 def : InstRW<[A53WriteFDivDP], (instregex "^FDIVv.*64$")>;
 def : InstRW<[A53WriteFSqrtSP], (instregex "^.*SQRT.*32$")>;
 def : InstRW<[A53WriteFSqrtDP], (instregex "^.*SQRT.*64$")>;
 
+//---
+// Cryptography Extensions
+//---
+// FIXME: resources and costs below were wildly guessed.
+def A53ReadAES  : SchedReadCluster<6, [WriteV]>;
+def : InstRW<[WriteV], (instregex "^AES[DE]")>;
+def : InstRW<[WriteV, A53ReadAES], (instregex "^AESI?MC")>;
+
 }

llvm/lib/Target/AArch64/AArch64SchedA57.td

 def : InstRW<[A57Write_1cyc_1I],    (instrs EXTRWrri)>;
 def : InstRW<[A57Write_3cyc_1I_1M], (instrs EXTRXrri)>;
 def : InstRW<[A57Write_2cyc_1M],    (instregex "BFM")>;
 
 
 // Cryptography Extensions
 // -----------------------------------------------------------------------------
 
-def A57ReadAES  : SchedReadAdvance<3, [A57Write_3cyc_1W]>;
+def A57ReadAES : SchedReadCluster<3, [A57Write_3cyc_1W]>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^AES[DE]")>;
 def : InstRW<[A57Write_3cyc_1W, A57ReadAES], (instregex "^AESI?MC")>;
 def : InstRW<[A57Write_6cyc_2V], (instregex "^SHA1SU0")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^SHA1(H|SU1)")>;
 def : InstRW<[A57Write_6cyc_2W], (instregex "^SHA1[CMP]")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^SHA256SU0")>;
 def : InstRW<[A57Write_6cyc_2W], (instregex "^SHA256(H|H2|SU1)")>;
 def : InstRW<[A57Write_3cyc_1W], (instregex "^CRC32")>;

llvm/lib/Target/AArch64/AArch64SchedM1.td

 def : InstRW<[M1WriteVSTB],   (instregex "ST1Threev(16b|8h|4s|2d)$")>;
 def : InstRW<[M1WriteVSTB,
               WriteAdr],      (instregex "ST1Threev(16b|8h|4s|2d)_POST$")>;
 def : InstRW<[M1WriteVSTC],   (instregex "ST1Fourv(8b|4h|2s|1d)$")>;
 def : InstRW<[M1WriteVSTC,
               WriteAdr],      (instregex "ST1Fourv(8b|4h|2s|1d)_POST$")>;
 def : InstRW<[M1WriteVSTC],   (instregex "ST1Fourv(16b|8h|4s|2d)$")>;
 def : InstRW<[M1WriteVSTC,
-              WriteAdr], (instregex "ST1Fourv(16b|8h|4s|2d)_POST$")>;
+              WriteAdr],      (instregex "ST1Fourv(16b|8h|4s|2d)_POST$")>;
 
 def : InstRW<[M1WriteVSTD],   (instregex "ST2i(8|16|32)$")>;
 def : InstRW<[M1WriteVSTD,
               WriteAdr],      (instregex "ST2i(8|16|32)_POST$")>;
 def : InstRW<[M1WriteVSTD],   (instregex "ST2i(64)$")>;
 def : InstRW<[M1WriteVSTD,
               WriteAdr],      (instregex "ST2i(64)_POST$")>;
 

 def : InstRW<[M1WriteVSTI,
               WriteAdr],      (instregex "ST4Fourv(16b|8h|4s)_POST$")>;
 def : InstRW<[M1WriteVSTI],   (instregex "ST4Fourv(2d)$")>;
 def : InstRW<[M1WriteVSTI,
               WriteAdr],      (instregex "ST4Fourv(2d)_POST$")>;
 
 // Cryptography instructions.
 def M1WriteAES : SchedWriteRes<[M1UnitNCRYPT]> { let Latency = 1; }
-def M1ReadAES  : SchedReadAdvance<1, [M1WriteAES]>;
+def M1ReadAES  : SchedReadCluster<1, [M1WriteAES]>;
 def : InstRW<[M1WriteAES], (instregex "^AES[DE]")>;
 def : InstRW<[M1WriteAES, M1ReadAES], (instregex "^AESI?MC")>;
 
 def : InstRW<[M1WriteNCRYPT1], (instregex "^PMUL")>;
 def : InstRW<[M1WriteNCRYPT1], (instregex "^SHA1(H|SU)")>;
 def : InstRW<[M1WriteNCRYPT5], (instregex "^SHA1[CMP]")>;
 def : InstRW<[M1WriteNCRYPT1], (instregex "^SHA256SU0")>;
 def : InstRW<[M1WriteNCRYPT5], (instregex "^SHA256(H|SU1)")>;
 
 // CRC instructions.
 def : InstRW<[M1WriteC2], (instregex "^CRC32")>;
 
 } // SchedModel = ExynosM1Model

llvm/lib/Target/AArch64/AArch64Subtarget.h

protected:
   bool BalanceFPOps = false;
   bool CustomAsCheapAsMove = false;
   bool UsePostRAScheduler = false;
   bool Misaligned128StoreIsSlow = false;
   bool Paired128IsSlow = false;
   bool UseAlternateSExtLoadCVTF32Pattern = false;
   bool HasArithmeticBccFusion = false;
   bool HasArithmeticCbzFusion = false;
-  bool HasFuseAES = false;
   bool HasFuseLiterals = false;
   bool DisableLatencySchedHeuristic = false;
   bool UseRSqrt = false;
   uint8_t MaxInterleaveFactor = 2;
   uint8_t VectorInsertExtractBaseCost = 3;
   uint16_t CacheLineSize = 0;
   uint16_t PrefetchDistance = 0;
   uint16_t MinPrefetchStride = 1;
public:
   bool hasCustomCheapAsMoveHandling() const { return CustomAsCheapAsMove; }
   bool isMisaligned128StoreSlow() const { return Misaligned128StoreIsSlow; }
   bool isPaired128Slow() const { return Paired128IsSlow; }
   bool useAlternateSExtLoadCVTF32Pattern() const {
     return UseAlternateSExtLoadCVTF32Pattern;
   }
   bool hasArithmeticBccFusion() const { return HasArithmeticBccFusion; }
   bool hasArithmeticCbzFusion() const { return HasArithmeticCbzFusion; }
-  bool hasFuseAES() const { return HasFuseAES; }
   bool hasFuseLiterals() const { return HasFuseLiterals; }
   bool useRSqrt() const { return UseRSqrt; }
   unsigned getMaxInterleaveFactor() const { return MaxInterleaveFactor; }
   unsigned getVectorInsertExtractBaseCost() const {
     return VectorInsertExtractBaseCost;
   }
   unsigned getCacheLineSize() const { return CacheLineSize; }
   unsigned getPrefetchDistance() const { return PrefetchDistance; }

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

createMachineScheduler(MachineSchedContext *C) const override {
     DAG->addMutation(createStoreClusterDAGMutation(DAG->TII, DAG->TRI));
     DAG->addMutation(createAArch64MacroFusionDAGMutation());
     return DAG;
   }
 
   ScheduleDAGInstrs *
   createPostMachineScheduler(MachineSchedContext *C) const override {
     const AArch64Subtarget &ST = C->MF->getSubtarget<AArch64Subtarget>();
-    if (ST.hasFuseAES() || ST.hasFuseLiterals()) {
+    if (ST.hasFuseLiterals()) {
       // Run the Macro Fusion after RA again since literals are expanded from
       // pseudos then (v. addPreSched2()).
       ScheduleDAGMI *DAG = createGenericSchedPostRA(C);
       DAG->addMutation(createAArch64MacroFusionDAGMutation());
       return DAG;
     }
 
     return nullptr;

llvm/test/CodeGen/AArch64/misched-fusion-aes.ll

define void @aesea(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   store <16 x i8> %h1, <16 x i8>* %c1
   %c2 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 2
   store <16 x i8> %h2, <16 x i8>* %c2
   %c3 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 3
   store <16 x i8> %h3, <16 x i8>* %c3
   ret void
 
 ; CHECK-LABEL: aesea:
+
 ; CHECKCORTEX: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesmc {{v[0-7].16b}}, [[VA]]
 ; CHECKCORTEX: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesmc {{v[0-7].16b}}, [[VB]]
 ; CHECKCORTEX: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesmc {{v[0-7].16b}}, [[VC]]
 ; CHECKCORTEX: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesmc {{v[0-7].16b}}, [[VD]]
 ; CHECKCORTEX: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesmc {{v[0-7].16b}}, [[VE]]
 ; CHECKCORTEX: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesmc {{v[0-7].16b}}, [[VF]]
 ; CHECKCORTEX: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesmc {{v[0-7].16b}}, [[VG]]
 ; CHECKCORTEX: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesmc {{v[0-7].16b}}, [[VH]]
 
 ; CHECKM1: aese [[VA:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VA]]
-; CHECKM1: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1: aese [[VB:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VB]]
-; CHECKM1: aese {{v[0-7].16b}}, {{v[0-7].16b}}
 ; CHECKM1: aese [[VC:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VC]]
+; CHECKM1: aese {{v[0-7].16b}}, {{v[0-7].16b}}
 ; CHECKM1: aese [[VD:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VD]]
-; CHECKM1: aesmc {{v[0-7].16b}}, [[VH]]
 ; CHECKM1: aese [[VE:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VE]]
+; CHECKM1: aesmc {{v[0-7].16b}}, [[VB]]
 ; CHECKM1: aese [[VF:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VF]]
 ; CHECKM1: aese [[VG:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VG]]
+; CHECKM1: aese [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesmc {{v[0-7].16b}}, [[VH]]
 }
 
 define void @aesda(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   %d0 = load <16 x i8>, <16 x i8>* %a0
   %a1 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 1
   %d1 = load <16 x i8>, <16 x i8>* %a1
   %a2 = getelementptr inbounds <16 x i8>, <16 x i8>* %a0, i64 2
   %d2 = load <16 x i8>, <16 x i8>* %a2
define void @aesda(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   store <16 x i8> %h1, <16 x i8>* %c1
   %c2 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 2
   store <16 x i8> %h2, <16 x i8>* %c2
   %c3 = getelementptr inbounds <16 x i8>, <16 x i8>* %c0, i64 3
   store <16 x i8> %h3, <16 x i8>* %c3
   ret void
 
 ; CHECK-LABEL: aesda:
+
 ; CHECKCORTEX: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesimc {{v[0-7].16b}}, [[VA]]
 ; CHECKCORTEX: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesimc {{v[0-7].16b}}, [[VB]]
 ; CHECKCORTEX: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesimc {{v[0-7].16b}}, [[VC]]
 ; CHECKCORTEX: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesimc {{v[0-7].16b}}, [[VD]]
 ; CHECKCORTEX: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesimc {{v[0-7].16b}}, [[VE]]
 ; CHECKCORTEX: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesimc {{v[0-7].16b}}, [[VF]]
 ; CHECKCORTEX: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesimc {{v[0-7].16b}}, [[VG]]
 ; CHECKCORTEX: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKCORTEX-NEXT: aesimc {{v[0-7].16b}}, [[VH]]
 
 ; CHECKM1: aesd [[VA:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VA]]
-; CHECKM1: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1: aesd [[VB:v[0-7].16b]], {{v[0-7].16b}}
-; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VB]]
-; CHECKM1: aesd {{v[0-7].16b}}, {{v[0-7].16b}}
 ; CHECKM1: aesd [[VC:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VC]]
+; CHECKM1: aesd {{v[0-7].16b}}, {{v[0-7].16b}}
 ; CHECKM1: aesd [[VD:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VD]]
-; CHECKM1: aesimc {{v[0-7].16b}}, [[VH]]
 ; CHECKM1: aesd [[VE:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VE]]
+; CHECKM1: aesimc {{v[0-7].16b}}, [[VB]]
 ; CHECKM1: aesd [[VF:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VF]]
 ; CHECKM1: aesd [[VG:v[0-7].16b]], {{v[0-7].16b}}
 ; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VG]]
+; CHECKM1: aesd [[VH:v[0-7].16b]], {{v[0-7].16b}}
+; CHECKM1-NEXT: aesimc {{v[0-7].16b}}, [[VH]]
 }
 
 define void @aes_load_store(<16 x i8> *%p1, <16 x i8> *%p2 , <16 x i8> *%p3) {
 entry:
   %x1 = alloca <16 x i8>, align 16
   %x2 = alloca <16 x i8>, align 16
   %x3 = alloca <16 x i8>, align 16
   %x4 = alloca <16 x i8>, align 16