[AArch64] Add some missing fusion subtarget features

[porglezomp]

Summary

Referencing ARM's software optimization guides:

A65 - 4.8 Instruction fusion

Address, AES, and MOVZ/MOVK literals

A72 - 4.11 Fast literal generation

• 4.12 PC-relative address calculation

A76 - 4.6. AES Encryption/Decryption
A77/A78/A78C/X1 - 4.13 Instruction fusion

CMP/CMN, TST, BICS + B.cond fusion
AES fusion

[AArch64] Make Cortex B.cc fusions more precise

The ArithmeticBccFusion feature expects to be able to fuse general
flag-updating arithmetic with a B.cc, for example an arbitrary SUBS
instructions and not just a CMP.

Since the Cortex cores are documented as fusing CMP/CMN/TST, and the A77
optimization guide specifies that BICS fusion must have a destination of
XZR or WZR, these cores should use a separate subtarget feature for
specifically fusing only comparisons with B.cc.

[porglezomp]

The existing fusion tests seem a little bit weak—the machine scheduler with the base Cortex-A57 machine models that most of those use seemed to pass the address and literal fusion on its own (by coincidence?) without the explicit features requested. I was unable to make that test case stronger to see the A57 machine model not exploiting those fusion pairs when the feature was disabled. Is that worth worrying about?

[SjoerdMeijer]

Thank, LGTM.

In D94457#2491639, @porglezomp wrote:

The existing fusion tests seem a little bit weak—the machine scheduler with the base Cortex-A57 machine models that most of those use seemed to pass the address and literal fusion on its own (by coincidence?) without the explicit features requested. I was unable to make that test case stronger to see the A57 machine model not exploiting those fusion pairs when the feature was disabled. Is that worth worrying about?

Yeah, perhaps there's room for improvement, it's not something that worries me very much.

llvm/lib/Target/AArch64/AArch64.td

221

Yep, makes sense I think.

624

Check

637

Check

669

Check

683

Check

694

Check

711

Check

739

Check

[porglezomp]

I don't have commit access, so could someone commit the change for me? Thanks.

[SjoerdMeijer]

Thanks for contributing this. I have committed this on your behalf in rG815dd4b29208.
If you plan to do more LLVM work, you can always consider requesting an account so you can commit patches yourself (but am of course happy to do it on your behalf).

[alexander-shaposhnikov]

@SjoerdMeijer, @porglezomp - many thanks in advance,
I'd like to clarify one aspect related to this diff- is it intentional that currently only pairs of cmp + bcc get fused (but e.g. not cmp + cbz or cmp + cbnz) (unless I'm missing something)
and another question - after looking at https://github.com/gcc-mirror/gcc/blob/master/gcc/config/aarch64/aarch64.cc#L1326 - would it make sense to enable CmpBccFusion by default.

cc: @dmgreen

[dmgreen]

I have for a while thought that we are missing a number of tuning features for AArch64 cores, but the (very brief) look into enabling more fusion on more cores ended up showing worse performance. The changes were not very large IIRC, but I worry that the more aggressive fusion was forcing instruction into worse positions. It may have just been that it was unlucky with noise. We get a lot of fusion naturally by the way the scheduler positions cmp/br and cmp/csel.

The optimization guides usually list the instructions that can be merged. If you have some evidence that any of them are producing better performance then that sounds like a useful patch. The performance number I got were done too quickly to draw any strong conclusions from.

As for the questions you actually asked - cbz and cbnz take a reg under aarch64 so don't have a separate cmp. For adding to -mcpu=generic, the general principle is that is needs to generally improve performance without hurting any other cores. Especially around the big-little cpus found in android phones. (Or it enables other optimizations like the linker relaxation from FeatureFuseAdrpAdd). GCC believes it is a benefit or benign, but they may have a slightly different algorithm for deciding when to fuse, I'm not sure. So long as we have some results that show it's improving things or flat for a selection of cores (big and little), then it should be OK.

llvm/lib/Target/AArch64/AArch64.td

 def FeatureArithmeticBccFusion : SubtargetFeature<
     "arith-bcc-fusion", "HasArithmeticBccFusion", "true",
     "CPU fuses arithmetic+bcc operations">;
 
 def FeatureArithmeticCbzFusion : SubtargetFeature<
     "arith-cbz-fusion", "HasArithmeticCbzFusion", "true",
     "CPU fuses arithmetic + cbz/cbnz operations">;
 
+def FeatureCmpBccFusion : SubtargetFeature<
+    "cmp-bcc-fusion", "HasCmpBccFusion", "true",
+    "CPU fuses cmp+bcc operations">;
+
 def FeatureFuseAddress : SubtargetFeature<
     "fuse-address", "HasFuseAddress", "true",
     "CPU fuses address generation and memory operations">;
 
 def FeatureFuseAES : SubtargetFeature<
     "fuse-aes", "HasFuseAES", "true",
     "CPU fuses AES crypto operations">;
 

 
 def ProcA65     : SubtargetFeature<"a65", "ARMProcFamily", "CortexA65",
                                    "Cortex-A65 ARM processors", [
                                    HasV8_2aOps,
                                    FeatureCrypto,
                                    FeatureDotProd,
                                    FeatureFPARMv8,
                                    FeatureFullFP16,
+                                   FeatureFuseAddress,
+                                   FeatureFuseAES,
+                                   FeatureFuseLiterals,
                                    FeatureNEON,
                                    FeatureRAS,
                                    FeatureRCPC,
                                    FeatureSSBS,
                                    ]>;
 
 def ProcA72     : SubtargetFeature<"a72", "ARMProcFamily", "CortexA72",
                                    "Cortex-A72 ARM processors", [
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureFPARMv8,
                                    FeatureFuseAES,
+                                   FeatureFuseLiterals,
                                    FeatureNEON,
                                    FeaturePerfMon
                                    ]>;
 
 def ProcA73     : SubtargetFeature<"a73", "ARMProcFamily", "CortexA73",
                                    "Cortex-A73 ARM processors", [
                                    FeatureCRC,
                                    FeatureCrypto,
def ProcA75     : SubtargetFeature<"a75", "ARMProcFamily", "CortexA75",
                                    FeatureRCPC,
                                    FeaturePerfMon
                                    ]>;
 
 def ProcA76     : SubtargetFeature<"a76", "ARMProcFamily", "CortexA76",
                                    "Cortex-A76 ARM processors", [
                                     HasV8_2aOps,
                                     FeatureFPARMv8,
+                                    FeatureFuseAES,
                                     FeatureNEON,
                                     FeatureRCPC,
                                     FeatureCrypto,
                                     FeatureFullFP16,
                                     FeatureDotProd,
                                     FeatureSSBS
                                     ]>;
 
 def ProcA77     : SubtargetFeature<"a77", "ARMProcFamily", "CortexA77",
                                    "Cortex-A77 ARM processors", [
                                     HasV8_2aOps,
+                                    FeatureCmpBccFusion,
                                     FeatureFPARMv8,
+                                    FeatureFuseAES,
                                     FeatureNEON, FeatureRCPC,
                                     FeatureCrypto,
                                     FeatureFullFP16,
                                     FeatureDotProd
                                     ]>;
 
 def ProcA78 : SubtargetFeature<"cortex-a78", "ARMProcFamily",
                                "CortexA78",
                                "Cortex-A78 ARM processors", [
                                HasV8_2aOps,
+                               FeatureCmpBccFusion,
                                FeatureCrypto,
                                FeatureFPARMv8,
                                FeatureFuseAES,
                                FeatureNEON,
                                FeatureRCPC,
                                FeaturePerfMon,
                                FeaturePostRAScheduler,
                                FeatureSPE,
                                FeatureFullFP16,
                                FeatureSSBS,
                                FeatureDotProd]>;
 
 def ProcA78C : SubtargetFeature<"cortex-a78c", "ARMProcFamily",
                                 "CortexA78C",
                                 "Cortex-A78C ARM processors", [
                                 HasV8_2aOps,
+                                FeatureCmpBccFusion,
                                 FeatureCrypto,
                                 FeatureDotProd,
                                 FeatureFlagM,
                                 FeatureFP16FML,
                                 FeatureFPARMv8,
                                 FeatureFullFP16,
                                 FeatureFuseAES,
                                 FeatureNEON,
def ProcR82 : SubtargetFeature<"cortex-r82", "ARMProcFamily",
                                // TODO: crypto and FuseAES
                                // All other features are implied by v8_0r ops:
                                HasV8_0rOps,
                                ]>;
 
 def ProcX1 : SubtargetFeature<"cortex-x1", "ARMProcFamily", "CortexX1",
                                   "Cortex-X1 ARM processors", [
                                   HasV8_2aOps,
+                                  FeatureCmpBccFusion,
                                   FeatureCrypto,
                                   FeatureFPARMv8,
                                   FeatureFuseAES,
                                   FeatureNEON,
                                   FeatureRCPC,
                                   FeaturePerfMon,
                                   FeaturePostRAScheduler,
                                   FeatureSPE,

llvm/lib/Target/AArch64/AArch64MacroFusion.cpp

 #include "llvm/CodeGen/TargetInstrInfo.h"
 
 using namespace llvm;
 
 namespace {
 
 /// CMN, CMP, TST followed by Bcc
 static bool isArithmeticBccPair(const MachineInstr *FirstMI,
-                                const MachineInstr &SecondMI) {
+                                const MachineInstr &SecondMI, bool CmpOnly) {
   if (SecondMI.getOpcode() != AArch64::Bcc)
     return false;
 
   // Assume the 1st instr to be a wildcard if it is unspecified.
   if (FirstMI == nullptr)
     return true;
 
+  // If we're in CmpOnly mode, we only fuse arithmetic instructions that
+  // discard their result.
+  if (CmpOnly && !(FirstMI->getOperand(0).getReg() == AArch64::XZR ||
+                   FirstMI->getOperand(0).getReg() == AArch64::WZR)) {
+    return false;
+  }
+
   switch (FirstMI->getOpcode()) {
   case AArch64::ADDSWri:
   case AArch64::ADDSWrr:
   case AArch64::ADDSXri:
   case AArch64::ADDSXrr:
   case AArch64::ANDSWri:
   case AArch64::ANDSWrr:
   case AArch64::ANDSXri:

 static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
                                    const TargetSubtargetInfo &TSI,
                                    const MachineInstr *FirstMI,
                                    const MachineInstr &SecondMI) {
   const AArch64Subtarget &ST = static_cast<const AArch64Subtarget&>(TSI);
 
   // All checking functions assume that the 1st instr is a wildcard if it is
   // unspecified.
-  if (ST.hasArithmeticBccFusion() && isArithmeticBccPair(FirstMI, SecondMI))
+  if (ST.hasCmpBccFusion() || ST.hasArithmeticBccFusion()) {
+    bool CmpOnly = !ST.hasArithmeticBccFusion();
+    if (isArithmeticBccPair(FirstMI, SecondMI, CmpOnly))
       return true;
+  }
   if (ST.hasArithmeticCbzFusion() && isArithmeticCbzPair(FirstMI, SecondMI))
     return true;
   if (ST.hasFuseAES() && isAESPair(FirstMI, SecondMI))
     return true;
   if (ST.hasFuseCryptoEOR() && isCryptoEORPair(FirstMI, SecondMI))
     return true;
   if (ST.hasFuseLiterals() && isLiteralsPair(FirstMI, SecondMI))
     return true;

llvm/lib/Target/AArch64/AArch64Subtarget.h

protected:
   bool ExynosAsCheapAsMove = false;
   bool UsePostRAScheduler = false;
   bool Misaligned128StoreIsSlow = false;
   bool Paired128IsSlow = false;
   bool STRQroIsSlow = false;
   bool UseAlternateSExtLoadCVTF32Pattern = false;
   bool HasArithmeticBccFusion = false;
   bool HasArithmeticCbzFusion = false;
+  bool HasCmpBccFusion = false;
   bool HasFuseAddress = false;
   bool HasFuseAES = false;
   bool HasFuseArithmeticLogic = false;
   bool HasFuseCCSelect = false;
   bool HasFuseCryptoEOR = false;
   bool HasFuseLiterals = false;
   bool DisableLatencySchedHeuristic = false;
   bool UseRSqrt = false;
public:
   bool isMisaligned128StoreSlow() const { return Misaligned128StoreIsSlow; }
   bool isPaired128Slow() const { return Paired128IsSlow; }
   bool isSTRQroSlow() const { return STRQroIsSlow; }
   bool useAlternateSExtLoadCVTF32Pattern() const {
     return UseAlternateSExtLoadCVTF32Pattern;
   }
   bool hasArithmeticBccFusion() const { return HasArithmeticBccFusion; }
   bool hasArithmeticCbzFusion() const { return HasArithmeticCbzFusion; }
+  bool hasCmpBccFusion() const { return HasCmpBccFusion; }
   bool hasFuseAddress() const { return HasFuseAddress; }
   bool hasFuseAES() const { return HasFuseAES; }
   bool hasFuseArithmeticLogic() const { return HasFuseArithmeticLogic; }
   bool hasFuseCCSelect() const { return HasFuseCCSelect; }
   bool hasFuseCryptoEOR() const { return HasFuseCryptoEOR; }
   bool hasFuseLiterals() const { return HasFuseLiterals; }
 
   /// Return true if the CPU supports any kind of instruction fusion.

llvm/test/CodeGen/AArch64/misched-fusion-addr.ll

 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=fuse-address | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a65    | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m3     | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m4     | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m5     | FileCheck %s
 
 target triple = "aarch64-unknown"
 
 @var_8bit = dso_local global i8 0
 @var_16bit = dso_local global i16 0

llvm/test/CodeGen/AArch64/misched-fusion-aes.ll

 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=+fuse-aes,+crypto | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=generic -mattr=+crypto | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a53 | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a57 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a65 | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a72 | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a73 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a76 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a77 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a78 | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a78c| FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-x1  | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m3  | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m4  | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m5  | FileCheck %s
 
 declare <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d, <16 x i8> %k)
 declare <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %d)
 declare <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d, <16 x i8> %k)
 declare <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %d)

llvm/test/CodeGen/AArch64/misched-fusion-lit.ll

 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=-fuse-literals | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKDONT
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=+fuse-literals | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a57      | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a65      | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a72      | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m3       | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m4       | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m5       | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
 
 @g = common local_unnamed_addr global i8* null, align 8
 
 define dso_local i8* @litp(i32 %a, i32 %b) {
 entry: