[AArch64][SchedModels] Handle virtual registers in FP/NEON predicates

[kpdev42]

Summary

Current implementation of Check[HSDQ]Form predicates doesn’t handle virtual registers and therefore isn’t useful for pre-RA scheduling. Patch fixes this implementing two function predicates: CheckQForm for checking that instruction writes 128-bit NEON register and CheckFpOrNEON which checks that instruction writes FP register (any width). The latter supersedes Check[HSD]Form predicates which are not used individually.

OS Laboratory. Huawei Russian Research Institute. Saint-Petersburg

[kpdev42]

Ping

[kpdev42]

Ping

[kpdev42]

Strip unneeded code and fix compilation issues

[kpdev42]

Ping

[andreadb]

In D114642#3257991, @kpdev42 wrote:

Ping

Sorry for the late reply.

The changes to the predicates look good to me.

However, I am not an expert of AArch64, so somebody else should review functions isFpOrNEON and isQForm.

[dmgreen]

This gets used by the exynos cpu schedule models, which are something I know little about. Ideally someone from that team would be able to check this and verify it was OK for performance. I'm not sure if that team exists in the same form it did in the past though.

llvm/lib/Target/AArch64/AArch64InstrInfo.h

106

This isn't a very general function if it only checks operand 0. There are AArch64 instructions that I would count as FP that return in X/W registers (the fcvt's for example). Perhaps make it clean in the description, that it is designed for use in scheduling models and only checks operand 0.

I would move this to bellow the getLoadStoreImmIdx and isPairableLdStInst too, so that the "load/store" functions can be kept together.

llvm/test/CodeGen/AArch64/misched-predicate-virtreg.mir

41

Is it possible to a better example? COPY instructions (especially no-op copys like these) are often expected to be removed by reg-alloc and any scheduling info you give them is likely misleading.

[kpdev42]

@dmgreen

Is it possible to a better example? COPY instructions (especially no-op copys like these) are often expected to be removed by reg-alloc and any scheduling info you give them is likely misleading.”

Well, after some further studying of Exynos M5 model the only instruction group where Q-Form predicate us used is FMINV/FMAXV/FMAXNMV/FMINNMV:

def : InstRW<[M5WriteNEONZ],  (instregex "^F(MAX|MIN)(NM)?Vv")>;

Q-Form of these instructions can be detected by operand 1, not operand 0. For instance default write resource for “fmaxv h0, v1.4h” is WriteVd and WriteVq for “fmaxv s0, v1.4s”. That’s why we’re now scanning all operands instead of just operand 0 (and both isFpOrNeon and isQForm can probably be considered general). See updated test case. Thank you!

[dmgreen]

In D114642#3314092, @kpdev42 wrote:

@dmgreen

Is it possible to a better example? COPY instructions (especially no-op copys like these) are often expected to be removed by reg-alloc and any scheduling info you give them is likely misleading.”

Well, after some further studying of Exynos M5 model the only instruction group where Q-Form predicate us used is FMINV/FMAXV/FMAXNMV/FMINNMV:

def : InstRW<[M5WriteNEONZ],  (instregex "^F(MAX|MIN)(NM)?Vv")>;

Q-Form of these instructions can be detected by operand 1, not operand 0. For instance default write resource for “fmaxv h0, v1.4h” is WriteVd and WriteVq for “fmaxv s0, v1.4s”. That’s why we’re now scanning all operands instead of just operand 0 (and both isFpOrNeon and isQForm can probably be considered general). See updated test case. Thank you!

I see. Perhaps it is sometimes clearer to be explicit in the schedule with opcode matching.

This looks like an improvement in either case, so LGTM.

llvm/lib/Target/AArch64/AArch64InstrInfo.h

public:
   static bool isPreLd(const MachineInstr &MI);
 
   /// Returns whether the instruction is a pre-indexed store.
   static bool isPreSt(const MachineInstr &MI);
 
   /// Returns whether the instruction is a pre-indexed load/store.
   static bool isPreLdSt(const MachineInstr &MI);
 
+  /// Returns whether the instruction is FP or NEON.
+  static bool isFpOrNEON(const MachineInstr &MI);
+
+  /// Returns whether the instruction is in Q form (128 bit operands)
+  static bool isQForm(const MachineInstr &MI);
+
   /// Returns the index for the immediate for a given instruction.
   static unsigned getLoadStoreImmIdx(unsigned Opc);
 
   /// Return true if pairing the given load or store may be paired with another.
   static bool isPairableLdStInst(const MachineInstr &MI);
 
   /// Return the opcode that set flags when possible.  The caller is
   /// responsible for ensuring the opc has a flag setting equivalent.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

case AArch64::STRQpre:
     return true;
   }
 }
 
 bool AArch64InstrInfo::isPreLdSt(const MachineInstr &MI) {
   return isPreLd(MI) || isPreSt(MI);
 }
 
+static const TargetRegisterClass *getRegClass(const MachineInstr &MI,
+                                              Register Reg) {
+  if (MI.getParent() == nullptr)
+    return nullptr;
+  const MachineFunction *MF = MI.getParent()->getParent();
+  return MF ? MF->getRegInfo().getRegClassOrNull(Reg) : nullptr;
+}
+
+bool AArch64InstrInfo::isQForm(const MachineInstr &MI) {
+  auto IsQFPR = [&](const MachineOperand &Op) {
+    if (!Op.isReg())
+      return false;
+    auto Reg = Op.getReg();
+    if (Reg.isPhysical())
+      return AArch64::FPR128RegClass.contains(Reg);
+    const TargetRegisterClass *TRC = ::getRegClass(MI, Reg);
+    return TRC == &AArch64::FPR128RegClass ||
+           TRC == &AArch64::FPR128_loRegClass;
+  };
+  return llvm::any_of(MI.operands(), IsQFPR);
+}
+
+bool AArch64InstrInfo::isFpOrNEON(const MachineInstr &MI) {
+  auto IsFPR = [&](const MachineOperand &Op) {
+    if (!Op.isReg())
+      return false;
+    auto Reg = Op.getReg();
+    if (Reg.isPhysical())
+      return AArch64::FPR128RegClass.contains(Reg) ||
+             AArch64::FPR64RegClass.contains(Reg) ||
+             AArch64::FPR32RegClass.contains(Reg) ||
+             AArch64::FPR16RegClass.contains(Reg) ||
+             AArch64::FPR8RegClass.contains(Reg);
+
+    const TargetRegisterClass *TRC = ::getRegClass(MI, Reg);
+    return TRC == &AArch64::FPR128RegClass ||
+           TRC == &AArch64::FPR128_loRegClass ||
+           TRC == &AArch64::FPR64RegClass ||
+           TRC == &AArch64::FPR64_loRegClass ||
+           TRC == &AArch64::FPR32RegClass || TRC == &AArch64::FPR16RegClass ||
+           TRC == &AArch64::FPR8RegClass;
+  };
+  return llvm::any_of(MI.operands(), IsFPR);
+}
+
 // Scale the unscaled offsets.  Returns false if the unscaled offset can't be
 // scaled.
 static bool scaleOffset(unsigned Opc, int64_t &Offset) {
   int Scale = AArch64InstrInfo::getMemScale(Opc);
 
   // If the byte-offset isn't a multiple of the stride, we can't scale this
   // offset.
   if (Offset % Scale != 0)

llvm/lib/Target/AArch64/AArch64SchedPredExynos.td

def ExynosScaledIdxFn   : TIIPredicate<"isExynosScaledAddr",
                                               CheckAny<
                                                 [CheckMemExtSXTW,
                                                  CheckMemExtUXTW,
                                                  CheckMemScaled]>>>],
                                          MCReturnStatement<FalsePred>>>;
 def ExynosScaledIdxPred : MCSchedPredicate<ExynosScaledIdxFn>;
 
 // Identify FP instructions.
-def ExynosFPPred : MCSchedPredicate<CheckAny<[CheckHForm,
-                                              CheckSForm,
-                                              CheckDForm,
-                                              CheckQForm]>>;
+def ExynosFPPred : MCSchedPredicate<CheckFpOrNEON>;
 
 // Identify 128-bit NEON instructions.
 def ExynosQFormPred : MCSchedPredicate<CheckQForm>;
 
 // Identify instructions that reset a register efficiently.
 def ExynosResetFn   : TIIPredicate<
                         "isExynosResetFast",
                         MCOpcodeSwitchStatement<

llvm/lib/Target/AArch64/AArch64SchedPredicates.td

 
 // Check for shifting in arithmetic and logic instructions.
 foreach I = {0-3, 8} in {
   let FunctionMapper = "AArch64_AM::getShiftValue" in
   def CheckShiftBy#I        : CheckImmOperand<3, I>;
 }
 
 // Generic predicates.
-
-// Identify whether an instruction is the 16-bit NEON form based on its result.
-def CheckHForm             : CheckAll<[CheckIsRegOperand<0>,
-                                       CheckAny<[CheckRegOperand<0, H0>,
-                                                 CheckRegOperand<0, H1>,
-                                                 CheckRegOperand<0, H2>,
-                                                 CheckRegOperand<0, H3>,
-                                                 CheckRegOperand<0, H4>,
-                                                 CheckRegOperand<0, H5>,
-                                                 CheckRegOperand<0, H6>,
-                                                 CheckRegOperand<0, H7>,
-                                                 CheckRegOperand<0, H8>,
-                                                 CheckRegOperand<0, H9>,
-                                                 CheckRegOperand<0, H10>,
-                                                 CheckRegOperand<0, H11>,
-                                                 CheckRegOperand<0, H12>,
-                                                 CheckRegOperand<0, H13>,
-                                                 CheckRegOperand<0, H14>,
-                                                 CheckRegOperand<0, H15>,
-                                                 CheckRegOperand<0, H16>,
-                                                 CheckRegOperand<0, H17>,
-                                                 CheckRegOperand<0, H18>,
-                                                 CheckRegOperand<0, H19>,
-                                                 CheckRegOperand<0, H20>,
-                                                 CheckRegOperand<0, H21>,
-                                                 CheckRegOperand<0, H22>,
-                                                 CheckRegOperand<0, H23>,
-                                                 CheckRegOperand<0, H24>,
-                                                 CheckRegOperand<0, H25>,
-                                                 CheckRegOperand<0, H26>,
-                                                 CheckRegOperand<0, H27>,
-                                                 CheckRegOperand<0, H28>,
-                                                 CheckRegOperand<0, H29>,
-                                                 CheckRegOperand<0, H30>,
-                                                 CheckRegOperand<0, H31>]>]>;
-
-// Identify whether an instruction is the 32-bit NEON form based on its result.
-def CheckSForm             : CheckAll<[CheckIsRegOperand<0>,
-                                       CheckAny<[CheckRegOperand<0, S0>,
-                                                 CheckRegOperand<0, S1>,
-                                                 CheckRegOperand<0, S2>,
-                                                 CheckRegOperand<0, S3>,
-                                                 CheckRegOperand<0, S4>,
-                                                 CheckRegOperand<0, S5>,
-                                                 CheckRegOperand<0, S6>,
-                                                 CheckRegOperand<0, S7>,
-                                                 CheckRegOperand<0, S8>,
-                                                 CheckRegOperand<0, S9>,
-                                                 CheckRegOperand<0, S10>,
-                                                 CheckRegOperand<0, S11>,
-                                                 CheckRegOperand<0, S12>,
-                                                 CheckRegOperand<0, S13>,
-                                                 CheckRegOperand<0, S14>,
-                                                 CheckRegOperand<0, S15>,
-                                                 CheckRegOperand<0, S16>,
-                                                 CheckRegOperand<0, S17>,
-                                                 CheckRegOperand<0, S18>,
-                                                 CheckRegOperand<0, S19>,
-                                                 CheckRegOperand<0, S20>,
-                                                 CheckRegOperand<0, S21>,
-                                                 CheckRegOperand<0, S22>,
-                                                 CheckRegOperand<0, S23>,
-                                                 CheckRegOperand<0, S24>,
-                                                 CheckRegOperand<0, S25>,
-                                                 CheckRegOperand<0, S26>,
-                                                 CheckRegOperand<0, S27>,
-                                                 CheckRegOperand<0, S28>,
-                                                 CheckRegOperand<0, S29>,
-                                                 CheckRegOperand<0, S30>,
-                                                 CheckRegOperand<0, S31>]>]>;
-
-// Identify whether an instruction is the 64-bit NEON form based on its result.
-def CheckDForm             : CheckAll<[CheckIsRegOperand<0>,
-                                       CheckAny<[CheckRegOperand<0, D0>,
-                                                 CheckRegOperand<0, D1>,
-                                                 CheckRegOperand<0, D2>,
-                                                 CheckRegOperand<0, D3>,
-                                                 CheckRegOperand<0, D4>,
-                                                 CheckRegOperand<0, D5>,
-                                                 CheckRegOperand<0, D6>,
-                                                 CheckRegOperand<0, D7>,
-                                                 CheckRegOperand<0, D8>,
-                                                 CheckRegOperand<0, D9>,
-                                                 CheckRegOperand<0, D10>,
-                                                 CheckRegOperand<0, D11>,
-                                                 CheckRegOperand<0, D12>,
-                                                 CheckRegOperand<0, D13>,
-                                                 CheckRegOperand<0, D14>,
-                                                 CheckRegOperand<0, D15>,
-                                                 CheckRegOperand<0, D16>,
-                                                 CheckRegOperand<0, D17>,
-                                                 CheckRegOperand<0, D18>,
-                                                 CheckRegOperand<0, D19>,
-                                                 CheckRegOperand<0, D20>,
-                                                 CheckRegOperand<0, D21>,
-                                                 CheckRegOperand<0, D22>,
-                                                 CheckRegOperand<0, D23>,
-                                                 CheckRegOperand<0, D24>,
-                                                 CheckRegOperand<0, D25>,
-                                                 CheckRegOperand<0, D26>,
-                                                 CheckRegOperand<0, D27>,
-                                                 CheckRegOperand<0, D28>,
-                                                 CheckRegOperand<0, D29>,
-                                                 CheckRegOperand<0, D30>,
-                                                 CheckRegOperand<0, D31>]>]>;
+// Identify whether an instruction is NEON or floating point
+def CheckFpOrNEON : CheckFunctionPredicateWithTII<
+  "AArch64_MC::isFpOrNEON",
+  "AArch64InstrInfo::isFpOrNEON"
+>;
 
 // Identify whether an instruction is the 128-bit NEON form based on its result.
-def CheckQForm             : CheckAll<[CheckIsRegOperand<0>,
-                                       CheckAny<[CheckRegOperand<0, Q0>,
-                                                 CheckRegOperand<0, Q1>,
-                                                 CheckRegOperand<0, Q2>,
-                                                 CheckRegOperand<0, Q3>,
-                                                 CheckRegOperand<0, Q4>,
-                                                 CheckRegOperand<0, Q5>,
-                                                 CheckRegOperand<0, Q6>,
-                                                 CheckRegOperand<0, Q7>,
-                                                 CheckRegOperand<0, Q8>,
-                                                 CheckRegOperand<0, Q9>,
-                                                 CheckRegOperand<0, Q10>,
-                                                 CheckRegOperand<0, Q11>,
-                                                 CheckRegOperand<0, Q12>,
-                                                 CheckRegOperand<0, Q13>,
-                                                 CheckRegOperand<0, Q14>,
-                                                 CheckRegOperand<0, Q15>,
-                                                 CheckRegOperand<0, Q16>,
-                                                 CheckRegOperand<0, Q17>,
-                                                 CheckRegOperand<0, Q18>,
-                                                 CheckRegOperand<0, Q19>,
-                                                 CheckRegOperand<0, Q20>,
-                                                 CheckRegOperand<0, Q21>,
-                                                 CheckRegOperand<0, Q22>,
-                                                 CheckRegOperand<0, Q23>,
-                                                 CheckRegOperand<0, Q24>,
-                                                 CheckRegOperand<0, Q25>,
-                                                 CheckRegOperand<0, Q26>,
-                                                 CheckRegOperand<0, Q27>,
-                                                 CheckRegOperand<0, Q28>,
-                                                 CheckRegOperand<0, Q29>,
-                                                 CheckRegOperand<0, Q30>,
-                                                 CheckRegOperand<0, Q31>]>]>;
+def CheckQForm : CheckFunctionPredicateWithTII<
+  "AArch64_MC::isQForm",
+  "AArch64InstrInfo::isQForm"
+>;
 
 // Identify arithmetic instructions with extend.
 def IsArithExtOp           : CheckOpcode<[ADDWrx, ADDXrx, ADDSWrx, ADDSXrx,
                                           SUBWrx, SUBXrx, SUBSWrx, SUBSXrx,
                                           ADDXrx64, ADDSXrx64,
                                           SUBXrx64, SUBSXrx64]>;
 
 // Identify arithmetic immediate instructions.

llvm/lib/Target/AArch64/MCTargetDesc/AArch64MCTargetDesc.h

 
 #include <memory>
 
 namespace llvm {
 class formatted_raw_ostream;
 class MCAsmBackend;
 class MCCodeEmitter;
 class MCContext;
+class MCInst;
 class MCInstrInfo;
 class MCInstPrinter;
 class MCRegisterInfo;
 class MCObjectTargetWriter;
 class MCStreamer;
 class MCSubtargetInfo;
 class MCTargetOptions;
 class MCTargetStreamer;

 
 MCTargetStreamer *createAArch64AsmTargetStreamer(MCStreamer &S,
                                                  formatted_raw_ostream &OS,
                                                  MCInstPrinter *InstPrint,
                                                  bool isVerboseAsm);
 
 namespace AArch64_MC {
 void initLLVMToCVRegMapping(MCRegisterInfo *MRI);
+bool isQForm(const MCInst &MI, const MCInstrInfo *MCII);
+bool isFpOrNEON(const MCInst &MI, const MCInstrInfo *MCII);
 }
 
 } // End llvm namespace
 
 // Defines symbolic names for AArch64 registers.  This defines a mapping from
 // register name to register number.
 //
 #define GET_REGINFO_ENUM

llvm/lib/Target/AArch64/MCTargetDesc/AArch64MCTargetDesc.cpp

MCPhysReg Reg;
       {codeview::RegisterId::ARM64_Q30, AArch64::Q30},
       {codeview::RegisterId::ARM64_Q31, AArch64::Q31},
 
   };
   for (const auto &I : RegMap)
     MRI->mapLLVMRegToCVReg(I.Reg, static_cast<int>(I.CVReg));
 }
 
+bool AArch64_MC::isQForm(const MCInst &MI, const MCInstrInfo *MCII) {
+  const auto &FPR128 = AArch64MCRegisterClasses[AArch64::FPR128RegClassID];
+  return llvm::any_of(MI, [](const MCOperand &Op) {
+    return Op.isReg() && FPR128.contains(Op.getReg());
+  });
+}
+
+bool AArch64_MC::isFpOrNEON(const MCInst &MI, const MCInstrInfo *MCII) {
+  const auto &FPR128 = AArch64MCRegisterClasses[AArch64::FPR128RegClassID];
+  const auto &FPR64 = AArch64MCRegisterClasses[AArch64::FPR64RegClassID];
+  const auto &FPR32 = AArch64MCRegisterClasses[AArch64::FPR32RegClassID];
+  const auto &FPR16 = AArch64MCRegisterClasses[AArch64::FPR16RegClassID];
+  const auto &FPR8 = AArch64MCRegisterClasses[AArch64::FPR8RegClassID];
+
+  auto IsFPR = [&](const MCOperand &Op) {
+    if (!Op.isReg())
+      return false;
+    auto Reg = Op.getReg();
+    return FPR128.contains(Reg) || FPR64.contains(Reg) || FPR32.contains(Reg) ||
+           FPR16.contains(Reg) || FPR8.contains(Reg);
+  };
+
+  return llvm::any_of(MI, IsFPR);
+}
+
 static MCRegisterInfo *createAArch64MCRegisterInfo(const Triple &Triple) {
   MCRegisterInfo *X = new MCRegisterInfo();
   InitAArch64MCRegisterInfo(X, AArch64::LR);
   AArch64_MC::initLLVMToCVRegMapping(X);
   return X;
 }
 
 static MCAsmInfo *createAArch64MCAsmInfo(const MCRegisterInfo &MRI,

llvm/test/CodeGen/AArch64/misched-predicate-virtreg.mir

+# RUN: llc -mcpu=exynos-m5 -mtriple=aarch64 -enable-misched -run-pass=machine-scheduler -debug-only=machine-scheduler %s -o /dev/null 2>&1 | FileCheck %s
+
+# CHECK-LABEL: ********** MI Scheduling **********
+# CHECK:       SU(0):   %0:fpr128 = COPY $q1
+# CHECK-NEXT:    # preds left       : 0
+# CHECK-NEXT:    # succs left       : 1
+# CHECK-NEXT:    # rdefs left       : 0
+# CHECK-NEXT:    Latency            : 2
+# CHECK-NEXT:    Depth              : 0
+# CHECK-NEXT:    Height             : 12
+# CHECK-NEXT:    Successors:
+# CHECK-NEXT:      SU(1): Data Latency=2 Reg=%0
+# CHECK-NEXT:    Single Issue       : false;
+# CHECK-NEXT:  SU(1):   %1:fpr32 = FMINVv4i32v %0:fpr128
+# CHECK-NEXT:    # preds left       : 1
+# CHECK-NEXT:    # succs left       : 1
+# CHECK-NEXT:    # rdefs left       : 0
+# CHECK-NEXT:    Latency            : 8
+# CHECK-NEXT:    Depth              : 2
+# CHECK-NEXT:    Height             : 10
+# CHECK-NEXT:    Predecessors:
+# CHECK-NEXT:      SU(0): Data Latency=2 Reg=%0
+# CHECK-NEXT:    Successors:
+# CHECK-NEXT:      SU(2): Data Latency=8 Reg=%1
+# CHECK-NEXT:    Single Issue       : false;
+
+name: test_qform_virtreg
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $s0, $q1
+    %0:fpr128 = COPY $q1
+    %1:fpr32 = FMINVv4i32v %0:fpr128
+    $s0 = COPY %1
+    RET_ReallyLR implicit $s0
+