[globalisel][tablegen] Import SelectionDAG's rule predicates and support the equivalent in GIRule.

[dsanders]

Summary

The SelectionDAG importer now imports rules with Predicate's attached via
Requires, PredicateControl, etc. These predicates are implemented as
bitset's to allow multiple predicates to be tested together. However,
unlike the MC layer subtarget features, each target only pays for it's own
predicates (e.g. AArch64 doesn't have 192 feature bits just because X86
needs a lot).

Both AArch64 and X86 derive at least one predicate from the MachineFunction
or Function so they must re-initialize AvailableFeatures before each
function. They also declare locals in <Target>InstructionSelector so that
computeAvailableFeatures() can use the code from SelectionDAG without
modification.

[dsanders]

Refresh

[ab]

Does this really depend on D31329? Can you invert the dependency and put the tblgen emission bits in that patch?

include/llvm/CodeGen/GlobalISel/InstructionSelector.h

36–39

Can we emit this in the .inc instead?

lib/Target/AArch64/AArch64InstructionSelector.cpp

81

If we're going to access things like OptForSize directly, I'm not sure a getter for the features is valuable.

583

We already specialize the selector (and everything else) per-subtarget. Should we also treat 'optsize' and 'optnone' the same way? We wouldn't need to do anything per-function then.

That might not be a good idea, but some of our features are already not hardware-related, and seem closer to 'optsize' than to other hardware features.

584

At least for features, this should be identical across functions that we select using the same subtarget (and nothing uses the MF anyway). Should this be in the ctor? It's unfortunate they would be separate from the other predicates, but Subtarget features are the essence of the subtarget, it seems reasonable to associate them with the subtarget-specific selector.

utils/TableGen/SubtargetFeatureInfo.cpp

107–119

It would be nice to use the same ComputeAvailableFeatures. A std::bitset does let us use more than 64 features, but is any target anywhere near that?

[dsanders]

In D31418#722861, @ab wrote:

Does this really depend on D31329? Can you invert the dependency and put the tblgen emission bits in that patch?

Yes, but only because of the test cases. I can remove the dependency fairly easily.

include/llvm/CodeGen/GlobalISel/InstructionSelector.h

36–39

I expect so. I'll look into it

lib/Target/AArch64/AArch64InstructionSelector.cpp

81

We don't normally access OptForSize directly in this implementation. It's used by ComputeAvailableFeatures() to initialize a feature bit but after that call it's unused. The only reason we store it in the class is so that the C++ code from the Predicate<> subclasses can be dropped in to ComputeAvailableFeatures() for that initialization.

583

My first attempt tried to deal with optsize/optnone inside <Target>TargetMachine::getSubtargetImpl() so that we didn't need this hook but I had problems with getting the wrong subtarget during the instruction selector. Unfortunately, I didn't manage to get to the bottom of that and I needed to move on to some dependent work so I had to fall back on this approach.

584

At least for features, this should be identical across functions that we select using the same subtarget

Agreed. There's a subtarget instance for each unique feature string.

(and nothing uses the MF anyway)

X86 needs it for NotWin64WithoutFP

Should this be in the ctor? It's unfortunate they would be separate from the other
predicates, but Subtarget features are the essence of the subtarget, it seems
reasonable to associate them with the subtarget-specific selector.

I originally wanted it in the ctor but had to move it out to support the predicates involving function attributes (e.g. optsize/optnone) and for X86's NotWin64WithoutFP.

utils/TableGen/SubtargetFeatureInfo.cpp

107–119

I agree but I think it will need to be done by moving users of the other one to this version. X86 defines 104 predicates of which 85 end up in tablegen-erated at the moment. The remainder are on rules that we don't import yet. The other implementation is limited to 64 features so we can't use that.

[dsanders]

Tablegen-erate the PredicateBitSet definitions.

[rovka]

LGTM with nits.

lib/Target/AArch64/AArch64InstructionSelector.cpp

81

This getter doesn't seem to be used anymore.

utils/TableGen/SubtargetFeatureInfo.h

59

I think this function needs a more verbose comment to highlight the differences between it and emitSubtargetFeatureFlagEnumeration.

100

This has the exact same comment as emitComputeAvailableFeatures. I think you should make it more clear why we need two different functions and what each of them does.

[dsanders]

Rebase and fix nits

[dsanders]

lib/Target/AArch64/AArch64InstructionSelector.cpp

583

I believe this is the remaining thread we still need to reach a conclusion on.

Seeing as there's quite a few LGTM'd patches queued up behind this one and this is about where we handle this rather than how, I'm going to commit this patch and I'll handle whatever conclusion we reach on this post-commit. I hope that's ok with you.

include/llvm/CodeGen/GlobalISel/InstructionSelector.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_GLOBALISEL_INSTRUCTIONSELECTOR_H
 #define LLVM_CODEGEN_GLOBALISEL_INSTRUCTIONSELECTOR_H
 
 #include "llvm/ADT/Optional.h"
 #include <cstdint>
+#include <bitset>
 
 namespace llvm {
 class MachineInstr;
+class MachineFunction;
 class MachineOperand;
 class MachineRegisterInfo;
 class RegisterBankInfo;
 class TargetInstrInfo;
 class TargetRegisterInfo;
 
+/// Container class for CodeGen predicate results.
+/// This is convenient because std::bitset does not have a constructor
+/// with an initializer list of set bits.
+///
+/// Each InstructionSelector subclass should define a PredicateBitset class with:
+///   const unsigned MAX_SUBTARGET_PREDICATES = 192;
+///   using PredicateBitset = PredicateBitsetImpl<MAX_SUBTARGET_PREDICATES>;
+/// and updating the constant to suit the target. Tablegen provides a suitable
+/// definition for the predicates in use in <Target>GenGlobalISel.inc when
+/// GET_GLOBALISEL_PREDICATE_BITSET is defined.
+template <std::size_t MaxPredicates>
+class PredicateBitsetImpl : public std::bitset<MaxPredicates> {
+public:
+  // Cannot inherit constructors because it's not supported by VC++..
+  PredicateBitsetImpl() = default;
+
+  PredicateBitsetImpl(const std::bitset<MaxPredicates> &B)
+      : std::bitset<MaxPredicates>(B) {}
+
+  PredicateBitsetImpl(std::initializer_list<unsigned> Init) {
+    for (auto I : Init)
+      std::bitset<MaxPredicates>::set(I);
+  }
+};
+
 /// Provides the logic to select generic machine instructions.
 class InstructionSelector {
 public:
   virtual ~InstructionSelector() {}
 
+  /// This is executed before selecting a function.
+  virtual void beginFunction(const MachineFunction &MF) {}
+
   /// Select the (possibly generic) instruction \p I to only use target-specific
   /// opcodes. It is OK to insert multiple instructions, but they cannot be
   /// generic pre-isel instructions.
   ///
   /// \returns whether selection succeeded.
   /// \pre  I.getParent() && I.getParent()->getParent()
   /// \post
   ///   if returns true:

lib/Target/AArch64/AArch64InstructionSelector.cpp

 using namespace llvm;
 
 #ifndef LLVM_BUILD_GLOBAL_ISEL
 #error "You shouldn't build this"
 #endif
 
 namespace {
 
+#define GET_GLOBALISEL_PREDICATE_BITSET
+#include "AArch64GenGlobalISel.inc"
+#undef GET_GLOBALISEL_PREDICATE_BITSET
+
 class AArch64InstructionSelector : public InstructionSelector {
 public:
   AArch64InstructionSelector(const AArch64TargetMachine &TM,
                              const AArch64Subtarget &STI,
                              const AArch64RegisterBankInfo &RBI);
 
+  void beginFunction(const MachineFunction &MF) override;
   bool select(MachineInstr &I) const override;
 
 private:
   /// tblgen-erated 'select' implementation, used as the initial selector for
   /// the patterns that don't require complex C++.
   bool selectImpl(MachineInstr &I) const;
 
   bool selectVaStartAAPCS(MachineInstr &I, MachineFunction &MF,
                           MachineRegisterInfo &MRI) const;
   bool selectVaStartDarwin(MachineInstr &I, MachineFunction &MF,
                            MachineRegisterInfo &MRI) const;
 
   bool selectCompareBranch(MachineInstr &I, MachineFunction &MF,
                            MachineRegisterInfo &MRI) const;
 
   bool selectArithImmed(MachineOperand &Root, MachineOperand &Result1,
                         MachineOperand &Result2) const;
 
   const AArch64TargetMachine &TM;
   const AArch64Subtarget &STI;
   const AArch64InstrInfo &TII;
   const AArch64RegisterInfo &TRI;
   const AArch64RegisterBankInfo &RBI;
+  bool ForCodeSize;
+
+  PredicateBitset AvailableFeatures;
+  PredicateBitset
+  computeAvailableFeatures(const MachineFunction *MF,
+                           const AArch64Subtarget *Subtarget) const;
 
 // We declare the temporaries used by selectImpl() in the class to minimize the
 // cost of constructing placeholder values.
 #define GET_GLOBALISEL_TEMPORARIES_DECL
 #include "AArch64GenGlobalISel.inc"
 #undef GET_GLOBALISEL_TEMPORARIES_DECL
 };
 
 } // end anonymous namespace
 
 #define GET_GLOBALISEL_IMPL
 #include "AArch64GenGlobalISel.inc"
 #undef GET_GLOBALISEL_IMPL
 
 AArch64InstructionSelector::AArch64InstructionSelector(
     const AArch64TargetMachine &TM, const AArch64Subtarget &STI,
     const AArch64RegisterBankInfo &RBI)
     : InstructionSelector(), TM(TM), STI(STI), TII(*STI.getInstrInfo()),
-      TRI(*STI.getRegisterInfo()), RBI(RBI)
+      TRI(*STI.getRegisterInfo()), RBI(RBI), ForCodeSize(), AvailableFeatures()
 #define GET_GLOBALISEL_TEMPORARIES_INIT
 #include "AArch64GenGlobalISel.inc"
 #undef GET_GLOBALISEL_TEMPORARIES_INIT
 {
 }
 
 // FIXME: This should be target-independent, inferred from the types declared
 // for each class in the bank.
MIB = BuildMI(*I.getParent(), I, I.getDebugLoc(), TII.get(AArch64::STRXui))
             .addImm(0)
             .addMemOperand(*I.memoperands_begin());
 
   constrainSelectedInstRegOperands(*MIB, TII, TRI, RBI);
   I.eraseFromParent();
   return true;
 }
 
+void AArch64InstructionSelector::beginFunction(
+    const MachineFunction &MF) {
+  ForCodeSize = MF.getFunction()->optForSize();
+  AvailableFeatures = computeAvailableFeatures(&MF, &STI);
+}
+
 bool AArch64InstructionSelector::select(MachineInstr &I) const {
   assert(I.getParent() && "Instruction should be in a basic block!");
   assert(I.getParent()->getParent() && "Instruction should be in a function!");
 
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 

lib/Target/X86/X86.h

 FunctionPass *createX86FixupBWInsts();
 
 void initializeFixupBWInstPassPass(PassRegistry &);
 
 /// This pass replaces EVEX ecnoded of AVX-512 instructiosn by VEX 
 /// encoding when possible in order to reduce code size.
 FunctionPass *createX86EvexToVexInsts();
 
-InstructionSelector *createX86InstructionSelector(X86Subtarget &,
+InstructionSelector *createX86InstructionSelector(const X86TargetMachine &TM,
+                                                  X86Subtarget &,
                                                   X86RegisterBankInfo &);
 
 void initializeEvexToVexInstPassPass(PassRegistry &);
 
 } // End llvm namespace
 
 #endif

lib/Target/X86/X86InstructionSelector.cpp

 using namespace llvm;
 
 #ifndef LLVM_BUILD_GLOBAL_ISEL
 #error "You shouldn't build this"
 #endif
 
 namespace {
 
+#define GET_GLOBALISEL_PREDICATE_BITSET
+#include "X86GenGlobalISel.inc"
+#undef GET_GLOBALISEL_PREDICATE_BITSET
+
 class X86InstructionSelector : public InstructionSelector {
 public:
-  X86InstructionSelector(const X86Subtarget &STI,
+  X86InstructionSelector(const X86TargetMachine &TM, const X86Subtarget &STI,
                          const X86RegisterBankInfo &RBI);
 
+  void beginFunction(const MachineFunction &MF) override;
   bool select(MachineInstr &I) const override;
 
 private:
   /// tblgen-erated 'select' implementation, used as the initial selector for
   /// the patterns that don't require complex C++.
   bool selectImpl(MachineInstr &I) const;
 
   // TODO: remove after selectImpl support pattern with a predicate.
bool selectLoadStoreOp(MachineInstr &I, MachineRegisterInfo &MRI,
                          MachineFunction &MF) const;
   bool selectFrameIndex(MachineInstr &I, MachineRegisterInfo &MRI,
                         MachineFunction &MF) const;
   bool selectConstant(MachineInstr &I, MachineRegisterInfo &MRI,
                       MachineFunction &MF) const;
   bool selectTrunc(MachineInstr &I, MachineRegisterInfo &MRI,
                    MachineFunction &MF) const;
 
+  const X86TargetMachine &TM;
   const X86Subtarget &STI;
   const X86InstrInfo &TII;
   const X86RegisterInfo &TRI;
   const X86RegisterBankInfo &RBI;
+  bool OptForSize;
+  bool OptForMinSize;
+
+  PredicateBitset AvailableFeatures;
+  PredicateBitset computeAvailableFeatures(const MachineFunction *MF,
+                                           const X86Subtarget *Subtarget) const;
 
 #define GET_GLOBALISEL_TEMPORARIES_DECL
 #include "X86GenGlobalISel.inc"
 #undef GET_GLOBALISEL_TEMPORARIES_DECL
 };
 
 } // end anonymous namespace
 
 #define GET_GLOBALISEL_IMPL
 #include "X86GenGlobalISel.inc"
 #undef GET_GLOBALISEL_IMPL
 
-X86InstructionSelector::X86InstructionSelector(const X86Subtarget &STI,
+X86InstructionSelector::X86InstructionSelector(const X86TargetMachine &TM,
+                                               const X86Subtarget &STI,
                                                const X86RegisterBankInfo &RBI)
-    : InstructionSelector(), STI(STI), TII(*STI.getInstrInfo()),
-      TRI(*STI.getRegisterInfo()), RBI(RBI)
+    : InstructionSelector(), TM(TM), STI(STI), TII(*STI.getInstrInfo()),
+      TRI(*STI.getRegisterInfo()), RBI(RBI), OptForSize(false),
+      OptForMinSize(false), AvailableFeatures()
 #define GET_GLOBALISEL_TEMPORARIES_INIT
 #include "X86GenGlobalISel.inc"
 #undef GET_GLOBALISEL_TEMPORARIES_INIT
 {
 }
 
 // FIXME: This should be target-independent, inferred from the types declared
 // for each class in the bank.
if (!RBI.constrainGenericRegister(DstReg, *RC, MRI)) {
                    << " operand\n");
       return false;
     }
   }
   I.setDesc(TII.get(X86::COPY));
   return true;
 }
 
+void X86InstructionSelector::beginFunction(const MachineFunction &MF) {
+  OptForSize = MF.getFunction()->optForSize();
+  OptForMinSize = MF.getFunction()->optForMinSize();
+  AvailableFeatures = computeAvailableFeatures(&MF, &STI);
+}
+
 bool X86InstructionSelector::select(MachineInstr &I) const {
   assert(I.getParent() && "Instruction should be in a basic block!");
   assert(I.getParent()->getParent() && "Instruction should be in a function!");
 
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
if (DstRC == SrcRC) {
     return false;
   }
 
   I.setDesc(TII.get(X86::COPY));
   return true;
 }
 
 InstructionSelector *
-llvm::createX86InstructionSelector(X86Subtarget &Subtarget,
+llvm::createX86InstructionSelector(const X86TargetMachine &TM,
+                                   X86Subtarget &Subtarget,
                                    X86RegisterBankInfo &RBI) {
-  return new X86InstructionSelector(Subtarget, RBI);
+  return new X86InstructionSelector(TM, Subtarget, RBI);
 }

lib/Target/X86/X86TargetMachine.cpp

 #else
     X86GISelActualAccessor *GISel = new X86GISelActualAccessor();
 
     GISel->CallLoweringInfo.reset(new X86CallLowering(*I->getTargetLowering()));
     GISel->Legalizer.reset(new X86LegalizerInfo(*I, *this));
 
     auto *RBI = new X86RegisterBankInfo(*I->getRegisterInfo());
     GISel->RegBankInfo.reset(RBI);
-    GISel->InstSelector.reset(createX86InstructionSelector(*I, *RBI));
+    GISel->InstSelector.reset(createX86InstructionSelector(*this, *I, *RBI));
 #endif
     I->setGISelAccessor(*GISel);
   }
   return I.get();
 }
 
 //===----------------------------------------------------------------------===//
 // Command line options for x86

test/TableGen/GlobalISelEmitter.td

 def gi_complex :
     GIComplexOperandMatcher<s32, (ops i32imm, i32imm), "selectComplexPattern">,
     GIComplexPatternEquiv<complex>;
 
 def m1 : OperandWithDefaultOps <i32, (ops (i32 -1))>;
 def Z : OperandWithDefaultOps <i32, (ops R0)>;
 def m1Z : OperandWithDefaultOps <i32, (ops (i32 -1), R0)>;
 
-//===- Test the function definition boilerplate. --------------------------===//
+def HasA : Predicate<"Subtarget->hasA()">;
+def HasB : Predicate<"Subtarget->hasB()">;
+
+//===- Test the function boilerplate. -------------------------------------===//
+
+// CHECK-LABEL: enum SubtargetFeatureBits : uint8_t {
+// CHECK-NEXT:    Feature_HasABit = 0,
+// CHECK-NEXT:    Feature_HasBBit = 1,
+// CHECK-NEXT:  };
+
+// CHECK-LABEL: static const char *SubtargetFeatureNames[] = {
+// CHECK-NEXT:    "Feature_HasA",
+// CHECK-NEXT:    "Feature_HasB",
+// CHECK-NEXT:    nullptr
+// CHECK-NEXT:  };
+
+// CHECK-LABEL: PredicateBitset MyTargetInstructionSelector::
+// CHECK-NEXT:  computeAvailableFeatures(const MachineFunction *MF, const MyTargetSubtarget *Subtarget) const {
+// CHECK-NEXT:    PredicateBitset Features;
+// CHECK-NEXT:    if (Subtarget->hasA())
+// CHECK-NEXT:      Features[Feature_HasABit] = 1;
+// CHECK-NEXT:    if (Subtarget->hasB())
+// CHECK-NEXT:      Features[Feature_HasBBit] = 1;
+// CHECK-NEXT:    return Features;
+// CHECK-NEXT:  }
 
 // CHECK: bool MyTargetInstructionSelector::selectImpl(MachineInstr &I) const {
 // CHECK: MachineFunction &MF = *I.getParent()->getParent();
 // CHECK: const MachineRegisterInfo &MRI = MF.getRegInfo();
 
 //===- Test a pattern with multiple ComplexPattern operands. --------------===//
 //
 

 
 
 def ADD : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2),
             [(set GPR32:$dst, (add GPR32:$src1, GPR32:$src2))]>;
 
 //===- Test a nested instruction match. -----------------------------------===//
 
 // CHECK-LABEL: if ([&]() {
+// CHECK-NEXT:    PredicateBitset ExpectedFeatures = {Feature_HasABit};
+// CHECK-NEXT:    if ((AvailableFeatures & ExpectedFeatures) != ExpectedFeatures)
+// CHECK-NEXT:      return false;
 // CHECK-NEXT:    MachineInstr &MI0 = I;
 // CHECK-NEXT:    if (MI0.getNumOperands() < 3)
 // CHECK-NEXT:      return false;
 // CHECK-NEXT:    if (!MI0.getOperand(1).isReg())
 // CHECK-NEXT:      return false;
 // CHECK-NEXT:    MachineInstr &MI1 = *MRI.getVRegDef(MI0.getOperand(1).getReg());
 // CHECK-NEXT:    if (MI1.getNumOperands() < 3)
 // CHECK-NEXT:      return false;

 // CHECK-NEXT:     I.eraseFromParent();
 // CHECK-NEXT:     MachineInstr &NewI = *MIB;
 // CHECK-NEXT:     constrainSelectedInstRegOperands(NewI, TII, TRI, RBI);
 // CHECK-NEXT:     return true;
 // CHECK-NEXT:   }
 
 // We also get a second rule by commutativity.
 // CHECK-LABEL: if ([&]() {
+// CHECK-NEXT:    PredicateBitset ExpectedFeatures = {Feature_HasABit};
+// CHECK-NEXT:    if ((AvailableFeatures & ExpectedFeatures) != ExpectedFeatures)
+// CHECK-NEXT:      return false;
 // CHECK-NEXT:    MachineInstr &MI0 = I;
 // CHECK-NEXT:    if (MI0.getNumOperands() < 3)
 // CHECK-NEXT:      return false;
 // CHECK-NEXT:    if (!MI0.getOperand(2).isReg())
 // CHECK-NEXT:      return false;
 // CHECK-NEXT:    MachineInstr &MI1 = *MRI.getVRegDef(MI0.getOperand(2).getReg());
 // CHECK-NEXT:    if (MI1.getNumOperands() < 3)
 // CHECK-NEXT:      return false;

 // CHECK-NEXT:     I.eraseFromParent();
 // CHECK-NEXT:     MachineInstr &NewI = *MIB;
 // CHECK-NEXT:     constrainSelectedInstRegOperands(NewI, TII, TRI, RBI);
 // CHECK-NEXT:     return true;
 // CHECK-NEXT:   }
 
 def MULADD : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2, GPR32:$src3),
                [(set GPR32:$dst,
-                     (mul (add GPR32:$src1, GPR32:$src2), GPR32:$src3))]>;
+                     (mul (add GPR32:$src1, GPR32:$src2), GPR32:$src3))]>,
+             Requires<[HasA]>;
 
 //===- Test another simple pattern with regclass operands. ----------------===//
 
 // CHECK-LABEL: if ([&]() {
+// CHECK-NEXT:    PredicateBitset ExpectedFeatures = {Feature_HasABit, Feature_HasBBit};
+// CHECK-NEXT:    if ((AvailableFeatures & ExpectedFeatures) != ExpectedFeatures)
+// CHECK-NEXT:      return false;
 // CHECK-NEXT:    MachineInstr &MI0 = I;
 // CHECK-NEXT:    if (MI0.getNumOperands() < 3)
 // CHECK-NEXT:      return false;
 // CHECK-NEXT:    if ((MI0.getOpcode() == TargetOpcode::G_MUL) &&
 // CHECK-NEXT:        ((/* dst */ (MRI.getType(MI0.getOperand(0).getReg()) == (LLT::scalar(32))) &&
 // CHECK-NEXT:         ((&RBI.getRegBankFromRegClass(MyTarget::GPR32RegClass) == RBI.getRegBank(MI0.getOperand(0).getReg(), MRI, TRI))))) &&
 // CHECK-NEXT:        ((/* src1 */ (MRI.getType(MI0.getOperand(1).getReg()) == (LLT::scalar(32))) &&
 // CHECK-NEXT:         ((&RBI.getRegBankFromRegClass(MyTarget::GPR32RegClass) == RBI.getRegBank(MI0.getOperand(1).getReg(), MRI, TRI))))) &&

 // CHECK-NEXT:      MachineInstr &NewI = *MIB;
 // CHECK-NEXT:      constrainSelectedInstRegOperands(NewI, TII, TRI, RBI);
 // CHECK-NEXT:      return true;
 // CHECK-NEXT:    }
 // CHECK-NEXT:    return false;
 // CHECK-NEXT:  }()) { return true; }
 
 def MUL : I<(outs GPR32:$dst), (ins GPR32:$src2, GPR32:$src1),
-             [(set GPR32:$dst, (mul GPR32:$src1, GPR32:$src2))]>;
+             [(set GPR32:$dst, (mul GPR32:$src1, GPR32:$src2))]>,
+          Requires<[HasA, HasB]>;
 
 //===- Test a pattern with ComplexPattern operands. -----------------------===//
 //
 
 // CHECK-LABEL: if ([&]() {
 // CHECK-NEXT:    MachineInstr &MI0 = I;
 // CHECK-NEXT:    if (MI0.getNumOperands() < 3)
 // CHECK-NEXT:      return false;

utils/TableGen/AsmMatcherEmitter.cpp

#endif
 
   // Emit the subclass predicate routine.
   emitIsSubclass(Target, Info.Classes, OS);
 
   // Emit the routine to validate an operand against a match class.
   emitValidateOperandClass(Info, OS);
 
   // Emit the available features compute function.
-  SubtargetFeatureInfo::emitComputeAvailableFeatures(
+  SubtargetFeatureInfo::emitComputeAssemblerAvailableFeatures(
       Info.Target.getName(), ClassName, "ComputeAvailableFeatures",
       Info.SubtargetFeatures, OS);
 
   StringToOffsetTable StringTable;
 
   size_t MaxNumOperands = 0;
   unsigned MaxMnemonicIndex = 0;
   bool HasDeprecation = false;

utils/TableGen/CodeEmitterGen.cpp

SubtargetFeatureInfo::emitSubtargetFeatureFlagEnumeration(SubtargetFeatures,
                                                             o);
 
   // Emit the name table for error messages.
   o << "#ifndef NDEBUG\n";
   SubtargetFeatureInfo::emitNameTable(SubtargetFeatures, o);
   o << "#endif // NDEBUG\n";
 
   // Emit the available features compute function.
-  SubtargetFeatureInfo::emitComputeAvailableFeatures(
+  SubtargetFeatureInfo::emitComputeAssemblerAvailableFeatures(
       Target.getName(), "MCCodeEmitter", "computeAvailableFeatures",
       SubtargetFeatures, o);
 
   // Emit the predicate verifier.
   o << "void " << Target.getName()
     << "MCCodeEmitter::verifyInstructionPredicates(\n"
     << "    const MCInst &Inst, uint64_t AvailableFeatures) const {\n"
     << "#ifndef NDEBUG\n"

utils/TableGen/GlobalISelEmitter.cpp

 /// selector for the patterns that don't require complex C++.
 ///
 /// FIXME: We'll probably want to eventually define a base
 /// "TargetGenInstructionSelector" class.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "CodeGenDAGPatterns.h"
+#include "SubtargetFeatureInfo.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/CodeGen/MachineValueType.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/LowLevelTypeImpl.h"
 #include "llvm/Support/ScopedPrinter.h"
for (const auto &P : N->getPredicateFns()) {
     if (P.isAlwaysTrue())
       Explanation += " always-true";
     if (P.isImmediatePattern())
       Explanation += " immediate";
   }
   return Explanation;
 }
 
-static std::string explainRulePredicates(const ArrayRef<Init *> Predicates) {
-  std::string Explanation = "";
-  StringRef Separator = "";
-  for (const auto *P : Predicates) {
-    Explanation += Separator;
-
-    if (const DefInit *PDef = dyn_cast<DefInit>(P)) {
-      Explanation += PDef->getDef()->getName();
-    } else
-      Explanation += "<unknown>";
-  }
-  return Explanation;
-}
-
 std::string explainOperator(Record *Operator) {
   if (Operator->isSubClassOf("SDNode"))
     return " (" + Operator->getValueAsString("Opcode") + ")";
 
   if (Operator->isSubClassOf("Intrinsic"))
     return (" (Operator is an Intrinsic, " + Operator->getName() + ")").str();
 
   return " (Operator not understood)";
class RuleMatcher {
 
   /// A map of instruction matchers to the local variables created by
   /// emitCxxCaptureStmts().
   std::map<const InstructionMatcher *, std::string> InsnVariableNames;
 
   /// ID for the next instruction variable defined with defineInsnVar()
   unsigned NextInsnVarID;
 
+  std::vector<Record *> RequiredFeatures;
+
 public:
   RuleMatcher()
       : Matchers(), Actions(), InsnVariableNames(), NextInsnVarID(0) {}
   RuleMatcher(RuleMatcher &&Other) = default;
   RuleMatcher &operator=(RuleMatcher &&Other) = default;
 
   InstructionMatcher &addInstructionMatcher();
+  void addRequiredFeature(Record *Feature);
 
   template <class Kind, class... Args> Kind &addAction(Args &&... args);
 
   std::string defineInsnVar(raw_ostream &OS, const InstructionMatcher &Matcher,
                             StringRef Value);
   StringRef getInsnVarName(const InstructionMatcher &InsnMatcher) const;
 
   void emitCxxCapturedInsnList(raw_ostream &OS);
   void emitCxxCaptureStmts(raw_ostream &OS, StringRef Expr);
 
-  void emit(raw_ostream &OS);
+  void emit(raw_ostream &OS,
+            std::map<Record *, SubtargetFeatureInfo, LessRecordByID>
+                SubtargetFeatures);
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool isHigherPriorityThan(const RuleMatcher &B) const;
 
   /// Report the maximum number of temporary operands needed by the rule
   /// matcher.
public:
   }
 };
 
 InstructionMatcher &RuleMatcher::addInstructionMatcher() {
   Matchers.emplace_back(new InstructionMatcher());
   return *Matchers.back();
 }
 
+void RuleMatcher::addRequiredFeature(Record *Feature) {
+  RequiredFeatures.push_back(Feature);
+}
+
 template <class Kind, class... Args>
 Kind &RuleMatcher::addAction(Args &&... args) {
   Actions.emplace_back(llvm::make_unique<Kind>(std::forward<Args>(args)...));
   return *static_cast<Kind *>(Actions.back().get());
 }
 
 std::string RuleMatcher::defineInsnVar(raw_ostream &OS,
                                        const InstructionMatcher &Matcher,

 /// Emit C++ statements to check the shape of the match and capture
 /// instructions into local variables.
 void RuleMatcher::emitCxxCaptureStmts(raw_ostream &OS, StringRef Expr) {
   assert(Matchers.size() == 1 && "Cannot handle multi-root matchers yet");
   std::string InsnVarName = defineInsnVar(OS, *Matchers.front(), Expr);
   Matchers.front()->emitCxxCaptureStmts(OS, *this, InsnVarName);
 }
 
-void RuleMatcher::emit(raw_ostream &OS) {
+void RuleMatcher::emit(raw_ostream &OS,
+                       std::map<Record *, SubtargetFeatureInfo, LessRecordByID>
+                           SubtargetFeatures) {
   if (Matchers.empty())
     llvm_unreachable("Unexpected empty matcher!");
 
   // The representation supports rules that require multiple roots such as:
   //    %ptr(p0) = ...
   //    %elt0(s32) = G_LOAD %ptr
   //    %1(p0) = G_ADD %ptr, 4
   //    %elt1(s32) = G_LOAD p0 %1
   // which could be usefully folded into:
   //    %ptr(p0) = ...
   //    %elt0(s32), %elt1(s32) = TGT_LOAD_PAIR %ptr
   // on some targets but we don't need to make use of that yet.
   assert(Matchers.size() == 1 && "Cannot handle multi-root matchers yet");
-  OS << "if ([&]() {\n";
+
+  OS << "if (";
+  OS << "[&]() {\n";
+  if (!RequiredFeatures.empty()) {
+    OS << "  PredicateBitset ExpectedFeatures = {";
+    StringRef Separator = "";
+    for (const auto &Predicate : RequiredFeatures) {
+      const auto &I = SubtargetFeatures.find(Predicate);
+      assert(I != SubtargetFeatures.end() && "Didn't import predicate?");
+      OS << Separator << I->second.getEnumBitName();
+      Separator = ", ";
+    }
+    OS << "};\n";
+    OS << "if ((AvailableFeatures & ExpectedFeatures) != ExpectedFeatures)\n"
+       << "  return false;\n";
+  }
 
   emitCxxCaptureStmts(OS, "I");
 
   OS << "    if (";
   Matchers.front()->emitCxxPredicateExpr(OS, *this,
                                          getInsnVarName(*Matchers.front()));
   OS << ") {\n";
 
private:
   /// This is defined using 'GINodeEquiv' in the target description.
   DenseMap<Record *, const CodeGenInstruction *> NodeEquivs;
 
   /// Keep track of the equivalence between ComplexPattern's and
   /// GIComplexOperandMatcher. Map entries are specified by subclassing
   /// GIComplexPatternEquiv.
   DenseMap<const Record *, const Record *> ComplexPatternEquivs;
 
+  // Map of predicates to their subtarget features.
+  std::map<Record *, SubtargetFeatureInfo, LessRecordByID> SubtargetFeatures;
+
   void gatherNodeEquivs();
   const CodeGenInstruction *findNodeEquiv(Record *N) const;
 
-  Error importRulePredicates(RuleMatcher &M, ArrayRef<Init *> Predicates) const;
+  Error importRulePredicates(RuleMatcher &M, ArrayRef<Init *> Predicates);
   Expected<InstructionMatcher &>
   createAndImportSelDAGMatcher(InstructionMatcher &InsnMatcher,
                                const TreePatternNode *Src) const;
   Error importChildMatcher(InstructionMatcher &InsnMatcher,
                            TreePatternNode *SrcChild, unsigned OpIdx,
                            unsigned &TempOpIdx) const;
   Expected<BuildMIAction &> createAndImportInstructionRenderer(
       RuleMatcher &M, const TreePatternNode *Dst,
       const InstructionMatcher &InsnMatcher) const;
   Error importExplicitUseRenderer(BuildMIAction &DstMIBuilder,
                                   TreePatternNode *DstChild,
                                   const InstructionMatcher &InsnMatcher) const;
   Error
   importImplicitDefRenderers(BuildMIAction &DstMIBuilder,
                              const std::vector<Record *> &ImplicitDefs) const;
 
   /// Analyze pattern \p P, returning a matcher for it if possible.
   /// Otherwise, return an Error explaining why we don't support it.
   Expected<RuleMatcher> runOnPattern(const PatternToMatch &P);
+
+  void declareSubtargetFeature(Record *Predicate);
 };
 
 void GlobalISelEmitter::gatherNodeEquivs() {
   assert(NodeEquivs.empty());
   for (Record *Equiv : RK.getAllDerivedDefinitions("GINodeEquiv"))
     NodeEquivs[Equiv->getValueAsDef("Node")] =
         &Target.getInstruction(Equiv->getValueAsDef("I"));
 

 
 GlobalISelEmitter::GlobalISelEmitter(RecordKeeper &RK)
     : RK(RK), CGP(RK), Target(CGP.getTargetInfo()) {}
 
 //===- Emitter ------------------------------------------------------------===//
 
 Error
 GlobalISelEmitter::importRulePredicates(RuleMatcher &M,
-                                        ArrayRef<Init *> Predicates) const {
-  if (!Predicates.empty())
-    return failedImport("Pattern has a rule predicate (" +
-                        explainRulePredicates(Predicates) + ")");
+                                        ArrayRef<Init *> Predicates) {
+  for (const Init *Predicate : Predicates) {
+    const DefInit *PredicateDef = static_cast<const DefInit *>(Predicate);
+    declareSubtargetFeature(PredicateDef->getDef());
+    M.addRequiredFeature(PredicateDef->getDef());
+  }
+
   return Error::success();
 }
 
 Expected<InstructionMatcher &> GlobalISelEmitter::createAndImportSelDAGMatcher(
     InstructionMatcher &InsnMatcher, const TreePatternNode *Src) const {
   // Start with the defined operands (i.e., the results of the root operator).
   if (Src->getExtTypes().size() > 1)
     return failedImport("Src pattern has multiple results");
std::stable_sort(Rules.begin(), Rules.end(),
               }
               return false;
             });
 
   unsigned MaxTemporaries = 0;
   for (const auto &Rule : Rules)
     MaxTemporaries = std::max(MaxTemporaries, Rule.countTemporaryOperands());
 
+  OS << "#ifdef GET_GLOBALISEL_PREDICATE_BITSET\n"
+     << "const unsigned MAX_SUBTARGET_PREDICATES = " << SubtargetFeatures.size()
+     << ";\n"
+     << "using PredicateBitset = "
+        "PredicateBitsetImpl<MAX_SUBTARGET_PREDICATES>;\n"
+     << "#endif // ifdef GET_GLOBALISEL_PREDICATE_BITSET\n\n";
+
   OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n";
   for (unsigned I = 0; I < MaxTemporaries; ++I)
     OS << "  mutable MachineOperand TempOp" << I << ";\n";
   OS << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n\n";
 
   OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n";
   for (unsigned I = 0; I < MaxTemporaries; ++I)
     OS << ", TempOp" << I << "(MachineOperand::CreatePlaceholder())\n";
   OS << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n\n";
 
-  OS << "#ifdef GET_GLOBALISEL_IMPL\n"
-     << "bool " << Target.getName()
+  OS << "#ifdef GET_GLOBALISEL_IMPL\n";
+  SubtargetFeatureInfo::emitSubtargetFeatureBitEnumeration(SubtargetFeatures,
+                                                           OS);
+  SubtargetFeatureInfo::emitNameTable(SubtargetFeatures, OS);
+  SubtargetFeatureInfo::emitComputeAvailableFeatures(
+      Target.getName(), "InstructionSelector", "computeAvailableFeatures",
+      SubtargetFeatures, OS);
+
+  OS << "bool " << Target.getName()
      << "InstructionSelector::selectImpl(MachineInstr &I) const {\n"
      << "  MachineFunction &MF = *I.getParent()->getParent();\n"
      << "  const MachineRegisterInfo &MRI = MF.getRegInfo();\n";
 
   for (auto &Rule : Rules) {
-    Rule.emit(OS);
+    Rule.emit(OS, SubtargetFeatures);
     ++NumPatternEmitted;
   }
 
   OS << "  return false;\n"
      << "}\n"
      << "#endif // ifdef GET_GLOBALISEL_IMPL\n";
 }
 
+void GlobalISelEmitter::declareSubtargetFeature(Record *Predicate) {
+  if (SubtargetFeatures.count(Predicate) == 0)
+    SubtargetFeatures.emplace(
+        Predicate, SubtargetFeatureInfo(Predicate, SubtargetFeatures.size()));
+}
+
 } // end anonymous namespace
 
 //===----------------------------------------------------------------------===//
 
 namespace llvm {
 void EmitGlobalISel(RecordKeeper &RK, raw_ostream &OS) {
   GlobalISelEmitter(RK).run(OS);
 }
 } // End llvm namespace

utils/TableGen/SubtargetFeatureInfo.h

struct SubtargetFeatureInfo {
 
   SubtargetFeatureInfo(Record *D, uint64_t Idx) : TheDef(D), Index(Idx) {}
 
   /// \brief The name of the enumerated constant identifying this feature.
   std::string getEnumName() const {
     return "Feature_" + TheDef->getName().str();
   }
 
+  /// \brief The name of the enumerated constant identifying the bitnumber for
+  /// this feature.
+  std::string getEnumBitName() const {
+    return "Feature_" + TheDef->getName().str() + "Bit";
+  }
+
   void dump() const;
   static std::vector<std::pair<Record *, SubtargetFeatureInfo>>
   getAll(const RecordKeeper &Records);
 
   /// Emit the subtarget feature flag definitions.
+  ///
+  /// This version emits the bit value for the feature and is therefore limited
+  /// to 64 feature bits.
   static void emitSubtargetFeatureFlagEnumeration(
       std::map<Record *, SubtargetFeatureInfo, LessRecordByID>
           &SubtargetFeatures,
       raw_ostream &OS);
 
+  /// Emit the subtarget feature flag definitions.
+  ///
+  /// This version emits the bit index for the feature and can therefore support
+  /// more than 64 feature bits.
+  static void emitSubtargetFeatureBitEnumeration(
+      std::map<Record *, SubtargetFeatureInfo, LessRecordByID>
+          &SubtargetFeatures,
+      raw_ostream &OS);
+
   static void emitNameTable(std::map<Record *, SubtargetFeatureInfo,
                                      LessRecordByID> &SubtargetFeatures,
                             raw_ostream &OS);
 
   /// Emit the function to compute the list of available features given a
   /// subtarget.
   ///
+  /// This version is used for subtarget features defined using Predicate<>
+  /// and supports more than 64 feature bits.
+  ///
   /// \param TargetName The name of the target as used in class prefixes (e.g.
   ///                   <TargetName>Subtarget)
   /// \param ClassName  The name of the class (without the <Target> prefix)
   ///                   that will contain the generated functions.
   /// \param FuncName   The name of the function to emit.
   /// \param SubtargetFeatures A map of TableGen records to the
   ///                          SubtargetFeatureInfo equivalent.
   static void emitComputeAvailableFeatures(
       StringRef TargetName, StringRef ClassName, StringRef FuncName,
       std::map<Record *, SubtargetFeatureInfo, LessRecordByID>
           &SubtargetFeatures,
       raw_ostream &OS);
+
+  /// Emit the function to compute the list of available features given a
+  /// subtarget.
+  ///
+  /// This version is used for subtarget features defined using
+  /// AssemblerPredicate<> and supports up to 64 feature bits.
+  ///
+  /// \param TargetName The name of the target as used in class prefixes (e.g.
+  ///                   <TargetName>Subtarget)
+  /// \param ClassName  The name of the class (without the <Target> prefix)
+  ///                   that will contain the generated functions.
+  /// \param FuncName   The name of the function to emit.
+  /// \param SubtargetFeatures A map of TableGen records to the
+  ///                          SubtargetFeatureInfo equivalent.
+  static void emitComputeAssemblerAvailableFeatures(
+      StringRef TargetName, StringRef ClassName, StringRef FuncName,
+      std::map<Record *, SubtargetFeatureInfo, LessRecordByID>
+          &SubtargetFeatures,
+      raw_ostream &OS);
 };
 } // end namespace llvm
 
 #endif // LLVM_UTIL_TABLEGEN_SUBTARGETFEATUREINFO_H

utils/TableGen/SubtargetFeatureInfo.cpp

void SubtargetFeatureInfo::emitSubtargetFeatureFlagEnumeration(
   for (const auto &SF : SubtargetFeatures) {
     const SubtargetFeatureInfo &SFI = SF.second;
     OS << "  " << SFI.getEnumName() << " = (1ULL << " << SFI.Index << "),\n";
   }
   OS << "  Feature_None = 0\n";
   OS << "};\n\n";
 }
 
+void SubtargetFeatureInfo::emitSubtargetFeatureBitEnumeration(
+    std::map<Record *, SubtargetFeatureInfo, LessRecordByID> &SubtargetFeatures,
+    raw_ostream &OS) {
+  OS << "// Bits for subtarget features that participate in "
+     << "instruction matching.\n";
+  OS << "enum SubtargetFeatureBits : "
+     << getMinimalTypeForRange(SubtargetFeatures.size()) << " {\n";
+  for (const auto &SF : SubtargetFeatures) {
+    const SubtargetFeatureInfo &SFI = SF.second;
+    OS << "  " << SFI.getEnumBitName() << " = " << SFI.Index << ",\n";
+  }
+  OS << "};\n\n";
+}
+
 void SubtargetFeatureInfo::emitNameTable(
     std::map<Record *, SubtargetFeatureInfo, LessRecordByID> &SubtargetFeatures,
     raw_ostream &OS) {
   // Need to sort the name table so that lookup by the log of the enum value
   // gives the proper name. More specifically, for a feature of value 1<<n,
   // SubtargetFeatureNames[n] should be the name of the feature.
   uint64_t IndexUB = 0;
   for (const auto &SF : SubtargetFeatures)
void SubtargetFeatureInfo::emitNameTable(
   OS << "  nullptr\n"
      << "};\n\n";
 }
 
 void SubtargetFeatureInfo::emitComputeAvailableFeatures(
     StringRef TargetName, StringRef ClassName, StringRef FuncName,
     std::map<Record *, SubtargetFeatureInfo, LessRecordByID> &SubtargetFeatures,
     raw_ostream &OS) {
+  OS << "PredicateBitset " << TargetName << ClassName << "::\n"
+     << FuncName << "(const MachineFunction *MF, const " << TargetName
+     << "Subtarget *Subtarget) const {\n";
+  OS << "  PredicateBitset Features;\n";
+  for (const auto &SF : SubtargetFeatures) {
+    const SubtargetFeatureInfo &SFI = SF.second;
+
+    OS << "  if (" << SFI.TheDef->getValueAsString("CondString") << ")\n";
+    OS << "    Features[" << SFI.getEnumBitName() << "] = 1;\n";
+  }
+  OS << "  return Features;\n";
+  OS << "}\n\n";
+}
+
+void SubtargetFeatureInfo::emitComputeAssemblerAvailableFeatures(
+    StringRef TargetName, StringRef ClassName, StringRef FuncName,
+    std::map<Record *, SubtargetFeatureInfo, LessRecordByID> &SubtargetFeatures,
+    raw_ostream &OS) {
   OS << "uint64_t " << TargetName << ClassName << "::\n"
      << FuncName << "(const FeatureBitset& FB) const {\n";
   OS << "  uint64_t Features = 0;\n";
   for (const auto &SF : SubtargetFeatures) {
     const SubtargetFeatureInfo &SFI = SF.second;
 
     OS << "  if (";
     std::string CondStorage =

utils/TableGen/Types.cpp

if (Range > 0xFF)
     return "uint16_t";
   return "uint8_t";
 }
 
 const char *llvm::getMinimalTypeForEnumBitfield(uint64_t Size) {
   uint64_t MaxIndex = Size;
   if (MaxIndex > 0)
     MaxIndex--;
+  assert(MaxIndex <= 64 && "Too many bits");
   return getMinimalTypeForRange(1ULL << MaxIndex);
 }