[TableGen] Store predicates in PatternToMatch as ListInit *. Add string for HwModeFeatures

[craig.topper]

Summary

This uses to be how predicates were handled prior to HwMode being
added. When the Predicates were converted to a std::vector it
significantly increased the cost of a compare in GenerateVariants.
Since ListInit's are uniquified by tablegen, we can use a simple
pointer comparison to check for identical lists.

In order to store the HwMode, we now add a separate string to
PatternToMatch. This will be appended separately to the predicate
string in getPredicateCheck. A new getPredicateRecords is added
to allow GlobalISel and getPredicateCheck to both get the sorted
list of Records. GlobalISel was ignoring any HwMode predicates
before and still is.

There is one slight change here, ListInits with different predicate
orders aren't sorted so the filtering in GenerateVariants might
fail to detect two isomorphic patterns with different predicate
orders. This doesn't seem to be happening in tree today.

My hope is this will allow us to remove all the BitVector tracking
in GenerateVariants that was making up for predicates beeing
expensive to compare. There's a decent amount of heap allocations
there on large targets like X86, AMDGPU, and RISCV.

[Paul-C-Anagnostopoulos]

I presume this passes all the TableGen tests.

How else did you test it? Knowing this may help me in the future.

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

1436

No need to mention initial size ", 4" in the function definition, is there?

1477

Any reason you didn't combine the catenations with '+' ?

[craig.topper]

In D100691#2696632, @Paul-C-Anagnostopoulos wrote:

I presume this passes all the TableGen tests.

How else did you test it? Knowing this may help me in the future.

All output files for all targets are identical before and after this patch.

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

1436

I think it's needed because I'm returning the vector object, not a reference. Maybe I should make the caller pass it by reference.

1477

Concatenating them would produce a temporary std::string that would need a heap allocation. That would then need to be copied into the SmallString. If the total string is small enough the SmallString won't heap allocate. If we copy each piece in separately we avoid heap allocations for temporaries.

[craig.topper]

Have caller create the SmallVector and pass by reference. This is much more common in the code base.

[Paul-C-Anagnostopoulos]

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

1477

Sounds like there's room for an optimization that looks at the complete sequence of catenations to decide whether there is room in the SmallString. But good to know.

[Paul-C-Anagnostopoulos]

For my future reference: What is the sequence you go through to ensure that no target output files change? Do you just build with the current system, then build with the revision and check that no output file dates changed?

Does the revision title want an "[NFC]"?

[frasercrmck]

Just a minor thing: the description has a typo in PatterntToMatch with an extra 't'.

llvm/utils/TableGen/CodeGenDAGPatterns.h

1080

Just wondering: would StringRef be more appropriate here? Maybe it's not idiomatic to tablegen code.

[craig.topper]

Return StringRef instead of const std::string &. Fix compilation errors form that.

[kparzysz]

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

1477

Isn't Twine specifically designed to optimize concatenations?

4363

Is this part removed in your patch? The "default" case could be vacuously true if there are no conditions attached to it, and it could be applied when a more specialized case should be applied instead.
The "default" was originally invented for situations where the specific modes don't cover all possibilities when combined. I guess in practice the "default" ends up being one of the explicitly defined modes, so this "accidental application" may not be happening.

[craig.topper]

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

1477

Yes, but Twine is used almost exclusively as an unnamed temporary object that is either passed to a function that takes a Twine argument or it is converted to a string before it goes out of scope. It has no operator+= so you can't incrementally update it.

4363

I removed it because it wasn't doing what the comment said, the DefaultPred vector is only pushed to it's not used otherwise. Since it was incomplete, I didn't know how to update it.

[RKSimon]

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

1451

Use llvm::sort ?

[craig.topper]

Use llvm::sort

[kparzysz]

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

4363

Yeah, see the comment below. It's a latent bug, unfortunately.

I added this change on top of the current main branch (i.e. without your changes) and both check-llvm and check-clang passed for all targets. I guess it hasn't caused trouble for the reasons I stated, but it could eventually. Another variant of the intent described above was to make sure that the predicated patterns can be tried in any order without affecting the result.

I'm not sure how you want to proceed. There is always the possibility of changing the initial intent if it doesn't break the current code...

4375

This should be

if (HasDefault) {
  ModeChecks[DefaultMode] = DefaultPred;
  AppendPattern(P, DefaultMode);
}

[craig.topper]

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

4375

That ends up not really working. The ModeChecks are cached for all patterns, but the DefaultPred vector lives inside the loop for a particular pattern. The DefaultPred vector only gets populated when a pattern contains a non-default mode that has never been seen by any pattern before due to the ModeChecks.find(M) != ModeChecks.end() in the first mode loop.

So only one or zero patterns end up getting a default check depending on whether the pattern applies in the default mode.

Is the DefaultPred supposed to be the inverse of the modes that apply to a particular pattern, or the inverse of the predicates for all modes? If the latter you can't calculate it inside the pattern loops and would need to calculate it based on the complete list of modes not just which modes apply to a particular pattern.

[kparzysz]

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

4375

Sorry, you're right. Here's a second attempt to fix this: https://reviews.llvm.org/D100889. That's not a real review, I just didn't want to clutter this one.

[craig.topper]

Rebase after the change to have negative predicates for HwMode. Now I'm building the full string when we add HwMode so I can do the negation without additional state.

llvm/utils/TableGen/CodeGenDAGPatterns.h

 //===- CodeGenDAGPatterns.h - Read DAG patterns from .td file ---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
Record *getImpResult(unsigned RN) const {
     assert(RN < ImpResults.size());
     return ImpResults[RN];
   }
 
   TreePatternNodePtr getSrcPattern() const { return SrcPattern; }
   TreePatternNodePtr getResultPattern() const { return ResultPattern; }
 };
 
-/// This class represents a condition that has to be satisfied for a pattern
-/// to be tried. It is a generalization of a class "Pattern" from Target.td:
-/// in addition to the Target.td's predicates, this class can also represent
-/// conditions associated with HW modes. Both types will eventually become
-/// strings containing C++ code to be executed, the difference is in how
-/// these strings are generated.
-class Predicate {
-public:
-  Predicate(Record *R, bool C = true) : Def(R), IfCond(C), IsHwMode(false) {
-    assert(R->isSubClassOf("Predicate") &&
-           "Predicate objects should only be created for records derived"
-           "from Predicate class");
-  }
-  Predicate(StringRef FS, bool C = true) : Def(nullptr), Features(FS.str()),
-    IfCond(C), IsHwMode(true) {}
-
-  /// Return a string which contains the C++ condition code that will serve
-  /// as a predicate during instruction selection.
-  std::string getCondString() const {
-    // The string will excute in a subclass of SelectionDAGISel.
-    // Cast to std::string explicitly to avoid ambiguity with StringRef.
-    std::string C = IsHwMode
-                        ? std::string("MF->getSubtarget().checkFeatures(\"" +
-                                      Features + "\")")
-                        : std::string(Def->getValueAsString("CondString"));
-    if (C.empty())
-      return "";
-    return IfCond ? C : "!("+C+')';
-  }
-
-  bool operator==(const Predicate &P) const {
-    return IfCond == P.IfCond && IsHwMode == P.IsHwMode && Def == P.Def &&
-           Features == P.Features;
-  }
-  bool operator<(const Predicate &P) const {
-    if (IsHwMode != P.IsHwMode)
-      return IsHwMode < P.IsHwMode;
-    assert(!Def == !P.Def && "Inconsistency between Def and IsHwMode");
-    if (IfCond != P.IfCond)
-      return IfCond < P.IfCond;
-    if (Def)
-      return LessRecord()(Def, P.Def);
-    return Features < P.Features;
-  }
-  Record *Def;            ///< Predicate definition from .td file, null for
-                          ///< HW modes.
-  std::string Features;   ///< Feature string for HW mode.
-  bool IfCond;            ///< The boolean value that the condition has to
-                          ///< evaluate to for this predicate to be true.
-  bool IsHwMode;          ///< Does this predicate correspond to a HW mode?
-};
-
 /// PatternToMatch - Used by CodeGenDAGPatterns to keep tab of patterns
 /// processed to produce isel.
 class PatternToMatch {
   Record          *SrcRecord;   // Originating Record for the pattern.
+  ListInit        *Predicates;  // Top level predicate conditions to match.
   TreePatternNodePtr SrcPattern;      // Source pattern to match.
   TreePatternNodePtr DstPattern;      // Resulting pattern.
-  std::vector<Predicate> Predicates;  // Top level predicate conditions
-                                      // to match.
   std::vector<Record*> Dstregs; // Physical register defs being matched.
+  std::string      HwModeFeatures;
   int              AddedComplexity; // Add to matching pattern complexity.
   unsigned         ID;          // Unique ID for the record.
   unsigned         ForceMode;   // Force this mode in type inference when set.
 
 public:
-  PatternToMatch(Record *srcrecord, std::vector<Predicate> preds,
-                 TreePatternNodePtr src, TreePatternNodePtr dst,
-                 std::vector<Record *> dstregs, int complexity,
-                 unsigned uid, unsigned setmode = 0)
-      : SrcRecord(srcrecord), SrcPattern(src), DstPattern(dst),
-        Predicates(std::move(preds)), Dstregs(std::move(dstregs)),
-        AddedComplexity(complexity), ID(uid), ForceMode(setmode) {}
+  PatternToMatch(Record *srcrecord, ListInit *preds, TreePatternNodePtr src,
+                 TreePatternNodePtr dst, std::vector<Record *> dstregs,
+                 int complexity, unsigned uid, unsigned setmode = 0,
+                 std::string hwmodefeatures = std::string())
+      : SrcRecord(srcrecord), Predicates(preds), SrcPattern(src),
+        DstPattern(dst), Dstregs(std::move(dstregs)),
+        HwModeFeatures(std::move(hwmodefeatures)), AddedComplexity(complexity),
+        ID(uid), ForceMode(setmode) {}
 
   Record          *getSrcRecord()  const { return SrcRecord; }
+  ListInit        *getPredicates() const { return Predicates; }
   TreePatternNode *getSrcPattern() const { return SrcPattern.get(); }
   TreePatternNodePtr getSrcPatternShared() const { return SrcPattern; }
   TreePatternNode *getDstPattern() const { return DstPattern.get(); }
   TreePatternNodePtr getDstPatternShared() const { return DstPattern; }
   const std::vector<Record*> &getDstRegs() const { return Dstregs; }
+  const std::string &getHwModeFeatures() const { return HwModeFeatures; }
   int         getAddedComplexity() const { return AddedComplexity; }
-  const std::vector<Predicate> &getPredicates() const { return Predicates; }
   unsigned getID() const { return ID; }
   unsigned getForceMode() const { return ForceMode; }
 
   std::string getPredicateCheck() const;
+  SmallVector<Record *, 4> getPredicateRecords() const;
 
   /// Compute the complexity metric for the input pattern.  This roughly
   /// corresponds to the number of nodes that are covered.
   int getPatternComplexity(const CodeGenDAGPatterns &CGP) const;
 };
 
 class CodeGenDAGPatterns {
   RecordKeeper &Records;
private:
   void ParseDefaultOperands();
   void ParseInstructions();
   void ParsePatterns();
   void ExpandHwModeBasedTypes();
   void InferInstructionFlags();
   void GenerateVariants();
   void VerifyInstructionFlags();
 
-  std::vector<Predicate> makePredList(ListInit *L);
-
   void ParseOnePattern(Record *TheDef,
                        TreePattern &Pattern, TreePattern &Result,
                        const std::vector<Record *> &InstImpResults);
   void AddPatternToMatch(TreePattern *Pattern, PatternToMatch &&PTM);
   void FindPatternInputsAndOutputs(
       TreePattern &I, TreePatternNodePtr Pat,
       std::map<std::string, TreePatternNodePtr> &InstInputs,
       MapVector<std::string, TreePatternNodePtr,

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

 //===- CodeGenDAGPatterns.cpp - Read DAG patterns from .td file -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 
 /// Compute the complexity metric for the input pattern.  This roughly
 /// corresponds to the number of nodes that are covered.
 int PatternToMatch::
 getPatternComplexity(const CodeGenDAGPatterns &CGP) const {
   return getPatternSize(getSrcPattern(), CGP) + getAddedComplexity();
 }
 
+SmallVector<Record *, 4> PatternToMatch::getPredicateRecords() const {
+  SmallVector<Record *, 4> PredicateRecs;
+  for (Init *I : Predicates->getValues()) {
+    if (DefInit *Pred = dyn_cast<DefInit>(I)) {
+      Record *Def = Pred->getDef();
+      if (!Def->isSubClassOf("Predicate")) {
+#ifndef NDEBUG
+        Def->dump();
+#endif
+        llvm_unreachable("Unknown predicate type!");
+      }
+      PredicateRecs.push_back(Def);
+    }
+  }
+  // Sort so that different orders get canonicalized to the same string.
+  std::sort(PredicateRecs.begin(), PredicateRecs.end(), LessRecord());
+
+  return PredicateRecs;
+}
+
 /// getPredicateCheck - Return a single string containing all of this
 /// pattern's predicates concatenated with "&&" operators.
 ///
 std::string PatternToMatch::getPredicateCheck() const {
-  SmallVector<const Predicate*,4> PredList;
-  for (const Predicate &P : Predicates) {
-    if (!P.getCondString().empty())
-      PredList.push_back(&P);
-  }
-  llvm::sort(PredList, deref<std::less<>>());
-
-  std::string Check;
-  for (unsigned i = 0, e = PredList.size(); i != e; ++i) {
-    if (i != 0)
-      Check += " && ";
-    Check += '(' + PredList[i]->getCondString() + ')';
+  SmallString<128> PredicateCheck;
+  for (Record *Pred : getPredicateRecords()) {
+    StringRef CondString = Pred->getValueAsString("CondString");
+    if (CondString.empty())
+      continue;
+    if (!PredicateCheck.empty())
+      PredicateCheck += " && ";
+    PredicateCheck += "(";
+    PredicateCheck += CondString;
+    PredicateCheck += ")";
+  }
+
+  if (!HwModeFeatures.empty()) {
+    if (!PredicateCheck.empty())
+      PredicateCheck += " && ";
+    PredicateCheck += "(MF->getSubtarget().checkFeatures(\"";
+    PredicateCheck += HwModeFeatures;
+    PredicateCheck += "\"))";
   }
-  return Check;
+
+  return std::string(PredicateCheck);
 }
 
 //===----------------------------------------------------------------------===//
 // SDTypeConstraint implementation
 //
 
 SDTypeConstraint::SDTypeConstraint(Record *R, const CodeGenHwModes &CGH) {
   OperandNo = R->getValueAsInt("OperandNum");
static void FindNames(TreePatternNode *P,
   }
 
   if (!P->isLeaf()) {
     for (unsigned i = 0, e = P->getNumChildren(); i != e; ++i)
       FindNames(P->getChild(i), Names, PatternTop);
   }
 }
 
-std::vector<Predicate> CodeGenDAGPatterns::makePredList(ListInit *L) {
-  std::vector<Predicate> Preds;
-  for (Init *I : L->getValues()) {
-    if (DefInit *Pred = dyn_cast<DefInit>(I))
-      Preds.push_back(Pred->getDef());
-    else
-      llvm_unreachable("Non-def on the list");
-  }
-
-  // Sort so that different orders get canonicalized to the same string.
-  llvm::sort(Preds);
-  return Preds;
-}
-
 void CodeGenDAGPatterns::AddPatternToMatch(TreePattern *Pattern,
                                            PatternToMatch &&PTM) {
   // Do some sanity checking on the pattern we're about to match.
   std::string Reason;
   if (!PTM.getSrcPattern()->canPatternMatch(Reason, *this)) {
     PrintWarning(Pattern->getRecord()->getLoc(),
       Twine("Pattern can never match: ") + Reason);
     return;
void CodeGenDAGPatterns::ParseOnePattern(Record *TheDef,
   // value type, and use a specific register class. If, for some mode,
   // that register class does not accept that type, the type inference
   // will lead to a contradiction, which is not an error however, but
   // a sign that this pattern will simply never match.
   if (Temp.getOnlyTree()->hasPossibleType())
     for (const auto &T : Pattern.getTrees())
       if (T->hasPossibleType())
         AddPatternToMatch(&Pattern,
-                          PatternToMatch(TheDef, makePredList(Preds),
-                                         T, Temp.getOnlyTree(),
+                          PatternToMatch(TheDef, Preds, T, Temp.getOnlyTree(),
                                          InstImpResults, Complexity,
                                          TheDef->getID()));
 }
 
 void CodeGenDAGPatterns::ParsePatterns() {
   std::vector<Record*> Patterns = Records.getAllDerivedDefinitions("Pattern");
 
   for (Record *CurPattern : Patterns) {
for (const auto &I : VTS)
       Modes.insert(I.first);
 
   for (unsigned i = 0, e = N->getNumChildren(); i != e; ++i)
     collectModes(Modes, N->getChild(i));
 }
 
 void CodeGenDAGPatterns::ExpandHwModeBasedTypes() {
   const CodeGenHwModes &CGH = getTargetInfo().getHwModes();
-  std::map<unsigned,std::vector<Predicate>> ModeChecks;
+  std::map<unsigned, std::string> ModeChecks;
   std::vector<PatternToMatch> Copy;
   PatternsToMatch.swap(Copy);
 
   auto AppendPattern = [this, &ModeChecks](PatternToMatch &P, unsigned Mode) {
     TreePatternNodePtr NewSrc = P.getSrcPattern()->clone();
     TreePatternNodePtr NewDst = P.getDstPattern()->clone();
     if (!NewSrc->setDefaultMode(Mode) || !NewDst->setDefaultMode(Mode)) {
       return;
     }
 
-    std::vector<Predicate> Preds = P.getPredicates();
-    const std::vector<Predicate> &MC = ModeChecks[Mode];
-    llvm::append_range(Preds, MC);
-    PatternsToMatch.emplace_back(P.getSrcRecord(), std::move(Preds),
+    PatternsToMatch.emplace_back(P.getSrcRecord(), P.getPredicates(),
                                  std::move(NewSrc), std::move(NewDst),
-                                 P.getDstRegs(),
-                                 P.getAddedComplexity(), Record::getNewUID(),
-                                 Mode);
+                                 P.getDstRegs(), P.getAddedComplexity(),
+                                 Record::getNewUID(), Mode, ModeChecks[Mode]);
   };
 
   for (PatternToMatch &P : Copy) {
     TreePatternNodePtr SrcP = nullptr, DstP = nullptr;
     if (P.getSrcPattern()->hasProperTypeByHwMode())
       SrcP = P.getSrcPatternShared();
     if (P.getDstPattern()->hasProperTypeByHwMode())
       DstP = P.getDstPatternShared();
     if (!SrcP && !DstP) {
       PatternsToMatch.push_back(P);
       continue;
     }
 
     std::set<unsigned> Modes;
     if (SrcP)
       collectModes(Modes, SrcP.get());
     if (DstP)
       collectModes(Modes, DstP.get());
 
-    // The predicate for the default mode needs to be constructed for each
-    // pattern separately.
-    // Since not all modes must be present in each pattern, if a mode m is
-    // absent, then there is no point in constructing a check for m. If such
-    // a check was created, it would be equivalent to checking the default
-    // mode, except not all modes' predicates would be a part of the checking
-    // code. The subsequently generated check for the default mode would then
-    // have the exact same patterns, but a different predicate code. To avoid
-    // duplicated patterns with different predicate checks, construct the
-    // default check as a negation of all predicates that are actually present
-    // in the source/destination patterns.
-    std::vector<Predicate> DefaultPred;
-
     for (unsigned M : Modes) {
       if (M == DefaultMode)
         continue;
       if (ModeChecks.find(M) != ModeChecks.end())
         continue;
 
       // Fill the map entry for this mode.
       const HwMode &HM = CGH.getMode(M);
-      ModeChecks[M].emplace_back(Predicate(HM.Features, true));
-
-      // Add negations of the HM's predicates to the default predicate.
-      DefaultPred.emplace_back(Predicate(HM.Features, false));
+      ModeChecks[M] = HM.Features;
     }
 
     for (unsigned M : Modes) {
       if (M == DefaultMode)
         continue;
       AppendPattern(P, M);
     }
 
     bool HasDefault = Modes.count(DefaultMode);
     if (HasDefault)
       AppendPattern(P, DefaultMode);
   }
 }
 
 /// Dependent variable map for CodeGenDAGPattern variant generation
 typedef StringMap<int> DepVarMap;
 
 static void FindDepVarsOf(TreePatternNode *N, DepVarMap &DepMap) {
   if (N->isLeaf()) {
LLVM_DEBUG(errs() << "FOUND VARIANTS OF: ";
                PatternsToMatch[i].getSrcPattern()->dump(); errs() << "\n");
 
     PatternsWithVariants[i] = std::make_pair(DepVars, Variants);
 
     // Cache matching predicates.
     if (MatchedPatterns[i])
       continue;
 
-    const std::vector<Predicate> &Predicates =
-        PatternsToMatch[i].getPredicates();
+    ListInit *Predicates = PatternsToMatch[i].getPredicates();
+    const std::string &HwModeFeatures = PatternsToMatch[i].getHwModeFeatures();
 
     BitVector &Matches = MatchedPredicates[i];
     MatchedPatterns.set(i);
     Matches.set(i);
 
     // Don't test patterns that have already been cached - it won't match.
     for (unsigned p = 0; p != NumOriginalPatterns; ++p)
       if (!MatchedPatterns[p])
-        Matches[p] = (Predicates == PatternsToMatch[p].getPredicates());
+        Matches[p] = (Predicates == PatternsToMatch[p].getPredicates()) &&
+                     (HwModeFeatures == PatternsToMatch[p].getHwModeFeatures());
 
     // Copy this to all the matching patterns.
     for (int p = Matches.find_first(); p != -1; p = Matches.find_next(p))
       if (p != (int)i) {
         MatchedPatterns.set(p);
         MatchedPredicates[p] = Matches;
       }
   }
for (unsigned v = 0, e = Variants.size(); v != e; ++v) {
       // If we already have it, ignore the variant.
       if (AlreadyExists) continue;
 
       // Otherwise, add it to the list of patterns we have.
       PatternsToMatch.emplace_back(
           PatternsToMatch[i].getSrcRecord(), PatternsToMatch[i].getPredicates(),
           Variant, PatternsToMatch[i].getDstPatternShared(),
           PatternsToMatch[i].getDstRegs(),
-          PatternsToMatch[i].getAddedComplexity(), Record::getNewUID());
+          PatternsToMatch[i].getAddedComplexity(), Record::getNewUID(),
+          PatternsToMatch[i].getForceMode(),
+          PatternsToMatch[i].getHwModeFeatures());
       MatchedPredicates.push_back(Matches);
 
       // Add a new match the same as this pattern.
       for (auto &P : MatchedPredicates)
         P.push_back(P[i]);
     }
 
     LLVM_DEBUG(errs() << "\n");
   }
 }

llvm/utils/TableGen/GlobalISelEmitter.cpp

 //===- GlobalISelEmitter.cpp - Generate an instruction selector -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
private:
   void gatherOpcodeValues();
   void gatherTypeIDValues();
   void gatherNodeEquivs();
 
   Record *findNodeEquiv(Record *N) const;
   const CodeGenInstruction *getEquivNode(Record &Equiv,
                                          const TreePatternNode *N) const;
 
-  Error importRulePredicates(RuleMatcher &M, ArrayRef<Predicate> Predicates);
+  Error importRulePredicates(RuleMatcher &M, ArrayRef<Record *> Predicates);
   Expected<InstructionMatcher &>
   createAndImportSelDAGMatcher(RuleMatcher &Rule,
                                InstructionMatcher &InsnMatcher,
                                const TreePatternNode *Src, unsigned &TempOpIdx);
   Error importComplexPatternOperandMatcher(OperandMatcher &OM, Record *R,
                                            unsigned &TempOpIdx) const;
   Error importChildMatcher(RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
                            const TreePatternNode *SrcChild,

 }
 
 GlobalISelEmitter::GlobalISelEmitter(RecordKeeper &RK)
     : RK(RK), CGP(RK), Target(CGP.getTargetInfo()),
       CGRegs(Target.getRegBank()) {}
 
 //===- Emitter ------------------------------------------------------------===//
 
-Error
-GlobalISelEmitter::importRulePredicates(RuleMatcher &M,
-                                        ArrayRef<Predicate> Predicates) {
-  for (const Predicate &P : Predicates) {
-    if (!P.Def || P.getCondString().empty())
+Error GlobalISelEmitter::importRulePredicates(RuleMatcher &M,
+                                              ArrayRef<Record *> Predicates) {
+  for (Record *Pred : Predicates) {
+    if (Pred->getValueAsString("CondString").empty())
       continue;
-    declareSubtargetFeature(P.Def);
-    M.addRequiredFeature(P.Def);
+    declareSubtargetFeature(Pred);
+    M.addRequiredFeature(Pred);
   }
 
   return Error::success();
 }
 
 Expected<InstructionMatcher &> GlobalISelEmitter::addBuiltinPredicates(
     const Record *SrcGIEquivOrNull, const TreePredicateFn &Predicate,
     InstructionMatcher &InsnMatcher, bool &HasAddedMatcher) {
Expected<RuleMatcher> GlobalISelEmitter::runOnPattern(const PatternToMatch &P) {
   // Keep track of the matchers and actions to emit.
   int Score = P.getPatternComplexity(CGP);
   RuleMatcher M(P.getSrcRecord()->getLoc());
   RuleMatcherScores[M.getRuleID()] = Score;
   M.addAction<DebugCommentAction>(llvm::to_string(*P.getSrcPattern()) +
                                   "  =>  " +
                                   llvm::to_string(*P.getDstPattern()));
 
-  if (auto Error = importRulePredicates(M, P.getPredicates()))
+  if (auto Error = importRulePredicates(M, P.getPredicateRecords()))
     return std::move(Error);
 
   // Next, analyze the pattern operators.
   TreePatternNode *Src = P.getSrcPattern();
   TreePatternNode *Dst = P.getDstPattern();
 
   // If the root of either pattern isn't a simple operator, ignore it.
   if (auto Err = isTrivialOperatorNode(Dst))