[SelectionDAG] Extend immAll(Ones|Zeros)V to handle ISD::SPLAT_VECTOR

[frasercrmck]

Summary

The TableGen immAllOnesV and immAllZerosV helpers implicitly wrapped the
ISD::isBuildVectorAll(Ones|Zeros) helper functions. This was inhibiting
their use for targets such as RISC-V which use ISD::SPLAT_VECTOR. In
particular, RISC-V had to define its own 'vnot' fragment.

In order to extend the scope of these nodes to include support for
ISD::SPLAT_VECTOR, two new ISD predicate functions have been introduced:
ISD::isConstantSplatVectorAll(Ones|Zeros). These effectively supersede
the older "isBuildVector" predicates, which are now simple wrappers for
the new functions. They pass a defaulted boolean toggle which preserves
the old behaviour. It is hoped that in time all call-sites can be ported
to the "isConstantSplatVector" functions.

While the use of ISD::isBuildVectorAll(Ones|Zeros) has not changed, the
behaviour of the TableGen immAll(Ones|Zeros)V has. To test the new
functionality, the custom RISC-V TableGen fragment has been removed and
replaced with the built-in 'vnot'. To test their use as pattern-roots, two
splat patterns have been updated accordingly.

[lebedev.ri]

This seems to be missing test coverage.

[frasercrmck]

• update documentation

[frasercrmck]

In D94223#2484023, @lebedev.ri wrote:

This seems to be missing test coverage.

The functionality is tested in part by replacing riscv_m_vnot with vnot and having the patterns still match. I've just identified that there are two further RVV patterns which can potentially be updated to use immAllOnesV and immAllZeroV and will try that out accordingly.

But I would gladly add more testing (for more targets) if someone can help me find a way in.

[frasercrmck]

Ah. I've encountered a hiccup when fixing this up so SPLAT_VECTOR can be used as a root of immAllOnesV and immAllZerosV, e.g.

def : Pat<(mti.Mask immAllOnesV),
          (!cast<Instruction>("PseudoVMSET_M_"#mti.BX) VLMax, mti.SEW)>;

@craig.topper, do you know if it's possible to extend the following code to allow SPLAT_VECTOR? My naive attempts are telling me that you can't just add two independent CheckOpcodeMatchers. I couldn't get SwitchOpcodeMatcher or ScopeMatcher (which seemed to optimize to SwitchOpcodeMatcher) to work. I'm suspecting that it can't place the emission code in the right place when I do that. We don't need a new kind of OPC, do we?

// If this is the root of the dag we're matching, we emit a redundant opcode
// check to ensure that this gets folded into the normal top-level
// OpcodeSwitch.
if (N == Pattern.getSrcPattern()) {
  const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed("build_vector"));
  AddMatcher(new CheckOpcodeMatcher(NI));
}
return AddMatcher(new CheckImmAllOnesVMatcher());

[craig.topper]

In D94223#2484326, @frasercrmck wrote:

Ah. I've encountered a hiccup when fixing this up so SPLAT_VECTOR can be used as a root of immAllOnesV and immAllZerosV, e.g.

def : Pat<(mti.Mask immAllOnesV),
          (!cast<Instruction>("PseudoVMSET_M_"#mti.BX) VLMax, mti.SEW)>;

@craig.topper, do you know if it's possible to extend the following code to allow SPLAT_VECTOR? My naive attempts are telling me that you can't just add two independent CheckOpcodeMatchers. I couldn't get SwitchOpcodeMatcher or ScopeMatcher (which seemed to optimize to SwitchOpcodeMatcher) to work. I'm suspecting that it can't place the emission code in the right place when I do that. We don't need a new kind of OPC, do we?

// If this is the root of the dag we're matching, we emit a redundant opcode
// check to ensure that this gets folded into the normal top-level
// OpcodeSwitch.
if (N == Pattern.getSrcPattern()) {
  const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed("build_vector"));
  AddMatcher(new CheckOpcodeMatcher(NI));
}
return AddMatcher(new CheckImmAllOnesVMatcher());

I think AddMatcher has some hidden state that keeps track of the last node added to build a linked list. You would need to put two entries into the PatternMatchers vector in DAGISelEmitter::run. I think we might be able to special case immAllZerosV/immAllOnesV at the top of MatcherGen::EmitMatcherCode where complex pattern is handled. We can emit a build_vector CheckOpcodeMatcher for variant 0 for build_vector and splat_vector for variant 1. And return true for variant >= 2. Then we remove the special case for "if (N == Pattern.getSrcPattern()) {" where these are handled in EmitLeafMatchCode.

[craig.topper]

In D94223#2485149, @craig.topper wrote:

In D94223#2484326, @frasercrmck wrote:

Ah. I've encountered a hiccup when fixing this up so SPLAT_VECTOR can be used as a root of immAllOnesV and immAllZerosV, e.g.

def : Pat<(mti.Mask immAllOnesV),
          (!cast<Instruction>("PseudoVMSET_M_"#mti.BX) VLMax, mti.SEW)>;

@craig.topper, do you know if it's possible to extend the following code to allow SPLAT_VECTOR? My naive attempts are telling me that you can't just add two independent CheckOpcodeMatchers. I couldn't get SwitchOpcodeMatcher or ScopeMatcher (which seemed to optimize to SwitchOpcodeMatcher) to work. I'm suspecting that it can't place the emission code in the right place when I do that. We don't need a new kind of OPC, do we?

// If this is the root of the dag we're matching, we emit a redundant opcode
// check to ensure that this gets folded into the normal top-level
// OpcodeSwitch.
if (N == Pattern.getSrcPattern()) {
  const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed("build_vector"));
  AddMatcher(new CheckOpcodeMatcher(NI));
}
return AddMatcher(new CheckImmAllOnesVMatcher());

I think AddMatcher has some hidden state that keeps track of the last node added to build a linked list. You would need to put two entries into the PatternMatchers vector in DAGISelEmitter::run. I think we might be able to special case immAllZerosV/immAllOnesV at the top of MatcherGen::EmitMatcherCode where complex pattern is handled. We can emit a build_vector CheckOpcodeMatcher for variant 0 for build_vector and splat_vector for variant 1. And return true for variant >= 2. Then we remove the special case for "if (N == Pattern.getSrcPattern()) {" where these are handled in EmitLeafMatchCode.

I tried this out and got it to work, but it does add redundant table entries to other targets that only support build_vector for their all zeros/ones. It was about 635 extra bytes on X86. For a given target and VT it should always be build_vector or splat_vector, not both. So I'm not sure if we should add the unneeded entries or just document that it doesn't work for splat_vector at the root node and keep the explicit splat_vector match on RISCV. If we keep the splat_vector explicit, we should change it to only look at bit 0 of the immediate.

[craig.topper]

New idea, just pick opcode based on the node value type being a scalable vector.

diff --git a/llvm/utils/TableGen/DAGISelMatcherGen.cpp b/llvm/utils/TableGen/DAGISelMatcherGen.cpp
index 792bf17..ad99304 100644
--- a/llvm/utils/TableGen/DAGISelMatcherGen.cpp
+++ b/llvm/utils/TableGen/DAGISelMatcherGen.cpp
@@ -282,7 +282,9 @@ void MatcherGen::EmitLeafMatchCode(const TreePatternNode *N) {
     // check to ensure that this gets folded into the normal top-level
     // OpcodeSwitch.
     if (N == Pattern.getSrcPattern()) {
-      const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed("build_vector"));
+      MVT VT = N->getSimpleType(0);
+      std::string Name = VT.isScalableVector() ? "splat_vector" : "build_vector";
+      const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed(Name));
       AddMatcher(new CheckOpcodeMatcher(NI));
     }
     return AddMatcher(new CheckImmAllOnesVMatcher());
@@ -292,7 +294,9 @@ void MatcherGen::EmitLeafMatchCode(const TreePatternNode *N) {
     // check to ensure that this gets folded into the normal top-level
     // OpcodeSwitch.
     if (N == Pattern.getSrcPattern()) {
-      const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed("build_vector"));
+      MVT VT = N->getSimpleType(0);
+      std::string Name = VT.isScalableVector() ? "splat_vector" : "build_vector";
+      const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed(Name));
       AddMatcher(new CheckOpcodeMatcher(NI));
     }
     return AddMatcher(new CheckImmAllZerosVMatcher());

Note, I had to use std::string because getSDNodeNamed takes a const std::string &. I'll investigate changing it to StringRef. We only ever call it with string literals so it's dumb to force a std::string temporary.

[frasercrmck]

In D94223#2485348, @craig.topper wrote:

New idea, just pick opcode based on the node value type being a scalable vector.

Note, I had to use std::string because getSDNodeNamed takes a const std::string &. I'll investigate changing it to StringRef. We only ever call it with string literals so it's dumb to force a std::string temporary.

Oh, neat! That hadn't occurred to me; I wasn't in the TableGen headspace. I was thinking last night that it's not ideal to add "redundant" SPLAT_VECTOR code to other targets' tables, so I like that your idea avoids that. I would prefer if it could act identically on both opcodes (and it would make the RVV fixed-length implementation simpler if we could assume splats everywhere) but I think the fixed-length<->BUILD_VECTOR relationship goes quite deep for obvious reasons so it's probably not achievable.

I'll add your idea to this patch if that's alright.

[frasercrmck]

• fix immAll(Ones|Zeros)V when used as root
• use that in RVV patterns

[craig.topper]

LGTM

llvm/utils/TableGen/DAGISelMatcherGen.cpp

286

This can be a StringRef now. I fixed getSDNodeNamed yesterday.

299

Same here

[frasercrmck]

rebase on main
change std::string to StringRef

llvm/include/llvm/CodeGen/SelectionDAGNodes.h

struct SDVTList {
   const EVT *VTs;
   unsigned int NumVTs;
 };
 
 namespace ISD {
 
   /// Node predicates
 
-  /// If N is a BUILD_VECTOR node whose elements are all the same constant or
-  /// undefined, return true and return the constant value in \p SplatValue.
-  bool isConstantSplatVector(const SDNode *N, APInt &SplatValue);
+/// If N is a BUILD_VECTOR or SPLAT_VECTOR node whose elements are all the
+/// same constant or undefined, return true and return the constant value in
+/// \p SplatValue.
 
-  /// Return true if the specified node is a BUILD_VECTOR where all of the
-  /// elements are ~0 or undef.
-  bool isBuildVectorAllOnes(const SDNode *N);
+/// Return true if the specified node is a BUILD_VECTOR or SPLAT_VECTOR where
+/// all of the elements are ~0 or undef. If \p BuildVectorOnly is set to
+/// true, it only checks BUILD_VECTOR.
+bool isConstantSplatVectorAllOnes(const SDNode *N,
+                                  bool BuildVectorOnly = false);
+
+/// Return true if the specified node is a BUILD_VECTOR or SPLAT_VECTOR where
+/// all of the elements are 0 or undef. If \p BuildVectorOnly is set to true, it
+/// only checks BUILD_VECTOR.
+bool isConstantSplatVectorAllZeros(const SDNode *N,
+                                   bool BuildVectorOnly = false);
+
 
-  /// Return true if the specified node is a BUILD_VECTOR where all of the
-  /// elements are 0 or undef.
-  bool isBuildVectorAllZeros(const SDNode *N);
 
-  /// Return true if the specified node is a BUILD_VECTOR node of all
-  /// ConstantSDNode or undef.
-  bool isBuildVectorOfConstantSDNodes(const SDNode *N);
 
-  /// Return true if the specified node is a BUILD_VECTOR node of all
-  /// ConstantFPSDNode or undef.
-  bool isBuildVectorOfConstantFPSDNodes(const SDNode *N);
 
-  /// Return true if the node has at least one operand and all operands of the
-  /// specified node are ISD::UNDEF.
-  bool allOperandsUndef(const SDNode *N);
 
 } // end namespace ISD
 
 //===----------------------------------------------------------------------===//
 /// Unlike LLVM values, Selection DAG nodes may return multiple
 /// values as the result of a computation.  Many nodes return multiple values,
 /// from loads (which define a token and a return value) to ADDC (which returns
 /// a result and a carry value), to calls (which may return an arbitrary number

llvm/include/llvm/Target/TargetSelectionDAG.td

class FPImmLeaf<ValueType vt, code pred, SDNodeXForm xform = NOOP_SDNodeXForm>
   let FastIselShouldIgnore = true;
 }
 
 // Leaf fragments.
 
 def vtInt      : PatLeaf<(vt),  [{ return N->getVT().isInteger(); }]>;
 def vtFP       : PatLeaf<(vt),  [{ return N->getVT().isFloatingPoint(); }]>;
 
-// Use ISD::isBuildVectorAllOnes or ISD::isBuildVectorAllZeros to look for
-// the corresponding build_vector. Will look through bitcasts except when used
-// as a pattern root.
-def immAllOnesV; // ISD::isBuildVectorAllOnes
-def immAllZerosV; // ISD::isBuildVectorAllZeros
+// Use ISD::isConstantSplatVectorAllOnes or ISD::isConstantSplatVectorAllZeros
+// to look for the corresponding build_vector or splat_vector. Will look through
+// bitcasts and check for either opcode, except when used as a pattern root.
+// When used as a pattern root, only fixed-length build_vector and scalable
+// splat_vector are supported.
+def immAllOnesV; // ISD::isConstantSplatVectorAllOnes
+def immAllZerosV; // ISD::isConstantSplatVectorAllZeros
 
 // Other helper fragments.
 def not  : PatFrag<(ops node:$in), (xor node:$in, -1)>;
 def vnot : PatFrag<(ops node:$in), (xor node:$in, immAllOnesV)>;
 def ineg : PatFrag<(ops node:$in), (sub 0, node:$in)>;
 
 def zanyext : PatFrags<(ops node:$op),
                        [(zext node:$op),

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

bool ISD::isConstantSplatVector(const SDNode *N, APInt &SplatVal) {
   return BV->isConstantSplat(SplatVal, SplatUndef, SplatBitSize, HasUndefs,
                              EltSize) &&
          EltSize == SplatBitSize;
 }
 
 // FIXME: AllOnes and AllZeros duplicate a lot of code. Could these be
 // specializations of the more general isConstantSplatVector()?
 
-bool ISD::isBuildVectorAllOnes(const SDNode *N) {
+bool ISD::isConstantSplatVectorAllOnes(const SDNode *N, bool BuildVectorOnly) {
   // Look through a bit convert.
   while (N->getOpcode() == ISD::BITCAST)
     N = N->getOperand(0).getNode();
 
+  if (!BuildVectorOnly && N->getOpcode() == ISD::SPLAT_VECTOR) {
+    APInt SplatVal;
+    return isConstantSplatVector(N, SplatVal) && SplatVal.isAllOnesValue();
+  }
+
   if (N->getOpcode() != ISD::BUILD_VECTOR) return false;
 
   unsigned i = 0, e = N->getNumOperands();
 
   // Skip over all of the undef values.
   while (i != e && N->getOperand(i).isUndef())
     ++i;
 
bool ISD::isConstantSplatVectorAllOnes(const SDNode *N, bool BuildVectorOnly) {
   // undefs. Even with the above element type twiddling, this should be OK, as
   // the same type legalization should have applied to all the elements.
   for (++i; i != e; ++i)
     if (N->getOperand(i) != NotZero && !N->getOperand(i).isUndef())
       return false;
   return true;
 }
 
-bool ISD::isBuildVectorAllZeros(const SDNode *N) {
+bool ISD::isConstantSplatVectorAllZeros(const SDNode *N, bool BuildVectorOnly) {
   // Look through a bit convert.
   while (N->getOpcode() == ISD::BITCAST)
     N = N->getOperand(0).getNode();
 
+  if (!BuildVectorOnly && N->getOpcode() == ISD::SPLAT_VECTOR) {
+    APInt SplatVal;
+    return isConstantSplatVector(N, SplatVal) && SplatVal.isNullValue();
+  }
+
   if (N->getOpcode() != ISD::BUILD_VECTOR) return false;
 
   bool IsAllUndef = true;
   for (const SDValue &Op : N->op_values()) {
     if (Op.isUndef())
       continue;
     IsAllUndef = false;
     // Do not accept build_vectors that aren't all constants or which have non-0
bool ISD::isConstantSplatVectorAllZeros(const SDNode *N, bool BuildVectorOnly) {
   }
 
   // Do not accept an all-undef vector.
   if (IsAllUndef)
     return false;
   return true;
 }
 
+bool ISD::isBuildVectorAllOnes(const SDNode *N) {
+  return isConstantSplatVectorAllOnes(N, /*BuildVectorOnly*/ true);
+}
+
+bool ISD::isBuildVectorAllZeros(const SDNode *N) {
+  return isConstantSplatVectorAllZeros(N, /*BuildVectorOnly*/ true);
+}
+
 bool ISD::isBuildVectorOfConstantSDNodes(const SDNode *N) {
   if (N->getOpcode() != ISD::BUILD_VECTOR)
     return false;
 
   for (const SDValue &Op : N->op_values()) {
     if (Op.isUndef())
       continue;
     if (!isa<ConstantSDNode>(Op))

llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

case OPC_CheckChild4Integer:
       continue;
     case OPC_CheckAndImm:
       if (!::CheckAndImm(MatcherTable, MatcherIndex, N, *this)) break;
       continue;
     case OPC_CheckOrImm:
       if (!::CheckOrImm(MatcherTable, MatcherIndex, N, *this)) break;
       continue;
     case OPC_CheckImmAllOnesV:
-      if (!ISD::isBuildVectorAllOnes(N.getNode())) break;
+      if (!ISD::isConstantSplatVectorAllOnes(N.getNode()))
+        break;
       continue;
     case OPC_CheckImmAllZerosV:
-      if (!ISD::isBuildVectorAllZeros(N.getNode())) break;
+      if (!ISD::isConstantSplatVectorAllZeros(N.getNode()))
+        break;
       continue;
 
     case OPC_CheckFoldableChainNode: {
       assert(NodeStack.size() != 1 && "No parent node");
       // Verify that all intermediate nodes between the root and this one have
       // a single use (ignoring chains, which are handled in UpdateChains).
       bool HasMultipleUses = false;
       for (unsigned i = 1, e = NodeStack.size()-1; i != e; ++i) {

llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td

 
 // Penalize the generic form with Complexity=1 to give the simm5/uimm5 variants
 // precedence
 def SplatPat       : ComplexPattern<vAny, 1, "selectVSplat", [], [], 1>;
 
 def SplatPat_simm5 : ComplexPattern<vAny, 1, "selectVSplatSimm5", []>;
 def SplatPat_uimm5 : ComplexPattern<vAny, 1, "selectVSplatUimm5", []>;
 
-// A mask-vector version of the standard 'vnot' fragment but using splat_vector
-// rather than (the implicit) build_vector
-def riscv_m_vnot : PatFrag<(ops node:$in),
-                           (xor node:$in, (splat_vector (XLenVT 1)))>;
-
 multiclass VPatUSLoadStoreSDNode<LLVMType type,
                                  LLVMType mask_type,
                                  int sew,
                                  LMULInfo vlmul,
                                  RegisterClass reg_rs1,
                                  VReg reg_class>
 {
   defvar load_instr = !cast<Instruction>("PseudoVLE"#sew#"_V_"#vlmul.MX);
def : Pat<(mti.Mask (and VR:$rs1, VR:$rs2)),
                  VR:$rs1, VR:$rs2, VLMax, mti.SEW)>;
   def : Pat<(mti.Mask (or VR:$rs1, VR:$rs2)),
             (!cast<Instruction>("PseudoVMOR_MM_"#mti.LMul.MX)
                  VR:$rs1, VR:$rs2, VLMax, mti.SEW)>;
   def : Pat<(mti.Mask (xor VR:$rs1, VR:$rs2)),
             (!cast<Instruction>("PseudoVMXOR_MM_"#mti.LMul.MX)
                  VR:$rs1, VR:$rs2, VLMax, mti.SEW)>;
 
-  def : Pat<(mti.Mask (riscv_m_vnot (and VR:$rs1, VR:$rs2))),
+  def : Pat<(mti.Mask (vnot (and VR:$rs1, VR:$rs2))),
             (!cast<Instruction>("PseudoVMNAND_MM_"#mti.LMul.MX)
                  VR:$rs1, VR:$rs2, VLMax, mti.SEW)>;
-  def : Pat<(mti.Mask (riscv_m_vnot (or VR:$rs1, VR:$rs2))),
+  def : Pat<(mti.Mask (vnot (or VR:$rs1, VR:$rs2))),
             (!cast<Instruction>("PseudoVMNOR_MM_"#mti.LMul.MX)
                  VR:$rs1, VR:$rs2, VLMax, mti.SEW)>;
-  def : Pat<(mti.Mask (riscv_m_vnot (xor VR:$rs1, VR:$rs2))),
+  def : Pat<(mti.Mask (vnot (xor VR:$rs1, VR:$rs2))),
             (!cast<Instruction>("PseudoVMXNOR_MM_"#mti.LMul.MX)
                  VR:$rs1, VR:$rs2, VLMax, mti.SEW)>;
 
-  def : Pat<(mti.Mask (and VR:$rs1, (riscv_m_vnot VR:$rs2))),
+  def : Pat<(mti.Mask (and VR:$rs1, (vnot VR:$rs2))),
             (!cast<Instruction>("PseudoVMANDNOT_MM_"#mti.LMul.MX)
                  VR:$rs1, VR:$rs2, VLMax, mti.SEW)>;
-  def : Pat<(mti.Mask (or VR:$rs1, (riscv_m_vnot VR:$rs2))),
+  def : Pat<(mti.Mask (or VR:$rs1, (vnot VR:$rs2))),
             (!cast<Instruction>("PseudoVMORNOT_MM_"#mti.LMul.MX)
                  VR:$rs1, VR:$rs2, VLMax, mti.SEW)>;
 }
 
 } // Predicates = [HasStdExtV]
 
 //===----------------------------------------------------------------------===//
 // Vector Splats
 //===----------------------------------------------------------------------===//
 
 let Predicates = [HasStdExtV] in {
 foreach vti = AllIntegerVectors in {
   def : Pat<(vti.Vector (splat_vector GPR:$rs1)),
             (!cast<Instruction>("PseudoVMV_V_X_" # vti.LMul.MX)
               GPR:$rs1, VLMax, vti.SEW)>;
   def : Pat<(vti.Vector (splat_vector simm5:$rs1)),
             (!cast<Instruction>("PseudoVMV_V_I_" # vti.LMul.MX)
               simm5:$rs1, VLMax, vti.SEW)>;
 }
 
 foreach mti = AllMasks in {
-  def : Pat<(mti.Mask (splat_vector (XLenVT 1))),
+  def : Pat<(mti.Mask immAllOnesV),
             (!cast<Instruction>("PseudoVMSET_M_"#mti.BX) VLMax, mti.SEW)>;
-  def : Pat<(mti.Mask (splat_vector (XLenVT 0))),
+  def : Pat<(mti.Mask immAllZerosV),
             (!cast<Instruction>("PseudoVMCLR_M_"#mti.BX) VLMax, mti.SEW)>;
 }
 } // Predicates = [HasStdExtV]
 
 let Predicates = [HasStdExtV, IsRV32] in {
 foreach vti = AllIntegerVectors in {
   if !eq(vti.SEW, 64) then {
     def : Pat<(vti.Vector (rv32_splat_i64 GPR:$rs1)),
               (!cast<Instruction>("PseudoVMV_V_X_" # vti.LMul.MX)
                 GPR:$rs1, VLMax, vti.SEW)>;
     def : Pat<(vti.Vector (rv32_splat_i64 simm5:$rs1)),
               (!cast<Instruction>("PseudoVMV_V_I_" # vti.LMul.MX)
                 simm5:$rs1, VLMax, vti.SEW)>;
   }
 }
 } // Predicates = [HasStdExtV, IsRV32]

llvm/utils/TableGen/DAGISelMatcher.h

 
 private:
   void printImpl(raw_ostream &OS, unsigned indent) const override;
   bool isEqualImpl(const Matcher *M) const override {
     return cast<CheckOrImmMatcher>(M)->Value == Value;
   }
 };
 
-/// CheckImmAllOnesVMatcher - This check if the current node is an build vector
-/// of all ones.
+/// CheckImmAllOnesVMatcher - This checks if the current node is a build_vector
+/// or splat_vector of all ones.
 class CheckImmAllOnesVMatcher : public Matcher {
 public:
   CheckImmAllOnesVMatcher() : Matcher(CheckImmAllOnesV) {}
 
   static bool classof(const Matcher *N) {
     return N->getKind() == CheckImmAllOnesV;
   }
 
 private:
   void printImpl(raw_ostream &OS, unsigned indent) const override;
   bool isEqualImpl(const Matcher *M) const override { return true; }
   bool isContradictoryImpl(const Matcher *M) const override;
 };
 
-/// CheckImmAllZerosVMatcher - This check if the current node is an build vector
-/// of all zeros.
+/// CheckImmAllZerosVMatcher - This checks if the current node is a
+/// build_vector or splat_vector of all zeros.
 class CheckImmAllZerosVMatcher : public Matcher {
 public:
   CheckImmAllZerosVMatcher() : Matcher(CheckImmAllZerosV) {}
 
   static bool classof(const Matcher *N) {
     return N->getKind() == CheckImmAllZerosV;
   }
 

llvm/utils/TableGen/DAGISelMatcherGen.cpp

if (LeafRec->isSubClassOf("ComplexPattern")) {
     return;
   }
 
   if (LeafRec->getName() == "immAllOnesV") {
     // If this is the root of the dag we're matching, we emit a redundant opcode
     // check to ensure that this gets folded into the normal top-level
     // OpcodeSwitch.
     if (N == Pattern.getSrcPattern()) {
-      const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed("build_vector"));
+      MVT VT = N->getSimpleType(0);
+      StringRef Name = VT.isScalableVector() ? "splat_vector" : "build_vector";
+      const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed(Name));
       AddMatcher(new CheckOpcodeMatcher(NI));
     }
     return AddMatcher(new CheckImmAllOnesVMatcher());
   }
   if (LeafRec->getName() == "immAllZerosV") {
     // If this is the root of the dag we're matching, we emit a redundant opcode
     // check to ensure that this gets folded into the normal top-level
     // OpcodeSwitch.
     if (N == Pattern.getSrcPattern()) {
-      const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed("build_vector"));
+      MVT VT = N->getSimpleType(0);
+      StringRef Name = VT.isScalableVector() ? "splat_vector" : "build_vector";
+      const SDNodeInfo &NI = CGP.getSDNodeInfo(CGP.getSDNodeNamed(Name));
       AddMatcher(new CheckOpcodeMatcher(NI));
     }
     return AddMatcher(new CheckImmAllZerosVMatcher());
   }
 
   errs() << "Unknown leaf kind: " << *N << "\n";
   abort();
 }