[RISCV] Move SHFLI matching to DAG combine. Add 32-bit support for RV64

[craig.topper]

Summary

We previously used isel patterns for this, but that used quite
a bit of space in the isel table due to OR being associative
and commutative. It also wouldn't handle shifts/ands being in
reversed order.

This generalizes the shift/and matching from GREVI to
take the expected mask table as input so we can reuse it for
SHFLI.

There is no SHFLIW instruction, but we can promote a 32-bit
SHFLI to i64 on RV64. As long as bit 4 of the control bit isn't
set, a 64-bit SHFLI will preserve 33 sign bits if the input had
at least 33 sign bits. ComputeNumSignBits has been updated to
account for that to avoid sext.w in the tests.

[frasercrmck]

I think there are also a few linter issues to fix up here and there.

In general, though, it looks good. I'll take a closer look soon when I find some time.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2690–2691

I wonder if this comment should be updated. It's more or less correct (for grev) but is now too specific.

2723–2725

This bit seems specific to SHFL in a roundabout way. Is there a way to make it more generic, or at least explicit? I know we're only matching GREV or SHFL so we can expect certain preconditions but I worry it'd catch someone out if they wanted to use this method for other purposes.

llvm/test/CodeGen/RISCV/rv32Zbp.ll

2879 ↗

(On Diff #323598)

Are these changes in operand order creating cases that weren't previously optimized? I take it it's not now required for the optimization to work.

[craig.topper]

llvm/test/CodeGen/RISCV/rv32Zbp.ll

2879 ↗

(On Diff #323598)

Tablegen would automatically generate patterns for all permutations of the operands for the two ORs, but we weren't testing it. I changed the tests to make sure my commuting and associativity worked. Each test case should have a different permutation now.

[craig.topper]

Update comments and clang-format.

[craig.topper]

I've pre-committed the test changes where I changed the OR operand order.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2723–2725

I've updated the header to provide a better description. We could add another parameter to make it explicit, but I figured the caller already had to size the array of expected masks correctly.

[frasercrmck]

Thanks for pre-committing the test cases, I think it's cleaner now.

There's one comment I left but apart from that LGTM.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2692

Should 1 be C2 here and in the last line?

[RKSimon]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2721

@craig.topper Shouldn't this always early-out if ShAmt >= Width? Reported by coverity.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2721

You mean should that be an || instead of &&?

[RKSimon]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2721

Yes - it might be a false positive but coverity is warning that *Mask can be shifted by 64 at line 2751.

llvm/lib/Target/RISCV/RISCVISelLowering.h

enum NodeType : unsigned {
   // TableGen chokes when faced with commutative permutations in deeply-nested
   // DAGs. Each node takes an input operand and a TargetConstant immediate
   // shift amount, and outputs a bit-manipulated version of input. All operands
   // are of type XLenVT.
   GREVI,
   GREVIW,
   GORCI,
   GORCIW,
+  SHFLI,
   // Vector Extension
   // VMV_V_X_VL matches the semantics of vmv.v.x but includes an extra operand
   // for the VL value to be used for the operation.
   VMV_V_X_VL,
   // VFMV_V_F_VL matches the semantics of vfmv.v.f but includes an extra operand
   // for the VL value to be used for the operation.
   VFMV_V_F_VL,
   // VMV_X_S matches the semantics of vmv.x.s. The result is always XLenVT sign

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

SDValue NewOp0 =
         DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, N->getOperand(0));
     SDValue NewRes =
         DAG.getNode(WOpcode, DL, MVT::i64, NewOp0, N->getOperand(1));
     // ReplaceNodeResults requires we maintain the same type for the return
     // value.
     Results.push_back(DAG.getNode(ISD::TRUNCATE, DL, MVT::i32, NewRes));
     break;
   }
+  case RISCVISD::SHFLI: {
+    // There is no SHFLIW instruction, but we can just promote the operation.
+    assert(N->getValueType(0) == MVT::i32 && Subtarget.is64Bit() &&
+           "Unexpected custom legalisation");
+    SDLoc DL(N);
+    SDValue NewOp0 =
+        DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, N->getOperand(0));
+    SDValue NewRes =
+        DAG.getNode(RISCVISD::SHFLI, DL, MVT::i64, NewOp0, N->getOperand(1));
+    // ReplaceNodeResults requires we maintain the same type for the return
+    // value.
+    Results.push_back(DAG.getNode(ISD::TRUNCATE, DL, MVT::i32, NewRes));
+    break;
+  }
   case ISD::BSWAP:
   case ISD::BITREVERSE: {
     assert(N->getValueType(0) == MVT::i32 && Subtarget.is64Bit() &&
            Subtarget.hasStdExtZbp() && "Unexpected custom legalisation");
     SDValue NewOp0 = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64,
                                  N->getOperand(0));
     unsigned Imm = N->getOpcode() == ISD::BITREVERSE ? 31 : 24;
     SDValue GREVIW = DAG.getNode(RISCVISD::GREVIW, DL, MVT::i64, NewOp0,
struct RISCVBitmanipPat {
   SDValue Op;
   unsigned ShAmt;
   bool IsSHL;
 
   bool formsPairWith(const RISCVBitmanipPat &Other) const {
     return Op == Other.Op && ShAmt == Other.ShAmt && IsSHL != Other.IsSHL;
   }
 };
 
-// Matches any of the following bit-manipulation patterns:
-//   (and (shl x, 1), (0x55555555 << 1))
-//   (and (srl x, 1), 0x55555555)
-//   (shl (and x, 0x55555555), 1)
-//   (srl (and x, (0x55555555 << 1)), 1)
-// where the shift amount and mask may vary thus:
-//   [1]  = 0x55555555 / 0xAAAAAAAA
-//   [2]  = 0x33333333 / 0xCCCCCCCC
-//   [4]  = 0x0F0F0F0F / 0xF0F0F0F0
-//   [8]  = 0x00FF00FF / 0xFF00FF00
-//   [16] = 0x0000FFFF / 0xFFFFFFFF
-//   [32] = 0x00000000FFFFFFFF / 0xFFFFFFFF00000000 (for RV64)
-static Optional<RISCVBitmanipPat> matchRISCVBitmanipPat(SDValue Op) {
+// Matches patterns of the form
+//   (and (shl x, C2), (C1 << C2))
+//   (and (srl x, C2), C1)
+//   (shl (and x, C1), C2)
+//   (srl (and x, (C1 << C2)), C2)
+// Where C2 is a power of 2 and C1 has at least that many leading zeroes.
+// The expected masks for each shift amount are specified in BitmanipMasks where
+// BitmanipMasks[log2(C2)] specifies the expected C1 value.
+// The max allowed shift amount is either XLen/2 or XLen/4 determined by whether
+// BitmanipMasks contains 6 or 5 entries assuming that the maximum possible
+// XLen is 64.
+static Optional<RISCVBitmanipPat>
+matchRISCVBitmanipPat(SDValue Op, ArrayRef<uint64_t> BitmanipMasks) {
+  assert((BitmanipMasks.size() == 5 || BitmanipMasks.size() == 6) &&
+         "Unexpected number of masks");
   Optional<uint64_t> Mask;
   // Optionally consume a mask around the shift operation.
   if (Op.getOpcode() == ISD::AND && isa<ConstantSDNode>(Op.getOperand(1))) {
     Mask = Op.getConstantOperandVal(1);
     Op = Op.getOperand(0);
   }
   if (Op.getOpcode() != ISD::SHL && Op.getOpcode() != ISD::SRL)
     return None;
   bool IsSHL = Op.getOpcode() == ISD::SHL;
 
   if (!isa<ConstantSDNode>(Op.getOperand(1)))
     return None;
-  auto ShAmt = Op.getConstantOperandVal(1);
+  uint64_t ShAmt = Op.getConstantOperandVal(1);
 
-  if (!isPowerOf2_64(ShAmt))
+  unsigned Width = Op.getValueType() == MVT::i64 ? 64 : 32;
+  if (ShAmt >= Width && !isPowerOf2_64(ShAmt))
     return None;
-
-  // These are the unshifted masks which we use to match bit-manipulation
-  // patterns. They may be shifted left in certain circumstances.
-  static const uint64_t BitmanipMasks[] = {
-      0x5555555555555555ULL, 0x3333333333333333ULL, 0x0F0F0F0F0F0F0F0FULL,
-      0x00FF00FF00FF00FFULL, 0x0000FFFF0000FFFFULL, 0x00000000FFFFFFFFULL,
-  };
-
-  unsigned MaskIdx = Log2_64(ShAmt);
-  if (MaskIdx >= array_lengthof(BitmanipMasks))
+  // If we don't have enough masks for 64 bit, then we must be trying to
+  // match SHFL so we're only allowed to shift 1/4 of the width.
+  if (BitmanipMasks.size() == 5 && ShAmt >= (Width / 2))
     return None;
 
-  auto Src = Op.getOperand(0);
-
-  unsigned Width = Op.getValueType() == MVT::i64 ? 64 : 32;
-  auto ExpMask = BitmanipMasks[MaskIdx] & maskTrailingOnes<uint64_t>(Width);
+  SDValue Src = Op.getOperand(0);
 
   // The expected mask is shifted left when the AND is found around SHL
   // patterns.
   //   ((x >> 1) & 0x55555555)
   //   ((x << 1) & 0xAAAAAAAA)
   bool SHLExpMask = IsSHL;
 
   if (!Mask) {
if (!Mask) {
     } else {
       // Use a default shifted mask of all-ones if there's no AND, truncated
       // down to the expected width. This simplifies the logic later on.
       Mask = maskTrailingOnes<uint64_t>(Width);
       *Mask &= (IsSHL ? *Mask << ShAmt : *Mask >> ShAmt);
     }
   }
 
+  unsigned MaskIdx = Log2_32(ShAmt);
+  uint64_t ExpMask = BitmanipMasks[MaskIdx] & maskTrailingOnes<uint64_t>(Width);
+
   if (SHLExpMask)
     ExpMask <<= ShAmt;
 
   if (Mask != ExpMask)
     return None;
 
   return RISCVBitmanipPat{Src, (unsigned)ShAmt, IsSHL};
 }
 
+// Matches any of the following bit-manipulation patterns:
+//   (and (shl x, 1), (0x55555555 << 1))
+//   (and (srl x, 1), 0x55555555)
+//   (shl (and x, 0x55555555), 1)
+//   (srl (and x, (0x55555555 << 1)), 1)
+// where the shift amount and mask may vary thus:
+//   [1]  = 0x55555555 / 0xAAAAAAAA
+//   [2]  = 0x33333333 / 0xCCCCCCCC
+//   [4]  = 0x0F0F0F0F / 0xF0F0F0F0
+//   [8]  = 0x00FF00FF / 0xFF00FF00
+//   [16] = 0x0000FFFF / 0xFFFFFFFF
+//   [32] = 0x00000000FFFFFFFF / 0xFFFFFFFF00000000 (for RV64)
+static Optional<RISCVBitmanipPat> matchGREVIPat(SDValue Op) {
+  // These are the unshifted masks which we use to match bit-manipulation
+  // patterns. They may be shifted left in certain circumstances.
+  static const uint64_t BitmanipMasks[] = {
+      0x5555555555555555ULL, 0x3333333333333333ULL, 0x0F0F0F0F0F0F0F0FULL,
+      0x00FF00FF00FF00FFULL, 0x0000FFFF0000FFFFULL, 0x00000000FFFFFFFFULL};
+
+  return matchRISCVBitmanipPat(Op, BitmanipMasks);
+}
+
 // Match the following pattern as a GREVI(W) operation
 //   (or (BITMANIP_SHL x), (BITMANIP_SRL x))
 static SDValue combineORToGREV(SDValue Op, SelectionDAG &DAG,
                                const RISCVSubtarget &Subtarget) {
+  assert(Subtarget.hasStdExtZbp() && "Expected Zbp extenson");
   EVT VT = Op.getValueType();
 
   if (VT == Subtarget.getXLenVT() || (Subtarget.is64Bit() && VT == MVT::i32)) {
-    auto LHS = matchRISCVBitmanipPat(Op.getOperand(0));
-    auto RHS = matchRISCVBitmanipPat(Op.getOperand(1));
+    auto LHS = matchGREVIPat(Op.getOperand(0));
+    auto RHS = matchGREVIPat(Op.getOperand(1));
     if (LHS && RHS && LHS->formsPairWith(*RHS)) {
       SDLoc DL(Op);
       return DAG.getNode(
           RISCVISD::GREVI, DL, VT, LHS->Op,
           DAG.getTargetConstant(LHS->ShAmt, DL, Subtarget.getXLenVT()));
     }
   }
   return SDValue();
 }
 
 // Matches any the following pattern as a GORCI(W) operation
 // 1.  (or (GREVI x, shamt), x) if shamt is a power of 2
 // 2.  (or x, (GREVI x, shamt)) if shamt is a power of 2
 // 3.  (or (or (BITMANIP_SHL x), x), (BITMANIP_SRL x))
 // Note that with the variant of 3.,
 //     (or (or (BITMANIP_SHL x), (BITMANIP_SRL x)), x)
 // the inner pattern will first be matched as GREVI and then the outer
 // pattern will be matched to GORC via the first rule above.
 // 4.  (or (rotl/rotr x, bitwidth/2), x)
 static SDValue combineORToGORC(SDValue Op, SelectionDAG &DAG,
                                const RISCVSubtarget &Subtarget) {
+  assert(Subtarget.hasStdExtZbp() && "Expected Zbp extenson");
   EVT VT = Op.getValueType();
 
   if (VT == Subtarget.getXLenVT() || (Subtarget.is64Bit() && VT == MVT::i32)) {
     SDLoc DL(Op);
     SDValue Op0 = Op.getOperand(0);
     SDValue Op1 = Op.getOperand(1);
 
     auto MatchOROfReverse = [&](SDValue Reverse, SDValue X) {
if (VT == Subtarget.getXLenVT() || (Subtarget.is64Bit() && VT == MVT::i32)) {
 
     // OR is commutable so canonicalize its OR operand to the left
     if (Op0.getOpcode() != ISD::OR && Op1.getOpcode() == ISD::OR)
       std::swap(Op0, Op1);
     if (Op0.getOpcode() != ISD::OR)
       return SDValue();
     SDValue OrOp0 = Op0.getOperand(0);
     SDValue OrOp1 = Op0.getOperand(1);
-    auto LHS = matchRISCVBitmanipPat(OrOp0);
+    auto LHS = matchGREVIPat(OrOp0);
     // OR is commutable so swap the operands and try again: x might have been
     // on the left
     if (!LHS) {
       std::swap(OrOp0, OrOp1);
-      LHS = matchRISCVBitmanipPat(OrOp0);
+      LHS = matchGREVIPat(OrOp0);
     }
-    auto RHS = matchRISCVBitmanipPat(Op1);
+    auto RHS = matchGREVIPat(Op1);
     if (LHS && RHS && LHS->formsPairWith(*RHS) && LHS->Op == OrOp1) {
       return DAG.getNode(
           RISCVISD::GORCI, DL, VT, LHS->Op,
           DAG.getTargetConstant(LHS->ShAmt, DL, Subtarget.getXLenVT()));
     }
   }
   return SDValue();
 }
 
+// Matches any of the following bit-manipulation patterns:
+//   (and (shl x, 1), (0x22222222 << 1))
+//   (and (srl x, 1), 0x22222222)
+//   (shl (and x, 0x22222222), 1)
+//   (srl (and x, (0x22222222 << 1)), 1)
+// where the shift amount and mask may vary thus:
+//   [1]  = 0x22222222 / 0x44444444
+//   [2]  = 0x0C0C0C0C / 0x3C3C3C3C
+//   [4]  = 0x00F000F0 / 0x0F000F00
+//   [8]  = 0x0000FF00 / 0x00FF0000
+//   [16] = 0x00000000FFFF0000 / 0x0000FFFF00000000 (for RV64)
+static Optional<RISCVBitmanipPat> matchSHFLPat(SDValue Op) {
+  // These are the unshifted masks which we use to match bit-manipulation
+  // patterns. They may be shifted left in certain circumstances.
+  static const uint64_t BitmanipMasks[] = {
+      0x2222222222222222ULL, 0x0C0C0C0C0C0C0C0CULL, 0x00F000F000F000F0ULL,
+      0x0000FF000000FF00ULL, 0x00000000FFFF0000ULL};
+
+  return matchRISCVBitmanipPat(Op, BitmanipMasks);
+}
+
+// Match (or (or (SHFL_SHL x), (SHFL_SHR x)), (SHFL_AND x)
+static SDValue combineORToSHFL(SDValue Op, SelectionDAG &DAG,
+                               const RISCVSubtarget &Subtarget) {
+  assert(Subtarget.hasStdExtZbp() && "Expected Zbp extenson");
+  EVT VT = Op.getValueType();
+
+  if (VT != MVT::i32 && VT != Subtarget.getXLenVT())
+    return SDValue();
+
+  SDValue Op0 = Op.getOperand(0);
+  SDValue Op1 = Op.getOperand(1);
+
+  // Or is commutable so canonicalize the second OR to the LHS.
+  if (Op0.getOpcode() != ISD::OR)
+    std::swap(Op0, Op1);
+  if (Op0.getOpcode() != ISD::OR)
+    return SDValue();
+
+  // We found an inner OR, so our operands are the operands of the inner OR
+  // and the other operand of the outer OR.
+  SDValue A = Op0.getOperand(0);
+  SDValue B = Op0.getOperand(1);
+  SDValue C = Op1;
+
+  auto Match1 = matchSHFLPat(A);
+  auto Match2 = matchSHFLPat(B);
+
+  // If neither matched, we failed.
+  if (!Match1 && !Match2)
+    return SDValue();
+
+  // We had at least one match. if one failed, try the remaining C operand.
+  if (!Match1) {
+    std::swap(A, C);
+    Match1 = matchSHFLPat(A);
+    if (!Match1)
+      return SDValue();
+  } else if (!Match2) {
+    std::swap(B, C);
+    Match2 = matchSHFLPat(B);
+    if (!Match2)
+      return SDValue();
+  }
+  assert(Match1 && Match2);
+
+  // Make sure our matches pair up.
+  if (!Match1->formsPairWith(*Match2))
+    return SDValue();
+
+  // All the remains is to make sure C is an AND with the same input, that masks
+  // out the bits that are being shuffled.
+  if (C.getOpcode() != ISD::AND || !isa<ConstantSDNode>(C.getOperand(1)) ||
+      C.getOperand(0) != Match1->Op)
+    return SDValue();
+
+  uint64_t Mask = C.getConstantOperandVal(1);
+
+  static const uint64_t BitmanipMasks[] = {
+      0x9999999999999999ULL, 0xC3C3C3C3C3C3C3C3ULL, 0xF00FF00FF00FF00FULL,
+      0xFF0000FFFF0000FFULL, 0xFFFF00000000FFFFULL,
+  };
+
+  unsigned Width = Op.getValueType() == MVT::i64 ? 64 : 32;
+  unsigned MaskIdx = Log2_32(Match1->ShAmt);
+  uint64_t ExpMask = BitmanipMasks[MaskIdx] & maskTrailingOnes<uint64_t>(Width);
+
+  if (Mask != ExpMask)
+    return SDValue();
+
+  SDLoc DL(Op);
+  return DAG.getNode(
+      RISCVISD::SHFLI, DL, VT, Match1->Op,
+      DAG.getTargetConstant(Match1->ShAmt, DL, Subtarget.getXLenVT()));
+}
+
 // Combine (GREVI (GREVI x, C2), C1) -> (GREVI x, C1^C2) when C1^C2 is
 // non-zero, and to x when it is. Any repeated GREVI stage undoes itself.
 // Combine (GORCI (GORCI x, C2), C1) -> (GORCI x, C1|C2). Repeated stage does
 // not undo itself, but they are redundant.
 static SDValue combineGREVI_GORCI(SDNode *N, SelectionDAG &DAG) {
   unsigned ShAmt1 = N->getConstantOperandVal(1);
   SDValue Src = N->getOperand(0);
 
SDValue RISCVTargetLowering::PerformDAGCombine(SDNode *N,
   case RISCVISD::GREVI:
   case RISCVISD::GORCI:
     return combineGREVI_GORCI(N, DCI.DAG);
   case ISD::OR:
     if (auto GREV = combineORToGREV(SDValue(N, 0), DCI.DAG, Subtarget))
       return GREV;
     if (auto GORC = combineORToGORC(SDValue(N, 0), DCI.DAG, Subtarget))
       return GORC;
+    if (auto SHFL = combineORToSHFL(SDValue(N, 0), DCI.DAG, Subtarget))
+      return SHFL;
     break;
   case RISCVISD::SELECT_CC: {
     // Transform
     // (select_cc (xor X, 1), 0, setne, trueV, falseV) ->
     // (select_cc X, 0, seteq, trueV, falseV) if we can prove X is 0/1.
     // This can occur when legalizing some floating point comparisons.
     SDValue LHS = N->getOperand(0);
     SDValue RHS = N->getOperand(1);
unsigned RISCVTargetLowering::ComputeNumSignBitsForTargetNode(
   case RISCVISD::GREVIW:
   case RISCVISD::GORCIW:
   case RISCVISD::FSLW:
   case RISCVISD::FSRW:
     // TODO: As the result is sign-extended, this is conservatively correct. A
     // more precise answer could be calculated for SRAW depending on known
     // bits in the shift amount.
     return 33;
+  case RISCVISD::SHFLI: {
+    // There is no SHFLIW, but a i64 SHFLI with bit 4 of the control word
+    // cleared doesn't affect bit 31. The upper 32 bits will be shuffled, but
+    // will stay within the upper 32 bits. If there were more than 32 sign bits
+    // before there will be at least 33 sign bits after.
+    if (Op.getValueType() == MVT::i64 &&
+        (Op.getConstantOperandVal(1) & 0x10) == 0) {
+      unsigned Tmp = DAG.ComputeNumSignBits(Op.getOperand(0), Depth + 1);
+      if (Tmp > 32)
+        return 33;
+    }
+    break;
+  }
   case RISCVISD::VMV_X_S:
     // The number of sign bits of the scalar result is computed by obtaining the
     // element type of the input vector operand, subtracting its width from the
     // XLEN, and then adding one (sign bit within the element type). If the
     // element type is wider than XLen, the least-significant XLEN bits are
     // taken.
     if (Op.getOperand(0).getScalarValueSizeInBits() > Subtarget.getXLen())
       return 1;
#define NODE_NAME_CASE(NODE)                                                   \
   NODE_NAME_CASE(FMV_X_ANYEXTH)
   NODE_NAME_CASE(FMV_W_X_RV64)
   NODE_NAME_CASE(FMV_X_ANYEXTW_RV64)
   NODE_NAME_CASE(READ_CYCLE_WIDE)
   NODE_NAME_CASE(GREVI)
   NODE_NAME_CASE(GREVIW)
   NODE_NAME_CASE(GORCI)
   NODE_NAME_CASE(GORCIW)
+  NODE_NAME_CASE(SHFLI)
   NODE_NAME_CASE(VMV_V_X_VL)
   NODE_NAME_CASE(VFMV_V_F_VL)
   NODE_NAME_CASE(VMV_X_S)
   NODE_NAME_CASE(SPLAT_VECTOR_I64)
   NODE_NAME_CASE(READ_VLENB)
   NODE_NAME_CASE(TRUNCATE_VECTOR)
   NODE_NAME_CASE(VLEFF)
   NODE_NAME_CASE(VLEFF_MASK)

llvm/lib/Target/RISCV/RISCVInstrInfoB.td

 def : Pat<(rotl GPR:$rs1, uimmlog2xlen:$shamt),
           (RORI GPR:$rs1, (ImmSubFromXLen uimmlog2xlen:$shamt))>;
 }
 
 def riscv_grevi    : SDNode<"RISCVISD::GREVI", SDTIntBinOp, []>;
 def riscv_greviw   : SDNode<"RISCVISD::GREVIW", SDTIntBinOp, []>;
 def riscv_gorci    : SDNode<"RISCVISD::GORCI", SDTIntBinOp, []>;
 def riscv_gorciw   : SDNode<"RISCVISD::GORCIW", SDTIntBinOp, []>;
+def riscv_shfli    : SDNode<"RISCVISD::SHFLI", SDTIntBinOp, []>;
 
 let Predicates = [HasStdExtZbp] in {
+def : Pat<(riscv_shfli GPR:$rs1, timm:$shamt), (SHFLI GPR:$rs1, timm:$shamt)>;
 def : Pat<(riscv_grevi GPR:$rs1, timm:$shamt), (GREVI GPR:$rs1, timm:$shamt)>;
 def : Pat<(riscv_gorci GPR:$rs1, timm:$shamt), (GORCI GPR:$rs1, timm:$shamt)>;
 
 // We treat orc.b as a separate instruction, so match it directly.
 def : Pat<(riscv_gorci GPR:$rs1, 7), (ORCB GPR:$rs1)>;
 } // Predicates = [HasStdExtZbp]
 
 let Predicates = [HasStdExtZbp, IsRV32] in {
def : Pat<(or (and (shl GPR:$rs2, (XLenVT 8)), 0xFF00),
           (PACKH GPR:$rs1, GPR:$rs2)>;
 
 let Predicates = [HasStdExtZbbOrZbp, IsRV32] in
 def : Pat<(i32 (and GPR:$rs, 0xFFFF)), (ZEXTH_RV32 GPR:$rs)>;
 let Predicates = [HasStdExtZbbOrZbp, IsRV64] in {
 def : Pat<(i64 (and GPR:$rs, 0xFFFF)), (ZEXTH_RV64 GPR:$rs)>;
 }
 
-let Predicates = [HasStdExtZbp, IsRV32] in {
-def : Pat<(or (or (and (shl GPR:$rs1, (i32 8)), (i32 0x00FF0000)),
-                  (and GPR:$rs1, (i32 0xFF0000FF))),
-              (and (srl GPR:$rs1, (i32 8)), (i32 0x0000FF00))),
-          (SHFLI GPR:$rs1, (i32 8))>;
-def : Pat<(or (or (and (shl GPR:$rs1, (i32 4)), (i32 0x0F000F00)),
-                  (and GPR:$rs1, (i32 0xF00FF00F))),
-              (and (srl GPR:$rs1, (i32 4)), (i32 0x00F000F0))),
-          (SHFLI GPR:$rs1, (i32 4))>;
-def : Pat<(or (or (and (shl GPR:$rs1, (i32 2)), (i32 0x30303030)),
-                  (and GPR:$rs1, (i32 0xC3C3C3C3))),
-              (and (srl GPR:$rs1, (i32 2)), (i32 0x0C0C0C0C))),
-          (SHFLI GPR:$rs1, (i32 2))>;
-def : Pat<(or (or (and (shl GPR:$rs1, (i32 1)), (i32 0x44444444)),
-                  (and GPR:$rs1, (i32 0x99999999))),
-              (and (srl GPR:$rs1, (i32 1)), (i32 0x22222222))),
-          (SHFLI GPR:$rs1, (i32 1))>;
-} // Predicates = [HasStdExtZbp, IsRV32]
-
-let Predicates = [HasStdExtZbp, IsRV64] in {
-def : Pat<(or (or (and (shl GPR:$rs1, (i64 16)), (i64 0x0000FFFF00000000)),
-                  (and GPR:$rs1, (i64 0xFFFF00000000FFFF))),
-              (and (srl GPR:$rs1, (i64 16)), (i64 0x00000000FFFF0000))),
-          (SHFLI GPR:$rs1, (i64 16))>;
-def : Pat<(or (or (and (shl GPR:$rs1, (i64 8)), (i64 0x00FF000000FF0000)),
-                  (and GPR:$rs1, (i64 0xFF0000FFFF0000FF))),
-              (and (srl GPR:$rs1, (i64 8)), (i64 0x0000FF000000FF00))),
-          (SHFLI GPR:$rs1, (i64 8))>;
-def : Pat<(or (or (and (shl GPR:$rs1, (i64 4)), (i64 0x0F000F000F000F00)),
-                  (and GPR:$rs1, (i64 0xF00FF00FF00FF00F))),
-              (and (srl GPR:$rs1, (i64 4)), (i64 0x00F000F000F000F0))),
-          (SHFLI GPR:$rs1, (i64 4))>;
-def : Pat<(or (or (and (shl GPR:$rs1, (i64 2)), (i64 0x3030303030303030)),
-                  (and GPR:$rs1, (i64 0xC3C3C3C3C3C3C3C3))),
-              (and (srl GPR:$rs1, (i64 2)), (i64 0x0C0C0C0C0C0C0C0C))),
-          (SHFLI GPR:$rs1, (i64 2))>;
-def : Pat<(or (or (and (shl GPR:$rs1, (i64 1)), (i64 0x4444444444444444)),
-                  (and GPR:$rs1, (i64 0x9999999999999999))),
-              (and (srl GPR:$rs1, (i64 1)), (i64 0x2222222222222222))),
-          (SHFLI GPR:$rs1, (i64 1))>;
-} // Predicates = [HasStdExtZbp, IsRV64]
-
 let Predicates = [HasStdExtZba] in {
 def : Pat<(add (shl GPR:$rs1, (XLenVT 1)), GPR:$rs2),
           (SH1ADD GPR:$rs1, GPR:$rs2)>;
 def : Pat<(add (shl GPR:$rs1, (XLenVT 2)), GPR:$rs2),
           (SH2ADD GPR:$rs1, GPR:$rs2)>;
 def : Pat<(add (shl GPR:$rs1, (XLenVT 3)), GPR:$rs2),
           (SH3ADD GPR:$rs1, GPR:$rs2)>;
 } // Predicates = [HasStdExtZba]

llvm/test/CodeGen/RISCV/rv64Zbp.ll

 ; RV64I-NEXT:    lui a2, 139810
 ; RV64I-NEXT:    addiw a2, a2, 546
 ; RV64I-NEXT:    and a0, a0, a2
 ; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: shfl1_i32:
 ; RV64IB:       # %bb.0:
-; RV64IB-NEXT:    lui a1, 629146
-; RV64IB-NEXT:    addiw a1, a1, -1639
-; RV64IB-NEXT:    and a1, a0, a1
-; RV64IB-NEXT:    slli a2, a0, 1
-; RV64IB-NEXT:    lui a3, 279620
-; RV64IB-NEXT:    addiw a3, a3, 1092
-; RV64IB-NEXT:    and a2, a2, a3
-; RV64IB-NEXT:    or a1, a2, a1
-; RV64IB-NEXT:    srli a0, a0, 1
-; RV64IB-NEXT:    lui a2, 139810
-; RV64IB-NEXT:    addiw a2, a2, 546
-; RV64IB-NEXT:    and a0, a0, a2
-; RV64IB-NEXT:    or a0, a1, a0
+; RV64IB-NEXT:    zip.n a0, a0
 ; RV64IB-NEXT:    ret
 ;
 ; RV64IBP-LABEL: shfl1_i32:
 ; RV64IBP:       # %bb.0:
-; RV64IBP-NEXT:    lui a1, 629146
-; RV64IBP-NEXT:    addiw a1, a1, -1639
-; RV64IBP-NEXT:    and a1, a0, a1
-; RV64IBP-NEXT:    slli a2, a0, 1
-; RV64IBP-NEXT:    lui a3, 279620
-; RV64IBP-NEXT:    addiw a3, a3, 1092
-; RV64IBP-NEXT:    and a2, a2, a3
-; RV64IBP-NEXT:    or a1, a2, a1
-; RV64IBP-NEXT:    srli a0, a0, 1
-; RV64IBP-NEXT:    lui a2, 139810
-; RV64IBP-NEXT:    addiw a2, a2, 546
-; RV64IBP-NEXT:    and a0, a0, a2
-; RV64IBP-NEXT:    or a0, a1, a0
+; RV64IBP-NEXT:    zip.n a0, a0
 ; RV64IBP-NEXT:    ret
   %and = and i32 %a, -1717986919
   %shl = shl i32 %a, 1
   %and1 = and i32 %shl, 1145324612
   %or = or i32 %and1, %and
   %shr = lshr i32 %a, 1
   %and2 = and i32 %shr, 572662306
   %or3 = or i32 %or, %and2

 ; RV64I-NEXT:    lui a2, 49345
 ; RV64I-NEXT:    addiw a2, a2, -1012
 ; RV64I-NEXT:    and a0, a0, a2
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: shfl2_i32:
 ; RV64IB:       # %bb.0:
-; RV64IB-NEXT:    lui a1, 801852
-; RV64IB-NEXT:    addiw a1, a1, 963
-; RV64IB-NEXT:    and a1, a0, a1
-; RV64IB-NEXT:    slli a2, a0, 2
-; RV64IB-NEXT:    lui a3, 197379
-; RV64IB-NEXT:    addiw a3, a3, 48
-; RV64IB-NEXT:    and a2, a2, a3
-; RV64IB-NEXT:    or a1, a2, a1
-; RV64IB-NEXT:    srli a0, a0, 2
-; RV64IB-NEXT:    lui a2, 49345
-; RV64IB-NEXT:    addiw a2, a2, -1012
-; RV64IB-NEXT:    and a0, a0, a2
-; RV64IB-NEXT:    or a0, a0, a1
+; RV64IB-NEXT:    zip2.b a0, a0
 ; RV64IB-NEXT:    ret
 ;
 ; RV64IBP-LABEL: shfl2_i32:
 ; RV64IBP:       # %bb.0:
-; RV64IBP-NEXT:    lui a1, 801852
-; RV64IBP-NEXT:    addiw a1, a1, 963
-; RV64IBP-NEXT:    and a1, a0, a1
-; RV64IBP-NEXT:    slli a2, a0, 2
-; RV64IBP-NEXT:    lui a3, 197379
-; RV64IBP-NEXT:    addiw a3, a3, 48
-; RV64IBP-NEXT:    and a2, a2, a3
-; RV64IBP-NEXT:    or a1, a2, a1
-; RV64IBP-NEXT:    srli a0, a0, 2
-; RV64IBP-NEXT:    lui a2, 49345
-; RV64IBP-NEXT:    addiw a2, a2, -1012
-; RV64IBP-NEXT:    and a0, a0, a2
-; RV64IBP-NEXT:    or a0, a0, a1
+; RV64IBP-NEXT:    zip2.b a0, a0
 ; RV64IBP-NEXT:    ret
   %and = and i32 %a, -1010580541
   %shl = shl i32 %a, 2
   %and1 = and i32 %shl, 808464432
   %or = or i32 %and1, %and
   %shr = lshr i32 %a, 2
   %and2 = and i32 %shr, 202116108
   %or3 = or i32 %and2, %or

 ; RV64I-NEXT:    addiw a3, a3, 240
 ; RV64I-NEXT:    and a0, a0, a3
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    or a0, a0, a2
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: shfl4_i32:
 ; RV64IB:       # %bb.0:
-; RV64IB-NEXT:    lui a1, 983295
-; RV64IB-NEXT:    addiw a1, a1, 15
-; RV64IB-NEXT:    and a1, a0, a1
-; RV64IB-NEXT:    slli a2, a0, 4
-; RV64IB-NEXT:    lui a3, 61441
-; RV64IB-NEXT:    addiw a3, a3, -256
-; RV64IB-NEXT:    and a2, a2, a3
-; RV64IB-NEXT:    srli a0, a0, 4
-; RV64IB-NEXT:    lui a3, 3840
-; RV64IB-NEXT:    addiw a3, a3, 240
-; RV64IB-NEXT:    and a0, a0, a3
-; RV64IB-NEXT:    or a0, a0, a1
-; RV64IB-NEXT:    or a0, a0, a2
+; RV64IB-NEXT:    zip4.h a0, a0
 ; RV64IB-NEXT:    ret
 ;
 ; RV64IBP-LABEL: shfl4_i32:
 ; RV64IBP:       # %bb.0:
-; RV64IBP-NEXT:    lui a1, 983295
-; RV64IBP-NEXT:    addiw a1, a1, 15
-; RV64IBP-NEXT:    and a1, a0, a1
-; RV64IBP-NEXT:    slli a2, a0, 4
-; RV64IBP-NEXT:    lui a3, 61441
-; RV64IBP-NEXT:    addiw a3, a3, -256
-; RV64IBP-NEXT:    and a2, a2, a3
-; RV64IBP-NEXT:    srli a0, a0, 4
-; RV64IBP-NEXT:    lui a3, 3840
-; RV64IBP-NEXT:    addiw a3, a3, 240
-; RV64IBP-NEXT:    and a0, a0, a3
-; RV64IBP-NEXT:    or a0, a0, a1
-; RV64IBP-NEXT:    or a0, a0, a2
+; RV64IBP-NEXT:    zip4.h a0, a0
 ; RV64IBP-NEXT:    ret
   %and = and i32 %a, -267390961
   %shl = shl i32 %a, 4
   %and1 = and i32 %shl, 251662080
   %shr = lshr i32 %a, 4
   %and2 = and i32 %shr, 15728880
   %or = or i32 %and2, %and
   %or3 = or i32 %or, %and1

 ; RV64I-NEXT:    addiw a3, a3, -256
 ; RV64I-NEXT:    and a0, a0, a3
 ; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    or a0, a0, a2
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: shfl8_i32:
 ; RV64IB:       # %bb.0:
-; RV64IB-NEXT:    lui a1, 1044480
-; RV64IB-NEXT:    addiw a1, a1, 255
-; RV64IB-NEXT:    and a1, a0, a1
-; RV64IB-NEXT:    slli a2, a0, 8
-; RV64IB-NEXT:    lui a3, 4080
-; RV64IB-NEXT:    and a2, a2, a3
-; RV64IB-NEXT:    srli a0, a0, 8
-; RV64IB-NEXT:    lui a3, 16
-; RV64IB-NEXT:    addiw a3, a3, -256
-; RV64IB-NEXT:    and a0, a0, a3
-; RV64IB-NEXT:    or a0, a1, a0
-; RV64IB-NEXT:    or a0, a0, a2
+; RV64IB-NEXT:    zip8.w a0, a0
 ; RV64IB-NEXT:    ret
 ;
 ; RV64IBP-LABEL: shfl8_i32:
 ; RV64IBP:       # %bb.0:
-; RV64IBP-NEXT:    lui a1, 1044480
-; RV64IBP-NEXT:    addiw a1, a1, 255
-; RV64IBP-NEXT:    and a1, a0, a1
-; RV64IBP-NEXT:    slli a2, a0, 8
-; RV64IBP-NEXT:    lui a3, 4080
-; RV64IBP-NEXT:    and a2, a2, a3
-; RV64IBP-NEXT:    srli a0, a0, 8
-; RV64IBP-NEXT:    lui a3, 16
-; RV64IBP-NEXT:    addiw a3, a3, -256
-; RV64IBP-NEXT:    and a0, a0, a3
-; RV64IBP-NEXT:    or a0, a1, a0
-; RV64IBP-NEXT:    or a0, a0, a2
+; RV64IBP-NEXT:    zip8.w a0, a0
 ; RV64IBP-NEXT:    ret
   %and = and i32 %a, -16776961
   %shl = shl i32 %a, 8
   %and1 = and i32 %shl, 16711680
   %shr = lshr i32 %a, 8
   %and2 = and i32 %shr, 65280
   %or = or i32 %and, %and2
   %or3 = or i32 %or, %and1