[CodeGen] Support folds of not(cmp(cc, ...)) -> cmp(!cc, ...) for scalable vectors

[david-arm]

Summary

I have updated TargetLowering::isConstTrueVal to also consider
SPLAT_VECTOR nodes with constant integer operands. This allows the
optimisation to also work for targets that support scalable vectors.

[frasercrmck]

Nice, I'd missed that one. I was going to ask about SPLAT_VECTOR_PARTS but I'm not sure we could get a test for that, seeing as I think it's only generated for 64-bit integer splats on RV32.

llvm/test/CodeGen/RISCV/rvv/cmp-folds.ll

3

I think using --check-prefixes=CHECK,RV32 is fairly common in our RVV testing and looks like it'd help remove all duplicate checks.

[david-arm]

• Used better check prefixes for RISCV tests.

[david-arm]

llvm/test/CodeGen/RISCV/rvv/cmp-folds.ll

3

Good suggestion, thanks!

[frasercrmck]

llvm/test/CodeGen/RISCV/rvv/cmp-folds.ll

3

Oh but of course if there's no use of the RV32 or RV64 checks then lit will fail, sorry about that: I always forget. Sounds like you can just use FileCheck %s

[david-arm]

• Removed unnecessary CHECK lines.

[frasercrmck]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

3175

I think we have to support the truncating case as with BUILD_VECTOR above, don't we?

[david-arm]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

3175

Hmm, I looked at the definition of SPLAT_VECTOR and you're right - the operand is allowed to be wider than the vector element type of the result. I hadn't realised that :(. The only problem is that I have absolutely no idea how to write a test case for this using IR! I'm not sure if there is a way to create a splat (using SPLAT_VECTOR) in IR that is truncating. Unless you know of any RISCV examples where this would happen?

[david-arm]

• Add support for truncating splats.

[frasercrmck]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

3175

Hmm, it would definitely happen during legalization as we only have either i32 (RV32) or i64 (RV64) as legal integer types, so most splats, really, like <vscale x 1 x i8> with a constant integer. But you'd need to be calling this during/after legalization. So no I'm not sure it's testable, per se. What you've got looks right, though?

[sdesmalen]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

3160

Instead of checking for SPLAT_VECTOR explicitly, how about:

if (auto *CN = dyn_cast<ConstantSDNode>(N)) {
  CVal = CN->getAPIntValue();
} else if (isConstantSplatVector(N, CVal)) {
  unsigned EltWidth = N->getValueType(0).getScalarSizeInBits();
  if (EltWidth < CVal.getBitwidth())
    CVal = CVal.trunc(EltWidth);
} else
  return false;

[david-arm]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

3160

Sadly this doesn't work because isConstantSplatVector demands the element width of the result be the same as the element width in the BuildVectorSDNode, which means we end up with lots of failing tests. It's just a minor point, but isConstantSplatVector also permits FP splats, so we'd have to restrict it to integer too.

[sdesmalen]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

3160

In that case, maybe you can use

ConstantSDNode* llvm::isConstOrConstSplat(SDValue N, bool AllowUndefs, bool AllowTruncation)

which allows you to specify whether truncation is allowed?

It's bonkers how many of these functions exist...

[david-arm]

• Updated isConsTrueVal to use isConstOrConstSplat.

[sdesmalen]

Thanks for the change, I think this is an improvement since we can now reuse some existing code for recognising the splat!

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

911 ↗

(On Diff #403963)

Can you update isConstFalseVal as well to only take an SDValue as part of this patch?

[sdesmalen]

Thanks @david-arm, LGTM!

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

 // FIXME: Ideally, this would use ISD::isConstantSplatVector(), but that must
 // work with truncating build vectors and vectors with elements of less than
 // 8 bits.
 bool TargetLowering::isConstTrueVal(const SDNode *N) const {
   if (!N)
     return false;
 
   APInt CVal;
   if (auto *CN = dyn_cast<ConstantSDNode>(N)) {
     CVal = CN->getAPIntValue();
   } else if (auto *BV = dyn_cast<BuildVectorSDNode>(N)) {
     auto *CN = BV->getConstantSplatNode();
     if (!CN)
       return false;
 
     // If this is a truncating build vector, truncate the splat value.
     // Otherwise, we may fail to match the expected values below.
     unsigned BVEltWidth = BV->getValueType(0).getScalarSizeInBits();
     CVal = CN->getAPIntValue();
     if (BVEltWidth < CVal.getBitWidth())
       CVal = CVal.trunc(BVEltWidth);
+  } else if (N->getOpcode() == ISD::SPLAT_VECTOR &&
+             isa<ConstantSDNode>(N->getOperand(0))) {
+    CVal = cast<ConstantSDNode>(N->getOperand(0))->getAPIntValue();
   } else {
     return false;
   }
 
   switch (getBooleanContents(N->getValueType(0))) {
   case UndefinedBooleanContent:
     return CVal[0];
   case ZeroOrOneBooleanContent:

llvm/test/CodeGen/AArch64/sve-cmp-folds.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-unknown -mattr=+sve -o - < %s | FileCheck %s
+
+define <vscale x 8 x i1> @not_icmp_sle_nxv8i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
+; CHECK-LABEL: not_icmp_sle_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    cmpgt p0.h, p0/z, z0.h, z1.h
+; CHECK-NEXT:    ret
+  %icmp = icmp sle <vscale x 8 x i16> %a, %b
+  %tmp = insertelement <vscale x 8 x i1> undef, i1 true, i32 0
+  %ones = shufflevector <vscale x 8 x i1> %tmp, <vscale x 8 x i1> undef, <vscale x 8 x i32> zeroinitializer
+  %not = xor <vscale x 8 x i1> %ones, %icmp
+  ret <vscale x 8 x i1> %not
+}
+
+define <vscale x 4 x i1> @not_icmp_sgt_nxv4i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: not_icmp_sgt_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    cmpge p0.s, p0/z, z1.s, z0.s
+; CHECK-NEXT:    ret
+  %icmp = icmp sgt <vscale x 4 x i32> %a, %b
+  %tmp = insertelement <vscale x 4 x i1> undef, i1 true, i32 0
+  %ones = shufflevector <vscale x 4 x i1> %tmp, <vscale x 4 x i1> undef, <vscale x 4 x i32> zeroinitializer
+  %not = xor <vscale x 4 x i1> %icmp, %ones
+  ret <vscale x 4 x i1> %not
+}
+
+define <vscale x 2 x i1> @not_fcmp_une_nxv2f64(<vscale x 2 x double> %a, <vscale x 2 x double> %b) {
+; CHECK-LABEL: not_fcmp_une_nxv2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    fcmeq p0.d, p0/z, z0.d, z1.d
+; CHECK-NEXT:    ret
+  %icmp = fcmp une <vscale x 2 x double> %a, %b
+  %tmp = insertelement <vscale x 2 x i1> undef, i1 true, i32 0
+  %ones = shufflevector <vscale x 2 x i1> %tmp, <vscale x 2 x i1> undef, <vscale x 2 x i32> zeroinitializer
+  %not = xor <vscale x 2 x i1> %icmp, %ones
+  ret <vscale x 2 x i1> %not
+}
+
+define <vscale x 4 x i1> @not_fcmp_uge_nxv4f32(<vscale x 4 x float> %a, <vscale x 4 x float> %b) {
+; CHECK-LABEL: not_fcmp_uge_nxv4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    fcmgt p0.s, p0/z, z1.s, z0.s
+; CHECK-NEXT:    ret
+  %icmp = fcmp uge <vscale x 4 x float> %a, %b
+  %tmp = insertelement <vscale x 4 x i1> undef, i1 true, i32 0
+  %ones = shufflevector <vscale x 4 x i1> %tmp, <vscale x 4 x i1> undef, <vscale x 4 x i32> zeroinitializer
+  %not = xor <vscale x 4 x i1> %icmp, %ones
+  ret <vscale x 4 x i1> %not
+}

llvm/test/CodeGen/RISCV/rvv/cmp-folds.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+experimental-zfh,+experimental-v \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s --check-prefixes=RV32
+; RUN: llc -mtriple=riscv64 -mattr=+d,+experimental-zfh,+experimental-v \
+; RUN:     -verify-machineinstrs < %s | FileCheck %s --check-prefixes=RV64
+
+define <vscale x 8 x i1> @not_icmp_sle_nxv8i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
+; RV32-LABEL: not_icmp_sle_nxv8i16:
+; RV32:       # %bb.0:
+; RV32-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV32-NEXT:    vmslt.vv v0, v10, v8
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: not_icmp_sle_nxv8i16:
+; RV64:       # %bb.0:
+; RV64-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV64-NEXT:    vmslt.vv v0, v10, v8
+; RV64-NEXT:    ret
+  %icmp = icmp sle <vscale x 8 x i16> %a, %b
+  %tmp = insertelement <vscale x 8 x i1> undef, i1 true, i32 0
+  %ones = shufflevector <vscale x 8 x i1> %tmp, <vscale x 8 x i1> undef, <vscale x 8 x i32> zeroinitializer
+  %not = xor <vscale x 8 x i1> %ones, %icmp
+  ret <vscale x 8 x i1> %not
+}
+
+define <vscale x 4 x i1> @not_icmp_sgt_nxv4i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
+; RV32-LABEL: not_icmp_sgt_nxv4i32:
+; RV32:       # %bb.0:
+; RV32-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32-NEXT:    vmsle.vv v0, v8, v10
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: not_icmp_sgt_nxv4i32:
+; RV64:       # %bb.0:
+; RV64-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64-NEXT:    vmsle.vv v0, v8, v10
+; RV64-NEXT:    ret
+  %icmp = icmp sgt <vscale x 4 x i32> %a, %b
+  %tmp = insertelement <vscale x 4 x i1> undef, i1 true, i32 0
+  %ones = shufflevector <vscale x 4 x i1> %tmp, <vscale x 4 x i1> undef, <vscale x 4 x i32> zeroinitializer
+  %not = xor <vscale x 4 x i1> %icmp, %ones
+  ret <vscale x 4 x i1> %not
+}
+
+define <vscale x 2 x i1> @not_fcmp_une_nxv2f64(<vscale x 2 x double> %a, <vscale x 2 x double> %b) {
+; RV32-LABEL: not_fcmp_une_nxv2f64:
+; RV32:       # %bb.0:
+; RV32-NEXT:    vsetvli a0, zero, e64, m2, ta, mu
+; RV32-NEXT:    vmfeq.vv v0, v8, v10
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: not_fcmp_une_nxv2f64:
+; RV64:       # %bb.0:
+; RV64-NEXT:    vsetvli a0, zero, e64, m2, ta, mu
+; RV64-NEXT:    vmfeq.vv v0, v8, v10
+; RV64-NEXT:    ret
+  %icmp = fcmp une <vscale x 2 x double> %a, %b
+  %tmp = insertelement <vscale x 2 x i1> undef, i1 true, i32 0
+  %ones = shufflevector <vscale x 2 x i1> %tmp, <vscale x 2 x i1> undef, <vscale x 2 x i32> zeroinitializer
+  %not = xor <vscale x 2 x i1> %icmp, %ones
+  ret <vscale x 2 x i1> %not
+}
+
+define <vscale x 4 x i1> @not_fcmp_uge_nxv4f32(<vscale x 4 x float> %a, <vscale x 4 x float> %b) {
+; RV32-LABEL: not_fcmp_uge_nxv4f32:
+; RV32:       # %bb.0:
+; RV32-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32-NEXT:    vmflt.vv v0, v8, v10
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: not_fcmp_uge_nxv4f32:
+; RV64:       # %bb.0:
+; RV64-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64-NEXT:    vmflt.vv v0, v8, v10
+; RV64-NEXT:    ret
+  %icmp = fcmp uge <vscale x 4 x float> %a, %b
+  %tmp = insertelement <vscale x 4 x i1> undef, i1 true, i32 0
+  %ones = shufflevector <vscale x 4 x i1> %tmp, <vscale x 4 x i1> undef, <vscale x 4 x i32> zeroinitializer
+  %not = xor <vscale x 4 x i1> %icmp, %ones
+  ret <vscale x 4 x i1> %not
+}