[AArch64][SVE] Add predicate reinterpret intrinsics

[c-rhodes]

Summary

Implements the following intrinsics:

• llvm.aarch64.sve.convert.to.svbool
• llvm.aarch64.sve.convert.from.svbool

For converting the ACLE svbool_t type (<n x 16 x i1>) to and from the
other predicate types: <n x 8 x i1>, <n x 4 x i1> and <n x 2 x i1>.

[sdesmalen]

Simple patch that implements the conversions to/from svbool_t, looks good to me!

llvm/test/CodeGen/AArch64/sve-intrinsics-reinterpret.ll

44

nit: the naming is a bit inconsistent here, maybe rename the tests to have either: _from_b, _from_h, _from_s, _from_d and _to_b, _to_h, _to_s, _to_d instead of the b2b, b2h, h2b, etc.

[efriedma]

When you convert from <vscale x 2 x i1> to <vscale x 16 x i1> using sve.convert.to.svbool, what values do the other bits contain? Zero? Poison? Something else?

[sdesmalen]

In D74471#1872978, @efriedma wrote:

When you convert from <vscale x 2 x i1> to <vscale x 16 x i1> using sve.convert.to.svbool, what values do the other bits contain? Zero? Poison? Something else?

When converting from <vscale x 2 x i1> to svbool_t, the other lanes are defined as zero.

As you probably spotted, this implementation does not actually guarantee that those lanes are zeroed. At the moment though, this is not really a problem because there are no optimizations on these intrinsics that rely on this definition and the instructions themselves produce predicates where the lanes are zeroed implicitly. If we don't fold the zeroing operation into the instruction that implicitly zeroes the lanes, there is a performance penalty, so we should write some peephole pass to remove the unnecessary instructions.

We can either choose to:

• Add the explicit zeroing in this patch and optimise these away later.
• Leave the patch as-is for now - given that it doesn't break anything - and do this in a follow-up patch (adding a TODO to the list: https://docs.google.com/document/d/1Psr3s3eh9PUlrpkuU7OKOi8my7xSt_v1BvjoQpbAGXc)

Without having properly discussed this and deciding on which way to go, my LGTM was probably a bit premature. Sorry for that!

[efriedma]

I'd prefer correctness first, optimizations later. If we miss the optimization, it's a giant red flag to anyone reading the assembly. If we miss correctness, some very unlucky person might waste days trying to track down a miscompile.

At the moment though, this is not really a problem

I'm not sure what you mean by this; you can trivially break the defined semantics by just writing convert.to.svbool(convert.from.svbool(ptrue)). Do you mean the way the C intrinsics are lowered gives some protection?

[sdesmalen]

In D74471#1881394, @efriedma wrote:

Do you mean the way the C intrinsics are lowered gives some protection?

Correct, the intrinsics are intended to be created by the Clang ACLE implementation, which will reinterpret-cast the predicate results to svbool_t. Without optimisations that introduce a pattern like you describe, nothing breaks in practice. Of course when we will start adding such optimisations, that changes things.

I'm happy with the decision to focus on correctness first though, it probably shouldn't be too hard to fold away the explicit zeroing.

@c-rhodes can you update this patch to explicitly zero the other lanes?

[c-rhodes]

• Explicitly zero lanes when converting to svbool_t.

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

3055

If the conversion isn't changing the type, do you need a REINTERPRET_CAST at all?

llvm/test/CodeGen/AArch64/sve-intrinsics-reinterpret.ll

33

Are you sure the explicit zeroing is working correctly? It looks like reinterpret_bool_h2b and reinterpret_bool_s2b are generating the same code.

[c-rhodes]

• Fix zeroing.
• Update tests.

[c-rhodes]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

3055

No the reinterpret isn't necessary, I was expecting it to be DAG combined with performNVCASTCombine but realised there's a case missing for the reinterpret ISD node where that's invoked. I've updated it so the operand is returned if it's a cast to the same type.

[efriedma]

Looks fine other than the change to LowerSPLAT_VECTOR

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

7478

We should not be calling getMachineNode before Select(). If we need a new AArch64ISD opcode, please add one.

I'm not sure how this is relevant to the rest of the patch, in any case.

[c-rhodes]

Remove change to lower SPLAT_VECTOR of i1s to pfalse and update test.

[c-rhodes]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

7478

It was necessary for how the test was written downstream with zeroinitializer but you're right it's not really relevant. I've removed this.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

3060

Can you just use OutVT instead of using a REINTERPET_CAST?

[sdesmalen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

3060

Sorry, ignore that comment, it of course needs the reinterpret cast because the AND operation is done on nxv16i1.

[efriedma]

LGTM

[sdesmalen]

Same here, LGTM!

llvm/include/llvm/IR/IntrinsicsAArch64.td

 // Testing predicates
 //
 
 def int_aarch64_sve_ptest_any   : AdvSIMD_SVE_PTEST_Intrinsic;
 def int_aarch64_sve_ptest_first : AdvSIMD_SVE_PTEST_Intrinsic;
 def int_aarch64_sve_ptest_last  : AdvSIMD_SVE_PTEST_Intrinsic;
 
 //
+// Reinterpreting data
+//
+
+def int_aarch64_sve_convert_from_svbool : Intrinsic<[llvm_anyvector_ty],
+                                                    [llvm_nxv16i1_ty],
+                                                    [IntrNoMem]>;
+
+def int_aarch64_sve_convert_to_svbool : Intrinsic<[llvm_nxv16i1_ty],
+                                                  [llvm_anyvector_ty],
+                                                  [IntrNoMem]>;
+
+//
 // Gather loads: scalar base + vector offsets
 //
 
 // 64 bit unscaled offsets
 def int_aarch64_sve_ld1_gather : AdvSIMD_GatherLoad_64bitOffset_Intrinsic;
 
 // 64 bit scaled offsets
 def int_aarch64_sve_ld1_gather_index : AdvSIMD_GatherLoad_64bitOffset_Intrinsic;

llvm/lib/Target/AArch64/AArch64ISelLowering.h

enum NodeType : unsigned {
   LASTB,
   REV,
   TBL,
 
   INSR,
   PTEST,
   PTRUE,
 
+  REINTERPRET_CAST,
+
   LDNF1,
   LDNF1S,
   LDFF1,
   LDFF1S,
 
   // Unsigned gather loads.
   GLD1,
   GLD1_SCALED,

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

const char *AArch64TargetLowering::getTargetNodeName(unsigned Opcode) const {
   case AArch64ISD::ORV_PRED:          return "AArch64ISD::ORV_PRED";
   case AArch64ISD::EORV_PRED:         return "AArch64ISD::EORV_PRED";
   case AArch64ISD::ANDV_PRED:         return "AArch64ISD::ANDV_PRED";
   case AArch64ISD::CLASTA_N:          return "AArch64ISD::CLASTA_N";
   case AArch64ISD::CLASTB_N:          return "AArch64ISD::CLASTB_N";
   case AArch64ISD::LASTA:             return "AArch64ISD::LASTA";
   case AArch64ISD::LASTB:             return "AArch64ISD::LASTB";
   case AArch64ISD::REV:               return "AArch64ISD::REV";
+  case AArch64ISD::REINTERPRET_CAST:  return "AArch64ISD::REINTERPRET_CAST";
   case AArch64ISD::TBL:               return "AArch64ISD::TBL";
   case AArch64ISD::NOT:               return "AArch64ISD::NOT";
   case AArch64ISD::BIT:               return "AArch64ISD::BIT";
   case AArch64ISD::CBZ:               return "AArch64ISD::CBZ";
   case AArch64ISD::CBNZ:              return "AArch64ISD::CBNZ";
   case AArch64ISD::TBZ:               return "AArch64ISD::TBZ";
   case AArch64ISD::TBNZ:              return "AArch64ISD::TBNZ";
   case AArch64ISD::TC_RETURN:         return "AArch64ISD::TC_RETURN";
static SDValue LowerMUL(SDValue Op, SelectionDAG &DAG) {
   EVT Op1VT = Op1.getValueType();
   return DAG.getNode(N0->getOpcode(), DL, VT,
                      DAG.getNode(NewOpc, DL, VT,
                                DAG.getNode(ISD::BITCAST, DL, Op1VT, N00), Op1),
                      DAG.getNode(NewOpc, DL, VT,
                                DAG.getNode(ISD::BITCAST, DL, Op1VT, N01), Op1));
 }
 
+static inline SDValue getPTrue(SelectionDAG &DAG, SDLoc DL, EVT VT,
+                               int Pattern) {
+  return DAG.getNode(AArch64ISD::PTRUE, DL, VT,
+                     DAG.getTargetConstant(Pattern, DL, MVT::i32));
+}
+
 SDValue AArch64TargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
                                                      SelectionDAG &DAG) const {
   unsigned IntNo = cast<ConstantSDNode>(Op.getOperand(0))->getZExtValue();
   SDLoc dl(Op);
   switch (IntNo) {
   default: return SDValue();    // Don't custom lower most intrinsics.
   case Intrinsic::thread_pointer: {
     EVT PtrVT = getPointerTy(DAG.getDataLayout());
case Intrinsic::aarch64_sve_zip1:
     return DAG.getNode(AArch64ISD::ZIP1, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   case Intrinsic::aarch64_sve_zip2:
     return DAG.getNode(AArch64ISD::ZIP2, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   case Intrinsic::aarch64_sve_ptrue:
     return DAG.getNode(AArch64ISD::PTRUE, dl, Op.getValueType(),
                        Op.getOperand(1));
+  case Intrinsic::aarch64_sve_convert_from_svbool:
+    return DAG.getNode(AArch64ISD::REINTERPRET_CAST, dl, Op.getValueType(),
+                       Op.getOperand(1));
+  case Intrinsic::aarch64_sve_convert_to_svbool: {
+    EVT VT = Op.getValueType();
+    SDValue Reinterpret = DAG.getNode(AArch64ISD::REINTERPRET_CAST, dl, VT,
+                                      Op.getOperand(1));
+    // Return the reintepret if the cast isn't changing type,
+    // i.e. <n x 16 x i1> -> <n x 16 x i1>
+    if (VT == Op.getOperand(1).getValueType())
+      return Reinterpret;
+    // Otherwise, zero the newly introduced lanes.
+    SDValue Mask = getPTrue(DAG, dl, VT, AArch64SVEPredPattern::all);
+    return DAG.getNode(ISD::AND, dl, VT, Reinterpret, Mask);
+  }
 
   case Intrinsic::aarch64_sve_insr: {
     SDValue Scalar = Op.getOperand(2);
     EVT ScalarTy = Scalar.getValueType();
     if ((ScalarTy == MVT::i8) || (ScalarTy == MVT::i16))
       Scalar = DAG.getNode(ISD::ANY_EXTEND, dl, MVT::i32, Scalar);
 
     return DAG.getNode(AArch64ISD::INSR, dl, Op.getValueType(),
SDValue AArch64TargetLowering::LowerSPLAT_VECTOR(SDValue Op,
   case MVT::i16:
   case MVT::i32:
     SplatVal = DAG.getAnyExtOrTrunc(SplatVal, dl, MVT::i32);
     return DAG.getNode(AArch64ISD::DUP, dl, VT, SplatVal);
   case MVT::i64:
     SplatVal = DAG.getAnyExtOrTrunc(SplatVal, dl, MVT::i64);
     return DAG.getNode(AArch64ISD::DUP, dl, VT, SplatVal);
   case MVT::i1: {
+    if (auto CSplatVal = dyn_cast<ConstantSDNode>(SplatVal))
+      if (CSplatVal->isNullValue())
+        return SDValue(DAG.getMachineNode(AArch64::PFALSE, dl, VT), 0);
     // The general case of i1.  There isn't any natural way to do this,
     // so we use some trickery with whilelo.
-    // TODO: Add special cases for splat of constant true/false.
+    // TODO: Add special case for splat of constant true.
     SplatVal = DAG.getAnyExtOrTrunc(SplatVal, dl, MVT::i64);
     SplatVal = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, MVT::i64, SplatVal,
                            DAG.getValueType(MVT::i1));
     SDValue ID = DAG.getTargetConstant(Intrinsic::aarch64_sve_whilelo, dl,
                                        MVT::i64);
     return DAG.getNode(ISD::INTRINSIC_WO_CHAIN, dl, VT, ID,
                        DAG.getConstant(0, dl, MVT::i64), SplatVal);
   }

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

 def AArch64clastb_n   : SDNode<"AArch64ISD::CLASTB_N",   SDT_AArch64ReduceWithInit>;
 
 def SDT_AArch64Rev   : SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisSameAs<0,1>]>;
 def AArch64rev       : SDNode<"AArch64ISD::REV", SDT_AArch64Rev>;
 
 def SDT_AArch64PTest : SDTypeProfile<0, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>]>;
 def AArch64ptest     : SDNode<"AArch64ISD::PTEST", SDT_AArch64PTest>;
 
+def reinterpret_cast : SDNode<"AArch64ISD::REINTERPRET_CAST", SDTUnaryOp>;
+
 let Predicates = [HasSVE] in {
 
   defm RDFFR_PPz  : sve_int_rdffr_pred<0b0, "rdffr", int_aarch64_sve_rdffr_z>;
   def  RDFFRS_PPz : sve_int_rdffr_pred<0b1, "rdffrs">;
   defm RDFFR_P    : sve_int_rdffr_unpred<"rdffr", int_aarch64_sve_rdffr>;
   def  SETFFR     : sve_int_setffr<"setffr", int_aarch64_sve_setffr>;
   def  WRFFR      : sve_int_wrffr<"wrffr", int_aarch64_sve_wrffr>;
 

 
   def : Pat<(nxv2f64 (bitconvert (nxv16i8 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv8i16 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv4i32 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv2i64 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv8f16 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv4f32 ZPR:$src))), (nxv2f64 ZPR:$src)>;
 
+  def : Pat<(nxv16i1 (reinterpret_cast (nxv16i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv16i1 (reinterpret_cast (nxv8i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv16i1 (reinterpret_cast (nxv4i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv16i1 (reinterpret_cast (nxv2i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv8i1 (reinterpret_cast (nxv16i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv8i1 (reinterpret_cast  (nxv4i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv8i1 (reinterpret_cast  (nxv2i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv4i1 (reinterpret_cast (nxv16i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv4i1 (reinterpret_cast  (nxv8i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv4i1 (reinterpret_cast  (nxv2i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv2i1 (reinterpret_cast (nxv16i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv2i1 (reinterpret_cast  (nxv8i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+  def : Pat<(nxv2i1 (reinterpret_cast  (nxv4i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
+
+  def : Pat<(nxv16i1 (and PPR:$Ps1, PPR:$Ps2)),
+            (AND_PPzPP (PTRUE_B 31), PPR:$Ps1, PPR:$Ps2)>;
+  def : Pat<(nxv8i1 (and PPR:$Ps1, PPR:$Ps2)),
+            (AND_PPzPP (PTRUE_H 31), PPR:$Ps1, PPR:$Ps2)>;
+  def : Pat<(nxv4i1 (and PPR:$Ps1, PPR:$Ps2)),
+            (AND_PPzPP (PTRUE_S 31), PPR:$Ps1, PPR:$Ps2)>;
+  def : Pat<(nxv2i1 (and PPR:$Ps1, PPR:$Ps2)),
+            (AND_PPzPP (PTRUE_D 31), PPR:$Ps1, PPR:$Ps2)>;
+
   // Add more complex addressing modes here as required
   multiclass pred_load<ValueType Ty, ValueType PredTy, SDPatternOperator Load,
                         Instruction RegImmInst> {
 
     def _default_z : Pat<(Ty (Load  GPR64:$base, (PredTy PPR:$gp), (SVEDup0Undef))),
                          (RegImmInst PPR:$gp, GPR64:$base, (i64 0))>;
   }
 

llvm/test/CodeGen/AArch64/sve-intrinsics-reinterpret.ll

+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+;
+; Converting to svbool_t (<vscale x 16 x i1>)
+;
+
+define <vscale x 16 x i1> @reinterpret_bool_b2b() {
+; CHECK-LABEL: reinterpret_bool_b2b:
+; CHECK: pfalse p0.b
+; CHECK-NEXT: ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.convert.to.svbool.nxv16i1(<vscale x 16 x i1> zeroinitializer)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @reinterpret_bool_h2b() {
+; CHECK-LABEL: reinterpret_bool_h2b:
+; CHECK: pfalse p0.b
+; CHECK-NEXT: ptrue p1.b
+; CHECK-NEXT: and p0.b, p1/z, p0.b, p1.b
+; CHECK-NEXT: ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.convert.to.svbool.nxv8i1(<vscale x 8 x i1> zeroinitializer)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @reinterpret_bool_s2b() {
+; CHECK-LABEL: reinterpret_bool_s2b:
+; CHECK: pfalse p0.b
+; CHECK-NEXT: ptrue p1.b
+; CHECK-NEXT: and p0.b, p1/z, p0.b, p1.b
+; CHECK-NEXT: ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.convert.to.svbool.nxv4i1(<vscale x 4 x i1> zeroinitializer)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @reinterpret_bool_d2b() {
+; CHECK-LABEL: reinterpret_bool_d2b:
+; CHECK: pfalse p0.b
+; CHECK-NEXT: ptrue p1.b
+; CHECK-NEXT: and p0.b, p1/z, p0.b, p1.b
+; CHECK-NEXT: ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.convert.to.svbool.nxv2i1(<vscale x 2 x i1> zeroinitializer)
+  ret <vscale x 16 x i1> %out
+}
+
+;
+; Converting from svbool_t
+;
+
+define <vscale x 16 x i1> @reinterpret_bool_b2b_from() {
+; CHECK-LABEL: reinterpret_bool_b2b_from:
+; CHECK: pfalse p0.b
+; CHECK-NEXT: ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv16i1(<vscale x 16 x i1> zeroinitializer)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 8 x i1> @reinterpret_bool_b2h() {
+; CHECK-LABEL: reinterpret_bool_b2h:
+; CHECK: pfalse p0.b
+; CHECK-NEXT: ret
+  %out = call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> zeroinitializer)
+  ret <vscale x 8 x i1> %out
+}
+
+define <vscale x 4 x i1> @reinterpret_bool_b2s() {
+; CHECK-LABEL: reinterpret_bool_b2s:
+; CHECK: pfalse p0.b
+; CHECK-NEXT: ret
+  %out = call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> zeroinitializer)
+  ret <vscale x 4 x i1> %out
+}
+
+define <vscale x 2 x i1> @reinterpret_bool_b2d() {
+; CHECK-LABEL: reinterpret_bool_b2d:
+; CHECK: pfalse p0.b
+; CHECK-NEXT: ret
+  %out = call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> zeroinitializer)
+  ret <vscale x 2 x i1> %out
+}
+
+declare <vscale x 16 x i1> @llvm.aarch64.sve.convert.to.svbool.nxv16i1(<vscale x 16 x i1>)
+declare <vscale x 16 x i1> @llvm.aarch64.sve.convert.to.svbool.nxv8i1(<vscale x 8 x i1>)
+declare <vscale x 16 x i1> @llvm.aarch64.sve.convert.to.svbool.nxv4i1(<vscale x 4 x i1>)
+declare <vscale x 16 x i1> @llvm.aarch64.sve.convert.to.svbool.nxv2i1(<vscale x 2 x i1>)
+
+declare <vscale x 16 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv16i1(<vscale x 16 x i1>)
+declare <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1>)
+declare <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1>)
+declare <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1>)