[AArch64][SVE] Replace destructive operand of vector zeros with a bundled MOVPRFX instruction

[Allen]

Summary

Replace unary instructions where the destructive operand is a vector of zeros
with a bundled MOVPRFX instruction, e.g:

Transform:
    %X0 = DUP_ZI_S 0, 0
    %X0 = FLOGB_ZPmZ_S X0, P0, X2
 into:
     X0 = MOVPRFX P0/z, X1  // doesn't introduce any fake register dependencies compare to X0 = MOVPRFX P0/z, X0
     X0 = FLOGB_ZPmZ_S X0, P0, X2

NOTE: This patch add a @earlyclobber constraint to PredOneOpPassthruPseudo to ensure safe register allocation for movprfx usage.

Depends on D105889

[Allen]

hi @paulwalker-arm, would you please give me some suggestion, thanks

[paulwalker-arm]

In D138888#3959465, @Allen wrote:

hi @paulwalker-arm, would you please give me some suggestion, thanks

I'll investigate further but this looks like an old implementation that existed before the requirement to fix D105889. With D105889 in place we should be able to handle zeroing unary operations much like the existing experimental-zeroing support (i.e. within isel by creating _ZERO pseudo instructions that get expanded later), albeit the unary variants likely don't need to be experimental because we can better control their register allocation requirements.

[Allen]

refactor with comment

[Allen]

In D138888#3964294, @paulwalker-arm wrote:

In D138888#3959465, @Allen wrote:

hi @paulwalker-arm, would you please give me some suggestion, thanks

I'll investigate further but this looks like an old implementation that existed before the requirement to fix D105889. With D105889 in place we should be able to handle zeroing unary operations much like the existing experimental-zeroing support (i.e. within isel by creating _ZERO pseudo instructions that get expanded later), albeit the unary variants likely don't need to be experimental because we can better control their register allocation requirements.

hi, @paulwalker-arm

Thanks for your advice. As instriction flogb and fneg have some different form in DAG ISEL,  so I only try to enable flogb in this patch. （fneg will be transformed into FNEG_MERGE_PASSTHRU）

[Allen]

ping ?

[nikic]

(Not familiar with AArch64 / SVE)

[paulwalker-arm]

@Allen: I've been out of the office throughout January, hence the sparse reviewing. I'll be back to normal from tomorrow and will start working my way through the backlog.

[Allen]

In D138888#4096167, @paulwalker-arm wrote:

@Allen: I've been out of the office throughout January, hence the sparse reviewing. I'll be back to normal from tomorrow and will start working my way through the backlog.

Thank you advance

[paulwalker-arm]

Hi @Allen, again sorry for the delay. The patch looks mostly good but I think for the unary instructions the zeroing can be handled completely during isel with perhaps no changes to AArch64ExpandPseudoInsts.cpp necessary. Given the unary instructions have a dedicated operand for the inactive lanes I believe we can add a constraint to PredOneOpPassthruPseudo to ensure safe register allocation for movprfx usage. Something like:

let Constraints = !if(!eq(flags, FalseLanesZero), "$Zd = $Passthru,@earlyclobber $Zd", "");

This will ensure @passthru is allocated the same register as the destination whilst also being unique to the real input operand (i.e. the one containing the active lanes). The existing instruction expansion should emit:

movprfx	z0.h, p0/z, z1.h
flogb	z0.h, p0/m, z1.h

This is preferred over the output shown in sve2-intrinsics-fp-int-binary-logarithm-zeroing.ll because it doesn't introduce any fake register dependencies, which is something we've already fixed for the UNDEF variants.

What do you think?

llvm/lib/Target/AArch64/SVEInstrFormats.td

2798–2799

Are these changes required? sve_fp_2op_p_zd looks to already set DestructiveInstType accordingly.

llvm/test/CodeGen/AArch64/sve2-intrinsics-fp-int-binary-logarithm-zeroing.ll

2

I believe this is the default and thus not required?

[Allen]

Handle review suggestions based on the current modification solution.

I agree new solution that doesn't affect AArch64ExpandPseudoInsts.cpp seems better, I'm still trying.

[Allen]

llvm/lib/Target/AArch64/SVEInstrFormats.td

2798–2799

Thanks, When I try all the modifications of multiclass sve2_fp_flogb, an assert error occurs when I execute test case sve2-intrinsics-fp-int-binary-logarithm-zeroing.ll. Of course, the condition let DestructiveInstType = flags in is unnecessary, I'll delete it.

#2  0x0000aaaaaceef5c8 in llvm::AArch64InstPrinter::printInstruction (this=0xaaaab2be5f70, MI=0xffffffffc1c8, Address=0, STI=..., O=...)
    at lib/Target/AArch64/AArch64GenAsmWriter.inc:16963
16963	  assert(Bits != 0 && "Cannot print this instruction.");
(gdb) l
16958	  auto MnemonicInfo = getMnemonic(MI);
16959	
16960	  O << MnemonicInfo.first;
16961	
16962	  uint64_t Bits = MnemonicInfo.second;
16963	  assert(Bits != 0 && "Cannot print this instruction.");                 ----------- here -----------------------

llvm/test/CodeGen/AArch64/sve2-intrinsics-fp-int-binary-logarithm-zeroing.ll

2

Done, thanks

[Allen]

add @earlyclobber constraint according the comment, and revert the unnecessary changes to AArch64ExpandPseudoInsts.cpp

[paulwalker-arm]

A couple of stylistic requests but otherwise looks good.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3584–3586

We typically put these blocks just after the instructions they're pseudos for so can this block be placed just after defm FLOGB_ZPmZ?

llvm/lib/Target/AArch64/SVEInstrFormats.td

515

Can this be SVE_1_Op_PassthruZero_Pat? and moved further up so they follow the existing SVE_1_Op_PassthruUndef_.... classes.

[Allen]

sorry, missing the above review, and I'll fix that.

• Related link https://reviews.llvm.org/D143459

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

if (DstReg == MI.getOperand(3).getReg()) {
       break;
     }
     [[fallthrough]];
   case AArch64::DestructiveBinary:
   case AArch64::DestructiveBinaryImm:
     std::tie(PredIdx, DOPIdx, SrcIdx) = std::make_tuple(1, 2, 3);
     break;
   case AArch64::DestructiveUnaryPassthru:
+    if (FalseZero)
+      std::tie(PredIdx, DOPIdx, SrcIdx) = std::make_tuple(2, 0, 3);
+    else
       std::tie(PredIdx, DOPIdx, SrcIdx) = std::make_tuple(2, 3, 3);
     break;
   case AArch64::DestructiveTernaryCommWithRev:
     std::tie(PredIdx, DOPIdx, SrcIdx, Src2Idx) = std::make_tuple(1, 2, 3, 4);
     if (DstReg == MI.getOperand(3).getReg()) {
       // FMLA Zd, Pg, Za, Zd, Zm ==> FMAD Zdn, Pg, Zm, Za
       std::tie(PredIdx, DOPIdx, SrcIdx, Src2Idx) = std::make_tuple(1, 3, 4, 2);
       UseRev = true;
     } else if (DstReg == MI.getOperand(4).getReg()) {
bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
                                    MachineBasicBlock::iterator &NextMBBI) {
   MachineInstr &MI = *MBBI;
   unsigned Opcode = MI.getOpcode();
 
   // Check if we can expand the destructive op
   int OrigInstr = AArch64::getSVEPseudoMap(MI.getOpcode());
   if (OrigInstr != -1) {
     auto &Orig = TII->get(OrigInstr);
-    if ((Orig.TSFlags & AArch64::DestructiveInstTypeMask)
-           != AArch64::NotDestructive) {
+    if ((Orig.TSFlags & AArch64::DestructiveInstTypeMask) !=
+        AArch64::NotDestructive) {
       return expand_DestructiveOp(MI, MBB, MBBI);
     }
   }
 
   switch (Opcode) {
   default:
     break;
 

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [HasSVE2] in {
   def HISTSEG_ZZZ : sve2_hist_gen_segment<"histseg", int_aarch64_sve_histseg>;
 
   // SVE2 histogram generation (vector)
   defm HISTCNT_ZPzZZ : sve2_hist_gen_vector<"histcnt", int_aarch64_sve_histcnt>;
 } // End HasSVE2
 
 let Predicates = [HasSVE2orSME] in {
   // SVE2 floating-point base 2 logarithm as integer
-  defm FLOGB_ZPmZ : sve2_fp_flogb<"flogb", int_aarch64_sve_flogb>;
+  defm FLOGB_ZPmZ : sve2_fp_flogb<"flogb", "FLOGB_ZPZZ", int_aarch64_sve_flogb, DestructiveUnaryPassthru>;
 
   // SVE2 floating-point convert precision
   defm FCVTXNT_ZPmZ : sve2_fp_convert_down_odd_rounding_top<"fcvtxnt", "int_aarch64_sve_fcvtxnt">;
   defm FCVTX_ZPmZ   : sve2_fp_convert_down_odd_rounding<"fcvtx",       "int_aarch64_sve_fcvtx">;
   defm FCVTNT_ZPmZ  : sve2_fp_convert_down_narrow<"fcvtnt",            "int_aarch64_sve_fcvtnt">;
   defm FCVTLT_ZPmZ  : sve2_fp_convert_up_long<"fcvtlt",                "int_aarch64_sve_fcvtlt">;
 
   // SVE2 floating-point pairwise operations
let Predicates = [HasSVE2orSME] in {
 
   // SVE2 bitwise xor and rotate right by immediate
   defm XAR_ZZZI : sve2_int_rotate_right_imm<"xar", int_aarch64_sve_xar>;
 
   // SVE2 extract vector (immediate offset, constructive)
   def EXT_ZZI_B : sve2_int_perm_extract_i_cons<"ext">;
 } // End HasSVE2orSME
 
+let Predicates = [HasSVE2orSME, UseExperimentalZeroingPseudos] in {
+  defm FLOGB_ZPZZ : sve2_fp_un_pred_zeroing_hsd<int_aarch64_sve_flogb>;
+} // End HasSVE2orSME, UseExperimentalZeroingPseudos
+
 let Predicates = [HasSVE2] in {
   // SVE2 non-temporal gather loads
   defm LDNT1SB_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00000, "ldnt1sb", AArch64ldnt1s_gather_z, nxv4i8>;
   defm LDNT1B_ZZR_S  : sve2_mem_gldnt_vs_32_ptrs<0b00001, "ldnt1b",  AArch64ldnt1_gather_z,  nxv4i8>;
   defm LDNT1SH_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00100, "ldnt1sh", AArch64ldnt1s_gather_z, nxv4i16>;
   defm LDNT1H_ZZR_S  : sve2_mem_gldnt_vs_32_ptrs<0b00101, "ldnt1h",  AArch64ldnt1_gather_z,  nxv4i16>;
   defm LDNT1W_ZZR_S  : sve2_mem_gldnt_vs_32_ptrs<0b01001, "ldnt1w",  AArch64ldnt1_gather_z,  nxv4i32>;
 
   defm LDNT1SB_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10000, "ldnt1sb", AArch64ldnt1s_gather_z, nxv2i8>;
   defm LDNT1B_ZZR_D  : sve2_mem_gldnt_vs_64_ptrs<0b10010, "ldnt1b",  AArch64ldnt1_gather_z,  nxv2i8>;
   defm LDNT1SH_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10100, "ldnt1sh", AArch64ldnt1s_gather_z, nxv2i16>;
   defm LDNT1H_ZZR_D  : sve2_mem_gldnt_vs_64_ptrs<0b10110, "ldnt1h",  AArch64ldnt1_gather_z,  nxv2i16>;
   defm LDNT1SW_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b11000, "ldnt1sw", AArch64ldnt1s_gather_z, nxv2i32>;
   defm LDNT1W_ZZR_D  : sve2_mem_gldnt_vs_64_ptrs<0b11010, "ldnt1w",  AArch64ldnt1_gather_z,  nxv2i32>;
   defm LDNT1D_ZZR_D  : sve2_mem_gldnt_vs_64_ptrs<0b11110, "ldnt1d",  AArch64ldnt1_gather_z,  nxv2i64>;
 } // End HasSVE2
 
 let Predicates = [HasSVE2orSME] in {
   // SVE2 vector splice (constructive)
   defm SPLICE_ZPZZ : sve2_int_perm_splice_cons<"splice">;
 } // End HasSVE2orSME
 
 let Predicates = [HasSVE2] in {

llvm/lib/Target/AArch64/SVEInstrFormats.td

 : Pat<(vtd (vtd (op vt1:$Op1, (vselect vt1:$Op1, vt2:$Op2, (SVEDup0)), vt3:$Op3))),
       (inst $Op1, $Op2, $Op3)>;
 
 class SVE_3_Op_Pat_Shift_Imm_SelZero<ValueType vtd, SDPatternOperator op,
                                      ValueType vt1, ValueType vt2,
                                      Operand vt3, Instruction inst>
 : Pat<(vtd (op vt1:$Op1, (vselect vt1:$Op1, vt2:$Op2, (SVEDup0)), (i32 (vt3:$Op3)))),
       (inst $Op1, $Op2, vt3:$Op3)>;
+
+class SVE_2_Op_Pat_Zero<ValueType vtd, SDPatternOperator op, ValueType vt1,
+                   ValueType vt2, Instruction inst>
+   : Pat<(vtd (op (vtd (SVEDup0)), vt1:$Op1, vt2:$Op2)),
+        (inst (IMPLICIT_DEF), $Op1, $Op2)>;
 }
 
 //
 // Common but less generic patterns.
 //
 
 class SVE_1_Op_AllActive_Pat<ValueType vtd, SDPatternOperator op, ValueType vt1,
                              Instruction inst, Instruction ptrue>
: Pat<(vt (op pt:$Pg, vt:$Src, inreg_vt, vt:$PassThru)),
       (inst $PassThru, $Pg, $Src)>;
 
 multiclass SVE_InReg_Extend_PassthruUndef<ValueType vt, SDPatternOperator op, ValueType pt,
                                           ValueType inreg_vt, Instruction inst> {
   def : Pat<(vt (op pt:$Pg, vt:$Src, inreg_vt, (vt undef))),
             (inst (IMPLICIT_DEF), $Pg, $Src)>;
   def : Pat<(vt (op (pt (SVEAllActive:$Pg)), vt:$Src, inreg_vt, vt:$PassThru)),
             (inst $PassThru, $Pg, $Src)>;
 }
 
 class SVE_Shift_DupImm_Pred_Pat<ValueType vt, SDPatternOperator op,
                                 ValueType pt, ValueType it,
                                 ComplexPattern cast, Instruction inst>
 : Pat<(vt (op pt:$Pg, vt:$Rn, (vt (splat_vector (it (cast i32:$imm)))))),
       (inst $Pg, $Rn, i32:$imm)>;
 
 class SVE_Shift_DupImm_All_Active_Pat<ValueType vt, SDPatternOperator op,
: SVEPseudo2Instr<name, 0>,
     let FalseLanes = flags;
   }
 }
 
 //
 // Pseudos for passthru operands
 //
 let hasNoSchedulingInfo = 1 in {
-  class PredOneOpPassthruPseudo<string name, ZPRRegOp zprty>
+  class PredOneOpPassthruPseudo<string name, ZPRRegOp zprty,
+                                FalseLanesEnum flags = FalseLanesNone>
   : SVEPseudo2Instr<name, 0>,
-    Pseudo<(outs zprty:$Zd), (ins zprty:$Passthru, PPR3bAny:$Pg, zprty:$Zs), []>;
+    Pseudo<(outs zprty:$Zd), (ins zprty:$Passthru, PPR3bAny:$Pg, zprty:$Zs), []> {
+    let FalseLanes = flags;
+  }
 }
 
 //===----------------------------------------------------------------------===//
 // SVE Predicate Misc Group
 //===----------------------------------------------------------------------===//
 
 class sve_int_pfalse<bits<6> opc, string asm>
 : I<(outs PPR8:$Pd), (ins),
multiclass sve_fp_2op_p_zd_HSD<bits<5> opc, string asm, SDPatternOperator op> {
   defm : SVE_1_Op_PassthruUndef_Pat<nxv8f16, op, nxv8i1, nxv8f16, !cast<Instruction>(NAME # _UNDEF_H)>;
   defm : SVE_1_Op_PassthruUndef_Pat<nxv4f16, op, nxv4i1, nxv4f16, !cast<Instruction>(NAME # _UNDEF_H)>;
   defm : SVE_1_Op_PassthruUndef_Pat<nxv2f16, op, nxv2i1, nxv2f16, !cast<Instruction>(NAME # _UNDEF_H)>;
   defm : SVE_1_Op_PassthruUndef_Pat<nxv4f32, op, nxv4i1, nxv4f32, !cast<Instruction>(NAME # _UNDEF_S)>;
   defm : SVE_1_Op_PassthruUndef_Pat<nxv2f32, op, nxv2i1, nxv2f32, !cast<Instruction>(NAME # _UNDEF_S)>;
   defm : SVE_1_Op_PassthruUndef_Pat<nxv2f64, op, nxv2i1, nxv2f64, !cast<Instruction>(NAME # _UNDEF_D)>;
 }
 
-multiclass sve2_fp_flogb<string asm, SDPatternOperator op> {
-  def _H : sve_fp_2op_p_zd<0b0011010, asm, ZPR16, ZPR16, ElementSizeH>;
-  def _S : sve_fp_2op_p_zd<0b0011100, asm, ZPR32, ZPR32, ElementSizeS>;
-  def _D : sve_fp_2op_p_zd<0b0011110, asm, ZPR64, ZPR64, ElementSizeD>;
+multiclass sve2_fp_flogb<string asm, string Ps, SDPatternOperator op, DestructiveInstTypeEnum flags> {
+  let DestructiveInstType = flags in {
+    def _H : sve_fp_2op_p_zd<0b0011010, asm, ZPR16, ZPR16, ElementSizeH>,
+               SVEPseudo2Instr<Ps # _H, 1>;
+    def _S : sve_fp_2op_p_zd<0b0011100, asm, ZPR32, ZPR32, ElementSizeS>,
+               SVEPseudo2Instr<Ps # _S, 1>;
+    def _D : sve_fp_2op_p_zd<0b0011110, asm, ZPR64, ZPR64, ElementSizeD>,
+               SVEPseudo2Instr<Ps # _D, 1>;
+  }
 
   def : SVE_3_Op_Pat<nxv8i16, op, nxv8i16, nxv8i1, nxv8f16, !cast<Instruction>(NAME # _H)>;
   def : SVE_3_Op_Pat<nxv4i32, op, nxv4i32, nxv4i1, nxv4f32, !cast<Instruction>(NAME # _S)>;
   def : SVE_3_Op_Pat<nxv2i64, op, nxv2i64, nxv2i1, nxv2f64, !cast<Instruction>(NAME # _D)>;
 }
 
+multiclass sve2_fp_un_pred_zeroing_hsd<SDPatternOperator op> {
+  def _ZERO_H : PredOneOpPassthruPseudo<NAME # _H, ZPR16, FalseLanesZero>;
+  def _ZERO_S : PredOneOpPassthruPseudo<NAME # _S, ZPR32, FalseLanesZero>;
+  def _ZERO_D : PredOneOpPassthruPseudo<NAME # _D, ZPR64, FalseLanesZero>;
+
+  def : SVE_2_Op_Pat_Zero<nxv8i16, op, nxv8i1, nxv8f16, !cast<Pseudo>(NAME # _ZERO_H)>;
+  def : SVE_2_Op_Pat_Zero<nxv4i32, op, nxv4i1, nxv4f32, !cast<Pseudo>(NAME # _ZERO_S)>;
+  def : SVE_2_Op_Pat_Zero<nxv2i64, op, nxv2i1, nxv2f64, !cast<Pseudo>(NAME # _ZERO_D)>;
+}
+
 multiclass sve2_fp_convert_down_odd_rounding<string asm, string op> {
   def _DtoS : sve_fp_2op_p_zd<0b0001010, asm, ZPR64, ZPR32, ElementSizeD>;
   def : SVE_3_Op_Pat<nxv4f32, !cast<SDPatternOperator>(op # _f32f64), nxv4f32, nxv2i1, nxv2f64, !cast<Instruction>(NAME # _DtoS)>;
 }
 
 //===----------------------------------------------------------------------===//
 // SVE Floating Point Unary Operations - Unpredicated Group
 //===----------------------------------------------------------------------===//

llvm/test/CodeGen/AArch64/sve2-intrinsics-fp-int-binary-logarithm-zeroing.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve2 -mattr=+use-experimental-zeroing-pseudos -asm-verbose=1 < %s | FileCheck %s
+
+;
+; FLOGB
+;
+
+; NOTE: The %unused paramter ensures z0 is free, leading to a simpler test.
+define <vscale x 8 x i16> @flogb_f16(<vscale x 8 x i16> %unused, <vscale x 8 x i1> %pg, <vscale x 8 x half> %a) {
+; CHECK-LABEL: flogb_f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    movprfx z0.h, p0/z, z0.h
+; CHECK-NEXT:    flogb z0.h, p0/m, z1.h
+; CHECK-NEXT:    ret
+  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.flogb.nxv8f16(<vscale x 8 x i16> zeroinitializer,
+                                                                 <vscale x 8 x i1> %pg,
+                                                                 <vscale x 8 x half> %a)
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @flogb_f32(<vscale x 4 x i32> %unused, <vscale x 4 x i1> %pg, <vscale x 4 x float> %a) {
+; CHECK-LABEL: flogb_f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    movprfx z0.s, p0/z, z0.s
+; CHECK-NEXT:    flogb z0.s, p0/m, z1.s
+; CHECK-NEXT:    ret
+  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.flogb.nxv4f32(<vscale x 4 x i32> zeroinitializer,
+                                                                 <vscale x 4 x i1> %pg,
+                                                                 <vscale x 4 x float> %a)
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @flogb_f64(<vscale x 2 x i64> %unused, <vscale x 2 x i1> %pg, <vscale x 2 x double> %a) {
+; CHECK-LABEL: flogb_f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    movprfx z0.d, p0/z, z0.d
+; CHECK-NEXT:    flogb z0.d, p0/m, z1.d
+; CHECK-NEXT:    ret
+  %out = call <vscale x 2 x i64> @llvm.aarch64.sve.flogb.nxv2f64(<vscale x 2 x i64> zeroinitializer,
+                                                                 <vscale x 2 x i1> %pg,
+                                                                 <vscale x 2 x double> %a)
+  ret <vscale x 2 x i64> %out
+}
+
+declare <vscale x 8 x i16> @llvm.aarch64.sve.flogb.nxv8f16(<vscale x 8 x i16>, <vscale x 8 x i1>, <vscale x 8 x half>)
+declare <vscale x 4 x i32> @llvm.aarch64.sve.flogb.nxv4f32(<vscale x 4 x i32>, <vscale x 4 x i1>, <vscale x 4 x float>)
+declare <vscale x 2 x i64> @llvm.aarch64.sve.flogb.nxv2f64(<vscale x 2 x i64>, <vscale x 2 x i1>, <vscale x 2 x double>)