[AArch64][GlobalISel] Lower G_BUILD_VECTOR -> G_DUP

[paquette]

Summary

If we have

%vec = G_BUILD_VECTOR %reg, %reg, ..., %reg

Then lower it to

%vec = G_DUP %reg

Also update the selector to handle constant splats on G_DUP.

Minor code size improvements on CTMark at -Os.

[aemerson]

llvm/lib/Target/AArch64/GISel/AArch64PostLegalizerLowering.cpp

475

I think we should this be a helper class and live in some AArch64 specific utils somewhere. We should programmatically have a single query to test for a splat operation.

[aemerson]

s/class/function.

[paquette]

• Fix a bug in G_DUP for preISelLower and add a testcase which exercises it to select-dup.mir
• Add getAArch64VectorConstantSplat, which checks for G_DUP as well as G_BUILD_VECTOR. Use that instead in places which now need to check for G_DUP.

(There's one place where the helper might make some difference, since getBuildVectorConstantSplat doesn't look through constants.)

[paquette]

Looks like this blocks selecting vector neg since we miss some imported patterns. E.g.

// (sub:{ *:[v2i32] } immAllZerosV:{ *:[v2i32] }, V64:{ *:[v2i32] }:$Rn)  =>  (NEGv2i32:{ *:[v2i32] } V64:{ *:[v2i32] }:$Rn)

Is there any way to get a G_DUP with 0 to match here?

[aemerson]

In D97731#2601251, @paquette wrote:

Looks like this blocks selecting vector neg since we miss some imported patterns. E.g.

// (sub:{ *:[v2i32] } immAllZerosV:{ *:[v2i32] }, V64:{ *:[v2i32] }:$Rn)  =>  (NEGv2i32:{ *:[v2i32] } V64:{ *:[v2i32] }:$Rn)

Is there any way to get a G_DUP with 0 to match here?

:(

I don't see any easy way. You could maybe replicate the patterns that use immAllZeroesV to have a variant that uses AArch64dup but that doesn't seem like the right approach. I guess we have to avoid using this for constants in the absence of a better idea.

[paquette]

This time, disallow 0 and 1 specifically.

I think there are some DUP combines we're going to want. If 90% of the time it's better to represent this as DUP, I think it's probably better to prefer that.

Also improve the helpers a little bit. I figured it'd be nice to have a general helper which gives you either a constant or a register for the dup matching.

[aemerson]

llvm/include/llvm/CodeGen/GlobalISel/Utils.h

263 ↗

(On Diff #328328)

A quick comment to explain what this is for would be helpful. I also think we have the space to fully spell out Constant in the type.

267 ↗

(On Diff #328328)

This is implementing a tagged union, and since it's better to not try to cast a Register to an int64 here, try:

class RegOrConstant {
  union {
    int64_t IntVal;
    Register Reg;
  };
  bool IsReg;
public:
  explicit RegOrConstant(int64_t Cst) : IntVal(Cst), IsReg(false) {}
  explicit RegOrConstant(Register R) : Reg(R), IsReg(true) {}
  Register getReg()...
... etc
};

llvm/lib/Target/AArch64/GISel/AArch64GlobalISelUtils.h

23 ↗

(On Diff #328328)

The types used in the declarations aren't opaque pointers, so forward declaring doesn't help here.

34 ↗

(On Diff #328328)

These helpers should be names could be more precise, because they seem to return a scalar, whereas "splat" usually refers to the vector itself. Something like getAArch64VectorSplatSrc() or getAArch64VectorSplatScalar() might be better.

llvm/lib/Target/AArch64/GISel/AArch64PostLegalizerLowering.cpp

707

I'd specifically call out the immAllZeroesV and immAllOnesV patterns in this comment.

[paquette]

llvm/include/llvm/CodeGen/GlobalISel/Utils.h

263 ↗

(On Diff #328328)

Like rename it to Constant?

267 ↗

(On Diff #328328)

I actually wrote this as a union originally, but it would do the wrong thing if Register has a non-trivial destructor or non-trivial copy constructor, so I just avoided the whole issue.

[aemerson]

llvm/include/llvm/CodeGen/GlobalISel/Utils.h

263 ↗

(On Diff #328328)

Yeah, RegOrConstant.

267 ↗

(On Diff #328328)

Ah yeah. In that case, I think it's still better to use extra memory and store both a Register and an in64_t, rather than casting.

[paquette]

Address review comments.

[aemerson]

LGTM.

llvm/lib/Target/AArch64/AArch64Combine.td

 def mul_const_matchdata : GIDefMatchData<"std::function<void(MachineIRBuilder&, Register)>">;
 def mul_const : GICombineRule<
   (defs root:$root, mul_const_matchdata:$matchinfo),
   (match (wip_match_opcode G_MUL):$root,
           [{ return matchAArch64MulConstCombine(*${root}, MRI, ${matchinfo}); }]),
   (apply [{ applyAArch64MulConstCombine(*${root}, MRI, B, ${matchinfo}); }])
 >;
 
+def build_vector_to_dup : GICombineRule<
+  (defs root:$root),
+  (match (wip_match_opcode G_BUILD_VECTOR):$root,
+          [{ return matchBuildVectorToDup(*${root}, MRI); }]),
+  (apply [{ return applyBuildVectorToDup(*${root}, MRI, B); }])
+>;
+
+def build_vector_lowering : GICombineGroup<[build_vector_to_dup]>;
+
 // Post-legalization combines which should happen at all optimization levels.
 // (E.g. ones that facilitate matching for the selector) For example, matching
 // pseudos.
 def AArch64PostLegalizerLoweringHelper
     : GICombinerHelper<"AArch64GenPostLegalizerLoweringHelper",
                        [shuffle_vector_lowering, vashr_vlshr_imm,
-                        icmp_lowering]> {
+                        icmp_lowering, build_vector_lowering]> {
   let DisableRuleOption = "aarch64postlegalizerlowering-disable-rule";
 }
 
 // Post-legalization combines which are primarily optimizations.
 def AArch64PostLegalizerCombinerHelper
     : GICombinerHelper<"AArch64GenPostLegalizerCombinerHelper",
                        [copy_prop, erase_undef_store, combines_for_extload,
                         sext_trunc_sextload,
                         hoist_logic_op_with_same_opcode_hands,
                         redundant_and, xor_of_and_with_same_reg,
                         extractvecelt_pairwise_add, redundant_or,
                         mul_const, redundant_sext_inreg]> {
   let DisableRuleOption = "aarch64postlegalizercombiner-disable-rule";
 }

llvm/lib/Target/AArch64/GISel/AArch64PostLegalizerLowering.cpp

 /// isVShiftRImm - Check if this is a valid vector for the immediate
 /// operand of a vector shift right operation. The value must be in the range:
 ///   1 <= Value <= ElementBits for a right shift.
 static bool isVShiftRImm(Register Reg, MachineRegisterInfo &MRI, LLT Ty,
                          int64_t &Cnt) {
   assert(Ty.isVector() && "vector shift count is not a vector type");
   MachineInstr *MI = MRI.getVRegDef(Reg);
   auto Cst = getBuildVectorConstantSplat(*MI, MRI);
   if (!Cst)
     return false;
   Cnt = *Cst;
   int64_t ElementBits = Ty.getScalarSizeInBits();
   return Cnt >= 1 && Cnt <= ElementBits;
 }
 
 /// Match a vector G_ASHR or G_LSHR with a valid immediate shift.
 static bool matchVAshrLshrImm(MachineInstr &MI, MachineRegisterInfo &MRI,
bool applyDupLane(MachineInstr &MI, MachineRegisterInfo &MRI,
   B.setInstrAndDebugLoc(MI);
   auto Lane = B.buildConstant(LLT::scalar(64), MatchInfo.second);
   B.buildInstr(MatchInfo.first, {MI.getOperand(0).getReg()},
                {MI.getOperand(1).getReg(), Lane});
   MI.eraseFromParent();
   return true;
 }
 
+static bool matchBuildVectorToDup(MachineInstr &MI, MachineRegisterInfo &MRI) {
+  assert(MI.getOpcode() == TargetOpcode::G_BUILD_VECTOR);
+  // Constant splats should be selected to MOV.
+  auto Reg = MI.getOperand(1).getReg();
+  auto Range = make_range(MI.operands_begin() + 2, MI.operands_end());
+  return !getConstantVRegValWithLookThrough(Reg, MRI) &&
+         all_of(Range, [&Reg](const MachineOperand &Op) {
+           return Op.getReg() == Reg;
+         });
+}
+
+static bool applyBuildVectorToDup(MachineInstr &MI, MachineRegisterInfo &MRI,
+                                  MachineIRBuilder &B) {
+  B.setInstrAndDebugLoc(MI);
+  B.buildInstr(AArch64::G_DUP, {MI.getOperand(0).getReg()},
+               {MI.getOperand(1).getReg()});
+  MI.eraseFromParent();
+  return true;
+}
+
 #define AARCH64POSTLEGALIZERLOWERINGHELPER_GENCOMBINERHELPER_DEPS
 #include "AArch64GenPostLegalizeGILowering.inc"
 #undef AARCH64POSTLEGALIZERLOWERINGHELPER_GENCOMBINERHELPER_DEPS
 
 namespace {
 #define AARCH64POSTLEGALIZERLOWERINGHELPER_GENCOMBINERHELPER_H
 #include "AArch64GenPostLegalizeGILowering.inc"
 #undef AARCH64POSTLEGALIZERLOWERINGHELPER_GENCOMBINERHELPER_H

llvm/test/CodeGen/AArch64/GlobalISel/postlegalizer-lowering-build-vector-to-dup.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple aarch64 -run-pass=aarch64-postlegalizer-lowering -verify-machineinstrs %s -o - | FileCheck %s
+
+...
+---
+name:            same_reg
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $d0
+    ; CHECK-LABEL: name: same_reg
+    ; CHECK: liveins: $d0
+    ; CHECK: %r:_(s8) = G_IMPLICIT_DEF
+    ; CHECK: %build_vector:_(<8 x s8>) = G_DUP %r(s8)
+    ; CHECK: $d0 = COPY %build_vector(<8 x s8>)
+    ; CHECK: RET_ReallyLR implicit $d0
+    %r:_(s8) = G_IMPLICIT_DEF
+    %build_vector:_(<8 x s8>) = G_BUILD_VECTOR %r, %r, %r, %r, %r, %r, %r, %r
+    $d0 = COPY %build_vector(<8 x s8>)
+    RET_ReallyLR implicit $d0
+
+...
+---
+name:            dont_combine_different_reg
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $d0, $w0, $w1
+    ; CHECK-LABEL: name: dont_combine_different_reg
+    ; CHECK: liveins: $d0, $w0, $w1
+    ; CHECK: %r:_(s32) = COPY $w0
+    ; CHECK: %q:_(s32) = COPY $w1
+    ; CHECK: %build_vector:_(<2 x s32>) = G_BUILD_VECTOR %r(s32), %q(s32)
+    ; CHECK: $d0 = COPY %build_vector(<2 x s32>)
+    ; CHECK: RET_ReallyLR implicit $d0
+    %r:_(s32) = COPY $w0
+    %q:_(s32) = COPY $w1
+    %build_vector:_(<2 x s32>) = G_BUILD_VECTOR %r, %q
+    $d0 = COPY %build_vector(<2 x s32>)
+    RET_ReallyLR implicit $d0
+
+...
+---
+name:            dont_combine_constant_splat
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $d0
+
+    ; These can be better selected as a MOV.
+
+    ; CHECK-LABEL: name: dont_combine_constant_splat
+    ; CHECK: liveins: $d0
+    ; CHECK: %cst:_(s32) = G_CONSTANT i32 0
+    ; CHECK: %build_vector:_(<2 x s32>) = G_BUILD_VECTOR %cst(s32), %cst(s32)
+    ; CHECK: $d0 = COPY %build_vector(<2 x s32>)
+    ; CHECK: RET_ReallyLR implicit $d0
+    %cst:_(s32) = G_CONSTANT i32 0
+    %build_vector:_(<2 x s32>) = G_BUILD_VECTOR %cst, %cst
+    $d0 = COPY %build_vector(<2 x s32>)
+    RET_ReallyLR implicit $d0
+
+...
+---
+name:            dont_combine_constant_fp_splat
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $d0
+
+    ; These can be better selected as a MOV.
+
+    ; CHECK-LABEL: name: dont_combine_constant_fp_splat
+    ; CHECK: liveins: $d0
+    ; CHECK: %cst:_(s32) = G_FCONSTANT float 0.000000e+00
+    ; CHECK: %build_vector:_(<2 x s32>) = G_BUILD_VECTOR %cst(s32), %cst(s32)
+    ; CHECK: $d0 = COPY %build_vector(<2 x s32>)
+    ; CHECK: RET_ReallyLR implicit $d0
+    %cst:_(s32) = G_FCONSTANT float 0.0
+    %build_vector:_(<2 x s32>) = G_BUILD_VECTOR %cst, %cst
+    $d0 = COPY %build_vector(<2 x s32>)
+    RET_ReallyLR implicit $d0