[AArch64][GlobalISel] Lower G_BUILD_VECTOR -> G_DUP

[paquette]

Summary

If we have

%vec = G_BUILD_VECTOR %reg, %reg, ..., %reg

Then lower it to

%vec = G_DUP %reg

Also update the selector to handle constant splats on G_DUP.

Minor code size improvements on CTMark at -Os.

[aemerson]

llvm/lib/Target/AArch64/GISel/AArch64PostLegalizerLowering.cpp

474

I think we should this be a helper class and live in some AArch64 specific utils somewhere. We should programmatically have a single query to test for a splat operation.

[aemerson]

s/class/function.

[paquette]

• Fix a bug in G_DUP for preISelLower and add a testcase which exercises it to select-dup.mir
• Add getAArch64VectorConstantSplat, which checks for G_DUP as well as G_BUILD_VECTOR. Use that instead in places which now need to check for G_DUP.

(There's one place where the helper might make some difference, since getBuildVectorConstantSplat doesn't look through constants.)

[paquette]

Looks like this blocks selecting vector neg since we miss some imported patterns. E.g.

// (sub:{ *:[v2i32] } immAllZerosV:{ *:[v2i32] }, V64:{ *:[v2i32] }:$Rn)  =>  (NEGv2i32:{ *:[v2i32] } V64:{ *:[v2i32] }:$Rn)

Is there any way to get a G_DUP with 0 to match here?

[aemerson]

In D97731#2601251, @paquette wrote:

Looks like this blocks selecting vector neg since we miss some imported patterns. E.g.

// (sub:{ *:[v2i32] } immAllZerosV:{ *:[v2i32] }, V64:{ *:[v2i32] }:$Rn)  =>  (NEGv2i32:{ *:[v2i32] } V64:{ *:[v2i32] }:$Rn)

Is there any way to get a G_DUP with 0 to match here?

:(

I don't see any easy way. You could maybe replicate the patterns that use immAllZeroesV to have a variant that uses AArch64dup but that doesn't seem like the right approach. I guess we have to avoid using this for constants in the absence of a better idea.

[paquette]

This time, disallow 0 and 1 specifically.

I think there are some DUP combines we're going to want. If 90% of the time it's better to represent this as DUP, I think it's probably better to prefer that.

Also improve the helpers a little bit. I figured it'd be nice to have a general helper which gives you either a constant or a register for the dup matching.

[aemerson]

llvm/include/llvm/CodeGen/GlobalISel/Utils.h

263 ↗

(On Diff #328328)

A quick comment to explain what this is for would be helpful. I also think we have the space to fully spell out Constant in the type.

267 ↗

(On Diff #328328)

This is implementing a tagged union, and since it's better to not try to cast a Register to an int64 here, try:

class RegOrConstant {
  union {
    int64_t IntVal;
    Register Reg;
  };
  bool IsReg;
public:
  explicit RegOrConstant(int64_t Cst) : IntVal(Cst), IsReg(false) {}
  explicit RegOrConstant(Register R) : Reg(R), IsReg(true) {}
  Register getReg()...
... etc
};

llvm/lib/Target/AArch64/GISel/AArch64GlobalISelUtils.h

23 ↗

(On Diff #328328)

The types used in the declarations aren't opaque pointers, so forward declaring doesn't help here.

34 ↗

(On Diff #328328)

These helpers should be names could be more precise, because they seem to return a scalar, whereas "splat" usually refers to the vector itself. Something like getAArch64VectorSplatSrc() or getAArch64VectorSplatScalar() might be better.

llvm/lib/Target/AArch64/GISel/AArch64PostLegalizerLowering.cpp

713

I'd specifically call out the immAllZeroesV and immAllOnesV patterns in this comment.

[paquette]

llvm/include/llvm/CodeGen/GlobalISel/Utils.h

263 ↗

(On Diff #328328)

Like rename it to Constant?

267 ↗

(On Diff #328328)

I actually wrote this as a union originally, but it would do the wrong thing if Register has a non-trivial destructor or non-trivial copy constructor, so I just avoided the whole issue.

[aemerson]

llvm/include/llvm/CodeGen/GlobalISel/Utils.h

263 ↗

(On Diff #328328)

Yeah, RegOrConstant.

267 ↗

(On Diff #328328)

Ah yeah. In that case, I think it's still better to use extra memory and store both a Register and an in64_t, rather than casting.

[paquette]

Address review comments.

[aemerson]

LGTM.

llvm/lib/Target/AArch64/AArch64Combine.td

 def mul_const_matchdata : GIDefMatchData<"std::function<void(MachineIRBuilder&, Register)>">;
 def mul_const : GICombineRule<
   (defs root:$root, mul_const_matchdata:$matchinfo),
   (match (wip_match_opcode G_MUL):$root,
           [{ return matchAArch64MulConstCombine(*${root}, MRI, ${matchinfo}); }]),
   (apply [{ applyAArch64MulConstCombine(*${root}, MRI, B, ${matchinfo}); }])
 >;
 
+def build_vector_to_dup : GICombineRule<
+  (defs root:$root),
+  (match (wip_match_opcode G_BUILD_VECTOR):$root,
+          [{ return matchBuildVectorToDup(*${root}, MRI); }]),
+  (apply [{ return applyBuildVectorToDup(*${root}, MRI, B); }])
+>;
+
+def build_vector_lowering : GICombineGroup<[build_vector_to_dup]>;
+
 // Post-legalization combines which should happen at all optimization levels.
 // (E.g. ones that facilitate matching for the selector) For example, matching
 // pseudos.
 def AArch64PostLegalizerLoweringHelper
     : GICombinerHelper<"AArch64GenPostLegalizerLoweringHelper",
                        [shuffle_vector_lowering, vashr_vlshr_imm,
-                        icmp_lowering]> {
+                        icmp_lowering, build_vector_lowering]> {
   let DisableRuleOption = "aarch64postlegalizerlowering-disable-rule";
 }
 
 // Post-legalization combines which are primarily optimizations.
 def AArch64PostLegalizerCombinerHelper
     : GICombinerHelper<"AArch64GenPostLegalizerCombinerHelper",
                        [copy_prop, erase_undef_store, combines_for_extload,
                         sext_trunc_sextload,
                         hoist_logic_op_with_same_opcode_hands,
                         redundant_and, xor_of_and_with_same_reg,
                         extractvecelt_pairwise_add, redundant_or,
                         mul_const, redundant_sext_inreg]> {
   let DisableRuleOption = "aarch64postlegalizercombiner-disable-rule";
 }

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

 #include "AArch64RegisterInfo.h"
 #include "AArch64Subtarget.h"
 #include "AArch64TargetMachine.h"
 #include "MCTargetDesc/AArch64AddressingModes.h"
 #include "MCTargetDesc/AArch64MCTargetDesc.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/CodeGen/GlobalISel/InstructionSelector.h"
 #include "llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h"
 #include "llvm/CodeGen/GlobalISel/MachineIRBuilder.h"
 #include "llvm/CodeGen/GlobalISel/MIPatternMatch.h"
 #include "llvm/CodeGen/GlobalISel/Utils.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/TargetOpcodes.h"
 #include "llvm/IR/Constants.h"
+#include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/PatternMatch.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/IntrinsicsAArch64.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 
 #define DEBUG_TYPE "aarch64-isel"
 
 using namespace llvm;
 using namespace MIPatternMatch;
private:
   ///
   /// The lane inserted into is defined by \p LaneIdx. The vector source
   /// register is given by \p SrcReg. The register containing the element is
   /// given by \p EltReg.
   MachineInstr *emitLaneInsert(Optional<Register> DstReg, Register SrcReg,
                                Register EltReg, unsigned LaneIdx,
                                const RegisterBank &RB,
                                MachineIRBuilder &MIRBuilder) const;
+
+  /// Emit a sequence of instructions representing a constant \p CV for a
+  /// vector register \p Dst. (E.g. a MOV, or a load from a constant pool.)
+  ///
+  /// \returns the last instruction in the sequence on success, and nullptr
+  /// otherwise.
+  MachineInstr *emitConstantVector(Register Dst, Constant *CV,
+                                  MachineIRBuilder &MIRBuilder,
+                                  MachineRegisterInfo &MRI) const;
+
   bool selectInsertElt(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool tryOptConstantBuildVec(MachineInstr &MI, LLT DstTy,
                               MachineRegisterInfo &MRI) const;
   bool selectBuildVector(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool selectMergeValues(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool selectUnmergeValues(MachineInstr &I, MachineRegisterInfo &MRI) const;
 
   bool selectShuffleVector(MachineInstr &I, MachineRegisterInfo &MRI) const;

 
 /// Returns the element immediate value of a vector shift operand if found.
 /// This needs to detect a splat-like operation, e.g. a G_BUILD_VECTOR.
 static Optional<int64_t> getVectorShiftImm(Register Reg,
                                            MachineRegisterInfo &MRI) {
   assert(MRI.getType(Reg).isVector() && "Expected a *vector* shift operand");
   MachineInstr *OpMI = MRI.getVRegDef(Reg);
   assert(OpMI && "Expected to find a vreg def for vector shift operand");
+
+  // We may combine these to G_DUP.
+  if (OpMI->getOpcode() == AArch64::G_DUP) {
+    auto VRegAndVal =
+        getConstantVRegValWithLookThrough(OpMI->getOperand(1).getReg(), MRI);
+    if (!VRegAndVal)
+      return None;
+    return VRegAndVal->Value.getSExtValue();
+  }
+
   if (OpMI->getOpcode() != TargetOpcode::G_BUILD_VECTOR)
     return None;
 
   // Check all operands are identical immediates.
   int64_t ImmVal = 0;
   for (unsigned Idx = 1; Idx < OpMI->getNumOperands(); ++Idx) {
     auto VRegAndVal = getConstantVRegValWithLookThrough(OpMI->getOperand(Idx).getReg(), MRI);
     if (!VRegAndVal)
bool AArch64InstructionSelector::earlySelect(MachineInstr &I) const {
   assert(I.getParent() && "Instruction should be in a basic block!");
   assert(I.getParent()->getParent() && "Instruction should be in a function!");
 
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
   switch (I.getOpcode()) {
+  case AArch64::G_DUP: {
+    // Before selecting a DUP instruction, check if it is better selected as a
+    // MOV or load from a constant pool.
+    Register Src = I.getOperand(1).getReg();
+    auto ValAndVReg = getConstantVRegValWithLookThrough(Src, MRI);
+    if (!ValAndVReg)
+      return false;
+    LLVMContext &Ctx = MF.getFunction().getContext();
+    Register Dst = I.getOperand(0).getReg();
+    auto *CV = ConstantDataVector::getSplat(
+        MRI.getType(Dst).getNumElements(),
+        ConstantInt::get(Type::getIntNTy(Ctx, MRI.getType(Src).getSizeInBits()),
+                         ValAndVReg->Value));
+    MachineIRBuilder MIRBuilder(I);
+    if (!emitConstantVector(Dst, CV, MIRBuilder, MRI))
+      return false;
+    I.eraseFromParent();
+    return true;
+  }
   case TargetOpcode::G_BR: {
     // If the branch jumps to the fallthrough block, don't bother emitting it.
     // Only do this for -O0 for a good code size improvement, because when
     // optimizations are enabled we want to leave this choice to
     // MachineBlockPlacement.
     bool EnableOpt = MF.getTarget().getOptLevel() != CodeGenOpt::None;
     if (EnableOpt || !MBB.isLayoutSuccessor(I.getOperand(0).getMBB()))
       return false;
if (VecSize < 128) {
     InsMI->getOperand(0).setReg(DstReg);
     constrainSelectedInstRegOperands(*InsMI, TII, TRI, RBI);
   }
 
   I.eraseFromParent();
   return true;
 }
 
+MachineInstr* AArch64InstructionSelector::emitConstantVector(
+    Register Dst, Constant *CV, MachineIRBuilder &MIRBuilder,
+    MachineRegisterInfo &MRI) const {
+  LLT DstTy = MRI.getType(Dst);
+  unsigned DstSize = DstTy.getSizeInBits();
+  if (CV->isNullValue()) {
+    if (DstSize == 128) {
+      auto Mov =
+          MIRBuilder.buildInstr(AArch64::MOVIv2d_ns, {Dst}, {}).addImm(0);
+      constrainSelectedInstRegOperands(*Mov, TII, TRI, RBI);
+      return &*Mov;
+    }
+
+    if (DstSize == 64) {
+      auto Mov =
+          MIRBuilder
+              .buildInstr(AArch64::MOVIv2d_ns, {&AArch64::FPR128RegClass}, {})
+              .addImm(0);
+      auto Copy = MIRBuilder.buildInstr(TargetOpcode::COPY, {Dst}, {})
+          .addReg(Mov.getReg(0), 0, AArch64::dsub);
+      RBI.constrainGenericRegister(Dst, AArch64::FPR64RegClass, MRI);
+      return &*Copy;
+    }
+  }
+
+  auto *CPLoad = emitLoadFromConstantPool(CV, MIRBuilder);
+  if (!CPLoad) {
+    LLVM_DEBUG(dbgs() << "Could not generate cp load for constant vector!");
+    return nullptr;
+  }
+
+  auto Copy = MIRBuilder.buildCopy(Dst, CPLoad->getOperand(0));
+  RBI.constrainGenericRegister(
+      Dst, *MRI.getRegClass(CPLoad->getOperand(0).getReg()), MRI);
+  return &*Copy;
+}
+
 bool AArch64InstructionSelector::tryOptConstantBuildVec(
     MachineInstr &I, LLT DstTy, MachineRegisterInfo &MRI) const {
   assert(I.getOpcode() == TargetOpcode::G_BUILD_VECTOR);
   unsigned DstSize = DstTy.getSizeInBits();
   assert(DstSize <= 128 && "Unexpected build_vec type!");
   if (DstSize < 32)
     return false;
   // Check if we're building a constant vector, in which case we want to
else if ((OpMI = getOpcodeDef(TargetOpcode::G_FCONSTANT,
                                   I.getOperand(Idx).getReg(), MRI)))
       Csts.emplace_back(
           const_cast<ConstantFP *>(OpMI->getOperand(1).getFPImm()));
     else
       return false;
   }
   Constant *CV = ConstantVector::get(Csts);
   MachineIRBuilder MIB(I);
-  if (CV->isNullValue()) {
-    // Until the importer can support immAllZerosV in pattern leaf nodes,
-    // select a zero move manually here.
-    Register DstReg = I.getOperand(0).getReg();
-    if (DstSize == 128) {
-      auto Mov = MIB.buildInstr(AArch64::MOVIv2d_ns, {DstReg}, {}).addImm(0);
-      I.eraseFromParent();
-      return constrainSelectedInstRegOperands(*Mov, TII, TRI, RBI);
-    } else if (DstSize == 64) {
-      auto Mov =
-          MIB.buildInstr(AArch64::MOVIv2d_ns, {&AArch64::FPR128RegClass}, {})
-              .addImm(0);
-      MIB.buildInstr(TargetOpcode::COPY, {DstReg}, {})
-          .addReg(Mov.getReg(0), 0, AArch64::dsub);
-      I.eraseFromParent();
-      return RBI.constrainGenericRegister(DstReg, AArch64::FPR64RegClass, MRI);
-    }
-  }
-  auto *CPLoad = emitLoadFromConstantPool(CV, MIB);
-  if (!CPLoad) {
-    LLVM_DEBUG(dbgs() << "Could not generate cp load for build_vector");
+  if (!emitConstantVector(I.getOperand(0).getReg(), CV, MIB, MRI))
     return false;
-  }
-  MIB.buildCopy(I.getOperand(0), CPLoad->getOperand(0));
-  RBI.constrainGenericRegister(I.getOperand(0).getReg(),
-                               *MRI.getRegClass(CPLoad->getOperand(0).getReg()),
-                               MRI);
   I.eraseFromParent();
   return true;
 }
 
 bool AArch64InstructionSelector::selectBuildVector(
     MachineInstr &I, MachineRegisterInfo &MRI) const {
   assert(I.getOpcode() == TargetOpcode::G_BUILD_VECTOR);
   // Until we port more of the optimized selections, for now just use a vector

llvm/lib/Target/AArch64/GISel/AArch64PostLegalizerLowering.cpp

 
 /// isVShiftRImm - Check if this is a valid vector for the immediate
 /// operand of a vector shift right operation. The value must be in the range:
 ///   1 <= Value <= ElementBits for a right shift.
 static bool isVShiftRImm(Register Reg, MachineRegisterInfo &MRI, LLT Ty,
                          int64_t &Cnt) {
   assert(Ty.isVector() && "vector shift count is not a vector type");
   MachineInstr *MI = MRI.getVRegDef(Reg);
-  auto Cst = getBuildVectorConstantSplat(*MI, MRI);
-  if (!Cst)
+  if (MI->getOpcode() == AArch64::G_DUP) {
+    auto VRegAndVal =
+        getConstantVRegValWithLookThrough(MI->getOperand(1).getReg(), MRI);
+    if (!VRegAndVal)
       return false;
+    Cnt = VRegAndVal->Value.getSExtValue();
+  } else if (auto Cst = getBuildVectorConstantSplat(*MI, MRI)) {
     Cnt = *Cst;
+  } else
+    return false;
   int64_t ElementBits = Ty.getScalarSizeInBits();
   return Cnt >= 1 && Cnt <= ElementBits;
 }
 
 /// Match a vector G_ASHR or G_LSHR with a valid immediate shift.
 static bool matchVAshrLshrImm(MachineInstr &MI, MachineRegisterInfo &MRI,
                               int64_t &Imm) {
   assert(MI.getOpcode() == TargetOpcode::G_ASHR ||
bool applyDupLane(MachineInstr &MI, MachineRegisterInfo &MRI,
   B.setInstrAndDebugLoc(MI);
   auto Lane = B.buildConstant(LLT::scalar(64), MatchInfo.second);
   B.buildInstr(MatchInfo.first, {MI.getOperand(0).getReg()},
                {MI.getOperand(1).getReg(), Lane});
   MI.eraseFromParent();
   return true;
 }
 
+static bool matchBuildVectorToDup(MachineInstr &MI, MachineRegisterInfo &MRI) {
+  assert(MI.getOpcode() == TargetOpcode::G_BUILD_VECTOR);
+  auto Reg = MI.getOperand(1).getReg();
+  return all_of(
+      make_range(MI.operands_begin() + 2, MI.operands_end()),
+      [&Reg](const MachineOperand &Op) { return Op.getReg() == Reg; });
+}
+
+static bool applyBuildVectorToDup(MachineInstr &MI, MachineRegisterInfo &MRI,
+                                  MachineIRBuilder &B) {
+  B.setInstrAndDebugLoc(MI);
+  B.buildInstr(AArch64::G_DUP, {MI.getOperand(0).getReg()},
+               {MI.getOperand(1).getReg()});
+  MI.eraseFromParent();
+  return true;
+}
+
 #define AARCH64POSTLEGALIZERLOWERINGHELPER_GENCOMBINERHELPER_DEPS
 #include "AArch64GenPostLegalizeGILowering.inc"
 #undef AARCH64POSTLEGALIZERLOWERINGHELPER_GENCOMBINERHELPER_DEPS
 
 namespace {
 #define AARCH64POSTLEGALIZERLOWERINGHELPER_GENCOMBINERHELPER_H
 #include "AArch64GenPostLegalizeGILowering.inc"
 #undef AARCH64POSTLEGALIZERLOWERINGHELPER_GENCOMBINERHELPER_H

llvm/test/CodeGen/AArch64/GlobalISel/postlegalizer-lowering-build-vector-to-dup.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple aarch64 -run-pass=aarch64-postlegalizer-lowering -verify-machineinstrs %s -o - | FileCheck %s
+
+...
+---
+name:            same_reg
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $d0
+    ; CHECK-LABEL: name: same_reg
+    ; CHECK: liveins: $d0
+    ; CHECK: %r:_(s8) = G_IMPLICIT_DEF
+    ; CHECK: %build_vector:_(<8 x s8>) = G_DUP %r(s8)
+    ; CHECK: $d0 = COPY %build_vector(<8 x s8>)
+    ; CHECK: RET_ReallyLR implicit $d0
+    %r:_(s8) = G_IMPLICIT_DEF
+    %build_vector:_(<8 x s8>) = G_BUILD_VECTOR %r, %r, %r, %r, %r, %r, %r, %r
+    $d0 = COPY %build_vector(<8 x s8>)
+    RET_ReallyLR implicit $d0
+
+...
+---
+name:            dont_combine_different_reg
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $d0, $w0, $w1
+    ; CHECK-LABEL: name: dont_combine_different_reg
+    ; CHECK: liveins: $d0, $w0, $w1
+    ; CHECK: %r:_(s32) = COPY $w0
+    ; CHECK: %q:_(s32) = COPY $w1
+    ; CHECK: %build_vector:_(<2 x s32>) = G_BUILD_VECTOR %r(s32), %q(s32)
+    ; CHECK: $d0 = COPY %build_vector(<2 x s32>)
+    ; CHECK: RET_ReallyLR implicit $d0
+    %r:_(s32) = COPY $w0
+    %q:_(s32) = COPY $w1
+    %build_vector:_(<2 x s32>) = G_BUILD_VECTOR %r, %q
+    $d0 = COPY %build_vector(<2 x s32>)
+    RET_ReallyLR implicit $d0

llvm/test/CodeGen/AArch64/GlobalISel/postlegalizer-lowering-vashr-vlshr.mir

 body:             |
   bb.1.entry:
     liveins: $d0, $d1
 
     ; CHECK-LABEL: name: imm_too_large
     ; CHECK: liveins: $d0, $d1
     ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $q0
     ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 40
-    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s32>) = G_BUILD_VECTOR [[C]](s32), [[C]](s32), [[C]](s32), [[C]](s32)
-    ; CHECK: [[LSHR:%[0-9]+]]:_(<4 x s32>) = G_LSHR [[COPY]], [[BUILD_VECTOR]](<4 x s32>)
+    ; CHECK: [[DUP:%[0-9]+]]:_(<4 x s32>) = G_DUP [[C]](s32)
+    ; CHECK: [[LSHR:%[0-9]+]]:_(<4 x s32>) = G_LSHR [[COPY]], [[DUP]](<4 x s32>)
     ; CHECK: $q0 = COPY [[LSHR]](<4 x s32>)
     ; CHECK: RET_ReallyLR implicit $q0
     %0:_(<4 x s32>) = COPY $q0
     %1:_(s32) = G_CONSTANT i32 40
     %2:_(<4 x s32>) = G_BUILD_VECTOR %1(s32), %1(s32), %1(s32), %1(s32)
     %3:_(<4 x s32>) = G_LSHR %0, %2(<4 x s32>)
     $q0 = COPY %3(<4 x s32>)
     RET_ReallyLR implicit $q0
 ...
 ---
 name:            imm_zero
 alignment:       4
 legalized:       true
 tracksRegLiveness: true
 body:             |
   bb.1.entry:
     liveins: $d0, $d1
 
     ; CHECK-LABEL: name: imm_zero
     ; CHECK: liveins: $d0, $d1
     ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $q0
     ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s32>) = G_BUILD_VECTOR [[C]](s32), [[C]](s32), [[C]](s32), [[C]](s32)
-    ; CHECK: [[LSHR:%[0-9]+]]:_(<4 x s32>) = G_LSHR [[COPY]], [[BUILD_VECTOR]](<4 x s32>)
+    ; CHECK: [[DUP:%[0-9]+]]:_(<4 x s32>) = G_DUP [[C]](s32)
+    ; CHECK: [[LSHR:%[0-9]+]]:_(<4 x s32>) = G_LSHR [[COPY]], [[DUP]](<4 x s32>)
     ; CHECK: $q0 = COPY [[LSHR]](<4 x s32>)
     ; CHECK: RET_ReallyLR implicit $q0
     %0:_(<4 x s32>) = COPY $q0
     %1:_(s32) = G_CONSTANT i32 0
     %2:_(<4 x s32>) = G_BUILD_VECTOR %1(s32), %1(s32), %1(s32), %1(s32)
     %3:_(<4 x s32>) = G_LSHR %0, %2(<4 x s32>)
     $q0 = COPY %3(<4 x s32>)
     RET_ReallyLR implicit $q0

llvm/test/CodeGen/AArch64/GlobalISel/select-dup.mir

bb.1:
     ; CHECK: $q0 = COPY [[DUPv2i64gpr]]
     ; CHECK: RET_ReallyLR implicit $q0
     %0:gpr(p0) = COPY $x0
     %4:fpr(<2 x p0>) = G_DUP %0(p0)
     $q0 = COPY %4(<2 x p0>)
     RET_ReallyLR implicit $q0
 
 ...
+---
+name:            zero_v4s32
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    liveins: $w0
+    ; CHECK-LABEL: name: zero_v4s32
+    ; CHECK: liveins: $w0
+    ; CHECK: %dup:fpr128 = MOVIv2d_ns 0
+    ; CHECK: $q0 = COPY %dup
+    ; CHECK: RET_ReallyLR implicit $q0
+    %zero:gpr(s32) = G_CONSTANT i32 0
+    %dup:fpr(<4 x s32>) = G_DUP %zero(s32)
+    $q0 = COPY %dup(<4 x s32>)
+    RET_ReallyLR implicit $q0
+
+...
+---
+name:            zero_v8s8
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    liveins: $w0
+    ; CHECK-LABEL: name: zero_v8s8
+    ; CHECK: liveins: $w0
+    ; CHECK: [[MOVIv2d_ns:%[0-9]+]]:fpr128 = MOVIv2d_ns 0
+    ; CHECK: %dup:fpr64 = COPY [[MOVIv2d_ns]].dsub
+    ; CHECK: $d0 = COPY %dup
+    ; CHECK: RET_ReallyLR implicit $d0
+    %zero:gpr(s8) = G_CONSTANT i8 0
+    %dup:fpr(<8 x s8>) = G_DUP %zero(s8)
+    $d0 = COPY %dup(<8 x s8>)
+    RET_ReallyLR implicit $d0
+
+...
+---
+name:            one_v4s32
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    liveins: $w0
+    ; CHECK-LABEL: name: one_v4s32
+    ; CHECK: liveins: $w0
+    ; CHECK: [[ADRP:%[0-9]+]]:gpr64common = ADRP target-flags(aarch64-page) %const.0
+    ; CHECK: [[LDRQui:%[0-9]+]]:fpr128 = LDRQui [[ADRP]], target-flags(aarch64-pageoff, aarch64-nc) %const.0
+    ; CHECK: $q0 = COPY [[LDRQui]]
+    ; CHECK: RET_ReallyLR implicit $q0
+    %zero:gpr(s32) = G_CONSTANT i32 1
+    %dup:fpr(<4 x s32>) = G_DUP %zero(s32)
+    $q0 = COPY %dup(<4 x s32>)
+    RET_ReallyLR implicit $q0

llvm/test/CodeGen/AArch64/GlobalISel/select-vector-shift.mir

bb.1:
     ; CHECK: $q0 = COPY [[USHLv16i8_]]
     ; CHECK: RET_ReallyLR implicit $q0
     %0:fpr(<16 x s8>) = COPY $q0
     %1:fpr(<16 x s8>) = COPY $q1
     %2:fpr(<16 x s8>) = G_LSHR %0, %1(<16 x s8>)
     $q0 = COPY %2(<16 x s8>)
     RET_ReallyLR implicit $q0
 ...
+---
+name:            shl_v2i32_imm_dup
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fpr }
+  - { id: 1, class: fpr }
+  - { id: 2, class: gpr }
+  - { id: 3, class: fpr }
+liveins:
+  - { reg: '$d0' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.1:
+    liveins: $d0
+
+    ; Should still be able to select immediate forms using a G_DUP from a
+    ; constant.
+
+    ; CHECK-LABEL: name: shl_v2i32_imm_dup
+    ; CHECK: liveins: $d0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK: [[SHLv2i32_shift:%[0-9]+]]:fpr64 = SHLv2i32_shift [[COPY]], 24
+    ; CHECK: $d0 = COPY [[SHLv2i32_shift]]
+    ; CHECK: RET_ReallyLR implicit $d0
+    %0:fpr(<2 x s32>) = COPY $d0
+    %2:gpr(s32) = G_CONSTANT i32 24
+    %1:fpr(<2 x s32>) = G_DUP %2(s32)
+    %3:fpr(<2 x s32>) = G_SHL %0, %1(<2 x s32>)
+    $d0 = COPY %3(<2 x s32>)
+    RET_ReallyLR implicit $d0