[GISel]: Few InsertVecElt combines

[aditya_nandakumar]

Summary

This adds the following combines that frequently occur

1. build_vector formation from insert_vectors.
2. insert_vec_elt(build_vector) -> build_vector

WRT (1), a few more cases need to be handled (such as handling multiple insert_vec_elts to the same index) and I'll try to add them shortly after this.

Also adds pattern matching for insert_vec_elts.

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2425–2426

Should guard against UB out of bounds accesses

2433

BitVector?

llvm/test/CodeGen/AArch64/GlobalISel/combine-insert-vec-elt.mir

43

Should add test with out of bounds constant index

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2425–2426

Not actually UB, but undef

[aditya_nandakumar]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2425–2426

Just to confirm, you mean wrt index(IntImm here)?

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2425–2426

Yes, index >= NumElts

[aditya_nandakumar]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2433

Removed this. This is not used in this change, but will be in the subsequent one.

[aditya_nandakumar]

Updated to abandon combine if index is out of bounds.

[arsenm]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

419

SmallVectorImpl for all of these?

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2428

emplace_back?

2432

s/no/number. Couldn't this add undefs for the missing elements?

[aditya_nandakumar]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2432

I was planning on handling the duplicate as well as missing cases in a follow up commit.

[aditya_nandakumar]

Updated to handle the missing index as well duplicate indices cases.

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2415

This could start at any index? Why give up at NumElts - 1?

2437

llvm::sort

2449

Optional doesn't really buy you anything here? You can just use Register()?

2453

I don't think you need to reset the insert point here

[aditya_nandakumar]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2415

This is poorly attempting to match the highest index (assuming insert_vec_elts are found with increasing indices) so we can collect as many components as possible.
For eg if we traverse top down (like we currently do)

a = insert_vec_elt undef, x, 0
b = insert_vec_elt a, y, 1

We'll turn a into

a = build_vector x, undef
b = insert_vec_elt a, y, 1

Subsequently the insert_vec_elt(build_vec) combine would turn this into

b = build_vector x, y

Instead starting with b would give the build_vector in one shot.

Maybe the better option to do is check if the current insert_vec_elt is used by insert_vec_elts and bail out.

2437

Will do.

2449

Will update.

2453

I'm not resetting. I'm making sure it's set correctly (MI) so I can insert IMPLICIT_DEFs.

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2453

Oh, actually this is modifying the MIR in the match function which seems like a problem. Can you defer this to apply?

[aditya_nandakumar]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2453

Updated.

[aditya_nandakumar]

Insert undef during the apply phase instead of the match as mentioned in the feedback.

[aditya_nandakumar]

s/std::sort/llvm::sort

[aditya_nandakumar]

Also missed some unstaged changes locally
s/push_back/emplace_back

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2415

It looks like the DAG handles this by separately canonicalizing the order of a pair of insert_vector_elts (it also seems to produce build_vector one element at a time, but I guess it makes sense to reduce the number of steps here)

2470

Braces

[aditya_nandakumar]

Address latest comments by Matt

[foad]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2413

Everything after this point looks way overcomplicated and could be replaced with something like:

MatchInfo.resize(NumElts);
while (mi_match(
    CurrInst->getOperand(0).getReg(), MRI,
    m_GInsertVecElt(m_MInstr(TmpInst), m_Reg(TmpReg), m_ICst(IntImm)))) {
  if (IntImm >= NumElts)
    return false;
  if (!MatchInfo[IntImm])
    MatchInfo[IntImm] = TmpReg;
  CurrInst = TmpInst;
}
return TmpInst->getOpcode() == TargetOpcode::G_IMPLICIT_DEF;

2491

Don't you have to defend against *IdxCst being out of range here too?

[foad]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2415

Maybe the better option to do is check if the current insert_vec_elt is used by insert_vec_elts and bail out.

Right, I think checking whether the current MI has a single use that is another eligible INSERT_VECTOR_ELT would work.

[foad]

Reverse ping! If you just don't have time to work on this, I'd be interested in commandeering the "insert_vec_elt(build_vector) -> build_vector" part.

[aditya_nandakumar]

In D88060#2329690, @foad wrote:

Reverse ping! If you just don't have time to work on this, I'd be interested in commandeering the "insert_vec_elt(build_vector) -> build_vector" part.

Sorry - your previous comment got lost in my mailbox. Update coming up.

[aditya_nandakumar]

Address Jay's comments.

[foad]

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

400

"Ternary" is the normal word for this.

[aditya_nandakumar]

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

400

Can do. Anything else besides this?

[foad]

llvm/include/llvm/Target/GlobalISel/Combine.td

496

Line length.

524

Line length.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2434

It seems like it would be very easy to handle G_BUILD_VECTOR here as well as G_IMPLICIT_DEF. Then it will handle a chain of G_INSERT_VECTOR_ELT ending in a G_BUILD_VECTOR, with no need for a separate match function. Win-win?

[aditya_nandakumar]

llvm/include/llvm/Target/GlobalISel/Combine.td

496

Can do.

524

Can do.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2434

While it's easy to do, I think it's important to not cram too many combines into one combine. IIUC, the goal with the current table gen wrapper/split is for us to be able to declare/express the same combines/matching ability with the new Combiner table gen framework. So conceptually we probably want to keep them separate for more match function/expressibility test cases for the new combiner (I guess how to separate is quite arbitrary at this point).
TLDR; I can certainly update this, but do we want to cram many peephole combines into one?

[foad]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2434

I see it as logically a single combine, which happens to handle "undef" sources as well as real build_vectors. I can't imagine why you'd want to split it into two and have it run twice as slow.

[aditya_nandakumar]

Updates as requested.

[aditya_nandakumar]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2434

In general it's possible to merge multiple combines into one c++ function and make the implementation simpler. However what's logically together is subjective.
In any case, I've updated as requested so we can get this in.

[foad]

LGTM, thanks! Just a load of optional nits inline.

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

421

Nit: make the "match" and "apply" function names match, now that they are one-to-one?

llvm/include/llvm/Target/GlobalISel/Combine.td

483

Nit: since you have given this a generic name, there are probably other places in this file that could make use of it instead of defining their own matchdata.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2412

Nit: the assert seems redundant since getNumElements will do it for you.

2429

Nit: I don't think you even need the .isValid() here, do you?

2434

Agreed it's subjective, but for compile time reasons I prefer to simplify the implementation unless there's a really good conceptual reason to separate it.

2451

Nit: could inline this into its only caller.

2459

Nit: don't need the .isValid() ?

[aditya_nandakumar]

rGbed839404784

llvm/include/llvm/Target/GlobalISel/Combine.td

483

I'll clean them all up in a follow up NFC commit.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2451

This attempts to just cache UNDEFs without relying on CSE mechanism.

[foad]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2451

I'm saying the function (lambda) GetUndef is only caled once, so there's no real need for it to be a separate function.

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

public:
 
   /// \return true if \p MI is a G_SEXT_INREG that can be erased.
   bool matchRedundantSExtInReg(MachineInstr &MI);
 
   /// Combine inverting a result of a compare into the opposite cond code.
   bool matchNotCmp(MachineInstr &MI, SmallVectorImpl<Register> &RegsToNegate);
   bool applyNotCmp(MachineInstr &MI, SmallVectorImpl<Register> &RegsToNegate);
 
+  bool matchConsecutiveInsertVecElts(MachineInstr &MI,
+                                     SmallVector<Register, 4> &MatchInfo);
+
+  bool applyBuildVecFromRegs(MachineInstr &MI,
+                             SmallVector<Register, 4> &MatchInfo);
+
+  bool matchInsertVecEltBuildVec(MachineInstr &MI,
+                                 SmallVector<Register, 4> &MatchInfo);
   /// Try to transform \p MI by using all of the above
   /// combine functions. Returns true if changed.
   bool tryCombine(MachineInstr &MI);
 
 private:
   // Memcpy family optimization helpers.
   bool optimizeMemcpy(MachineInstr &MI, Register Dst, Register Src,
                       unsigned KnownLen, Align DstAlign, Align SrcAlign,

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

struct CheckType {
 
   bool match(const MachineRegisterInfo &MRI, Register Reg) {
     return MRI.getType(Reg) == Ty;
   }
 };
 
 inline CheckType m_SpecificType(LLT Ty) { return Ty; }
 
+template <typename Src0Ty, typename Src1Ty, typename Src2Ty, unsigned Opcode>
+struct TrinaryOp_match {
+  Src0Ty Src0;
+  Src1Ty Src1;
+  Src2Ty Src2;
+
+  TrinaryOp_match(const Src0Ty &Src0, const Src1Ty &Src1, const Src2Ty &Src2)
+      : Src0(Src0), Src1(Src1), Src2(Src2) {}
+  template <typename OpTy>
+  bool match(const MachineRegisterInfo &MRI, OpTy &&Op) {
+    MachineInstr *TmpMI;
+    if (mi_match(Op, MRI, m_MInstr(TmpMI))) {
+      if (TmpMI->getOpcode() == Opcode && TmpMI->getNumOperands() == 4) {
+        return (Src0.match(MRI, TmpMI->getOperand(1).getReg()) &&
+                Src1.match(MRI, TmpMI->getOperand(2).getReg()) &&
+                Src2.match(MRI, TmpMI->getOperand(3).getReg()));
+      }
+    }
+    return false;
+  }
+};
+template <typename Src0Ty, typename Src1Ty, typename Src2Ty>
+inline TrinaryOp_match<Src0Ty, Src1Ty, Src2Ty,
+                       TargetOpcode::G_INSERT_VECTOR_ELT>
+m_GInsertVecElt(const Src0Ty &Src0, const Src1Ty &Src1, const Src2Ty &Src2) {
+  return TrinaryOp_match<Src0Ty, Src1Ty, Src2Ty,
+                         TargetOpcode::G_INSERT_VECTOR_ELT>(Src0, Src1, Src2);
+}
+
 } // namespace GMIPatternMatch
 } // namespace llvm
 
 #endif

llvm/include/llvm/Target/GlobalISel/Combine.td

 // Transform (mul x, -1) -> (sub 0, x)
 def mul_by_neg_one: GICombineRule <
   (defs root:$root),
   (match (wip_match_opcode G_MUL):$root,
          [{ return Helper.matchConstantOp(${root}->getOperand(2), -1); }]),
   (apply [{ return Helper.applyCombineMulByNegativeOne(*${root}); }])
 >;
 
+def regs_small_vec : GIDefMatchData<"SmallVector<Register, 4>">;
+def combine_consecutive_insert_vec_elts : GICombineRule<
+  (defs root:$root, regs_small_vec:$info),
+  (match (wip_match_opcode G_INSERT_VECTOR_ELT):$root,
+    [{ return Helper.matchConsecutiveInsertVecElts(*${root}, ${info}); }]),
+  (apply [{ return Helper.applyBuildVecFromRegs(*${root}, ${info}); }])>;
+
+def combine_ins_vec_build_vec : GICombineRule<
+  (defs root:$root, regs_small_vec:$info),
+  (match (wip_match_opcode G_INSERT_VECTOR_ELT):$root,
+    [{ return Helper.matchInsertVecEltBuildVec(*${root}, ${info}); }]),
+  (apply [{ return Helper.applyBuildVecFromRegs(*${root}, ${info}); }])>;
+
+def insert_vec_elt_combines : GICombineGroup<[combine_consecutive_insert_vec_elts,
+                                              combine_ins_vec_build_vec]>;
+
 // FIXME: These should use the custom predicate feature once it lands.
 def undef_combines : GICombineGroup<[undef_to_fp_zero, undef_to_int_zero,
                                      undef_to_negative_one,
                                      binop_left_undef_to_zero,
                                      propagate_undef_any_op,
                                      propagate_undef_all_ops,
                                      propagate_undef_shuffle_mask,
                                      erase_undef_store]>;

 
 def width_reduction_combines : GICombineGroup<[reduce_shl_of_extend]>;
 
 def select_combines : GICombineGroup<[select_undef_cmp, select_constant_cmp]>;
 
 def trivial_combines : GICombineGroup<[copy_prop, mul_to_shl, add_p2i_to_ptradd,
                                        mul_by_neg_one]>;
 
-def all_combines : GICombineGroup<[trivial_combines, ptr_add_immed_chain,
+def all_combines : GICombineGroup<[trivial_combines, insert_vec_elt_combines, ptr_add_immed_chain,
     combines_for_extload, combine_indexed_load_store, undef_combines,
     identity_combines, simplify_add_to_sub,
     hoist_logic_op_with_same_opcode_hands,
     shl_ashr_to_sext_inreg, sext_inreg_of_load,
     width_reduction_combines, select_combines,
     known_bits_simplifications, ext_ext_fold,
     not_cmp_fold, opt_brcond_by_inverting_cond,
     unmerge_merge, fabs_fabs_fold, unmerge_cst, unmerge_dead_to_trunc,
     unmerge_zext_to_zext, trunc_ext_fold, trunc_shl,
     constant_fp_op]>;

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

if (!mi_match(MaybeSub, MRI, m_GSub(m_ICst(Cst), m_Reg(NewRHS))) ||
       return false;
     NewLHS = MaybeNewLHS;
     return true;
   };
 
   return CheckFold(LHS, RHS) || CheckFold(RHS, LHS);
 }
 
+bool CombinerHelper::matchConsecutiveInsertVecElts(
+    MachineInstr &MI, SmallVector<Register, 4> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_INSERT_VECTOR_ELT &&
+         "Invalid opcode");
+  Register DstReg = MI.getOperand(0).getReg();
+  LLT DstTy = MRI.getType(DstReg);
+  assert(DstTy.isVector() && "Invalid G_INSERT_VECTOR_ELT?");
+  unsigned NumElts = DstTy.getNumElements();
+  auto Index = getConstantVRegVal(MI.getOperand(3).getReg(), MRI);
+  if (!Index || *Index != (NumElts - 1))
+    return false;
+  SmallVector<std::pair<Register, unsigned>, 4> Inserts;
+  // Try to collect various InsertElts.
+  MachineInstr *CurrInst = &MI;
+  MachineInstr *TmpInst;
+  int64_t IntImm;
+  Register TmpReg;
+  while (mi_match(
+      CurrInst->getOperand(0).getReg(), MRI,
+      m_GInsertVecElt(m_MInstr(TmpInst), m_Reg(TmpReg), m_ICst(IntImm)))) {
+    Inserts.push_back(std::make_pair(TmpReg, IntImm));
+    CurrInst = TmpInst;
+  }
+
+  // Make sure we have correct no of elements.
+  if (Inserts.size() != NumElts)
+    return false;
+  SmallVector<bool, 4> SeenIndex(NumElts, false);
+  std::sort(Inserts.begin(), Inserts.end(),
+            [](const std::pair<Register, unsigned> &P1,
+               const std::pair<Register, unsigned> &P2) {
+              return P1.second < P2.second;
+            });
+  for (unsigned i = 0; i < NumElts; ++i) {
+    if (Inserts[i].second != i)
+      return false;
+  }
+  for (auto &P : Inserts)
+    MatchInfo.push_back(P.first);
+  return true;
+}
+
+bool CombinerHelper::applyBuildVecFromRegs(
+    MachineInstr &MI, SmallVector<Register, 4> &MatchInfo) {
+  Builder.setInstr(MI);
+  Builder.buildBuildVector(MI.getOperand(0).getReg(), MatchInfo);
+  MI.eraseFromParent();
+  return true;
+}
+
+bool CombinerHelper::matchInsertVecEltBuildVec(
+    MachineInstr &MI, SmallVector<Register, 4> &MatchInfo) {
+  Register SrcVecReg = MI.getOperand(1).getReg();
+  MachineInstr *SrcMI = MRI.getVRegDef(SrcVecReg);
+  Register EltReg = MI.getOperand(2).getReg();
+  Register IdxReg = MI.getOperand(3).getReg();
+  auto IdxCst = getConstantVRegVal(IdxReg, MRI);
+  if (!IdxCst)
+    return false;
+  if (SrcMI->getOpcode() == TargetOpcode::G_BUILD_VECTOR) {
+    for (auto &Op : SrcMI->uses())
+      MatchInfo.push_back(Op.getReg());
+    MatchInfo[*IdxCst] = EltReg;
+    return true;
+  }
+  return false;
+}
+
 bool CombinerHelper::applySimplifyAddToSub(
     MachineInstr &MI, std::tuple<Register, Register> &MatchInfo) {
   Builder.setInstr(MI);
   Register SubLHS, SubRHS;
   std::tie(SubLHS, SubRHS) = MatchInfo;
   Builder.buildSub(MI.getOperand(0).getReg(), SubLHS, SubRHS);
   MI.eraseFromParent();
   return true;
 }
 
 bool CombinerHelper::matchHoistLogicOpWithSameOpcodeHands(
     MachineInstr &MI, InstructionStepsMatchInfo &MatchInfo) {
   // Matches: logic (hand x, ...), (hand y, ...) -> hand (logic x, y), ...
   //
   // Creates the new hand + logic instruction (but does not insert them.)
   //
   // On success, MatchInfo is populated with the new instructions. These are
   // inserted in applyHoistLogicOpWithSameOpcodeHands.
   unsigned LogicOpcode = MI.getOpcode();
   assert(LogicOpcode == TargetOpcode::G_AND ||
          LogicOpcode == TargetOpcode::G_OR ||
          LogicOpcode == TargetOpcode::G_XOR);
   MachineIRBuilder MIB(MI);
   Register Dst = MI.getOperand(0).getReg();
   Register LHSReg = MI.getOperand(1).getReg();
   Register RHSReg = MI.getOperand(2).getReg();

llvm/test/CodeGen/AArch64/GlobalISel/combine-insert-vec-elt.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -o - -march=aarch64 -run-pass=aarch64-prelegalizer-combiner %s | FileCheck %s
+---
+name:            test_combine_consecutive
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+
+    ; CHECK-LABEL: name: test_combine_consecutive
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $w1
+    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32)
+    ; CHECK: $x0 = COPY [[BUILD_VECTOR]](<2 x s32>)
+    %0:_(s32) = COPY $w0
+    %1:_(s32) = COPY $w1
+    %2:_(<2 x s32>) = G_IMPLICIT_DEF
+    %7:_(s32) = G_CONSTANT i32 0
+    %8:_(s32) = G_CONSTANT i32 1
+    %3:_(<2 x s32>) = G_INSERT_VECTOR_ELT %2, %0(s32), %7(s32)
+    %4:_(<2 x s32>) = G_INSERT_VECTOR_ELT %3, %1(s32), %8(s32)
+    $x0 = COPY %4
+...
+---
+name:            test_combine_insert_vec_build_vec_idx_1
+body:             |
+  bb.1:
+    liveins: $w0, $w1, $w2, $w3
+
+    ; CHECK-LABEL: name: test_combine_insert_vec_build_vec_idx_1
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $w2
+    ; CHECK: [[COPY2:%[0-9]+]]:_(s32) = COPY $w3
+    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32)
+    ; CHECK: $q0 = COPY [[BUILD_VECTOR]](<4 x s32>)
+    %0:_(s32) = COPY $w0
+    %1:_(s32) = COPY $w1
+    %6:_(s32) = COPY $w2
+    %7:_(s32) = COPY $w3
+    %2:_(<4 x s32>) = G_BUILD_VECTOR %0, %1, %6, %7
+    %3:_(s32) = G_CONSTANT i32 1
+    %4:_(<4 x s32>) = G_INSERT_VECTOR_ELT %2, %0(s32), %3(s32)
+    $q0 = COPY %4
+...

llvm/test/CodeGen/AArch64/combine-loads.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -O0 -mtriple=aarch64-unknown-unknown | FileCheck %s
 
 define <2 x i64> @z(i64* nocapture nonnull readonly %p) {
 ; CHECK-LABEL: z:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    adrp x8, .LCPI0_0
-; CHECK-NEXT:    ldr q0, [x8, :lo12:.LCPI0_0]
 ; CHECK-NEXT:    ldr x9, [x0]
 ; CHECK-NEXT:    ldr x8, [x0, #8]
-; CHECK-NEXT:    mov v0.d[0], x9
+; CHECK-NEXT:    // implicit-def: $q0
+; CHECK-NEXT:    fmov d0, x9
 ; CHECK-NEXT:    mov v0.d[1], x8
 ; CHECK-NEXT:    ret
   %b = load i64, i64* %p
   %p2 = getelementptr i64, i64* %p, i64 1
   %bb = load i64, i64* %p2
   %r1 = insertelement <2 x i64> zeroinitializer, i64 %b, i32 0
   %r2 = insertelement <2 x i64> %r1, i64 %bb, i32 1
   ret <2 x i64> %r2
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/udiv.i64.ll

 ; GISEL-NEXT:    v_cndmask_b32_e32 v1, v6, v8, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e64 v3, v7, v3, s[4:5]
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_udiv_v2i64_24bit:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    s_mov_b32 s6, 0xffffff
-; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:    v_and_b32_e32 v0, s6, v0
-; CGP-NEXT:    v_and_b32_e32 v2, s6, v2
-; CGP-NEXT:    v_and_b32_e32 v3, s6, v4
-; CGP-NEXT:    v_and_b32_e32 v4, s6, v6
+; CGP-NEXT:    v_and_b32_e32 v1, s6, v2
+; CGP-NEXT:    v_and_b32_e32 v2, s6, v4
+; CGP-NEXT:    v_and_b32_e32 v3, s6, v6
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v0
-; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v3
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v2
-; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v4
+; CGP-NEXT:    v_cvt_f32_u32_e32 v1, v1
+; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v3
+; CGP-NEXT:    v_rcp_f32_e32 v4, v2
 ; CGP-NEXT:    v_rcp_f32_e32 v5, v3
-; CGP-NEXT:    v_rcp_f32_e32 v6, v4
-; CGP-NEXT:    v_mul_f32_e32 v5, v0, v5
-; CGP-NEXT:    v_mul_f32_e32 v6, v2, v6
+; CGP-NEXT:    v_mul_f32_e32 v4, v0, v4
+; CGP-NEXT:    v_mul_f32_e32 v5, v1, v5
+; CGP-NEXT:    v_trunc_f32_e32 v4, v4
 ; CGP-NEXT:    v_trunc_f32_e32 v5, v5
-; CGP-NEXT:    v_trunc_f32_e32 v6, v6
-; CGP-NEXT:    v_mad_f32 v0, -v5, v3, v0
+; CGP-NEXT:    v_mad_f32 v0, -v4, v2, v0
+; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
+; CGP-NEXT:    v_mad_f32 v1, -v5, v3, v1
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v5, v5
-; CGP-NEXT:    v_mad_f32 v2, -v6, v4, v2
-; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
-; CGP-NEXT:    v_cmp_ge_f32_e64 s[4:5], |v0|, v3
+; CGP-NEXT:    v_cmp_ge_f32_e64 s[4:5], |v0|, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s[4:5]
-; CGP-NEXT:    v_cmp_ge_f32_e64 s[4:5], |v2|, v4
-; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[4:5]
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v6, v2
+; CGP-NEXT:    v_cmp_ge_f32_e64 s[4:5], |v1|, v3
+; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[4:5]
+; CGP-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
 ; CGP-NEXT:    v_and_b32_e32 v0, s6, v0
-; CGP-NEXT:    v_and_b32_e32 v2, s6, v2
-; CGP-NEXT:    v_mov_b32_e32 v3, v1
+; CGP-NEXT:    v_and_b32_e32 v2, s6, v1
+; CGP-NEXT:    v_mov_b32_e32 v1, 0
+; CGP-NEXT:    v_mov_b32_e32 v3, 0
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %num.mask = and <2 x i64> %num, <i64 16777215, i64 16777215>
   %den.mask = and <2 x i64> %den, <i64 16777215, i64 16777215>
   %result = udiv <2 x i64> %num.mask, %den.mask
   ret <2 x i64> %result
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/urem.i64.ll

 ; GISEL-NEXT:    v_cndmask_b32_e32 v1, v6, v1, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e64 v3, v8, v3, s[4:5]
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_urem_v2i64_24bit:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    s_mov_b32 s6, 0xffffff
-; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:    v_and_b32_e32 v0, s6, v0
-; CGP-NEXT:    v_and_b32_e32 v2, s6, v2
-; CGP-NEXT:    v_and_b32_e32 v3, s6, v4
-; CGP-NEXT:    v_and_b32_e32 v4, s6, v6
-; CGP-NEXT:    v_cvt_f32_u32_e32 v5, v0
-; CGP-NEXT:    v_cvt_f32_u32_e32 v6, v3
-; CGP-NEXT:    v_cvt_f32_u32_e32 v7, v2
-; CGP-NEXT:    v_cvt_f32_u32_e32 v8, v4
-; CGP-NEXT:    v_rcp_f32_e32 v9, v6
-; CGP-NEXT:    v_rcp_f32_e32 v10, v8
-; CGP-NEXT:    v_mul_f32_e32 v9, v5, v9
-; CGP-NEXT:    v_mul_f32_e32 v10, v7, v10
+; CGP-NEXT:    v_and_b32_e32 v1, s6, v2
+; CGP-NEXT:    v_and_b32_e32 v2, s6, v4
+; CGP-NEXT:    v_and_b32_e32 v3, s6, v6
+; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v0
+; CGP-NEXT:    v_cvt_f32_u32_e32 v5, v2
+; CGP-NEXT:    v_cvt_f32_u32_e32 v6, v1
+; CGP-NEXT:    v_cvt_f32_u32_e32 v7, v3
+; CGP-NEXT:    v_rcp_f32_e32 v8, v5
+; CGP-NEXT:    v_rcp_f32_e32 v9, v7
+; CGP-NEXT:    v_mul_f32_e32 v8, v4, v8
+; CGP-NEXT:    v_mul_f32_e32 v9, v6, v9
+; CGP-NEXT:    v_trunc_f32_e32 v8, v8
 ; CGP-NEXT:    v_trunc_f32_e32 v9, v9
-; CGP-NEXT:    v_trunc_f32_e32 v10, v10
-; CGP-NEXT:    v_mad_f32 v5, -v9, v6, v5
+; CGP-NEXT:    v_mad_f32 v4, -v8, v5, v4
+; CGP-NEXT:    v_cvt_u32_f32_e32 v8, v8
+; CGP-NEXT:    v_mad_f32 v6, -v9, v7, v6
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v9, v9
-; CGP-NEXT:    v_mad_f32 v7, -v10, v8, v7
-; CGP-NEXT:    v_cvt_u32_f32_e32 v10, v10
-; CGP-NEXT:    v_cmp_ge_f32_e64 s[4:5], |v5|, v6
+; CGP-NEXT:    v_cmp_ge_f32_e64 s[4:5], |v4|, v5
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, s[4:5]
+; CGP-NEXT:    v_cmp_ge_f32_e64 s[4:5], |v6|, v7
 ; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s[4:5]
-; CGP-NEXT:    v_cmp_ge_f32_e64 s[4:5], |v7|, v8
-; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, s[4:5]
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v9, v5
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, v10, v6
+; CGP-NEXT:    v_mul_lo_u32 v2, v4, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v5, v3
-; CGP-NEXT:    v_mul_lo_u32 v4, v6, v4
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v4
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
+; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v3
 ; CGP-NEXT:    v_and_b32_e32 v0, s6, v0
-; CGP-NEXT:    v_and_b32_e32 v2, s6, v2
-; CGP-NEXT:    v_mov_b32_e32 v3, v1
+; CGP-NEXT:    v_and_b32_e32 v2, s6, v1
+; CGP-NEXT:    v_mov_b32_e32 v1, 0
+; CGP-NEXT:    v_mov_b32_e32 v3, 0
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %num.mask = and <2 x i64> %num, <i64 16777215, i64 16777215>
   %den.mask = and <2 x i64> %den, <i64 16777215, i64 16777215>
   %result = urem <2 x i64> %num.mask, %den.mask
   ret <2 x i64> %result
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/xnor.ll

; GFX906-NEXT:    ; return to shader part epilog
   %ret = xor <4 x i16> %xor, <i16 -1, i16 -1, i16 -1, i16 -1>
   %cast = bitcast <4 x i16> %ret to i64
   ret i64 %cast
 }
 
 define amdgpu_ps <2 x i64> @scalar_xnor_i64_mul_use(i64 inreg %a, i64 inreg %b) {
 ; GCN-LABEL: scalar_xnor_i64_mul_use:
 ; GCN:       ; %bb.0:
-; GCN-NEXT:    s_mov_b32 s4, s0
-; GCN-NEXT:    s_mov_b32 s5, s1
-; GCN-NEXT:    s_xor_b64 s[2:3], s[4:5], s[2:3]
-; GCN-NEXT:    s_not_b64 s[0:1], s[2:3]
-; GCN-NEXT:    s_add_u32 s2, s2, s4
-; GCN-NEXT:    s_cselect_b32 s4, 1, 0
-; GCN-NEXT:    s_and_b32 s4, s4, 1
-; GCN-NEXT:    s_cmp_lg_u32 s4, 0
-; GCN-NEXT:    s_addc_u32 s3, s3, s5
+; GCN-NEXT:    s_xor_b64 s[2:3], s[0:1], s[2:3]
+; GCN-NEXT:    s_not_b64 s[4:5], s[2:3]
+; GCN-NEXT:    s_add_u32 s2, s2, s0
+; GCN-NEXT:    s_cselect_b32 s0, 1, 0
+; GCN-NEXT:    s_and_b32 s0, s0, 1
+; GCN-NEXT:    s_cmp_lg_u32 s0, 0
+; GCN-NEXT:    s_addc_u32 s3, s3, s1
+; GCN-NEXT:    s_mov_b32 s0, s4
+; GCN-NEXT:    s_mov_b32 s1, s5
 ; GCN-NEXT:    ; return to shader part epilog
   %xor = xor i64 %a, %b
   %r0.val = xor i64 %xor, -1
   %r1.val = add i64 %xor, %a
   %ins0 = insertelement <2 x i64> undef, i64 %r0.val, i32 0
   %ins1 = insertelement <2 x i64> %ins0, i64 %r1.val, i32 1
   ret <2 x i64> %ins1
 }