[DAGCombiner][RISCV][AMDGPU] Call SimplifyDemandedBits at the end of visitMULHU to enable known bits contant folding.

[craig.topper]

Summary

We don't have real demanded bits support for MULHU, but we can
still use the known bits based constant folding support at the end
of SimplifyDemandedBits to simplify a MULHU. This helps with cases
where we know the LHS and RHS have enough leading zeros so that
the high multiply result is always 0.

[craig.topper]

llvm/test/CodeGen/AMDGPU/sdiv64.ll

1397–1398

This also points to a failure to canonicalize constants to the RHS so the isNullValue check in visitMULHU would work. I tried to add the canonicalization without this patch, but ended up with a verifier failure on some AMDGPU tests.

It appears the simplification introduced in this patch catches something even earlier and produces simpler code that doesn't hit the verifier error.

[RKSimon]

llvm/test/CodeGen/AMDGPU/sdiv64.ll

1397–1398

@foad @arsenm - any ideas?

*** Bad machine code: VOP* instruction violates constant bus restriction ***
- function:    v_test_sdiv_k_num_i64
- basic block: %bb.0  (0x238f3f455f0)
- instruction: %160:vgpr_32 = V_ADDC_U32_e32 %70:sreg_32, %161:vgpr_32, implicit-def dead $vcc, implicit $vcc, implicit $exec

[critson]

llvm/test/CodeGen/AMDGPU/sdiv64.ll

1397–1398

There appears to be a verifier bug, where VCC is being counted toward constant bus usage for V_ADDC (where it is implicit).
The instruction is legal.

[critson]

llvm/test/CodeGen/AMDGPU/sdiv64.ll

1397–1398

Please ignore me - I misread the documentation.
The verifier is correct.

[foad]

llvm/test/CodeGen/AMDGPU/sdiv64.ll

1397–1398

How did you "add the canonicalization"? Have you got a patch for that?

[RKSimon]

llvm/test/CodeGen/AMDGPU/sdiv64.ll

1397–1398

Yes, I'll dig it out and post it on a bug

[RKSimon]

llvm/test/CodeGen/AMDGPU/sdiv64.ll

1397–1398

https://bugs.llvm.org/show_bug.cgi?id=51217

[RKSimon]

@craig.topper once D106868 goes in, are you happy to add the canonicalization of mulh constants to rhs (from PR51217) first?

[craig.topper]

In D106471#2907860, @RKSimon wrote:

@craig.topper once D106868 goes in, are you happy to add the canonicalization of mulh constants to rhs (from PR51217) first?

Yes we can add the canonicalization first.

[craig.topper]

Rebase

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

4582

Out of interest, do we need MULHU support in SimplifyDemandedBits?

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

4582

I'm not sure there's much you can do. If you demand any of the bits from MULHU, then I think you demand all bits of the input. Maybe there's something you can do if you have known bits from one input, but I'd need to think about it a lot more.

[RKSimon]

LGTM - cheers

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

4582

Since a common reason for creating these nodes is divide by constants (TargetLowering.BuildUDIV et al) then RHS is likely to be constant - but yes this should probably wait until we have some actual examples.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (TLI.isOperationLegal(ISD::MUL, NewVT)) {
       N1 = DAG.getNode(ISD::MUL, DL, NewVT, N0, N1);
       N1 = DAG.getNode(ISD::SRL, DL, NewVT, N1,
             DAG.getConstant(SimpleSize, DL,
                             getShiftAmountTy(N1.getValueType())));
       return DAG.getNode(ISD::TRUNCATE, DL, VT, N1);
     }
   }
 
+  // Simplify the operands using demanded-bits information.
+  // We don't have demanded bits support for MULHU so this just enables constant
+  // folding based on known bits.
+  if (SimplifyDemandedBits(SDValue(N, 0)))
+    return SDValue(N, 0);
+
   return SDValue();
 }
 
 /// Perform optimizations common to nodes that compute two values. LoOp and HiOp
 /// give the opcodes for the two computations that are being performed. Return
 /// true if a simplification was made.
 SDValue DAGCombiner::SimplifyNodeWithTwoResults(SDNode *N, unsigned LoOp,
                                                 unsigned HiOp) {

llvm/test/CodeGen/AMDGPU/sdiv64.ll

 ; GCN-NEXT:    s_ashr_i32 s2, s7, 31
 ; GCN-NEXT:    s_add_u32 s0, s6, s2
 ; GCN-NEXT:    s_addc_u32 s1, s7, s2
 ; GCN-NEXT:    s_mov_b32 s3, s2
 ; GCN-NEXT:    s_xor_b64 s[8:9], s[0:1], s[2:3]
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v0, s8
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v1, s9
 ; GCN-NEXT:    s_sub_u32 s3, 0, s8
-; GCN-NEXT:    s_subb_u32 s10, 0, s9
+; GCN-NEXT:    s_subb_u32 s6, 0, s9
 ; GCN-NEXT:    s_mov_b32 s7, 0xf000
 ; GCN-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; GCN-NEXT:    v_rcp_f32_e32 v0, v0
 ; GCN-NEXT:    v_mov_b32_e32 v1, 0
-; GCN-NEXT:    s_mov_b32 s6, -1
 ; GCN-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; GCN-NEXT:    v_mul_f32_e32 v3, 0x2f800000, v0
 ; GCN-NEXT:    v_trunc_f32_e32 v3, v3
 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v3
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v3, v3
 ; GCN-NEXT:    v_mul_hi_u32 v5, s3, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s3, v3
-; GCN-NEXT:    v_mul_lo_u32 v7, s10, v0
+; GCN-NEXT:    v_mul_lo_u32 v7, s6, v0
 ; GCN-NEXT:    v_mul_lo_u32 v6, s3, v0
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GCN-NEXT:    v_mul_hi_u32 v5, v0, v6
 ; GCN-NEXT:    v_mul_lo_u32 v7, v0, v4
 ; GCN-NEXT:    v_mul_hi_u32 v9, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v8, v3, v6
 ; GCN-NEXT:    v_mul_hi_u32 v6, v3, v6
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; GCN-NEXT:    v_mul_hi_u32 v10, v3, v4
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, v2, v9, vcc
-; GCN-NEXT:    v_mul_hi_u32 v9, v3, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v3, v4
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v8
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, v7, v6, vcc
-; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v9, v1, vcc
+; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v10, v1, vcc
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GCN-NEXT:    v_add_i32_e64 v0, s[0:1], v0, v4
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, v2, v6, vcc
 ; GCN-NEXT:    v_addc_u32_e64 v4, vcc, v3, v5, s[0:1]
 ; GCN-NEXT:    v_mul_lo_u32 v6, s3, v4
 ; GCN-NEXT:    v_mul_hi_u32 v7, s3, v0
-; GCN-NEXT:    v_mul_lo_u32 v8, s10, v0
+; GCN-NEXT:    v_mul_lo_u32 v8, s6, v0
+; GCN-NEXT:    s_mov_b32 s6, -1
 ; GCN-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GCN-NEXT:    v_mul_lo_u32 v7, s3, v0
 ; GCN-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
 ; GCN-NEXT:    v_mul_lo_u32 v10, v0, v6
 ; GCN-NEXT:    v_mul_hi_u32 v12, v0, v6
 ; GCN-NEXT:    v_mul_hi_u32 v11, v0, v7
 ; GCN-NEXT:    v_mul_hi_u32 v9, v4, v7
 ; GCN-NEXT:    v_mul_lo_u32 v7, v4, v7
 ; GCN-NEXT:    v_mul_hi_u32 v8, v4, v6
 ; GCN-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
 ; GCN-NEXT:    v_addc_u32_e32 v11, vcc, v2, v12, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v4, v4, v6
 ; GCN-NEXT:    v_add_i32_e32 v7, vcc, v10, v7
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, v11, v9, vcc
-; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v8, v1, vcc
+; GCN-NEXT:    v_addc_u32_e32 v1, vcc, v8, v1, vcc
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v7, v4
-; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v2, v6, vcc
+; GCN-NEXT:    v_addc_u32_e32 v1, vcc, v2, v1, vcc
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
-; GCN-NEXT:    v_addc_u32_e64 v3, vcc, v3, v6, s[0:1]
+; GCN-NEXT:    v_addc_u32_e64 v1, vcc, v3, v1, s[0:1]
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
-; GCN-NEXT:    v_mul_lo_u32 v4, v3, 24
+; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GCN-NEXT:    v_mul_lo_u32 v3, v1, 24
 ; GCN-NEXT:    v_mul_hi_u32 v0, v0, 24
-; GCN-NEXT:    v_mul_hi_u32 v3, v3, 24
+; GCN-NEXT:    v_mul_hi_u32 v1, v1, 24
 ; GCN-NEXT:    v_mov_b32_e32 v5, s9
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; GCN-NEXT:    v_addc_u32_e32 v2, vcc, v2, v3, vcc
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, 0, v0
-; GCN-NEXT:    v_addc_u32_e32 v0, vcc, 0, v2, vcc
-; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GCN-NEXT:    v_mul_lo_u32 v2, s8, v1
+; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
+; GCN-NEXT:    v_addc_u32_e32 v0, vcc, v2, v1, vcc
+; GCN-NEXT:    v_mul_lo_u32 v1, s9, v0
 ; GCN-NEXT:    v_mul_hi_u32 v3, s8, v0
-; GCN-NEXT:    v_mul_lo_u32 v4, s9, v0
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
 ; GCN-NEXT:    v_mul_lo_u32 v3, s8, v0
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
-; GCN-NEXT:    v_sub_i32_e32 v4, vcc, 0, v2
+; GCN-NEXT:    v_sub_i32_e32 v4, vcc, 0, v1
 ; GCN-NEXT:    v_sub_i32_e32 v3, vcc, 24, v3
 ; GCN-NEXT:    v_subb_u32_e64 v4, s[0:1], v4, v5, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v5, s[0:1], s8, v3
 ; GCN-NEXT:    v_subbrev_u32_e64 v4, s[0:1], 0, v4, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[0:1], s9, v4
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[0:1], s8, v5
 ; GCN-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[0:1]
 ; GCN-NEXT:    v_cmp_eq_u32_e64 s[0:1], s9, v4
 ; GCN-NEXT:    v_cndmask_b32_e64 v4, v6, v5, s[0:1]
 ; GCN-NEXT:    v_add_i32_e64 v5, s[0:1], 2, v0
-; GCN-NEXT:    v_addc_u32_e64 v6, s[0:1], 0, v1, s[0:1]
+; GCN-NEXT:    v_addc_u32_e64 v6, s[0:1], 0, v2, s[0:1]
+; GCN-NEXT:    v_subb_u32_e32 v1, vcc, 0, v1, vcc
 ; GCN-NEXT:    v_add_i32_e64 v7, s[0:1], 1, v0
-; GCN-NEXT:    v_subb_u32_e32 v2, vcc, 0, v2, vcc
-; GCN-NEXT:    v_addc_u32_e64 v8, s[0:1], 0, v1, s[0:1]
+; GCN-NEXT:    v_addc_u32_e64 v2, s[0:1], 0, v2, s[0:1]
+; GCN-NEXT:    v_cmp_le_u32_e32 vcc, s9, v1
 ; GCN-NEXT:    v_cmp_ne_u32_e64 s[0:1], 0, v4
-; GCN-NEXT:    v_cmp_le_u32_e32 vcc, s9, v2
-; GCN-NEXT:    v_cndmask_b32_e64 v4, v8, v6, s[0:1]
-; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, vcc
+; GCN-NEXT:    v_cndmask_b32_e64 v4, 0, -1, vcc
 ; GCN-NEXT:    v_cmp_le_u32_e32 vcc, s8, v3
 ; GCN-NEXT:    v_cndmask_b32_e64 v3, 0, -1, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, s9, v2
-; GCN-NEXT:    v_cndmask_b32_e32 v2, v6, v3, vcc
-; GCN-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, s9, v1
+; GCN-NEXT:    v_cndmask_b32_e32 v1, v4, v3, vcc
+; GCN-NEXT:    v_cndmask_b32_e64 v2, v2, v6, s[0:1]
+; GCN-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
+; GCN-NEXT:    v_cndmask_b32_e32 v1, 0, v2, vcc
 ; GCN-NEXT:    v_cndmask_b32_e64 v2, v7, v5, s[0:1]
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
-; GCN-NEXT:    v_cndmask_b32_e32 v1, v1, v4, vcc
 ; GCN-NEXT:    v_xor_b32_e32 v0, s2, v0
 ; GCN-NEXT:    v_xor_b32_e32 v1, s2, v1
 ; GCN-NEXT:    v_mov_b32_e32 v2, s2
 ; GCN-NEXT:    v_subrev_i32_e32 v0, vcc, s2, v0
 ; GCN-NEXT:    v_subb_u32_e32 v1, vcc, v1, v2, vcc
 ; GCN-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GCN-NEXT:    s_endpgm
 ;

 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, v9, v12, vcc
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
 ; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v13, v7, vcc
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GCN-NEXT:    v_addc_u32_e64 v4, vcc, v4, v6, s[4:5]
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v4, vcc, 0, v4, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v5, v4, 24
 ; GCN-NEXT:    v_mul_hi_u32 v3, v3, 24
 ; GCN-NEXT:    v_mul_hi_u32 v4, v4, 24
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
-; GCN-NEXT:    v_addc_u32_e32 v4, vcc, v13, v4, vcc
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, 0, v3
-; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
-; GCN-NEXT:    v_addc_u32_e32 v4, vcc, 0, v12, vcc
-; GCN-NEXT:    v_mul_lo_u32 v5, v0, v4
-; GCN-NEXT:    v_mul_hi_u32 v6, v0, v3
-; GCN-NEXT:    v_mul_lo_u32 v7, v1, v3
-; GCN-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; GCN-NEXT:    v_mul_lo_u32 v6, v0, v3
-; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; GCN-NEXT:    v_sub_i32_e32 v7, vcc, 0, v5
-; GCN-NEXT:    v_sub_i32_e32 v6, vcc, 24, v6
-; GCN-NEXT:    v_subb_u32_e64 v7, s[4:5], v7, v1, vcc
-; GCN-NEXT:    v_sub_i32_e64 v8, s[4:5], v6, v0
-; GCN-NEXT:    v_subbrev_u32_e64 v7, s[4:5], 0, v7, s[4:5]
-; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v7, v1
-; GCN-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
-; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v8, v0
+; GCN-NEXT:    v_addc_u32_e32 v3, vcc, v13, v4, vcc
+; GCN-NEXT:    v_mul_lo_u32 v4, v1, v3
+; GCN-NEXT:    v_mul_hi_u32 v5, v0, v3
+; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; GCN-NEXT:    v_mul_lo_u32 v5, v0, v3
+; GCN-NEXT:    v_sub_i32_e32 v6, vcc, 0, v4
+; GCN-NEXT:    v_sub_i32_e32 v5, vcc, 24, v5
+; GCN-NEXT:    v_subb_u32_e64 v6, s[4:5], v6, v1, vcc
+; GCN-NEXT:    v_sub_i32_e64 v7, s[4:5], v5, v0
+; GCN-NEXT:    v_subbrev_u32_e64 v6, s[4:5], 0, v6, s[4:5]
+; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v6, v1
 ; GCN-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[4:5]
-; GCN-NEXT:    v_cmp_eq_u32_e64 s[4:5], v7, v1
-; GCN-NEXT:    v_cndmask_b32_e64 v7, v9, v8, s[4:5]
-; GCN-NEXT:    v_add_i32_e64 v8, s[4:5], 2, v3
-; GCN-NEXT:    v_addc_u32_e64 v9, s[4:5], 0, v4, s[4:5]
-; GCN-NEXT:    v_add_i32_e64 v10, s[4:5], 1, v3
-; GCN-NEXT:    v_subb_u32_e32 v5, vcc, 0, v5, vcc
-; GCN-NEXT:    v_addc_u32_e64 v11, s[4:5], 0, v4, s[4:5]
-; GCN-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, v7
-; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v5, v1
-; GCN-NEXT:    v_cndmask_b32_e64 v7, v11, v9, s[4:5]
-; GCN-NEXT:    v_cndmask_b32_e64 v9, 0, -1, vcc
-; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v6, v0
+; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v7, v0
+; GCN-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
+; GCN-NEXT:    v_cmp_eq_u32_e64 s[4:5], v6, v1
+; GCN-NEXT:    v_cndmask_b32_e64 v6, v8, v7, s[4:5]
+; GCN-NEXT:    v_add_i32_e64 v7, s[4:5], 2, v3
+; GCN-NEXT:    v_addc_u32_e64 v8, s[4:5], 0, v13, s[4:5]
+; GCN-NEXT:    v_add_i32_e64 v9, s[4:5], 1, v3
+; GCN-NEXT:    v_subb_u32_e32 v4, vcc, 0, v4, vcc
+; GCN-NEXT:    v_addc_u32_e64 v10, s[4:5], 0, v13, s[4:5]
+; GCN-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, v6
+; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v4, v1
+; GCN-NEXT:    v_cndmask_b32_e64 v6, v10, v8, s[4:5]
+; GCN-NEXT:    v_cndmask_b32_e64 v8, 0, -1, vcc
+; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v5, v0
 ; GCN-NEXT:    v_cndmask_b32_e64 v0, 0, -1, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, v5, v1
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v9, v0, vcc
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, v4, v1
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v8, v0, vcc
 ; GCN-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
-; GCN-NEXT:    v_cndmask_b32_e64 v1, v10, v8, s[4:5]
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v4, v7, vcc
+; GCN-NEXT:    v_cndmask_b32_e64 v1, v9, v7, s[4:5]
+; GCN-NEXT:    v_cndmask_b32_e32 v0, 0, v6, vcc
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
 ; GCN-NEXT:    v_xor_b32_e32 v3, v0, v2
 ; GCN-NEXT:    v_xor_b32_e32 v0, v1, v2
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_subb_u32_e32 v1, vcc, v3, v2, vcc
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GCN-IR-LABEL: v_test_sdiv_k_num_i64:

 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GCN-NEXT:    v_addc_u32_e64 v4, vcc, v4, v6, s[4:5]
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v4, vcc, 0, v4, vcc
 ; GCN-NEXT:    v_lshrrev_b32_e32 v5, 17, v4
 ; GCN-NEXT:    v_lshlrev_b32_e32 v4, 15, v4
 ; GCN-NEXT:    v_lshrrev_b32_e32 v3, 17, v3
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
-; GCN-NEXT:    v_addc_u32_e32 v4, vcc, v13, v5, vcc
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, 0, v3
-; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
-; GCN-NEXT:    v_addc_u32_e32 v4, vcc, 0, v12, vcc
-; GCN-NEXT:    v_mul_lo_u32 v5, v0, v4
-; GCN-NEXT:    v_mul_hi_u32 v6, v0, v3
-; GCN-NEXT:    v_mul_lo_u32 v7, v1, v3
-; GCN-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; GCN-NEXT:    v_mul_lo_u32 v6, v0, v3
-; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; GCN-NEXT:    v_sub_i32_e32 v7, vcc, 0, v5
-; GCN-NEXT:    v_sub_i32_e32 v6, vcc, 0x8000, v6
-; GCN-NEXT:    v_subb_u32_e64 v7, s[4:5], v7, v1, vcc
-; GCN-NEXT:    v_sub_i32_e64 v8, s[4:5], v6, v0
-; GCN-NEXT:    v_subbrev_u32_e64 v7, s[4:5], 0, v7, s[4:5]
-; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v7, v1
-; GCN-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
-; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v8, v0
+; GCN-NEXT:    v_addc_u32_e32 v3, vcc, v13, v5, vcc
+; GCN-NEXT:    v_mul_lo_u32 v4, v1, v3
+; GCN-NEXT:    v_mul_hi_u32 v5, v0, v3
+; GCN-NEXT:    s_mov_b32 s4, 0x8000
+; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; GCN-NEXT:    v_mul_lo_u32 v5, v0, v3
+; GCN-NEXT:    v_sub_i32_e32 v6, vcc, 0, v4
+; GCN-NEXT:    v_sub_i32_e32 v5, vcc, s4, v5
+; GCN-NEXT:    v_subb_u32_e64 v6, s[4:5], v6, v1, vcc
+; GCN-NEXT:    v_sub_i32_e64 v7, s[4:5], v5, v0
+; GCN-NEXT:    v_subbrev_u32_e64 v6, s[4:5], 0, v6, s[4:5]
+; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v6, v1
 ; GCN-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[4:5]
-; GCN-NEXT:    v_cmp_eq_u32_e64 s[4:5], v7, v1
-; GCN-NEXT:    v_cndmask_b32_e64 v7, v9, v8, s[4:5]
-; GCN-NEXT:    v_add_i32_e64 v8, s[4:5], 2, v3
-; GCN-NEXT:    v_addc_u32_e64 v9, s[4:5], 0, v4, s[4:5]
-; GCN-NEXT:    v_add_i32_e64 v10, s[4:5], 1, v3
-; GCN-NEXT:    v_subb_u32_e32 v5, vcc, 0, v5, vcc
-; GCN-NEXT:    v_addc_u32_e64 v11, s[4:5], 0, v4, s[4:5]
-; GCN-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, v7
-; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v5, v1
-; GCN-NEXT:    v_cndmask_b32_e64 v7, v11, v9, s[4:5]
-; GCN-NEXT:    v_cndmask_b32_e64 v9, 0, -1, vcc
-; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v6, v0
+; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v7, v0
+; GCN-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
+; GCN-NEXT:    v_cmp_eq_u32_e64 s[4:5], v6, v1
+; GCN-NEXT:    v_cndmask_b32_e64 v6, v8, v7, s[4:5]
+; GCN-NEXT:    v_add_i32_e64 v7, s[4:5], 2, v3
+; GCN-NEXT:    v_addc_u32_e64 v8, s[4:5], 0, v13, s[4:5]
+; GCN-NEXT:    v_add_i32_e64 v9, s[4:5], 1, v3
+; GCN-NEXT:    v_subb_u32_e32 v4, vcc, 0, v4, vcc
+; GCN-NEXT:    v_addc_u32_e64 v10, s[4:5], 0, v13, s[4:5]
+; GCN-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, v6
+; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v4, v1
+; GCN-NEXT:    v_cndmask_b32_e64 v6, v10, v8, s[4:5]
+; GCN-NEXT:    v_cndmask_b32_e64 v8, 0, -1, vcc
+; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v5, v0
 ; GCN-NEXT:    v_cndmask_b32_e64 v0, 0, -1, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, v5, v1
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v9, v0, vcc
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, v4, v1
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v8, v0, vcc
 ; GCN-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
-; GCN-NEXT:    v_cndmask_b32_e64 v1, v10, v8, s[4:5]
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v4, v7, vcc
+; GCN-NEXT:    v_cndmask_b32_e64 v1, v9, v7, s[4:5]
+; GCN-NEXT:    v_cndmask_b32_e32 v0, 0, v6, vcc
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
 ; GCN-NEXT:    v_xor_b32_e32 v3, v0, v2
 ; GCN-NEXT:    v_xor_b32_e32 v0, v1, v2
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_subb_u32_e32 v1, vcc, v3, v2, vcc
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GCN-IR-LABEL: v_test_sdiv_pow2_k_num_i64:

llvm/test/CodeGen/AMDGPU/srem64.ll

 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GCN-NEXT:    v_mul_hi_u32 v5, v0, v6
 ; GCN-NEXT:    v_mul_lo_u32 v7, v0, v4
 ; GCN-NEXT:    v_mul_hi_u32 v9, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v8, v3, v6
 ; GCN-NEXT:    v_mul_hi_u32 v6, v3, v6
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; GCN-NEXT:    v_mul_hi_u32 v10, v3, v4
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, v2, v9, vcc
-; GCN-NEXT:    v_mul_hi_u32 v9, v3, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v3, v4
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v8
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, v7, v6, vcc
-; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v9, v1, vcc
+; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v10, v1, vcc
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GCN-NEXT:    v_add_i32_e64 v0, s[0:1], v0, v4
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, v2, v6, vcc
 ; GCN-NEXT:    v_addc_u32_e64 v4, vcc, v3, v5, s[0:1]
 ; GCN-NEXT:    v_mul_lo_u32 v6, s2, v4
 ; GCN-NEXT:    v_mul_hi_u32 v7, s2, v0
 ; GCN-NEXT:    v_mul_lo_u32 v8, s3, v0
 ; GCN-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GCN-NEXT:    v_mul_lo_u32 v7, s2, v0
 ; GCN-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
 ; GCN-NEXT:    v_mul_lo_u32 v10, v0, v6
 ; GCN-NEXT:    v_mul_hi_u32 v12, v0, v6
 ; GCN-NEXT:    v_mul_hi_u32 v11, v0, v7
 ; GCN-NEXT:    v_mul_hi_u32 v9, v4, v7
 ; GCN-NEXT:    v_mul_lo_u32 v7, v4, v7
 ; GCN-NEXT:    v_mul_hi_u32 v8, v4, v6
 ; GCN-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
 ; GCN-NEXT:    v_addc_u32_e32 v11, vcc, v2, v12, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v4, v4, v6
 ; GCN-NEXT:    v_add_i32_e32 v7, vcc, v10, v7
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, v11, v9, vcc
-; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v8, v1, vcc
+; GCN-NEXT:    v_addc_u32_e32 v1, vcc, v8, v1, vcc
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v7, v4
-; GCN-NEXT:    v_addc_u32_e32 v6, vcc, v2, v6, vcc
+; GCN-NEXT:    v_addc_u32_e32 v1, vcc, v2, v1, vcc
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
-; GCN-NEXT:    v_addc_u32_e64 v3, vcc, v3, v6, s[0:1]
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
-; GCN-NEXT:    v_mul_lo_u32 v4, v3, 24
-; GCN-NEXT:    v_mul_hi_u32 v0, v0, 24
-; GCN-NEXT:    v_mul_hi_u32 v3, v3, 24
+; GCN-NEXT:    v_addc_u32_e64 v1, vcc, v3, v1, s[0:1]
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; GCN-NEXT:    v_addc_u32_e32 v2, vcc, v2, v3, vcc
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, 0, v0
-; GCN-NEXT:    v_addc_u32_e32 v0, vcc, 0, v2, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GCN-NEXT:    v_mul_lo_u32 v1, s8, v1
+; GCN-NEXT:    v_mul_lo_u32 v3, v1, 24
+; GCN-NEXT:    v_mul_hi_u32 v0, v0, 24
+; GCN-NEXT:    v_mul_hi_u32 v1, v1, 24
+; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
+; GCN-NEXT:    v_addc_u32_e32 v0, vcc, v2, v1, vcc
+; GCN-NEXT:    v_mul_lo_u32 v1, s9, v0
 ; GCN-NEXT:    v_mul_hi_u32 v2, s8, v0
-; GCN-NEXT:    v_mul_lo_u32 v3, s9, v0
 ; GCN-NEXT:    v_mul_lo_u32 v0, s8, v0
+; GCN-NEXT:    v_mov_b32_e32 v3, s9
 ; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, 0, v1
-; GCN-NEXT:    v_mov_b32_e32 v3, s9
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, 24, v0
 ; GCN-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v3, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v4, s[0:1], s8, v0
 ; GCN-NEXT:    v_subbrev_u32_e64 v5, s[2:3], 0, v2, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s9, v5
 ; GCN-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v3, s[0:1]
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[2:3]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s8, v4

 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
 ; GCN-NEXT:    v_addc_u32_e64 v3, vcc, v3, v5, s[4:5]
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v4, v3, 24
 ; GCN-NEXT:    v_mul_hi_u32 v2, v2, 24
 ; GCN-NEXT:    v_mul_hi_u32 v3, v3, 24
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
-; GCN-NEXT:    v_addc_u32_e32 v3, vcc, v12, v3, vcc
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, 0, v2
-; GCN-NEXT:    v_addc_u32_e32 v2, vcc, 0, v3, vcc
-; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v11, vcc
-; GCN-NEXT:    v_mul_lo_u32 v3, v0, v3
+; GCN-NEXT:    v_addc_u32_e32 v2, vcc, v12, v3, vcc
+; GCN-NEXT:    v_mul_lo_u32 v3, v1, v2
 ; GCN-NEXT:    v_mul_hi_u32 v4, v0, v2
-; GCN-NEXT:    v_mul_lo_u32 v5, v1, v2
 ; GCN-NEXT:    v_mul_lo_u32 v2, v0, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_sub_i32_e32 v4, vcc, 0, v3
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, 24, v2
 ; GCN-NEXT:    v_subb_u32_e64 v4, s[4:5], v4, v1, vcc
 ; GCN-NEXT:    v_sub_i32_e64 v5, s[4:5], v2, v0
 ; GCN-NEXT:    v_subbrev_u32_e64 v6, s[6:7], 0, v4, s[4:5]
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[6:7], v6, v1
 ; GCN-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[6:7]
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[6:7], v5, v0

 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
 ; GCN-NEXT:    v_addc_u32_e64 v3, vcc, v3, v5, s[4:5]
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; GCN-NEXT:    v_lshrrev_b32_e32 v4, 17, v3
 ; GCN-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
 ; GCN-NEXT:    v_lshrrev_b32_e32 v2, 17, v2
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_addc_u32_e32 v3, vcc, v12, v4, vcc
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, 0, v2
-; GCN-NEXT:    v_addc_u32_e32 v2, vcc, 0, v3, vcc
-; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v11, vcc
-; GCN-NEXT:    v_mul_lo_u32 v3, v0, v3
+; GCN-NEXT:    v_addc_u32_e32 v2, vcc, v12, v4, vcc
+; GCN-NEXT:    v_mul_lo_u32 v3, v1, v2
 ; GCN-NEXT:    v_mul_hi_u32 v4, v0, v2
-; GCN-NEXT:    v_mul_lo_u32 v5, v1, v2
 ; GCN-NEXT:    v_mul_lo_u32 v2, v0, v2
+; GCN-NEXT:    s_mov_b32 s4, 0x8000
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_sub_i32_e32 v4, vcc, 0, v3
-; GCN-NEXT:    v_sub_i32_e32 v2, vcc, 0x8000, v2
+; GCN-NEXT:    v_sub_i32_e32 v2, vcc, s4, v2
 ; GCN-NEXT:    v_subb_u32_e64 v4, s[4:5], v4, v1, vcc
 ; GCN-NEXT:    v_sub_i32_e64 v5, s[4:5], v2, v0
 ; GCN-NEXT:    v_subbrev_u32_e64 v6, s[6:7], 0, v4, s[4:5]
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[6:7], v6, v1
 ; GCN-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[6:7]
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[6:7], v5, v0
 ; GCN-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[6:7]
 ; GCN-NEXT:    v_cmp_eq_u32_e64 s[6:7], v6, v1

llvm/test/CodeGen/RISCV/rvv/rvv-vscale.i64.ll

 ; RV64:       # %bb.0: # %entry
 ; RV64-NEXT:    csrr a0, vlenb
 ; RV64-NEXT:    slli a1, a0, 1
 ; RV64-NEXT:    add a0, a1, a0
 ; RV64-NEXT:    ret
 ;
 ; RV32-LABEL: vscale_non_pow2:
 ; RV32:       # %bb.0: # %entry
-; RV32-NEXT:    csrr a1, vlenb
-; RV32-NEXT:    slli a0, a1, 1
-; RV32-NEXT:    add a0, a0, a1
-; RV32-NEXT:    srli a1, a1, 3
-; RV32-NEXT:    addi a2, zero, 24
-; RV32-NEXT:    mulhu a1, a1, a2
+; RV32-NEXT:    csrr a0, vlenb
+; RV32-NEXT:    slli a1, a0, 1
+; RV32-NEXT:    add a0, a1, a0
+; RV32-NEXT:    mv a1, zero
 ; RV32-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = mul i64 %0, 24
   ret i64 %1
 }
 
 declare i64 @llvm.vscale.i64()

llvm/test/CodeGen/RISCV/rvv/stepvector.ll

entry:
   ret <vscale x 16 x i64> %2
 }
 
 define <vscale x 16 x i64> @mul_stepvector_nxv16i64() {
 ; RV32-LABEL: mul_stepvector_nxv16i64:
 ; RV32:       # %bb.0: # %entry
 ; RV32-NEXT:    addi sp, sp, -16
 ; RV32-NEXT:    .cfi_def_cfa_offset 16
+; RV32-NEXT:    sw zero, 12(sp)
 ; RV32-NEXT:    csrr a0, vlenb
-; RV32-NEXT:    srli a1, a0, 3
-; RV32-NEXT:    addi a2, zero, 24
-; RV32-NEXT:    mulhu a1, a1, a2
-; RV32-NEXT:    sw a1, 12(sp)
 ; RV32-NEXT:    slli a1, a0, 1
 ; RV32-NEXT:    add a0, a1, a0
 ; RV32-NEXT:    sw a0, 8(sp)
 ; RV32-NEXT:    vsetvli a0, zero, e64, m8, ta, mu
 ; RV32-NEXT:    addi a0, sp, 8
 ; RV32-NEXT:    vlse64.v v16, (a0), zero
 ; RV32-NEXT:    vid.v v8
 ; RV32-NEXT:    addi a0, zero, 3