[AMDGPU] Improved wide multiplies

[OutOfCache]

Summary

These checks show optimized instructions if an operand is known to be
(partially) zero.

Change-Id: Ie2f6d0d3ee9d5b279d1f4c1dd0787492e39cc77a

[OutOfCache]

Added missing commits

[arsenm]

llvm/include/llvm/CodeGen/GlobalISel/LegalizerHelper.h

85–87

Can you pre-commit the change to add GISelKnown bits to the legalizer

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

109–115

Don't need both constructors, just the one with default argument

[OutOfCache]

Fixing indentation

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i64.ll

2253–2255

Missing mir test updates? I thought we had mir coverage for mul legalize already

[OutOfCache]

Adding missing commits.

[foad]

I guess we could get the same improvement by running the binop_right_to_zero combine post-legalization?

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

2932

Can inline this into the loop

3058

Intendation issue?

3184

Naming is a bit unfortunate, as the function returning the pointer to the KBAnalysis is named the same as the function getting the known bits.

[tsymalla]

Somethign went wrong during applying the patch. Could you please have a look?
https://buildkite.com/llvm-project/diff-checks/builds/142783#01851b23-3226-4e98-bac5-219865bb07b5

[OutOfCache]

Adressing the redundant constructor and minor format issues

[OutOfCache]

Adding the new test file

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3184

Can be merged to AtLeastOneSrcIsZero (just like with AtLeastOneArgIsZero)

3184

Intendation (maybe run clang-format)?

llvm/test/CodeGen/AMDGPU/GlobalISel/mul-known-bits.i64.ll

6

Can you give some more insightful names to the tests (appending _xy vs. naming the test @v_mul_i64_zext_no_args or something similar)?

7

Can you pre-commit the tests so it is easier to identify the changes in the assembly?

146

Maybe add some test to show whats happening when both high and low bits are being masked (0x0000FFFF000...)

235

Should this tests and the ones following also be prefixed with v_?

270

Typo: differnt

272

Maybe add some 32-bit tests to show that your changes are being applied correctly.

[tsymalla]

It seems like this is breaking the LegalizerHelper.h / the LegalizerHelperTest:
/var/lib/buildkite-agent/builds/llvm-project/llvm/include/llvm/CodeGen/GlobalISel/LegalizerHelper.h:47:7: note: candidate constructor (the implicit move constructor) not viable: requires 1 argument, but 4 were provided
/var/lib/buildkite-agent/builds/llvm-project/llvm/unittests/CodeGen/GlobalISel/LegalizerHelperTest.cpp:641:19: error: no matching constructor for initialization of 'llvm::LegalizerHelper'

LegalizerHelper Helper(*MF, Info, Observer, B);

[OutOfCache]

Improved tests with better naming.
Better variable naming in LegalizerInfo.
Added constructor again because that was causing issues.

[OutOfCache]

This comment was removed by OutOfCache.

[OutOfCache]

• [AMDGPU] Improved naming of mul-known-bits tests
• [AMDGPU] Better variable naming in LegalizerInfo

[OutOfCache]

Improved tests with better naming.
Better variable naming in LegalizerInfo.
Added constructor again because that was causing issues.

[OutOfCache]

Fixing constructor issues by moving the default parameter to the header.

The default parameter for the new LegalizerHelper constructor has to be set
in the header file and not the definition in the .cpp file.

[OutOfCache]

Fixing constructor issues by moving the default parameter to the header.

The default parameter for the new LegalizerHelper constructor has to be set
in the header file and not the definition in the .cpp file.

[OutOfCache]

• [GISel] Adding KnownBitsAnalysis to Legalizer
• [AMDGPU] Wide multiplies with Known Bits Analysis
• [AMDGPU] Improved wide multiplies tests

Fixed clang-format issue.

[OutOfCache]

• [AMDGPU] Improved wide multiplies tests

Fixed clang-format.

[OutOfCache]

• [AMDGPU] Wide multiplies with Known Bits Analysis
• [AMDGPU] Improved wide multiplies tests

Fixed clang-format for good.

[tsymalla]

A few nits / questions, otherwise LGTM.

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3002

It looks like this is doing the same thing like in LOC 3040. Does it make sense to cache the results (is there anything computed when invoking isZero())? You could also cache the pairs that are known to be zero, but this is just optional. I'd just think about re-implementing the same thing a few lines later.

3185

I'd still prefer to have the getter method Helper::getKnownBits() differently named. The one thing is returning a pointer to the known bits analysis, the other one is returning the known bits for a register.

llvm/test/CodeGen/AMDGPU/GlobalISel/mul-known-bits.i64.ll

396

Can you please pre-commit these tests?

[foad]

Can you rebase now that D140907 has landed please?

[nhaehnle]

Thanks, this already looks good to me. I do have a small number of comments inline still.

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

2901–2907

Did you run clang-format on this? Usually it chooses a different layout.

2933–2936

Please remove the assumption that Src0 and Src1 are equally long. I'd suggest using two for-each loops.

3002–3004

nit: I don't think having the separate variable is worth it anymore, just put the condition directly into the if-statement (same below)

3185

I think it's fine. The class is called GISelKnownBits, not GISelKnownBitsAnalysis or similar.

3188–3192

Does this actually happen? (Is there a test case that changes when you remove this?) I would have thought that such a G_MUL would have been eliminated by an earlier combine.

[OutOfCache]

• [GISel] Adding KnownBitsAnalysis to Legalizer
• [AMDGPU] Wide multiplies with Known Bits Analysis
• [AMDGPU] Improved wide multiplies tests
• [GISel/AMDGPU] caching results of isZero()
• [AMDGPU] Inlining conditions in buildMultiply
• [AMDGPU] Removing redundant zero-check for mults

• Rebasing after D140907 landed
• Addressing comments

[OutOfCache]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

2901–2907

I did. It even caused failing pre-merge checks if I remember correctly. I ran clang-format again and this is still the result.

3002

isZero() is calculating the number of zeroes and checking if the sum is the same as the total number of bits. Because of that, I decided to cache the results of isZero() in the vectors.

3007

This check is required, when the accumulator is a zero register.

!LocalAccum[0] only checks for the existence of a Register. It is still true, if the Register is known to be all zeroes.
This particular case occurs when the lower bytes of an operand are masked.
In that case, the check in line 3048 will fail and no G_MAD will be created. LocalAccum[0] will still be set to the result of the Unmerge of the Tmp register in line 3060. Tmp is set to a zero register in line 3041, so it is all zeroes at this point.

By stepping through the debugger, I confirmed that in that case the first condition, !LocalAccum[0] will be false, but the second condition will be correctly evaluated to true and therefore skip the addition to 0.

3184

I can see the issue. I am not sure what to rename it to, though. getKnownBitsAnalysis? Not quite correct since the type is GISelKnownBits, but perhaps less confusing?

3185

I kept it as is because of the type name and because it has the same name in the CombinerHelper

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

2936

Can use ranged for-loops

3047

Can one of these accesses be out-of-bounds now that you removed the assumption that both Src0 and Src1 are of equal length?

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

2901–2907

You can try clang-format -style=file:path and see if this changes something. I think you can also manually invoke git-clang-format to see if anything would have been changed by clang-format.

[OutOfCache]

• [AMDGPU] Removing redundant zero-check for mults

[OutOfCache]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

2901–2907

I ran git clang-format but it didn't change this part. clang-format file -i changed the file in a lot of places, but not here either. Maybe someone could double-check on their machine?

3047

I double-checked and the way the code works currently makes sure that both Src0 and Src1 are the same length. In line 3179, both arrays are created in the same for-loop. I don't know if there is a case where they will have different sizes. If the operands are not of the same size, an error is thrown and it does not compile. So in theory yes, but in practice no? Is this a case that should be considered or would that be too much?

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

2901–2907

I just re-ran git-clang-format HEAD~1 and it did not change anything for me. The Buildbot does not seem to spill out an error due to clang-format, so I guess it's alright.

3047

@nhaehnle If both arrays are initialized in one loop, shouldn't we just move back to initialize Src0KnownZeros and Src1KnownZeros in one loop, too?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3007

If you're just looking for zero, just looking for the constant zero is cheaper than going through getKnownBits

[OutOfCache]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3007

Sounds like a good idea, but how do I do that?

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3007

I guess he meant checking the operands for being zero explicitly. I think using getKnownBits is fine.

[OutOfCache]

• [AMDGPU] reverting separation of for-loops

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3007

Check if it's G_CONSTANT i32 0. There are a few too many ways to check for it (I'd suggest MIPatternMatch's m_ZeroInt)

[tsymalla]

Can you re-apply clang-format on AMDGPULegalizerInfo.cpp please?

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

2933

I think you wanted to iterate with a classical for loop and access Src0 and Src via the subscript operator?

3007

Is that check correct? Can you remove the comment please?

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3047

Can you add some GISel tests showing the behavior by transforming MIR to MIR? You had an initial test case, you could re-use that. Maybe there are already MIR test cases you could extend. This should also show that Src0 and Src1 are handled correctly.

[OutOfCache]

• [AMDGPU] reverting separation of for-loops

[OutOfCache]

• [AMDGPU] reverting separation of for-loops

[OutOfCache]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3007

I tried mi_match(LocalAccum[0], MRI, m_ZeroInt()), but for some reason it always returned false.

I also tried replacing the SrcXKnownZeros.push_back(KB.getKnownBits(SrcX[i]).isZero()) with Src0KnownZeros.push_back(mi_match(SrcX[i], MRI, m_ZeroInt()) and similarly, it returned false when the first one returned true.

This also caused the @v_mul_i64_masked_src0_lo and @v_mul_i64_masked_src1_lo tests to fail and produce multiplications with 0.

[OutOfCache]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3007

This is the Code before the Legalizer:

bb.1.entry:
  liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3
  %2:_(s32) = COPY $vgpr0
  %3:_(s32) = COPY $vgpr1
  %0:_(s64) = G_MERGE_VALUES %2:_(s32), %3:_(s32)
  %4:_(s32) = COPY $vgpr2
  %5:_(s32) = COPY $vgpr3
  %1:_(s64) = G_MERGE_VALUES %4:_(s32), %5:_(s32)
  %6:_(s64) = G_CONSTANT i64 -4294967296
  %7:_(s64) = G_AND %1:_, %6:_
  %8:_(s64) = G_MUL %0:_, %7:_
  %9:_(s32), %10:_(s32) = G_UNMERGE_VALUES %8:_(s64)
  $vgpr0 = COPY %9:_(s32)
  $vgpr1 = COPY %10:_(s32)
  SI_RETURN implicit $vgpr0, implicit $vgpr1

The only G_CONSTANTs are the mask for the G_AND and a 64-bit 0 for the G_MAD addition

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3007

Your MIR should look something like that at the time of the LocalAccum[0] check, so you need to query appropriately.

/Users/seuchomat/Documents/Projekte/C++/llvm-project/build/bin/llc -global-isel -march=amdgcn -mcpu=gfx1010 /Users/seuchomat/Documents/Projekte/C++/llvm-project/llvm/test/CodeGen/AMDGPU/GlobalISel/mad.mir -run-pass=legalizer
  %18:_(s32) = G_MUL %11:_, %14:_
# Machine code for function test_mad: IsSSA, NoPHIs

bb.0.entry:
  %0:_(s32) = COPY $vgpr0
  %1:_(s32) = COPY $vgpr1
  %2:_(s64) = G_MERGE_VALUES %0:_(s32), %1:_(s32)
  %3:_(s32) = COPY $vgpr2
  %4:_(s32) = COPY $vgpr3
  %5:_(s64) = G_MERGE_VALUES %3:_(s32), %4:_(s32)
  %6:_(s64) = G_CONSTANT i64 -4294967296
  %7:_(s64) = G_AND %5:_, %6:_
  %11:_(s32), %13:_(s32) = G_UNMERGE_VALUES %2:_(s64)
  %12:_(s32), %14:_(s32) = G_UNMERGE_VALUES %7:_(s64)
  %15:_(s64) = G_CONSTANT i64 0
  %16:_(s32), %17:_(s32) = G_UNMERGE_VALUES %15:_(s64)
  %18:_(s32) = G_MUL %11:_, %14:_
  %8:_(s64) = G_MUL %2:_, %7:_
  %9:_(s32), %10:_(s32) = G_UNMERGE_VALUES %8:_(s64)
  $vgpr0 = COPY %9:_(s32)
  $vgpr1 = COPY %10:_(s32)
  SI_RETURN implicit $vgpr0, implicit $vgpr1

# End machine code for function test_mad.

So, I think, by applying your pattern matching to the register itself, it tries to find the last instruction that uses the constant 0 which is the G_UNMERGE_VALUES itself in llvm::getConstantVRegValWithLookThrough.

[OutOfCache]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3007

I tried following the suggestion of using mi_match. Unfortunately, it did not work. There is no i32 G_CONSTANT that can be matched.

The LocalAccum has the result of the G_MAD, which is not using a G_CONSTANT, even if the result is 0.

%16:_(s64), %17:_(s1) = G_AMDGPU_MAD_U64_U32 %2:_(s32), %12:_, %15:_
%18:_(s32), %19:_(s32) = G_UNMERGE_VALUES %16:_(s64)

Essentially we are looking at %18 and %19. If any of these are all zeroes, which we only know using the Known Bits Analysis (to my knowledge), we can save the following G_ADDs here:

%20:_(s32) = G_MUL %2:_, %14:_
%21:_(s32) = G_ADD %19:_, %20:_
%22:_(s32) = G_MUL %3:_, %12:_
%23:_(s32) = G_ADD %21:_, %22:_

The Multiplication Arguments are within VGPRs, for which the mi_match does not work.

Thank you for the suggestion, though, @arsenm! It would have been more efficient if I managed to make it work. Plus, I learned a lot along the way.

In case I should try something else, feel free to let me know.

I apologize for the delay regarding my answer, I took time off for my exams.

[tsymalla]

Can you please add some MIR test cases in addition to your LLVM IR tests? You already put some example code in the discussion, but I believe a test that shows the transformation on a GMIR level could be helpful.

Other than that, it looks good to me.

[OutOfCache]

Added MIR Tests for gfx10 and gfx11.

• [GISel] Adding KnownBitsAnalysis to Legalizer
• [AMDGPU] Wide multiplies with Known Bits Analysis
• [AMDGPU] Improved wide multiplies tests
• [GISel/AMDGPU] caching results of isZero()
• [AMDGPU] Inlining conditions in buildMultiply
• [AMDGPU] Removing redundant zero-check for mults
• [AMDGPU] reverting separation of for-loops
• [AMDGPU] Update Tests for multiplies using KBA

[tsymalla]

llvm/test/CodeGen/AMDGPU/GlobalISel/mul-known-bits.i64.mir

10

Can you please pre-commit the tests based on the new test names so they don't show up in the diff?

Btw, it seems like gfx10 and gfx11 always get the same changes in the GMIR test. In that case, I'd rather remove the gfx11 tests and just execute one test. This keeps the GMIR test less cluttered.

[OutOfCache]

Remove gfx11 from MIR tests

• [GISel] Adding KnownBitsAnalysis to Legalizer
• [AMDGPU] Wide multiplies with Known Bits Analysis
• [AMDGPU] Improved wide multiplies tests
• [GISel/AMDGPU] caching results of isZero()
• [AMDGPU] Inlining conditions in buildMultiply
• [AMDGPU] Removing redundant zero-check for mults
• [AMDGPU] reverting separation of for-loops
• [AMDGPU] Update Tests for multiplies using KBA

[tsymalla]

LGTM

llvm/include/llvm/CodeGen/GlobalISel/Legalizer.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_GLOBALISEL_LEGALIZER_H
 #define LLVM_CODEGEN_GLOBALISEL_LEGALIZER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/CodeGen/GlobalISel/GISelKnownBits.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 
 namespace llvm {
 
 class LegalizerInfo;
 class MachineIRBuilder;
 class MachineInstr;
public:
   }
 
   bool runOnMachineFunction(MachineFunction &MF) override;
 
   static MFResult
   legalizeMachineFunction(MachineFunction &MF, const LegalizerInfo &LI,
                           ArrayRef<GISelChangeObserver *> AuxObservers,
                           LostDebugLocObserver &LocObserver,
-                          MachineIRBuilder &MIRBuilder);
+                          MachineIRBuilder &MIRBuilder, GISelKnownBits *KB);
 };
 } // End namespace llvm.
 
 #endif

llvm/include/llvm/CodeGen/GlobalISel/LegalizerHelper.h

 /// illegal instruction.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_GLOBALISEL_LEGALIZERHELPER_H
 #define LLVM_CODEGEN_GLOBALISEL_LEGALIZERHELPER_H
 
 #include "llvm/CodeGen/GlobalISel/CallLowering.h"
+#include "llvm/CodeGen/GlobalISel/GISelKnownBits.h"
 #include "llvm/CodeGen/RuntimeLibcalls.h"
 #include "llvm/CodeGen/TargetOpcodes.h"
 
 namespace llvm {
 // Forward declarations.
 class APInt;
 class GAnyLoad;
 class GLoadStore;
public:
 
   /// To keep track of changes made by the LegalizerHelper.
   GISelChangeObserver &Observer;
 
 private:
   MachineRegisterInfo &MRI;
   const LegalizerInfo &LI;
   const TargetLowering &TLI;
+  GISelKnownBits *KB;
 
 public:
   enum LegalizeResult {
     /// Instruction was already legal and no change was made to the
     /// MachineFunction.
     AlreadyLegal,
 
     /// Instruction has been legalized and the MachineFunction changed.
     Legalized,
 
     /// Some kind of error has occurred and we could not legalize this
     /// instruction.
     UnableToLegalize,
   };
 
   /// Expose LegalizerInfo so the clients can re-use.
   const LegalizerInfo &getLegalizerInfo() const { return LI; }
   const TargetLowering &getTargetLowering() const { return TLI; }
+  GISelKnownBits *getKnownBits() const { return KB; }
 
   LegalizerHelper(MachineFunction &MF, GISelChangeObserver &Observer,
                   MachineIRBuilder &B);
   LegalizerHelper(MachineFunction &MF, const LegalizerInfo &LI,
-                  GISelChangeObserver &Observer, MachineIRBuilder &B);
+                  GISelChangeObserver &Observer, MachineIRBuilder &B,
+                  GISelKnownBits *KB = nullptr);
 
   /// Replace \p MI by a sequence of legal instructions that can implement the
   /// same operation. Note that this means \p MI may be deleted, so any iterator
   /// steps should be performed before calling this function. \p Helper should
   /// be initialized to the MachineFunction containing \p MI.
   ///
   /// Considered as an opaque blob, the legal code will use and define the same
   /// registers as \p MI.
   LegalizeResult legalizeInstrStep(MachineInstr &MI,
                                    LostDebugLocObserver &LocObserver);

llvm/lib/CodeGen/GlobalISel/Legalizer.cpp

 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/GlobalISel/Legalizer.h"
 #include "llvm/ADT/PostOrderIterator.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/CodeGen/GlobalISel/CSEInfo.h"
 #include "llvm/CodeGen/GlobalISel/CSEMIRBuilder.h"
 #include "llvm/CodeGen/GlobalISel/GISelChangeObserver.h"
+#include "llvm/CodeGen/GlobalISel/GISelKnownBits.h"
 #include "llvm/CodeGen/GlobalISel/GISelWorkList.h"
 #include "llvm/CodeGen/GlobalISel/LegalizationArtifactCombiner.h"
 #include "llvm/CodeGen/GlobalISel/LegalizerHelper.h"
 #include "llvm/CodeGen/GlobalISel/LostDebugLocObserver.h"
 #include "llvm/CodeGen/GlobalISel/Utils.h"
 #include "llvm/CodeGen/MachineOptimizationRemarkEmitter.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"

 #endif
 
 char Legalizer::ID = 0;
 INITIALIZE_PASS_BEGIN(Legalizer, DEBUG_TYPE,
                       "Legalize the Machine IR a function's Machine IR", false,
                       false)
 INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
 INITIALIZE_PASS_DEPENDENCY(GISelCSEAnalysisWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(GISelKnownBitsAnalysis)
 INITIALIZE_PASS_END(Legalizer, DEBUG_TYPE,
                     "Legalize the Machine IR a function's Machine IR", false,
                     false)
 
 Legalizer::Legalizer() : MachineFunctionPass(ID) { }
 
 void Legalizer::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.addRequired<TargetPassConfig>();
   AU.addRequired<GISelCSEAnalysisWrapperPass>();
   AU.addPreserved<GISelCSEAnalysisWrapperPass>();
+  AU.addRequired<GISelKnownBitsAnalysis>();
+  AU.addPreserved<GISelKnownBitsAnalysis>();
   getSelectionDAGFallbackAnalysisUsage(AU);
   MachineFunctionPass::getAnalysisUsage(AU);
 }
 
 void Legalizer::init(MachineFunction &MF) {
 }
 
 static bool isArtifact(const MachineInstr &MI) {
public:
   }
 };
 } // namespace
 
 Legalizer::MFResult
 Legalizer::legalizeMachineFunction(MachineFunction &MF, const LegalizerInfo &LI,
                                    ArrayRef<GISelChangeObserver *> AuxObservers,
                                    LostDebugLocObserver &LocObserver,
-                                   MachineIRBuilder &MIRBuilder) {
+                                   MachineIRBuilder &MIRBuilder,
+                                   GISelKnownBits *KB) {
   MIRBuilder.setMF(MF);
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
   // Populate worklists.
   InstListTy InstList;
   ArtifactListTy ArtifactList;
   ReversePostOrderTraversal<MachineFunction *> RPOT(&MF);
   // Perform legalization bottom up so we can DCE as we legalize.
Legalizer::legalizeMachineFunction(MachineFunction &MF, const LegalizerInfo &LI,
   // CSEInfo) to observe all changes. Use the wrapper observer.
   GISelObserverWrapper WrapperObserver(&WorkListObserver);
   for (GISelChangeObserver *Observer : AuxObservers)
     WrapperObserver.addObserver(Observer);
 
   // Now install the observer as the delegate to MF.
   // This will keep all the observers notified about new insertions/deletions.
   RAIIMFObsDelInstaller Installer(MF, WrapperObserver);
-  LegalizerHelper Helper(MF, LI, WrapperObserver, MIRBuilder);
+  LegalizerHelper Helper(MF, LI, WrapperObserver, MIRBuilder, KB);
   LegalizationArtifactCombiner ArtCombiner(MIRBuilder, MRI, LI);
   bool Changed = false;
   SmallVector<MachineInstr *, 128> RetryList;
   do {
     LLVM_DEBUG(dbgs() << "=== New Iteration ===\n");
     assert(RetryList.empty() && "Expected no instructions in RetryList");
     unsigned NumArtifacts = ArtifactList.size();
     while (!InstList.empty()) {
if (EnableCSE && CSEInfo) {
     // We want CSEInfo in addition to WorkListObserver to observe all changes.
     AuxObservers.push_back(CSEInfo);
   }
   assert(!CSEInfo || !errorToBool(CSEInfo->verify()));
   LostDebugLocObserver LocObserver(DEBUG_TYPE);
   if (VerifyDebugLocs > DebugLocVerifyLevel::None)
     AuxObservers.push_back(&LocObserver);
 
+  // This allows Known Bits Analysis in the legalizer.
+  GISelKnownBits *KB = &getAnalysis<GISelKnownBitsAnalysis>().get(MF);
+
   const LegalizerInfo &LI = *MF.getSubtarget().getLegalizerInfo();
-  MFResult Result =
-      legalizeMachineFunction(MF, LI, AuxObservers, LocObserver, *MIRBuilder);
+  MFResult Result = legalizeMachineFunction(MF, LI, AuxObservers, LocObserver,
+                                            *MIRBuilder, KB);
 
   if (Result.FailedOn) {
     reportGISelFailure(MF, TPC, MORE, "gisel-legalize",
                        "unable to legalize instruction", *Result.FailedOn);
     return false;
   }
   // For now don't support if new blocks are inserted - we would need to fix the
   // outer loop for that.

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

 /// individual instructions and the LegalizeMachineIR wrapper pass for the
 /// primary legalization.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/GlobalISel/LegalizerHelper.h"
 #include "llvm/CodeGen/GlobalISel/CallLowering.h"
 #include "llvm/CodeGen/GlobalISel/GISelChangeObserver.h"
+#include "llvm/CodeGen/GlobalISel/GISelKnownBits.h"
 #include "llvm/CodeGen/GlobalISel/GenericMachineInstrs.h"
 #include "llvm/CodeGen/GlobalISel/LegalizerInfo.h"
 #include "llvm/CodeGen/GlobalISel/LostDebugLocObserver.h"
 #include "llvm/CodeGen/GlobalISel/MIPatternMatch.h"
 #include "llvm/CodeGen/GlobalISel/MachineIRBuilder.h"
 #include "llvm/CodeGen/GlobalISel/Utils.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
static Type *getFloatTypeForLLT(LLVMContext &Ctx, LLT Ty) {
   }
 }
 
 LegalizerHelper::LegalizerHelper(MachineFunction &MF,
                                  GISelChangeObserver &Observer,
                                  MachineIRBuilder &Builder)
     : MIRBuilder(Builder), Observer(Observer), MRI(MF.getRegInfo()),
       LI(*MF.getSubtarget().getLegalizerInfo()),
-      TLI(*MF.getSubtarget().getTargetLowering()) { }
+      TLI(*MF.getSubtarget().getTargetLowering()), KB(nullptr) {}
 
 LegalizerHelper::LegalizerHelper(MachineFunction &MF, const LegalizerInfo &LI,
                                  GISelChangeObserver &Observer,
-                                 MachineIRBuilder &B)
+                                 MachineIRBuilder &B, GISelKnownBits *KB)
     : MIRBuilder(B), Observer(Observer), MRI(MF.getRegInfo()), LI(LI),
-    TLI(*MF.getSubtarget().getTargetLowering()) { }
+      TLI(*MF.getSubtarget().getTargetLowering()), KB(KB) {}
 
 LegalizerHelper::LegalizeResult
 LegalizerHelper::legalizeInstrStep(MachineInstr &MI,
                                    LostDebugLocObserver &LocObserver) {
   LLVM_DEBUG(dbgs() << "Legalizing: " << MI);
 
   MIRBuilder.setInstrAndDebugLoc(MI);
 
   if (MI.getOpcode() == TargetOpcode::G_INTRINSIC ||
       MI.getOpcode() == TargetOpcode::G_INTRINSIC_W_SIDE_EFFECTS)

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

 //
 // Source and accumulation registers must all be 32-bits.
 //
 // TODO: When the multiply is uniform, we should produce a code sequence
 // that is better suited to instruction selection on the SALU. Instead of
 // the outer loop going over parts of the result, the outer loop should go
 // over parts of one of the factors. This should result in instruction
 // selection that makes full use of S_ADDC_U32 instructions.
-void AMDGPULegalizerInfo::buildMultiply(
-    LegalizerHelper &Helper, MutableArrayRef<Register> Accum,
-    ArrayRef<Register> Src0, ArrayRef<Register> Src1,
-    bool UsePartialMad64_32, bool SeparateOddAlignedProducts) const {
+void AMDGPULegalizerInfo::buildMultiply(LegalizerHelper &Helper,
+                                        MutableArrayRef<Register> Accum,
+                                        ArrayRef<Register> Src0,
+                                        ArrayRef<Register> Src1,
+                                        bool UsePartialMad64_32,
+                                        bool SeparateOddAlignedProducts) const {
   // Use (possibly empty) vectors of S1 registers to represent the set of
   // carries from one pair of positions to the next.
   using Carry = SmallVector<Register, 2>;
 
   MachineIRBuilder &B = Helper.MIRBuilder;
+  GISelKnownBits &KB = *Helper.getKnownBits();
 
   const LLT S1 = LLT::scalar(1);
   const LLT S32 = LLT::scalar(32);
   const LLT S64 = LLT::scalar(64);
 
   Register Zero32;
   Register Zero64;
 
   auto getZero32 = [&]() -> Register {
     if (!Zero32)
       Zero32 = B.buildConstant(S32, 0).getReg(0);
     return Zero32;
   };
   auto getZero64 = [&]() -> Register {
     if (!Zero64)
       Zero64 = B.buildConstant(S64, 0).getReg(0);
     return Zero64;
   };
 
+  SmallVector<bool, 2> Src0KnownZeros, Src1KnownZeros;
+  for (unsigned i = 0; i < Src0.size(); ++i) {
+    Src0KnownZeros.push_back(KB.getKnownBits(Src0[i]).isZero());
+    Src1KnownZeros.push_back(KB.getKnownBits(Src1[i]).isZero());
+  }
+
   // Merge the given carries into the 32-bit LocalAccum, which is modified
   // in-place.
   //
   // Returns the carry-out, which is a single S1 register or null.
   auto mergeCarry =
       [&](Register &LocalAccum, const Carry &CarryIn) -> Register {
         if (CarryIn.empty())
           return Register();
auto buildMadChain =
         Carry CarryOut;
         unsigned j0 = 0;
 
         // Use plain 32-bit multiplication for the most significant part of the
         // result by default.
         if (LocalAccum.size() == 1 &&
             (!UsePartialMad64_32 || !CarryIn.empty())) {
           do {
+            // Skip multiplication if one of the operands is 0
             unsigned j1 = DstIndex - j0;
+            if (Src0KnownZeros[j0] || Src1KnownZeros[j1]) {
+              ++j0;
+              continue;
+            }
             auto Mul = B.buildMul(S32, Src0[j0], Src1[j1]);
-            if (!LocalAccum[0]) {
+            if (!LocalAccum[0] || KB.getKnownBits(LocalAccum[0]).isZero()) {
               LocalAccum[0] = Mul.getReg(0);
             } else {
               if (CarryIn.empty()) {
                 LocalAccum[0] = B.buildAdd(S32, LocalAccum[0], Mul).getReg(0);
               } else {
                 LocalAccum[0] =
                     B.buildUAdde(S32, S1, LocalAccum[0], Mul, CarryIn.back())
                         .getReg(0);
auto buildMadChain =
           } else {
             assert(LocalAccum.size() == 1 || !LocalAccum[1]);
             Tmp = getZero64();
             HaveSmallAccum = true;
           }
 
           do {
             unsigned j1 = DstIndex - j0;
+            if (Src0KnownZeros[j0] || Src1KnownZeros[j1]) {
+              ++j0;
+              continue;
+            }
             auto Mad = B.buildInstr(AMDGPU::G_AMDGPU_MAD_U64_U32, {S64, S1},
                                     {Src0[j0], Src1[j1], Tmp});
             Tmp = Mad.getReg(0);
             if (!HaveSmallAccum)
               CarryOut.push_back(Mad.getReg(1));
             HaveSmallAccum = false;
+
             ++j0;
           } while (j0 <= DstIndex);
 
           auto Unmerge = B.buildUnmerge(S32, Tmp);
           LocalAccum[0] = Unmerge.getReg(0);
           if (LocalAccum.size() > 1)
             LocalAccum[1] = Unmerge.getReg(1);
         }
 
bool AMDGPULegalizerInfo::legalizeMul(LegalizerHelper &Helper,
   // stalls on some subtargets in some cases.
   const bool UsePartialMad64_32 = ST.getGeneration() < AMDGPUSubtarget::GFX10;
 
   // Whether to compute odd-aligned partial products separately. This is
   // advisable on subtargets where the accumulator of MAD_64_32 must be placed
   // in an even-aligned VGPR.
   const bool SeparateOddAlignedProducts = ST.hasFullRate64Ops();
 
   LLT S32 = LLT::scalar(32);
   SmallVector<Register, 2> Src0Parts, Src1Parts;
   for (unsigned i = 0; i < NumParts; ++i) {
     Src0Parts.push_back(MRI.createGenericVirtualRegister(S32));
     Src1Parts.push_back(MRI.createGenericVirtualRegister(S32));
   }
   B.buildUnmerge(Src0Parts, Src0);
   B.buildUnmerge(Src1Parts, Src1);
 
   SmallVector<Register, 2> AccumRegs(NumParts);
   buildMultiply(Helper, AccumRegs, Src0Parts, Src1Parts, UsePartialMad64_32,
                 SeparateOddAlignedProducts);
 
   B.buildMergeLikeInstr(DstReg, AccumRegs);
   MI.eraseFromParent();
   return true;
-
 }
 
 // Legalize ctlz/cttz to ffbh/ffbl instead of the default legalization to
 // ctlz/cttz_zero_undef. This allows us to fix up the result for the zero input
 // case with a single min instruction instead of a compare+select.
 bool AMDGPULegalizerInfo::legalizeCTLZ_CTTZ(MachineInstr &MI,
                                             MachineRegisterInfo &MRI,
                                             MachineIRBuilder &B) const {

llvm/test/CodeGen/AMDGPU/GlobalISel/mul-known-bits.i64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN:  llc -march=amdgcn -mcpu=gfx1010 -global-isel=1 -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=GFX10 %s
 ; RUN:  llc -march=amdgcn -mcpu=gfx1100 -global-isel=1 -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=GFX11 %s
 declare i32 @llvm.amdgcn.workitem.id.x()
 
 ; A 64-bit multiplication where no arguments were zero extended.
-define amdgpu_kernel void @v_mul_i64_zext_00(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) nounwind {
-; GFX10-LABEL: v_mul_i64_zext_00:
+define amdgpu_kernel void @v_mul_i64_no_zext(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) nounwind {
+; GFX10-LABEL: v_mul_i64_no_zext:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x2c
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v6, 3, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    global_load_dwordx2 v[0:1], v6, s[0:1]
 ; GFX10-NEXT:    global_load_dwordx2 v[2:3], v6, s[2:3]
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_mad_u64_u32 v[4:5], s0, v0, v2, 0
 ; GFX10-NEXT:    v_mul_lo_u32 v0, v0, v3
 ; GFX10-NEXT:    v_mul_lo_u32 v1, v1, v2
 ; GFX10-NEXT:    v_add3_u32 v5, v5, v0, v1
 ; GFX10-NEXT:    global_store_dwordx2 v6, v[4:5], s[2:3]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: v_mul_i64_zext_00:
+; GFX11-LABEL: v_mul_i64_no_zext:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x2c
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v6, 3, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    global_load_b64 v[0:1], v6, s[0:1]
 ; GFX11-NEXT:    global_load_b64 v[2:3], v6, s[2:3]
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
; GFX11-NEXT:    s_endpgm
   %a = load i64, ptr addrspace(1) %gep.a
   %b = load i64, ptr addrspace(1) %gep.b
   %mul = mul i64 %a, %b
   store i64 %mul, ptr addrspace(1) %gep.out
   ret void
 }
 
 ; a 64 bit multiplication where the second argument was zero extended.
-define amdgpu_kernel void @v_mul_i64_zext_01(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: v_mul_i64_zext_01:
+define amdgpu_kernel void @v_mul_i64_zext_src1(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul_i64_zext_src1:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v3, 2, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    global_load_dwordx2 v[0:1], v2, s[6:7]
 ; GFX10-NEXT:    global_load_dword v4, v3, s[2:3]
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_mad_u64_u32 v[2:3], s0, v0, v4, 0
 ; GFX10-NEXT:    v_mul_lo_u32 v0, v1, v4
 ; GFX10-NEXT:    v_add_nc_u32_e32 v3, v3, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-NEXT:    global_store_dwordx2 v0, v[2:3], s[4:5]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: v_mul_i64_zext_01:
+; GFX11-LABEL: v_mul_i64_zext_src1:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 3, v0
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    global_load_b64 v[0:1], v1, s[6:7]
; GFX11-NEXT:    s_endpgm
   %b = load i32, ptr addrspace(1) %gep.b
   %b_ext = zext i32 %b to i64
   %mul = mul i64 %a, %b_ext
   store i64 %mul, ptr addrspace(1) %out
   ret void
 }
 
 ; 64 bit multiplication where the first argument was zero extended.
-define amdgpu_kernel void @v_mul_i64_zext_10(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: v_mul_i64_zext_10:
+define amdgpu_kernel void @v_mul_i64_zext_src0(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul_i64_zext_src0:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v3, 3, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    global_load_dword v4, v2, s[6:7]
 ; GFX10-NEXT:    global_load_dwordx2 v[0:1], v3, s[2:3]
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_mad_u64_u32 v[2:3], s0, v4, v0, 0
 ; GFX10-NEXT:    v_mul_lo_u32 v0, v4, v1
 ; GFX10-NEXT:    v_add_nc_u32_e32 v3, v3, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-NEXT:    global_store_dwordx2 v0, v[2:3], s[4:5]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: v_mul_i64_zext_10:
+; GFX11-LABEL: v_mul_i64_zext_src0:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    global_load_b32 v4, v1, s[6:7]
 ; GFX11-NEXT:    global_load_b64 v[0:1], v0, s[0:1]
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    v_mad_u64_u32 v[2:3], null, v4, v0, 0
 ; GFX11-NEXT:    v_mul_lo_u32 v0, v4, v1
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
 ; GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_add_nc_u32 v3, v3, v0
 ; GFX11-NEXT:    global_store_b64 v0, v[2:3], s[4:5]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep.a = getelementptr inbounds i32, ptr addrspace(1) %aptr, i32 %tid
   %gep.b = getelementptr inbounds i64, ptr addrspace(1) %bptr, i32 %tid
   %a = load i32, ptr addrspace(1) %gep.a
   %b = load i64, ptr addrspace(1) %gep.b
   %a_ext = zext i32 %a to i64
   %mul = mul i64 %a_ext, %b
   store i64 %mul, ptr addrspace(1) %out
   ret void
 }
 
 ; 64-bit multiplication where both arguments were zero extended.
-define amdgpu_kernel void @v_mul_i64_zext_11(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: v_mul_i64_zext_11:
+define amdgpu_kernel void @v_mul_i64_zext_src0_src1(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul_i64_zext_src0_src1:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    global_load_dword v1, v0, s[6:7]
 ; GFX10-NEXT:    global_load_dword v2, v0, s[2:3]
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_mad_u64_u32 v[0:1], s0, v1, v2, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10-NEXT:    global_store_dwordx2 v2, v[0:1], s[4:5]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: v_mul_i64_zext_11:
+; GFX11-LABEL: v_mul_i64_zext_src0_src1:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX11-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    s_clause 0x1
; GFX11-NEXT:    s_endpgm
   %a_ext = zext i32 %a to i64
   %b_ext = zext i32 %b to i64
   %mul = mul i64 %a_ext, %b_ext
   store i64 %mul, ptr addrspace(1) %out
   ret void
 }
 
 ; 64-bit multiplication where the upper bytes of the first argument were masked.
-define amdgpu_kernel void @v_mul_i64_and_a_hi(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: v_mul_i64_and_a_hi:
+define amdgpu_kernel void @v_mul_i64_masked_src0_hi(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul_i64_masked_src0_hi:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    global_load_dword v4, v2, s[6:7]
 ; GFX10-NEXT:    global_load_dwordx2 v[0:1], v2, s[2:3]
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_mad_u64_u32 v[2:3], s0, v4, v0, 0
 ; GFX10-NEXT:    v_mul_lo_u32 v0, v4, v1
 ; GFX10-NEXT:    v_add_nc_u32_e32 v3, v3, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-NEXT:    global_store_dwordx2 v0, v[2:3], s[4:5]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: v_mul_i64_and_a_hi:
+; GFX11-LABEL: v_mul_i64_masked_src0_hi:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    global_load_b32 v4, v0, s[6:7]
 ; GFX11-NEXT:    global_load_b64 v[0:1], v0, s[0:1]
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    v_mad_u64_u32 v[2:3], null, v4, v0, 0
 ; GFX11-NEXT:    v_mul_lo_u32 v0, v4, v1
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
 ; GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_add_nc_u32 v3, v3, v0
 ; GFX11-NEXT:    global_store_b64 v0, v[2:3], s[4:5]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
  %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep.a = getelementptr inbounds i64, ptr addrspace(1) %aptr, i32 %tid
   %gep.b = getelementptr inbounds i64, ptr addrspace(1) %bptr, i32 %tid
   %a = load i64, ptr addrspace(1) %gep.a
   %b = load i64, ptr addrspace(1) %gep.b
   %a_and = and i64 %a, u0x00000000FFFFFFFF
   %mul = mul i64 %a_and, %b
   store i64 %mul, ptr addrspace(1) %out
   ret void
 }
 
 ; 64-bit multiplication where lower bytes of first argument were masked.
-define amdgpu_kernel void @v_mul_i64_and_a_lo(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: v_mul_i64_and_a_lo:
+define amdgpu_kernel void @v_mul_i64_masked_src0_lo(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul_i64_masked_src0_lo:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[2:3]
-; GFX10-NEXT:    global_load_dwordx2 v[2:3], v4, s[6:7]
+; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
+; GFX10-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
 ; GFX10-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-NEXT:    v_mad_u64_u32 v[4:5], s0, 0, v0, 0
-; GFX10-NEXT:    v_mul_lo_u32 v1, 0, v1
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_mul_lo_u32 v0, v3, v0
-; GFX10-NEXT:    v_add3_u32 v5, v5, v1, v0
 ; GFX10-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10-NEXT:    global_store_dwordx2 v0, v[4:5], s[4:5]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_mul_lo_u32 v1, v1, v2
+; GFX10-NEXT:    global_store_dwordx2 v0, v[0:1], s[4:5]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: v_mul_i64_and_a_lo:
+; GFX11-LABEL: v_mul_i64_masked_src0_lo:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    s_load_b64 s[4:5], s[0:1], 0x34
-; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    s_clause 0x1
-; GFX11-NEXT:    global_load_b64 v[0:1], v2, s[4:5]
-; GFX11-NEXT:    global_load_b64 v[2:3], v2, s[2:3]
+; GFX11-NEXT:    global_load_b64 v[0:1], v2, s[6:7]
+; GFX11-NEXT:    global_load_b64 v[2:3], v2, s[0:1]
 ; GFX11-NEXT:    s_waitcnt vmcnt(1)
-; GFX11-NEXT:    v_mad_u64_u32 v[4:5], null, 0, v0, 0
-; GFX11-NEXT:    v_mul_lo_u32 v1, 0, v1
-; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_mul_lo_u32 v0, v3, v0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11-NEXT:    v_add3_u32 v5, v5, v1, v0
 ; GFX11-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11-NEXT:    global_store_b64 v0, v[4:5], s[0:1]
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_mul_lo_u32 v1, v1, v2
+; GFX11-NEXT:    global_store_b64 v0, v[0:1], s[4:5]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep.a = getelementptr inbounds i64, ptr addrspace(1) %aptr, i32 %tid
   %gep.b = getelementptr inbounds i64, ptr addrspace(1) %bptr, i32 %tid
   %a = load i64, ptr addrspace(1) %gep.a
   %b = load i64, ptr addrspace(1) %gep.b
   %a_and = and i64 %a, u0xFFFFFFFF00000000
   %mul = mul i64 %a_and, %b
   store i64 %mul, ptr addrspace(1) %out
   ret void
 }
 
 ; 64-bit multiplication where the lower bytes of the second argument were masked.
-define amdgpu_kernel void @v_mul_i64_and_b_lo(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: v_mul_i64_and_b_lo:
+define amdgpu_kernel void @v_mul_i64_masked_src1_lo(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul_i64_masked_src1_lo:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
-; GFX10-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
+; GFX10-NEXT:    v_lshlrev_b32_e32 v3, 3, v0
+; GFX10-NEXT:    ; kill: killed $vgpr3
+; GFX10-NEXT:    ; kill: killed $sgpr6_sgpr7
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
-; GFX10-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
-; GFX10-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-NEXT:    v_mad_u64_u32 v[4:5], s0, v0, 0, 0
+; GFX10-NEXT:    global_load_dwordx2 v[0:1], v3, s[6:7]
+; GFX10-NEXT:    global_load_dwordx2 v[1:2], v3, s[2:3]
+; GFX10-NEXT:    ; kill: killed $sgpr2_sgpr3
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_mul_lo_u32 v0, v0, v3
-; GFX10-NEXT:    v_mul_lo_u32 v1, v1, 0
-; GFX10-NEXT:    v_add3_u32 v5, v5, v0, v1
+; GFX10-NEXT:    v_mul_lo_u32 v1, v0, v2
 ; GFX10-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10-NEXT:    global_store_dwordx2 v0, v[4:5], s[4:5]
+; GFX10-NEXT:    global_store_dwordx2 v0, v[0:1], s[4:5]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: v_mul_i64_and_b_lo:
+; GFX11-LABEL: v_mul_i64_masked_src1_lo:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    global_load_b64 v[0:1], v2, s[6:7]
-; GFX11-NEXT:    global_load_b64 v[2:3], v2, s[0:1]
-; GFX11-NEXT:    s_waitcnt vmcnt(1)
-; GFX11-NEXT:    v_mad_u64_u32 v[4:5], null, v0, 0, 0
+; GFX11-NEXT:    global_load_b64 v[1:2], v2, s[0:1]
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_mul_lo_u32 v0, v0, v3
-; GFX11-NEXT:    v_mul_lo_u32 v1, v1, 0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11-NEXT:    v_add3_u32 v5, v5, v0, v1
+; GFX11-NEXT:    v_mul_lo_u32 v1, v0, v2
 ; GFX11-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11-NEXT:    global_store_b64 v0, v[4:5], s[4:5]
+; GFX11-NEXT:    global_store_b64 v0, v[0:1], s[4:5]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep.a = getelementptr inbounds i64, ptr addrspace(1) %aptr, i32 %tid
   %gep.b = getelementptr inbounds i64, ptr addrspace(1) %bptr, i32 %tid
   %a = load i64, ptr addrspace(1) %gep.a
   %b = load i64, ptr addrspace(1) %gep.b
   %b_and = and i64 %b, u0xFFFFFFFF00000000
   %mul = mul i64 %a, %b_and
   store i64 %mul, ptr addrspace(1) %out
   ret void
 }
 
 ; 64-bit multiplication where the entire first argument is masked.
-define amdgpu_kernel void @v_mul_i64_and_hilo(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: v_mul_i64_and_hilo:
+define amdgpu_kernel void @v_mul_i64_masked_src0(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul_i64_masked_src0:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: v_mul_i64_and_hilo:
+; GFX11-LABEL: v_mul_i64_masked_src0:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX11-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_mov_b32 v2, 0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm

 ; GFX10-NEXT:    v_mad_u64_u32 v[0:1], s0, v4, v2, 0
 ; GFX10-NEXT:    v_mul_lo_u32 v3, v4, v3
 ; GFX10-NEXT:    v_mul_lo_u32 v2, v5, v2
 ; GFX10-NEXT:    v_add3_u32 v1, v1, v3, v2
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10-NEXT:    global_store_dwordx2 v2, v[0:1], s[4:5]
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: v_mul_i64_partially_masked_src0:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    global_load_b64 v[0:1], v2, s[6:7]
; GFX11-NEXT:    s_endpgm
   %b = load i64, ptr addrspace(1) %gep.b
   %a_and = and i64 %a, u0x0000F00FFFF00000
   %mul = mul i64 %a_and, %b
   store i64 %mul, ptr addrspace(1) %out
   ret void
 }
 
 ; 64-bit multiplication, where the first argument is masked before a branch
-define amdgpu_kernel void @mul64_and_in_branch(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: mul64_and_in_branch:
+define amdgpu_kernel void @v_mul64_masked_before_branch(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul64_masked_before_branch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: mul64_and_in_branch:
+; GFX11-LABEL: v_mul64_masked_before_branch:
 ; GFX11:       ; %bb.0: ; %entry
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX11-NEXT:    v_dual_mov_b32 v1, 0 :: v_dual_mov_b32 v2, 0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
else:
   br label %endif
 
 endif:
   %3 = phi i64 [%1, %if], [%2, %else]
   store i64 %3, ptr addrspace(1) %out
   ret void
 }
 
-; 64-bit multiplication with both arguments changed in differnt basic blocks.
-define amdgpu_kernel void @mul64_and_in_branch_2(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
-; GFX10-LABEL: mul64_and_in_branch_2:
+; 64-bit multiplication with both arguments changed in different basic blocks.
+define amdgpu_kernel void @v_mul64_masked_before_and_in_branch(ptr addrspace(1) %out, ptr addrspace(1) %aptr, ptr addrspace(1) %bptr) {
+; GFX10-LABEL: v_mul64_masked_before_and_in_branch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
-; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
+; GFX10-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    s_clause 0x1
-; GFX10-NEXT:    global_load_dwordx2 v[2:3], v0, s[6:7]
-; GFX10-NEXT:    global_load_dwordx2 v[4:5], v0, s[2:3]
-; GFX10-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; GFX10-NEXT:    global_load_dwordx2 v[2:3], v4, s[6:7]
+; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[2:3]
 ; GFX10-NEXT:    s_waitcnt vmcnt(1)
 ; GFX10-NEXT:    v_cmp_ge_u64_e32 vcc_lo, 0, v[2:3]
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_mul_lo_u32 v5, v2, v5
+; GFX10-NEXT:    v_mul_lo_u32 v1, v2, v1
 ; GFX10-NEXT:    s_and_saveexec_b32 s0, vcc_lo
 ; GFX10-NEXT:    s_xor_b32 s0, exec_lo, s0
 ; GFX10-NEXT:  ; %bb.1: ; %else
-; GFX10-NEXT:    v_mad_u64_u32 v[0:1], s1, v2, v4, 0
-; GFX10-NEXT:    v_mul_lo_u32 v2, 0, v4
-; GFX10-NEXT:    v_add3_u32 v1, v1, v5, v2
-; GFX10-NEXT:    ; implicit-def: $vgpr2_vgpr3
-; GFX10-NEXT:    ; implicit-def: $vgpr5
+; GFX10-NEXT:    v_mad_u64_u32 v[2:3], s1, v2, v0, 0
+; GFX10-NEXT:    v_add_nc_u32_e32 v3, v3, v1
+; GFX10-NEXT:    v_mov_b32_e32 v0, v2
+; GFX10-NEXT:    v_mov_b32_e32 v1, v3
 ; GFX10-NEXT:  ; %bb.2: ; %Flow
 ; GFX10-NEXT:    s_andn2_saveexec_b32 s0, s0
 ; GFX10-NEXT:  ; %bb.3: ; %if
-; GFX10-NEXT:    v_mad_u64_u32 v[0:1], s1, v2, 0, 0
-; GFX10-NEXT:    v_mul_lo_u32 v2, 0, 0
-; GFX10-NEXT:    v_add3_u32 v1, v1, v5, v2
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-NEXT:  ; %bb.4: ; %endif
 ; GFX10-NEXT:    s_or_b32 exec_lo, exec_lo, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10-NEXT:    global_store_dwordx2 v2, v[0:1], s[4:5]
 ; GFX10-NEXT:    s_endpgm
 ;
-; GFX11-LABEL: mul64_and_in_branch_2:
+; GFX11-LABEL: v_mul64_masked_before_and_in_branch:
 ; GFX11:       ; %bb.0: ; %entry
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    s_load_b128 s[4:7], s[0:1], 0x24
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x34
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    s_clause 0x1
 ; GFX11-NEXT:    global_load_b64 v[2:3], v0, s[6:7]
-; GFX11-NEXT:    global_load_b64 v[4:5], v0, s[0:1]
+; GFX11-NEXT:    global_load_b64 v[0:1], v0, s[0:1]
 ; GFX11-NEXT:    s_mov_b32 s0, exec_lo
-; GFX11-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_mul_lo_u32 v5, v2, v5
+; GFX11-NEXT:    v_mul_lo_u32 v1, v2, v1
 ; GFX11-NEXT:    v_cmpx_ge_u64_e32 0, v[2:3]
 ; GFX11-NEXT:    s_xor_b32 s0, exec_lo, s0
 ; GFX11-NEXT:  ; %bb.1: ; %else
-; GFX11-NEXT:    v_mad_u64_u32 v[0:1], null, v2, v4, 0
-; GFX11-NEXT:    v_mul_lo_u32 v2, 0, v4
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11-NEXT:    v_add3_u32 v1, v1, v5, v2
-; GFX11-NEXT:    ; implicit-def: $vgpr2_vgpr3
-; GFX11-NEXT:    ; implicit-def: $vgpr5
+; GFX11-NEXT:    v_mad_u64_u32 v[2:3], null, v2, v0, 0
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT:    v_dual_mov_b32 v0, v2 :: v_dual_add_nc_u32 v3, v3, v1
+; GFX11-NEXT:    v_mov_b32_e32 v1, v3
 ; GFX11-NEXT:  ; %bb.2: ; %Flow
 ; GFX11-NEXT:    s_and_not1_saveexec_b32 s0, s0
 ; GFX11-NEXT:  ; %bb.3: ; %if
-; GFX11-NEXT:    v_mad_u64_u32 v[0:1], null, v2, 0, 0
-; GFX11-NEXT:    v_mul_lo_u32 v2, 0, 0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11-NEXT:    v_add3_u32 v1, v1, v5, v2
+; GFX11-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX11-NEXT:  ; %bb.4: ; %endif
 ; GFX11-NEXT:    s_or_b32 exec_lo, exec_lo, s0
 ; GFX11-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[4:5]
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()

llvm/test/CodeGen/AMDGPU/GlobalISel/mul-known-bits.i64.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs -global-isel=1 -run-pass=legalizer -o - %s | FileCheck -check-prefix=GFX10 %s
 
 ---
 name:            v_mul_i64_no_zext
 body:             |
   bb.0:
     liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
 
     ; GFX10-LABEL: name: v_mul_i64_no_zext
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
     ; GFX10-NEXT: {{  $}}
     ; GFX10-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY $vgpr0_vgpr1
     ; GFX10-NEXT: [[COPY1:%[0-9]+]]:_(s64) = COPY $vgpr2_vgpr3
     ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY]](s64)
     ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY1]](s64)
     ; GFX10-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
     ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C]]
bb.0:
     ; GFX10-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY $vgpr0_vgpr1
     ; GFX10-NEXT: [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr2
     ; GFX10-NEXT: [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[COPY1]](s32)
     ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY]](s64)
     ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[ZEXT]](s64)
     ; GFX10-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
     ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C]]
     ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_MAD_U64_U32_]](s64)
-    ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV]], [[UV3]]
+    ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
     ; GFX10-NEXT: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[UV5]], [[MUL]]
-    ; GFX10-NEXT: [[MUL1:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
-    ; GFX10-NEXT: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ADD]], [[MUL1]]
-    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD1]](s32)
+    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD]](s32)
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[MV]](s64)
     %0:_(s64) = COPY $vgpr0_vgpr1
     %1:_(s32) = COPY $vgpr2
     %2:_(s64) = G_ZEXT %1(s32)
     %3:_(s64) = G_MUL %0, %2
     $vgpr0_vgpr1 = COPY %3
 
 ...
bb.0:
     ; GFX10-NEXT: [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[COPY]](s32)
     ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[ZEXT]](s64)
     ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY1]](s64)
     ; GFX10-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
     ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C]]
     ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_MAD_U64_U32_]](s64)
     ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV]], [[UV3]]
     ; GFX10-NEXT: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[UV5]], [[MUL]]
-    ; GFX10-NEXT: [[MUL1:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
-    ; GFX10-NEXT: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ADD]], [[MUL1]]
-    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD1]](s32)
+    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD]](s32)
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[MV]](s64)
     %0:_(s32) = COPY $vgpr0
     %1:_(s64) = COPY $vgpr2_vgpr3
     %2:_(s64) = G_ZEXT %0(s32)
     %3:_(s64) = G_MUL %2, %1
     $vgpr0_vgpr1 = COPY %3
 
 ...
bb.0:
     ; GFX10-NEXT: [[AND:%[0-9]+]]:_(s64) = G_AND [[COPY]], [[C]]
     ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AND]](s64)
     ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY1]](s64)
     ; GFX10-NEXT: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
     ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C1]]
     ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_MAD_U64_U32_]](s64)
     ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV]], [[UV3]]
     ; GFX10-NEXT: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[UV5]], [[MUL]]
-    ; GFX10-NEXT: [[MUL1:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
-    ; GFX10-NEXT: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ADD]], [[MUL1]]
-    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD1]](s32)
+    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD]](s32)
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[MV]](s64)
     %0:_(s64) = COPY $vgpr0_vgpr1
     %1:_(s64) = COPY $vgpr2_vgpr3
     %2:_(s64) = G_CONSTANT i64 4294967295
     %3:_(s64) = G_AND %0, %2
     %4:_(s64) = G_MUL %3, %1
     $vgpr0_vgpr1 = COPY %4
 ...
 ---
 name:            v_mul_i64_masked_src0_lo
 body:             |
   bb.0:
     liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
 
     ; GFX10-LABEL: name: v_mul_i64_masked_src0_lo
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
     ; GFX10-NEXT: {{  $}}
     ; GFX10-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY $vgpr0_vgpr1
     ; GFX10-NEXT: [[COPY1:%[0-9]+]]:_(s64) = COPY $vgpr2_vgpr3
     ; GFX10-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 -4294967296
     ; GFX10-NEXT: [[AND:%[0-9]+]]:_(s64) = G_AND [[COPY]], [[C]]
     ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AND]](s64)
     ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY1]](s64)
     ; GFX10-NEXT: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
-    ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C1]]
-    ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_MAD_U64_U32_]](s64)
-    ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV]], [[UV3]]
-    ; GFX10-NEXT: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[UV5]], [[MUL]]
-    ; GFX10-NEXT: [[MUL1:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
-    ; GFX10-NEXT: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ADD]], [[MUL1]]
-    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD1]](s32)
+    ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[C1]](s64)
+    ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
+    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[MUL]](s32)
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[MV]](s64)
     %0:_(s64) = COPY $vgpr0_vgpr1
     %1:_(s64) = COPY $vgpr2_vgpr3
     %2:_(s64) = G_CONSTANT i64 -4294967296
     %3:_(s64) = G_AND %0, %2
     %4:_(s64) = G_MUL %3, %1
     $vgpr0_vgpr1 = COPY %4
 
bb.0:
     ; GFX10-NEXT: {{  $}}
     ; GFX10-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY $vgpr0_vgpr1
     ; GFX10-NEXT: [[COPY1:%[0-9]+]]:_(s64) = COPY $vgpr2_vgpr3
     ; GFX10-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 -4294967296
     ; GFX10-NEXT: [[AND:%[0-9]+]]:_(s64) = G_AND [[COPY1]], [[C]]
     ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY]](s64)
     ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AND]](s64)
     ; GFX10-NEXT: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
-    ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C1]]
-    ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_MAD_U64_U32_]](s64)
+    ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[C1]](s64)
     ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV]], [[UV3]]
-    ; GFX10-NEXT: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[UV5]], [[MUL]]
-    ; GFX10-NEXT: [[MUL1:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
-    ; GFX10-NEXT: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ADD]], [[MUL1]]
-    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD1]](s32)
+    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[MUL]](s32)
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[MV]](s64)
     %0:_(s64) = COPY $vgpr0_vgpr1
     %1:_(s64) = COPY $vgpr2_vgpr3
     %2:_(s64) = G_CONSTANT i64 -4294967296
     %3:_(s64) = G_AND %1, %2
     %4:_(s64) = G_MUL %0, %3
     $vgpr0_vgpr1 = COPY %4
 
 ...
 ---
 name:            v_mul_i64_masked_src0
 body:             |
   bb.0:
     liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
 
     ; GFX10-LABEL: name: v_mul_i64_masked_src0
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2_vgpr3
     ; GFX10-NEXT: {{  $}}
-    ; GFX10-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY $vgpr0_vgpr1
-    ; GFX10-NEXT: [[COPY1:%[0-9]+]]:_(s64) = COPY $vgpr2_vgpr3
     ; GFX10-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
-    ; GFX10-NEXT: [[AND:%[0-9]+]]:_(s64) = G_AND [[COPY]], [[C]]
-    ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AND]](s64)
-    ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY1]](s64)
-    ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C]]
-    ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_MAD_U64_U32_]](s64)
-    ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV]], [[UV3]]
-    ; GFX10-NEXT: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[UV5]], [[MUL]]
-    ; GFX10-NEXT: [[MUL1:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
-    ; GFX10-NEXT: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ADD]], [[MUL1]]
-    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD1]](s32)
-    ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[MV]](s64)
+    ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[C]](s64)
     %0:_(s64) = COPY $vgpr0_vgpr1
     %1:_(s64) = COPY $vgpr2_vgpr3
     %2:_(s64) = G_CONSTANT i64 0
     %3:_(s64) = G_AND %0, %2
     %4:_(s64) = G_MUL %3, %1
     $vgpr0_vgpr1 = COPY %4
 ...
 ---
bb.0:
     ; GFX10-LABEL: name: v_mul_i64_constant_hi
     ; GFX10: liveins: $vgpr0_vgpr1
     ; GFX10-NEXT: {{  $}}
     ; GFX10-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY $vgpr0_vgpr1
     ; GFX10-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 -4294967296
     ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY]](s64)
     ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[C]](s64)
     ; GFX10-NEXT: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
-    ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C1]]
-    ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_MAD_U64_U32_]](s64)
+    ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[C1]](s64)
     ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV]], [[UV3]]
-    ; GFX10-NEXT: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[UV5]], [[MUL]]
-    ; GFX10-NEXT: [[MUL1:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
-    ; GFX10-NEXT: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ADD]], [[MUL1]]
-    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD1]](s32)
+    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[MUL]](s32)
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[MV]](s64)
     %0:_(s64) = COPY $vgpr0_vgpr1
     %1:_(s64) = G_CONSTANT i64 -4294967296
     %2:_(s64) = G_MUL %0, %1
     $vgpr0_vgpr1 = COPY %2
 ...
 ---
 name:            v_mul_i64_constant_lo
 body:             |
   bb.0:
     liveins: $vgpr0_vgpr1
 
     ; GFX10-LABEL: name: v_mul_i64_constant_lo
     ; GFX10: liveins: $vgpr0_vgpr1
     ; GFX10-NEXT: {{  $}}
     ; GFX10-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY $vgpr0_vgpr1
     ; GFX10-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 4294967295
     ; GFX10-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[COPY]](s64)
     ; GFX10-NEXT: [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[C]](s64)
     ; GFX10-NEXT: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
     ; GFX10-NEXT: [[AMDGPU_MAD_U64_U32_:%[0-9]+]]:_(s64), [[AMDGPU_MAD_U64_U32_1:%[0-9]+]]:_(s1) = G_AMDGPU_MAD_U64_U32 [[UV]](s32), [[UV2]], [[C1]]
     ; GFX10-NEXT: [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_MAD_U64_U32_]](s64)
-    ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV]], [[UV3]]
+    ; GFX10-NEXT: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
     ; GFX10-NEXT: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[UV5]], [[MUL]]
-    ; GFX10-NEXT: [[MUL1:%[0-9]+]]:_(s32) = G_MUL [[UV1]], [[UV2]]
-    ; GFX10-NEXT: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ADD]], [[MUL1]]
-    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD1]](s32)
+    ; GFX10-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[UV4]](s32), [[ADD]](s32)
     ; GFX10-NEXT: $vgpr0_vgpr1 = COPY [[MV]](s64)
     %0:_(s64) = COPY $vgpr0_vgpr1
     %1:_(s64) = G_CONSTANT i64 4294967295
     %2:_(s64) = G_MUL %0, %1
     $vgpr0_vgpr1 = COPY %2
 ...

llvm/test/CodeGen/AMDGPU/GlobalISel/sdiv.i64.ll

 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v7, v4
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v6, vcc, v6, v3, vcc
 ; CHECK-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s6, v5, 0
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v7, 31, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v0, v7
 ; CHECK-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], s6, v6, v[3:4]
-; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v1, v7, vcc
-; CHECK-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], -1, v5, v[3:4]
-; CHECK-NEXT:    v_xor_b32_e32 v4, v0, v7
-; CHECK-NEXT:    v_mul_lo_u32 v0, v6, v2
-; CHECK-NEXT:    v_mul_lo_u32 v8, v5, v3
-; CHECK-NEXT:    v_xor_b32_e32 v9, v1, v7
-; CHECK-NEXT:    v_mul_hi_u32 v1, v5, v2
+; CHECK-NEXT:    v_addc_u32_e32 v9, vcc, v1, v7, vcc
+; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], -1, v5, v[3:4]
+; CHECK-NEXT:    v_xor_b32_e32 v3, v8, v7
+; CHECK-NEXT:    v_mul_lo_u32 v1, v6, v2
+; CHECK-NEXT:    v_mul_lo_u32 v8, v5, v0
+; CHECK-NEXT:    v_xor_b32_e32 v4, v9, v7
+; CHECK-NEXT:    v_mul_hi_u32 v9, v5, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v2, v6, v2
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
-; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v1, v6, v3
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
-; CHECK-NEXT:    v_mul_hi_u32 v8, v5, v3
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v8
 ; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
-; CHECK-NEXT:    v_mul_hi_u32 v3, v6, v3
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v9
 ; CHECK-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
+; CHECK-NEXT:    v_mul_lo_u32 v9, v6, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v8, v1
+; CHECK-NEXT:    v_mul_hi_u32 v8, v5, v0
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v9, v2
+; CHECK-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; CHECK-NEXT:    v_mul_hi_u32 v0, v6, v0
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v6, v1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v2, v9, v0
-; CHECK-NEXT:    v_mul_lo_u32 v3, v4, v1
-; CHECK-NEXT:    v_mul_hi_u32 v5, v4, v0
-; CHECK-NEXT:    v_mul_hi_u32 v0, v9, v0
+; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v8, v2
+; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
+; CHECK-NEXT:    v_addc_u32_e32 v0, vcc, v6, v0, vcc
+; CHECK-NEXT:    v_mul_lo_u32 v2, v4, v1
+; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v0
+; CHECK-NEXT:    v_mul_hi_u32 v8, v3, v1
+; CHECK-NEXT:    v_mul_hi_u32 v1, v4, v1
 ; CHECK-NEXT:    s_movk_i32 s6, 0x1000
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
+; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v5, v9, v1
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; CHECK-NEXT:    v_mul_hi_u32 v3, v4, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
+; CHECK-NEXT:    v_mul_lo_u32 v8, v4, v0
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CHECK-NEXT:    v_mul_hi_u32 v5, v3, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v8, v1
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v0, v2
-; CHECK-NEXT:    v_mul_hi_u32 v6, v9, v1
-; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v5, 0
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v1, v2
+; CHECK-NEXT:    v_mul_hi_u32 v9, v4, v0
+; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v8, 0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v6, v2
-; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], s6, v3, v[1:2]
-; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v4, v0
-; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], 0, v5, v[1:2]
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v9, v2
+; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], s6, v5, v[1:2]
+; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v3, v0
+; CHECK-NEXT:    v_subb_u32_e64 v2, s[4:5], v4, v1, vcc
+; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v4, v1
 ; CHECK-NEXT:    v_mov_b32_e32 v6, 0x1000
-; CHECK-NEXT:    s_bfe_i32 s6, 1, 0x10000
-; CHECK-NEXT:    v_subb_u32_e64 v2, s[4:5], v9, v1, vcc
-; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v9, v1
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
+; CHECK-NEXT:    s_bfe_i32 s6, 1, 0x10000
 ; CHECK-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v6
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
-; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
-; CHECK-NEXT:    v_mov_b32_e32 v8, s6
+; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, -1, s[4:5]
+; CHECK-NEXT:    v_mov_b32_e32 v4, s6
 ; CHECK-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v2
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, v8, v4, s[4:5]
-; CHECK-NEXT:    v_add_i32_e32 v4, vcc, 1, v5
-; CHECK-NEXT:    v_addc_u32_e32 v8, vcc, 0, v3, vcc
+; CHECK-NEXT:    v_cndmask_b32_e64 v2, v4, v3, s[4:5]
+; CHECK-NEXT:    v_add_i32_e32 v3, vcc, 1, v8
+; CHECK-NEXT:    v_addc_u32_e32 v4, vcc, 0, v5, vcc
 ; CHECK-NEXT:    s_bfe_i32 s4, 1, 0x10000
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v6
 ; CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, -1, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v6, s4
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v6, v0, vcc
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, 1, v4
-; CHECK-NEXT:    v_addc_u32_e32 v6, vcc, 0, v8, vcc
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, 1, v3
+; CHECK-NEXT:    v_addc_u32_e32 v6, vcc, 0, v4, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
-; CHECK-NEXT:    v_cndmask_b32_e32 v0, v4, v1, vcc
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v8, v6, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v0, v3, v1, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v4, v6, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
-; CHECK-NEXT:    v_cndmask_b32_e32 v0, v5, v0, vcc
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v0, v8, v0, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v5, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v0, v7
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v1, v7
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v7
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v7, vcc
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv i64 %num, 4096
   ret i64 %result
 }

 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v7, v4
-; CGP-NEXT:    v_addc_u32_e32 v8, vcc, v8, v5, vcc
-; CGP-NEXT:    v_mad_u64_u32 v[5:6], s[4:5], s6, v9, 0
-; CGP-NEXT:    v_mov_b32_e32 v4, v6
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v8, v[4:5]
-; CGP-NEXT:    v_ashrrev_i32_e32 v4, 31, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], -1, v9, v[6:7]
-; CGP-NEXT:    v_addc_u32_e32 v1, vcc, v1, v4, vcc
-; CGP-NEXT:    v_xor_b32_e32 v10, v0, v4
-; CGP-NEXT:    v_mul_lo_u32 v0, v8, v5
-; CGP-NEXT:    v_mul_lo_u32 v7, v9, v6
-; CGP-NEXT:    v_xor_b32_e32 v11, v1, v4
-; CGP-NEXT:    v_mul_hi_u32 v1, v9, v5
-; CGP-NEXT:    v_mul_hi_u32 v5, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v1, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v9, 0
+; CGP-NEXT:    v_addc_u32_e32 v10, vcc, v8, v5, vcc
+; CGP-NEXT:    v_ashrrev_i32_e32 v5, 31, v1
+; CGP-NEXT:    v_mov_b32_e32 v4, v7
+; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s6, v10, v[4:5]
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v0, v5
+; CGP-NEXT:    v_addc_u32_e32 v11, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], -1, v9, v[7:8]
+; CGP-NEXT:    v_xor_b32_e32 v8, v4, v5
+; CGP-NEXT:    v_mul_lo_u32 v1, v10, v6
+; CGP-NEXT:    v_mul_lo_u32 v4, v9, v0
 ; CGP-NEXT:    v_mul_hi_u32 v7, v9, v6
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_mul_hi_u32 v6, v10, v6
+; CGP-NEXT:    v_xor_b32_e32 v11, v11, v5
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; CGP-NEXT:    v_mul_hi_u32 v6, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v6, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v9, v0
-; CGP-NEXT:    v_addc_u32_e32 v1, vcc, v8, v1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v5, v11, v0
-; CGP-NEXT:    v_mul_lo_u32 v6, v10, v1
-; CGP-NEXT:    v_mul_hi_u32 v7, v10, v0
-; CGP-NEXT:    v_mul_hi_u32 v0, v11, v0
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
+; CGP-NEXT:    v_mul_lo_u32 v7, v10, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; CGP-NEXT:    v_mul_hi_u32 v4, v9, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v7, v11, v1
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; CGP-NEXT:    v_mul_hi_u32 v6, v10, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_mul_hi_u32 v0, v10, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v9, v1
+; CGP-NEXT:    v_addc_u32_e32 v0, vcc, v10, v0, vcc
+; CGP-NEXT:    v_mul_lo_u32 v6, v11, v1
+; CGP-NEXT:    v_mul_lo_u32 v7, v8, v0
+; CGP-NEXT:    v_mul_hi_u32 v9, v8, v1
+; CGP-NEXT:    v_mul_hi_u32 v1, v11, v1
+; CGP-NEXT:    v_mul_hi_u32 v10, v11, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v7
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v6
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v9, v11, v0
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v0, v5
-; CGP-NEXT:    v_mul_hi_u32 v7, v11, v1
-; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s7, v8, 0
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v7, v5
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v9, v[1:2]
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v10, v0
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], 0, v8, v[6:7]
-; CGP-NEXT:    v_mov_b32_e32 v5, 0x1000
-; CGP-NEXT:    v_mov_b32_e32 v10, s8
+; CGP-NEXT:    v_mul_hi_u32 v7, v8, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v9, v1
+; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v7
+; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v1, v6
+; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s7, v9, 0
+; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_add_i32_e32 v10, vcc, v10, v6
+; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v10, v[1:2]
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v8, v0
+; CGP-NEXT:    v_mov_b32_e32 v4, 0x1000
 ; CGP-NEXT:    v_subb_u32_e64 v1, s[4:5], v11, v6, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v6, s[4:5], v11, v6
-; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v5
+; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v1
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v6, vcc
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v6, 0x1000
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v10, v10, v7, s[4:5]
+; CGP-NEXT:    v_mov_b32_e32 v8, s8
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v8, v8, v7, s[4:5]
 ; CGP-NEXT:    v_subbrev_u32_e32 v7, vcc, 0, v1, vcc
 ; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v1, 0
 ; CGP-NEXT:    v_mac_f32_e32 v6, 0x4f800000, v1
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v1, v6
-; CGP-NEXT:    v_add_i32_e32 v11, vcc, 1, v8
-; CGP-NEXT:    v_addc_u32_e32 v12, vcc, 0, v9, vcc
+; CGP-NEXT:    v_add_i32_e32 v11, vcc, 1, v9
+; CGP-NEXT:    v_addc_u32_e32 v12, vcc, 0, v10, vcc
 ; CGP-NEXT:    v_mul_f32_e32 v1, 0x5f7ffffc, v1
 ; CGP-NEXT:    v_mul_f32_e32 v6, 0x2f800000, v1
 ; CGP-NEXT:    v_trunc_f32_e32 v6, v6
 ; CGP-NEXT:    v_mac_f32_e32 v1, 0xcf800000, v6
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v13, v1
 ; CGP-NEXT:    s_bfe_i32 s4, 1, 0x10000
-; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v5
+; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v4
 ; CGP-NEXT:    v_mov_b32_e32 v15, s4
 ; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v13, 0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v16, v6
 ; CGP-NEXT:    v_cndmask_b32_e64 v14, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v7
 ; CGP-NEXT:    v_cndmask_b32_e32 v14, v15, v14, vcc
 ; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v16, v[1:2]
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, 1, v11

 ; CGP-NEXT:    v_mul_hi_u32 v6, v16, v6
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v11, v1
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v6, v1
 ; CGP-NEXT:    v_add_i32_e32 v11, vcc, v13, v0
 ; CGP-NEXT:    v_addc_u32_e32 v13, vcc, v16, v1, vcc
 ; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v11, 0
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
-; CGP-NEXT:    v_cndmask_b32_e32 v8, v8, v7, vcc
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
+; CGP-NEXT:    v_cndmask_b32_e32 v6, v9, v7, vcc
+; CGP-NEXT:    v_xor_b32_e32 v9, v6, v5
 ; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v13, v[1:2]
-; CGP-NEXT:    v_xor_b32_e32 v1, v8, v4
-; CGP-NEXT:    v_ashrrev_i32_e32 v8, 31, v3
+; CGP-NEXT:    v_cndmask_b32_e32 v8, v10, v12, vcc
+; CGP-NEXT:    v_xor_b32_e32 v1, v8, v5
 ; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], -1, v11, v[6:7]
-; CGP-NEXT:    v_cndmask_b32_e32 v9, v9, v12, vcc
+; CGP-NEXT:    v_ashrrev_i32_e32 v8, 31, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
 ; CGP-NEXT:    v_addc_u32_e32 v3, vcc, v3, v8, vcc
-; CGP-NEXT:    v_xor_b32_e32 v10, v2, v8
+; CGP-NEXT:    v_xor_b32_e32 v7, v2, v8
 ; CGP-NEXT:    v_mul_lo_u32 v2, v13, v0
-; CGP-NEXT:    v_mul_lo_u32 v7, v11, v6
+; CGP-NEXT:    v_mul_lo_u32 v10, v11, v6
 ; CGP-NEXT:    v_xor_b32_e32 v12, v3, v8
 ; CGP-NEXT:    v_mul_hi_u32 v3, v11, v0
 ; CGP-NEXT:    v_mul_hi_u32 v0, v13, v0
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v3, v13, v6
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v7, v2
-; CGP-NEXT:    v_mul_hi_u32 v7, v11, v6
+; CGP-NEXT:    v_add_i32_e32 v2, vcc, v10, v2
+; CGP-NEXT:    v_mul_hi_u32 v10, v11, v6
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
+; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v10
 ; CGP-NEXT:    v_mul_hi_u32 v6, v13, v6
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v6, v2
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v11, v0
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v11, v0
 ; CGP-NEXT:    v_addc_u32_e32 v2, vcc, v13, v2, vcc
-; CGP-NEXT:    v_mul_lo_u32 v3, v12, v0
-; CGP-NEXT:    v_mul_lo_u32 v6, v10, v2
-; CGP-NEXT:    v_mul_hi_u32 v7, v10, v0
-; CGP-NEXT:    v_mul_hi_u32 v0, v12, v0
-; CGP-NEXT:    v_xor_b32_e32 v9, v9, v4
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
-; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v7, v12, v2
+; CGP-NEXT:    v_mul_lo_u32 v6, v12, v3
+; CGP-NEXT:    v_mul_lo_u32 v10, v7, v2
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v9, v5
+; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mul_hi_u32 v5, v7, v3
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v6, v12, v2
+; CGP-NEXT:    v_mul_hi_u32 v3, v12, v3
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v9, v5
+; CGP-NEXT:    v_mul_hi_u32 v9, v7, v2
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v6, v3
-; CGP-NEXT:    v_mul_hi_u32 v6, v10, v2
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v6
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
-; CGP-NEXT:    v_add_i32_e32 v11, vcc, v0, v3
-; CGP-NEXT:    v_mul_hi_u32 v7, v12, v2
-; CGP-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s7, v11, 0
-; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v6, v0
-; CGP-NEXT:    v_add_i32_e32 v13, vcc, v7, v0
-; CGP-NEXT:    v_mov_b32_e32 v0, v3
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v13, v[0:1]
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v1, v4
-; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v9, v4, vcc
-; CGP-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], 0, v11, v[6:7]
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v10, v2
-; CGP-NEXT:    v_subb_u32_e64 v4, s[4:5], v12, v3, vcc
-; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v12, v3
-; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v9
+; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v9
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v3, v5
+; CGP-NEXT:    v_mul_hi_u32 v10, v12, v2
+; CGP-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s7, v9, 0
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
+; CGP-NEXT:    v_add_i32_e32 v10, vcc, v10, v5
+; CGP-NEXT:    v_mad_u64_u32 v[5:6], s[4:5], s7, v10, v[3:4]
+; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v7, v2
+; CGP-NEXT:    v_subb_u32_e64 v3, s[4:5], v12, v5, vcc
+; CGP-NEXT:    v_sub_i32_e64 v5, s[4:5], v12, v5
+; CGP-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v5, vcc
 ; CGP-NEXT:    s_bfe_i32 s6, 1, 0x10000
-; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v5
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v5
+; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v4
+; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[4:5]
 ; CGP-NEXT:    v_mov_b32_e32 v7, s6
-; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v4
-; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_cndmask_b32_e64 v4, v7, v6, s[4:5]
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, 1, v11
-; CGP-NEXT:    v_addc_u32_e32 v7, vcc, 0, v13, vcc
+; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v3
+; CGP-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v5, vcc
+; CGP-NEXT:    v_cndmask_b32_e64 v3, v7, v6, s[4:5]
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, 1, v9
+; CGP-NEXT:    v_addc_u32_e32 v7, vcc, 0, v10, vcc
 ; CGP-NEXT:    s_bfe_i32 s4, 1, 0x10000
-; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v5
+; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, -1, vcc
-; CGP-NEXT:    v_mov_b32_e32 v5, s4
-; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
-; CGP-NEXT:    v_cndmask_b32_e32 v2, v5, v2, vcc
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, 1, v6
+; CGP-NEXT:    v_mov_b32_e32 v4, s4
+; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v5
+; CGP-NEXT:    v_cndmask_b32_e32 v2, v4, v2, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v6
 ; CGP-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
-; CGP-NEXT:    v_cndmask_b32_e32 v2, v6, v3, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v7, v5, vcc
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
-; CGP-NEXT:    v_cndmask_b32_e32 v2, v11, v2, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v13, v3, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v2, v6, v4, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v4, v7, v5, vcc
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
+; CGP-NEXT:    v_cndmask_b32_e32 v2, v9, v2, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v3, v10, v4, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v8
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v8
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v8
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v8, vcc
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv <2 x i64> %num, <i64 4096, i64 4096>
   ret <2 x i64> %result
 }

 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v7, v4
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v6, vcc, v6, v3, vcc
 ; CHECK-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s6, v5, 0
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v7, 31, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v0, v7
 ; CHECK-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], s6, v6, v[3:4]
-; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v1, v7, vcc
-; CHECK-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], -1, v5, v[3:4]
-; CHECK-NEXT:    v_xor_b32_e32 v4, v0, v7
-; CHECK-NEXT:    v_mul_lo_u32 v0, v6, v2
-; CHECK-NEXT:    v_mul_lo_u32 v8, v5, v3
-; CHECK-NEXT:    v_xor_b32_e32 v9, v1, v7
-; CHECK-NEXT:    v_mul_hi_u32 v1, v5, v2
+; CHECK-NEXT:    v_addc_u32_e32 v9, vcc, v1, v7, vcc
+; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], -1, v5, v[3:4]
+; CHECK-NEXT:    v_xor_b32_e32 v3, v8, v7
+; CHECK-NEXT:    v_mul_lo_u32 v1, v6, v2
+; CHECK-NEXT:    v_mul_lo_u32 v8, v5, v0
+; CHECK-NEXT:    v_xor_b32_e32 v4, v9, v7
+; CHECK-NEXT:    v_mul_hi_u32 v9, v5, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v2, v6, v2
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
-; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v1, v6, v3
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
-; CHECK-NEXT:    v_mul_hi_u32 v8, v5, v3
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v8
 ; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
-; CHECK-NEXT:    v_mul_hi_u32 v3, v6, v3
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v9
 ; CHECK-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
+; CHECK-NEXT:    v_mul_lo_u32 v9, v6, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v8, v1
+; CHECK-NEXT:    v_mul_hi_u32 v8, v5, v0
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v9, v2
+; CHECK-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; CHECK-NEXT:    v_mul_hi_u32 v0, v6, v0
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v6, v1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v2, v9, v0
-; CHECK-NEXT:    v_mul_lo_u32 v3, v4, v1
-; CHECK-NEXT:    v_mul_hi_u32 v5, v4, v0
-; CHECK-NEXT:    v_mul_hi_u32 v0, v9, v0
+; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v8, v2
+; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
+; CHECK-NEXT:    v_addc_u32_e32 v0, vcc, v6, v0, vcc
+; CHECK-NEXT:    v_mul_lo_u32 v2, v4, v1
+; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v0
+; CHECK-NEXT:    v_mul_hi_u32 v8, v3, v1
+; CHECK-NEXT:    v_mul_hi_u32 v1, v4, v1
 ; CHECK-NEXT:    s_mov_b32 s6, 0x12d8fb
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
+; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v5, v9, v1
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; CHECK-NEXT:    v_mul_hi_u32 v3, v4, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
+; CHECK-NEXT:    v_mul_lo_u32 v8, v4, v0
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CHECK-NEXT:    v_mul_hi_u32 v5, v3, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v8, v1
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v0, v2
-; CHECK-NEXT:    v_mul_hi_u32 v6, v9, v1
-; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v5, 0
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v1, v2
+; CHECK-NEXT:    v_mul_hi_u32 v9, v4, v0
+; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v8, 0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v6, v2
-; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], s6, v3, v[1:2]
-; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v4, v0
-; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], 0, v5, v[1:2]
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v9, v2
+; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], s6, v5, v[1:2]
+; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v3, v0
+; CHECK-NEXT:    v_subb_u32_e64 v2, s[4:5], v4, v1, vcc
+; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v4, v1
 ; CHECK-NEXT:    v_mov_b32_e32 v6, 0x12d8fb
-; CHECK-NEXT:    s_bfe_i32 s6, 1, 0x10000
-; CHECK-NEXT:    v_subb_u32_e64 v2, s[4:5], v9, v1, vcc
-; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v9, v1
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
+; CHECK-NEXT:    s_bfe_i32 s6, 1, 0x10000
 ; CHECK-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v6
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
-; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
-; CHECK-NEXT:    v_mov_b32_e32 v8, s6
+; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, -1, s[4:5]
+; CHECK-NEXT:    v_mov_b32_e32 v4, s6
 ; CHECK-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v2
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, v8, v4, s[4:5]
-; CHECK-NEXT:    v_add_i32_e32 v4, vcc, 1, v5
-; CHECK-NEXT:    v_addc_u32_e32 v8, vcc, 0, v3, vcc
+; CHECK-NEXT:    v_cndmask_b32_e64 v2, v4, v3, s[4:5]
+; CHECK-NEXT:    v_add_i32_e32 v3, vcc, 1, v8
+; CHECK-NEXT:    v_addc_u32_e32 v4, vcc, 0, v5, vcc
 ; CHECK-NEXT:    s_bfe_i32 s4, 1, 0x10000
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v6
 ; CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, -1, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v6, s4
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v6, v0, vcc
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, 1, v4
-; CHECK-NEXT:    v_addc_u32_e32 v6, vcc, 0, v8, vcc
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, 1, v3
+; CHECK-NEXT:    v_addc_u32_e32 v6, vcc, 0, v4, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
-; CHECK-NEXT:    v_cndmask_b32_e32 v0, v4, v1, vcc
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v8, v6, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v0, v3, v1, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v4, v6, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
-; CHECK-NEXT:    v_cndmask_b32_e32 v0, v5, v0, vcc
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v0, v8, v0, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v5, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v0, v7
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v1, v7
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v7
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v7, vcc
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv i64 %num, 1235195
   ret i64 %result
 }

 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v7, v4
-; CGP-NEXT:    v_addc_u32_e32 v8, vcc, v8, v5, vcc
-; CGP-NEXT:    v_mad_u64_u32 v[5:6], s[4:5], s6, v9, 0
-; CGP-NEXT:    v_mov_b32_e32 v4, v6
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v8, v[4:5]
-; CGP-NEXT:    v_ashrrev_i32_e32 v4, 31, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], -1, v9, v[6:7]
-; CGP-NEXT:    v_addc_u32_e32 v1, vcc, v1, v4, vcc
-; CGP-NEXT:    v_xor_b32_e32 v10, v0, v4
-; CGP-NEXT:    v_mul_lo_u32 v0, v8, v5
-; CGP-NEXT:    v_mul_lo_u32 v7, v9, v6
-; CGP-NEXT:    v_xor_b32_e32 v11, v1, v4
-; CGP-NEXT:    v_mul_hi_u32 v1, v9, v5
-; CGP-NEXT:    v_mul_hi_u32 v5, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v1, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v9, 0
+; CGP-NEXT:    v_addc_u32_e32 v10, vcc, v8, v5, vcc
+; CGP-NEXT:    v_ashrrev_i32_e32 v5, 31, v1
+; CGP-NEXT:    v_mov_b32_e32 v4, v7
+; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s6, v10, v[4:5]
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v0, v5
+; CGP-NEXT:    v_addc_u32_e32 v11, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], -1, v9, v[7:8]
+; CGP-NEXT:    v_xor_b32_e32 v8, v4, v5
+; CGP-NEXT:    v_mul_lo_u32 v1, v10, v6
+; CGP-NEXT:    v_mul_lo_u32 v4, v9, v0
 ; CGP-NEXT:    v_mul_hi_u32 v7, v9, v6
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_mul_hi_u32 v6, v10, v6
+; CGP-NEXT:    v_xor_b32_e32 v11, v11, v5
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; CGP-NEXT:    v_mul_hi_u32 v6, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v6, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v9, v0
-; CGP-NEXT:    v_addc_u32_e32 v1, vcc, v8, v1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v5, v11, v0
-; CGP-NEXT:    v_mul_lo_u32 v6, v10, v1
-; CGP-NEXT:    v_mul_hi_u32 v7, v10, v0
-; CGP-NEXT:    v_mul_hi_u32 v0, v11, v0
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
+; CGP-NEXT:    v_mul_lo_u32 v7, v10, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; CGP-NEXT:    v_mul_hi_u32 v4, v9, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v7, v11, v1
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; CGP-NEXT:    v_mul_hi_u32 v6, v10, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_mul_hi_u32 v0, v10, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v9, v1
+; CGP-NEXT:    v_addc_u32_e32 v0, vcc, v10, v0, vcc
+; CGP-NEXT:    v_mul_lo_u32 v6, v11, v1
+; CGP-NEXT:    v_mul_lo_u32 v7, v8, v0
+; CGP-NEXT:    v_mul_hi_u32 v9, v8, v1
+; CGP-NEXT:    v_mul_hi_u32 v1, v11, v1
+; CGP-NEXT:    v_mul_hi_u32 v10, v11, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v7
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v6
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v9, v11, v0
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v0, v5
-; CGP-NEXT:    v_mul_hi_u32 v7, v11, v1
-; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s7, v8, 0
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v7, v5
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v9, v[1:2]
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v10, v0
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], 0, v8, v[6:7]
-; CGP-NEXT:    v_mov_b32_e32 v5, 0x12d8fb
-; CGP-NEXT:    v_mov_b32_e32 v10, s8
+; CGP-NEXT:    v_mul_hi_u32 v7, v8, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v9, v1
+; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v7
+; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v1, v6
+; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s7, v9, 0
+; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_add_i32_e32 v10, vcc, v10, v6
+; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v10, v[1:2]
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v8, v0
+; CGP-NEXT:    v_mov_b32_e32 v4, 0x12d8fb
 ; CGP-NEXT:    v_subb_u32_e64 v1, s[4:5], v11, v6, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v6, s[4:5], v11, v6
-; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v5
+; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v1
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v6, vcc
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v6, 0x12d8fb
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v10, v10, v7, s[4:5]
+; CGP-NEXT:    v_mov_b32_e32 v8, s8
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v8, v8, v7, s[4:5]
 ; CGP-NEXT:    v_subbrev_u32_e32 v7, vcc, 0, v1, vcc
 ; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v1, 0
 ; CGP-NEXT:    v_mac_f32_e32 v6, 0x4f800000, v1
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v1, v6
-; CGP-NEXT:    v_add_i32_e32 v11, vcc, 1, v8
-; CGP-NEXT:    v_addc_u32_e32 v12, vcc, 0, v9, vcc
+; CGP-NEXT:    v_add_i32_e32 v11, vcc, 1, v9
+; CGP-NEXT:    v_addc_u32_e32 v12, vcc, 0, v10, vcc
 ; CGP-NEXT:    v_mul_f32_e32 v1, 0x5f7ffffc, v1
 ; CGP-NEXT:    v_mul_f32_e32 v6, 0x2f800000, v1
 ; CGP-NEXT:    v_trunc_f32_e32 v6, v6
 ; CGP-NEXT:    v_mac_f32_e32 v1, 0xcf800000, v6
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v13, v1
 ; CGP-NEXT:    s_bfe_i32 s4, 1, 0x10000
-; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v5
+; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v4
 ; CGP-NEXT:    v_mov_b32_e32 v15, s4
 ; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v13, 0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v16, v6
 ; CGP-NEXT:    v_cndmask_b32_e64 v14, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v7
 ; CGP-NEXT:    v_cndmask_b32_e32 v14, v15, v14, vcc
 ; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v16, v[1:2]
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, 1, v11

 ; CGP-NEXT:    v_mul_hi_u32 v6, v16, v6
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v11, v1
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v6, v1
 ; CGP-NEXT:    v_add_i32_e32 v11, vcc, v13, v0
 ; CGP-NEXT:    v_addc_u32_e32 v13, vcc, v16, v1, vcc
 ; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v11, 0
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
-; CGP-NEXT:    v_cndmask_b32_e32 v8, v8, v7, vcc
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
+; CGP-NEXT:    v_cndmask_b32_e32 v6, v9, v7, vcc
+; CGP-NEXT:    v_xor_b32_e32 v9, v6, v5
 ; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v13, v[1:2]
-; CGP-NEXT:    v_xor_b32_e32 v1, v8, v4
-; CGP-NEXT:    v_ashrrev_i32_e32 v8, 31, v3
+; CGP-NEXT:    v_cndmask_b32_e32 v8, v10, v12, vcc
+; CGP-NEXT:    v_xor_b32_e32 v1, v8, v5
 ; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], -1, v11, v[6:7]
-; CGP-NEXT:    v_cndmask_b32_e32 v9, v9, v12, vcc
+; CGP-NEXT:    v_ashrrev_i32_e32 v8, 31, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
 ; CGP-NEXT:    v_addc_u32_e32 v3, vcc, v3, v8, vcc
-; CGP-NEXT:    v_xor_b32_e32 v10, v2, v8
+; CGP-NEXT:    v_xor_b32_e32 v7, v2, v8
 ; CGP-NEXT:    v_mul_lo_u32 v2, v13, v0
-; CGP-NEXT:    v_mul_lo_u32 v7, v11, v6
+; CGP-NEXT:    v_mul_lo_u32 v10, v11, v6
 ; CGP-NEXT:    v_xor_b32_e32 v12, v3, v8
 ; CGP-NEXT:    v_mul_hi_u32 v3, v11, v0
 ; CGP-NEXT:    v_mul_hi_u32 v0, v13, v0
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v3, v13, v6
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v7, v2
-; CGP-NEXT:    v_mul_hi_u32 v7, v11, v6
+; CGP-NEXT:    v_add_i32_e32 v2, vcc, v10, v2
+; CGP-NEXT:    v_mul_hi_u32 v10, v11, v6
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
+; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v10
 ; CGP-NEXT:    v_mul_hi_u32 v6, v13, v6
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v6, v2
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v11, v0
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v11, v0
 ; CGP-NEXT:    v_addc_u32_e32 v2, vcc, v13, v2, vcc
-; CGP-NEXT:    v_mul_lo_u32 v3, v12, v0
-; CGP-NEXT:    v_mul_lo_u32 v6, v10, v2
-; CGP-NEXT:    v_mul_hi_u32 v7, v10, v0
-; CGP-NEXT:    v_mul_hi_u32 v0, v12, v0
-; CGP-NEXT:    v_xor_b32_e32 v9, v9, v4
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
-; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v7, v12, v2
+; CGP-NEXT:    v_mul_lo_u32 v6, v12, v3
+; CGP-NEXT:    v_mul_lo_u32 v10, v7, v2
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v9, v5
+; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mul_hi_u32 v5, v7, v3
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v6, v12, v2
+; CGP-NEXT:    v_mul_hi_u32 v3, v12, v3
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v9, v5
+; CGP-NEXT:    v_mul_hi_u32 v9, v7, v2
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v6, v3
-; CGP-NEXT:    v_mul_hi_u32 v6, v10, v2
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v6
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
-; CGP-NEXT:    v_add_i32_e32 v11, vcc, v0, v3
-; CGP-NEXT:    v_mul_hi_u32 v7, v12, v2
-; CGP-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s7, v11, 0
-; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v6, v0
-; CGP-NEXT:    v_add_i32_e32 v13, vcc, v7, v0
-; CGP-NEXT:    v_mov_b32_e32 v0, v3
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v13, v[0:1]
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v1, v4
-; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v9, v4, vcc
-; CGP-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], 0, v11, v[6:7]
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v10, v2
-; CGP-NEXT:    v_subb_u32_e64 v4, s[4:5], v12, v3, vcc
-; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v12, v3
-; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v9
+; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v9
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v3, v5
+; CGP-NEXT:    v_mul_hi_u32 v10, v12, v2
+; CGP-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s7, v9, 0
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
+; CGP-NEXT:    v_add_i32_e32 v10, vcc, v10, v5
+; CGP-NEXT:    v_mad_u64_u32 v[5:6], s[4:5], s7, v10, v[3:4]
+; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v7, v2
+; CGP-NEXT:    v_subb_u32_e64 v3, s[4:5], v12, v5, vcc
+; CGP-NEXT:    v_sub_i32_e64 v5, s[4:5], v12, v5
+; CGP-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v5, vcc
 ; CGP-NEXT:    s_bfe_i32 s6, 1, 0x10000
-; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v5
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v5
+; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v4
+; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[4:5]
 ; CGP-NEXT:    v_mov_b32_e32 v7, s6
-; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v4
-; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_cndmask_b32_e64 v4, v7, v6, s[4:5]
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, 1, v11
-; CGP-NEXT:    v_addc_u32_e32 v7, vcc, 0, v13, vcc
+; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v3
+; CGP-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v5, vcc
+; CGP-NEXT:    v_cndmask_b32_e64 v3, v7, v6, s[4:5]
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, 1, v9
+; CGP-NEXT:    v_addc_u32_e32 v7, vcc, 0, v10, vcc
 ; CGP-NEXT:    s_bfe_i32 s4, 1, 0x10000
-; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v5
+; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, -1, vcc
-; CGP-NEXT:    v_mov_b32_e32 v5, s4
-; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
-; CGP-NEXT:    v_cndmask_b32_e32 v2, v5, v2, vcc
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, 1, v6
+; CGP-NEXT:    v_mov_b32_e32 v4, s4
+; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v5
+; CGP-NEXT:    v_cndmask_b32_e32 v2, v4, v2, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v6
 ; CGP-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
-; CGP-NEXT:    v_cndmask_b32_e32 v2, v6, v3, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v7, v5, vcc
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
-; CGP-NEXT:    v_cndmask_b32_e32 v2, v11, v2, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v13, v3, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v2, v6, v4, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v4, v7, v5, vcc
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
+; CGP-NEXT:    v_cndmask_b32_e32 v2, v9, v2, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v3, v10, v4, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v8
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v8
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v8
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v8, vcc
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv <2 x i64> %num, <i64 1235195, i64 1235195>
   ret <2 x i64> %result
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i64.ll

 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v7, v4
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v6, vcc, v6, v3, vcc
 ; CHECK-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s6, v5, 0
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v7, 31, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v0, v7
 ; CHECK-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], s6, v6, v[3:4]
-; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v1, v7, vcc
-; CHECK-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], -1, v5, v[3:4]
-; CHECK-NEXT:    v_xor_b32_e32 v4, v0, v7
-; CHECK-NEXT:    v_mul_lo_u32 v0, v6, v2
-; CHECK-NEXT:    v_mul_lo_u32 v8, v5, v3
-; CHECK-NEXT:    v_xor_b32_e32 v9, v1, v7
-; CHECK-NEXT:    v_mul_hi_u32 v1, v5, v2
+; CHECK-NEXT:    v_addc_u32_e32 v9, vcc, v1, v7, vcc
+; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], -1, v5, v[3:4]
+; CHECK-NEXT:    v_xor_b32_e32 v3, v8, v7
+; CHECK-NEXT:    v_mul_lo_u32 v1, v6, v2
+; CHECK-NEXT:    v_mul_lo_u32 v8, v5, v0
+; CHECK-NEXT:    v_xor_b32_e32 v4, v9, v7
+; CHECK-NEXT:    v_mul_hi_u32 v9, v5, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v2, v6, v2
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
-; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v1, v6, v3
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
-; CHECK-NEXT:    v_mul_hi_u32 v8, v5, v3
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v8
 ; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
-; CHECK-NEXT:    v_mul_hi_u32 v3, v6, v3
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v9
 ; CHECK-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
+; CHECK-NEXT:    v_mul_lo_u32 v9, v6, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v8, v1
+; CHECK-NEXT:    v_mul_hi_u32 v8, v5, v0
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v9, v2
+; CHECK-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; CHECK-NEXT:    v_mul_hi_u32 v0, v6, v0
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v6, v1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v2, v9, v0
-; CHECK-NEXT:    v_mul_lo_u32 v3, v4, v1
-; CHECK-NEXT:    v_mul_hi_u32 v5, v4, v0
-; CHECK-NEXT:    v_mul_hi_u32 v0, v9, v0
+; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v8, v2
+; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
+; CHECK-NEXT:    v_addc_u32_e32 v0, vcc, v6, v0, vcc
+; CHECK-NEXT:    v_mul_lo_u32 v2, v4, v1
+; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v0
+; CHECK-NEXT:    v_mul_hi_u32 v8, v3, v1
+; CHECK-NEXT:    v_mul_hi_u32 v1, v4, v1
 ; CHECK-NEXT:    s_movk_i32 s6, 0x1000
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
+; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v5, v9, v1
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; CHECK-NEXT:    v_mul_hi_u32 v3, v4, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
+; CHECK-NEXT:    v_mul_lo_u32 v8, v4, v0
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CHECK-NEXT:    v_mul_hi_u32 v5, v3, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v8, v1
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v0, v2
-; CHECK-NEXT:    v_mul_hi_u32 v6, v9, v1
-; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v5, 0
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
+; CHECK-NEXT:    v_mul_hi_u32 v8, v4, v0
+; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v1, 0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v6, v2
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v8, v2
 ; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], s6, v2, v[1:2]
-; CHECK-NEXT:    v_mov_b32_e32 v3, 0x1000
+; CHECK-NEXT:    v_sub_i32_e64 v0, s[4:5], v3, v0
+; CHECK-NEXT:    v_mov_b32_e32 v6, 0x1000
+; CHECK-NEXT:    v_subb_u32_e64 v2, vcc, v4, v1, s[4:5]
+; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v4, v1
 ; CHECK-NEXT:    s_bfe_i32 s6, 1, 0x10000
-; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], 0, v5, v[1:2]
-; CHECK-NEXT:    v_sub_i32_e64 v0, s[4:5], v4, v0
-; CHECK-NEXT:    v_subb_u32_e64 v2, vcc, v9, v1, s[4:5]
-; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v9, v1
-; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, vcc
-; CHECK-NEXT:    v_mov_b32_e32 v5, s6
+; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v6
+; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, -1, vcc
+; CHECK-NEXT:    v_mov_b32_e32 v4, s6
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
-; CHECK-NEXT:    v_cndmask_b32_e32 v4, v5, v4, vcc
-; CHECK-NEXT:    v_sub_i32_e32 v6, vcc, v0, v3
+; CHECK-NEXT:    v_cndmask_b32_e32 v3, v4, v3, vcc
+; CHECK-NEXT:    v_sub_i32_e32 v5, vcc, v0, v6
 ; CHECK-NEXT:    v_subbrev_u32_e64 v1, s[4:5], 0, v1, s[4:5]
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
-; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v6, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, -1, vcc
+; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v5, v6
+; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
-; CHECK-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
-; CHECK-NEXT:    v_subrev_i32_e32 v5, vcc, 0x1000, v6
+; CHECK-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
+; CHECK-NEXT:    v_subrev_i32_e32 v6, vcc, 0x1000, v5
 ; CHECK-NEXT:    v_subbrev_u32_e32 v8, vcc, 0, v1, vcc
-; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
-; CHECK-NEXT:    v_cndmask_b32_e32 v3, v6, v5, vcc
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
-; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v4, v5, v6, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc
+; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
+; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v0, v7
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v1, v7
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v7
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v7, vcc
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = srem i64 %num, 4096
   ret i64 %result

 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v7, v4
-; CGP-NEXT:    v_addc_u32_e32 v8, vcc, v8, v5, vcc
-; CGP-NEXT:    v_mad_u64_u32 v[5:6], s[4:5], s6, v9, 0
-; CGP-NEXT:    v_mov_b32_e32 v4, v6
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v8, v[4:5]
-; CGP-NEXT:    v_ashrrev_i32_e32 v4, 31, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], -1, v9, v[6:7]
-; CGP-NEXT:    v_addc_u32_e32 v1, vcc, v1, v4, vcc
-; CGP-NEXT:    v_xor_b32_e32 v10, v0, v4
-; CGP-NEXT:    v_mul_lo_u32 v0, v8, v5
-; CGP-NEXT:    v_mul_lo_u32 v7, v9, v6
-; CGP-NEXT:    v_xor_b32_e32 v11, v1, v4
-; CGP-NEXT:    v_mul_hi_u32 v1, v9, v5
-; CGP-NEXT:    v_mul_hi_u32 v5, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v1, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v9, 0
+; CGP-NEXT:    v_addc_u32_e32 v10, vcc, v8, v5, vcc
+; CGP-NEXT:    v_ashrrev_i32_e32 v5, 31, v1
+; CGP-NEXT:    v_mov_b32_e32 v4, v7
+; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s6, v10, v[4:5]
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v0, v5
+; CGP-NEXT:    v_addc_u32_e32 v11, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], -1, v9, v[7:8]
+; CGP-NEXT:    v_xor_b32_e32 v8, v4, v5
+; CGP-NEXT:    v_mul_lo_u32 v1, v10, v6
+; CGP-NEXT:    v_mul_lo_u32 v4, v9, v0
 ; CGP-NEXT:    v_mul_hi_u32 v7, v9, v6
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_mul_hi_u32 v6, v10, v6
+; CGP-NEXT:    v_xor_b32_e32 v11, v11, v5
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; CGP-NEXT:    v_mul_hi_u32 v6, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v6, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v9, v0
-; CGP-NEXT:    v_addc_u32_e32 v1, vcc, v8, v1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v5, v11, v0
-; CGP-NEXT:    v_mul_lo_u32 v6, v10, v1
-; CGP-NEXT:    v_mul_hi_u32 v7, v10, v0
-; CGP-NEXT:    v_mul_hi_u32 v0, v11, v0
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
+; CGP-NEXT:    v_mul_lo_u32 v7, v10, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; CGP-NEXT:    v_mul_hi_u32 v4, v9, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v7, v11, v1
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; CGP-NEXT:    v_mul_hi_u32 v6, v10, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_mul_hi_u32 v0, v10, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v9, v1
+; CGP-NEXT:    v_addc_u32_e32 v0, vcc, v10, v0, vcc
+; CGP-NEXT:    v_mul_lo_u32 v6, v11, v1
+; CGP-NEXT:    v_mul_lo_u32 v7, v8, v0
+; CGP-NEXT:    v_mul_hi_u32 v9, v8, v1
+; CGP-NEXT:    v_mul_hi_u32 v1, v11, v1
+; CGP-NEXT:    v_mov_b32_e32 v4, 0x1000
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v7
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v6
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v9, v11, v0
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v0, v5
-; CGP-NEXT:    v_mul_hi_u32 v7, v11, v1
-; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s7, v8, 0
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v5, v[1:2]
-; CGP-NEXT:    v_sub_i32_e32 v9, vcc, v10, v0
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], 0, v8, v[6:7]
-; CGP-NEXT:    v_mov_b32_e32 v5, 0x1000
-; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v7, 0
+; CGP-NEXT:    v_mul_hi_u32 v7, v8, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v9, v1
+; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v7
+; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v6
+; CGP-NEXT:    v_mul_hi_u32 v9, v11, v0
+; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s7, v1, 0
+; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
+; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v6, v[1:2]
+; CGP-NEXT:    v_sub_i32_e32 v9, vcc, v8, v0
 ; CGP-NEXT:    v_subb_u32_e64 v10, s[4:5], v11, v6, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v0, s[4:5], v11, v6
-; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v9, v5
+; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v9, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, -1, s[4:5]
 ; CGP-NEXT:    v_mov_b32_e32 v6, s8
 ; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v10
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, v6, v1, s[4:5]
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v1, 0x1000
+; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v7, 0
 ; CGP-NEXT:    v_subbrev_u32_e32 v0, vcc, 0, v0, vcc
-; CGP-NEXT:    v_sub_i32_e32 v12, vcc, v9, v5
 ; CGP-NEXT:    v_mac_f32_e32 v1, 0x4f800000, v7
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v1, v1
+; CGP-NEXT:    v_sub_i32_e32 v12, vcc, v9, v4
 ; CGP-NEXT:    v_subbrev_u32_e32 v13, vcc, 0, v0, vcc
-; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v12, v5
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v1
 ; CGP-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0
 ; CGP-NEXT:    v_trunc_f32_e32 v7, v1
 ; CGP-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v7
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v14, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v15, v7
+; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v12, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, -1, vcc
-; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v13
 ; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v14, 0
+; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v13
 ; CGP-NEXT:    v_cndmask_b32_e32 v16, v6, v8, vcc
 ; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s6, v15, v[1:2]
-; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v12, v5
+; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v12, v4
 ; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], -1, v14, v[7:8]
 ; CGP-NEXT:    v_subbrev_u32_e32 v17, vcc, 0, v13, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v16
 ; CGP-NEXT:    v_cndmask_b32_e32 v8, v12, v1, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v1, v15, v0
 ; CGP-NEXT:    v_mul_lo_u32 v12, v14, v7
 ; CGP-NEXT:    v_mul_hi_u32 v16, v14, v0
 ; CGP-NEXT:    v_cndmask_b32_e32 v13, v13, v17, vcc

 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v12, v1
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v7, v1
 ; CGP-NEXT:    v_add_i32_e32 v12, vcc, v14, v0
 ; CGP-NEXT:    v_addc_u32_e32 v14, vcc, v15, v1, vcc
 ; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v12, 0
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v11
-; CGP-NEXT:    v_cndmask_b32_e32 v9, v9, v8, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v7, v9, v8, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v9, v10, v13, vcc
+; CGP-NEXT:    v_xor_b32_e32 v10, v7, v5
 ; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s6, v14, v[1:2]
-; CGP-NEXT:    v_xor_b32_e32 v1, v9, v4
+; CGP-NEXT:    v_xor_b32_e32 v1, v9, v5
 ; CGP-NEXT:    v_ashrrev_i32_e32 v9, 31, v3
 ; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], -1, v12, v[7:8]
-; CGP-NEXT:    v_cndmask_b32_e32 v10, v10, v13, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v9
 ; CGP-NEXT:    v_addc_u32_e32 v3, vcc, v3, v9, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v11, v2, v9
 ; CGP-NEXT:    v_mul_lo_u32 v2, v14, v0
 ; CGP-NEXT:    v_mul_lo_u32 v8, v12, v7
 ; CGP-NEXT:    v_xor_b32_e32 v13, v3, v9
 ; CGP-NEXT:    v_mul_hi_u32 v3, v12, v0
 ; CGP-NEXT:    v_mul_hi_u32 v0, v14, v0

 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v8
 ; CGP-NEXT:    v_mul_hi_u32 v7, v14, v7
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v7, v2
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v12, v0
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v12, v0
 ; CGP-NEXT:    v_addc_u32_e32 v2, vcc, v14, v2, vcc
-; CGP-NEXT:    v_mul_lo_u32 v3, v13, v0
-; CGP-NEXT:    v_mul_lo_u32 v7, v11, v2
-; CGP-NEXT:    v_mul_hi_u32 v8, v11, v0
-; CGP-NEXT:    v_mul_hi_u32 v0, v13, v0
-; CGP-NEXT:    v_xor_b32_e32 v10, v10, v4
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
+; CGP-NEXT:    v_mul_lo_u32 v7, v13, v3
+; CGP-NEXT:    v_mul_lo_u32 v8, v11, v2
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v10, v5
+; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mul_hi_u32 v5, v11, v3
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v7, v13, v2
+; CGP-NEXT:    v_mul_hi_u32 v3, v13, v3
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CGP-NEXT:    v_mul_hi_u32 v8, v11, v2
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v7, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v8
-; CGP-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v8, v13, v2
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v7, v3
-; CGP-NEXT:    v_mul_hi_u32 v7, v11, v2
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
-; CGP-NEXT:    v_add_i32_e32 v12, vcc, v0, v3
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; CGP-NEXT:    v_mul_hi_u32 v8, v13, v2
-; CGP-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s7, v12, 0
-; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v0
-; CGP-NEXT:    v_mov_b32_e32 v0, v3
-; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s7, v7, v[0:1]
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v1, v4
-; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v10, v4, vcc
-; CGP-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], 0, v12, v[7:8]
+; CGP-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s7, v3, 0
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s7, v5, v[3:4]
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v11, v2
-; CGP-NEXT:    v_subb_u32_e64 v4, s[4:5], v13, v3, vcc
-; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v13, v3
-; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_sub_i32_e32 v8, vcc, v2, v5
-; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v5
-; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v8, v5
+; CGP-NEXT:    v_sub_i32_e64 v5, s[4:5], v13, v7
+; CGP-NEXT:    v_subb_u32_e64 v3, s[4:5], v13, v7, vcc
+; CGP-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v5, vcc
+; CGP-NEXT:    v_sub_i32_e32 v8, vcc, v2, v4
+; CGP-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v5, vcc
+; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v4
+; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v8, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
-; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v4
+; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
-; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v5
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, v6, v7, s[4:5]
 ; CGP-NEXT:    v_cndmask_b32_e32 v6, v6, v10, vcc
-; CGP-NEXT:    v_sub_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_subbrev_u32_e32 v10, vcc, 0, v3, vcc
+; CGP-NEXT:    v_sub_i32_e32 v4, vcc, v8, v4
+; CGP-NEXT:    v_subbrev_u32_e32 v10, vcc, 0, v5, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
-; CGP-NEXT:    v_cndmask_b32_e32 v5, v8, v5, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v10, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v4, v8, v4, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v5, v5, v10, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
-; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v5, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v4, v3, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v9
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v9
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v9
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v9, vcc
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = srem <2 x i64> %num, <i64 4096, i64 4096>
   ret <2 x i64> %result
 }

 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v7, v4
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v6, vcc, v6, v3, vcc
 ; CHECK-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s6, v5, 0
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v7, 31, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v0, v7
 ; CHECK-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], s6, v6, v[3:4]
-; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v1, v7, vcc
-; CHECK-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], -1, v5, v[3:4]
-; CHECK-NEXT:    v_xor_b32_e32 v4, v0, v7
-; CHECK-NEXT:    v_mul_lo_u32 v0, v6, v2
-; CHECK-NEXT:    v_mul_lo_u32 v8, v5, v3
-; CHECK-NEXT:    v_xor_b32_e32 v9, v1, v7
-; CHECK-NEXT:    v_mul_hi_u32 v1, v5, v2
+; CHECK-NEXT:    v_addc_u32_e32 v9, vcc, v1, v7, vcc
+; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], -1, v5, v[3:4]
+; CHECK-NEXT:    v_xor_b32_e32 v3, v8, v7
+; CHECK-NEXT:    v_mul_lo_u32 v1, v6, v2
+; CHECK-NEXT:    v_mul_lo_u32 v8, v5, v0
+; CHECK-NEXT:    v_xor_b32_e32 v4, v9, v7
+; CHECK-NEXT:    v_mul_hi_u32 v9, v5, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v2, v6, v2
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
-; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v1, v6, v3
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
-; CHECK-NEXT:    v_mul_hi_u32 v8, v5, v3
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v8
 ; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
-; CHECK-NEXT:    v_mul_hi_u32 v3, v6, v3
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v9
 ; CHECK-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
+; CHECK-NEXT:    v_mul_lo_u32 v9, v6, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v8, v1
+; CHECK-NEXT:    v_mul_hi_u32 v8, v5, v0
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v9, v2
+; CHECK-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; CHECK-NEXT:    v_mul_hi_u32 v0, v6, v0
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v6, v1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v2, v9, v0
-; CHECK-NEXT:    v_mul_lo_u32 v3, v4, v1
-; CHECK-NEXT:    v_mul_hi_u32 v5, v4, v0
-; CHECK-NEXT:    v_mul_hi_u32 v0, v9, v0
+; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v8, v2
+; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
+; CHECK-NEXT:    v_addc_u32_e32 v0, vcc, v6, v0, vcc
+; CHECK-NEXT:    v_mul_lo_u32 v2, v4, v1
+; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v0
+; CHECK-NEXT:    v_mul_hi_u32 v8, v3, v1
+; CHECK-NEXT:    v_mul_hi_u32 v1, v4, v1
 ; CHECK-NEXT:    s_mov_b32 s6, 0x12d8fb
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
+; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v8
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v5, v9, v1
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; CHECK-NEXT:    v_mul_hi_u32 v3, v4, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
+; CHECK-NEXT:    v_mul_lo_u32 v8, v4, v0
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CHECK-NEXT:    v_mul_hi_u32 v5, v3, v0
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v8, v1
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v0, v2
-; CHECK-NEXT:    v_mul_hi_u32 v6, v9, v1
-; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v5, 0
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
+; CHECK-NEXT:    v_mul_hi_u32 v8, v4, v0
+; CHECK-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v1, 0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v6, v2
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v8, v2
 ; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], s6, v2, v[1:2]
-; CHECK-NEXT:    v_mov_b32_e32 v3, 0x12d8fb
+; CHECK-NEXT:    v_sub_i32_e64 v0, s[4:5], v3, v0
+; CHECK-NEXT:    v_mov_b32_e32 v6, 0x12d8fb
+; CHECK-NEXT:    v_subb_u32_e64 v2, vcc, v4, v1, s[4:5]
+; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v4, v1
 ; CHECK-NEXT:    s_bfe_i32 s6, 1, 0x10000
-; CHECK-NEXT:    v_mad_u64_u32 v[1:2], s[4:5], 0, v5, v[1:2]
-; CHECK-NEXT:    v_sub_i32_e64 v0, s[4:5], v4, v0
-; CHECK-NEXT:    v_subb_u32_e64 v2, vcc, v9, v1, s[4:5]
-; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v9, v1
-; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, vcc
-; CHECK-NEXT:    v_mov_b32_e32 v5, s6
+; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v6
+; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, -1, vcc
+; CHECK-NEXT:    v_mov_b32_e32 v4, s6
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
-; CHECK-NEXT:    v_cndmask_b32_e32 v4, v5, v4, vcc
-; CHECK-NEXT:    v_sub_i32_e32 v6, vcc, v0, v3
+; CHECK-NEXT:    v_cndmask_b32_e32 v3, v4, v3, vcc
+; CHECK-NEXT:    v_sub_i32_e32 v5, vcc, v0, v6
 ; CHECK-NEXT:    v_subbrev_u32_e64 v1, s[4:5], 0, v1, s[4:5]
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
-; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v6, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, -1, vcc
+; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v5, v6
+; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
-; CHECK-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
-; CHECK-NEXT:    v_subrev_i32_e32 v5, vcc, 0x12d8fb, v6
+; CHECK-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
+; CHECK-NEXT:    v_subrev_i32_e32 v6, vcc, 0x12d8fb, v5
 ; CHECK-NEXT:    v_subbrev_u32_e32 v8, vcc, 0, v1, vcc
-; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
-; CHECK-NEXT:    v_cndmask_b32_e32 v3, v6, v5, vcc
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
-; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v4, v5, v6, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc
+; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
+; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v0, v7
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v1, v7
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v7
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v7, vcc
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = srem i64 %num, 1235195
   ret i64 %result

 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v7, v4
-; CGP-NEXT:    v_addc_u32_e32 v8, vcc, v8, v5, vcc
-; CGP-NEXT:    v_mad_u64_u32 v[5:6], s[4:5], s6, v9, 0
-; CGP-NEXT:    v_mov_b32_e32 v4, v6
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v8, v[4:5]
-; CGP-NEXT:    v_ashrrev_i32_e32 v4, 31, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], -1, v9, v[6:7]
-; CGP-NEXT:    v_addc_u32_e32 v1, vcc, v1, v4, vcc
-; CGP-NEXT:    v_xor_b32_e32 v10, v0, v4
-; CGP-NEXT:    v_mul_lo_u32 v0, v8, v5
-; CGP-NEXT:    v_mul_lo_u32 v7, v9, v6
-; CGP-NEXT:    v_xor_b32_e32 v11, v1, v4
-; CGP-NEXT:    v_mul_hi_u32 v1, v9, v5
-; CGP-NEXT:    v_mul_hi_u32 v5, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v1, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s6, v9, 0
+; CGP-NEXT:    v_addc_u32_e32 v10, vcc, v8, v5, vcc
+; CGP-NEXT:    v_ashrrev_i32_e32 v5, 31, v1
+; CGP-NEXT:    v_mov_b32_e32 v4, v7
+; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s6, v10, v[4:5]
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v0, v5
+; CGP-NEXT:    v_addc_u32_e32 v11, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], -1, v9, v[7:8]
+; CGP-NEXT:    v_xor_b32_e32 v8, v4, v5
+; CGP-NEXT:    v_mul_lo_u32 v1, v10, v6
+; CGP-NEXT:    v_mul_lo_u32 v4, v9, v0
 ; CGP-NEXT:    v_mul_hi_u32 v7, v9, v6
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_mul_hi_u32 v6, v10, v6
+; CGP-NEXT:    v_xor_b32_e32 v11, v11, v5
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; CGP-NEXT:    v_mul_hi_u32 v6, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v6, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v9, v0
-; CGP-NEXT:    v_addc_u32_e32 v1, vcc, v8, v1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v5, v11, v0
-; CGP-NEXT:    v_mul_lo_u32 v6, v10, v1
-; CGP-NEXT:    v_mul_hi_u32 v7, v10, v0
-; CGP-NEXT:    v_mul_hi_u32 v0, v11, v0
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
+; CGP-NEXT:    v_mul_lo_u32 v7, v10, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; CGP-NEXT:    v_mul_hi_u32 v4, v9, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v7, v11, v1
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; CGP-NEXT:    v_mul_hi_u32 v6, v10, v1
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_mul_hi_u32 v0, v10, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v9, v1
+; CGP-NEXT:    v_addc_u32_e32 v0, vcc, v10, v0, vcc
+; CGP-NEXT:    v_mul_lo_u32 v6, v11, v1
+; CGP-NEXT:    v_mul_lo_u32 v7, v8, v0
+; CGP-NEXT:    v_mul_hi_u32 v9, v8, v1
+; CGP-NEXT:    v_mul_hi_u32 v1, v11, v1
+; CGP-NEXT:    v_mov_b32_e32 v4, 0x12d8fb
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v7
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v6
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v9, v11, v0
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v0, v5
-; CGP-NEXT:    v_mul_hi_u32 v7, v11, v1
-; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s7, v8, 0
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v5, v[1:2]
-; CGP-NEXT:    v_sub_i32_e32 v9, vcc, v10, v0
-; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], 0, v8, v[6:7]
-; CGP-NEXT:    v_mov_b32_e32 v5, 0x12d8fb
-; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v7, 0
+; CGP-NEXT:    v_mul_hi_u32 v7, v8, v0
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v9, v1
+; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v7
+; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_add_i32_e32 v1, vcc, v1, v6
+; CGP-NEXT:    v_mul_hi_u32 v9, v11, v0
+; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s7, v1, 0
+; CGP-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
+; CGP-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
+; CGP-NEXT:    v_mad_u64_u32 v[6:7], s[4:5], s7, v6, v[1:2]
+; CGP-NEXT:    v_sub_i32_e32 v9, vcc, v8, v0
 ; CGP-NEXT:    v_subb_u32_e64 v10, s[4:5], v11, v6, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v0, s[4:5], v11, v6
-; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v9, v5
+; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v9, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, -1, s[4:5]
 ; CGP-NEXT:    v_mov_b32_e32 v6, s8
 ; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v10
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, v6, v1, s[4:5]
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v1, 0x12d8fb
+; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v7, 0
 ; CGP-NEXT:    v_subbrev_u32_e32 v0, vcc, 0, v0, vcc
-; CGP-NEXT:    v_sub_i32_e32 v12, vcc, v9, v5
 ; CGP-NEXT:    v_mac_f32_e32 v1, 0x4f800000, v7
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v1, v1
+; CGP-NEXT:    v_sub_i32_e32 v12, vcc, v9, v4
 ; CGP-NEXT:    v_subbrev_u32_e32 v13, vcc, 0, v0, vcc
-; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v12, v5
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v1
 ; CGP-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0
 ; CGP-NEXT:    v_trunc_f32_e32 v7, v1
 ; CGP-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v7
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v14, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v15, v7
+; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v12, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, -1, vcc
-; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v13
 ; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v14, 0
+; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v13
 ; CGP-NEXT:    v_cndmask_b32_e32 v16, v6, v8, vcc
 ; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s6, v15, v[1:2]
-; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v12, v5
+; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v12, v4
 ; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], -1, v14, v[7:8]
 ; CGP-NEXT:    v_subbrev_u32_e32 v17, vcc, 0, v13, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v16
 ; CGP-NEXT:    v_cndmask_b32_e32 v8, v12, v1, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v1, v15, v0
 ; CGP-NEXT:    v_mul_lo_u32 v12, v14, v7
 ; CGP-NEXT:    v_mul_hi_u32 v16, v14, v0
 ; CGP-NEXT:    v_cndmask_b32_e32 v13, v13, v17, vcc

 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e64 v1, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v12, v1
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v7, v1
 ; CGP-NEXT:    v_add_i32_e32 v12, vcc, v14, v0
 ; CGP-NEXT:    v_addc_u32_e32 v14, vcc, v15, v1, vcc
 ; CGP-NEXT:    v_mad_u64_u32 v[0:1], s[4:5], s6, v12, 0
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v11
-; CGP-NEXT:    v_cndmask_b32_e32 v9, v9, v8, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v7, v9, v8, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v9, v10, v13, vcc
+; CGP-NEXT:    v_xor_b32_e32 v10, v7, v5
 ; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s6, v14, v[1:2]
-; CGP-NEXT:    v_xor_b32_e32 v1, v9, v4
+; CGP-NEXT:    v_xor_b32_e32 v1, v9, v5
 ; CGP-NEXT:    v_ashrrev_i32_e32 v9, 31, v3
 ; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], -1, v12, v[7:8]
-; CGP-NEXT:    v_cndmask_b32_e32 v10, v10, v13, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v9
 ; CGP-NEXT:    v_addc_u32_e32 v3, vcc, v3, v9, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v11, v2, v9
 ; CGP-NEXT:    v_mul_lo_u32 v2, v14, v0
 ; CGP-NEXT:    v_mul_lo_u32 v8, v12, v7
 ; CGP-NEXT:    v_xor_b32_e32 v13, v3, v9
 ; CGP-NEXT:    v_mul_hi_u32 v3, v12, v0
 ; CGP-NEXT:    v_mul_hi_u32 v0, v14, v0

 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v8
 ; CGP-NEXT:    v_mul_hi_u32 v7, v14, v7
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v7, v2
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v12, v0
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v12, v0
 ; CGP-NEXT:    v_addc_u32_e32 v2, vcc, v14, v2, vcc
-; CGP-NEXT:    v_mul_lo_u32 v3, v13, v0
-; CGP-NEXT:    v_mul_lo_u32 v7, v11, v2
-; CGP-NEXT:    v_mul_hi_u32 v8, v11, v0
-; CGP-NEXT:    v_mul_hi_u32 v0, v13, v0
-; CGP-NEXT:    v_xor_b32_e32 v10, v10, v4
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
+; CGP-NEXT:    v_mul_lo_u32 v7, v13, v3
+; CGP-NEXT:    v_mul_lo_u32 v8, v11, v2
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v10, v5
+; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mul_hi_u32 v5, v11, v3
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v7, v13, v2
+; CGP-NEXT:    v_mul_hi_u32 v3, v13, v3
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CGP-NEXT:    v_mul_hi_u32 v8, v11, v2
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v7, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v8
-; CGP-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v8, v13, v2
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v7, v3
-; CGP-NEXT:    v_mul_hi_u32 v7, v11, v2
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
-; CGP-NEXT:    v_add_i32_e32 v12, vcc, v0, v3
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; CGP-NEXT:    v_mul_hi_u32 v8, v13, v2
-; CGP-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s7, v12, 0
-; CGP-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v0
-; CGP-NEXT:    v_mov_b32_e32 v0, v3
-; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s7, v7, v[0:1]
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v1, v4
-; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v10, v4, vcc
-; CGP-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], 0, v12, v[7:8]
+; CGP-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], s7, v3, 0
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CGP-NEXT:    v_mad_u64_u32 v[7:8], s[4:5], s7, v5, v[3:4]
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v11, v2
-; CGP-NEXT:    v_subb_u32_e64 v4, s[4:5], v13, v3, vcc
-; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v13, v3
-; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_sub_i32_e32 v8, vcc, v2, v5
-; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v5
-; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v8, v5
+; CGP-NEXT:    v_sub_i32_e64 v5, s[4:5], v13, v7
+; CGP-NEXT:    v_subb_u32_e64 v3, s[4:5], v13, v7, vcc
+; CGP-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v5, vcc
+; CGP-NEXT:    v_sub_i32_e32 v8, vcc, v2, v4
+; CGP-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v5, vcc
+; CGP-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v4
+; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v8, v4
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
-; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v4
+; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
-; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v5
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, v6, v7, s[4:5]
 ; CGP-NEXT:    v_cndmask_b32_e32 v6, v6, v10, vcc
-; CGP-NEXT:    v_sub_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_subbrev_u32_e32 v10, vcc, 0, v3, vcc
+; CGP-NEXT:    v_sub_i32_e32 v4, vcc, v8, v4
+; CGP-NEXT:    v_subbrev_u32_e32 v10, vcc, 0, v5, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
-; CGP-NEXT:    v_cndmask_b32_e32 v5, v8, v5, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v10, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v4, v8, v4, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v5, v5, v10, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
-; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v5, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v4, v3, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v9
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v9
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v9
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v9, vcc
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = srem <2 x i64> %num, <i64 1235195, i64 1235195>
   ret <2 x i64> %result
 }
 

llvm/unittests/CodeGen/GlobalISel/LegalizerTest.cpp

 //===- LegalizerTest.cpp --------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/GlobalISel/Legalizer.h"
 #include "GISelMITest.h"
+#include "llvm/CodeGen/GlobalISel/GISelKnownBits.h"
 #include "llvm/CodeGen/GlobalISel/LostDebugLocObserver.h"
 
 #define DEBUG_TYPE "legalizer-test"
 
 using namespace LegalizeActions;
 using namespace LegalizeMutations;
 using namespace LegalityPredicates;
 
StringRef MIRString = R"(
     $h4 = COPY %v:_(<2 x s8>)
   )";
   setUp(MIRString.rtrim(' '));
   if (!TM)
     GTEST_SKIP();
 
   ALegalizerInfo LI(MF->getSubtarget());
   LostDebugLocObserver LocObserver(DEBUG_TYPE);
+  GISelKnownBits KB(*MF);
 
   Legalizer::MFResult Result = Legalizer::legalizeMachineFunction(
-      *MF, LI, {&LocObserver}, LocObserver, B);
+      *MF, LI, {&LocObserver}, LocObserver, B, &KB);
 
   EXPECT_TRUE(isNullMIPtr(Result.FailedOn));
   EXPECT_TRUE(Result.Changed);
 
   StringRef CheckString = R"(
     CHECK:      %vptr:_(p0) = COPY $x4
     CHECK-NEXT: [[LOAD_0:%[0-9]+]]:_(s16) = G_LOAD %vptr:_(p0) :: (load (s8))
     CHECK-NEXT: [[OFFSET_1:%[0-9]+]]:_(s64) = G_CONSTANT i64 1
StringRef MIRString = R"(
     $h4 = COPY %v0_ext:_(s16)
   )";
   setUp(MIRString.rtrim(' '));
   if (!TM)
     GTEST_SKIP();
 
   ALegalizerInfo LI(MF->getSubtarget());
   LostDebugLocObserver LocObserver(DEBUG_TYPE);
+  GISelKnownBits KB(*MF);
 
   // The events here unfold as follows:
   // 1. First, the function is scanned pre-forming the worklist of artifacts:
   //
   //      UNMERGE  (1): pushed into the worklist first, will be processed last.
   //         |
   //       ANYEXT  (2)
   //
TEST_F(AArch64GISelMITest, UnorderedArtifactCombiningTest) {
   //
   //  We're looking into making sure that (4.B) happens here, not (4.A). Note
   //  that in that case the first scan through the artifacts worklist, while not
   //  being done in any guaranteed order, only needs to find the innermost
   //  pair(s) of artifacts that could be immediately combined out. After that
   //  the process follows def-use chains, making them shorter at each step, thus
   //  combining everything that can be combined in O(n) time.
   Legalizer::MFResult Result = Legalizer::legalizeMachineFunction(
-      *MF, LI, {&LocObserver}, LocObserver, B);
+      *MF, LI, {&LocObserver}, LocObserver, B, &KB);
 
   EXPECT_TRUE(isNullMIPtr(Result.FailedOn));
   EXPECT_TRUE(Result.Changed);
 
   StringRef CheckString = R"(
     CHECK:      %vptr:_(p0) = COPY $x4
     CHECK-NEXT: [[LOAD_0:%[0-9]+]]:_(s16) = G_LOAD %vptr:_(p0) :: (load (s8))
     CHECK:      $h4 = COPY [[LOAD_0]]:_(s16)
StringRef MIRString = R"(
     $w5 = COPY %v1_sext:_(s32)
   )";
   setUp(MIRString.rtrim(' '));
   if (!TM)
     GTEST_SKIP();
 
   ALegalizerInfo LI(MF->getSubtarget());
   LostDebugLocObserver LocObserver(DEBUG_TYPE);
+  GISelKnownBits KB(*MF);
 
   Legalizer::MFResult Result = Legalizer::legalizeMachineFunction(
-      *MF, LI, {&LocObserver}, LocObserver, B);
+      *MF, LI, {&LocObserver}, LocObserver, B, &KB);
 
   EXPECT_TRUE(isNullMIPtr(Result.FailedOn));
   EXPECT_TRUE(Result.Changed);
 
   StringRef CheckString = R"(
     CHECK:      %vptr:_(p0) = COPY $x4
     CHECK-NEXT: [[LOAD_0:%[0-9]+]]:_(s16) = G_LOAD %vptr:_(p0) :: (load (s8))
     CHECK-NEXT: [[OFFSET_1:%[0-9]+]]:_(s64) = G_CONSTANT i64 1