[MachineCopyPropagation] Do more backward copy propagations

[yshui]

Summary

Previously we only do backward copy propagations if there is no read or write to both the Def and Src of the copy between the definition site and the copy site.

However, this restriction can be relaxed. Reads of the Src can be allowed as long as those Reads are rewritten to use Def instead during the propagation.

This commit implements that, potentially improve the generated code. In particular, this resolve an inconsistency between optimized build and optimized build with debuginfo, where DBG_VALUE instructions could prevent backward copy propagation from happening (See: https://bugs.llvm.org/show_bug.cgi?id=49446)

[yshui]

Things still need to be done:

1. Review the failed tests and see which ones are actual bad CodeGen, and which ones need to be updated.
2. Add a test case for this propagation case.

I need some help with (2) as I can't come up with a simple test case that generates the kind of machine IR pattern I need.

[yshui]

Fix typos in comments.

[yshui]

Adjust how DBG_VALUE is rewritten.

[yshui]

Remove a TODO which is no longer relevant

[yshui]

Adjust a debug message.

[condy]

Please do move https://reviews.llvm.org/D98401 regression test to this patch set

[foad]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

331

Hi, I tried testing this on AMDGPU but quite a lot of tests crash here because Copy is nullptr.

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

331

Thanks, I will have a look.

[yshui]

Strengthen the invariant maintained by the CopyTracker.

trackUse assumed that when a copy instruction exists in Copies, it can be found via both its Def and its Src, and vice-versa. This wasn't true previously, causing segfaults because findCopyDefViaUnit returned nullptr.

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

331

Looks like I made an assumption that was stronger than what CopyTracker actually guaranteed.

This should be fixed now.

[foad]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

331

Yes that fixes the crashes thanks. There are no affected tests for AMDGPU -- of course I was hoping for some massive codegen improvents!

[yshui]

I reviewed the changes in CodeGen tests to the best of my ability, and they look fine to me. If someone else could review and confirm, that would be appreciated.

Also fix some clang-tidy complaints.

[lebedev.ri]

Please upload all patches with full context (-U9999)

[yshui]

Include full context in the patch

[yshui]

Update Mips tests.

My knowledge of Mips is much more limited. I am especially not sure about the $gp motion. Someone more knowledgeable should definitely double check.

[jrtc27]

I don't know why it only affects microMIPS (presumably it does its lowering subtly differently from normal MIPS) but those changes look sensible to me

[atanasyan]

In D98659#2631768, @jrtc27 wrote:

I don't know why it only affects microMIPS (presumably it does its lowering subtly differently from normal MIPS) but those changes look sensible to me

MIIPS changes look good to me. Using $gp register in instructions like lw $25, %call16(__divti3)($gp) is a common way. I do not know why initially another register were involved.

[yshui]

Since this patch already changed output of many test cases, do we still need a test case specifically for this patch?

[lkail]

I haven't looked into implementation details. There are some coding style issues need to be fixed.

llvm/lib/CodeGen/MachineCopyPropagation.cpp

100

Any reason of moving these two functions?

378

This comment should be ended with a period.

382

Comment should be a complete sentence.

922

Please follow https://llvm.org/docs/CodingStandards.html#don-t-use-braces-on-simple-single-statement-bodies-of-if-else-loop-statements

940

Missing period.

[lkail]

In D98659#2632859, @yshui wrote:

Since this patch already changed output of many test cases, do we still need a test case specifically for this patch?

You'd better add some mir tests to demonstrate the idea, w/o -g.

[condy]

In D98659#2633657, @lkail wrote:

In D98659#2632859, @yshui wrote:

Since this patch already changed output of many test cases, do we still need a test case specifically for this patch?

You'd better add some mir tests to demonstrate the idea, w/o -g.

A piece of tests can be copied from https://reviews.llvm.org/D98401 @yshui

[yshui]

In D98659#2633657, @lkail wrote:

In D98659#2632859, @yshui wrote:

Since this patch already changed output of many test cases, do we still need a test case specifically for this patch?

You'd better add some mir tests to demonstrate the idea, w/o -g.

Thanks for your review. There is a test case in https://reviews.llvm.org/D98401, which is quite big. I wasn't able to reduce it further, is it OK to just include that as test case?

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

100

I realized these two functions don't have to be part of CopyTracker. If this is not fine I can put them back.

[lkail]

Thanks for your review. There is a test case in https://reviews.llvm.org/D98401, which is quite big. I wasn't able to reduce it further, is it OK to just include that as test case?

Have you tried using bugpoint to reduce it further, or writing MIR test manually?

llvm/lib/CodeGen/MachineCopyPropagation.cpp

100

I would prefer leave them as they are to make code diff less noisy. If you have strong desire to move, please post another NFC patch.

[yshui]

In D98659#2634206, @lkail wrote:

Thanks for your review. There is a test case in https://reviews.llvm.org/D98401, which is quite big. I wasn't able to reduce it further, is it OK to just include that as test case?

Have you tried using bugpoint to reduce it further, or writing MIR test manually?

I am not sure how to use bugpoint for this, as this is not a miscompilation, bad codegen, or crash. But I did try using llvm-reduce, which did help, but the resulting IR is still pretty big.

I find it difficult to manually create the kind of MIR pattern I want.

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

100

Ah, sorry, my mistake. These functions were originally in class MachineCopyPropagation, but I needed them in class CopyTracker, so I moved them out.

[yshui]

Style fixes.

[jmorse]

There isn't a lot of difference to how debug-info is updated, as far as I can see, but the summary says a fault is resolved. Is this patch just masking the debug-info issue by doing more propagation? (I might have missed something in the implementation).

For the codegen change -- I would suggest this additional feature deserves a dedicated MIR test. All the tests modified are IR-to-assembly, where unrelated changes to other parts of the compiler might obscure this feature being accidentally disabled or otherwise modified. A MIR test for this specific behaviour in this specific pass will guarantee it isn't broken in the future. Plus, it'll make it easier for reviewers and post-review-readers to understand the patch.

llvm/lib/CodeGen/MachineCopyPropagation.cpp

295–298

Performance nit -- is it safe to return a SmallVector * in the pair? That avoids needless copying and potential allocations.

[yshui]

In D98659#2648161, @jmorse wrote:

There isn't a lot of difference to how debug-info is updated, as far as I can see, but the summary says a fault is resolved. Is this patch just masking the debug-info issue by doing more propagation? (I might have missed something in the implementation).

I am not exactly sure what you mean here. The fault is caused by debug-info preventing propagation from happening, this patch fixes that problem.

For the codegen change -- I would suggest this additional feature deserves a dedicated MIR test. All the tests modified are IR-to-assembly, where unrelated changes to other parts of the compiler might obscure this feature being accidentally disabled or otherwise modified. A MIR test for this specific behaviour in this specific pass will guarantee it isn't broken in the future. Plus, it'll make it easier for reviewers and post-review-readers to understand the patch.

I agree, but I mentioned above I have problem creating an small IR input that would generate the kind of MIR pattern I need. There is a test case attached to the issue but it's quite big.

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

295–298

I don't see a problem with this. Will update.

[yshui]

Thanks @jmorse for the review.

Update findAvailBackwardCopy to return a pointer to the SmallVector instead.

[yshui]

Rebasing

[yshui]

Uh, I just figured out how mir tests work. Guess nobody wanted to point out how dumb I am. 😅

Adapted the MIR test from D98401, thanks @condy

[jmorse]

Uh, I just figured out how mir tests work.

No worries, there's a first time for everything :),

I am not exactly sure what you mean here. The fault is caused by debug-info preventing propagation from happening, this patch fixes that problem.

This is true, but I think the patch fixes it by masking the underlying debug-info problem. I've taken a closer look at MCP, I reckon there's a fault in MachineCopyPropagation::BackwardCopyPropagateBlock where it treats all register operands the same, i.e. a "use" by a DBG_VALUE is treated the same as a "use" by any ordinary instruction, causing the debug instruction to prevent an optimisation. The simplest and easiest fix for that, is to have the function ignore debug-info instructions by adding a "if (MI.isDebugInstr()) continue;" statement in the loop before interpreting its register-reads.

That fits with why the extra optimisation gets rid of the different-code-with-debug problem, optimising through register uses would also optimise through debug uses.

If I'm right about that -- could you add that guard, with a debug specific test (ideally in the llvm/tests/DebugInfo/X86 directory), in a separate patch so that we can consider the debug-info matter separate from the optimisation.

That being said: the extra optimisation in this patch is good, as shown by the changes to ll-to-assembly tests, some completely un-necessary copies get eliminated. It's well worth pursuing -- I think more MIR test coverage is needed though (as suggested in D98659#2633657), for example:

• A full example without debug-info being involved,
• Testing that no optimisation happens if the "renamable" flag is absent,
• Checking that propagation doesn't occur (or occurs correctly modified) when sub-registers are read

Plus the scenarios you list in the comment in the trackUse function. The best way to guarantee that those situations are not incorrectly optimised by mistake in the future is by having a test that detects such a scenario. (It's often the case that the tests are a lot more important than the code in the patch itself).

[yshui]

In D98659#2661440, @jmorse wrote:

The simplest and easiest fix for that, is to have the function ignore debug-info instructions by adding a "if (MI.isDebugInstr()) continue;" statement in the loop before interpreting its register-reads.

This is what @condy did in D98401, but this is incorrect. A debug-info use is still a use, you still need to update them when doing propagation, otherwise the debug-info would be incorrect. And that is what this patch is trying to do.

That being said: the extra optimisation in this patch is good, as shown by the changes to ll-to-assembly tests, some completely un-necessary copies get eliminated. It's well worth pursuing -- I think more MIR test coverage is needed though (as suggested in D98659#2633657), for example:

• A full example without debug-info being involved,
• Testing that no optimisation happens if the "renamable" flag is absent,
• Checking that propagation doesn't occur (or occurs correctly modified) when sub-registers are read

Plus the scenarios you list in the comment in the trackUse function. The best way to guarantee that those situations are not incorrectly optimised by mistake in the future is by having a test that detects such a scenario. (It's often the case that the tests are a lot more important than the code in the patch itself).

Makes sense, I'll get on with those.

Thanks!

[yshui]

Add tests suggested by @jmorse

[jmorse]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

172–174

The key is only MCRegisters, but invalidateRegister and similar functions only work with register units. How are Uses found so that they can be invalidated too?

210–211

Could you explain how this is different from the original code -- after all, the original implementation invalidated copes for the reguints for source and destinations of copies found. I don't quite understand "entire register" -- is it the set of regunits, the aliases, or something else?

318–320

This would be a lot easier to understand if it had a motivating example of what it's doing -- see the top of MachineCopyPropagation.cpp for a few samples of what I mean.

327

clang-format? (And other single-statement-continues)

400–404

Shouldn't this be left to the rest of the machine-copy-propagate implementation -- after all, this method is called "trackUse", and it seems to be doing more than that.

948

I ran a stage2 RelWithDebInfo build of clang to test this (I was curious), and got an assertion failure in this function:

llvm::MachineRegisterInfo::updateDbgUsersToReg(llvm::Register, ArrayRef<llvm::MachineInstr *>) const: Assertion `MI->getOperand(0).isReg()' failed.

llvm/test/CodeGen/X86/machine-copy-prop.mir

139–149

The rbp-to-rbx copy is deleted without this patch too, the only difference is that some of the registers change. I think this is fragile, and doesn't demonstrate the intention of the optimisation. Could you use a test body that

a) deletes a COPY that wasn't already dead, and
b) rewrites a non-COPY operand, another NOOP for example.

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

172–174

Uses is always accessed with registers as key (instead of register units), whether it's to find entries or to invalidate entries.

Unless I misunderstood your question?

210–211

So entries of Copies form pairs. Say we have a Def = COPY Src, then Copies will contain {RegUnits of Def} and {RegUnits of Src} as a pair. trackUse relies on the invariant that when Def = COPY Src is no longer a candidate, all related entries in Copies are removed.

The original code doesn't guarantee this. When a RegUnit of a Src register is passed to the original invalidateRegister, only that RegUnit, as well as the related Def RegUnits will be removed, leaving all other RegUnits of that Src register around.

This change makes sure all the related RegUnits are removed.

400–404

trackUse tracks the uses that are rewrite candidates, so if a use is no longer a candidate for rewrite, trackUse removes it.

I think this makes sense.

948

Interesting. I assume this is on X86? Do you happen to have the offending MIR around?

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

948

I can successfully build a RelWithDebInfo stage 2 clang with this patch applied to ab158d35b5a09b2541071ec8351a6ad57dfd7b6e.

Maybe a recent change broke this patch. I will test again after updating.

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

948

Hmm, still cannot reproduce with this patch applied to d4528cbb0e703ee17ee6fb4abe72b7246ccb38f1

[yshui]

Update:

• Rebase
• Add an example of the new copy propagation in the top comments
• One more updated test cases: CodeGen/X86/vector-interleaved-load-i16-stride-4.ll
• Style fixes

[yshui]

Rebasing

[yshui]

Update:

• Rebasing
• More affected tests:
  • CodeGen/RISCV/rvv/vselect-fp-rv64.ll
  • CodeGen/RISCV/rvv/vselect-fp-rv32.ll

[yshui]

Rebasing, no new affected tests.

[yshui]

@jmorse Ping.

[yshui]

One more affected test: CodeGen/ARM/umulo-128-legalisation-lowering.ll

[lkail]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

54

Is it possible forward copy propagation can also eliminate this COPY if $R1 defined by this COPY is killed at the BB later? If so, I think we should extend forward copy propagation and should be less complex than what current patch does.

283

Missing perid.

[yshui]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

54

One case I can think of where this can't done is if $R0 got clobbered between $R1 = $R0 and the use of $R1 later.

[yshui]

Fix a missing period

[jmorse]

Looks like the verifier error I experienced must have been transient / my fault, I can't reproduce it now,

I think I understand what's going on here now, however it involves re-purposing the "Avail" flag attached to copies in an unexpected way. At the very least, it needs re-naming. As far as I understand it before this patch, avail means:

• False: candidate for def being rewritten in backward copy propagation
• False: is a clobbered register, in forward copy propagation
• True: candidate for forward copy propagation

With this patch the "false" value takes on even more meaning: "uses can be rewritten along the way". Realistically, I think if this is to all be understood, the "Avail" flag needs to become an enum or something to communicate more information to the reader about what it means.

How does this new form of backwards copy propagation interact with the existing backwards copy propagation? The reason being, modifying one your tests a little:

---

name: copyprop3
body: |
  bb.0:
    renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
    NOOP implicit renamable $rbp
    renamable $rax = COPY renamable $rbp
    renamable $rbx = COPY killed renamable $rbp
    NOOP implicit $rax
...

I was expecting $rbp to be rewritten to $rax and the copies dropped -- instead the dead COPY is dropped, and the uses left un-rewritten:

bb.0:
  renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
  NOOP implicit renamable $rbp
  renamable $rax = COPY renamable $rbp
  NOOP implicit $rax

I suspect (90% confidence) that this is because CopyTracker::invalidateRegister is deleting the copy because it's not eligible for the "plain" backwards copy propagation. The design of these two things needs to be unified.

[yshui]

In D98659#2830241, @jmorse wrote:

Looks like the verifier error I experienced must have been transient / my fault, I can't reproduce it now,

I think I understand what's going on here now, however it involves re-purposing the "Avail" flag attached to copies in an unexpected way. At the very least, it needs re-naming. As far as I understand it before this patch, avail means:

• False: candidate for def being rewritten in backward copy propagation
• False: is a clobbered register, in forward copy propagation
• True: candidate for forward copy propagation

With this patch the "false" value takes on even more meaning: "uses can be rewritten along the way". Realistically, I think if this is to all be understood, the "Avail" flag needs to become an enum or something to communicate more information to the reader about what it means.

I think it's just "Avail" has different meanings in forward/backward propagation. In backward propagation, it means "def is a candidate for rewriting", this patch doesn't add a new meaning.

How does this new form of backwards copy propagation interact with the existing backwards copy propagation? The reason being, modifying one your tests a little:

---

name: copyprop3
body: |
  bb.0:
    renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
    NOOP implicit renamable $rbp
    renamable $rax = COPY renamable $rbp
    renamable $rbx = COPY killed renamable $rbp
    NOOP implicit $rax
...

I was expecting $rbp to be rewritten to $rax and the copies dropped -- instead the dead COPY is dropped, and the uses left un-rewritten:

bb.0:
  renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
  NOOP implicit renamable $rbp
  renamable $rax = COPY renamable $rbp
  NOOP implicit $rax

I suspect (90% confidence) that this is because CopyTracker::invalidateRegister is deleting the copy because it's not eligible for the "plain" backwards copy propagation. The design of these two things needs to be unified.

renamable $rax = COPY renamable $rbp isn't a candidate for propagation because $rbp isn't killed in that instruction. The later copy is probably eliminated because it's dead code.

[yshui]

Rebasing, new affected test:

• CodeGen/Thumb2/mve-vqdmulh.ll

llvm/lib/CodeGen/MachineCopyPropagation.cpp

 //
 //    $R0 = OP ...
 //    ... // No read/clobber of $R0 and $R1
 //    $R1 = COPY $R0 // $R0 is killed
 // Replace $R0 with $R1 and remove the COPY
 //    $R1 = OP ...
 //    ...
 //
+// or
+//
+//    $R0 = OP ...
+//    ... // No read/clobber of $R1, no clobber of $R0
+//    $R2 = OP $R0 // $R0 is used
+//    $R1 = COPY $R0 // $R0 is killed
+// Replace $R0 and its uses with $R1 and remove the COPY
+//    $R1 = OP ...
+//    $R2 = OP $R1
+//    ...
+//
+//
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"

 STATISTIC(NumDeletes, "Number of dead copies deleted");
 STATISTIC(NumCopyForwards, "Number of copy uses forwarded");
 STATISTIC(NumCopyBackwardPropagated, "Number of copy defs backward propagated");
 DEBUG_COUNTER(FwdCounter, "machine-cp-fwd",
               "Controls which register COPYs are forwarded");
 
 namespace {
 
+/// Check that \p MI does not have implicit uses that overlap with it's \p Use
+/// operand (the register being replaced), since these can sometimes be
+/// implicitly tied to other operands.  For example, on AMDGPU:
+///
+/// V_MOVRELS_B32_e32 %VGPR2, %M0<imp-use>, %EXEC<imp-use>,
+/// %VGPR2_VGPR3_VGPR4_VGPR5<imp-use>
+///
+/// the %VGPR2 is implicitly tied to the larger reg operand, but we have no
+/// way of knowing we need to update the latter when updating the former.
+bool hasImplicitOverlap(const MachineInstr &MI, const MachineOperand &Use,
+                        const TargetRegisterInfo &TRI) {
+  for (const MachineOperand &MIUse : MI.uses())
+    if (&MIUse != &Use && MIUse.isReg() && MIUse.isImplicit() &&
+        MIUse.isUse() && TRI.regsOverlap(Use.getReg(), MIUse.getReg()))
+      return true;
+
+  return false;
+}
+
+/// Decide whether we should forward the source of \param Copy to its use in
+/// \param UseI based on the physical register class constraints of the opcode
+/// and avoiding introducing more cross-class COPYs.
+bool isForwardableRegClassCopy(const Register &NewReg, const MachineInstr &UseI,
+                               unsigned UseIdx, const TargetInstrInfo &TII,
+                               const TargetRegisterInfo &TRI) {
+
+  // If the new register meets the opcode register constraints, then allow
+  // forwarding.
+  if (const TargetRegisterClass *URC =
+          UseI.getRegClassConstraint(UseIdx, &TII, &TRI))
+    return URC->contains(NewReg);
+
+  if (!UseI.isCopy())
+    return false;
+
+  /// COPYs don't have register class constraints, so if the user instruction
+  /// is a COPY, we just try to avoid introducing additional cross-class
+  /// COPYs.  For example:
+  ///
+  ///   RegClassA = COPY RegClassB  // Copy parameter
+  ///   ...
+  ///   RegClassB = COPY RegClassA  // UseI parameter
+  ///
+  /// which after forwarding becomes
+  ///
+  ///   RegClassA = COPY RegClassB
+  ///   ...
+  ///   RegClassB = COPY RegClassB
+  ///
+  /// so we have reduced the number of cross-class COPYs and potentially
+  /// introduced a nop COPY that can be removed.
+  const TargetRegisterClass *UseDstRC =
+      TRI.getMinimalPhysRegClass(UseI.getOperand(0).getReg());
+
+  const TargetRegisterClass *SuperRC = UseDstRC;
+  for (TargetRegisterClass::sc_iterator SuperRCI = UseDstRC->getSuperClasses();
+       SuperRC; SuperRC = *SuperRCI++)
+    if (SuperRC->contains(NewReg))
+      return true;
+
+  return false;
+}
+
 class CopyTracker {
   struct CopyInfo {
     MachineInstr *MI;
     SmallVector<MCRegister, 4> DefRegs;
     bool Avail;
   };
 
   DenseMap<MCRegister, CopyInfo> Copies;
 
+  /// Mapping registers to where they are used later. Key is actual registers
+  /// (not register units) Only used for Backward Propagation
+  DenseMap<MCRegister, SmallVector<MachineInstr *>> Uses;
+
 public:
   /// Mark all of the given registers and their subregisters as unavailable for
   /// copying.
   void markRegsUnavailable(ArrayRef<MCRegister> Regs,
                            const TargetRegisterInfo &TRI) {
     for (MCRegister Reg : Regs) {
       // Source of copy is no longer available for propagation.
       for (MCRegUnitIterator RUI(Reg, &TRI); RUI.isValid(); ++RUI) {
         auto CI = Copies.find(*RUI);
         if (CI != Copies.end())
           CI->second.Avail = false;
       }
     }
   }
 
   /// Remove register from copy maps.
   void invalidateRegister(MCRegister Reg, const TargetRegisterInfo &TRI) {
     // Since Reg might be a subreg of some registers, only invalidate Reg is not
     // enough. We have to find the COPY defines Reg or registers defined by Reg
     // and invalidate all of them.
     SmallSet<MCRegister, 8> RegsToInvalidate;
-    RegsToInvalidate.insert(Reg);
+
+    // Remove in 2 stages to make sure when we invalidate a register, all traces
+    // of the corresponding copy instruction are removed from Copies
     for (MCRegUnitIterator RUI(Reg, &TRI); RUI.isValid(); ++RUI) {
       auto I = Copies.find(*RUI);
       if (I != Copies.end()) {
-        if (MachineInstr *MI = I->second.MI) {
+        if (MachineInstr *MI = I->second.MI)
           RegsToInvalidate.insert(MI->getOperand(0).getReg().asMCReg());
-          RegsToInvalidate.insert(MI->getOperand(1).getReg().asMCReg());
-        }
         RegsToInvalidate.insert(I->second.DefRegs.begin(),
                                 I->second.DefRegs.end());
       }
     }
-    for (MCRegister InvalidReg : RegsToInvalidate)
+
+    // Make sure when we remove Src registers from Copies, we remove the entire
+    // register used in the copy instruction.
+    for (MCRegister InvalidDefReg : RegsToInvalidate) {
+      for (MCRegUnitIterator RUI(InvalidDefReg, &TRI); RUI.isValid(); ++RUI) {
+        auto I = Copies.find(*RUI);
+        if (I != Copies.end()) {
+          assert(I->second.MI);
+          RegsToInvalidate.insert(
+              I->second.MI->getOperand(1).getReg().asMCReg());
+        }
+      }
+    }
+
+    for (MCRegister InvalidReg : RegsToInvalidate) {
       for (MCRegUnitIterator RUI(InvalidReg, &TRI); RUI.isValid(); ++RUI)
         Copies.erase(*RUI);
+      Uses.erase(InvalidReg);
+    }
   }
 
   /// Clobber a single register, removing it from the tracker's copy maps.
   void clobberRegister(MCRegister Reg, const TargetRegisterInfo &TRI) {
     for (MCRegUnitIterator RUI(Reg, &TRI); RUI.isValid(); ++RUI) {
       auto I = Copies.find(*RUI);
       if (I != Copies.end()) {
         // When we clobber the source of a copy, we need to clobber everything
void trackCopy(MachineInstr *MI, const TargetRegisterInfo &TRI) {
     for (MCRegUnitIterator RUI(Src, &TRI); RUI.isValid(); ++RUI) {
       auto I = Copies.insert({*RUI, {nullptr, {}, false}});
       auto &Copy = I.first->second;
       if (!is_contained(Copy.DefRegs, Def))
         Copy.DefRegs.push_back(Def);
     }
   }
 
-  bool hasAnyCopies() {
-    return !Copies.empty();
-  }
+  bool hasAnyCopies() { return !Copies.empty(); }
 
   MachineInstr *findCopyForUnit(MCRegister RegUnit,
                                 const TargetRegisterInfo &TRI,
                                 bool MustBeAvailable = false) {
     auto CI = Copies.find(RegUnit);
     if (CI == Copies.end())
       return nullptr;
     if (MustBeAvailable && !CI->second.Avail)
       return nullptr;
     return CI->second.MI;
   }
 
+  /// Find the corresponding copy instruction where \p RegUnit appeared as the
+  /// Src.
   MachineInstr *findCopyDefViaUnit(MCRegister RegUnit,
                                    const TargetRegisterInfo &TRI) {
     auto CI = Copies.find(RegUnit);
     if (CI == Copies.end())
       return nullptr;
     if (CI->second.DefRegs.size() != 1)
       return nullptr;
     MCRegUnitIterator RUI(CI->second.DefRegs[0], &TRI);
     return findCopyForUnit(*RUI, TRI, true);
   }
 
-  MachineInstr *findAvailBackwardCopy(MachineInstr &I, MCRegister Reg,
-                                      const TargetRegisterInfo &TRI) {
+  std::pair<MachineInstr *, SmallVector<MachineInstr *> *>
+  findAvailBackwardCopy(MachineInstr &I, MCRegister Reg,
     MCRegUnitIterator RUI(Reg, &TRI);
     MachineInstr *AvailCopy = findCopyDefViaUnit(*RUI, TRI);
     if (!AvailCopy ||
         !TRI.isSubRegisterEq(AvailCopy->getOperand(1).getReg(), Reg))
-      return nullptr;
+      return {nullptr, nullptr};
 
     Register AvailSrc = AvailCopy->getOperand(1).getReg();
     Register AvailDef = AvailCopy->getOperand(0).getReg();
     for (const MachineInstr &MI :
          make_range(AvailCopy->getReverseIterator(), I.getReverseIterator()))
       for (const MachineOperand &MO : MI.operands())
         if (MO.isRegMask())
           // FIXME: Shall we simultaneously invalidate AvailSrc or AvailDef?
           if (MO.clobbersPhysReg(AvailSrc) || MO.clobbersPhysReg(AvailDef))
-            return nullptr;
+            return {nullptr, nullptr};
 
-    return AvailCopy;
+    auto UseI = Uses.find(AvailSrc);
+    return {AvailCopy, UseI == Uses.end() ? nullptr : &UseI->second};
+  }
+
+  /// Track uses of registers that are candidates for backward copy propagation
+  void trackUse(MachineInstr *MI, const TargetRegisterInfo &TRI,
+                const TargetInstrInfo &TII) {
+    SmallSet<MCRegister, 8> RegsToInvalidate;
+    for (unsigned OpIdx = 0, OpEnd = MI->getNumOperands(); OpIdx != OpEnd;
+         ++OpIdx) {
+      MachineOperand &MOUse = MI->getOperand(OpIdx);
+      if (!MOUse.isReg() || !MOUse.isUse() || MOUse.isUndef())
+        continue;
+      if (!MOUse.getReg())
+        continue;
+
+      MCRegister Use = MOUse.getReg().asMCReg();
+
+      // Three cases where we should give up propagating copies:
+      //
+      //   1) If a read register overlaps, but is not equal to, some of the
+      //   candidate src registers, we need to give up on propagating those
+      //   overlapping registers.
+      //
+      //   2) If a copy candidate's Def is read or partially read.
+      //
+      //   3) This instruction has uses, but we don't know how to
+      //   rewrite those uses because of overlaps/ties/unrenamble registers, so
+      //   give up on propagating copies related to these uses
+      bool IsOverlappingUse = false;
+      Register CandidateSrc = {0}, CandidateDef = {0};
+      for (MCRegUnitIterator RUI(Use, &TRI); RUI.isValid(); ++RUI) {
+        auto CopyInfoI = Copies.find(*RUI);
+        if (CopyInfoI == Copies.end())
+          continue;
+        if (!CopyInfoI->second.Avail) {
+          // Use matches or overlaps with an Src
+          // Find the actual Src in the copy instruction
+          MachineInstr *Copy = findCopyDefViaUnit(*RUI, TRI);
+          MCRegister Src = Copy->getOperand(1).getReg().asMCReg();
+          MCRegister Def = Copy->getOperand(0).getReg().asMCReg();
+          if (Src != Use) {
+            // Case (1)
+            IsOverlappingUse = true;
+            RegsToInvalidate.insert(Src);
+            RegsToInvalidate.insert(Def);
+          } else {
+            CandidateSrc = Src;
+            CandidateDef = Def;
+            break;
+          }
+        } else {
+          // Case (2)
+          RegsToInvalidate.insert(
+              CopyInfoI->second.MI->getOperand(0).getReg().asMCReg());
+          RegsToInvalidate.insert(
+              CopyInfoI->second.MI->getOperand(1).getReg().asMCReg());
+        }
+      }
+
+      // Can't have matching and overlapping Srcs at the same time.
+      assert(!CandidateSrc || !IsOverlappingUse);
+
+      if (CandidateSrc) {
+        // Implies: !IsOverlappingUse.
+        if (!MI->isDebugValue() &&
+            (hasImplicitOverlap(*MI, MOUse, TRI) || !MOUse.isRenamable() ||
+             MOUse.isTied() ||
+             !isForwardableRegClassCopy(CandidateSrc, *MI, OpIdx, TII, TRI))) {
+          // Case (3)
+          RegsToInvalidate.insert(CandidateSrc.asMCReg());
+          RegsToInvalidate.insert(CandidateDef.asMCReg());
+          continue;
+        }
+
+        // Add to Uses for future rewrite
+        auto I = Uses.insert({Use, {MI}});
+        if (!I.second) {
+          I.first->second.push_back(MI);
+        }
+      }
+    }
+
+    for (MCRegister InvalidReg : RegsToInvalidate) {
+      for (MCRegUnitIterator RUI(InvalidReg, &TRI); RUI.isValid(); ++RUI)
+        Copies.erase(*RUI);
+      Uses.erase(InvalidReg);
+    }
   }
 
   MachineInstr *findAvailCopy(MachineInstr &DestCopy, MCRegister Reg,
                               const TargetRegisterInfo &TRI) {
     // We check the first RegUnit here, since we'll only be interested in the
     // copy if it copies the entire register anyway.
     MCRegUnitIterator RUI(Reg, &TRI);
     MachineInstr *AvailCopy =
         findCopyForUnit(*RUI, TRI, /*MustBeAvailable=*/true);
     if (!AvailCopy ||
         !TRI.isSubRegisterEq(AvailCopy->getOperand(0).getReg(), Reg))
for (const MachineInstr &MI :
           if (MO.clobbersPhysReg(AvailSrc) || MO.clobbersPhysReg(AvailDef))
             return nullptr;
 
     return AvailCopy;
   }
 
   void clear() {
     Copies.clear();
+    Uses.clear();
   }
 };
 
 class MachineCopyPropagation : public MachineFunctionPass {
   const TargetRegisterInfo *TRI;
   const TargetInstrInfo *TII;
   const MachineRegisterInfo *MRI;
 
private:
   typedef enum { DebugUse = false, RegularUse = true } DebugType;
 
   void ReadRegister(MCRegister Reg, MachineInstr &Reader, DebugType DT);
   void ForwardCopyPropagateBlock(MachineBasicBlock &MBB);
   void BackwardCopyPropagateBlock(MachineBasicBlock &MBB);
   bool eraseIfRedundant(MachineInstr &Copy, MCRegister Src, MCRegister Def);
   void forwardUses(MachineInstr &MI);
   void propagateDefs(MachineInstr &MI);
-  bool isForwardableRegClassCopy(const MachineInstr &Copy,
-                                 const MachineInstr &UseI, unsigned UseIdx);
   bool isBackwardPropagatableRegClassCopy(const MachineInstr &Copy,
                                           const MachineInstr &UseI,
                                           unsigned UseIdx);
-  bool hasImplicitOverlap(const MachineInstr &MI, const MachineOperand &Use);
   bool hasOverlappingMultipleDef(const MachineInstr &MI,
                                  const MachineOperand &MODef, Register Def);
 
   /// Candidates for deletion.
   SmallSetVector<MachineInstr *, 8> MaybeDeadCopies;
 
   /// Multimap tracking debug users in current BB
   DenseMap<MachineInstr *, SmallSet<MachineInstr *, 2>> CopyDbgUsers;
if (const TargetRegisterClass *URC =
           UseI.getRegClassConstraint(UseIdx, TII, TRI))
     return URC->contains(Def);
 
   // We don't process further if UseI is a COPY, since forward copy propagation
   // should handle that.
   return false;
 }
 
-/// Decide whether we should forward the source of \param Copy to its use in
-/// \param UseI based on the physical register class constraints of the opcode
-/// and avoiding introducing more cross-class COPYs.
-bool MachineCopyPropagation::isForwardableRegClassCopy(const MachineInstr &Copy,
-                                                       const MachineInstr &UseI,
-                                                       unsigned UseIdx) {
-
-  Register CopySrcReg = Copy.getOperand(1).getReg();
-
-  // If the new register meets the opcode register constraints, then allow
-  // forwarding.
-  if (const TargetRegisterClass *URC =
-          UseI.getRegClassConstraint(UseIdx, TII, TRI))
-    return URC->contains(CopySrcReg);
-
-  if (!UseI.isCopy())
-    return false;
-
-  /// COPYs don't have register class constraints, so if the user instruction
-  /// is a COPY, we just try to avoid introducing additional cross-class
-  /// COPYs.  For example:
-  ///
-  ///   RegClassA = COPY RegClassB  // Copy parameter
-  ///   ...
-  ///   RegClassB = COPY RegClassA  // UseI parameter
-  ///
-  /// which after forwarding becomes
-  ///
-  ///   RegClassA = COPY RegClassB
-  ///   ...
-  ///   RegClassB = COPY RegClassB
-  ///
-  /// so we have reduced the number of cross-class COPYs and potentially
-  /// introduced a nop COPY that can be removed.
-  const TargetRegisterClass *UseDstRC =
-      TRI->getMinimalPhysRegClass(UseI.getOperand(0).getReg());
-
-  const TargetRegisterClass *SuperRC = UseDstRC;
-  for (TargetRegisterClass::sc_iterator SuperRCI = UseDstRC->getSuperClasses();
-       SuperRC; SuperRC = *SuperRCI++)
-    if (SuperRC->contains(CopySrcReg))
-      return true;
-
-  return false;
-}
-
-/// Check that \p MI does not have implicit uses that overlap with it's \p Use
-/// operand (the register being replaced), since these can sometimes be
-/// implicitly tied to other operands.  For example, on AMDGPU:
-///
-/// V_MOVRELS_B32_e32 %VGPR2, %M0<imp-use>, %EXEC<imp-use>, %VGPR2_VGPR3_VGPR4_VGPR5<imp-use>
-///
-/// the %VGPR2 is implicitly tied to the larger reg operand, but we have no
-/// way of knowing we need to update the latter when updating the former.
-bool MachineCopyPropagation::hasImplicitOverlap(const MachineInstr &MI,
-                                                const MachineOperand &Use) {
-  for (const MachineOperand &MIUse : MI.uses())
-    if (&MIUse != &Use && MIUse.isReg() && MIUse.isImplicit() &&
-        MIUse.isUse() && TRI->regsOverlap(Use.getReg(), MIUse.getReg()))
-      return true;
-
-  return false;
-}
-
 /// For an MI that has multiple definitions, check whether \p MI has
 /// a definition that overlaps with another of its definitions.
 /// For example, on ARM: umull   r9, r9, lr, r0
 /// The umull instruction is unpredictable unless RdHi and RdLo are different.
 bool MachineCopyPropagation::hasOverlappingMultipleDef(
     const MachineInstr &MI, const MachineOperand &MODef, Register Def) {
   for (const MachineOperand &MIDef : MI.defs()) {
     if ((&MIDef != &MODef) && MIDef.isReg() &&
if (MOUse.getReg() != CopyDstReg) {
                  << MI);
       continue;
     }
 
     // Don't forward COPYs of reserved regs unless they are constant.
     if (MRI->isReserved(CopySrcReg) && !MRI->isConstantPhysReg(CopySrcReg))
       continue;
 
-    if (!isForwardableRegClassCopy(*Copy, MI, OpIdx))
+    if (!isForwardableRegClassCopy(CopySrcReg, MI, OpIdx, *TII, *TRI))
       continue;
 
-    if (hasImplicitOverlap(MI, MOUse))
+    if (hasImplicitOverlap(MI, MOUse, *TRI))
       continue;
 
     // Check that the instruction is not a copy that partially overwrites the
     // original copy source that we are about to use. The tracker mechanism
     // cannot cope with that.
     if (MI.isCopy() && MI.modifiesRegister(CopySrcReg, TRI) &&
         !MI.definesRegister(CopySrcReg)) {
       LLVM_DEBUG(dbgs() << "MCP: Copy source overlap with dest in " << MI);
for (unsigned OpIdx = 0, OpEnd = MI.getNumOperands(); OpIdx < OpEnd;
     Changed = true;
   }
 }
 
 void MachineCopyPropagation::ForwardCopyPropagateBlock(MachineBasicBlock &MBB) {
   LLVM_DEBUG(dbgs() << "MCP: ForwardCopyPropagateBlock " << MBB.getName()
                     << "\n");
 
-  for (MachineBasicBlock::iterator I = MBB.begin(), E = MBB.end(); I != E; ) {
+  for (MachineBasicBlock::iterator I = MBB.begin(), E = MBB.end(); I != E;) {
     MachineInstr *MI = &*I;
     ++I;
 
     // Analyze copies (which don't overlap themselves).
     if (MI->isCopy() && !TRI->regsOverlap(MI->getOperand(0).getReg(),
                                           MI->getOperand(1).getReg())) {
       assert(MI->getOperand(0).getReg().isPhysical() &&
              MI->getOperand(1).getReg().isPhysical() &&
for (unsigned OpIdx = 0, OpEnd = MI.getNumOperands(); OpIdx != OpEnd;
 
     if (!MODef.getReg())
       continue;
 
     // We only handle if the register comes from a vreg.
     if (!MODef.isRenamable())
       continue;
 
-    MachineInstr *Copy =
+    auto Copy =
         Tracker.findAvailBackwardCopy(MI, MODef.getReg().asMCReg(), *TRI);
-    if (!Copy)
+    if (!Copy.first)
       continue;
 
-    Register Def = Copy->getOperand(0).getReg();
-    Register Src = Copy->getOperand(1).getReg();
+    Register Def = Copy.first->getOperand(0).getReg();
+    Register Src = Copy.first->getOperand(1).getReg();
 
     if (MODef.getReg() != Src)
       continue;
 
-    if (!isBackwardPropagatableRegClassCopy(*Copy, MI, OpIdx))
+    if (!isBackwardPropagatableRegClassCopy(*Copy.first, MI, OpIdx))
       continue;
 
-    if (hasImplicitOverlap(MI, MODef))
+    if (hasImplicitOverlap(MI, MODef, *TRI))
       continue;
 
     if (hasOverlappingMultipleDef(MI, MODef, Def))
       continue;
 
     LLVM_DEBUG(dbgs() << "MCP: Replacing " << printReg(MODef.getReg(), TRI)
                       << "\n     with " << printReg(Def, TRI) << "\n     in "
-                      << MI << "     from " << *Copy);
+                      << MI << "     from " << *Copy.first);
 
+    bool IsRenamable = Copy.first->getOperand(0).isRenamable();
     MODef.setReg(Def);
-    MODef.setIsRenamable(Copy->getOperand(0).isRenamable());
+    MODef.setIsRenamable(IsRenamable);
 
     LLVM_DEBUG(dbgs() << "MCP: After replacement: " << MI << "\n");
-    MaybeDeadCopies.insert(Copy);
+
+    // Update uses of the original Def.
+    // We don't need to perform checks here, Uses only contains rewrittable
+    // uses.
+    if (Copy.second) {
+      for (MachineInstr *User : *Copy.second) {
+        LLVM_DEBUG(dbgs() << "MCP: Replacing " << printReg(Src, TRI)
+                          << "\n     with " << printReg(Def, TRI)
+                          << "\n     in " << *User << "     from "
+                          << *Copy.first);
+        if (User->isDebugValue()) {
+          MRI->updateDbgUsersToReg(Src, Def, User);
+        } else {
+          for (unsigned UseIdx = 0, UseEnd = User->getNumOperands();
+               UseIdx != UseEnd; ++UseIdx) {
+            MachineOperand &MOUse = User->getOperand(UseIdx);
+            if (!MOUse.isReg() || MOUse.getReg() != Src)
+              continue;
+            MOUse.setReg(Def);
+            MOUse.setIsRenamable(IsRenamable);
+          }
+        }
+        LLVM_DEBUG(dbgs() << "MCP: After replacement: " << *User << "\n");
+      }
+    }
+
+    MaybeDeadCopies.insert(Copy.first);
     Changed = true;
     ++NumCopyBackwardPropagated;
   }
 }
 
 void MachineCopyPropagation::BackwardCopyPropagateBlock(
     MachineBasicBlock &MBB) {
   LLVM_DEBUG(dbgs() << "MCP: BackwardCopyPropagateBlock " << MBB.getName()
if (MI->isCopy() && MI->getNumOperands() == 2 &&
                           MI->getOperand(1).getReg())) {
 
       MCRegister Def = MI->getOperand(0).getReg().asMCReg();
       MCRegister Src = MI->getOperand(1).getReg().asMCReg();
 
       // Unlike forward cp, we don't invoke propagateDefs here,
       // just let forward cp do COPY-to-COPY propagation.
       if (isBackwardPropagatableCopy(*MI, *MRI)) {
+        // Remove copies related to Src. Src can only possibly appear
+        // in copy candidates as define, because otherwise the current copy
+        // won't kill Src.
         Tracker.invalidateRegister(Src, *TRI);
+        // Remove copies related to Def.
         Tracker.invalidateRegister(Def, *TRI);
         Tracker.trackCopy(MI, *TRI);
         continue;
       }
     }
 
     // Invalidate any earlyclobber regs first.
     for (const MachineOperand &MO : MI->operands())
       if (MO.isReg() && MO.isEarlyClobber()) {
         MCRegister Reg = MO.getReg().asMCReg();
         if (!Reg)
           continue;
         Tracker.invalidateRegister(Reg, *TRI);
       }
 
     propagateDefs(*MI);
+
+    // Track uses after propagation, because we need the correct Def register.
+    Tracker.trackUse(MI, *TRI, *TII);
     for (const MachineOperand &MO : MI->operands()) {
       if (!MO.isReg())
         continue;
 
       if (!MO.getReg())
         continue;
 
-      if (MO.isDef())
+      if (MO.isDef()) {
+        // Even if we did apply propagation, the relevant copy instruction
+        // would still get invalidated here. The original instruction would
+        // trigger invalidation because its Def matches the Src of the relevant
+        // copy, the updated instruction would still do because its Def now
+        // matches the Def of the relevant copy.
         Tracker.invalidateRegister(MO.getReg().asMCReg(), *TRI);
-
-      if (MO.readsReg()) {
-        if (MO.isDebug()) {
-          //  Check if the register in the debug instruction is utilized
-          // in a copy instruction, so we can update the debug info if the
-          // register is changed.
-          for (MCRegUnitIterator RUI(MO.getReg().asMCReg(), TRI); RUI.isValid();
-               ++RUI) {
-            if (auto *Copy = Tracker.findCopyDefViaUnit(*RUI, *TRI)) {
-              CopyDbgUsers[Copy].insert(MI);
-            }
-          }
-        } else {
-          Tracker.invalidateRegister(MO.getReg().asMCReg(), *TRI);
-        }
       }
     }
   }
 
   for (auto *Copy : MaybeDeadCopies) {
-
-    Register Src = Copy->getOperand(1).getReg();
-    Register Def = Copy->getOperand(0).getReg();
-    SmallVector<MachineInstr *> MaybeDeadDbgUsers(CopyDbgUsers[Copy].begin(),
-                                                  CopyDbgUsers[Copy].end());
-
-    MRI->updateDbgUsersToReg(Src.asMCReg(), Def.asMCReg(), MaybeDeadDbgUsers);
     Copy->eraseFromParent();
     ++NumDeletes;
   }
 
   MaybeDeadCopies.clear();
   CopyDbgUsers.clear();
   Tracker.clear();
 }

llvm/test/CodeGen/ARM/umulo-128-legalisation-lowering.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=armv6-unknown-linux-gnu | FileCheck %s --check-prefixes=ARMV6
 ; RUN: llc < %s -mtriple=armv7-unknown-linux-gnu | FileCheck %s --check-prefixes=ARMV7
 
 define { i128, i8 } @muloti_test(i128 %l, i128 %r) unnamed_addr #0 {
 ; ARMV6-LABEL: muloti_test:
 ; ARMV6:       @ %bb.0: @ %start
 ; ARMV6-NEXT:    push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; ARMV6-NEXT:    sub sp, sp, #28
-; ARMV6-NEXT:    ldr r7, [sp, #72]
+; ARMV6-NEXT:    ldr lr, [sp, #72]
 ; ARMV6-NEXT:    mov r6, r0
 ; ARMV6-NEXT:    str r0, [sp, #8]            @ 4-byte Spill
 ; ARMV6-NEXT:    ldr r4, [sp, #84]
-; ARMV6-NEXT:    umull r1, r0, r2, r7
-; ARMV6-NEXT:    mov lr, r7
+; ARMV6-NEXT:    umull r1, r0, r2, lr
 ; ARMV6-NEXT:    umull r5, r10, r4, r2
 ; ARMV6-NEXT:    str r1, [r6]
 ; ARMV6-NEXT:    ldr r6, [sp, #80]
 ; ARMV6-NEXT:    umull r1, r7, r3, r6
 ; ARMV6-NEXT:    str r7, [sp, #12]           @ 4-byte Spill
 ; ARMV6-NEXT:    add r1, r5, r1
 ; ARMV6-NEXT:    umull r7, r5, r6, r2
 ; ARMV6-NEXT:    mov r6, lr

 ; ARMV7-NEXT:    adc r11, r11, #0
 ; ARMV7-NEXT:    adds r7, r7, r6
 ; ARMV7-NEXT:    ldr r6, [sp, #92]
 ; ARMV7-NEXT:    adc r0, r2, r0
 ; ARMV7-NEXT:    str r0, [sp, #28]           @ 4-byte Spill
 ; ARMV7-NEXT:    ldr r0, [sp, #92]
 ; ARMV7-NEXT:    cmp r3, #0
 ; ARMV7-NEXT:    movwne r3, #1
-; ARMV7-NEXT:    ldr r2, [sp, #76]
+; ARMV7-NEXT:    ldr r2, [sp, #72]
 ; ARMV7-NEXT:    cmp r0, #0
 ; ARMV7-NEXT:    movwne r0, #1
 ; ARMV7-NEXT:    cmp r1, #0
+; ARMV7-NEXT:    and r0, r0, r3
+; ARMV7-NEXT:    ldr r3, [sp, #76]
 ; ARMV7-NEXT:    movwne r1, #1
 ; ARMV7-NEXT:    cmp r12, #0
-; ARMV7-NEXT:    and r0, r0, r3
 ; ARMV7-NEXT:    movwne r12, #1
 ; ARMV7-NEXT:    cmp r5, #0
 ; ARMV7-NEXT:    orr r0, r0, r1
 ; ARMV7-NEXT:    movwne r5, #1
-; ARMV7-NEXT:    cmp r2, #0
-; ARMV7-NEXT:    mov r1, r2
-; ARMV7-NEXT:    mov r3, r2
+; ARMV7-NEXT:    cmp r3, #0
+; ARMV7-NEXT:    mov r1, r3
 ; ARMV7-NEXT:    movwne r1, #1
 ; ARMV7-NEXT:    cmp r4, #0
-; ARMV7-NEXT:    ldr r2, [sp, #72]
 ; ARMV7-NEXT:    movwne r4, #1
 ; ARMV7-NEXT:    cmp lr, #0
-; ARMV7-NEXT:    and r1, r1, r5
 ; ARMV7-NEXT:    movwne lr, #1
 ; ARMV7-NEXT:    orrs r2, r2, r3
 ; ARMV7-NEXT:    ldr r3, [sp, #88]
 ; ARMV7-NEXT:    movwne r2, #1
-; ARMV7-NEXT:    orr r1, r1, r4
+; ARMV7-NEXT:    and r1, r1, r5
 ; ARMV7-NEXT:    orr r0, r0, r12
 ; ARMV7-NEXT:    orrs r3, r3, r6
-; ARMV7-NEXT:    orr r1, r1, lr
+; ARMV7-NEXT:    orr r1, r1, r4
 ; ARMV7-NEXT:    movwne r3, #1
 ; ARMV7-NEXT:    adds r7, r9, r7
 ; ARMV7-NEXT:    str r7, [r8, #8]
-; ARMV7-NEXT:    and r2, r2, r3
+; ARMV7-NEXT:    orr r1, r1, lr
 ; ARMV7-NEXT:    ldr r7, [sp, #28]           @ 4-byte Reload
+; ARMV7-NEXT:    and r2, r2, r3
 ; ARMV7-NEXT:    orr r0, r0, r11
 ; ARMV7-NEXT:    adcs r7, r10, r7
 ; ARMV7-NEXT:    str r7, [r8, #12]
 ; ARMV7-NEXT:    ldr r7, [sp, #32]           @ 4-byte Reload
 ; ARMV7-NEXT:    orr r1, r1, r7
 ; ARMV7-NEXT:    orr r1, r2, r1
 ; ARMV7-NEXT:    orr r0, r1, r0
 ; ARMV7-NEXT:    mov r1, #0

llvm/test/CodeGen/Mips/llvm-ir/mul.ll

; ALL-LABEL: mul_i128:
 
   ; 64R6-DAG:       dmul    $[[T1:[0-9]+]], $5, $6
   ; 64R6:           dmuhu   $[[T2:[0-9]+]], $5, $7
   ; 64R6:           daddu   $[[T3:[0-9]+]], $[[T2]], $[[T1]]
   ; 64R6-DAG:       dmul    $[[T0:[0-9]+]], $4, $7
   ; 64R6:           daddu   $2, $[[T1]], $[[T0]]
   ; 64R6-DAG:       dmul    $3, $5, $7
 
-  ; MM32:           lw      $25, %call16(__multi3)($16)
+  ; MM32:           lw      $25, %call16(__multi3)($gp)
 
   %r = mul i128 %a, %b
   ret i128 %r
 }

llvm/test/CodeGen/Mips/llvm-ir/sdiv.ll

 ; MMR3-LABEL: sdiv_i64:
 ; MMR3:       # %bb.0: # %entry
 ; MMR3-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR3-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MMR3-NEXT:    addiusp -24
 ; MMR3-NEXT:    .cfi_def_cfa_offset 24
 ; MMR3-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MMR3-NEXT:    .cfi_offset 31, -4
-; MMR3-NEXT:    addu $2, $2, $25
-; MMR3-NEXT:    lw $25, %call16(__divdi3)($2)
-; MMR3-NEXT:    move $gp, $2
+; MMR3-NEXT:    addu $gp, $2, $25
+; MMR3-NEXT:    lw $25, %call16(__divdi3)($gp)
 ; MMR3-NEXT:    jalr $25
 ; MMR3-NEXT:    nop
 ; MMR3-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    addiusp 24
 ; MMR3-NEXT:    jrc $ra
 ;
 ; MMR6-LABEL: sdiv_i64:
 ; MMR6:       # %bb.0: # %entry
 ; MMR6-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR6-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MMR6-NEXT:    addiu $sp, $sp, -24
 ; MMR6-NEXT:    .cfi_def_cfa_offset 24
 ; MMR6-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MMR6-NEXT:    .cfi_offset 31, -4
-; MMR6-NEXT:    addu $2, $2, $25
-; MMR6-NEXT:    lw $25, %call16(__divdi3)($2)
-; MMR6-NEXT:    move $gp, $2
+; MMR6-NEXT:    addu $gp, $2, $25
+; MMR6-NEXT:    lw $25, %call16(__divdi3)($gp)
 ; MMR6-NEXT:    jalr $25
 ; MMR6-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MMR6-NEXT:    addiu $sp, $sp, 24
 ; MMR6-NEXT:    jrc $ra
 entry:
   %r = sdiv i64 %a, %b
   ret i64 %r
 }

 ; GP64R6-NEXT:    ld $ra, 8($sp) # 8-byte Folded Reload
 ; GP64R6-NEXT:    jr $ra
 ; GP64R6-NEXT:    daddiu $sp, $sp, 16
 ;
 ; MMR3-LABEL: sdiv_i128:
 ; MMR3:       # %bb.0: # %entry
 ; MMR3-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR3-NEXT:    addiu $2, $2, %lo(_gp_disp)
-; MMR3-NEXT:    addiusp -48
-; MMR3-NEXT:    .cfi_def_cfa_offset 48
-; MMR3-NEXT:    sw $ra, 44($sp) # 4-byte Folded Spill
-; MMR3-NEXT:    swp $16, 36($sp)
+; MMR3-NEXT:    addiusp -40
+; MMR3-NEXT:    .cfi_def_cfa_offset 40
+; MMR3-NEXT:    sw $ra, 36($sp) # 4-byte Folded Spill
+; MMR3-NEXT:    sw $17, 32($sp) # 4-byte Folded Spill
 ; MMR3-NEXT:    .cfi_offset 31, -4
 ; MMR3-NEXT:    .cfi_offset 17, -8
-; MMR3-NEXT:    .cfi_offset 16, -12
-; MMR3-NEXT:    addu $16, $2, $25
+; MMR3-NEXT:    addu $gp, $2, $25
 ; MMR3-NEXT:    move $1, $7
-; MMR3-NEXT:    lw $7, 68($sp)
-; MMR3-NEXT:    lw $17, 72($sp)
-; MMR3-NEXT:    lw $3, 76($sp)
+; MMR3-NEXT:    lw $7, 60($sp)
+; MMR3-NEXT:    lw $17, 64($sp)
+; MMR3-NEXT:    lw $3, 68($sp)
 ; MMR3-NEXT:    move $2, $sp
 ; MMR3-NEXT:    sw16 $3, 28($2)
 ; MMR3-NEXT:    sw16 $17, 24($2)
 ; MMR3-NEXT:    sw16 $7, 20($2)
-; MMR3-NEXT:    lw $3, 64($sp)
+; MMR3-NEXT:    lw $3, 56($sp)
 ; MMR3-NEXT:    sw16 $3, 16($2)
-; MMR3-NEXT:    lw $25, %call16(__divti3)($16)
+; MMR3-NEXT:    lw $25, %call16(__divti3)($gp)
 ; MMR3-NEXT:    move $7, $1
-; MMR3-NEXT:    move $gp, $16
 ; MMR3-NEXT:    jalr $25
 ; MMR3-NEXT:    nop
-; MMR3-NEXT:    lwp $16, 36($sp)
-; MMR3-NEXT:    lw $ra, 44($sp) # 4-byte Folded Reload
-; MMR3-NEXT:    addiusp 48
+; MMR3-NEXT:    lw $17, 32($sp) # 4-byte Folded Reload
+; MMR3-NEXT:    lw $ra, 36($sp) # 4-byte Folded Reload
+; MMR3-NEXT:    addiusp 40
 ; MMR3-NEXT:    jrc $ra
 ;
 ; MMR6-LABEL: sdiv_i128:
 ; MMR6:       # %bb.0: # %entry
 ; MMR6-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR6-NEXT:    addiu $2, $2, %lo(_gp_disp)
-; MMR6-NEXT:    addiu $sp, $sp, -48
-; MMR6-NEXT:    .cfi_def_cfa_offset 48
-; MMR6-NEXT:    sw $ra, 44($sp) # 4-byte Folded Spill
-; MMR6-NEXT:    sw $17, 40($sp) # 4-byte Folded Spill
-; MMR6-NEXT:    sw $16, 36($sp) # 4-byte Folded Spill
+; MMR6-NEXT:    addiu $sp, $sp, -40
+; MMR6-NEXT:    .cfi_def_cfa_offset 40
+; MMR6-NEXT:    sw $ra, 36($sp) # 4-byte Folded Spill
+; MMR6-NEXT:    sw $17, 32($sp) # 4-byte Folded Spill
 ; MMR6-NEXT:    .cfi_offset 31, -4
 ; MMR6-NEXT:    .cfi_offset 17, -8
-; MMR6-NEXT:    .cfi_offset 16, -12
-; MMR6-NEXT:    addu $16, $2, $25
+; MMR6-NEXT:    addu $gp, $2, $25
 ; MMR6-NEXT:    move $1, $7
-; MMR6-NEXT:    lw $7, 68($sp)
-; MMR6-NEXT:    lw $17, 72($sp)
-; MMR6-NEXT:    lw $3, 76($sp)
+; MMR6-NEXT:    lw $7, 60($sp)
+; MMR6-NEXT:    lw $17, 64($sp)
+; MMR6-NEXT:    lw $3, 68($sp)
 ; MMR6-NEXT:    move $2, $sp
 ; MMR6-NEXT:    sw16 $3, 28($2)
 ; MMR6-NEXT:    sw16 $17, 24($2)
 ; MMR6-NEXT:    sw16 $7, 20($2)
-; MMR6-NEXT:    lw $3, 64($sp)
+; MMR6-NEXT:    lw $3, 56($sp)
 ; MMR6-NEXT:    sw16 $3, 16($2)
-; MMR6-NEXT:    lw $25, %call16(__divti3)($16)
+; MMR6-NEXT:    lw $25, %call16(__divti3)($gp)
 ; MMR6-NEXT:    move $7, $1
-; MMR6-NEXT:    move $gp, $16
 ; MMR6-NEXT:    jalr $25
-; MMR6-NEXT:    lw $16, 36($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    lw $17, 40($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    lw $ra, 44($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    addiu $sp, $sp, 48
+; MMR6-NEXT:    lw $17, 32($sp) # 4-byte Folded Reload
+; MMR6-NEXT:    lw $ra, 36($sp) # 4-byte Folded Reload
+; MMR6-NEXT:    addiu $sp, $sp, 40
 ; MMR6-NEXT:    jrc $ra
 entry:
   %r = sdiv i128 %a, %b
   ret i128 %r
 }

llvm/test/CodeGen/Mips/llvm-ir/srem.ll

 ; MMR3-LABEL: srem_i64:
 ; MMR3:       # %bb.0: # %entry
 ; MMR3-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR3-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MMR3-NEXT:    addiusp -24
 ; MMR3-NEXT:    .cfi_def_cfa_offset 24
 ; MMR3-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MMR3-NEXT:    .cfi_offset 31, -4
-; MMR3-NEXT:    addu $2, $2, $25
-; MMR3-NEXT:    lw $25, %call16(__moddi3)($2)
-; MMR3-NEXT:    move $gp, $2
+; MMR3-NEXT:    addu $gp, $2, $25
+; MMR3-NEXT:    lw $25, %call16(__moddi3)($gp)
 ; MMR3-NEXT:    jalr $25
 ; MMR3-NEXT:    nop
 ; MMR3-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    addiusp 24
 ; MMR3-NEXT:    jrc $ra
 ;
 ; MMR6-LABEL: srem_i64:
 ; MMR6:       # %bb.0: # %entry
 ; MMR6-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR6-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MMR6-NEXT:    addiu $sp, $sp, -24
 ; MMR6-NEXT:    .cfi_def_cfa_offset 24
 ; MMR6-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MMR6-NEXT:    .cfi_offset 31, -4
-; MMR6-NEXT:    addu $2, $2, $25
-; MMR6-NEXT:    lw $25, %call16(__moddi3)($2)
-; MMR6-NEXT:    move $gp, $2
+; MMR6-NEXT:    addu $gp, $2, $25
+; MMR6-NEXT:    lw $25, %call16(__moddi3)($gp)
 ; MMR6-NEXT:    jalr $25
 ; MMR6-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MMR6-NEXT:    addiu $sp, $sp, 24
 ; MMR6-NEXT:    jrc $ra
 entry:
   %r = srem i64 %a, %b
   ret i64 %r
 }

 ; GP64R6-NEXT:    ld $ra, 8($sp) # 8-byte Folded Reload
 ; GP64R6-NEXT:    jr $ra
 ; GP64R6-NEXT:    daddiu $sp, $sp, 16
 ;
 ; MMR3-LABEL: srem_i128:
 ; MMR3:       # %bb.0: # %entry
 ; MMR3-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR3-NEXT:    addiu $2, $2, %lo(_gp_disp)
-; MMR3-NEXT:    addiusp -48
-; MMR3-NEXT:    .cfi_def_cfa_offset 48
-; MMR3-NEXT:    sw $ra, 44($sp) # 4-byte Folded Spill
-; MMR3-NEXT:    swp $16, 36($sp)
+; MMR3-NEXT:    addiusp -40
+; MMR3-NEXT:    .cfi_def_cfa_offset 40
+; MMR3-NEXT:    sw $ra, 36($sp) # 4-byte Folded Spill
+; MMR3-NEXT:    sw $17, 32($sp) # 4-byte Folded Spill
 ; MMR3-NEXT:    .cfi_offset 31, -4
 ; MMR3-NEXT:    .cfi_offset 17, -8
-; MMR3-NEXT:    .cfi_offset 16, -12
-; MMR3-NEXT:    addu $16, $2, $25
+; MMR3-NEXT:    addu $gp, $2, $25
 ; MMR3-NEXT:    move $1, $7
-; MMR3-NEXT:    lw $7, 68($sp)
-; MMR3-NEXT:    lw $17, 72($sp)
-; MMR3-NEXT:    lw $3, 76($sp)
+; MMR3-NEXT:    lw $7, 60($sp)
+; MMR3-NEXT:    lw $17, 64($sp)
+; MMR3-NEXT:    lw $3, 68($sp)
 ; MMR3-NEXT:    move $2, $sp
 ; MMR3-NEXT:    sw16 $3, 28($2)
 ; MMR3-NEXT:    sw16 $17, 24($2)
 ; MMR3-NEXT:    sw16 $7, 20($2)
-; MMR3-NEXT:    lw $3, 64($sp)
+; MMR3-NEXT:    lw $3, 56($sp)
 ; MMR3-NEXT:    sw16 $3, 16($2)
-; MMR3-NEXT:    lw $25, %call16(__modti3)($16)
+; MMR3-NEXT:    lw $25, %call16(__modti3)($gp)
 ; MMR3-NEXT:    move $7, $1
-; MMR3-NEXT:    move $gp, $16
 ; MMR3-NEXT:    jalr $25
 ; MMR3-NEXT:    nop
-; MMR3-NEXT:    lwp $16, 36($sp)
-; MMR3-NEXT:    lw $ra, 44($sp) # 4-byte Folded Reload
-; MMR3-NEXT:    addiusp 48
+; MMR3-NEXT:    lw $17, 32($sp) # 4-byte Folded Reload
+; MMR3-NEXT:    lw $ra, 36($sp) # 4-byte Folded Reload
+; MMR3-NEXT:    addiusp 40
 ; MMR3-NEXT:    jrc $ra
 ;
 ; MMR6-LABEL: srem_i128:
 ; MMR6:       # %bb.0: # %entry
 ; MMR6-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR6-NEXT:    addiu $2, $2, %lo(_gp_disp)
-; MMR6-NEXT:    addiu $sp, $sp, -48
-; MMR6-NEXT:    .cfi_def_cfa_offset 48
-; MMR6-NEXT:    sw $ra, 44($sp) # 4-byte Folded Spill
-; MMR6-NEXT:    sw $17, 40($sp) # 4-byte Folded Spill
-; MMR6-NEXT:    sw $16, 36($sp) # 4-byte Folded Spill
+; MMR6-NEXT:    addiu $sp, $sp, -40
+; MMR6-NEXT:    .cfi_def_cfa_offset 40
+; MMR6-NEXT:    sw $ra, 36($sp) # 4-byte Folded Spill
+; MMR6-NEXT:    sw $17, 32($sp) # 4-byte Folded Spill
 ; MMR6-NEXT:    .cfi_offset 31, -4
 ; MMR6-NEXT:    .cfi_offset 17, -8
-; MMR6-NEXT:    .cfi_offset 16, -12
-; MMR6-NEXT:    addu $16, $2, $25
+; MMR6-NEXT:    addu $gp, $2, $25
 ; MMR6-NEXT:    move $1, $7
-; MMR6-NEXT:    lw $7, 68($sp)
-; MMR6-NEXT:    lw $17, 72($sp)
-; MMR6-NEXT:    lw $3, 76($sp)
+; MMR6-NEXT:    lw $7, 60($sp)
+; MMR6-NEXT:    lw $17, 64($sp)
+; MMR6-NEXT:    lw $3, 68($sp)
 ; MMR6-NEXT:    move $2, $sp
 ; MMR6-NEXT:    sw16 $3, 28($2)
 ; MMR6-NEXT:    sw16 $17, 24($2)
 ; MMR6-NEXT:    sw16 $7, 20($2)
-; MMR6-NEXT:    lw $3, 64($sp)
+; MMR6-NEXT:    lw $3, 56($sp)
 ; MMR6-NEXT:    sw16 $3, 16($2)
-; MMR6-NEXT:    lw $25, %call16(__modti3)($16)
+; MMR6-NEXT:    lw $25, %call16(__modti3)($gp)
 ; MMR6-NEXT:    move $7, $1
-; MMR6-NEXT:    move $gp, $16
 ; MMR6-NEXT:    jalr $25
-; MMR6-NEXT:    lw $16, 36($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    lw $17, 40($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    lw $ra, 44($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    addiu $sp, $sp, 48
+; MMR6-NEXT:    lw $17, 32($sp) # 4-byte Folded Reload
+; MMR6-NEXT:    lw $ra, 36($sp) # 4-byte Folded Reload
+; MMR6-NEXT:    addiu $sp, $sp, 40
 ; MMR6-NEXT:    jrc $ra
 entry:
   %r = srem i128 %a, %b
   ret i128 %r
 }

llvm/test/CodeGen/Mips/llvm-ir/udiv.ll

 ; MMR3-LABEL: udiv_i64:
 ; MMR3:       # %bb.0: # %entry
 ; MMR3-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR3-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MMR3-NEXT:    addiusp -24
 ; MMR3-NEXT:    .cfi_def_cfa_offset 24
 ; MMR3-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MMR3-NEXT:    .cfi_offset 31, -4
-; MMR3-NEXT:    addu $2, $2, $25
-; MMR3-NEXT:    lw $25, %call16(__udivdi3)($2)
-; MMR3-NEXT:    move $gp, $2
+; MMR3-NEXT:    addu $gp, $2, $25
+; MMR3-NEXT:    lw $25, %call16(__udivdi3)($gp)
 ; MMR3-NEXT:    jalr $25
 ; MMR3-NEXT:    nop
 ; MMR3-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    addiusp 24
 ; MMR3-NEXT:    jrc $ra
 ;
 ; MMR6-LABEL: udiv_i64:
 ; MMR6:       # %bb.0: # %entry
 ; MMR6-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR6-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MMR6-NEXT:    addiu $sp, $sp, -24
 ; MMR6-NEXT:    .cfi_def_cfa_offset 24
 ; MMR6-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MMR6-NEXT:    .cfi_offset 31, -4
-; MMR6-NEXT:    addu $2, $2, $25
-; MMR6-NEXT:    lw $25, %call16(__udivdi3)($2)
-; MMR6-NEXT:    move $gp, $2
+; MMR6-NEXT:    addu $gp, $2, $25
+; MMR6-NEXT:    lw $25, %call16(__udivdi3)($gp)
 ; MMR6-NEXT:    jalr $25
 ; MMR6-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MMR6-NEXT:    addiu $sp, $sp, 24
 ; MMR6-NEXT:    jrc $ra
 entry:
   %r = udiv i64 %a, %b
   ret i64 %r
 }

 ; GP64R6-NEXT:    ld $ra, 8($sp) # 8-byte Folded Reload
 ; GP64R6-NEXT:    jr $ra
 ; GP64R6-NEXT:    daddiu $sp, $sp, 16
 ;
 ; MMR3-LABEL: udiv_i128:
 ; MMR3:       # %bb.0: # %entry
 ; MMR3-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR3-NEXT:    addiu $2, $2, %lo(_gp_disp)
-; MMR3-NEXT:    addiusp -48
-; MMR3-NEXT:    .cfi_def_cfa_offset 48
-; MMR3-NEXT:    sw $ra, 44($sp) # 4-byte Folded Spill
-; MMR3-NEXT:    swp $16, 36($sp)
+; MMR3-NEXT:    addiusp -40
+; MMR3-NEXT:    .cfi_def_cfa_offset 40
+; MMR3-NEXT:    sw $ra, 36($sp) # 4-byte Folded Spill
+; MMR3-NEXT:    sw $17, 32($sp) # 4-byte Folded Spill
 ; MMR3-NEXT:    .cfi_offset 31, -4
 ; MMR3-NEXT:    .cfi_offset 17, -8
-; MMR3-NEXT:    .cfi_offset 16, -12
-; MMR3-NEXT:    addu $16, $2, $25
+; MMR3-NEXT:    addu $gp, $2, $25
 ; MMR3-NEXT:    move $1, $7
-; MMR3-NEXT:    lw $7, 68($sp)
-; MMR3-NEXT:    lw $17, 72($sp)
-; MMR3-NEXT:    lw $3, 76($sp)
+; MMR3-NEXT:    lw $7, 60($sp)
+; MMR3-NEXT:    lw $17, 64($sp)
+; MMR3-NEXT:    lw $3, 68($sp)
 ; MMR3-NEXT:    move $2, $sp
 ; MMR3-NEXT:    sw16 $3, 28($2)
 ; MMR3-NEXT:    sw16 $17, 24($2)
 ; MMR3-NEXT:    sw16 $7, 20($2)
-; MMR3-NEXT:    lw $3, 64($sp)
+; MMR3-NEXT:    lw $3, 56($sp)
 ; MMR3-NEXT:    sw16 $3, 16($2)
-; MMR3-NEXT:    lw $25, %call16(__udivti3)($16)
+; MMR3-NEXT:    lw $25, %call16(__udivti3)($gp)
 ; MMR3-NEXT:    move $7, $1
-; MMR3-NEXT:    move $gp, $16
 ; MMR3-NEXT:    jalr $25
 ; MMR3-NEXT:    nop
-; MMR3-NEXT:    lwp $16, 36($sp)
-; MMR3-NEXT:    lw $ra, 44($sp) # 4-byte Folded Reload
-; MMR3-NEXT:    addiusp 48
+; MMR3-NEXT:    lw $17, 32($sp) # 4-byte Folded Reload
+; MMR3-NEXT:    lw $ra, 36($sp) # 4-byte Folded Reload
+; MMR3-NEXT:    addiusp 40
 ; MMR3-NEXT:    jrc $ra
 ;
 ; MMR6-LABEL: udiv_i128:
 ; MMR6:       # %bb.0: # %entry
 ; MMR6-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR6-NEXT:    addiu $2, $2, %lo(_gp_disp)
-; MMR6-NEXT:    addiu $sp, $sp, -48
-; MMR6-NEXT:    .cfi_def_cfa_offset 48
-; MMR6-NEXT:    sw $ra, 44($sp) # 4-byte Folded Spill
-; MMR6-NEXT:    sw $17, 40($sp) # 4-byte Folded Spill
-; MMR6-NEXT:    sw $16, 36($sp) # 4-byte Folded Spill
+; MMR6-NEXT:    addiu $sp, $sp, -40
+; MMR6-NEXT:    .cfi_def_cfa_offset 40
+; MMR6-NEXT:    sw $ra, 36($sp) # 4-byte Folded Spill
+; MMR6-NEXT:    sw $17, 32($sp) # 4-byte Folded Spill
 ; MMR6-NEXT:    .cfi_offset 31, -4
 ; MMR6-NEXT:    .cfi_offset 17, -8
-; MMR6-NEXT:    .cfi_offset 16, -12
-; MMR6-NEXT:    addu $16, $2, $25
+; MMR6-NEXT:    addu $gp, $2, $25
 ; MMR6-NEXT:    move $1, $7
-; MMR6-NEXT:    lw $7, 68($sp)
-; MMR6-NEXT:    lw $17, 72($sp)
-; MMR6-NEXT:    lw $3, 76($sp)
+; MMR6-NEXT:    lw $7, 60($sp)
+; MMR6-NEXT:    lw $17, 64($sp)
+; MMR6-NEXT:    lw $3, 68($sp)
 ; MMR6-NEXT:    move $2, $sp
 ; MMR6-NEXT:    sw16 $3, 28($2)
 ; MMR6-NEXT:    sw16 $17, 24($2)
 ; MMR6-NEXT:    sw16 $7, 20($2)
-; MMR6-NEXT:    lw $3, 64($sp)
+; MMR6-NEXT:    lw $3, 56($sp)
 ; MMR6-NEXT:    sw16 $3, 16($2)
-; MMR6-NEXT:    lw $25, %call16(__udivti3)($16)
+; MMR6-NEXT:    lw $25, %call16(__udivti3)($gp)
 ; MMR6-NEXT:    move $7, $1
-; MMR6-NEXT:    move $gp, $16
 ; MMR6-NEXT:    jalr $25
-; MMR6-NEXT:    lw $16, 36($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    lw $17, 40($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    lw $ra, 44($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    addiu $sp, $sp, 48
+; MMR6-NEXT:    lw $17, 32($sp) # 4-byte Folded Reload
+; MMR6-NEXT:    lw $ra, 36($sp) # 4-byte Folded Reload
+; MMR6-NEXT:    addiu $sp, $sp, 40
 ; MMR6-NEXT:    jrc $ra
 entry:
   %r = udiv i128 %a, %b
   ret i128 %r
 }

llvm/test/CodeGen/Mips/llvm-ir/urem.ll

 ; MMR3-LABEL: urem_i64:
 ; MMR3:       # %bb.0: # %entry
 ; MMR3-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR3-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MMR3-NEXT:    addiusp -24
 ; MMR3-NEXT:    .cfi_def_cfa_offset 24
 ; MMR3-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MMR3-NEXT:    .cfi_offset 31, -4
-; MMR3-NEXT:    addu $2, $2, $25
-; MMR3-NEXT:    lw $25, %call16(__umoddi3)($2)
-; MMR3-NEXT:    move $gp, $2
+; MMR3-NEXT:    addu $gp, $2, $25
+; MMR3-NEXT:    lw $25, %call16(__umoddi3)($gp)
 ; MMR3-NEXT:    jalr $25
 ; MMR3-NEXT:    nop
 ; MMR3-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    addiusp 24
 ; MMR3-NEXT:    jrc $ra
 ;
 ; MMR6-LABEL: urem_i64:
 ; MMR6:       # %bb.0: # %entry
 ; MMR6-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR6-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MMR6-NEXT:    addiu $sp, $sp, -24
 ; MMR6-NEXT:    .cfi_def_cfa_offset 24
 ; MMR6-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MMR6-NEXT:    .cfi_offset 31, -4
-; MMR6-NEXT:    addu $2, $2, $25
-; MMR6-NEXT:    lw $25, %call16(__umoddi3)($2)
-; MMR6-NEXT:    move $gp, $2
+; MMR6-NEXT:    addu $gp, $2, $25
+; MMR6-NEXT:    lw $25, %call16(__umoddi3)($gp)
 ; MMR6-NEXT:    jalr $25
 ; MMR6-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MMR6-NEXT:    addiu $sp, $sp, 24
 ; MMR6-NEXT:    jrc $ra
 entry:
   %r = urem i64 %a, %b
   ret i64 %r
 }

 ; GP64R6-NEXT:    ld $ra, 8($sp) # 8-byte Folded Reload
 ; GP64R6-NEXT:    jr $ra
 ; GP64R6-NEXT:    daddiu $sp, $sp, 16
 ;
 ; MMR3-LABEL: urem_i128:
 ; MMR3:       # %bb.0: # %entry
 ; MMR3-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR3-NEXT:    addiu $2, $2, %lo(_gp_disp)
-; MMR3-NEXT:    addiusp -48
-; MMR3-NEXT:    .cfi_def_cfa_offset 48
-; MMR3-NEXT:    sw $ra, 44($sp) # 4-byte Folded Spill
-; MMR3-NEXT:    swp $16, 36($sp)
+; MMR3-NEXT:    addiusp -40
+; MMR3-NEXT:    .cfi_def_cfa_offset 40
+; MMR3-NEXT:    sw $ra, 36($sp) # 4-byte Folded Spill
+; MMR3-NEXT:    sw $17, 32($sp) # 4-byte Folded Spill
 ; MMR3-NEXT:    .cfi_offset 31, -4
 ; MMR3-NEXT:    .cfi_offset 17, -8
-; MMR3-NEXT:    .cfi_offset 16, -12
-; MMR3-NEXT:    addu $16, $2, $25
+; MMR3-NEXT:    addu $gp, $2, $25
 ; MMR3-NEXT:    move $1, $7
-; MMR3-NEXT:    lw $7, 68($sp)
-; MMR3-NEXT:    lw $17, 72($sp)
-; MMR3-NEXT:    lw $3, 76($sp)
+; MMR3-NEXT:    lw $7, 60($sp)
+; MMR3-NEXT:    lw $17, 64($sp)
+; MMR3-NEXT:    lw $3, 68($sp)
 ; MMR3-NEXT:    move $2, $sp
 ; MMR3-NEXT:    sw16 $3, 28($2)
 ; MMR3-NEXT:    sw16 $17, 24($2)
 ; MMR3-NEXT:    sw16 $7, 20($2)
-; MMR3-NEXT:    lw $3, 64($sp)
+; MMR3-NEXT:    lw $3, 56($sp)
 ; MMR3-NEXT:    sw16 $3, 16($2)
-; MMR3-NEXT:    lw $25, %call16(__umodti3)($16)
+; MMR3-NEXT:    lw $25, %call16(__umodti3)($gp)
 ; MMR3-NEXT:    move $7, $1
-; MMR3-NEXT:    move $gp, $16
 ; MMR3-NEXT:    jalr $25
 ; MMR3-NEXT:    nop
-; MMR3-NEXT:    lwp $16, 36($sp)
-; MMR3-NEXT:    lw $ra, 44($sp) # 4-byte Folded Reload
-; MMR3-NEXT:    addiusp 48
+; MMR3-NEXT:    lw $17, 32($sp) # 4-byte Folded Reload
+; MMR3-NEXT:    lw $ra, 36($sp) # 4-byte Folded Reload
+; MMR3-NEXT:    addiusp 40
 ; MMR3-NEXT:    jrc $ra
 ;
 ; MMR6-LABEL: urem_i128:
 ; MMR6:       # %bb.0: # %entry
 ; MMR6-NEXT:    lui $2, %hi(_gp_disp)
 ; MMR6-NEXT:    addiu $2, $2, %lo(_gp_disp)
-; MMR6-NEXT:    addiu $sp, $sp, -48
-; MMR6-NEXT:    .cfi_def_cfa_offset 48
-; MMR6-NEXT:    sw $ra, 44($sp) # 4-byte Folded Spill
-; MMR6-NEXT:    sw $17, 40($sp) # 4-byte Folded Spill
-; MMR6-NEXT:    sw $16, 36($sp) # 4-byte Folded Spill
+; MMR6-NEXT:    addiu $sp, $sp, -40
+; MMR6-NEXT:    .cfi_def_cfa_offset 40
+; MMR6-NEXT:    sw $ra, 36($sp) # 4-byte Folded Spill
+; MMR6-NEXT:    sw $17, 32($sp) # 4-byte Folded Spill
 ; MMR6-NEXT:    .cfi_offset 31, -4
 ; MMR6-NEXT:    .cfi_offset 17, -8
-; MMR6-NEXT:    .cfi_offset 16, -12
-; MMR6-NEXT:    addu $16, $2, $25
+; MMR6-NEXT:    addu $gp, $2, $25
 ; MMR6-NEXT:    move $1, $7
-; MMR6-NEXT:    lw $7, 68($sp)
-; MMR6-NEXT:    lw $17, 72($sp)
-; MMR6-NEXT:    lw $3, 76($sp)
+; MMR6-NEXT:    lw $7, 60($sp)
+; MMR6-NEXT:    lw $17, 64($sp)
+; MMR6-NEXT:    lw $3, 68($sp)
 ; MMR6-NEXT:    move $2, $sp
 ; MMR6-NEXT:    sw16 $3, 28($2)
 ; MMR6-NEXT:    sw16 $17, 24($2)
 ; MMR6-NEXT:    sw16 $7, 20($2)
-; MMR6-NEXT:    lw $3, 64($sp)
+; MMR6-NEXT:    lw $3, 56($sp)
 ; MMR6-NEXT:    sw16 $3, 16($2)
-; MMR6-NEXT:    lw $25, %call16(__umodti3)($16)
+; MMR6-NEXT:    lw $25, %call16(__umodti3)($gp)
 ; MMR6-NEXT:    move $7, $1
-; MMR6-NEXT:    move $gp, $16
 ; MMR6-NEXT:    jalr $25
-; MMR6-NEXT:    lw $16, 36($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    lw $17, 40($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    lw $ra, 44($sp) # 4-byte Folded Reload
-; MMR6-NEXT:    addiu $sp, $sp, 48
+; MMR6-NEXT:    lw $17, 32($sp) # 4-byte Folded Reload
+; MMR6-NEXT:    lw $ra, 36($sp) # 4-byte Folded Reload
+; MMR6-NEXT:    addiu $sp, $sp, 40
 ; MMR6-NEXT:    jrc $ra
 entry:
   %r = urem i128 %a, %b
   ret i128 %r
 }

llvm/test/CodeGen/Mips/mcount.ll

 ; MIPS32-MM-PIC-LABEL: foo:
 ; MIPS32-MM-PIC:       # %bb.0: # %entry
 ; MIPS32-MM-PIC-NEXT:    lui $2, %hi(_gp_disp)
 ; MIPS32-MM-PIC-NEXT:    addiu $2, $2, %lo(_gp_disp)
 ; MIPS32-MM-PIC-NEXT:    addiu $sp, $sp, -24
 ; MIPS32-MM-PIC-NEXT:    .cfi_def_cfa_offset 24
 ; MIPS32-MM-PIC-NEXT:    sw $ra, 20($sp) # 4-byte Folded Spill
 ; MIPS32-MM-PIC-NEXT:    .cfi_offset 31, -4
-; MIPS32-MM-PIC-NEXT:    addu $2, $2, $25
-; MIPS32-MM-PIC-NEXT:    lw $25, %call16(_mcount)($2)
-; MIPS32-MM-PIC-NEXT:    move $gp, $2
+; MIPS32-MM-PIC-NEXT:    addu $gp, $2, $25
+; MIPS32-MM-PIC-NEXT:    lw $25, %call16(_mcount)($gp)
 ; MIPS32-MM-PIC-NEXT:    move $1, $ra
 ; MIPS32-MM-PIC-NEXT:    .reloc ($tmp0), R_MICROMIPS_JALR, _mcount
 ; MIPS32-MM-PIC-NEXT:  $tmp0:
 ; MIPS32-MM-PIC-NEXT:    jalr $25
 ; MIPS32-MM-PIC-NEXT:    addiu $sp, $sp, -8
 ; MIPS32-MM-PIC-NEXT:    lw $ra, 20($sp) # 4-byte Folded Reload
 ; MIPS32-MM-PIC-NEXT:    jr $ra
 ; MIPS32-MM-PIC-NEXT:    addiu $sp, $sp, 24
 entry:
   call void @_mcount()
   ret void
 }
 
 declare void @_mcount()

llvm/test/CodeGen/Mips/micromips-gp-rc.ll

 ; RUN: llc -march=mipsel -mcpu=mips32r2 -mattr=+micromips \
 ; RUN:   -relocation-model=pic -O3 < %s | FileCheck %s
 
 @g = external global i32
 
 ; Function Attrs: noreturn nounwind
 define void @foo() #0 {
 entry:
   %0 = load i32, i32* @g, align 4
   tail call void @exit(i32 signext %0)
   unreachable
 }
 
 ; Function Attrs: noreturn
 declare void @exit(i32 signext)
 
-; CHECK: move $gp, ${{[0-9]+}}
+; CHECK: addu $gp, ${{[0-9]+}}, ${{[0-9]+}}
 

llvm/test/CodeGen/Mips/tailcall/tailcall.ll

 @g7 = common global i32 0, align 4
 @g8 = common global i32 0, align 4
 @g9 = common global i32 0, align 4
 
 define i32 @caller1(i32 %a0) nounwind {
 entry:
 ; ALL-LABEL: caller1:
 ; PIC32: jalr $25
-; PIC32MM: jalr $25
+; PIC32MM: jalr{{(s16)?}} $25
 ; PIC32R6: jalr $25
 ; STATIC32: jal
 ; STATIC32MMR6: balc
 ; N64: jalr $25
 ; N64R6: jalr $25
 ; PIC16: jalrc
 
   %call = tail call i32 @callee1(i32 1, i32 1, i32 1, i32 %a0) nounwind
; PIC16: jalrc
   ret i32 %call
 }
 
 define internal fastcc i32 @caller8_1() nounwind noinline {
 entry:
 ; ALL-LABEL: caller8_1:
 ; PIC32: jalr $25
 ; PIC32R6: jalr $25
-; PIC32MM: jalr $25
+; PIC32MM: jalr{{(s16)?}} $25
 ; STATIC32: jal
 ; STATIC32MMR6: balc
 ; PIC64: jalr $25
 ; STATIC64: jal
 ; PIC16: jalrc
 
   %call = tail call i32 (i32, ...) @callee8(i32 2, i32 1) nounwind
   ret i32 %call

 
 declare i32 @callee13(i32, ...)
 
 define i32 @caller13() nounwind {
 entry:
 ; ALL-LABEL: caller13:
 ; PIC32: jalr $25
 ; PIC32R6: jalr $25
-; PIC32MM: jalr $25
+; PIC32MM: jalr{{(s16)?}} $25
 ; STATIC32: jal
 ; STATIC32MMR6: balc
 ; STATIC64: jal
 ; PIC64R6: jalr $25
 ; PIC64: jalr $25
 ; PIC16: jalrc
 
   %call = tail call i32 (i32, ...) @callee13(i32 1, i32 2) nounwind

llvm/test/CodeGen/Mips/tls.ll

 ; PIC64-DAG:   ld      $25, %call16(__tls_get_addr)($[[R0]])
 ; PIC64-DAG:   jalr    $25
 ; PIC64-DAG:   lw      $2, 0($2)
 
 ; MM-LABEL:       f1:
 ; MM-DAG:   addu    $[[R0:[a-z0-9]+]], $2, $25
 ; MM-DAG:   addiu   $4, $[[R0]], %tlsgd(t1)
 ; MM-DAG:   lw      $25, %call16(__tls_get_addr)($[[R0]])
-; MM-DAG:   move    $gp, $2
 ; MM-DAG:   jalr    $25
 ; MM-DAG:   lw16    $2, 0($2)
 }
 
 @t2 = external thread_local global i32
 
 define dso_preemptable i32 @f2() nounwind {
 entry:

llvm/test/CodeGen/RISCV/rvv/vselect-fp-rv32.ll

; CHECK-NEXT:    ret
   ret <vscale x 16 x double> %sel
 }
 
 define void @vselect_legalize_regression(<vscale x 16 x double> %a, <vscale x 16 x i1> %ma, <vscale x 16 x i1> %mb, <vscale x 16 x double>* %out) {
 ; CHECK-LABEL: vselect_legalize_regression:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli a2, zero, e8, m2, ta, mu
 ; CHECK-NEXT:    vle1.v v25, (a0)
-; CHECK-NEXT:    vmand.mm v25, v0, v25
+; CHECK-NEXT:    vmand.mm v2, v0, v25
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    srli a0, a0, 3
 ; CHECK-NEXT:    vsetvli a2, zero, e8, mf4, ta, mu
-; CHECK-NEXT:    vslidedown.vx v0, v25, a0
-; CHECK-NEXT:    vmv1r.v v2, v25
+; CHECK-NEXT:    vslidedown.vx v0, v2, a0
 ; CHECK-NEXT:    vsetvli a2, zero, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv.v.i v24, 0
 ; CHECK-NEXT:    vmerge.vvm v16, v24, v16, v0
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vmerge.vvm v8, v24, v8, v0
 ; CHECK-NEXT:    vs8r.v v8, (a1)
 ; CHECK-NEXT:    slli a0, a0, 6
 ; CHECK-NEXT:    add a0, a1, a0
 ; CHECK-NEXT:    vs8r.v v16, (a0)
 ; CHECK-NEXT:    ret
   %cond = and <vscale x 16 x i1> %ma, %mb
   %sel = select <vscale x 16 x i1> %cond, <vscale x 16 x double> %a, <vscale x 16 x double> zeroinitializer
   store <vscale x 16 x double> %sel, <vscale x 16 x double>* %out
   ret void
 }

llvm/test/CodeGen/RISCV/rvv/vselect-fp-rv64.ll

; CHECK-NEXT:    ret
   ret <vscale x 16 x double> %sel
 }
 
 define void @vselect_legalize_regression(<vscale x 16 x double> %a, <vscale x 16 x i1> %ma, <vscale x 16 x i1> %mb, <vscale x 16 x double>* %out) {
 ; CHECK-LABEL: vselect_legalize_regression:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli a2, zero, e8, m2, ta, mu
 ; CHECK-NEXT:    vle1.v v25, (a0)
-; CHECK-NEXT:    vmand.mm v25, v0, v25
+; CHECK-NEXT:    vmand.mm v2, v0, v25
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    srli a0, a0, 3
 ; CHECK-NEXT:    vsetvli a2, zero, e8, mf4, ta, mu
-; CHECK-NEXT:    vslidedown.vx v0, v25, a0
-; CHECK-NEXT:    vmv1r.v v2, v25
+; CHECK-NEXT:    vslidedown.vx v0, v2, a0
 ; CHECK-NEXT:    vsetvli a2, zero, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv.v.i v24, 0
 ; CHECK-NEXT:    vmerge.vvm v16, v24, v16, v0
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vmerge.vvm v8, v24, v8, v0
 ; CHECK-NEXT:    vs8r.v v8, (a1)
 ; CHECK-NEXT:    slli a0, a0, 6
 ; CHECK-NEXT:    add a0, a1, a0
 ; CHECK-NEXT:    vs8r.v v16, (a0)
 ; CHECK-NEXT:    ret
   %cond = and <vscale x 16 x i1> %ma, %mb
   %sel = select <vscale x 16 x i1> %cond, <vscale x 16 x double> %a, <vscale x 16 x double> zeroinitializer
   store <vscale x 16 x double> %sel, <vscale x 16 x double>* %out
   ret void
 }

llvm/test/CodeGen/Thumb2/mve-vqdmulh.ll

 ; CHECK-NEXT:    vstrw.32 q3, [r1]
 ; CHECK-NEXT:    vldrb.s16 q0, [r3, #8]
 ; CHECK-NEXT:    add r7, sp, #64
 ; CHECK-NEXT:    add r0, sp, #128
 ; CHECK-NEXT:    add r5, sp, #80
 ; CHECK-NEXT:    vstrw.32 q0, [r7]
 ; CHECK-NEXT:    add r6, sp, #144
 ; CHECK-NEXT:    vldrb.s16 q0, [r4, #8]
+; CHECK-NEXT:    add.w r12, sp, #112
 ; CHECK-NEXT:    vstrw.32 q0, [r0]
 ; CHECK-NEXT:    vldrb.s16 q0, [r3]
 ; CHECK-NEXT:    add r3, sp, #96
 ; CHECK-NEXT:    vstrw.32 q0, [r5]
 ; CHECK-NEXT:    vldrb.s16 q0, [r4]
 ; CHECK-NEXT:    add r4, sp, #160
 ; CHECK-NEXT:    vstrw.32 q0, [r6]
 ; CHECK-NEXT:    vldrb.s16 q0, [r2, #8]
 ; CHECK-NEXT:    vstrw.32 q0, [r3]
 ; CHECK-NEXT:    vldrb.s16 q0, [r1, #8]
 ; CHECK-NEXT:    vstrw.32 q0, [r4]
 ; CHECK-NEXT:    vldrb.s16 q0, [r2]
-; CHECK-NEXT:    add r2, sp, #112
-; CHECK-NEXT:    vstrw.32 q0, [r2]
-; CHECK-NEXT:    mov r12, r2
+; CHECK-NEXT:    add r2, sp, #192
+; CHECK-NEXT:    vstrw.32 q0, [r12]
 ; CHECK-NEXT:    vldrb.s16 q0, [r1]
 ; CHECK-NEXT:    add r1, sp, #176
-; CHECK-NEXT:    add r2, sp, #192
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    vldrh.s32 q0, [r7, #8]
 ; CHECK-NEXT:    vldrh.s32 q1, [r0, #8]
 ; CHECK-NEXT:    vmul.i32 q0, q1, q0
 ; CHECK-NEXT:    vshr.s32 q1, q0, #7
 ; CHECK-NEXT:    vmov.i32 q0, #0x7f
 ; CHECK-NEXT:    vmin.s32 q1, q1, q0
 ; CHECK-NEXT:    vstrb.32 q1, [r2, #12]

llvm/test/CodeGen/X86/bitreverse.ll

 ; X64-NEXT:    movq %rdi, %r12
 ; X64-NEXT:    movq {{[0-9]+}}(%rsp), %rbp
 ; X64-NEXT:    movq {{[0-9]+}}(%rsp), %rbx
 ; X64-NEXT:    bswapq %rbx
 ; X64-NEXT:    movabsq $1085102592571150095, %r13 # imm = 0xF0F0F0F0F0F0F0F
 ; X64-NEXT:    movq %rbx, %r10
 ; X64-NEXT:    andq %r13, %r10
 ; X64-NEXT:    shlq $4, %r10
-; X64-NEXT:    movabsq $-1085102592571150096, %rax # imm = 0xF0F0F0F0F0F0F0F0
-; X64-NEXT:    andq %rax, %rbx
+; X64-NEXT:    movabsq $-1085102592571150096, %r15 # imm = 0xF0F0F0F0F0F0F0F0
+; X64-NEXT:    andq %r15, %rbx
 ; X64-NEXT:    shrq $4, %rbx
 ; X64-NEXT:    orq %r10, %rbx
 ; X64-NEXT:    movabsq $3689348814741910323, %r11 # imm = 0x3333333333333333
 ; X64-NEXT:    movq %rbx, %r10
 ; X64-NEXT:    andq %r11, %r10
 ; X64-NEXT:    movabsq $-3689348814741910324, %r14 # imm = 0xCCCCCCCCCCCCCCCC
 ; X64-NEXT:    andq %r14, %rbx
 ; X64-NEXT:    shrq $2, %rbx
 ; X64-NEXT:    leaq (%rbx,%r10,4), %r10
 ; X64-NEXT:    movabsq $6148820866244280320, %rbx # imm = 0x5555000000000000
 ; X64-NEXT:    andq %r10, %rbx
 ; X64-NEXT:    movabsq $-6149102341220990976, %rdi # imm = 0xAAAA000000000000
 ; X64-NEXT:    andq %r10, %rdi
 ; X64-NEXT:    shrq %rdi
 ; X64-NEXT:    leaq (%rdi,%rbx,2), %rdi
 ; X64-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    bswapq %rbp
 ; X64-NEXT:    movq %rbp, %rdi
 ; X64-NEXT:    andq %r13, %rdi
 ; X64-NEXT:    shlq $4, %rdi
-; X64-NEXT:    andq %rax, %rbp
+; X64-NEXT:    andq %r15, %rbp
 ; X64-NEXT:    shrq $4, %rbp
 ; X64-NEXT:    orq %rdi, %rbp
 ; X64-NEXT:    movq %rbp, %rdi
 ; X64-NEXT:    andq %r11, %rdi
 ; X64-NEXT:    andq %r14, %rbp
 ; X64-NEXT:    shrq $2, %rbp
 ; X64-NEXT:    leaq (%rbp,%rdi,4), %rbp
 ; X64-NEXT:    movabsq $6148914691236517205, %rbx # imm = 0x5555555555555555
 ; X64-NEXT:    movq %rbp, %r10
 ; X64-NEXT:    andq %rbx, %r10
 ; X64-NEXT:    movabsq $-6148914691236517206, %rdi # imm = 0xAAAAAAAAAAAAAAAA
 ; X64-NEXT:    andq %rdi, %rbp
 ; X64-NEXT:    shrq %rbp
 ; X64-NEXT:    leaq (%rbp,%r10,2), %rbp
 ; X64-NEXT:    movq %rbp, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    movq {{[0-9]+}}(%rsp), %rbp
 ; X64-NEXT:    bswapq %rbp
 ; X64-NEXT:    movq %rbp, %r10
 ; X64-NEXT:    andq %r13, %r10
 ; X64-NEXT:    shlq $4, %r10
-; X64-NEXT:    andq %rax, %rbp
-; X64-NEXT:    movq %rax, %r15
+; X64-NEXT:    andq %r15, %rbp
 ; X64-NEXT:    shrq $4, %rbp
 ; X64-NEXT:    orq %r10, %rbp
 ; X64-NEXT:    movq %rbp, %r10
 ; X64-NEXT:    andq %r11, %r10
 ; X64-NEXT:    andq %r14, %rbp
 ; X64-NEXT:    shrq $2, %rbp
 ; X64-NEXT:    leaq (%rbp,%r10,4), %rbp
 ; X64-NEXT:    movq %rbp, %r10

llvm/test/CodeGen/X86/fp128-libcalls-strict.ll

 ; X86:       # %bb.0:
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %esi
 ; X86-NEXT:    subl $12, %esp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
-; X86-NEXT:    pushl %ebx
-; X86-NEXT:    movl %ebx, %esi
+; X86-NEXT:    pushl %esi
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
 ; X86-NEXT:    calll __eqtf2
 ; X86-NEXT:    addl $32, %esp
 ; X86-NEXT:    testl %eax, %eax
 ; X86-NEXT:    sete %bl
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)

 ; X86:       # %bb.0:
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %esi
 ; X86-NEXT:    subl $12, %esp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
-; X86-NEXT:    pushl %ebx
-; X86-NEXT:    movl %ebx, %esi
+; X86-NEXT:    pushl %esi
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)
 ; X86-NEXT:    calll __eqtf2
 ; X86-NEXT:    addl $32, %esp
 ; X86-NEXT:    testl %eax, %eax
 ; X86-NEXT:    setne %bl
 ; X86-NEXT:    pushl {{[0-9]+}}(%esp)

llvm/test/CodeGen/X86/machine-copy-prop.mir

 # RUN: llc -mtriple=i686-- -run-pass machine-cp -verify-machineinstrs -o - %s | FileCheck %s
 
 --- |
   declare void @foo()
   define void @copyprop_remove_kill0() { ret void }
   define void @copyprop_remove_kill1() { ret void }
   define void @copyprop_remove_kill2() { ret void }
+  define void @copyprop_dbg_value() { ret void }
   define void @copyprop0() { ret void }
   define void @copyprop1() { ret void }
   define void @copyprop2() { ret void }
+  define void @copyprop3() { ret void }
   define void @nocopyprop0() { ret void }
   define void @nocopyprop1() { ret void }
   define void @nocopyprop2() { ret void }
   define void @nocopyprop3() { ret void }
   define void @nocopyprop4() { ret void }
   define void @nocopyprop5() { ret void }
+  define void @nocopyprop6() { ret void }
+  define void @nocopyprop7() { ret void }
+  define void @nocopyprop8() { ret void }
 ...
 ---
 # The second copy is redundant and will be removed, check that we also remove
 # the kill flag of intermediate instructions.
 # CHECK-LABEL: name: copyprop_remove_kill0
 # CHECK: bb.0:
 # CHECK-NEXT: $rax = COPY $rdi
 # CHECK-NEXT: NOOP implicit $rdi

 body: |
   bb.0:
     $ax = COPY $di
     NOOP implicit killed $rdi
     $di = COPY $ax
     NOOP implicit $rax, implicit $rdi
 ...
 ---
+# DBG_VALUE shouldn't stop backward copy propagation
+# CHECK-LABEL: name: copyprop_dbg_value
+# CHECK: renamable $rbx = LEA64r $rax, 1, $noreg, 8, $noreg
+# CHECK-NEXT: DBG_VALUE $rbx, $noreg
+# CHECK-NEXT: DBG_VALUE $noreg, $noreg
+name: copyprop_dbg_value
+body: |
+  bb.0:
+    renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
+    DBG_VALUE $rbp, $noreg
+    renamable $rbx = COPY killed renamable $rbp
+    DBG_VALUE $noreg, $noreg
+...
+---
 # The second copy is redundant; the call preserves the source and dest register.
 # CHECK-LABEL: name: copyprop0
 # CHECK: bb.0:
 # CHECK-NEXT: $rax = COPY $rdi
 # CHECK-NEXT: CALL64pcrel32 @foo, csr_64_rt_mostregs
 # CHECK-NEXT: NOOP implicit $edi
 # CHECK-NOT: COPY
 # CHECK-NEXT: NOOP implicit $rax, implicit $rdi
body: |
   bb.0:
     $rax = COPY $rdi
     NOOP implicit killed $ax
     CALL64pcrel32 @foo, csr_64_rt_mostregs
     $rax = COPY $rdi
     NOOP implicit $rax, implicit $rdi
 ...
 ---
+# CHECK-LABEL: name: copyprop3
+# CHECK: renamable $rbx = LEA64r $rax, 1, $noreg, 8, $noreg
+# CHECK-NEXT: renamable $rax = COPY renamable $rbx
+# CHECK-NEXT: NOOP implicit $rax
+name: copyprop3
+body: |
+  bb.0:
+    renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
+    renamable $rax = COPY renamable $rbp
+    renamable $rbx = COPY killed renamable $rbp
+    NOOP implicit $rax
+...
+---
 # The second copy is not redundant if the source register ($rax) is clobbered
 # even if the dest ($rbp) is not.
 # CHECK-LABEL: name: nocopyprop0
 # CHECK: bb.0:
 # CHECK-NEXT: $rax = COPY $rbp
 # CHECK-NEXT: CALL64pcrel32 @foo, csr_64, implicit $rax, implicit $rbp
 # CHECK-NEXT: $rbp = COPY $rax
 # CHECK-NEXT: NOOP implicit $rax, implicit $rbp

 # CHECK-NEXT: $rip = COPY $rax
 # CHECK-NEXT: $rip = COPY $rax
 name: nocopyprop5
 body: |
   bb.0:
     $rip = COPY $rax
     $rip = COPY $rax
 ...
+---
+# Use of a sub-register should stop backward propagation
+# CHECK-LABEL: name: nocopyprop6
+# CHECK: renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
+# CHECK-NEXT: renamable $rax = COPY renamable $ebp
+# CHECK-NEXT: renamable $rbx = COPY killed renamable $rbp
+# CHECK-NEXT: NOOP implicit $rax, implicit $rbx
+name: nocopyprop6
+body: |
+  bb.0:
+    renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
+    renamable $rax = COPY renamable $ebp
+    renamable $rbx = COPY killed renamable $rbp
+    NOOP implicit $rax, implicit $rbx
+...
+---
+# Non-renamable use of a register should stop backward propagation
+# CHECK-LABEL: name: nocopyprop7
+# CHECK: renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
+# CHECK-NEXT: renamable $rax = COPY $rbp
+# CHECK-NEXT: renamable $rbx = COPY killed renamable $rbp
+# CHECK-NEXT: NOOP implicit $rax, implicit $rbx
+name: nocopyprop7
+body: |
+  bb.0:
+    renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
+    renamable $rax = COPY $rbp
+    renamable $rbx = COPY killed renamable $rbp
+    NOOP implicit $rax, implicit $rbx
+...
+---
+# Read of the Def register should stop backward propagation
+# CHECK-LABEL: name: nocopyprop8
+# CHECK: renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
+# CHECK-NEXT: renamable $rax = COPY $rbx
+# CHECK-NEXT: renamable $rbx = COPY killed renamable $rbp
+# CHECK-NEXT: NOOP implicit $rax, implicit $rbx
+name: nocopyprop8
+body: |
+  bb.0:
+    renamable $rbp = LEA64r $rax, 1, $noreg, 8, $noreg
+    renamable $rax = COPY $rbx
+    renamable $rbx = COPY killed renamable $rbp
+    NOOP implicit $rax, implicit $rbx
+...

llvm/test/CodeGen/X86/mul-i1024.ll

 ; X32-NEXT:    mull %ebp
 ; X32-NEXT:    addl %ecx, %eax
 ; X32-NEXT:    movzbl %bl, %ecx
 ; X32-NEXT:    adcl %ecx, %edx
 ; X32-NEXT:    addl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X32-NEXT:    adcl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X32-NEXT:    movl 124(%edi), %ecx
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X32-NEXT:    movl 124(%ebx), %ecx
 ; X32-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
 ; X32-NEXT:    imull %eax, %ecx
-; X32-NEXT:    movl 120(%edi), %esi
-; X32-NEXT:    movl %edi, %ebx
+; X32-NEXT:    movl 120(%ebx), %esi
 ; X32-NEXT:    mull %esi
 ; X32-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X32-NEXT:    addl %ecx, %edx
 ; X32-NEXT:    imull (%esp), %esi # 4-byte Folded Reload
 ; X32-NEXT:    addl %edx, %esi
-; X32-NEXT:    movl 112(%edi), %edi
+; X32-NEXT:    movl 112(%ebx), %edi
 ; X32-NEXT:    movl 116(%ebx), %ebp
 ; X32-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
 ; X32-NEXT:    movl %eax, %ebx
 ; X32-NEXT:    imull %ebp, %ebx
 ; X32-NEXT:    movl %ebp, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X32-NEXT:    mull %edi
 ; X32-NEXT:    addl %ebx, %edx
 ; X32-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload

 ; X64-NEXT:    movq %rbp, %rax
 ; X64-NEXT:    movq %rbp, %r11
 ; X64-NEXT:    movq %rbp, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    mulq %r10
 ; X64-NEXT:    movq %rdx, %rbx
 ; X64-NEXT:    movq %rax, %rsi
 ; X64-NEXT:    addq %r12, %rsi
 ; X64-NEXT:    adcq $0, %rbx
-; X64-NEXT:    movq 24(%rdi), %rdi
+; X64-NEXT:    movq 24(%rdi), %r12
 ; X64-NEXT:    movq %r9, %rax
-; X64-NEXT:    mulq %rdi
-; X64-NEXT:    movq %rdi, %r12
+; X64-NEXT:    mulq %r12
 ; X64-NEXT:    movq %rdx, %rbp
 ; X64-NEXT:    addq %rsi, %rax
 ; X64-NEXT:    movq %rax, %rsi
 ; X64-NEXT:    adcq %rbx, %rbp
 ; X64-NEXT:    setb %r9b
 ; X64-NEXT:    movq %r11, %rax
-; X64-NEXT:    mulq %rdi
+; X64-NEXT:    mulq %r12
 ; X64-NEXT:    movq %rdx, %rdi
 ; X64-NEXT:    movq %rax, %rbx
 ; X64-NEXT:    addq %rbp, %rbx
 ; X64-NEXT:    movzbl %r9b, %eax
 ; X64-NEXT:    adcq %rax, %rdi
 ; X64-NEXT:    addq %rcx, %r13
 ; X64-NEXT:    movq %r13, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq %r15, %rsi

 ; X64-NEXT:    movq %r14, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq %r13, %rdi
 ; X64-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq $0, %rbx
 ; X64-NEXT:    adcq $0, %r8
 ; X64-NEXT:    addq %r12, %rbx
 ; X64-NEXT:    adcq %r11, %r8
 ; X64-NEXT:    setb %r14b
-; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
-; X64-NEXT:    movq %rsi, %rax
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r15 # 8-byte Reload
+; X64-NEXT:    movq %r15, %rax
 ; X64-NEXT:    movq %r10, (%rsp) # 8-byte Spill
 ; X64-NEXT:    mulq %r10
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, %r11
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
 ; X64-NEXT:    movq %r9, %rax
 ; X64-NEXT:    mulq %r10
 ; X64-NEXT:    movq %rdx, %rdi
 ; X64-NEXT:    movq %rax, %rbp
 ; X64-NEXT:    addq %rcx, %rbp
 ; X64-NEXT:    adcq $0, %rdi
-; X64-NEXT:    movq %rsi, %rax
-; X64-NEXT:    movq %rsi, %r15
+; X64-NEXT:    movq %r15, %rax
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
 ; X64-NEXT:    mulq %rsi
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, %r10
 ; X64-NEXT:    addq %rbp, %r10
 ; X64-NEXT:    adcq %rdi, %rcx
 ; X64-NEXT:    setb %dil
 ; X64-NEXT:    movq %r9, %rax

 ; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Folded Reload
 ; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Folded Reload
 ; X64-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq $0, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Folded Spill
 ; X64-NEXT:    adcq $0, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Folded Spill
 ; X64-NEXT:    adcq $0, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Folded Spill
 ; X64-NEXT:    adcq $0, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Folded Spill
-; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
-; X64-NEXT:    movq 32(%rsi), %rdi
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r13 # 8-byte Reload
+; X64-NEXT:    movq 32(%r13), %r11
 ; X64-NEXT:    movq %r15, %rax
-; X64-NEXT:    mulq %rdi
+; X64-NEXT:    mulq %r11
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    movq %r9, %rbx
 ; X64-NEXT:    movq %r9, %rax
-; X64-NEXT:    mulq %rdi
-; X64-NEXT:    movq %rdi, %r11
+; X64-NEXT:    mulq %r11
 ; X64-NEXT:    movq %rdx, %rdi
 ; X64-NEXT:    movq %rax, %rbp
 ; X64-NEXT:    addq %rcx, %rbp
 ; X64-NEXT:    adcq $0, %rdi
-; X64-NEXT:    movq 40(%rsi), %rcx
-; X64-NEXT:    movq %rsi, %r13
+; X64-NEXT:    movq 40(%r13), %r8
 ; X64-NEXT:    movq %r15, %rax
-; X64-NEXT:    mulq %rcx
-; X64-NEXT:    movq %rcx, %r8
+; X64-NEXT:    mulq %r8
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, %r9
 ; X64-NEXT:    addq %rbp, %r9
 ; X64-NEXT:    adcq %rdi, %rcx
 ; X64-NEXT:    setb %sil
 ; X64-NEXT:    movq %rbx, %rax
 ; X64-NEXT:    mulq %r8
 ; X64-NEXT:    movq %rdx, %r10

 ; X64-NEXT:    movq %r8, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Folded Reload
 ; X64-NEXT:    movq %r9, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r14 # 8-byte Folded Reload
 ; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r10 # 8-byte Folded Reload
 ; X64-NEXT:    setb {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Folded Spill
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
 ; X64-NEXT:    movq %rdi, %rax
-; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
-; X64-NEXT:    mulq %rsi
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; X64-NEXT:    mulq %r8
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbp # 8-byte Reload
 ; X64-NEXT:    movq %rbp, %rax
-; X64-NEXT:    mulq %rsi
-; X64-NEXT:    movq %rsi, %r8
+; X64-NEXT:    mulq %r8
 ; X64-NEXT:    movq %rdx, %rsi
 ; X64-NEXT:    movq %rax, %rbx
 ; X64-NEXT:    addq %rcx, %rbx
 ; X64-NEXT:    adcq $0, %rsi
 ; X64-NEXT:    movq %rdi, %rax
-; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
-; X64-NEXT:    mulq %rdi
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; X64-NEXT:    mulq %r9
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, %r12
 ; X64-NEXT:    addq %rbx, %r12
 ; X64-NEXT:    adcq %rsi, %rcx
 ; X64-NEXT:    setb %bl
 ; X64-NEXT:    movq %rbp, %rax
-; X64-NEXT:    mulq %rdi
-; X64-NEXT:    movq %rdi, %r9
+; X64-NEXT:    mulq %r9
 ; X64-NEXT:    movq %rdx, %r11
 ; X64-NEXT:    movq %rax, %r13
 ; X64-NEXT:    addq %rcx, %r13
 ; X64-NEXT:    movzbl %bl, %eax
 ; X64-NEXT:    adcq %rax, %r11
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbp # 8-byte Reload
 ; X64-NEXT:    movq %rbp, %rax
 ; X64-NEXT:    mulq %r8

 ; X64-NEXT:    adcq %rax, %r13
 ; X64-NEXT:    movq %r13, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq $0, %r15
 ; X64-NEXT:    movq %r15, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq $0, %rcx
 ; X64-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq $0, %rdx
 ; X64-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
-; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
-; X64-NEXT:    movq 64(%rsi), %rdi
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r13 # 8-byte Reload
+; X64-NEXT:    movq 64(%r13), %r8
 ; X64-NEXT:    movq (%rsp), %rbx # 8-byte Reload
 ; X64-NEXT:    movq %rbx, %rax
-; X64-NEXT:    mulq %rdi
+; X64-NEXT:    mulq %r8
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, %r11
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
 ; X64-NEXT:    movq %r9, %rax
-; X64-NEXT:    mulq %rdi
-; X64-NEXT:    movq %rdi, %r8
+; X64-NEXT:    mulq %r8
 ; X64-NEXT:    movq %rdx, %rdi
 ; X64-NEXT:    movq %rax, %rbp
 ; X64-NEXT:    addq %rcx, %rbp
 ; X64-NEXT:    adcq $0, %rdi
-; X64-NEXT:    movq 72(%rsi), %rcx
-; X64-NEXT:    movq %rsi, %r13
+; X64-NEXT:    movq 72(%r13), %rsi
 ; X64-NEXT:    movq %rbx, %rax
-; X64-NEXT:    mulq %rcx
-; X64-NEXT:    movq %rcx, %rsi
+; X64-NEXT:    mulq %rsi
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, %r14
 ; X64-NEXT:    addq %rbp, %r14
 ; X64-NEXT:    adcq %rdi, %rcx
 ; X64-NEXT:    setb %bl
 ; X64-NEXT:    movq %r9, %rax
 ; X64-NEXT:    mulq %rsi
 ; X64-NEXT:    movq %rsi, %r12

 ; X64-NEXT:    movq %r14, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq %r12, %rbp
 ; X64-NEXT:    movq %rbp, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq $0, %rsi
 ; X64-NEXT:    adcq $0, %rbx
 ; X64-NEXT:    addq %r15, %rsi
 ; X64-NEXT:    adcq %r10, %rbx
 ; X64-NEXT:    setb %r9b
-; X64-NEXT:    movq (%rsp), %r14 # 8-byte Reload
-; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    movq (%rsp), %r15 # 8-byte Reload
+; X64-NEXT:    movq %r15, %rax
 ; X64-NEXT:    mulq %r11
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    movq %rax, %r10
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
 ; X64-NEXT:    movq %r8, %rax
 ; X64-NEXT:    mulq %r11
 ; X64-NEXT:    movq %rdx, %rdi
 ; X64-NEXT:    movq %rax, %rbp
 ; X64-NEXT:    addq %rcx, %rbp
 ; X64-NEXT:    adcq $0, %rdi
-; X64-NEXT:    movq %r14, %rax
-; X64-NEXT:    movq %r14, %r15
+; X64-NEXT:    movq %r15, %rax
 ; X64-NEXT:    mulq %r13
 ; X64-NEXT:    movq %rdx, %rcx
 ; X64-NEXT:    addq %rbp, %rax
 ; X64-NEXT:    movq %rax, %rbp
 ; X64-NEXT:    adcq %rdi, %rcx
 ; X64-NEXT:    setb %dil
 ; X64-NEXT:    movq %r8, %rax
 ; X64-NEXT:    mulq %r13

 ; X64-NEXT:    mulq %rdi
 ; X64-NEXT:    movq %rdx, %r13
 ; X64-NEXT:    movq %rax, %r11
 ; X64-NEXT:    addq %rbp, %r11
 ; X64-NEXT:    movzbl %cl, %eax
 ; X64-NEXT:    adcq %rax, %r13
 ; X64-NEXT:    addq %r9, %r11
 ; X64-NEXT:    adcq %r8, %r13
-; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
-; X64-NEXT:    movq 120(%rdx), %rcx
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
+; X64-NEXT:    movq 120(%rdi), %rcx
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
 ; X64-NEXT:    imulq %rax, %rcx
-; X64-NEXT:    movq 112(%rdx), %rsi
-; X64-NEXT:    movq %rdx, %rdi
+; X64-NEXT:    movq 112(%rdi), %rsi
 ; X64-NEXT:    movq %rax, %rbx
 ; X64-NEXT:    mulq %rsi
 ; X64-NEXT:    movq %rax, %r10
 ; X64-NEXT:    addq %rcx, %rdx
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
 ; X64-NEXT:    imulq %r8, %rsi
 ; X64-NEXT:    addq %rdx, %rsi
 ; X64-NEXT:    movq 96(%rdi), %rcx

 ; X64-NEXT:    movq %r12, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq %r15, %rcx
 ; X64-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    movzbl %r8b, %ecx
 ; X64-NEXT:    adcq %rcx, %rax
 ; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; X64-NEXT:    adcq $0, %rdx
 ; X64-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
-; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbp # 8-byte Reload
-; X64-NEXT:    movq 96(%rbp), %rsi
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
+; X64-NEXT:    movq 96(%rdi), %rsi
 ; X64-NEXT:    imulq %rsi, %r9
 ; X64-NEXT:    movq %rsi, %rax
 ; X64-NEXT:    mulq %rbx
 ; X64-NEXT:    movq %rax, %r14
 ; X64-NEXT:    addq %r9, %rdx
-; X64-NEXT:    movq 104(%rbp), %r15
+; X64-NEXT:    movq 104(%rdi), %r15
 ; X64-NEXT:    imulq %r15, %rbx
 ; X64-NEXT:    addq %rdx, %rbx
 ; X64-NEXT:    movq %rbx, %r9
-; X64-NEXT:    movq 112(%rbp), %rax
-; X64-NEXT:    movq %rbp, %rdi
+; X64-NEXT:    movq 112(%rdi), %rax
 ; X64-NEXT:    movq %rax, %rcx
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbx # 8-byte Reload
 ; X64-NEXT:    imulq %rbx, %rcx
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbp # 8-byte Reload
 ; X64-NEXT:    mulq %rbp
 ; X64-NEXT:    movq %rax, %r13
 ; X64-NEXT:    addq %rcx, %rdx
 ; X64-NEXT:    movq 120(%rdi), %r8

llvm/test/CodeGen/X86/mul-i512.ll

 ; X32-NEXT:    mull %ebp
 ; X32-NEXT:    addl %ecx, %eax
 ; X32-NEXT:    movzbl %bl, %ecx
 ; X32-NEXT:    adcl %ecx, %edx
 ; X32-NEXT:    addl (%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X32-NEXT:    adcl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X32-NEXT:    movl 60(%edi), %ecx
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X32-NEXT:    movl 60(%ebx), %ecx
 ; X32-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
 ; X32-NEXT:    imull %eax, %ecx
-; X32-NEXT:    movl 56(%edi), %esi
-; X32-NEXT:    movl %edi, %ebx
+; X32-NEXT:    movl 56(%ebx), %esi
 ; X32-NEXT:    mull %esi
 ; X32-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X32-NEXT:    addl %ecx, %edx
 ; X32-NEXT:    imull {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Folded Reload
 ; X32-NEXT:    addl %edx, %esi
-; X32-NEXT:    movl 48(%edi), %edi
+; X32-NEXT:    movl 48(%ebx), %edi
 ; X32-NEXT:    movl 52(%ebx), %ebp
 ; X32-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
 ; X32-NEXT:    movl %eax, %ebx
 ; X32-NEXT:    imull %ebp, %ebx
 ; X32-NEXT:    movl %ebp, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X32-NEXT:    mull %edi
 ; X32-NEXT:    addl %ebx, %edx
 ; X32-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload

 ; X64-NEXT:    mulq %r9
 ; X64-NEXT:    movq %rdx, %r15
 ; X64-NEXT:    movq %rax, %r13
 ; X64-NEXT:    addq %rdi, %r13
 ; X64-NEXT:    movzbl %bl, %eax
 ; X64-NEXT:    adcq %rax, %r15
 ; X64-NEXT:    addq %r12, %r13
 ; X64-NEXT:    adcq %rbp, %r15
-; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
-; X64-NEXT:    movq 56(%rdx), %rcx
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; X64-NEXT:    movq 56(%rsi), %rcx
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
 ; X64-NEXT:    imulq %rax, %rcx
-; X64-NEXT:    movq 48(%rdx), %rbp
-; X64-NEXT:    movq %rdx, %rsi
+; X64-NEXT:    movq 48(%rsi), %rbp
 ; X64-NEXT:    movq %rax, %r11
 ; X64-NEXT:    mulq %rbp
 ; X64-NEXT:    movq %rax, %r12
 ; X64-NEXT:    addq %rcx, %rdx
 ; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
 ; X64-NEXT:    imulq %r8, %rbp
 ; X64-NEXT:    addq %rdx, %rbp
 ; X64-NEXT:    movq 32(%rsi), %rdi

llvm/test/CodeGen/X86/pr43820.ll

 ; CHECK-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %r10
 ; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %rbx
 ; CHECK-NEXT:    bswapq %rbx
 ; CHECK-NEXT:    movabsq $1085102592571150095, %rdi # imm = 0xF0F0F0F0F0F0F0F
 ; CHECK-NEXT:    movq %rbx, %rbp
 ; CHECK-NEXT:    andq %rdi, %rbp
 ; CHECK-NEXT:    shlq $4, %rbp
-; CHECK-NEXT:    movabsq $-1085102592571150096, %r11 # imm = 0xF0F0F0F0F0F0F0F0
-; CHECK-NEXT:    andq %r11, %rbx
-; CHECK-NEXT:    movq %r11, %rax
+; CHECK-NEXT:    movabsq $-1085102592571150096, %rax # imm = 0xF0F0F0F0F0F0F0F0
+; CHECK-NEXT:    andq %rax, %rbx
 ; CHECK-NEXT:    shrq $4, %rbx
 ; CHECK-NEXT:    orq %rbp, %rbx
 ; CHECK-NEXT:    movabsq $3689348814741910323, %r11 # imm = 0x3333333333333333
 ; CHECK-NEXT:    movq %rbx, %r14
 ; CHECK-NEXT:    andq %r11, %r14
-; CHECK-NEXT:    movabsq $-3689348814741910324, %rbp # imm = 0xCCCCCCCCCCCCCCCC
-; CHECK-NEXT:    andq %rbp, %rbx
-; CHECK-NEXT:    movq %rbp, %r15
+; CHECK-NEXT:    movabsq $-3689348814741910324, %r15 # imm = 0xCCCCCCCCCCCCCCCC
+; CHECK-NEXT:    andq %r15, %rbx
 ; CHECK-NEXT:    shrq $2, %rbx
 ; CHECK-NEXT:    leaq (%rbx,%r14,4), %r14
 ; CHECK-NEXT:    movabsq $6148914691230924800, %rbx # imm = 0x5555555555000000
 ; CHECK-NEXT:    andq %r14, %rbx
 ; CHECK-NEXT:    movabsq $-6148914691247702016, %rbp # imm = 0xAAAAAAAAAA000000
 ; CHECK-NEXT:    andq %r14, %rbp
 ; CHECK-NEXT:    shrq %rbp
 ; CHECK-NEXT:    leaq (%rbp,%rbx,2), %rbx

llvm/test/CodeGen/X86/sdiv_fix.ll

 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %eax
 ; X86-NEXT:    pushl %ecx
 ; X86-NEXT:    leal {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    pushl %eax
 ; X86-NEXT:    calll __divti3
 ; X86-NEXT:    addl $32, %esp
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X86-NEXT:    movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:    subl $1, %eax
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    sbbl $0, %ebx
 ; X86-NEXT:    testl %esi, %esi
 ; X86-NEXT:    sets %al
 ; X86-NEXT:    testl %edi, %edi
 ; X86-NEXT:    sets %cl
 ; X86-NEXT:    xorb %al, %cl

 ; X86-NEXT:    movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    pushl %esi
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    calll __divdi3
 ; X86-NEXT:    addl $16, %esp
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    movl %eax, %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    movl %esi, %ecx
 ; X86-NEXT:    sarl $31, %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movl %edx, %ebx
 ; X86-NEXT:    shll $31, %ebx
 ; X86-NEXT:    movl %edx, %edi
 ; X86-NEXT:    shrl $31, %edi
 ; X86-NEXT:    shldl $31, %edx, %edi
 ; X86-NEXT:    pushl %ecx
 ; X86-NEXT:    movl %ecx, %ebp
 ; X86-NEXT:    movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT:    pushl %eax
-; X86-NEXT:    movl %eax, %esi
+; X86-NEXT:    pushl %esi
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    calll __moddi3
 ; X86-NEXT:    addl $16, %esp
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    pushl %esi

llvm/test/CodeGen/X86/sdiv_fix_sat.ll

 ; X86-NEXT:    pushl %ecx
 ; X86-NEXT:    pushl %eax
 ; X86-NEXT:    calll __divti3
 ; X86-NEXT:    addl $32, %esp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    subl $1, %esi
-; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:    sbbl $0, %eax
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    movl %edx, %eax
 ; X86-NEXT:    sbbl $0, %eax
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X86-NEXT:    movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    sbbl $0, %ebx

 ; X86-NEXT:    calll __divti3
 ; X86-NEXT:    addl $32, %esp
 ; X86-NEXT:    leal {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
 ; X86-NEXT:    pushl %ecx
 ; X86-NEXT:    pushl %ecx
 ; X86-NEXT:    pushl %ecx
 ; X86-NEXT:    pushl 36(%ebp)
-; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
-; X86-NEXT:    pushl %ecx
-; X86-NEXT:    pushl %ecx
-; X86-NEXT:    movl %ecx, %edi
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
+; X86-NEXT:    pushl %edi
+; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Reload
 ; X86-NEXT:    pushl {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Reload
 ; X86-NEXT:    pushl %eax
 ; X86-NEXT:    calll __divti3
 ; X86-NEXT:    addl $32, %esp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    subl $1, %eax

llvm/test/CodeGen/X86/shift-and.ll

 ; rdar://11866926
 define i64 @t6(i64 %key, i64* nocapture %val) nounwind {
 ; X32-LABEL: t6:
 ; X32:       # %bb.0:
 ; X32-NEXT:    pushl %edi
 ; X32-NEXT:    pushl %esi
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    shrdl $3, %eax, %esi
-; X32-NEXT:    movl %eax, %edi
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; X32-NEXT:    shrdl $3, %edi, %esi
 ; X32-NEXT:    shrl $3, %edi
 ; X32-NEXT:    movl (%ecx), %eax
 ; X32-NEXT:    movl 4(%ecx), %edx
 ; X32-NEXT:    addl $-1, %eax
 ; X32-NEXT:    adcl $-1, %edx
 ; X32-NEXT:    andl %esi, %eax
 ; X32-NEXT:    andl %edi, %edx
 ; X32-NEXT:    popl %esi

llvm/test/CodeGen/X86/shrink_vmul.ll

 ; X86-SSE-NEXT:    pushl %ebx
 ; X86-SSE-NEXT:    pushl %edi
 ; X86-SSE-NEXT:    pushl %esi
 ; X86-SSE-NEXT:    pushl %eax
 ; X86-SSE-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-SSE-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-SSE-NEXT:    movzwl 16(%eax), %edx
 ; X86-SSE-NEXT:    movl %edx, (%esp) # 4-byte Spill
-; X86-SSE-NEXT:    movdqa (%eax), %xmm3
+; X86-SSE-NEXT:    movdqa (%eax), %xmm5
 ; X86-SSE-NEXT:    movdqa (%ecx), %xmm0
 ; X86-SSE-NEXT:    movdqa 16(%ecx), %xmm1
 ; X86-SSE-NEXT:    pxor %xmm4, %xmm4
-; X86-SSE-NEXT:    movdqa %xmm3, %xmm2
-; X86-SSE-NEXT:    pextrw $7, %xmm3, %eax
-; X86-SSE-NEXT:    pextrw $4, %xmm3, %esi
-; X86-SSE-NEXT:    pextrw $0, %xmm3, %edi
-; X86-SSE-NEXT:    pextrw $1, %xmm3, %ebx
-; X86-SSE-NEXT:    pextrw $3, %xmm3, %ebp
-; X86-SSE-NEXT:    movdqa %xmm3, %xmm5
+; X86-SSE-NEXT:    movdqa %xmm5, %xmm2
+; X86-SSE-NEXT:    pextrw $7, %xmm5, %eax
+; X86-SSE-NEXT:    pextrw $4, %xmm5, %esi
+; X86-SSE-NEXT:    pextrw $0, %xmm5, %edi
+; X86-SSE-NEXT:    pextrw $1, %xmm5, %ebx
+; X86-SSE-NEXT:    pextrw $3, %xmm5, %ebp
 ; X86-SSE-NEXT:    punpcklwd {{.*#+}} xmm5 = xmm5[0],xmm4[0],xmm5[1],xmm4[1],xmm5[2],xmm4[2],xmm5[3],xmm4[3]
 ; X86-SSE-NEXT:    punpckhwd {{.*#+}} xmm2 = xmm2[4],xmm4[4],xmm2[5],xmm4[5],xmm2[6],xmm4[6],xmm2[7],xmm4[7]
 ; X86-SSE-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[3,3,3,3]
 ; X86-SSE-NEXT:    movd %xmm3, %ecx
 ; X86-SSE-NEXT:    xorl %edx, %edx
 ; X86-SSE-NEXT:    divl %ecx
 ; X86-SSE-NEXT:    movd %edx, %xmm3
 ; X86-SSE-NEXT:    pshufd {{.*#+}} xmm4 = xmm2[2,3,2,3]

 ; X86-AVX2-NEXT:    popl %esi
 ; X86-AVX2-NEXT:    popl %edi
 ; X86-AVX2-NEXT:    vzeroupper
 ; X86-AVX2-NEXT:    retl
 ;
 ; X64-SSE-LABEL: PR34947:
 ; X64-SSE:       # %bb.0:
 ; X64-SSE-NEXT:    movzwl 16(%rdi), %r8d
-; X64-SSE-NEXT:    movdqa (%rdi), %xmm3
+; X64-SSE-NEXT:    movdqa (%rdi), %xmm5
 ; X64-SSE-NEXT:    movdqa (%rsi), %xmm0
 ; X64-SSE-NEXT:    movdqa 16(%rsi), %xmm1
 ; X64-SSE-NEXT:    pxor %xmm4, %xmm4
-; X64-SSE-NEXT:    movdqa %xmm3, %xmm2
-; X64-SSE-NEXT:    pextrw $7, %xmm3, %eax
-; X64-SSE-NEXT:    pextrw $4, %xmm3, %r9d
-; X64-SSE-NEXT:    pextrw $0, %xmm3, %r10d
-; X64-SSE-NEXT:    pextrw $1, %xmm3, %r11d
-; X64-SSE-NEXT:    pextrw $3, %xmm3, %ecx
-; X64-SSE-NEXT:    movdqa %xmm3, %xmm5
+; X64-SSE-NEXT:    movdqa %xmm5, %xmm2
+; X64-SSE-NEXT:    pextrw $7, %xmm5, %eax
+; X64-SSE-NEXT:    pextrw $4, %xmm5, %r9d
+; X64-SSE-NEXT:    pextrw $0, %xmm5, %r10d
+; X64-SSE-NEXT:    pextrw $1, %xmm5, %r11d
+; X64-SSE-NEXT:    pextrw $3, %xmm5, %ecx
 ; X64-SSE-NEXT:    punpcklwd {{.*#+}} xmm5 = xmm5[0],xmm4[0],xmm5[1],xmm4[1],xmm5[2],xmm4[2],xmm5[3],xmm4[3]
 ; X64-SSE-NEXT:    punpckhwd {{.*#+}} xmm2 = xmm2[4],xmm4[4],xmm2[5],xmm4[5],xmm2[6],xmm4[6],xmm2[7],xmm4[7]
 ; X64-SSE-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[3,3,3,3]
 ; X64-SSE-NEXT:    movd %xmm3, %edi
 ; X64-SSE-NEXT:    xorl %edx, %edx
 ; X64-SSE-NEXT:    divl %edi
 ; X64-SSE-NEXT:    movd %edx, %xmm3
 ; X64-SSE-NEXT:    pshufd {{.*#+}} xmm4 = xmm2[2,3,2,3]

llvm/test/CodeGen/X86/sqrt-fastmath.ll

; AVX512-NEXT:    retq
   %sqrt = tail call <16 x float> @llvm.sqrt.v16f32(<16 x float> %x)
   %div = fdiv fast <16 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %sqrt
   ret <16 x float> %div
 }
 
 define <16 x float> @v16f32_estimate(<16 x float> %x) #1 {
 ; SSE-LABEL: v16f32_estimate:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    rsqrtps %xmm0, %xmm5
+; SSE-NEXT:    rsqrtps %xmm0, %xmm6
 ; SSE-NEXT:    movaps {{.*#+}} xmm4 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; SSE-NEXT:    mulps %xmm5, %xmm0
-; SSE-NEXT:    mulps %xmm5, %xmm0
-; SSE-NEXT:    movaps %xmm5, %xmm6
+; SSE-NEXT:    mulps %xmm6, %xmm0
+; SSE-NEXT:    mulps %xmm6, %xmm0
 ; SSE-NEXT:    mulps %xmm4, %xmm6
 ; SSE-NEXT:    movaps {{.*#+}} xmm5 = [-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0]
 ; SSE-NEXT:    addps %xmm5, %xmm0
 ; SSE-NEXT:    mulps %xmm6, %xmm0
 ; SSE-NEXT:    rsqrtps %xmm1, %xmm6
 ; SSE-NEXT:    mulps %xmm6, %xmm1
 ; SSE-NEXT:    mulps %xmm6, %xmm1
 ; SSE-NEXT:    mulps %xmm4, %xmm6

llvm/test/CodeGen/X86/umulo-128-legalisation-lowering.ll

 ; X86-NEXT:    seto {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
 ; X86-NEXT:    addl %ecx, %esi
 ; X86-NEXT:    movl %edi, %eax
 ; X86-NEXT:    mull %ebx
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    movl %edx, %ecx
 ; X86-NEXT:    addl %esi, %ecx
 ; X86-NEXT:    setb {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    mull %esi
-; X86-NEXT:    movl %esi, %ebx
+; X86-NEXT:    mull %ebx
 ; X86-NEXT:    movl %eax, %esi
 ; X86-NEXT:    seto {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X86-NEXT:    movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    mull %edi
 ; X86-NEXT:    movl %eax, %ebp
 ; X86-NEXT:    seto {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill

llvm/test/CodeGen/X86/vector-interleaved-load-i16-stride-4.ll

 ; AVX2-SLOW-LABEL: vf32:
 ; AVX2-SLOW:       # %bb.0:
 ; AVX2-SLOW-NEXT:    subq $248, %rsp
 ; AVX2-SLOW-NEXT:    vmovdqa (%rdi), %xmm5
 ; AVX2-SLOW-NEXT:    vmovdqa %xmm5, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-SLOW-NEXT:    vmovdqa 16(%rdi), %xmm9
 ; AVX2-SLOW-NEXT:    vmovdqa 32(%rdi), %xmm4
 ; AVX2-SLOW-NEXT:    vmovdqa %xmm4, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; AVX2-SLOW-NEXT:    vmovdqa 48(%rdi), %xmm3
+; AVX2-SLOW-NEXT:    vmovdqa 48(%rdi), %xmm14
 ; AVX2-SLOW-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm2 = xmm3[0],xmm0[1,2,3],xmm3[4],xmm0[5,6,7]
-; AVX2-SLOW-NEXT:    vmovdqa %xmm3, %xmm14
-; AVX2-SLOW-NEXT:    vmovdqa %xmm3, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm2 = xmm14[0],xmm0[1,2,3],xmm14[4],xmm0[5,6,7]
+; AVX2-SLOW-NEXT:    vmovdqa %xmm14, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm3 = xmm4[0],xmm0[1,2,3],xmm4[4],xmm0[5,6,7]
 ; AVX2-SLOW-NEXT:    vpackusdw %xmm2, %xmm3, %xmm2
 ; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm3 = xmm9[0],xmm0[1,2,3],xmm9[4],xmm0[5,6,7]
 ; AVX2-SLOW-NEXT:    vmovdqa %xmm9, (%rsp) # 16-byte Spill
 ; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm4 = xmm5[0],xmm0[1,2,3],xmm5[4],xmm0[5,6,7]
 ; AVX2-SLOW-NEXT:    vpackusdw %xmm3, %xmm4, %xmm3
 ; AVX2-SLOW-NEXT:    vpackusdw %xmm2, %xmm3, %xmm2
 ; AVX2-SLOW-NEXT:    vmovdqa 112(%rdi), %xmm13

 ; AVX2-SLOW-NEXT:    vmovdqu %ymm1, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
 ; AVX2-SLOW-NEXT:    vmovdqa 176(%rdi), %xmm1
 ; AVX2-SLOW-NEXT:    vmovdqa %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm1 = xmm1[0],xmm0[1,2,3],xmm1[4],xmm0[5,6,7]
 ; AVX2-SLOW-NEXT:    vmovdqa 160(%rdi), %xmm2
 ; AVX2-SLOW-NEXT:    vmovdqa %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm2 = xmm2[0],xmm0[1,2,3],xmm2[4],xmm0[5,6,7]
 ; AVX2-SLOW-NEXT:    vpackusdw %xmm1, %xmm2, %xmm1
-; AVX2-SLOW-NEXT:    vmovdqa 144(%rdi), %xmm3
-; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm2 = xmm3[0],xmm0[1,2,3],xmm3[4],xmm0[5,6,7]
-; AVX2-SLOW-NEXT:    vmovdqa %xmm3, %xmm15
-; AVX2-SLOW-NEXT:    vmovdqa %xmm3, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; AVX2-SLOW-NEXT:    vmovdqa 144(%rdi), %xmm15
+; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm2 = xmm15[0],xmm0[1,2,3],xmm15[4],xmm0[5,6,7]
+; AVX2-SLOW-NEXT:    vmovdqa %xmm15, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-SLOW-NEXT:    vmovdqa 128(%rdi), %xmm3
 ; AVX2-SLOW-NEXT:    vmovdqa %xmm3, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm3 = xmm3[0],xmm0[1,2,3],xmm3[4],xmm0[5,6,7]
 ; AVX2-SLOW-NEXT:    vpackusdw %xmm2, %xmm3, %xmm2
 ; AVX2-SLOW-NEXT:    vpackusdw %xmm1, %xmm2, %xmm4
 ; AVX2-SLOW-NEXT:    vmovdqa 240(%rdi), %xmm10
 ; AVX2-SLOW-NEXT:    vpblendw {{.*#+}} xmm2 = xmm10[0],xmm0[1,2,3],xmm10[4],xmm0[5,6,7]
 ; AVX2-SLOW-NEXT:    vmovdqa 224(%rdi), %xmm11

 ; AVX2-FAST-ALL-NEXT:    vpblendw {{.*#+}} xmm1 = xmm11[0],xmm8[1,2,3],xmm11[4],xmm8[5,6,7]
 ; AVX2-FAST-ALL-NEXT:    vmovdqa %xmm11, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-FAST-ALL-NEXT:    vpblendw {{.*#+}} xmm4 = xmm2[0],xmm8[1,2,3],xmm2[4],xmm8[5,6,7]
 ; AVX2-FAST-ALL-NEXT:    vpackusdw %xmm1, %xmm4, %xmm1
 ; AVX2-FAST-ALL-NEXT:    vpackusdw %xmm0, %xmm1, %xmm1
 ; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} ymm15 = [0,2,2,3,0,2,4,6]
 ; AVX2-FAST-ALL-NEXT:    vpermd %ymm6, %ymm15, %ymm2
 ; AVX2-FAST-ALL-NEXT:    vmovdqu %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
-; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} ymm0 = <u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,16,17,20,21,24,25,28,29>
-; AVX2-FAST-ALL-NEXT:    vpshufb %ymm0, %ymm2, %ymm4
-; AVX2-FAST-ALL-NEXT:    vmovdqa %ymm0, %ymm3
+; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} ymm3 = <u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,16,17,20,21,24,25,28,29>
+; AVX2-FAST-ALL-NEXT:    vpshufb %ymm3, %ymm2, %ymm4
 ; AVX2-FAST-ALL-NEXT:    vpermd %ymm5, %ymm15, %ymm2
-; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} ymm0 = <0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15,16,17,20,21,24,25,28,29,u,u,u,u,u,u,u,u>
-; AVX2-FAST-ALL-NEXT:    vpshufb %ymm0, %ymm2, %ymm5
-; AVX2-FAST-ALL-NEXT:    vmovdqa %ymm0, %ymm14
+; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} ymm14 = <0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15,16,17,20,21,24,25,28,29,u,u,u,u,u,u,u,u>
+; AVX2-FAST-ALL-NEXT:    vpshufb %ymm14, %ymm2, %ymm5
 ; AVX2-FAST-ALL-NEXT:    vpblendd {{.*#+}} ymm4 = ymm5[0,1,2,3,4,5],ymm4[6,7]
 ; AVX2-FAST-ALL-NEXT:    vpblendd {{.*#+}} ymm0 = ymm1[0,1,2,3],ymm4[4,5,6,7]
 ; AVX2-FAST-ALL-NEXT:    vmovdqu %ymm0, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
 ; AVX2-FAST-ALL-NEXT:    vmovdqa 176(%rdi), %xmm13
 ; AVX2-FAST-ALL-NEXT:    vpblendw {{.*#+}} xmm5 = xmm13[0],xmm8[1,2,3],xmm13[4],xmm8[5,6,7]
 ; AVX2-FAST-ALL-NEXT:    vmovdqa %xmm13, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-FAST-ALL-NEXT:    vmovdqa 160(%rdi), %xmm0
 ; AVX2-FAST-ALL-NEXT:    vmovdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill

 ; AVX2-FAST-ALL-NEXT:    vpshufb %xmm14, %xmm12, %xmm3
 ; AVX2-FAST-ALL-NEXT:    vpunpckldq {{.*#+}} xmm15 = xmm3[0],xmm0[0],xmm3[1],xmm0[1]
 ; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} xmm3 = [2,3,10,11,8,9,10,11,8,9,10,11,12,13,14,15]
 ; AVX2-FAST-ALL-NEXT:    vpshufb %xmm3, %xmm11, %xmm0
 ; AVX2-FAST-ALL-NEXT:    vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm8 # 16-byte Reload
 ; AVX2-FAST-ALL-NEXT:    vpshufb %xmm3, %xmm8, %xmm12
 ; AVX2-FAST-ALL-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm12[0],xmm0[0],xmm12[1],xmm0[1]
 ; AVX2-FAST-ALL-NEXT:    vpblendd {{.*#+}} xmm0 = xmm0[0,1],xmm15[2,3]
-; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} ymm11 = <u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,18,19,22,23,26,27,30,31>
+; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} ymm15 = <u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,u,18,19,22,23,26,27,30,31>
 ; AVX2-FAST-ALL-NEXT:    vmovdqu {{[-0-9]+}}(%r{{[sb]}}p), %ymm12 # 32-byte Reload
-; AVX2-FAST-ALL-NEXT:    vpshufb %ymm11, %ymm12, %ymm12
-; AVX2-FAST-ALL-NEXT:    vmovdqa %ymm11, %ymm15
+; AVX2-FAST-ALL-NEXT:    vpshufb %ymm15, %ymm12, %ymm12
 ; AVX2-FAST-ALL-NEXT:    vmovdqa {{.*#+}} ymm11 = <2,3,6,7,10,11,14,15,14,15,10,11,12,13,14,15,18,19,22,23,26,27,30,31,u,u,u,u,u,u,u,u>
 ; AVX2-FAST-ALL-NEXT:    vpshufb %ymm11, %ymm2, %ymm2
 ; AVX2-FAST-ALL-NEXT:    vpblendd {{.*#+}} ymm2 = ymm2[0,1,2,3,4,5],ymm12[6,7]
 ; AVX2-FAST-ALL-NEXT:    vpblendd {{.*#+}} ymm0 = ymm0[0,1,2,3],ymm2[4,5,6,7]
 ; AVX2-FAST-ALL-NEXT:    vmovdqu %ymm0, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
 ; AVX2-FAST-ALL-NEXT:    vpshufb %xmm14, %xmm13, %xmm0
 ; AVX2-FAST-ALL-NEXT:    vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm13 # 16-byte Reload
 ; AVX2-FAST-ALL-NEXT:    vpshufb %xmm14, %xmm13, %xmm2

 ; AVX2-FAST-ALL-NEXT:    vzeroupper
 ; AVX2-FAST-ALL-NEXT:    retq
 ;
 ; AVX2-FAST-PERLANE-LABEL: vf32:
 ; AVX2-FAST-PERLANE:       # %bb.0:
 ; AVX2-FAST-PERLANE-NEXT:    subq $248, %rsp
 ; AVX2-FAST-PERLANE-NEXT:    vmovdqa (%rdi), %xmm6
 ; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm6, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; AVX2-FAST-PERLANE-NEXT:    vmovdqa 16(%rdi), %xmm1
+; AVX2-FAST-PERLANE-NEXT:    vmovdqa 16(%rdi), %xmm9
 ; AVX2-FAST-PERLANE-NEXT:    vmovdqa 32(%rdi), %xmm5
-; AVX2-FAST-PERLANE-NEXT:    vmovdqa 48(%rdi), %xmm3
+; AVX2-FAST-PERLANE-NEXT:    vmovdqa 48(%rdi), %xmm8
 ; AVX2-FAST-PERLANE-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm2 = xmm3[0],xmm0[1,2,3],xmm3[4],xmm0[5,6,7]
-; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm3, %xmm8
-; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm3, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm2 = xmm8[0],xmm0[1,2,3],xmm8[4],xmm0[5,6,7]
+; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm8, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm3 = xmm5[0],xmm0[1,2,3],xmm5[4],xmm0[5,6,7]
 ; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm5, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-FAST-PERLANE-NEXT:    vpackusdw %xmm2, %xmm3, %xmm2
-; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm3 = xmm1[0],xmm0[1,2,3],xmm1[4],xmm0[5,6,7]
-; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm1, %xmm9
+; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm3 = xmm9[0],xmm0[1,2,3],xmm9[4],xmm0[5,6,7]
 ; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm4 = xmm6[0],xmm0[1,2,3],xmm6[4],xmm0[5,6,7]
 ; AVX2-FAST-PERLANE-NEXT:    vpackusdw %xmm3, %xmm4, %xmm3
 ; AVX2-FAST-PERLANE-NEXT:    vpackusdw %xmm2, %xmm3, %xmm2
 ; AVX2-FAST-PERLANE-NEXT:    vmovdqa 112(%rdi), %xmm15
 ; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm3 = xmm15[0],xmm0[1,2,3],xmm15[4],xmm0[5,6,7]
-; AVX2-FAST-PERLANE-NEXT:    vmovdqa 96(%rdi), %xmm1
-; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm4 = xmm1[0],xmm0[1,2,3],xmm1[4],xmm0[5,6,7]
-; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm1, %xmm6
-; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; AVX2-FAST-PERLANE-NEXT:    vmovdqa 96(%rdi), %xmm6
+; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm4 = xmm6[0],xmm0[1,2,3],xmm6[4],xmm0[5,6,7]
+; AVX2-FAST-PERLANE-NEXT:    vmovdqa %xmm6, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; AVX2-FAST-PERLANE-NEXT:    vpackusdw %xmm3, %xmm4, %xmm3
 ; AVX2-FAST-PERLANE-NEXT:    vpackusdw %xmm3, %xmm3, %xmm3
 ; AVX2-FAST-PERLANE-NEXT:    vinserti128 $1, %xmm3, %ymm0, %ymm3
 ; AVX2-FAST-PERLANE-NEXT:    vmovdqa 80(%rdi), %xmm12
 ; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm7 = xmm12[0],xmm0[1,2,3],xmm12[4],xmm0[5,6,7]
 ; AVX2-FAST-PERLANE-NEXT:    vmovdqa 64(%rdi), %xmm14
 ; AVX2-FAST-PERLANE-NEXT:    vpblendw {{.*#+}} xmm1 = xmm14[0],xmm0[1,2,3],xmm14[4],xmm0[5,6,7]
 ; AVX2-FAST-PERLANE-NEXT:    vpackusdw %xmm7, %xmm1, %xmm1