[MachineSinking] sink more profitable loads

[shchenz]

Summary

Instead of setting Store to true conservatively, handle some cases where Store can be determined by some analysis if MI.parent() dom SuccToSinkTo and SuccToSinkTo post dom MI.parent().

Thus we can get more profitable loads to be sunk.

[shchenz]

handle further more cases to sink profitable loads

[efriedma]

Do we need some threshold to limit the analysis? I'm worried this could get very expensive.

llvm/lib/CodeGen/MachineSink.cpp

889

Are the post-dominance checks necessary? I'm not sure what invariants you're trying to enforce.

[shchenz]

1: can not check alias for call instruction by using mayAlias
2: add one more cache to save compiling time.

[shchenz]

In D86864#2250785, @efriedma wrote:

Do we need some threshold to limit the analysis? I'm worried this could get very expensive.

I add a new cache to save compiling time. Could you be more specific on how to add the threshold? Thanks.

llvm/lib/CodeGen/MachineSink.cpp

889

they are not necessary. I add the post-dominance check for the convenience of finding all blocks which are in the path from block From to To.
With these checks, it is easy to judge if a block BB(reachable from From) is in the path from From to To. Just check PDT->dominates(To, BB))
`

[shchenz]

gentle ping

[shchenz]

gentle ping

[qcolombet]

Hi @efriedma,

Given you've already taken a look, can you do the review or do you want me to step in?

Cheers,
-Quentin

[efriedma]

I'd appreciate it if you'd step in, Quentin. I have a backlog of reviews, and I still haven't quite wrapped my head around the use of post-dominance here.

[shchenz]

@efriedma @qcolombet haha, thanks for your review. Please take your time. ^-^ . I want to make the alias check simple(find all basic blocks in all paths from block From to block To), so I guard the dominance between From and To as From dominates To and To post dominates From. This will make us still miss some cases in which load instruction can be sunk. But it can make the complexity simple.

[shchenz]

gentle ping ^^

[qcolombet]

Hi,

Looks reasonable but I'd like to have an idea of the compile time impact of this patch.

Make sure to fix the Lint comments as well.

Cheers,
-Quentin

llvm/lib/CodeGen/MachineSink.cpp

343

Would it make sense to preserve the cache as long as we don't move stores, calls and loads with ordered memory ref?

I'm worried about the compile time impact of this patch so I am wondering if we want to be smarter about invalidating the cache.

912

While we traverse BB, should we create an entry for the pair (To, BB).

Put differently, how is the compile time impact of this patch and do we have to do more to avoid computations?

[shchenz]

1: clear the cache just before function returns
2: cache handled bb while calling depth_first() for a block

[shchenz]

Thanks very much for your review. @qcolombet Update accordingly. I collected some compiling time numbers for llvm test-suite. No obivious degradation found.

llvm/lib/CodeGen/MachineSink.cpp

343

Good idea. But seems we still need to clear the cache before we return from MachineSinking::runOnMachineFunction. (MachineSink instance is shared with different callings to MachineSinking::runOnMachineFunction for different functions), otherwise I meet some weird memory issue.

912

I collected some data from PowerPC, seems the compiling time difference is not obivious in llvm test-suite. the tool hides some small diff tests. Among them, I saw some tests takes 10s and some of them takes more than 30s, but no big reg for them. The biggest diff 9.6% and 7.4%, compile time are around 0.1s. And these two degradations can not reproduciable in other runs.

./utils/compare.py -m compile_time base.json fix.json  
Tests: 309
Metric: compile_time

Program                                        base  fix   diff 
                                                                  
 test-suite...enchmarks/Stanford/Queens.test       9.6%
 test-suite...d-warshall/floyd-warshall.test       7.4%
 test-suite...ImageProcessing/Blur/blur.test      -4.7%
 test-suite...SubsetBRawLoops/lcalsBRaw.test       2.7%
 test-suite...math/automotive-basicmath.test     -2.5%
 test-suite.../Benchmarks/Stanford/Perm.test      -2.4%
 test-suite...rks/tramp3d-v4/tramp3d-v4.test     -2.2%
 test-suite...ow-dbl/GlobalDataFlow-dbl.test      -2.2%
 test-suite...enchmarks/Stanford/RealMM.test      -2.0%
 test-suite...ctions-flt/Reductions-flt.test      -1.8%
 test-suite...t/StatementReordering-flt.test      -1.7%
 test-suite.../Trimaran/enc-rc4/enc-rc4.test      -1.7%
 test-suite...l/StatementReordering-dbl.test      1.6%
 test-suite...pansion-dbl/Expansion-dbl.test      -1.4%
 test-suite...bl/IndirectAddressing-dbl.test      -1.2%
 Geomean difference                                           nan%
            base        fix        diff
count  309.000000  309.000000  277.000000
mean   2.696964    2.685603   -0.003232  
std    6.283106    6.251353    0.009456  
min    0.000000    0.000000   -0.047321  
25%    0.077944    0.077862   -0.005599  
50%    0.291912    0.289780   -0.003404  
75%    2.475318    2.478657   -0.001450  
max    64.257411   64.088470   0.095601

[shchenz]

1: address Lint suggestion

[shchenz]

Ah, I used @nikic perf testing tool for this patch on the X86 target(?). (Thanks for this great tool ^-^) Here is the result of the compiling time: https://llvm-compile-time-tracker.com/compare.php?from=2d25004a137223a02aa06e8bfd512a648f3b3f94&to=abf39366779bed8b9f2d276cd199c3b78471e3b1&stat=instructions

GEO degradates about 0.02% and no obivious outlier. I guess the compiling time increase on these benchmarks should be acceptable?

[qcolombet]

I guess the compiling time increase on these benchmarks should be acceptable?

Yes, that's fine by me.

llvm/lib/CodeGen/MachineSink.cpp

343

Yeah, that's expected.

[LiuChen3]

Hi, @shchenz. Our several opecncl benchmarks have appeared great compile time regression.
For only one function, the time consume on Machine code sinking pass increased form 6.0711s to 366.5713.
According to your algorithm, this patch will obviously increase the compile time for some cases.

[shchenz]

In D86864#2469289, @LiuChen3 wrote:

Hi, @shchenz. Our several opecncl benchmarks have appeared great compile time regression.
For only one function, the time consume on Machine code sinking pass increased form 6.0711s to 366.5713.
According to your algorithm, this patch will obviously increase the compile time for some cases.

Yes, we have been aware that this patch may introduce compiling time degradations. And as you can see in previous comments, I already tested the compiling time on X86 arch. Sadly, the tested benchmarks don't expose any regressions.

Could you please help to send me your regression function/IR? So I can have a look about how to fix it? Thanks.

[LiuChen3]

Yes, we have been aware that this patch may introduce compiling time degradations. And as you can see in previous comments, I already tested the compiling time on X86 arch. Sadly, the tested benchmarks don't expose any regressions.

Could you please help to send me your regression function/IR? So I can have a look about how to fix it? Thanks.

I am sorry that I can't provide the case to you directly. I am trying making a small reproduce but encountered some problems.
However, can you adding an option to control it?
Thanks.

[shchenz]

In D86864#2469576, @LiuChen3 wrote:

Yes, we have been aware that this patch may introduce compiling time degradations. And as you can see in previous comments, I already tested the compiling time on X86 arch. Sadly, the tested benchmarks don't expose any regressions.

Could you please help to send me your regression function/IR? So I can have a look about how to fix it? Thanks.

I am sorry that I can't provide the case to you directly. I am trying making a small reproduce but encountered some problems.
However, can you adding an option to control it?
Thanks.

I can surely do that. But I think the most reasonable solution would be fix the compiling time issue. Since compiling time tests I did before does not expose any regression, your test case must be a little special. Could you find out the special point, for example the function has too many blocks or some/many blocks in the function has too many instructions? Thanks.

[LiuChen3]

I can surely do that. But I think the most reasonable solution would be fix the compiling time issue. Since compiling time tests I did before does not expose any regression, your test case must be a little special. Could you find out the special point, for example the function has too many blocks or some/many blocks in the function has too many instructions? Thanks.

I think the increase in compile time is because the function has too many instructions and blocks. The function has Thousands of lines of instructions. Can you add limitation for the number of instructions or the number of blocks so the check for 'store' can end early?

[shchenz]

In D86864#2472369, @LiuChen3 wrote:

I can surely do that. But I think the most reasonable solution would be fix the compiling time issue. Since compiling time tests I did before does not expose any regression, your test case must be a little special. Could you find out the special point, for example the function has too many blocks or some/many blocks in the function has too many instructions? Thanks.

I think the increase in compile time is because the function has too many instructions and blocks. The function has Thousands of lines of instructions. Can you add limitation for the number of instructions or the number of blocks so the check for 'store' can end early?

Could you please help to check if the following change can solve your issue?

C
diff --git a/llvm/lib/CodeGen/MachineSink.cpp b/llvm/lib/CodeGen/MachineSink.cpp
index 0abdf89..8ca3520 100644
--- a/llvm/lib/CodeGen/MachineSink.cpp
+++ b/llvm/lib/CodeGen/MachineSink.cpp
@@ -79,6 +79,12 @@ static cl::opt<unsigned> SplitEdgeProbabilityThreshold(
         "splitted critical edge"),
     cl::init(40), cl::Hidden);
 
+static cl::opt<unsigned> SinkLoadInstsPerBlockThreshold(
+    "machine-sink-load-instrs-threshold",
+    cl::desc("Do not try to find alias store for a load if there is a in-path "
+             "block whose instruction number is higher than this threshold."),
+    cl::init(2000), cl::Hidden);
+
 STATISTIC(NumSunk,      "Number of machine instructions sunk");
 STATISTIC(NumSplit,     "Number of critical edges split");
 STATISTIC(NumCoalesces, "Number of copies coalesced");
@@ -1036,6 +1042,12 @@ bool MachineSinking::hasStoreBetween(MachineBasicBlock *From,
     HandledBlocks.insert(BB);
     // To post dominates BB, it must be a path from block From.
     if (PDT->dominates(To, BB)) {
+      // If this BB is too big, stop searching to save compiling time.
+      if (BB->size() > SinkLoadInstsPerBlockThreshold) {
+        HasStoreCache[BlockPair] = true;
+        return true;
+      }
+
       for (MachineInstr &I : *BB) {
         // Treat as alias conservatively for a call or an ordered memory
         // operation.

[LiuChen3]

In D86864#2472375, @shchenz wrote:

In D86864#2472369, @LiuChen3 wrote:

I can surely do that. But I think the most reasonable solution would be fix the compiling time issue. Since compiling time tests I did before does not expose any regression, your test case must be a little special. Could you find out the special point, for example the function has too many blocks or some/many blocks in the function has too many instructions? Thanks.

I think the increase in compile time is because the function has too many instructions and blocks. The function has Thousands of lines of instructions. Can you add limitation for the number of instructions or the number of blocks so the check for 'store' can end early?

Could you please help to check if the following change can solve your issue?

C
diff --git a/llvm/lib/CodeGen/MachineSink.cpp b/llvm/lib/CodeGen/MachineSink.cpp
index 0abdf89..8ca3520 100644
--- a/llvm/lib/CodeGen/MachineSink.cpp
+++ b/llvm/lib/CodeGen/MachineSink.cpp
@@ -79,6 +79,12 @@ static cl::opt<unsigned> SplitEdgeProbabilityThreshold(
         "splitted critical edge"),
     cl::init(40), cl::Hidden);
 
+static cl::opt<unsigned> SinkLoadInstsPerBlockThreshold(
+    "machine-sink-load-instrs-threshold",
+    cl::desc("Do not try to find alias store for a load if there is a in-path "
+             "block whose instruction number is higher than this threshold."),
+    cl::init(2000), cl::Hidden);
+
 STATISTIC(NumSunk,      "Number of machine instructions sunk");
 STATISTIC(NumSplit,     "Number of critical edges split");
 STATISTIC(NumCoalesces, "Number of copies coalesced");
@@ -1036,6 +1042,12 @@ bool MachineSinking::hasStoreBetween(MachineBasicBlock *From,
     HandledBlocks.insert(BB);
     // To post dominates BB, it must be a path from block From.
     if (PDT->dominates(To, BB)) {
+      // If this BB is too big, stop searching to save compiling time.
+      if (BB->size() > SinkLoadInstsPerBlockThreshold) {
+        HasStoreCache[BlockPair] = true;
+        return true;
+      }
+
       for (MachineInstr &I : *BB) {
         // Treat as alias conservatively for a call or an ordered memory
         // operation.

Unfortunately, this patch doesn't work. I don't think the number of blocks is necessarily related to the number of instructions.
For one function, with your default threshold, The time spent on Machine code sinking is 11.2464s, no obvious difference from before. When set the threshold to 1, the time spent on Machine code sinking reduced 0.6937.
I think it would be better if you can limit the number of MIs.
Anyway, this patch provides us with an option. Thanks for your help.

[shchenz]

In D86864#2472387, @LiuChen3 wrote:

In D86864#2472375, @shchenz wrote:

In D86864#2472369, @LiuChen3 wrote:

I can surely do that. But I think the most reasonable solution would be fix the compiling time issue. Since compiling time tests I did before does not expose any regression, your test case must be a little special. Could you find out the special point, for example the function has too many blocks or some/many blocks in the function has too many instructions? Thanks.

I think the increase in compile time is because the function has too many instructions and blocks. The function has Thousands of lines of instructions. Can you add limitation for the number of instructions or the number of blocks so the check for 'store' can end early?

Could you please help to check if the following change can solve your issue?

C
diff --git a/llvm/lib/CodeGen/MachineSink.cpp b/llvm/lib/CodeGen/MachineSink.cpp
index 0abdf89..8ca3520 100644
--- a/llvm/lib/CodeGen/MachineSink.cpp
+++ b/llvm/lib/CodeGen/MachineSink.cpp
@@ -79,6 +79,12 @@ static cl::opt<unsigned> SplitEdgeProbabilityThreshold(
         "splitted critical edge"),
     cl::init(40), cl::Hidden);
 
+static cl::opt<unsigned> SinkLoadInstsPerBlockThreshold(
+    "machine-sink-load-instrs-threshold",
+    cl::desc("Do not try to find alias store for a load if there is a in-path "
+             "block whose instruction number is higher than this threshold."),
+    cl::init(2000), cl::Hidden);
+
 STATISTIC(NumSunk,      "Number of machine instructions sunk");
 STATISTIC(NumSplit,     "Number of critical edges split");
 STATISTIC(NumCoalesces, "Number of copies coalesced");
@@ -1036,6 +1042,12 @@ bool MachineSinking::hasStoreBetween(MachineBasicBlock *From,
     HandledBlocks.insert(BB);
     // To post dominates BB, it must be a path from block From.
     if (PDT->dominates(To, BB)) {
+      // If this BB is too big, stop searching to save compiling time.
+      if (BB->size() > SinkLoadInstsPerBlockThreshold) {
+        HasStoreCache[BlockPair] = true;
+        return true;
+      }
+
       for (MachineInstr &I : *BB) {
         // Treat as alias conservatively for a call or an ordered memory
         // operation.

Unfortunately, this patch doesn't work. I don't think the number of blocks is necessarily related to the number of instructions.
For one function, with your default threshold, The time spent on Machine code sinking is 11.2464s, no obvious difference from before. When set the threshold to 1, the time spent on Machine code sinking reduced 0.6937.
I think it would be better if you can limit the number of MIs.
Anyway, this patch provides us with an option. Thanks for your help.

The above threshold is for number of MIs. BB->size() is to get instruction number of BB. I committed 31c2b93d83f63ce7f9bb4977f58de2e00bf18e0f to further reduce compiling time. You can have a try

[steven.zhang]

Is it better to have the check before 'if (PDT->dominates(To, BB)) {' ?

[shchenz]

In D86864#2472470, @steven.zhang wrote:

Is it better to have the check before 'if (PDT->dominates(To, BB)) {' ?

I don't think so. If there is a huge block that is reachable from block From but is not dominated by From and can not flow to To, we will mark {From, To} as hasStore. This is not true, we should only care about the blocks which are dominated by From and can flow to To.

[LiuChen3]

The above threshold is for number of MIs. BB->size() is to get instruction number of BB. I committed 31c2b93d83f63ce7f9bb4977f58de2e00bf18e0f to further reduce compiling time. You can have a try

Thanks. It works for our tests.

llvm/lib/CodeGen/MachineSink.cpp

class MachineSinking : public MachineFunctionPass {
     /// Record of DBG_VALUE uses of vregs in a block, so that we can identify
     /// debug instructions to sink.
     SmallDenseMap<unsigned, TinyPtrVector<SeenDbgUser>> SeenDbgUsers;
 
     /// Record of debug variables that have had their locations set in the
     /// current block.
     DenseSet<DebugVariable> SeenDbgVars;
 
+    std::map<std::pair<MachineBasicBlock *, MachineBasicBlock *>, bool> StoreCache;
+
   public:
     static char ID; // Pass identification
 
     MachineSinking() : MachineFunctionPass(ID) {
       initializeMachineSinkingPass(*PassRegistry::getPassRegistry());
     }
 
     bool runOnMachineFunction(MachineFunction &MF) override;
namespace {
 
   private:
     bool ProcessBlock(MachineBasicBlock &MBB);
     void ProcessDbgInst(MachineInstr &MI);
     bool isWorthBreakingCriticalEdge(MachineInstr &MI,
                                      MachineBasicBlock *From,
                                      MachineBasicBlock *To);
 
+    bool hasStoreBetween(MachineBasicBlock *From, MachineBasicBlock *To);
+
     /// Postpone the splitting of the given critical
     /// edge (\p From, \p To).
     ///
     /// We do not split the edges on the fly. Indeed, this invalidates
     /// the dominance information and thus triggers a lot of updates
     /// of that information underneath.
     /// Instead, we postpone all the splits after each iteration of
     /// the main loop. That way, the information is at least valid
bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
   bool EverMadeChange = false;
 
   while (true) {
     bool MadeChange = false;
 
     // Process all basic blocks.
     CEBCandidates.clear();
     ToSplit.clear();
     for (auto &MBB: MF)
       MadeChange |= ProcessBlock(MBB);
 
     // If we have anything we marked as toSplit, split it now.
     for (auto &Pair : ToSplit) {
       auto NewSucc = Pair.first->SplitCriticalEdge(Pair.second, *this);
       if (NewSucc != nullptr) {
         LLVM_DEBUG(dbgs() << " *** Splitting critical edge: "
                           << printMBBReference(*Pair.first) << " -- "
for (SmallVectorImpl<MachineInstr *>::iterator DBI = DbgValuesToSink.begin(),
     MachineInstr *NewDbgMI = DbgMI->getMF()->CloneMachineInstr(*DBI);
     SuccToSinkTo.insert(InsertPos, NewDbgMI);
 
     if (!attemptDebugCopyProp(MI, *DbgMI))
       DbgMI->setDebugValueUndef();
   }
 }
 
+/// hasStoreBetween - check if there is store betweeen straight line blocks From
+/// and To.
+bool MachineSinking::hasStoreBetween(MachineBasicBlock *From,
+                                     MachineBasicBlock *To) {
+  // Make sure From and To are in straight line which means From dominates To
+  // and To post dominates From.
+  if (!DT->dominates(From, To) || !PDT->dominates(To, From))
+    return true;
+
+  // Does these two blocks pair query before?
+  if (StoreCache.find(std::make_pair(From, To)) != StoreCache.end())
+    return StoreCache[std::make_pair(From, To)];
+
+  // Go through all reachable blocks from From.
+  for (MachineBasicBlock *BB : depth_first(From)) {
+    // We insert the instruction at the start of block To, so no need to worry
+    // about stores inside To.
+    // Store in block From should be considered just enter SinkInstruction.
+    if (BB == To || BB == From)
+      continue;
+    // To post dominates BB, it must be a path from block From.
+    if (PDT->dominates(To, BB)) {
+      for (const MachineInstr &MI : *BB)
+        if (MI.mayStore() || MI.isCall() ||
+            (MI.mayLoad() && MI.hasOrderedMemoryRef())) {
+          StoreCache[std::make_pair(From, To)] = true;
+          return true;
+        }
+    }
+  }
+  StoreCache[std::make_pair(From, To)] = false;
+  return false;
+}
+
 /// SinkInstruction - Determine whether it is safe to sink the specified machine
 /// instruction out of its current block into a successor.
 bool MachineSinking::SinkInstruction(MachineInstr &MI, bool &SawStore,
                                      AllSuccsCache &AllSuccessors) {
   // Don't sink instructions that the target prefers not to sink.
   if (!TII->shouldSink(MI))
     return false;
 
bool MachineSinking::SinkInstruction(MachineInstr &MI, bool &SawStore,
   LLVM_DEBUG(dbgs() << "Sink instr " << MI << "\tinto block " << *SuccToSinkTo);
 
   // If the block has multiple predecessors, this is a critical edge.
   // Decide if we can sink along it or need to break the edge.
   if (SuccToSinkTo->pred_size() > 1) {
     // We cannot sink a load across a critical edge - there may be stores in
     // other code paths.
     bool TryBreak = false;
-    bool store = true;
-    if (!MI.isSafeToMove(AA, store)) {
+    bool Store =
+        MI.mayLoad() ? hasStoreBetween(ParentBlock, SuccToSinkTo) : true;
+    if (!MI.isSafeToMove(AA, Store)) {
       LLVM_DEBUG(dbgs() << " *** NOTE: Won't sink load along critical edge.\n");
       TryBreak = true;
     }
 
     // We don't want to sink across a critical edge if we don't dominate the
     // successor. We could be introducing calculations to new code paths.
     if (!TryBreak && !DT->dominates(ParentBlock, SuccToSinkTo)) {
       LLVM_DEBUG(dbgs() << " *** NOTE: Critical edge found\n");

llvm/test/CodeGen/RISCV/select-optimize-multiple.ll

entry:
   %cond = select i1 %cmp, i64 %b, i64 %c
   ret i64 %cond
 }
 
 define i128 @cmovcc128(i64 signext %a, i128 %b, i128 %c) nounwind {
 ; RV32I-LABEL: cmovcc128:
 ; RV32I:       # %bb.0: # %entry
 ; RV32I-NEXT:    xori a1, a1, 123
-; RV32I-NEXT:    or a2, a1, a2
-; RV32I-NEXT:    mv a1, a3
-; RV32I-NEXT:    beqz a2, .LBB1_2
+; RV32I-NEXT:    or a1, a1, a2
+; RV32I-NEXT:    mv a2, a3
+; RV32I-NEXT:    beqz a1, .LBB1_2
 ; RV32I-NEXT:  # %bb.1: # %entry
-; RV32I-NEXT:    mv a1, a4
+; RV32I-NEXT:    mv a2, a4
 ; RV32I-NEXT:  .LBB1_2: # %entry
-; RV32I-NEXT:    lw a6, 0(a1)
-; RV32I-NEXT:    beqz a2, .LBB1_6
+; RV32I-NEXT:    beqz a1, .LBB1_5
 ; RV32I-NEXT:  # %bb.3: # %entry
-; RV32I-NEXT:    addi a1, a4, 4
-; RV32I-NEXT:    lw a5, 0(a1)
-; RV32I-NEXT:    bnez a2, .LBB1_7
+; RV32I-NEXT:    addi a7, a4, 4
+; RV32I-NEXT:    bnez a1, .LBB1_6
 ; RV32I-NEXT:  .LBB1_4:
-; RV32I-NEXT:    addi a1, a3, 8
-; RV32I-NEXT:    lw a1, 0(a1)
-; RV32I-NEXT:    bnez a2, .LBB1_8
+; RV32I-NEXT:    addi a5, a3, 8
+; RV32I-NEXT:    j .LBB1_7
 ; RV32I-NEXT:  .LBB1_5:
-; RV32I-NEXT:    addi a2, a3, 12
-; RV32I-NEXT:    j .LBB1_9
-; RV32I-NEXT:  .LBB1_6:
-; RV32I-NEXT:    addi a1, a3, 4
-; RV32I-NEXT:    lw a5, 0(a1)
-; RV32I-NEXT:    beqz a2, .LBB1_4
+; RV32I-NEXT:    addi a7, a3, 4
+; RV32I-NEXT:    beqz a1, .LBB1_4
+; RV32I-NEXT:  .LBB1_6: # %entry
+; RV32I-NEXT:    addi a5, a4, 8
 ; RV32I-NEXT:  .LBB1_7: # %entry
-; RV32I-NEXT:    addi a1, a4, 8
+; RV32I-NEXT:    lw a6, 0(a2)
+; RV32I-NEXT:    lw a7, 0(a7)
+; RV32I-NEXT:    lw a2, 0(a5)
+; RV32I-NEXT:    beqz a1, .LBB1_9
+; RV32I-NEXT:  # %bb.8: # %entry
+; RV32I-NEXT:    addi a1, a4, 12
+; RV32I-NEXT:    j .LBB1_10
+; RV32I-NEXT:  .LBB1_9:
+; RV32I-NEXT:    addi a1, a3, 12
+; RV32I-NEXT:  .LBB1_10: # %entry
 ; RV32I-NEXT:    lw a1, 0(a1)
-; RV32I-NEXT:    beqz a2, .LBB1_5
-; RV32I-NEXT:  .LBB1_8: # %entry
-; RV32I-NEXT:    addi a2, a4, 12
-; RV32I-NEXT:  .LBB1_9: # %entry
-; RV32I-NEXT:    lw a2, 0(a2)
-; RV32I-NEXT:    sw a2, 12(a0)
-; RV32I-NEXT:    sw a1, 8(a0)
-; RV32I-NEXT:    sw a5, 4(a0)
+; RV32I-NEXT:    sw a1, 12(a0)
+; RV32I-NEXT:    sw a2, 8(a0)
+; RV32I-NEXT:    sw a7, 4(a0)
 ; RV32I-NEXT:    sw a6, 0(a0)
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: cmovcc128:
 ; RV64I:       # %bb.0: # %entry
 ; RV64I-NEXT:    addi a5, zero, 123
 ; RV64I-NEXT:    beq a0, a5, .LBB1_2
 ; RV64I-NEXT:  # %bb.1: # %entry

 entry:
   %cond = select i1 %a, i64 %b, i64 %c
   ret i64 %cond
 }
 
 define i128 @cmov128(i1 %a, i128 %b, i128 %c) nounwind {
 ; RV32I-LABEL: cmov128:
 ; RV32I:       # %bb.0: # %entry
-; RV32I-NEXT:    andi a4, a1, 1
-; RV32I-NEXT:    mv a1, a2
-; RV32I-NEXT:    bnez a4, .LBB3_2
+; RV32I-NEXT:    andi a1, a1, 1
+; RV32I-NEXT:    mv a4, a2
+; RV32I-NEXT:    bnez a1, .LBB3_2
 ; RV32I-NEXT:  # %bb.1: # %entry
-; RV32I-NEXT:    mv a1, a3
+; RV32I-NEXT:    mv a4, a3
 ; RV32I-NEXT:  .LBB3_2: # %entry
-; RV32I-NEXT:    lw a6, 0(a1)
-; RV32I-NEXT:    bnez a4, .LBB3_6
+; RV32I-NEXT:    bnez a1, .LBB3_5
 ; RV32I-NEXT:  # %bb.3: # %entry
-; RV32I-NEXT:    addi a1, a3, 4
-; RV32I-NEXT:    lw a5, 0(a1)
-; RV32I-NEXT:    beqz a4, .LBB3_7
+; RV32I-NEXT:    addi a7, a3, 4
+; RV32I-NEXT:    beqz a1, .LBB3_6
 ; RV32I-NEXT:  .LBB3_4:
-; RV32I-NEXT:    addi a1, a2, 8
-; RV32I-NEXT:    lw a1, 0(a1)
-; RV32I-NEXT:    beqz a4, .LBB3_8
+; RV32I-NEXT:    addi a5, a2, 8
+; RV32I-NEXT:    j .LBB3_7
 ; RV32I-NEXT:  .LBB3_5:
-; RV32I-NEXT:    addi a2, a2, 12
-; RV32I-NEXT:    j .LBB3_9
-; RV32I-NEXT:  .LBB3_6:
-; RV32I-NEXT:    addi a1, a2, 4
-; RV32I-NEXT:    lw a5, 0(a1)
-; RV32I-NEXT:    bnez a4, .LBB3_4
+; RV32I-NEXT:    addi a7, a2, 4
+; RV32I-NEXT:    bnez a1, .LBB3_4
+; RV32I-NEXT:  .LBB3_6: # %entry
+; RV32I-NEXT:    addi a5, a3, 8
 ; RV32I-NEXT:  .LBB3_7: # %entry
-; RV32I-NEXT:    addi a1, a3, 8
+; RV32I-NEXT:    lw a6, 0(a4)
+; RV32I-NEXT:    lw a7, 0(a7)
+; RV32I-NEXT:    lw a4, 0(a5)
+; RV32I-NEXT:    bnez a1, .LBB3_9
+; RV32I-NEXT:  # %bb.8: # %entry
+; RV32I-NEXT:    addi a1, a3, 12
+; RV32I-NEXT:    j .LBB3_10
+; RV32I-NEXT:  .LBB3_9:
+; RV32I-NEXT:    addi a1, a2, 12
+; RV32I-NEXT:  .LBB3_10: # %entry
 ; RV32I-NEXT:    lw a1, 0(a1)
-; RV32I-NEXT:    bnez a4, .LBB3_5
-; RV32I-NEXT:  .LBB3_8: # %entry
-; RV32I-NEXT:    addi a2, a3, 12
-; RV32I-NEXT:  .LBB3_9: # %entry
-; RV32I-NEXT:    lw a2, 0(a2)
-; RV32I-NEXT:    sw a2, 12(a0)
-; RV32I-NEXT:    sw a1, 8(a0)
-; RV32I-NEXT:    sw a5, 4(a0)
+; RV32I-NEXT:    sw a1, 12(a0)
+; RV32I-NEXT:    sw a4, 8(a0)
+; RV32I-NEXT:    sw a7, 4(a0)
 ; RV32I-NEXT:    sw a6, 0(a0)
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: cmov128:
 ; RV64I:       # %bb.0: # %entry
 ; RV64I-NEXT:    andi a5, a0, 1
 ; RV64I-NEXT:    mv a0, a1
 ; RV64I-NEXT:    bnez a5, .LBB3_2

llvm/test/CodeGen/Thumb2/LowOverheadLoops/reductions.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve -tail-predication=enabled %s -o - | FileCheck %s
 
 define dso_local arm_aapcs_vfpcc zeroext i8 @one_loop_add_add_v16i8(i8* nocapture readonly %a, i8* nocapture readonly %b, i32 %N) local_unnamed_addr {
 ; CHECK-LABEL: one_loop_add_add_v16i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cbz r2, .LBB0_4
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    dlstp.8 lr, r2
-; CHECK:  .LBB0_2: @ %vector.body
+; CHECK-NEXT:  .LBB0_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrb.u8 q1, [r1], #16
 ; CHECK-NEXT:    vldrb.u8 q2, [r0], #16
 ; CHECK-NEXT:    vadd.i8 q0, q2, q1
 ; CHECK-NEXT:    vaddv.u8 r12, q0
 ; CHECK-NEXT:    letp lr, .LBB0_2
 ; CHECK-NEXT:  @ %bb.3: @ %for.cond.cleanup
 ; CHECK-NEXT:    uxtb.w r0, r12

 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    adds r3, r2, #7
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #7
 ; CHECK-NEXT:    sub.w r12, r3, #8
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w lr, r3, r12, lsr #3
 ; CHECK-NEXT:    dls lr, lr
-; CHECK:  .LBB1_2: @ %vector.body
+; CHECK-NEXT:  .LBB1_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.16 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vldrbt.u16 q1, [r0], #8
 ; CHECK-NEXT:    subs r2, #8
 ; CHECK-NEXT:    vadd.i16 q1, q0, q1
 ; CHECK-NEXT:    vpst

 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    add.w r3, r2, #15
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #15
 ; CHECK-NEXT:    sub.w r12, r3, #16
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w lr, r3, r12, lsr #4
 ; CHECK-NEXT:    dls lr, lr
-; CHECK:  .LBB2_2: @ %vector.body
+; CHECK-NEXT:  .LBB2_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.8 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u8 q1, [r1], #16
 ; CHECK-NEXT:    vldrbt.u8 q2, [r0], #16
 ; CHECK-NEXT:    subs r2, #16
 ; CHECK-NEXT:    vsub.i8 q1, q2, q1

 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    adds r3, r2, #7
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #7
 ; CHECK-NEXT:    sub.w r12, r3, #8
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w lr, r3, r12, lsr #3
 ; CHECK-NEXT:    dls lr, lr
-; CHECK:  .LBB3_2: @ %vector.body
+; CHECK-NEXT:  .LBB3_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.16 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u16 q1, [r0], #8
 ; CHECK-NEXT:    vldrbt.u16 q2, [r1], #8
 ; CHECK-NEXT:    subs r2, #8
 ; CHECK-NEXT:    vsub.i16 q1, q2, q1

 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    add.w r3, r2, #15
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #15
 ; CHECK-NEXT:    sub.w r12, r3, #16
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w lr, r3, r12, lsr #4
 ; CHECK-NEXT:    dls lr, lr
-; CHECK:  .LBB4_2: @ %vector.body
+; CHECK-NEXT:  .LBB4_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.8 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u8 q1, [r0], #16
 ; CHECK-NEXT:    vldrbt.u8 q2, [r1], #16
 ; CHECK-NEXT:    subs r2, #16
 ; CHECK-NEXT:    vmul.i8 q1, q2, q1

 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    adds r3, r2, #7
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #7
 ; CHECK-NEXT:    sub.w r12, r3, #8
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w lr, r3, r12, lsr #3
 ; CHECK-NEXT:    dls lr, lr
-; CHECK:  .LBB5_2: @ %vector.body
+; CHECK-NEXT:  .LBB5_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.16 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u16 q1, [r0], #8
 ; CHECK-NEXT:    vldrbt.u16 q2, [r1], #8
 ; CHECK-NEXT:    subs r2, #8
 ; CHECK-NEXT:    vmul.i16 q1, q2, q1

 ; CHECK-NEXT:    bic r3, r3, #3
 ; CHECK-NEXT:    mov r4, r0
 ; CHECK-NEXT:    subs r6, r3, #4
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    mov r5, r1
 ; CHECK-NEXT:    add.w lr, r3, r6, lsr #2
 ; CHECK-NEXT:    mov r3, r2
 ; CHECK-NEXT:    dls lr, lr
-; CHECK:  .LBB6_2: @ %vector.body
+; CHECK-NEXT:  .LBB6_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.32 r3
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u32 q1, [r4], #4
 ; CHECK-NEXT:    vldrbt.u32 q2, [r5], #4
 ; CHECK-NEXT:    subs r3, #4
 ; CHECK-NEXT:    vmul.i32 q1, q2, q1
 ; CHECK-NEXT:    vadd.i32 q1, q1, q0
 ; CHECK-NEXT:    le lr, .LBB6_2
 ; CHECK-NEXT:  @ %bb.3: @ %middle.block
 ; CHECK-NEXT:    vpsel q0, q1, q0
 ; CHECK-NEXT:    vaddv.u32 r12, q0
 ; CHECK-NEXT:    cbz r2, .LBB6_7
 ; CHECK-NEXT:  @ %bb.4: @ %vector.ph47
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w lr, r3, r6, lsr #2
 ; CHECK-NEXT:    movs r3, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vdup.32 q0, r3
 ; CHECK-NEXT:    vmov.32 q0[0], r12
-; CHECK:  .LBB6_5: @ %vector.body46
+; CHECK-NEXT:  .LBB6_5: @ %vector.body46
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.32 r2
 ; CHECK-NEXT:    vmov q1, q0
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u32 q0, [r0], #4
 ; CHECK-NEXT:    vldrbt.u32 q2, [r1], #4
 ; CHECK-NEXT:    subs r2, #4
 ; CHECK-NEXT:    vmul.i32 q0, q2, q0

 ; CHECK-NEXT:    bic r3, r3, #7
 ; CHECK-NEXT:    movs r4, #1
 ; CHECK-NEXT:    subs r3, #8
 ; CHECK-NEXT:    vmov q3, q1
 ; CHECK-NEXT:    add.w lr, r4, r3, lsr #3
 ; CHECK-NEXT:    mov r3, r0
 ; CHECK-NEXT:    mov r4, r1
 ; CHECK-NEXT:    dls lr, lr
-; CHECK:  .LBB7_2: @ %vector.body
+; CHECK-NEXT:  .LBB7_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.16 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u16 q1, [r3], #8
 ; CHECK-NEXT:    vldrbt.u16 q4, [r4], #8
 ; CHECK-NEXT:    vmov q2, q3
 ; CHECK-NEXT:    vsub.i16 q3, q4, q1
for.cond.cleanup:                                 ; preds = %middle.block, %entry
   ret void
 }
 
 %struct.date = type { i32, i32, i32, i32 }
 @days = internal unnamed_addr constant [2 x [13 x i32]] [[13 x i32] [i32 0, i32 31, i32 28, i32 31, i32 30, i32 31, i32 30, i32 31, i32 31, i32 30, i32 31, i32 30, i32 31], [13 x i32] [i32 0, i32 31, i32 29, i32 31, i32 30, i32 31, i32 30, i32 31, i32 31, i32 30, i32 31, i32 30, i32 31]], align 4
 define i32 @wrongop(%struct.date* nocapture readonly %pd) {
 ; CHECK-LABEL: wrongop:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push {r4, lr}
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr.w r2, [r12, #8]
-; CHECK-NEXT:    lsls r3, r2, #30
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    ldr r1, [r0, #8]
+; CHECK-NEXT:    lsls r2, r1, #30
 ; CHECK-NEXT:    bne .LBB8_3
 ; CHECK-NEXT:  @ %bb.1: @ %entry
-; CHECK-NEXT:    movw r3, #34079
-; CHECK-NEXT:    movt r3, #20971
-; CHECK-NEXT:    smmul r3, r2, r3
-; CHECK-NEXT:    asrs r1, r3, #5
-; CHECK-NEXT:    add.w r1, r1, r3, lsr #31
+; CHECK-NEXT:    movw r2, #34079
+; CHECK-NEXT:    movt r2, #20971
+; CHECK-NEXT:    smmul r2, r1, r2
+; CHECK-NEXT:    asrs r3, r2, #5
+; CHECK-NEXT:    add.w r2, r3, r2, lsr #31
 ; CHECK-NEXT:    movs r3, #100
-; CHECK-NEXT:    mls r1, r1, r3, r2
-; CHECK-NEXT:    cbz r1, .LBB8_3
+; CHECK-NEXT:    mls r2, r2, r3, r1
+; CHECK-NEXT:    cbz r2, .LBB8_3
 ; CHECK-NEXT:  @ %bb.2:
-; CHECK-NEXT:    movs r4, #1
+; CHECK-NEXT:    mov.w r12, #1
 ; CHECK-NEXT:    b .LBB8_4
 ; CHECK-NEXT:  .LBB8_3: @ %lor.rhs
-; CHECK-NEXT:    movw r1, #47184
+; CHECK-NEXT:    movw r2, #47184
 ; CHECK-NEXT:    movw r3, #23593
-; CHECK-NEXT:    movt r1, #1310
+; CHECK-NEXT:    movt r2, #1310
 ; CHECK-NEXT:    movt r3, #49807
-; CHECK-NEXT:    mla r1, r2, r3, r1
+; CHECK-NEXT:    mla r1, r1, r3, r2
 ; CHECK-NEXT:    movw r2, #55051
 ; CHECK-NEXT:    movt r2, #163
 ; CHECK-NEXT:    ror.w r1, r1, #4
 ; CHECK-NEXT:    cmp r1, r2
-; CHECK-NEXT:    cset r4, lo
+; CHECK-NEXT:    cset r12, lo
 ; CHECK-NEXT:  .LBB8_4: @ %lor.end
-; CHECK-NEXT:    ldr.w r3, [r12, #4]
+; CHECK-NEXT:    ldrd r2, r3, [r0]
 ; CHECK-NEXT:    cmp r3, #1
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    poplt {r4, pc}
-; CHECK-NEXT:  .LBB8_5: @ %vector.ph
-; CHECK-NEXT:    adds r1, r3, #3
-; CHECK-NEXT:    movs r2, #1
-; CHECK-NEXT:    bic r1, r1, #3
-; CHECK-NEXT:    subs r1, #4
-; CHECK-NEXT:    add.w lr, r2, r1, lsr #2
-; CHECK-NEXT:    movw r1, :lower16:days
-; CHECK-NEXT:    movt r1, :upper16:days
-; CHECK-NEXT:    movs r2, #52
-; CHECK-NEXT:    mla r1, r4, r2, r1
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    vdup.32 q0, r2
+; CHECK-NEXT:    blt .LBB8_8
+; CHECK-NEXT:  @ %bb.5: @ %vector.ph
+; CHECK-NEXT:    adds r0, r3, #3
+; CHECK-NEXT:    movs r1, #1
+; CHECK-NEXT:    bic r0, r0, #3
+; CHECK-NEXT:    subs r0, #4
+; CHECK-NEXT:    add.w lr, r1, r0, lsr #2
+; CHECK-NEXT:    movw r0, :lower16:days
+; CHECK-NEXT:    movt r0, :upper16:days
+; CHECK-NEXT:    movs r1, #52
+; CHECK-NEXT:    mla r0, r12, r1, r0
+; CHECK-NEXT:    movs r1, #0
+; CHECK-NEXT:    vdup.32 q0, r1
+; CHECK-NEXT:    subs r1, r3, #1
+; CHECK-NEXT:    vmov.32 q0[0], r2
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    subs r0, r3, #1
-; CHECK:  .LBB8_6: @ %vector.body
+; CHECK-NEXT:  .LBB8_6: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vctp.32 r0
+; CHECK-NEXT:    vctp.32 r1
 ; CHECK-NEXT:    vmov q1, q0
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vldrwt.u32 q0, [r1], #16
-; CHECK-NEXT:    subs r0, #4
+; CHECK-NEXT:    vldrwt.u32 q0, [r0], #16
+; CHECK-NEXT:    subs r1, #4
 ; CHECK-NEXT:    vadd.i32 q0, q0, q1
 ; CHECK-NEXT:    le lr, .LBB8_6
 ; CHECK-NEXT:  @ %bb.7: @ %middle.block
 ; CHECK-NEXT:    vpsel q0, q0, q1
-; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    pop {r4, pc}
+; CHECK-NEXT:    vaddv.u32 r2, q0
+; CHECK-NEXT:  .LBB8_8: @ %for.end
+; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    pop {r7, pc}
 entry:
   %day1 = getelementptr inbounds %struct.date, %struct.date* %pd, i32 0, i32 0
   %0 = load i32, i32* %day1, align 4
   %year = getelementptr inbounds %struct.date, %struct.date* %pd, i32 0, i32 2
   %1 = load i32, i32* %year, align 4
   %2 = and i32 %1, 3
   %cmp = icmp ne i32 %2, 0
   %rem3 = srem i32 %1, 100

llvm/test/CodeGen/X86/2007-01-13-StackPtrIndex.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s
 ; PR1103
 
 target datalayout = "e-p:64:64"
 @i6000 = global [128 x i64] zeroinitializer, align 16
 
 
 define void @foo(i32* %a0, i32* %a1, i32* %a2, i32* %a3, i32* %a4, i32* %a5) {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0: # %b
 ; CHECK-NEXT:    pushq %rbp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
 ; CHECK-NEXT:    .cfi_def_cfa_register %rbp
-; CHECK-NEXT:    movslq (%rdi), %rax
+; CHECK-NEXT:    movslq (%rdi), %rdi
 ; CHECK-NEXT:    movslq (%rsi), %r8
 ; CHECK-NEXT:    movslq (%rdx), %r10
-; CHECK-NEXT:    movl (%rcx), %edi
-; CHECK-NEXT:    movslq (%r9), %rcx
-; CHECK-NEXT:    movq %rsp, %rdx
-; CHECK-NEXT:    subl %eax, %r8d
-; CHECK-NEXT:    movslq %r8d, %rsi
+; CHECK-NEXT:    movl (%rcx), %esi
+; CHECK-NEXT:    movq %rsp, %rcx
+; CHECK-NEXT:    subl %edi, %r8d
+; CHECK-NEXT:    movslq %r8d, %rdx
 ; CHECK-NEXT:    js .LBB0_1
 ; CHECK-NEXT:  # %bb.11: # %b63
-; CHECK-NEXT:    testq %rsi, %rsi
+; CHECK-NEXT:    testq %rdx, %rdx
 ; CHECK-NEXT:    js .LBB0_14
 ; CHECK-NEXT:  # %bb.12:
-; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    xorl %edi, %edi
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_13: # %a25b
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    testb %al, %al
+; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_13
 ; CHECK-NEXT:  .LBB0_14: # %b85
 ; CHECK-NEXT:    movb $1, %al
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    jne .LBB0_1
 ; CHECK-NEXT:  # %bb.15:
-; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    xorl %edi, %edi
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_16: # %a25b140
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    testb %al, %al
+; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_16
 ; CHECK-NEXT:  .LBB0_1: # %a29b
-; CHECK-NEXT:    cmpl %r10d, %edi
+; CHECK-NEXT:    cmpl %r10d, %esi
 ; CHECK-NEXT:    js .LBB0_10
 ; CHECK-NEXT:  # %bb.2: # %b158
+; CHECK-NEXT:    movslq (%r9), %rsi
 ; CHECK-NEXT:    xorl %edi, %edi
 ; CHECK-NEXT:    xorps %xmm0, %xmm0
 ; CHECK-NEXT:    movb $1, %r10b
 ; CHECK-NEXT:    jmp .LBB0_3
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_9: # %b1606
 ; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
 ; CHECK-NEXT:    testb %dil, %dil

 ; CHECK-NEXT:    # Child Loop BB0_39 Depth 3
 ; CHECK-NEXT:    # Child Loop BB0_33 Depth 3
 ; CHECK-NEXT:    # Child Loop BB0_34 Depth 2
 ; CHECK-NEXT:    # Child Loop BB0_36 Depth 2
 ; CHECK-NEXT:    testl %r8d, %r8d
 ; CHECK-NEXT:    js .LBB0_4
 ; CHECK-NEXT:  # %bb.17: # %b179
 ; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
-; CHECK-NEXT:    testq %rsi, %rsi
+; CHECK-NEXT:    testq %rdx, %rdx
 ; CHECK-NEXT:    js .LBB0_18
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_37: # %a30b
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_37
 ; CHECK-NEXT:  .LBB0_18: # %b188
 ; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
 ; CHECK-NEXT:    testb %r10b, %r10b
 ; CHECK-NEXT:    jne .LBB0_4
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_19: # %a30b294
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_19
 ; CHECK-NEXT:  .LBB0_4: # %a33b
 ; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
-; CHECK-NEXT:    movl %ecx, %eax
+; CHECK-NEXT:    movl %esi, %eax
 ; CHECK-NEXT:    orl %r8d, %eax
 ; CHECK-NEXT:    movl %eax, %r9d
 ; CHECK-NEXT:    shrl $31, %r9d
 ; CHECK-NEXT:    testl %eax, %eax
 ; CHECK-NEXT:    jns .LBB0_20
 ; CHECK-NEXT:  .LBB0_5: # %a50b
 ; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
 ; CHECK-NEXT:    movl %r8d, %eax
-; CHECK-NEXT:    orl %ecx, %eax
+; CHECK-NEXT:    orl %esi, %eax
 ; CHECK-NEXT:    movl %eax, %r11d
 ; CHECK-NEXT:    shrl $31, %r11d
 ; CHECK-NEXT:    testl %eax, %eax
 ; CHECK-NEXT:    jns .LBB0_26
 ; CHECK-NEXT:  .LBB0_6: # %a57b
 ; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
 ; CHECK-NEXT:    testb %r9b, %r9b
 ; CHECK-NEXT:    je .LBB0_30

 ; CHECK-NEXT:  .LBB0_22: # %b463
 ; CHECK-NEXT:    # in Loop: Header=BB0_20 Depth=2
 ; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_23
 ; CHECK-NEXT:  .LBB0_20: # %b341
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # => This Loop Header: Depth=2
 ; CHECK-NEXT:    # Child Loop BB0_21 Depth 3
-; CHECK-NEXT:    testq %rcx, %rcx
+; CHECK-NEXT:    testq %rsi, %rsi
 ; CHECK-NEXT:    js .LBB0_22
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_21: # %a35b
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # Parent Loop BB0_20 Depth=2
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=3
 ; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_21
 ; CHECK-NEXT:    jmp .LBB0_22
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_28: # %b1016
 ; CHECK-NEXT:    # in Loop: Header=BB0_26 Depth=2
-; CHECK-NEXT:    testq %rcx, %rcx
+; CHECK-NEXT:    testq %rsi, %rsi
 ; CHECK-NEXT:    jle .LBB0_6
 ; CHECK-NEXT:  .LBB0_26: # %b858
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # => This Loop Header: Depth=2
 ; CHECK-NEXT:    # Child Loop BB0_38 Depth 3
 ; CHECK-NEXT:    # Child Loop BB0_29 Depth 3
-; CHECK-NEXT:    testq %rsi, %rsi
+; CHECK-NEXT:    testq %rdx, %rdx
 ; CHECK-NEXT:    js .LBB0_27
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_38: # %a53b
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # Parent Loop BB0_26 Depth=2
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=3
 ; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_38
 ; CHECK-NEXT:  .LBB0_27: # %b879
 ; CHECK-NEXT:    # in Loop: Header=BB0_26 Depth=2
 ; CHECK-NEXT:    testb %r10b, %r10b
 ; CHECK-NEXT:    jne .LBB0_28
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_29: # %a53b1019
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # Parent Loop BB0_26 Depth=2
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=3
-; CHECK-NEXT:    testq %rsi, %rsi
+; CHECK-NEXT:    testq %rdx, %rdx
 ; CHECK-NEXT:    jle .LBB0_29
 ; CHECK-NEXT:    jmp .LBB0_28
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_32: # %b1263
 ; CHECK-NEXT:    # in Loop: Header=BB0_30 Depth=2
-; CHECK-NEXT:    testq %rsi, %rsi
+; CHECK-NEXT:    testq %rdx, %rdx
 ; CHECK-NEXT:    jle .LBB0_7
 ; CHECK-NEXT:  .LBB0_30: # %b1117
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # => This Loop Header: Depth=2
 ; CHECK-NEXT:    # Child Loop BB0_39 Depth 3
 ; CHECK-NEXT:    # Child Loop BB0_33 Depth 3
-; CHECK-NEXT:    testq %rcx, %rcx
+; CHECK-NEXT:    testq %rsi, %rsi
 ; CHECK-NEXT:    js .LBB0_31
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_39: # %a63b
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # Parent Loop BB0_30 Depth=2
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=3
-; CHECK-NEXT:    testq %rcx, %rcx
+; CHECK-NEXT:    testq %rsi, %rsi
 ; CHECK-NEXT:    jle .LBB0_39
 ; CHECK-NEXT:  .LBB0_31: # %b1139
 ; CHECK-NEXT:    # in Loop: Header=BB0_30 Depth=2
-; CHECK-NEXT:    testq %rcx, %rcx
+; CHECK-NEXT:    testq %rsi, %rsi
 ; CHECK-NEXT:    jle .LBB0_32
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_33: # %a63b1266
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # Parent Loop BB0_30 Depth=2
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=3
-; CHECK-NEXT:    testq %rcx, %rcx
+; CHECK-NEXT:    testq %rsi, %rsi
 ; CHECK-NEXT:    jle .LBB0_33
 ; CHECK-NEXT:    jmp .LBB0_32
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_25: # %b712
 ; CHECK-NEXT:    # in Loop: Header=BB0_23 Depth=2
 ; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_5
 ; CHECK-NEXT:  .LBB0_23: # %b535
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # => This Loop Header: Depth=2
 ; CHECK-NEXT:    # Child Loop BB0_24 Depth 3
-; CHECK-NEXT:    testq %rsi, %rsi
+; CHECK-NEXT:    testq %rdx, %rdx
 ; CHECK-NEXT:    js .LBB0_25
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_24: # %a45b
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # Parent Loop BB0_23 Depth=2
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=3
 ; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    je .LBB0_24

llvm/test/CodeGen/X86/MachineSink-eflags.ll

 %4 = type <{ %5*, i8*, i32, i32, [4 x i64], [4 x i64], [4 x i64], [4 x i64], [4 x i64] }>
 %5 = type <{ void (i32)*, i8*, i32 (i8*, ...)* }>
 
 define void @foo(i8* nocapture %_stubArgs) nounwind {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    subq $152, %rsp
 ; CHECK-NEXT:    movq 48(%rdi), %rax
-; CHECK-NEXT:    movl 64(%rdi), %edx
+; CHECK-NEXT:    movl 64(%rdi), %ecx
 ; CHECK-NEXT:    movl $200, %esi
 ; CHECK-NEXT:    addl 68(%rdi), %esi
-; CHECK-NEXT:    imull $46, %edx, %ecx
-; CHECK-NEXT:    addq %rsi, %rcx
-; CHECK-NEXT:    shlq $4, %rcx
-; CHECK-NEXT:    imull $47, %edx, %edx
+; CHECK-NEXT:    imull $46, %ecx, %edx
 ; CHECK-NEXT:    addq %rsi, %rdx
 ; CHECK-NEXT:    shlq $4, %rdx
-; CHECK-NEXT:    movaps (%rax,%rdx), %xmm0
+; CHECK-NEXT:    imull $47, %ecx, %ecx
+; CHECK-NEXT:    addq %rsi, %rcx
+; CHECK-NEXT:    shlq $4, %rcx
 ; CHECK-NEXT:    cmpl $0, (%rdi)
 ; CHECK-NEXT:    jne .LBB0_1
 ; CHECK-NEXT:  # %bb.2: # %entry
-; CHECK-NEXT:    xorps %xmm1, %xmm1
-; CHECK-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    je .LBB0_4
-; CHECK-NEXT:    jmp .LBB0_5
+; CHECK-NEXT:    xorps %xmm0, %xmm0
+; CHECK-NEXT:    jmp .LBB0_3
 ; CHECK-NEXT:  .LBB0_1:
+; CHECK-NEXT:    movaps (%rax,%rdx), %xmm0
+; CHECK-NEXT:  .LBB0_3: # %entry
 ; CHECK-NEXT:    movaps (%rax,%rcx), %xmm1
-; CHECK-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    jne .LBB0_5
-; CHECK-NEXT:  .LBB0_4: # %entry
-; CHECK-NEXT:    xorps %xmm0, %xmm0
+; CHECK-NEXT:  # %bb.4: # %entry
+; CHECK-NEXT:    xorps %xmm1, %xmm1
 ; CHECK-NEXT:  .LBB0_5: # %entry
-; CHECK-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    addq $152, %rsp
 ; CHECK-NEXT:    retq
 entry:
  %i0 = alloca i8*, align 8
  %i2 = alloca i8*, align 8
  %b.i = alloca [16 x <2 x double>], align 16
  %conv = bitcast i8* %_stubArgs to i32*
  %tmp1 = load i32, i32* %conv, align 4

llvm/test/CodeGen/X86/avx2-masked-gather.ll

 ; X64-NEXT:    vextracti128 $1, %ymm0, %xmm5
 ; X64-NEXT:    vpgatherqd %xmm5, (,%ymm3), %xmm4
 ; X64-NEXT:    vpgatherqd %xmm0, (,%ymm2), %xmm1
 ; X64-NEXT:    vinserti128 $1, %xmm4, %ymm1, %ymm0
 ; X64-NEXT:    retq
 ;
 ; NOGATHER-LABEL: masked_gather_v8i32:
 ; NOGATHER:       # %bb.0: # %entry
-; NOGATHER-NEXT:    vmovdqa (%rdi), %ymm3
-; NOGATHER-NEXT:    vmovdqa 32(%rdi), %ymm2
+; NOGATHER-NEXT:    vmovdqa (%rdi), %ymm2
 ; NOGATHER-NEXT:    vpsllw $15, %xmm0, %xmm0
 ; NOGATHER-NEXT:    vpacksswb %xmm0, %xmm0, %xmm0
 ; NOGATHER-NEXT:    vpmovmskb %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB6_2
 ; NOGATHER-NEXT:  # %bb.1: # %cond.load
-; NOGATHER-NEXT:    vmovq %xmm3, %rcx
+; NOGATHER-NEXT:    vmovq %xmm2, %rcx
 ; NOGATHER-NEXT:    vpinsrd $0, (%rcx), %xmm1, %xmm0
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm0[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB6_2: # %else
 ; NOGATHER-NEXT:    testb $2, %al
 ; NOGATHER-NEXT:    je .LBB6_4
 ; NOGATHER-NEXT:  # %bb.3: # %cond.load1
-; NOGATHER-NEXT:    vpextrq $1, %xmm3, %rcx
+; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rcx
 ; NOGATHER-NEXT:    vpinsrd $1, (%rcx), %xmm1, %xmm0
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm0[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB6_4: # %else2
-; NOGATHER-NEXT:    vextractf128 $1, %ymm3, %xmm0
+; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm0
 ; NOGATHER-NEXT:    testb $4, %al
-; NOGATHER-NEXT:    jne .LBB6_5
-; NOGATHER-NEXT:  # %bb.6: # %else5
+; NOGATHER-NEXT:    je .LBB6_6
+; NOGATHER-NEXT:  # %bb.5: # %cond.load4
+; NOGATHER-NEXT:    vmovq %xmm0, %rcx
+; NOGATHER-NEXT:    vpinsrd $2, (%rcx), %xmm1, %xmm2
+; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm2[0,1,2,3],ymm1[4,5,6,7]
+; NOGATHER-NEXT:  .LBB6_6: # %else5
 ; NOGATHER-NEXT:    testb $8, %al
-; NOGATHER-NEXT:    jne .LBB6_7
+; NOGATHER-NEXT:    je .LBB6_8
+; NOGATHER-NEXT:  # %bb.7: # %cond.load7
+; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rcx
+; NOGATHER-NEXT:    vpinsrd $3, (%rcx), %xmm1, %xmm0
+; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm0[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB6_8: # %else8
+; NOGATHER-NEXT:    vmovdqa 32(%rdi), %ymm0
 ; NOGATHER-NEXT:    testb $16, %al
-; NOGATHER-NEXT:    jne .LBB6_9
+; NOGATHER-NEXT:    je .LBB6_10
+; NOGATHER-NEXT:  # %bb.9: # %cond.load10
+; NOGATHER-NEXT:    vmovq %xmm0, %rcx
+; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; NOGATHER-NEXT:    vpinsrd $0, (%rcx), %xmm2, %xmm2
+; NOGATHER-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB6_10: # %else11
 ; NOGATHER-NEXT:    testb $32, %al
 ; NOGATHER-NEXT:    je .LBB6_12
-; NOGATHER-NEXT:  .LBB6_11: # %cond.load13
-; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rcx
-; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm0
-; NOGATHER-NEXT:    vpinsrd $1, (%rcx), %xmm0, %xmm0
-; NOGATHER-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm1
+; NOGATHER-NEXT:  # %bb.11: # %cond.load13
+; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rcx
+; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; NOGATHER-NEXT:    vpinsrd $1, (%rcx), %xmm2, %xmm2
+; NOGATHER-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB6_12: # %else14
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm0
+; NOGATHER-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; NOGATHER-NEXT:    testb $64, %al
 ; NOGATHER-NEXT:    jne .LBB6_13
 ; NOGATHER-NEXT:  # %bb.14: # %else17
 ; NOGATHER-NEXT:    testb $-128, %al
 ; NOGATHER-NEXT:    jne .LBB6_15
 ; NOGATHER-NEXT:  .LBB6_16: # %else20
 ; NOGATHER-NEXT:    vmovaps %ymm1, %ymm0
 ; NOGATHER-NEXT:    retq
-; NOGATHER-NEXT:  .LBB6_5: # %cond.load4
-; NOGATHER-NEXT:    vmovq %xmm0, %rcx
-; NOGATHER-NEXT:    vpinsrd $2, (%rcx), %xmm1, %xmm3
-; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm3[0,1,2,3],ymm1[4,5,6,7]
-; NOGATHER-NEXT:    testb $8, %al
-; NOGATHER-NEXT:    je .LBB6_8
-; NOGATHER-NEXT:  .LBB6_7: # %cond.load7
-; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rcx
-; NOGATHER-NEXT:    vpinsrd $3, (%rcx), %xmm1, %xmm0
-; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm0[0,1,2,3],ymm1[4,5,6,7]
-; NOGATHER-NEXT:    testb $16, %al
-; NOGATHER-NEXT:    je .LBB6_10
-; NOGATHER-NEXT:  .LBB6_9: # %cond.load10
-; NOGATHER-NEXT:    vmovq %xmm2, %rcx
-; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm0
-; NOGATHER-NEXT:    vpinsrd $0, (%rcx), %xmm0, %xmm0
-; NOGATHER-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm1
-; NOGATHER-NEXT:    testb $32, %al
-; NOGATHER-NEXT:    jne .LBB6_11
-; NOGATHER-NEXT:    jmp .LBB6_12
 ; NOGATHER-NEXT:  .LBB6_13: # %cond.load16
 ; NOGATHER-NEXT:    vmovq %xmm0, %rcx
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; NOGATHER-NEXT:    vpinsrd $2, (%rcx), %xmm2, %xmm2
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; NOGATHER-NEXT:    testb $-128, %al
 ; NOGATHER-NEXT:    je .LBB6_16
 ; NOGATHER-NEXT:  .LBB6_15: # %cond.load19

 ; X64-NEXT:    vextracti128 $1, %ymm0, %xmm5
 ; X64-NEXT:    vgatherqps %xmm5, (,%ymm3), %xmm4
 ; X64-NEXT:    vgatherqps %xmm0, (,%ymm2), %xmm1
 ; X64-NEXT:    vinsertf128 $1, %xmm4, %ymm1, %ymm0
 ; X64-NEXT:    retq
 ;
 ; NOGATHER-LABEL: masked_gather_v8float:
 ; NOGATHER:       # %bb.0: # %entry
-; NOGATHER-NEXT:    vmovdqa (%rdi), %ymm3
-; NOGATHER-NEXT:    vmovdqa 32(%rdi), %ymm2
+; NOGATHER-NEXT:    vmovdqa (%rdi), %ymm2
 ; NOGATHER-NEXT:    vpsllw $15, %xmm0, %xmm0
 ; NOGATHER-NEXT:    vpacksswb %xmm0, %xmm0, %xmm0
 ; NOGATHER-NEXT:    vpmovmskb %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB7_2
 ; NOGATHER-NEXT:  # %bb.1: # %cond.load
-; NOGATHER-NEXT:    vmovq %xmm3, %rcx
+; NOGATHER-NEXT:    vmovq %xmm2, %rcx
 ; NOGATHER-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm0[0],ymm1[1,2,3,4,5,6,7]
 ; NOGATHER-NEXT:  .LBB7_2: # %else
 ; NOGATHER-NEXT:    testb $2, %al
 ; NOGATHER-NEXT:    je .LBB7_4
 ; NOGATHER-NEXT:  # %bb.3: # %cond.load1
-; NOGATHER-NEXT:    vpextrq $1, %xmm3, %rcx
+; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rcx
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm0 = xmm1[0],mem[0],xmm1[2,3]
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm0[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB7_4: # %else2
-; NOGATHER-NEXT:    vextractf128 $1, %ymm3, %xmm0
+; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm0
 ; NOGATHER-NEXT:    testb $4, %al
-; NOGATHER-NEXT:    jne .LBB7_5
-; NOGATHER-NEXT:  # %bb.6: # %else5
+; NOGATHER-NEXT:    je .LBB7_6
+; NOGATHER-NEXT:  # %bb.5: # %cond.load4
+; NOGATHER-NEXT:    vmovq %xmm0, %rcx
+; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm2 = xmm1[0,1],mem[0],xmm1[3]
+; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm2[0,1,2,3],ymm1[4,5,6,7]
+; NOGATHER-NEXT:  .LBB7_6: # %else5
 ; NOGATHER-NEXT:    testb $8, %al
-; NOGATHER-NEXT:    jne .LBB7_7
+; NOGATHER-NEXT:    je .LBB7_8
+; NOGATHER-NEXT:  # %bb.7: # %cond.load7
+; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rcx
+; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm0 = xmm1[0,1,2],mem[0]
+; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm0[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB7_8: # %else8
+; NOGATHER-NEXT:    vmovdqa 32(%rdi), %ymm0
 ; NOGATHER-NEXT:    testb $16, %al
-; NOGATHER-NEXT:    jne .LBB7_9
+; NOGATHER-NEXT:    je .LBB7_10
+; NOGATHER-NEXT:  # %bb.9: # %cond.load10
+; NOGATHER-NEXT:    vmovq %xmm0, %rcx
+; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; NOGATHER-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
+; NOGATHER-NEXT:    vblendps {{.*#+}} xmm2 = xmm3[0],xmm2[1,2,3]
+; NOGATHER-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB7_10: # %else11
 ; NOGATHER-NEXT:    testb $32, %al
 ; NOGATHER-NEXT:    je .LBB7_12
-; NOGATHER-NEXT:  .LBB7_11: # %cond.load13
-; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rcx
-; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm0
-; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[2,3]
-; NOGATHER-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm1
+; NOGATHER-NEXT:  # %bb.11: # %cond.load13
+; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rcx
+; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0],mem[0],xmm2[2,3]
+; NOGATHER-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB7_12: # %else14
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm0
+; NOGATHER-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; NOGATHER-NEXT:    testb $64, %al
 ; NOGATHER-NEXT:    jne .LBB7_13
 ; NOGATHER-NEXT:  # %bb.14: # %else17
 ; NOGATHER-NEXT:    testb $-128, %al
 ; NOGATHER-NEXT:    jne .LBB7_15
 ; NOGATHER-NEXT:  .LBB7_16: # %else20
 ; NOGATHER-NEXT:    vmovaps %ymm1, %ymm0
 ; NOGATHER-NEXT:    retq
-; NOGATHER-NEXT:  .LBB7_5: # %cond.load4
-; NOGATHER-NEXT:    vmovq %xmm0, %rcx
-; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm3 = xmm1[0,1],mem[0],xmm1[3]
-; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm3[0,1,2,3],ymm1[4,5,6,7]
-; NOGATHER-NEXT:    testb $8, %al
-; NOGATHER-NEXT:    je .LBB7_8
-; NOGATHER-NEXT:  .LBB7_7: # %cond.load7
-; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rcx
-; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm0 = xmm1[0,1,2],mem[0]
-; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm0[0,1,2,3],ymm1[4,5,6,7]
-; NOGATHER-NEXT:    testb $16, %al
-; NOGATHER-NEXT:    je .LBB7_10
-; NOGATHER-NEXT:  .LBB7_9: # %cond.load10
-; NOGATHER-NEXT:    vmovq %xmm2, %rcx
-; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm0
-; NOGATHER-NEXT:    vmovd {{.*#+}} xmm3 = mem[0],zero,zero,zero
-; NOGATHER-NEXT:    vpblendw {{.*#+}} xmm0 = xmm3[0,1],xmm0[2,3,4,5,6,7]
-; NOGATHER-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm1
-; NOGATHER-NEXT:    testb $32, %al
-; NOGATHER-NEXT:    jne .LBB7_11
-; NOGATHER-NEXT:    jmp .LBB7_12
 ; NOGATHER-NEXT:  .LBB7_13: # %cond.load16
 ; NOGATHER-NEXT:    vmovq %xmm0, %rcx
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],mem[0],xmm2[3]
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; NOGATHER-NEXT:    testb $-128, %al
 ; NOGATHER-NEXT:    je .LBB7_16
 ; NOGATHER-NEXT:  .LBB7_15: # %cond.load19

llvm/test/CodeGen/X86/cmovcmov.ll

 ; NOCMOV-NEXT:    pushl %edi
 ; NOCMOV-NEXT:    pushl %esi
 ; NOCMOV-NEXT:    flds {{[0-9]+}}(%esp)
 ; NOCMOV-NEXT:    flds {{[0-9]+}}(%esp)
 ; NOCMOV-NEXT:    fucompp
 ; NOCMOV-NEXT:    fnstsw %ax
 ; NOCMOV-NEXT:    # kill: def $ah killed $ah killed $ax
 ; NOCMOV-NEXT:    sahf
-; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %eax
+; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %ecx
 ; NOCMOV-NEXT:    jne .LBB4_3
 ; NOCMOV-NEXT:  # %bb.1: # %entry
 ; NOCMOV-NEXT:    jp .LBB4_3
 ; NOCMOV-NEXT:  # %bb.2: # %entry
-; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %eax
+; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %ecx
 ; NOCMOV-NEXT:  .LBB4_3: # %entry
-; NOCMOV-NEXT:    movl (%eax), %ecx
 ; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %edx
 ; NOCMOV-NEXT:    jne .LBB4_6
 ; NOCMOV-NEXT:  # %bb.4: # %entry
 ; NOCMOV-NEXT:    jp .LBB4_6
 ; NOCMOV-NEXT:  # %bb.5: # %entry
 ; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %edx
 ; NOCMOV-NEXT:  .LBB4_6: # %entry
 ; NOCMOV-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; NOCMOV-NEXT:    movl (%edx), %edx
 ; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %esi
 ; NOCMOV-NEXT:    jne .LBB4_9
 ; NOCMOV-NEXT:  # %bb.7: # %entry
 ; NOCMOV-NEXT:    jp .LBB4_9
 ; NOCMOV-NEXT:  # %bb.8: # %entry
 ; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %esi
 ; NOCMOV-NEXT:  .LBB4_9: # %entry
+; NOCMOV-NEXT:    movl (%ecx), %ecx
+; NOCMOV-NEXT:    movl (%edx), %edx
 ; NOCMOV-NEXT:    movl (%esi), %esi
 ; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %edi
 ; NOCMOV-NEXT:    jne .LBB4_12
 ; NOCMOV-NEXT:  # %bb.10: # %entry
 ; NOCMOV-NEXT:    jp .LBB4_12
 ; NOCMOV-NEXT:  # %bb.11: # %entry
 ; NOCMOV-NEXT:    leal {{[0-9]+}}(%esp), %edi
 ; NOCMOV-NEXT:  .LBB4_12: # %entry

llvm/test/CodeGen/X86/select.ll

 ; MCU:       # %bb.0:
 ; MCU-NEXT:    pushl %ebp
 ; MCU-NEXT:    pushl %ebx
 ; MCU-NEXT:    pushl %edi
 ; MCU-NEXT:    pushl %esi
 ; MCU-NEXT:    testb $1, %al
 ; MCU-NEXT:    jne .LBB7_1
 ; MCU-NEXT:  # %bb.2:
-; MCU-NEXT:    leal {{[0-9]+}}(%esp), %eax
-; MCU-NEXT:    movl (%eax), %eax
+; MCU-NEXT:    leal {{[0-9]+}}(%esp), %edi
 ; MCU-NEXT:    je .LBB7_5
 ; MCU-NEXT:  .LBB7_4:
 ; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ecx
-; MCU-NEXT:    movl (%ecx), %ecx
 ; MCU-NEXT:    je .LBB7_8
 ; MCU-NEXT:  .LBB7_7:
 ; MCU-NEXT:    leal {{[0-9]+}}(%esp), %esi
-; MCU-NEXT:    movl (%esi), %esi
 ; MCU-NEXT:    je .LBB7_11
 ; MCU-NEXT:  .LBB7_10:
-; MCU-NEXT:    leal {{[0-9]+}}(%esp), %edi
-; MCU-NEXT:    movl (%edi), %edi
+; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ebp
 ; MCU-NEXT:    je .LBB7_14
 ; MCU-NEXT:  .LBB7_13:
-; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ebx
-; MCU-NEXT:    movl (%ebx), %ebx
-; MCU-NEXT:    je .LBB7_17
-; MCU-NEXT:  .LBB7_16:
-; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ebp
-; MCU-NEXT:    jmp .LBB7_18
-; MCU-NEXT:  .LBB7_1:
 ; MCU-NEXT:    leal {{[0-9]+}}(%esp), %eax
-; MCU-NEXT:    movl (%eax), %eax
+; MCU-NEXT:    jmp .LBB7_15
+; MCU-NEXT:  .LBB7_1:
+; MCU-NEXT:    leal {{[0-9]+}}(%esp), %edi
 ; MCU-NEXT:    jne .LBB7_4
 ; MCU-NEXT:  .LBB7_5:
 ; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ecx
-; MCU-NEXT:    movl (%ecx), %ecx
 ; MCU-NEXT:    jne .LBB7_7
 ; MCU-NEXT:  .LBB7_8:
 ; MCU-NEXT:    leal {{[0-9]+}}(%esp), %esi
-; MCU-NEXT:    movl (%esi), %esi
 ; MCU-NEXT:    jne .LBB7_10
 ; MCU-NEXT:  .LBB7_11:
-; MCU-NEXT:    leal {{[0-9]+}}(%esp), %edi
-; MCU-NEXT:    movl (%edi), %edi
+; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ebp
 ; MCU-NEXT:    jne .LBB7_13
 ; MCU-NEXT:  .LBB7_14:
-; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ebx
-; MCU-NEXT:    movl (%ebx), %ebx
+; MCU-NEXT:    leal {{[0-9]+}}(%esp), %eax
+; MCU-NEXT:  .LBB7_15:
+; MCU-NEXT:    movl (%edi), %ebx
+; MCU-NEXT:    movl (%ecx), %edi
+; MCU-NEXT:    movl (%esi), %esi
+; MCU-NEXT:    movl (%ebp), %ecx
+; MCU-NEXT:    movl (%eax), %eax
 ; MCU-NEXT:    jne .LBB7_16
-; MCU-NEXT:  .LBB7_17:
+; MCU-NEXT:  # %bb.17:
+; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ebp
+; MCU-NEXT:    jmp .LBB7_18
+; MCU-NEXT:  .LBB7_16:
 ; MCU-NEXT:    leal {{[0-9]+}}(%esp), %ebp
 ; MCU-NEXT:  .LBB7_18:
 ; MCU-NEXT:    movl (%ebp), %ebp
 ; MCU-NEXT:    decl %ebp
-; MCU-NEXT:    decl %ebx
-; MCU-NEXT:    decl %edi
-; MCU-NEXT:    decl %esi
-; MCU-NEXT:    decl %ecx
 ; MCU-NEXT:    decl %eax
-; MCU-NEXT:    movl %eax, 20(%edx)
-; MCU-NEXT:    movl %ecx, 16(%edx)
+; MCU-NEXT:    decl %ecx
+; MCU-NEXT:    decl %esi
+; MCU-NEXT:    decl %edi
+; MCU-NEXT:    decl %ebx
+; MCU-NEXT:    movl %ebx, 20(%edx)
+; MCU-NEXT:    movl %edi, 16(%edx)
 ; MCU-NEXT:    movl %esi, 12(%edx)
-; MCU-NEXT:    movl %edi, 8(%edx)
-; MCU-NEXT:    movl %ebx, 4(%edx)
+; MCU-NEXT:    movl %ecx, 8(%edx)
+; MCU-NEXT:    movl %eax, 4(%edx)
 ; MCU-NEXT:    movl %ebp, (%edx)
 ; MCU-NEXT:    popl %esi
 ; MCU-NEXT:    popl %edi
 ; MCU-NEXT:    popl %ebx
 ; MCU-NEXT:    popl %ebp
 ; MCU-NEXT:    retl
   %x = select i1 %c, <6 x i32> %src1, <6 x i32> %src2
   %val = sub <6 x i32> %x, < i32 1, i32 1, i32 1, i32 1, i32 1, i32 1 >

llvm/test/CodeGen/X86/vec_int_to_fp.ll

 
 ;
 ; Load Unsigned Integer to Float
 ;
 
 define <4 x float> @uitofp_load_4i64_to_4f32(<4 x i64> *%a) {
 ; SSE2-LABEL: uitofp_load_4i64_to_4f32:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    movdqa (%rdi), %xmm2
 ; SSE2-NEXT:    movdqa 16(%rdi), %xmm0
 ; SSE2-NEXT:    movq %xmm0, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB83_1
 ; SSE2-NEXT:  # %bb.2:
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm1
 ; SSE2-NEXT:    jmp .LBB83_3
 ; SSE2-NEXT:  .LBB83_1:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm1
 ; SSE2-NEXT:    addss %xmm1, %xmm1
 ; SSE2-NEXT:  .LBB83_3:
+; SSE2-NEXT:    movdqa (%rdi), %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
 ; SSE2-NEXT:    movq %xmm0, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB83_4
 ; SSE2-NEXT:  # %bb.5:
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm3
 ; SSE2-NEXT:    jmp .LBB83_6
 ; SSE2-NEXT:  .LBB83_4:
; AVX-NEXT:    retq
   %ld = load <4 x i8>, <4 x i8> *%a
   %cvt = uitofp <4 x i8> %ld to <4 x float>
   ret <4 x float> %cvt
 }
 
 define <8 x float> @uitofp_load_8i64_to_8f32(<8 x i64> *%a) {
 ; SSE2-LABEL: uitofp_load_8i64_to_8f32:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    movdqa (%rdi), %xmm5
 ; SSE2-NEXT:    movdqa 16(%rdi), %xmm0
-; SSE2-NEXT:    movdqa 32(%rdi), %xmm2
-; SSE2-NEXT:    movdqa 48(%rdi), %xmm1
 ; SSE2-NEXT:    movq %xmm0, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB87_1
 ; SSE2-NEXT:  # %bb.2:
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm3
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm2
 ; SSE2-NEXT:    jmp .LBB87_3
 ; SSE2-NEXT:  .LBB87_1:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm3
-; SSE2-NEXT:    addss %xmm3, %xmm3
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm2
+; SSE2-NEXT:    addss %xmm2, %xmm2
 ; SSE2-NEXT:  .LBB87_3:
+; SSE2-NEXT:    movdqa (%rdi), %xmm3
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
 ; SSE2-NEXT:    movq %xmm0, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB87_4
 ; SSE2-NEXT:  # %bb.5:
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm4
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm1
 ; SSE2-NEXT:    jmp .LBB87_6
 ; SSE2-NEXT:  .LBB87_4:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm4
-; SSE2-NEXT:    addss %xmm4, %xmm4
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm1
+; SSE2-NEXT:    addss %xmm1, %xmm1
 ; SSE2-NEXT:  .LBB87_6:
-; SSE2-NEXT:    movq %xmm5, %rax
+; SSE2-NEXT:    movq %xmm3, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB87_7
 ; SSE2-NEXT:  # %bb.8:
 ; SSE2-NEXT:    xorps %xmm0, %xmm0
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm0
 ; SSE2-NEXT:    jmp .LBB87_9
 ; SSE2-NEXT:  .LBB87_7:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
 ; SSE2-NEXT:    xorps %xmm0, %xmm0
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm0
 ; SSE2-NEXT:    addss %xmm0, %xmm0
 ; SSE2-NEXT:  .LBB87_9:
-; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[2,3,2,3]
-; SSE2-NEXT:    movq %xmm5, %rax
+; SSE2-NEXT:    movdqa 48(%rdi), %xmm6
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm3[2,3,2,3]
+; SSE2-NEXT:    movq %xmm3, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB87_10
 ; SSE2-NEXT:  # %bb.11:
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm6
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm4
 ; SSE2-NEXT:    jmp .LBB87_12
 ; SSE2-NEXT:  .LBB87_10:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm6
-; SSE2-NEXT:    addss %xmm6, %xmm6
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm4
+; SSE2-NEXT:    addss %xmm4, %xmm4
 ; SSE2-NEXT:  .LBB87_12:
-; SSE2-NEXT:    movq %xmm1, %rax
+; SSE2-NEXT:    movq %xmm6, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB87_13
 ; SSE2-NEXT:  # %bb.14:
-; SSE2-NEXT:    xorps %xmm5, %xmm5
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm5
+; SSE2-NEXT:    xorps %xmm3, %xmm3
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm3
 ; SSE2-NEXT:    jmp .LBB87_15
 ; SSE2-NEXT:  .LBB87_13:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
-; SSE2-NEXT:    xorps %xmm5, %xmm5
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm5
-; SSE2-NEXT:    addss %xmm5, %xmm5
+; SSE2-NEXT:    xorps %xmm3, %xmm3
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm3
+; SSE2-NEXT:    addss %xmm3, %xmm3
 ; SSE2-NEXT:  .LBB87_15:
-; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,2,3]
-; SSE2-NEXT:    movq %xmm1, %rax
+; SSE2-NEXT:    movdqa 32(%rdi), %xmm5
+; SSE2-NEXT:    pshufd {{.*#+}} xmm6 = xmm6[2,3,2,3]
+; SSE2-NEXT:    movq %xmm6, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB87_16
 ; SSE2-NEXT:  # %bb.17:
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm7
+; SSE2-NEXT:    xorps %xmm6, %xmm6
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm6
 ; SSE2-NEXT:    jmp .LBB87_18
 ; SSE2-NEXT:  .LBB87_16:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
-; SSE2-NEXT:    cvtsi2ss %rax, %xmm7
-; SSE2-NEXT:    addss %xmm7, %xmm7
+; SSE2-NEXT:    xorps %xmm6, %xmm6
+; SSE2-NEXT:    cvtsi2ss %rax, %xmm6
+; SSE2-NEXT:    addss %xmm6, %xmm6
 ; SSE2-NEXT:  .LBB87_18:
-; SSE2-NEXT:    unpcklps {{.*#+}} xmm3 = xmm3[0],xmm4[0],xmm3[1],xmm4[1]
-; SSE2-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm6[0],xmm0[1],xmm6[1]
-; SSE2-NEXT:    movq %xmm2, %rax
+; SSE2-NEXT:    unpcklps {{.*#+}} xmm2 = xmm2[0],xmm1[0],xmm2[1],xmm1[1]
+; SSE2-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm4[0],xmm0[1],xmm4[1]
+; SSE2-NEXT:    movq %xmm5, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB87_19
 ; SSE2-NEXT:  # %bb.20:
 ; SSE2-NEXT:    xorps %xmm1, %xmm1
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm1
 ; SSE2-NEXT:    jmp .LBB87_21
 ; SSE2-NEXT:  .LBB87_19:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
 ; SSE2-NEXT:    xorps %xmm1, %xmm1
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm1
 ; SSE2-NEXT:    addss %xmm1, %xmm1
 ; SSE2-NEXT:  .LBB87_21:
-; SSE2-NEXT:    movlhps {{.*#+}} xmm0 = xmm0[0],xmm3[0]
-; SSE2-NEXT:    unpcklps {{.*#+}} xmm5 = xmm5[0],xmm7[0],xmm5[1],xmm7[1]
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,2,3]
+; SSE2-NEXT:    movlhps {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; SSE2-NEXT:    unpcklps {{.*#+}} xmm3 = xmm3[0],xmm6[0],xmm3[1],xmm6[1]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm5[2,3,2,3]
 ; SSE2-NEXT:    movq %xmm2, %rax
 ; SSE2-NEXT:    testq %rax, %rax
 ; SSE2-NEXT:    js .LBB87_22
 ; SSE2-NEXT:  # %bb.23:
 ; SSE2-NEXT:    xorps %xmm2, %xmm2
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm2
 ; SSE2-NEXT:    jmp .LBB87_24
 ; SSE2-NEXT:  .LBB87_22:
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
 ; SSE2-NEXT:    xorps %xmm2, %xmm2
 ; SSE2-NEXT:    cvtsi2ss %rax, %xmm2
 ; SSE2-NEXT:    addss %xmm2, %xmm2
 ; SSE2-NEXT:  .LBB87_24:
 ; SSE2-NEXT:    unpcklps {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
-; SSE2-NEXT:    movlhps {{.*#+}} xmm1 = xmm1[0],xmm5[0]
+; SSE2-NEXT:    movlhps {{.*#+}} xmm1 = xmm1[0],xmm3[0]
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: uitofp_load_8i64_to_8f32:
 ; SSE41:       # %bb.0:
 ; SSE41-NEXT:    movdqa (%rdi), %xmm4
 ; SSE41-NEXT:    movdqa 16(%rdi), %xmm5
 ; SSE41-NEXT:    movdqa 32(%rdi), %xmm6
 ; SSE41-NEXT:    movdqa 48(%rdi), %xmm2