[AMDGPU] Add MachineDCE pass after RenameIndependentSubregs

[rampitec]

Summary

Detect dead lanes can create some dead defs. Then RenameIndependentSubregs
will break a REG_SEQUENCE which may use these dead defs. At this point
a dead instruction can be removed but we do not run a DCE anymore.

MachineDCE was only running before live variable analysis. The patch
adds a mean to preserve LiveIntervals and SlotIndexes in case it works
past this.

[arsenm]

I feel like we just keep spamming DCE runs to solve issues. Is there some reason it's difficult for RenameIndependentSubregs to clean up after itself?

[arsenm]

In D59626#1438255, @arsenm wrote:

I feel like we just keep spamming DCE runs to solve issues. Is there some reason it's difficult for RenameIndependentSubregs to clean up after itself?

Also, I think making more passes to have to care about LiveIntervals should generally be avoided

[rampitec]

In D59626#1438258, @arsenm wrote:

In D59626#1438255, @arsenm wrote:

I feel like we just keep spamming DCE runs to solve issues. Is there some reason it's difficult for RenameIndependentSubregs to clean up after itself?

Also, I think making more passes to have to care about LiveIntervals should generally be avoided

It is not only RenameIndependentSubregs, detect dead lanes also just mark them dead. I guess the assumption was it will be later deadcoded. And that never happened.
Even if both passes will cleanup after themselves both will need to handle LIS anyway.

[arsenm]

lib/CodeGen/DeadMachineInstructionElim.cpp

193–197

Can you just use shrinkToUses?

[rampitec]

lib/CodeGen/DeadMachineInstructionElim.cpp

193–197

I am afraid not. It does far less than needed. In particular it does not handle defs, and this cannot be solved by just removing a dead interval.

[rampitec]

Rebased.

[rampitec]

Ping.

Since the primary goal of the DCE here is to clean dead subregs I can bail this pass invocation if subreg liveness if not supported by target. What do you think?

[rampitec]

In D59626#1449373, @rampitec wrote:

Ping.

Since the primary goal of the DCE here is to clean dead subregs I can bail this pass invocation if subreg liveness if not supported by target. What do you think?

In fact the BE affected most if x86. Adding @craig.topper as a reviewer. At the same time x86 does not override subRegLivenessEnabled, so it should not be limited to targets with subreg liveness enabled only.

[craig.topper]

test/CodeGen/X86/combine-bitselect.ll

657

This test should be fixed to not be an infinite loop. The middle end wouldn't have given this code to the backend.

test/CodeGen/X86/speculative-load-hardening-gather.ll

10

This looks like speculative load hardening is creating an instruction it doesn't end up using. The speculative loading hardening pass runs after the previous times the MachineDCE was ran. Maybe we should fix the pass to not insert dead instructions.

test/CodeGen/X86/speculative-load-hardening-indirect.ll

134

This looks like new regular expressioning the script wasn't doing previously. Can you drop this from this from your diff or rerun the script on an unmodified trunk and commit it.

test/CodeGen/X86/tail-dup-merge-loop-headers.ll

262

This basic block isn't reachable and is keeping other code alive. That would normally have been removed by IR passes.

[rampitec]

test/CodeGen/X86/combine-bitselect.ll

657

Does it change anything? The test has no side effects anyway.
I am not really sure what does it test, so I am a little afraid to touch it. I can add an option not run extra DCE pass here if needed.

test/CodeGen/X86/speculative-load-hardening-indirect.ll

134

Will do.

test/CodeGen/X86/tail-dup-merge-loop-headers.ll

262

Do I conclude x86 does not really need this pass, since most of the stuff can be handled separatetly, like speculative load hardening?
If that is the case I can only enable it if subreg liveness is enabled, like proposed earlier.

[rampitec]

Rebased.

test/CodeGen/X86/speculative-load-hardening-indirect.ll

134

Done.

[craig.topper]

I don't think X86 needs this pass. Everything here looks like bad test construction or the issue in speculative load hardening.

test/CodeGen/X86/combine-bitselect.ll

657

I updated this test in trunk so that it no longer has an infinite loop that would allow things to be deleted.

[rampitec]

In D59626#1453765, @craig.topper wrote:

I don't think X86 needs this pass. Everything here looks like bad test construction or the issue in speculative load hardening.

OK, thanks! Than I will skip it for targets which do not track subreg liveness.

[rampitec]

Skip this invocation for targets which do not have subregister liveness.

[qcolombet]

Hi @rampitec,

Are the dead instructions marked during the detect dead lanes pass or during the rename independent SubReg pass?

I would expect the former and in that case, it may be easier to just do the code deletion there.

Cheers,
-Quentin

[rampitec]

In D59626#1454322, @qcolombet wrote:

Hi @rampitec,

Are the dead instructions marked during the detect dead lanes pass or during the rename independent SubReg pass?

I would expect the former and in that case, it may be easier to just do the code deletion there.

Cheers,
-Quentin

The detect dead lanes pass cannot really delete dead lanes, just mark them. So a subreg definition becomes dead, but in most cases retain a use by REG_SEQUENCE.
Rename independent subregs deletes REQ_SEQENCEs and the subreg looses that use, so finally can be deleted.

I.e. both passes can free up new dead instructions independently, but in most cases they need to work together.

[qcolombet]

lib/CodeGen/DeadMachineInstructionElim.cpp

120

I feel that the pass shouldn't be making that decision.
If someone put that in their pipeline, the pass should just run.

[rampitec]

lib/CodeGen/DeadMachineInstructionElim.cpp

120

TargetPassConfig does not have subtarget access when it adds a pass to the pipeline.
So the choice is: 1) always run even if it will not find anything 2) perform this check 3) insert it from the target and not from common TargetPassConfig.
In all cases code to handle LIS is needed though.

I can make it only running from AMDGPU target since we always track subreg liveness. Would that be better?

[rampitec]

Moving pass insertion from common TargetPassConfig into AMDGPU, since most targets do not need it.
Changes to maintain liveness if available remain in the MachineDCE pass, but are a no-op for any other target until they decide to use it late in the RA.

[rampitec]

lib/CodeGen/DeadMachineInstructionElim.cpp

120

I have reverted the check and only add it from AMDGPU. If any other target feels it may need it there is an option to do the same.
If this is committed it will be possible since pass will not destroy liveness anymore.

[qcolombet]

Looks reasonable to me.

I leave the final approval to @arsenm for the AMDGPU changes.

[arsenm]

lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

168–170 ↗

(On Diff #193766)

I prefer avoiding double negative flags, so -amdgpu-dce-in-ra and default to true

test/CodeGen/AMDGPU/dead-lane.mir

11–16

You can drop the registers section (although %0 will need a class marker)

[rampitec]

Addressed review comments.

[arsenm]

LGTM except for the option description string

lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

170 ↗

(On Diff #193922)

Description needs to be updated

[rampitec]

Changed option description.

[arsenm]

LGTM

[bjope]

I see a couple of problems with the changes to DeadMachineInstructionElim.cpp in this patch:

• I get MachineVerifier problems because the LIS update does not take into consideration that the removal of a dead use can introduce disjoint live ranges (resulting in verifier complaining about "Multiple connected components in live interval".
• I get MachineVerifier problems because the LIS update does not update live ranges for physical registers. Consider an instruction like this dead %42:regs = COPY $a1, if that is removed and it is the last use of the physical register $a1 then we need to shrink the live range for $a1 , otherwise the MachineVerifier will complain about "Live segment doesn't end at a valid instruction".
• I do not think the !MRI->reg_empty(Reg) check takes debug info into consideration. Possibly introducing a debug invariance problem.
• It does not seem like this pass is using the LIS itself. So why do we even bother keeping it update? It is destroyed, and then it is up to the pass manager to detect if it needs to be recalculated by a later pass, right? Or do we know that it will be used by a later pass, and thereby we want to bother about preserving it (even if it isn't for the greater good of the pass itself). If it is like that, then we probably need to add some code comments to highlight that we only do it because some other pass needs it (for example in case that pass is removed in the future).
• Is it really true that we now preserve all possible analyses (both existing and future, both upstream and downstream), motivating the change from setPreservesCFG() to setPreservesAll(). I mean, the pass is pretty "destructive" as it removes instructions. You have only added (afaict incomplete) support to preserve the LIS. Notice that I'm not sure exactly what the difference between "PreservesAll" and "PreservesCFG" is, but I suppose it includes more analysis results besides LIS (and SlotIndices).

Maybe we can back out the change in DeadMachineInstructionElim.cpp until the above problems have been sorted out?

[rampitec]

Bjorn,

did you enable MachineDCE in RA in some other target? So far it is only enabled in the AMDGPU, so I am curious where do you see these problems?

In D59626#1458186, @bjope wrote:

I see a couple of problems with the changes to DeadMachineInstructionElim.cpp in this patch:

• I get MachineVerifier problems because the LIS update does not take into consideration that the removal of a dead use can introduce disjoint live ranges (resulting in verifier complaining about "Multiple connected components in live interval".

I am recomputing liveness for all affected uses, how could that be?

• I get MachineVerifier problems because the LIS update does not update live ranges for physical registers. Consider an instruction like this dead %42:regs = COPY $a1, if that is removed and it is the last use of the physical register $a1 then we need to shrink the live range for $a1 , otherwise the MachineVerifier will complain about "Live segment doesn't end at a valid instruction".

That is probably right. I avoid deleting dead phys, but not their uses. I think shrinkToUses() will be appropriate here.

• I do not think the !MRI->reg_empty(Reg) check takes debug info into consideration. Possibly introducing a debug invariance problem.

I thought about this, so I did not use reg_nodbg_empty() in this instance. I.e. this LI gets recreated.

• It does not seem like this pass is using the LIS itself. So why do we even bother keeping it update? It is destroyed, and then it is up to the pass manager to detect if it needs to be recalculated by a later pass, right? Or do we know that it will be used by a later pass, and thereby we want to bother about preserving it (even if it isn't for the greater good of the pass itself). If it is like that, then we probably need to add some code comments to highlight that we only do it because some other pass needs it (for example in case that pass is removed in the future).

The pass does not use it, but should update if it exists. That is true, if I do not claim it is preserved then it will be recalculated, but that is very slow.
So far only AMDGPU should be using it with liveness available. Am I missing some other invocation?

• Is it really true that we now preserve all possible analyses (both existing and future, both upstream and downstream), motivating the change from setPreservesCFG() to setPreservesAll(). I mean, the pass is pretty "destructive" as it removes instructions. You have only added (afaict incomplete) support to preserve the LIS. Notice that I'm not sure exactly what the difference between "PreservesAll" and "PreservesCFG" is, but I suppose it includes more analysis results besides LIS (and SlotIndices).

It does preserve LIS and SlotIndexes. The rest of the analysis passes (like PDT) are just unaffected by the pass itself. I believe that is more or less the same assumption for all passes which use PreservesAll() that we are considering exiting analysis passes. Otherwise this method is not future proof in general.

Maybe we can back out the change in DeadMachineInstructionElim.cpp until the above problems have been sorted out?

If there is no better way. If you see the problems do you mind to share an IR?

[rampitec]

In D59626#1458594, @rampitec wrote:

• I get MachineVerifier problems because the LIS update does not update live ranges for physical registers. Consider an instruction like this dead %42:regs = COPY $a1, if that is removed and it is the last use of the physical register $a1 then we need to shrink the live range for $a1 , otherwise the MachineVerifier will complain about "Live segment doesn't end at a valid instruction".

Actually I cannot reproduce it:

# RUN: llc -march=amdgcn -mcpu=tonga %s -run-pass liveintervals,dead-mi-elimination -verify-machineinstrs -o - | FileCheck -check-prefix=GCN %s
  
---
name:            dead_phys_use
tracksRegLiveness: true
liveins:
  - { reg: '$vgpr0', virtual-reg: '%2' }
body:             |
  bb.0:
    liveins: $vgpr0

    %2:vgpr_32 = COPY $vgpr0
    $vgpr1 = COPY %2:vgpr_32
    %1:vgpr_32 = COPY $vgpr0, implicit $exec
    SI_RETURN_TO_EPILOG $vgpr1
...

The last COPY is dead and $vgpr0 will be affected, but verifier did not find any errors:

bb.0:
  liveins: $vgpr0

  %0:vgpr_32 = COPY $vgpr0
  $vgpr1 = COPY %0
  SI_RETURN_TO_EPILOG $vgpr1

[bjope]

In D59626#1458594, @rampitec wrote:

Bjorn,

did you enable MachineDCE in RA in some other target? So far it is only enabled in the AMDGPU, so I am curious where do you see these problems?

Well, our OOT target is using the DeadMachineInstructionElim pass in (maybe non-standard according to in-tree target) positions in the backend pipeline.

In D59626#1458186, @bjope wrote:

I see a couple of problems with the changes to DeadMachineInstructionElim.cpp in this patch:

• I get MachineVerifier problems because the LIS update does not take into consideration that the removal of a dead use can introduce disjoint live ranges (resulting in verifier complaining about "Multiple connected components in live interval".

I am recomputing liveness for all affected uses, how could that be?

Ok, so the test case is from the OOT target. Here is an excerpt of the MIR:

0B	bb.0:

16B	  %0:rn = ...
32B	  %10:an32pairs = ...
80B	  %4:an32_0_7 = ... %10.hiAcc:an32pairs, ...
96B	  brr_cond %bb.3, ...
112B	  brr_uncond %bb.1

128B	bb.1:

144B	  brr_cond %bb.2, ...
160B	  brr_uncond %bb.3

176B	bb.2:

192B	  %8:anh_rn = COPY %4.lo16:an32_0_7
224B	  %7:anh_rn = COPY %10.hiAcc_then_lo16:an32pairs
256B	  brr_uncond %bb.4

272B	bb.3:

288B	  %9:rn = ...
304B	  %10:an32pairs = ... %9:rn, ...
352B	  %7:anh_rn = COPY %10.hiAcc_then_lo16:an32pairs
368B	  %8:anh_rn = COPY %10.hiAcc_then_hi16:an32pairs

384B	bb.4::

400B	  nop 0, $noreg, 0, implicit %10.loAcc:an32pairs
416B	  $a0h = COPY %8:anh_rn
432B	  $a1h = COPY %7:anh_rn
448B	  rets implicit $a0h, implicit $a1h

The nop is removed by by DMIE, resulting in

*** Bad machine code: Multiple connected components in live interval ***
- function:    f3
- interval:    %10 [32r,224r:0)[304r,368r:1)  0@32r 1@304r weight:0.000000e+00

• I get MachineVerifier problems because the LIS update does not update live ranges for physical registers. Consider an instruction like this dead %42:regs = COPY $a1, if that is removed and it is the last use of the physical register $a1 then we need to shrink the live range for $a1 , otherwise the MachineVerifier will complain about "Live segment doesn't end at a valid instruction".

That is probably right. I avoid deleting dead phys, but not their uses. I think shrinkToUses() will be appropriate here.

• I do not think the !MRI->reg_empty(Reg) check takes debug info into consideration. Possibly introducing a debug invariance problem.

I thought about this, so I did not use reg_nodbg_empty() in this instance. I.e. this LI gets recreated.

So it might be that that the call to createAndComputeVirtRegInterval only happens when having debug info. And afaict that might give a different codegen compared to the situation when not having any debug info. I think we usually try to avoid that. Codegen should not be impacted by existence of debug info.

• It does not seem like this pass is using the LIS itself. So why do we even bother keeping it update? It is destroyed, and then it is up to the pass manager to detect if it needs to be recalculated by a later pass, right? Or do we know that it will be used by a later pass, and thereby we want to bother about preserving it (even if it isn't for the greater good of the pass itself). If it is like that, then we probably need to add some code comments to highlight that we only do it because some other pass needs it (for example in case that pass is removed in the future).

The pass does not use it, but should update if it exists. That is true, if I do not claim it is preserved then it will be recalculated, but that is very slow.
So far only AMDGPU should be using it with liveness available. Am I missing some other invocation?

Well, in my case it is an OOT target. So I'm can't blame you for missing that.

• Is it really true that we now preserve all possible analyses (both existing and future, both upstream and downstream), motivating the change from setPreservesCFG() to setPreservesAll(). I mean, the pass is pretty "destructive" as it removes instructions. You have only added (afaict incomplete) support to preserve the LIS. Notice that I'm not sure exactly what the difference between "PreservesAll" and "PreservesCFG" is, but I suppose it includes more analysis results besides LIS (and SlotIndices).

It does preserve LIS and SlotIndexes. The rest of the analysis passes (like PDT) are just unaffected by the pass itself. I believe that is more or less the same assumption for all passes which use PreservesAll() that we are considering exiting analysis passes. Otherwise this method is not future proof in general.

Ok. A little bit scary. But if that is how PreservesAll() works in general I guess it is ok.

Maybe we can back out the change in DeadMachineInstructionElim.cpp until the above problems have been sorted out?

If there is no better way. If you see the problems do you mind to share an IR?

A little bit hard due to that I've only triggered it for the OOT target so far. For now I hacked around it in our downstream clone (although using setPreservesCFG instead gives some churn in some AMDGPU test cases, so it seems like we get different results if LIS is recalculated instead of just being preserved by DeadMachineInstructionElim, I would have expected to get the same result, at least if the preserve worked correctly).

Anyway. I can try to make in-tree reproducers tomorrow.

[rampitec]

In D59626#1458686, @bjope wrote:

Actually shrinkToUses does not work with phys. That should be possible to avoid removing dead phys uses if that phys is allocatable (otherwise liveness is not calculated for it).
I also do not see how to reproduce disjoint intervals.
And yes, changing it back to PreservesCFG() should effectively switch this off.

[rampitec]

@bjope could you please check if D60419 solves your problem with physreg use removal?

[rampitec]

In D59626#1458773, @rampitec wrote:

@bjope could you please check if D60419 solves your problem with physreg use removal?

WRT disjoint liveranges. I only see such possibility with a physreg, because virtual should be recomputed. So maybe the same patch would help.

[bjope]

Here is a MIR-reproducer triggering "Multiple connected components in live interval".

For some reason I needed to add "simple-register-coalescing" at the end of the list of passes to run. Otherwise the pass manager decides to discard LIS before dead-mi-elimination.

# RUN: llc -mtriple=amdgcn-- -run-pass=liveintervals,machineverifier,dead-mi-elimination,machineverifier,simple-register-coalescing -o /dev/null %s

---
name:            foo
tracksRegLiveness: true
body:             |
  bb.0:
    liveins: $sgpr0_sgpr1

    %10:sreg_128 = S_LOAD_DWORDX4_IMM killed $noreg, 9, 0
    S_NOP 0, implicit-def %4:sreg_128, implicit %10.sub1:sreg_128
    S_CBRANCH_SCC0 %bb.3, implicit undef $scc
    S_BRANCH %bb.1

  bb.1:
    S_CBRANCH_SCC0 %bb.2, implicit undef $scc
    S_BRANCH %bb.3

  bb.2:
    %8:sreg_32_xm0 = COPY %4.sub1:sreg_128
    %7:sreg_32_xm0 = COPY %10.sub1:sreg_128
    S_BRANCH %bb.4

  bb.3:
    %10:sreg_128 = S_LOAD_DWORDX4_IMM killed $noreg, 10, 0
    %7:sreg_32_xm0 = COPY %10.sub1:sreg_128
    %8:sreg_32_xm0 = COPY %10.sub2:sreg_128

  bb.4:
    S_NOP 0, implicit %10
    $sgpr0 = COPY %8:sreg_32_xm0
    $sgpr1 = COPY %7:sreg_32_xm0
    S_ENDPGM 0, implicit $sgpr0, implicit $sgpr1
...

[bjope]

In D59626#1458773, @rampitec wrote:

@bjope could you please check if D60419 solves your problem with physreg use removal?

Isn't it weird that this pass now is doing less optimizations when an analysis pass is provided? It might be OK for your use case, but it seems weird from a more general perspective.

FWIW, I'm not sure exactly why we are running this pass in a lot of places in our OOT target (after ISel, before coalescing, post-coalescing, etc). But we have done it for years (since before I started working with this target), and now some of those runs won't do as much cleanup as they used to do (when a LIS is found). Not sure, but maybe it is exactly these physreg removals we aim for, so I'm not sure that solving the LIS preservation problem by not eliminating certain dead instructions is good enough for us. It will take some time to analyse that, so at the moment it is easier for me to just make a work around patch downstream.

[rampitec]

In D59626#1459918, @bjope wrote:

In D59626#1458773, @rampitec wrote:

@bjope could you please check if D60419 solves your problem with physreg use removal?

Isn't it weird that this pass now is doing less optimizations when an analysis pass is provided? It might be OK for your use case, but it seems weird from a more general perspective.

FWIW, I'm not sure exactly why we are running this pass in a lot of places in our OOT target (after ISel, before coalescing, post-coalescing, etc). But we have done it for years (since before I started working with this target), and now some of those runs won't do as much cleanup as they used to do (when a LIS is found). Not sure, but maybe it is exactly these physreg removals we aim for, so I'm not sure that solving the LIS preservation problem by not eliminating certain dead instructions is good enough for us. It will take some time to analyse that, so at the moment it is easier for me to just make a work around patch downstream.

Bjorn, thank you for the testcase. It doesn't seem to be easy to quick fix, while I am going for vacation tomorrow. I will create a revert patch for now and try to sort it out later.

I understood someone might be aiming for dead phys regs, but handling phys without recompute can be much more challenging, so probably a full recompute is not unreasonable under such scenario too.

[rampitec]

In D59626#1459926, @rampitec wrote:

In D59626#1459918, @bjope wrote:

In D59626#1458773, @rampitec wrote:

@bjope could you please check if D60419 solves your problem with physreg use removal?

Isn't it weird that this pass now is doing less optimizations when an analysis pass is provided? It might be OK for your use case, but it seems weird from a more general perspective.

FWIW, I'm not sure exactly why we are running this pass in a lot of places in our OOT target (after ISel, before coalescing, post-coalescing, etc). But we have done it for years (since before I started working with this target), and now some of those runs won't do as much cleanup as they used to do (when a LIS is found). Not sure, but maybe it is exactly these physreg removals we aim for, so I'm not sure that solving the LIS preservation problem by not eliminating certain dead instructions is good enough for us. It will take some time to analyse that, so at the moment it is easier for me to just make a work around patch downstream.

Bjorn, thank you for the testcase. It doesn't seem to be easy to quick fix, while I am going for vacation tomorrow. I will create a revert patch for now and try to sort it out later.

I understood someone might be aiming for dead phys regs, but handling phys without recompute can be much more challenging, so probably a full recompute is not unreasonable under such scenario too.

Revert patch: D60466

[bjope]

For the record, I think the "multiple connected components" problem could have been solved by something like this after the call to "createAndComputeVirtRegInterval"

LiveInterval &LI = LIS->getInterval(Reg);
assert(Reg == LI.reg && "Invalid reg to interval mapping");
if (TargetRegisterInfo::isVirtualRegister(LI.reg)) {
  // Check whether or not LI is composed by multiple connected
  // components and if that is the case, fix that.
  SmallVector<LiveInterval*, 8> SplitLIs;
  LIS->splitSeparateComponents(LI, SplitLIs);
}

However, that still leaves the questions about if the pass should do less optimizations related to physregs when LIS is available, or how to preserve LIS if optimizing on physregs.
And I'm not sure about the cost of using splitSeparateComponents compared to a full recompute.

[rampitec]

In D59626#1460006, @bjope wrote:

For the record, I think the "multiple connected components" problem could have been solved by something like this after the call to "createAndComputeVirtRegInterval"

LiveInterval &LI = LIS->getInterval(Reg);
assert(Reg == LI.reg && "Invalid reg to interval mapping");
if (TargetRegisterInfo::isVirtualRegister(LI.reg)) {
  // Check whether or not LI is composed by multiple connected
  // components and if that is the case, fix that.
  SmallVector<LiveInterval*, 8> SplitLIs;
  LIS->splitSeparateComponents(LI, SplitLIs);
}

However, that still leaves the questions about if the pass should do less optimizations related to physregs when LIS is available, or how to preserve LIS if optimizing on physregs.
And I'm not sure about the cost of using splitSeparateComponents compared to a full recompute.

Thank you for the hint and test case. For the phys I could remove all regunits, but for some reason I cannot always get them reconstructed. It will need some more debugging.

[rampitec]

In D59626#1460006, @bjope wrote:

For the record, I think the "multiple connected components" problem could have been solved by something like this after the call to "createAndComputeVirtRegInterval"

LiveInterval &LI = LIS->getInterval(Reg);
assert(Reg == LI.reg && "Invalid reg to interval mapping");
if (TargetRegisterInfo::isVirtualRegister(LI.reg)) {
  // Check whether or not LI is composed by multiple connected
  // components and if that is the case, fix that.
  SmallVector<LiveInterval*, 8> SplitLIs;
  LIS->splitSeparateComponents(LI, SplitLIs);
}

However, that still leaves the questions about if the pass should do less optimizations related to physregs when LIS is available, or how to preserve LIS if optimizing on physregs.
And I'm not sure about the cost of using splitSeparateComponents compared to a full recompute.

For the record, this code works if applied after createAndComputeVirtRegInterval() call.

[rampitec]

In D59626#1460173, @rampitec wrote:

In D59626#1460006, @bjope wrote:

For the record, I think the "multiple connected components" problem could have been solved by something like this after the call to "createAndComputeVirtRegInterval"

LiveInterval &LI = LIS->getInterval(Reg);
assert(Reg == LI.reg && "Invalid reg to interval mapping");
if (TargetRegisterInfo::isVirtualRegister(LI.reg)) {
  // Check whether or not LI is composed by multiple connected
  // components and if that is the case, fix that.
  SmallVector<LiveInterval*, 8> SplitLIs;
  LIS->splitSeparateComponents(LI, SplitLIs);
}

However, that still leaves the questions about if the pass should do less optimizations related to physregs when LIS is available, or how to preserve LIS if optimizing on physregs.
And I'm not sure about the cost of using splitSeparateComponents compared to a full recompute.

For the record, this code works if applied after createAndComputeVirtRegInterval() call.

The problem with phys regs is that I see no way to recompute liveins. If I remove reg units for affected register they are reconstructed on the next query. However if it was a livein I will drop it as well, and it is only recomputed in private LiveIntervals::computeLiveInRegUnits(), which is only called from LiveIntervals::runOnMachineFunction()...

lib/CodeGen/DeadMachineInstructionElim.cpp

 //===- DeadMachineInstructionElim.cpp - Remove dead machine instructions --===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This is an extremely simple MachineInstr-level dead-code-elimination pass.
 //
 //===----------------------------------------------------------------------===//
 
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/Statistic.h"
+#include "llvm/CodeGen/LiveIntervals.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "dead-mi-elimination"
 
 STATISTIC(NumDeletes,          "Number of dead instructions deleted");
 
 namespace {
   class DeadMachineInstructionElim : public MachineFunctionPass {
     bool runOnMachineFunction(MachineFunction &MF) override;
 
     const TargetRegisterInfo *TRI;
     const MachineRegisterInfo *MRI;
     const TargetInstrInfo *TII;
+    LiveIntervals *LIS;
     BitVector LivePhysRegs;
 
   public:
     static char ID; // Pass identification, replacement for typeid
     DeadMachineInstructionElim() : MachineFunctionPass(ID) {
      initializeDeadMachineInstructionElimPass(*PassRegistry::getPassRegistry());
     }
 
     void getAnalysisUsage(AnalysisUsage &AU) const override {
-      AU.setPreservesCFG();
+      AU.setPreservesAll();
       MachineFunctionPass::getAnalysisUsage(AU);
     }
 
   private:
     bool isDead(const MachineInstr *MI) const;
   };
 }
 char DeadMachineInstructionElim::ID = 0;
bool DeadMachineInstructionElim::isDead(const MachineInstr *MI) const {
 
   // Examine each operand.
   for (unsigned i = 0, e = MI->getNumOperands(); i != e; ++i) {
     const MachineOperand &MO = MI->getOperand(i);
     if (MO.isReg() && MO.isDef()) {
       unsigned Reg = MO.getReg();
       if (TargetRegisterInfo::isPhysicalRegister(Reg)) {
         // Don't delete live physreg defs, or any reserved register defs.
-        if (LivePhysRegs.test(Reg) || MRI->isReserved(Reg))
+        // Do not remove physreg defs if we have LIS as we may be unable
+        // to accurately recompute its liveness.
+        if (LivePhysRegs.test(Reg) || MRI->isReserved(Reg) || LIS)
           return false;
       } else {
+        // An instruction can also use its def in case if it is a tied operand.
+        // TODO: Technically we can also remove it if def dominates the use.
+        //       This can happen when two instructions define different subregs
+        //       of the same register.
         for (const MachineInstr &Use : MRI->use_nodbg_instructions(Reg)) {
           if (&Use != MI)
             // This def has a non-debug use. Don't delete the instruction!
             return false;
         }
       }
     }
   }
 
   // If there are no defs with uses, the instruction is dead.
   return true;
 }
 
 bool DeadMachineInstructionElim::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   bool AnyChanges = false;
   MRI = &MF.getRegInfo();
   TRI = MF.getSubtarget().getRegisterInfo();
   TII = MF.getSubtarget().getInstrInfo();
+  LIS = getAnalysisIfAvailable<LiveIntervals>();
+  DenseSet<unsigned> RecalcRegs;
 
   // Loop over all instructions in all blocks, from bottom to top, so that it's
   // more likely that chains of dependent but ultimately dead instructions will
   // be cleaned up.
   for (MachineBasicBlock &MBB : make_range(MF.rbegin(), MF.rend())) {
     // Start out assuming that reserved registers are live out of this block.
     LivePhysRegs = MRI->getReservedRegs();
 
     // Add live-ins from successors to LivePhysRegs. Normally, physregs are not
     // live across blocks, but some targets (x86) can have flags live out of a
     // block.
     for (MachineBasicBlock::succ_iterator S = MBB.succ_begin(),
            E = MBB.succ_end(); S != E; S++)
       for (const auto &LI : (*S)->liveins())
         LivePhysRegs.set(LI.PhysReg);
 
     // Now scan the instructions and delete dead ones, tracking physreg
     // liveness as we go.
     for (MachineBasicBlock::reverse_iterator MII = MBB.rbegin(),
          MIE = MBB.rend(); MII != MIE; ) {
       MachineInstr *MI = &*MII++;
 
       // If the instruction is dead, delete it!
       if (isDead(MI)) {
         LLVM_DEBUG(dbgs() << "DeadMachineInstructionElim: DELETING: " << *MI);
+        if (LIS) {
+          for (const MachineOperand &MO : MI->operands()) {
+            if (MO.isReg() && TRI->isVirtualRegister(MO.getReg()))
+              RecalcRegs.insert(MO.getReg());
+          }
+          LIS->RemoveMachineInstrFromMaps(*MI);
+        }
+
         // It is possible that some DBG_VALUE instructions refer to this
         // instruction.  They get marked as undef and will be deleted
         // in the live debug variable analysis.
         MI->eraseFromParentAndMarkDBGValuesForRemoval();
         AnyChanges = true;
         ++NumDeletes;
         continue;
       }
for (MachineBasicBlock::reverse_iterator MII = MBB.rbegin(),
               LivePhysRegs.set(*AI);
           }
         }
       }
     }
   }
 
   LivePhysRegs.clear();
+
+  for (auto Reg : RecalcRegs) {
+    LIS->removeInterval(Reg);
+    if (!MRI->reg_empty(Reg))
+      LIS->createAndComputeVirtRegInterval(Reg);
+  }
+
   return AnyChanges;
 }

lib/CodeGen/TargetPassConfig.cpp

 // insert a PostRA scheduling pass wherever it wants.
 cl::opt<bool> MISchedPostRA("misched-postra", cl::Hidden,
   cl::desc("Run MachineScheduler post regalloc (independent of preRA sched)"));
 
 // Experimental option to run live interval analysis early.
 static cl::opt<bool> EarlyLiveIntervals("early-live-intervals", cl::Hidden,
     cl::desc("Run live interval analysis earlier in the pipeline"));
 
+// Option is used in lit tests to prevent deadcoding of patterns inspected.
+static cl::opt<bool>
+DisableDCEInRA("disable-dce-in-ra",
+    cl::init(false), cl::Hidden,
+    cl::desc("Disable machine DCE inside regalloc"));
+
 // Experimental option to use CFL-AA in codegen
 enum class CFLAAType { None, Steensgaard, Andersen, Both };
 static cl::opt<CFLAAType> UseCFLAA(
     "use-cfl-aa-in-codegen", cl::init(CFLAAType::None), cl::Hidden,
     cl::desc("Enable the new, experimental CFL alias analysis in CodeGen"),
     cl::values(clEnumValN(CFLAAType::None, "none", "Disable CFL-AA"),
                clEnumValN(CFLAAType::Steensgaard, "steens",
                           "Enable unification-based CFL-AA"),
void TargetPassConfig::addOptimizedRegAlloc() {
   addPass(&TwoAddressInstructionPassID, false);
   addPass(&RegisterCoalescerID);
 
   // The machine scheduler may accidentally create disconnected components
   // when moving subregister definitions around, avoid this by splitting them to
   // separate vregs before. Splitting can also improve reg. allocation quality.
   addPass(&RenameIndependentSubregsID);
 
+  if (!DisableDCEInRA)
+    addPass(&DeadMachineInstructionElimID);
+
   // PreRA instruction scheduling.
   addPass(&MachineSchedulerID);
 
   if (addRegAssignmentOptimized()) {
     // Copy propagate to forward register uses and try to eliminate COPYs that
     // were not coalesced.
     addPass(&MachineCopyPropagationID);
 

test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       Remove unreachable machine basic blocks
 ; CHECK-NEXT:       Live Variable Analysis
 ; CHECK-NEXT:       Eliminate PHI nodes for register allocation
 ; CHECK-NEXT:       Two-Address instruction pass
 ; CHECK-NEXT:       Slot index numbering
 ; CHECK-NEXT:       Live Interval Analysis
 ; CHECK-NEXT:       Simple Register Coalescing
 ; CHECK-NEXT:       Rename Disconnected Subregister Components
+; CHECK-NEXT:       Remove dead machine instructions
 ; CHECK-NEXT:       Machine Instruction Scheduler
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       Debug Variable Analysis
 ; CHECK-NEXT:       Live Stack Slot Analysis
 ; CHECK-NEXT:       Virtual Register Map
 ; CHECK-NEXT:       Live Register Matrix
 ; CHECK-NEXT:       Bundle Machine CFG Edges
 ; CHECK-NEXT:       Spill Code Placement Analysis

test/CodeGen/AArch64/arm64-AdvSIMD-Scalar.ll

-; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-eabi -aarch64-neon-syntax=apple -aarch64-enable-simd-scalar=true -asm-verbose=false -disable-adv-copy-opt=true | FileCheck %s -check-prefix=CHECK -check-prefix=CHECK-NOOPT
-; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-eabi -aarch64-neon-syntax=apple -aarch64-enable-simd-scalar=true -asm-verbose=false -disable-adv-copy-opt=false | FileCheck %s -check-prefix=CHECK -check-prefix=CHECK-OPT
-; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-eabi -aarch64-neon-syntax=generic -aarch64-enable-simd-scalar=true -asm-verbose=false -disable-adv-copy-opt=true | FileCheck %s -check-prefix=GENERIC -check-prefix=GENERIC-NOOPT
-; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-eabi -aarch64-neon-syntax=generic -aarch64-enable-simd-scalar=true -asm-verbose=false -disable-adv-copy-opt=false | FileCheck %s -check-prefix=GENERIC -check-prefix=GENERIC-OPT
+; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-eabi -aarch64-neon-syntax=apple -aarch64-enable-simd-scalar=true -asm-verbose=false -disable-adv-copy-opt=true -disable-dce-in-ra | FileCheck %s -check-prefix=CHECK -check-prefix=CHECK-NOOPT
+; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-eabi -aarch64-neon-syntax=apple -aarch64-enable-simd-scalar=true -asm-verbose=false -disable-adv-copy-opt=false -disable-dce-in-ra | FileCheck %s -check-prefix=CHECK -check-prefix=CHECK-OPT
+; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-eabi -aarch64-neon-syntax=generic -aarch64-enable-simd-scalar=true -asm-verbose=false -disable-adv-copy-opt=true -disable-dce-in-ra | FileCheck %s -check-prefix=GENERIC -check-prefix=GENERIC-NOOPT
+; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-eabi -aarch64-neon-syntax=generic -aarch64-enable-simd-scalar=true -asm-verbose=false -disable-adv-copy-opt=false -disable-dce-in-ra | FileCheck %s -check-prefix=GENERIC -check-prefix=GENERIC-OPT
 
 define <2 x i64> @bar(<2 x i64> %a, <2 x i64> %b) nounwind readnone {
 ; CHECK-LABEL: bar:
 ; CHECK: add.2d v[[REG:[0-9]+]], v0, v1
 ; CHECK: add  d[[REG3:[0-9]+]], d[[REG]], d1
 ; CHECK: sub  d[[REG2:[0-9]+]], d[[REG]], d1
 ; CHECK-NOT: fmov
 ; CHECK: fmov [[COPY_REG2:x[0-9]+]], d[[REG2]]

test/CodeGen/AMDGPU/dead-lane.mir

+# RUN: llc -march=amdgcn -mcpu=tonga %s -start-before detect-dead-lanes -stop-before machine-scheduler -verify-machineinstrs -o - | FileCheck -check-prefix=GCN %s
+
+# GCN-LABEL: name: dead_lane
+# GCN:      bb.0:
+# GCN-NEXT: undef %3.sub0:vreg_64 = V_MAC_F32_e32 undef %0:vgpr_32, undef %0:vgpr_32, undef %3.sub0, implicit $exec
+# GCN-NEXT: FLAT_STORE_DWORD undef %4:vreg_64, %3.sub0,
+---
+name:            dead_lane
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: vgpr_32}
+  - { id: 1, class: vgpr_32}
+  - { id: 2, class: vgpr_32}
+  - { id: 3, class: vreg_64}
+  - { id: 4, class: vreg_64}
+body:             |
+  bb.0:
+    %1:vgpr_32 = V_MAC_F32_e32 undef %0, undef %0, undef %0, implicit $exec
+    %2:vgpr_32 = V_MAC_F32_e32 undef %0, undef %0, undef %0, implicit $exec
+    %3:vreg_64 = REG_SEQUENCE %1, %subreg.sub0, %2, %subreg.sub1
+    FLAT_STORE_DWORD undef %4, %3.sub0, 0, 0, 0, implicit $exec, implicit $flat_scr
+    S_ENDPGM 0
+
+...

test/CodeGen/AMDGPU/salu-to-valu.ll

bb1:
   br i1 %cnd, label %bb4, label %bb2
 
 bb2:
   %tmp3 = add nsw i32 %tmp, 1
   br label %bb4
 
 bb4:
   %tmp5 = phi i32 [ %tmp3, %bb2 ], [ %tmp, %bb1 ]
+  store volatile i32 %tmp5, i32 addrspace(1)* undef
   br label %bb1
 }
 
 ; GCN-LABEL: {{^}}phi_imm_in_sgprs
 ; GCN: s_movk_i32 [[A:s[0-9]+]], 0x400
 ; GCN: s_movk_i32 [[B:s[0-9]+]], 0x400
 ; GCN: [[LOOP_LABEL:[0-9a-zA-Z_]+]]:
 ; GCN: s_xor_b32 [[B]], [[B]], [[A]]

test/CodeGen/AMDGPU/sdwa-peephole.ll

 bb2:                                              ; preds = %bb1
   %tmp3 = call i32 asm "v_and_b32_e32 $0, $1, $2", "=v,s,v"(i32 65535, i32 undef) #1
   %tmp5 = or i32 %tmp3, 65536
   %tmp6 = insertelement <2 x i32> %tmp, i32 %tmp5, i64 0
   br label %bb11
 
 bb11:                                             ; preds = %bb10, %bb2
   %tmp12 = phi <2 x i32> [ %tmp6, %bb2 ], [ %tmp, %bb1 ]
+  store volatile <2 x i32> %tmp12, <2 x i32> addrspace(1)* undef
   br label %bb1
 }

test/CodeGen/AMDGPU/shrink-carry.mir

 body:             |
   bb.0:
 
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     %2 = IMPLICIT_DEF
     %3 = V_CMP_GT_U32_e64 %0, %1, implicit $exec
     %4, %5 = V_SUBBREV_U32_e64 0, %0, %3, 0, implicit $exec
+    GLOBAL_STORE_DWORD undef $vgpr0_vgpr1, %4, 0, 0, 0, implicit $exec
 
 ...
 
 # GCN-LABEL: name: subb{{$}}
 # GCN:       V_SUBBREV_U32_e32 0, undef $vgpr0, implicit-def $vcc, implicit killed $vcc, implicit $exec
 
 ---
 name:            subb
 tracksRegLiveness: true
 registers:
   - { id: 0, class: vgpr_32 }
   - { id: 1, class: vgpr_32 }
   - { id: 2, class: vgpr_32 }
   - { id: 3, class: sreg_64_xexec }
   - { id: 4, class: vgpr_32 }
   - { id: 5, class: sreg_64_xexec }
 body:             |
   bb.0:
 
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     %2 = IMPLICIT_DEF
     %3 = V_CMP_GT_U32_e64 %0, %1, implicit $exec
     %4, %5 = V_SUBB_U32_e64 %0, 0, %3, 0, implicit $exec
+    GLOBAL_STORE_DWORD undef $vgpr0_vgpr1, %4, 0, 0, 0, implicit $exec
 
 ...
 
 # GCN-LABEL: name: addc{{$}}
 # GCN:       V_ADDC_U32_e32 0, undef $vgpr0, implicit-def $vcc, implicit killed $vcc, implicit $exec
 
 ---
 name:            addc
 tracksRegLiveness: true
 registers:
   - { id: 0, class: vgpr_32 }
   - { id: 1, class: vgpr_32 }
   - { id: 2, class: vgpr_32 }
   - { id: 3, class: sreg_64_xexec }
   - { id: 4, class: vgpr_32 }
   - { id: 5, class: sreg_64_xexec }
 body:             |
   bb.0:
 
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     %2 = IMPLICIT_DEF
     %3 = V_CMP_GT_U32_e64 %0, %1, implicit $exec
     %4, %5 = V_ADDC_U32_e64 0, %0, %3, 0, implicit $exec
+    GLOBAL_STORE_DWORD undef $vgpr0_vgpr1, %4, 0, 0, 0, implicit $exec
 
 ...
 
 # GCN-LABEL: name: addc2{{$}}
 # GCN:       V_ADDC_U32_e32 0, undef $vgpr0, implicit-def $vcc, implicit killed $vcc, implicit $exec
 
 ---
 name:            addc2
 tracksRegLiveness: true
 registers:
   - { id: 0, class: vgpr_32 }
   - { id: 1, class: vgpr_32 }
   - { id: 2, class: vgpr_32 }
   - { id: 3, class: sreg_64_xexec }
   - { id: 4, class: vgpr_32 }
   - { id: 5, class: sreg_64_xexec }
 body:             |
   bb.0:
 
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     %2 = IMPLICIT_DEF
     %3 = V_CMP_GT_U32_e64 %0, %1, implicit $exec
     %4, %5 = V_ADDC_U32_e64 %0, 0, %3, 0, implicit $exec
+    GLOBAL_STORE_DWORD undef $vgpr0_vgpr1, %4, 0, 0, 0, implicit $exec
 
 ...

test/CodeGen/AMDGPU/spill-empty-live-interval.mir

-# RUN: llc -mtriple=amdgcn-amd-amdhsa -verify-machineinstrs -stress-regalloc=1 -start-before=simple-register-coalescing -stop-after=greedy -o - %s | FileCheck %s
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -disable-dce-in-ra -verify-machineinstrs -stress-regalloc=1 -start-before=simple-register-coalescing -stop-after=greedy -o - %s | FileCheck %s
 # https://bugs.llvm.org/show_bug.cgi?id=33620
 
 ---
 # This would assert due to the empty live interval created for %9
 # on the last S_NOP with an undef subreg use.
 
 # CHECK-LABEL: name: expecting_non_empty_interval
 

test/CodeGen/AMDGPU/subreg-coalescer-undef-use.ll

-; RUN: llc -march=amdgcn -mcpu=tahiti -o - %s | FileCheck %s
+; RUN: llc -march=amdgcn -mcpu=tahiti -disable-dce-in-ra -o - %s | FileCheck %s
 ; Don't crash when the use of an undefined value is only detected by the
 ; register coalescer because it is hidden with subregister insert/extract.
 target triple="amdgcn--"
 
 ; CHECK-LABEL: foobar:
 ; CHECK: s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; CHECK-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; CHECK-NEXT: v_mbcnt_lo_u32_b32_e64

test/CodeGen/ARM/select-imm.ll

-; RUN: llc -mtriple=arm-eabi %s -o - | FileCheck %s --check-prefix=ARM
+; RUN: llc -mtriple=arm-eabi -disable-dce-in-ra %s -o - | FileCheck %s --check-prefix=ARM
 
-; RUN: llc -mtriple=arm-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - \
+; RUN: llc -mtriple=arm-eabi -mcpu=arm1156t2-s -mattr=+thumb2 -disable-dce-in-ra %s -o - \
 ; RUN:  | FileCheck %s --check-prefix=ARMT2
 
-; RUN: llc -mtriple=thumb-eabi -mcpu=cortex-m0 %s -o - \
+; RUN: llc -mtriple=thumb-eabi -mcpu=cortex-m0 -disable-dce-in-ra %s -o - \
 ; RUN:  | FileCheck %s --check-prefix=THUMB1
 
-; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - \
+; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 -disable-dce-in-ra %s -o - \
 ; RUN:  | FileCheck %s --check-prefix=THUMB2
 
-; RUN: llc -mtriple=thumbv8m.base-eabi %s -o - \
+; RUN: llc -mtriple=thumbv8m.base-eabi -disable-dce-in-ra %s -o - \
 ; RUN:  | FileCheck %s --check-prefix=V8MBASE
 
 define i32 @t1(i32 %c) nounwind readnone {
 entry:
 ; ARM-LABEL: t1:
 ; ARM: mov [[R1:r[0-9]+]], #101
 ; ARM: orr [[R1b:r[0-9]+]], [[R1]], #256
 ; ARM: movgt {{r[0-1]}}, #123

test/CodeGen/Hexagon/v6-unaligned-spill.ll

-; RUN: llc -march=hexagon < %s | FileCheck %s
+; RUN: llc -march=hexagon -disable-dce-in-ra < %s | FileCheck %s
 
 ; Test that we generate an unaligned vector store for a spill when a function
 ; has an alloca. Also, make sure the addressing mode for unaligned store does
 ; is not a base+offset with a non-zero offset that is not a multiple of 128.
 
 ; CHECK: vmemu(r{{[0-9]+}}+#0)
 
 %s.0 = type { [5 x [4 x i8]], i32, i32, i32, i32 }

test/CodeGen/X86/O3-pipeline.ll

 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       Machine Natural Loop Construction
 ; CHECK-NEXT:       Eliminate PHI nodes for register allocation
 ; CHECK-NEXT:       Two-Address instruction pass
 ; CHECK-NEXT:       Slot index numbering
 ; CHECK-NEXT:       Live Interval Analysis
 ; CHECK-NEXT:       Simple Register Coalescing
 ; CHECK-NEXT:       Rename Disconnected Subregister Components
+; CHECK-NEXT:       Remove dead machine instructions
 ; CHECK-NEXT:       Machine Instruction Scheduler
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       Debug Variable Analysis
 ; CHECK-NEXT:       Live Stack Slot Analysis
 ; CHECK-NEXT:       Virtual Register Map
 ; CHECK-NEXT:       Live Register Matrix
 ; CHECK-NEXT:       Bundle Machine CFG Edges
 ; CHECK-NEXT:       Spill Code Placement Analysis

test/CodeGen/X86/combine-bitselect.ll

 ; SSE-NEXT:  .LBB12_1: # %bb1
 ; SSE-NEXT:    # =>This Inner Loop Header: Depth=1
 ; SSE-NEXT:    pand %xmm1, %xmm2
 ; SSE-NEXT:    por %xmm3, %xmm2
 ; SSE-NEXT:    jmp .LBB12_1
 ;
 ; XOP-LABEL: bitselect_v4i1_loop:
 ; XOP:       # %bb.0: # %bb
-; XOP-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; XOP-NEXT:    vpcomneqd %xmm1, %xmm0, %xmm1
-; XOP-NEXT:    vpcomeqd {{.*}}(%rip), %xmm0, %xmm0
-; XOP-NEXT:    vmovdqa %xmm1, %xmm2
 ; XOP-NEXT:    .p2align 4, 0x90
 ; XOP-NEXT:  .LBB12_1: # %bb1
 ; XOP-NEXT:    # =>This Inner Loop Header: Depth=1
-; XOP-NEXT:    vblendvps %xmm1, %xmm0, %xmm2, %xmm2
 ; XOP-NEXT:    jmp .LBB12_1
 ;
 ; AVX1-LABEL: bitselect_v4i1_loop:
 ; AVX1:       # %bb.0: # %bb
-; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:    vpcmpeqd %xmm1, %xmm0, %xmm1
-; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm2, %xmm2
-; AVX1-NEXT:    vpxor %xmm2, %xmm1, %xmm2
-; AVX1-NEXT:    vpcmpeqd {{.*}}(%rip), %xmm0, %xmm0
 ; AVX1-NEXT:    .p2align 4, 0x90
 ; AVX1-NEXT:  .LBB12_1: # %bb1
 ; AVX1-NEXT:    # =>This Inner Loop Header: Depth=1
-; AVX1-NEXT:    vblendvps %xmm1, %xmm2, %xmm0, %xmm2
 ; AVX1-NEXT:    jmp .LBB12_1
 ;
 ; AVX2-LABEL: bitselect_v4i1_loop:
 ; AVX2:       # %bb.0: # %bb
-; AVX2-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX2-NEXT:    vpcmpeqd %xmm1, %xmm0, %xmm1
-; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm2, %xmm2
-; AVX2-NEXT:    vpxor %xmm2, %xmm1, %xmm2
-; AVX2-NEXT:    vpbroadcastd {{.*#+}} xmm3 = [12,12,12,12]
-; AVX2-NEXT:    vpcmpeqd %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    .p2align 4, 0x90
 ; AVX2-NEXT:  .LBB12_1: # %bb1
 ; AVX2-NEXT:    # =>This Inner Loop Header: Depth=1
-; AVX2-NEXT:    vblendvps %xmm1, %xmm2, %xmm0, %xmm2
 ; AVX2-NEXT:    jmp .LBB12_1
 ;
 ; AVX512F-LABEL: bitselect_v4i1_loop:
 ; AVX512F:       # %bb.0: # %bb
 ; AVX512F-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
 ; AVX512F-NEXT:    vptestmd %zmm0, %zmm0, %k1
 ; AVX512F-NEXT:    vpbroadcastd {{.*#+}} xmm1 = [12,12,12,12]
 ; AVX512F-NEXT:    vpcmpeqd %zmm1, %zmm0, %k2
bb:
   br label %bb1
 
 bb1:                                              ; preds = %bb1, %bb
   %tmp2 = phi <4 x i1> [ %tmp, %bb ], [ %tmp4, %bb1 ]
   %tmp3 = icmp eq <4 x i32> %a0, <i32 12, i32 12, i32 12, i32 12>
   %tmp4 = select <4 x i1> %tmp, <4 x i1> %tmp3, <4 x i1> %tmp2
   %tmp5 = and <4 x i1> %tmp4, %tmp
   br label %bb1
 }

test/CodeGen/X86/llc-start-stop-instance.ll

-; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -stop-after=dead-mi-elimination,1 %s -o /dev/null 2>&1 \
+; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -stop-after=dead-mi-elimination,1 -disable-dce-in-ra %s -o /dev/null 2>&1 \
 ; RUN:   | FileCheck -check-prefix=STOP-AFTER-DEAD1 %s
 
-; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -stop-after=dead-mi-elimination,0 %s -o /dev/null 2>&1 \
+; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -stop-after=dead-mi-elimination,0 -disable-dce-in-ra %s -o /dev/null 2>&1 \
 ; RUN:   | FileCheck -check-prefix=STOP-AFTER-DEAD0 %s
 
 
-; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -stop-before=dead-mi-elimination,1 %s -o /dev/null 2>&1 \
+; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -stop-before=dead-mi-elimination,1 -disable-dce-in-ra %s -o /dev/null 2>&1 \
 ; RUN:   | FileCheck -check-prefix=STOP-BEFORE-DEAD1 %s
 
 
-; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -start-before=dead-mi-elimination,1 %s -o /dev/null 2>&1 \
+; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -start-before=dead-mi-elimination,1 -disable-dce-in-ra %s -o /dev/null 2>&1 \
 ; RUN:   | FileCheck -check-prefix=START-BEFORE-DEAD1 %s
 
-; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -start-after=dead-mi-elimination,1 %s -o /dev/null 2>&1 \
+; RUN: llc -mtriple=x86_64-- -debug-pass=Structure -start-after=dead-mi-elimination,1 -disable-dce-in-ra %s -o /dev/null 2>&1 \
 ; RUN:   | FileCheck -check-prefix=START-AFTER-DEAD1 %s
 
 
 
 ; STOP-AFTER-DEAD1: -dead-mi-elimination
 ; STOP-AFTER-DEAD1-SAME: -dead-mi-elimination
 
 ; STOP-AFTER-DEAD1: Remove dead machine instructions

test/CodeGen/X86/speculative-load-hardening-gather.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
 
 declare <4 x float> @llvm.x86.avx2.gather.d.ps(<4 x float>, i8*, <4 x i32>, <4 x float>, i8)
 
 define <4 x float> @test_llvm_x86_avx2_gather_d_ps(i8* %b, <4 x i32> %iv, <4 x float> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_d_ps:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vgatherdps %xmm1, (%rdi,%xmm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x float> @llvm.x86.avx2.gather.d.ps(<4 x float> zeroinitializer, i8* %b, <4 x i32> %iv, <4 x float> %mask, i8 1)
   ret <4 x float> %v
 }
 
 declare <4 x float> @llvm.x86.avx2.gather.q.ps(<4 x float>, i8*, <2 x i64>, <4 x float>, i8)
 
 define <4 x float> @test_llvm_x86_avx2_gather_q_ps(i8* %b, <2 x i64> %iv, <4 x float> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_q_ps:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vgatherqps %xmm1, (%rdi,%xmm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x float> @llvm.x86.avx2.gather.q.ps(<4 x float> zeroinitializer, i8* %b, <2 x i64> %iv, <4 x float> %mask, i8 1)
   ret <4 x float> %v
 }
 
 declare <2 x double> @llvm.x86.avx2.gather.d.pd(<2 x double>, i8*, <4 x i32>, <2 x double>, i8)
 
 define <2 x double> @test_llvm_x86_avx2_gather_d_pd(i8* %b, <4 x i32> %iv, <2 x double> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_d_pd:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vxorpd %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vgatherdpd %xmm1, (%rdi,%xmm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <2 x double> @llvm.x86.avx2.gather.d.pd(<2 x double> zeroinitializer, i8* %b, <4 x i32> %iv, <2 x double> %mask, i8 1)
   ret <2 x double> %v
 }
 
 declare <2 x double> @llvm.x86.avx2.gather.q.pd(<2 x double>, i8*, <2 x i64>, <2 x double>, i8)
 
 define <2 x double> @test_llvm_x86_avx2_gather_q_pd(i8* %b, <2 x i64> %iv, <2 x double> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_q_pd:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vxorpd %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vgatherqpd %xmm1, (%rdi,%xmm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <2 x double> @llvm.x86.avx2.gather.q.pd(<2 x double> zeroinitializer, i8* %b, <2 x i64> %iv, <2 x double> %mask, i8 1)
   ret <2 x double> %v
 }
 
 declare <8 x float> @llvm.x86.avx2.gather.d.ps.256(<8 x float>, i8*, <8 x i32>, <8 x float>, i8)
 
 define <8 x float> @test_llvm_x86_avx2_gather_d_ps_256(i8* %b, <8 x i32> %iv, <8 x float> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_d_ps_256:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %ymm3
 ; CHECK-NEXT:    vpor %ymm0, %ymm3, %ymm0
 ; CHECK-NEXT:    vgatherdps %ymm1, (%rdi,%ymm0), %ymm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %ymm2, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x float> @llvm.x86.avx2.gather.d.ps.256(<8 x float> zeroinitializer, i8* %b, <8 x i32> %iv, <8 x float> %mask, i8 1)
   ret <8 x float> %v
 }
 
 declare <4 x float> @llvm.x86.avx2.gather.q.ps.256(<4 x float>, i8*, <4 x i64>, <4 x float>, i8)
 
 define <4 x float> @test_llvm_x86_avx2_gather_q_ps_256(i8* %b, <4 x i64> %iv, <4 x float> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_q_ps_256:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %ymm3
 ; CHECK-NEXT:    vpor %ymm0, %ymm3, %ymm0
 ; CHECK-NEXT:    vgatherqps %xmm1, (%rdi,%ymm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x float> @llvm.x86.avx2.gather.q.ps.256(<4 x float> zeroinitializer, i8* %b, <4 x i64> %iv, <4 x float> %mask, i8 1)
   ret <4 x float> %v
 }
 
 declare <4 x double> @llvm.x86.avx2.gather.d.pd.256(<4 x double>, i8*, <4 x i32>, <4 x double>, i8)
 
 define <4 x double> @test_llvm_x86_avx2_gather_d_pd_256(i8* %b, <4 x i32> %iv, <4 x double> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_d_pd_256:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vxorpd %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vgatherdpd %ymm1, (%rdi,%xmm0), %ymm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %ymm2, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x double> @llvm.x86.avx2.gather.d.pd.256(<4 x double> zeroinitializer, i8* %b, <4 x i32> %iv, <4 x double> %mask, i8 1)
   ret <4 x double> %v
 }
 
 declare <4 x double> @llvm.x86.avx2.gather.q.pd.256(<4 x double>, i8*, <4 x i64>, <4 x double>, i8)
 
 define <4 x double> @test_llvm_x86_avx2_gather_q_pd_256(i8* %b, <4 x i64> %iv, <4 x double> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_q_pd_256:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vxorpd %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %ymm3
 ; CHECK-NEXT:    vpor %ymm0, %ymm3, %ymm0
 ; CHECK-NEXT:    vgatherqpd %ymm1, (%rdi,%ymm0), %ymm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %ymm2, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x double> @llvm.x86.avx2.gather.q.pd.256(<4 x double> zeroinitializer, i8* %b, <4 x i64> %iv, <4 x double> %mask, i8 1)
   ret <4 x double> %v
 }
 
 declare <4 x i32> @llvm.x86.avx2.gather.d.d(<4 x i32>, i8*, <4 x i32>, <4 x i32>, i8)
 
 define <4 x i32> @test_llvm_x86_avx2_gather_d_d(i8* %b, <4 x i32> %iv, <4 x i32> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_d_d:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vpgatherdd %xmm1, (%rdi,%xmm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i32> @llvm.x86.avx2.gather.d.d(<4 x i32> zeroinitializer, i8* %b, <4 x i32> %iv, <4 x i32> %mask, i8 1)
   ret <4 x i32> %v
 }
 
 declare <4 x i32> @llvm.x86.avx2.gather.q.d(<4 x i32>, i8*, <2 x i64>, <4 x i32>, i8)
 
 define <4 x i32> @test_llvm_x86_avx2_gather_q_d(i8* %b, <2 x i64> %iv, <4 x i32> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_q_d:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vpgatherqd %xmm1, (%rdi,%xmm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i32> @llvm.x86.avx2.gather.q.d(<4 x i32> zeroinitializer, i8* %b, <2 x i64> %iv, <4 x i32> %mask, i8 1)
   ret <4 x i32> %v
 }
 
 declare <2 x i64> @llvm.x86.avx2.gather.d.q(<2 x i64>, i8*, <4 x i32>, <2 x i64>, i8)
 
 define <2 x i64> @test_llvm_x86_avx2_gather_d_q(i8* %b, <4 x i32> %iv, <2 x i64> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_d_q:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vpgatherdq %xmm1, (%rdi,%xmm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <2 x i64> @llvm.x86.avx2.gather.d.q(<2 x i64> zeroinitializer, i8* %b, <4 x i32> %iv, <2 x i64> %mask, i8 1)
   ret <2 x i64> %v
 }
 
 declare <2 x i64> @llvm.x86.avx2.gather.q.q(<2 x i64>, i8*, <2 x i64>, <2 x i64>, i8)
 
 define <2 x i64> @test_llvm_x86_avx2_gather_q_q(i8* %b, <2 x i64> %iv, <2 x i64> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_q_q:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vpgatherqq %xmm1, (%rdi,%xmm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <2 x i64> @llvm.x86.avx2.gather.q.q(<2 x i64> zeroinitializer, i8* %b, <2 x i64> %iv, <2 x i64> %mask, i8 1)
   ret <2 x i64> %v
 }
 
 declare <8 x i32> @llvm.x86.avx2.gather.d.d.256(<8 x i32>, i8*, <8 x i32>, <8 x i32>, i8)
 
 define <8 x i32> @test_llvm_x86_avx2_gather_d_d_256(i8* %b, <8 x i32> %iv, <8 x i32> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_d_d_256:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %ymm3
 ; CHECK-NEXT:    vpor %ymm0, %ymm3, %ymm0
 ; CHECK-NEXT:    vpgatherdd %ymm1, (%rdi,%ymm0), %ymm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %ymm2, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x i32> @llvm.x86.avx2.gather.d.d.256(<8 x i32> zeroinitializer, i8* %b, <8 x i32> %iv, <8 x i32> %mask, i8 1)
   ret <8 x i32> %v
 }
 
 declare <4 x i32> @llvm.x86.avx2.gather.q.d.256(<4 x i32>, i8*, <4 x i64>, <4 x i32>, i8)
 
 define <4 x i32> @test_llvm_x86_avx2_gather_q_d_256(i8* %b, <4 x i64> %iv, <4 x i32> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_q_d_256:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %ymm3
 ; CHECK-NEXT:    vpor %ymm0, %ymm3, %ymm0
 ; CHECK-NEXT:    vpgatherqd %xmm1, (%rdi,%ymm0), %xmm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm2, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i32> @llvm.x86.avx2.gather.q.d.256(<4 x i32> zeroinitializer, i8* %b, <4 x i64> %iv, <4 x i32> %mask, i8 1)
   ret <4 x i32> %v
 }
 
 declare <4 x i64> @llvm.x86.avx2.gather.d.q.256(<4 x i64>, i8*, <4 x i32>, <4 x i64>, i8)
 
 define <4 x i64> @test_llvm_x86_avx2_gather_d_q_256(i8* %b, <4 x i32> %iv, <4 x i64> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_d_q_256:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %xmm3
 ; CHECK-NEXT:    vpor %xmm0, %xmm3, %xmm0
 ; CHECK-NEXT:    vpgatherdq %ymm1, (%rdi,%xmm0), %ymm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %ymm2, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i64> @llvm.x86.avx2.gather.d.q.256(<4 x i64> zeroinitializer, i8* %b, <4 x i32> %iv, <4 x i64> %mask, i8 1)
   ret <4 x i64> %v
 }
 
 declare <4 x i64> @llvm.x86.avx2.gather.q.q.256(<4 x i64>, i8*, <4 x i64>, <4 x i64>, i8)
 
 define <4 x i64> @test_llvm_x86_avx2_gather_q_q_256(i8* %b, <4 x i64> %iv, <4 x i64> %mask) #0 {
 ; CHECK-LABEL: test_llvm_x86_avx2_gather_q_q_256:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm3
 ; CHECK-NEXT:    vpbroadcastq %xmm3, %ymm3
 ; CHECK-NEXT:    vpor %ymm0, %ymm3, %ymm0
 ; CHECK-NEXT:    vpgatherqq %ymm1, (%rdi,%ymm0), %ymm2
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %ymm2, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i64> @llvm.x86.avx2.gather.q.q.256(<4 x i64> zeroinitializer, i8* %b, <4 x i64> %iv, <4 x i64> %mask, i8 1)
   ret <4 x i64> %v
 }
 
 declare <16 x float> @llvm.x86.avx512.gather.dps.512(<16 x float>, i8*, <16 x i32>, i16, i32)
 
 define <16 x float> @test_llvm_x86_avx512_gather_dps_512(i8* %b, <16 x i32> %iv) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather_dps_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %zmm2
 ; CHECK-NEXT:    vporq %zmm0, %zmm2, %zmm0
 ; CHECK-NEXT:    vgatherdps (%rdi,%zmm0), %zmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %zmm1, %zmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <16 x float> @llvm.x86.avx512.gather.dps.512(<16 x float> zeroinitializer, i8* %b, <16 x i32> %iv, i16 -1, i32 1)
   ret <16 x float> %v
 }
 
 declare <8 x double> @llvm.x86.avx512.gather.dpd.512(<8 x double>, i8*, <8 x i32>, i8, i32)
 
 define <8 x double> @test_llvm_x86_avx512_gather_dpd_512(i8* %b, <8 x i32> %iv) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather_dpd_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm2
 ; CHECK-NEXT:    vpbroadcastq %xmm2, %ymm2
 ; CHECK-NEXT:    vpor %ymm0, %ymm2, %ymm0
 ; CHECK-NEXT:    vgatherdpd (%rdi,%ymm0), %zmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %zmm1, %zmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x double> @llvm.x86.avx512.gather.dpd.512(<8 x double> zeroinitializer, i8* %b, <8 x i32> %iv, i8 -1, i32 1)
   ret <8 x double> %v
 }
 
 declare <8 x float> @llvm.x86.avx512.gather.qps.512(<8 x float>, i8*, <8 x i64>, i8, i32)
 
 define <8 x float> @test_llvm_x86_avx512_gather_qps_512(i8* %b, <8 x i64> %iv) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather_qps_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %zmm2
 ; CHECK-NEXT:    vporq %zmm0, %zmm2, %zmm0
 ; CHECK-NEXT:    vgatherqps (%rdi,%zmm0), %ymm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %ymm1, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x float> @llvm.x86.avx512.gather.qps.512(<8 x float> zeroinitializer, i8* %b, <8 x i64> %iv, i8 -1, i32 1)
   ret <8 x float> %v
 }
 
 declare <8 x double> @llvm.x86.avx512.gather.qpd.512(<8 x double>, i8*, <8 x i64>, i8, i32)
 
 define <8 x double> @test_llvm_x86_avx512_gather_qpd_512(i8* %b, <8 x i64> %iv) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather_qpd_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %zmm2
 ; CHECK-NEXT:    vporq %zmm0, %zmm2, %zmm0
 ; CHECK-NEXT:    vgatherqpd (%rdi,%zmm0), %zmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %zmm1, %zmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x double> @llvm.x86.avx512.gather.qpd.512(<8 x double> zeroinitializer, i8* %b, <8 x i64> %iv, i8 -1, i32 1)
   ret <8 x double> %v
 }
 
 declare <16 x i32> @llvm.x86.avx512.gather.dpi.512(<16 x i32>, i8*, <16 x i32>, i16, i32)
 
 define <16 x i32> @test_llvm_x86_avx512_gather_dpi_512(i8* %b, <16 x i32> %iv) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather_dpi_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %zmm2
 ; CHECK-NEXT:    vporq %zmm0, %zmm2, %zmm0
 ; CHECK-NEXT:    vpgatherdd (%rdi,%zmm0), %zmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa64 %zmm1, %zmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <16 x i32> @llvm.x86.avx512.gather.dpi.512(<16 x i32> zeroinitializer, i8* %b, <16 x i32> %iv, i16 -1, i32 1)
   ret <16 x i32> %v
 }
 
 declare <8 x i64> @llvm.x86.avx512.gather.dpq.512(<8 x i64>, i8*, <8 x i32>, i8, i32)
 
 define <8 x i64> @test_llvm_x86_avx512_gather_dpq_512(i8* %b, <8 x i32> %iv) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather_dpq_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vmovq %rax, %xmm2
 ; CHECK-NEXT:    vpbroadcastq %xmm2, %ymm2
 ; CHECK-NEXT:    vpor %ymm0, %ymm2, %ymm0
 ; CHECK-NEXT:    vpgatherdq (%rdi,%ymm0), %zmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa64 %zmm1, %zmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x i64> @llvm.x86.avx512.gather.dpq.512(<8 x i64> zeroinitializer, i8* %b, <8 x i32> %iv, i8 -1, i32 1)
   ret <8 x i64> %v
 }
 
 
 declare <8 x i32> @llvm.x86.avx512.gather.qpi.512(<8 x i32>, i8*, <8 x i64>, i8, i32)
 
 define <8 x i32> @test_llvm_x86_avx512_gather_qpi_512(i8* %b, <8 x i64> %iv) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather_qpi_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %zmm2
 ; CHECK-NEXT:    vporq %zmm0, %zmm2, %zmm0
 ; CHECK-NEXT:    vpgatherqd (%rdi,%zmm0), %ymm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %ymm1, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x i32> @llvm.x86.avx512.gather.qpi.512(<8 x i32> zeroinitializer, i8* %b, <8 x i64> %iv, i8 -1, i32 1)
   ret <8 x i32> %v
 }
 
 declare <8 x i64> @llvm.x86.avx512.gather.qpq.512(<8 x i64>, i8*, <8 x i64>, i8, i32)
 
 define <8 x i64> @test_llvm_x86_avx512_gather_qpq_512(i8* %b, <8 x i64> %iv) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather_qpq_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %zmm2
 ; CHECK-NEXT:    vporq %zmm0, %zmm2, %zmm0
 ; CHECK-NEXT:    vpgatherqq (%rdi,%zmm0), %zmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa64 %zmm1, %zmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x i64> @llvm.x86.avx512.gather.qpq.512(<8 x i64> zeroinitializer, i8* %b, <8 x i64> %iv, i8 -1, i32 1)
   ret <8 x i64> %v
 }
 
 declare void @llvm.x86.avx512.gatherpf.qps.512(i8, <8 x i64>, i8*, i32, i32);
 
 define void @test_llvm_x86_avx512_gatherpf_qps_512(<8 x i64> %iv, i8* %b) #1 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gatherpf_qps_512:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %zmm1
 ; CHECK-NEXT:    vporq %zmm0, %zmm1, %zmm0
 ; CHECK-NEXT:    vgatherpf0qps (%rdi,%zmm0,4) {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   call void @llvm.x86.avx512.gatherpf.qps.512(i8 -1, <8 x i64> %iv, i8* %b, i32 4, i32 3)
   ret void
 }
 
 declare <4 x float> @llvm.x86.avx512.gather3siv4.sf(<4 x float>, i8*, <4 x i32>, i8, i32)
 
 define <4 x float> @test_llvm_x86_avx512_gather3siv4_sf(i8* %b, <4 x i32> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3siv4_sf:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vgatherdps (%rdi,%xmm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x float> @llvm.x86.avx512.gather3siv4.sf(<4 x float> zeroinitializer, i8* %b, <4 x i32> %iv, i8 -1, i32 1)
   ret <4 x float> %v
 }
 
 declare <4 x float> @llvm.x86.avx512.gather3div4.sf(<4 x float>, i8*, <2 x i64>, i8, i32)
 
 define <4 x float> @test_llvm_x86_avx512_gather3div4_sf(i8* %b, <2 x i64> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3div4_sf:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vgatherqps (%rdi,%xmm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x float> @llvm.x86.avx512.gather3div4.sf(<4 x float> zeroinitializer, i8* %b, <2 x i64> %iv, i8 -1, i32 1)
   ret <4 x float> %v
 }
 
 declare <2 x double> @llvm.x86.avx512.gather3siv2.df(<2 x double>, i8*, <4 x i32>, i8, i32)
 
 define <2 x double> @test_llvm_x86_avx512_gather3siv2_df(i8* %b, <4 x i32> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3siv2_df:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vgatherdpd (%rdi,%xmm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <2 x double> @llvm.x86.avx512.gather3siv2.df(<2 x double> zeroinitializer, i8* %b, <4 x i32> %iv, i8 -1, i32 1)
   ret <2 x double> %v
 }
 
 declare <2 x double> @llvm.x86.avx512.gather3div2.df(<2 x double>, i8*, <2 x i64>, i8, i32)
 
 define <2 x double> @test_llvm_x86_avx512_gather3div2_df(i8* %b, <2 x i64> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3div2_df:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vgatherqpd (%rdi,%xmm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <2 x double> @llvm.x86.avx512.gather3div2.df(<2 x double> zeroinitializer, i8* %b, <2 x i64> %iv, i8 -1, i32 1)
   ret <2 x double> %v
 }
 
 declare <8 x float> @llvm.x86.avx512.gather3siv8.sf(<8 x float>, i8*, <8 x i32>, i8, i32)
 
 define <8 x float> @test_llvm_x86_avx512_gather3siv8_sf(i8* %b, <8 x i32> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3siv8_sf:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %ymm2
 ; CHECK-NEXT:    vpor %ymm0, %ymm2, %ymm0
 ; CHECK-NEXT:    vgatherdps (%rdi,%ymm0), %ymm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %ymm1, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x float> @llvm.x86.avx512.gather3siv8.sf(<8 x float> zeroinitializer, i8* %b, <8 x i32> %iv, i8 -1, i32 1)
   ret <8 x float> %v
 }
 
 declare <4 x float> @llvm.x86.avx512.gather3div8.sf(<4 x float>, i8*, <4 x i64>, i8, i32)
 
 define <4 x float> @test_llvm_x86_avx512_gather3div8_sf(i8* %b, <4 x i64> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3div8_sf:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %ymm2
 ; CHECK-NEXT:    vpor %ymm0, %ymm2, %ymm0
 ; CHECK-NEXT:    vgatherqps (%rdi,%ymm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x float> @llvm.x86.avx512.gather3div8.sf(<4 x float> zeroinitializer, i8* %b, <4 x i64> %iv, i8 -1, i32 1)
   ret <4 x float> %v
 }
 
 declare <4 x double> @llvm.x86.avx512.gather3siv4.df(<4 x double>, i8*, <4 x i32>, i8, i32)
 
 define <4 x double> @test_llvm_x86_avx512_gather3siv4_df(i8* %b, <4 x i32> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3siv4_df:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vgatherdpd (%rdi,%xmm0), %ymm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %ymm1, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x double> @llvm.x86.avx512.gather3siv4.df(<4 x double> zeroinitializer, i8* %b, <4 x i32> %iv, i8 -1, i32 1)
   ret <4 x double> %v
 }
 
 declare <4 x double> @llvm.x86.avx512.gather3div4.df(<4 x double>, i8*, <4 x i64>, i8, i32)
 
 define <4 x double> @test_llvm_x86_avx512_gather3div4_df(i8* %b, <4 x i64> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3div4_df:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %ymm2
 ; CHECK-NEXT:    vpor %ymm0, %ymm2, %ymm0
 ; CHECK-NEXT:    vgatherqpd (%rdi,%ymm0), %ymm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovapd %ymm1, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x double> @llvm.x86.avx512.gather3div4.df(<4 x double> zeroinitializer, i8* %b, <4 x i64> %iv, i8 -1, i32 1)
   ret <4 x double> %v
 }
 
 declare <4 x i32> @llvm.x86.avx512.gather3siv4.si(<4 x i32>, i8*, <4 x i32>, i8, i32)
 
 define <4 x i32> @test_llvm_x86_avx512_gather3siv4_si(i8* %b, <4 x i32> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3siv4_si:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vpgatherdd (%rdi,%xmm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i32> @llvm.x86.avx512.gather3siv4.si(<4 x i32> zeroinitializer, i8* %b, <4 x i32> %iv, i8 -1, i32 1)
   ret <4 x i32> %v
 }
 
 declare <4 x i32> @llvm.x86.avx512.gather3div4.si(<4 x i32>, i8*, <2 x i64>, i8, i32)
 
 define <4 x i32> @test_llvm_x86_avx512_gather3div4_si(i8* %b, <2 x i64> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3div4_si:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vpgatherqd (%rdi,%xmm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i32> @llvm.x86.avx512.gather3div4.si(<4 x i32> zeroinitializer, i8* %b, <2 x i64> %iv, i8 -1, i32 1)
   ret <4 x i32> %v
 }
 
 declare <2 x i64> @llvm.x86.avx512.gather3siv2.di(<2 x i64>, i8*, <4 x i32>, i8, i32)
 
 define <2 x i64> @test_llvm_x86_avx512_gather3siv2_di(i8* %b, <4 x i32> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3siv2_di:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vpgatherdq (%rdi,%xmm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <2 x i64> @llvm.x86.avx512.gather3siv2.di(<2 x i64> zeroinitializer, i8* %b, <4 x i32> %iv, i8 -1, i32 1)
   ret <2 x i64> %v
 }
 
 declare <2 x i64> @llvm.x86.avx512.gather3div2.di(<2 x i64>, i8*, <2 x i64>, i8, i32)
 
 define <2 x i64> @test_llvm_x86_avx512_gather3div2_di(i8* %b, <2 x i64> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3div2_di:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vpgatherqq (%rdi,%xmm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <2 x i64> @llvm.x86.avx512.gather3div2.di(<2 x i64> zeroinitializer, i8* %b, <2 x i64> %iv, i8 -1, i32 1)
   ret <2 x i64> %v
 }
 
 declare <8 x i32> @llvm.x86.avx512.gather3siv8.si(<8 x i32>, i8*, <8 x i32>, i8, i32)
 
 define <8 x i32> @test_llvm_x86_avx512_gather3siv8_si(i8* %b, <8 x i32> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3siv8_si:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %ymm2
 ; CHECK-NEXT:    vpor %ymm0, %ymm2, %ymm0
 ; CHECK-NEXT:    vpgatherdd (%rdi,%ymm0), %ymm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %ymm1, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <8 x i32> @llvm.x86.avx512.gather3siv8.si(<8 x i32> zeroinitializer, i8* %b, <8 x i32> %iv, i8 -1, i32 1)
   ret <8 x i32> %v
 }
 
 declare <4 x i32> @llvm.x86.avx512.gather3div8.si(<4 x i32>, i8*, <4 x i64>, i8, i32)
 
 define <4 x i32> @test_llvm_x86_avx512_gather3div8_si(i8* %b, <4 x i64> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3div8_si:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %ymm2
 ; CHECK-NEXT:    vpor %ymm0, %ymm2, %ymm0
 ; CHECK-NEXT:    vpgatherqd (%rdi,%ymm0), %xmm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %xmm1, %xmm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i32> @llvm.x86.avx512.gather3div8.si(<4 x i32> zeroinitializer, i8* %b, <4 x i64> %iv, i8 -1, i32 1)
   ret <4 x i32> %v
 }
 
 declare <4 x i64> @llvm.x86.avx512.gather3siv4.di(<4 x i64>, i8*, <4 x i32>, i8, i32)
 
 define <4 x i64> @test_llvm_x86_avx512_gather3siv4_di(i8* %b, <4 x i32> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3siv4_di:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %xmm2
 ; CHECK-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-NEXT:    vpgatherdq (%rdi,%xmm0), %ymm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax
 ; CHECK-NEXT:    vmovdqa %ymm1, %ymm0
 ; CHECK-NEXT:    orq %rax, %rsp
 ; CHECK-NEXT:    retq
 entry:
   %v = call <4 x i64> @llvm.x86.avx512.gather3siv4.di(<4 x i64> zeroinitializer, i8* %b, <4 x i32> %iv, i8 -1, i32 1)
   ret <4 x i64> %v
 }
 
 declare <4 x i64> @llvm.x86.avx512.gather3div4.di(<4 x i64>, i8*, <4 x i64>, i8, i32)
 
 define <4 x i64> @test_llvm_x86_avx512_gather3div4_di(i8* %b, <4 x i64> %iv) #2 {
 ; CHECK-LABEL: test_llvm_x86_avx512_gather3div4_di:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq %rsp, %rax
-; CHECK-NEXT:    movq $-1, %rcx
 ; CHECK-NEXT:    sarq $63, %rax
 ; CHECK-NEXT:    kxnorw %k0, %k0, %k1
 ; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    orq %rax, %rdi
 ; CHECK-NEXT:    vpbroadcastq %rax, %ymm2
 ; CHECK-NEXT:    vpor %ymm0, %ymm2, %ymm0
 ; CHECK-NEXT:    vpgatherqq (%rdi,%ymm0), %ymm1 {%k1}
 ; CHECK-NEXT:    shlq $47, %rax

test/CodeGen/X86/speculative-load-hardening-indirect.ll

 ; X64-PIC-NEXT:    orq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rax
 ; X64-PIC-NEXT:    orq %rax, %rsp
 ; X64-PIC-NEXT:    callq *%rcx
 ; X64-PIC-NEXT:  .Lslh_ret_addr0:
 ; X64-PIC-NEXT:    movq %rsp, %rcx
 ; X64-PIC-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
 ; X64-PIC-NEXT:    sarq $63, %rcx
-; X64-PIC-NEXT:    leaq .Lslh_ret_addr0(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rbx, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    popq %rbx
 ; X64-PIC-NEXT:    retq
 ;
 ; X64-RETPOLINE-LABEL: test_indirect_call:
entry:
   %v = call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_tail_call(i32 ()** %ptr) nounwind {
 ; X64-LABEL: test_indirect_tail_call:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    movq $-1, %rcx
 ; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    movq (%rdi), %rcx
 ; X64-NEXT:    orq %rax, %rcx
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    jmpq *%rcx # TAILCALL
 ;
 ; X64-PIC-LABEL: test_indirect_tail_call:
 ; X64-PIC:       # %bb.0: # %entry
 ; X64-PIC-NEXT:    movq %rsp, %rax
-; X64-PIC-NEXT:    movq $-1, %rcx
 ; X64-PIC-NEXT:    sarq $63, %rax
 ; X64-PIC-NEXT:    movq (%rdi), %rcx
 ; X64-PIC-NEXT:    orq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rax
 ; X64-PIC-NEXT:    orq %rax, %rsp
 ; X64-PIC-NEXT:    jmpq *%rcx # TAILCALL
 ;
 ; X64-RETPOLINE-LABEL: test_indirect_tail_call:
 ; X64-RETPOLINE:       # %bb.0: # %entry
 ; X64-RETPOLINE-NEXT:    movq %rsp, %rax
-; X64-RETPOLINE-NEXT:    movq $-1, %rcx
 ; X64-RETPOLINE-NEXT:    sarq $63, %rax
 ; X64-RETPOLINE-NEXT:    movq (%rdi), %r11
 ; X64-RETPOLINE-NEXT:    orq %rax, %r11
 ; X64-RETPOLINE-NEXT:    shlq $47, %rax
 ; X64-RETPOLINE-NEXT:    orq %rax, %rsp
 ; X64-RETPOLINE-NEXT:    jmp __llvm_retpoline_r11 # TAILCALL
 entry:
   %fp = load i32 ()*, i32 ()** %ptr
   %v = tail call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_call_global() nounwind {
 ; X64-LABEL: test_indirect_call_global:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    pushq %rbx
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    movq $-1, %rbx
 ; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    movq global_fnptr(%rip), %rcx
+; X64-NEXT:    movq {{.*}}(%rip), %rcx
 ; X64-NEXT:    orq %rax, %rcx
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq *%rcx
 ; X64-NEXT:  .Lslh_ret_addr1:
 ; X64-NEXT:    movq %rsp, %rcx
 ; X64-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
 ; X64-NEXT:    sarq $63, %rcx
 ; X64-NEXT:    cmpq $.Lslh_ret_addr1, %rdx
 ; X64-NEXT:    cmovneq %rbx, %rcx
 ; X64-NEXT:    shlq $47, %rcx
 ; X64-NEXT:    orq %rcx, %rsp
 ; X64-NEXT:    popq %rbx
 ; X64-NEXT:    retq
 ;
 ; X64-PIC-LABEL: test_indirect_call_global:
 ; X64-PIC:       # %bb.0: # %entry
 ; X64-PIC-NEXT:    pushq %rbx
 ; X64-PIC-NEXT:    movq %rsp, %rax
 ; X64-PIC-NEXT:    movq $-1, %rbx
 ; X64-PIC-NEXT:    sarq $63, %rax
-; X64-PIC-NEXT:    movq global_fnptr@GOTPCREL(%rip), %rcx
+; X64-PIC-NEXT:    movq global_fnptr@{{.*}}(%rip), %rcx
 ; X64-PIC-NEXT:    movq (%rcx), %rcx
 ; X64-PIC-NEXT:    orq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rax
 ; X64-PIC-NEXT:    orq %rax, %rsp
 ; X64-PIC-NEXT:    callq *%rcx
 ; X64-PIC-NEXT:  .Lslh_ret_addr1:
 ; X64-PIC-NEXT:    movq %rsp, %rcx
 ; X64-PIC-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
 ; X64-PIC-NEXT:    sarq $63, %rcx
-; X64-PIC-NEXT:    leaq .Lslh_ret_addr1(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rbx, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    popq %rbx
 ; X64-PIC-NEXT:    retq
 ;
 ; X64-RETPOLINE-LABEL: test_indirect_call_global:
 ; X64-RETPOLINE:       # %bb.0: # %entry
 ; X64-RETPOLINE-NEXT:    pushq %rbx
 ; X64-RETPOLINE-NEXT:    movq %rsp, %rax
 ; X64-RETPOLINE-NEXT:    movq $-1, %rbx
 ; X64-RETPOLINE-NEXT:    sarq $63, %rax
-; X64-RETPOLINE-NEXT:    movq global_fnptr(%rip), %r11
+; X64-RETPOLINE-NEXT:    movq {{.*}}(%rip), %r11
 ; X64-RETPOLINE-NEXT:    shlq $47, %rax
 ; X64-RETPOLINE-NEXT:    orq %rax, %rsp
 ; X64-RETPOLINE-NEXT:    callq __llvm_retpoline_r11
 ; X64-RETPOLINE-NEXT:  .Lslh_ret_addr1:
 ; X64-RETPOLINE-NEXT:    movq %rsp, %rcx
 ; X64-RETPOLINE-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
 ; X64-RETPOLINE-NEXT:    sarq $63, %rcx
 ; X64-RETPOLINE-NEXT:    cmpq $.Lslh_ret_addr1, %rdx
 ; X64-RETPOLINE-NEXT:    cmovneq %rbx, %rcx
 ; X64-RETPOLINE-NEXT:    shlq $47, %rcx
 ; X64-RETPOLINE-NEXT:    orq %rcx, %rsp
 ; X64-RETPOLINE-NEXT:    popq %rbx
 ; X64-RETPOLINE-NEXT:    retq
 entry:
   %fp = load i32 ()*, i32 ()** @global_fnptr
   %v = call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_tail_call_global() nounwind {
 ; X64-LABEL: test_indirect_tail_call_global:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    movq $-1, %rcx
 ; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    movq global_fnptr(%rip), %rcx
+; X64-NEXT:    movq {{.*}}(%rip), %rcx
 ; X64-NEXT:    orq %rax, %rcx
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    jmpq *%rcx # TAILCALL
 ;
 ; X64-PIC-LABEL: test_indirect_tail_call_global:
 ; X64-PIC:       # %bb.0: # %entry
 ; X64-PIC-NEXT:    movq %rsp, %rax
-; X64-PIC-NEXT:    movq $-1, %rcx
 ; X64-PIC-NEXT:    sarq $63, %rax
-; X64-PIC-NEXT:    movq global_fnptr@GOTPCREL(%rip), %rcx
+; X64-PIC-NEXT:    movq global_fnptr@{{.*}}(%rip), %rcx
 ; X64-PIC-NEXT:    movq (%rcx), %rcx
 ; X64-PIC-NEXT:    orq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rax
 ; X64-PIC-NEXT:    orq %rax, %rsp
 ; X64-PIC-NEXT:    jmpq *%rcx # TAILCALL
 ;
 ; X64-RETPOLINE-LABEL: test_indirect_tail_call_global:
 ; X64-RETPOLINE:       # %bb.0: # %entry
 ; X64-RETPOLINE-NEXT:    movq %rsp, %rax
-; X64-RETPOLINE-NEXT:    movq $-1, %rcx
 ; X64-RETPOLINE-NEXT:    sarq $63, %rax
-; X64-RETPOLINE-NEXT:    movq global_fnptr(%rip), %r11
+; X64-RETPOLINE-NEXT:    movq {{.*}}(%rip), %r11
 ; X64-RETPOLINE-NEXT:    shlq $47, %rax
 ; X64-RETPOLINE-NEXT:    orq %rax, %rsp
 ; X64-RETPOLINE-NEXT:    jmp __llvm_retpoline_r11 # TAILCALL
 entry:
   %fp = load i32 ()*, i32 ()** @global_fnptr
   %v = tail call i32 %fp()
   ret i32 %v
 }

 ; X64-PIC-NEXT:    movq %rsp, %rcx
 ; X64-PIC-NEXT:    movq $-1, %rax
 ; X64-PIC-NEXT:    sarq $63, %rcx
 ; X64-PIC-NEXT:    movq (%rdi), %rdx
 ; X64-PIC-NEXT:    orq %rcx, %rdx
 ; X64-PIC-NEXT:    jmpq *%rdx
 ; X64-PIC-NEXT:  .LBB4_1: # Block address taken
 ; X64-PIC-NEXT:    # %bb0
-; X64-PIC-NEXT:    leaq .LBB4_1(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $2, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .LBB4_3: # Block address taken
 ; X64-PIC-NEXT:    # %bb2
-; X64-PIC-NEXT:    leaq .LBB4_3(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $13, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .LBB4_4: # Block address taken
 ; X64-PIC-NEXT:    # %bb3
-; X64-PIC-NEXT:    leaq .LBB4_4(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $42, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .LBB4_2: # Block address taken
 ; X64-PIC-NEXT:    # %bb1
-; X64-PIC-NEXT:    leaq .LBB4_2(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $7, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ;
 ; X64-RETPOLINE-LABEL: test_indirectbr:

 ; X64-NEXT:    retq
 ;
 ; X64-PIC-LABEL: test_indirectbr_global:
 ; X64-PIC:       # %bb.0: # %entry
 ; X64-PIC-NEXT:    movq %rsp, %rcx
 ; X64-PIC-NEXT:    movq $-1, %rax
 ; X64-PIC-NEXT:    sarq $63, %rcx
 ; X64-PIC-NEXT:    movslq %edi, %rdx
-; X64-PIC-NEXT:    movq global_blockaddrs@GOTPCREL(%rip), %rsi
+; X64-PIC-NEXT:    movq global_blockaddrs@{{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    movq (%rsi,%rdx,8), %rdx
 ; X64-PIC-NEXT:    orq %rcx, %rdx
 ; X64-PIC-NEXT:    jmpq *%rdx
 ; X64-PIC-NEXT:  .Ltmp0: # Block address taken
 ; X64-PIC-NEXT:  .LBB5_1: # %bb0
-; X64-PIC-NEXT:    leaq .LBB5_1(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $2, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .Ltmp1: # Block address taken
 ; X64-PIC-NEXT:  .LBB5_3: # %bb2
-; X64-PIC-NEXT:    leaq .LBB5_3(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $13, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .Ltmp2: # Block address taken
 ; X64-PIC-NEXT:  .LBB5_4: # %bb3
-; X64-PIC-NEXT:    leaq .LBB5_4(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $42, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .Ltmp3: # Block address taken
 ; X64-PIC-NEXT:  .LBB5_2: # %bb1
-; X64-PIC-NEXT:    leaq .LBB5_2(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $7, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ;
 ; X64-RETPOLINE-LABEL: test_indirectbr_global:

 ; X64-PIC-NEXT:    movq %rsp, %rcx
 ; X64-PIC-NEXT:    movq $-1, %rax
 ; X64-PIC-NEXT:    sarq $63, %rcx
 ; X64-PIC-NEXT:    cmpl $3, %edi
 ; X64-PIC-NEXT:    ja .LBB6_2
 ; X64-PIC-NEXT:  # %bb.1: # %entry
 ; X64-PIC-NEXT:    cmovaq %rax, %rcx
 ; X64-PIC-NEXT:    movl %edi, %edx
-; X64-PIC-NEXT:    leaq .LJTI6_0(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    movslq (%rsi,%rdx,4), %rdx
 ; X64-PIC-NEXT:    addq %rsi, %rdx
 ; X64-PIC-NEXT:    orq %rcx, %rdx
 ; X64-PIC-NEXT:    jmpq *%rdx
 ; X64-PIC-NEXT:  .LBB6_3: # Block address taken
 ; X64-PIC-NEXT:    # %bb1
-; X64-PIC-NEXT:    leaq .LBB6_3(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $7, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .LBB6_2: # %bb0
 ; X64-PIC-NEXT:    cmovbeq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $2, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .LBB6_4: # Block address taken
 ; X64-PIC-NEXT:    # %bb2
-; X64-PIC-NEXT:    leaq .LBB6_4(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $13, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .LBB6_5: # Block address taken
 ; X64-PIC-NEXT:    # %bb3
-; X64-PIC-NEXT:    leaq .LBB6_5(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $42, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ; X64-PIC-NEXT:  .LBB6_6: # Block address taken
 ; X64-PIC-NEXT:    # %bb5
-; X64-PIC-NEXT:    leaq .LBB6_6(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:    cmpq %rsi, %rdx
 ; X64-PIC-NEXT:    cmovneq %rax, %rcx
 ; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl $11, %eax
 ; X64-PIC-NEXT:    orq %rcx, %rsp
 ; X64-PIC-NEXT:    retq
 ;
 ; X64-RETPOLINE-LABEL: test_switch_jumptable:

 ; X64-PIC-NEXT:    movq $-1, %r10
 ; X64-PIC-NEXT:    sarq $63, %r9
 ; X64-PIC-NEXT:    cmpl $3, %edi
 ; X64-PIC-NEXT:    ja .LBB7_2
 ; X64-PIC-NEXT:  # %bb.1: # %entry
 ; X64-PIC-NEXT:    cmovaq %r10, %r9
 ; X64-PIC-NEXT:    xorl %eax, %eax
 ; X64-PIC-NEXT:    movl %edi, %esi
-; X64-PIC-NEXT:    leaq .LJTI7_0(%rip), %rdi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rdi
 ; X64-PIC-NEXT:    movslq (%rdi,%rsi,4), %rsi
 ; X64-PIC-NEXT:    addq %rdi, %rsi
 ; X64-PIC-NEXT:    orq %r9, %rsi
 ; X64-PIC-NEXT:    jmpq *%rsi
 ; X64-PIC-NEXT:  .LBB7_2: # %bb0
 ; X64-PIC-NEXT:    cmovbeq %r10, %r9
 ; X64-PIC-NEXT:    movl (%rsi), %eax
 ; X64-PIC-NEXT:    orl %r9d, %eax
-; X64-PIC-NEXT:    leaq .LBB7_3(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:  .LBB7_3: # Block address taken
 ; X64-PIC-NEXT:    # %bb1
-; X64-PIC-NEXT:    leaq .LBB7_3(%rip), %rdi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rdi
 ; X64-PIC-NEXT:    cmpq %rdi, %rsi
 ; X64-PIC-NEXT:    cmovneq %r10, %r9
 ; X64-PIC-NEXT:    addl (%rdx), %eax
 ; X64-PIC-NEXT:    orl %r9d, %eax
-; X64-PIC-NEXT:    leaq .LBB7_4(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:  .LBB7_4: # Block address taken
 ; X64-PIC-NEXT:    # %bb2
-; X64-PIC-NEXT:    leaq .LBB7_4(%rip), %rdx
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rdx
 ; X64-PIC-NEXT:    cmpq %rdx, %rsi
 ; X64-PIC-NEXT:    cmovneq %r10, %r9
 ; X64-PIC-NEXT:    addl (%rcx), %eax
 ; X64-PIC-NEXT:    orl %r9d, %eax
-; X64-PIC-NEXT:    leaq .LBB7_5(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:  .LBB7_5: # Block address taken
 ; X64-PIC-NEXT:    # %bb3
-; X64-PIC-NEXT:    leaq .LBB7_5(%rip), %rcx
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rcx
 ; X64-PIC-NEXT:    cmpq %rcx, %rsi
 ; X64-PIC-NEXT:    cmovneq %r10, %r9
 ; X64-PIC-NEXT:    addl (%r8), %eax
 ; X64-PIC-NEXT:    orl %r9d, %eax
-; X64-PIC-NEXT:    leaq .LBB7_6(%rip), %rsi
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rsi
 ; X64-PIC-NEXT:  .LBB7_6: # Block address taken
 ; X64-PIC-NEXT:    # %bb4
-; X64-PIC-NEXT:    leaq .LBB7_6(%rip), %rcx
+; X64-PIC-NEXT:    leaq {{.*}}(%rip), %rcx
 ; X64-PIC-NEXT:    cmpq %rcx, %rsi
 ; X64-PIC-NEXT:    cmovneq %r10, %r9
 ; X64-PIC-NEXT:    shlq $47, %r9
 ; X64-PIC-NEXT:    orq %r9, %rsp
 ; X64-PIC-NEXT:    retq
 ;
 ; X64-RETPOLINE-LABEL: test_switch_jumptable_fallthrough:
 ; X64-RETPOLINE:       # %bb.0: # %entry

test/CodeGen/X86/speculative-load-hardening.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s --check-prefix=X64
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -x86-slh-lfence | FileCheck %s --check-prefix=X64-LFENCE
 ;
 ; FIXME: Add support for 32-bit and other EH ABIs.
 
 declare void @leak(i32 %v1, i32 %v2)
 
 declare void @sink(i32)
 
 define i32 @test_trivial_entry_load(i32* %ptr) speculative_load_hardening {
 ; X64-LABEL: test_trivial_entry_load:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movq %rsp, %rcx
-; X64-NEXT:    movq $-1, %rax
 ; X64-NEXT:    sarq $63, %rcx
 ; X64-NEXT:    movl (%rdi), %eax
 ; X64-NEXT:    orl %ecx, %eax
 ; X64-NEXT:    shlq $47, %rcx
 ; X64-NEXT:    orq %rcx, %rsp
 ; X64-NEXT:    retq
 ;
 ; X64-LFENCE-LABEL: test_trivial_entry_load:

test/CodeGen/X86/tail-dup-merge-loop-headers.ll

 ; CHECK-NEXT:    jb .LBB1_25
 ; CHECK-NEXT:  # %bb.5: # %if.end50
 ; CHECK-NEXT:    movq %rbx, %rdi
 ; CHECK-NEXT:    movq %r15, %rdx
 ; CHECK-NEXT:    callq memcpy
 ; CHECK-NEXT:    cmpl $4, %r14d
 ; CHECK-NEXT:    jb .LBB1_28
 ; CHECK-NEXT:  # %bb.6: # %shared_preheader
-; CHECK-NEXT:    movb $32, %dl
+; CHECK-NEXT:    movb $32, %cl
 ; CHECK-NEXT:    xorl %eax, %eax
-; CHECK-NEXT:    # implicit-def: $rcx
 ; CHECK-NEXT:    testl %ebp, %ebp
 ; CHECK-NEXT:    je .LBB1_18
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB1_8: # %shared_loop_header
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    testq %rbx, %rbx
 ; CHECK-NEXT:    jne .LBB1_27
 ; CHECK-NEXT:  # %bb.9: # %inner_loop_body
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
 ; CHECK-NEXT:    testl %eax, %eax
 ; CHECK-NEXT:    jns .LBB1_8
 ; CHECK-NEXT:  # %bb.10: # %if.end96.i
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
 ; CHECK-NEXT:    cmpl $3, %ebp
 ; CHECK-NEXT:    jae .LBB1_22
 ; CHECK-NEXT:  # %bb.11: # %if.end287.i
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    xorl %esi, %esi
+; CHECK-NEXT:    xorl %edx, %edx
 ; CHECK-NEXT:    cmpl $1, %ebp
-; CHECK-NEXT:    setne %dl
+; CHECK-NEXT:    setne %cl
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    jne .LBB1_15
 ; CHECK-NEXT:  # %bb.12: # %if.end308.i
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    je .LBB1_17
 ; CHECK-NEXT:  # %bb.13: # %if.end335.i
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    xorl %edx, %edx
-; CHECK-NEXT:    testb %dl, %dl
-; CHECK-NEXT:    movl $0, %esi
+; CHECK-NEXT:    xorl %ecx, %ecx
+; CHECK-NEXT:    testb %cl, %cl
+; CHECK-NEXT:    movl $0, %edx
 ; CHECK-NEXT:    jne .LBB1_7
 ; CHECK-NEXT:  # %bb.14: # %merge_other
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    xorl %esi, %esi
+; CHECK-NEXT:    xorl %edx, %edx
 ; CHECK-NEXT:    jmp .LBB1_16
 ; CHECK-NEXT:  .LBB1_15: # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    movb %dl, %sil
-; CHECK-NEXT:    addl $3, %esi
+; CHECK-NEXT:    movb %cl, %dl
+; CHECK-NEXT:    addl $3, %edx
 ; CHECK-NEXT:  .LBB1_16: # %outer_loop_latch
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    # implicit-def: $dl
+; CHECK-NEXT:    # implicit-def: $cl
 ; CHECK-NEXT:    jmp .LBB1_7
 ; CHECK-NEXT:  .LBB1_17: # %merge_predecessor_split
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    movb $32, %dl
-; CHECK-NEXT:    xorl %esi, %esi
+; CHECK-NEXT:    movb $32, %cl
+; CHECK-NEXT:    xorl %edx, %edx
 ; CHECK-NEXT:  .LBB1_7: # %outer_loop_latch
 ; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    movzwl %si, %esi
-; CHECK-NEXT:    decl %esi
-; CHECK-NEXT:    movzwl %si, %esi
-; CHECK-NEXT:    leaq 1(%rcx,%rsi), %rcx
+; CHECK-NEXT:    movzwl %dx, %edx
+; CHECK-NEXT:    decl %edx
 ; CHECK-NEXT:    testl %ebp, %ebp
 ; CHECK-NEXT:    jne .LBB1_8
 ; CHECK-NEXT:  .LBB1_18: # %while.cond.us1412.i
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    movl $1, %ebx
 ; CHECK-NEXT:    jne .LBB1_20
 ; CHECK-NEXT:  # %bb.19: # %while.cond.us1412.i
-; CHECK-NEXT:    decb %dl
+; CHECK-NEXT:    decb %cl
 ; CHECK-NEXT:    jne .LBB1_26
 ; CHECK-NEXT:  .LBB1_20: # %if.end41.us1436.i
 ; CHECK-NEXT:  .LBB1_25:
 ; CHECK-NEXT:    movl $1, %ebx
 ; CHECK-NEXT:    jmp .LBB1_26
 ; CHECK-NEXT:  .LBB1_22: # %if.then99.i
 ; CHECK-NEXT:    xorl %ebx, %ebx
 ; CHECK-NEXT:    movl $.str.6, %edi
shared_loop_header:                               ; preds = %dup_early2, %dup_early1
   %dst.0.i = phi i8* [ undef, %inner_loop_body ], [ %dst.0.ph.i, %outer_loop_header ], [ undef, %dead_block ]
   %cmp3.i1172.i = icmp ult i8* null, %call
   br i1 %cmp3.i1172.i, label %wunpsect.exit.thread.loopexit389, label %inner_loop_body
 
 inner_loop_body:                                  ; preds = %shared_loop_header
   %3 = icmp slt i32 undef, 0
   br i1 %3, label %if.end96.i, label %shared_loop_header
 
 dead_block:                                       ; preds = %inner_loop_body
   %cmp75.i = icmp ult i8* %dst.0.i, null
   br label %shared_loop_header
 
 if.end96.i:                                       ; preds = %inner_loop_body
   %cmp97.i = icmp ugt i32 %0, 2
   br i1 %cmp97.i, label %if.then99.i, label %if.end287.i
 
 if.then99.i:                                      ; preds = %if.end96.i