[CodeGen] Late cleanup of redundant address/immediate definitions.

[jonpa]

Summary

On SystemZ, eliminateFrameIndex() may use a scratch register to load part of an offset that is out of range for the using instruction (this is likely also done on other targets - I believe at least AArch64 also does this). Since this is done per instruction (FI operand), several identical addressing "anchor points" may result in the MBB where actually only the first one is needed (provided the GPR is the same). I was expecting this to be cleaned up somehow afterwards, but to my surprise it is not.

I have therefore now made an initial experiment consisting of making PrologEpilogInserter try to clean up redundant definitions after all frame indices have been eliminated, and it would be great to get feedback from other targets to see if this is generally useful. It seems beneficial on SystemZ at least (see below). The patch isn't quite ready yet but I hope it can be the start of a discussion at least. It actually now seems that even though there are "load address" instructions removed, there are in addition many more redundant immediate loads removed (see below). This makes me think that maybe this would better belong in a separate pass run after PEI (or perhaps even later)?

I should also say that I looked into MachineCSE and DeadMachineInstructionElim, but they are at least not readily available: MachineCSE requires SSA, and DeadMachineInstructionElim works bottom-up which does not quite seem to work (the redundant definitions are not "dead").

The patch currently looks for simple "candidate" instructions which may be removed if an identical preceding one exists without any intervening clobbering of the value. The CFG is traversed breadth-first so that live-in definitions can be reused.

Given that the the global analysis adds complexity, I tried also doing this just locally (by passing -pei-localonly). I did find however that the reuse from predecessors is what gives the bulk of improvement, so I would say it is worth that extra effort. On SystemZ/SPEC17:

main <> "local only":
lghi           :               445967               443571    -2396  // Immediate load
lay            :                55090                54783     -307  // Load address
vgbm           :                10974                10848     -126  // Immediate load
...
OPCDIFFS: -2969

main <> patch:
lghi           :               445967               432678   -13289  // Immediate load
lhi            :               219663               216594    -3069  // Immediate load
la             :               533531               531952    -1579  // Load address
lay            :                55090                54774     -316  // Load address
...
OPCDIFFS: -19862

I would say that this is a number of instructions removed (~20k) that makes this look interesting to me... (*) Some examples:

LAYs removed in ./f507.cactuBSSN_r/build/ML_BSSN_Advect.s:

        lay     %r1, 4096(%r15)
        std     %f4, 1024(%r1)
-       lay     %r1, 4096(%r15)
        std     %f1, 1000(%r1)
-       lay     %r1, 4096(%r15)
        std     %f5, 1016(%r1)
-       lay     %r1, 4096(%r15)

LGHI removed in ./f507.cactuBSSN_r/build/RadiationBoundary.s:

        lghi    %r3, 0
        clgijl  %r5, 12, .LBB2_23     // Conditional branch
        risbgn  %r4, %r4, 1, 189, 0
        lghi    %r5, 0
-       lghi    %r3, 0

VGBMs (vector immediate loads) removed in ./f510.parest_r/build/derivative_approximation.s

...
        vst     %v0, 344(%r15), 3
-       vgbm    %v0, 0
        vst     %v0, 360(%r15), 3
-       vgbm    %v0, 0
        vst     %v0, 312(%r15), 3
-       vgbm    %v0, 0
        vst     %v0, 328(%r15), 3
-       vgbm    %v0, 0
        vst     %v0, 280(%r15), 3
-       vgbm    %v0, 0
        vst     %v0, 296(%r15), 3

(*) I also see a few hundred more lmg/br insructions in total, which I am guessing come from some missed CFG optimization, but not sure if this is important. For example:

-       j       .LBB50_33
+       lochie  %r13, 1
+       llgfr   %r2, %r13
+       lmg     %r10, %r15, 240(%r15)
+       br      %r14
 .LBB50_32:
        lghi    %r0, -1
        sllg    %r0, %r0, 0(%r12)
        xihf    %r0, 4294967295
        xilf    %r0, 4294967295
        cg      %r0, 24(%r11)
-.LBB50_33:
-       lhi     %r13, 0
        lochie  %r13, 1
        llgfr   %r2, %r13
        lmg     %r10, %r15, 240(%r15)
        br      %r14

Current status is that benchmarks build with -verify-machineinstrs, but there are many regression tests failing (hopefully due to removed redundant defs :-). I am hoping that people working with those targets can help me understand those...

[jonpa]

If we're not generating enough code for the giant integers, increasing the bitwidth of the giant integers is the first thing to try, I think.

Thanks! I increased it to i8500, and it now passes and seems to have the same block structure.

[arsenm]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

135–136 ↗

(On Diff #425193)

DenseMap?

138–139 ↗

(On Diff #425193)

SmallSetVector?

1492–1495 ↗

(On Diff #425256)

Can't you use one of the existing block visiting iterators?

1514 ↗

(On Diff #425256)

If PEI's loop was reversed and scavenge register backwards was used, we wouldn't need kill flags at all

1629 ↗

(On Diff #425256)

*MI works, don't need dump

1643 ↗

(On Diff #425256)

Ditto

[jonpa]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1514 ↗

(On Diff #425256)

Thanks for review. That is a good point, but I have discovered that not only address anchors from Frame Indices lowering that can be found redundant and cleaned, but also many (more) immediate loads. So changing PEI would not handle all cases, unfortunately.

I have asked before here if this should be part of PEI or become a separate pass, but so far we have not agreed on anything. Given the above, it seems to me now more reasonable to have this as a separate pass run before MachineCopyPropagation. Does this make sense to you?

[arsenm]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1514 ↗

(On Diff #425256)

I think PEI should be split into a number of separate passes as it is. It might make sense to keep it in PEI if you were to do something ahead of or as part of the frame finalization (which might be better than looking for references to getFrameRegister)

[jonpa]

Thank you for review.

I have made some progress but there is yet more to do - any suggestions are welcome.

I think PEI should be split into a number of separate passes as it is. It might make sense to keep it in PEI if you were to do something ahead of or as part of the frame finalization (which might be better than looking for references to getFrameRegister)

I have experimented with putting this into a new pass at different positions in TargetPassConfig::addMachineLateOptimization(). It is possible to maintain NFC by putting this first in that method (which I ended up doing), but I thought it would be interesting to see if it would make any difference putting it after Branchfolder / TailDup. I saw some minor changes only, and these are the "-stats" summarized over SPEC17/SystemZ:

Before Branchfolder <> After Branchfolder:

159682           160143    branch-folder - Number of block tails merged         461  0.3%
561638           560901    branch-folder - Number of branches optimized        -737  0.1%
139108           139050  tailduplication - Number of tail duplicated blocks     -58
107964           108330  tailduplication - Number of tails duplicated           366

After Branchfolder <> Before MachinCopyPropagation (After TailDuplication)

160143           160126    branch-folder - Number of block tails merged         -17
560901           561491    branch-folder - Number of branches optimized         590
139050           138961  tailduplication - Number of tail duplicated blocks     -89
108330           107959  tailduplication - Number of tails duplicated          -371

Before Branchfolder <> Before MachinCopyPropagation (After TailDuplication)

159682           160126    branch-folder - Number of block tails merged         444
561638           561491    branch-folder - Number of branches optimized        -147
139108           138961  tailduplication - Number of tail duplicated blocks    -147
107964           107959  tailduplication - Number of tails duplicated            -5

This doesn't look like making much difference, so I picked the original place for now (with a bias towards more tails dups/less tail merging) .

[jonpa]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1492–1495 ↗

(On Diff #425256)

Using breadth_first() seems to work well (very nearly NFC on SPEC17).

[jonpa]

The handling of the kill flags have been improved, but they are still updated one at a time. Not sure if there is a better way, or if that would be needed.

Simplified the handling of values from predecessors, but however then got into trouble with DenseMap. It seems I must have been updating it somehow in cases while iterating which is not allowed, so I was scared back to std::map for now (this was not due to the operator[], at least).

I experimented with something like a "register substitution", where a previous def was reused even if the current register is a different one, after changing the register in the users. After sorting out all illegal cases this however seemed to give only a very marginal improvement, so it's not included in the patch. Also tried to not merge predecessor entries, but just handle the case of 1 predecessor, but that did have an impact, so I kept it as it was. So I think at the moment at least this looks to be close to a final version, as I have no more ideas to try.

All tests are updated. It would be nice to get review (it's less than 300 lines) and an agreement on on what could be the next step - perhaps at some point enable it for one or more specific targets?

[jonpa]

Also handle cases of invariant loads (GOT, CP) and more constant operand types, such as Global, CPI, and some minor improvements.

Number of removed redundant instructions on SystemZ / SPEC17 is now 21200 (was 19500).

It would be nice to get some numbers for other targets as well!

[nemanjai]

This is fine from a PPC perspective. What would make more of an impact would be the ability to common partial immediate materializations such as happen in prologue/epilogue when a stack frame is very large. Perhaps this can be extended to do that in the future. It would eliminate things like:

lis 11, 16
ori 11, 11, 256 
stxvx 53, 31, 11                        # 16-byte Folded Spill
lis 11, 16
ori 11, 11, 272 
stxvx 54, 31, 11                        # 16-byte Folded Spill
lis 11, 16
ori 11, 11, 288 
stxvx 55, 31, 11                        # 16-byte Folded Spill
lis 11, 16
ori 11, 11, 304 
stxvx 56, 31, 11                        # 16-byte Folded Spill

Of course, that would involve changing the subsequent ori instructions to addi and would likely be very PPC-specific. I just bring it up here in case SystemZ has the same issue.

[jonpa]

In D123394#3499422, @nemanjai wrote:

This is fine from a PPC perspective. What would make more of an impact would be the ability to common partial immediate materializations such as happen in prologue/epilogue when a stack frame is very large. Perhaps this can be extended to do that in the future. It would eliminate things like:

lis 11, 16
ori 11, 11, 256 
stxvx 53, 31, 11                        # 16-byte Folded Spill
lis 11, 16
ori 11, 11, 272 
stxvx 54, 31, 11                        # 16-byte Folded Spill
lis 11, 16
ori 11, 11, 288 
stxvx 55, 31, 11                        # 16-byte Folded Spill
lis 11, 16
ori 11, 11, 304 
stxvx 56, 31, 11                        # 16-byte Folded Spill

Of course, that would involve changing the subsequent ori instructions to addi and would likely be very PPC-specific. I just bring it up here in case SystemZ has the same issue.

That's a good point - my idea so far about that is to try to use a postRA pseudo for e.g. an immediate load requiring two target instructions so that they can be handled just the same way during this pass. Alternatively, one could as you say probably extend this current algorithm to have some kind of "look-ahead" or something so that it could recognize e.g. pairs of instructions as well. That is a bit more complex, though... I will try pseudo on SystemZ and see how it goes with 32-bit immediate loads.

So you did not get any significant results on SPEC with this patch yet?

I checked the compile time and realized that I have to work on that a bit as I see a bit of slowdowns on some files.

[asb]

The RISC-V changes for this look good to me.

[jonpa]

NFC updates.

• Compile time improved: The regression I saw was remedied with a better iteration strategy over the MBBs. I tried to find further improvements, like tracking kill flags in a map instead of doing the search backwards over instructions, but that was slower. I also tried to use DenseMap for the mapping of Register -> MachineInstr*. Even the best init() alloc with a value of 8 was not better than std::map. IndexedMap did not seem like a good choice given that the typical number of mapped registers are not that many. It looks now like the Wall time for this pass on SPEC/SystemZ is on average 0.6%, much like Machine Copy Propagation Pass #2. There are no big compile time blowups that I am aware of.

I can't find much more to improve, at least not at the moment. This pass is "better than nothing" but it is not impossible that there could be more elegant/powerful solutions like emitting frame address anchors in an intelligent way, cleaning up rematerialized immedate-loads somehow, and maybe other things. Personally, I think this looks pretty good to have as long as there is no better way...

However, if we were to actually go ahead and use this, I think there should be some kind of broader agreement on this. There is no strong SystemZ benefit from this - it was just something that seemed missing (looking at those depressing multiple identical load adress instructions). The most benefit is probably on in-order targets. I think there should at least be two or three targets that "vote for" this. It's good to know that the test changes on e.g. RISC-V look correct, and now the next step would be for those targets to evaluate if it worth adding to the pass pipeline.

@nemanjai My previous idea to try post-RA pseudos for immediates on SystemZ is actually not very useful as it is only 64-bit immediates that require two instructions (very rare). But it still looks like this could be worth trying on e.g. PowerPC per your example above. Maybe you could give that a try (like in FrameLowering emitting a pseduo instruction instead of two actual instructions which is expanded in ExpandPostRAPseudos)? If you think that looks good, it would be great to know that...

[jonpa]

ping!

I haven't heard about any other target testing this on benchmarks yet. On SystemZ we see one or two slight improvements on SPEC, but not enough to motivate having this pass in the SystemZ backend. We would however run it if there were other targets as well wanting to do it. So please, take the chance and see if this is anything beneficial for your target!

[bjope]

Hi @jonpa ,

FWIW, I've made a quick test downstream for our OOT target and can see that this triggers occasionally.

Found one benchmark so far that was improved by almost 2%.
Except for that I've only seen some benchmarks where materialization of zero constants are eliminated. And for our VLIW target such loads of zero often only has a cost in code size as we can bundle them with other instructions, so I did not really see any improvements in cycles when disregarding potential I-cache effects.

Anyway, it doesn't seem to be totally useless for us. I'll see if I can run some more tests to later.

[asb]

@jonpa: From your perspective, what's the threshold on whether this should be added to the pass pipeline or not? A pass like this that always results in better code (as opposed to sometimes producing better, sometimes worse code) is attractive even if it triggers rarely. But obviously being too eager to add such code will hit compile time.

[jonpa]

In D123394#3595772, @asb wrote:

@jonpa: From your perspective, what's the threshold on whether this should be added to the pass pipeline or not? A pass like this that always results in better code (as opposed to sometimes producing better, sometimes worse code) is attractive even if it triggers rarely. But obviously being too eager to add such code will hit compile time.

I think it should be enabled on targets where there is some kind of real performance improvement and compile time is not top-priority. If there are only static code-generation improvements that do not really make a difference in execution time, it seems better to leave it out. On SystemZ, there are just one or two slight performance improvements (~1%), enough to want to use it, I think.

Generally, loads of immediates and address anchors are easily done ahead of time on an OOO machine, so just because it is a nice cleanup doesn't mean it would be worth adding. I am hoping that the other targets would care to try it on benchmarks, and if there are then some more interest we could enable it for those targets.

[jonpa]

Ping!

Patch rebased and waiting for evaluation on different targets.

[jonpa]

As I have spent a certain amount of time developing this new target independent pass, I would be very happy to get some feedback.

Short summary of what this patch does:

• Cleaning up of rematerialized immediates: After ISel, an immediate is typically copied to all it's users from the original def, while regalloc then rematerializes it which causes a lot of identical immediate loads which are currently never cleaned away even if they are close in the same MBB.

• Cleaning up of big address offsets forced into registers: SystemZRegisterInfo::eliminateFrameIndex() loads the bigger part of an out of range offset into a register for each FI access, with the idea that these anchors should be reused between accesses. There is however no late "machine cleanup" of identical instructions currently, so there is no reuse at all. This surprised me and was the motivation for this patch, and I think this should be relevant to other targets as well.

It's ~300 LOC, with a small compile time approximately similar to MachineCopyPropagation.

On SystemZ I see ~22k instructions cleaned on SPEC and some slight benchmark improvements which is nice, but perhaps not enough to add a new pass to the backend. I expected this to be done in common code in the first place, and I think this is where it belongs as it is not really target specific.

The question for me now is whether this cleanup should be purposefully omitted (e.g. "doesn't matter enough on OOO CPUs"), or if there is a performance improvement available here. If there is, this patch could be used, or perhaps some other way of achieving this (i.e. earlier in the compilation - any ideas anyone?). If there *isn't*, then that would also be good to know (and perhaps make the comment somewhere in CodeGen)!

I hope you agree that it would be worth at least trying once if this cleanup matters or not. These targets have these number of test files improved: RISCV: 21, AMDGPU: 15, X86: 13, AArch64/ARM/Thumb: 12, PowerPC: 4, Mips: 2, BPF: 1. It would be very nice to learn how much of a cleanup targets achieve in terms of number of instructions eliminated and benchmark improvements. If there is no static improvement on benchmarks (no eliminated instructions) for a target, that would also be interesting to see why this was not even needed.

I am not really strongly arguing for this pass to be added, but I would very much like to come to some consensus on this one way or the other, thanks.

[foad]

llvm/test/CodeGen/AMDGPU/GlobalISel/saddsat.ll

4710 ↗

(On Diff #459019)

The redundant s_mov_b32 instructions in saddsat.ll and ssubsat.ll should be fixed by D133702.

[asb]

My natural bias would be towards accepting a pass that wasn't excessively complicated and demonstrably improved static code quality (especially like this one does, in a way where it's never a regression, even if the benefit may be small or not measurable).

There's a cost in terms of complicating the compiler and adding a new pass, but there's also cost in having generated code that's known to be substandard. Every single time someone invests time in analysing compiler output, it will show up when looking for redundant code and people are likely to waste time figuring out if it's measureable, or perhaps if it's a symptom of some wider codegen problem.

What do others think?

[lattner]

Do you have a sense of what (in practice) is causing the dead operations to get emitted? Can they practically be handled upstream? Such a thing would be a win for all the upstream cost models and would reduce compile time.

I wonder if a version of your pass working as an "assertion" could shake out some obvious missed optimizations or other bugs that cause extraneous instructions to get emitted.

If there are well known things that are impractical to handle (for one reason or another) then something like this would make sense to me.

[foad]

I can understand the temptation to commit this patch. I also agree with Chris that it's useful as a diagnostic tool to find deficiencies that should probably be fixed elsewhere. That's what I did with D133702.

Do you have a sense of what (in practice) is causing the dead operations to get emitted?

Speaking only for AMDGPU, I think there are quite a few different reasons. So far I've seen:

• Missed optimizations in globalisel (D133702).
• PrologEpilogInserter calls TargetRegisterInfo::eliminateFrameIndex which materializes a frame offset into a register, too late for the resulting register-immediate moves to be commoned up.
• RegisterCoalescer(?) not doing a good job with register tuples? The problem is that a 64-bit constant like 0x1 is materialized in a register pair like s[0:1], and then 32-bit 0x0 is materialized in s1 -- but s1 was already 0 because it was the high part of the 64-bit constant.
• Some problems related to CFG structurization: I suspect that the highly-AMDGPU-specific parts of this like the SIOptimizeExecMasking pass are not doing a good enough job of cleaning up redundant code.

[jonpa]

In D123394#3786525, @asb wrote:

There's a cost in terms of complicating the compiler and adding a new pass, but there's also cost in having generated code that's known to be substandard. Every single time someone invests time in analysing compiler output, it will show up when looking for redundant code and people are likely to waste time figuring out if it's measureable, or perhaps if it's a symptom of some wider codegen problem.

Thanks for mentioning this important aspect which I agree very much with, only to fail to put it down into words here.

[jonpa]

In D123394#3790449, @lattner wrote:

Do you have a sense of what (in practice) is causing the dead operations to get emitted? Can they practically be handled upstream? Such a thing would be a win for all the upstream cost models and would reduce compile time.

I wonder if a version of your pass working as an "assertion" could shake out some obvious missed optimizations or other bugs that cause extraneous instructions to get emitted.

If there are well known things that are impractical to handle (for one reason or another) then something like this would make sense to me.

I did another round today of trying to understand the causes here while building SPEC on SystemZ. The patch applied as it is removes ~22k instructions. I could see that rematerialization was the cause in some cases so I rebuilt the comparison without it. In the next comparison I in addition also disabled the register coalescer. These disablings were done both on main and the patch when comparing in each step:

• main <> patch: ~22k instructions less

• main <> patch, while disabling all rematerialization in the backend (returning false in isTriviallyReMaterializable()): ~15k instructions less

• main <> patch, while disabling all rematerialization in the backend, and also disabling register coalescing (-join-intervals=false): ~8k instructions less

The register coalescer seems to play a role here as it can take two unrelated but identical immediate loads and make them define the same virtual register. This causes them to end up in the same physreg, and in some cases then the second one can be removed. This looks kind of similar to the results of rematerialization (having two smaller live-ranges instead of one).

For the remaining 8k instruction it seems that many of these are instructions present after isel already. Prior to isel it is an operand of an instruction, e.g. a call operand, and then (in some contexts) target decides to load it into a register. Then there are also on SystemZ the already mentioned duplicated address offsets loads into registers, emitted post-RA, which is different.

It seems to me that these redundant instructions are the effect of the overall heuristic in the compiler of reducing register pressure at the cost of more immediate loads, in all of the above. It's the way it should be, but it wouldn't hurt to clean things up somewhere in the end as much as possible. I don't think that can be done until the final allocation of registers is done (how would one know if there is a clobbering of the reg in between two identical immediate loads?). That's what it looks like to me on SystemZ, at least.

[jonpa]

rebase (tests)

[RKSimon]

The X86 tests LGTM (including masked_load.ll and oddshuffles.ll)

[jonpa]

PING!

Patch updated after rebase (just two AMDGPU tests).

[jonpa]

Speaking only for AMDGPU, I think there are quite a few different reasons. So far I've seen:
...

• PrologEpilogInserter calls TargetRegisterInfo::eliminateFrameIndex which materializes a frame offset into a register, too late for the resulting register-immediate moves to be commoned up.

...

Yes, this is exactly the same issue as on SystemZ. So far I cannot see any other solution for this - can you?

[jonpa]

PING! Further comments would be very nice.

Another dozen or so tests updated (RISCV, X86) after rebase (improved).

I know there have been at least some interest in this patch, but so far I am still waiting for a consensus from everybody. Am I the only person who would actually like to see this go in?

[RKSimon]

X86 LGTM (and I'd like to see this committed as well!)

llvm/include/llvm/InitializePasses.h

277

This looks like it should be sorted?

llvm/lib/CodeGen/CMakeLists.txt

194

sorting

[jonpa]

llvm/include/llvm/InitializePasses.h

277

yes... Should the pass maybe better be named something with Machine... to make it clear that it is a late CodeGen pass? Maybe MachineImmLoadsCleanupPass or something?

[RKSimon]

llvm/include/llvm/InitializePasses.h

277

SGTM - no strong preference tbh

[jonpa]

Patch updated per review (sorted order in two files), and also a name change into something hopefully better (it seems good to me that it begins with Machine...).

Thanks for taking a look - I guess then we could enable this for at least SystemZ and X86 to begin with if there are no objections. Other targets that seem to benefit like AMDGPU and RISCV could enable it at some later point if they would rather wait for some reason.

[foad]

I'm happy to have this enabled for AMDGPU.

[RKSimon]

+1 to fast track AMDGPU/SystemZ/X86 - I don't how you want to handle helping other targets in the longer term (raise bugs? TODO comments?)

[jonpa]

In D123394#3895939, @RKSimon wrote:

+1 to fast track AMDGPU/SystemZ/X86 - I don't how you want to handle helping other targets in the longer term (raise bugs? TODO comments?)

So far I have only thought that targets should decide for themselves if they want to run it or not. I hope they will give it a try eventually and either enable it or make a comment for the reasons not to. Perhaps a TODO comment next to the disablePass in the target file saying that this needs to be evaluated before enabling would be nice?

[craig.topper]

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

48

Can these be DenseMaps?

53

Every MachineBasicBlock has a number assigned to it. This could possibly be a BitVector using the basic block numbering. See MachineBasicBlock::getNumber() and MachineFunction::getNumBlockIDs

You might also be able to use the numbering instead of a map for MBB2RegDefsMap. But probably depends on how spare that map is.

139

No underscores in variable names. Use VisitedPreds

156

!pred_empty

207

!MBB->pred_empty()?

223

Could this use llvm::all_of with a lambda?

282

Would std::queue be better here?

284

!Worklist.empty()?

[asb]

LGTM to enable this on RISC-V too, thank you!

[craig.topper]

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

197

Use DefedReg.isValid(). Someday we should remove the implicited casts to unsigned integers from the Register class.

[jonpa]

Thanks for review! The pass compilation time percentage has with these improvements decreased on average by 0.1%. I did all the changes except DenseMap (see inline comments below).

Tests updated after rebase:

AMDGPU/si-annotate-cf.ll
RISCV/rvv/fixed-vectors-rint-vp.ll
RISCV/rvv/rint-vp.ll

[jonpa]

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

48

I did make some experiments with DenseMap and other alternatives earlier (see May 16 above) and never got any improvements in compile time by doing so.

In addition, it currently seems unwise to do that since processBlock ("Clear any entries in map that MI clobbers") is looping over the entries while erasing some of them. I could not do this now even after rewriting the loop like:

  // Data structures to map regs to their definitions per MBB.
-  using Reg2DefMap = std::map<Register, MachineInstr*>;
-  using MBB2RegDefsMap = std::map<MachineBasicBlock *, Reg2DefMap>;
+  using Reg2DefMap = DenseMap<Register, MachineInstr*>;
+  using MBB2RegDefsMap = DenseMap<MachineBasicBlock *, Reg2DefMap>;
   MBB2RegDefsMap RegDefs;
 
   // Set of visited MBBs.
@@ -257,11 +258,10 @@ bool MachineLateInstrsCleanup::processBlock(MachineBasicBlock *MBB) {
 
     // Clear any entries in map that MI clobbers.
     for (auto DefI = MBBDefs.begin(); DefI != MBBDefs.end();) {
-      Register Reg = DefI->first;
+      Reg2DefMap::iterator CurrI = DefI++;
+      Register Reg = CurrI->first;
       if (MI->modifiesRegister(Reg, TRI))
-        DefI = MBBDefs.erase(DefI);
-      else
-        ++DefI;
+        MBBDefs.erase(CurrI);
     }

I then got "Assertion `isHandleInSync() && "invalid iterator access!"' failed.", so it seems to me that this isn't supported.

53

Using BitVector for Visited and Visited_preds seems to lower the average compile time (on the files it shows up) by about ~0.04%, so it looks like a very slight but noticeable improvement.

For the MBB2RegDefsMap I saw a similar slight further improvement on average, so with both of these changes an average improvement of 0.1% seems to result. It may be that with this change (MBB2RegDefsMap) some of the slower cases get slower, but the average is improved.

I have only tested this on SystemZ, but I guess a BitMap really should be faster than a map, so it makes sense to use it to me.

223

yeah

282

It's 0.01% faster on average, so why not :-)

[jonpa]

@craig.topper : LGTY?

[danielkiss]

LGTM to enable this on Arm/AArch64.

[jonpa]

Ping!

Now that many targets are willing to try this, it would be nice to have it properly reviewed as well before I commit it.

From what I understand nearly all targets with updated tests except BPF, Mips and XCore have said they would like to have it enabled. Not sure if I should disable those targets...

[RKSimon]

@sdardis Any thoughts on the MIPS changes?

[craig.topper]

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

111

Can we use RPOT(reverse post order traversal)? That will guarantee this property except for loops.

164

I think you can use getMF() instead of getParent()->getParent()

[jonpa]

Updated per review. Using RPOT was a nice improvement - see inline comment. A few more improvements in tests also resulted from this.

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

111

Yes - awesome!

55 lines of iteration logic removed, about the same amount of success (even very slightly better on SPEC), and improved compile time (ave 0.57% -> 0.50%).

[jonpa]

Updated tests were:

llvm/test/CodeGen/AMDGPU/exec-mask-opt-cannot-create-empty-or-backward-segment.ll |  2 --
llvm/test/CodeGen/RISCV/rvv/bitreverse-sdnode.ll                                  |  2 +-
llvm/test/CodeGen/RISCV/rvv/bswap-sdnode.ll                                       |  1 -
llvm/test/CodeGen/RISCV/rvv/bswap-vp.ll                                           |  6 ------
llvm/test/CodeGen/RISCV/rvv/fixed-vectors-bswap-vp.ll                             |  4 ----
llvm/test/CodeGen/SystemZ/frame-28.mir                                            |  5 +++--

[kparzysz]

In Hexagon we try to find a pre-existing register that we can reuse:
https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/Hexagon/HexagonRegisterInfo.cpp#L286-L333

[jonpa]

In D123394#3938078, @kparzysz wrote:

In Hexagon we try to find a pre-existing register that we can reuse:
https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/Hexagon/HexagonRegisterInfo.cpp#L286-L333

That's done after regalloc in eliminateFrameIndex - interesting. My thoughts on this have been that it is more efficient to scan the entire function once rather than for each single frame index elimination. That should be possible if the used register is the same for consecutive FI operands, which I think it should typically be. It would be interesting to see what would happen if you removed that code and instead enabled this pass.

The other thing is that the bulk of the instructions being cleaned up seems to be loads of immediates which result from rematerialization rather than FIs.

[jonpa]

Rebase.

This time the test CodeGen/AMDGPU/flat-scratch.ll failed with the verifier as a kill flag of a super register was not removed. To remedy this, I changed clearKillsForDef() to check for any overlapping reg instead of just the identical one.

Updated tests:

AMDGPU/GlobalISel/call-outgoing-stack-args.ll
AMDGPU/GlobalISel/flat-scratch.ll
AMDGPU/chain-hi-to-lo.ll
AMDGPU/flat-scratch.ll
AMDGPU/si-annotate-cf.ll
AMDGPU/spill-offset-calculation.ll
AMDGPU/spill-scavenge-offset.ll

[foad]

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

183

Could use structured bindings: for (auto [Reg, DefMI] : ...)

186

Could use all_of(drop_begin(MBB->predecessors()), ...)?

194–195

Use << printMBBReference(*MBB) << instead of rolling your own MBB#number syntax, here and elsewhere.

204–205

This looks like for (auto &MI : make_early_inc_range(MBB)))

207

Comment should explain why rather than what.

[jonpa]

Thanks (again) for nice review suggestions. Patch updated (NFC, no new test updates).

[jonpa]

Ping!

Patch rebased, with one test updated: AMDGPU/GlobalISel/call-outgoing-stack-args.ll.

Still waiting for a final approval...

[RKSimon]

LGTM - @foad @craig.topper any additional feedback?

[foad]

In D123394#3957323, @RKSimon wrote:

LGTM - @foad @craig.topper any additional feedback?

Still seems fine to me - but I still think we should also try to clean up some of these redundant instructions earlier, or avoid generating them in the first place, if possible.

[RKSimon]

LGTM - cheers!

[reames]

This looks very interesting. I've been thinking about ways to improve stack addressing sequences on RISCV, and this nicely tackles many of the cases I was just thinking about.

Unfortunately, I think the patch as landed is wrong. See the inline comment on an example where we can't forward due to a clobber which doesn't appear to be respected.

llvm/test/CodeGen/RISCV/rvv/rv64-spill-zvlsseg.ll

44

This line looks to be wrong. a0 is clobbered on line 40 which is after the previous definition being forwarded from line 37.

[jonpa]

Patch reverted because of build failure, relating to capturing structured bindings, which builds fine with gcc but not yet with clang.

I added init captures in processBlock(), which should remedy the build problems and also allow easy removal once it is fully accepted by clang.

for (auto [Reg, DefMI] : RegDefs[FirstPred->getNumber()])
  if (llvm::all_of(
       drop_begin(MBB->predecessors()),
       **[&, &Reg = Reg, &DefMI **= DefMI](const MachineBasicBlock *Pred) {
       ...

@reames: Thanks for catching that problem that was previously missed. I looked into it and found:

$x10 = ADDI $x2, 16
$x11 = PseudoReadVLENB
PseudoVSPILL2_M1 killed renamable $v8_v9, killed $x10, killed $x11 :: (store unknown-size into %stack.0, align 8)
INLINEASM &"" [sideeffect] [attdialect], $0:[clobber] ...
$x10 = ADDI $x2, 16
$x11 = PseudoReadVLENB
renamable $v7_v8 = PseudoVRELOAD2_M1 killed $x10, killed $x11 :: (load unknown-size from %stack.0, align 8)

After Machine Late Instructions Cleanup Pass (machine-latecleanup)

$x10 = ADDI $x2, 16
 $x11 = PseudoReadVLENB
 PseudoVSPILL2_M1 killed renamable $v8_v9, $x10, $x11 :: (store unknown-size into %stack.0, align 8)
 INLINEASM &"" [sideeffect] [attdialect], $0:[clobber], ...
 renamable $v7_v8 = PseudoVRELOAD2_M1 killed $x10, killed $x11 :: (load unknown-size from %stack.0, align 8)

After RISCV pseudo instruction expansion pass (riscv-expand-pseudo)

$x10 = ADDI $x2, 16
$x11 = PseudoReadVLENB
VS1R_V $v8, $x10, implicit $v8_v9 :: (store unknown-size into %stack.0, align 8)
$x10 = ADD $x10, $x11
VS1R_V $v9, $x10, implicit $v8_v9 :: (store unknown-size into %stack.0, align 8)
INLINEASM &"" [sideeffect] [attdialect], $0:[clobber], ...
$v7 = VL1RE8_V $x10 :: (load unknown-size from %stack.0, align 8)
$x10 = ADD $x10, $x11
$v8 = VL1RE8_V $x10 :: (load unknown-size from %stack.0, align 8)

The problem here is that $x10 is only marked as a use operand in the pseudo instruction (PseudoVSPILL2_M1) when the cleanup pass sees it, but it is in fact later getting expanded to a sequence that actually clobbers the register. I think the fix should be to make sure any pseudo instruction is properly modeled to reflect the final expansion, in this case with a def operand of the base register.

Could I disable the pass for now for RISCV and recommit?

[reames]

In D123394#3964717, @jonpa wrote:

@reames: Thanks for catching that problem that was previously missed. I looked into it and found:
...
Could I disable the pass for now for RISCV and recommit?

If you're sure this is a RISCV specific problem, sure. Your description does sound like one to me, so it basically comes down to how sure of your analysis you are.

[craig.topper]

In D123394#3964960, @reames wrote:

In D123394#3964717, @jonpa wrote:

@reames: Thanks for catching that problem that was previously missed. I looked into it and found:
...
Could I disable the pass for now for RISCV and recommit?

If you're sure this is a RISCV specific problem, sure. Your description does sound like one to me, so it basically comes down to how sure of your analysis you are.

Looks like I tried to fix the RISCV issue once. https://reviews.llvm.org/D109405 but it caused other problems and got reverted. Then I guess I lost track of it.

[jonpa]

OK - I will recommit this then as it does seem like a problem in the RISCV backend. I'll wait a day or so to let other targets check that they don't have this potential problem...

[jonpa]

Recommitted as 17db0de, with RISCV disabling it for now.

Posting the RISCV part of the patch here for future use when you want to enable it.

[jonpa]

[jonpa]

Reverted again with 122efef.

[MaskRay]

Hi, it's useful to include Differential Revision: https://reviews.llvm.org/D123394 for all reland commits so that others can easily tell the state by visiting this page.
For relands, please include the original description (don't let readers trace through the original commit to get the description).

[MaskRay]

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

87

This can be moved closer to Changed |= processBlock(MBB);

113

--I https://llvm.org/docs/CodingStandards.html#prefer-preincrement

177

Don't add unneeded blank lines between variable definitions or a variable definition. If the next statement has a comment, it is fine to keep a blank line before the comment.

Applies to elsewhere in the patch.

203

This condition is somewhat strange.

[mgorny]

@MaskRay, I've reverted your M68k test fix since this change was reverted too, to get M68k tests to pass again.

@jonpa, can you please include the change from f55880e830e150d98e5340cdc3c4c41867a5514d in your patch?

[jonpa]

Latest patch as committed (did not update correctly automatically).

[uabelho]

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

107–111

Hi @jonpa

It looks like clearKillsForDef doesn't really handle bundled input. E.g. for our OOT target I've seen cases where "kill" flags were cleared in the BUNDLE instruction, but not in the individual bundled instructions which then lead to verifier complaints.

Do you know if there are other parts of the pass that have problems with bundled input?

[uabelho]

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

115–124

I wonder if this won't abort too early? Or are we really guaranteed to find an implicit kill of the super-reg within this instruction if we find a kill of a sub-reg?

At least for my target I've found cases like

DEF superreg
USE killed subreg1
USE killed subreg2
DEF superreg
USE killed subreg1

and with the current implementation we get

DEF super reg
USE killed subreg1
USE subreg2
USE killed subreg1

and then the verifier complains on the second use of subreg1.

[jonpa]

Hi @uabelho , I'm glad you may find some use for this pass as well :)

It looks like clearKillsForDef doesn't really handle bundled input. E.g. for our OOT target I've seen cases where "kill" flags were cleared in the BUNDLE instruction, but not in the individual bundled instructions which then lead to verifier complaints.

Do you know if there are other parts of the pass that have problems with bundled input?

TBH, it was a while since I thought about BUNDLEs, but IIRC it should be possible to use an instr_iterator with NFC in the *absence* of BUNDLEs, so perhaps that would be a somewhat trivial change..?
Maybe @kparzysz would be interested in this as well, as I believe Hexagon is also emitting BUNDLEs..?

Not quite sure how BUNDLEs need to be handled, for instance when a bundled instruction is deleted, I guess the BUNDLE operands must also be updated.

I wonder if this won't abort too early? Or are we really guaranteed to find an implicit kill of the super-reg within this instruction if we find a kill of a sub-reg?

Are you returning true from enableSubRegLiveness()? I suspect that could cause a difference that I haven't yet run into myself.

As I have mentioned before, I still think the most valuable thing here would be to consider dropping kill flags entirely since this is run so late. I don't think e.g. the buildSchedGraph() uses them. Perhaps they could be regenerated quickly when they are actually useful. The benefits here would be to not have to worry about this tedious updating, which also costs potentially in compile time. Alternatively, it seems that this has to be fixed here for the case of enabled subreg liveness. I have at least so far found it sufficient with the current search for kill flags (without subreg liveness enabled).

[uabelho]

In D123394#3996342, @jonpa wrote:

I wonder if this won't abort too early? Or are we really guaranteed to find an implicit kill of the super-reg within this instruction if we find a kill of a sub-reg?

Are you returning true from enableSubRegLiveness()? I suspect that could cause a difference that I haven't yet run into myself.

As I have mentioned before, I still think the most valuable thing here would be to consider dropping kill flags entirely since this is run so late. I don't think e.g. the buildSchedGraph() uses them. Perhaps they could be regenerated quickly when they are actually useful. The benefits here would be to not have to worry about this tedious updating, which also costs potentially in compile time. Alternatively, it seems that this has to be fixed here for the case of enabled subreg liveness. I have at least so far found it sufficient with the current search for kill flags (without subreg liveness enabled).

We haven't enabled subreg liveness. What we do have though is a downstream hack in VirtRegMap.cpp that throws away implicit uses of superregisters since we haven't understood what they are good for and they just limit scheduling later.
But ok, that explains why those implicit killed uses are missing for us :) I'm still not sure if they are always guaranteed to be there though. Do you know if this is documented somewhere?

Right now, downstream I just made clearKillsForDef continue searhing until it finds a def, and not do the early abort you can do due to the implicit superreg kill. (no idea what compiletime impact this has.)

[jonpa]

We haven't enabled subreg liveness. What we do have though is a downstream hack in VirtRegMap.cpp that throws away implicit uses of superregisters since we haven't understood what they are good for and they just limit scheduling later.

Ah - yeah have some vague notion of that :-)

But ok, that explains why those implicit killed uses are missing for us :) I'm still not sure if they are always guaranteed to be there though. Do you know if this is documented somewhere?

Not really - the idea was basically to stop the search backwards also when a use of the reg is found as there is no point in continuing. I then encountered a problem in a case of the extra super-reg operand, and added a handling for that. I guess I was then still assuming that there can't be an earlier kill of the register. Are you seeing a problem here also without the removal of the superregisters-hack you are using? Something like a def of a superregister, and then a kill of a subreg, in which case the search should continue for the other subreg kill?

Right now, downstream I just made clearKillsForDef continue searhing until it finds a def, and not do the early abort you can do due to the implicit superreg kill. (no idea what compiletime impact this has.)

Makes sense to me - I tried also earlier to store the kills in a map but I found (at least on SystemZ) that this backwards search was actually quicker than that, so I don't think it should be an issue in the normal case...

[vpykhtin]

Makes sense to me - I tried also earlier to store the kills in a map but I found (at least on SystemZ) that this backwards search was actually quicker than that, so I don't think it should be an issue in the normal case...

Hello Jonas, just FYI I got case:

Total Execution Time: 670.4078 seconds (670.7149 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
  289.2998 ( 43.7%)   0.1439 (  1.8%)  289.4437 ( 43.2%)  289.5775 ( 43.2%)  Structurize control flow
  124.9852 ( 18.9%)   0.0000 (  0.0%)  124.9852 ( 18.6%)  125.0289 ( 18.6%)  Machine Late Instructions Cleanup Pass
  45.2408 (  6.8%)   0.0000 (  0.0%)  45.2408 (  6.7%)  45.2740 (  6.8%)  SI Form memory clauses
  38.0358 (  5.7%)   0.0240 (  0.3%)  38.0598 (  5.7%)  38.0767 (  5.7%)  Simple Register Coalescing

Most of the time is spent in clearKillsForDef.

[jonpa]

In D123394#4186531, @vpykhtin wrote:

Makes sense to me - I tried also earlier to store the kills in a map but I found (at least on SystemZ) that this backwards search was actually quicker than that, so I don't think it should be an issue in the normal case...

Hello Jonas, just FYI I got case:

Total Execution Time: 670.4078 seconds (670.7149 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
  289.2998 ( 43.7%)   0.1439 (  1.8%)  289.4437 ( 43.2%)  289.5775 ( 43.2%)  Structurize control flow
  124.9852 ( 18.9%)   0.0000 (  0.0%)  124.9852 ( 18.6%)  125.0289 ( 18.6%)  Machine Late Instructions Cleanup Pass
  45.2408 (  6.8%)   0.0000 (  0.0%)  45.2408 (  6.7%)  45.2740 (  6.8%)  SI Form memory clauses
  38.0358 (  5.7%)   0.0240 (  0.3%)  38.0598 (  5.7%)  38.0767 (  5.7%)  Simple Register Coalescing

Most of the time is spent in clearKillsForDef.

Hi Valery, thanks for the report! It would be nice if you posted your test case on github issues and assigned me (and/or yourself :)... I would think this could be fixed.

[vpykhtin]

Hi Valery, thanks for the report! It would be nice if you posted your test case on github issues and assigned me (and/or yourself :)... I would think this could be fixed.

Sorry for delay, done https://github.com/llvm/llvm-project/issues/61397. Maybe if you have the version with kill flag maps we can check if it works better on the testcase.

[jonpa]

In D123394#4191917, @vpykhtin wrote:

Hi Valery, thanks for the report! It would be nice if you posted your test case on github issues and assigned me (and/or yourself :)... I would think this could be fixed.

Sorry for delay, done https://github.com/llvm/llvm-project/issues/61397. Maybe if you have the version with kill flag maps we can check if it works better on the testcase.

Indeed, the kill flags map does seem to remedy this huge test case. My results and patch posted on github issues.

Anyone interested in compile time with this pass with huge functions, please feel free to contribute your opinion here.

llvm/include/llvm/CodeGen/CodeGenPassBuilder.h

void CodeGenPassBuilder<Derived>::addMachineLateOptimization(
 
   // Tail duplication.
   // Note that duplicating tail just increases code size and degrades
   // performance for targets that require Structured Control Flow.
   // In addition it can also make CFG irreducible. Thus we disable it.
   if (!TM.requiresStructuredCFG())
     addPass(TailDuplicatePass());
 
+  // Cleanup of redundant (identical) address/immediate loads.
+  addPass(MachineLateInstrsCleanupPass());
+
   // Copy propagation.
   addPass(MachineCopyPropagationPass());
 }
 
 /// Add standard basic block placement passes.
 template <typename Derived>
 void CodeGenPassBuilder<Derived>::addBlockPlacement(
     AddMachinePass &addPass) const {

llvm/include/llvm/CodeGen/MachinePassRegistry.def

 DUMMY_MACHINE_FUNCTION_PASS("localstackalloc", LocalStackSlotPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("shrink-wrap", ShrinkWrapPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("prologepilog", PrologEpilogInserterPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("postrapseudos", ExpandPostRAPseudosPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("implicit-null-checks", ImplicitNullChecksPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("postmisched", PostMachineSchedulerPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("machine-scheduler", MachineSchedulerPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("machine-cp", MachineCopyPropagationPass, ())
+DUMMY_MACHINE_FUNCTION_PASS("machine-latecleanup", MachineLateInstrsCleanupPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("post-RA-sched", PostRASchedulerPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("fentry-insert", FEntryInserterPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("xray-instrumentation", XRayInstrumentationPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("patchable-function", PatchableFunctionPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("reg-usage-propagation", RegUsageInfoPropagationPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("reg-usage-collector", RegUsageInfoCollectorPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("funclet-layout", FuncletLayoutPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("stackmap-liveness", StackMapLivenessPass, ())

llvm/include/llvm/CodeGen/Passes.h

namespace llvm {
   extern char &MachineSinkingID;
 
   /// MachineCopyPropagation - This pass performs copy propagation on
   /// machine instructions.
   extern char &MachineCopyPropagationID;
 
   MachineFunctionPass *createMachineCopyPropagationPass(bool UseCopyInstr);
 
+  /// MachineLateInstrsCleanup - This pass removes redundant identical
+  /// instructions after register allocation and rematerialization.
+  extern char &MachineLateInstrsCleanupID;
+
   /// PeepholeOptimizer - This pass performs peephole optimizations -
   /// like extension and comparison eliminations.
   extern char &PeepholeOptimizerID;
 
   /// OptimizePHIs - This pass optimizes machine instruction PHIs
   /// to take advantage of opportunities created during DAG legalization.
   extern char &OptimizePHIsID;
 

llvm/include/llvm/InitializePasses.h

 void initializeMachineBlockFrequencyInfoPass(PassRegistry&);
 void initializeMachineBlockPlacementPass(PassRegistry&);
 void initializeMachineBlockPlacementStatsPass(PassRegistry&);
 void initializeMachineBranchProbabilityInfoPass(PassRegistry&);
 void initializeMachineCFGPrinterPass(PassRegistry &);
 void initializeMachineCSEPass(PassRegistry&);
 void initializeMachineCombinerPass(PassRegistry&);
 void initializeMachineCopyPropagationPass(PassRegistry&);
 void initializeMachineCycleInfoPrinterPassPass(PassRegistry &);
 void initializeMachineCycleInfoWrapperPassPass(PassRegistry &);
 void initializeMachineDominanceFrontierPass(PassRegistry&);
 void initializeMachineDominatorTreePass(PassRegistry&);
 void initializeMachineFunctionPrinterPassPass(PassRegistry&);
 void initializeMachineFunctionSplitterPass(PassRegistry &);
+void initializeMachineLateInstrsCleanupPass(PassRegistry&);
 void initializeMachineLICMPass(PassRegistry&);
 void initializeMachineLoopInfoPass(PassRegistry&);
 void initializeMachineModuleInfoWrapperPassPass(PassRegistry &);
 void initializeMachineOptimizationRemarkEmitterPassPass(PassRegistry&);
 void initializeMachineOutlinerPass(PassRegistry&);
 void initializeMachinePipelinerPass(PassRegistry&);
 void initializeMachinePostDominatorTreePass(PassRegistry&);
 void initializeMachineRegionInfoPassPass(PassRegistry&);

llvm/lib/CodeGen/CMakeLists.txt

add_llvm_component_library(LLVMCodeGen
   MachineDominators.cpp
   MachineFrameInfo.cpp
   MachineFunction.cpp
   MachineFunctionPass.cpp
   MachineFunctionPrinterPass.cpp
   MachineFunctionSplitter.cpp
   MachineInstrBundle.cpp
   MachineInstr.cpp
+  MachineLateInstrsCleanup.cpp
   MachineLICM.cpp
   MachineLoopInfo.cpp
   MachineLoopUtils.cpp
   MachineModuleInfo.cpp
   MachineModuleInfoImpls.cpp
   MachineModuleSlotTracker.cpp
   MachineOperand.cpp
   MachineOptimizationRemarkEmitter.cpp
add_llvm_component_library(LLVMCodeGen
   MachineStableHash.cpp
   MIRVRegNamerUtils.cpp
   MIRNamerPass.cpp
   MIRCanonicalizerPass.cpp
   RegisterUsageInfo.cpp
   RegUsageInfoCollector.cpp
   RegUsageInfoPropagate.cpp
   ReplaceWithVeclib.cpp
   ResetMachineFunctionPass.cpp
   RegisterBank.cpp
   RegisterBankInfo.cpp
   SafeStack.cpp
   SafeStackLayout.cpp
   ScheduleDAG.cpp
   ScheduleDAGInstrs.cpp
   ScheduleDAGPrinter.cpp
   ScoreboardHazardRecognizer.cpp

llvm/lib/CodeGen/CodeGen.cpp

void llvm::initializeCodeGen(PassRegistry &Registry) {
   initializeMachineCFGPrinterPass(Registry);
   initializeMachineCSEPass(Registry);
   initializeMachineCombinerPass(Registry);
   initializeMachineCopyPropagationPass(Registry);
   initializeMachineCycleInfoPrinterPassPass(Registry);
   initializeMachineCycleInfoWrapperPassPass(Registry);
   initializeMachineDominatorTreePass(Registry);
   initializeMachineFunctionPrinterPassPass(Registry);
+  initializeMachineLateInstrsCleanupPass(Registry);
   initializeMachineLICMPass(Registry);
   initializeMachineLoopInfoPass(Registry);
   initializeMachineModuleInfoWrapperPassPass(Registry);
   initializeMachineOptimizationRemarkEmitterPassPass(Registry);
   initializeMachineOutlinerPass(Registry);
   initializeMachinePipelinerPass(Registry);
   initializeModuloScheduleTestPass(Registry);
   initializeMachinePostDominatorTreePass(Registry);

llvm/lib/CodeGen/MachineLateInstrsCleanup.cpp

+//==--- MachineLateInstrsCleanup.cpp - Late Instructions Cleanup Pass -----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This simple pass removes any identical and redundant immediate or address
+// loads to the same register. The immediate loads removed can originally be
+// the result of rematerialization, while the addresses are redundant frame
+// addressing anchor points created during Frame Indices elimination.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/BitVector.h"
+#include "llvm/ADT/BreadthFirstIterator.h"
+#include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/CodeGen/MachineFunction.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstr.h"
+#include "llvm/CodeGen/MachineOperand.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/TargetInstrInfo.h"
+#include "llvm/CodeGen/TargetRegisterInfo.h"
+#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include "llvm/Support/Debug.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "machine-latecleanup"
+
+STATISTIC(NumRemoved, "Number of redundant instructions removed.");
+
+namespace {
+
+class MachineLateInstrsCleanup : public MachineFunctionPass {
+  const TargetRegisterInfo *TRI;
+  const TargetInstrInfo *TII;
+
+  // Data structures to map regs to their definitions per MBB.
+  using Reg2DefMap = std::map<Register, MachineInstr*>;
+  std::vector<Reg2DefMap> RegDefs;
+
+  // Set of visited MBBs.
+  BitVector Visited;
+
+  // Return true if all predecessors of MBB have been visited.
+  bool allPredsVisited(MachineBasicBlock *MBB);
+
+  // Walk through the instructions in MBB and remove any redundant
+  // instructions.
+  bool processBlock(MachineBasicBlock *MBB);
+
+  // Visit MBB and then any of its successors that become ready.
+  bool visitBlock(MachineBasicBlock *MBB);
+
+public:
+  static char ID; // Pass identification, replacement for typeid
+
+  MachineLateInstrsCleanup() : MachineFunctionPass(ID) {
+    initializeMachineLateInstrsCleanupPass(*PassRegistry::getPassRegistry());
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesCFG();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+  MachineFunctionProperties getRequiredProperties() const override {
+    return MachineFunctionProperties().set(
+        MachineFunctionProperties::Property::NoVRegs);
+  }
+};
+
+} // end anonymous namespace
+
+char MachineLateInstrsCleanup::ID = 0;
+
+char &llvm::MachineLateInstrsCleanupID = MachineLateInstrsCleanup::ID;
+
+INITIALIZE_PASS(MachineLateInstrsCleanup, DEBUG_TYPE,
+                "Machine Late Instructions Cleanup Pass", false, false)
+
+bool MachineLateInstrsCleanup::allPredsVisited(MachineBasicBlock *MBB) {
+  return llvm::all_of(MBB->predecessors(), [&](const MachineBasicBlock *Pred) {
+      return Visited.test(Pred->getNumber());
+    });
+}
+
+bool MachineLateInstrsCleanup::runOnMachineFunction(MachineFunction &MF) {
+  if (skipFunction(MF.getFunction()))
+    return false;
+
+  bool Changed = false;
+
+  TRI = MF.getSubtarget().getRegisterInfo();
+  TII = MF.getSubtarget().getInstrInfo();
+
+  RegDefs.clear();
+  RegDefs.resize(MF.getNumBlockIDs());
+  Visited.clear();
+  Visited.resize(MF.getNumBlockIDs());
+
+  // Visit all MBBs in an order that maximises the reuse from predecessors.
+  // Put the ones that are not ready in Pending and handle them in a second
+  // round.
+  std::vector<MachineBasicBlock *> Pending;
+  for (auto *MBB : breadth_first(&MF))
+    if (!Visited.test(MBB->getNumber())) {
+      if (allPredsVisited(MBB))
+        Changed |= visitBlock(MBB);
+      else
+        Pending.push_back(MBB);
+    }
+
+  for (auto *MBB : Pending)
+    if (!Visited.test(MBB->getNumber()))
+      Changed |= visitBlock(MBB);
+
+  return Changed;
+}
+
+// Clear any previous kill flag on Reg found before I in MBB. Walk backwards
+// in MBB and if needed continue in predecessors until a use/def of Reg is
+// encountered. This seems to be faster than in practice than tracking kill
+// flags in a map.
+static void clearKillsForDef(Register Reg, MachineBasicBlock *MBB,
+                             MachineBasicBlock::iterator I,
+                             BitVector &VisitedPreds,
+                             const TargetRegisterInfo *TRI) {
+  VisitedPreds.set(MBB->getNumber());
+  while (I != MBB->begin()) {
+    I--;
+    for (auto &MO : I->operands())
+      if (MO.isReg()) {
+        if (MO.isDef() && TRI->regsOverlap(MO.getReg(), Reg))
+          return;
+        if (MO.readsReg() && MO.getReg() == Reg) {
+          MO.setIsKill(false);
+          return;
+        }
+      }
+  }
+
+  // If earlier def is not in MBB, continue in predecessors.
+  if (!MBB->isLiveIn(Reg))
+    MBB->addLiveIn(Reg);
+  assert(!MBB->pred_empty() && "Predecessor def not found!");
+  for (MachineBasicBlock *Pred : MBB->predecessors())
+    if (!VisitedPreds.test(Pred->getNumber()))
+      clearKillsForDef(Reg, Pred, Pred->end(), VisitedPreds, TRI);
+}
+
+static void removeRedundantDef(MachineInstr *MI,
+                               const TargetRegisterInfo *TRI) {
+  Register Reg = MI->getOperand(0).getReg();
+  BitVector VisitedPreds(MI->getParent()->getParent()->getNumBlockIDs());
+  clearKillsForDef(Reg, MI->getParent(), MI->getIterator(), VisitedPreds, TRI);
+  MI->eraseFromParent();
+  ++NumRemoved;
+}
+
+// Return true if MI is a potential candidate for reuse/removal and if so
+// also the register it defines in DefedReg.  A candidate is a simple
+// instruction that does not touch memory, has only one register definition
+// and the only reg it may use is FrameReg. Typically this is an immediate
+// load or a load-address instruction.
+static bool isCandidate(const MachineInstr *MI, Register &DefedReg,
+                        Register FrameReg) {
+  DefedReg = MCRegister::NoRegister;
+  bool SawStore = true;
+  if (!MI->isSafeToMove(nullptr, SawStore) || MI->isImplicitDef() ||
+      MI->isInlineAsm())
+    return false;
+  for (unsigned i = 0, e = MI->getNumOperands(); i != e; ++i) {
+    const MachineOperand &MO = MI->getOperand(i);
+    if (MO.isReg()) {
+      if (MO.isDef()) {
+        if (i == 0 && !MO.isImplicit() && !MO.isDead())
+          DefedReg = MO.getReg();
+        else
+          return false;
+      } else if (MO.getReg() && MO.getReg() != FrameReg)
+        return false;
+    } else if (!(MO.isImm() || MO.isCImm() || MO.isFPImm() || MO.isCPI() ||
+                 MO.isGlobal() || MO.isSymbol()))
+      return false;
+  }
+  return DefedReg.isValid();
+}
+
+bool MachineLateInstrsCleanup::processBlock(MachineBasicBlock *MBB) {
+  bool Changed = false;
+  Visited.set(MBB->getNumber());
+
+  Reg2DefMap &MBBDefs = RegDefs[MBB->getNumber()];
+
+  // Find reusable definitions in the predecessor(s).
+  if (!MBB->pred_empty()) {
+    MachineBasicBlock *FirstPred = *MBB->pred_begin();
+    for (auto I : RegDefs[FirstPred->getNumber()]) {
+      Register Reg = I.first;
+      MachineInstr *DefMI = I.second;
+      if (llvm::all_of(MBB->predecessors(), [&](const MachineBasicBlock *Pred) {
+            if (Pred == FirstPred)
+              return true;
+            auto PredDefI = RegDefs[Pred->getNumber()].find(Reg);
+            return PredDefI != RegDefs[Pred->getNumber()].end() &&
+                   DefMI->isIdenticalTo(*PredDefI->second);
+          })) {
+        MBBDefs[Reg] = DefMI;
+        LLVM_DEBUG(dbgs() << "Reusable instruction from pred(s): in MBB#"
+                          << MBB->getNumber() << ":  " << *DefMI;);
+      }
+    }
+  }
+
+  // Process MBB.
+  MachineFunction *MF = MBB->getParent();
+  const TargetRegisterInfo *TRI = MF->getSubtarget().getRegisterInfo();
+  Register FrameReg = TRI->getFrameRegister(*MF);
+  for (MachineBasicBlock::iterator I = MBB->begin(); I != MBB->end();) {
+    MachineInstr *MI = &*(I++);
+
+    // Clear map if the FrameReg is modified.
+    if (MI->modifiesRegister(FrameReg, TRI)) {
+      MBBDefs.clear();
+      continue;
+    }
+
+    Register DefedReg;
+    bool IsCandidate = isCandidate(MI, DefedReg, FrameReg);
+
+    // Check for an earlier identical and reusable instruction.
+    if (IsCandidate) {
+      auto DefI = MBBDefs.find(DefedReg);
+      if (DefI != MBBDefs.end() && MI->isIdenticalTo(*DefI->second)) {
+        LLVM_DEBUG(dbgs() << "Removing redundant instruction in MBB#"
+                          << MBB->getNumber() << ":  " << *MI;);
+        removeRedundantDef(MI, TRI);
+        Changed = true;
+        continue;
+      }
+    }
+
+    // Clear any entries in map that MI clobbers.
+    for (auto DefI = MBBDefs.begin(); DefI != MBBDefs.end();) {
+      Register Reg = DefI->first;
+      if (MI->modifiesRegister(Reg, TRI))
+        DefI = MBBDefs.erase(DefI);
+      else
+        ++DefI;
+    }
+
+    // Record this MI for potential later reuse.
+    if (IsCandidate) {
+      LLVM_DEBUG(dbgs() << "Found interesting instruction in MBB#"
+                        << MBB->getNumber() << ":  " << *MI;);
+      MBBDefs[DefedReg] = MI;
+    }
+  }
+
+  return Changed;
+}
+
+bool MachineLateInstrsCleanup::visitBlock(MachineBasicBlock *PendingMBB) {
+  bool Changed = false;
+
+  std::queue<MachineBasicBlock *> Worklist;
+  Worklist.push(PendingMBB);
+  while (!Worklist.empty()) {
+    MachineBasicBlock *CurrMBB = Worklist.front();
+    Worklist.pop();
+    Changed |= processBlock(CurrMBB);
+    for (MachineBasicBlock *SuccMBB : CurrMBB->successors())
+      if (!Visited.test(SuccMBB->getNumber()) && allPredsVisited(SuccMBB))
+        Worklist.push(SuccMBB);
+  }
+
+  return Changed;
+}

llvm/lib/CodeGen/TargetPassConfig.cpp

 }
 
 //===---------------------------------------------------------------------===//
 /// Post RegAlloc Pass Configuration
 //===---------------------------------------------------------------------===//
 
 /// Add passes that optimize machine instructions after register allocation.
 void TargetPassConfig::addMachineLateOptimization() {
+  // Cleanup of redundant immediate/address loads.
+  addPass(&MachineLateInstrsCleanupID);
+
   // Branch folding must be run after regalloc and prolog/epilog insertion.
   addPass(&BranchFolderPassID);
 
   // Tail duplication.
   // Note that duplicating tail just increases code size and degrades
   // performance for targets that require Structured Control Flow.
   // In addition it can also make CFG irreducible. Thus we disable it.
   if (!TM->requiresStructuredCFG())

llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp

 
 void NVPTXPassConfig::addIRPasses() {
   // The following passes are known to not play well with virtual regs hanging
   // around after register allocation (which in our case, is *all* registers).
   // We explicitly disable them here.  We do, however, need some functionality
   // of the PrologEpilogCodeInserter pass, so we emulate that behavior in the
   // NVPTXPrologEpilog pass (see NVPTXPrologEpilogPass.cpp).
   disablePass(&PrologEpilogCodeInserterID);
+  disablePass(&MachineLateInstrsCleanupID);
   disablePass(&MachineCopyPropagationID);
   disablePass(&TailDuplicateID);
   disablePass(&StackMapLivenessID);
   disablePass(&LiveDebugValuesID);
   disablePass(&PostRAMachineSinkingID);
   disablePass(&PostRASchedulerID);
   disablePass(&FuncletLayoutID);
   disablePass(&PatchableFunctionID);

llvm/lib/Target/WebAssembly/WebAssemblyTargetMachine.cpp

 }
 
 void WebAssemblyPassConfig::addPostRegAlloc() {
   // TODO: The following CodeGen passes don't currently support code containing
   // virtual registers. Consider removing their restrictions and re-enabling
   // them.
 
   // These functions all require the NoVRegs property.
+  disablePass(&MachineLateInstrsCleanupID);
   disablePass(&MachineCopyPropagationID);
   disablePass(&PostRAMachineSinkingID);
   disablePass(&PostRASchedulerID);
   disablePass(&FuncletLayoutID);
   disablePass(&StackMapLivenessID);
   disablePass(&LiveDebugValuesID);
   disablePass(&PatchableFunctionID);
   disablePass(&ShrinkWrapID);

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       Machine Natural Loop Construction
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       MachinePostDominator Tree Construction
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Shrink Wrapping analysis
 ; CHECK-NEXT:       Prologue/Epilogue Insertion & Frame Finalization
+; CHECK-NEXT:       Machine Late Instructions Cleanup Pass
 ; CHECK-NEXT:       Control Flow Optimizer
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Tail Duplication
 ; CHECK-NEXT:       Machine Copy Propagation Pass
 ; CHECK-NEXT:       Post-RA pseudo instruction expansion pass
 ; CHECK-NEXT:       AArch64 pseudo instruction expansion pass
 ; CHECK-NEXT:       AArch64 load / store optimization pass
 ; CHECK-NEXT:       Insert KCFI indirect call checks

llvm/test/CodeGen/AArch64/stack-guard-remat-bitcast.ll

 ; CHECK-NEXT:    ldr x8, [x8, ___stack_chk_guard@GOTPAGEOFF]
 ; CHECK-NEXT:  Lloh3:
 ; CHECK-NEXT:    ldr x9, [x9, ___stack_chk_guard@GOTPAGEOFF]
 ; CHECK-NEXT:  Lloh4:
 ; CHECK-NEXT:    ldr x8, [x8]
 ; CHECK-NEXT:  Lloh5:
 ; CHECK-NEXT:    ldr x9, [x9]
 ; CHECK-NEXT:    str x8, [sp]
-; CHECK-NEXT:  Lloh6:
-; CHECK-NEXT:    adrp x8, ___stack_chk_guard@GOTPAGE
 ; CHECK-NEXT:    stur x9, [x29, #-8]
-; CHECK-NEXT:  Lloh7:
-; CHECK-NEXT:    ldr x8, [x8, ___stack_chk_guard@GOTPAGEOFF]
 ; CHECK-NEXT:    ldur x9, [x29, #-8]
-; CHECK-NEXT:  Lloh8:
-; CHECK-NEXT:    ldr x8, [x8]
 ; CHECK-NEXT:    cmp x8, x9
 ; CHECK-NEXT:    b.ne LBB0_2
 ; CHECK-NEXT:  ; %bb.1: ; %entry
 ; CHECK-NEXT:    ldp x29, x30, [sp, #48] ; 16-byte Folded Reload
 ; CHECK-NEXT:    mov w0, #-1
 ; CHECK-NEXT:    add sp, sp, #64
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  LBB0_2: ; %entry
 ; CHECK-NEXT:    bl ___stack_chk_fail
-; CHECK-NEXT:    .loh AdrpLdrGotLdr Lloh6, Lloh7, Lloh8
 ; CHECK-NEXT:    .loh AdrpLdrGotLdr Lloh1, Lloh3, Lloh5
 ; CHECK-NEXT:    .loh AdrpLdrGotLdr Lloh0, Lloh2, Lloh4
 entry:
   %StackGuardSlot = alloca i8*
   %StackGuard = load i8*, i8** bitcast (i64** @__stack_chk_guard to i8**)
   call void @llvm.stackprotector(i8* %StackGuard, i8** %StackGuardSlot)
   %container = alloca [32 x i8], align 1
   call void @llvm.stackprotectorcheck(i8** bitcast (i64** @__stack_chk_guard to i8**))
   ret i32 -1
 }
 
 declare void @llvm.stackprotector(i8*, i8**) ssp
 declare void @llvm.stackprotectorcheck(i8**) ssp

llvm/test/CodeGen/AArch64/sve-calling-convention-mixed.ll

 
 define float @foo2(double* %x0, double* %x1) nounwind {
 ; CHECK-LABEL: foo2:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    addvl sp, sp, #-4
 ; CHECK-NEXT:    sub sp, sp, #16
 ; CHECK-NEXT:    ptrue p0.b
-; CHECK-NEXT:    add x9, sp, #16
+; CHECK-NEXT:    add x8, sp, #16
 ; CHECK-NEXT:    ld4d { z1.d - z4.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld4d { z16.d - z19.d }, p0/z, [x1]
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    add x8, sp, #16
 ; CHECK-NEXT:    fmov s0, #1.00000000
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    mov w1, #1
 ; CHECK-NEXT:    mov w2, #2
-; CHECK-NEXT:    st1d { z16.d }, p0, [x9]
-; CHECK-NEXT:    add x9, sp, #16
 ; CHECK-NEXT:    mov w3, #3
 ; CHECK-NEXT:    mov w4, #4
 ; CHECK-NEXT:    mov w5, #5
 ; CHECK-NEXT:    mov w6, #6
-; CHECK-NEXT:    st1d { z17.d }, p0, [x9, #1, mul vl]
-; CHECK-NEXT:    add x9, sp, #16
 ; CHECK-NEXT:    mov w7, #7
-; CHECK-NEXT:    st1d { z18.d }, p0, [x9, #2, mul vl]
 ; CHECK-NEXT:    add x9, sp, #16
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    st1d { z16.d }, p0, [x9]
+; CHECK-NEXT:    st1d { z17.d }, p0, [x9, #1, mul vl]
+; CHECK-NEXT:    st1d { z18.d }, p0, [x9, #2, mul vl]
 ; CHECK-NEXT:    st1d { z19.d }, p0, [x9, #3, mul vl]
 ; CHECK-NEXT:    str x8, [sp]
 ; CHECK-NEXT:    bl callee2
 ; CHECK-NEXT:    addvl sp, sp, #4
 ; CHECK-NEXT:    add sp, sp, #16
 ; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
 entry:

llvm/test/CodeGen/AMDGPU/GlobalISel/call-outgoing-stack-args.ll

 ; FLATSCR:       ; %bb.0:
 ; FLATSCR-NEXT:    s_add_u32 flat_scratch_lo, s0, s3
 ; FLATSCR-NEXT:    v_mov_b32_e32 v0, 0
 ; FLATSCR-NEXT:    s_addc_u32 flat_scratch_hi, s1, 0
 ; FLATSCR-NEXT:    v_mov_b32_e32 v1, 0
 ; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
 ; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:8
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:72
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:16
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:80
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:24
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:88
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:32
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:96
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:40
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:104
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:48
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:112
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:56
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
-; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:120
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], vcc_hi offset:64
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:72
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:80
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:88
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:96
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:104
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:112
+; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:120
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s33 offset:128
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[0:1], off, s33 offset:8
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
+; FLATSCR-NEXT:    s_nop 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[2:3], off, s33 offset:16
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[4:5], off, s33 offset:24
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[6:7], off, s33 offset:32
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[8:9], off, s33 offset:40
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[10:11], off, s33 offset:48
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[12:13], off, s33 offset:56
-; FLATSCR-NEXT:    s_mov_b32 s33, 0
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[14:15], off, s33 offset:64
 ; FLATSCR-NEXT:    s_movk_i32 s32, 0x50
 ; FLATSCR-NEXT:    s_getpc_b64 s[0:1]
 ; FLATSCR-NEXT:    s_add_u32 s0, s0, external_void_func_byval@rel32@lo+4
 ; FLATSCR-NEXT:    s_addc_u32 s1, s1, external_void_func_byval@rel32@hi+12
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)
 ; FLATSCR-NEXT:    scratch_store_dwordx2 off, v[0:1], s32
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(7)

llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch.ll

 ; GFX940-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX940-NEXT:    s_lshl_b32 s1, s0, 2
 ; GFX940-NEXT:    s_and_b32 s0, s0, 15
 ; GFX940-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX940-NEXT:    s_lshl_b32 s0, s0, 2
 ; GFX940-NEXT:    scratch_store_dword v1, v0, vcc_hi sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_mov_b32_e32 v0, s0
-; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    scratch_load_dword v0, v0, vcc_hi sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_sindex_large_offset_kernel:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    s_load_b32 s0, s[0:1], 0x24
 ; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    v_mov_b32_e32 v1, 15
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX11-NEXT:    s_lshl_b32 s1, s0, 2
 ; GFX11-NEXT:    s_and_b32 s0, s0, 15
 ; GFX11-NEXT:    v_mov_b32_e32 v0, s1
 ; GFX11-NEXT:    s_lshl_b32 s0, s0, 2
 ; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
 ; GFX11-NEXT:    v_mov_b32_e32 v2, s0
 ; GFX11-NEXT:    scratch_store_b32 v0, v1, vcc_lo dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    scratch_load_b32 v0, v2, vcc_lo glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
   %padding = alloca [4096 x i32], align 4, addrspace(5)
   %i = alloca [32 x float], align 4, addrspace(5)
   %pad_gep = getelementptr inbounds [4096 x i32], [4096 x i32] addrspace(5)* %padding, i32 0, i32 undef
   %pad_load = load volatile i32, i32 addrspace(5)* %pad_gep, align 4

 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
 ; GFX940-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    v_sub_u32_e32 v0, 0, v0
 ; GFX940-NEXT:    scratch_store_dword v1, v2, vcc_hi sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    scratch_load_dword v0, v0, vcc_hi offset:124 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: store_load_vindex_large_offset_kernel:
 ; GFX11:       ; %bb.0: ; %bb
 ; GFX11-NEXT:    v_sub_nc_u32_e32 v1, 0, v0
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX11-NEXT:    v_mov_b32_e32 v2, 15
 ; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    scratch_load_b32 v3, off, off offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
 ; GFX11-NEXT:    scratch_store_b32 v0, v2, vcc_lo dlc
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    scratch_load_b32 v0, v1, vcc_lo offset:124 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_endpgm
 bb:
   %padding = alloca [4096 x i32], align 4, addrspace(5)
   %i = alloca [32 x float], align 4, addrspace(5)
   %pad_gep = getelementptr inbounds [4096 x i32], [4096 x i32] addrspace(5)* %padding, i32 0, i32 undef
   %pad_load = load volatile i32, i32 addrspace(5)* %pad_gep, align 4

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.div.fmas.ll

 ; GFX7:       ; %bb.0: ; %entry
 ; GFX7-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0xa
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX7-NEXT:    s_mov_b32 s6, 0
 ; GFX7-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    buffer_load_dwordx3 v[1:3], v[1:2], s[4:7], 0 addr64
-; GFX7-NEXT:    s_mov_b32 s6, 0
 ; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
 ; GFX7-NEXT:    s_and_saveexec_b64 s[2:3], vcc
 ; GFX7-NEXT:    s_cbranch_execz .LBB13_2
 ; GFX7-NEXT:  ; %bb.1: ; %bb
 ; GFX7-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x14
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_load_dword s4, s[4:5], 0x0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)

llvm/test/CodeGen/AMDGPU/cc-update.ll

 ; GFX803-NEXT:    s_add_u32 s0, s0, s7
 ; GFX803-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX803-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:8 glc
 ; GFX803-NEXT:    s_waitcnt vmcnt(0)
 ; GFX803-NEXT:    s_mov_b32 s4, 0x40000
 ; GFX803-NEXT:    buffer_store_dword v0, off, s[0:3], s4 ; 4-byte Folded Spill
 ; GFX803-NEXT:    ;;#ASMSTART
 ; GFX803-NEXT:    ;;#ASMEND
-; GFX803-NEXT:    s_mov_b32 s4, 0x40000
 ; GFX803-NEXT:    buffer_load_dword v0, off, s[0:3], s4 ; 4-byte Folded Reload
 ; GFX803-NEXT:    s_waitcnt vmcnt(0)
 ; GFX803-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:8
 ; GFX803-NEXT:    s_waitcnt vmcnt(0)
 ; GFX803-NEXT:    s_endpgm
 ;
 ; GFX900-LABEL: test_sgpr_offset_kernel:
 ; GFX900:       ; %bb.0: ; %entry
 ; GFX900-NEXT:    s_add_u32 s0, s0, s7
 ; GFX900-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX900-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:8 glc
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
 ; GFX900-NEXT:    s_mov_b32 s4, 0x40000
 ; GFX900-NEXT:    buffer_store_dword v0, off, s[0:3], s4 ; 4-byte Folded Spill
 ; GFX900-NEXT:    ;;#ASMSTART
 ; GFX900-NEXT:    ;;#ASMEND
-; GFX900-NEXT:    s_mov_b32 s4, 0x40000
 ; GFX900-NEXT:    buffer_load_dword v0, off, s[0:3], s4 ; 4-byte Folded Reload
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
 ; GFX900-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:8
 ; GFX900-NEXT:    s_waitcnt vmcnt(0)
 ; GFX900-NEXT:    s_endpgm
 ;
 ; GFX1010-LABEL: test_sgpr_offset_kernel:
 ; GFX1010:       ; %bb.0: ; %entry
 ; GFX1010-NEXT:    s_add_u32 s0, s0, s7
 ; GFX1010-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX1010-NEXT:    s_mov_b32 s4, 0x20000
 ; GFX1010-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:8 glc dlc
 ; GFX1010-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1010-NEXT:    buffer_store_dword v0, off, s[0:3], s4 ; 4-byte Folded Spill
-; GFX1010-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-NEXT:    s_mov_b32 s4, 0x20000
 ; GFX1010-NEXT:    ;;#ASMSTART
 ; GFX1010-NEXT:    ;;#ASMEND
 ; GFX1010-NEXT:    buffer_load_dword v0, off, s[0:3], s4 ; 4-byte Folded Reload
 ; GFX1010-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1010-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:8
 ; GFX1010-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX1010-NEXT:    s_endpgm
 ;
 ; GFX1100-LABEL: test_sgpr_offset_kernel:
 ; GFX1100:       ; %bb.0: ; %entry
 ; GFX1100-NEXT:    scratch_load_b32 v0, off, off offset:8 glc dlc
 ; GFX1100-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1100-NEXT:    s_movk_i32 s0, 0x1000
 ; GFX1100-NEXT:    scratch_store_b32 off, v0, s0 ; 4-byte Folded Spill
-; GFX1100-NEXT:    s_movk_i32 s0, 0x1000
 ; GFX1100-NEXT:    ;;#ASMSTART
 ; GFX1100-NEXT:    ;;#ASMEND
 ; GFX1100-NEXT:    scratch_load_b32 v0, off, s0 ; 4-byte Folded Reload
 ; GFX1100-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1100-NEXT:    scratch_store_b32 off, v0, off offset:8 dlc
 ; GFX1100-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX1100-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1100-NEXT:    s_endpgm

llvm/test/CodeGen/AMDGPU/chain-hi-to-lo.ll

 ; FLATSCR-NEXT:    v_mov_b32_e32 v2, 0
 ; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
 ; FLATSCR-NEXT:    s_waitcnt lgkmcnt(0)
 ; FLATSCR-NEXT:    global_load_ushort v0, v2, s[0:1]
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    scratch_store_short off, v0, vcc_hi offset:4
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    global_load_ushort v0, v2, s[0:1] offset:2
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    scratch_store_short off, v0, vcc_hi offset:6
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    global_load_ushort v0, v2, s[0:1] offset:4
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    scratch_store_short off, v0, vcc_hi offset:8
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
 ; FLATSCR-NEXT:    scratch_load_dword v0, off, vcc_hi offset:4
-; FLATSCR-NEXT:    s_mov_b32 vcc_hi, 0
 ; FLATSCR-NEXT:    scratch_load_dword v1, off, vcc_hi offset:6
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
 ; FLATSCR-NEXT:    s_endpgm
 ;
 ; GFX10_DEFAULT-LABEL: vload2_private:
 ; GFX10_DEFAULT:       ; %bb.0: ; %entry
 ; GFX10_DEFAULT-NEXT:    s_load_dwordx4 s[4:7], s[4:5], 0x0

 ; FLATSCR_GFX10-NEXT:    v_mov_b32_e32 v2, 0
 ; FLATSCR_GFX10-NEXT:    s_mov_b32 vcc_lo, 0
 ; FLATSCR_GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; FLATSCR_GFX10-NEXT:    global_load_ushort v0, v2, s[0:1]
 ; FLATSCR_GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR_GFX10-NEXT:    scratch_store_short off, v0, vcc_lo offset:4
 ; FLATSCR_GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; FLATSCR_GFX10-NEXT:    global_load_ushort v0, v2, s[0:1] offset:2
-; FLATSCR_GFX10-NEXT:    s_waitcnt_depctr 0xffe3
-; FLATSCR_GFX10-NEXT:    s_mov_b32 vcc_lo, 0
 ; FLATSCR_GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR_GFX10-NEXT:    scratch_store_short off, v0, vcc_lo offset:6
 ; FLATSCR_GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; FLATSCR_GFX10-NEXT:    global_load_ushort v0, v2, s[0:1] offset:4
-; FLATSCR_GFX10-NEXT:    s_waitcnt_depctr 0xffe3
-; FLATSCR_GFX10-NEXT:    s_mov_b32 vcc_lo, 0
 ; FLATSCR_GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR_GFX10-NEXT:    scratch_store_short off, v0, vcc_lo offset:8
 ; FLATSCR_GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; FLATSCR_GFX10-NEXT:    s_waitcnt_depctr 0xffe3
-; FLATSCR_GFX10-NEXT:    s_mov_b32 vcc_lo, 0
+; FLATSCR_GFX10-NEXT:    s_clause 0x1
 ; FLATSCR_GFX10-NEXT:    scratch_load_dword v0, off, vcc_lo offset:4
-; FLATSCR_GFX10-NEXT:    s_waitcnt_depctr 0xffe3
-; FLATSCR_GFX10-NEXT:    s_mov_b32 vcc_lo, 0
 ; FLATSCR_GFX10-NEXT:    scratch_load_dword v1, off, vcc_lo offset:6
 ; FLATSCR_GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR_GFX10-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
 ; FLATSCR_GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: vload2_private:
 ; GFX11:       ; %bb.0: ; %entry
 ; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x0

llvm/test/CodeGen/AMDGPU/flat-scratch.ll

 ; GFX9-NEXT:    s_mov_b32 s2, s0
 ; GFX9-NEXT:    s_mov_b32 s3, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:52
-; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:36
-; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:20
-; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:4
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: zero_init_kernel:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_add_u32 s0, s0, s3
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0

 ; GFX9-PAL-NEXT:    s_mov_b32 s1, s0
 ; GFX9-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX9-PAL-NEXT:    s_mov_b32 s3, s0
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:52
-; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:36
-; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:20
-; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:4
 ; GFX9-PAL-NEXT:    s_endpgm
 ;
 ; GFX940-LABEL: zero_init_kernel:
 ; GFX940:       ; %bb.0:
 ; GFX940-NEXT:    s_mov_b32 s0, 0
 ; GFX940-NEXT:    s_mov_b32 s1, s0
 ; GFX940-NEXT:    s_mov_b32 s2, s0

 ; GFX1010-PAL-NEXT:    s_mov_b32 s1, s0
 ; GFX1010-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX1010-PAL-NEXT:    s_mov_b32 s3, s0
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:52
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:36
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:20
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:4
 ; GFX1010-PAL-NEXT:    s_endpgm
 ;
 ; GFX1030-PAL-LABEL: zero_init_kernel:
 ; GFX1030-PAL:       ; %bb.0:
 ; GFX1030-PAL-NEXT:    s_getpc_b64 s[2:3]
 ; GFX1030-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX1030-PAL-NEXT:    s_load_dwordx2 s[2:3], s[2:3], 0x0

 ; GFX9-NEXT:    s_mov_b32 s0, 0
 ; GFX9-NEXT:    s_mov_b32 s1, s0
 ; GFX9-NEXT:    s_mov_b32 s2, s0
 ; GFX9-NEXT:    s_mov_b32 s3, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s3
-; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:260
-; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:276
-; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:292
-; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:308
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: zero_init_small_offset_kernel:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_add_u32 s0, s0, s3
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0

 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    s_mov_b32 s1, s0
 ; GFX9-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX9-PAL-NEXT:    s_mov_b32 s3, s0
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v3, s3
-; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:260
-; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:276
-; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:292
-; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:308
 ; GFX9-PAL-NEXT:    s_endpgm
 ;
 ; GFX940-LABEL: zero_init_small_offset_kernel:
 ; GFX940:       ; %bb.0:
 ; GFX940-NEXT:    scratch_load_dword v0, off, off offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_mov_b32 s0, 0

 ; GFX1010-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1010-PAL-NEXT:    s_mov_b32 s1, s0
 ; GFX1010-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX1010-PAL-NEXT:    s_mov_b32 s3, s0
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v3, s3
-; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:260
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:276
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:292
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_mov_b32 vcc_lo, 0
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:308
 ; GFX1010-PAL-NEXT:    s_endpgm
 ;
 ; GFX1030-PAL-LABEL: zero_init_small_offset_kernel:
 ; GFX1030-PAL:       ; %bb.0:
 ; GFX1030-PAL-NEXT:    s_getpc_b64 s[2:3]
 ; GFX1030-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX1030-PAL-NEXT:    s_load_dwordx2 s[2:3], s[2:3], 0x0

 ; GFX9-NEXT:    s_mov_b32 s2, s0
 ; GFX9-NEXT:    s_mov_b32 s3, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi
-; GFX9-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:16
-; GFX9-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:32
-; GFX9-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:48
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: zero_init_large_offset_kernel:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_add_u32 s0, s0, s3
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s0
 ; GFX10-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s1
 ; GFX10-NEXT:    scratch_load_dword v0, off, off offset:4 glc dlc
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    s_mov_b32 s0, 0
 ; GFX10-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX10-NEXT:    s_mov_b32 s1, s0
 ; GFX10-NEXT:    s_mov_b32 s2, s0
 ; GFX10-NEXT:    s_mov_b32 s3, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX10-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX10-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX10-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo
-; GFX10-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX10-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:16
-; GFX10-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX10-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:32
-; GFX10-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX10-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:48
 ; GFX10-NEXT:    s_endpgm
 ;
 ; GFX11-LABEL: zero_init_large_offset_kernel:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    scratch_load_b32 v0, off, off offset:4 glc dlc
 ; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    s_mov_b32 s0, 0
 ; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    s_mov_b32 s1, s0
 ; GFX11-NEXT:    s_mov_b32 s2, s0
 ; GFX11-NEXT:    s_mov_b32 s3, s0
 ; GFX11-NEXT:    v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v1, s1
 ; GFX11-NEXT:    v_dual_mov_b32 v2, s2 :: v_dual_mov_b32 v3, s3
+; GFX11-NEXT:    s_clause 0x3
 ; GFX11-NEXT:    scratch_store_b128 off, v[0:3], vcc_lo
-; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    scratch_store_b128 off, v[0:3], vcc_lo offset:16
-; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    scratch_store_b128 off, v[0:3], vcc_lo offset:32
-; GFX11-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-NEXT:    scratch_store_b128 off, v[0:3], vcc_lo offset:48
 ; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-NEXT:    s_endpgm
 ;
 ; GFX9-PAL-LABEL: zero_init_large_offset_kernel:
 ; GFX9-PAL:       ; %bb.0:
 ; GFX9-PAL-NEXT:    s_getpc_b64 s[2:3]
 ; GFX9-PAL-NEXT:    s_mov_b32 s2, s0

 ; GFX9-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX9-PAL-NEXT:    s_mov_b32 s3, s0
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-PAL-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi
-; GFX9-PAL-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:16
-; GFX9-PAL-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:32
-; GFX9-PAL-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:48
 ; GFX9-PAL-NEXT:    s_endpgm
 ;
 ; GFX940-LABEL: zero_init_large_offset_kernel:
 ; GFX940:       ; %bb.0:
 ; GFX940-NEXT:    scratch_load_dword v0, off, off offset:4 sc0 sc1
 ; GFX940-NEXT:    s_waitcnt vmcnt(0)
 ; GFX940-NEXT:    s_mov_b32 s0, 0
 ; GFX940-NEXT:    s_mov_b32 s1, s0
 ; GFX940-NEXT:    s_mov_b32 s2, s0
 ; GFX940-NEXT:    s_mov_b32 s3, s0
 ; GFX940-NEXT:    v_mov_b64_e32 v[0:1], s[0:1]
 ; GFX940-NEXT:    v_mov_b64_e32 v[2:3], s[2:3]
 ; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi
-; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:16
-; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:32
-; GFX940-NEXT:    s_movk_i32 vcc_hi, 0x4004
 ; GFX940-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:48
 ; GFX940-NEXT:    s_endpgm
 ;
 ; GFX1010-PAL-LABEL: zero_init_large_offset_kernel:
 ; GFX1010-PAL:       ; %bb.0:
 ; GFX1010-PAL-NEXT:    s_getpc_b64 s[2:3]
 ; GFX1010-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX1010-PAL-NEXT:    s_load_dwordx2 s[2:3], s[2:3], 0x0

 ; GFX1010-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX1010-PAL-NEXT:    s_mov_b32 s3, s0
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1010-PAL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX1010-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:16
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:32
-; GFX1010-PAL-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1010-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX1010-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:48
 ; GFX1010-PAL-NEXT:    s_endpgm
 ;
 ; GFX1030-PAL-LABEL: zero_init_large_offset_kernel:
 ; GFX1030-PAL:       ; %bb.0:
 ; GFX1030-PAL-NEXT:    s_getpc_b64 s[2:3]
 ; GFX1030-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX1030-PAL-NEXT:    s_load_dwordx2 s[2:3], s[2:3], 0x0

 ; GFX1030-PAL-NEXT:    s_mov_b32 s1, s0
 ; GFX1030-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX1030-PAL-NEXT:    s_mov_b32 s3, s0
 ; GFX1030-PAL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX1030-PAL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX1030-PAL-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1030-PAL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX1030-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo
-; GFX1030-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX1030-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:16
-; GFX1030-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX1030-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:32
-; GFX1030-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX1030-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_lo offset:48
 ; GFX1030-PAL-NEXT:    s_endpgm
 ;
 ; GFX11-PAL-LABEL: zero_init_large_offset_kernel:
 ; GFX11-PAL:       ; %bb.0:
 ; GFX11-PAL-NEXT:    scratch_load_b32 v0, off, off offset:4 glc dlc
 ; GFX11-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-PAL-NEXT:    s_mov_b32 s0, 0
 ; GFX11-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-PAL-NEXT:    s_mov_b32 s1, s0
 ; GFX11-PAL-NEXT:    s_mov_b32 s2, s0
 ; GFX11-PAL-NEXT:    s_mov_b32 s3, s0
 ; GFX11-PAL-NEXT:    v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v1, s1
 ; GFX11-PAL-NEXT:    v_dual_mov_b32 v2, s2 :: v_dual_mov_b32 v3, s3
+; GFX11-PAL-NEXT:    s_clause 0x3
 ; GFX11-PAL-NEXT:    scratch_store_b128 off, v[0:3], vcc_lo
-; GFX11-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-PAL-NEXT:    scratch_store_b128 off, v[0:3], vcc_lo offset:16
-; GFX11-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-PAL-NEXT:    scratch_store_b128 off, v[0:3], vcc_lo offset:32
-; GFX11-PAL-NEXT:    s_movk_i32 vcc_lo, 0x4004
 ; GFX11-PAL-NEXT:    scratch_store_b128 off, v[0:3], vcc_lo offset:48
 ; GFX11-PAL-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11-PAL-NEXT:    s_endpgm
   %padding = alloca [4096 x i32], align 4, addrspace(5)
   %alloca = alloca [32 x i16], align 2, addrspace(5)
   %pad_gep = getelementptr inbounds [4096 x i32], [4096 x i32] addrspace(5)* %padding, i32 0, i32 undef
   %pad_load = load volatile i32, i32 addrspace(5)* %pad_gep, align 4
   %cast = bitcast [32 x i16] addrspace(5)* %alloca to i8 addrspace(5)*

 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_addc_u32 flat_scratch_hi, s1, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v2, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v3, v0
 ; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:3024
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-NEXT:    scratch_load_dwordx4 v[0:3], off, vcc_hi offset:3024 glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 16
 ; GFX9-NEXT:    ;;#ASMSTART
 ; GFX9-NEXT:    ; use v0
 ; GFX9-NEXT:    ;;#ASMEND
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0x810
 ; GFX9-NEXT:    ;;#ASMSTART

 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v3, v0
 ; GFX9-PAL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-PAL-NEXT:    s_and_b32 s3, s3, 0xffff
 ; GFX9-PAL-NEXT:    s_add_u32 flat_scratch_lo, s2, s0
 ; GFX9-PAL-NEXT:    s_addc_u32 flat_scratch_hi, s3, 0
 ; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_store_dwordx4 off, v[0:3], vcc_hi offset:3024
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-PAL-NEXT:    s_mov_b32 vcc_hi, 0
 ; GFX9-PAL-NEXT:    scratch_load_dwordx4 v[0:3], off, vcc_hi offset:3024 glc
 ; GFX9-PAL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v0, 16
 ; GFX9-PAL-NEXT:    ;;#ASMSTART
 ; GFX9-PAL-NEXT:    ; use v0
 ; GFX9-PAL-NEXT:    ;;#ASMEND
 ; GFX9-PAL-NEXT:    v_mov_b32_e32 v0, 0x810
 ; GFX9-PAL-NEXT:    ;;#ASMSTART

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

 ; GCN-O1-NEXT:        MachineDominator Tree Construction
 ; GCN-O1-NEXT:        Machine Natural Loop Construction
 ; GCN-O1-NEXT:        Machine Block Frequency Analysis
 ; GCN-O1-NEXT:        MachinePostDominator Tree Construction
 ; GCN-O1-NEXT:        Lazy Machine Block Frequency Analysis
 ; GCN-O1-NEXT:        Machine Optimization Remark Emitter
 ; GCN-O1-NEXT:        Shrink Wrapping analysis
 ; GCN-O1-NEXT:        Prologue/Epilogue Insertion & Frame Finalization
+; GCN-O1-NEXT:        Machine Late Instructions Cleanup Pass
 ; GCN-O1-NEXT:        Control Flow Optimizer
 ; GCN-O1-NEXT:        Lazy Machine Block Frequency Analysis
 ; GCN-O1-NEXT:        Tail Duplication
 ; GCN-O1-NEXT:        Machine Copy Propagation Pass
 ; GCN-O1-NEXT:        Post-RA pseudo instruction expansion pass
 ; GCN-O1-NEXT:        SI Shrink Instructions
 ; GCN-O1-NEXT:        SI post-RA bundler
 ; GCN-O1-NEXT:        MachineDominator Tree Construction

 ; GCN-O1-OPTS-NEXT:        MachineDominator Tree Construction
 ; GCN-O1-OPTS-NEXT:        Machine Natural Loop Construction
 ; GCN-O1-OPTS-NEXT:        Machine Block Frequency Analysis
 ; GCN-O1-OPTS-NEXT:        MachinePostDominator Tree Construction
 ; GCN-O1-OPTS-NEXT:        Lazy Machine Block Frequency Analysis
 ; GCN-O1-OPTS-NEXT:        Machine Optimization Remark Emitter
 ; GCN-O1-OPTS-NEXT:        Shrink Wrapping analysis
 ; GCN-O1-OPTS-NEXT:        Prologue/Epilogue Insertion & Frame Finalization
+; GCN-O1-OPTS-NEXT:        Machine Late Instructions Cleanup Pass
 ; GCN-O1-OPTS-NEXT:        Control Flow Optimizer
 ; GCN-O1-OPTS-NEXT:        Lazy Machine Block Frequency Analysis
 ; GCN-O1-OPTS-NEXT:        Tail Duplication
 ; GCN-O1-OPTS-NEXT:        Machine Copy Propagation Pass
 ; GCN-O1-OPTS-NEXT:        Post-RA pseudo instruction expansion pass
 ; GCN-O1-OPTS-NEXT:        SI Shrink Instructions
 ; GCN-O1-OPTS-NEXT:        SI post-RA bundler
 ; GCN-O1-OPTS-NEXT:        MachineDominator Tree Construction

 ; GCN-O2-NEXT:        MachineDominator Tree Construction
 ; GCN-O2-NEXT:        Machine Natural Loop Construction
 ; GCN-O2-NEXT:        Machine Block Frequency Analysis
 ; GCN-O2-NEXT:        MachinePostDominator Tree Construction
 ; GCN-O2-NEXT:        Lazy Machine Block Frequency Analysis
 ; GCN-O2-NEXT:        Machine Optimization Remark Emitter
 ; GCN-O2-NEXT:        Shrink Wrapping analysis
 ; GCN-O2-NEXT:        Prologue/Epilogue Insertion & Frame Finalization
+; GCN-O2-NEXT:        Machine Late Instructions Cleanup Pass
 ; GCN-O2-NEXT:        Control Flow Optimizer
 ; GCN-O2-NEXT:        Lazy Machine Block Frequency Analysis
 ; GCN-O2-NEXT:        Tail Duplication
 ; GCN-O2-NEXT:        Machine Copy Propagation Pass
 ; GCN-O2-NEXT:        Post-RA pseudo instruction expansion pass
 ; GCN-O2-NEXT:        SI Shrink Instructions
 ; GCN-O2-NEXT:        SI post-RA bundler
 ; GCN-O2-NEXT:        MachineDominator Tree Construction

 ; GCN-O3-NEXT:        MachineDominator Tree Construction
 ; GCN-O3-NEXT:        Machine Natural Loop Construction
 ; GCN-O3-NEXT:        Machine Block Frequency Analysis
 ; GCN-O3-NEXT:        MachinePostDominator Tree Construction
 ; GCN-O3-NEXT:        Lazy Machine Block Frequency Analysis
 ; GCN-O3-NEXT:        Machine Optimization Remark Emitter
 ; GCN-O3-NEXT:        Shrink Wrapping analysis
 ; GCN-O3-NEXT:        Prologue/Epilogue Insertion & Frame Finalization
+; GCN-O3-NEXT:        Machine Late Instructions Cleanup Pass
 ; GCN-O3-NEXT:        Control Flow Optimizer
 ; GCN-O3-NEXT:        Lazy Machine Block Frequency Analysis
 ; GCN-O3-NEXT:        Tail Duplication
 ; GCN-O3-NEXT:        Machine Copy Propagation Pass
 ; GCN-O3-NEXT:        Post-RA pseudo instruction expansion pass
 ; GCN-O3-NEXT:        SI Shrink Instructions
 ; GCN-O3-NEXT:        SI post-RA bundler
 ; GCN-O3-NEXT:        MachineDominator Tree Construction

llvm/test/CodeGen/AMDGPU/local-stack-alloc-block-sp-reference.ll

 ; FLATSCR-NEXT:    s_addk_i32 s2, 0x2000
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[8:9], off, s2 offset:720 glc
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    scratch_load_dwordx4 v[0:3], off, s2 offset:704 glc
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    s_movk_i32 s2, 0x2000
 ; FLATSCR-NEXT:    scratch_load_dwordx2 v[10:11], off, s2 offset:16 glc
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
-; FLATSCR-NEXT:    s_movk_i32 s2, 0x2000
 ; FLATSCR-NEXT:    scratch_load_dwordx4 v[4:7], off, s2 glc
 ; FLATSCR-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR-NEXT:    v_mov_b32_e32 v12, 0
 ; FLATSCR-NEXT:    v_add_co_u32_e32 v2, vcc, v2, v6
 ; FLATSCR-NEXT:    v_addc_co_u32_e32 v3, vcc, v3, v7, vcc
 ; FLATSCR-NEXT:    v_add_co_u32_e32 v0, vcc, v0, v4
 ; FLATSCR-NEXT:    v_addc_co_u32_e32 v1, vcc, v1, v5, vcc
 ; FLATSCR-NEXT:    v_add_co_u32_e32 v4, vcc, v8, v10

llvm/test/CodeGen/AMDGPU/multilevel-break.ll

 ; GCN-NEXT:    v_readfirstlane_b32 s8, v1
 ; GCN-NEXT:    s_mov_b64 s[4:5], -1
 ; GCN-NEXT:    s_cmp_lt_i32 s8, 1
 ; GCN-NEXT:    s_mov_b64 s[6:7], -1
 ; GCN-NEXT:    s_cbranch_scc1 .LBB1_6
 ; GCN-NEXT:  ; %bb.3: ; %LeafBlock1
 ; GCN-NEXT:    ; in Loop: Header=BB1_2 Depth=1
 ; GCN-NEXT:    s_cmp_eq_u32 s8, 1
-; GCN-NEXT:    s_mov_b64 s[4:5], -1
 ; GCN-NEXT:    s_cbranch_scc0 .LBB1_5
 ; GCN-NEXT:  ; %bb.4: ; %case1
 ; GCN-NEXT:    ; in Loop: Header=BB1_2 Depth=1
 ; GCN-NEXT:    buffer_load_dword v1, off, s[0:3], 0 glc
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    v_cmp_ge_i32_e32 vcc, v0, v1
 ; GCN-NEXT:    s_orn2_b64 s[4:5], vcc, exec
 ; GCN-NEXT:  .LBB1_5: ; %Flow3

llvm/test/CodeGen/AMDGPU/si-annotate-cf.ll

 ; SI-NEXT:    s_and_b64 s[2:3], exec, s[2:3]
 ; SI-NEXT:    s_and_b64 s[4:5], exec, s[6:7]
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
 ; SI-NEXT:    v_mov_b32_e32 v0, 3
 ; SI-NEXT:    s_branch .LBB3_3
 ; SI-NEXT:  .LBB3_1: ; in Loop: Header=BB3_3 Depth=1
 ; SI-NEXT:    s_mov_b64 s[8:9], 0
-; SI-NEXT:    s_mov_b64 s[10:11], -1
-; SI-NEXT:    s_mov_b64 s[12:13], -1
 ; SI-NEXT:  .LBB3_2: ; %Flow
 ; SI-NEXT:    ; in Loop: Header=BB3_3 Depth=1
 ; SI-NEXT:    s_and_b64 vcc, exec, s[12:13]
 ; SI-NEXT:    s_cbranch_vccnz .LBB3_8
 ; SI-NEXT:  .LBB3_3: ; %while.cond
 ; SI-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; SI-NEXT:    s_mov_b64 s[10:11], -1
 ; SI-NEXT:    s_mov_b64 s[8:9], -1
 ; SI-NEXT:    s_mov_b64 s[12:13], -1
 ; SI-NEXT:    s_mov_b64 vcc, s[0:1]
 ; SI-NEXT:    s_cbranch_vccz .LBB3_2
 ; SI-NEXT:  ; %bb.4: ; %convex.exit
 ; SI-NEXT:    ; in Loop: Header=BB3_3 Depth=1
 ; SI-NEXT:    s_mov_b64 vcc, s[2:3]
 ; SI-NEXT:    s_cbranch_vccz .LBB3_1
 ; SI-NEXT:  ; %bb.5: ; %if.end
 ; SI-NEXT:    ; in Loop: Header=BB3_3 Depth=1
-; SI-NEXT:    s_mov_b64 s[12:13], -1
 ; SI-NEXT:    s_mov_b64 vcc, s[4:5]
 ; SI-NEXT:    s_cbranch_vccz .LBB3_7
 ; SI-NEXT:  ; %bb.6: ; %if.else
 ; SI-NEXT:    ; in Loop: Header=BB3_3 Depth=1
 ; SI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    s_mov_b64 s[12:13], 0
 ; SI-NEXT:  .LBB3_7: ; %Flow6

 ; FLAT-NEXT:    s_and_b64 s[2:3], exec, s[2:3]
 ; FLAT-NEXT:    s_and_b64 s[4:5], exec, s[6:7]
 ; FLAT-NEXT:    s_mov_b32 s7, 0xf000
 ; FLAT-NEXT:    s_mov_b32 s6, -1
 ; FLAT-NEXT:    v_mov_b32_e32 v0, 3
 ; FLAT-NEXT:    s_branch .LBB3_3
 ; FLAT-NEXT:  .LBB3_1: ; in Loop: Header=BB3_3 Depth=1
 ; FLAT-NEXT:    s_mov_b64 s[8:9], 0
-; FLAT-NEXT:    s_mov_b64 s[10:11], -1
-; FLAT-NEXT:    s_mov_b64 s[12:13], -1
 ; FLAT-NEXT:  .LBB3_2: ; %Flow
 ; FLAT-NEXT:    ; in Loop: Header=BB3_3 Depth=1
 ; FLAT-NEXT:    s_and_b64 vcc, exec, s[12:13]
 ; FLAT-NEXT:    s_cbranch_vccnz .LBB3_8
 ; FLAT-NEXT:  .LBB3_3: ; %while.cond
 ; FLAT-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; FLAT-NEXT:    s_mov_b64 s[10:11], -1
 ; FLAT-NEXT:    s_mov_b64 s[8:9], -1
 ; FLAT-NEXT:    s_mov_b64 s[12:13], -1
 ; FLAT-NEXT:    s_mov_b64 vcc, s[0:1]
 ; FLAT-NEXT:    s_cbranch_vccz .LBB3_2
 ; FLAT-NEXT:  ; %bb.4: ; %convex.exit
 ; FLAT-NEXT:    ; in Loop: Header=BB3_3 Depth=1
 ; FLAT-NEXT:    s_mov_b64 vcc, s[2:3]
 ; FLAT-NEXT:    s_cbranch_vccz .LBB3_1
 ; FLAT-NEXT:  ; %bb.5: ; %if.end
 ; FLAT-NEXT:    ; in Loop: Header=BB3_3 Depth=1
-; FLAT-NEXT:    s_mov_b64 s[12:13], -1
 ; FLAT-NEXT:    s_mov_b64 vcc, s[4:5]
 ; FLAT-NEXT:    s_cbranch_vccz .LBB3_7
 ; FLAT-NEXT:  ; %bb.6: ; %if.else
 ; FLAT-NEXT:    ; in Loop: Header=BB3_3 Depth=1
 ; FLAT-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; FLAT-NEXT:    s_waitcnt vmcnt(0)
 ; FLAT-NEXT:    s_mov_b64 s[12:13], 0
 ; FLAT-NEXT:  .LBB3_7: ; %Flow6

llvm/test/CodeGen/AMDGPU/si-unify-exit-multiple-unreachables.ll

 ; CHECK-NEXT:    s_mov_b64 s[2:3], 0
 ; CHECK-NEXT:    s_mov_b64 s[0:1], 0
 ; CHECK-NEXT:    s_and_saveexec_b64 s[8:9], vcc
 ; CHECK-NEXT:    s_cbranch_execz .LBB0_5
 ; CHECK-NEXT:  ; %bb.2: ; %if.then3
 ; CHECK-NEXT:    s_cmp_lg_u32 s10, 0
 ; CHECK-NEXT:    s_cbranch_scc1 .LBB0_14
 ; CHECK-NEXT:  ; %bb.3:
-; CHECK-NEXT:    s_mov_b64 s[2:3], 0
 ; CHECK-NEXT:    s_mov_b64 s[0:1], -1
 ; CHECK-NEXT:  .LBB0_4: ; %Flow3
 ; CHECK-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; CHECK-NEXT:    s_and_b64 s[2:3], s[2:3], exec
 ; CHECK-NEXT:  .LBB0_5: ; %Flow2
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[8:9]
 ; CHECK-NEXT:    s_and_b64 vcc, exec, s[6:7]
 ; CHECK-NEXT:    s_cbranch_vccz .LBB0_8

 ; CHECK-NEXT:    s_mov_b64 s[0:1], 0
 ; CHECK-NEXT:    s_or_b64 s[2:3], s[2:3], exec
 ; CHECK-NEXT:    s_trap 2
 ; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[2:3]
 ; CHECK-NEXT:    s_cbranch_execnz .LBB0_9
 ; CHECK-NEXT:    s_branch .LBB0_10
 ; CHECK-NEXT:  .LBB0_14: ; %cond.false.i8
 ; CHECK-NEXT:    s_mov_b64 s[2:3], -1
-; CHECK-NEXT:    s_mov_b64 s[0:1], 0
 ; CHECK-NEXT:    s_trap 2
 ; CHECK-NEXT:    s_branch .LBB0_4
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %cmp = icmp eq i32 %n, 256
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:

llvm/test/CodeGen/AMDGPU/spill-offset-calculation.ll

entry:
   %asm5 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32 } %asm, 5
   %asm6 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32 } %asm, 6
   %asm7 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32 } %asm, 7
 
   call void asm sideeffect "", "~{v0},~{v1},~{v2},~{v3},~{v4},~{v5},~{v6},~{v7}"() #0
 
   ; MUBUF: v_mov_b32_e32 [[OFFSET:v[0-9]+]], 0x1004
   ; MUBUF: buffer_load_dword v{{[0-9]+}}, [[OFFSET]], s[{{[0-9]+:[0-9]+}}], 0 offen ; 4-byte Folded Reload
-  ; FLATSCR: s_movk_i32 [[SOFF:s[0-9]+]], 0x1004
   ; FLATSCR: scratch_load_dword v{{[0-9]+}}, off, [[SOFF]] ; 4-byte Folded Reload
 
    ; Force %a to spill with no free SGPRs
   call void asm sideeffect "", "s,s,s,s,s,s,s,s,v"(i32 %asm0, i32 %asm1, i32 %asm2, i32 %asm3, i32 %asm4, i32 %asm5, i32 %asm6, i32 %asm7, i32 %a)
   ret void
 }
 
 ; GCN-LABEL: test_sgpr_offset_subregs_kernel

llvm/test/CodeGen/AMDGPU/spill-scavenge-offset.ll

 ; GFX6: s_mov_b64 exec, s
 
 ; GFX6: NumSgprs: 48
 ; GFX6: ScratchSize: 8608
 
 ; FLATSCR:           s_movk_i32 [[SOFF1:s[0-9]+]], 0x
 ; GFX9-FLATSCR:      s_waitcnt vmcnt(0)
 ; FLATSCR:           scratch_store_dwordx4 off, v[{{[0-9:]+}}], [[SOFF1]] ; 16-byte Folded Spill
-; FLATSCR:           s_movk_i32 [[SOFF2:s[0-9]+]], 0x
-; FLATSCR:           scratch_load_dwordx4 v[{{[0-9:]+}}], off, [[SOFF2]] ; 16-byte Folded Reload
+; FLATSCR:           scratch_load_dwordx4 v[{{[0-9:]+}}], off, [[SOFF1]] ; 16-byte Folded Reload
 define amdgpu_kernel void @test_limited_sgpr(<64 x i32> addrspace(1)* %out, <64 x i32> addrspace(1)* %in) #0 {
 entry:
   %lo = call i32 @llvm.amdgcn.mbcnt.lo(i32 -1, i32 0)
   %tid = call i32 @llvm.amdgcn.mbcnt.hi(i32 -1, i32 %lo)
 
 ; allocate enough scratch to go beyond 2^12 addressing
   %scratch = alloca <1280 x i32>, align 8, addrspace(5)
 

llvm/test/CodeGen/ARM/O3-pipeline.ll

 ; CHECK-NEXT:      PostRA Machine Sink
 ; CHECK-NEXT:      Machine Block Frequency Analysis
 ; CHECK-NEXT:      MachineDominator Tree Construction
 ; CHECK-NEXT:      MachinePostDominator Tree Construction
 ; CHECK-NEXT:      Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:      Machine Optimization Remark Emitter
 ; CHECK-NEXT:      Shrink Wrapping analysis
 ; CHECK-NEXT:      Prologue/Epilogue Insertion & Frame Finalization
+; CHECK-NEXT:      Machine Late Instructions Cleanup Pass
 ; CHECK-NEXT:      Control Flow Optimizer
 ; CHECK-NEXT:      Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:      Tail Duplication
 ; CHECK-NEXT:      Machine Copy Propagation Pass
 ; CHECK-NEXT:      Post-RA pseudo instruction expansion pass
 ; CHECK-NEXT:      ARM load / store optimization pass
 ; CHECK-NEXT:      ReachingDefAnalysis
 ; CHECK-NEXT:      ARM Execution Domain Fix

llvm/test/CodeGen/ARM/arm-shrink-wrapping.ll

 ; ARM-DISABLE-NEXT:  LBB11_5: @ %end
 ; ARM-DISABLE-NEXT:    bx lr
 ;
 ; THUMB-ENABLE-LABEL: infiniteloop3:
 ; THUMB-ENABLE:       @ %bb.0: @ %entry
 ; THUMB-ENABLE-NEXT:    movs r0, #0
 ; THUMB-ENABLE-NEXT:    cbnz r0, LBB11_5
 ; THUMB-ENABLE-NEXT:  @ %bb.1: @ %loop2a.preheader
-; THUMB-ENABLE-NEXT:    movs r0, #0
 ; THUMB-ENABLE-NEXT:    movs r1, #0
 ; THUMB-ENABLE-NEXT:    mov r2, r0
 ; THUMB-ENABLE-NEXT:    b LBB11_3
 ; THUMB-ENABLE-NEXT:  LBB11_2: @ %loop2b
 ; THUMB-ENABLE-NEXT:    @ in Loop: Header=BB11_3 Depth=1
 ; THUMB-ENABLE-NEXT:    str r1, [r2]
 ; THUMB-ENABLE-NEXT:    mov r2, r1
 ; THUMB-ENABLE-NEXT:    mov r1, r3

 ; THUMB-ENABLE-NEXT:  LBB11_5: @ %end
 ; THUMB-ENABLE-NEXT:    bx lr
 ;
 ; THUMB-DISABLE-LABEL: infiniteloop3:
 ; THUMB-DISABLE:       @ %bb.0: @ %entry
 ; THUMB-DISABLE-NEXT:    movs r0, #0
 ; THUMB-DISABLE-NEXT:    cbnz r0, LBB11_5
 ; THUMB-DISABLE-NEXT:  @ %bb.1: @ %loop2a.preheader
-; THUMB-DISABLE-NEXT:    movs r0, #0
 ; THUMB-DISABLE-NEXT:    movs r1, #0
 ; THUMB-DISABLE-NEXT:    mov r2, r0
 ; THUMB-DISABLE-NEXT:    b LBB11_3
 ; THUMB-DISABLE-NEXT:  LBB11_2: @ %loop2b
 ; THUMB-DISABLE-NEXT:    @ in Loop: Header=BB11_3 Depth=1
 ; THUMB-DISABLE-NEXT:    str r1, [r2]
 ; THUMB-DISABLE-NEXT:    mov r2, r1
 ; THUMB-DISABLE-NEXT:    mov r1, r3

llvm/test/CodeGen/ARM/fpclamptosat.ll

 ; SOFT-NEXT:    mvns r6, r3
 ; SOFT-NEXT:    ldr r0, .LCPI48_0
 ; SOFT-NEXT:    cmp r4, r0
 ; SOFT-NEXT:    ldr r3, [sp, #16] @ 4-byte Reload
 ; SOFT-NEXT:    blo .LBB48_19
 ; SOFT-NEXT:  @ %bb.18: @ %entry
 ; SOFT-NEXT:    mov r3, r6
 ; SOFT-NEXT:  .LBB48_19: @ %entry
-; SOFT-NEXT:    ldr r0, .LCPI48_0
 ; SOFT-NEXT:    cmp r4, r0
 ; SOFT-NEXT:    ldr r4, [sp, #16] @ 4-byte Reload
 ; SOFT-NEXT:    beq .LBB48_21
 ; SOFT-NEXT:  @ %bb.20: @ %entry
 ; SOFT-NEXT:    mov r4, r3
 ; SOFT-NEXT:  .LBB48_21: @ %entry
 ; SOFT-NEXT:    cmp r7, #0
 ; SOFT-NEXT:    bmi .LBB48_23

 ; SOFT-NEXT:    mvns r6, r3
 ; SOFT-NEXT:    ldr r0, .LCPI51_0
 ; SOFT-NEXT:    cmp r4, r0
 ; SOFT-NEXT:    ldr r3, [sp, #16] @ 4-byte Reload
 ; SOFT-NEXT:    blo .LBB51_19
 ; SOFT-NEXT:  @ %bb.18: @ %entry
 ; SOFT-NEXT:    mov r3, r6
 ; SOFT-NEXT:  .LBB51_19: @ %entry
-; SOFT-NEXT:    ldr r0, .LCPI51_0
 ; SOFT-NEXT:    cmp r4, r0
 ; SOFT-NEXT:    ldr r4, [sp, #16] @ 4-byte Reload
 ; SOFT-NEXT:    beq .LBB51_21
 ; SOFT-NEXT:  @ %bb.20: @ %entry
 ; SOFT-NEXT:    mov r4, r3
 ; SOFT-NEXT:  .LBB51_21: @ %entry
 ; SOFT-NEXT:    cmp r7, #0
 ; SOFT-NEXT:    bmi .LBB51_23

llvm/test/CodeGen/ARM/ifcvt-branch-weight-bug.ll

 ; for.body -> lor.lhs.false.i (50%)
 ;          -> for.cond.backedge (50%)
 ; lor.lhs.false.i -> for.cond.backedge (100%)
 ;                 -> cond.false.i (0%)
 ; Afer if conversion, we have
 ; for.body -> for.cond.backedge (100%)
 ;          -> cond.false.i (0%)
 ; CHECK: bb.1.for.body:
-; CHECK: successors: %bb.2(0x80000000), %bb.4(0x00000000)
+; CHECK: successors: %bb.2(0x80000000), %bb.5(0x00000000)
 for.body:
   br i1 undef, label %for.cond.backedge, label %lor.lhs.false.i, !prof !1
 
 for.cond.backedge:
   %tobool = icmp eq %classL* %p0, null
   br i1 %tobool, label %for.end, label %for.body
 
 lor.lhs.false.i:

llvm/test/CodeGen/ARM/jump-table-islands.ll

 ; RUN: llc -mtriple=armv7-apple-ios8.0 -o - %s | FileCheck %s
 
-%BigInt = type i5500
+%BigInt = type i8500
 
 define %BigInt @test_moved_jumptable(i1 %tst, i32 %sw, %BigInt %l) {
 ; CHECK-LABEL: test_moved_jumptable:
 
 ; CHECK:   adr {{r[0-9]+}}, [[JUMP_TABLE:LJTI[0-9]+_[0-9]+]]
 ; CHECK:   b [[SKIP_TABLE:LBB[0-9]+_[0-9]+]]
 
 ; CHECK: [[JUMP_TABLE]]:

llvm/test/CodeGen/ARM/reg_sequence.ll

 ; CHECK-NEXT:  vadd.f32 [[Q1:q[0-9]+]], [[Q8]], [[Q8]]
 ; CHECK-NEXT:  vmul.f32 [[Q8]], [[Q9]], d1[0]
 ; CHECK-NEXT:  vmul.f32 [[Q8]], [[Q8]], [[Q8]]
 ; CHECK-NEXT:  vadd.f32 [[Q8]], [[Q8]], [[Q8]]
 ; CHECK-NEXT:  vmul.f32 [[Q8]], [[Q8]], [[Q8]]
 ; CHECK-NEXT:  vst1.32  {d17[1]}, [r0:32]
 ; CHECK-NEXT:  mov  r0, #0
 ; CHECK-NEXT:  cmp  r0, #0
-; CHECK-NEXT:  movne  r0, #0
 ; CHECK-NEXT:  bxne lr
 ; CHECK-NEXT:  LBB9_1:
 ; CHECK-NEXT:  trap
 entry:
   %0 = shufflevector <4 x float> zeroinitializer, <4 x float> undef, <4 x i32> zeroinitializer ; <<4 x float>> [#uses=1]
   %1 = insertelement <4 x float> %0, float %x, i32 1 ; <<4 x float>> [#uses=1]
   %2 = insertelement <4 x float> %1, float %x, i32 2 ; <<4 x float>> [#uses=1]
   %3 = insertelement <4 x float> %2, float %x, i32 3 ; <<4 x float>> [#uses=1]

llvm/test/CodeGen/BPF/objdump_cond_op_2.ll

 ; RUN: llc -march=bpfel -filetype=obj -o - %s | llvm-objdump --no-print-imm-hex -d - | FileCheck %s
 
 ; Source Code:
 ; int test(int a, int b) {
 ;   int s = 0;
 ;   while (a < b) { s++; a += s; b -= s; }
 ;   return s;
 ; }
 
 define i32 @test(i32, i32) local_unnamed_addr #0 {
 ; CHECK-LABEL: <test>:
   %3 = icmp slt i32 %0, %1
   br i1 %3, label %4, label %13
 
 ; <label>:4:                                      ; preds = %2
   br label %5
-; CHECK: if r4 s>= r3 goto +11 <LBB0_3>
-; CHECK: r0 = 0
-; CHECK-LABEL: <LBB0_2>:
+; CHECK: if r4 s>= r3 goto +10 <LBB0_2>
+; CHECK-LABEL: <LBB0_1>:
 
 ; <label>:5:                                      ; preds = %4, %5
   %6 = phi i32 [ %9, %5 ], [ 0, %4 ]
   %7 = phi i32 [ %11, %5 ], [ %1, %4 ]
   %8 = phi i32 [ %10, %5 ], [ %0, %4 ]
   %9 = add nuw nsw i32 %6, 1
   %10 = add nsw i32 %9, %8
   %11 = sub nsw i32 %7, %9
   %12 = icmp slt i32 %10, %11
   br i1 %12, label %5, label %13
 ; CHECK: r1 = r3
-; CHECK: if r2 s> r3 goto -10 <LBB0_2>
+; CHECK: if r2 s> r3 goto -10 <LBB0_1>
 
 ; <label>:13:                                     ; preds = %5, %2
   %14 = phi i32 [ 0, %2 ], [ %9, %5 ]
   ret i32 %14
-; CHECK-LABEL: <LBB0_3>:
+; CHECK-LABEL: <LBB0_2>:
 ; CHECK: exit
 }
 attributes #0 = { norecurse nounwind readnone }

llvm/test/CodeGen/Mips/llvm-ir/lshr.ll

 ; MMR3-NEXT:    lw $7, 20($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    movn $4, $9, $7
 ; MMR3-NEXT:    lw $6, 4($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    li16 $7, 0
 ; MMR3-NEXT:    movn $6, $7, $17
 ; MMR3-NEXT:    or16 $6, $4
 ; MMR3-NEXT:    lw $4, 8($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    movn $1, $7, $4
-; MMR3-NEXT:    li16 $7, 0
 ; MMR3-NEXT:    movn $1, $6, $10
 ; MMR3-NEXT:    lw $4, 24($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    movz $1, $4, $16
 ; MMR3-NEXT:    movn $2, $7, $17
 ; MMR3-NEXT:    li16 $4, 0
 ; MMR3-NEXT:    movz $2, $4, $10
 ; MMR3-NEXT:    move $4, $1
 ; MMR3-NEXT:    lwp $16, 32($sp)

llvm/test/CodeGen/Mips/llvm-ir/shl.ll

 ; MMR3-NEXT:    lw $7, 24($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    movn $3, $9, $7
 ; MMR3-NEXT:    lw $5, 8($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    li16 $7, 0
 ; MMR3-NEXT:    movn $5, $7, $17
 ; MMR3-NEXT:    or16 $5, $3
 ; MMR3-NEXT:    lw $3, 12($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    movn $8, $7, $3
-; MMR3-NEXT:    li16 $7, 0
 ; MMR3-NEXT:    movn $8, $5, $10
 ; MMR3-NEXT:    lw $3, 28($sp) # 4-byte Folded Reload
 ; MMR3-NEXT:    movz $8, $3, $16
 ; MMR3-NEXT:    movn $6, $7, $17
 ; MMR3-NEXT:    li16 $3, 0
 ; MMR3-NEXT:    movz $6, $3, $10
 ; MMR3-NEXT:    move $3, $8
 ; MMR3-NEXT:    move $5, $6

llvm/test/CodeGen/PowerPC/O3-pipeline.ll

 ; CHECK-NEXT:       PostRA Machine Sink
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       MachinePostDominator Tree Construction
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Shrink Wrapping analysis
 ; CHECK-NEXT:       Prologue/Epilogue Insertion & Frame Finalization
+; CHECK-NEXT:       Machine Late Instructions Cleanup Pass
 ; CHECK-NEXT:       Control Flow Optimizer
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Tail Duplication
 ; CHECK-NEXT:       Machine Copy Propagation Pass
 ; CHECK-NEXT:       Post-RA pseudo instruction expansion pass
 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       Machine Natural Loop Construction
 ; CHECK-NEXT:       Machine Block Frequency Analysis

llvm/test/CodeGen/PowerPC/cgp-select.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -O3 -mcpu=pwr9 -verify-machineinstrs -mtriple=powerpc64le-unknown-unknown < %s | FileCheck %s
 
 define dso_local void @wibble(ptr nocapture readonly %arg, i32 signext %arg1, ptr nocapture %arg2, ptr nocapture %arg3) {
 ; CHECK-LABEL: wibble:
 ; CHECK:       # %bb.0: # %bb
 ; CHECK-NEXT:    lfs 0, 0(3)
 ; CHECK-NEXT:    li 7, 7
 ; CHECK-NEXT:    cmpwi 4, 2
 ; CHECK-NEXT:    xsaddsp 0, 0, 0
 ; CHECK-NEXT:    blt 0, .LBB0_5
 ; CHECK-NEXT:  # %bb.1: # %bb6
 ; CHECK-NEXT:    clrldi 4, 4, 32
-; CHECK-NEXT:    li 7, 7
 ; CHECK-NEXT:    addi 4, 4, -1
 ; CHECK-NEXT:    mtctr 4
 ; CHECK-NEXT:    li 4, 8
 ; CHECK-NEXT:    b .LBB0_3
 ; CHECK-NEXT:    .p2align 5
 ; CHECK-NEXT:  .LBB0_2: # %bb11
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    iselgt 7, 4, 7

llvm/test/CodeGen/PowerPC/fp-strict-conv-f128.ll

 ; P8-NEXT:    .cfi_offset lr, 16
 ; P8-NEXT:    .cfi_offset r30, -16
 ; P8-NEXT:    addis r3, r2, .LCPI13_0@toc@ha
 ; P8-NEXT:    xxlxor f3, f3, f3
 ; P8-NEXT:    std r30, 112(r1) # 8-byte Folded Spill
 ; P8-NEXT:    lfs f0, .LCPI13_0@toc@l(r3)
 ; P8-NEXT:    lis r3, -32768
 ; P8-NEXT:    fcmpo cr0, f2, f3
-; P8-NEXT:    xxlxor f3, f3, f3
 ; P8-NEXT:    fcmpo cr1, f1, f0
 ; P8-NEXT:    crand 4*cr5+lt, 4*cr1+eq, lt
 ; P8-NEXT:    crandc 4*cr5+gt, 4*cr1+lt, 4*cr1+eq
 ; P8-NEXT:    cror 4*cr5+lt, 4*cr5+gt, 4*cr5+lt
 ; P8-NEXT:    isel r30, 0, r3, 4*cr5+lt
 ; P8-NEXT:    bc 12, 4*cr5+lt, .LBB13_2
 ; P8-NEXT:  # %bb.1: # %entry
 ; P8-NEXT:    fmr f3, f0

 ; P9-NEXT:    std r0, 16(r1)
 ; P9-NEXT:    stdu r1, -48(r1)
 ; P9-NEXT:    addis r3, r2, .LCPI13_0@toc@ha
 ; P9-NEXT:    xxlxor f3, f3, f3
 ; P9-NEXT:    lfs f0, .LCPI13_0@toc@l(r3)
 ; P9-NEXT:    fcmpo cr1, f2, f3
 ; P9-NEXT:    lis r3, -32768
 ; P9-NEXT:    fcmpo cr0, f1, f0
-; P9-NEXT:    xxlxor f3, f3, f3
 ; P9-NEXT:    crand 4*cr5+lt, eq, 4*cr1+lt
 ; P9-NEXT:    crandc 4*cr5+gt, lt, eq
 ; P9-NEXT:    cror 4*cr5+lt, 4*cr5+gt, 4*cr5+lt
 ; P9-NEXT:    isel r30, 0, r3, 4*cr5+lt
 ; P9-NEXT:    bc 12, 4*cr5+lt, .LBB13_2
 ; P9-NEXT:  # %bb.1: # %entry
 ; P9-NEXT:    fmr f3, f0
 ; P9-NEXT:  .LBB13_2: # %entry

llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll

 ; PC64LE-NEXT:    std 30, -16(1) # 8-byte Folded Spill
 ; PC64LE-NEXT:    std 0, 16(1)
 ; PC64LE-NEXT:    stdu 1, -48(1)
 ; PC64LE-NEXT:    addis 3, 2, .LCPI31_0@toc@ha
 ; PC64LE-NEXT:    xxlxor 3, 3, 3
 ; PC64LE-NEXT:    lfs 0, .LCPI31_0@toc@l(3)
 ; PC64LE-NEXT:    lis 3, -32768
 ; PC64LE-NEXT:    fcmpo 0, 2, 3
-; PC64LE-NEXT:    xxlxor 3, 3, 3
 ; PC64LE-NEXT:    fcmpo 1, 1, 0
 ; PC64LE-NEXT:    crand 20, 6, 0
 ; PC64LE-NEXT:    crandc 21, 4, 6
 ; PC64LE-NEXT:    cror 20, 21, 20
 ; PC64LE-NEXT:    isel 30, 0, 3, 20
 ; PC64LE-NEXT:    bc 12, 20, .LBB31_2
 ; PC64LE-NEXT:  # %bb.1: # %entry
 ; PC64LE-NEXT:    fmr 3, 0

 ; PC64LE9-NEXT:    std 0, 16(1)
 ; PC64LE9-NEXT:    stdu 1, -48(1)
 ; PC64LE9-NEXT:    addis 3, 2, .LCPI31_0@toc@ha
 ; PC64LE9-NEXT:    xxlxor 3, 3, 3
 ; PC64LE9-NEXT:    lfs 0, .LCPI31_0@toc@l(3)
 ; PC64LE9-NEXT:    fcmpo 1, 2, 3
 ; PC64LE9-NEXT:    lis 3, -32768
 ; PC64LE9-NEXT:    fcmpo 0, 1, 0
-; PC64LE9-NEXT:    xxlxor 3, 3, 3
 ; PC64LE9-NEXT:    crand 20, 2, 4
 ; PC64LE9-NEXT:    crandc 21, 0, 2
 ; PC64LE9-NEXT:    cror 20, 21, 20
 ; PC64LE9-NEXT:    isel 30, 0, 3, 20
 ; PC64LE9-NEXT:    bc 12, 20, .LBB31_2
 ; PC64LE9-NEXT:  # %bb.1: # %entry
 ; PC64LE9-NEXT:    fmr 3, 0
 ; PC64LE9-NEXT:  .LBB31_2: # %entry

llvm/test/CodeGen/RISCV/O3-pipeline.ll

 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       Machine Natural Loop Construction
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       MachinePostDominator Tree Construction
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Shrink Wrapping analysis
 ; CHECK-NEXT:       Prologue/Epilogue Insertion & Frame Finalization
+; CHECK-NEXT:       Machine Late Instructions Cleanup Pass
 ; CHECK-NEXT:       Control Flow Optimizer
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Tail Duplication
 ; CHECK-NEXT:       Machine Copy Propagation Pass
 ; CHECK-NEXT:       Post-RA pseudo instruction expansion pass
 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       Machine Natural Loop Construction
 ; CHECK-NEXT:       Post RA top-down list latency scheduler

llvm/test/CodeGen/RISCV/branch-relaxation.ll

 ; CHECK-RV32-NEXT:    #APP
 ; CHECK-RV32-NEXT:    # reg use t5
 ; CHECK-RV32-NEXT:    #NO_APP
 ; CHECK-RV32-NEXT:    #APP
 ; CHECK-RV32-NEXT:    # reg use t6
 ; CHECK-RV32-NEXT:    #NO_APP
 ; CHECK-RV32-NEXT:    lui a0, 2
 ; CHECK-RV32-NEXT:    sub sp, s0, a0
-; CHECK-RV32-NEXT:    lui a0, 2
 ; CHECK-RV32-NEXT:    addi a0, a0, -2032
 ; CHECK-RV32-NEXT:    add sp, sp, a0
 ; CHECK-RV32-NEXT:    lw ra, 2028(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s0, 2024(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s1, 2020(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s2, 2016(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s3, 2012(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s4, 2008(sp) # 4-byte Folded Reload

 ; CHECK-RV64-NEXT:    #APP
 ; CHECK-RV64-NEXT:    # reg use t5
 ; CHECK-RV64-NEXT:    #NO_APP
 ; CHECK-RV64-NEXT:    #APP
 ; CHECK-RV64-NEXT:    # reg use t6
 ; CHECK-RV64-NEXT:    #NO_APP
 ; CHECK-RV64-NEXT:    lui a0, 2
 ; CHECK-RV64-NEXT:    sub sp, s0, a0
-; CHECK-RV64-NEXT:    lui a0, 2
 ; CHECK-RV64-NEXT:    addiw a0, a0, -2032
 ; CHECK-RV64-NEXT:    add sp, sp, a0
 ; CHECK-RV64-NEXT:    ld ra, 2024(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s0, 2016(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s1, 2008(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s2, 2000(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s3, 1992(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s4, 1984(sp) # 8-byte Folded Reload

 ; CHECK-RV32-NEXT:    addi a0, a0, -200
 ; CHECK-RV32-NEXT:    add a0, sp, a0
 ; CHECK-RV32-NEXT:    lw t6, 0(a0) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    #APP
 ; CHECK-RV32-NEXT:    # reg use t6
 ; CHECK-RV32-NEXT:    #NO_APP
 ; CHECK-RV32-NEXT:    lui a0, 2
 ; CHECK-RV32-NEXT:    sub sp, s0, a0
-; CHECK-RV32-NEXT:    lui a0, 2
 ; CHECK-RV32-NEXT:    addi a0, a0, -2032
 ; CHECK-RV32-NEXT:    add sp, sp, a0
 ; CHECK-RV32-NEXT:    lw ra, 2028(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s0, 2024(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s1, 2020(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s2, 2016(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s3, 2012(sp) # 4-byte Folded Reload
 ; CHECK-RV32-NEXT:    lw s4, 2008(sp) # 4-byte Folded Reload

 ; CHECK-RV64-NEXT:    #APP
 ; CHECK-RV64-NEXT:    # reg use t5
 ; CHECK-RV64-NEXT:    #NO_APP
 ; CHECK-RV64-NEXT:    #APP
 ; CHECK-RV64-NEXT:    # reg use t6
 ; CHECK-RV64-NEXT:    #NO_APP
 ; CHECK-RV64-NEXT:    lui a0, 2
 ; CHECK-RV64-NEXT:    sub sp, s0, a0
-; CHECK-RV64-NEXT:    lui a0, 2
 ; CHECK-RV64-NEXT:    addiw a0, a0, -2032
 ; CHECK-RV64-NEXT:    add sp, sp, a0
 ; CHECK-RV64-NEXT:    ld ra, 2024(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s0, 2016(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s1, 2008(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s2, 2000(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s3, 1992(sp) # 8-byte Folded Reload
 ; CHECK-RV64-NEXT:    ld s4, 1984(sp) # 8-byte Folded Reload

llvm/test/CodeGen/RISCV/out-of-reach-emergency-slot.mir

--- |
   ; CHECK-NEXT:    lui a1, 1
   ; CHECK-NEXT:    addiw a1, a1, -8
   ; CHECK-NEXT:    add a1, sp, a1
   ; CHECK-NEXT:    sd a0, 0(a1)
   ; CHECK-NEXT:    ld a1, 0(sp)
   ; CHECK-NEXT:    call foo@plt
   ; CHECK-NEXT:    lui a0, 2
   ; CHECK-NEXT:    sub sp, s0, a0
-  ; CHECK-NEXT:    lui a0, 2
   ; CHECK-NEXT:    addiw a0, a0, -2032
   ; CHECK-NEXT:    add sp, sp, a0
   ; CHECK-NEXT:    ld ra, 2024(sp) # 8-byte Folded Reload
   ; CHECK-NEXT:    ld s0, 2016(sp) # 8-byte Folded Reload
   ; CHECK-NEXT:    addi sp, sp, 2032
   ; CHECK-NEXT:    ret
   entry:
     ret void

llvm/test/CodeGen/RISCV/rvv/bitreverse-sdnode.ll

 ; RV32-NEXT:    vs8r.v v16, (a0) # Unknown-size Folded Spill
 ; RV32-NEXT:    vlse64.v v16, (a1), zero
 ; RV32-NEXT:    vsll.vi v24, v8, 8
 ; RV32-NEXT:    vand.vv v24, v24, v0
 ; RV32-NEXT:    vsll.vi v8, v8, 24
 ; RV32-NEXT:    vand.vv v8, v8, v16
 ; RV32-NEXT:    vor.vv v8, v8, v24
 ; RV32-NEXT:    vlse64.v v16, (a1), zero
-; RV32-NEXT:    addi a0, sp, 16
 ; RV32-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; RV32-NEXT:    vor.vv v8, v24, v8
 ; RV32-NEXT:    csrr a0, vlenb
 ; RV32-NEXT:    slli a0, a0, 3
 ; RV32-NEXT:    add a0, sp, a0
 ; RV32-NEXT:    addi a0, a0, 16
 ; RV32-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; RV32-NEXT:    vor.vv v8, v8, v24

llvm/test/CodeGen/RISCV/rvv/bswap-sdnode.ll

 ; RV32-NEXT:    addi a0, sp, 16
 ; RV32-NEXT:    vs8r.v v16, (a0) # Unknown-size Folded Spill
 ; RV32-NEXT:    vlse64.v v0, (a1), zero
 ; RV32-NEXT:    vsll.vi v16, v8, 8
 ; RV32-NEXT:    vand.vv v16, v16, v24
 ; RV32-NEXT:    vsll.vi v8, v8, 24
 ; RV32-NEXT:    vand.vv v8, v8, v0
 ; RV32-NEXT:    vor.vv v8, v8, v16
-; RV32-NEXT:    addi a0, sp, 16
 ; RV32-NEXT:    vl8re8.v v16, (a0) # Unknown-size Folded Reload
 ; RV32-NEXT:    vor.vv v8, v16, v8
 ; RV32-NEXT:    csrr a0, vlenb
 ; RV32-NEXT:    slli a0, a0, 3
 ; RV32-NEXT:    add a0, sp, a0
 ; RV32-NEXT:    addi a0, a0, 16
 ; RV32-NEXT:    vl8re8.v v16, (a0) # Unknown-size Folded Reload
 ; RV32-NEXT:    vor.vv v8, v8, v16

llvm/test/CodeGen/RISCV/rvv/ceil-vp.ll

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a2
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a2) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v16, v24, v16, v0.t
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v16, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    bltu a0, a1, .LBB32_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB32_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfabs.v v16, v8, v0.t

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ceil-vp.ll

 ; CHECK-NEXT:    add a0, sp, a0
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    vl8re8.v v16, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfabs.v v24, v16, v0.t
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v1
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vmflt.vf v1, v24, ft0, v0.t
 ; CHECK-NEXT:    fsrmi a0, 3
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a0
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v8, v0.t
 ; CHECK-NEXT:    fsrm a1
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v8, v24, v8, v0.t
 ; CHECK-NEXT:    addi a1, a0, -16
 ; CHECK-NEXT:    sltu a0, a0, a1
 ; CHECK-NEXT:    addi a0, a0, -1
 ; CHECK-NEXT:    and a0, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-floor-vp.ll

 ; CHECK-NEXT:    add a0, sp, a0
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    vl8re8.v v16, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfabs.v v24, v16, v0.t
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v1
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vmflt.vf v1, v24, ft0, v0.t
 ; CHECK-NEXT:    fsrmi a0, 2
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a0
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v8, v0.t
 ; CHECK-NEXT:    fsrm a1
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v8, v24, v8, v0.t
 ; CHECK-NEXT:    addi a1, a0, -16
 ; CHECK-NEXT:    sltu a0, a0, a1
 ; CHECK-NEXT:    addi a0, a0, -1
 ; CHECK-NEXT:    and a0, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-rint-vp.ll

 ; CHECK-NEXT:    vmflt.vf v2, v24, ft0, v0.t
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v8, v0.t
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v8, v24, v8, v0.t
 ; CHECK-NEXT:    addi a1, a0, -16
 ; CHECK-NEXT:    sltu a0, a0, a1
 ; CHECK-NEXT:    addi a0, a0, -1
 ; CHECK-NEXT:    and a0, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1

 ; CHECK-NEXT:    vmflt.vf v2, v24, ft0, v0.t
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v8, v0.t
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v8, v24, v8, v0.t
 ; CHECK-NEXT:    addi a1, a0, -16
 ; CHECK-NEXT:    sltu a0, a0, a1
 ; CHECK-NEXT:    addi a0, a0, -1
 ; CHECK-NEXT:    and a0, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-round-vp.ll

 ; CHECK-NEXT:    add a0, sp, a0
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    vl8re8.v v16, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfabs.v v24, v16, v0.t
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v1
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vmflt.vf v1, v24, ft0, v0.t
 ; CHECK-NEXT:    fsrmi a0, 4
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a0
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v8, v0.t
 ; CHECK-NEXT:    fsrm a1
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v8, v24, v8, v0.t
 ; CHECK-NEXT:    addi a1, a0, -16
 ; CHECK-NEXT:    sltu a0, a0, a1
 ; CHECK-NEXT:    addi a0, a0, -1
 ; CHECK-NEXT:    and a0, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-roundeven-vp.ll

 ; CHECK-NEXT:    add a0, sp, a0
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    vl8re8.v v16, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfabs.v v24, v16, v0.t
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v1
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vmflt.vf v1, v24, ft0, v0.t
 ; CHECK-NEXT:    fsrmi a0, 0
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a0
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v8, v0.t
 ; CHECK-NEXT:    fsrm a1
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v8, v24, v8, v0.t
 ; CHECK-NEXT:    addi a1, a0, -16
 ; CHECK-NEXT:    sltu a0, a0, a1
 ; CHECK-NEXT:    addi a0, a0, -1
 ; CHECK-NEXT:    and a0, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-roundtozero-vp.ll

 ; CHECK-NEXT:    add a0, sp, a0
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    vl8re8.v v16, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfabs.v v24, v16, v0.t
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v1
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vmflt.vf v1, v24, ft0, v0.t
 ; CHECK-NEXT:    fsrmi a0, 1
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a0
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v8, v0.t
 ; CHECK-NEXT:    fsrm a1
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a1) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a1, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a1) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v8, v24, v8, v0.t
 ; CHECK-NEXT:    addi a1, a0, -16
 ; CHECK-NEXT:    sltu a0, a0, a1
 ; CHECK-NEXT:    addi a0, a0, -1
 ; CHECK-NEXT:    and a0, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfwadd.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vfwadd.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfwadd.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x float>, <32 x float>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfwmul.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vfwmul.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfwmul.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x float>, <32 x float>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfwsub.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vfwsub.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfwsub.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x float>, <32 x float>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vpgather.ll

 ; RV64-NEXT:    vs1r.v v0, (a2) # Unknown-size Folded Spill
 ; RV64-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; RV64-NEXT:    vslidedown.vi v16, v8, 16
 ; RV64-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
 ; RV64-NEXT:    vsext.vf2 v0, v16
 ; RV64-NEXT:    vsext.vf2 v24, v8
 ; RV64-NEXT:    vsll.vi v16, v0, 3
 ; RV64-NEXT:    vsll.vi v8, v24, 3
-; RV64-NEXT:    addi a2, sp, 16
 ; RV64-NEXT:    vl1r.v v24, (a2) # Unknown-size Folded Reload
 ; RV64-NEXT:    addi a2, a1, -16
 ; RV64-NEXT:    sltu a3, a1, a2
 ; RV64-NEXT:    addi a3, a3, -1
 ; RV64-NEXT:    and a2, a3, a2
 ; RV64-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; RV64-NEXT:    vslidedown.vi v0, v24, 2
 ; RV64-NEXT:    vsetvli zero, a2, e64, m8, ta, ma

 ; RV64-NEXT:    vs1r.v v0, (a2) # Unknown-size Folded Spill
 ; RV64-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; RV64-NEXT:    vslidedown.vi v16, v8, 16
 ; RV64-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
 ; RV64-NEXT:    vzext.vf2 v0, v16
 ; RV64-NEXT:    vzext.vf2 v24, v8
 ; RV64-NEXT:    vsll.vi v16, v0, 3
 ; RV64-NEXT:    vsll.vi v8, v24, 3
-; RV64-NEXT:    addi a2, sp, 16
 ; RV64-NEXT:    vl1r.v v24, (a2) # Unknown-size Folded Reload
 ; RV64-NEXT:    addi a2, a1, -16
 ; RV64-NEXT:    sltu a3, a1, a2
 ; RV64-NEXT:    addi a3, a3, -1
 ; RV64-NEXT:    and a2, a3, a2
 ; RV64-NEXT:    vsetivli zero, 2, e8, mf4, ta, ma
 ; RV64-NEXT:    vslidedown.vi v0, v24, 2
 ; RV64-NEXT:    vsetvli zero, a2, e64, m8, ta, ma

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vwadd.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vwadd.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vwadd.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x i32>, <32 x i32>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vwaddu.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vwaddu.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vwaddu.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x i32>, <32 x i32>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vwmul.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vwmul.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vwmul.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x i32>, <32 x i32>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vwmulsu.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vwmulsu.vv v8, v24, v16
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vwmulsu.vv v16, v0, v24
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x i32>, <32 x i32>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vwmulu.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vwmulu.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vwmulu.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x i32>, <32 x i32>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vwsub.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vwsub.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vwsub.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x i32>, <32 x i32>* %x

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vwsubu.ll

 ; CHECK-NEXT:    vle32.v v24, (a1)
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m8, ta, ma
 ; CHECK-NEXT:    vslidedown.vi v8, v16, 16
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vslidedown.vi v0, v24, 16
 ; CHECK-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; CHECK-NEXT:    vwsubu.vv v8, v16, v24
-; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a0) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vwsubu.vv v16, v24, v0
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   %a = load <32 x i32>, <32 x i32>* %x

llvm/test/CodeGen/RISCV/rvv/floor-vp.ll

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a2
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a2) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v16, v24, v16, v0.t
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v16, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    bltu a0, a1, .LBB32_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB32_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfabs.v v16, v8, v0.t

llvm/test/CodeGen/RISCV/rvv/rint-vp.ll

 ; CHECK-NEXT:    vmflt.vf v2, v24, ft0, v0.t
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a2) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v16, v24, v16, v0.t
 ; CHECK-NEXT:    bltu a0, a1, .LBB32_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB32_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1

llvm/test/CodeGen/RISCV/rvv/round-vp.ll

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a2
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a2) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v16, v24, v16, v0.t
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v16, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    bltu a0, a1, .LBB32_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB32_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfabs.v v16, v8, v0.t

llvm/test/CodeGen/RISCV/rvv/roundeven-vp.ll

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a2
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a2) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v16, v24, v16, v0.t
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v16, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    bltu a0, a1, .LBB32_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB32_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfabs.v v16, v8, v0.t

llvm/test/CodeGen/RISCV/rvv/roundtozero-vp.ll

 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v2
 ; CHECK-NEXT:    vfcvt.x.f.v v24, v16, v0.t
 ; CHECK-NEXT:    fsrm a2
 ; CHECK-NEXT:    vfcvt.f.x.v v24, v24, v0.t
 ; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v24, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vl8re8.v v24, (a2) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vfsgnj.vv v16, v24, v16, v0.t
-; CHECK-NEXT:    addi a2, sp, 16
 ; CHECK-NEXT:    vs8r.v v16, (a2) # Unknown-size Folded Spill
 ; CHECK-NEXT:    bltu a0, a1, .LBB32_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB32_2:
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, ma
 ; CHECK-NEXT:    vmv1r.v v0, v1
 ; CHECK-NEXT:    vfabs.v v16, v8, v0.t

llvm/test/CodeGen/RISCV/rvv/rv32-spill-vector.ll

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs1r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs1r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs2r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl2re8.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 2
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs4r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl4re8.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 2
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 3
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl8re8.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 3
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",
   "~{v0},~{v1},~{v2},~{v3},~{v4},~{v5},~{v6},~{v7},~{v8},~{v9},~{v10},~{v11},~{v12},~{v13},~{v14},~{v15},~{v16},~{v17},~{v18},~{v19},~{v20},~{v21},~{v22},~{v23},~{v24},~{v25},~{v26},~{v27},~{v28},~{v29},~{v30},~{v31}"()
 
   ret <vscale x 16 x i32> %va
 }

llvm/test/CodeGen/RISCV/rvv/rv32-spill-zvlsseg.ll

 ; SPILL-O2-NEXT:    vlseg2e32.v v8, (a0)
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    vs1r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs1r.v v9, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
-; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    vl1r.v v7, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16

 ; SPILL-O2-NEXT:    vlseg2e32.v v8, (a0)
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    vs1r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs1r.v v9, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
-; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    vl1r.v v7, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16

 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    vs2r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs2r.v v10, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    vl2r.v v6, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 2

 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 2
 ; SPILL-O2-NEXT:    vs4r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs4r.v v12, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 2
 ; SPILL-O2-NEXT:    vl4r.v v4, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl4r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 3

 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    vs2r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs2r.v v10, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs2r.v v12, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    vl2r.v v6, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v10, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2

llvm/test/CodeGen/RISCV/rvv/rv64-spill-vector.ll

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs1r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs2r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl2re8.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 2
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs4r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl4re8.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 2
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",

 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 3
 ; SPILL-O2-NEXT:    sub sp, sp, a0
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vs8r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    vl8re8.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 3
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 entry:
   call void asm sideeffect "",
   "~{v0},~{v1},~{v2},~{v3},~{v4},~{v5},~{v6},~{v7},~{v8},~{v9},~{v10},~{v11},~{v12},~{v13},~{v14},~{v15},~{v16},~{v17},~{v18},~{v19},~{v20},~{v21},~{v22},~{v23},~{v24},~{v25},~{v26},~{v27},~{v28},~{v29},~{v30},~{v31}"()
 
   ret <vscale x 8 x i64> %va
 }

llvm/test/CodeGen/RISCV/rvv/rv64-spill-zvlsseg.ll

 ; SPILL-O2-NEXT:    vlseg2e32.v v8, (a0)
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    vs1r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs1r.v v9, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
-; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    vl1r.v v7, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16

 ; SPILL-O2-NEXT:    vlseg2e32.v v8, (a0)
 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    vs1r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs1r.v v9, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
-; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    vl1r.v v7, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl1r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8 killed $v8 killed $v7_v8
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    addi sp, sp, 16

 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    vs2r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs2r.v v10, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    vl2r.v v6, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 2

 ; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 2
 ; SPILL-O2-NEXT:    vs4r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs4r.v v12, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 2
 ; SPILL-O2-NEXT:    vl4r.v v4, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl4r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8m4 killed $v8m4 killed $v4m4_v8m4
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 3

 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    vs2r.v v8, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs2r.v v10, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vs2r.v v12, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    #APP
 ; SPILL-O2-NEXT:    #NO_APP
-; SPILL-O2-NEXT:    addi a0, sp, 16
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    vl2r.v v6, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v8, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    add a0, a0, a1
 ; SPILL-O2-NEXT:    vl2r.v v10, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    # kill: def $v8m2 killed $v8m2 killed $v6m2_v8m2_v10m2

llvm/test/CodeGen/RISCV/rvv/zvlsseg-spill.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-# RUN: llc -march=riscv64 -mattr=+v -stop-after=prologepilog %s -o - 2>&1 | FileCheck %s
+# RUN: llc -march=riscv64 -mattr=+v -stop-after=machine-latecleanup %s -o - 2>&1 | FileCheck %s
 
 --- |
   target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n64-S128"
   target triple = "riscv64"
 
   define void @zvlsseg_spill(i64 *%base, i64 %vl) {
     ret void
   }
bb.0:
     ; CHECK-NEXT: frame-setup CFI_INSTRUCTION def_cfa_offset 16
     ; CHECK-NEXT: $x12 = frame-setup PseudoReadVLENB
     ; CHECK-NEXT: $x12 = frame-setup SLLI killed $x12, 3
     ; CHECK-NEXT: $x2 = frame-setup SUB $x2, killed $x12
     ; CHECK-NEXT: dead $x0 = PseudoVSETVLI killed renamable $x11, 216 /* e64, m1, ta, ma */, implicit-def $vl, implicit-def $vtype
     ; CHECK-NEXT: $v0_v1_v2_v3_v4_v5_v6 = PseudoVLSEG7E64_V_M1 renamable $x10, $noreg, 6 /* e64 */, implicit $vl, implicit $vtype
     ; CHECK-NEXT: $x11 = ADDI $x2, 16
     ; CHECK-NEXT: $x12 = PseudoReadVLENB
-    ; CHECK-NEXT: PseudoVSPILL7_M1 killed renamable $v0_v1_v2_v3_v4_v5_v6, killed $x11, killed $x12
-    ; CHECK-NEXT: $x11 = ADDI $x2, 16
-    ; CHECK-NEXT: $x12 = PseudoReadVLENB
+    ; CHECK-NEXT: PseudoVSPILL7_M1 killed renamable $v0_v1_v2_v3_v4_v5_v6, $x11, $x12
     ; CHECK-NEXT: dead renamable $v7_v8_v9_v10_v11_v12_v13 = PseudoVRELOAD7_M1 killed $x11, killed $x12, implicit-def $v8
     ; CHECK-NEXT: VS1R_V killed $v8, killed renamable $x10
     ; CHECK-NEXT: $x10 = frame-destroy PseudoReadVLENB
     ; CHECK-NEXT: $x10 = frame-destroy SLLI killed $x10, 3
     ; CHECK-NEXT: $x2 = frame-destroy ADD $x2, killed $x10
     ; CHECK-NEXT: $x2 = frame-destroy ADDI $x2, 16
     ; CHECK-NEXT: PseudoRET
     %0:gpr = COPY $x10
     %1:gprnox0 = COPY $x11
     $v0_v1_v2_v3_v4_v5_v6 = PseudoVLSEG7E64_V_M1 %0, %1, 6
     PseudoVSPILL7_M1 killed renamable $v0_v1_v2_v3_v4_v5_v6, %stack.0, $x0
     renamable $v7_v8_v9_v10_v11_v12_v13 = PseudoVRELOAD7_M1 %stack.0, $x0
     VS1R_V killed $v8, %0:gpr
     PseudoRET
 ...

llvm/test/CodeGen/RISCV/stack-realignment.ll

 ; RV32I-NEXT:    sub sp, sp, a0
 ; RV32I-NEXT:    srli a0, sp, 12
 ; RV32I-NEXT:    slli sp, a0, 12
 ; RV32I-NEXT:    lui a0, 1
 ; RV32I-NEXT:    add a0, sp, a0
 ; RV32I-NEXT:    call callee@plt
 ; RV32I-NEXT:    lui a0, 2
 ; RV32I-NEXT:    sub sp, s0, a0
-; RV32I-NEXT:    lui a0, 2
 ; RV32I-NEXT:    addi a0, a0, -2032
 ; RV32I-NEXT:    add sp, sp, a0
 ; RV32I-NEXT:    lw ra, 2028(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    lw s0, 2024(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 2032
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: caller4096:

 ; RV64I-NEXT:    sub sp, sp, a0
 ; RV64I-NEXT:    srli a0, sp, 12
 ; RV64I-NEXT:    slli sp, a0, 12
 ; RV64I-NEXT:    lui a0, 1
 ; RV64I-NEXT:    add a0, sp, a0
 ; RV64I-NEXT:    call callee@plt
 ; RV64I-NEXT:    lui a0, 2
 ; RV64I-NEXT:    sub sp, s0, a0
-; RV64I-NEXT:    lui a0, 2
 ; RV64I-NEXT:    addiw a0, a0, -2032
 ; RV64I-NEXT:    add sp, sp, a0
 ; RV64I-NEXT:    ld ra, 2024(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    ld s0, 2016(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 2032
 ; RV64I-NEXT:    ret
   %1 = alloca i8, align 4096
   call void @callee(i8* %1)

llvm/test/CodeGen/SystemZ/frame-28.mir

+# RUN: llc -mtriple=s390x-linux-gnu -start-before=prologepilog %s -o - -mcpu=z14 \
+# RUN:   -verify-machineinstrs 2>&1 | FileCheck %s
+# REQUIRES: asserts
+#
+# Test that redundant frame addressing anchor points are removed by PEI.
+
+--- |
+  define void @fun1() { ret void }
+  define void @fun2() { ret void }
+  define void @fun3() { ret void }
+  define void @fun4() { ret void }
+  define void @fun5() { ret void }
+  define void @fun6() { ret void }
+  define void @fun7() { ret void }
+  define void @fun8() { ret void }
+
+  declare i32 @foo()
+
+  @ptr = external dso_local local_unnamed_addr global ptr
+---
+
+# Test elimination of redundant LAYs in successor blocks.
+# CHECK-LABEL: fun1:
+# CHECK: lay %r1, 4096(%r15)
+# CHECK: # %bb.1:
+# CHECK-NOT: lay
+# CHECK: .LBB0_2:
+# CHECK-NOT: lay
+---
+name:            fun1
+tracksRegLiveness: true
+stack:
+  - { id: 0, size: 5000 }
+  - { id: 1, size: 2500 }
+  - { id: 2, size: 2500 }
+
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $f16d
+    successors: %bb.2(0x00000001), %bb.1(0x7fffffff)
+
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.1, 0, $noreg
+    CHIMux undef $r0l, 3, implicit-def $cc
+    BRC 14, 8, %bb.2, implicit killed $cc
+    J %bb.1
+
+  bb.1:
+    liveins: $f16d
+    VST64 renamable $f16d, %stack.2, 0, $noreg
+    J %bb.2
+
+  bb.2:
+    liveins: $f16d
+    VST64 renamable $f16d, %stack.1, 0, $noreg
+    Return
+...
+
+# In this function the LAY in bb.1 will have a different offset, so the first
+# LAY in bb.2 must remain.
+# CHECK-LABEL: fun2:
+# CHECK: lay %r1, 4096(%r15)
+# CHECK: # %bb.1:
+# CHECK: lay %r1, 8192(%r15)
+# CHECK: .LBB1_2:
+# CHECK: lay %r1, 4096(%r15)
+# CHECK-NOT: lay
+---
+name:            fun2
+tracksRegLiveness: true
+stack:
+  - { id: 0, size: 5000 }
+  - { id: 1, size: 5000 }
+  - { id: 2, size: 2500 }
+
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $f16d
+    successors: %bb.2(0x00000001), %bb.1(0x7fffffff)
+
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.1, 0, $noreg
+    CHIMux undef $r0l, 3, implicit-def $cc
+    BRC 14, 8, %bb.2, implicit killed $cc
+    J %bb.1
+
+  bb.1:
+    liveins: $f16d
+    VST64 renamable $f16d, %stack.2, 0, $noreg
+    J %bb.2
+
+  bb.2:
+    liveins: $f16d
+    VST64 renamable $f16d, %stack.1, 0, $noreg
+    VST64 renamable $f16d, %stack.1, 0, $noreg
+    Return
+...
+
+# Test case with a loop (with room for improvement: since %r1 is not clobbered
+# inside the loop only the first LAY is needed).
+# CHECK-LABEL: fun3:
+# CHECK: lay %r1, 4096(%r15)
+# CHECK: .LBB2_1:
+# CHECK: lay %r1, 4096(%r15)
+# CHECK: .LBB2_2:
+# CHECK: lay %r1, 4096(%r15)
+---
+name:            fun3
+tracksRegLiveness: true
+stack:
+  - { id: 0, size: 5000 }
+  - { id: 1, size: 2500 }
+  - { id: 2, size: 2500 }
+
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $f16d
+    successors: %bb.2(0x00000001), %bb.1(0x7fffffff)
+
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.1, 0, $noreg
+    CHIMux undef $r0l, 3, implicit-def $cc
+    BRC 14, 8, %bb.2, implicit killed $cc
+    J %bb.1
+
+  bb.1:
+    liveins: $f16d
+    successors: %bb.2(0x00000001), %bb.1(0x7fffffff)
+
+    VST64 renamable $f16d, %stack.2, 0, $noreg
+    CHIMux undef $r0l, 3, implicit-def $cc
+    BRC 14, 8, %bb.1, implicit killed $cc
+    J %bb.2
+
+  bb.2:
+    liveins: $f16d
+    VST64 renamable $f16d, %stack.1, 0, $noreg
+    Return
+...
+
+# Test case with a call which clobbers r1: the second LAY after the call is needed.
+# CHECK-LABEL: fun4:
+# CHECK: lay %r1, 4096(%r15)
+# CHECK: brasl
+# CHECK: lay %r1, 4096(%r15)
+---
+name:            fun4
+tracksRegLiveness: true
+stack:
+  - { id: 0, size: 5000 }
+  - { id: 1, size: 2500 }
+
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $f16d
+
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.1, 0, $noreg
+    ADJCALLSTACKDOWN 0, 0
+    CallBRASL @foo, csr_systemz_elf, implicit-def dead $r14d, implicit-def dead $cc, implicit $fpc, implicit-def $r2l
+    ADJCALLSTACKUP 0, 0
+    $f17d = IMPLICIT_DEF
+    VST64 renamable $f17d, %stack.1, 0, $noreg
+    Return
+...
+
+# Test case where index reg is loaded instead of using an LAY. Only one LGHI is needed.
+# CHECK-LABEL: fun5:
+# CHECK: lghi %r1, 4096
+# CHECK-NOT: lghi
+---
+name:            fun5
+tracksRegLiveness: true
+stack:
+  - { id: 0, size: 5000 }
+  - { id: 1, size: 2500 }
+
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $f16d
+
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    VST64 renamable $f16d, %stack.0, 0, $noreg
+    $f0q = nofpexcept LXEB %stack.1, 0, $noreg, implicit $fpc
+    $f1q = nofpexcept LXEB %stack.1, 0, $noreg, implicit $fpc
+    Return
+...
+
+# Test where the constant is a Global. Only one LARL is needed.
+# CHECK-LABEL: fun6:
+# CHECK: larl %r1, ptr
+# CHECK-NOT: larl
+---
+name:            fun6
+alignment:       16
+tracksRegLiveness: true
+tracksDebugUserValues: true
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+fixedStack:
+  - { id: 0, offset: -160, size: 8, alignment: 8 }
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    successors: %bb.2(0x30000000), %bb.1(0x50000000)
+
+    renamable $r1d = LARL @ptr
+    CGHSI killed renamable $r1d, 0, 0, implicit-def $cc :: (volatile dereferenceable load (s64) from @ptr)
+    BRC 14, 8, %bb.2, implicit killed $cc
+    J %bb.1
+
+  bb.1:
+    renamable $r1d = LARL @ptr
+    MVGHI killed renamable $r1d, 0, 0
+
+  bb.2:
+    Return
+
+...
+
+# Load of an invariant location (GOT). Only one LGRL is needed.
+# CHECK-LABEL: fun7:
+# CHECK: lgrl %r1, ptr
+# CHECK-NOT: lgrl
+---
+name:            fun7
+alignment:       16
+tracksRegLiveness: true
+tracksDebugUserValues: true
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+fixedStack:
+  - { id: 0, offset: -160, size: 8, alignment: 8 }
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    successors: %bb.2(0x30000000), %bb.1(0x50000000)
+  
+    renamable $r1d = LGRL @ptr :: (load (s64) from got)
+    CGHSI killed renamable $r1d, 0, 0, implicit-def $cc :: (volatile dereferenceable load (s64) from @ptr)
+    BRC 14, 8, %bb.2, implicit killed $cc
+    J %bb.1
+  
+  bb.1:
+    renamable $r1d = LGRL @ptr :: (load (s64) from got)
+    MVGHI killed renamable $r1d, 0, 0
+  
+  bb.2:
+    Return
+
+...
+
+# Load from constant pool. Only one LARL is needed.
+# CHECK-LABEL: fun8:
+# CHECK: larl %r1, .LCPI7_0
+# CHECK-NOT: larl
+---
+name:            fun8
+alignment:       16
+tracksRegLiveness: true
+tracksDebugUserValues: true
+liveins:
+  - { reg: '$f0s' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+fixedStack:
+  - { id: 0, offset: -160, size: 8, alignment: 8 }
+constants:
+  - id:              0
+    value:           float 0x43E0000000000000
+    alignment:       4
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    successors: %bb.1, %bb.2
+    liveins: $f0s
+
+    renamable $r1d = LARL %const.0
+    renamable $f1s = LE killed renamable $r1d, 0, $noreg :: (load (s32) from constant-pool)
+    nofpexcept CEBR renamable $f0s, renamable $f1s, implicit-def $cc, implicit $fpc
+    BRC 15, 11, %bb.2, implicit killed $cc
+
+  bb.1:
+    liveins: $f0s
+
+    J %bb.3
+
+  bb.2 (%ir-block.0):
+    liveins: $f0s, $f1s
+
+    renamable $r1d = LARL %const.0
+    renamable $f1s = LE killed renamable $r1d, 0, $noreg :: (load (s32) from constant-pool)
+
+  bb.3 (%ir-block.0):
+    liveins: $r2d
+
+    Return
+
+...

llvm/test/CodeGen/Thumb/frame-access.ll

 ; CHECK:       sub sp, #28
 ; Incoming arguments area is accessed via SP if FP is not available
 ; CHECK-NOFP:  add r0, sp, #36
 ; CHECK-NOFP:  stm r0!, {r1, r2, r3}
 ; CHECK-FP-ATPCS: mov r0, r7
 ; CHECK-FP-ATPCS: adds r0, #8
 ; CHECK-FP-ATPCS: stm r0!, {r1, r2, r3}
 ; CHECK-FP-AAPCS: mov r0, r11
-; CHECK-FP-AAPCS: str r1, [r0, #8]
-; CHECK-FP-AAPCS: mov r0, r11
-; CHECK-FP-AAPCS: str r2, [r0, #12]
-; CHECK-FP-AAPCS: mov r0, r11
-; CHECK-FP-AAPCS: str r3, [r0, #16]
-
+; CHECK-FP-AAPCS: mov r7, r0
+; CHECK-FP-AAPCS: adds r7, #8
+; CHECK-FP-AAPCS: stm r7!, {r1, r2, r3}
 ; Re-aligned stack, access via FP
 ; int test_args_realign(int a, int b, int c, int d, int e) {
 ;   __attribute__((aligned(16))) int v[4];
 ;   return g(v, a, b, c, d, e);
 ; }
 ; Function Attrs: nounwind
 define dso_local i32 @test_args_realign(i32 %a, i32 %b, i32 %c, i32 %d, i32 %e) local_unnamed_addr  {
 entry:

 ; CHECK-NEXT:  lsrs r4, r4, #4
 ; CHECK-NEXT:  lsls r4, r4, #4
 ; CHECK-NEXT:  mov  sp, r4
 ; Incoming register varargs stored via FP
 ; CHECK-ATPCS: mov r0, r7
 ; CHECK-ATPCS-NEXT: adds r0, #8
 ; CHECK-ATPCS-NEXT: stm r0!, {r1, r2, r3}
 ; CHECK-AAPCS: mov r0, r11
-; CHECK-AAPCS: str r1, [r0, #8]
-; CHECK-AAPCS: mov r0, r11
-; CHECK-AAPCS: str r2, [r0, #12]
-; CHECK-AAPCS: mov r0, r11
-; CHECK-AAPCS: str r3, [r0, #16]
+; CHECK-AAPCS: mov r7, r0
+; CHECK-AAPCS: adds r7, #8
+; CHECK-AAPCS: stm r7!, {r1, r2, r3}
 ; VLAs present, access via FP
 ; int test_args_vla(int a, int b, int c, int d, int e) {
 ;   int v[a];
 ;   return g(v, a, b, c, d, e);
 ; }
 define dso_local i32 @test_args_vla(i32 %a, i32 %b, i32 %c, i32 %d, i32 %e) local_unnamed_addr  {
 entry:
   %vla = alloca i32, i32 %a, align 4

 ; CHECK-NOFP:      mov r0, r6
 ; CHECK-NOFP-NEXT: adds r0, #36
 ; CHECK-NOFP-NEXT: stm r0!, {r1, r2, r3}
 ; Incoming varargs stored via FP otherwise
 ; CHECK-FP-ATPCS:      mov r0, r7
 ; CHECK-FP-ATPCS-NEXT: adds r0, #8
 ; CHECK-FP-ATPCS-NEXT: stm r0!, {r1, r2, r3}
 ; CHECK-FP-AAPCS:      mov r0, r11
-; CHECK-FP-AAPCS-NEXT: str r1, [r0, #8]
-; CHECK-FP-AAPCS-NEXT: mov r0, r11
-; CHECK-FP-AAPCS-NEXT: str r2, [r0, #12]
-; CHECK-FP-AAPCS-NEXT: mov r0, r11
-; CHECK-FP-AAPCS-NEXT: str r3, [r0, #16]
+; CHECK-FP-AAPCS-NEXT: mov r5, r0
+; CHECK-FP-AAPCS-NEXT: adds r5, #8
+; CHECK-FP-AAPCS-NEXT: stm r5!, {r1, r2, r3}
 
 ; struct S { int x[128]; } s;
 ; int test(S a, int b) {
 ;   return i(b);
 ; }
 define dso_local i32 @test_args_large_offset(%struct.S* byval(%struct.S) align 4 %0, i32 %1) local_unnamed_addr {
   %3 = alloca i32, align 4
   store i32 %1, i32* %3, align 4

llvm/test/CodeGen/Thumb2/mve-fpclamptosat_vec.ll

 ; CHECK-NEXT:    cmp.w r8, #-2147483648
 ; CHECK-NEXT:    mov.w r8, #0
 ; CHECK-NEXT:    ldr r2, [sp, #16] @ 4-byte Reload
 ; CHECK-NEXT:    csel r6, r5, r8, hi
 ; CHECK-NEXT:    csel r6, r5, r6, eq
 ; CHECK-NEXT:    cmp.w r2, #-1
 ; CHECK-NEXT:    ldr r2, [sp, #8] @ 4-byte Reload
 ; CHECK-NEXT:    csel r5, r5, r8, gt
-; CHECK-NEXT:    mov.w r8, #0
 ; CHECK-NEXT:    cmp r2, #0
 ; CHECK-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    csel r5, r6, r5, eq
 ; CHECK-NEXT:    cmp r1, r11
 ; CHECK-NEXT:    csel r1, lr, r3, lo
 ; CHECK-NEXT:    csel r1, lr, r1, eq
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    csel r0, lr, r3, mi

 ; CHECK-NEXT:    cmp.w r8, #-2147483648
 ; CHECK-NEXT:    mov.w r8, #0
 ; CHECK-NEXT:    ldr r2, [sp, #20] @ 4-byte Reload
 ; CHECK-NEXT:    csel r6, r5, r8, hi
 ; CHECK-NEXT:    csel r6, r5, r6, eq
 ; CHECK-NEXT:    cmp.w r2, #-1
 ; CHECK-NEXT:    ldr r2, [sp, #12] @ 4-byte Reload
 ; CHECK-NEXT:    csel r5, r5, r8, gt
-; CHECK-NEXT:    mov.w r8, #0
 ; CHECK-NEXT:    cmp r2, #0
 ; CHECK-NEXT:    ldr r2, [sp, #8] @ 4-byte Reload
 ; CHECK-NEXT:    csel r5, r6, r5, eq
 ; CHECK-NEXT:    cmp r1, r10
 ; CHECK-NEXT:    csel r1, lr, r3, lo
 ; CHECK-NEXT:    csel r1, lr, r1, eq
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    csel r0, lr, r3, mi

 ; CHECK-NEXT:    cmp.w r8, #-2147483648
 ; CHECK-NEXT:    mov.w r8, #0
 ; CHECK-NEXT:    ldr r2, [sp, #16] @ 4-byte Reload
 ; CHECK-NEXT:    csel r6, r5, r8, hi
 ; CHECK-NEXT:    csel r6, r5, r6, eq
 ; CHECK-NEXT:    cmp.w r2, #-1
 ; CHECK-NEXT:    ldr r2, [sp, #8] @ 4-byte Reload
 ; CHECK-NEXT:    csel r5, r5, r8, gt
-; CHECK-NEXT:    mov.w r8, #0
 ; CHECK-NEXT:    cmp r2, #0
 ; CHECK-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    csel r5, r6, r5, eq
 ; CHECK-NEXT:    cmp r1, r11
 ; CHECK-NEXT:    csel r1, lr, r3, lo
 ; CHECK-NEXT:    csel r1, lr, r1, eq
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    csel r0, lr, r3, mi

llvm/test/CodeGen/X86/2008-04-09-BranchFolding.ll

 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movb $1, %al
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    jne .LBB0_1
 ; CHECK-NEXT:  # %bb.2: # %bb17.i
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    retl
 ; CHECK-NEXT:  .LBB0_1: # %bb160
-; CHECK-NEXT:    movb $1, %al
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    retl
 entry:
   %tmp3.i40 = icmp eq ptr null, null    ; <i1> [#uses=2]
   br label %bb140
 bb140:    ; preds = %entry
   br i1 %tmp3.i40, label %bb160, label %bb17.i

llvm/test/CodeGen/X86/2008-04-16-ReMatBug.ll

 ; CHECK-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    movzwl {{[0-9]+}}(%esp), %ebx
 ; CHECK-NEXT:    movzwl {{[0-9]+}}(%esp), %ebp
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; CHECK-NEXT:    movw $-2, %si
 ; CHECK-NEXT:    jne LBB0_6
 ; CHECK-NEXT:  ## %bb.4: ## %bb37
 ; CHECK-NEXT:    movw $0, 40(%edi)
-; CHECK-NEXT:    movb $1, %al
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    leal (,%ecx,4), %ecx
 ; CHECK-NEXT:    leal (,%ebx,4), %edx
 ; CHECK-NEXT:    subl $12, %esp
 ; CHECK-NEXT:    movzwl %bp, %eax
 ; CHECK-NEXT:    movswl %cx, %ecx
 ; CHECK-NEXT:    movswl %dx, %edx
 ; CHECK-NEXT:    pushl $87

llvm/test/CodeGen/X86/AMX/amx-across-func.ll

 ; CHECK-NEXT:    tilestored %tmm2, 64(%rsp,%rax) # 1024-byte Folded Spill
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    callq foo
 ; CHECK-NEXT:    ldtilecfg (%rsp)
 ; CHECK-NEXT:    movl $buf+2048, %eax
 ; CHECK-NEXT:    tileloadd (%rax,%r14), %tmm0
 ; CHECK-NEXT:    movabsq $64, %rcx
 ; CHECK-NEXT:    tileloadd 1088(%rsp,%rcx), %tmm1 # 1024-byte Folded Reload
-; CHECK-NEXT:    movabsq $64, %rcx
 ; CHECK-NEXT:    tileloadd 64(%rsp,%rcx), %tmm2 # 1024-byte Folded Reload
 ; CHECK-NEXT:    tdpbssd %tmm2, %tmm1, %tmm0
 ; CHECK-NEXT:    tilestored %tmm0, (%rax,%r14)
 ; CHECK-NEXT:    addq $2120, %rsp # imm = 0x848
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %r14
 ; CHECK-NEXT:    popq %r15
 ; CHECK-NEXT:    popq %rbp

llvm/test/CodeGen/X86/AMX/amx-spill-merge.ll

 ; CHECK-NEXT:    movl $buf, %eax
 ; CHECK-NEXT:    movw $8, %cx
 ; CHECK-NEXT:    tileloadd (%rax,%r14), %tmm0
 ; CHECK-NEXT:    movl $buf+1024, %eax
 ; CHECK-NEXT:    tileloadd (%rax,%r14), %tmm1
 ; CHECK-NEXT:    movabsq $64, %rax
 ; CHECK-NEXT:    tilestored %tmm5, 1088(%rsp,%rax) # 1024-byte Folded Spill
 ; CHECK-NEXT:    tdpbssd %tmm1, %tmm0, %tmm5
-; CHECK-NEXT:    movabsq $64, %rax
 ; CHECK-NEXT:    tilestored %tmm5, 64(%rsp,%rax) # 1024-byte Folded Spill
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    callq foo
 ; CHECK-NEXT:    ldtilecfg (%rsp)
 ; CHECK-NEXT:    movabsq $64, %rax
 ; CHECK-NEXT:    tileloadd 64(%rsp,%rax), %tmm6 # 1024-byte Folded Reload
 ; CHECK-NEXT:    jmp .LBB0_3
 ; CHECK-NEXT:  .LBB0_2: # %if.false
 ; CHECK-NEXT:    movl $buf, %eax
 ; CHECK-NEXT:    movw $8, %cx
 ; CHECK-NEXT:    tileloadd (%rax,%r14), %tmm2
 ; CHECK-NEXT:    movl $buf+1024, %eax
 ; CHECK-NEXT:    tileloadd (%rax,%r14), %tmm3
 ; CHECK-NEXT:    movabsq $64, %rax
 ; CHECK-NEXT:    tilestored %tmm5, 1088(%rsp,%rax) # 1024-byte Folded Spill
 ; CHECK-NEXT:    tdpbssd %tmm3, %tmm2, %tmm5
-; CHECK-NEXT:    movabsq $64, %rax
 ; CHECK-NEXT:    tilestored %tmm5, 64(%rsp,%rax) # 1024-byte Folded Spill
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    callq foo
 ; CHECK-NEXT:    ldtilecfg (%rsp)
 ; CHECK-NEXT:    movabsq $64, %rax
 ; CHECK-NEXT:    tileloadd 64(%rsp,%rax), %tmm6 # 1024-byte Folded Reload
 ; CHECK-NEXT:    tilestored %tmm6, (%r15,%r14)

llvm/test/CodeGen/X86/fast-isel-stackcheck.ll

 ; RUN: llc -o - %s | FileCheck %s
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx"
 
 ; selectiondag stack protector uses a GuardReg which the fast-isel stack
 ; protection code did not but the state was not reset properly.
 ; The optnone attribute on @bar forces fast-isel.
 
 ; CHECK-LABEL: foo:
 ; CHECK: movq ___stack_chk_guard@GOTPCREL(%rip), %rax
-; CHECK: movq ___stack_chk_guard@GOTPCREL(%rip), %rax
 define void @foo() #0 {
 entry:
   %_tags = alloca [3 x i32], align 4
   ret void
 }
 
 ; CHECK-LABEL: bar:
 ; CHECK: movq ___stack_chk_guard@GOTPCREL(%rip), %{{r.x}}

llvm/test/CodeGen/X86/fshl.ll

 ; X86-SLOW-NEXT:    jne .LBB6_1
 ; X86-SLOW-NEXT:  # %bb.2:
 ; X86-SLOW-NEXT:    movl %ebp, %ecx
 ; X86-SLOW-NEXT:    movl %edi, %ebp
 ; X86-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X86-SLOW-NEXT:    movl %edx, %ebx
 ; X86-SLOW-NEXT:    movl %esi, %edx
 ; X86-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %esi
-; X86-SLOW-NEXT:    jmp .LBB6_3
+; X86-SLOW-NEXT:    testb $32, %al
+; X86-SLOW-NEXT:    je .LBB6_5
+; X86-SLOW-NEXT:  .LBB6_4:
+; X86-SLOW-NEXT:    movl %esi, (%esp) # 4-byte Spill
+; X86-SLOW-NEXT:    movl %ebp, %esi
+; X86-SLOW-NEXT:    movl %edx, %ebp
+; X86-SLOW-NEXT:    movl %ecx, %edx
+; X86-SLOW-NEXT:    jmp .LBB6_6
 ; X86-SLOW-NEXT:  .LBB6_1:
 ; X86-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X86-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-SLOW-NEXT:  .LBB6_3:
-; X86-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-SLOW-NEXT:    testb $32, %al
 ; X86-SLOW-NEXT:    jne .LBB6_4
-; X86-SLOW-NEXT:  # %bb.5:
+; X86-SLOW-NEXT:  .LBB6_5:
 ; X86-SLOW-NEXT:    movl %ecx, %ebx
 ; X86-SLOW-NEXT:    movl %edi, (%esp) # 4-byte Spill
-; X86-SLOW-NEXT:    jmp .LBB6_6
-; X86-SLOW-NEXT:  .LBB6_4:
-; X86-SLOW-NEXT:    movl %esi, (%esp) # 4-byte Spill
-; X86-SLOW-NEXT:    movl %ebp, %esi
-; X86-SLOW-NEXT:    movl %edx, %ebp
-; X86-SLOW-NEXT:    movl %ecx, %edx
 ; X86-SLOW-NEXT:  .LBB6_6:
 ; X86-SLOW-NEXT:    movl %edx, %edi
-; X86-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-SLOW-NEXT:    movl %eax, %ecx
 ; X86-SLOW-NEXT:    shll %cl, %edi
 ; X86-SLOW-NEXT:    shrl %ebx
 ; X86-SLOW-NEXT:    movb %al, %ch
 ; X86-SLOW-NEXT:    notb %ch
 ; X86-SLOW-NEXT:    movb %ch, %cl
 ; X86-SLOW-NEXT:    shrl %cl, %ebx
 ; X86-SLOW-NEXT:    orl %edi, %ebx

llvm/test/CodeGen/X86/masked_load.ll

 ; SSE-NEXT:    jne LBB3_5
 ; SSE-NEXT:  LBB3_6: ## %else5
 ; SSE-NEXT:    testb $8, %al
 ; SSE-NEXT:    jne LBB3_7
 ; SSE-NEXT:  LBB3_8: ## %else8
 ; SSE-NEXT:    retq
 ; SSE-NEXT:  LBB3_1: ## %cond.load
 ; SSE-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
-; SSE-NEXT:    xorps %xmm1, %xmm1
 ; SSE-NEXT:    testb $2, %al
 ; SSE-NEXT:    je LBB3_4
 ; SSE-NEXT:  LBB3_3: ## %cond.load1
 ; SSE-NEXT:    movhps {{.*#+}} xmm0 = xmm0[0,1],mem[0,1]
 ; SSE-NEXT:    testb $4, %al
 ; SSE-NEXT:    je LBB3_6
 ; SSE-NEXT:  LBB3_5: ## %cond.load4
 ; SSE-NEXT:    movlps {{.*#+}} xmm1 = mem[0,1],xmm1[2,3]

 ; SSE2-NEXT:    jne LBB10_13
 ; SSE2-NEXT:  LBB10_14: ## %else17
 ; SSE2-NEXT:    testb $-128, %al
 ; SSE2-NEXT:    jne LBB10_15
 ; SSE2-NEXT:  LBB10_16: ## %else20
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  LBB10_1: ## %cond.load
 ; SSE2-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE2-NEXT:    xorps %xmm1, %xmm1
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    je LBB10_4
 ; SSE2-NEXT:  LBB10_3: ## %cond.load1
 ; SSE2-NEXT:    movd {{.*#+}} xmm2 = mem[0],zero,zero,zero
 ; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm0[0]
 ; SSE2-NEXT:    shufps {{.*#+}} xmm2 = xmm2[2,0],xmm0[2,3]
 ; SSE2-NEXT:    movaps %xmm2, %xmm0
 ; SSE2-NEXT:    testb $4, %al

 ; SSE42-NEXT:    jne LBB10_13
 ; SSE42-NEXT:  LBB10_14: ## %else17
 ; SSE42-NEXT:    testb $-128, %al
 ; SSE42-NEXT:    jne LBB10_15
 ; SSE42-NEXT:  LBB10_16: ## %else20
 ; SSE42-NEXT:    retq
 ; SSE42-NEXT:  LBB10_1: ## %cond.load
 ; SSE42-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE42-NEXT:    xorps %xmm1, %xmm1
 ; SSE42-NEXT:    testb $2, %al
 ; SSE42-NEXT:    je LBB10_4
 ; SSE42-NEXT:  LBB10_3: ## %cond.load1
 ; SSE42-NEXT:    insertps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[2,3]
 ; SSE42-NEXT:    testb $4, %al
 ; SSE42-NEXT:    je LBB10_6
 ; SSE42-NEXT:  LBB10_5: ## %cond.load4
 ; SSE42-NEXT:    insertps {{.*#+}} xmm0 = xmm0[0,1],mem[0],xmm0[3]

 ; SSE2-NEXT:    jne LBB20_13
 ; SSE2-NEXT:  LBB20_14: ## %else17
 ; SSE2-NEXT:    testb $-128, %al
 ; SSE2-NEXT:    jne LBB20_15
 ; SSE2-NEXT:  LBB20_16: ## %else20
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  LBB20_1: ## %cond.load
 ; SSE2-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE2-NEXT:    xorps %xmm1, %xmm1
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    je LBB20_4
 ; SSE2-NEXT:  LBB20_3: ## %cond.load1
 ; SSE2-NEXT:    movd {{.*#+}} xmm2 = mem[0],zero,zero,zero
 ; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm0[0]
 ; SSE2-NEXT:    shufps {{.*#+}} xmm2 = xmm2[2,0],xmm0[2,3]
 ; SSE2-NEXT:    movaps %xmm2, %xmm0
 ; SSE2-NEXT:    testb $4, %al

 ; SSE42-NEXT:    jne LBB20_13
 ; SSE42-NEXT:  LBB20_14: ## %else17
 ; SSE42-NEXT:    testb $-128, %al
 ; SSE42-NEXT:    jne LBB20_15
 ; SSE42-NEXT:  LBB20_16: ## %else20
 ; SSE42-NEXT:    retq
 ; SSE42-NEXT:  LBB20_1: ## %cond.load
 ; SSE42-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE42-NEXT:    pxor %xmm1, %xmm1
 ; SSE42-NEXT:    testb $2, %al
 ; SSE42-NEXT:    je LBB20_4
 ; SSE42-NEXT:  LBB20_3: ## %cond.load1
 ; SSE42-NEXT:    pinsrd $1, 4(%rdi), %xmm0
 ; SSE42-NEXT:    testb $4, %al
 ; SSE42-NEXT:    je LBB20_6
 ; SSE42-NEXT:  LBB20_5: ## %cond.load4
 ; SSE42-NEXT:    pinsrd $2, 8(%rdi), %xmm0

llvm/test/CodeGen/X86/oddshuffles.ll

 ; SSE42-LABEL: splat_v3i32:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
 ; SSE42-NEXT:    pxor %xmm1, %xmm1
 ; SSE42-NEXT:    pxor %xmm2, %xmm2
 ; SSE42-NEXT:    pblendw {{.*#+}} xmm2 = xmm0[0,1],xmm2[2,3,4,5,6,7]
 ; SSE42-NEXT:    pblendw {{.*#+}} xmm0 = xmm1[0,1],xmm0[2,3],xmm1[4,5,6,7]
 ; SSE42-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[1,1,0,1]
-; SSE42-NEXT:    pxor %xmm1, %xmm1
 ; SSE42-NEXT:    xorps %xmm3, %xmm3
 ; SSE42-NEXT:    retq
 ;
 ; AVX1-LABEL: splat_v3i32:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vmovddup {{.*#+}} xmm1 = mem[0,0]
 ; AVX1-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; AVX1-NEXT:    vblendps {{.*#+}} ymm0 = ymm2[0],ymm1[1],ymm2[2,3,4,5,6,7]

llvm/test/CodeGen/X86/opt-pipeline.ll

 ; CHECK-NEXT:       Fixup Statepoint Caller Saved
 ; CHECK-NEXT:       PostRA Machine Sink
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       MachinePostDominator Tree Construction
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Shrink Wrapping analysis
 ; CHECK-NEXT:       Prologue/Epilogue Insertion & Frame Finalization
+; CHECK-NEXT:       Machine Late Instructions Cleanup Pass
 ; CHECK-NEXT:       Control Flow Optimizer
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Tail Duplication
 ; CHECK-NEXT:       Machine Copy Propagation Pass
 ; CHECK-NEXT:       Post-RA pseudo instruction expansion pass
 ; CHECK-NEXT:       X86 pseudo instruction expansion pass
 ; CHECK-NEXT:       Insert KCFI indirect call checks
 ; CHECK-NEXT:       MachineDominator Tree Construction

llvm/test/CodeGen/X86/sdiv_fix_sat.ll

 ; X86-NEXT:    movl $0, %esi
 ; X86-NEXT:    cmovel %esi, %edi
 ; X86-NEXT:    cmpl $-1, %edi
 ; X86-NEXT:    movl $0, %edx
 ; X86-NEXT:    cmovel %eax, %edx
 ; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
 ; X86-NEXT:    testl %ecx, %ecx
 ; X86-NEXT:    cmovsl %esi, %eax
-; X86-NEXT:    movl $0, %esi
 ; X86-NEXT:    movl $-1, %ebx
 ; X86-NEXT:    cmovsl %ebx, %edi
 ; X86-NEXT:    andl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Folded Reload
 ; X86-NEXT:    cmpl $-1, %ecx
 ; X86-NEXT:    cmovel %edx, %eax
 ; X86-NEXT:    cmovnel %edi, %ecx
 ; X86-NEXT:    shldl $31, %eax, %ecx
 ; X86-NEXT:    movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill

llvm/test/CodeGen/X86/shift-i128.ll

 ; i686-NEXT:    shrdl %cl, %ebp, %esi
 ; i686-NEXT:    testb $32, %dl
 ; i686-NEXT:    jne .LBB6_9
 ; i686-NEXT:  # %bb.8: # %entry
 ; i686-NEXT:    movl %esi, %ebx
 ; i686-NEXT:  .LBB6_9: # %entry
 ; i686-NEXT:    movl %edi, %esi
 ; i686-NEXT:    movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; i686-NEXT:    movl {{[0-9]+}}(%esp), %ebp
 ; i686-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
 ; i686-NEXT:    shrl %cl, %ebp
 ; i686-NEXT:    testb $32, %cl
 ; i686-NEXT:    movl $0, %ecx
 ; i686-NEXT:    jne .LBB6_11
 ; i686-NEXT:  # %bb.10: # %entry
 ; i686-NEXT:    movl %ebp, %ecx
 ; i686-NEXT:  .LBB6_11: # %entry

 ; i686-NEXT:  # %bb.10: # %entry
 ; i686-NEXT:    movl %esi, %ecx
 ; i686-NEXT:  .LBB7_11: # %entry
 ; i686-NEXT:    movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; i686-NEXT:    movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; i686-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; i686-NEXT:    movb $64, %cl
 ; i686-NEXT:    subb %dl, %cl
-; i686-NEXT:    movl {{[0-9]+}}(%esp), %ebp
 ; i686-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; i686-NEXT:    shldl %cl, %ebx, %ebp
 ; i686-NEXT:    movl %ebp, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; i686-NEXT:    movl %ebx, %ebp
 ; i686-NEXT:    shll %cl, %ebp
 ; i686-NEXT:    testb $32, %cl
 ; i686-NEXT:    movb $64, %bl
 ; i686-NEXT:    movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill

llvm/test/CodeGen/X86/ushl_sat_vec.ll

 ; X86-NEXT:    movl %edx, %esi
 ; X86-NEXT:    shll %cl, %esi
 ; X86-NEXT:    movzwl %si, %ebx
 ; X86-NEXT:    movl %ebx, %esi
 ; X86-NEXT:    shrl %cl, %esi
 ; X86-NEXT:    cmpw %si, %dx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    cmovnel %eax, %ebx
-; X86-NEXT:    movl $65535, %eax # imm = 0xFFFF
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl %edx, %esi
 ; X86-NEXT:    shll %cl, %esi
 ; X86-NEXT:    movzwl %si, %edi
 ; X86-NEXT:    movl %edi, %esi
 ; X86-NEXT:    shrl %cl, %esi
 ; X86-NEXT:    cmpw %si, %dx
 ; X86-NEXT:    cmovnel %eax, %edi

llvm/test/CodeGen/X86/vec_extract.ll

 
 ; OSS-Fuzz #15662
 ; https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=15662
 define <4 x i32> @ossfuzz15662(ptr %in) {
 ; X32-LABEL: ossfuzz15662:
 ; X32:       # %bb.0:
 ; X32-NEXT:    xorps %xmm0, %xmm0
 ; X32-NEXT:    movaps %xmm0, (%eax)
-; X32-NEXT:    xorps %xmm0, %xmm0
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: ossfuzz15662:
 ; X64:       # %bb.0:
 ; X64-NEXT:    xorps %xmm0, %xmm0
 ; X64-NEXT:    movaps %xmm0, (%rax)
-; X64-NEXT:    xorps %xmm0, %xmm0
 ; X64-NEXT:    retq
    %C10 = icmp ule i1 false, false
    %C3 = icmp ule i1 true, undef
    %B = sdiv i1 %C10, %C3
    %I = insertelement <4 x i32> zeroinitializer, i32 0, i1 %B
    store <4 x i32> %I, ptr undef
    ret <4 x i32> zeroinitializer
 }

llvm/test/CodeGen/X86/vec_shift5.ll

 ; Make sure we fold fully undef input vectors. We previously folded only when
 ; undef had a single use so use 2 undefs.
 define <4 x i32> @test17(<4 x i32> %a0, ptr %dummy) {
 ; X86-LABEL: test17:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    xorps %xmm0, %xmm0
 ; X86-NEXT:    movaps %xmm0, (%eax)
-; X86-NEXT:    xorps %xmm0, %xmm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test17:
 ; X64:       # %bb.0:
 ; X64-NEXT:    xorps %xmm0, %xmm0
 ; X64-NEXT:    movaps %xmm0, (%rdi)
-; X64-NEXT:    xorps %xmm0, %xmm0
 ; X64-NEXT:    retq
   %a = call <4 x i32> @llvm.x86.sse2.pslli.d(<4 x i32> undef, i32 6)
   store <4 x i32> %a, ptr %dummy
   %res = call <4 x i32> @llvm.x86.sse2.pslli.d(<4 x i32> undef, i32 7)
   ret <4 x i32> %res
 }
 
 define <4 x i32> @test18(<4 x i32> %a0, ptr %dummy) {
 ; X86-LABEL: test18:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    xorps %xmm0, %xmm0
 ; X86-NEXT:    movaps %xmm0, (%eax)
-; X86-NEXT:    xorps %xmm0, %xmm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test18:
 ; X64:       # %bb.0:
 ; X64-NEXT:    xorps %xmm0, %xmm0
 ; X64-NEXT:    movaps %xmm0, (%rdi)
-; X64-NEXT:    xorps %xmm0, %xmm0
 ; X64-NEXT:    retq
   %a = call <4 x i32> @llvm.x86.sse2.pslli.d(<4 x i32> undef, i32 3)
   store <4 x i32> %a, ptr %dummy
   %res = call <4 x i32> @llvm.x86.sse2.pslli.d(<4 x i32> undef, i32 1)
   ret <4 x i32> %res
 }
 
 ; PR39482

llvm/test/CodeGen/XCore/scavenging.ll

 ; scavenge r5 using SR spill slot
 ; CHECK: stw r5, sp[0]
 ; CHECK: ldw r5, cp[[[INDEX0]]]
 ; r4 & r5 used by InsertSPConstInst() to emit STW_l3r instruction.
 ; CHECK: stw r0, r4[r5]
 ; CHECK: ldaw r0, sp[0]
 ; CHECK: ldw r5, cp[[[INDEX1]]]
 ; CHECK: stw r1, r0[r5]
-; CHECK: ldaw r0, sp[0]
 ; CHECK: ldw r1, cp[[[INDEX2]]]
 ; CHECK: stw r2, r0[r1]
-; CHECK: ldaw r0, sp[0]
 ; CHECK: ldw r1, cp[[[INDEX3]]]
 ; CHECK: stw r3, r0[r1]
-; CHECK: ldaw r0, sp[0]
 ; CHECK: ldw r1, cp[[[INDEX4]]]
 ; CHECK: stw r11, r0[r1]
 ; CHECK: ldaw sp, sp[65535]
 ; CHECK: ldw r4, sp[1]
 ; CHECK: ldw r5, sp[0]
 ; CHECK: retsp 34467
 define void @ScavengeSlots(i32 %r0, i32 %r1, i32 %r2, i32 %r3, i32 %r4) nounwind {
 entry: