Actually, now that I think about it, I believe we realized that SIFixWWMLiveness has a giant hole in that if any of the extra live ranges we insert are split, it'll fall over. I don't think anyone has come up with a way to express the constraints only with extra defs and uses in a way that always works, and I'm not sure it's possible. The issue is that we're lying to LLVM RA by pretending that vector instructions always fully clobber their destinations, and while before we were careful to never write to any inactive channels in order to keep up the charade, but WWM instructions force us to deal with it somehow. Fully informing LLVM of what's going on would involve marking every vector instruction as partially clobbering its destination, even the move instructions and load/store instructions LLVM emits during RA, which of course would tank performance unless LLVM is taught about predicated liveness -- but of course that's a whole lot of work that opens another can of worms (register pressure is suddenly not that meaningful anymore...).

In D59295#1427864, @arsenm wrote:

If you can express the constraints with some series of uses and defs, that would be preferable.

In D59295#1427734, @arsenm wrote:

I really don't like introducing new, dynamically reserved registers for this. It's going to introduce hell for dealing with any kind of ABI, and reserved registers are generally a bad idea. There's also nothing guaranteeing there are any free registers available to reserve, since you are just grabbing totally unused ones. This is going to just hit some variant of the problem I've been working on solving for handling SGPR->VGPR spills. Can WWM code be moved into a bundle or something?

Fixed a regression in the llvm.amdgcn.kill tests.

I'm not going to read everything in detail, but the combining rules look correct to me and everything passes with this pass enabled. Feel free to re-enable it.

Adding a pattern for this wouldn't work for what I wanted to do, which was a ballot/inverseballot pair to operate directly on the bitmask representation of a boolean, since there's a bug where SelectionDAG forgets that ballot removes divergence, and it needs to be non-divergent for the pattern to fire. That being said, inserting two readlanes isn't that much better, so maybe I should just fix that instead...

• Remove spurious whitespace change.
• Lower S_INV_BALLOT with EmitInstrWithCustomInserter.

• Added a pseudoinstruction lowered in SILowerI1Copies, and legalized it more

similarly to how other instructions are legalized.

• Added tests where the source is a uniform and non-uniform phi node. I had to

to use the amdgpu_ps calling convention here to get the arguments into SGPR's.

In D57748#1385046, @arsenm wrote:

The user can't actually guarantee the argument is uniform, since transforms are allowed to touch the argument. We're accumulating quite a few places that assume this though

I figured it would be a little easier if I looked at these cases by myself. It turns out there are more problems with isIdentityValue, including some correctness issues. After fixing these, everything works correctly now.

Sorry, I just got back from break this week. I've run CTS with the pass enabled, and it now passes, although it seems most of the patterns we use don't get folded. Firstly AND, XOR, unsigned max, and unsigned min are most troubling, since the code that gets generated looks like it should be optimized:

In D55314#1321326, @vpykhtin wrote:

Ok, I'll disable it. I'm not sure about 3rd point: are you sayng the pass doesn't actually perform the optimization or it's fundamentally wrong? Because it implemented to handle "identity" cases for add, mul and min/max.

In D55314#1321222, @vpykhtin wrote:

In D55314#1321167, @hakzsam wrote:

Hi,

This change breaks most of the subgroups tests with RADV (ie. dEQP-VK.subgroups.arithmetic.*).

Any reasons why you enabled it by default? Looks like it now triggers a new bug in the AMDGPU backend.

Thanks!

Hi,

this is awaited feature and other reason is to detect a situation such yours. I think it's better to turn it off and reproduce the failures: how can I do this?

In D54516#1300997, @nhaehnle wrote:

In D54516#1298178, @cwabbott wrote:

I believe the combination of Convergent + not Speculatable should mean that the compiler shouldn't hoist it to a non-control-equivalent block and shouldn't CSE it. In particular, IIRC it's not guaranteed that that a readnone function always return the same value when it's called with the same arguments, so it's not safe to CSE -- it just means that LLVM can move other things across it, since it doesn't modify *caller-visible* state. What pass is causing a problem? Maybe it's a bug in the pass?

readnone is literally readnone. Maybe you're mixing this up with inaccessiblememonly?

I believe the combination of Convergent + not Speculatable should mean that the compiler shouldn't hoist it to a non-control-equivalent block and shouldn't CSE it. In particular, IIRC it's not guaranteed that that a readnone function always return the same value when it's called with the same arguments, so it's not safe to CSE -- it just means that LLVM can move other things across it, since it doesn't modify *caller-visible* state. What pass is causing a problem? Maybe it's a bug in the pass?

In D46470#1088984, @tpr wrote:

You made a comment in this code that "this is a workaround anyways until LLVM gains the notion of predicated uses and definitions of variables". How do you see that working? Are you aware of anyone else thinking along those lines in the LLVM community? Something to think about for the future.

For the liveness issue, maybe a better way to solve it would be to add a new ENTER_WWM pseudoinstruction similar to EXIT_WWM, and add a matching implicit def to the matching ENTER_WWM whenever we insert an implicit use on EXIT_WWM, and mark both of them as kills. After all, any affected registers only need to interfere with the instructions run in WWM, so that should help with code quality too. I'm not sure why I didn't do that in the first place.

Ping on this one. This is the last outstanding patch for implementing AMD_shader_ballot in Mesa.

It looks like Sam has worked a lot on the assembler, including adding support for DPP instructions, so I'm adding him for the assembler bits. I'd like to get this in before I leave next week, though.

Remove spurious change to AMDGPUAsmParser.cpp

Fix assembling DPP instructions. Also, adopt a more conservative version of
D34715. In particular, we ignore Constraints/DisableEncoding from the original
instruction for the DPP version. The only instruction with any special
constraints is MAC, because of its fake third source, and there it doesn't make
sense to keep the fake third source since it has to be the same as the normal
"old" source anyways. We can revisit this if something else comes up, but I
think this is a good plan for now.

I realized today that this change is bogus, since these lines aren't copying the Constraints from the Profile. They're copying the constraints set via the pseudoinstruction, and removing these lines breaks a few things, in particular assembling MAC instructions, and probably liveness tracking for them too. Originally the changes in D34716 wouldn't work without this, but I'm going to update D34716 with a better solution that passes all the tests.

Rebase on master.

Remove unneccesary stuff from fix-wwm-liveness.mir, based on Matt's comment for wqm.mir.

Fix one last style issue.

Clarify what the code that implements the WQM semantics is doing.

• rename dest of EXIT_WWM to be consistent

• fix style issues
• add test for WWM with integers
• remove unneccesary stuff from MIR test

Ping.

Fixup lowerCopyInstrs() after using setDesc() by removing extra arguments.

Remove leftover $wqm_ctrl.

Rebase on using s_or_saveexec_b64 for WWM (fix test).

• Fix style issues.
• Rebased on using setDesc() in lowerCopyInstrs()
• Reworked logic for determining safe points to transition where SCC isn't live, to make using s_or_saveexec_b64 for entering WWM possible.
• Added tests for WWM across multiple basic blocks and correct SCC handling. For the latter, I couldn't figure out how to make an IR test that exercised that path, so I made a MIR test for it.

Reset to previous version. Whoops.

Whoops, I accidentally squashed this with D35524 while rebasing. Those changes were meant to be applied to that.

Fix WWM SCC test to actually test what it's supposed to, and fix a bug caught by it. Whoops.

• Fixed style issues.
• Rebased on using setDesc() in lowerCopyInstrs()
• Reworked logic for determining safe points to transition where SCC isn't live, to make using s_or_saveexec_b64 for entering WWM possible.
• Added tests for WWM across multiple basic blocks and correct SCC handling. For the latter, I couldn't figure out how to make an IR test that exercised that path, so I made a MIR test for it.

Mask out the flag first in markInstruction() to simplify the code a little and
avoid processing instructions unnecessarily once WWM is enabled.

Minor comment style fixes, simplify lowerCopyInstrs by using setDesc()

• Check for correct S_NOP argument and use VI-NEXT
• Only check for VALU EXEC write as per the docs, and update test

Abadon in favor of D35524. While I based that change off of this one, things have changed so much that it's probably better to abandon this and do the review there.

Remove wqm_ctrl in favor of llvm.amdgcn.wqm and llvm.amdgcn.wqm
intrinsics, and remove tests for them.

Remove wqm_mode in favor of llvm.amdgcn.wqm and llvm.amdgcn.wwm intrinsics.

rebase on latest WWM implementation, tweak semantics and implementation
to force WQM whenever WQM is used.

Avoid illegal SGPR<->VGPR copies by treating WQM more like COPY, add test that
exposes the problem.

Ping.

I just posted D35167, which seems closer to how we want to deal with WQM and WWM in the long term, and should provide the same functionality as this intrinsic.

Actually disable WWM on exit of a block.

After some more testing, it seems like my hypothesis about the issue was wrong. I think the real problem was that WWM wasn't being disabled in the face of control flow like I intended it to, and then stuff was getting messed up because inactive lanes were enabled when the code assumed they weren't. At least, that's my theory for now. I'm going to update D34717 to fix that.