Page MenuHomePhabricator

[LoopVectorize] Fix bug where predicated loads/stores were dropped

Authored by joechrisellis on Mar 30 2021, 2:08 AM.



This commit fixes a bug where the loop vectoriser fails to predicate
loads/stores when interleaving for targets that support masked
loads and stores.

Code such as:

1  void foo(int *restrict data1, int *restrict data2)
2  {
3    int counter = 1024;
4    while (counter--)
5      if (data1[counter] > data2[counter])
6        data1[counter] = data2[counter];
7  }

... could previously be transformed in such a way that the predicated
store implied by:

if (data1[counter] > data2[counter])
   data1[counter] = data2[counter];

... was lost, resulting in miscompiles.

This bug was causing some tests in llvm-test-suite to fail when built
for SVE.

Diff Detail

Event Timeline

joechrisellis created this revision.Mar 30 2021, 2:08 AM
joechrisellis requested review of this revision.Mar 30 2021, 2:08 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 30 2021, 2:08 AM
joechrisellis planned changes to this revision.Mar 30 2021, 3:07 AM

Hmm. I anticipated there might be some test failures for this one.

Minor change to x86-predication.ll test.

The codegen from before the patch had a simpler CFG because we weren’t
predicating on loads. In this particular case, it turns out that it’s actually
fine to not explicitly predicate on the loads, but only because they exist in a
basic block next to a udiv which does require predication, so we get it ‘for
free’ in a sense. This is not guaranteed to always be the case, though, so we
have to be careful.

The result of this patch is that the IR CFG for scalarize_and_sink_gather in
x86-predication.ll is larger, and so the test needs to be updated to reflect
that. Ultimately, though, the machine code lowering is the same, so no loss in

joechrisellis retitled this revision from [AArch64][SVE] Fix vectoriser bug where predicated stores were dropped to [LoopVectorize] Fix bug where predicated loads/stores were dropped.Mar 31 2021, 5:29 AM
joechrisellis edited the summary of this revision. (Show Details)
peterwaller-arm added inline comments.

I suspect using -debug needs a REQUIRES: asserts line, since the tests will be run presumably with a release build where -debug is unavailable.

The output of:

rg -C3 -- 'RUN.* -debug-only'

... in the llvm/test directory appears to show that they generally have such a REQUIRES line.

I find this a little surprising since presumably this doesn't work on an asserts+release build; but my reading is that enabling assertions may enable debug output:

joechrisellis marked an inline comment as done.

Improve test and address review comments.

david-arm added inline comments.

@peterwaller-arm yep you're right this needs:

REQUIRES: asserts

and it will enable debug.


Hi @joechrisellis you've lost the variable T0 here that was used to ensure the load fed into the udiv. It would be good if you could re-add this.

joechrisellis marked an inline comment as done.

Address comments.

Gentle ping. 🙂


Sure, will re-add this. For context, I removed this because the control flow is now slightly different now that the loads are predicated, so I thought the cleanest thing to do would be to remove the use of this variable. I've added it back now.

fhahn added inline comments.Apr 12 2021, 4:39 AM

I'm not sure about the default. Why does it make sense to default to VF = 1? Can all users of the function instead pass the right VF?


as you are only checking specifically for small IR snippets of the predicated blocks, do we actually need -dce -instcombine? Running those potentially make the test more fragile, because it may be impacted by changes to -dce -instcombine.


can we instead just do 2>&1 | FileCheck %s in one go? Splitting them up makes it a bit more inconvenient to reproduce a failure, because 2 lines need to copied/executed?


this seems already out of sync with the IR below.... Shouldn't the restrict be translated to noalias in the function definition below? FWIW I think it would be more helpful if you'd add a comment to the test saying this checks that the store in the if.then block gets properly predicated if VF = 1 or something like that, as the IR is already quite compact :)


Is this check crucial? Seems like the IR checks already check this and we could remove it and execute it even with asserts.


do we need 2 loads here or could one be removed and compare to a constant?


If it's not SVE specific, can we have 2 run lines, one with -mattr=+sve and one without?

joechrisellis marked 5 inline comments as done.

Address review comments.

Hi @fhahn -- thanks for the review!


The reason I added this optional parameter is because isPredicatedInst is used in this context:

bool IsPredicated = LoopVectorizationPlanner::getDecisionAndClampRange(
    [&](ElementCount VF) { return CM.isPredicatedInst(I, VF); }, Range);

I wanted to make sure that the VF (which is used to make decisions on how to clamp the range) is made available to isScalarWithPredication in the fallback case for this particular call, but keep the same signature for calls elsewhere. Given what I think the function is supposed to do, I can see why it might not make sense to pass a VF parameter for the general case, but here I _think_ we need it. Other references to isPredicatedInst do not supply a VF parameter, so in the fallback cases, they will default to VF = 1 when we reach the call for isScalarWithPredication. This is just pushing the optional parameter up the stack so that it can be used for isPredicatedInst too.

If there's a better thing to do here I'm happy to make changes. 😄


Good spot. I'll remove the restrict in the C code example since it isn't strictly necessary here anyways.


Fair enough, done!


I would like to keep this IR as-is for consistency with the C code at the top of the test. 🙂

david-arm accepted this revision.Apr 19 2021, 5:59 AM

LGTM! I think there is potentially a regression in the X86 test 'scalarize_and_sink_gather' in terms of number of generated lines of assembly, but the fix looks right for now. We could potentially improve the code quality in a future patch that looks for instructions with matching predicates and folds them into the same predicated block.

This revision is now accepted and ready to land.Apr 19 2021, 5:59 AM
fhahn added inline comments.Apr 19 2021, 6:11 AM

his is just pushing the optional parameter up the stack so that it can be used for isPredicatedInst too.

That's fair enough, but my point was that there are only 2 uses of the function I think, so the default parameter is not really that useful and I think it would be better to just explicitly pass the VF in at both call sites. Also, the default of VF(1) is surprising (but I realize it is used elsewhere) and it's probably better to force the caller to think about what VF to pass in.


I would like to keep this IR as-is for consistency with the C code at the top of the test. 🙂

Personally I think that the C example code should not stand in the way of having a simpler IR test. If you really want to keep the C code, can't you simplify the C as well? (With the even simpler IR I think the C code adds more noise than value it adds).

Address review comments.

  • @fhahn:
    • make VF parameter to isPredicatedInst required.
    • simplify IR test.
This revision was landed with ongoing or failed builds.Apr 22 2021, 8:06 AM
This revision was automatically updated to reflect the committed changes.