[AArch64][LoopVectorize] Enable tail-folding of simple loops on neoverse-v1

[david-arm]

Summary

This patch enables the tail-folding of simple loops by default
when targeting the neoverse-v1 CPU. Simple loops exclude those
with recurrences or reductions or loops that are reversed.

New tests have been added here:

Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll

In terms of SPEC2017 only one benchmark is really affected when
building with "-Ofast -mcpu=neoverse-v1 -flto", which is
(+ faster, - slower):

525.x264: +7.0%

[tschuett]

Is the restriction the vector length in disguise or is something really special, i.e., the N2 has short SVE vectors?

[david-arm]

In D130618#3682600, @tschuett wrote:

Is the restriction the vector length in disguise or is something really special, i.e., the N2 has short SVE vectors?

By 'restriction' are you referring to how this patch limits the types of loop we consider for tail-folding on the N1? This decision is based purely on the observed results from running benchmarks on hardware.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

73–74

This class seems to be mixing two concepts:

• Features that the loop requires for tail folding (i.e. recurrences/reductions/anything-else).
• What the target wants as the default and the parsing-logic for user-visible toggles.

This makes it quite tricky to follow the logic, especially the need for both NeedsDefault + DefaultBits and Add/RemoveBits`.

Perhaps it would make more sense to have two classes:

class TailFoldingKind;    // This encodes which features the loop requires
class TailFoldingOption;  // This encodes the default (which itself will be a TailFoldingKind object), and has logic to parse strings.

Then you can add a method such as TailFoldingKind TailFoldingOption::getSupportedMode() const { .. } that you can use to query if the target's mode satisfies the required TailFoldingKind from the Loop.

120–125

Perhaps I'm missing something, but this logic with AddBits and RemoveBits seems unnecessarily complicated. Can't you have a single bitfield and do something like this:

void set(uint8_t Flags, bool Enable) {
  if (Enable)
    Bits |= Flags;
  else
    Bits &= ~Flags;
}

?

3495–3496

This seems redundant given the line above that says 'Defaults to 0'.

How about creating a constructor for it that takes a TailFoldingOpts as operand, so that you can write TailFoldingKind Required(TFDisabled)?

3510

Is it worth creating a method for this, so that you don't have to expose the bit-field to users, e.g.

return TailFoldingKindLoc.satisfies(Required);

?

[david-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

120–125

The reason for this is because at the time of parsing the string we have no idea what the default options are going to be. I basically wanted to avoid creating a dependency on the CPU such that the user has to put the -sve-tail-folding option after the -mcpu flag. In the tests I added two RUN lines for both "-mcpu=neoverse-v1 -sve-tail-folding=default" and "-sve-tail-folding=default -mcpu=neoverse-v1". In the latter case we can't build on top of the default bits because we don't yet have them at the time of parsing. An example I'm thinking of is this:

-sve-tail-folding=default+nosimple -mcpu=neoverse-v1

which is a bit daft (and we don't even have a nosimple option yet!). We only know the default (simple only for neoverse-v1) once we've parsed the -mcpu flag and therefore we can't remove the simple flag until later. So I tried doing this by keeping a record of the bits we want to add and remove, and apply them later. That's why a single 'Bits' field like you mentioned above doesn't work. I can have a look at your suggestion above and see if that solves the problem.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

73–74

This doesn't quite work because you've lost the position of the default value compared to the explicitly enabled/disabled options. It's almost as if you want to defer parsing until when the data is required and then have something like TailFoldingKindLoc.parseWithDefault(getTailFoldingDefaultForCPU()) . I think if you can do this, many of the other changes in the patch become necessary.

[david-arm]

• Renamed TailFoldingKind -> TailFoldingOption and introduced a simple TailFoldingKind class.
• TailFoldingOption now stashes a copy of the option string and parses it on demand to ensure that bits are added and removed in the correct order.
• Added a new satisfies(TailFoldingKind Required) interface to TailFoldingOption and simplified the logic in getScalarEpilogueLowering

[david-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

73–74

I've tried to separate out the two concepts into two different classes, but I've kept the DefaultBits as part of the new TailFoldingOption class.

[david-arm]

• Let cortex-a710 and x2 have the same SVE tail-folding defaults.

[dmgreen]

Can you explain more about why reductions are a problem for certain cpus? What about the cortex-a510? And if it being disabled for all these cpus, should it be disabled for -mcpu=generic too? I'm not sure why we would disable sve reductions though - first-order-recurrences make more sense, but that might be something that is better is done in general, not per-subtarget.

And is tail-folding expected to be beneficial in general? As far as I can see it might currently be losing the interleaving, which can be important for performance. And it should ideally not be altering the NEON codegen, if that could be preferable. Is this currently one option that alters both scalable and fixed width vectorization?

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

131

Is this setting a global variable? I think it should be just a field in the subtarget (maybe a subtarget feature), that is potentially overridden by the option if it is present.

[david-arm]

In D130618#3696283, @dmgreen wrote:

Can you explain more about why reductions are a problem for certain cpus? What about the cortex-a510? And if it being disabled for all these cpus, should it be disabled for -mcpu=generic too? I'm not sure why we would disable sve reductions though - first-order-recurrences make more sense, but that might be something that is better is done in general, not per-subtarget.

And is tail-folding expected to be beneficial in general? As far as I can see it might currently be losing the interleaving, which can be important for performance. And it should ideally not be altering the NEON codegen, if that could be preferable. Is this currently one option that alters both scalable and fixed width vectorization?

Hi @dmgreen, through benchmarking and performance analysis we discovered that on some cores (neoverse-v1, x2, a64fx) if you use tail-folding with reductions the performance is significantly worse (i.e. 80-100% slower on some loops!) than using unpredicated vector loops. This was observed consistently across a range of loops, benchmarks and CPUs, although we don't know exactly why. Our best guess is that it's to do with the chain of loop-carried dependencies in the loops, i.e. reduction PHI + scalar IV + loop predicate PHI. So it's absolutely critical that we avoid signficant regressions for benchmarks that contain reductions or first-order recurrences and this patch is a sort of a compromise. If you don't specify the CPU then we will follow architectural intent and always tail-fold for all loops, but when targeting CPUs with this issue we disable tail-folding for such loops.

In general, tail-folding is beneficial for reducing code size and mopping up the scalar tail, as well following the intentions of the architecture. For example, x264 in SPEC2k17 sees 6-7% performance improvements on neoverse-v1 CPUs due to the low trip counts in hot loops.

With regards to interleaving, the fundamental problem lies with how we do tail-folding in the loop vectoriser, which forces us to make cost-based decisions about whether to use tail-folding or not before we've calculated any loop costs. Enabling tail-folding has consequences because suddenly your loops become predicated and the costs change accordingly. For example, NEON does not support masked interleaved memory accesses, so enabling tail-folding leads to insane fixed-width VF costs. At the same time the loop vectoriser does not support vectorising interleaved memory accesses for scalable vectors either, so we end up in a situation where the vectoriser decides not to vectorise at all! Whereas if we don't enable tail-folding we will vectorise using a fixed-width VF and use NEON's ld2/st2/etc instructions, which is often still faster than a scalar loop. Ultimately in the long term we would like to change the loop vectoriser to consider a matrix of costs, with vectorisation style on one axis and VF on the other, then choose the most optimal cost in that matrix. But this is a non-trivial piece of work, so in the short term we opted for this temporary solution.

[dmgreen]

Hi @dmgreen, through benchmarking and performance analysis we discovered that on some cores (neoverse-v1, x2, a64fx) if you use tail-folding with reductions the performance is significantly worse (i.e. 80-100% slower on some loops!) than using unpredicated vector loops. This was observed consistently across a range of loops, benchmarks and CPUs, although we don't know exactly why. Our best guess is that it's to do with the chain of loop-carried dependencies in the loops, i.e. reduction PHI + scalar IV + loop predicate PHI. So it's absolutely critical that we avoid signficant regressions for benchmarks that contain reductions or first-order recurrences and this patch is a sort of a compromise. If you don't specify the CPU then we will follow architectural intent and always tail-fold for all loops, but when targeting CPUs with this issue we disable tail-folding for such loops.

In general, tail-folding is beneficial for reducing code size and mopping up the scalar tail, as well following the intentions of the architecture. For example, x264 in SPEC2k17 sees 6-7% performance improvements on neoverse-v1 CPUs due to the low trip counts in hot loops.

With regards to interleaving, the fundamental problem lies with how we do tail-folding in the loop vectoriser, which forces us to make cost-based decisions about whether to use tail-folding or not before we've calculated any loop costs. Enabling tail-folding has consequences because suddenly your loops become predicated and the costs change accordingly. For example, NEON does not support masked interleaved memory accesses, so enabling tail-folding leads to insane fixed-width VF costs. At the same time the loop vectoriser does not support vectorising interleaved memory accesses for scalable vectors either, so we end up in a situation where the vectoriser decides not to vectorise at all! Whereas if we don't enable tail-folding we will vectorise using a fixed-width VF and use NEON's ld2/st2/etc instructions, which is often still faster than a scalar loop. Ultimately in the long term we would like to change the loop vectoriser to consider a matrix of costs, with vectorisation style on one axis and VF on the other, then choose the most optimal cost in that matrix. But this is a non-trivial piece of work, so in the short term we opted for this temporary solution.

Hello. That sounds like the loops need unrolling - that's common in order to get the best ILP out of cores. Especially for in-order cores, but it is important for out of order cores too. Something like the Neoverse-N2 has 2 FP/ASIMD pipes, and in order to keep them busy you need to unroll small loops. The Neoverse-V1 has 4 FP/ASIMD pipes. But that's not limited to reductions, it applies to any small loop as far as I understand. This is the reason that we use a MaxInterleaveFactor of at least 2 for all cores in llvm, and some set it higher. I don't think that changes with SVE. It still needs to unroll the loop body, preferably with a predicated epilogue.

It is true that sometimes this extra unrolling is unhelpful for loops with small trip counts. But I was hoping that the predicated epilogue would help with that. I thought that was the plan? What happened to making an unrolled/unpredicated body with a predicated remainder? There will be a loops that are large enough that we do not unroll. Those probably make sense to predicate, so long as the performance is good (it will depend on the core). For Os too. For MVE it certainly made sense once we had worked through fixing all the performance degradations we could find, but those are very different cores which pay more heavily for code-structure inefficiencies. And they (almost) never need the unrolling.

For AArch64 the interleaving count will be based on the cost of the loop, among other things. Maybe this should be based on the cost of the loop too? Does tail folding really mean we cannot unroll?

[dmgreen]

Oh - and about -mcpu=generic - it'd difficult but it needs to be the best we can do for the most cores possible. Especially on Android systems where the chances of using a correct -mcpu are almost none. What that is I'm not entirely sure. I think maybe unpredicated loop with predicated remainder, with the option to change to something else in the future?

[david-arm]

Hi @dmgreen, we had exactly the same thought process as you. We have already explored unrolling tail-folded loops with reductions and even with the best code quality it makes zero difference to performance on these cores. Sadly it doesn't make the loops faster in the slightest - there appears to be a fundamental bottleneck that cannot be surpassed. No amount of unrolling (using LLVM or by hand) helps in the case of reductions or first-order recurrences. I have also tried unrolling plus manually rescheduling instructions in different ways, but to no avail. We were also very suprised by this, and wish it were different!

[dmgreen]

In D130618#3699606, @david-arm wrote:

Hi @dmgreen, we had exactly the same thought process as you. We have already explored unrolling tail-folded loops with reductions and even with the best code quality it makes zero difference to performance on these cores. Sadly it doesn't make the loops faster in the slightest - there appears to be a fundamental bottleneck that cannot be surpassed. No amount of unrolling (using LLVM or by hand) helps in the case of reductions or first-order recurrences. I have also tried unrolling plus manually rescheduling instructions in different ways, but to no avail. We were also very suprised by this, and wish it were different!

It sounds like you are hitting another bottleneck. Perhaps the amount of predication resources.

[SjoerdMeijer]

Hi @SjoerdMeijer, thanks for looking into this. I do actually already have a patch to enable this by default (https://reviews.llvm.org/D130618), where the default behaviour is tuned according to the CPU. I think this is what we want because the profile will change according to what CPU you're running on - some CPUs may handle reductions better than others.

I didn't know what the problem is with reductions. I now see you had a discussion with Dave here about reductions, but it looks it is still unclear why exactly that is a problem (at least to me).
But I thought that "simple" would be a good start as a default, which is what you're more or less doing here too. Well, you're setting it for a few cpus, but my feeling is that "simple" should be a very safe bet for all.

The decision in this patch may be incorrect for 128-bit vector implementations. I also ran SPEC2k17 on a SVE-enabled CPU as well and I remember I saw a small (2-3%) regression in parest or something like that, which is one of the reasons I didn't push the patch any further. I also think it's really important to run a much larger set of benchmarks besides SPEC2k17 and collect numbers to show the benefits, since there isn't much vectorisation actually going on in SPEC2k17.

I will collect data for SPEC FP. I don't know exactly how much vectorisation is going on there, but will see. By the way, most vectorisation happens in x264 which sees a significant uplift. My numbers on a 256 bit vector implementation sees neutral results for the other SPEC INT apps. I think this is a very good start.

One of the major problems with the currrent tail-folding implementation is that we make the decision before doing any cost analysis in the vectoriser, which isn't great because we may be forcing the vectoriser to take different code paths to if we didn't tail-fold. Ideally what we really want is to move to a model where the vectoriser has a two-dimensional matrix of costs considering the combination of VF and vectorisation style (e.g. tail-folding vs whole vector loops, etc.), and choose the most optimal combination.

Yeah, I am aware and noticed this while implementing tail-folding for MVE. This is a problem, but something we have to live with for a while I think as it is not very low hanging fruit to change this. That's my impression at least. However, with the "simple" tail-folding it is difficult to see how it would lead to regressions.

What are your ideas to progress this?

[SjoerdMeijer]

I saw a small (2-3%) regression in parest or something like that

Yep, I see a 2.7% regression in parest.

[david-arm]

• Rebase.
• The patch now only enables tail-folding by default for neoverse-v1.

[sdesmalen]

Hi @david-arm, I wonder if it makes sense to move some of these changes to a separate (NFC?) patch where you do the refactoring of the class, so that this patch becomes as simple as calling setSVETailFoldingDefaultOpts(TFSimple);. That way, if for any reason the patch needs to be reverted, we don't loose out on the refactoring. We can still test the current code with the llc options.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

43

Does this class still have any value? It only does bitwise |, & and ~, which you can inline below.

60

nit: Could you rename this to UnparsedOptionString or something?

60–61

Rather than storing the string and substrings, can you just do the parsing as part of the constructor of TailFoldingOption so that you only need to store the bits?

90

Instead of an llvm_unreachable, do you want to print the error message here?

112

This is only printing to stderr. Should this be a fatal error?

llvm/lib/Target/AArch64/Utils/AArch64BaseInfo.h

537

I know you're just moving this code around, but could you add a description of what 'simple' means? And maybe also add a description for TFRecurrences (are those first-order recurrences that reqequire a splice operation?)

(very minor nit: would it make sense to make this enum value 0x01 so that it's clear that anything not '1' is no longer a 'simple' loop?)

[david-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

60–61

I seem to remember there was a problem with ordering where we had to ensure we got the same behaviour for each version of this command line:

-mcpu=native -mllvm -sve-tail-folding=default

and

-mllvm -sve-tail-folding=default -mcpu=native

which is tested in sve-tail-folding-option.ll. I think it meant that we could only deal with the options after the entire command line has been parsed, so we couldn't do it in the constructor. I'll double check though as I may have remembered incorrectly!

[dmgreen]

Do you intend this to be a real patch to review? It doesn't match my understanding of the theory, but perhaps something has changed?

[david-arm]

In D130618#4294921, @dmgreen wrote:

Do you intend this to be a real patch to review? It doesn't match my understanding of the theory, but perhaps something has changed?

The short answer is - yes. As the patch says we want to enable tail-folding by default for simple loops on neoverse-v1 CPUs.

[dmgreen]

OK. My understanding was that the Neoverse-V1 was one of the worst cpus for tail folding due to the limited throughput of while instructions. Doesn't tail folding prevent interleaving too? Newer generations should do better than the V1.

This is one of the "simplest" loops I can think of. My understanding is that it will go around half the speed with this patch: https://godbolt.org/z/K4e3PMdar

Are you sure this shouldn't be based on whether the loop is small enough to desire interleaving? As in the difference between Simple and Reductions was never really about reductions, those were just the loops that hit the problem the hardest. (There may be some other issues with reductions codegen, but the limited interleaving/throughput still remains). The real problem is that we need to make sure we don't limit the interleaving for small loops, or that the while instructions become a bottleneck for throughput.

[david-arm]

• Addressed review comments, split off some of the refactoring into a NFC patch (D149659).
• Restricted tail-folding only to loops that exceed a certain instruction threshold, so that we get the benefit of interleaving for tight loops.

[david-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

60–61

Hi @sdesmalen, so this isn't possible because (as mentioned above) you can't guarantee that the construction (in this case the operator=) of the option happens before setting the CPU with -mcpu=neoverse-v1. So you have to construct the bits only after you're guaranteed to have parsed *all* options on the command line. This is why we have to dynamically reconstruct the bits each time.

90

So we now do report an error when parsing the option with the operator= flag below, which means that this else case should really be unreachable now.

3510

Hi @dmgreen, I've tried to address your concerns about using tail-folding for very tight loops here. It's a little crude, but I've introduced an instruction threshold, below which we don't tail-fold. For the godbolt example you gave we won't use tail-folding, but for the SPEC2017 benchmarks (and many others) we will.

[dmgreen]

Thanks. This still makes me a bit nervous, considering what we know about predication and the performance results I've seen.

Can you explain more about why the limit is 10 instructions? As far as I could see the limit on interleaving in the vectorizer is a cost of 20, and with many instructions like geps, phis and branches being free that will be quite a bit more than 10 instructions. We could have the limit lower than the default for interleaving if that makes more sense, but 10 seems quite low.

[david-arm]

In D130618#4315481, @dmgreen wrote:

Thanks. This still makes me a bit nervous, considering what we know about predication and the performance results I've seen.

Can you explain more about why the limit is 10 instructions? As far as I could see the limit on interleaving in the vectorizer is a cost of 20, and with many instructions like geps, phis and branches being free that will be quite a bit more than 10 instructions. We could have the limit lower than the default for interleaving if that makes more sense, but 10 seems quite low.

I guess we have to choose a limit somewhere. Whatever number we pick there will always be an example that proves it's bad. The goal here is not to make it the fastest for every single case, which is not really possible as shown by holes we sometimes find in our cost model. We want to make it good overall for the majority of cases. This patch and the number chosen here are not fixed - these are things that we can evolve over time based on real evidence.

I specifically chose 10 as a starting point because that's based on the example you gave me:

void test(float *x, float *y, int n) {
  for (int i = 0; i < n; i++)
    x[i] = -y[i];
}

which has 9 LLVM IR instructions. Ignoring tail-folding completely, if you build this with current HEAD of LLVM you'll notice that interleaving is slightly faster than not interleaving. However, when you change this to:

void test(float *x, float *y, int n) {
  for (int i = 0; i < n; i++)
    x[i] -= y[i];
}

suddenly the non-interleaved version (-mllvm -force-vector-interleave=1) becomes 18% faster than the interleaved version on neoverse-v1. There is only one extra instruction in the loop, which makes that 10. This means that today we already have an upstream performance bug caused by a poor interleaving choice. Just by chance more than anything else, this loop becomes faster when applying this patch.

Using your suggestion of 20 proved detrimental for x264 performance on neoverse-v1, with ~5% performance regression.

[dmgreen]

Those number do not match what I see. The second version is 25% slower without interleaving in the example I tried. It can be dependant on the trip count though, or there could be something else going on. I will try to contact you to see where the discrepancy might lie.

[david-arm]

• Rebase.

[david-arm]

• Bumped up the instruction threshold from 10 -> 15 in order to reduce the risk of causing possible regressions for tight loops. This still leaves us with a 7% win for x264, but there is also no longer a regression for parest because the loops are now too small to be tail-folded. This is more by chance however, since the problem with tail-folding for parest seems unrelated to loop size and due to problems with code quality.

[paulwalker-arm]

I'm happy with this patch. It's perhaps not 100% clear cut but on average we're seeing more (important?) wins than losses and there's nothing binding here. We're in the middle of the LLVM 17 development cycle and thus if evidence presents a more valuable configuration then fine let's use that. Making the change also gives us more eyes on the testing and benchmarking of tail folding, which I see as a good thing.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

42

Please can this be tail-folding to match the existing command line option naming.

[dmgreen]

Thanks, a threshold of 15 will certainly help mitigate some of the issues in common cases. The results I have are still not looking great in places, but like we discussed some of this will be dependant on alignment and other issues like multiple ir instructions becoming a single aarch64 instruction. And there are certainly cases where this is making improvements, even if it still makes me nervous.

On a more technical note I don't think this works without setting -sve-tail-folding=default as nothing will set NeedsDefault.

[david-arm]

• Rebase.

[david-arm]

• Rename instruction threshold option to "sve-tail-folding-insn-threshold"

[david-arm]

In D130618#4341270, @dmgreen wrote:

On a more technical note I don't think this works without setting -sve-tail-folding=default as nothing will set NeedsDefault.

Thanks for pointing this out @dmgreen! I noticed this too - I've fixed the parent NFC patch so that we always set NeedsDefault=true in the absence of any explicit user-specified option. I added an extra RUN line in sve-tail-folding-option.ll to test this too.

[dmgreen]

Thanks

[david-arm]

• Rebase

llvm/lib/Target/AArch64/AArch64Subtarget.cpp

case NeoverseV2:
     MaxBytesForLoopAlignment = 16;
     VScaleForTuning = 1;
     break;
   case NeoverseV1:
     PrefFunctionAlignment = Align(16);
     PrefLoopAlignment = Align(32);
     MaxBytesForLoopAlignment = 16;
     VScaleForTuning = 2;
+    setSVETailFoldingDefaultOpts(TFSimple);
     break;
   case Neoverse512TVB:
     PrefFunctionAlignment = Align(16);
     VScaleForTuning = 1;
     MaxInterleaveFactor = 4;
     break;
   case Saphira:
     MaxInterleaveFactor = 4;

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

static cl::opt<bool> EnableFalkorHWPFUnrollFix("enable-falkor-hwpf-unroll-fix",
                                                cl::init(true), cl::Hidden);
 
 static cl::opt<unsigned> SVEGatherOverhead("sve-gather-overhead", cl::init(10),
                                            cl::Hidden);
 
 static cl::opt<unsigned> SVEScatterOverhead("sve-scatter-overhead",
                                             cl::init(10), cl::Hidden);
 
 namespace {
 class TailFoldingKind {
 private:
-  uint8_t Bits = 0; // Currently defaults to disabled.
+  uint8_t Bits;
 
 public:
-  enum TailFoldingOpts {
-    TFDisabled = 0x0,
-    TFReductions = 0x01,
-    TFRecurrences = 0x02,
-    TFReverse = 0x04,
-    TFSimple = 0x80,
-    TFAll = TFReductions | TFRecurrences | TFReverse | TFSimple
-  };
+  TailFoldingKind(uint8_t Bits) : Bits(Bits) {}
+
+  void add(uint8_t Bit) { Bits |= Bit; }
+
+  void remove(uint8_t Bit) { Bits &= ~Bit; }
+
+  operator uint8_t() const { return Bits; }
 
-  void operator=(const std::string &Val) {
-    if (Val.empty())
-      return;
-    SmallVector<StringRef, 6> TailFoldTypes;
-    StringRef(Val).split(TailFoldTypes, '+', -1, false);
+class TailFoldingOption {
+private:
+  uint8_t DefaultBits = TFDisabled;
+  std::string OrigVal;
+  SmallVector<StringRef, 4> TailFoldTypes;
+
+  uint8_t getBits() const {
+    if (!TailFoldTypes.size())
+      return DefaultBits;
+
+    TailFoldingKind Bits(0);
     for (auto TailFoldType : TailFoldTypes) {
       if (TailFoldType == "disabled")
-        Bits = 0;
+        Bits.remove(TFAll);
       else if (TailFoldType == "all")
-        Bits = TFAll;
+        Bits.add(TFAll);
       else if (TailFoldType == "default")
-        Bits = 0; // Currently defaults to never tail-folding.
+        Bits.add(DefaultBits);
       else if (TailFoldType == "simple")
-        add(TFSimple);
+        Bits.add(TFSimple);
       else if (TailFoldType == "reductions")
-        add(TFReductions);
+        Bits.add(TFReductions);
       else if (TailFoldType == "recurrences")
-        add(TFRecurrences);
+        Bits.add(TFRecurrences);
       else if (TailFoldType == "reverse")
-        add(TFReverse);
+        Bits.add(TFReverse);
       else if (TailFoldType == "noreductions")
-        remove(TFReductions);
+        Bits.remove(TFReductions);
       else if (TailFoldType == "norecurrences")
-        remove(TFRecurrences);
+        Bits.remove(TFRecurrences);
       else if (TailFoldType == "noreverse")
-        remove(TFReverse);
-      else {
+        Bits.remove(TFReverse);
+      else
+        llvm_unreachable("No! That's impossible!");
+    }
+
+    return Bits;
+  }
+
+public:
+
+  void setDefault(uint8_t V) { DefaultBits = V; }
+
+  void operator=(const std::string &Val) {
+    if (Val.empty())
+      return;
+
+    OrigVal = Val;
+    StringRef(OrigVal).split(TailFoldTypes, '+', -1, false);
+    for (auto TailFoldType : TailFoldTypes) {
+      if (TailFoldType != "disabled" && TailFoldType != "all" &&
+          TailFoldType != "default" && TailFoldType != "simple" &&
+          TailFoldType != "reductions" && TailFoldType != "recurrences" &&
+          TailFoldType != "reverse" && TailFoldType != "noreductions" &&
+          TailFoldType != "norecurrences" && TailFoldType != "noreverse") {
         errs()
             << "invalid argument " << TailFoldType.str()
             << " to -sve-tail-folding=; each element must be one of: disabled, "
                "all, default, simple, reductions, noreductions, recurrences, "
-               "norecurrences\n";
+               "norecurrences, reverse, noreverse\n";
       }
     }
   }
 
-  operator uint8_t() const { return Bits; }
-
-  void add(uint8_t Flag) { Bits |= Flag; }
-  void remove(uint8_t Flag) { Bits &= ~Flag; }
+  bool satisfies(TailFoldingKind Required) const {
+    return (getBits() & Required) == Required;
+  }
 };
 } // namespace
 
-TailFoldingKind TailFoldingKindLoc;
+TailFoldingOption TailFoldingOptionLoc;
+
+namespace llvm {
+void setSVETailFoldingDefaultOpts(uint8_t V) {
+  TailFoldingOptionLoc.setDefault(V);
+}
+} // namespace llvm
 
-cl::opt<TailFoldingKind, true, cl::parser<std::string>> SVETailFolding(
+cl::opt<TailFoldingOption, true, cl::parser<std::string>> SVETailFolding(
     "sve-tail-folding",
     cl::desc(
         "Control the use of vectorisation using tail-folding for SVE:"
         "\ndisabled    No loop types will vectorize using tail-folding"
         "\ndefault     Uses the default tail-folding settings for the target "
         "CPU"
         "\nall         All legal loop types will vectorize using tail-folding"
         "\nsimple      Use tail-folding for simple loops (not reductions or "
         "recurrences)"
         "\nreductions  Use tail-folding for loops containing reductions"
         "\nrecurrences Use tail-folding for loops containing fixed order "
         "recurrences"
         "\nreverse     Use tail-folding for loops requiring reversed "
         "predicates"),
-    cl::location(TailFoldingKindLoc));
+    cl::location(TailFoldingOptionLoc));
 
 // Experimental option that will only be fully functional when the
 // code-generator is changed to use SVE instead of NEON for all fixed-width
 // operations.
 static cl::opt<bool> EnableFixedwidthAutovecInStreamingMode(
     "enable-fixedwidth-autovec-in-streaming-mode", cl::init(false), cl::Hidden);
 
 // Experimental option that will only be fully functional when the cost-model
for (Instruction &I : *BB) {
           return true;
       }
     }
   }
   return false;
 }
 
 bool AArch64TTIImpl::preferPredicateOverEpilogue(TailFoldingInfo *TFI) {
-  if (!ST->hasSVE() || TailFoldingKindLoc == TailFoldingKind::TFDisabled)
+  if (!ST->hasSVE())
     return false;
 
   // We don't currently support vectorisation with interleaving for SVE - with
   // such loops we're better off not using tail-folding. This gives us a chance
   // to fall back on fixed-width vectorisation using NEON's ld2/st2/etc.
   if (TFI->IAI->hasGroups())
     return false;
 
-  TailFoldingKind Required; // Defaults to 0.
+  TailFoldingKind Required(0);
   if (TFI->LVL->getReductionVars().size())
-    Required.add(TailFoldingKind::TFReductions);
+    Required.add(TFReductions);
   if (TFI->LVL->getFixedOrderRecurrences().size())
-    Required.add(TailFoldingKind::TFRecurrences);
+    Required.add(TFRecurrences);
 
   // We call this to discover whether any load/store pointers in the loop have
   // negative strides. This will require extra work to reverse the loop
   // predicate, which may be expensive.
   if (containsDecreasingPointers(TFI->LVL->getLoop(),
                                  TFI->LVL->getPredicatedScalarEvolution()))
-    Required.add(TailFoldingKind::TFReverse);
+    Required.add(TFReverse);
   if (!Required)
-    Required.add(TailFoldingKind::TFSimple);
+    Required.add(TFSimple);
 
-  return (TailFoldingKindLoc & Required) == Required;
+  return TailFoldingOptionLoc.satisfies(Required);
 }
 
 InstructionCost
 AArch64TTIImpl::getScalingFactorCost(Type *Ty, GlobalValue *BaseGV,
                                      int64_t BaseOffset, bool HasBaseReg,
                                      int64_t Scale, unsigned AddrSpace) const {
   // Scaling factors are not free at all.
   // Operands                     | Rt Latency

llvm/lib/Target/AArch64/Utils/AArch64BaseInfo.h

case 64:
     return AArch64SVEPredPattern::vl64;
   case 128:
     return AArch64SVEPredPattern::vl128;
   case 256:
     return AArch64SVEPredPattern::vl256;
   }
 }
 
+enum TailFoldingOpts : uint8_t {
+  TFDisabled = 0x0,
+  TFReductions = 0x01,
+  TFRecurrences = 0x02,
+  TFReverse = 0x04,
+  TFSimple = 0x80,
+  TFAll = TFReductions | TFRecurrences | TFSimple | TFReverse
+};
+
+void setSVETailFoldingDefaultOpts(uint8_t);
+
 namespace AArch64ExactFPImm {
   struct ExactFPImm {
     const char *Name;
     int Enum;
     const char *Repr;
   };
 #define GET_EXACTFPIMM_DECL
 #include "AArch64GenSystemOperands.inc"

llvm/test/Transforms/LoopVectorize/AArch64/sve-epilog-vect-vscale-tune.ll

 ; RUN: opt -S -passes=loop-vectorize,instsimplify -force-vector-interleave=1 \
-; RUN:   -mcpu=neoverse-v1 < %s | FileCheck %s --check-prefix=CHECK-EPILOG
-; RUN: opt -S -passes=loop-vectorize,instsimplify -force-vector-interleave=1 \
-; RUN:   -mcpu=neoverse-v1 < %s | FileCheck %s --check-prefix=CHECK-EPILOG
+; RUN:   -mcpu=neoverse-v1 -sve-tail-folding=disabled < %s | FileCheck %s --check-prefix=CHECK-EPILOG
 ; RUN: opt -S -passes=loop-vectorize,instsimplify -force-vector-interleave=1 \
 ; RUN:   -mcpu=neoverse-v2 < %s | FileCheck %s --check-prefix=CHECK-NO-EPILOG
 ; RUN: opt -S -passes=loop-vectorize,instsimplify -force-vector-interleave=1 \
 ; RUN:   -mcpu=cortex-x2 < %s | FileCheck %s --check-prefix=CHECK-NO-EPILOG
 
 target triple = "aarch64-unknown-linux-gnu"
 
 define void @foo(ptr noalias nocapture readonly %p, ptr noalias nocapture %q, i64 %len) #0 {

llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll

 ; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=disabled -S | FileCheck %s -check-prefix=CHECK-NOTF
 ; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=default -S | FileCheck %s -check-prefix=CHECK-NOTF
+; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -S | FileCheck %s -check-prefix=CHECK-NOTF
 ; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=all -S | FileCheck %s -check-prefix=CHECK-TF
-; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=disabled+simple+reductions+recurrences+reverse -S | FileCheck %s -check-prefix=CHECK-TF
+; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=default+disabled+simple+reductions+recurrences+reverse -S | FileCheck %s -check-prefix=CHECK-TF
+; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -S -mcpu=neoverse-v1 -sve-tail-folding=default+reductions+recurrences+reverse | FileCheck %s -check-prefix=CHECK-TF
 ; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=all+noreductions -S | FileCheck %s -check-prefix=CHECK-TF-NORED
 ; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=all+norecurrences -S | FileCheck %s -check-prefix=CHECK-TF-NOREC
 ; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=all+noreverse -S | FileCheck %s -check-prefix=CHECK-TF-NOREV
 ; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -sve-tail-folding=reductions -S | FileCheck %s -check-prefix=CHECK-TF-ONLYRED
+; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -S -sve-tail-folding=default -mcpu=neoverse-v1 | FileCheck %s -check-prefix=CHECK-NEOVERSE-V1
+; RUN: opt -opaque-pointers=0 < %s -passes=loop-vectorize -S -mcpu=neoverse-v1 -sve-tail-folding=default | FileCheck %s -check-prefix=CHECK-NEOVERSE-V1
 
 target triple = "aarch64-unknown-linux-gnu"
 
 define void @simple_memset(i32 %val, i32* %ptr, i64 %n) #0 {
 ; CHECK-NOTF-LABEL: @simple_memset(
 ; CHECK-NOTF:       vector.ph:
 ; CHECK-NOTF:         %[[INSERT:.*]] = insertelement <vscale x 4 x i32> poison, i32 %val, i64 0
 ; CHECK-NOTF:         %[[SPLAT:.*]] = shufflevector <vscale x 4 x i32> %[[INSERT]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer

 ; CHECK-TF-ONLYRED-LABEL: @simple_memset(
 ; CHECK-TF-ONLYRED:       vector.ph:
 ; CHECK-TF-ONLYRED:         %[[INSERT:.*]] = insertelement <vscale x 4 x i32> poison, i32 %val, i64 0
 ; CHECK-TF-ONLYRED:         %[[SPLAT:.*]] = shufflevector <vscale x 4 x i32> %[[INSERT]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
 ; CHECK-TF-ONLYRED:       vector.body:
 ; CHECK-TF-ONLYRED-NOT:     %{{.*}} = phi <vscale x 4 x i1>
 ; CHECK-TF-ONLYRED:         store <vscale x 4 x i32> %[[SPLAT]], <vscale x 4 x i32>*
 
+; CHECK-NEOVERSE-V1-LABEL: @simple_memset(
+; CHECK-NEOVERSE-V1:       vector.ph:
+; CHECK-NEOVERSE-V1:         %[[INSERT:.*]] = insertelement <vscale x 4 x i32> poison, i32 %val, i64 0
+; CHECK-NEOVERSE-V1:         %[[SPLAT:.*]] = shufflevector <vscale x 4 x i32> %[[INSERT]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEOVERSE-V1:       vector.body:
+; CHECK-NEOVERSE-V1:         %[[ACTIVE_LANE_MASK:.*]] = phi <vscale x 4 x i1>
+; CHECK-NEOVERSE-V1:         call void @llvm.masked.store.nxv4i32.p0nxv4i32(<vscale x 4 x i32> %[[SPLAT]], {{.*}} %[[ACTIVE_LANE_MASK]]
+
 entry:
   br label %while.body
 
 while.body:                                       ; preds = %while.body, %entry
   %index = phi i64 [ %index.next, %while.body ], [ 0, %entry ]
   %gep = getelementptr i32, i32* %ptr, i64 %index
   store i32 %val, i32* %gep
   %index.next = add nsw i64 %index, 1

 ; CHECK-TF-ONLYRED:       vector.body:
 ; CHECK-TF-ONLYRED:         %[[ACTIVE_LANE_MASK:.*]] = phi <vscale x 4 x i1>
 ; CHECK-TF-ONLYRED:         %[[VEC_PHI:.*]] = phi <vscale x 4 x float>
 ; CHECK-TF-ONLYRED:         %[[LOAD:.*]] = call <vscale x 4 x float> @llvm.masked.load.nxv4f32.p0nxv4f32(<vscale x 4 x float>{{.*}} %[[ACTIVE_LANE_MASK]]
 ; CHECK-TF-ONLYRED:         %[[ADD:.*]] = fadd fast <vscale x 4 x float> %[[LOAD]]
 ; CHECK-TF-ONLYRED:         %[[SEL:.*]] = select fast <vscale x 4 x i1> %[[ACTIVE_LANE_MASK]], <vscale x 4 x float> %[[ADD]], <vscale x 4 x float> %[[VEC_PHI]]
 ; CHECK-TF-ONLYRED:       middle.block:
 ; CHECK-TF-ONLYRED-NEXT:    call fast float @llvm.vector.reduce.fadd.nxv4f32(float -0.000000e+00, <vscale x 4 x float> %[[SEL]])
+
+; CHECK-NEOVERSE-V1-LABEL: @fadd_red_fast
+; CHECK-NEOVERSE-V1:       vector.body:
+; CHECK-NEOVERSE-V1-NOT:     %{{.*}} = phi <vscale x 4 x i1>
+; CHECK-NEOVERSE-V1:         %[[LOAD:.*]] = load <vscale x 4 x float>
+; CHECK-NEOVERSE-V1:         %[[ADD:.*]] = fadd fast <vscale x 4 x float> %[[LOAD]]
+; CHECK-NEOVERSE-V1:       middle.block:
+; CHECK-NEOVERSE-V1-NEXT:    call fast float @llvm.vector.reduce.fadd.nxv4f32(float -0.000000e+00, <vscale x 4 x float> %[[ADD]])
+
 entry:
   br label %for.body
 
 for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %sum.07 = phi float [ 0.000000e+00, %entry ], [ %add, %for.body ]
   %arrayidx = getelementptr inbounds float, float* %a, i64 %iv
   %0 = load float, float* %arrayidx, align 4

 ; CHECK-TF-ONLYRED:       vector.body:
 ; CHECK-TF-ONLYRED-NOT:     %{{.*}} = phi <vscale x 4 x i1>
 ; CHECK-TF-ONLYRED:         %[[VECTOR_RECUR:.*]] = phi <vscale x 4 x i32> [ %[[RECUR_INIT]], %vector.ph ], [ %[[LOAD:.*]], %vector.body ]
 ; CHECK-TF-ONLYRED:         %[[LOAD]] = load <vscale x 4 x i32>
 ; CHECK-TF-ONLYRED:         %[[SPLICE:.*]] = call <vscale x 4 x i32> @llvm.experimental.vector.splice.nxv4i32(<vscale x 4 x i32> %[[VECTOR_RECUR]], <vscale x 4 x i32> %[[LOAD]], i32 -1)
 ; CHECK-TF-ONLYRED:         %[[ADD:.*]] = add nsw <vscale x 4 x i32> %[[LOAD]], %[[SPLICE]]
 ; CHECK-TF-ONLYRED:         store <vscale x 4 x i32> %[[ADD]]
 
+; CHECK-NEOVERSE-V1-LABEL: @add_recur
+; CHECK-NEOVERSE-V1:       entry:
+; CHECK-NEOVERSE-V1:         %[[PRE:.*]] = load i32, i32* %src, align 4
+; CHECK-NEOVERSE-V1:       vector.ph:
+; CHECK-NEOVERSE-V1:         %[[RECUR_INIT:.*]] = insertelement <vscale x 4 x i32> poison, i32 %[[PRE]]
+; CHECK-NEOVERSE-V1:       vector.body:
+; CHECK-NEOVERSE-V1-NOT:     %{{.*}} = phi <vscale x 4 x i1>
+; CHECK-NEOVERSE-V1:         %[[VECTOR_RECUR:.*]] = phi <vscale x 4 x i32> [ %[[RECUR_INIT]], %vector.ph ], [ %[[LOAD:.*]], %vector.body ]
+; CHECK-NEOVERSE-V1:         %[[LOAD]] = load <vscale x 4 x i32>
+; CHECK-NEOVERSE-V1:         %[[SPLICE:.*]] = call <vscale x 4 x i32> @llvm.experimental.vector.splice.nxv4i32(<vscale x 4 x i32> %[[VECTOR_RECUR]], <vscale x 4 x i32> %[[LOAD]], i32 -1)
+; CHECK-NEOVERSE-V1:         %[[ADD:.*]] = add nsw <vscale x 4 x i32> %[[LOAD]], %[[SPLICE]]
+; CHECK-NEOVERSE-V1:         store <vscale x 4 x i32> %[[ADD]]
+
 entry:
   %.pre = load i32, i32* %src, align 4
   br label %for.body
 
 for.body:                                         ; preds = %entry, %for.body
   %0 = phi i32 [ %1, %for.body ], [ %.pre, %entry ]
   %i.010 = phi i64 [ %add, %for.body ], [ 0, %entry ]
   %add = add nuw nsw i64 %i.010, 1

 ; CHECK-TF-NOREC:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
 
 ; CHECK-TF-NOREV-LABEL: @interleave(
 ; CHECK-TF-NOREV:       vector.body:
 ; CHECK-TF-NOREV:         %[[LOAD:.*]] = load <8 x float>, <8 x float>
 ; CHECK-TF-NOREV:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
 ; CHECK-TF-NOREV:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
 
+; CHECK-NEOVERSE-V1-LABEL: @interleave(
+; CHECK-NEOVERSE-V1:       vector.body:
+; CHECK-NEOVERSE-V1:         %[[LOAD:.*]] = load <8 x float>, <8 x float>
+; CHECK-NEOVERSE-V1:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
+; CHECK-NEOVERSE-V1:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %entry, %for.body
   %i.021 = phi i64 [ %inc, %for.body ], [ 0, %entry ]
   %mul = shl nuw nsw i64 %i.021, 1
   %arrayidx = getelementptr inbounds float, float* %src, i64 %mul
   %0 = load float, float* %arrayidx, align 4

 ; CHECK-TF-NORED:         %[[MASKED_LOAD:.*]] = call <vscale x 2 x double> @llvm.masked.load.nxv2f64.p0nxv2f64(<vscale x 2 x double>{{.*}} <vscale x 2 x i1> %reverse
 
 ; CHECK-TF-NOREC-LABEL: @reverse(
 ; CHECK-TF-NOREC:       vector.body:
 ; CHECK-TF-NOREC:         %[[ACTIVE_LANE_MASK:.*]] = phi <vscale x 2 x i1>
 ; CHECK-TF-NOREC:         %[[REVERSE_MASK:.*]] = call <vscale x 2 x i1> @llvm.experimental.vector.reverse.nxv2i1(<vscale x 2 x i1> %[[ACTIVE_LANE_MASK]])
 ; CHECK-TF-NOREC:         %[[MASKED_LOAD:.*]] = call <vscale x 2 x double> @llvm.masked.load.nxv2f64.p0nxv2f64(<vscale x 2 x double>{{.*}} <vscale x 2 x i1> %reverse
 
+; CHECK-TF-NEOVERSE-V1-LABEL: @reverse(
+; CHECK-TF-NEOVERSE-V1:       vector.body:
+; CHECK-TF-NEOVERSE-V1-NOT:     %{{.*}} = phi <vscale x 4 x i1>
+; CHECK-TF-NEOVERSE-V1:         %[[LOAD:.*]] = load <vscale x 2 x double>, <vscale x 2 x double>* %18, align 8
+; CHECK-TF-NEOVERSE-V1:         %{{.*}} = call <vscale x 2 x double> @llvm.experimental.vector.reverse.nxv2f64(<vscale x 2 x double> %[[LOAD]])
+
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %entry, %for.body
   %indvars.iv = phi i64 [ 1023, %entry ], [ %indvars.iv.next, %for.body ]
   %arrayidx = getelementptr inbounds double, double* %src, i64 %indvars.iv
   %0 = load double, double* %arrayidx, align 8
   %add = fadd double %0, 1.000000e+00