[SVE] Add support to vectorize_width loop pragma for scalable vectors

[david-arm]

Summary

This patch adds support for two new variants of the vectorize_width
pragma:

1. vectorize_width(X[, fixed|scalable]) where an optional second

parameter is passed to the vectorize_width pragma, which indicates if
the user wishes to use fixed width or scalable vectorization. For
example the user can now write something like:

#pragma clang loop vectorize_width(4, fixed)

or

#pragma clang loop vectorize_width(4, scalable)

In the absence of a second parameter it is assumed the user wants
fixed width vectorization, in order to maintain compatibility with
existing code.

2. vectorize_width(fixed|scalable) where the width is left unspecified,

but the user hints what type of vectorization they prefer, either
fixed width or scalable.

I have implemented this by making use of the LLVM loop hint attribute:

llvm.loop.vectorize.scalable.enable

Tests were added to

clang/test/CodeGenCXX/pragma-loop.cpp

for both the 'fixed' and 'scalable' optional parameter.

See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html

[fhahn]

clang/lib/Parse/ParsePragma.cpp

1098

nit: unrelated change?

clang/lib/Sema/SemaStmtAttr.cpp

145

Is there a way to only accept fixed_width/scalable for targets that support it? Not sure if we have enough information here, but we might be able to reject it eg per target basis or something

[david-arm]

• Rebase.

[david-arm]

clang/lib/Sema/SemaStmtAttr.cpp

145

Hi @fhahn, I think if possible we'd prefer not to reject scalable vectors at this point. Theoretically there is no reason why we can't perform scalable vectorisation for targets that don't have hardware support for scalable vectors. In this case it simply means that vscale is 1. If you want we could add some kind of opt-remark in the vectoriser that says something like "target does not support scalable vectors, vectorising for vscale=1"?

[sdesmalen]

LGTM

clang/lib/Sema/SemaStmtAttr.cpp

145

I agree with @david-arm that we shouldn't prevent this in the front-end. Even if the architecture may not support scalable vectors natively, there may still be reasons to want to create scalable vectors in IR, for example to have more portable IR.

clang/test/CodeGenCXX/pragma-loop.cpp

167

out of curiosity, is there a particular reason you're testing it with a do-while loop instead of a shorter for-loop like the tests in for_template_constant_expression_test ?

[fhahn]

clang/lib/Sema/SemaStmtAttr.cpp

145

Hm, I am just a bit worried that it might be a bit confusing to users that do not know what scalable vectors are (it is obvious when knowing all about SVE, but I would assume most people don't necessarily know what this means). I guess that is not the biggest deal, as long vectorize_width(X, scalable) works for every target.

Even if the architecture may not support scalable vectors natively, there may still be reasons to want to create scalable vectors in IR, for example to have more portable IR.

Sure, but there are so many other target-specific things encoded that make the IR really un-portable between targets. Granted, it is not impossible to convert IR between some architectures (as in arm64_32)

[sdesmalen]

clang/docs/LanguageExtensions.rst

3049

Can you add a comment saying that the use of "scalable" is still experimental and is currently only intended to work for targets that support scalable vectors?

clang/lib/Sema/SemaStmtAttr.cpp

145

Sorry, forgot to reply to this.

Hm, I am just a bit worried that it might be a bit confusing to users that do not know what scalable vectors are (it is obvious when knowing all about SVE, but I would assume most people don't necessarily know what this means). I guess that is not the biggest deal, as long vectorize_width(X, scalable) works for every target.

At the moment this feature is still experimental, so I don't think any target would be able to return true to the question if this is supported :) That said, I agree that the compiler shouldn't crash for other targets after support in the loop-vectorizer stops being experimental. So I'm changing my mind here, and am happy to go with your suggestion to ignore the flag for other targets. When some default mechanism is added to lower scalable vectors to fixed-width vectors (for targets that don't natively support them), this check can probably be removed. @david-arm can you add some target hook to ignore the hint?

Sure, but there are so many other target-specific things encoded that make the IR really un-portable between targets. Granted, it is not impossible to convert IR between some architectures (as in arm64_32)

I didn't mean portable between targets, but more as keeping the length of the vector agnostic in IR and leaving it until code-generation to pick a suitable/available vector extension, so that the same IR could be used to generate code for Neon or 256bit SVE for example. This is more a hypothetical use-case at the moment though.

[sdesmalen]

clang/include/clang/Basic/DiagnosticSemaKinds.td

939

From what I can see, the vectorize_width flag is not ignored, only the scalable property is. That means this should be:

'scalable' not supported by the target so assuming 'fixed' instead.

clang/lib/Sema/SemaStmtAttr.cpp

148

If the target does not support scalable vectors, it currently assumes "fixed". If we want to stick with that approach, the diagnostic message should be changed (see my other comment). The alternative is dropping the hint entirely by returning nullptr and changing the diagnostic message to say the hint is ignored. I could live with both options. @fhahn do you have a preference here?

nit: to reduce nesting, can you hoist this out one level, e.g.

if (StateLoc && StateLoc->Ident & ...)
  State = LoopHintAttr::ScalableNumeric;
else
  State = LoopHintAttr::Numeric;

if (State == LoopHintAttr::ScalableNumeric &&
    !S.Context.getTargetInfo().supportsScalableVectors()) {
  S.Diag(....);
  State = LoopHintAttr::Numeric;
}

[david-arm]

I'll hold off on any more changes for now to give @fhahn a chance to reply to your comment @sdesmalen about the fallback behaviour when scalable vectorisation is unsupported.

clang/include/clang/Basic/DiagnosticSemaKinds.td

939

OK. I guess it's just when the warning comes out it appears at the start of the line so I wanted to emphasise that this relates to the scalable property passed to the vectorize_width attribute (rather than other attributes) as there could potentially be several pragmas on one line. I think it would be good to mention the vectorize_width pragma/attribute somewhere in the warning message to make it clear. I'll see if I can reword it.

[sdesmalen]

This comment was removed by sdesmalen.

[c-rhodes]

clang/docs/LanguageExtensions.rst

3032

nit: s/__value__/_value_/

[SjoerdMeijer]

I am very sorry that I am late to this... but I do have some concerns.

The concern that I have is that we extend vecorize_width with a scalable/fixed boolean, but there are more vectorisation pragma that set vectorisation options which imply enabling vectorisation:

Pragmas setting transformation options imply the transformation is enabled, as if it was enabled via the corresponding transformation pragma (e.g. vectorize(enable))

Thus, unless I miss something, I don't think this should be an option to vectorize_width, but to me it looks like we need a separate one, e.g.:

vectorize_scalable(enable|disable)

what do you think?

[david-arm]

Hi @SjoerdMeijer I think that given we now support scalable vectors we thought it made sense to be able to specify whether the user wants 'fixed' or 'scalable' vectorisation along with the vector width, although without specifying the additional property the default continues to remain 'fixed'. However, what you said about having a vectorize_scalable pragma is correct and we are intending to also add a pragma like this in a future patch.

[SjoerdMeijer]

In D89031#2391160, @david-arm wrote:

Hi @SjoerdMeijer I think that given we now support scalable vectors we thought it made sense to be able to specify whether the user wants 'fixed' or 'scalable' vectorisation along with the vector width, although without specifying the additional property the default continues to remain 'fixed'. However, what you said about having a vectorize_scalable pragma is correct and we are intending to also add a pragma like this in a future patch.

Okay, I haven't looked at the implementation to be honest, but am just trying to understand the different use cases of this first.
I just seem to be missing or not understanding why fixed/scalable is an option to only vectorize_width, why not to vectorize(enable) or just a separate one like vectorize_scalable? By making scalable/fixed and option to vectorize_width, you can't toggle this for other pragmas like interleave(enable) that enable vectorisation, which would be inconsistent? It also seems to be more work to me to do this first for vectorize_width, and then fix up other pragmas later. But I might be missing something (obivous) here.

[sdesmalen]

In D89031#2391248, @SjoerdMeijer wrote:

In D89031#2391160, @david-arm wrote:

Hi @SjoerdMeijer I think that given we now support scalable vectors we thought it made sense to be able to specify whether the user wants 'fixed' or 'scalable' vectorisation along with the vector width, although without specifying the additional property the default continues to remain 'fixed'. However, what you said about having a vectorize_scalable pragma is correct and we are intending to also add a pragma like this in a future patch.

Okay, I haven't looked at the implementation to be honest, but am just trying to understand the different use cases of this first.
I just seem to be missing or not understanding why fixed/scalable is an option to only vectorize_width, why not to vectorize(enable) or just a separate one like vectorize_scalable? By making scalable/fixed and option to vectorize_width, you can't toggle this for other pragmas like interleave(enable) that enable vectorisation, which would be inconsistent? It also seems to be more work to me to do this first for vectorize_width, and then fix up other pragmas later. But I might be missing something (obivous) here.

Hi @SjoerdMeijer, all valid and good questions. We think it makes sense to allow specifying explicitly what the meaning of '4' is when specifying the width. So that vectorize_width(4, fixed) means vectorizing with <4 x eltty> and vectorize_width(4, scalable) means vectorizing with <vscale x 4 x eltty>. Like @david-arm said, we also plan to add something like vectorize_style(fixed|scalable). This approach should be fully complementary to vectorize_with so that it would be possible to have:

// Use scalable vectors, but leave it to the cost-model to choose the most efficient N in <vscale x N x eltty>.
// If the pragma is not specified, it defaults to vectorize_style(fixed).
#pragma clang loop vectorize_style(scalable)

// Use <4 x eltty>
#pragma clang loop vectorize_width(4, fixed)

// Use <vscale x 4 x eltty>
#pragma clang loop vectorize_width(4, scalable)

// If vectorize_style(scalable) is specified, then use <vscale x 4 x eltty>, otherwise <4 x eltty>
#pragma clang loop vectorize_width(4)                           // uses <4 x eltty>
#pragma clang loop vectorize_width(4) vectorize_style(scalable) // uses <vscale x 4 x eltty>

// Conflicting options, clang should print diagnostic and error or ignore the hint.
#pragma clang loop vectorize_width(4, fixed) vectorize_style(scalable)

I hope that gives a bit more context.

[SjoerdMeijer]

This approach should be fully complementary to vectorize_with so that it would be possible to have:

// Use scalable vectors, but leave it to the cost-model to choose the most efficient N in <vscale x N x eltty>.
// If the pragma is not specified, it defaults to vectorize_style(fixed).
#pragma clang loop vectorize_style(scalable)

// Use <4 x eltty>
#pragma clang loop vectorize_width(4, fixed)

// Use <vscale x 4 x eltty>
#pragma clang loop vectorize_width(4, scalable)

// If vectorize_style(scalable) is specified, then use <vscale x 4 x eltty>, otherwise <4 x eltty>
#pragma clang loop vectorize_width(4)                           // uses <4 x eltty>
#pragma clang loop vectorize_width(4) vectorize_style(scalable) // uses <vscale x 4 x eltty>

// Conflicting options, clang should print diagnostic and error or ignore the hint.
#pragma clang loop vectorize_width(4, fixed) vectorize_style(scalable)

I hope that gives a bit more context.

Ok, thanks for clarifying that!

If:

// Use <vscale x 4 x eltty>
#pragma clang loop vectorize_width(4, scalable)

is equivalent to:

// uses <vscale x 4 x eltty>
#pragma clang loop vectorize_width(4) vectorize_style(scalable)

then I think that illustrates that I don't see the point of extending vectorize_width because we still can't express scalable vectorisation for:

// <VF x eltty>
#pragma clang loop vectorize_predicate(enable)

and also for interleave_count(4)?

Again, when the idea is to have vectorize_style anyway, wouldn't it be easier not to bother extending vectorize_width and just go for vectorize_style? It allows you to specify fixed/scalable vectorisation in one way, and avoids having conflicting options.

The other thing I thought about: this is extending an existing user-facing pragma, and notifying the list would probably be best thing to do.

[paulwalker-arm]

As I see it there are a bunch of pragmas that all enable vectorisation, with each pragma providing a unit of information. One component of this information is the vectorisation factor hint provided by vectorize_width.

With the introduction of scalable vectors this hint is using the wrong datatype and thus needs to be updated to allow vectorize_width(#num,[fixed|scalable]) and vectorize_width([fixed|scalable]) along side the existing vectorize_width(#num) representation that effectively becomes an alias to vectorize_width(#num, fixed).

Doing this means all existing usages work as expected and there's now extra power to better guide the chosen vectorisation factor.

[SjoerdMeijer]

Because I was not understanding, we have discussed this further offline.

I think the conclusion was: pragma vectorize_width controls the vectorisation vector VF in <vscale x VF x elty>. where vscale is not just a separate thing but it defines a VectorType. That's why it would make sense to attach scalable|fixed to vectorize_width. I agree with this and seems reasonable.

I still don't see how the proposed extension here allows you to specify fixed width vectorisation for:

#pragma clang loop interleave_count(4)

targeting SVE and would appreciate if someone can comment on this example, but I won't be holding up this work anymore as this might be addressed later.

[SjoerdMeijer]

Thanks @david-arm for posting this proposal to the cfe list.
My confusion has been cleared up. The (new) proposal is to have:

1. vectorize_width(X) where X is an integer.
2. vectorize_width(X, fixed|scalable)
3. vectorize_width(fixed|scalable)

And with that 3rd option I agree that this allows us to express everything we want, and this patch needs adapting to this new proposal (just stating the obvious for clarity/completeness)

If scalable is used on a target that doesn't support this, a warning and falling back to fixed seems like the right thing to do.

I did have concerns about this, similarly like @fhahn:

Hm, I am just a bit worried that it might be a bit confusing to users that do not know what scalable vectors are (it is obvious when knowing all about SVE, but I would assume most people don't necessarily know what this means). I guess that is not the biggest deal, as long vectorize_width(X, scalable) works for every target.

But since the new scalable option is opt-in, people don't need know about this if they don't want/need to, this should (hopefully) not be the case.

[aaron.ballman]

clang/include/clang/Basic/Attr.td

3302–3303

Should the documentation in AttrDocs.td be updated for this change?

[david-arm]

clang/include/clang/Basic/Attr.td

3302–3303

Hi @aaron.ballman I had a look at LoopHintDocs in AttrDocs.td and it didn't explicitly mention these states, i.e. "assume_safety", "numeric", etc., so I'm not sure if it's necessary to add anything there?

[aaron.ballman]

clang/include/clang/Basic/Attr.td

3302–3303

Oh, I see now, we're deferring to the documentation in the language extensions document. I suppose that's fine as far as this patch goes, sorry for the noise.

[david-arm]

Hi everyone, I realise that most people have probably been on holiday recently, but just a gentle ping here to see if anyone could take another look? Thanks!

[SjoerdMeijer]

LGTM, perhaps wait a day with committing in case there are more comments.

clang/include/clang/Basic/Attr.td

3302–3303

Nit: formatting, exceeding 80 columns?

3303

same?

[sdesmalen]

clang/docs/LanguageExtensions.rst

3034

nit: s/In this case//

3036

nit: Another use of vectorize_width is

3038–3040

nit: In both variants of the pragma the vectorizer may decide to fall back on fixed width vectorization if the target does not support scalable vectors.

clang/include/clang/Basic/DiagnosticParseKinds.td

1390

use vectorize_width(X, fixed) or vectorize_width(X, scalable)
(it may otherwise lead to confusion whether fixed/scalable needs quotes, same below)

clang/lib/AST/AttrImpl.cpp

46 ↗

(On Diff #312802)

is there always a value, even when "vectorize_width(scalable)" is specified?

clang/lib/CodeGen/CGLoopInfo.cpp

754–756

is that not something to fix in the code that conditionally sets vectorize.enable later on instead of working around it here?

[david-arm]

clang/lib/CodeGen/CGLoopInfo.cpp

754–756

I did originally try to do that, but I had trouble with it and found it broke other places too. It ended up being simpler to fix it here, but I can play around with it again. Even if this is still the simplest solution I can come back with a more detailed explanation at least!

[david-arm]

• Updated documentation as per review comments.
• Fixed an issue with using value->prettyPrint on a null ptr.
• Reworked the code that sets vectorize.enable.

[sdesmalen]

LGTM, thanks for all the changes @david-arm!

clang/lib/AST/AttrImpl.cpp

46 ↗

(On Diff #315121)

nit:

if (value) {
  value->printPretty(OS, nullptr, Policy);
  OS << ", ";
}
OS << (state == ScalableWidth ? "scalable" : "fixed";

?

clang/lib/CodeGen/CGLoopInfo.cpp

307

nit: != 1 (it should be functionally the same because the >1 is already caught above, but this is specifically testing that VF=1 (scalar) is not specified)

clang/docs/LanguageExtensions.rst

 .. code-block:: c++
 
   #pragma clang loop vectorize(enable)
   #pragma clang loop interleave(enable)
   for(...) {
     ...
   }
 
-The vector width is specified by ``vectorize_width(_value_)`` and the interleave
-count is specified by ``interleave_count(_value_)``, where
+The vector width is specified by
+``vectorize_width(_value_[, fixed|scalable])``, where __value__ is a positive
+integer and the type of vectorization can be specified with an optional
+second parameter. In this case 'fixed' is the default and refers to fixed width
+vectorization, whereas 'scalable' indicates the compiler should use scalable
+vectors instead. The 'scalable' option is currently experimental and is only
+intended to work for targets that support scalable vectors.
+
+The interleave count is specified by ``interleave_count(_value_)``, where
 _value_ is a positive integer. This is useful for specifying the optimal
 width/count of the set of target architectures supported by your application.
 
 .. code-block:: c++
 
   #pragma clang loop vectorize_width(2)
   #pragma clang loop interleave_count(2)
   for(...) {
     ...
   }
 
 Specifying a width/count of 1 disables the optimization, and is equivalent to
 ``vectorize(disable)`` or ``interleave(disable)``.
 
 Vector predication is enabled by ``vectorize_predicate(enable)``, for example:
 
 .. code-block:: c++
 

clang/include/clang/Basic/Attr.td

let Args = [EnumArgument<"Option", "OptionType",
                            "unroll", "unroll_count", "unroll_and_jam", "unroll_and_jam_count",
                            "pipeline", "pipeline_initiation_interval", "distribute",
                            "vectorize_predicate"],
                           ["Vectorize", "VectorizeWidth", "Interleave", "InterleaveCount",
                            "Unroll", "UnrollCount", "UnrollAndJam", "UnrollAndJamCount",
                            "PipelineDisabled", "PipelineInitiationInterval", "Distribute",
                            "VectorizePredicate"]>,
               EnumArgument<"State", "LoopHintState",
-                           ["enable", "disable", "numeric", "assume_safety", "full"],
-                           ["Enable", "Disable", "Numeric", "AssumeSafety", "Full"]>,
+                           ["enable", "disable", "numeric", "scalable_numeric", "assume_safety", "full"],
+                           ["Enable", "Disable", "Numeric", "ScalableNumeric", "AssumeSafety", "Full"]>,
               ExprArgument<"Value">];
 
   let AdditionalMembers = [{
   static const char *getOptionName(int Option) {
     switch(Option) {
     case Vectorize: return "vectorize";
     case VectorizeWidth: return "vectorize_width";
     case Interleave: return "interleave";

clang/include/clang/Basic/DiagnosticParseKinds.td

 // Pragma loop support.
 def err_pragma_loop_missing_argument : Error<
   "missing argument; expected %select{an integer value|"
   "'enable'%select{|, 'full'}1%select{|, 'assume_safety'}2 or 'disable'}0">;
 def err_pragma_loop_invalid_option : Error<
   "%select{invalid|missing}0 option%select{ %1|}0; expected vectorize, "
   "vectorize_width, interleave, interleave_count, unroll, unroll_count, "
   "pipeline, pipeline_initiation_interval, vectorize_predicate, or distribute">;
+def err_pragma_loop_invalid_vectorize_option : Error<
+  "vectorize_width loop hint malformed; use 'vectorize_width(X, scalable)' or "
+  "'vectorize_width(X, fixed)' where X is an integer">;
 
 def err_pragma_fp_invalid_option : Error<
   "%select{invalid|missing}0 option%select{ %1|}0; expected 'contract', 'reassociate' or 'exceptions'">;
 def err_pragma_fp_invalid_argument : Error<
   "unexpected argument '%0' to '#pragma clang fp %1'; expected "
   "%select{"
   "'fast' or 'on' or 'off'|"
   "'on' or 'off'|"

clang/include/clang/Basic/DiagnosticSemaKinds.td

def note_pragma_attribute_region_ends_here : Note<
   "'#pragma clang attribute push' regions ends here">;
 def err_pragma_attribute_no_pop_eof : Error<"unterminated "
   "'#pragma clang attribute push' at end of file">;
 def note_pragma_attribute_applied_decl_here : Note<
   "when applied to this declaration">;
 def err_pragma_attr_attr_no_push : Error<
   "'#pragma clang attribute' attribute with no matching "
   "'#pragma clang attribute push'">;
+def warn_pragma_attribute_scalable_unused : Warning<
+  "ignoring scalable vectorize_width flag due to lack of target support">,
+  InGroup<PragmaClangAttribute>;
 
 /// Objective-C parser diagnostics
 def err_duplicate_class_def : Error<
   "duplicate interface definition for class %0">;
 def err_undef_superclass : Error<
   "cannot find interface declaration for %0, superclass of %1">;
 def err_forward_superclass : Error<
   "attempting to use the forward class %0 as superclass of %1">;

clang/include/clang/Basic/TargetInfo.h

public:
 
   /// Identify whether this target supports multiversioning of functions,
   /// which requires support for cpu_supports and cpu_is functionality.
   bool supportsMultiVersioning() const { return getTriple().isX86(); }
 
   /// Identify whether this target supports IFuncs.
   bool supportsIFunc() const { return getTriple().isOSBinFormatELF(); }
 
+  virtual bool supportsScalableVectors() const { return false; }
+
   // Validate the contents of the __builtin_cpu_supports(const char*)
   // argument.
   virtual bool validateCpuSupports(StringRef Name) const { return false; }
 
   // Return the target-specific priority for features/cpus/vendors so
   // that they can be properly sorted for checking.
   virtual unsigned multiVersionSortPriority(StringRef Name) const {
     return 0;

clang/lib/Basic/Targets/AArch64.h

public:
   }
 
   int getEHDataRegisterNumber(unsigned RegNo) const override;
 
   const char *getBFloat16Mangling() const override { return "u6__bf16"; };
   bool hasInt128Type() const override;
 
   bool hasExtIntType() const override { return true; }
+
+  bool supportsScalableVectors() const override { return FPU & SveMode; }
 };
 
 class LLVM_LIBRARY_VISIBILITY AArch64leTargetInfo : public AArch64TargetInfo {
 public:
   AArch64leTargetInfo(const llvm::Triple &Triple, const TargetOptions &Opts);
 
   void getTargetDefines(const LangOptions &Opts,
                             MacroBuilder &Builder) const override;

clang/lib/CodeGen/CGLoopInfo.h

 #ifndef LLVM_CLANG_LIB_CODEGEN_CGLOOPINFO_H
 #define LLVM_CLANG_LIB_CODEGEN_CGLOOPINFO_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/TypeSize.h"
 
 namespace llvm {
 class BasicBlock;
 class Instruction;
 class MDNode;
 } // end namespace llvm
 
 namespace clang {
struct LoopAttributes {
 
   /// Value for llvm.loop.unroll_and_jam.* metadata (enable, disable, or full).
   LVEnableState UnrollAndJamEnable;
 
   /// Value for llvm.loop.vectorize.predicate metadata
   LVEnableState VectorizePredicateEnable;
 
   /// Value for llvm.loop.vectorize.width metadata.
-  unsigned VectorizeWidth;
+  llvm::ElementCount VectorizeWidth;
 
   /// Value for llvm.loop.interleave.count metadata.
   unsigned InterleaveCount;
 
   /// llvm.unroll.
   unsigned UnrollCount;
 
   /// llvm.unroll.
public:
   }
 
   /// Set the next pushed loop unroll_and_jam state.
   void setUnrollAndJamState(const LoopAttributes::LVEnableState &State) {
     StagedAttrs.UnrollAndJamEnable = State;
   }
 
   /// Set the vectorize width for the next loop pushed.
-  void setVectorizeWidth(unsigned W) { StagedAttrs.VectorizeWidth = W; }
+  void setVectorizeWidth(unsigned W, bool IsScalable = false) {
+    StagedAttrs.VectorizeWidth = llvm::ElementCount::get(W, IsScalable);
+  }
 
   /// Set the interleave count for the next loop pushed.
   void setInterleaveCount(unsigned C) { StagedAttrs.InterleaveCount = C; }
 
   /// Set the unroll count for the next loop pushed.
   void setUnrollCount(unsigned C) { StagedAttrs.UnrollCount = C; }
 
   /// \brief Set the unroll count for the next loop pushed.

clang/lib/CodeGen/CGLoopInfo.cpp

LoopInfo::createLoopVectorizeMetadata(const LoopAttributes &Attrs,
                                       bool &HasUserTransforms) {
   LLVMContext &Ctx = Header->getContext();
 
   Optional<bool> Enabled;
   if (Attrs.VectorizeEnable == LoopAttributes::Disable)
     Enabled = false;
   else if (Attrs.VectorizeEnable != LoopAttributes::Unspecified ||
            Attrs.VectorizePredicateEnable != LoopAttributes::Unspecified ||
-           Attrs.InterleaveCount != 0 || Attrs.VectorizeWidth != 0)
+           Attrs.InterleaveCount != 0 || Attrs.VectorizeWidth.isNonZero())
     Enabled = true;
 
   if (Enabled != true) {
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
       NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
       NewLoopProperties.push_back(
           MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
LoopInfo::createLoopVectorizeMetadata(const LoopAttributes &Attrs,
   SmallVector<Metadata *, 4> Args;
   Args.push_back(nullptr);
   Args.append(LoopProperties.begin(), LoopProperties.end());
 
   // Setting vectorize.predicate
   bool IsVectorPredicateEnabled = false;
   if (Attrs.VectorizePredicateEnable != LoopAttributes::Unspecified &&
       Attrs.VectorizeEnable != LoopAttributes::Disable &&
-      Attrs.VectorizeWidth < 1) {
+      Attrs.VectorizeWidth.getKnownMinValue() < 1) {
 
     IsVectorPredicateEnabled =
         (Attrs.VectorizePredicateEnable == LoopAttributes::Enable);
 
     Metadata *Vals[] = {
         MDString::get(Ctx, "llvm.loop.vectorize.predicate.enable"),
         ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt1Ty(Ctx),
                                                  IsVectorPredicateEnabled))};
     Args.push_back(MDNode::get(Ctx, Vals));
   }
 
   // Setting vectorize.width
-  if (Attrs.VectorizeWidth > 0) {
-    Metadata *Vals[] = {
-        MDString::get(Ctx, "llvm.loop.vectorize.width"),
-        ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
-                                                 Attrs.VectorizeWidth))};
+  if (Attrs.VectorizeWidth.getKnownMinValue() > 0) {
+    if (Attrs.VectorizeWidth.isScalable()) {
+      Metadata *ECArgs[] = {
+          ConstantAsMetadata::get(
+              ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
+                               Attrs.VectorizeWidth.getKnownMinValue())),
+          ConstantAsMetadata::get(ConstantInt::get(
+              llvm::Type::getInt1Ty(Ctx), Attrs.VectorizeWidth.isScalable()))};
+      MDNode *EC = MDNode::get(Ctx, ECArgs);
+      Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.vectorize.width"), EC};
       Args.push_back(MDNode::get(Ctx, Vals));
+    } else {
+      Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.vectorize.width"),
+                          ConstantAsMetadata::get(ConstantInt::get(
+                              llvm::Type::getInt32Ty(Ctx),
+                              Attrs.VectorizeWidth.getKnownMinValue()))};
+
+      Args.push_back(MDNode::get(Ctx, Vals));
+    }
   }
 
   // Setting interleave.count
   if (Attrs.InterleaveCount > 0) {
     Metadata *Vals[] = {
         MDString::get(Ctx, "llvm.loop.interleave.count"),
         ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
                                                  Attrs.InterleaveCount))};
     Args.push_back(MDNode::get(Ctx, Vals));
   }
 
   // vectorize.enable is set if:
   // 1) loop hint vectorize.enable is set, or
   // 2) it is implied when vectorize.predicate is set, or
   // 3) it is implied when vectorize.width is set.
   if (Attrs.VectorizeEnable != LoopAttributes::Unspecified ||
-      IsVectorPredicateEnabled ||
-      Attrs.VectorizeWidth > 1 ) {
+      IsVectorPredicateEnabled || Attrs.VectorizeWidth.getKnownMinValue() > 1 ||
+      (Attrs.VectorizeWidth.getKnownMinValue() == 1 &&
+       Attrs.VectorizeWidth.isScalable())) {
     bool AttrVal = Attrs.VectorizeEnable != LoopAttributes::Disable;
     Args.push_back(
         MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
                           ConstantAsMetadata::get(ConstantInt::get(
                               llvm::Type::getInt1Ty(Ctx), AttrVal))}));
   }
 
   if (FollowupHasTransforms)
     Args.push_back(MDNode::get(
LoopProperties.insert(LoopProperties.end(), AdditionalLoopProperties.begin(),
                         AdditionalLoopProperties.end());
   return createFullUnrollMetadata(Attrs, LoopProperties, HasUserTransforms);
 }
 
 LoopAttributes::LoopAttributes(bool IsParallel)
     : IsParallel(IsParallel), VectorizeEnable(LoopAttributes::Unspecified),
       UnrollEnable(LoopAttributes::Unspecified),
       UnrollAndJamEnable(LoopAttributes::Unspecified),
-      VectorizePredicateEnable(LoopAttributes::Unspecified), VectorizeWidth(0),
-      InterleaveCount(0), UnrollCount(0), UnrollAndJamCount(0),
+      VectorizePredicateEnable(LoopAttributes::Unspecified),
+      VectorizeWidth(ElementCount::getFixed(0)), InterleaveCount(0),
+      UnrollCount(0), UnrollAndJamCount(0),
       DistributeEnable(LoopAttributes::Unspecified), PipelineDisabled(false),
       PipelineInitiationInterval(0) {}
 
 void LoopAttributes::clear() {
   IsParallel = false;
-  VectorizeWidth = 0;
+  VectorizeWidth = ElementCount::getFixed(0);
   InterleaveCount = 0;
   UnrollCount = 0;
   UnrollAndJamCount = 0;
   VectorizeEnable = LoopAttributes::Unspecified;
   UnrollEnable = LoopAttributes::Unspecified;
   UnrollAndJamEnable = LoopAttributes::Unspecified;
   VectorizePredicateEnable = LoopAttributes::Unspecified;
   DistributeEnable = LoopAttributes::Unspecified;
   PipelineDisabled = false;
   PipelineInitiationInterval = 0;
 }
 
 LoopInfo::LoopInfo(BasicBlock *Header, const LoopAttributes &Attrs,
                    const llvm::DebugLoc &StartLoc, const llvm::DebugLoc &EndLoc,
                    LoopInfo *Parent)
     : Header(Header), Attrs(Attrs), StartLoc(StartLoc), EndLoc(EndLoc),
       Parent(Parent) {
 
   if (Attrs.IsParallel) {
     // Create an access group for this loop.
     LLVMContext &Ctx = Header->getContext();
     AccGroup = MDNode::getDistinct(Ctx, {});
   }
 
-  if (!Attrs.IsParallel && Attrs.VectorizeWidth == 0 &&
+  if (!Attrs.IsParallel && Attrs.VectorizeWidth.isZero() &&
       Attrs.InterleaveCount == 0 && Attrs.UnrollCount == 0 &&
       Attrs.UnrollAndJamCount == 0 && !Attrs.PipelineDisabled &&
       Attrs.PipelineInitiationInterval == 0 &&
       Attrs.VectorizePredicateEnable == LoopAttributes::Unspecified &&
       Attrs.VectorizeEnable == LoopAttributes::Unspecified &&
       Attrs.UnrollEnable == LoopAttributes::Unspecified &&
       Attrs.UnrollAndJamEnable == LoopAttributes::Unspecified &&
       Attrs.DistributeEnable == LoopAttributes::Unspecified && !StartLoc &&
if (Parent && (Parent->Attrs.UnrollAndJamEnable ||
     if (!Parent->UnrollAndJamInnerFollowup) {
       // Splitting the attributes into a BeforeJam and an AfterJam part will
       // stop 'llvm.loop.isvectorized' (generated by vectorization in BeforeJam)
       // to be forwarded to the AfterJam part. We detect the situation here and
       // add it manually.
       SmallVector<Metadata *, 1> BeforeLoopProperties;
       if (BeforeJam.VectorizeEnable != LoopAttributes::Unspecified ||
           BeforeJam.VectorizePredicateEnable != LoopAttributes::Unspecified ||
-          BeforeJam.InterleaveCount != 0 || BeforeJam.VectorizeWidth != 0)
+          BeforeJam.InterleaveCount != 0 || BeforeJam.VectorizeWidth.isNonZero())
         BeforeLoopProperties.push_back(
             MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.isvectorized")));
 
       bool InnerFollowupHasTransform = false;
       MDNode *InnerFollowup = createMetadata(AfterJam, BeforeLoopProperties,
                                              InnerFollowupHasTransform);
       if (InnerFollowupHasTransform)
         Parent->UnrollAndJamInnerFollowup = InnerFollowup;
case LoopHintAttr::Full:
       case LoopHintAttr::Distribute:
       case LoopHintAttr::PipelineDisabled:
       case LoopHintAttr::PipelineInitiationInterval:
       case LoopHintAttr::VectorizePredicate:
         llvm_unreachable("Options cannot be used with 'full' hint.");
         break;
       }
       break;
+    case LoopHintAttr::ScalableNumeric:
+      switch (Option) {
+      case LoopHintAttr::VectorizeWidth:
+        setVectorizeWidth(ValueInt, true);
+        break;
+      default:
+        llvm_unreachable("Options cannot be used with 'scalable' hint.");
+        break;
+      }
+      break;
     case LoopHintAttr::Numeric:
       switch (Option) {
       case LoopHintAttr::VectorizeWidth:
         setVectorizeWidth(ValueInt);
         break;
       case LoopHintAttr::InterleaveCount:
         setInterleaveCount(ValueInt);
         break;
       case LoopHintAttr::UnrollCount:
         setUnrollCount(ValueInt);
         break;
       case LoopHintAttr::UnrollAndJamCount:
         setUnrollAndJamCount(ValueInt);
         break;
       case LoopHintAttr::PipelineInitiationInterval:
         setPipelineInitiationInterval(ValueInt);
         break;
       case LoopHintAttr::Unroll:
       case LoopHintAttr::UnrollAndJam:
       case LoopHintAttr::VectorizePredicate:

clang/lib/Parse/ParsePragma.cpp

return std::string(llvm::StringSwitch<StringRef>(Str)
                          .Case("unroll", Str)
                          .Default(""));
 }
 
 bool Parser::HandlePragmaLoopHint(LoopHint &Hint) {
   assert(Tok.is(tok::annot_pragma_loop_hint));
   PragmaLoopHintInfo *Info =
       static_cast<PragmaLoopHintInfo *>(Tok.getAnnotationValue());
 
   IdentifierInfo *PragmaNameInfo = Info->PragmaName.getIdentifierInfo();
   Hint.PragmaNameLoc = IdentifierLoc::create(
       Actions.Context, Info->PragmaName.getLocation(), PragmaNameInfo);
 
   // It is possible that the loop hint has no option identifier, such as
   // #pragma unroll(4).
   IdentifierInfo *OptionInfo = Info->Option.is(tok::identifier)
                                    ? Info->Option.getIdentifierInfo()
bool Parser::HandlePragmaLoopHint(LoopHint &Hint) {
   } else {
     // Enter constant expression including eof terminator into token stream.
     PP.EnterTokenStream(Toks, /*DisableMacroExpansion=*/false,
                         /*IsReinject=*/false);
     ConsumeAnnotationToken();
 
     ExprResult R = ParseConstantExpression();
 
+    if (OptionInfo && OptionInfo->getName() == "vectorize_width" &&
+        Tok.is(tok::comma)) {
+      PP.Lex(Tok); // ,
+
+      SourceLocation StateLoc = Tok.getLocation();
+      IdentifierInfo *StateInfo = Tok.getIdentifierInfo();
+      StringRef IsScalableStr = StateInfo->getName();
+
+      if (IsScalableStr != "scalable" && IsScalableStr != "fixed") {
+        Diag(Tok.getLocation(), diag::err_pragma_loop_invalid_vectorize_option);
+        return false;
+      }
+      PP.Lex(Tok); // Identifier
+
+      Hint.StateLoc =
+          IdentifierLoc::create(Actions.Context, StateLoc, StateInfo);
+    }
+
     // Tokens following an error in an ill-formed constant expression will
     // remain in the token stream and must be removed.
     if (Tok.isNot(tok::eof)) {
       Diag(Tok.getLocation(), diag::warn_pragma_extra_tokens_at_eol)
           << PragmaLoopHintString(Info->PragmaName, Info->Option);
       while (Tok.isNot(tok::eof))
         ConsumeAnyToken();
     }

clang/lib/Sema/SemaStmtAttr.cpp

 //===--- SemaStmtAttr.cpp - Statement Attribute Handling ------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //  This file implements stmt-related attribute processing.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/AST/EvaluatedExprVisitor.h"
 #include "clang/Sema/SemaInternal.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
 #include "clang/Sema/DelayedDiagnostic.h"
 #include "clang/Sema/Lookup.h"
 #include "clang/Sema/ScopeInfo.h"
 #include "llvm/ADT/StringExtras.h"
 
 using namespace clang;
 using namespace sema;
 
Option = llvm::StringSwitch<LoopHintAttr::OptionType>(
                  .Case("interleave_count", LoopHintAttr::InterleaveCount)
                  .Case("unroll", LoopHintAttr::Unroll)
                  .Case("unroll_count", LoopHintAttr::UnrollCount)
                  .Case("pipeline", LoopHintAttr::PipelineDisabled)
                  .Case("pipeline_initiation_interval",
                        LoopHintAttr::PipelineInitiationInterval)
                  .Case("distribute", LoopHintAttr::Distribute)
                  .Default(LoopHintAttr::Vectorize);
-    if (Option == LoopHintAttr::VectorizeWidth ||
-        Option == LoopHintAttr::InterleaveCount ||
+    if (Option == LoopHintAttr::VectorizeWidth) {
+      assert(ValueExpr && "Attribute must have a valid value expression.");
+      if (S.CheckLoopHintExpr(ValueExpr, St->getBeginLoc()))
+        return nullptr;
+      if (StateLoc && StateLoc->Ident && StateLoc->Ident->isStr("scalable")) {
+        if (!S.Context.getTargetInfo().supportsScalableVectors()) {
+          S.Diag(St->getBeginLoc(), diag::warn_pragma_attribute_scalable_unused);
+          State = LoopHintAttr::Numeric;
+        } else
+          State = LoopHintAttr::ScalableNumeric;
+      } else
+        State = LoopHintAttr::Numeric;
+    } else if (Option == LoopHintAttr::InterleaveCount ||
                Option == LoopHintAttr::UnrollCount ||
                Option == LoopHintAttr::PipelineInitiationInterval) {
       assert(ValueExpr && "Attribute must have a valid value expression.");
       if (S.CheckLoopHintExpr(ValueExpr, St->getBeginLoc()))
         return nullptr;
       State = LoopHintAttr::Numeric;
     } else if (Option == LoopHintAttr::Vectorize ||
                Option == LoopHintAttr::Interleave ||
                Option == LoopHintAttr::VectorizePredicate ||
                Option == LoopHintAttr::Unroll ||

clang/test/CodeGenCXX/pragma-loop.cpp

-// RUN: %clang_cc1 -triple x86_64-apple-darwin -std=c++11 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -std=c++11 -emit-llvm -o - %s 2>%t | FileCheck %s
+// RUN: FileCheck --check-prefix=CHECK-SCALABLE %s < %t
 
 // Verify while loop is recognized after sequence of pragma clang loop directives.
 void while_test(int *List, int Length) {
   // CHECK: define {{.*}} @_Z10while_test
   int i = 0;
 
 #pragma clang loop vectorize(enable)
 #pragma clang loop interleave_count(4)

 void template_test(double *List, int Length) {
   double Value = 10;
 
   for_template_test<double>(List, Length, Value);
   for_template_define_test<double, int>(List, Length, Value);
   for_template_constant_expression_test<double, 2, 4, 8>(List, Length);
 }
 
+// Verify for loop is performing fixed width vectorization
+void for_test_fixed(int *List, int Length) {
+#pragma clang loop vectorize_width(16, fixed) interleave_count(4) unroll(disable) distribute(disable)
+  for (int i = 0; i < Length; i++) {
+    // CHECK: br label {{.*}}, !llvm.loop ![[LOOP_15:.*]]
+    List[i] = i * 2;
+  }
+}
+
+// Verify for loop rejects scalable vectorization due to lack of target support
+// CHECK-SCALABLE: ignoring scalable vectorize_width flag due to lack of target support
+void for_test_scalable(int *List, int Length) {
+#pragma clang loop vectorize_width(16, scalable) interleave_count(4) unroll(disable) distribute(disable)
+  for (int i = 0; i < Length; i++) {
+    // CHECK: br label {{.*}}, !llvm.loop ![[LOOP_16:.*]]
+    // CHECK-SVE: br label {{.*}}, !llvm.loop ![[LOOP_16_SVE:.*]]
+    List[i] = i * 2;
+  }
+}
+
 // CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], ![[UNROLL_FULL:.*]]}
 // CHECK: ![[UNROLL_FULL]] = !{!"llvm.loop.unroll.full"}
 
 // CHECK: ![[LOOP_2]] = distinct !{![[LOOP_2]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_8:.*]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
 // CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
 // CHECK: ![[DISTRIBUTE_DISABLE]] = !{!"llvm.loop.distribute.enable", i1 false}
 // CHECK: ![[WIDTH_8]] = !{!"llvm.loop.vectorize.width", i32 8}
 // CHECK: ![[INTERLEAVE_4]] = !{!"llvm.loop.interleave.count", i32 4}

 // CHECK: ![[LOOP_13]] = distinct !{![[LOOP_13]], ![[WIDTH_8:.*]], ![[INTERLEAVE_16:.*]], ![[VECTORIZE_ENABLE]], ![[FOLLOWUP_VECTOR_13:.*]]}
 // CHECK: ![[INTERLEAVE_16]] = !{!"llvm.loop.interleave.count", i32 16}
 // CHECK: ![[FOLLOWUP_VECTOR_13]] = !{!"llvm.loop.vectorize.followup_all", ![[AFTER_VECTOR_13:.*]]}
 // CHECK: ![[AFTER_VECTOR_13]] = distinct !{![[AFTER_VECTOR_13]], ![[ISVECTORIZED:.*]], ![[UNROLL_32:.*]]}
 // CHECK: ![[UNROLL_32]] = !{!"llvm.loop.unroll.count", i32 32}
 
 // CHECK: ![[LOOP_14]] = distinct !{![[LOOP_14]], ![[WIDTH_10:.*]], ![[VECTORIZE_ENABLE]]}
 // CHECK: ![[WIDTH_10]] = !{!"llvm.loop.vectorize.width", i32 10}
+
+// CHECK: ![[LOOP_15]] = distinct !{![[LOOP_15]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_16_FIXED:.*]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
+// CHECK: ![[WIDTH_16_FIXED]] = !{!"llvm.loop.vectorize.width", i32 16}
+
+// CHECK: ![[LOOP_16]] = distinct !{![[LOOP_16]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_16_FIXED:.*]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}

clang/test/CodeGenCXX/pragma-scalable-loop.cpp

+// RUN: %clang_cc1 -triple aarch64-linux-gnu -target-feature +sve -std=c++11 -emit-llvm -o - %s | FileCheck %s
+
+// Verify do loop is performing scalable vectorization
+void for_test_scalable(int *List, int Length) {
+#pragma clang loop vectorize_width(16, scalable) interleave_count(4) unroll(disable) distribute(disable)
+  for (int i = 0; i < Length; i++) {
+    // CHECK: br label {{.*}}, !llvm.loop ![[LOOP_1:.*]]
+    List[i] = i * 2;
+  }
+}
+
+// CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_16_SCALABLE:.*]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
+// CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
+// CHECK: ![[DISTRIBUTE_DISABLE]] = !{!"llvm.loop.distribute.enable", i1 false}
+// CHECK: ![[WIDTH_16_SCALABLE]] = !{!"llvm.loop.vectorize.width", ![[ELEMENT_COUNT_16_SCALABLE:.*]]}
+// CHECK: ![[ELEMENT_COUNT_16_SCALABLE]] = !{i32 16, i1 true}
+// CHECK: ![[INTERLEAVE_4]] = !{!"llvm.loop.interleave.count", i32 4}
+// CHECK: ![[VECTORIZE_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}