[SVE] Add support to vectorize_width loop pragma for scalable vectors

[david-arm]

Summary

This patch adds support for two new variants of the vectorize_width
pragma:

1. vectorize_width(X[, fixed|scalable]) where an optional second

parameter is passed to the vectorize_width pragma, which indicates if
the user wishes to use fixed width or scalable vectorization. For
example the user can now write something like:

#pragma clang loop vectorize_width(4, fixed)

or

#pragma clang loop vectorize_width(4, scalable)

In the absence of a second parameter it is assumed the user wants
fixed width vectorization, in order to maintain compatibility with
existing code.

2. vectorize_width(fixed|scalable) where the width is left unspecified,

but the user hints what type of vectorization they prefer, either
fixed width or scalable.

I have implemented this by making use of the LLVM loop hint attribute:

llvm.loop.vectorize.scalable.enable

Tests were added to

clang/test/CodeGenCXX/pragma-loop.cpp

for both the 'fixed' and 'scalable' optional parameter.

See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html

[fhahn]

clang/lib/Parse/ParsePragma.cpp

1098

nit: unrelated change?

clang/lib/Sema/SemaStmtAttr.cpp

145

Is there a way to only accept fixed_width/scalable for targets that support it? Not sure if we have enough information here, but we might be able to reject it eg per target basis or something

[david-arm]

• Rebase.

[david-arm]

clang/lib/Sema/SemaStmtAttr.cpp

145

Hi @fhahn, I think if possible we'd prefer not to reject scalable vectors at this point. Theoretically there is no reason why we can't perform scalable vectorisation for targets that don't have hardware support for scalable vectors. In this case it simply means that vscale is 1. If you want we could add some kind of opt-remark in the vectoriser that says something like "target does not support scalable vectors, vectorising for vscale=1"?

[sdesmalen]

LGTM

clang/lib/Sema/SemaStmtAttr.cpp

145

I agree with @david-arm that we shouldn't prevent this in the front-end. Even if the architecture may not support scalable vectors natively, there may still be reasons to want to create scalable vectors in IR, for example to have more portable IR.

clang/test/CodeGenCXX/pragma-loop.cpp

166

out of curiosity, is there a particular reason you're testing it with a do-while loop instead of a shorter for-loop like the tests in for_template_constant_expression_test ?

[fhahn]

clang/lib/Sema/SemaStmtAttr.cpp

145

Hm, I am just a bit worried that it might be a bit confusing to users that do not know what scalable vectors are (it is obvious when knowing all about SVE, but I would assume most people don't necessarily know what this means). I guess that is not the biggest deal, as long vectorize_width(X, scalable) works for every target.

Even if the architecture may not support scalable vectors natively, there may still be reasons to want to create scalable vectors in IR, for example to have more portable IR.

Sure, but there are so many other target-specific things encoded that make the IR really un-portable between targets. Granted, it is not impossible to convert IR between some architectures (as in arm64_32)

[sdesmalen]

clang/docs/LanguageExtensions.rst

3131

Can you add a comment saying that the use of "scalable" is still experimental and is currently only intended to work for targets that support scalable vectors?

clang/lib/Sema/SemaStmtAttr.cpp

145

Sorry, forgot to reply to this.

Hm, I am just a bit worried that it might be a bit confusing to users that do not know what scalable vectors are (it is obvious when knowing all about SVE, but I would assume most people don't necessarily know what this means). I guess that is not the biggest deal, as long vectorize_width(X, scalable) works for every target.

At the moment this feature is still experimental, so I don't think any target would be able to return true to the question if this is supported :) That said, I agree that the compiler shouldn't crash for other targets after support in the loop-vectorizer stops being experimental. So I'm changing my mind here, and am happy to go with your suggestion to ignore the flag for other targets. When some default mechanism is added to lower scalable vectors to fixed-width vectors (for targets that don't natively support them), this check can probably be removed. @david-arm can you add some target hook to ignore the hint?

Sure, but there are so many other target-specific things encoded that make the IR really un-portable between targets. Granted, it is not impossible to convert IR between some architectures (as in arm64_32)

I didn't mean portable between targets, but more as keeping the length of the vector agnostic in IR and leaving it until code-generation to pick a suitable/available vector extension, so that the same IR could be used to generate code for Neon or 256bit SVE for example. This is more a hypothetical use-case at the moment though.

[sdesmalen]

clang/include/clang/Basic/DiagnosticSemaKinds.td

939 ↗

(On Diff #302773)

From what I can see, the vectorize_width flag is not ignored, only the scalable property is. That means this should be:

'scalable' not supported by the target so assuming 'fixed' instead.

clang/lib/Sema/SemaStmtAttr.cpp

148

If the target does not support scalable vectors, it currently assumes "fixed". If we want to stick with that approach, the diagnostic message should be changed (see my other comment). The alternative is dropping the hint entirely by returning nullptr and changing the diagnostic message to say the hint is ignored. I could live with both options. @fhahn do you have a preference here?

nit: to reduce nesting, can you hoist this out one level, e.g.

if (StateLoc && StateLoc->Ident & ...)
  State = LoopHintAttr::ScalableNumeric;
else
  State = LoopHintAttr::Numeric;

if (State == LoopHintAttr::ScalableNumeric &&
    !S.Context.getTargetInfo().supportsScalableVectors()) {
  S.Diag(....);
  State = LoopHintAttr::Numeric;
}

[david-arm]

I'll hold off on any more changes for now to give @fhahn a chance to reply to your comment @sdesmalen about the fallback behaviour when scalable vectorisation is unsupported.

clang/include/clang/Basic/DiagnosticSemaKinds.td

939 ↗

(On Diff #302773)

OK. I guess it's just when the warning comes out it appears at the start of the line so I wanted to emphasise that this relates to the scalable property passed to the vectorize_width attribute (rather than other attributes) as there could potentially be several pragmas on one line. I think it would be good to mention the vectorize_width pragma/attribute somewhere in the warning message to make it clear. I'll see if I can reword it.

[sdesmalen]

This comment was removed by sdesmalen.

[c-rhodes]

clang/docs/LanguageExtensions.rst

3111

nit: s/__value__/_value_/

[SjoerdMeijer]

I am very sorry that I am late to this... but I do have some concerns.

The concern that I have is that we extend vecorize_width with a scalable/fixed boolean, but there are more vectorisation pragma that set vectorisation options which imply enabling vectorisation:

Pragmas setting transformation options imply the transformation is enabled, as if it was enabled via the corresponding transformation pragma (e.g. vectorize(enable))

Thus, unless I miss something, I don't think this should be an option to vectorize_width, but to me it looks like we need a separate one, e.g.:

vectorize_scalable(enable|disable)

what do you think?

[david-arm]

Hi @SjoerdMeijer I think that given we now support scalable vectors we thought it made sense to be able to specify whether the user wants 'fixed' or 'scalable' vectorisation along with the vector width, although without specifying the additional property the default continues to remain 'fixed'. However, what you said about having a vectorize_scalable pragma is correct and we are intending to also add a pragma like this in a future patch.

[SjoerdMeijer]

In D89031#2391160, @david-arm wrote:

Hi @SjoerdMeijer I think that given we now support scalable vectors we thought it made sense to be able to specify whether the user wants 'fixed' or 'scalable' vectorisation along with the vector width, although without specifying the additional property the default continues to remain 'fixed'. However, what you said about having a vectorize_scalable pragma is correct and we are intending to also add a pragma like this in a future patch.

Okay, I haven't looked at the implementation to be honest, but am just trying to understand the different use cases of this first.
I just seem to be missing or not understanding why fixed/scalable is an option to only vectorize_width, why not to vectorize(enable) or just a separate one like vectorize_scalable? By making scalable/fixed and option to vectorize_width, you can't toggle this for other pragmas like interleave(enable) that enable vectorisation, which would be inconsistent? It also seems to be more work to me to do this first for vectorize_width, and then fix up other pragmas later. But I might be missing something (obivous) here.

[sdesmalen]

In D89031#2391248, @SjoerdMeijer wrote:

In D89031#2391160, @david-arm wrote:

Hi @SjoerdMeijer I think that given we now support scalable vectors we thought it made sense to be able to specify whether the user wants 'fixed' or 'scalable' vectorisation along with the vector width, although without specifying the additional property the default continues to remain 'fixed'. However, what you said about having a vectorize_scalable pragma is correct and we are intending to also add a pragma like this in a future patch.

Okay, I haven't looked at the implementation to be honest, but am just trying to understand the different use cases of this first.
I just seem to be missing or not understanding why fixed/scalable is an option to only vectorize_width, why not to vectorize(enable) or just a separate one like vectorize_scalable? By making scalable/fixed and option to vectorize_width, you can't toggle this for other pragmas like interleave(enable) that enable vectorisation, which would be inconsistent? It also seems to be more work to me to do this first for vectorize_width, and then fix up other pragmas later. But I might be missing something (obivous) here.

Hi @SjoerdMeijer, all valid and good questions. We think it makes sense to allow specifying explicitly what the meaning of '4' is when specifying the width. So that vectorize_width(4, fixed) means vectorizing with <4 x eltty> and vectorize_width(4, scalable) means vectorizing with <vscale x 4 x eltty>. Like @david-arm said, we also plan to add something like vectorize_style(fixed|scalable). This approach should be fully complementary to vectorize_with so that it would be possible to have:

// Use scalable vectors, but leave it to the cost-model to choose the most efficient N in <vscale x N x eltty>.
// If the pragma is not specified, it defaults to vectorize_style(fixed).
#pragma clang loop vectorize_style(scalable)

// Use <4 x eltty>
#pragma clang loop vectorize_width(4, fixed)

// Use <vscale x 4 x eltty>
#pragma clang loop vectorize_width(4, scalable)

// If vectorize_style(scalable) is specified, then use <vscale x 4 x eltty>, otherwise <4 x eltty>
#pragma clang loop vectorize_width(4)                           // uses <4 x eltty>
#pragma clang loop vectorize_width(4) vectorize_style(scalable) // uses <vscale x 4 x eltty>

// Conflicting options, clang should print diagnostic and error or ignore the hint.
#pragma clang loop vectorize_width(4, fixed) vectorize_style(scalable)

I hope that gives a bit more context.

[SjoerdMeijer]

This approach should be fully complementary to vectorize_with so that it would be possible to have:

// Use scalable vectors, but leave it to the cost-model to choose the most efficient N in <vscale x N x eltty>.
// If the pragma is not specified, it defaults to vectorize_style(fixed).
#pragma clang loop vectorize_style(scalable)

// Use <4 x eltty>
#pragma clang loop vectorize_width(4, fixed)

// Use <vscale x 4 x eltty>
#pragma clang loop vectorize_width(4, scalable)

// If vectorize_style(scalable) is specified, then use <vscale x 4 x eltty>, otherwise <4 x eltty>
#pragma clang loop vectorize_width(4)                           // uses <4 x eltty>
#pragma clang loop vectorize_width(4) vectorize_style(scalable) // uses <vscale x 4 x eltty>

// Conflicting options, clang should print diagnostic and error or ignore the hint.
#pragma clang loop vectorize_width(4, fixed) vectorize_style(scalable)

I hope that gives a bit more context.

Ok, thanks for clarifying that!

If:

// Use <vscale x 4 x eltty>
#pragma clang loop vectorize_width(4, scalable)

is equivalent to:

// uses <vscale x 4 x eltty>
#pragma clang loop vectorize_width(4) vectorize_style(scalable)

then I think that illustrates that I don't see the point of extending vectorize_width because we still can't express scalable vectorisation for:

// <VF x eltty>
#pragma clang loop vectorize_predicate(enable)

and also for interleave_count(4)?

Again, when the idea is to have vectorize_style anyway, wouldn't it be easier not to bother extending vectorize_width and just go for vectorize_style? It allows you to specify fixed/scalable vectorisation in one way, and avoids having conflicting options.

The other thing I thought about: this is extending an existing user-facing pragma, and notifying the list would probably be best thing to do.

[paulwalker-arm]

As I see it there are a bunch of pragmas that all enable vectorisation, with each pragma providing a unit of information. One component of this information is the vectorisation factor hint provided by vectorize_width.

With the introduction of scalable vectors this hint is using the wrong datatype and thus needs to be updated to allow vectorize_width(#num,[fixed|scalable]) and vectorize_width([fixed|scalable]) along side the existing vectorize_width(#num) representation that effectively becomes an alias to vectorize_width(#num, fixed).

Doing this means all existing usages work as expected and there's now extra power to better guide the chosen vectorisation factor.

[SjoerdMeijer]

Because I was not understanding, we have discussed this further offline.

I think the conclusion was: pragma vectorize_width controls the vectorisation vector VF in <vscale x VF x elty>. where vscale is not just a separate thing but it defines a VectorType. That's why it would make sense to attach scalable|fixed to vectorize_width. I agree with this and seems reasonable.

I still don't see how the proposed extension here allows you to specify fixed width vectorisation for:

#pragma clang loop interleave_count(4)

targeting SVE and would appreciate if someone can comment on this example, but I won't be holding up this work anymore as this might be addressed later.

[SjoerdMeijer]

Thanks @david-arm for posting this proposal to the cfe list.
My confusion has been cleared up. The (new) proposal is to have:

1. vectorize_width(X) where X is an integer.
2. vectorize_width(X, fixed|scalable)
3. vectorize_width(fixed|scalable)

And with that 3rd option I agree that this allows us to express everything we want, and this patch needs adapting to this new proposal (just stating the obvious for clarity/completeness)

If scalable is used on a target that doesn't support this, a warning and falling back to fixed seems like the right thing to do.

I did have concerns about this, similarly like @fhahn:

Hm, I am just a bit worried that it might be a bit confusing to users that do not know what scalable vectors are (it is obvious when knowing all about SVE, but I would assume most people don't necessarily know what this means). I guess that is not the biggest deal, as long vectorize_width(X, scalable) works for every target.

But since the new scalable option is opt-in, people don't need know about this if they don't want/need to, this should (hopefully) not be the case.

[aaron.ballman]

clang/include/clang/Basic/Attr.td

3378–3381

Should the documentation in AttrDocs.td be updated for this change?

[david-arm]

clang/include/clang/Basic/Attr.td

3378–3381

Hi @aaron.ballman I had a look at LoopHintDocs in AttrDocs.td and it didn't explicitly mention these states, i.e. "assume_safety", "numeric", etc., so I'm not sure if it's necessary to add anything there?

[aaron.ballman]

clang/include/clang/Basic/Attr.td

3378–3381

Oh, I see now, we're deferring to the documentation in the language extensions document. I suppose that's fine as far as this patch goes, sorry for the noise.

[david-arm]

Hi everyone, I realise that most people have probably been on holiday recently, but just a gentle ping here to see if anyone could take another look? Thanks!

[SjoerdMeijer]

LGTM, perhaps wait a day with committing in case there are more comments.

clang/include/clang/Basic/Attr.td

3378–3381

Nit: formatting, exceeding 80 columns?

3379

same?

[sdesmalen]

clang/docs/LanguageExtensions.rst

3113

nit: s/In this case//

3115

nit: Another use of vectorize_width is

3117–3119

nit: In both variants of the pragma the vectorizer may decide to fall back on fixed width vectorization if the target does not support scalable vectors.

clang/include/clang/Basic/DiagnosticParseKinds.td

1400

use vectorize_width(X, fixed) or vectorize_width(X, scalable)
(it may otherwise lead to confusion whether fixed/scalable needs quotes, same below)

clang/lib/AST/AttrImpl.cpp

46

is there always a value, even when "vectorize_width(scalable)" is specified?

clang/lib/CodeGen/CGLoopInfo.cpp

757–759

is that not something to fix in the code that conditionally sets vectorize.enable later on instead of working around it here?

[david-arm]

clang/lib/CodeGen/CGLoopInfo.cpp

757–759

I did originally try to do that, but I had trouble with it and found it broke other places too. It ended up being simpler to fix it here, but I can play around with it again. Even if this is still the simplest solution I can come back with a more detailed explanation at least!

[david-arm]

• Updated documentation as per review comments.
• Fixed an issue with using value->prettyPrint on a null ptr.
• Reworked the code that sets vectorize.enable.

[sdesmalen]

LGTM, thanks for all the changes @david-arm!

clang/lib/AST/AttrImpl.cpp

46

nit:

if (value) {
  value->printPretty(OS, nullptr, Policy);
  OS << ", ";
}
OS << (state == ScalableWidth ? "scalable" : "fixed";

?

clang/lib/CodeGen/CGLoopInfo.cpp

309

nit: != 1 (it should be functionally the same because the >1 is already caught above, but this is specifically testing that VF=1 (scalar) is not specified)

clang/docs/LanguageExtensions.rst

 .. code-block:: c++
 
   #pragma clang loop vectorize(enable)
   #pragma clang loop interleave(enable)
   for(...) {
     ...
   }
 
-The vector width is specified by ``vectorize_width(_value_)`` and the interleave
-count is specified by ``interleave_count(_value_)``, where
+The vector width is specified by
+``vectorize_width(_value_[, fixed|scalable])``, where _value_ is a positive
+integer and the type of vectorization can be specified with an optional
+second parameter. The default for the second parameter is 'fixed' and
+refers to fixed width vectorization, whereas 'scalable' indicates the
+compiler should use scalable vectors instead. Another use of vectorize_width
+is ``vectorize_width(fixed|scalable)`` where the user can hint at the type
+of vectorization to use without specifying the exact width. In both variants
+of the pragma the vectorizer may decide to fall back on fixed width
+vectorization if the target does not support scalable vectors.
+
+The interleave count is specified by ``interleave_count(_value_)``, where
 _value_ is a positive integer. This is useful for specifying the optimal
 width/count of the set of target architectures supported by your application.
 
 .. code-block:: c++
 
   #pragma clang loop vectorize_width(2)
   #pragma clang loop interleave_count(2)
   for(...) {
     ...
   }
 
 Specifying a width/count of 1 disables the optimization, and is equivalent to
 ``vectorize(disable)`` or ``interleave(disable)``.
 
 Vector predication is enabled by ``vectorize_predicate(enable)``, for example:
 
 .. code-block:: c++
 

clang/include/clang/Basic/Attr.td

let Args = [EnumArgument<"Option", "OptionType",
                            "unroll", "unroll_count", "unroll_and_jam", "unroll_and_jam_count",
                            "pipeline", "pipeline_initiation_interval", "distribute",
                            "vectorize_predicate"],
                           ["Vectorize", "VectorizeWidth", "Interleave", "InterleaveCount",
                            "Unroll", "UnrollCount", "UnrollAndJam", "UnrollAndJamCount",
                            "PipelineDisabled", "PipelineInitiationInterval", "Distribute",
                            "VectorizePredicate"]>,
               EnumArgument<"State", "LoopHintState",
-                           ["enable", "disable", "numeric", "assume_safety", "full"],
-                           ["Enable", "Disable", "Numeric", "AssumeSafety", "Full"]>,
+                           ["enable", "disable", "numeric", "fixed_width",
+                            "scalable_width", "assume_safety", "full"],
+                           ["Enable", "Disable", "Numeric", "FixedWidth",
+                            "ScalableWidth", "AssumeSafety", "Full"]>,
               ExprArgument<"Value">];
 
   let AdditionalMembers = [{
   static const char *getOptionName(int Option) {
     switch(Option) {
     case Vectorize: return "vectorize";
     case VectorizeWidth: return "vectorize_width";
     case Interleave: return "interleave";

clang/include/clang/Basic/DiagnosticParseKinds.td

 // Pragma loop support.
 def err_pragma_loop_missing_argument : Error<
   "missing argument; expected %select{an integer value|"
   "'enable'%select{|, 'full'}1%select{|, 'assume_safety'}2 or 'disable'}0">;
 def err_pragma_loop_invalid_option : Error<
   "%select{invalid|missing}0 option%select{ %1|}0; expected vectorize, "
   "vectorize_width, interleave, interleave_count, unroll, unroll_count, "
   "pipeline, pipeline_initiation_interval, vectorize_predicate, or distribute">;
+def err_pragma_loop_invalid_vectorize_option : Error<
+  "vectorize_width loop hint malformed; use vectorize_width(X, fixed) or "
+  "vectorize_width(X, scalable) where X is an integer, or vectorize_width('fixed' or 'scalable')">;
+def note_pragma_loop_invalid_vectorize_option : Note<
+  "vectorize_width loop hint malformed; use vectorize_width(X, fixed) or "
+  "vectorize_width(X, scalable) where X is an integer, or vectorize_width('fixed' or 'scalable')">;
 
 def err_pragma_fp_invalid_option : Error<
   "%select{invalid|missing}0 option%select{ %1|}0; expected 'contract', 'reassociate' or 'exceptions'">;
 def err_pragma_fp_invalid_argument : Error<
   "unexpected argument '%0' to '#pragma clang fp %1'; expected "
   "%select{"
   "'fast' or 'on' or 'off'|"
   "'on' or 'off'|"

clang/lib/AST/AttrImpl.cpp

 // Return a string containing the loop hint argument including the
 // enclosing parentheses.
 std::string LoopHintAttr::getValueString(const PrintingPolicy &Policy) const {
   std::string ValueName;
   llvm::raw_string_ostream OS(ValueName);
   OS << "(";
   if (state == Numeric)
     value->printPretty(OS, nullptr, Policy);
-  else if (state == Enable)
+  else if (state == FixedWidth || state == ScalableWidth) {
+    if (value) {
+      value->printPretty(OS, nullptr, Policy);
+      if (state == ScalableWidth)
+        OS << ", scalable";
+    } else if (state == ScalableWidth)
+      OS << "scalable";
+    else
+      OS << "fixed";
+  } else if (state == Enable)
     OS << "enable";
   else if (state == Full)
     OS << "full";
   else if (state == AssumeSafety)
     OS << "assume_safety";
   else
     OS << "disable";
   OS << ")";

clang/lib/CodeGen/CGLoopInfo.h

struct LoopAttributes {
   LVEnableState UnrollAndJamEnable;
 
   /// Value for llvm.loop.vectorize.predicate metadata
   LVEnableState VectorizePredicateEnable;
 
   /// Value for llvm.loop.vectorize.width metadata.
   unsigned VectorizeWidth;
 
+  // Value for llvm.loop.vectorize.scalable.enable
+  LVEnableState VectorizeScalable;
+
   /// Value for llvm.loop.interleave.count metadata.
   unsigned InterleaveCount;
 
   /// llvm.unroll.
   unsigned UnrollCount;
 
   /// llvm.unroll.
   unsigned UnrollAndJamCount;
public:
   /// Set the next pushed loop unroll_and_jam state.
   void setUnrollAndJamState(const LoopAttributes::LVEnableState &State) {
     StagedAttrs.UnrollAndJamEnable = State;
   }
 
   /// Set the vectorize width for the next loop pushed.
   void setVectorizeWidth(unsigned W) { StagedAttrs.VectorizeWidth = W; }
 
+  void setVectorizeScalable(const LoopAttributes::LVEnableState &State) {
+    StagedAttrs.VectorizeScalable = State;
+  }
+
   /// Set the interleave count for the next loop pushed.
   void setInterleaveCount(unsigned C) { StagedAttrs.InterleaveCount = C; }
 
   /// Set the unroll count for the next loop pushed.
   void setUnrollCount(unsigned C) { StagedAttrs.UnrollCount = C; }
 
   /// \brief Set the unroll count for the next loop pushed.
   void setUnrollAndJamCount(unsigned C) { StagedAttrs.UnrollAndJamCount = C; }

clang/lib/CodeGen/CGLoopInfo.cpp

LoopInfo::createLoopVectorizeMetadata(const LoopAttributes &Attrs,
                                       bool &HasUserTransforms) {
   LLVMContext &Ctx = Header->getContext();
 
   Optional<bool> Enabled;
   if (Attrs.VectorizeEnable == LoopAttributes::Disable)
     Enabled = false;
   else if (Attrs.VectorizeEnable != LoopAttributes::Unspecified ||
            Attrs.VectorizePredicateEnable != LoopAttributes::Unspecified ||
-           Attrs.InterleaveCount != 0 || Attrs.VectorizeWidth != 0)
+           Attrs.InterleaveCount != 0 || Attrs.VectorizeWidth != 0 ||
+           Attrs.VectorizeScalable != LoopAttributes::Unspecified)
     Enabled = true;
 
   if (Enabled != true) {
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
       NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
       NewLoopProperties.push_back(
           MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
LoopInfo::createLoopVectorizeMetadata(const LoopAttributes &Attrs,
   }
 
   // Setting vectorize.width
   if (Attrs.VectorizeWidth > 0) {
     Metadata *Vals[] = {
         MDString::get(Ctx, "llvm.loop.vectorize.width"),
         ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
                                                  Attrs.VectorizeWidth))};
+
+    Args.push_back(MDNode::get(Ctx, Vals));
+  }
+
+  if (Attrs.VectorizeScalable != LoopAttributes::Unspecified) {
+    bool IsScalable = Attrs.VectorizeScalable == LoopAttributes::Enable;
+    Metadata *Vals[] = {
+        MDString::get(Ctx, "llvm.loop.vectorize.scalable.enable"),
+        ConstantAsMetadata::get(
+            ConstantInt::get(llvm::Type::getInt1Ty(Ctx), IsScalable))};
     Args.push_back(MDNode::get(Ctx, Vals));
   }
 
   // Setting interleave.count
   if (Attrs.InterleaveCount > 0) {
     Metadata *Vals[] = {
         MDString::get(Ctx, "llvm.loop.interleave.count"),
         ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
                                                  Attrs.InterleaveCount))};
     Args.push_back(MDNode::get(Ctx, Vals));
   }
 
   // vectorize.enable is set if:
   // 1) loop hint vectorize.enable is set, or
   // 2) it is implied when vectorize.predicate is set, or
-  // 3) it is implied when vectorize.width is set.
+  // 3) it is implied when vectorize.width is set to a value > 1
+  // 4) it is implied when vectorize.scalable.enable is true
+  // 5) it is implied when vectorize.width is unset (0) and the user
+  //    explicitly requested fixed-width vectorization, i.e.
+  //    vectorize.scalable.enable is false.
   if (Attrs.VectorizeEnable != LoopAttributes::Unspecified ||
-      IsVectorPredicateEnabled ||
-      Attrs.VectorizeWidth > 1 ) {
+      IsVectorPredicateEnabled || Attrs.VectorizeWidth > 1 ||
+      Attrs.VectorizeScalable == LoopAttributes::Enable ||
+      (Attrs.VectorizeScalable == LoopAttributes::Disable &&
+       Attrs.VectorizeWidth != 1)) {
     bool AttrVal = Attrs.VectorizeEnable != LoopAttributes::Disable;
     Args.push_back(
         MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
                           ConstantAsMetadata::get(ConstantInt::get(
                               llvm::Type::getInt1Ty(Ctx), AttrVal))}));
   }
 
   if (FollowupHasTransforms)
MDNode *LoopInfo::createMetadata(
   return createFullUnrollMetadata(Attrs, LoopProperties, HasUserTransforms);
 }
 
 LoopAttributes::LoopAttributes(bool IsParallel)
     : IsParallel(IsParallel), VectorizeEnable(LoopAttributes::Unspecified),
       UnrollEnable(LoopAttributes::Unspecified),
       UnrollAndJamEnable(LoopAttributes::Unspecified),
       VectorizePredicateEnable(LoopAttributes::Unspecified), VectorizeWidth(0),
-      InterleaveCount(0), UnrollCount(0), UnrollAndJamCount(0),
+      VectorizeScalable(LoopAttributes::Unspecified), InterleaveCount(0),
+      UnrollCount(0), UnrollAndJamCount(0),
       DistributeEnable(LoopAttributes::Unspecified), PipelineDisabled(false),
       PipelineInitiationInterval(0), MustProgress(false) {}
 
 void LoopAttributes::clear() {
   IsParallel = false;
   VectorizeWidth = 0;
+  VectorizeScalable = LoopAttributes::Unspecified;
   InterleaveCount = 0;
   UnrollCount = 0;
   UnrollAndJamCount = 0;
   VectorizeEnable = LoopAttributes::Unspecified;
   UnrollEnable = LoopAttributes::Unspecified;
   UnrollAndJamEnable = LoopAttributes::Unspecified;
   VectorizePredicateEnable = LoopAttributes::Unspecified;
   DistributeEnable = LoopAttributes::Unspecified;
LoopInfo::LoopInfo(BasicBlock *Header, const LoopAttributes &Attrs,
 
   if (Attrs.IsParallel) {
     // Create an access group for this loop.
     LLVMContext &Ctx = Header->getContext();
     AccGroup = MDNode::getDistinct(Ctx, {});
   }
 
   if (!Attrs.IsParallel && Attrs.VectorizeWidth == 0 &&
+      Attrs.VectorizeScalable == LoopAttributes::Unspecified &&
       Attrs.InterleaveCount == 0 && Attrs.UnrollCount == 0 &&
       Attrs.UnrollAndJamCount == 0 && !Attrs.PipelineDisabled &&
       Attrs.PipelineInitiationInterval == 0 &&
       Attrs.VectorizePredicateEnable == LoopAttributes::Unspecified &&
       Attrs.VectorizeEnable == LoopAttributes::Unspecified &&
       Attrs.UnrollEnable == LoopAttributes::Unspecified &&
       Attrs.UnrollAndJamEnable == LoopAttributes::Unspecified &&
       Attrs.DistributeEnable == LoopAttributes::Unspecified && !StartLoc &&
if (Parent && (Parent->Attrs.UnrollAndJamEnable ||
     // Split the transformations in those that happens before the unroll-and-jam
     // and those after.
 
     LoopAttributes BeforeJam, AfterJam;
 
     BeforeJam.IsParallel = AfterJam.IsParallel = Attrs.IsParallel;
 
     BeforeJam.VectorizeWidth = Attrs.VectorizeWidth;
+    BeforeJam.VectorizeScalable = Attrs.VectorizeScalable;
     BeforeJam.InterleaveCount = Attrs.InterleaveCount;
     BeforeJam.VectorizeEnable = Attrs.VectorizeEnable;
     BeforeJam.DistributeEnable = Attrs.DistributeEnable;
     BeforeJam.VectorizePredicateEnable = Attrs.VectorizePredicateEnable;
 
     switch (Attrs.UnrollEnable) {
     case LoopAttributes::Unspecified:
     case LoopAttributes::Disable:
if (Parent && (Parent->Attrs.UnrollAndJamEnable ||
     if (!Parent->UnrollAndJamInnerFollowup) {
       // Splitting the attributes into a BeforeJam and an AfterJam part will
       // stop 'llvm.loop.isvectorized' (generated by vectorization in BeforeJam)
       // to be forwarded to the AfterJam part. We detect the situation here and
       // add it manually.
       SmallVector<Metadata *, 1> BeforeLoopProperties;
       if (BeforeJam.VectorizeEnable != LoopAttributes::Unspecified ||
           BeforeJam.VectorizePredicateEnable != LoopAttributes::Unspecified ||
-          BeforeJam.InterleaveCount != 0 || BeforeJam.VectorizeWidth != 0)
+          BeforeJam.InterleaveCount != 0 || BeforeJam.VectorizeWidth != 0 ||
+          BeforeJam.VectorizeScalable == LoopAttributes::Enable)
         BeforeLoopProperties.push_back(
             MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.isvectorized")));
 
       bool InnerFollowupHasTransform = false;
       MDNode *InnerFollowup = createMetadata(AfterJam, BeforeLoopProperties,
                                              InnerFollowupHasTransform);
       if (InnerFollowupHasTransform)
         Parent->UnrollAndJamInnerFollowup = InnerFollowup;
if (OpenCLHint) {
       State = LH->getState();
     }
     switch (State) {
     case LoopHintAttr::Disable:
       switch (Option) {
       case LoopHintAttr::Vectorize:
         // Disable vectorization by specifying a width of 1.
         setVectorizeWidth(1);
+        setVectorizeScalable(LoopAttributes::Unspecified);
         break;
       case LoopHintAttr::Interleave:
         // Disable interleaving by speciyfing a count of 1.
         setInterleaveCount(1);
         break;
       case LoopHintAttr::Unroll:
         setUnrollState(LoopAttributes::Disable);
         break;
case LoopHintAttr::Full:
       case LoopHintAttr::Distribute:
       case LoopHintAttr::PipelineDisabled:
       case LoopHintAttr::PipelineInitiationInterval:
       case LoopHintAttr::VectorizePredicate:
         llvm_unreachable("Options cannot be used with 'full' hint.");
         break;
       }
       break;
-    case LoopHintAttr::Numeric:
+    case LoopHintAttr::FixedWidth:
+    case LoopHintAttr::ScalableWidth:
       switch (Option) {
       case LoopHintAttr::VectorizeWidth:
+        setVectorizeScalable(State == LoopHintAttr::ScalableWidth
+                                 ? LoopAttributes::Enable
+                                 : LoopAttributes::Disable);
+        if (LH->getValue())
           setVectorizeWidth(ValueInt);
         break;
+      default:
+        llvm_unreachable("Options cannot be used with 'scalable' hint.");
+        break;
+      }
+      break;
+    case LoopHintAttr::Numeric:
+      switch (Option) {
       case LoopHintAttr::InterleaveCount:
         setInterleaveCount(ValueInt);
         break;
       case LoopHintAttr::UnrollCount:
         setUnrollCount(ValueInt);
         break;
       case LoopHintAttr::UnrollAndJamCount:
         setUnrollAndJamCount(ValueInt);
         break;
       case LoopHintAttr::PipelineInitiationInterval:
         setPipelineInitiationInterval(ValueInt);
         break;
       case LoopHintAttr::Unroll:
       case LoopHintAttr::UnrollAndJam:
       case LoopHintAttr::VectorizePredicate:
       case LoopHintAttr::Vectorize:
+      case LoopHintAttr::VectorizeWidth:
       case LoopHintAttr::Interleave:
       case LoopHintAttr::Distribute:
       case LoopHintAttr::PipelineDisabled:
         llvm_unreachable("Options cannot be assigned a value.");
         break;
       }
       break;
     }

clang/lib/Parse/ParsePragma.cpp

return std::string(llvm::StringSwitch<StringRef>(Str)
                          .Case("unroll", Str)
                          .Default(""));
 }
 
 bool Parser::HandlePragmaLoopHint(LoopHint &Hint) {
   assert(Tok.is(tok::annot_pragma_loop_hint));
   PragmaLoopHintInfo *Info =
       static_cast<PragmaLoopHintInfo *>(Tok.getAnnotationValue());
 
   IdentifierInfo *PragmaNameInfo = Info->PragmaName.getIdentifierInfo();
   Hint.PragmaNameLoc = IdentifierLoc::create(
       Actions.Context, Info->PragmaName.getLocation(), PragmaNameInfo);
 
   // It is possible that the loop hint has no option identifier, such as
   // #pragma unroll(4).
   IdentifierInfo *OptionInfo = Info->Option.is(tok::identifier)
                                    ? Info->Option.getIdentifierInfo()
if (!Valid) {
             << /*AssumeSafetyKeyword=*/AssumeSafetyArg;
       }
       return false;
     }
     if (Toks.size() > 2)
       Diag(Tok.getLocation(), diag::warn_pragma_extra_tokens_at_eol)
           << PragmaLoopHintString(Info->PragmaName, Info->Option);
     Hint.StateLoc = IdentifierLoc::create(Actions.Context, StateLoc, StateInfo);
+  } else if (OptionInfo && OptionInfo->getName() == "vectorize_width") {
+    PP.EnterTokenStream(Toks, /*DisableMacroExpansion=*/false,
+                        /*IsReinject=*/false);
+    ConsumeAnnotationToken();
+
+    SourceLocation StateLoc = Toks[0].getLocation();
+    IdentifierInfo *StateInfo = Toks[0].getIdentifierInfo();
+    StringRef IsScalableStr = StateInfo ? StateInfo->getName() : "";
+
+    // Look for vectorize_width(fixed|scalable)
+    if (IsScalableStr == "scalable" || IsScalableStr == "fixed") {
+      PP.Lex(Tok); // Identifier
+
+      if (Toks.size() > 2) {
+        Diag(Tok.getLocation(), diag::warn_pragma_extra_tokens_at_eol)
+            << PragmaLoopHintString(Info->PragmaName, Info->Option);
+        while (Tok.isNot(tok::eof))
+          ConsumeAnyToken();
+      }
+
+      Hint.StateLoc =
+          IdentifierLoc::create(Actions.Context, StateLoc, StateInfo);
+
+      ConsumeToken(); // Consume the constant expression eof terminator.
+    } else {
+      // Enter constant expression including eof terminator into token stream.
+      ExprResult R = ParseConstantExpression();
+
+      if (R.isInvalid() && !Tok.is(tok::comma))
+        Diag(Toks[0].getLocation(),
+             diag::note_pragma_loop_invalid_vectorize_option);
+
+      bool Arg2Error = false;
+      if (Tok.is(tok::comma)) {
+        PP.Lex(Tok); // ,
+
+        StateInfo = Tok.getIdentifierInfo();
+        IsScalableStr = StateInfo->getName();
+
+        if (IsScalableStr != "scalable" && IsScalableStr != "fixed") {
+          Diag(Tok.getLocation(),
+               diag::err_pragma_loop_invalid_vectorize_option);
+          Arg2Error = true;
+        } else
+          Hint.StateLoc =
+              IdentifierLoc::create(Actions.Context, StateLoc, StateInfo);
+
+        PP.Lex(Tok); // Identifier
+      }
+
+      // Tokens following an error in an ill-formed constant expression will
+      // remain in the token stream and must be removed.
+      if (Tok.isNot(tok::eof)) {
+        Diag(Tok.getLocation(), diag::warn_pragma_extra_tokens_at_eol)
+            << PragmaLoopHintString(Info->PragmaName, Info->Option);
+        while (Tok.isNot(tok::eof))
+          ConsumeAnyToken();
+      }
+
+      ConsumeToken(); // Consume the constant expression eof terminator.
+
+      if (Arg2Error || R.isInvalid() ||
+          Actions.CheckLoopHintExpr(R.get(), Toks[0].getLocation()))
+        return false;
+
+      // Argument is a constant expression with an integer type.
+      Hint.ValueExpr = R.get();
+    }
   } else {
     // Enter constant expression including eof terminator into token stream.
     PP.EnterTokenStream(Toks, /*DisableMacroExpansion=*/false,
                         /*IsReinject=*/false);
     ConsumeAnnotationToken();
-
     ExprResult R = ParseConstantExpression();
 
     // Tokens following an error in an ill-formed constant expression will
     // remain in the token stream and must be removed.
     if (Tok.isNot(tok::eof)) {
       Diag(Tok.getLocation(), diag::warn_pragma_extra_tokens_at_eol)
           << PragmaLoopHintString(Info->PragmaName, Info->Option);
       while (Tok.isNot(tok::eof))

clang/lib/Sema/SemaStmtAttr.cpp

 //===--- SemaStmtAttr.cpp - Statement Attribute Handling ------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //  This file implements stmt-related attribute processing.
 //
 //===----------------------------------------------------------------------===//
 
-#include "clang/AST/EvaluatedExprVisitor.h"
-#include "clang/Sema/SemaInternal.h"
 #include "clang/AST/ASTContext.h"
+#include "clang/AST/EvaluatedExprVisitor.h"
 #include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
 #include "clang/Sema/DelayedDiagnostic.h"
 #include "clang/Sema/Lookup.h"
 #include "clang/Sema/ScopeInfo.h"
+#include "clang/Sema/SemaInternal.h"
 #include "llvm/ADT/StringExtras.h"
 
 using namespace clang;
 using namespace sema;
 
 static Attr *handleFallThroughAttr(Sema &S, Stmt *St, const ParsedAttr &A,
                                    SourceRange Range) {
   FallThroughAttr Attr(S.Context, A);
Option = llvm::StringSwitch<LoopHintAttr::OptionType>(
                  .Case("interleave_count", LoopHintAttr::InterleaveCount)
                  .Case("unroll", LoopHintAttr::Unroll)
                  .Case("unroll_count", LoopHintAttr::UnrollCount)
                  .Case("pipeline", LoopHintAttr::PipelineDisabled)
                  .Case("pipeline_initiation_interval",
                        LoopHintAttr::PipelineInitiationInterval)
                  .Case("distribute", LoopHintAttr::Distribute)
                  .Default(LoopHintAttr::Vectorize);
-    if (Option == LoopHintAttr::VectorizeWidth ||
-        Option == LoopHintAttr::InterleaveCount ||
+    if (Option == LoopHintAttr::VectorizeWidth) {
+      assert((ValueExpr || (StateLoc && StateLoc->Ident)) &&
+             "Attribute must have a valid value expression or argument.");
+      if (ValueExpr && S.CheckLoopHintExpr(ValueExpr, St->getBeginLoc()))
+        return nullptr;
+      if (StateLoc && StateLoc->Ident && StateLoc->Ident->isStr("scalable"))
+        State = LoopHintAttr::ScalableWidth;
+      else
+        State = LoopHintAttr::FixedWidth;
+    } else if (Option == LoopHintAttr::InterleaveCount ||
                Option == LoopHintAttr::UnrollCount ||
                Option == LoopHintAttr::PipelineInitiationInterval) {
       assert(ValueExpr && "Attribute must have a valid value expression.");
       if (S.CheckLoopHintExpr(ValueExpr, St->getBeginLoc()))
         return nullptr;
       State = LoopHintAttr::Numeric;
     } else if (Option == LoopHintAttr::Vectorize ||
                Option == LoopHintAttr::Interleave ||
                Option == LoopHintAttr::VectorizePredicate ||
                Option == LoopHintAttr::Unroll ||

clang/test/AST/ast-print-pragmas.cpp

 // RUN: %clang_cc1 -ast-print %s -o - | FileCheck %s
 // RUN: %clang_cc1 -DMS_EXT -fsyntax-only -fms-extensions %s -triple x86_64-pc-win32 -ast-print | FileCheck %s --check-prefix=MS-EXT
 
 // CHECK: #pragma clang loop vectorize_width(4)
 // CHECK-NEXT: #pragma clang loop interleave_count(8){{$}}
 
 void test(int *List, int Length) {
   int i = 0;
 #pragma clang loop vectorize_width(4)
 #pragma clang loop interleave_count(8)
 // CHECK-NEXT: while (i < Length)
   while (i < Length) {
     List[i] = i * 2;
     i++;
   }
+  i = 0;
+
+// CHECK: #pragma clang loop vectorize_width(4, scalable)
+
+#pragma clang loop vectorize_width(4, scalable)
+// CHECK-NEXT: while (i < Length)
+  while (i < Length) {
+    List[i] = i * 2;
+    i++;
+  }
+  i = 0;
+
+// CHECK: #pragma clang loop vectorize_width(fixed)
+
+#pragma clang loop vectorize_width(fixed)
+// CHECK-NEXT: while (i < Length)
+  while (i < Length) {
+    List[i] = i * 2;
+    i++;
+  }
+  i = 0;
+
+// CHECK: #pragma clang loop vectorize_width(scalable)
+
+#pragma clang loop vectorize_width(scalable)
+// CHECK-NEXT: while (i < Length)
+  while (i < Length) {
+    List[i] = i * 2;
+    i++;
+  }
 
 // CHECK: #pragma clang loop distribute(disable)
 // CHECK-NEXT: #pragma clang loop vectorize(enable)
 // CHECK-NEXT: #pragma clang loop interleave(disable)
 // CHECK-NEXT: #pragma clang loop vectorize_predicate(disable)
 
 #pragma clang loop distribute(disable)
 #pragma clang loop vectorize(enable)

clang/test/CodeGenCXX/pragma-loop-pr27643.cpp

 // CHECK-LABEL: @{{.*}}loop4{{.*}}(
 // CHECK: br label {{.*}}, !llvm.loop ![[LOOP4:.*]]
 
   #pragma clang loop vectorize_width(2)
   for (int i = 0; i < Length; i++)
     List[i] = i * 2;
 }
 
-// CHECK: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP:![0-9]+]], ![[VEC_WIDTH_1:.*]], ![[VEC_ENABLE:.*]]}
+// CHECK: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP:![0-9]+]], ![[VEC_WIDTH_1:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE:.*]]}
 // CHECK: ![[VEC_WIDTH_1]] = !{!"llvm.loop.vectorize.width", i32 1}
+// CHECK: ![[FIXED_WIDTH]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
 // CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
 
-// CHECK: ![[LOOP2]] = distinct !{![[LOOP2]], [[MP]], ![[VEC_WIDTH_2:.*]], ![[VEC_ENABLE]]}
+// CHECK: ![[LOOP2]] = distinct !{![[LOOP2]], [[MP]], ![[VEC_WIDTH_2:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE]]}
 // CHECK: ![[VEC_WIDTH_2]] = !{!"llvm.loop.vectorize.width", i32 2}
 
-// CHECK: ![[LOOP3]] = distinct !{![[LOOP3]], [[MP]], ![[VEC_WIDTH_1]]}
+// CHECK: ![[LOOP3]] = distinct !{![[LOOP3]], [[MP]], ![[VEC_WIDTH_1]], ![[FIXED_WIDTH]]}
 
-// CHECK: ![[LOOP4]] = distinct !{![[LOOP4]], [[MP]], ![[VEC_WIDTH_2]], ![[VEC_ENABLE]]}
+// CHECK: ![[LOOP4]] = distinct !{![[LOOP4]], [[MP]], ![[VEC_WIDTH_2]], ![[FIXED_WIDTH]], ![[VEC_ENABLE]]}

clang/test/CodeGenCXX/pragma-loop.cpp

 void template_test(double *List, int Length) {
   double Value = 10;
 
   for_template_test<double>(List, Length, Value);
   for_template_define_test<double, int>(List, Length, Value);
   for_template_constant_expression_test<double, 2, 4, 8>(List, Length);
 }
 
+// Verify for loop is performing fixed width vectorization
+void for_test_fixed_16(int *List, int Length) {
+#pragma clang loop vectorize_width(16, fixed) interleave_count(4) unroll(disable) distribute(disable)
+  for (int i = 0; i < Length; i++) {
+    // CHECK: br label {{.*}}, !llvm.loop ![[LOOP_15:.*]]
+    List[i] = i * 2;
+  }
+}
+
+// Verify for loop is performing scalable vectorization
+void for_test_scalable_16(int *List, int Length) {
+#pragma clang loop vectorize_width(16, scalable) interleave_count(4) unroll(disable) distribute(disable)
+  for (int i = 0; i < Length; i++) {
+    // CHECK: br label {{.*}}, !llvm.loop ![[LOOP_16:.*]]
+    List[i] = i * 2;
+  }
+}
+
+// Verify for loop is performing fixed width vectorization
+void for_test_fixed(int *List, int Length) {
+#pragma clang loop vectorize_width(fixed) interleave_count(4) unroll(disable) distribute(disable)
+  for (int i = 0; i < Length; i++) {
+    // CHECK: br label {{.*}}, !llvm.loop ![[LOOP_17:.*]]
+    List[i] = i * 2;
+  }
+}
+
+// Verify for loop is performing scalable vectorization
+void for_test_scalable(int *List, int Length) {
+#pragma clang loop vectorize_width(scalable) interleave_count(4) unroll(disable) distribute(disable)
+  for (int i = 0; i < Length; i++) {
+    // CHECK: br label {{.*}}, !llvm.loop ![[LOOP_18:.*]]
+    List[i] = i * 2;
+  }
+}
+
+// Verify for loop is performing scalable vectorization
+void for_test_scalable_1(int *List, int Length) {
+#pragma clang loop vectorize_width(1, scalable) interleave_count(4) unroll(disable) distribute(disable)
+  for (int i = 0; i < Length; i++) {
+    // CHECK: br label {{.*}}, !llvm.loop ![[LOOP_19:.*]]
+    List[i] = i * 2;
+  }
+}
+
 // CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], [[MP:![0-9]+]], ![[UNROLL_FULL:.*]]}
 // CHECK: ![[UNROLL_FULL]] = !{!"llvm.loop.unroll.full"}
 
-// CHECK: ![[LOOP_2]] = distinct !{![[LOOP_2]], [[MP]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_8:.*]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
+// CHECK: ![[LOOP_2]] = distinct !{![[LOOP_2]], [[MP]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_8:.*]], ![[FIXED_VEC:.*]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
 // CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
 // CHECK: ![[DISTRIBUTE_DISABLE]] = !{!"llvm.loop.distribute.enable", i1 false}
 // CHECK: ![[WIDTH_8]] = !{!"llvm.loop.vectorize.width", i32 8}
+// CHECK: ![[FIXED_VEC]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
 // CHECK: ![[INTERLEAVE_4]] = !{!"llvm.loop.interleave.count", i32 4}
 // CHECK: ![[VECTORIZE_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
 
-// CHECK: ![[LOOP_3]] = distinct !{![[LOOP_3]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE]], ![[FOLLOWUP_VECTOR_3:.*]]}
+// CHECK: ![[LOOP_3]] = distinct !{![[LOOP_3]], [[MP]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE]], ![[FOLLOWUP_VECTOR_3:.*]]}
 // CHECK: ![[FOLLOWUP_VECTOR_3]] = !{!"llvm.loop.vectorize.followup_all", ![[AFTER_VECTOR_3:.*]]}
 // CHECK: ![[AFTER_VECTOR_3]] = distinct !{![[AFTER_VECTOR_3]], [[MP]], ![[ISVECTORIZED:.*]], ![[UNROLL_8:.*]]}
 // CHECK: ![[ISVECTORIZED]] = !{!"llvm.loop.isvectorized"}
 // CHECK: ![[UNROLL_8]] = !{!"llvm.loop.unroll.count", i32 8}
 
-// CHECK: ![[LOOP_4]] = distinct !{![[LOOP_4]], ![[WIDTH_2:.*]], ![[INTERLEAVE_2:.*]], ![[VECTORIZE_ENABLE]]}
+// CHECK: ![[LOOP_4]] = distinct !{![[LOOP_4]], ![[WIDTH_2:.*]], ![[FIXED_VEC]], ![[INTERLEAVE_2:.*]], ![[VECTORIZE_ENABLE]]}
 // CHECK: ![[WIDTH_2]] = !{!"llvm.loop.vectorize.width", i32 2}
 // CHECK: ![[INTERLEAVE_2]] = !{!"llvm.loop.interleave.count", i32 2}
 
 // CHECK: ![[LOOP_5]] = distinct !{![[LOOP_5]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_1:.*]]}
 // CHECK: ![[WIDTH_1]] = !{!"llvm.loop.vectorize.width", i32 1}
 
-// CHECK: ![[LOOP_6]] = distinct !{![[LOOP_6]], ![[WIDTH_2:.*]], ![[INTERLEAVE_2:.*]], ![[FOLLOWUP_VECTOR_6:.*]]}
+// CHECK: ![[LOOP_6]] = distinct !{![[LOOP_6]], [[MP]], ![[WIDTH_2:.*]], ![[FIXED_VEC]], ![[INTERLEAVE_2:.*]], ![[FOLLOWUP_VECTOR_6:.*]]}
 // CHECK: ![[FOLLOWUP_VECTOR_6]] = !{!"llvm.loop.vectorize.followup_all", ![[AFTER_VECTOR_6:.*]]}
 // CHECK: ![[AFTER_VECTOR_6]] = distinct !{![[AFTER_VECTOR_6]], ![[ISVECTORIZED:.*]], ![[UNROLL_8:.*]]}
 
-// CHECK: ![[LOOP_7]] = distinct !{![[LOOP_7]], [[MP]], ![[WIDTH_5:.*]], ![[VECTORIZE_ENABLE]]}
+// CHECK: ![[LOOP_7]] = distinct !{![[LOOP_7]], [[MP]], ![[WIDTH_5:.*]], ![[FIXED_VEC]], ![[VECTORIZE_ENABLE]]}
 // CHECK: ![[WIDTH_5]] = !{!"llvm.loop.vectorize.width", i32 5}
 
-// CHECK: ![[LOOP_8]] = distinct !{![[LOOP_8]], ![[WIDTH_5:.*]]}
+// CHECK: ![[LOOP_8]] = distinct !{![[LOOP_8]], [[MP]], ![[WIDTH_5:.*]], ![[FIXED_VEC]], ![[VECTORIZE_ENABLE]]}
 
-// CHECK: ![[LOOP_9]] = distinct !{![[LOOP_9]], ![[WIDTH_8:.*]], ![[INTERLEAVE_8:.*]], ![[FOLLOWUP_VECTOR_9:.*]]}
+// CHECK: ![[LOOP_9]] = distinct !{![[LOOP_9]], ![[WIDTH_8:.*]], ![[FIXED_VEC]], ![[INTERLEAVE_8:.*]], ![[FOLLOWUP_VECTOR_9:.*]]}
 // CHECK: ![[FOLLOWUP_VECTOR_9]] = !{!"llvm.loop.vectorize.followup_all", ![[AFTER_VECTOR_9:.*]]}
 // CHECK: ![[AFTER_VECTOR_9]] = distinct !{![[AFTER_VECTOR_9]], ![[ISVECTORIZED:.*]], ![[UNROLL_8:.*]]}
 
-// CHECK: ![[LOOP_10]] = distinct !{![[LOOP_10]], ![[WIDTH_2:.*]], ![[INTERLEAVE_2:.*]], ![[FOLLOWUP_VECTOR_10:.*]]}
+// CHECK: ![[LOOP_10]] = distinct !{![[LOOP_10]], ![[WIDTH_2:.*]], ![[FIXED_VEC]], ![[INTERLEAVE_2:.*]], ![[FOLLOWUP_VECTOR_10:.*]]}
 // CHECK: ![[FOLLOWUP_VECTOR_10]] = !{!"llvm.loop.vectorize.followup_all", ![[AFTER_VECTOR_10:.*]]}
 // CHECK: ![[AFTER_VECTOR_10]] = distinct !{![[AFTER_VECTOR_10]], ![[ISVECTORIZED:.*]], ![[UNROLL_8:.*]]}
 
-// CHECK: ![[LOOP_11]] = distinct !{![[LOOP_11]], ![[WIDTH_2:.*]], ![[INTERLEAVE_4:.*]], ![[FOLLOWUP_VECTOR_11:.*]]}
+// CHECK: ![[LOOP_11]] = distinct !{![[LOOP_11]], ![[WIDTH_2:.*]], ![[FIXED_VEC]], ![[INTERLEAVE_4:.*]], ![[FOLLOWUP_VECTOR_11:.*]]}
 // CHECK: ![[FOLLOWUP_VECTOR_11]] = !{!"llvm.loop.vectorize.followup_all", ![[AFTER_VECTOR_11:.*]]}
 // CHECK: ![[AFTER_VECTOR_11]] = distinct !{![[AFTER_VECTOR_11]], ![[ISVECTORIZED:.*]], ![[UNROLL_8:.*]]}
 
-// CHECK: ![[LOOP_12]] = distinct !{![[LOOP_12]], ![[WIDTH_6:.*]], ![[INTERLEAVE_10:.*]], ![[FOLLOWUP_VECTOR_12:.*]]}
+// CHECK: ![[LOOP_12]] = distinct !{![[LOOP_12]], ![[WIDTH_6:.*]], ![[FIXED_VEC]], ![[INTERLEAVE_10:.*]], ![[FOLLOWUP_VECTOR_12:.*]]}
 // CHECK: ![[FOLLOWUP_VECTOR_12]] = !{!"llvm.loop.vectorize.followup_all", ![[AFTER_VECTOR_12:.*]]}
 // CHECK: ![[AFTER_VECTOR_12]] = distinct !{![[AFTER_VECTOR_12]], ![[ISVECTORIZED:.*]], ![[UNROLL_24:.*]]}
 // CHECK: ![[UNROLL_24]] = !{!"llvm.loop.unroll.count", i32 24}
 
 // CHECK: ![[LOOP_13]] = distinct !{![[LOOP_13]], ![[WIDTH_8:.*]], ![[INTERLEAVE_16:.*]], ![[VECTORIZE_ENABLE]], ![[FOLLOWUP_VECTOR_13:.*]]}
 // CHECK: ![[INTERLEAVE_16]] = !{!"llvm.loop.interleave.count", i32 16}
 // CHECK: ![[FOLLOWUP_VECTOR_13]] = !{!"llvm.loop.vectorize.followup_all", ![[AFTER_VECTOR_13:.*]]}
 // CHECK: ![[AFTER_VECTOR_13]] = distinct !{![[AFTER_VECTOR_13]], ![[ISVECTORIZED:.*]], ![[UNROLL_32:.*]]}
 // CHECK: ![[UNROLL_32]] = !{!"llvm.loop.unroll.count", i32 32}
 
-// CHECK: ![[LOOP_14]] = distinct !{![[LOOP_14]], [[MP]], ![[WIDTH_10:.*]], ![[VECTORIZE_ENABLE]]}
+// CHECK: ![[LOOP_14]] = distinct !{![[LOOP_14]], [[MP]], ![[WIDTH_10:.*]], ![[FIXED_VEC]], ![[VECTORIZE_ENABLE]]}
 // CHECK: ![[WIDTH_10]] = !{!"llvm.loop.vectorize.width", i32 10}
+
+// CHECK: ![[LOOP_15]] = distinct !{![[LOOP_15]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_16:.*]], ![[FIXED_VEC]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
+// CHECK: ![[WIDTH_16]] = !{!"llvm.loop.vectorize.width", i32 16}
+
+// CHECK: ![[LOOP_16]] = distinct !{![[LOOP_16]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_16]], ![[SCALABLE_VEC:.*]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
+// CHECK: ![[SCALABLE_VEC]] = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
+
+// CHECK: ![[LOOP_17]] = distinct !{![[LOOP_17]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[FIXED_VEC]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
+// CHECK: ![[LOOP_18]] = distinct !{![[LOOP_18]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[SCALABLE_VEC]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}
+// CHECK: ![[LOOP_19]] = distinct !{![[LOOP_19]], ![[UNROLL_DISABLE:.*]], ![[DISTRIBUTE_DISABLE:.*]], ![[WIDTH_1]], ![[SCALABLE_VEC]], ![[INTERLEAVE_4:.*]], ![[VECTORIZE_ENABLE:.*]]}

clang/test/Parser/pragma-loop.cpp

void test_nontype_template_bool(int *List, int Length) {
   /* expected-error {{invalid argument of type 'bool'; expected an integer type}} */ #pragma clang loop vectorize_width(V)
   for (int i = 0; i < Length; i++) {
     List[i] = i;
   }
 }
 
 template <int V, int I>
 void test_nontype_template_badarg(int *List, int Length) {
-  /* expected-error {{use of undeclared identifier 'Vec'}} */ #pragma clang loop vectorize_width(Vec) interleave_count(I)
+  /* expected-error {{use of undeclared identifier 'Vec'}} */ #pragma clang loop vectorize_width(Vec) interleave_count(I) /*
+     expected-note {{vectorize_width loop hint malformed; use vectorize_width(X, fixed) or vectorize_width(X, scalable) where X is an integer, or vectorize_width('fixed' or 'scalable')}} */
   /* expected-error {{use of undeclared identifier 'Int'}} */ #pragma clang loop vectorize_width(V) interleave_count(Int)
   for (int i = 0; i < Length; i++) {
     List[i] = i;
   }
 }
 
 template <typename T>
 void test_type_template_vectorize(int *List, int Length) {
   const T Value = -1;
   /* expected-error {{invalid value '-1'; must be positive}} */ #pragma clang loop vectorize_width(Value)
   for (int i = 0; i < Length; i++) {
     List[i] = i;
   }
+
+  /* expected-error {{invalid value '-1'; must be positive}} */ #pragma clang loop vectorize_width(Value, fixed)
+  for (int i = 0; i < Length; i++) {
+    List[i] = i;
+  }
 }
 
 void test(int *List, int Length) {
   int i = 0;
 
 #pragma clang loop vectorize(enable)
 #pragma clang loop interleave(enable)
 #pragma clang loop vectorize_predicate(enable)

 /* expected-error {{value '3000000000' is too large}} */ #pragma clang loop unroll_count(3000000000)
   while (i-6 < Length) {
     List[i] = i;
   }
 
 /* expected-warning {{extra tokens at end of '#pragma clang loop'}} */ #pragma clang loop vectorize_width(1 +) 1
 /* expected-warning {{extra tokens at end of '#pragma clang loop'}} */ #pragma clang loop vectorize_width(1) +1
 const int VV = 4;
-/* expected-error {{expected expression}} */ #pragma clang loop vectorize_width(VV +/ 2)
-/* expected-error {{use of undeclared identifier 'undefined'}} */ #pragma clang loop vectorize_width(VV+undefined)
+/* expected-error {{expected expression}} */ #pragma clang loop vectorize_width(VV +/ 2) /*
+   expected-note {{vectorize_width loop hint malformed; use vectorize_width(X, fixed) or vectorize_width(X, scalable) where X is an integer, or vectorize_width('fixed' or 'scalable')}} */
+/* expected-error {{use of undeclared identifier 'undefined'}} */ #pragma clang loop vectorize_width(VV+undefined) /*
+   expected-note {{vectorize_width loop hint malformed; use vectorize_width(X, fixed) or vectorize_width(X, scalable) where X is an integer, or vectorize_width('fixed' or 'scalable')}} */
 /* expected-error {{expected ')'}} */ #pragma clang loop vectorize_width(1+(^*/2 * ()
 /* expected-warning {{extra tokens at end of '#pragma clang loop' - ignored}} */ #pragma clang loop vectorize_width(1+(-0[0]))))))
 
-/* expected-error {{use of undeclared identifier 'badvalue'}} */ #pragma clang loop vectorize_width(badvalue)
+/* expected-error {{use of undeclared identifier 'badvalue'}} */ #pragma clang loop vectorize_width(badvalue) /*
+   expected-note {{vectorize_width loop hint malformed; use vectorize_width(X, fixed) or vectorize_width(X, scalable) where X is an integer, or vectorize_width('fixed' or 'scalable')}} */
 /* expected-error {{use of undeclared identifier 'badvalue'}} */ #pragma clang loop interleave_count(badvalue)
 /* expected-error {{use of undeclared identifier 'badvalue'}} */ #pragma clang loop unroll_count(badvalue)
   while (i-6 < Length) {
     List[i] = i;
   }
 
 /* expected-error {{invalid argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop vectorize(badidentifier)
 /* expected-error {{invalid argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop interleave(badidentifier)
 /* expected-error {{invalid argument; expected 'enable', 'full' or 'disable'}} */ #pragma clang loop unroll(badidentifier)
 /* expected-error {{invalid argument; expected 'enable' or 'disable'}} */ #pragma clang loop distribute(badidentifier)
   while (i-7 < Length) {
     List[i] = i;
   }
 
 // PR20069 - Loop pragma arguments that are not identifiers or numeric
 // constants crash FE.
 /* expected-error {{expected ')'}} */ #pragma clang loop vectorize(()
 /* expected-error {{invalid argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop interleave(*)
 /* expected-error {{invalid argument; expected 'enable', 'full' or 'disable'}} */ #pragma clang loop unroll(=)
 /* expected-error {{invalid argument; expected 'enable' or 'disable'}} */ #pragma clang loop distribute(+)
-/* expected-error {{type name requires a specifier or qualifier}} expected-error {{expected expression}} */ #pragma clang loop vectorize_width(^)
+/* expected-error {{type name requires a specifier or qualifier}} expected-error {{expected expression}} */ #pragma clang loop vectorize_width(^) /* expected-note {{vectorize_width loop hint malformed; use vectorize_width(X, fixed) or vectorize_width(X, scalable) where X is an integer, or vectorize_width('fixed' or 'scalable')}} */
 /* expected-error {{expected expression}} expected-error {{expected expression}} */ #pragma clang loop interleave_count(/)
 /* expected-error {{expected expression}} expected-error {{expected expression}} */ #pragma clang loop unroll_count(==)
   while (i-8 < Length) {
     List[i] = i;
   }
 
 #pragma clang loop vectorize(enable)
 /* expected-error {{expected a for, while, or do-while loop to follow '#pragma clang loop'}} */ int j = Length;