#pragma clang loop vectorize_predicate(enable|disable)

[SjoerdMeijer]

Summary

This adds a vectorize ''predicate" loop hint, e.g.:

#pragma loop vectorize(enable) vectorize_predicate(enable)

that can be used to indicate to the vectoriser that all statements should be predicated. This allows, for example, 'folding' of the remainder loop into the main loop.

[Meinersbur]

IMHO, this loops like an option of a particular transformation, not an independent pragma. E.g.

#pragma clang loop vectorize(enable) vectorize_remainder(predicated)

There could be multiple choices for how to execute remainder iterations, e.g. instead of an epilogue, the first iterations could be executed in an prologue. Or an option where the compiler may assume that the iteration-count is always a multiple of the vector width, etc.

Also consider interactions with other transformations: What would happen with the llvm.loop.tailpredicate metadata after, e.g. loop fusion/distribution? How does the user know whether the pragma had an effect?

[SjoerdMeijer]

Thanks for taking a look and your suggestions!

I noticed your comment here after I replied to the list. As I wrote there, and long story short, I thought I could kill 2 birds with 1 stone, but that doesn't seem to be the case. I agree that for the vectorizer an option like this is much better:

#pragma clang loop vectorize(enable) vectorize_remainder(enable)

I will implement that here because that looks a useful addition that people like, and I think this will also work for me initially.

[SjoerdMeijer]

So I went for:

predicate(enable)

as I think that is most general, best capturing it, but I am of course completely open to bikeshedding names :-)

[Meinersbur]

[serious] Need to update docs.

I'd expect predicate to be an option of another transformation. For vectorization, by convention the #pragma name would be vectorize_predicate and the metadata llvm.loop.vectorize.predicate. As a standalone, it is not clear what it is supposed to do (docs missing) and I am against mixing similar semantics of options of different loop transformations.

I am currently trying to update the semantics of loop transformations such that a transformation order can be defined. An option without transformation it belongs to does not fit well into that. E.g. what happends with llvm.loop.predicate after unrolling/vectorization/distribution/interchange/unroll-and-jam/fusion?

Michael

clang/lib/CodeGen/CGLoopInfo.cpp

426 ↗

(On Diff #210314)

[nit] run clang-format over your patch

[SjoerdMeijer]

Hi Michael, thanks for taking a look again!

Completely agree what you suggested, so I've change the pragma to vectorize_predicate(enable) and the metadata to llvm.loop.vectorize.predicate.

My little plan is as follows:

• Finish D64916: thanks for reviewing that too!
• I will follow up on that. I have a local patch that I need to finish that shows how all moving parts work together. I.e., it picks up the metadata, and enables the folding.
• Finally, I was thinking to follow up with a doc patch. I don't want to advertise this just yet, but want to get everything in, and then add it to the docs.

Does that sound like an idea?

[SjoerdMeijer]

Removed the separate function that created the loop.llvm.vectorize.predicate metadata. This is now just part of function createLoopVectorizeMetadata, that creates all other vectorize metadata.

[Meinersbur]

I prefer having the documentation change to be in the same patch as the functional change. Makes it easier to check whether they match.

clang/test/CodeGenCXX/pragma-loop.cpp

163 ↗

(On Diff #210857)

I spent quite some time in the past to ensure that the MDNode matchers use regexes instead of literals. I'd suggest you create a new test file for vectorize_predicate, then you don't have to change anything here.

clang/test/Parser/pragma-loop.cpp

253 ↗

(On Diff #210857)

Can you make the #pragma have its own line. You can use expected-error@+1 to make it match another line.

[SjoerdMeijer]

• Moved the codegen test to a separate file
• Added a langref description for this new metadata node.

[aaron.ballman]

Is it intentional that this review has no reviewers listed (like, is this a work in progress you don't expect review on yet)?

clang/include/clang/Basic/Attr.td

2985 ↗

(On Diff #211042)

You need to update AttrDocs.td for this as well.

[SjoerdMeijer]

Is it intentional that this review has no reviewers listed (like, is this a work in progress you don't expect review on yet)?

No, sorry about this, that's not intentional. It started indeed as a work-in-progress patch when I wrote to the clang/llvm list with an idea about this. I wanted to show some code too, but didn't add reviewers at that point. In the mean time, this patch has evolved quite a bit, but I never bothered to add reviewers, but will do.

You need to update AttrDocs.td for this as well.

Thanks for the suggestion! Will do.

[SjoerdMeijer]

More doc changes added to AttrDocs.td and LangRef.rst

[SjoerdMeijer]

Apologies for the early ping! But I'm off next weeks, so it would be nice to get this in before that if there are no further comments.
Tomorrow, I will upload another diff that builds on top D64916, which enables code-generation with tail loops folded, thus also demonstrating an end to end test.

[aaron.ballman]

LGTM!

[SjoerdMeijer]

Many thanks for reviewing!

cfe/trunk/docs/LanguageExtensions.rst

 these two instantiations, ``twice`` will be optimized (because its definition
 was outside the region) and ``thrice`` will not be optimized.
 
 Extensions for loop hint optimizations
 ======================================
 
 The ``#pragma clang loop`` directive is used to specify hints for optimizing the
 subsequent for, while, do-while, or c++11 range-based for loop. The directive
-provides options for vectorization, interleaving, unrolling and
+provides options for vectorization, interleaving, predication, unrolling and
 distribution. Loop hints can be specified before any loop and will be ignored if
 the optimization is not safe to apply.
 
-Vectorization and Interleaving
-------------------------------
+Vectorization, Interleaving, and Predication
+--------------------------------------------
 
 A vectorized loop performs multiple iterations of the original loop
 in parallel using vector instructions. The instruction set of the target
 processor determines which vector instructions are available and their vector
 widths. This restricts the types of loops that can be vectorized. The vectorizer
 automatically determines if the loop is safe and profitable to vectorize. A
 vector instruction cost model is used to select the vector width.
 
.. code-block:: c++
   #pragma clang loop interleave_count(2)
   for(...) {
     ...
   }
 
 Specifying a width/count of 1 disables the optimization, and is equivalent to
 ``vectorize(disable)`` or ``interleave(disable)``.
 
+Vector predication is enabled by ``vectorize_predicate(enable)``, for example:
+
+.. code-block:: c++
+
+  #pragma clang loop vectorize(enable)
+  #pragma clang loop vectorize_predicate(enable)
+  for(...) {
+    ...
+  }
+
+This predicates (masks) all instructions in the loop, which allows the scalar
+remainder loop (the tail) to be folded into the main vectorized loop. This
+might be more efficient when vector predication is efficiently supported by the
+target platform.
+
 Loop Unrolling
 --------------
 
 Unrolling a loop reduces the loop control overhead and exposes more
 opportunities for ILP. Loops can be fully or partially unrolled. Full unrolling
 eliminates the loop and replaces it with an enumerated sequence of loop
 iterations. Full unrolling is only possible if the loop trip count is known at
 compile time. Partial unrolling replicates the loop body within the loop and

cfe/trunk/include/clang/Basic/Attr.td

def LoopHint : Attr {
   let Spellings = [Pragma<"clang", "loop">, Pragma<"", "unroll">,
                    Pragma<"", "nounroll">, Pragma<"", "unroll_and_jam">,
                    Pragma<"", "nounroll_and_jam">];
 
   /// State of the loop optimization specified by the spelling.
   let Args = [EnumArgument<"Option", "OptionType",
                           ["vectorize", "vectorize_width", "interleave", "interleave_count",
                            "unroll", "unroll_count", "unroll_and_jam", "unroll_and_jam_count",
-                           "pipeline", "pipeline_initiation_interval", "distribute"],
+                           "pipeline", "pipeline_initiation_interval", "distribute",
+                           "vectorize_predicate"],
                           ["Vectorize", "VectorizeWidth", "Interleave", "InterleaveCount",
                            "Unroll", "UnrollCount", "UnrollAndJam", "UnrollAndJamCount",
-                           "PipelineDisabled", "PipelineInitiationInterval", "Distribute"]>,
+                           "PipelineDisabled", "PipelineInitiationInterval", "Distribute",
+                           "VectorizePredicate"]>,
               EnumArgument<"State", "LoopHintState",
                            ["enable", "disable", "numeric", "assume_safety", "full"],
                            ["Enable", "Disable", "Numeric", "AssumeSafety", "Full"]>,
               ExprArgument<"Value">];
 
   let AdditionalMembers = [{
   static const char *getOptionName(int Option) {
     switch(Option) {
     case Vectorize: return "vectorize";
     case VectorizeWidth: return "vectorize_width";
     case Interleave: return "interleave";
     case InterleaveCount: return "interleave_count";
     case Unroll: return "unroll";
     case UnrollCount: return "unroll_count";
     case UnrollAndJam: return "unroll_and_jam";
     case UnrollAndJamCount: return "unroll_and_jam_count";
     case PipelineDisabled: return "pipeline";
     case PipelineInitiationInterval: return "pipeline_initiation_interval";
     case Distribute: return "distribute";
+    case VectorizePredicate: return "vectorize_predicate";
     }
     llvm_unreachable("Unhandled LoopHint option.");
   }
 
   void printPrettyPragma(raw_ostream &OS, const PrintingPolicy &Policy) const {
     unsigned SpellingIndex = getSpellingListIndex();
     // For "#pragma unroll" and "#pragma nounroll" the string "unroll" or
     // "nounroll" is already emitted as the pragma name.

cfe/trunk/include/clang/Basic/AttrDocs.td

 }
 
 def LoopHintDocs : Documentation {
   let Category = DocCatStmt;
   let Heading = "#pragma clang loop";
   let Content = [{
 The ``#pragma clang loop`` directive allows loop optimization hints to be
 specified for the subsequent loop. The directive allows pipelining to be
-disabled, or vectorization, interleaving, and unrolling to be enabled or disabled.
-Vector width, interleave count, unrolling count, and the initiation interval
-for pipelining can be explicitly specified. See `language extensions
+disabled, or vectorization, vector predication, interleaving, and unrolling to
+be enabled or disabled. Vector width, vector predication, interleave count,
+unrolling count, and the initiation interval for pipelining can be explicitly
+specified. See `language extensions
 <http://clang.llvm.org/docs/LanguageExtensions.html#extensions-for-loop-hint-optimizations>`_
 for details.
   }];
 }
 
 def UnrollHintDocs : Documentation {
   let Category = DocCatStmt;
   let Heading = "#pragma unroll, #pragma nounroll";
def HIPPinnedShadowDocs : Documentation {
   let Content = [{
 The GNU style attribute __attribute__((hip_pinned_shadow)) or MSVC style attribute
 __declspec(hip_pinned_shadow) can be added to the definition of a global variable
 to indicate it is a HIP pinned shadow variable. A HIP pinned shadow variable can
 be accessed on both device side and host side. It has external linkage and is
 not initialized on device side. It has internal linkage and is initialized by
 the initializer on host side.
   }];
-}
+}
 

cfe/trunk/include/clang/Basic/DiagnosticParseKinds.td

 
 // Pragma loop support.
 def err_pragma_loop_missing_argument : Error<
   "missing argument; expected %select{an integer value|"
   "'enable'%select{|, 'full'}1%select{|, 'assume_safety'}2 or 'disable'}0">;
 def err_pragma_loop_invalid_option : Error<
   "%select{invalid|missing}0 option%select{ %1|}0; expected vectorize, "
   "vectorize_width, interleave, interleave_count, unroll, unroll_count, "
-  "pipeline, pipeline_initiation_interval, or distribute">;
+  "pipeline, pipeline_initiation_interval, vectorize_predicate, or distribute">;
 
 def err_pragma_fp_invalid_option : Error<
   "%select{invalid|missing}0 option%select{ %1|}0; expected contract">;
 def err_pragma_fp_invalid_argument : Error<
   "unexpected argument '%0' to '#pragma clang fp %1'; "
   "expected 'on', 'fast' or 'off'">;
 def err_pragma_fp_scope : Error<
   "'#pragma clang fp' can only appear at file scope or at the start of a "

cfe/trunk/lib/CodeGen/CGLoopInfo.h

struct LoopAttributes {
   LVEnableState VectorizeEnable;
 
   /// Value for llvm.loop.unroll.* metadata (enable, disable, or full).
   LVEnableState UnrollEnable;
 
   /// Value for llvm.loop.unroll_and_jam.* metadata (enable, disable, or full).
   LVEnableState UnrollAndJamEnable;
 
+  /// Value for llvm.loop.vectorize.predicate metadata
+  LVEnableState VectorizePredicateEnable;
+
   /// Value for llvm.loop.vectorize.width metadata.
   unsigned VectorizeWidth;
 
   /// Value for llvm.loop.interleave.count metadata.
   unsigned InterleaveCount;
 
   /// llvm.unroll.
   unsigned UnrollCount;
StagedAttrs.DistributeEnable =
         Enable ? LoopAttributes::Enable : LoopAttributes::Disable;
   }
 
   /// Set the next pushed loop unroll state.
   void setUnrollState(const LoopAttributes::LVEnableState &State) {
     StagedAttrs.UnrollEnable = State;
   }
 
+  /// Set the next pushed vectorize predicate state.
+  void setVectorizePredicateState(const LoopAttributes::LVEnableState &State) {
+    StagedAttrs.VectorizePredicateEnable = State;
+  }
+
   /// Set the next pushed loop unroll_and_jam state.
   void setUnrollAndJamState(const LoopAttributes::LVEnableState &State) {
     StagedAttrs.UnrollAndJamEnable = State;
   }
 
   /// Set the vectorize width for the next loop pushed.
   void setVectorizeWidth(unsigned W) { StagedAttrs.VectorizeWidth = W; }
 

cfe/trunk/lib/CodeGen/CGLoopInfo.cpp

LoopInfo::createLoopVectorizeMetadata(const LoopAttributes &Attrs,
                                       ArrayRef<Metadata *> LoopProperties,
                                       bool &HasUserTransforms) {
   LLVMContext &Ctx = Header->getContext();
 
   Optional<bool> Enabled;
   if (Attrs.VectorizeEnable == LoopAttributes::Disable)
     Enabled = false;
   else if (Attrs.VectorizeEnable != LoopAttributes::Unspecified ||
+           Attrs.VectorizePredicateEnable != LoopAttributes::Unspecified ||
            Attrs.InterleaveCount != 0 || Attrs.VectorizeWidth != 0)
     Enabled = true;
 
   if (Enabled != true) {
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
       NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
       NewLoopProperties.push_back(
LoopInfo::createLoopVectorizeMetadata(const LoopAttributes &Attrs,
   MDNode *Followup = createUnrollAndJamMetadata(Attrs, FollowupLoopProperties,
                                                 FollowupHasTransforms);
 
   SmallVector<Metadata *, 4> Args;
   TempMDTuple TempNode = MDNode::getTemporary(Ctx, None);
   Args.push_back(TempNode.get());
   Args.append(LoopProperties.begin(), LoopProperties.end());
 
+  // Setting vectorize.predicate
+  if (Attrs.VectorizePredicateEnable != LoopAttributes::Unspecified) {
+    Metadata *Vals[] = {
+        MDString::get(Ctx, "llvm.loop.vectorize.predicate.enable"),
+        ConstantAsMetadata::get(ConstantInt::get(
+            llvm::Type::getInt1Ty(Ctx),
+            (Attrs.VectorizePredicateEnable == LoopAttributes::Enable)))};
+    Args.push_back(MDNode::get(Ctx, Vals));
+  }
+
   // Setting vectorize.width
   if (Attrs.VectorizeWidth > 0) {
     Metadata *Vals[] = {
         MDString::get(Ctx, "llvm.loop.vectorize.width"),
         ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
                                                  Attrs.VectorizeWidth))};
     Args.push_back(MDNode::get(Ctx, Vals));
   }
MDNode *LoopInfo::createMetadata(
   LoopProperties.insert(LoopProperties.end(), AdditionalLoopProperties.begin(),
                         AdditionalLoopProperties.end());
   return createFullUnrollMetadata(Attrs, LoopProperties, HasUserTransforms);
 }
 
 LoopAttributes::LoopAttributes(bool IsParallel)
     : IsParallel(IsParallel), VectorizeEnable(LoopAttributes::Unspecified),
       UnrollEnable(LoopAttributes::Unspecified),
-      UnrollAndJamEnable(LoopAttributes::Unspecified), VectorizeWidth(0),
+      UnrollAndJamEnable(LoopAttributes::Unspecified),
+      VectorizePredicateEnable(LoopAttributes::Unspecified), VectorizeWidth(0),
       InterleaveCount(0), UnrollCount(0), UnrollAndJamCount(0),
       DistributeEnable(LoopAttributes::Unspecified), PipelineDisabled(false),
       PipelineInitiationInterval(0) {}
 
 void LoopAttributes::clear() {
   IsParallel = false;
   VectorizeWidth = 0;
   InterleaveCount = 0;
   UnrollCount = 0;
   UnrollAndJamCount = 0;
   VectorizeEnable = LoopAttributes::Unspecified;
   UnrollEnable = LoopAttributes::Unspecified;
   UnrollAndJamEnable = LoopAttributes::Unspecified;
+  VectorizePredicateEnable = LoopAttributes::Unspecified;
   DistributeEnable = LoopAttributes::Unspecified;
   PipelineDisabled = false;
   PipelineInitiationInterval = 0;
 }
 
 LoopInfo::LoopInfo(BasicBlock *Header, const LoopAttributes &Attrs,
                    const llvm::DebugLoc &StartLoc, const llvm::DebugLoc &EndLoc,
                    LoopInfo *Parent)
     : Header(Header), Attrs(Attrs), StartLoc(StartLoc), EndLoc(EndLoc),
       Parent(Parent) {
 
   if (Attrs.IsParallel) {
     // Create an access group for this loop.
     LLVMContext &Ctx = Header->getContext();
     AccGroup = MDNode::getDistinct(Ctx, {});
   }
 
   if (!Attrs.IsParallel && Attrs.VectorizeWidth == 0 &&
       Attrs.InterleaveCount == 0 && Attrs.UnrollCount == 0 &&
       Attrs.UnrollAndJamCount == 0 && !Attrs.PipelineDisabled &&
       Attrs.PipelineInitiationInterval == 0 &&
+      Attrs.VectorizePredicateEnable == LoopAttributes::Unspecified &&
       Attrs.VectorizeEnable == LoopAttributes::Unspecified &&
       Attrs.UnrollEnable == LoopAttributes::Unspecified &&
       Attrs.UnrollAndJamEnable == LoopAttributes::Unspecified &&
       Attrs.DistributeEnable == LoopAttributes::Unspecified && !StartLoc &&
       !EndLoc)
     return;
 
   TempLoopID = MDNode::getTemporary(Header->getContext(), None);
if (Parent && (Parent->Attrs.UnrollAndJamEnable ||
     LoopAttributes BeforeJam, AfterJam;
 
     BeforeJam.IsParallel = AfterJam.IsParallel = Attrs.IsParallel;
 
     BeforeJam.VectorizeWidth = Attrs.VectorizeWidth;
     BeforeJam.InterleaveCount = Attrs.InterleaveCount;
     BeforeJam.VectorizeEnable = Attrs.VectorizeEnable;
     BeforeJam.DistributeEnable = Attrs.DistributeEnable;
+    BeforeJam.VectorizePredicateEnable = Attrs.VectorizePredicateEnable;
 
     switch (Attrs.UnrollEnable) {
     case LoopAttributes::Unspecified:
     case LoopAttributes::Disable:
       BeforeJam.UnrollEnable = Attrs.UnrollEnable;
       AfterJam.UnrollEnable = Attrs.UnrollEnable;
       break;
     case LoopAttributes::Full:
       BeforeJam.UnrollEnable = LoopAttributes::Full;
       break;
     case LoopAttributes::Enable:
       AfterJam.UnrollEnable = LoopAttributes::Enable;
       break;
     }
 
+    AfterJam.VectorizePredicateEnable = Attrs.VectorizePredicateEnable;
     AfterJam.UnrollCount = Attrs.UnrollCount;
     AfterJam.PipelineDisabled = Attrs.PipelineDisabled;
     AfterJam.PipelineInitiationInterval = Attrs.PipelineInitiationInterval;
 
     // If this loop is subject of an unroll-and-jam by the parent loop, and has
     // an unroll-and-jam annotation itself, we have to decide whether to first
     // apply the parent's unroll-and-jam or this loop's unroll-and-jam. The
     // UnrollAndJam pass processes loops from inner to outer, so we apply the
     // inner first.
     BeforeJam.UnrollAndJamCount = Attrs.UnrollAndJamCount;
     BeforeJam.UnrollAndJamEnable = Attrs.UnrollAndJamEnable;
 
     // Set the inner followup metadata to process by the outer loop. Only
     // consider the first inner loop.
     if (!Parent->UnrollAndJamInnerFollowup) {
       // Splitting the attributes into a BeforeJam and an AfterJam part will
       // stop 'llvm.loop.isvectorized' (generated by vectorization in BeforeJam)
       // to be forwarded to the AfterJam part. We detect the situation here and
       // add it manually.
       SmallVector<Metadata *, 1> BeforeLoopProperties;
       if (BeforeJam.VectorizeEnable != LoopAttributes::Unspecified ||
+          BeforeJam.VectorizePredicateEnable != LoopAttributes::Unspecified ||
           BeforeJam.InterleaveCount != 0 || BeforeJam.VectorizeWidth != 0)
         BeforeLoopProperties.push_back(
             MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.isvectorized")));
 
       bool InnerFollowupHasTransform = false;
       MDNode *InnerFollowup = createMetadata(AfterJam, BeforeLoopProperties,
                                              InnerFollowupHasTransform);
       if (InnerFollowupHasTransform)
case LoopHintAttr::Disable:
         setInterleaveCount(1);
         break;
       case LoopHintAttr::Unroll:
         setUnrollState(LoopAttributes::Disable);
         break;
       case LoopHintAttr::UnrollAndJam:
         setUnrollAndJamState(LoopAttributes::Disable);
         break;
+      case LoopHintAttr::VectorizePredicate:
+        setVectorizePredicateState(LoopAttributes::Disable);
+        break;
       case LoopHintAttr::Distribute:
         setDistributeState(false);
         break;
       case LoopHintAttr::PipelineDisabled:
         setPipelineDisabled(true);
         break;
       case LoopHintAttr::UnrollCount:
       case LoopHintAttr::UnrollAndJamCount:
case LoopHintAttr::Enable:
         setVectorizeEnable(true);
         break;
       case LoopHintAttr::Unroll:
         setUnrollState(LoopAttributes::Enable);
         break;
       case LoopHintAttr::UnrollAndJam:
         setUnrollAndJamState(LoopAttributes::Enable);
         break;
+      case LoopHintAttr::VectorizePredicate:
+        setVectorizePredicateState(LoopAttributes::Enable);
+        break;
       case LoopHintAttr::Distribute:
         setDistributeState(true);
         break;
       case LoopHintAttr::UnrollCount:
       case LoopHintAttr::UnrollAndJamCount:
       case LoopHintAttr::VectorizeWidth:
       case LoopHintAttr::InterleaveCount:
       case LoopHintAttr::PipelineDisabled:
       case LoopHintAttr::PipelineInitiationInterval:
         llvm_unreachable("Options cannot enabled.");
         break;
       }
       break;
     case LoopHintAttr::AssumeSafety:
       switch (Option) {
       case LoopHintAttr::Vectorize:
       case LoopHintAttr::Interleave:
         // Apply "llvm.mem.parallel_loop_access" metadata to load/stores.
         setParallel(true);
         setVectorizeEnable(true);
         break;
       case LoopHintAttr::Unroll:
       case LoopHintAttr::UnrollAndJam:
+      case LoopHintAttr::VectorizePredicate:
       case LoopHintAttr::UnrollCount:
       case LoopHintAttr::UnrollAndJamCount:
       case LoopHintAttr::VectorizeWidth:
       case LoopHintAttr::InterleaveCount:
       case LoopHintAttr::Distribute:
       case LoopHintAttr::PipelineDisabled:
       case LoopHintAttr::PipelineInitiationInterval:
         llvm_unreachable("Options cannot be used to assume mem safety.");
case LoopHintAttr::Full:
       case LoopHintAttr::Interleave:
       case LoopHintAttr::UnrollCount:
       case LoopHintAttr::UnrollAndJamCount:
       case LoopHintAttr::VectorizeWidth:
       case LoopHintAttr::InterleaveCount:
       case LoopHintAttr::Distribute:
       case LoopHintAttr::PipelineDisabled:
       case LoopHintAttr::PipelineInitiationInterval:
+      case LoopHintAttr::VectorizePredicate:
         llvm_unreachable("Options cannot be used with 'full' hint.");
         break;
       }
       break;
     case LoopHintAttr::Numeric:
       switch (Option) {
       case LoopHintAttr::VectorizeWidth:
         setVectorizeWidth(ValueInt);
         break;
       case LoopHintAttr::InterleaveCount:
         setInterleaveCount(ValueInt);
         break;
       case LoopHintAttr::UnrollCount:
         setUnrollCount(ValueInt);
         break;
       case LoopHintAttr::UnrollAndJamCount:
         setUnrollAndJamCount(ValueInt);
         break;
       case LoopHintAttr::PipelineInitiationInterval:
         setPipelineInitiationInterval(ValueInt);
         break;
       case LoopHintAttr::Unroll:
       case LoopHintAttr::UnrollAndJam:
+      case LoopHintAttr::VectorizePredicate:
       case LoopHintAttr::Vectorize:
       case LoopHintAttr::Interleave:
       case LoopHintAttr::Distribute:
       case LoopHintAttr::PipelineDisabled:
         llvm_unreachable("Options cannot be assigned a value.");
         break;
       }
       break;

cfe/trunk/lib/Parse/ParsePragma.cpp

bool Parser::HandlePragmaLoopHint(LoopHint &Hint) {
   if (OptionInfo) { // Pragma Unroll does not specify an option.
     OptionUnroll = OptionInfo->isStr("unroll");
     OptionUnrollAndJam = OptionInfo->isStr("unroll_and_jam");
     OptionDistribute = OptionInfo->isStr("distribute");
     OptionPipelineDisabled = OptionInfo->isStr("pipeline");
     StateOption = llvm::StringSwitch<bool>(OptionInfo->getName())
                       .Case("vectorize", true)
                       .Case("interleave", true)
+                      .Case("vectorize_predicate", true)
                       .Default(false) ||
                   OptionUnroll || OptionUnrollAndJam || OptionDistribute ||
                   OptionPipelineDisabled;
   }
 
   bool AssumeSafetyArg = !OptionUnroll && !OptionUnrollAndJam &&
                          !OptionDistribute && !OptionPipelineDisabled;
   // Verify loop hint has an argument.

 ///
 ///  loop-hints:
 ///    loop-hint loop-hints[opt]
 ///
 ///  loop-hint:
 ///    'vectorize' '(' loop-hint-keyword ')'
 ///    'interleave' '(' loop-hint-keyword ')'
 ///    'unroll' '(' unroll-hint-keyword ')'
+///    'vectorize_predicate' '(' loop-hint-keyword ')'
 ///    'vectorize_width' '(' loop-hint-value ')'
 ///    'interleave_count' '(' loop-hint-value ')'
 ///    'unroll_count' '(' loop-hint-value ')'
 ///    'pipeline' '(' disable ')'
 ///    'pipeline_initiation_interval' '(' loop-hint-value ')'
 ///
 ///  loop-hint-keyword:
 ///    'enable'
while (Tok.is(tok::identifier)) {
     Token Option = Tok;
     IdentifierInfo *OptionInfo = Tok.getIdentifierInfo();
 
     bool OptionValid = llvm::StringSwitch<bool>(OptionInfo->getName())
                            .Case("vectorize", true)
                            .Case("interleave", true)
                            .Case("unroll", true)
                            .Case("distribute", true)
+                           .Case("vectorize_predicate", true)
                            .Case("vectorize_width", true)
                            .Case("interleave_count", true)
                            .Case("unroll_count", true)
                            .Case("pipeline", true)
                            .Case("pipeline_initiation_interval", true)
                            .Default(false);
     if (!OptionValid) {
       PP.Diag(Tok.getLocation(), diag::err_pragma_loop_invalid_option)

cfe/trunk/lib/Sema/SemaStmtAttr.cpp

if (PragmaName == "nounroll") {
     // #pragma clang loop ...
     assert(OptionLoc && OptionLoc->Ident &&
            "Attribute must have valid option info.");
     Option = llvm::StringSwitch<LoopHintAttr::OptionType>(
                  OptionLoc->Ident->getName())
                  .Case("vectorize", LoopHintAttr::Vectorize)
                  .Case("vectorize_width", LoopHintAttr::VectorizeWidth)
                  .Case("interleave", LoopHintAttr::Interleave)
+                 .Case("vectorize_predicate", LoopHintAttr::VectorizePredicate)
                  .Case("interleave_count", LoopHintAttr::InterleaveCount)
                  .Case("unroll", LoopHintAttr::Unroll)
                  .Case("unroll_count", LoopHintAttr::UnrollCount)
                  .Case("pipeline", LoopHintAttr::PipelineDisabled)
                  .Case("pipeline_initiation_interval",
                        LoopHintAttr::PipelineInitiationInterval)
                  .Case("distribute", LoopHintAttr::Distribute)
                  .Default(LoopHintAttr::Vectorize);
     if (Option == LoopHintAttr::VectorizeWidth ||
         Option == LoopHintAttr::InterleaveCount ||
         Option == LoopHintAttr::UnrollCount ||
         Option == LoopHintAttr::PipelineInitiationInterval) {
       assert(ValueExpr && "Attribute must have a valid value expression.");
       if (S.CheckLoopHintExpr(ValueExpr, St->getBeginLoc()))
         return nullptr;
       State = LoopHintAttr::Numeric;
     } else if (Option == LoopHintAttr::Vectorize ||
                Option == LoopHintAttr::Interleave ||
+               Option == LoopHintAttr::VectorizePredicate ||
                Option == LoopHintAttr::Unroll ||
                Option == LoopHintAttr::Distribute ||
                Option == LoopHintAttr::PipelineDisabled) {
       assert(StateLoc && StateLoc->Ident && "Loop hint must have an argument");
       if (StateLoc->Ident->isStr("disable"))
         State = LoopHintAttr::Disable;
       else if (StateLoc->Ident->isStr("assume_safety"))
         State = LoopHintAttr::AssumeSafety;
CheckForIncompatibleAttributes(Sema &S,
   // transformation). The numeric form form provides an integer hint (for
   // example, unroll count) to the transformer. The following array accumulates
   // the hints encountered while iterating through the attributes to check for
   // compatibility.
   struct {
     const LoopHintAttr *StateAttr;
     const LoopHintAttr *NumericAttr;
   } HintAttrs[] = {{nullptr, nullptr}, {nullptr, nullptr}, {nullptr, nullptr},
-                   {nullptr, nullptr}, {nullptr, nullptr}, {nullptr, nullptr}};
+                   {nullptr, nullptr}, {nullptr, nullptr}, {nullptr, nullptr},
+                   {nullptr, nullptr}};
 
   for (const auto *I : Attrs) {
     const LoopHintAttr *LH = dyn_cast<LoopHintAttr>(I);
 
     // Skip non loop hint attributes
     if (!LH)
       continue;
 
     LoopHintAttr::OptionType Option = LH->getOption();
     enum {
       Vectorize,
       Interleave,
       Unroll,
       UnrollAndJam,
       Distribute,
-      Pipeline
+      Pipeline,
+      VectorizePredicate
     } Category;
     switch (Option) {
     case LoopHintAttr::Vectorize:
     case LoopHintAttr::VectorizeWidth:
       Category = Vectorize;
       break;
     case LoopHintAttr::Interleave:
     case LoopHintAttr::InterleaveCount:
for (const auto *I : Attrs) {
     case LoopHintAttr::Distribute:
       // Perform the check for duplicated 'distribute' hints.
       Category = Distribute;
       break;
     case LoopHintAttr::PipelineDisabled:
     case LoopHintAttr::PipelineInitiationInterval:
       Category = Pipeline;
       break;
+    case LoopHintAttr::VectorizePredicate:
+      Category = VectorizePredicate;
+      break;
     };
 
     assert(Category < sizeof(HintAttrs) / sizeof(HintAttrs[0]));
     auto &CategoryState = HintAttrs[Category];
     const LoopHintAttr *PrevAttr;
     if (Option == LoopHintAttr::Vectorize ||
         Option == LoopHintAttr::Interleave || Option == LoopHintAttr::Unroll ||
         Option == LoopHintAttr::UnrollAndJam ||
+        Option == LoopHintAttr::VectorizePredicate ||
         Option == LoopHintAttr::PipelineDisabled ||
         Option == LoopHintAttr::Distribute) {
       // Enable|Disable|AssumeSafety hint.  For example, vectorize(enable).
       PrevAttr = CategoryState.StateAttr;
       CategoryState.StateAttr = LH;
     } else {
       // Numeric hint.  For example, vectorize_width(8).
       PrevAttr = CategoryState.NumericAttr;

cfe/trunk/test/AST/ast-print-pragmas.cpp

// CHECK-NEXT: while (i < Length)
   while (i < Length) {
     List[i] = i * 2;
     i++;
   }
 
 // CHECK: #pragma clang loop distribute(disable)
 // CHECK-NEXT: #pragma clang loop vectorize(enable)
 // CHECK-NEXT: #pragma clang loop interleave(disable)
+// CHECK-NEXT: #pragma clang loop vectorize_predicate(disable)
 
 #pragma clang loop distribute(disable)
 #pragma clang loop vectorize(enable)
 #pragma clang loop interleave(disable)
+#pragma clang loop vectorize_predicate(disable)
 // CHECK-NEXT: while (i - 1 < Length)
   while (i - 1 < Length) {
     List[i] = i * 2;
     i++;
   }
 
 // CHECK: #pragma clang loop distribute(enable)
 // CHECK-NEXT: #pragma clang loop vectorize(disable)
 // CHECK-NEXT: #pragma clang loop interleave(enable)
+// CHECK-NEXT: #pragma clang loop vectorize_predicate(enable)
 
 #pragma clang loop distribute(enable)
 #pragma clang loop vectorize(disable)
 #pragma clang loop interleave(enable)
+#pragma clang loop vectorize_predicate(enable)
 // CHECK-NEXT: while (i - 2 < Length)
   while (i - 2 < Length) {
     List[i] = i * 2;
     i++;
   }
 }
 
 template <int V, int I>

cfe/trunk/test/CodeGenCXX/pragma-loop-predicate.cpp

+// RUN: %clang_cc1 -triple x86_64-apple-darwin -std=c++11 -emit-llvm -o - %s | FileCheck %s
+
+void test0(int *List, int Length) {
+// CHECK-LABEL: @{{.*}}test0{{.*}}(
+// CHECK: br label {{.*}}, !llvm.loop ![[LOOP0:.*]]
+
+  #pragma clang loop vectorize(enable)
+  for (int i = 0; i < Length; i++)
+    List[i] = i * 2;
+}
+
+void test1(int *List, int Length) {
+// CHECK-LABEL: @{{.*}}test1{{.*}}(
+// CHECK: br label {{.*}}, !llvm.loop ![[LOOP1:.*]]
+
+  #pragma clang loop vectorize(enable) vectorize_predicate(enable)
+  for (int i = 0; i < Length; i++)
+    List[i] = i * 2;
+}
+
+void test2(int *List, int Length) {
+// CHECK-LABEL: @{{.*}}test2{{.*}}(
+// CHECK:       br label {{.*}}, !llvm.loop ![[LOOP2:.*]]
+
+  #pragma clang loop vectorize(enable) vectorize_predicate(disable)
+  for (int i = 0; i < Length; i++)
+    List[i] = i * 2;
+}
+
+// CHECK:      ![[LOOP0]] = distinct !{![[LOOP0]], !3}
+// CHECK-NEXT: !3 = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: ![[LOOP1]] = distinct !{![[LOOP1]], !5, !3}
+// CHECK-NEXT: !5 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
+// CHECK-NEXT: ![[LOOP2]] = distinct !{![[LOOP2]], !7, !3}
+// CHECK-NEXT: !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 false}

cfe/trunk/test/Parser/pragma-loop.cpp

void test_type_template_vectorize(int *List, int Length) {
   }
 }
 
 void test(int *List, int Length) {
   int i = 0;
 
 #pragma clang loop vectorize(enable)
 #pragma clang loop interleave(enable)
+#pragma clang loop vectorize_predicate(enable)
 #pragma clang loop unroll(full)
   while (i + 1 < Length) {
     List[i] = i;
   }
 
 #pragma clang loop vectorize_width(4)
 #pragma clang loop interleave_count(8)
 #pragma clang loop unroll_count(16)
   while (i < Length) {
     List[i] = i;
   }
 
 #pragma clang loop vectorize(disable)
 #pragma clang loop interleave(disable)
+#pragma clang loop vectorize_predicate(disable)
 #pragma clang loop unroll(disable)
   while (i - 1 < Length) {
     List[i] = i;
   }
 
 #pragma clang loop vectorize_width(4) interleave_count(8) unroll_count(16)
   while (i - 2 < Length) {
     List[i] = i;
   }
 
 #pragma clang loop interleave_count(16)
   while (i - 3 < Length) {
     List[i] = i;
   }
 
   int VList[Length];
-#pragma clang loop vectorize(disable) interleave(disable) unroll(disable)
+#pragma clang loop vectorize(disable) interleave(disable) unroll(disable) vectorize_predicate(disable)
   for (int j : VList) {
     VList[j] = List[j];
   }
 
 #pragma clang loop distribute(enable)
   for (int j : VList) {
     VList[j] = List[j];
   }
 
 #pragma clang loop distribute(disable)
   for (int j : VList) {
     VList[j] = List[j];
   }
 
   test_nontype_template_param<4, 8>(List, Length);
 
 /* expected-error {{expected '('}} */ #pragma clang loop vectorize
 /* expected-error {{expected '('}} */ #pragma clang loop interleave
+/* expected-error {{expected '('}} */ #pragma clang loop vectorize_predicate
 /* expected-error {{expected '('}} */ #pragma clang loop unroll
 /* expected-error {{expected '('}} */ #pragma clang loop distribute
 
 /* expected-error {{expected ')'}} */ #pragma clang loop vectorize(enable
 /* expected-error {{expected ')'}} */ #pragma clang loop interleave(enable
+/* expected-error {{expected ')'}} */ #pragma clang loop vectorize_predicate(enable
 /* expected-error {{expected ')'}} */ #pragma clang loop unroll(full
 /* expected-error {{expected ')'}} */ #pragma clang loop distribute(enable
 
 /* expected-error {{expected ')'}} */ #pragma clang loop vectorize_width(4
 /* expected-error {{expected ')'}} */ #pragma clang loop interleave_count(4
 /* expected-error {{expected ')'}} */ #pragma clang loop unroll_count(4
 
 /* expected-error {{missing argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop vectorize()
 /* expected-error {{missing argument; expected an integer value}} */ #pragma clang loop interleave_count()
 /* expected-error {{missing argument; expected 'enable', 'full' or 'disable'}} */ #pragma clang loop unroll()
 /* expected-error {{missing argument; expected 'enable' or 'disable'}} */ #pragma clang loop distribute()
 
-/* expected-error {{missing option; expected vectorize, vectorize_width, interleave, interleave_count, unroll, unroll_count, pipeline, pipeline_initiation_interval, or distribute}} */ #pragma clang loop
+/* expected-error {{missing option; expected vectorize, vectorize_width, interleave, interleave_count, unroll, unroll_count, pipeline, pipeline_initiation_interval, vectorize_predicate, or distribute}} */ #pragma clang loop
 /* expected-error {{invalid option 'badkeyword'}} */ #pragma clang loop badkeyword
 /* expected-error {{invalid option 'badkeyword'}} */ #pragma clang loop badkeyword(enable)
 /* expected-error {{invalid option 'badkeyword'}} */ #pragma clang loop vectorize(enable) badkeyword(4)
 /* expected-warning {{extra tokens at end of '#pragma clang loop'}} */ #pragma clang loop vectorize(enable) ,
   while (i-4 < Length) {
     List[i] = i;
   }
 
/* expected-error {{incompatible directives 'unroll(disable)' and 'unroll_count(4)'}} */ #pragma clang loop unroll(disable)
   while (i-8 < Length) {
     List[i] = i;
   }
 
 #pragma clang loop vectorize(enable)
 /* expected-error {{duplicate directives 'vectorize(enable)' and 'vectorize(disable)'}} */ #pragma clang loop vectorize(disable)
 #pragma clang loop interleave(enable)
 /* expected-error {{duplicate directives 'interleave(enable)' and 'interleave(disable)'}} */ #pragma clang loop interleave(disable)
+#pragma clang loop vectorize_predicate(enable)
+/* expected-error@+1 {{duplicate directives 'vectorize_predicate(enable)' and 'vectorize_predicate(disable)'}} */
+#pragma clang loop vectorize_predicate(disable)
 #pragma clang loop unroll(full)
 /* expected-error {{duplicate directives 'unroll(full)' and 'unroll(disable)'}} */ #pragma clang loop unroll(disable)
 #pragma clang loop distribute(enable)
 /* expected-error {{duplicate directives 'distribute(enable)' and 'distribute(disable)'}} */ #pragma clang loop distribute(disable)
   while (i-9 < Length) {
     List[i] = i;
   }
 

 #pragma clang loop unroll(full)
 /* expected-error {{incompatible directives 'unroll(full)' and 'unroll_count(4)'}} */ #pragma clang loop unroll_count(4)
   while (i-11 < Length) {
     List[i] = i;
   }
 
 #pragma clang loop interleave(enable)
 /* expected-error {{expected statement}} */ }
+
+void foo(void) {
+#pragma clang loop vectorize_predicate(enable)
+/* expected-error {{expected statement}} */ }

cfe/trunk/test/Parser/pragma-unroll-and-jam.cpp

 #pragma unroll(4)
   for (int i = 0; i < Length; i++) {
     for (int j = 0; j < Length; j++) {
       List[i * Length + j] = Value;
     }
   }
 
 // pragma clang unroll_and_jam is disabled for the moment
-/* expected-error {{invalid option 'unroll_and_jam'; expected vectorize, vectorize_width, interleave, interleave_count, unroll, unroll_count, pipeline, pipeline_initiation_interval, or distribute}} */ #pragma clang loop unroll_and_jam(4)
+/* expected-error {{invalid option 'unroll_and_jam'; expected vectorize, vectorize_width, interleave, interleave_count, unroll, unroll_count, pipeline, pipeline_initiation_interval, vectorize_predicate, or distribute}} */ #pragma clang loop unroll_and_jam(4)
   for (int i = 0; i < Length; i++) {
     for (int j = 0; j < Length; j++) {
       List[i * Length + j] = Value;
     }
   }
 
 #pragma unroll_and_jam
 /* expected-error {{expected statement}} */ }

llvm/trunk/docs/LangRef.rst

 is a bit. If the bit operand value is 1 vectorization is enabled. A value of
 0 disables vectorization:
 
 .. code-block:: llvm
 
    !0 = !{!"llvm.loop.vectorize.enable", i1 0}
    !1 = !{!"llvm.loop.vectorize.enable", i1 1}
 
+'``llvm.loop.vectorize.predicate.enable``' Metadata
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This metadata selectively enables or disables creating predicated instructions
+for the loop, which can enable folding of the scalar epilogue loop into the
+main loop. The first operand is the string
+``llvm.loop.vectorize.predicate.enable`` and the second operand is a bit. If
+the bit operand value is 1 vectorization is enabled. A value of 0 disables
+vectorization:
+
+.. code-block:: llvm
+
+   !0 = !{!"llvm.loop.vectorize.predicate.enable", i1 0}
+   !1 = !{!"llvm.loop.vectorize.predicate.enable", i1 1}
+
 '``llvm.loop.vectorize.width``' Metadata
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This metadata sets the target width of the vectorizer. The first
 operand is the string ``llvm.loop.vectorize.width`` and the second
 operand is an integer specifying the width. For example:
 
 .. code-block:: llvm