[mlir][spirv] Convert linalg.generic for reduction to SPIR-V ops

[antiagainst]

Summary

This commit adds a pattern to lower linalg.generic for reduction
to spv.GroupNonUniform* ops. Right now this only supports integer
reduction on 1-D input memref. Shader entry point ABI is queried
to make sure that the input memref's shape matches the local
workgroup's invocation configuration. This makes sure that the
workload fits in one local workgroup so that we can leverage
SPIR-V group non-uniform operations.

linglg.generic is a structured op that preserves the right level
of information. It is easier to recognize reduction at this level
than performing analysis on loops.

This commit also exposes getElementPtr in SPIRVLowering.h given
that it's a generally useful utility function.

[merge_guards_bot]

Unit tests: pass. 62199 tests passed, 0 failed and 815 were skipped.

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[rriddle]

mlir/include/mlir/Conversion/LinalgToSPIRV/LinalgToSPIRV.h

4

Should be LLVM Project now.

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

28

Why the inline keyword?

61

Don't use size, it is O(N). Use 'has_single_element' instead.

71

Same here, size() is O(N): !has_single_element(block.without_terminator())

104

This function is huge, is there any way to split it up a bit?

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRVPass.cpp

43

Drop the trivial brace.

mlir/lib/Dialect/SPIRV/TargetAndABI.cpp

70

nit: Drop the newline here.

[mravishankar]

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

37

Nit : s/interators/iterators/

74

I feel like this is not required. If the linalg.generic operation says this is a reduction, we should not be checking the region to verify that it is. linalg as a contract is guaranteeing this is a reduction.

183

This is hard-wiring to IAdd. We probably need to structure this differently. We need to have a pattern to lower the linalg.generic with reduction iterator into the kernel generated here, and then lower the operations within the region separately.

[antiagainst]

Address comments

[antiagainst]

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

28

Added initially because this function was trivial enough. I guess right now it's better to let the compiler decide.

61

Oh, good to know! Thanks!

71

Nice, thanks!

74

I'm not sure I get your idea correctly; but this is checking we are doing the expected kind of reduction (BinaryOp).

104

Good point. Split out two helper functions.

183

It was intentional to only support IAdd. I've re-structured this a bit so it's extensible to other binary op kinds.

[antiagainst]

Revert unrelated commits

[antiagainst]

Clean up unrelated commits again

[antiagainst]

Messed up the revision history with Arc... Please show the diff between "Diff 1" and "Diff 4" to check the modifications.

[merge_guards_bot]

Unit tests: pass. 62199 tests passed, 0 failed and 815 were skipped.

clang-tidy: pass.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[merge_guards_bot]

Unit tests: pass. 62199 tests passed, 0 failed and 815 were skipped.

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[rriddle]

mlir/include/mlir/Dialect/SPIRV/SPIRVControlFlowOps.td

463

Drop the llvm::.

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

33

nit: You can use getOperandTypes() && getResultTypes() respectively.

39

Can you use size() directly on the ArrayAttr?

67

nit: Missing end namespace comment.

81

You should be able to remove the .empty() call as well as the .getBlocks()

[nicolasvasilache]

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

28

Please use LinalgOp::hasBufferSemantics instead of rolling your own

38

Can we create a singleReductionIterator() somewhere in the existing utils and directly equality compare the ArrayAttr iterators against that?
I anticipate this type of things is soon going to become more and more useful, including from Tablegen.

79

Same thing here, we should move these to utils on the Linalg side and make reusable as things are soon going to blow up out of control otherwise.

126

Please split this into a precondition and an apply that must succeed so we can use the precondition as a Constraint and the apply as a simple Rewrite.
I have found this split to be crucial to write custom fused passes with DRR.
I claim this is generally the way we should be writing transformations when we can, which is the case for Linalg ops.

157

I have similar comments re utils here but it will take a bit longer to put things in a reasonable form so please just add a TODO that this should use generic Linalg utils when available.

261

Thanks for doing this as a pattern, it will compose nicely thanks to all of @rriddle's great work!

[mravishankar]

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

74

That is what I am not sure we need to do. You have already checked the generic op is a 1D loop with reduction iterator type. So the body of the generic op is assumed to be a reduction right, i.e. a "binaryOp". Here you are checking whether its an operation which takes two operands, but not necessarily a binary operation that can be used for reduction. In some sense you dont need to check that since the generic op description already told you to assume that the region of the op represents a binary op that can be used for reduction (this is based on my understanding of "reduction" loops in linalg, but I need to double check).

183

The way I see this structured is

1. You check the "structure" of the linalg.generic op to see if it is a 1D reduction. You assume that the body of the generic op represents a binary operation that can be used for the reduction.
2. You write a separate pattern that converts the operations of the body itself into the spirv::GroupNonUniform*Op.

The dialect conversion will first convert the generic op, and then it will attempt to convert the body of the op. The way this is strucutred, it can only handle straight-forward binary operations. THere could be more complicated operations in the region of a generic op that implements the reduction, which would be hard to incorporate in this approach.

With your updated changes, it is easy to extend to other ops, but I think a more general solution is needed where we are not constrained in handling just simple reductions. It might need some more careful design though, which I dont have a full picture of right now. So for now this OK, but please leave a TODO to say something like "handle more general reductions".

[mravishankar]

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

74

Ok, looked into this a little bit more. This is indeed required as of now. Ideally we dont need to do this, but that is orthogonal. No issues here.

183

Thought about this a little bit more. It will be a more sane strategy to not handle cases where the region is "inlined" into the generic op, but rather the region is specified as a function using the fun attribute of the linalg.generic op. Then we can split the lowering of the generic op and the lowering of the function that does the reduction separately.
This too not needed for this CL. Should revisit this

[antiagainst]

Address comments

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

28

Oh, didn't know we have that. Thanks!

81

Nice, thanks!

126

I feel I don't have all the backgrounds here so I'd appreciate some explanation as for why it's necessary to separate the match and rewrite. I think that's the original design of patterns (that we have match and rewrite separately) and then later we found it's too cumbersome to carry over a large matched state between them and then came up with matchAndRewrite? IIUC, every pattern is applied as a whole. I can understand that the match/rewrite side can be potentially reusable across patterns as components if they are exposed as helper functions, but that's more from a library organization and code reuse's perspective; I feel you've more reasons than that so I'd like to learn about them. :)

Another question here is that I've matches that is generally useful to other patterns (checking whether this is a linalg doing reduction) and matches that is only useful to SPIR-V here (checking whether the workload can be fit in one local workgroup). How do you handle such cases?

For now I separated the check that whether this is a linalg doing reduction as a helper function. I haven't put it in some linalg file yet because currently it's too rigid and overfitting to a specific case. I plan to extend it and then later maybe we can move it to some linalg file.

157

Right. I guess here we need some thinking about how to better design the API to make it generic and usable across different cases. I'm not so versed on that at the current moment. So putting your name in the TODO for now. :)

183

Yeah, right now linalg.generic is an integrated entity; I'm not sure how we can easily convert part of it (the "structure") and then the inlined scalar region. I guess it's fine for now to overfit a bit until later we see concrete cases that we want to support, which will give us a more tangible problem to solve.

[merge_guards_bot]

Unit tests: pass. 62199 tests passed, 0 failed and 815 were skipped.

clang-tidy: pass.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[nicolasvasilache]

Thanks Lei, this looks great, glad to see you pushing on this front!

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

126

Let's sync up over video for this, this is a longer discussion and touches phase ordering issues, local patterns etc.
If you want some of the deeper details please look at the rationale doc: https://mlir.llvm.org/docs/Dialects/Linalg/ (which will soon move to https://mlir.llvm.org/docs/Dialects/LinalgRationale/).

In the meantime, this is not a blocker but it would be great to sync up in particular on the implementation on those ideas today in DRR.

157

SG, thanks!

[antiagainst]

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

126

Awesome, thanks! I've opened https://mlir.llvm.org/docs/Dialects/Linalg/ in a tab but haven't read it through yet. ;-P Will do now.

clang/include/clang/AST/ParentMapContext.h

public:
   Expr *traverseIgnored(Expr *E) const;
   ast_type_traits::DynTypedNode
   traverseIgnored(const ast_type_traits::DynTypedNode &N) const;
 
 private:
   ASTContext &ASTCtx;
   class ParentMap;
   ast_type_traits::TraversalKind Traversal = ast_type_traits::TK_AsIs;
-  std::map<ast_type_traits::TraversalKind, std::unique_ptr<ParentMap>> Parents;
+  std::unique_ptr<ParentMap> Parents;
 };
 
 class TraversalKindScope {
   ParentMapContext &Ctx;
   ast_type_traits::TraversalKind TK = ast_type_traits::TK_AsIs;
 
 public:
   TraversalKindScope(ASTContext &ASTCtx,

clang/lib/AST/ParentMapContext.cpp

 #include "clang/AST/TemplateBase.h"
 
 using namespace clang;
 
 ParentMapContext::ParentMapContext(ASTContext &Ctx) : ASTCtx(Ctx) {}
 
 ParentMapContext::~ParentMapContext() = default;
 
-void ParentMapContext::clear() { Parents.clear(); }
+void ParentMapContext::clear() { Parents.reset(); }
 
 const Expr *ParentMapContext::traverseIgnored(const Expr *E) const {
   return traverseIgnored(const_cast<Expr *>(E));
 }
 
 Expr *ParentMapContext::traverseIgnored(Expr *E) const {
   if (!E)
     return nullptr;
for (const auto &Entry : OtherParents) {
       if (Entry.second.is<ast_type_traits::DynTypedNode *>()) {
         delete Entry.second.get<ast_type_traits::DynTypedNode *>();
       } else if (Entry.second.is<ParentVector *>()) {
         delete Entry.second.get<ParentVector *>();
       }
     }
   }
 
-  DynTypedNodeList getParents(const ast_type_traits::DynTypedNode &Node) {
-    if (Node.getNodeKind().hasPointerIdentity())
-      return getDynNodeFromMap(Node.getMemoizationData(), PointerParents);
+  DynTypedNodeList getParents(ast_type_traits::TraversalKind TK,
+                              const ast_type_traits::DynTypedNode &Node) {
+    if (Node.getNodeKind().hasPointerIdentity()) {
+      auto ParentList =
+          getDynNodeFromMap(Node.getMemoizationData(), PointerParents);
+      if (ParentList.size() == 1 &&
+          TK == ast_type_traits::TK_IgnoreUnlessSpelledInSource) {
+        const auto *E = ParentList[0].get<Expr>();
+        const auto *Child = Node.get<Expr>();
+        if (E && Child)
+          return AscendIgnoreUnlessSpelledInSource(E, Child);
+      }
+      return ParentList;
+    }
     return getDynNodeFromMap(Node, OtherParents);
   }
+
+  ast_type_traits::DynTypedNode
+  AscendIgnoreUnlessSpelledInSource(const Expr *E, const Expr *Child) {
+
+    auto ShouldSkip = [](const Expr *E, const Expr *Child) {
+      if (isa<ImplicitCastExpr>(E))
+        return true;
+
+      if (isa<FullExpr>(E))
+        return true;
+
+      if (isa<MaterializeTemporaryExpr>(E))
+        return true;
+
+      if (isa<CXXBindTemporaryExpr>(E))
+        return true;
+
+      if (isa<ParenExpr>(E))
+        return true;
+
+      if (isa<ExprWithCleanups>(E))
+        return true;
+
+      auto SR = Child->getSourceRange();
+
+      if (const auto *C = dyn_cast<CXXConstructExpr>(E)) {
+        if (C->getSourceRange() == SR || !isa<CXXTemporaryObjectExpr>(C))
+          return true;
+      }
+
+      if (const auto *C = dyn_cast<CXXMemberCallExpr>(E)) {
+        if (C->getSourceRange() == SR)
+          return true;
+      }
+
+      if (const auto *C = dyn_cast<MemberExpr>(E)) {
+        if (C->getSourceRange() == SR)
+          return true;
+      }
+      return false;
+    };
+
+    while (ShouldSkip(E, Child)) {
+      auto It = PointerParents.find(E);
+      if (It == PointerParents.end())
+        break;
+      const auto *S = It->second.dyn_cast<const Stmt *>();
+      if (!S)
+        return getSingleDynTypedNodeFromParentMap(It->second);
+      const auto *P = dyn_cast<Expr>(S);
+      if (!P)
+        return ast_type_traits::DynTypedNode::create(*S);
+      Child = E;
+      E = P;
+    }
+    return ast_type_traits::DynTypedNode::create(*E);
+  }
 };
 
 /// Template specializations to abstract away from pointers and TypeLocs.
 /// @{
 template <typename T>
 static ast_type_traits::DynTypedNode createDynTypedNode(const T &Node) {
   return ast_type_traits::DynTypedNode::create(*Node);
 }

 /// Note that the relationship described here is purely in terms of AST
 /// traversal - there are other relationships (for example declaration context)
 /// in the AST that are better modeled by special matchers.
 ///
 /// FIXME: Currently only builds up the map using \c Stmt and \c Decl nodes.
 class ParentMapContext::ParentMap::ASTVisitor
     : public RecursiveASTVisitor<ASTVisitor> {
 public:
-  ASTVisitor(ParentMap &Map, ParentMapContext &MapCtx)
-      : Map(Map), MapCtx(MapCtx) {}
+  ASTVisitor(ParentMap &Map) : Map(Map) {}
 
 private:
   friend class RecursiveASTVisitor<ASTVisitor>;
 
   using VisitorBase = RecursiveASTVisitor<ASTVisitor>;
 
   bool shouldVisitTemplateInstantiations() const { return true; }
 
private:
 
   bool TraverseDecl(Decl *DeclNode) {
     return TraverseNode(
         DeclNode, DeclNode, [&] { return VisitorBase::TraverseDecl(DeclNode); },
         &Map.PointerParents);
   }
 
   bool TraverseStmt(Stmt *StmtNode) {
-    Stmt *FilteredNode = StmtNode;
-    if (auto *ExprNode = dyn_cast_or_null<Expr>(FilteredNode))
-      FilteredNode = MapCtx.traverseIgnored(ExprNode);
-    return TraverseNode(FilteredNode, FilteredNode,
-                        [&] { return VisitorBase::TraverseStmt(FilteredNode); },
+    return TraverseNode(StmtNode, StmtNode,
+                        [&] { return VisitorBase::TraverseStmt(StmtNode); },
                         &Map.PointerParents);
   }
 
   bool TraverseTypeLoc(TypeLoc TypeLocNode) {
     return TraverseNode(
         TypeLocNode, ast_type_traits::DynTypedNode::create(TypeLocNode),
         [&] { return VisitorBase::TraverseTypeLoc(TypeLocNode); },
         &Map.OtherParents);
   }
 
   bool TraverseNestedNameSpecifierLoc(NestedNameSpecifierLoc NNSLocNode) {
     return TraverseNode(
         NNSLocNode, ast_type_traits::DynTypedNode::create(NNSLocNode),
         [&] { return VisitorBase::TraverseNestedNameSpecifierLoc(NNSLocNode); },
         &Map.OtherParents);
   }
 
   ParentMap &Map;
-  ParentMapContext &MapCtx;
   llvm::SmallVector<ast_type_traits::DynTypedNode, 16> ParentStack;
 };
 
 ParentMapContext::ParentMap::ParentMap(ASTContext &Ctx) {
-  ASTVisitor(*this, Ctx.getParentMapContext()).TraverseAST(Ctx);
+  ASTVisitor(*this).TraverseAST(Ctx);
 }
 
 DynTypedNodeList
 ParentMapContext::getParents(const ast_type_traits::DynTypedNode &Node) {
-  std::unique_ptr<ParentMap> &P = Parents[Traversal];
-  if (!P)
+  if (!Parents)
     // We build the parent map for the traversal scope (usually whole TU), as
     // hasAncestor can escape any subtree.
-    P = std::make_unique<ParentMap>(ASTCtx);
-  return P->getParents(Node);
+    Parents = std::make_unique<ParentMap>(ASTCtx);
+  return Parents->getParents(getTraversalKind(), Node);
 }
-

clang/lib/Parse/ParseExprCXX.cpp

while (!Tok.is(tok::r_brace)) {
             Requirements.push_back(Req);
           break;
         }
         if (!TryConsumeToken(tok::arrow))
           // User probably forgot the arrow, remind them and try to continue.
           Diag(Tok, diag::err_requires_expr_missing_arrow)
               << FixItHint::CreateInsertion(Tok.getLocation(), "->");
         // Try to parse a 'type-constraint'
-        CXXScopeSpec SS;
-        if (ParseOptionalCXXScopeSpecifier(SS, ParsedType(),
-                                           /*EnteringContext=*/false,
-                                           /*MayBePseudoDestructor=*/nullptr,
-                                           // If this is not a type-constraint,
-                                           // then this scope-spec is part of
-                                           // the typename of a non-type
-                                           // template parameter
-                                           /*IsTypename=*/true,
-                                           /*LastII=*/nullptr,
-                                           // We won't find concepts in
-                                           // non-namespaces anyway, so might as
-                                           // well parse this correctly for
-                                           // possible type names.
-                                           /*OnlyNamespace=*/false,
-                                           /*SuppressDiagnostic=*/true)) {
-          SkipUntil(tok::semi, tok::r_brace, SkipUntilFlags::StopBeforeMatch);
-          break;
-        }
         if (TryAnnotateTypeConstraint()) {
           SkipUntil(tok::semi, tok::r_brace, SkipUntilFlags::StopBeforeMatch);
           break;
         }
         if (!isTypeConstraintAnnotation()) {
           Diag(Tok, diag::err_requires_expr_expected_type_constraint);
           SkipUntil(tok::semi, tok::r_brace, SkipUntilFlags::StopBeforeMatch);
           break;
         }
-        if (Tok.is(tok::annot_cxxscope))
+        CXXScopeSpec SS;
+        if (Tok.is(tok::annot_cxxscope)) {
+          Actions.RestoreNestedNameSpecifierAnnotation(Tok.getAnnotationValue(),
+                                                       Tok.getAnnotationRange(),
+                                                       SS);
           ConsumeAnnotationToken();
+        }
 
         Req = Actions.ActOnCompoundRequirement(
             Expression.get(), NoexceptLoc, SS, takeTemplateIdAnnotation(Tok),
             TemplateParameterDepth);
         ConsumeAnnotationToken();
         if (Req)
           Requirements.push_back(Req);
         break;

clang/lib/Sema/SemaTemplate.cpp

if (auto *TTP = dyn_cast<TemplateTypeParmDecl>(TemplateParam)) {
           /*Depth*/ 0, Depth1IndexAdjustment + TTP->getIndex(),
           TTP->getIdentifier(), TTP->wasDeclaredWithTypename(),
           TTP->isParameterPack(), TTP->hasTypeConstraint(),
           TTP->isExpandedParameterPack() ?
           llvm::Optional<unsigned>(TTP->getNumExpansionParameters()) : None);
       if (const auto *TC = TTP->getTypeConstraint()) {
         TemplateArgumentListInfo TransformedArgs;
         const auto *ArgsAsWritten = TC->getTemplateArgsAsWritten();
-        if (SemaRef.Subst(ArgsAsWritten->getTemplateArgs(),
+        if (!ArgsAsWritten ||
+            SemaRef.Subst(ArgsAsWritten->getTemplateArgs(),
                           ArgsAsWritten->NumTemplateArgs, TransformedArgs,
                           Args))
           SemaRef.AttachTypeConstraint(
               TC->getNestedNameSpecifierLoc(), TC->getConceptNameInfo(),
-              TC->getNamedConcept(), &TransformedArgs, NewTTP,
+              TC->getNamedConcept(), ArgsAsWritten ? &TransformedArgs : nullptr,
+              NewTTP,
               NewTTP->isParameterPack()
                  ? cast<CXXFoldExpr>(TC->getImmediatelyDeclaredConstraint())
                      ->getEllipsisLoc()
                  : SourceLocation());
       }
       if (TTP->hasDefaultArgument()) {
         TypeSourceInfo *InstantiatedDefaultArg =
             SemaRef.SubstType(TTP->getDefaultArgumentInfo(), Args,
                               TTP->getDefaultArgumentLoc(), TTP->getDeclName());
         if (InstantiatedDefaultArg)

clang/test/Parser/cxx2a-abbreviated-templates.cpp

+// RUN: %clang_cc1 -std=c++2a -x c++ %s -verify
+// expected-no-diagnostics
+
+template<typename T, typename U=void>
+concept C = true;
+
+namespace ns {
+  template<typename T, typename U=void>
+  concept D = true;
+}
+
+void foo(C auto a,
+         C<int> auto b,
+         ns::D auto c,
+         ns::D<int> auto d,
+         const C auto e,
+         const C<int> auto f,
+         const ns::D auto g,
+         const ns::D<int> auto h);
 No newline at end of file

clang/test/Parser/cxx2a-concepts-requires-expr.cpp

 template<typename T, typename U>
 concept C2 = true;
 
 bool r29 = requires { { 0 } noexcept C1; };
 // expected-error@-1 {{expected '->' before expression type requirement}}
 
 bool r30 = requires { { 0 } noexcept -> C2<int>; };
 
+namespace ns { template<typename T> concept C = true; }
+
+bool r31 = requires { { 0 } noexcept -> ns::C; };
+
 template<typename T>
 T i1 = 0;
 
-bool r31 = requires { requires false, 1; };
+bool r32 = requires { requires false, 1; };
 // expected-error@-1 {{expected ';' at end of requirement}}
 
-bool r32 = requires { 0 noexcept; };
+bool r33 = requires { 0 noexcept; };
 // expected-error@-1 {{'noexcept' can only be used in a compound requirement (with '{' '}' around the expression)}}
 
-bool r33 = requires { 0 int; };
+bool r34 = requires { 0 int; };
 // expected-error@-1 {{expected ';' at end of requirement}}
 
-bool r34 = requires { requires true };
+bool r35 = requires { requires true };
 // expected-error@-1 {{expected ';' at end of requirement}}
 
-bool r35 = requires (bool b) { requires sizeof(b) == 1; };
+bool r36 = requires (bool b) { requires sizeof(b) == 1; };
 
-void r36(bool b) requires requires { 1 } {}
+void r37(bool b) requires requires { 1 } {}
 // expected-error@-1 {{expected ';' at end of requirement}}
 
-bool r37 = requires { requires { 1; }; };
+bool r38 = requires { requires { 1; }; };
 // expected-warning@-1 {{this requires expression will only be checked for syntactic validity; did you intend to place it in a nested requirement? (add another 'requires' before the expression)}}
 
-bool r38 = requires { requires () { 1; }; };
+bool r39 = requires { requires () { 1; }; };
 // expected-warning@-1 {{this requires expression will only be checked for syntactic validity; did you intend to place it in a nested requirement? (add another 'requires' before the expression)}}
 
-bool r39 = requires { requires (int i) { i; }; };
+bool r40 = requires { requires (int i) { i; }; };
 // expected-warning@-1 {{this requires expression will only be checked for syntactic validity; did you intend to place it in a nested requirement? (add another 'requires' before the expression)}}
 
-bool r40 = requires { requires (); };
+bool r41 = requires { requires (); };
 // expected-error@-1 {{expected expression}}

clang/test/Parser/cxx2a-placeholder-type-constraint.cpp

 // RUN: %clang_cc1 -std=c++2a -x c++ %s -verify
 
 template<typename T, typename U=void>
 concept C = true;
 
+namespace ns {
+  template<typename T, typename U=void>
+  concept D = true;
+}
+
 int foo() {
-  C auto a4 = 1;
-  C<> auto a5 = 1;
-  C<int> auto a6 = 1;
-  const C auto &a7 = 1;
-  const C<> auto &a8 = 1;
-  const C<int> auto &a9 = 1;
-  C decltype(auto) a10 = 1;
-  C<> decltype(auto) a11 = 1;
-  C<int> decltype(auto) a12 = 1;
-  const C<> decltype(auto) &a13 = 1; // expected-error{{'decltype(auto)' cannot be combined with other type specifiers}}
+  {ns::D auto a = 1;}
+  {C auto a = 1;}
+  {C<> auto a = 1;}
+  {C<int> auto a = 1;}
+  {ns::D<int> auto a = 1;}
+  {const ns::D auto &a = 1;}
+  {const C auto &a = 1;}
+  {const C<> auto &a = 1;}
+  {const C<int> auto &a = 1;}
+  {const ns::D<int> auto &a = 1;}
+  {C decltype(auto) a = 1;}
+  {C<> decltype(auto) a = 1;}
+  {C<int> decltype(auto) a = 1;}
+  {const C<> decltype(auto) &a = 1;} // expected-error{{'decltype(auto)' cannot be combined with other type specifiers}}
   // expected-error@-1{{non-const lvalue reference to type 'int' cannot bind to a temporary of type 'int'}}
-  const C<int> decltype(auto) &a14 = 1; // expected-error{{'decltype(auto)' cannot be combined with other type specifiers}}
+  {const C<int> decltype(auto) &a = 1;} // expected-error{{'decltype(auto)' cannot be combined with other type specifiers}}
   // expected-error@-1{{non-const lvalue reference to type 'int' cannot bind to a temporary of type 'int'}}
-  C a15 = 1;
+  {C a = 1;}
   // expected-error@-1{{expected 'auto' or 'decltype(auto)' after concept name}}
-  C decltype a19 = 1;
+  {C decltype a19 = 1;}
   // expected-error@-1{{expected '('}}
-  C decltype(1) a20 = 1;
+  {C decltype(1) a20 = 1;}
   // expected-error@-1{{expected 'auto' or 'decltype(auto)' after concept name}}
 }
 No newline at end of file

libunwind/CMakeLists.txt

 # Add path for custom modules
 set(CMAKE_MODULE_PATH
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake"
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules"
   ${CMAKE_MODULE_PATH}
   )
 
 if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR OR LIBUNWIND_STANDALONE_BUILD)
-  project(libunwind)
+  project(libunwind LANGUAGES C CXX ASM)
 
   # Rely on llvm-config.
   set(CONFIG_OUTPUT)
   if(NOT LLVM_CONFIG_PATH)
     find_program(LLVM_CONFIG_PATH "llvm-config")
   endif()
   if (DEFINED LLVM_PATH)
     set(LLVM_INCLUDE_DIR ${LLVM_INCLUDE_DIR} CACHE PATH "Path to llvm/include")

llvm/docs/LangRef.rst

 """""""""
 
 Reads scalar values from arbitrary memory locations and gathers them into one vector. The memory locations are provided in the vector of pointers '``ptrs``'. The memory is accessed according to the provided mask. The mask holds a bit for each vector lane, and is used to prevent memory accesses to the masked-off lanes. The masked-off lanes in the result vector are taken from the corresponding lanes of the '``passthru``' operand.
 
 
 Arguments:
 """"""""""
 
-The first operand is a vector of pointers which holds all memory addresses to read. The second operand is an alignment of the source addresses. It must be a constant integer value. The third operand, mask, is a vector of boolean values with the same number of elements as the return type. The fourth is a pass-through value that is used to fill the masked-off lanes of the result. The return type, underlying type of the vector of pointers and the type of the '``passthru``' operand are the same vector types.
-
+The first operand is a vector of pointers which holds all memory addresses to read. The second operand is an alignment of the source addresses. It must be 0 or a power of two constant integer value. The third operand, mask, is a vector of boolean values with the same number of elements as the return type. The fourth is a pass-through value that is used to fill the masked-off lanes of the result. The return type, underlying type of the vector of pointers and the type of the '``passthru``' operand are the same vector types.
 
 Semantics:
 """"""""""
 
 The '``llvm.masked.gather``' intrinsic is designed for conditional reading of multiple scalar values from arbitrary memory locations in a single IR operation. It is useful for targets that support vector masked gathers and allows vectorizing basic blocks with data and control divergence. Other targets may support this intrinsic differently, for example by lowering it into a sequence of scalar load operations.
 The semantics of this operation are equivalent to a sequence of conditional scalar loads with subsequent gathering all loaded values into a single vector. The mask restricts memory access to certain lanes and facilitates vectorization of predicated basic blocks.
 
 

 Overview:
 """""""""
 
 Writes each element from the value vector to the corresponding memory address. The memory addresses are represented as a vector of pointers. Writing is done according to the provided mask. The mask holds a bit for each vector lane, and is used to prevent memory accesses to the masked-off lanes.
 
 Arguments:
 """"""""""
 
-The first operand is a vector value to be written to memory. The second operand is a vector of pointers, pointing to where the value elements should be stored. It has the same underlying type as the value operand. The third operand is an alignment of the destination addresses. The fourth operand, mask, is a vector of boolean values. The types of the mask and the value operand must have the same number of vector elements.
-
+The first operand is a vector value to be written to memory. The second operand is a vector of pointers, pointing to where the value elements should be stored. It has the same underlying type as the value operand. The third operand is an alignment of the destination addresses. It must be 0 or a power of two constant integer value. The fourth operand, mask, is a vector of boolean values. The types of the mask and the value operand must have the same number of vector elements.
 
 Semantics:
 """"""""""
 
 The '``llvm.masked.scatter``' intrinsics is designed for writing selected vector elements to arbitrary memory addresses in a single IR operation. The operation may be conditional, when not all bits in the mask are switched on. It is useful for targets that support vector masked scatter and allows vectorizing basic blocks with data and control divergence. Other targets may support this intrinsic differently, for example by lowering it into a sequence of branches that guard scalar store operations.
 
 ::
 

llvm/lib/IR/Verifier.cpp

case Intrinsic::masked_store: {
     Assert(DataTy == Val->getType(),
            "masked_store: storee must match pointer type", Call);
     Assert(Mask->getType()->getVectorNumElements() ==
                DataTy->getVectorNumElements(),
            "masked_store: vector mask must be same length as data", Call);
     break;
   }
 
+  case Intrinsic::masked_gather: {
+    const APInt &Alignment =
+        cast<ConstantInt>(Call.getArgOperand(1))->getValue();
+    Assert(Alignment.isNullValue() || Alignment.isPowerOf2(),
+           "masked_gather: alignment must be 0 or a power of 2", Call);
+    break;
+  }
+  case Intrinsic::masked_scatter: {
+    const APInt &Alignment =
+        cast<ConstantInt>(Call.getArgOperand(2))->getValue();
+    Assert(Alignment.isNullValue() || Alignment.isPowerOf2(),
+           "masked_scatter: alignment must be 0 or a power of 2", Call);
+    break;
+  }
+
   case Intrinsic::experimental_guard: {
     Assert(isa<CallInst>(Call), "experimental_guard cannot be invoked", Call);
     Assert(Call.countOperandBundlesOfType(LLVMContext::OB_deopt) == 1,
            "experimental_guard must have exactly one "
            "\"deopt\" operand bundle");
     break;
   }
 

llvm/lib/Target/X86/X86ISelLowering.cpp

   // Return the new chain to replace N.
   return V;
 }
 
 // Attempt to commute shufps LHS loads:
 // permilps(shufps(load(),x)) --> permilps(shufps(x,load()))
 static SDValue combineCommutableSHUFP(SDValue N, MVT VT, const SDLoc &DL,
                                       SelectionDAG &DAG) {
-  // TODO: Add general vXf32 + vXf64 support.
-  if (VT != MVT::v4f32)
+  // TODO: Add vXf64 support.
+  if (VT != MVT::v4f32 && VT != MVT::v8f32 && VT != MVT::v16f32)
     return SDValue();
 
   // SHUFP(LHS, RHS) -> SHUFP(RHS, LHS) iff LHS is foldable + RHS is not.
   auto commuteSHUFP = [&VT, &DL, &DAG](SDValue Parent, SDValue V) {
     if (V.getOpcode() != X86ISD::SHUFP || !Parent->isOnlyUserOf(V.getNode()))
       return SDValue();
     SDValue N0 = V.getOperand(0);
     SDValue N1 = V.getOperand(1);

llvm/lib/Target/X86/X86ScheduleBdVer2.td

 def PdWriteBTSRm : SchedWriteRes<[PdEX01, PdLoad]> {
   let Latency = 7;
   let ResourceCycles = [44, 1];
   let NumMicroOps = 10;
 }
 def : SchedAlias<WriteBitTestSetRegRMW, PdWriteBTSRm>;
 
 // This is for simple LEAs with one or two input operands.
-// FIXME: SAGU 3-operand LEA
-def : WriteRes<WriteLEA,              [PdEX01]> { let NumMicroOps = 2; }
+def : WriteRes<WriteLEA,              [PdEX01]> { let ResourceCycles = [2]; }
+
+// This write is used for slow LEA instructions.
+def PdWrite3OpsLEA : SchedWriteRes<[PdEX01]> {
+  let Latency = 2;
+  let ResourceCycles = [2];
+}
+
+// On Piledriver, a slow LEA is either a 3Ops LEA (base, index, offset),
+// or an LEA with a `Scale` value different than 1.
+def PdSlowLEAPredicate : MCSchedPredicate<
+  CheckAny<[
+    // A 3-operand LEA (base, index, offset).
+    IsThreeOperandsLEAFn,
+    // An LEA with a "Scale" different than 1.
+    CheckAll<[
+      CheckIsImmOperand<2>,
+      CheckNot<CheckImmOperand<2, 1>>
+    ]>
+  ]>
+>;
+
+def PdWriteLEA : SchedWriteVariant<[
+    SchedVar<PdSlowLEAPredicate, [PdWrite3OpsLEA]>,
+    SchedVar<NoSchedPred,        [WriteLEA]>
+]>;
+
+def : InstRW<[PdWriteLEA], (instrs LEA32r, LEA64r, LEA64_32r)>;
+
+def PdWriteLEA16r : SchedWriteRes<[PdEX01]> {
+  let ResourceCycles = [3];
+  let NumMicroOps = 2;
+}
+def : InstRW<[PdWriteLEA16r], (instrs LEA16r)>;
 
 // Bit counts.
 defm : PdWriteResExPair<WriteBSF,     [PdEX01],          3,  [6],     6, 2>;
 defm : PdWriteResExPair<WriteBSR,     [PdEX01],          4,  [8],     7, 2>;
 defm : PdWriteResExPair<WritePOPCNT,  [PdEX01],          4,  [4]>;
 defm : PdWriteResExPair<WriteLZCNT,   [PdEX0],           2,  [2],     2>;
 defm : PdWriteResExPair<WriteTZCNT,   [PdEX0],           2,  [2],     2>;
 

llvm/test/Assembler/auto_upgrade_intrinsics.ll

; CHECK: @llvm.masked.gather.v2f64.v2p0f64
   ret <2 x double> %res
 }
 
 declare void @llvm.masked.scatter.v2f64(<2 x double> %val, <2 x double*> %ptrs, i32, <2 x i1> %mask)
 
 define void @tests.masked.scatter(<2 x double*> %ptr, <2 x i1> %mask, <2 x double> %val)  {
 ; CHECK-LABEL: @tests.masked.scatter(
 ; CHECK: @llvm.masked.scatter.v2f64.v2p0f64
-  call void @llvm.masked.scatter.v2f64(<2 x double> %val, <2 x double*> %ptr, i32 3, <2 x i1> %mask)
+  call void @llvm.masked.scatter.v2f64(<2 x double> %val, <2 x double*> %ptr, i32 1, <2 x i1> %mask)
   ret void
 }
 
 declare {}* @llvm.invariant.start(i64, i8* nocapture) nounwind readonly
 declare void @llvm.invariant.end({}*, i64, i8* nocapture) nounwind
 
 define void @tests.invariant.start.end() {
   ; CHECK-LABEL: @tests.invariant.start.end(

llvm/test/CodeGen/X86/vector-shuffle-256-v8.ll

; ALL-NEXT:    retq
   %2 = shufflevector <4 x float> %b, <4 x float> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
   %3 = shufflevector <4 x float> %1, <4 x float> %2, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   ret <8 x float> %3
 }
 
 define <8 x float> @shuffle_mem_v8f32_8BA0CFE4(<8 x float> %a0, <8 x float>* %a1) {
 ; AVX1OR2-LABEL: shuffle_mem_v8f32_8BA0CFE4:
 ; AVX1OR2:       # %bb.0:
-; AVX1OR2-NEXT:    vmovaps (%rdi), %ymm1
-; AVX1OR2-NEXT:    vshufps {{.*#+}} ymm1 = ymm1[0,0],ymm0[2,0],ymm1[4,4],ymm0[6,4]
-; AVX1OR2-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[0,3],ymm1[2,0],ymm0[4,7],ymm1[6,4]
+; AVX1OR2-NEXT:    vshufps {{.*#+}} ymm1 = ymm0[2,0],mem[0,0],ymm0[6,4],mem[4,4]
+; AVX1OR2-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[0,3],ymm1[0,2],ymm0[4,7],ymm1[4,6]
 ; AVX1OR2-NEXT:    retq
 ;
 ; AVX512VL-SLOW-LABEL: shuffle_mem_v8f32_8BA0CFE4:
 ; AVX512VL-SLOW:       # %bb.0:
-; AVX512VL-SLOW-NEXT:    vmovaps (%rdi), %ymm1
-; AVX512VL-SLOW-NEXT:    vshufps {{.*#+}} ymm1 = ymm1[0,0],ymm0[2,0],ymm1[4,4],ymm0[6,4]
-; AVX512VL-SLOW-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[0,3],ymm1[2,0],ymm0[4,7],ymm1[6,4]
+; AVX512VL-SLOW-NEXT:    vshufps {{.*#+}} ymm1 = ymm0[2,0],mem[0,0],ymm0[6,4],mem[4,4]
+; AVX512VL-SLOW-NEXT:    vshufps {{.*#+}} ymm0 = ymm0[0,3],ymm1[0,2],ymm0[4,7],ymm1[4,6]
 ; AVX512VL-SLOW-NEXT:    retq
 ;
 ; AVX512VL-FAST-LABEL: shuffle_mem_v8f32_8BA0CFE4:
 ; AVX512VL-FAST:       # %bb.0:
 ; AVX512VL-FAST-NEXT:    vmovaps {{.*#+}} ymm1 = [0,3,2,8,4,7,6,12]
 ; AVX512VL-FAST-NEXT:    vpermt2ps (%rdi), %ymm1, %ymm0
 ; AVX512VL-FAST-NEXT:    retq
   %1 = load <8 x float>, <8 x float>* %a1

llvm/test/CodeGen/X86/vector-shuffle-512-v16.ll

; ALL-NEXT:    retq
   %c = load <16 x float>, <16 x float>* %b
   %d = shufflevector <16 x float> %a, <16 x float> %c, <16 x i32> <i32 15, i32 31, i32 14, i32 22, i32 13, i32 29, i32 4, i32 28, i32 11, i32 27, i32 10, i32 26, i32 9, i32 25, i32 8, i32 24>
   ret <16 x float> %d
 }
 
 define <16 x float> @shuffle_v16f32_load_08_11_10_00_12_15_14_04(<16 x float> %a0, <16 x float>* %a1) {
 ; ALL-LABEL: shuffle_v16f32_load_08_11_10_00_12_15_14_04:
 ; ALL:       # %bb.0:
-; ALL-NEXT:    vmovaps (%rdi), %zmm1
-; ALL-NEXT:    vshufps {{.*#+}} zmm1 = zmm1[0,0],zmm0[2,0],zmm1[4,4],zmm0[6,4],zmm1[8,8],zmm0[10,8],zmm1[12,12],zmm0[14,12]
-; ALL-NEXT:    vshufps {{.*#+}} zmm0 = zmm0[0,3],zmm1[2,0],zmm0[4,7],zmm1[6,4],zmm0[8,11],zmm1[10,8],zmm0[12,15],zmm1[14,12]
+; ALL-NEXT:    vshufps {{.*#+}} zmm1 = zmm0[2,0],mem[0,0],zmm0[6,4],mem[4,4],zmm0[10,8],mem[8,8],zmm0[14,12],mem[12,12]
+; ALL-NEXT:    vshufps {{.*#+}} zmm0 = zmm0[0,3],zmm1[0,2],zmm0[4,7],zmm1[4,6],zmm0[8,11],zmm1[8,10],zmm0[12,15],zmm1[12,14]
 ; ALL-NEXT:    retq
   %1 = load <16 x float>, <16 x float>* %a1
   %2 = shufflevector <16 x float> %1, <16 x float> %a0, <16 x i32> <i32 16, i32 19, i32 18, i32 0, i32 20, i32 23, i32 22, i32 4, i32 24, i32 27, i32 26, i32 8, i32 28, i32 31, i32 30, i32 12>
   ret <16 x float> %2
 }
 
 define <16 x i32> @shuffle_v16i32_load_0f_1f_0e_16_0d_1d_04_1e_0b_1b_0a_1a_09_19_08_18(<16 x i32> %a, <16 x i32>* %b)  {
 ; ALL-LABEL: shuffle_v16i32_load_0f_1f_0e_16_0d_1d_04_1e_0b_1b_0a_1a_09_19_08_18:

llvm/test/Transforms/InstCombine/masked_intrinsics.ll

;
   ret <2 x double> %res
 }
 
 
 define void @scatter_zeromask(<2 x double*> %ptrs, <2 x double> %val)  {
 ; CHECK-LABEL: @scatter_zeromask(
 ; CHECK-NEXT:    ret void
 ;
-  call void @llvm.masked.scatter.v2f64.v2p0f64(<2 x double> %val, <2 x double*> %ptrs, i32 6, <2 x i1> zeroinitializer)
+  call void @llvm.masked.scatter.v2f64.v2p0f64(<2 x double> %val, <2 x double*> %ptrs, i32 8, <2 x i1> zeroinitializer)
   ret void
 }
 
 define void @scatter_demandedelts(double* %ptr, double %val)  {
 ; CHECK-LABEL: @scatter_demandedelts(
 ; CHECK-NEXT:    [[PTRS:%.*]] = getelementptr double, double* [[PTR:%.*]], <2 x i64> <i64 0, i64 undef>
 ; CHECK-NEXT:    [[VALVEC2:%.*]] = insertelement <2 x double> undef, double [[VAL:%.*]], i32 0
 ; CHECK-NEXT:    call void @llvm.masked.scatter.v2f64.v2p0f64(<2 x double> [[VALVEC2]], <2 x double*> [[PTRS]], i32 8, <2 x i1> <i1 true, i1 false>)
 ; CHECK-NEXT:    ret void
 ;
   %ptrs = getelementptr double, double* %ptr, <2 x i64> <i64 0, i64 1>
   %valvec1 = insertelement <2 x double> undef, double %val, i32 0
   %valvec2 = insertelement <2 x double> %valvec1, double %val, i32 1
   call void @llvm.masked.scatter.v2f64.v2p0f64(<2 x double> %valvec2, <2 x double*> %ptrs, i32 8, <2 x i1> <i1 true, i1 false>)
   ret void
 }

llvm/test/Transforms/InstCombine/shift-add.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; This test makes sure that these instructions are properly eliminated.
 ;
 ; RUN: opt < %s -instcombine -S | FileCheck %s
 
 define i32 @shl_C1_add_A_C2_i32(i16 %A) {
 ; CHECK-LABEL: @shl_C1_add_A_C2_i32(
-; CHECK-NEXT:    [[B:%.*]] = zext i16 %A to i32
+; CHECK-NEXT:    [[B:%.*]] = zext i16 [[A:%.*]] to i32
 ; CHECK-NEXT:    [[D:%.*]] = shl i32 192, [[B]]
 ; CHECK-NEXT:    ret i32 [[D]]
 ;
   %B = zext i16 %A to i32
   %C = add i32 %B, 5
   %D = shl i32 6, %C
   ret i32 %D
 }
 
 define i32 @ashr_C1_add_A_C2_i32(i32 %A) {
 ; CHECK-LABEL: @ashr_C1_add_A_C2_i32(
 ; CHECK-NEXT:    ret i32 0
 ;
   %B = and i32 %A, 65535
   %C = add i32 %B, 5
   %D = ashr i32 6, %C
   ret i32 %D
 }
 
 define i32 @lshr_C1_add_A_C2_i32(i32 %A) {
 ; CHECK-LABEL: @lshr_C1_add_A_C2_i32(
-; CHECK-NEXT:    [[B:%.*]] = and i32 %A, 65535
+; CHECK-NEXT:    [[B:%.*]] = and i32 [[A:%.*]], 65535
 ; CHECK-NEXT:    [[D:%.*]] = shl i32 192, [[B]]
 ; CHECK-NEXT:    ret i32 [[D]]
 ;
   %B = and i32 %A, 65535
   %C = add i32 %B, 5
   %D = shl i32 6, %C
   ret i32 %D
 }
 
 define <4 x i32> @shl_C1_add_A_C2_v4i32(<4 x i16> %A) {
 ; CHECK-LABEL: @shl_C1_add_A_C2_v4i32(
-; CHECK-NEXT:    [[B:%.*]] = zext <4 x i16> %A to <4 x i32>
+; CHECK-NEXT:    [[B:%.*]] = zext <4 x i16> [[A:%.*]] to <4 x i32>
 ; CHECK-NEXT:    [[D:%.*]] = shl <4 x i32> <i32 6, i32 4, i32 undef, i32 -458752>, [[B]]
 ; CHECK-NEXT:    ret <4 x i32> [[D]]
 ;
   %B = zext <4 x i16> %A to <4 x i32>
   %C = add <4 x i32> %B, <i32 0, i32 1, i32 50, i32 16>
   %D = shl <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, %C
   ret <4 x i32> %D
 }
 
 define <4 x i32> @ashr_C1_add_A_C2_v4i32(<4 x i32> %A) {
 ; CHECK-LABEL: @ashr_C1_add_A_C2_v4i32(
-; CHECK-NEXT:    [[B:%.*]] = and <4 x i32> %A, <i32 0, i32 15, i32 255, i32 65535>
+; CHECK-NEXT:    [[B:%.*]] = and <4 x i32> [[A:%.*]], <i32 0, i32 15, i32 255, i32 65535>
 ; CHECK-NEXT:    [[D:%.*]] = ashr <4 x i32> <i32 6, i32 1, i32 undef, i32 -1>, [[B]]
 ; CHECK-NEXT:    ret <4 x i32> [[D]]
 ;
   %B = and <4 x i32> %A, <i32 0, i32 15, i32 255, i32 65535>
   %C = add <4 x i32> %B, <i32 0, i32 1, i32 50, i32 16>
   %D = ashr <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, %C
   ret <4 x i32> %D
 }
 
 define <4 x i32> @lshr_C1_add_A_C2_v4i32(<4 x i32> %A) {
 ; CHECK-LABEL: @lshr_C1_add_A_C2_v4i32(
-; CHECK-NEXT:    [[B:%.*]] = and <4 x i32> %A, <i32 0, i32 15, i32 255, i32 65535>
+; CHECK-NEXT:    [[B:%.*]] = and <4 x i32> [[A:%.*]], <i32 0, i32 15, i32 255, i32 65535>
 ; CHECK-NEXT:    [[D:%.*]] = lshr <4 x i32> <i32 6, i32 1, i32 undef, i32 65535>, [[B]]
 ; CHECK-NEXT:    ret <4 x i32> [[D]]
 ;
   %B = and <4 x i32> %A, <i32 0, i32 15, i32 255, i32 65535>
   %C = add <4 x i32> %B, <i32 0, i32 1, i32 50, i32 16>
   %D = lshr <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, %C
   ret <4 x i32> %D
 }
+
+define <4 x i32> @shl_C1_add_A_C2_v4i32_splat(i16 %I) {
+; CHECK-LABEL: @shl_C1_add_A_C2_v4i32_splat(
+; CHECK-NEXT:    [[A:%.*]] = zext i16 [[I:%.*]] to i32
+; CHECK-NEXT:    [[B:%.*]] = insertelement <4 x i32> undef, i32 [[A]], i32 0
+; CHECK-NEXT:    [[C:%.*]] = shufflevector <4 x i32> [[B]], <4 x i32> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[D:%.*]] = add <4 x i32> [[C]], <i32 0, i32 1, i32 50, i32 16>
+; CHECK-NEXT:    [[E:%.*]] = shl <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, [[D]]
+; CHECK-NEXT:    ret <4 x i32> [[E]]
+;
+  %A = zext i16 %I to i32
+  %B = insertelement <4 x i32> undef, i32 %A, i32 0
+  %C = shufflevector <4 x i32> %B, <4 x i32> undef, <4 x i32> zeroinitializer
+  %D = add <4 x i32> %C, <i32 0, i32 1, i32 50, i32 16>
+  %E = shl <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, %D
+  ret <4 x i32> %E
+}
+
+define <4 x i32> @ashr_C1_add_A_C2_v4i32_splat(i16 %I) {
+; CHECK-LABEL: @ashr_C1_add_A_C2_v4i32_splat(
+; CHECK-NEXT:    [[A:%.*]] = zext i16 [[I:%.*]] to i32
+; CHECK-NEXT:    [[B:%.*]] = insertelement <4 x i32> undef, i32 [[A]], i32 0
+; CHECK-NEXT:    [[C:%.*]] = shufflevector <4 x i32> [[B]], <4 x i32> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[D:%.*]] = add <4 x i32> [[C]], <i32 0, i32 1, i32 50, i32 16>
+; CHECK-NEXT:    [[E:%.*]] = ashr <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, [[D]]
+; CHECK-NEXT:    ret <4 x i32> [[E]]
+;
+  %A = zext i16 %I to i32
+  %B = insertelement <4 x i32> undef, i32 %A, i32 0
+  %C = shufflevector <4 x i32> %B, <4 x i32> undef, <4 x i32> zeroinitializer
+  %D = add <4 x i32> %C, <i32 0, i32 1, i32 50, i32 16>
+  %E = ashr <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, %D
+  ret <4 x i32> %E
+}
+
+define <4 x i32> @lshr_C1_add_A_C2_v4i32_splat(i16 %I) {
+; CHECK-LABEL: @lshr_C1_add_A_C2_v4i32_splat(
+; CHECK-NEXT:    [[A:%.*]] = zext i16 [[I:%.*]] to i32
+; CHECK-NEXT:    [[B:%.*]] = insertelement <4 x i32> undef, i32 [[A]], i32 0
+; CHECK-NEXT:    [[C:%.*]] = shufflevector <4 x i32> [[B]], <4 x i32> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[D:%.*]] = add <4 x i32> [[C]], <i32 0, i32 1, i32 50, i32 16>
+; CHECK-NEXT:    [[E:%.*]] = lshr <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, [[D]]
+; CHECK-NEXT:    ret <4 x i32> [[E]]
+;
+  %A = zext i16 %I to i32
+  %B = insertelement <4 x i32> undef, i32 %A, i32 0
+  %C = shufflevector <4 x i32> %B, <4 x i32> undef, <4 x i32> zeroinitializer
+  %D = add <4 x i32> %C, <i32 0, i32 1, i32 50, i32 16>
+  %E = lshr <4 x i32> <i32 6, i32 2, i32 1, i32 -7>, %D
+  ret <4 x i32> %E
+}

llvm/test/tools/llvm-mca/X86/BdVer2/resources-lea.s

 # CHECK-NEXT: [1]: #uOps
 # CHECK-NEXT: [2]: Latency
 # CHECK-NEXT: [3]: RThroughput
 # CHECK-NEXT: [4]: MayLoad
 # CHECK-NEXT: [5]: MayStore
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  2      1     0.50                        leaw  0, %cx
-# CHECK-NEXT:  2      1     0.50                        leal  0, %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  0, %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (%eax), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (%eax), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (%eax), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (%rax), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (%rax), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (%rax), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (,%ebx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (,%ebx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (,%ebx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (,%rbx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (,%rbx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (,%rbx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (%eax,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (%eax,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (%eax,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (%rax,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (%rax,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (%rax,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (%eax,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (%eax,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (%eax,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (%rax,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (%rax,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (%rax,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (%eax,%ebx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (%eax,%ebx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (%eax,%ebx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  (%rax,%rbx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  (%rax,%rbx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  (%rax,%rbx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16, %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16, %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16, %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(%eax), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(%eax), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(%eax), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(%rax), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(%rax), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(%rax), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(,%ebx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(,%ebx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(,%ebx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(,%rbx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(,%rbx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(,%rbx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(%eax,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(%eax,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(%rax,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(%rax,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(%eax,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(%eax,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(%rax,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(%rax,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(%eax,%ebx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(%eax,%ebx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(%eax,%ebx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  -16(%rax,%rbx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  -16(%rax,%rbx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  -16(%rax,%rbx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024, %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024, %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024, %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(%eax), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(%eax), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(%eax), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(%rax), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(%rax), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(%rax), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(,%ebx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(,%ebx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(,%ebx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(,%rbx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(,%rbx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(,%rbx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(%eax,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(%eax,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(%rax,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(%rax,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(%eax,%ebx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(%eax,%ebx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(%eax,%ebx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(%rax,%rbx), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(%rax,%rbx), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(%rax,%rbx), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(%eax,%ebx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(%eax,%ebx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(%eax,%ebx,2), %rcx
-# CHECK-NEXT:  2      1     0.50                        leaw  1024(%rax,%rbx,2), %cx
-# CHECK-NEXT:  2      1     0.50                        leal  1024(%rax,%rbx,2), %ecx
-# CHECK-NEXT:  2      1     0.50                        leaq  1024(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  0, %cx
+# CHECK-NEXT:  1      1     1.00                        leal  0, %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  0, %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (%eax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (%eax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (%eax), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (%rax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (%rax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (%rax), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (,%ebx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  (,%ebx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  (,%ebx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (,%rbx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  (,%rbx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  (,%rbx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (%eax,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (%rax,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (%eax,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (%eax,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (%eax,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (%rax,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  (%rax,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  (%rax,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (%eax,%ebx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  (%eax,%ebx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  (%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  (%rax,%rbx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  (%rax,%rbx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  (%rax,%rbx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16, %cx
+# CHECK-NEXT:  1      1     1.00                        leal  -16, %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  -16, %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(%eax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  -16(%eax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  -16(%eax), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(%rax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  -16(%rax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  -16(%rax), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  -16(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  -16(,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  -16(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  -16(,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  -16(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  -16(,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  -16(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  -16(,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(,%ebx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  -16(,%ebx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  -16(,%ebx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(,%rbx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  -16(,%rbx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  -16(,%rbx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(%eax,%ebx), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  -16(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  -16(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(%rax,%rbx), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  -16(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  -16(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(%eax,%ebx), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  -16(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  -16(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(%rax,%rbx), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  -16(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  -16(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(%eax,%ebx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  -16(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  -16(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  -16(%rax,%rbx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  -16(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  -16(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024, %cx
+# CHECK-NEXT:  1      1     1.00                        leal  1024, %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  1024, %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(%eax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  1024(%eax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  1024(%eax), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(%rax), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  1024(%rax), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  1024(%rax), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  1024(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  1024(,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  1024(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  1024(,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(,%ebx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  1024(,%ebx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  1024(,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(,%rbx), %cx
+# CHECK-NEXT:  1      1     1.00                        leal  1024(,%rbx), %ecx
+# CHECK-NEXT:  1      1     1.00                        leaq  1024(,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(,%ebx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  1024(,%ebx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  1024(,%ebx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(,%rbx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  1024(,%rbx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  1024(,%rbx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(%eax,%ebx), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(%rax,%rbx), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(%eax,%ebx), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(%rax,%rbx), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(%eax,%ebx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  1024(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  1024(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  2      1     1.50                        leaw  1024(%rax,%rbx,2), %cx
+# CHECK-NEXT:  1      2     1.00                        leal  1024(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  1      2     1.00                        leaq  1024(%rax,%rbx,2), %rcx
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0.0] - PdAGLU01
 # CHECK-NEXT: [0.1] - PdAGLU01
 # CHECK-NEXT: [1]   - PdBranch
 # CHECK-NEXT: [2]   - PdCount
 # CHECK-NEXT: [3]   - PdDiv
 # CHECK-NEXT: [4]   - PdEX0

 # CHECK-NEXT: [15]  - PdFPXBR
 # CHECK-NEXT: [16.0] - PdLoad
 # CHECK-NEXT: [16.1] - PdLoad
 # CHECK-NEXT: [17]  - PdMul
 # CHECK-NEXT: [18]  - PdStore
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3]    [4]    [5]    [6]    [7.0]  [7.1]  [8.0]  [8.1]  [9]    [10]   [11]   [12]   [13]   [14]   [15]   [16.0] [16.1] [17]   [18]
-# CHECK-NEXT:  -      -      -      -      -     67.50  67.50   -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
+# CHECK-NEXT:  -      -      -      -      -     157.50 157.50  -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3]    [4]    [5]    [6]    [7.0]  [7.1]  [8.0]  [8.1]  [9]    [10]   [11]   [12]   [13]   [14]   [15]   [16.0] [16.1] [17]   [18]   Instructions:
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  0, %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  0, %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  0, %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%eax), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%eax), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%eax), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%rax), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%rax), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%rax), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%ebx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%ebx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%ebx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%rbx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%rbx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%rbx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%eax,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%eax,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%eax,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%rax,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%rax,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%rax,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%eax,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%eax,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%eax,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%rax,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%rax,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%rax,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%eax,%ebx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%eax,%ebx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%eax,%ebx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%rax,%rbx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%rax,%rbx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%rax,%rbx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16, %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16, %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16, %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%eax), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%eax), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%eax), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%rax), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%rax), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%rax), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%ebx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%ebx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%ebx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%rbx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%rbx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%rbx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%eax,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%eax,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%eax,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%rax,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%rax,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%rax,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%eax,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%eax,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%eax,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%rax,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%rax,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%rax,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%eax,%ebx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%eax,%ebx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%eax,%ebx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%rax,%rbx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%rax,%rbx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%rax,%rbx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024, %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024, %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024, %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%eax), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%eax), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%eax), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%rax), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%rax), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%rax), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%ebx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%ebx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%ebx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%rbx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%rbx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%rbx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%eax,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%eax,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%eax,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%rax,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%rax,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%rax,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%eax,%ebx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%eax,%ebx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%eax,%ebx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%rax,%rbx), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%rax,%rbx), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%rax,%rbx), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%eax,%ebx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%eax,%ebx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%eax,%ebx,2), %rcx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%rax,%rbx,2), %cx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%rax,%rbx,2), %ecx
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  0, %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  0, %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  0, %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%eax), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%eax), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%eax), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%rax), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%eax,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%eax,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  (%rax,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  (%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  (%rax,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16, %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16, %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16, %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%eax), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%eax), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%eax), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%rax), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%eax,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  -16(%rax,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  -16(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  -16(%rax,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024, %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024, %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024, %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%eax), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%eax), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%eax), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%rax), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%rax), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%rax), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(,%rbx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%eax,%ebx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%eax,%ebx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%eax,%ebx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%rax,%rbx), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%rax,%rbx), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%rax,%rbx), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%eax,%ebx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%eax,%ebx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%eax,%ebx,2), %rcx
+# CHECK-NEXT:  -      -      -      -      -     1.50   1.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaw  1024(%rax,%rbx,2), %cx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leal  1024(%rax,%rbx,2), %ecx
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  1024(%rax,%rbx,2), %rcx

llvm/test/tools/llvm-mca/X86/BdVer2/vbroadcast-operand-latency.s

 # NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
 # RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=bdver2 -timeline -timeline-max-iterations=3 < %s | FileCheck %s
 
 leaq 8(%rsp, %rdi, 2), %rax
 vbroadcastss (%rax), %ymm0
 
 # CHECK:      Iterations:        100
 # CHECK-NEXT: Instructions:      200
 # CHECK-NEXT: Total Cycles:      206
-# CHECK-NEXT: Total uOps:        400
+# CHECK-NEXT: Total uOps:        300
 
 # CHECK:      Dispatch Width:    4
-# CHECK-NEXT: uOps Per Cycle:    1.94
+# CHECK-NEXT: uOps Per Cycle:    1.46
 # CHECK-NEXT: IPC:               0.97
 # CHECK-NEXT: Block RThroughput: 2.0
 
 # CHECK:      Instruction Info:
 # CHECK-NEXT: [1]: #uOps
 # CHECK-NEXT: [2]: Latency
 # CHECK-NEXT: [3]: RThroughput
 # CHECK-NEXT: [4]: MayLoad
 # CHECK-NEXT: [5]: MayStore
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  2      1     0.50                        leaq  8(%rsp,%rdi,2), %rax
+# CHECK-NEXT:  1      2     1.00                        leaq  8(%rsp,%rdi,2), %rax
 # CHECK-NEXT:  2      6     2.00    *                   vbroadcastss  (%rax), %ymm0
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0.0] - PdAGLU01
 # CHECK-NEXT: [0.1] - PdAGLU01
 # CHECK-NEXT: [1]   - PdBranch
 # CHECK-NEXT: [2]   - PdCount
 # CHECK-NEXT: [3]   - PdDiv

 # CHECK-NEXT: [15]  - PdFPXBR
 # CHECK-NEXT: [16.0] - PdLoad
 # CHECK-NEXT: [16.1] - PdLoad
 # CHECK-NEXT: [17]  - PdMul
 # CHECK-NEXT: [18]  - PdStore
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3]    [4]    [5]    [6]    [7.0]  [7.1]  [8.0]  [8.1]  [9]    [10]   [11]   [12]   [13]   [14]   [15]   [16.0] [16.1] [17]   [18]
-# CHECK-NEXT: 0.50   0.50    -      -      -     0.50   0.50    -     2.00   2.00    -      -      -      -     1.00   1.00    -      -      -     0.50   0.50    -      -
+# CHECK-NEXT: 0.50   0.50    -      -      -     1.00   1.00    -     2.00   2.00    -      -      -      -     1.00   1.00    -      -      -     0.50   0.50    -      -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3]    [4]    [5]    [6]    [7.0]  [7.1]  [8.0]  [8.1]  [9]    [10]   [11]   [12]   [13]   [14]   [15]   [16.0] [16.1] [17]   [18]   Instructions:
-# CHECK-NEXT:  -      -      -      -      -     0.50   0.50    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  8(%rsp,%rdi,2), %rax
+# CHECK-NEXT:  -      -      -      -      -     1.00   1.00    -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -     leaq  8(%rsp,%rdi,2), %rax
 # CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -     2.00   2.00    -      -      -      -     1.00   1.00    -      -      -     0.50   0.50    -      -     vbroadcastss (%rax), %ymm0
 
 # CHECK:      Timeline view:
 # CHECK-NEXT:                     012
 # CHECK-NEXT: Index     0123456789
 
-# CHECK:      [0,0]     DeER .    . .   leaq  8(%rsp,%rdi,2), %rax
+# CHECK:      [0,0]     DeeER.    . .   leaq  8(%rsp,%rdi,2), %rax
 # CHECK-NEXT: [0,1]     DeeeeeeER . .   vbroadcastss  (%rax), %ymm0
-# CHECK-NEXT: [1,0]     .DeE----R . .   leaq  8(%rsp,%rdi,2), %rax
+# CHECK-NEXT: [1,0]     DeeE----R . .   leaq  8(%rsp,%rdi,2), %rax
 # CHECK-NEXT: [1,1]     .DeeeeeeER. .   vbroadcastss  (%rax), %ymm0
-# CHECK-NEXT: [2,0]     . DeE----R. .   leaq  8(%rsp,%rdi,2), %rax
+# CHECK-NEXT: [2,0]     .D=eeE---R. .   leaq  8(%rsp,%rdi,2), %rax
 # CHECK-NEXT: [2,1]     . D==eeeeeeER   vbroadcastss  (%rax), %ymm0
 
 # CHECK:      Average Wait times (based on the timeline view):
 # CHECK-NEXT: [0]: Executions
 # CHECK-NEXT: [1]: Average time spent waiting in a scheduler's queue
 # CHECK-NEXT: [2]: Average time spent waiting in a scheduler's queue while ready
 # CHECK-NEXT: [3]: Average time elapsed from WB until retire stage
 
 # CHECK:            [0]    [1]    [2]    [3]
-# CHECK-NEXT: 0.     3     1.0    1.0    2.7       leaq 8(%rsp,%rdi,2), %rax
-# CHECK-NEXT: 1.     3     1.7    0.7    0.0       vbroadcastss (%rax), %ymm0
-# CHECK-NEXT:        3     1.3    0.8    1.3       <total>
+# CHECK-NEXT: 0.     3     1.3    1.3    2.3       leaq 8(%rsp,%rdi,2), %rax
+# CHECK-NEXT: 1.     3     1.7    1.0    0.0       vbroadcastss (%rax), %ymm0
+# CHECK-NEXT:        3     1.5    1.2    1.2       <total>

llvm/test/tools/llvm-mca/X86/Generic/resources-clzero.s

 # NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
 # RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=x86-64 -instruction-tables < %s | FileCheck %s
 
 clzero
 
 # CHECK:      Instruction Info:
 # CHECK-NEXT: [1]: #uOps
 # CHECK-NEXT: [2]: Latency
 # CHECK-NEXT: [3]: RThroughput
 # CHECK-NEXT: [4]: MayLoad
 # CHECK-NEXT: [5]: MayStore
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      5   0.50                  U     clzero
+# CHECK-NEXT:  1      5     0.50                  U     clzero
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - SBDivider
 # CHECK-NEXT: [1]   - SBFPDivider
 # CHECK-NEXT: [2]   - SBPort0
 # CHECK-NEXT: [3]   - SBPort1
 # CHECK-NEXT: [4]   - SBPort4
 # CHECK-NEXT: [5]   - SBPort5
 # CHECK-NEXT: [6.0] - SBPort23
 # CHECK-NEXT: [6.1] - SBPort23
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6.0]  [6.1]
 # CHECK-NEXT:  -      -      -      -      -      -     0.50   0.50
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6.0]  [6.1]  Instructions:
-# CHECK-NEXT: -      -      -      -      -      -     0.50   0.50   clzero
+# CHECK-NEXT:  -      -      -      -      -      -     0.50   0.50   clzero

llvm/test/tools/llvm-mca/X86/Znver1/resources-clzero.s

 # NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
 # RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver1 -instruction-tables < %s | FileCheck %s
 
 clzero
 
 # CHECK:      Instruction Info:
 # CHECK-NEXT: [1]: #uOps
 # CHECK-NEXT: [2]: Latency
 # CHECK-NEXT: [3]: RThroughput
 # CHECK-NEXT: [4]: MayLoad
 # CHECK-NEXT: [5]: MayStore
 # CHECK-NEXT: [6]: HasSideEffects (U)
 
 # CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
-# CHECK-NEXT:  1      8    0.50                   U       clzero
+# CHECK-NEXT:  1      8     0.50                  U     clzero
 
 # CHECK:      Resources:
 # CHECK-NEXT: [0]   - ZnAGU0
 # CHECK-NEXT: [1]   - ZnAGU1
 # CHECK-NEXT: [2]   - ZnALU0
 # CHECK-NEXT: [3]   - ZnALU1
 # CHECK-NEXT: [4]   - ZnALU2
 # CHECK-NEXT: [5]   - ZnALU3
 # CHECK-NEXT: [6]   - ZnDivider
 # CHECK-NEXT: [7]   - ZnFPU0
 # CHECK-NEXT: [8]   - ZnFPU1
 # CHECK-NEXT: [9]   - ZnFPU2
 # CHECK-NEXT: [10]  - ZnFPU3
 # CHECK-NEXT: [11]  - ZnMultiplier
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]
-# CHECK-NEXT:  0.50   0.50    -      -      -      -      -      -      -      -      -      -
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   Instructions:
 # CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     clzero

llvm/test/tools/llvm-mca/X86/bextr-read-after-ld.s

 # SKYLAKE-NEXT: IPC:               0.20
 # SKYLAKE-NEXT: Block RThroughput: 0.7
 
 # ZNVER1:       Dispatch Width:    4
 # ZNVER1-NEXT:  uOps Per Cycle:    0.38
 # ZNVER1-NEXT:  IPC:               0.25
 # ZNVER1-NEXT:  Block RThroughput: 0.8
 
+# ZNVER2:       Dispatch Width:    4
+# ZNVER2-NEXT:  uOps Per Cycle:    0.38
+# ZNVER2-NEXT:  IPC:               0.25
+# ZNVER2-NEXT:  Block RThroughput: 0.8
+
 # ALL:          Instruction Info:
 # ALL-NEXT:     [1]: #uOps
 # ALL-NEXT:     [2]: Latency
 # ALL-NEXT:     [3]: RThroughput
 # ALL-NEXT:     [4]: MayLoad
 # ALL-NEXT:     [5]: MayStore
 # ALL-NEXT:     [6]: HasSideEffects (U)
 

 # HASWELL-NEXT:  3      7     0.50    *                   bextrl  %esi, (%rdi), %eax
 
 # SKYLAKE-NEXT:  1      1     0.25                        addl  %edi, %esi
 # SKYLAKE-NEXT:  3      7     0.50    *                   bextrl  %esi, (%rdi), %eax
 
 # ZNVER1-NEXT:   1      1     0.25                        addl  %edi, %esi
 # ZNVER1-NEXT:   2      5     0.50    *                   bextrl  %esi, (%rdi), %eax
 
+# ZNVER2-NEXT:   1      1     0.25                        addl  %edi, %esi
+# ZNVER2-NEXT:   2      5     0.33    *                   bextrl  %esi, (%rdi), %eax
+
 # ALL:          Timeline view:
 
 # BDVER2-NEXT:  Index     012345678
 # BDWELL-NEXT:  Index     0123456789
 # BTVER2-NEXT:  Index     0123456
 # HASWELL-NEXT: Index     0123456789
 # SKYLAKE-NEXT: Index     0123456789
 # ZNVER1-NEXT:  Index     01234567
+# ZNVER2-NEXT:  Index     01234567
 
 # BDVER2:       [0,0]     DeER .  .   addl  %edi, %esi
 # BDVER2-NEXT:  [0,1]     DeeeeeeER   bextrl  %esi, (%rdi), %eax
 
 # BDWELL:       [0,0]     DeER .   .   addl %edi, %esi
 # BDWELL-NEXT:  [0,1]     DeeeeeeeER   bextrl %esi, (%rdi), %eax
 
 # BTVER2:       [0,0]     DeER ..   addl  %edi, %esi
 # BTVER2-NEXT:  [0,1]     DeeeeER   bextrl  %esi, (%rdi), %eax
 
 # HASWELL:      [0,0]     DeER .   .   addl %edi, %esi
 # HASWELL-NEXT: [0,1]     DeeeeeeeER   bextrl %esi, (%rdi), %eax
 
 # SKYLAKE:      [0,0]     DeER .   .   addl %edi, %esi
 # SKYLAKE-NEXT: [0,1]     DeeeeeeeER   bextrl %esi, (%rdi), %eax
 
 # ZNVER1:       [0,0]     DeER . .   addl %edi, %esi
 # ZNVER1-NEXT:  [0,1]     DeeeeeER   bextrl %esi, (%rdi), %eax
 
+# ZNVER2:       [0,0]     DeER . .   addl %edi, %esi
+# ZNVER2-NEXT:  [0,1]     DeeeeeER   bextrl %esi, (%rdi), %eax
+
 # ALL:          Average Wait times (based on the timeline view):
 # ALL-NEXT:     [0]: Executions
 # ALL-NEXT:     [1]: Average time spent waiting in a scheduler's queue
 # ALL-NEXT:     [2]: Average time spent waiting in a scheduler's queue while ready
 # ALL-NEXT:     [3]: Average time elapsed from WB until retire stage
 
 # ALL:                [0]    [1]    [2]    [3]
 # ALL-NEXT:     0.     1     1.0    1.0    0.0       addl %edi, %esi
 # ALL-NEXT:     1.     1     1.0    0.0    0.0       bextrl %esi, (%rdi), %eax
 # ALL-NEXT:            1     1.0    0.5    0.0       <total>

llvm/test/tools/llvm-mca/X86/bzhi-read-after-ld.s

 # SKYLAKE-NEXT: IPC:               0.22
 # SKYLAKE-NEXT: Block RThroughput: 0.5
 
 # ZNVER1:       Dispatch Width:    4
 # ZNVER1-NEXT:  uOps Per Cycle:    0.38
 # ZNVER1-NEXT:  IPC:               0.25
 # ZNVER1-NEXT:  Block RThroughput: 0.8
 
+# ZNVER2:       Dispatch Width:    4
+# ZNVER2-NEXT:  uOps Per Cycle:    0.38
+# ZNVER2-NEXT:  IPC:               0.25
+# ZNVER2-NEXT:  Block RThroughput: 0.8
+
 # ALL:          Instruction Info:
 # ALL-NEXT:     [1]: #uOps
 # ALL-NEXT:     [2]: Latency
 # ALL-NEXT:     [3]: RThroughput
 # ALL-NEXT:     [4]: MayLoad
 # ALL-NEXT:     [5]: MayStore
 # ALL-NEXT:     [6]: HasSideEffects (U)
 
 # ALL:          [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 # ALL-NEXT:      1      1     0.25                        addl  %edi, %esi
 
 # BDWELL-NEXT:   2      6     0.50    *                   bzhil %esi, (%rdi), %eax
 # HASWELL-NEXT:  2      6     0.50    *                   bzhil %esi, (%rdi), %eax
 # SKYLAKE-NEXT:  2      6     0.50    *                   bzhil %esi, (%rdi), %eax
 # ZNVER1-NEXT:   2      5     0.50    *                   bzhil %esi, (%rdi), %eax
+# ZNVER2-NEXT:   2      5     0.33    *                   bzhil %esi, (%rdi), %eax
 
 # ALL:          Timeline view:
 
 # BDWELL-NEXT:  Index     012345678
 # HASWELL-NEXT: Index     012345678
 # SKYLAKE-NEXT: Index     012345678
 # ZNVER1-NEXT:  Index     01234567
+# ZNVER2-NEXT:  Index     01234567
 
 # BDWELL:       [0,0]     DeER .  .   addl  %edi, %esi
 # BDWELL-NEXT:  [0,1]     DeeeeeeER   bzhil %esi, (%rdi), %eax
 
 # HASWELL:      [0,0]     DeER .  .   addl  %edi, %esi
 # HASWELL-NEXT: [0,1]     DeeeeeeER   bzhil %esi, (%rdi), %eax
 
 # SKYLAKE:      [0,0]     DeER .  .   addl  %edi, %esi
 # SKYLAKE-NEXT: [0,1]     DeeeeeeER   bzhil %esi, (%rdi), %eax
 
 # ZNVER1:       [0,0]     DeER . .   addl %edi, %esi
 # ZNVER1-NEXT:  [0,1]     DeeeeeER   bzhil  %esi, (%rdi), %eax
 
+# ZNVER2:       [0,0]     DeER . .   addl %edi, %esi
+# ZNVER2-NEXT:  [0,1]     DeeeeeER   bzhil  %esi, (%rdi), %eax
+
 # ALL:          Average Wait times (based on the timeline view):
 # ALL-NEXT:     [0]: Executions
 # ALL-NEXT:     [1]: Average time spent waiting in a scheduler's queue
 # ALL-NEXT:     [2]: Average time spent waiting in a scheduler's queue while ready
 # ALL-NEXT:     [3]: Average time elapsed from WB until retire stage
 
 # ALL:                [0]    [1]    [2]    [3]
 # ALL-NEXT:     0.     1     1.0    1.0    0.0       addl %edi, %esi
 # ALL-NEXT:     1.     1     1.0    0.0    0.0       bzhil  %esi, (%rdi), %eax
 # ALL-NEXT:            1     1.0    0.5    0.0       <total>

llvm/test/tools/llvm-mca/X86/read-after-ld-1.s

 
 # SKYLAKE-NEXT:                       01234567
 # SKYLAKE-NEXT:   Index     0123456789
 
 # ZNVER1-NEXT:                        0123456789
 # ZNVER1-NEXT:    Index     0123456789
 
 # ZNVER2-NEXT:                        0123456789
-# ZNVER2-NEXT:    Index     0123456789
+# ZNVER2-NEXT:    Index     0123456789          0
 
 # BARCELONA:      [0,0]     DeeeeeeeeeeeeeeER  .   vdivps %xmm0, %xmm1, %xmm1
 # BARCELONA-NEXT: [0,1]     D========eeeeeeeeeER   vaddps (%rax), %xmm1, %xmm1
 
 # BDVER2:         [0,0]     DeeeeeeeeeER   ..   vdivps  %xmm0, %xmm1, %xmm1
 # BDVER2-NEXT:    [0,1]     D====eeeeeeeeeeER   vaddps  (%rax), %xmm1, %xmm1
 
 # BDWELL:         [0,0]     DeeeeeeeeeeeER ..   vdivps  %xmm0, %xmm1, %xmm1

 # SANDY-NEXT:     [0,1]     D========eeeeeeeeeER   vaddps (%rax), %xmm1, %xmm1
 
 # SKYLAKE:        [0,0]     DeeeeeeeeeeeER . .   vdivps %xmm0, %xmm1, %xmm1
 # SKYLAKE-NEXT:   [0,1]     D=====eeeeeeeeeeER   vaddps (%rax), %xmm1, %xmm1
 
 # ZNVER1:         [0,0]     DeeeeeeeeeeeeeeeER .   vdivps %xmm0, %xmm1, %xmm1
 # ZNVER1-NEXT:    [0,1]     D=======eeeeeeeeeeER   vaddps (%rax), %xmm1, %xmm1
 
-# ZNVER2:         [0,0]     DeeeeeeeeeeeeeeeER .   vdivps %xmm0, %xmm1, %xmm1
+# ZNVER2:         [0,0]     DeeeeeeeeeeeeeeeER  .   vdivps  %xmm0, %xmm1, %xmm1
 # ZNVER2-NEXT:    [0,1]     D========eeeeeeeeeeER   vaddps  (%rax), %xmm1, %xmm1
 
 # ALL:            Average Wait times (based on the timeline view):
 # ALL-NEXT:       [0]: Executions
 # ALL-NEXT:       [1]: Average time spent waiting in a scheduler's queue
 # ALL-NEXT:       [2]: Average time spent waiting in a scheduler's queue while ready
 # ALL-NEXT:       [3]: Average time elapsed from WB until retire stage
 

llvm/test/tools/llvm-mca/X86/read-after-ld-2.s

 # HASWELL-NEXT: Total uOps:        500
 
 # SKYLAKE-NEXT: Total Cycles:      110
 # SKYLAKE-NEXT: Total uOps:        500
 
 # ZNVER1-NEXT:  Total Cycles:      110
 # ZNVER1-NEXT:  Total uOps:        400
 
+# ZNVER2-NEXT:  Total Cycles:      110
+# ZNVER2-NEXT:  Total uOps:        400
+
 # BDWELL:       Dispatch Width:    4
 # BDWELL-NEXT:  uOps Per Cycle:    3.52
 # BDWELL-NEXT:  IPC:               2.82
 # BDWELL-NEXT:  Block RThroughput: 1.3
 
 # HASWELL:      Dispatch Width:    4
 # HASWELL-NEXT: uOps Per Cycle:    3.50
 # HASWELL-NEXT: IPC:               2.80

 # ZNVER1-NEXT:  [8,1]     .    .  DeeeeeeeeER.   vpaddd (%r8), %ymm0, %ymm0
 # ZNVER1-NEXT:  [8,2]     .    .  DeE-------R.   addq $32, %r8
 # ZNVER1-NEXT:  [8,3]     .    .  D=eE------R.   cmpl %edi, %edx
 # ZNVER1-NEXT:  [9,0]     .    .   DeE------R.   addl $1, %edx
 # ZNVER1-NEXT:  [9,1]     .    .   DeeeeeeeeER   vpaddd (%r8), %ymm0, %ymm0
 # ZNVER1-NEXT:  [9,2]     .    .   DeE-------R   addq $32, %r8
 # ZNVER1-NEXT:  [9,3]     .    .   D=eE------R   cmpl %edi, %edx
 
-# ZNVER2:       [0,0]     DeER .    .    .   .   addl   $1, %edx
-# ZNVER2-NEXT:  [0,1]     DeeeeeeeeER    .   .   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [0,2]     DeE-------R    .   .   addq   $32, %r8
-# ZNVER2-NEXT:  [0,3]     D=eE------R    .   .   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [1,0]     .DeE------R    .   .   addl   $1, %edx
-# ZNVER2-NEXT:  [1,1]     .DeeeeeeeeER   .   .   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [1,2]     .DeE-------R   .   .   addq   $32, %r8
-# ZNVER2-NEXT:  [1,3]     .D=eE------R   .   .   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [2,0]     . DeE------R   .   .   addl   $1, %edx
-# ZNVER2-NEXT:  [2,1]     . DeeeeeeeeER  .   .   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [2,2]     . DeE-------R  .   .   addq   $32, %r8
-# ZNVER2-NEXT:  [2,3]     . D=eE------R  .   .   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [3,0]     .  DeE------R  .   .   addl   $1, %edx
-# ZNVER2-NEXT:  [3,1]     .  DeeeeeeeeER .   .   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [3,2]     .  DeE-------R .   .   addq   $32, %r8
-# ZNVER2-NEXT:  [3,3]     .  D=eE------R .   .   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [4,0]     .   DeE------R .   .   addl   $1, %edx
-# ZNVER2-NEXT:  [4,1]     .   DeeeeeeeeER.   .   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [4,2]     .   DeE-------R.   .   addq   $32, %r8
-# ZNVER2-NEXT:  [4,3]     .   D=eE------R.   .   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [5,0]     .    DeE------R.   .   addl   $1, %edx
-# ZNVER2-NEXT:  [5,1]     .    DeeeeeeeeER   .   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [5,2]     .    DeE-------R   .   addq   $32, %r8
-# ZNVER2-NEXT:  [5,3]     .    D=eE------R   .   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [6,0]     .    .DeE------R   .   addl   $1, %edx
-# ZNVER2-NEXT:  [6,1]     .    .DeeeeeeeeER  .   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [6,2]     .    .DeE-------R  .   addq   $32, %r8
-# ZNVER2-NEXT:  [6,3]     .    .D=eE------R  .   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [7,0]     .    . DeE------R  .   addl   $1, %edx
-# ZNVER2-NEXT:  [7,1]     .    . DeeeeeeeeER .   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [7,2]     .    . DeE-------R .   addq   $32, %r8
-# ZNVER2-NEXT:  [7,3]     .    . D=eE------R .   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [8,0]     .    .  DeE------R .   addl   $1, %edx
-# ZNVER2-NEXT:  [8,1]     .    .  DeeeeeeeeER.   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [8,2]     .    .  DeE-------R.   addq   $32, %r8
-# ZNVER2-NEXT:  [8,3]     .    .  D=eE------R.   cmpl   %edi, %edx
-# ZNVER2-NEXT:  [9,0]     .    .   DeE------R.   addl   $1, %edx
-# ZNVER2-NEXT:  [9,1]     .    .   DeeeeeeeeER   vpaddd (%r8), %ymm0, %ymm0
-# ZNVER2-NEXT:  [9,2]     .    .   DeE-------R   addq   $32, %r8
-# ZNVER2-NEXT:  [9,3]     .    .   D=eE------R   cmpl   %edi, %edx
+# ZNVER2:       [0,0]     DeER .    .    .   .   addl $1, %edx
+# ZNVER2-NEXT:  [0,1]     DeeeeeeeeER    .   .   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [0,2]     DeE-------R    .   .   addq $32, %r8
+# ZNVER2-NEXT:  [0,3]     D=eE------R    .   .   cmpl %edi, %edx
+# ZNVER2-NEXT:  [1,0]     .DeE------R    .   .   addl $1, %edx
+# ZNVER2-NEXT:  [1,1]     .DeeeeeeeeER   .   .   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [1,2]     .DeE-------R   .   .   addq $32, %r8
+# ZNVER2-NEXT:  [1,3]     .D=eE------R   .   .   cmpl %edi, %edx
+# ZNVER2-NEXT:  [2,0]     . DeE------R   .   .   addl $1, %edx
+# ZNVER2-NEXT:  [2,1]     . DeeeeeeeeER  .   .   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [2,2]     . DeE-------R  .   .   addq $32, %r8
+# ZNVER2-NEXT:  [2,3]     . D=eE------R  .   .   cmpl %edi, %edx
+# ZNVER2-NEXT:  [3,0]     .  DeE------R  .   .   addl $1, %edx
+# ZNVER2-NEXT:  [3,1]     .  DeeeeeeeeER .   .   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [3,2]     .  DeE-------R .   .   addq $32, %r8
+# ZNVER2-NEXT:  [3,3]     .  D=eE------R .   .   cmpl %edi, %edx
+# ZNVER2-NEXT:  [4,0]     .   DeE------R .   .   addl $1, %edx
+# ZNVER2-NEXT:  [4,1]     .   DeeeeeeeeER.   .   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [4,2]     .   DeE-------R.   .   addq $32, %r8
+# ZNVER2-NEXT:  [4,3]     .   D=eE------R.   .   cmpl %edi, %edx
+# ZNVER2-NEXT:  [5,0]     .    DeE------R.   .   addl $1, %edx
+# ZNVER2-NEXT:  [5,1]     .    DeeeeeeeeER   .   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [5,2]     .    DeE-------R   .   addq $32, %r8
+# ZNVER2-NEXT:  [5,3]     .    D=eE------R   .   cmpl %edi, %edx
+# ZNVER2-NEXT:  [6,0]     .    .DeE------R   .   addl $1, %edx
+# ZNVER2-NEXT:  [6,1]     .    .DeeeeeeeeER  .   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [6,2]     .    .DeE-------R  .   addq $32, %r8
+# ZNVER2-NEXT:  [6,3]     .    .D=eE------R  .   cmpl %edi, %edx
+# ZNVER2-NEXT:  [7,0]     .    . DeE------R  .   addl $1, %edx
+# ZNVER2-NEXT:  [7,1]     .    . DeeeeeeeeER .   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [7,2]     .    . DeE-------R .   addq $32, %r8
+# ZNVER2-NEXT:  [7,3]     .    . D=eE------R .   cmpl %edi, %edx
+# ZNVER2-NEXT:  [8,0]     .    .  DeE------R .   addl $1, %edx
+# ZNVER2-NEXT:  [8,1]     .    .  DeeeeeeeeER.   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [8,2]     .    .  DeE-------R.   addq $32, %r8
+# ZNVER2-NEXT:  [8,3]     .    .  D=eE------R.   cmpl %edi, %edx
+# ZNVER2-NEXT:  [9,0]     .    .   DeE------R.   addl $1, %edx
+# ZNVER2-NEXT:  [9,1]     .    .   DeeeeeeeeER   vpaddd (%r8), %ymm0, %ymm0
+# ZNVER2-NEXT:  [9,2]     .    .   DeE-------R   addq $32, %r8
+# ZNVER2-NEXT:  [9,3]     .    .   D=eE------R   cmpl %edi, %edx
 
 # ALL:          Average Wait times (based on the timeline view):
 # ALL-NEXT:     [0]: Executions
 # ALL-NEXT:     [1]: Average time spent waiting in a scheduler's queue
 # ALL-NEXT:     [2]: Average time spent waiting in a scheduler's queue while ready
 # ALL-NEXT:     [3]: Average time elapsed from WB until retire stage
 
 # ALL:                [0]    [1]    [2]    [3]

llvm/test/tools/llvm-mca/X86/scheduler-queue-usage.s

 # SNB-NEXT:        [4] Total number of buffer entries.
 
 # ZNVER1:          Scheduler's queue usage:
 # ZNVER1-NEXT:     [1] Resource name.
 # ZNVER1-NEXT:     [2] Average number of used buffer entries.
 # ZNVER1-NEXT:     [3] Maximum number of used buffer entries.
 # ZNVER1-NEXT:     [4] Total number of buffer entries.
 
-# ZNVER2:         Scheduler's queue usage:
+# ZNVER2:          Scheduler's queue usage:
 # ZNVER2-NEXT:     [1] Resource name.
 # ZNVER2-NEXT:     [2] Average number of used buffer entries.
 # ZNVER2-NEXT:     [3] Maximum number of used buffer entries.
 # ZNVER2-NEXT:     [4] Total number of buffer entries.
 
 # BARCELONA:        [1]            [2]        [3]        [4]
 # BARCELONA-NEXT:  SBPortAny        0          1          54
 

 
 # SNB:              [1]            [2]        [3]        [4]
 # SNB-NEXT:        SBPortAny        0          1          54
 
 # ZNVER1:           [1]            [2]        [3]        [4]
 # ZNVER1-NEXT:     ZnAGU            0          0          28
 # ZNVER1-NEXT:     ZnALU            0          1          56
 # ZNVER1-NEXT:     ZnFPU            0          0          36
+
+# ZNVER2:           [1]            [2]        [3]        [4]
+# ZNVER2-NEXT:     Zn2AGU           0          0          28
+# ZNVER2-NEXT:     Zn2ALU           0          1          64
+# ZNVER2-NEXT:     Zn2FPU           0          0          36

llvm/test/tools/llvm-mca/X86/sqrt-rsqrt-rcp-memop.s

 
 # ALL:            [0] Code Region - test_sqrtss
 
 # ALL:            Timeline view:
 
 # BARCELONA-NEXT:                     0123456789
 # BARCELONA-NEXT: Index     0123456789          0123
 
-# BDVER2-NEXT:                        01234567
+# BDVER2-NEXT:                        012345678
 # BDVER2-NEXT:    Index     0123456789
 
 # BROADWELL-NEXT:                     0123456789
 # BROADWELL-NEXT: Index     0123456789
 
 # BTVER2-NEXT:                        0123456789          0
 # BTVER2-NEXT:    Index     0123456789          0123456789
 
 # HASWELL-NEXT:                       0123456789
 # HASWELL-NEXT:   Index     0123456789
 
 # SKYLAKE-NEXT:                       0123456789
 # SKYLAKE-NEXT:   Index     0123456789          0
 
 # ZNVER1-NEXT:                        0123456789          0
 # ZNVER1-NEXT:    Index     0123456789          0123456789
 
 # ZNVER2-NEXT:                        0123456789          0
 # ZNVER2-NEXT:    Index     0123456789          0123456789
 
 # BARCELONA:      [0,0]     DeER .    .    .    .  .   leaq 8(%rsp,%rdi,2), %rax
 # BARCELONA-NEXT: [0,1]     D=eeeeeeeeeeeeeeeeeeeeER   sqrtss (%rax), %xmm1
 
-# BDVER2:         [0,0]     DeER .    .    . .   leaq 8(%rsp,%rdi,2), %rax
-# BDVER2-NEXT:    [0,1]     D=eeeeeeeeeeeeeeER   sqrtss (%rax), %xmm1
+# BDVER2:         [0,0]     DeeER.    .    .  .   leaq  8(%rsp,%rdi,2), %rax
+# BDVER2-NEXT:    [0,1]     D==eeeeeeeeeeeeeeER   sqrtss  (%rax), %xmm1
 
 # BROADWELL:      [0,0]     DeER .    .    .   .   leaq 8(%rsp,%rdi,2), %rax
 # BROADWELL-NEXT: [0,1]     D=eeeeeeeeeeeeeeeeER   sqrtss (%rax), %xmm1
 
 # BTVER2:         [0,0]     DeeER.    .    .    .    .    .   leaq  8(%rsp,%rdi,2), %rax
 # BTVER2-NEXT:    [0,1]     D==eeeeeeeeeeeeeeeeeeeeeeeeeeER   sqrtss  (%rax), %xmm1
 
 # HASWELL:        [0,0]     DeER .    .    .   .   leaq 8(%rsp,%rdi,2), %rax

 # ALL-NEXT:       [3]: Average time elapsed from WB until retire stage
 
 # ALL:                  [0]    [1]    [2]    [3]
 # ALL-NEXT:       0.     1     1.0    1.0    0.0       leaq 8(%rsp,%rdi,2), %rax
 
 # BARCELONA-NEXT: 1.     1     2.0    0.0    0.0       sqrtss (%rax), %xmm1
 # BARCELONA-NEXT:        1     1.5    0.5    0.0       <total>
 
-# BDVER2-NEXT:    1.     1     2.0    0.0    0.0       sqrtss (%rax), %xmm1
-# BDVER2-NEXT:           1     1.5    0.5    0.0       <total>
+# BDVER2-NEXT:    1.     1     3.0    0.0    0.0       sqrtss (%rax), %xmm1
+# BDVER2-NEXT:           1     2.0    0.5    0.0       <total>
 
 # BROADWELL-NEXT: 1.     1     2.0    0.0    0.0       sqrtss (%rax), %xmm1
 # BROADWELL-NEXT:        1     1.5    0.5    0.0       <total>
 
 # BTVER2-NEXT:    1.     1     3.0    0.0    0.0       sqrtss (%rax), %xmm1
 # BTVER2-NEXT:           1     2.0    0.5    0.0       <total>
 
 # HASWELL-NEXT:   1.     1     2.0    0.0    0.0       sqrtss (%rax), %xmm1

 
 # ALL:            [1] Code Region - test_sqrtsd
 
 # ALL:            Timeline view:
 
 # BARCELONA-NEXT:                     0123456789          0
 # BARCELONA-NEXT: Index     0123456789          0123456789
 
-# BDVER2-NEXT:                        01234567
+# BDVER2-NEXT:                        012345678
 # BDVER2-NEXT:    Index     0123456789
 
 # BROADWELL-NEXT:                     0123456789
 # BROADWELL-NEXT: Index     0123456789          01234
 
 # BTVER2-NEXT:                        0123456789          0123456
 # BTVER2-NEXT:    Index     0123456789          0123456789
 
 # HASWELL-NEXT:                       0123456789
 # HASWELL-NEXT:   Index     0123456789          01234
 
 # SKYLAKE-NEXT:                       0123456789
 # SKYLAKE-NEXT:   Index     0123456789          0123456
 
 # ZNVER1-NEXT:                        0123456789          0
 # ZNVER1-NEXT:    Index     0123456789          0123456789
 
 # ZNVER2-NEXT:                        0123456789          0
 # ZNVER2-NEXT:    Index     0123456789          0123456789
 
 # BARCELONA:      [0,0]     DeER .    .    .    .    .    .   leaq  8(%rsp,%rdi,2), %rax
 # BARCELONA-NEXT: [0,1]     D=eeeeeeeeeeeeeeeeeeeeeeeeeeeER   sqrtsd  (%rax), %xmm1
 
-# BDVER2:         [0,0]     DeER .    .    . .   leaq 8(%rsp,%rdi,2), %rax
-# BDVER2-NEXT:    [0,1]     D=eeeeeeeeeeeeeeER   sqrtsd (%rax), %xmm1
+# BDVER2:         [0,0]     DeeER.    .    .  .   leaq  8(%rsp,%rdi,2), %rax
+# BDVER2-NEXT:    [0,1]     D==eeeeeeeeeeeeeeER   sqrtsd  (%rax), %xmm1
 
 # BROADWELL:      [0,0]     DeER .    .    .    .   .   leaq  8(%rsp,%rdi,2), %rax
 # BROADWELL-NEXT: [0,1]     D=eeeeeeeeeeeeeeeeeeeeeER   sqrtsd  (%rax), %xmm1
 
 # BTVER2:         [0,0]     DeeER.    .    .    .    .    .    ..   leaq  8(%rsp,%rdi,2), %rax
 # BTVER2-NEXT:    [0,1]     D==eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeER   sqrtsd  (%rax), %xmm1
 
 # HASWELL:        [0,0]     DeER .    .    .    .   .   leaq  8(%rsp,%rdi,2), %rax

 # ALL-NEXT:       [3]: Average time elapsed from WB until retire stage
 
 # ALL:                  [0]    [1]    [2]    [3]
 # ALL-NEXT:       0.     1     1.0    1.0    0.0       leaq 8(%rsp,%rdi,2), %rax
 
 # BARCELONA-NEXT: 1.     1     2.0    0.0    0.0       sqrtsd (%rax), %xmm1
 # BARCELONA-NEXT:        1     1.5    0.5    0.0       <total>
 
-# BDVER2-NEXT:    1.     1     2.0    0.0    0.0       sqrtsd (%rax), %xmm1
-# BDVER2-NEXT:           1     1.5    0.5    0.0       <total>
+# BDVER2-NEXT:    1.     1     3.0    0.0    0.0       sqrtsd (%rax), %xmm1
+# BDVER2-NEXT:           1     2.0    0.5    0.0       <total>
 
 # BROADWELL-NEXT: 1.     1     2.0    0.0    0.0       sqrtsd (%rax), %xmm1
 # BROADWELL-NEXT:        1     1.5    0.5    0.0       <total>
 
 # BTVER2-NEXT:    1.     1     3.0    0.0    0.0       sqrtsd (%rax), %xmm1
 # BTVER2-NEXT:           1     2.0    0.5    0.0       <total>
 
 # HASWELL-NEXT:   1.     1     2.0    0.0    0.0       sqrtsd (%rax), %xmm1
 # HASWELL-NEXT:          1     1.5    0.5    0.0       <total>
 
 # SKYLAKE-NEXT:   1.     1     2.0    0.0    0.0       sqrtsd (%rax), %xmm1
 # SKYLAKE-NEXT:          1     1.5    0.5    0.0       <total>
 
 # ZNVER1-NEXT:    1.     1     2.0    0.0    0.0       sqrtsd (%rax), %xmm1
 # ZNVER1-NEXT:           1     1.5    0.5    0.0       <total>
 
 # ZNVER2-NEXT:    1.     1     2.0    0.0    0.0       sqrtsd (%rax), %xmm1
 # ZNVER2-NEXT:           1     1.5    0.5    0.0       <total>
 
 # ALL:            [2] Code Region - test_rsqrtss
 
 # ALL:            Timeline view:
 
 # BARCELONA-NEXT:                     01234
-# BDVER2-NEXT:                        0123
+# BDVER2-NEXT:                        01234
 # BROADWELL-NEXT:                     0123
 # BTVER2-NEXT:                        01
 # HASWELL-NEXT:                       0123
 # SKYLAKE-NEXT:                       012
 # ZNVER1-NEXT:                        012345
 # ZNVER2-NEXT:                        012345
 
 # ALL-NEXT:       Index     0123456789
 
 # BARCELONA:      [0,0]     DeER .    .   .   leaq  8(%rsp,%rdi,2), %rax
 # BARCELONA-NEXT: [0,1]     D=eeeeeeeeeeeER   rsqrtss (%rax), %xmm1
 
-# BDVER2:         [0,0]     DeER .    .  .   leaq 8(%rsp,%rdi,2), %rax
-# BDVER2-NEXT:    [0,1]     D=eeeeeeeeeeER   rsqrtss  (%rax), %xmm1
+# BDVER2:         [0,0]     DeeER.    .   .   leaq  8(%rsp,%rdi,2), %rax
+# BDVER2-NEXT:    [0,1]     D==eeeeeeeeeeER   rsqrtss (%rax), %xmm1
 
 # BROADWELL:      [0,0]     DeER .    .  .   leaq 8(%rsp,%rdi,2), %rax
 # BROADWELL-NEXT: [0,1]     D=eeeeeeeeeeER   rsqrtss  (%rax), %xmm1
 
 # BTVER2:         [0,0]     DeeER.    ..   leaq 8(%rsp,%rdi,2), %rax
 # BTVER2-NEXT:    [0,1]     D==eeeeeeeER   rsqrtss  (%rax), %xmm1
 
 # HASWELL:        [0,0]     DeER .    .  .   leaq 8(%rsp,%rdi,2), %rax

 # ALL-NEXT:       [3]: Average time elapsed from WB until retire stage
 
 # ALL:                  [0]    [1]    [2]    [3]
 # ALL-NEXT:       0.     1     1.0    1.0    0.0       leaq 8(%rsp,%rdi,2), %rax
 
 # BARCELONA-NEXT: 1.     1     2.0    0.0    0.0       rsqrtss  (%rax), %xmm1
 # BARCELONA-NEXT:        1     1.5    0.5    0.0       <total>
 
-# BDVER2-NEXT:    1.     1     2.0    0.0    0.0       rsqrtss  (%rax), %xmm1
-# BDVER2-NEXT:           1     1.5    0.5    0.0       <total>
+# BDVER2-NEXT:    1.     1     3.0    0.0    0.0       rsqrtss  (%rax), %xmm1
+# BDVER2-NEXT:           1     2.0    0.5    0.0       <total>
 
 # BROADWELL-NEXT: 1.     1     2.0    0.0    0.0       rsqrtss  (%rax), %xmm1
 # BROADWELL-NEXT:        1     1.5    0.5    0.0       <total>
 
 # BTVER2-NEXT:    1.     1     3.0    0.0    0.0       rsqrtss  (%rax), %xmm1
 # BTVER2-NEXT:           1     2.0    0.5    0.0       <total>
 
 # HASWELL-NEXT:   1.     1     2.0    0.0    0.0       rsqrtss  (%rax), %xmm1
 # HASWELL-NEXT:          1     1.5    0.5    0.0       <total>
 
 # SKYLAKE-NEXT:   1.     1     2.0    0.0    0.0       rsqrtss  (%rax), %xmm1
 # SKYLAKE-NEXT:          1     1.5    0.5    0.0       <total>
 
 # ZNVER1-NEXT:    1.     1     2.0    0.0    0.0       rsqrtss  (%rax), %xmm1
 # ZNVER1-NEXT:           1     1.5    0.5    0.0       <total>
 
 # ZNVER2-NEXT:    1.     1     2.0    0.0    0.0       rsqrtss  (%rax), %xmm1
 # ZNVER2-NEXT:           1     1.5    0.5    0.0       <total>
 
 # ALL:            [3] Code Region - test_rcp
 
 # ALL:            Timeline view:
 
 # BARCELONA-NEXT:                     01234
-# BDVER2-NEXT:                        0123
+# BDVER2-NEXT:                        01234
 # BROADWELL-NEXT:                     0123
 # BTVER2-NEXT:                        01
 # HASWELL-NEXT:                       0123
 # SKYLAKE-NEXT:                       012
 # ZNVER1-NEXT:                        012345
 # ZNVER2-NEXT:                        012345
 
 # ALL-NEXT:       Index     0123456789
 
 # BARCELONA:      [0,0]     DeER .    .   .   leaq  8(%rsp,%rdi,2), %rax
 # BARCELONA-NEXT: [0,1]     D=eeeeeeeeeeeER   rcpss (%rax), %xmm1
 
-# BDVER2:         [0,0]     DeER .    .  .   leaq 8(%rsp,%rdi,2), %rax
-# BDVER2-NEXT:    [0,1]     D=eeeeeeeeeeER   rcpss  (%rax), %xmm1
+# BDVER2:         [0,0]     DeeER.    .   .   leaq  8(%rsp,%rdi,2), %rax
+# BDVER2-NEXT:    [0,1]     D==eeeeeeeeeeER   rcpss (%rax), %xmm1
 
 # BROADWELL:      [0,0]     DeER .    .  .   leaq 8(%rsp,%rdi,2), %rax
 # BROADWELL-NEXT: [0,1]     D=eeeeeeeeeeER   rcpss  (%rax), %xmm1
 
 # BTVER2:         [0,0]     DeeER.    ..   leaq 8(%rsp,%rdi,2), %rax
 # BTVER2-NEXT:    [0,1]     D==eeeeeeeER   rcpss  (%rax), %xmm1
 
 # HASWELL:        [0,0]     DeER .    .  .   leaq 8(%rsp,%rdi,2), %rax

 # ALL-NEXT:       [3]: Average time elapsed from WB until retire stage
 
 # ALL:                  [0]    [1]    [2]    [3]
 # ALL-NEXT:       0.     1     1.0    1.0    0.0       leaq 8(%rsp,%rdi,2), %rax
 
 # BARCELONA-NEXT: 1.     1     2.0    0.0    0.0       rcpss  (%rax), %xmm1
 # BARCELONA-NEXT:        1     1.5    0.5    0.0       <total>
 
-# BDVER2-NEXT:    1.     1     2.0    0.0    0.0       rcpss  (%rax), %xmm1
-# BDVER2-NEXT:           1     1.5    0.5    0.0       <total>
+# BDVER2-NEXT:    1.     1     3.0    0.0    0.0       rcpss  (%rax), %xmm1
+# BDVER2-NEXT:           1     2.0    0.5    0.0       <total>
 
 # BROADWELL-NEXT: 1.     1     2.0    0.0    0.0       rcpss  (%rax), %xmm1
 # BROADWELL-NEXT:        1     1.5    0.5    0.0       <total>
 
 # BTVER2-NEXT:    1.     1     3.0    0.0    0.0       rcpss  (%rax), %xmm1
 # BTVER2-NEXT:           1     2.0    0.5    0.0       <total>
 
 # HASWELL-NEXT:   1.     1     2.0    0.0    0.0       rcpss  (%rax), %xmm1

llvm/test/tools/llvm-mca/X86/variable-blend-read-after-ld-2.s

 # SKYLAKE-NEXT: 1.     1     1.0    0.0    0.0       vblendvps  %xmm1, (%rdi), %xmm2, %xmm3
 # SKYLAKE-NEXT:        1     1.0    0.5    0.0       <total>
 
 # ZNVER1-NEXT:  1.     1     1.0    0.0    0.0       vblendvps  %xmm1, (%rdi), %xmm2, %xmm3
 # ZNVER1-NEXT:         1     1.0    0.5    0.0       <total>
 
 # ZNVER2-NEXT:  1.     1     1.0    0.0    0.0       vblendvps  %xmm1, (%rdi), %xmm2, %xmm3
 # ZNVER2-NEXT:         1     1.0    0.5    0.0       <total>
-

mlir/include/mlir/Conversion/LinalgToSPIRV/LinalgToSPIRV.h

+//===- LinalgToSPIRV.h - Linalg to SPIR-V dialect conversion ----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides patterns for Linalg to SPIR-V dialect conversion.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_CONVERSION_LINALGTOSPIRV_LINALGTOSPIRV_H
+#define MLIR_CONVERSION_LINALGTOSPIRV_LINALGTOSPIRV_H
+
+namespace mlir {
+class MLIRContext;
+class OwningRewritePatternList;
+class SPIRVTypeConverter;
+
+/// Appends to a pattern list additional patterns for translating Linalg ops to
+/// SPIR-V ops.
+void populateLinalgToSPIRVPatterns(MLIRContext *context,
+                                   SPIRVTypeConverter &typeConverter,
+                                   OwningRewritePatternList &patterns);
+
+} // namespace mlir
+
+#endif // MLIR_CONVERSION_LINALGTOSPIRV_LINALGTOSPIRV_H

mlir/include/mlir/Conversion/LinalgToSPIRV/LinalgToSPIRVPass.h

+//===- LinalgToSPIRVPass.h -  Linalg to SPIR-V conversion pass --*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides a pass for Linalg to SPIR-V dialect conversion.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_CONVERSION_STANDARDTOSPIRV_LINALGTOSPIRVPASS_H
+#define MLIR_CONVERSION_STANDARDTOSPIRV_LINALGTOSPIRVPASS_H
+
+#include "mlir/Pass/Pass.h"
+
+namespace mlir {
+
+/// Creates and returns a pass to convert Linalg ops to SPIR-V ops.
+std::unique_ptr<OpPassBase<ModuleOp>> createLinalgToSPIRVPass();
+
+} // namespace mlir
+
+#endif // MLIR_CONVERSION_STANDARDTOSPIRV_LINALGTOSPIRVPASS_H

mlir/include/mlir/Dialect/SPIRV/SPIRVAtomicOps.td

class SPV_AtomicUpdateOp<string mnemonic, list<OpTrait> traits = []> :
   let printer = [{ return ::printAtomicUpdateOp(getOperation(), p); }];
   let verifier = [{ return ::verifyAtomicUpdateOp(getOperation()); }];
 
   let arguments = (ins
     SPV_AnyPtr:$pointer,
     SPV_ScopeAttr:$memory_scope,
     SPV_MemorySemanticsAttr:$semantics
   );
+
   let results = (outs
     SPV_Integer:$result
   );
 }
 
 class SPV_AtomicUpdateWithValueOp<string mnemonic, list<OpTrait> traits = []> :
   SPV_Op<mnemonic, traits> {
   let parser = [{ return ::parseAtomicUpdateOp(parser, result, true); }];
   let printer = [{ return ::printAtomicUpdateOp(getOperation(), p); }];
   let verifier = [{ return ::verifyAtomicUpdateOp(getOperation()); }];
 
   let arguments = (ins
     SPV_AnyPtr:$pointer,
     SPV_ScopeAttr:$memory_scope,
     SPV_MemorySemanticsAttr:$semantics,
     SPV_Integer:$value
   );
+
   let results = (outs
     SPV_Integer:$result
   );
+
+  let builders = [
+    OpBuilder<
+      [{Builder *builder, OperationState &state, Value pointer,
+        ::mlir::spirv::Scope scope, ::mlir::spirv::MemorySemantics memory,
+        Value value}],
+      [{build(builder, state, value.getType(), pointer, scope, memory, value);}]
+    >
+  ];
 }
 
 // -----
 
 def SPV_AtomicAndOp : SPV_AtomicUpdateWithValueOp<"AtomicAnd", []> {
   let summary = [{
     Perform the following steps atomically with respect to any other atomic
     accesses within Scope to the same location:

mlir/include/mlir/Dialect/SPIRV/SPIRVControlFlowOps.td

let arguments = (ins
     SPV_SelectionControlAttr:$selection_control
   );
 
   let results = (outs);
 
   let regions = (region AnyRegion:$body);
 
   let extraClassDeclaration = [{
-    // Returns the selection header block.
+    /// Returns the selection header block.
     Block *getHeaderBlock();
 
-    // Returns the selection merge block.
+    /// Returns the selection merge block.
     Block *getMergeBlock();
 
-    // Adds a selection merge block containing one spv._merge op.
+    /// Adds a selection merge block containing one spv._merge op.
     void addMergeBlock();
+
+    /// Creates a spv.selection op for `if (<condition>) then { <thenBody> }`
+    /// with `builder`. `builder`'s insertion point will remain at after the
+    /// newly inserted spv.selection op afterwards.
+    static SelectionOp createIfThen(
+        Location loc, Value condition,
+        llvm::function_ref<void(OpBuilder *builder)> thenBody,
+        OpBuilder *builder);
   }];
 
   let hasOpcode = 0;
 
   let autogenSerialization = 0;
 
   let hasCanonicalizer = 1;
 }
 
 #endif // SPIRV_CONTROLFLOW_OPS

mlir/include/mlir/Dialect/SPIRV/SPIRVLowering.h

 /// `func` op to the SPIR-V dialect. These patterns do not handle shader
 /// interface/ABI; they convert function parameters to be of SPIR-V allowed
 /// types.
 void populateBuiltinFuncToSPIRVPatterns(MLIRContext *context,
                                         SPIRVTypeConverter &typeConverter,
                                         OwningRewritePatternList &patterns);
 
 namespace spirv {
+class AccessChainOp;
+
 class SPIRVConversionTarget : public ConversionTarget {
 public:
   /// Creates a SPIR-V conversion target for the given target environment.
   static std::unique_ptr<SPIRVConversionTarget> get(TargetEnvAttr targetEnv,
                                                     MLIRContext *context);
 
 private:
   SPIRVConversionTarget(TargetEnvAttr targetEnv, MLIRContext *context);

 
 /// Returns the value for the given `builtin` variable. This function gets or
 /// inserts the global variable associated for the builtin within the nearest
 /// enclosing op that has a symbol table. Returns null Value if such an
 /// enclosing op cannot be found.
 Value getBuiltinVariableValue(Operation *op, BuiltIn builtin,
                               OpBuilder &builder);
 
+/// Performs the index computation to get to the element at `indices` of the
+/// memory pointed to by `basePtr`, using the layout map of `baseType`.
+
+// TODO(ravishankarm) : This method assumes that the `baseType` is a MemRefType
+// with AffineMap that has static strides. Extend to handle dynamic strides.
+spirv::AccessChainOp getElementPtr(SPIRVTypeConverter &typeConverter,
+                                   MemRefType baseType, Value basePtr,
+                                   ArrayRef<Value> indices, Location loc,
+                                   OpBuilder &builder);
+
 /// Sets the InterfaceVarABIAttr and EntryPointABIAttr for a function and its
 /// arguments.
 LogicalResult setABIAttrs(FuncOp funcOp, EntryPointABIAttr entryPointInfo,
                           ArrayRef<InterfaceVarABIAttr> argABIInfo);
 } // namespace spirv
 } // namespace mlir
 
 #endif // MLIR_DIALECT_SPIRV_SPIRVLOWERING_H

mlir/include/mlir/Dialect/SPIRV/TargetAndABI.h

 /// Returns the default target environment: SPIR-V 1.0 with Shader capability
 /// and no extra extensions.
 TargetEnvAttr getDefaultTargetEnv(MLIRContext *context);
 
 /// Queries the target environment from the given `op` or returns the default
 /// target environment (SPIR-V 1.0 with Shader capability and no extra
 /// extensions) if not provided.
 TargetEnvAttr lookupTargetEnvOrDefault(Operation *op);
+
+/// Queries the local workgroup size from entry point ABI on the nearest
+/// function-like op containing the given `op`. Returns null attribute if not
+/// found.
+DenseIntElementsAttr lookupLocalWorkGroupSize(Operation *op);
+
 } // namespace spirv
 } // namespace mlir
 
 #endif // MLIR_DIALECT_SPIRV_TARGETANDABI_H

mlir/lib/Conversion/CMakeLists.txt

 add_subdirectory(AffineToStandard)
 add_subdirectory(GPUToCUDA)
 add_subdirectory(GPUToNVVM)
 add_subdirectory(GPUToROCDL)
 add_subdirectory(GPUToSPIRV)
 add_subdirectory(LinalgToLLVM)
+add_subdirectory(LinalgToSPIRV)
 add_subdirectory(LoopsToGPU)
 add_subdirectory(LoopToStandard)
 add_subdirectory(StandardToLLVM)
 add_subdirectory(StandardToSPIRV)
 add_subdirectory(VectorToLLVM)
 add_subdirectory(VectorToLoops)

mlir/lib/Conversion/LinalgToSPIRV/CMakeLists.txt

+add_llvm_library(MLIRLinalgToSPIRVTransforms
+  LinalgToSPIRV.cpp
+  LinalgToSPIRVPass.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/SPIRV
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/IR
+  )
+
+target_link_libraries(MLIRLinalgToSPIRVTransforms
+  MLIRIR
+  MLIRLinalgOps
+  MLIRPass
+  MLIRSPIRV
+  MLIRSupport
+  )

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRV.cpp

+//===- LinalgToSPIRV.cpp - Linalg to SPIR-V dialect conversion ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/LinalgToSPIRV/LinalgToSPIRV.h"
+#include "mlir/Dialect/Linalg/IR/LinalgOps.h"
+#include "mlir/Dialect/SPIRV/SPIRVDialect.h"
+#include "mlir/Dialect/SPIRV/SPIRVLowering.h"
+#include "mlir/Dialect/SPIRV/SPIRVOps.h"
+#include "mlir/Dialect/StandardOps/Ops.h"
+#include "mlir/Dialect/Utils/StructuredOpsUtils.h"
+#include "mlir/IR/AffineExpr.h"
+#include "mlir/IR/Matchers.h"
+
+using namespace mlir;
+
+//===----------------------------------------------------------------------===//
+// Utilities
+//===----------------------------------------------------------------------===//
+
+/// Returns true if all of the given `op`'s operands and values are of memref
+/// types.
+static bool areAllValuesMemref(Operation *op) {
+  auto isOfMemrefType = [](Value val) {
+    return val.getType().isa<MemRefType>();
+  };
+
+  return llvm::all_of(op->getOperands(), isOfMemrefType) &&
+         llvm::all_of(op->getResults(), isOfMemrefType);
+}
+
+/// Returns true if the given Linalg `iterators` is one reduction.
+static bool isLinalgSingleReductionIterator(ArrayAttr iterators) {
+  if (iterators.getValue().size() != 1)
+    return false;
+
+  auto iterator = (*iterators.begin()).dyn_cast<StringAttr>();
+  if (iterator.getValue() != getReductionIteratorTypeName())
+    return false;
+
+  return true;
+}
+
+/// Returns a `Value` containing the `dim`-th dimension's size of SPIR-V
+/// location invocation ID. This function will create necessary operations with
+/// `builder` at the proper region containing `op`.
+static Value getLocalInvocationDimSize(Operation *op, int dim, Location loc,
+                                       OpBuilder *builder) {
+  assert(dim >= 0 && dim < 3 && "local invocation only has three dimensions");
+  Value invocation = spirv::getBuiltinVariableValue(
+      op, spirv::BuiltIn::LocalInvocationId, *builder);
+  Type xType = invocation.getType().cast<ShapedType>().getElementType();
+  return builder->create<spirv::CompositeExtractOp>(
+      loc, xType, invocation, builder->getI32ArrayAttr({dim}));
+}
+
+namespace {
+enum class BinaryOpKind {
+  Unknown,
+  IAdd,
+};
+}
+
+/// Returns the binary op kind if the given linalg.generic op has the following
+/// body:
+///
+/// ```
+/// linalg.generic ... {
+///   ^bb(%a: <scalar-type>, %b: <scalar-type>):
+///     %0 = <binary-op> %a, %b: <scalar-type>
+///     linalg.yield %0: <scalar-type>
+/// }
+/// ```
+static BinaryOpKind getScalarBinaryOpKind(linalg::GenericOp op) {
+  auto &region = op.region();
+  if (region.empty() || !has_single_element(region.getBlocks()))
+    return BinaryOpKind::Unknown;
+
+  Block &block = region.front();
+  if (block.getNumArguments() != 2 ||
+      !block.getArgument(0).getType().isIntOrFloat() ||
+      !block.getArgument(1).getType().isIntOrFloat())
+    return BinaryOpKind::Unknown;
+
+  auto &ops = block.getOperations();
+  if (!has_single_element(block.without_terminator()))
+    return BinaryOpKind::Unknown;
+
+  using mlir::matchers::m_Val;
+  auto a = m_Val(block.getArgument(0));
+  auto b = m_Val(block.getArgument(1));
+
+  auto addPattern = m_Op<linalg::YieldOp>(m_Op<AddIOp>(a, b));
+  if (addPattern.match(&ops.back()))
+    return BinaryOpKind::IAdd;
+
+  return BinaryOpKind::Unknown;
+}
+
+//===----------------------------------------------------------------------===//
+// Reduction (single workgroup)
+//===----------------------------------------------------------------------===//
+
+namespace {
+
+/// A pattern to convert a linalg.generic op to SPIR-V ops under the condition
+/// that the linalg.generic op is performing reduction with a workload size that
+/// can fit in one workgroup.
+class SingleWorkgroupReduction final
+    : public SPIRVOpLowering<linalg::GenericOp> {
+public:
+  using SPIRVOpLowering<linalg::GenericOp>::SPIRVOpLowering;
+
+  PatternMatchResult
+  matchAndRewrite(linalg::GenericOp genericOp, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override;
+};
+
+} // namespace
+
+PatternMatchResult SingleWorkgroupReduction::matchAndRewrite(
+    linalg::GenericOp genericOp, ArrayRef<Value> operands,
+    ConversionPatternRewriter &rewriter) const {
+  Operation *op = genericOp.getOperation();
+
+  // Make sure the linalg.generic is working on memrefs.
+  if (!areAllValuesMemref(op))
+    return matchFailure();
+
+  // Make sure this is reudction with one input and one output.
+  if (genericOp.args_in().getZExtValue() != 1 ||
+      genericOp.args_out().getZExtValue() != 1)
+    return matchFailure();
+
+  auto originalInputType = op->getOperand(0).getType().cast<MemRefType>();
+  auto originalOutputType = op->getOperand(1).getType().cast<MemRefType>();
+
+  // Make sure the original input has one dimension.
+  if (!originalInputType.hasStaticShape() || originalInputType.getRank() != 1)
+    return matchFailure();
+  // Make sure the original output has one element.
+  if (!originalOutputType.hasStaticShape() ||
+      originalOutputType.getNumElements() != 1)
+    return matchFailure();
+
+  if (!isLinalgSingleReductionIterator(genericOp.iterator_types()))
+    return matchFailure();
+
+  if (genericOp.indexing_maps().getValue().size() != 2)
+    return matchFailure();
+
+  auto inputMap = genericOp.indexing_maps().getValue()[0].cast<AffineMapAttr>();
+  auto outputMap =
+      genericOp.indexing_maps().getValue()[1].cast<AffineMapAttr>();
+  // The indexing map for the input should be `(i) -> (i)`.
+  if (inputMap.getValue() !=
+      AffineMap::get(1, 0, {getAffineDimExpr(0, op->getContext())}))
+    return matchFailure();
+  // The indexing map for the input should be `(i) -> (0)`.
+  if (outputMap.getValue() !=
+      AffineMap::get(1, 0, {getAffineConstantExpr(0, op->getContext())}))
+    return matchFailure();
+
+  auto binaryOpKind = getScalarBinaryOpKind(genericOp);
+  if (binaryOpKind == BinaryOpKind::Unknown)
+    return matchFailure();
+
+  // Query the shader interface for local workgroup size to make sure the
+  // invocation configuration fits with the input memref's shape.
+  DenseIntElementsAttr localSize = spirv::lookupLocalWorkGroupSize(genericOp);
+  if (!localSize)
+    return matchFailure();
+
+  if ((*localSize.begin()).getSExtValue() != originalInputType.getDimSize(0))
+    return matchFailure();
+  if (llvm::any_of(llvm::drop_begin(localSize.getIntValues(), 1),
+                   [](const APInt &size) { return !size.isOneValue(); }))
+    return matchFailure();
+
+  // TODO(antiagainst): Query the target environment to make sure the current
+  // workload fits in a local workgroup.
+
+  Value convertedInput = operands[0], convertedOutput = operands[1];
+  Location loc = genericOp.getLoc();
+
+  // Get the invocation ID.
+  Value x = getLocalInvocationDimSize(genericOp, /*dim=*/0, loc, &rewriter);
+
+  // TODO(antiagainst): Load to Workgroup storage class first.
+
+  // Get the input element accessed by this invocation.
+  Value inputElementPtr = spirv::getElementPtr(
+      typeConverter, originalInputType, convertedInput, {x}, loc, rewriter);
+  Value inputElement = rewriter.create<spirv::LoadOp>(loc, inputElementPtr);
+
+  // Perform the group reduction operation.
+  Value groupOperation;
+#define CREATE_GROUP_NON_UNIFORM_BIN_OP(opKind, spvOp)                         \
+  case BinaryOpKind::opKind: {                                                 \
+    groupOperation = rewriter.create<spirv::spvOp>(                            \
+        loc, originalInputType.getElementType(), spirv::Scope::Subgroup,       \
+        spirv::GroupOperation::Reduce, inputElement,                           \
+        /*cluster_size=*/ArrayRef<Value>());                                   \
+  } break
+  switch (binaryOpKind) {
+    CREATE_GROUP_NON_UNIFORM_BIN_OP(IAdd, GroupNonUniformIAddOp);
+  case BinaryOpKind::Unknown:
+    llvm_unreachable("failed to reject match");
+  }
+#undef CREATE_GROUP_NON_UNIFORM_BIN_OP
+
+  // Get the output element accessed by this reduction.
+  Value zero = spirv::ConstantOp::getZero(
+      typeConverter.getIndexType(rewriter.getContext()), loc, &rewriter);
+  SmallVector<Value, 1> zeroIndices(originalOutputType.getRank(), zero);
+  Value outputElementPtr =
+      spirv::getElementPtr(typeConverter, originalOutputType, convertedOutput,
+                           zeroIndices, loc, rewriter);
+
+  // Write out the final reduction result. This should be only conducted by one
+  // invocation. We use spv.GroupNonUniformElect to find the invocation with the
+  // lowest ID.
+  //
+  // ```
+  // if (spv.GroupNonUniformElect) { output = ... }
+  // ```
+
+  Value condition = rewriter.create<spirv::GroupNonUniformElectOp>(
+      loc, spirv::Scope::Subgroup);
+
+  auto createAtomicOp = [&](OpBuilder *builder) {
+#define CREATE_ATOMIC_BIN_OP(opKind, spvOp)                                    \
+  case BinaryOpKind::opKind: {                                                 \
+    builder->create<spirv::spvOp>(loc, outputElementPtr, spirv::Scope::Device, \
+                                  spirv::MemorySemantics::AcquireRelease,      \
+                                  groupOperation);                             \
+  } break
+    switch (binaryOpKind) {
+      CREATE_ATOMIC_BIN_OP(IAdd, AtomicIAddOp);
+    case BinaryOpKind::Unknown:
+      llvm_unreachable("failed to reject match");
+    }
+#undef CREATE_ATOMIC_BIN_OP
+  };
+
+  spirv::SelectionOp::createIfThen(loc, condition, createAtomicOp, &rewriter);
+
+  rewriter.eraseOp(genericOp);
+  return matchSuccess();
+}
+
+//===----------------------------------------------------------------------===//
+// Pattern population
+//===----------------------------------------------------------------------===//
+
+void mlir::populateLinalgToSPIRVPatterns(MLIRContext *context,
+                                         SPIRVTypeConverter &typeConverter,
+                                         OwningRewritePatternList &patterns) {
+  patterns.insert<SingleWorkgroupReduction>(context, typeConverter);
+}

mlir/lib/Conversion/LinalgToSPIRV/LinalgToSPIRVPass.cpp

+//===- LinalgToSPIRVPass.cpp - Linalg to SPIR-V conversion pass -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/LinalgToSPIRV/LinalgToSPIRVPass.h"
+#include "mlir/Conversion/LinalgToSPIRV/LinalgToSPIRV.h"
+#include "mlir/Dialect/SPIRV/SPIRVDialect.h"
+#include "mlir/Dialect/SPIRV/SPIRVLowering.h"
+#include "mlir/Pass/Pass.h"
+
+using namespace mlir;
+
+namespace {
+/// A pass converting MLIR Linalg ops into SPIR-V ops.
+class LinalgToSPIRVPass : public ModulePass<LinalgToSPIRVPass> {
+  void runOnModule() override;
+};
+} // namespace
+
+void LinalgToSPIRVPass::runOnModule() {
+  MLIRContext *context = &getContext();
+  ModuleOp module = getModule();
+
+  SPIRVTypeConverter typeConverter;
+  OwningRewritePatternList patterns;
+  populateLinalgToSPIRVPatterns(context, typeConverter, patterns);
+  populateBuiltinFuncToSPIRVPatterns(context, typeConverter, patterns);
+
+  auto targetEnv = spirv::lookupTargetEnvOrDefault(module);
+  std::unique_ptr<ConversionTarget> target =
+      spirv::SPIRVConversionTarget::get(targetEnv, context);
+
+  // Allow builtin ops.
+  target->addLegalOp<ModuleOp, ModuleTerminatorOp>();
+  target->addDynamicallyLegalOp<FuncOp>(
+      [&](FuncOp op) { return typeConverter.isSignatureLegal(op.getType()); });
+
+  if (failed(applyFullConversion(module, *target, patterns)))
+    return signalPassFailure();
+}
+
+std::unique_ptr<OpPassBase<ModuleOp>> mlir::createLinalgToSPIRVPass() {
+  return std::make_unique<LinalgToSPIRVPass>();
+}
+
+static PassRegistration<LinalgToSPIRVPass>
+    pass("convert-linalg-to-spirv", "Convert Linalg ops to SPIR-V ops");

mlir/lib/Conversion/StandardToSPIRV/ConvertStandardToSPIRV.cpp

public:
   PatternMatchResult
   matchAndRewrite(StoreOp storeOp, ArrayRef<Value> operands,
                   ConversionPatternRewriter &rewriter) const override;
 };
 
 } // namespace
 
 //===----------------------------------------------------------------------===//
-// Utility functions for operation conversion
-//===----------------------------------------------------------------------===//
-
-/// Performs the index computation to get to the element pointed to by
-/// `indices` using the layout map of `baseType`.
-
-// TODO(ravishankarm) : This method assumes that the `origBaseType` is a
-// MemRefType with AffineMap that has static strides. Handle dynamic strides
-static spirv::AccessChainOp getElementPtr(OpBuilder &builder,
-                                          SPIRVTypeConverter &typeConverter,
-                                          Location loc, MemRefType origBaseType,
-                                          Value basePtr,
-                                          ArrayRef<Value> indices) {
-  // Get base and offset of the MemRefType and verify they are static.
-  int64_t offset;
-  SmallVector<int64_t, 4> strides;
-  if (failed(getStridesAndOffset(origBaseType, strides, offset)) ||
-      llvm::is_contained(strides, MemRefType::getDynamicStrideOrOffset())) {
-    return nullptr;
-  }
-
-  auto indexType = typeConverter.getIndexType(builder.getContext());
-
-  Value ptrLoc = nullptr;
-  assert(indices.size() == strides.size());
-  for (auto index : enumerate(indices)) {
-    Value strideVal = builder.create<spirv::ConstantOp>(
-        loc, indexType, IntegerAttr::get(indexType, strides[index.index()]));
-    Value update = builder.create<spirv::IMulOp>(loc, strideVal, index.value());
-    ptrLoc =
-        (ptrLoc ? builder.create<spirv::IAddOp>(loc, ptrLoc, update).getResult()
-                : update);
-  }
-  SmallVector<Value, 2> linearizedIndices;
-  // Add a '0' at the start to index into the struct.
-  linearizedIndices.push_back(builder.create<spirv::ConstantOp>(
-      loc, indexType, IntegerAttr::get(indexType, 0)));
-  linearizedIndices.push_back(ptrLoc);
-  return builder.create<spirv::AccessChainOp>(loc, basePtr, linearizedIndices);
-}
-
-//===----------------------------------------------------------------------===//
 // ConstantOp with composite type.
 //===----------------------------------------------------------------------===//
 
 PatternMatchResult ConstantCompositeOpConversion::matchAndRewrite(
     ConstantOp constCompositeOp, ArrayRef<Value> operands,
     ConversionPatternRewriter &rewriter) const {
   auto compositeType =
       constCompositeOp.getResult().getType().dyn_cast<RankedTensorType>();

 //===----------------------------------------------------------------------===//
 // LoadOp
 //===----------------------------------------------------------------------===//
 
 PatternMatchResult
 LoadOpConversion::matchAndRewrite(LoadOp loadOp, ArrayRef<Value> operands,
                                   ConversionPatternRewriter &rewriter) const {
   LoadOpOperandAdaptor loadOperands(operands);
-  auto loadPtr = getElementPtr(rewriter, typeConverter, loadOp.getLoc(),
-                               loadOp.memref().getType().cast<MemRefType>(),
-                               loadOperands.memref(), loadOperands.indices());
+  auto loadPtr = spirv::getElementPtr(
+      typeConverter, loadOp.memref().getType().cast<MemRefType>(),
+      loadOperands.memref(), loadOperands.indices(), loadOp.getLoc(), rewriter);
   rewriter.replaceOpWithNewOp<spirv::LoadOp>(loadOp, loadPtr);
   return matchSuccess();
 }
 
 //===----------------------------------------------------------------------===//
 // ReturnOp
 //===----------------------------------------------------------------------===//
 

 //===----------------------------------------------------------------------===//
 // StoreOp
 //===----------------------------------------------------------------------===//
 
 PatternMatchResult
 StoreOpConversion::matchAndRewrite(StoreOp storeOp, ArrayRef<Value> operands,
                                    ConversionPatternRewriter &rewriter) const {
   StoreOpOperandAdaptor storeOperands(operands);
-  auto storePtr =
-      getElementPtr(rewriter, typeConverter, storeOp.getLoc(),
-                    storeOp.memref().getType().cast<MemRefType>(),
-                    storeOperands.memref(), storeOperands.indices());
+  auto storePtr = spirv::getElementPtr(
+      typeConverter, storeOp.memref().getType().cast<MemRefType>(),
+      storeOperands.memref(), storeOperands.indices(), storeOp.getLoc(),
+      rewriter);
   rewriter.replaceOpWithNewOp<spirv::StoreOp>(storeOp, storePtr,
                                               storeOperands.value());
   return matchSuccess();
 }
 
 namespace {
 /// Import the Standard Ops to SPIR-V Patterns.
 #include "StandardToSPIRV.cpp.inc"

mlir/lib/Dialect/SPIRV/SPIRVLowering.cpp

Value mlir::spirv::getBuiltinVariableValue(Operation *op,
 
   spirv::GlobalVariableOp varOp = getOrInsertBuiltinVariable(
       *parent->getRegion(0).begin(), op->getLoc(), builtin, builder);
   Value ptr = builder.create<spirv::AddressOfOp>(op->getLoc(), varOp);
   return builder.create<spirv::LoadOp>(op->getLoc(), ptr);
 }
 
 //===----------------------------------------------------------------------===//
+// Index calculation
+//===----------------------------------------------------------------------===//
+
+spirv::AccessChainOp mlir::spirv::getElementPtr(
+    SPIRVTypeConverter &typeConverter, MemRefType baseType, Value basePtr,
+    ArrayRef<Value> indices, Location loc, OpBuilder &builder) {
+  // Get base and offset of the MemRefType and verify they are static.
+  int64_t offset;
+  SmallVector<int64_t, 4> strides;
+  if (failed(getStridesAndOffset(baseType, strides, offset)) ||
+      llvm::is_contained(strides, MemRefType::getDynamicStrideOrOffset())) {
+    return nullptr;
+  }
+
+  auto indexType = typeConverter.getIndexType(builder.getContext());
+
+  Value ptrLoc = nullptr;
+  assert(indices.size() == strides.size());
+  for (auto index : enumerate(indices)) {
+    Value strideVal = builder.create<spirv::ConstantOp>(
+        loc, indexType, IntegerAttr::get(indexType, strides[index.index()]));
+    Value update = builder.create<spirv::IMulOp>(loc, strideVal, index.value());
+    ptrLoc =
+        (ptrLoc ? builder.create<spirv::IAddOp>(loc, ptrLoc, update).getResult()
+                : update);
+  }
+  SmallVector<Value, 2> linearizedIndices;
+  // Add a '0' at the start to index into the struct.
+  linearizedIndices.push_back(builder.create<spirv::ConstantOp>(
+      loc, indexType, IntegerAttr::get(indexType, 0)));
+  linearizedIndices.push_back(ptrLoc);
+  return builder.create<spirv::AccessChainOp>(loc, basePtr, linearizedIndices);
+}
+
+//===----------------------------------------------------------------------===//
 // Set ABI attributes for lowering entry functions.
 //===----------------------------------------------------------------------===//
 
 LogicalResult
 mlir::spirv::setABIAttrs(FuncOp funcOp, spirv::EntryPointABIAttr entryPointInfo,
                          ArrayRef<spirv::InterfaceVarABIAttr> argABIInfo) {
   // Set the attributes for argument and the function.
   StringRef argABIAttrName = spirv::getInterfaceVarABIAttrName();

mlir/lib/Dialect/SPIRV/SPIRVOps.cpp

void spirv::SelectionOp::addMergeBlock() {
   auto *mergeBlock = new Block();
   body().push_back(mergeBlock);
   OpBuilder builder(mergeBlock);
 
   // Add a spv._merge op into the merge block.
   builder.create<spirv::MergeOp>(getLoc());
 }
 
+spirv::SelectionOp spirv::SelectionOp::createIfThen(
+    Location loc, Value condition,
+    llvm::function_ref<void(OpBuilder *builder)> thenBody, OpBuilder *builder) {
+  auto selectionControl = builder->getI32IntegerAttr(
+      static_cast<uint32_t>(spirv::SelectionControl::None));
+  auto selectionOp = builder->create<spirv::SelectionOp>(loc, selectionControl);
+
+  selectionOp.addMergeBlock();
+  Block *mergeBlock = selectionOp.getMergeBlock();
+  Block *thenBlock = nullptr;
+
+  // Build the "then" block.
+  {
+    OpBuilder::InsertionGuard guard(*builder);
+    thenBlock = builder->createBlock(mergeBlock);
+    thenBody(builder);
+    builder->create<spirv::BranchOp>(loc, mergeBlock);
+  }
+
+  // Build the header block.
+  {
+    OpBuilder::InsertionGuard guard(*builder);
+    builder->createBlock(thenBlock);
+    builder->create<spirv::BranchConditionalOp>(
+        loc, condition, thenBlock,
+        /*trueArguments=*/ArrayRef<Value>(), mergeBlock,
+        /*falseArguments=*/ArrayRef<Value>());
+  }
+
+  return selectionOp;
+}
+
 namespace {
 // Blocks from the given `spv.selection` operation must satisfy the following
 // layout:
 //
 //       +-----------------------------------------------+
 //       | header block                                  |
 //       | spv.BranchConditionalOp %cond, ^case0, ^case1 |
 //       +-----------------------------------------------+

mlir/lib/Dialect/SPIRV/TargetAndABI.cpp

 //===- TargetAndABI.cpp - SPIR-V target and ABI utilities -----------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/SPIRV/TargetAndABI.h"
 #include "mlir/Dialect/SPIRV/SPIRVTypes.h"
 #include "mlir/IR/Builders.h"
+#include "mlir/IR/FunctionSupport.h"
 #include "mlir/IR/Operation.h"
 
 using namespace mlir;
 
 namespace mlir {
 #include "mlir/Dialect/SPIRV/TargetAndABI.cpp.inc"
 }
 

 }
 
 spirv::TargetEnvAttr spirv::lookupTargetEnvOrDefault(Operation *op) {
   if (auto attr = op->getAttrOfType<spirv::TargetEnvAttr>(
           spirv::getTargetEnvAttrName()))
     return attr;
   return getDefaultTargetEnv(op->getContext());
 }
+
+DenseIntElementsAttr spirv::lookupLocalWorkGroupSize(Operation *op) {
+  while (op && !op->hasTrait<OpTrait::FunctionLike>())
+    op = op->getParentOp();
+  if (!op)
+    return {};
+
+  if (auto attr = op->getAttrOfType<spirv::EntryPointABIAttr>(
+          spirv::getEntryPointABIAttrName()))
+    return attr.local_size();
+
+  return {};
+}

mlir/test/Conversion/LinalgToSPIRV/linalg-to-spirv.mlir

+// RUN: mlir-opt -split-input-file -convert-linalg-to-spirv -canonicalize -verify-diagnostics %s -o - | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+// Single workgroup reduction
+//===----------------------------------------------------------------------===//
+
+#single_workgroup_reduction_trait = {
+  args_in = 1,
+  args_out = 1,
+  iterator_types = ["reduction"],
+  indexing_maps = [
+    affine_map<(i) -> (i)>,
+    affine_map<(i) -> (0)>
+  ]
+}
+
+module attributes {
+  spv.target_env = {
+    version = 3 : i32,
+    extensions = [],
+    capabilities = [1: i32, 63: i32] // Shader, GroupNonUniformArithmetic
+  }
+} {
+
+// CHECK:      spv.globalVariable
+// CHECK-SAME: built_in("LocalInvocationId")
+
+// CHECK:      func @single_workgroup_reduction
+// CHECK-SAME: (%[[INPUT:.+]]: !spv.ptr{{.+}}, %[[OUTPUT:.+]]: !spv.ptr{{.+}})
+
+// CHECK:        %[[ZERO:.+]] = spv.constant 0 : i32
+// CHECK:        %[[ID:.+]] = spv.Load "Input" %{{.+}} : vector<3xi32>
+// CHECK:        %[[X:.+]] = spv.CompositeExtract %[[ID]][0 : i32]
+
+// CHECK:        %[[INPTR:.+]] = spv.AccessChain %[[INPUT]][%[[ZERO]], %[[X]]]
+// CHECK:        %[[VAL:.+]] = spv.Load "StorageBuffer" %[[INPTR]] : i32
+// CHECK:        %[[ADD:.+]] = spv.GroupNonUniformIAdd "Subgroup" "Reduce" %[[VAL]] : i32
+
+// CHECK:        %[[OUTPTR:.+]] = spv.AccessChain %[[OUTPUT]][%[[ZERO]], %[[ZERO]]]
+// CHECK:        %[[ELECT:.+]] = spv.GroupNonUniformElect "Subgroup" : i1
+
+// CHECK:        spv.selection {
+// CHECK:          spv.BranchConditional %[[ELECT]], ^bb1, ^bb2
+// CHECK:        ^bb1:
+// CHECK:          spv.AtomicIAdd "Device" "AcquireRelease" %[[OUTPTR]], %[[ADD]]
+// CHECK:          spv.Branch ^bb2
+// CHECK:        ^bb2:
+// CHECK:          spv._merge
+// CHECK:        }
+// CHECK:        spv.Return
+
+func @single_workgroup_reduction(%input: memref<16xi32>, %output: memref<1xi32>) attributes {
+  spv.entry_point_abi = {local_size = dense<[16, 1, 1]>: vector<3xi32>}
+} {
+  linalg.generic #single_workgroup_reduction_trait %input, %output {
+    ^bb(%in: i32, %out: i32):
+      %sum = addi %in, %out : i32
+      linalg.yield %sum : i32
+  } : memref<16xi32>, memref<1xi32>
+  spv.Return
+}
+}
+
+// -----
+
+// Missing shader entry point ABI
+
+#single_workgroup_reduction_trait = {
+  args_in = 1,
+  args_out = 1,
+  iterator_types = ["reduction"],
+  indexing_maps = [
+    affine_map<(i) -> (i)>,
+    affine_map<(i) -> (0)>
+  ]
+}
+
+module attributes {
+  spv.target_env = {
+    version = 3 : i32,
+    extensions = [],
+    capabilities = [1: i32, 63: i32] // Shader, GroupNonUniformArithmetic
+  }
+} {
+func @single_workgroup_reduction(%input: memref<16xi32>, %output: memref<1xi32>) {
+  // expected-error @+1 {{failed to legalize operation 'linalg.generic'}}
+  linalg.generic #single_workgroup_reduction_trait %input, %output {
+    ^bb(%in: i32, %out: i32):
+      %sum = addi %in, %out : i32
+      linalg.yield %sum : i32
+  } : memref<16xi32>, memref<1xi32>
+  return
+}
+}
+
+// -----
+
+// Mismatch between shader entry point ABI and input memref shape
+
+#single_workgroup_reduction_trait = {
+  args_in = 1,
+  args_out = 1,
+  iterator_types = ["reduction"],
+  indexing_maps = [
+    affine_map<(i) -> (i)>,
+    affine_map<(i) -> (0)>
+  ]
+}
+
+module attributes {
+  spv.target_env = {
+    version = 3 : i32,
+    extensions = [],
+    capabilities = [1: i32, 63: i32] // Shader, GroupNonUniformArithmetic
+  }
+} {
+func @single_workgroup_reduction(%input: memref<16xi32>, %output: memref<1xi32>) attributes {
+  spv.entry_point_abi = {local_size = dense<[32, 1, 1]>: vector<3xi32>}
+} {
+  // expected-error @+1 {{failed to legalize operation 'linalg.generic'}}
+  linalg.generic #single_workgroup_reduction_trait %input, %output {
+    ^bb(%in: i32, %out: i32):
+      %sum = addi %in, %out : i32
+      linalg.yield %sum : i32
+  } : memref<16xi32>, memref<1xi32>
+  spv.Return
+}
+}
+
+// -----
+
+// Unsupported multi-dimension input memref
+
+#single_workgroup_reduction_trait = {
+  args_in = 1,
+  args_out = 1,
+  iterator_types = ["parallel", "reduction"],
+  indexing_maps = [
+    affine_map<(i, j) -> (i, j)>,
+    affine_map<(i, j) -> (i)>
+  ]
+}
+
+module attributes {
+  spv.target_env = {
+    version = 3 : i32,
+    extensions = [],
+    capabilities = [1: i32, 63: i32] // Shader, GroupNonUniformArithmetic
+  }
+} {
+func @single_workgroup_reduction(%input: memref<16x8xi32>, %output: memref<16xi32>) attributes {
+  spv.entry_point_abi = {local_size = dense<[16, 8, 1]>: vector<3xi32>}
+} {
+  // expected-error @+1 {{failed to legalize operation 'linalg.generic'}}
+  linalg.generic #single_workgroup_reduction_trait %input, %output {
+    ^bb(%in: i32, %out: i32):
+      %sum = addi %in, %out : i32
+      linalg.yield %sum : i32
+  } : memref<16x8xi32>, memref<16xi32>
+  spv.Return
+}
+}

mlir/tools/mlir-opt/CMakeLists.txt

set(LIBS
   MLIRNVVMIR
   MLIROptMain
   MLIRParser
   MLIRPass
   MLIRQuantizerTransforms
   MLIRQuantOps
   MLIRROCDLIR
   MLIRSPIRV
+  MLIRLinalgToSPIRVTransforms
   MLIRStandardToSPIRVTransforms
   MLIRSPIRVTestPasses
   MLIRSPIRVTransforms
   MLIRStandardOps
   MLIRStandardToLLVM
   MLIRTransforms
   MLIRTestDialect
   MLIRTestIR