[WIP] Replace std::forward & std::move by cast expressions during Sema

[cor3ntin]

Summary

This is a very early draft, mostly looking for direction/guidance at this point (notably whether the cost/benefit ratio of the overall idea is sensible).
I need to do a lot of refactors, some (a lot of) tests, etc.

std::move/std::forward are just casts, and are used a lot (230K instances in the VCPKG repository for std::move).

This code some issues:

• Worse debug experience
• More symbols
• Slower build times through template instantiations
• Worse Codegen

push    rbp
mov     rbp, rsp
sub     rsp, 16
mov     dword ptr [rbp - 4], edi
lea     rdi, [rbp - 4]
call    std::remove_reference<int&>::type&& std::move<int&>(int&)
mov     eax, dword ptr [rax]
add     rsp, 16
pop     rbp
ret

These functions could be marked force inline, but that does not address the build time concerns, and it doesn't solve O0 (some people seem to care?)
I also consider an LLVM pass to replace calls to std::move/std::forward, but that doesn't solve build time concerns.

In the end, I landed on the idea to replace these calls by a cast expression in the front end.

The general idea is to try to intercept qualified calls before overload resolution, and after overload resolutions for all calls.
The functions are still required to be declared, so lookup is not modified.
When a replacement fails, we try to create the CallExpr as usual so any diagnostic is emitted from there.
The intent is of course that this replacement cannot be observable.

• Is this an avenue worth pursuing ?
• Do we want an even more generic solution so that calls to eg, is_constant_evaluated can be replaced too?
• Do we want to expand to as_const, to_integer, to_underlying, bit_cast?

In terms of AST nodes, do we want to preserve the original replaced expression?
My current CXXStdLibraryCastExpr assumes the call is resolved to a CallExpr, but that's no longer the case since I also replace calls before template instantiations,
so I would probably do something to keep the arguments.

I don't know how much information we need to retain, and as is certainly very visible, I have no idea what I am doing.

Nevertheless, I figured the path might be in a good enough state to start a discussion.
I am also thinking that making std::forward(x) or std::fwd(x) (notice the lack of angle brackets) a magic expression in C++23 using the same mechanism might be worth pursuing.
(instead of new syntax as in http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0644r1.html)

[aaron.ballman]

Adding a few more reviewers for early feedback.

[lebedev.ri]

To be noted, this generally goes against the [current] spirit of the AST,
those were calls in the source code, but they won't be represented as such any longer,
which is likely not very expected by whatever tool that could be consuming the AST.
Though, i'm not sure how much that matters.

I would guess, having clang emit MLIR C++ dialect and not LLVM IR
might allow such transforms without degrading AST,
though i will be surprised if that happens [upstream] soon.

[rjmccall]

I think a pattern-match in IRGen is probably a much more reasonable alternative if the goal is primarily to avoid the code-generation overheads at -O0 / optimization costs at -O<n>. You'd still have template instantiation overheads; I don't know how significant those really are.

[rsmith]

In D99893#2669441, @rjmccall wrote:

I think a pattern-match in IRGen is probably a much more reasonable alternative if the goal is primarily to avoid the code-generation overheads at -O0 / optimization costs at -O<n>. You'd still have template instantiation overheads; I don't know how significant those really are.

I agree, this kind of special-casing really goes against the spirit of Clang's AST model, and such special-casing seems to fit better in IR generation than in the AST representation. The approach in this patch is going to require additional complexity in all AST traversals that want to treat function calls uniformly, and will break all existing AST matchers looking for std::move / std::forward calls.

Perhaps we could instead treat std::move and std::forward as builtin functions? We could then skip instantiating the definition given that we already have a builtin definition, and IR generation is already set up to perform custom code generation for built-in functions. That should minimize the amount of special-casing and non-uniform AST representation we need here, and closely matches how we model other library functions that have well-known semantic effects. Note that the "builtin function" approach won't allow us to avoid performing overload resolution, but I don't know how important that is for what you're trying to achieve here -- and in any case, skipping overload resolution seems fraught with problems given that there is another (algorithm) overload of std::move and that both std::move and (especially) std::forward will reject some calls in overload resolution.

Treating a namespace-std function template as a builtin isn't entirely novel; we already do this for MSVC's std::_GetExceptionInfo (though we don't actually handle that properly: we're missing the "namespace std" check, at least). Treating the builtin definition as overriding an inline library definition might be novel, though that doesn't seem like a huge problem.

[cor3ntin]

Treating a namespace-std function template as a builtin isn't entirely novel; we already do this for MSVC's std::_GetExceptionInfo (though we don't actually handle that properly: we're missing the "namespace std" check, at least). Treating the builtin definition as overriding an inline library definition might be novel, though that doesn't seem like a huge problem.

This was my initial approach, I found some issues

• I struggled getting a built in forward to return a reference
• I had no clue how to inject them in namespace std (knowing about std::_GetExceptionInfo will certainly help, thanks!
• I really didn't want to modify standard headers and wasn't sure how to override the declaration.

But overall, I agree with all of you, it's a better design if we can make it work. Back to the drawing board, and thanks for the feedbacks :)

[rjmccall]

Our builtin logic is capable of recognizing user declarations as builtins without any sort of explicit annotation in source. That's how we recognize C library builtins, for example. As Richard points out, that logic isn't really set up to do the right thing for namespaced declarations (or template ones, for that matter), but that seems fixable.

The namespace issue might be annoying because of the inline namespaces that some standard libraries used for ABI versioning.

[cor3ntin]

In D99893#2672258, @rjmccall wrote:

Our builtin logic is capable of recognizing user declarations as builtins without any sort of explicit annotation in source. That's how we recognize C library builtins, for example. As Richard points out, that logic isn't really set up to do the right thing for namespaced declarations (or template ones, for that matter), but that seems fixable.

The namespace issue might be annoying because of the inline namespaces that some standard libraries used for ABI versioning.

I managed to make a somewhat generic mechanism to mark functions declared in the namespace std ( and I do think we want there to be a declaration in the header always) as built-ins.
Currently it's using the number of parameters as a filter - which I think is reasonable ( the issue as Richard pointed out is that std::move is overloaded with the algorithm version).

I'm now struggling to handle code generation and constant evaluation, but hopefully I will find a way to get there.

This would solve some of the issues (perfect code-gen / no body instantiation), but the declarations are still instantiated of course.
And I have a few folks asking me whether I could get rid of it. Unfortunately, I think the goals of preserving the AST as is today and getting rid of these template instantiations are contradictory.

[• Quuxplusone]

No matter how it works internally, I think that (nearer the end of the process) someone should insist that you add some Clang tests to verify e.g.

Widget&& a = static_cast<Widget&&>(Widget());  // is lifetime-extended, but
Widget&& b = std::move(Widget());              // is not

https://godbolt.org/z/9Ka7hcode
and whatever other corner-cases people can think of. That is, these may turn into not-a-function-calls internally, but they'd better continue to behave observably as if they were function-calls.

[rsmith]

In D99893#2672563, @cor3ntin wrote:

This would solve some of the issues (perfect code-gen / no body instantiation), but the declarations are still instantiated of course.
And I have a few folks asking me whether I could get rid of it. Unfortunately, I think the goals of preserving the AST as is today and getting rid of these template instantiations are contradictory.

Do you know how much of the cost here is instantiating the function declaration, and how much is instantiating std::remove_reference? I imagine we could make the latter cheaper by adding a __remove_reference builtin and changing libc++'s forward and move to use it if that's worthwhile.

[cor3ntin]

Richard committed a better approach so this is no longer needed

clang/include/clang/AST/ExprCXX.h

public:
   SourceLocation getBeginLoc() const LLVM_READONLY;
   SourceLocation getEndLoc() const LLVM_READONLY;
 
   static bool classof(const Stmt *T) {
     return T->getStmtClass() == CXXFunctionalCastExprClass;
   }
 };
 
+class CXXStandardLibraryCastExpr final
+    : public CastExpr,
+      private llvm::TrailingObjects<CXXStandardLibraryCastExpr,
+                                    CXXBaseSpecifier *, FPOptionsOverride> {
+  friend class ASTStmtReader;
+
+  CXXStandardLibraryCastExpr(QualType exprTy, ExprValueKind VK, CastKind kind,
+                             Expr *op, Expr *OriginalExpression,
+                             unsigned PathSize, const CXXCastPath *BasePath,
+                             FPOptionsOverride FPO)
+      : CastExpr(CXXStandardLibraryCastExprClass, exprTy, VK, kind, op,
+                 PathSize, FPO.requiresTrailingStorage()),
+        OriginalExpression(OriginalExpression) {}
+
+  explicit CXXStandardLibraryCastExpr(EmptyShell Shell, unsigned PathSize,
+                                      bool HasFPFeatures)
+      : CastExpr(CXXStandardLibraryCastExprClass, Shell, PathSize,
+                 HasFPFeatures) {}
+
+  unsigned numTrailingObjects(OverloadToken<CXXBaseSpecifier *>) const {
+    return path_size();
+  }
+
+  Expr *OriginalExpression;
+
+public:
+  friend TrailingObjects;
+  friend class CastExpr;
+
+  static CXXStandardLibraryCastExpr *
+  Create(const ASTContext &Context, QualType T, ExprValueKind VK, CastKind Kind,
+         Expr *Op, Expr *OriginalExpression, const CXXCastPath *BasePath,
+         FPOptionsOverride FPO);
+
+  static CXXStandardLibraryCastExpr *
+  CreateEmpty(const ASTContext &Context, unsigned PathSize, bool HasFPFeatures);
+
+  /* Expr* getOriginalExpression() const {
+       return OriginalExpression;
+   }*/
+
+  SourceLocation getBeginLoc() const LLVM_READONLY;
+  SourceLocation getEndLoc() const LLVM_READONLY;
+
+  static bool classof(const Stmt *T) {
+    return T->getStmtClass() == CXXStandardLibraryCastExprClass;
+  }
+};
+
 /// Represents a C++ functional cast expression that builds a
 /// temporary object.
 ///
 /// This expression type represents a C++ "functional" cast
 /// (C++[expr.type.conv]) with N != 1 arguments that invokes a
 /// constructor to build a temporary object. With N == 1 arguments the
 /// functional cast expression will be represented by CXXFunctionalCastExpr.
 /// Example:

clang/include/clang/AST/RecursiveASTVisitor.h

 })
 
 DEF_TRAVERSE_STMT(
     ImplicitCastExpr,
     {// We don't traverse the cast type, as it's not written in the
      // source code.
     })
 
+DEF_TRAVERSE_STMT(CXXStandardLibraryCastExpr, {})
+
 DEF_TRAVERSE_STMT(CStyleCastExpr, {
   TRY_TO(TraverseTypeLoc(S->getTypeInfoAsWritten()->getTypeLoc()));
 })
 
 DEF_TRAVERSE_STMT(CXXFunctionalCastExpr, {
   TRY_TO(TraverseTypeLoc(S->getTypeInfoAsWritten()->getTypeLoc()));
 })
 

clang/include/clang/AST/TextNodeDumper.h

#include "clang/AST/AttrTextNodeDump.inc"
   void VisitBinaryOperator(const BinaryOperator *Node);
   void VisitCompoundAssignOperator(const CompoundAssignOperator *Node);
   void VisitAddrLabelExpr(const AddrLabelExpr *Node);
   void VisitCXXNamedCastExpr(const CXXNamedCastExpr *Node);
   void VisitCXXBoolLiteralExpr(const CXXBoolLiteralExpr *Node);
   void VisitCXXThisExpr(const CXXThisExpr *Node);
   void VisitCXXFunctionalCastExpr(const CXXFunctionalCastExpr *Node);
   void VisitCXXStaticCastExpr(const CXXStaticCastExpr *Node);
+  void VisitCXXStandardLibraryCast(const CXXStandardLibraryCastExpr *Node);
   void VisitCXXUnresolvedConstructExpr(const CXXUnresolvedConstructExpr *Node);
   void VisitCXXConstructExpr(const CXXConstructExpr *Node);
   void VisitCXXBindTemporaryExpr(const CXXBindTemporaryExpr *Node);
   void VisitCXXNewExpr(const CXXNewExpr *Node);
   void VisitCXXDeleteExpr(const CXXDeleteExpr *Node);
   void VisitTypeTraitExpr(const TypeTraitExpr *Node);
   void VisitArrayTypeTraitExpr(const ArrayTypeTraitExpr *Node);
   void VisitExpressionTraitExpr(const ExpressionTraitExpr *Node);

clang/include/clang/ASTMatchers/ASTMatchers.h

 ///
 /// Example: Matches (int) 2.2f in
 /// \code
 ///   int i = (int) 2.2f;
 /// \endcode
 extern const internal::VariadicDynCastAllOfMatcher<Stmt, CStyleCastExpr>
     cStyleCastExpr;
 
+extern const internal::VariadicDynCastAllOfMatcher<Stmt,
+                                                   CXXStandardLibraryCastExpr>
+    cxxStandardLibraryCastExpr;
+
 /// Matches explicit cast expressions.
 ///
 /// Matches any cast expression written in user code, whether it be a
 /// C-style cast, a functional-style cast, or a keyword cast.
 ///
 /// Does not match implicit conversions.
 ///
 /// Note: the name "explicitCast" is chosen to match Clang's terminology, as

clang/include/clang/Basic/StmtNodes.td

 def CXXRewrittenBinaryOperator : StmtNode<Expr>;
 def CXXNamedCastExpr : StmtNode<ExplicitCastExpr, 1>;
 def CXXStaticCastExpr : StmtNode<CXXNamedCastExpr>;
 def CXXDynamicCastExpr : StmtNode<CXXNamedCastExpr>;
 def CXXReinterpretCastExpr : StmtNode<CXXNamedCastExpr>;
 def CXXConstCastExpr : StmtNode<CXXNamedCastExpr>;
 def CXXAddrspaceCastExpr : StmtNode<CXXNamedCastExpr>;
 def CXXFunctionalCastExpr : StmtNode<ExplicitCastExpr>;
+def CXXStandardLibraryCastExpr : StmtNode<CastExpr>;
 def CXXTypeidExpr : StmtNode<Expr>;
 def UserDefinedLiteral : StmtNode<CallExpr>;
 def CXXBoolLiteralExpr : StmtNode<Expr>;
 def CXXNullPtrLiteralExpr : StmtNode<Expr>;
 def CXXThisExpr : StmtNode<Expr>;
 def CXXThrowExpr : StmtNode<Expr>;
 def CXXDefaultArgExpr : StmtNode<Expr>;
 def CXXDefaultInitExpr : StmtNode<Expr>;

clang/include/clang/Serialization/ASTBitCodes.h

enum StmtCode {
   EXPR_CXX_INHERITED_CTOR_INIT,
 
   /// A CXXTemporaryObjectExpr record.
   EXPR_CXX_TEMPORARY_OBJECT,
 
   /// A CXXStaticCastExpr record.
   EXPR_CXX_STATIC_CAST,
 
+  /// A CXXStandardLibraryCastExpr record.
+  EXPR_CXX_STL_CAST,
+
   /// A CXXDynamicCastExpr record.
   EXPR_CXX_DYNAMIC_CAST,
 
   /// A CXXReinterpretCastExpr record.
   EXPR_CXX_REINTERPRET_CAST,
 
   /// A CXXConstCastExpr record.
   EXPR_CXX_CONST_CAST,

clang/lib/AST/Expr.cpp

if (DCE->getTypeAsWritten()->isReferenceType() &&
       return true;
     }
     LLVM_FALLTHROUGH;
   case ImplicitCastExprClass:
   case CStyleCastExprClass:
   case CXXStaticCastExprClass:
   case CXXReinterpretCastExprClass:
   case CXXConstCastExprClass:
+  case CXXStandardLibraryCastExprClass:
   case CXXAddrspaceCastExprClass:
   case CXXFunctionalCastExprClass:
   case BuiltinBitCastExprClass: {
     // While volatile reads are side-effecting in both C and C++, we treat them
     // as having possible (not definite) side-effects. This allows idiomatic
     // code to behave without warning, such as sizeof(*v) for a volatile-
     // qualified pointer.
     if (!IncludePossibleEffects)

clang/lib/AST/ExprCXX.cpp

 SourceLocation CXXFunctionalCastExpr::getBeginLoc() const {
   return getTypeInfoAsWritten()->getTypeLoc().getBeginLoc();
 }
 
 SourceLocation CXXFunctionalCastExpr::getEndLoc() const {
   return RParenLoc.isValid() ? RParenLoc : getSubExpr()->getEndLoc();
 }
 
+CXXStandardLibraryCastExpr *CXXStandardLibraryCastExpr::Create(
+    const ASTContext &C, QualType T, ExprValueKind VK, CastKind Kind, Expr *Op,
+    Expr *OriginalExpression, const CXXCastPath *BasePath,
+    FPOptionsOverride FPO) {
+  unsigned PathSize = (BasePath ? BasePath->size() : 0);
+  void *Buffer =
+      C.Allocate(totalSizeToAlloc<CXXBaseSpecifier *, FPOptionsOverride>(
+          PathSize, FPO.requiresTrailingStorage()));
+  auto *E = new (Buffer) CXXStandardLibraryCastExpr(
+      T, VK, Kind, Op, OriginalExpression, PathSize, BasePath, FPO);
+  if (PathSize)
+    std::uninitialized_copy_n(BasePath->data(), BasePath->size(),
+                              E->getTrailingObjects<CXXBaseSpecifier *>());
+  return E;
+}
+
+CXXStandardLibraryCastExpr *
+CXXStandardLibraryCastExpr::CreateEmpty(const ASTContext &C, unsigned PathSize,
+                                        bool HasFPFeatures) {
+  void *Buffer =
+      C.Allocate(totalSizeToAlloc<CXXBaseSpecifier *, FPOptionsOverride>(
+          PathSize, HasFPFeatures));
+  return new (Buffer)
+      CXXStandardLibraryCastExpr(EmptyShell(), PathSize, HasFPFeatures);
+}
+
+SourceLocation CXXStandardLibraryCastExpr::getBeginLoc() const {
+  return OriginalExpression->getBeginLoc();
+}
+
+SourceLocation CXXStandardLibraryCastExpr::getEndLoc() const {
+  return OriginalExpression->getEndLoc();
+}
+
 UserDefinedLiteral::UserDefinedLiteral(Expr *Fn, ArrayRef<Expr *> Args,
                                        QualType Ty, ExprValueKind VK,
                                        SourceLocation LitEndLoc,
                                        SourceLocation SuffixLoc,
                                        FPOptionsOverride FPFeatures)
     : CallExpr(UserDefinedLiteralClass, Fn, /*PreArgs=*/{}, Args, Ty, VK,
                LitEndLoc, FPFeatures, /*MinNumArgs=*/0, NotADL),
       UDSuffixLoc(SuffixLoc) {}

clang/lib/AST/ExprClassification.cpp

#include "clang/AST/StmtNodes.inc"
   case Expr::CXXConstCastExprClass:
   case Expr::CXXAddrspaceCastExprClass:
   case Expr::ObjCBridgedCastExprClass:
   case Expr::BuiltinBitCastExprClass:
     // Only in C++ can casts be interesting at all.
     if (!Lang.CPlusPlus) return Cl::CL_PRValue;
     return ClassifyUnnamed(Ctx, cast<ExplicitCastExpr>(E)->getTypeAsWritten());
 
+  case Expr::CXXStandardLibraryCastExprClass:
+    return ClassifyExprValueKind(Lang, E, E->getValueKind());
+
   case Expr::CXXUnresolvedConstructExprClass:
     return ClassifyUnnamed(Ctx,
                       cast<CXXUnresolvedConstructExpr>(E)->getTypeAsWritten());
 
   case Expr::BinaryConditionalOperatorClass: {
     if (!Lang.CPlusPlus) return Cl::CL_PRValue;
     const auto *co = cast<BinaryConditionalOperator>(E);
     return ClassifyConditional(Ctx, co->getTrueExpr(), co->getFalseExpr());

clang/lib/AST/ExprConstant.cpp

case Expr::BinaryOperatorClass: {
     }
     llvm_unreachable("invalid binary operator kind");
   }
   case Expr::ImplicitCastExprClass:
   case Expr::CStyleCastExprClass:
   case Expr::CXXFunctionalCastExprClass:
   case Expr::CXXStaticCastExprClass:
   case Expr::CXXReinterpretCastExprClass:
+  case Expr::CXXStandardLibraryCastExprClass:
   case Expr::CXXConstCastExprClass:
   case Expr::ObjCBridgedCastExprClass: {
     const Expr *SubExpr = cast<CastExpr>(E)->getSubExpr();
     if (isa<ExplicitCastExpr>(E)) {
       if (const FloatingLiteral *FL
             = dyn_cast<FloatingLiteral>(SubExpr->IgnoreParenImpCasts())) {
         unsigned DestWidth = Ctx.getIntWidth(E->getType());
         bool DestSigned = E->getType()->isSignedIntegerOrEnumerationType();

clang/lib/AST/ItaniumMangle.cpp

if (auto *IL = dyn_cast<InitListExpr>(Sub)) {
       mangleInitListElements(IL);
       Out << "E";
     } else {
       mangleCastExpression(E, "cv");
     }
     break;
   }
 
+  case Expr::CXXStandardLibraryCastExprClass:
   case Expr::CXXStaticCastExprClass:
     NotPrimaryExpr();
     mangleCastExpression(E, "sc");
     break;
   case Expr::CXXDynamicCastExprClass:
     NotPrimaryExpr();
     mangleCastExpression(E, "dc");
     break;

clang/lib/AST/StmtPrinter.cpp

void StmtPrinter::VisitCompoundLiteralExpr(CompoundLiteralExpr *Node) {
   PrintExpr(Node->getInitializer());
 }
 
 void StmtPrinter::VisitImplicitCastExpr(ImplicitCastExpr *Node) {
   // No need to print anything, simply forward to the subexpression.
   PrintExpr(Node->getSubExpr());
 }
 
+void StmtPrinter::VisitCXXStandardLibraryCastExpr(
+    CXXStandardLibraryCastExpr *Node) {
+  // No need to print anything, simply forward to the subexpression.
+  PrintExpr(Node->getSubExpr());
+}
+
 void StmtPrinter::VisitBinaryOperator(BinaryOperator *Node) {
   PrintExpr(Node->getLHS());
   OS << " " << BinaryOperator::getOpcodeStr(Node->getOpcode()) << " ";
   PrintExpr(Node->getRHS());
 }
 
 void StmtPrinter::VisitCompoundAssignOperator(CompoundAssignOperator *Node) {
   PrintExpr(Node->getLHS());

clang/lib/AST/StmtProfile.cpp

void StmtProfiler::VisitCastExpr(const CastExpr *S) {
   VisitExpr(S);
 }
 
 void StmtProfiler::VisitImplicitCastExpr(const ImplicitCastExpr *S) {
   VisitCastExpr(S);
   ID.AddInteger(S->getValueKind());
 }
 
+void StmtProfiler::VisitCXXStandardLibraryCastExpr(
+    const CXXStandardLibraryCastExpr *S) {
+  VisitCastExpr(S);
+}
+
 void StmtProfiler::VisitExplicitCastExpr(const ExplicitCastExpr *S) {
   VisitCastExpr(S);
   VisitType(S->getTypeAsWritten());
 }
 
 void StmtProfiler::VisitCStyleCastExpr(const CStyleCastExpr *S) {
   VisitExplicitCastExpr(S);
 }

clang/lib/AST/TextNodeDumper.cpp

 }
 
 void TextNodeDumper::VisitCXXStaticCastExpr(const CXXStaticCastExpr *Node) {
   VisitCXXNamedCastExpr(Node);
   if (Node->hasStoredFPFeatures())
     printFPOptions(Node->getFPFeatures());
 }
 
+void TextNodeDumper::VisitCXXStandardLibraryCast(
+    const CXXStandardLibraryCastExpr *Node) {
+  VisitCastExpr(Node);
+  OS << "<" << Node->getType().getAsString() << ">"
+     << "<" << Node->getCastKindName();
+  dumpBasePath(OS, Node);
+  OS << ">";
+
+  // Visit(Node->getOriginalExpression());
+}
+
 void TextNodeDumper::VisitCXXUnresolvedConstructExpr(
     const CXXUnresolvedConstructExpr *Node) {
   dumpType(Node->getTypeAsWritten());
   if (Node->isListInitialization())
     OS << " list";
 }
 
 void TextNodeDumper::VisitCXXConstructExpr(const CXXConstructExpr *Node) {

clang/lib/ASTMatchers/ASTMatchersInternal.cpp

 const internal::VariadicDynCastAllOfMatcher<Stmt, CXXConstCastExpr>
     cxxConstCastExpr;
 const internal::VariadicDynCastAllOfMatcher<Stmt, CStyleCastExpr>
     cStyleCastExpr;
 const internal::VariadicDynCastAllOfMatcher<Stmt, ExplicitCastExpr>
     explicitCastExpr;
 const internal::VariadicDynCastAllOfMatcher<Stmt, ImplicitCastExpr>
     implicitCastExpr;
+const internal::VariadicDynCastAllOfMatcher<Stmt, CXXStandardLibraryCastExpr>
+    cssStandardLibraryCastExpr;
 const internal::VariadicDynCastAllOfMatcher<Stmt, CastExpr> castExpr;
 const internal::VariadicDynCastAllOfMatcher<Stmt, CXXFunctionalCastExpr>
     cxxFunctionalCastExpr;
 const internal::VariadicDynCastAllOfMatcher<Stmt, CXXTemporaryObjectExpr>
     cxxTemporaryObjectExpr;
 const internal::VariadicDynCastAllOfMatcher<Stmt, PredefinedExpr>
     predefinedExpr;
 const internal::VariadicDynCastAllOfMatcher<Stmt, DesignatedInitExpr>

clang/lib/ASTMatchers/Dynamic/Registry.cpp

RegistryMaps::RegistryMaps() {
   REGISTER_MATCHER(cxxNewExpr);
   REGISTER_MATCHER(cxxNoexceptExpr);
   REGISTER_MATCHER(cxxNullPtrLiteralExpr);
   REGISTER_MATCHER(cxxOperatorCallExpr);
   REGISTER_MATCHER(cxxRecordDecl);
   REGISTER_MATCHER(cxxReinterpretCastExpr);
   REGISTER_MATCHER(cxxRewrittenBinaryOperator);
   REGISTER_MATCHER(cxxStaticCastExpr);
+  REGISTER_MATCHER(cxxStandardLibraryCastExpr);
   REGISTER_MATCHER(cxxStdInitializerListExpr);
   REGISTER_MATCHER(cxxTemporaryObjectExpr);
   REGISTER_MATCHER(cxxThisExpr);
   REGISTER_MATCHER(cxxThrowExpr);
   REGISTER_MATCHER(cxxTryStmt);
   REGISTER_MATCHER(cxxUnresolvedConstructExpr);
   REGISTER_MATCHER(decayedType);
   REGISTER_MATCHER(decl);

clang/lib/CodeGen/CGExpr.cpp

LValue CodeGenFunction::EmitLValue(const Expr *E) {
   case Expr::OpaqueValueExprClass:
     return EmitOpaqueValueLValue(cast<OpaqueValueExpr>(E));
   case Expr::SubstNonTypeTemplateParmExprClass:
     return EmitLValue(cast<SubstNonTypeTemplateParmExpr>(E)->getReplacement());
   case Expr::ImplicitCastExprClass:
   case Expr::CStyleCastExprClass:
   case Expr::CXXFunctionalCastExprClass:
   case Expr::CXXStaticCastExprClass:
+  case Expr::CXXStandardLibraryCastExprClass:
   case Expr::CXXDynamicCastExprClass:
   case Expr::CXXReinterpretCastExprClass:
   case Expr::CXXConstCastExprClass:
   case Expr::CXXAddrspaceCastExprClass:
   case Expr::ObjCBridgedCastExprClass:
     return EmitCastLValue(cast<CastExpr>(E));
 
   case Expr::MaterializeTemporaryExprClass:

clang/lib/Sema/SemaExceptionSpec.cpp

CanThrowResult Sema::canThrow(const Stmt *S) {
   case Expr::OMPArraySectionExprClass:
   case Expr::OMPArrayShapingExprClass:
   case Expr::OMPIteratorExprClass:
   case Expr::BinaryOperatorClass:
   case Expr::DependentCoawaitExprClass:
   case Expr::CompoundAssignOperatorClass:
   case Expr::CStyleCastExprClass:
   case Expr::CXXStaticCastExprClass:
+  case Expr::CXXStandardLibraryCastExprClass:
   case Expr::CXXFunctionalCastExprClass:
   case Expr::ImplicitCastExprClass:
   case Expr::MaterializeTemporaryExprClass:
   case Expr::UnaryOperatorClass: {
     // FIXME: Properly determine whether a variably-modified type can throw.
     if (auto *CE = dyn_cast<CastExpr>(S))
       if (CE->getType()->isVariablyModifiedType())
         return CT_Can;

clang/lib/Sema/SemaExpr.cpp

if (getLangOpts().CPlusPlus) {
     }
 
     if (Fn->getType() == Context.BoundMemberTy) {
       return BuildCallToMemberFunction(Scope, Fn, LParenLoc, ArgExprs,
                                        RParenLoc, AllowRecovery);
     }
   }
 
+  auto RewriteQualifiedCallsToStandardCastFunction =
+      [&](Expr *Fn) -> ExprResult {
+    auto *ULE = dyn_cast_or_null<OverloadExpr>(Fn->IgnoreParens());
+    if (ArgExprs.size() != 1 || Expr::hasAnyTypeDependentArguments(ArgExprs) ||
+        !ULE || !ULE->getQualifier() ||
+        !ULE->getQualifier()->getAsNamespace()->isStdNamespace())
+      return {};
+
+    ULE->dumpColor();
+    llvm::outs() << "Template Args: " << ULE->getNumTemplateArgs() << "\n";
+
+    IdentifierInfo *ID = ULE->getName().isIdentifier()
+                             ? ULE->getName().getAsIdentifierInfo()
+                             : nullptr;
+    if (!ID)
+      return {};
+
+    Expr *Arg = ArgExprs.front();
+
+    if (ULE->getNumTemplateArgs() == 0 && ID->isStr("move")) {
+      llvm::outs()
+          << "Replacing a call to std::move before overload resolution\n";
+      return CXXStandardLibraryCastExpr::Create(
+          Context, Arg->getType().getNonReferenceType(), VK_XValue, CK_NoOp,
+          Arg, Fn, nullptr, {});
+    }
+
+    if (ULE->getNumTemplateArgs() == 1 && ID->isStr("forward")) {
+      auto TemplateArg = ULE->getTemplateArgs()->getArgument();
+      if (TemplateArg.isDependent() ||
+          TemplateArg.getKind() != TemplateArgument::ArgKind::Type)
+        return {};
+
+      const auto TemplateType = TemplateArg.getAsType().getNonReferenceType();
+      const auto ExpressionType = Arg->getType().getNonReferenceType();
+
+      if (!Context.hasSameType(TemplateType, ExpressionType))
+        return {};
+
+      llvm::outs()
+          << "Replacing a call to std::forward before overload resolution\n";
+      return CXXStandardLibraryCastExpr::Create(Context, TemplateType,
+                                                Arg->getValueKind(), CK_NoOp,
+                                                Arg, Fn, nullptr, {});
+    }
+
+    return {};
+  };
+
+  ExprResult E = RewriteQualifiedCallsToStandardCastFunction(Fn);
+  if (E.isUsable()) {
+    return E;
+  }
+
   // Check for overloaded calls.  This can happen even in C due to extensions.
   if (Fn->getType() == Context.OverloadTy) {
     OverloadExpr::FindResult find = OverloadExpr::find(Fn);
 
     // We aren't supposed to apply this logic if there's an '&' involved.
     if (!find.HasFormOfMemberPointer) {
       if (Expr::hasAnyTypeDependentArguments(ArgExprs))
         return CallExpr::Create(Context, Fn, ArgExprs, Context.DependentTy,
ExprResult Sema::BuildResolvedCallExpr(Expr *Fn, NamedDecl *NDecl,
   } else if (NDecl) {
     if (CheckPointerCall(NDecl, TheCall, Proto))
       return ExprError();
   } else {
     if (CheckOtherCall(TheCall, Proto))
       return ExprError();
   }
 
-  return CheckForImmediateInvocation(MaybeBindToTemporary(TheCall), FDecl);
+  auto TransformStdMoveAndForwardIntoCast = [&](CallExpr *Call) -> Expr * {
+    const bool isStd = NDecl && NDecl->isInStdNamespace();
+    if (!getLangOpts().CPlusPlus || !isStd || !NDecl)
+      return Call;
+
+    if (Call->getNumArgs() != 1 && !(NDecl->getName().compare("move") == 0 ||
+                                     NDecl->getName().compare("forward") == 0))
+      return Call;
+
+    Expr *A = TheCall->getArg(0);
+    if (A->getType().getCanonicalType().getNonReferenceType() !=
+        TheCall->getType().getCanonicalType().getNonReferenceType())
+      return Call;
+
+    if (A->isRValue() && Context.hasSameType(A->getType(), TheCall->getType()))
+      return A;
+
+    return CXXStandardLibraryCastExpr::Create(
+        Context, TheCall->getType(), TheCall->getValueKind(), CastKind::CK_NoOp,
+        A, TheCall, nullptr, {});
+  };
+  return CheckForImmediateInvocation(
+      MaybeBindToTemporary(TransformStdMoveAndForwardIntoCast(TheCall)), FDecl);
 }
 
 ExprResult
 Sema::ActOnCompoundLiteral(SourceLocation LParenLoc, ParsedType Ty,
                            SourceLocation RParenLoc, Expr *InitExpr) {
   assert(Ty && "ActOnCompoundLiteral(): missing type");
   assert(InitExpr && "ActOnCompoundLiteral(): missing expression");
 

clang/lib/Sema/TreeTransform.h

 template<typename Derived>
 ExprResult
 TreeTransform<Derived>::TransformImplicitCastExpr(ImplicitCastExpr *E) {
   // Implicit casts are eliminated during transformation, since they
   // will be recomputed by semantic analysis after transformation.
   return getDerived().TransformExpr(E->getSubExprAsWritten());
 }
 
-template<typename Derived>
+template <typename Derived>
+ExprResult TreeTransform<Derived>::TransformCXXStandardLibraryCastExpr(
+    CXXStandardLibraryCastExpr *E) {
+  // CXXStandardLibraryCastExpr casts are eliminated during transformation,
+  // since they will be recomputed by semantic analysis after transformation.
+  return E;
+}
+
+template<typename Derived>
 ExprResult
 TreeTransform<Derived>::TransformCStyleCastExpr(CStyleCastExpr *E) {
   TypeSourceInfo *Type = getDerived().TransformType(E->getTypeInfoAsWritten());
   if (!Type)
     return ExprError();
 
   ExprResult SubExpr
     = getDerived().TransformExpr(E->getSubExprAsWritten());

clang/lib/Serialization/ASTReaderStmt.cpp

ASTStmtReader::VisitBinaryConditionalOperator(BinaryConditionalOperator *E) {
   E->ColonLoc = readSourceLocation();
 }
 
 void ASTStmtReader::VisitImplicitCastExpr(ImplicitCastExpr *E) {
   VisitCastExpr(E);
   E->setIsPartOfExplicitCast(Record.readInt());
 }
 
+void ASTStmtReader::VisitCXXStandardLibraryCastExpr(
+    CXXStandardLibraryCastExpr *E) {
+  VisitCastExpr(E);
+  E->OriginalExpression = Record.readSubExpr();
+}
+
 void ASTStmtReader::VisitExplicitCastExpr(ExplicitCastExpr *E) {
   VisitCastExpr(E);
   E->setTypeInfoAsWritten(readTypeSourceInfo());
 }
 
 void ASTStmtReader::VisitCStyleCastExpr(CStyleCastExpr *E) {
   VisitExplicitCastExpr(E);
   E->setLParenLoc(readSourceLocation());
while (true) {
 
     case EXPR_IMPLICIT_CAST:
       S = ImplicitCastExpr::CreateEmpty(
           Context,
           /*PathSize*/ Record[ASTStmtReader::NumExprFields],
           /*HasFPFeatures*/ Record[ASTStmtReader::NumExprFields + 1]);
       break;
 
+    case EXPR_CXX_STL_CAST:
+      S = CXXStandardLibraryCastExpr::CreateEmpty(
+          Context,
+          /*PathSize*/ Record[ASTStmtReader::NumExprFields],
+          /*HasFPFeatures*/ Record[ASTStmtReader::NumExprFields + 1]);
+      break;
+
     case EXPR_CSTYLE_CAST:
       S = CStyleCastExpr::CreateEmpty(
           Context,
           /*PathSize*/ Record[ASTStmtReader::NumExprFields],
           /*HasFPFeatures*/ Record[ASTStmtReader::NumExprFields + 1]);
       break;
 
     case EXPR_COMPOUND_LITERAL:

clang/lib/Serialization/ASTWriterStmt.cpp

void ASTStmtWriter::VisitCXXNamedCastExpr(CXXNamedCastExpr *E) {
   Record.AddSourceRange(E->getAngleBrackets());
 }
 
 void ASTStmtWriter::VisitCXXStaticCastExpr(CXXStaticCastExpr *E) {
   VisitCXXNamedCastExpr(E);
   Code = serialization::EXPR_CXX_STATIC_CAST;
 }
 
+void ASTStmtWriter::VisitCXXStandardLibraryCastExpr(
+    CXXStandardLibraryCastExpr *E) {
+  VisitCastExpr(E);
+  VisitExpr(E);
+  Code = serialization::EXPR_CXX_STATIC_CAST;
+}
+
 void ASTStmtWriter::VisitCXXDynamicCastExpr(CXXDynamicCastExpr *E) {
   VisitCXXNamedCastExpr(E);
   Code = serialization::EXPR_CXX_DYNAMIC_CAST;
 }
 
 void ASTStmtWriter::VisitCXXReinterpretCastExpr(CXXReinterpretCastExpr *E) {
   VisitCXXNamedCastExpr(E);
   Code = serialization::EXPR_CXX_REINTERPRET_CAST;

clang/lib/StaticAnalyzer/Core/ExprEngine.cpp

case Stmt::DeclStmtClass:
       Bldr.takeNodes(Pred);
       VisitDeclStmt(cast<DeclStmt>(S), Pred, Dst);
       Bldr.addNodes(Dst);
       break;
 
     case Stmt::ImplicitCastExprClass:
     case Stmt::CStyleCastExprClass:
     case Stmt::CXXStaticCastExprClass:
+    case Stmt::CXXStandardLibraryCastExprClass:
     case Stmt::CXXDynamicCastExprClass:
     case Stmt::CXXReinterpretCastExprClass:
     case Stmt::CXXConstCastExprClass:
     case Stmt::CXXFunctionalCastExprClass:
     case Stmt::BuiltinBitCastExprClass:
     case Stmt::ObjCBridgedCastExprClass:
     case Stmt::CXXAddrspaceCastExprClass: {
       Bldr.takeNodes(Pred);

clang/tools/libclang/CXCursor.cpp

CXCursor cxcursor::MakeCXCursor(const Stmt *S, const Decl *Parent,
   case Stmt::GNUNullExprClass:
     K = CXCursor_GNUNullExpr;
     break;
 
   case Stmt::CXXStaticCastExprClass:
     K = CXCursor_CXXStaticCastExpr;
     break;
 
+  case Stmt::CXXStandardLibraryCastExprClass:
+    return MakeCXCursor(
+        cast<CXXStandardLibraryCastExpr>(S)->getOriginalExpression(), Parent,
+        TU, RegionOfInterest);
+
   case Stmt::CXXDynamicCastExprClass:
     K = CXCursor_CXXDynamicCastExpr;
     break;
 
   case Stmt::CXXReinterpretCastExprClass:
     K = CXCursor_CXXReinterpretCastExpr;
     break;