Add reference type transformation builtins

[zoecarver]

Summary

This patch adds builtin type traits to transform reference types. Specifically, it adds __add_lvalue_reference, __add_rvalue_reference, and __remove_reference. The first two builtins speed up builds by around 3x while the last builtin only sees small improvements (we may be able to optimize it more, though). Once added to the standard library, this should make libc++ (and other code) much faster to compile.

I tried to generalize as much of the builtin as possible so, the only functional difference between the three builtins is in the file BuildUnaryTransformType.

[zoecarver]

libcxx/test/libcxx/utilities/meta/stress_tests/stress_test_remove_reference.sh.cpp

13

I think there is a good chance the reason that my added builtin isn't significantly faster is that std::remove_reference is already optimized somewhere. Thoughts?

[zoecarver]

• remove accedentally added file

[Mordante]

clang/lib/Parse/ParseDeclCXX.cpp

1095

Wouldn't it be better to use llvm_unreachable("Not a reference type specifier");
Maybe also move this line out of the switch, allowing the compiler to warn about not handled enumeration values.

clang/lib/Sema/SemaType.cpp

1266

Same here?

[mclow.lists]

If you're going to do __add__lvalue_reference, __add_rvalue_reference, and __remove_reference, why not go all the way and add __is_reference, __is_lvalue_reference and __is_rvalue_reference?

[zoecarver]

In D67052#1656319, @mclow.lists wrote:

If you're going to do __add__lvalue_reference, __add_rvalue_reference, and __remove_reference, why not go all the way and add __is_reference, __is_lvalue_reference and __is_rvalue_reference?

Those already exist :)

I am planning on adding others, __remove_cv, __decay, etc. but wanted to land this first and make sure I was doing it right.

I'll update type_traits in libc++ after this lands. I'm also planning on making a patch to update a few of the builtin type traits. That should significantly speed up build times. FYI here is all the "type trait primitives" clang supports. At some point, we should try to support all of them.

[zoecarver]

clang/lib/Parse/ParseDeclCXX.cpp

1095

I'll update it to use llvm_unreachable but, I don't think we want the compiler to warn about unhandled enumeration values.

[zoecarver]

libcxx/test/libcxx/utilities/meta/stress_tests/stress_test_remove_reference.sh.cpp

13

I think if I updated this test to actually remove a reference, it might show more of a difference.

[zoecarver]

clang/include/clang/Parse/Parser.h

2606

These methods are generalized more in D67588.

[EricWF]

clang/include/clang/AST/Type.h

4353

This should say ref in the name.

clang/lib/Parse/ParseDeclCXX.cpp

1094

Don't use default switch cases. It suppresses non-covered switch warnings, which we want fire when a new enumerator is added.

[EricWF]

clang/lib/Parse/ParseDeclCXX.cpp

1099

I think this should be generalized and merged with ParseUnderlyingTypeSpecifier.

Maybe called ParseTransformationTypeSpecifier.

clang/lib/Sema/SemaType.cpp

1612

This should be merged with the above case.

5490

DS.getTypeSpecType() >= DeclSpec::TST_underlyingType && DS.getTypeSpecType() <= TST_removeReferenceType would correct this assertion.

clang/test/SemaCXX/add_reference.cpp

67

You should test a lot more types here. reference types, const types, pointer types, function types, non-referenceable types.

[zoecarver]

clang/lib/Sema/SemaType.cpp

1612

The reason I haven't done that is because __underlying_type will default to IntTy if it fails. We don't want that for the others (maybe we don't want that here either?).

[zoecarver]

• address review comments
• fix warnings in build

[zoecarver]

clang/lib/AST/ItaniumMangle.cpp

3412 ↗

(On Diff #221014)

Eric, I looked at your comment, but it seems that we no longer unary type transformations that way. @rsmith is this correct?

[zoecarver]

Friendly ping. Other than type mangling, does anything need to be fixed?

[zoecarver]

@EricWF ping.

[zoecarver]

• Combine D67052 and D67588.
• Remove __add* type traits.
• Update mangling/dumping/encoding of all traits.
• Remove stress tests.
• Address comments in D67588.

I'll work on some failing tests and add those when they're ready.

[zoecarver]

clang/lib/AST/TypePrinter.cpp

1028 ↗

(On Diff #243834)

These need the opening paren.

1062 ↗

(On Diff #243834)

Missing RemoveReference.

clang/lib/Parse/ParseDeclCXX.cpp

1102

I'll work on your comment:

It would be really cool if we could simplify these conversions to static_cast<DeclSpec::TST>(Tok.getKind() - base);

[miscco]

Some nits, looks great, especially for library concepts. Pinging @cjdb

clang/lib/AST/TypePrinter.cpp

1020 ↗

(On Diff #243834)

Couldn't we use TextNodeDumper::VisitUnaryTransformType to dump the string and then simplify it to a single case.

Given the possibility that maybe some day one might add the add_foo builtins

clang/lib/Parse/ParseDeclCXX.cpp

1107

I dont think that this message is valid anymore now that you also remove qualifiers.

[EricWF]

LGTM once all of the inline comments are addressed.

After that, this is ready to land.

clang/lib/AST/ItaniumMangle.cpp

3412 ↗

(On Diff #221014)

@rsmith Can you double check the mangling changes?

clang/lib/Sema/SemaType.cpp

1601

Why can't we share the implementation with TST_underlyingType?

[miscco]

clang/include/clang/Sema/DeclSpec.h

414–416

There is a bug here, this should probably be

return (TST_typename <= T && T <= TST_atomic);

[zoecarver]

@EricWF and @miscco thanks for the review comments. Sorry for the delay, I forgot about this patch. All your comments have been addressed/fixed.

clang/lib/AST/TypePrinter.cpp

1020 ↗

(On Diff #243834)

Couldn't we use TextNodeDumper::VisitUnaryTransformType to dump the string and then simplify it to a single case.

Good call. Will do.

Given the possibility that maybe some day one might add the add_foo builtins

I had a patch that added those builtins but, we decided that there was no real reason for them to exist.

clang/lib/Sema/SemaType.cpp

1601

Yep, fixed.

[zoecarver]

• Rebase
• Fix based on review comments

[zoecarver]

• Format using clang-format

clang/include/clang/AST/Type.h

public:
   static void Profile(llvm::FoldingSetNodeID &ID, const ASTContext &Context,
                       Expr *E);
 };
 
 /// A unary type transform, which is a type constructed from another.
 class UnaryTransformType : public Type {
 public:
   enum UTTKind {
-    EnumUnderlyingType
+    EnumUnderlyingType,
+    RemoveReferenceType,
+    AddRValueType,
+    AddLValueType
   };
 
 private:
   /// The untransformed type.
   QualType BaseType;
 
   /// The transformed type if not dependent, otherwise the same as BaseType.
   QualType UnderlyingType;

clang/include/clang/Basic/Specifiers.h

enum TypeSpecifierType {
     TST_struct,
     TST_class,        // C++ class type
     TST_interface,    // C++ (Microsoft-specific) __interface type
     TST_typename,     // Typedef, C++ class-name or enum name, etc.
     TST_typeofType,
     TST_typeofExpr,
     TST_decltype,         // C++11 decltype
     TST_underlyingType,   // __underlying_type for C++11
+    TST_addLValueReferenceType, // __add_lvalue_reference
+    TST_addRValueReferenceType, // __add_rvalue_reference
+    TST_removeReferenceType, // __remove_reference
     TST_auto,             // C++11 auto
     TST_decltype_auto,    // C++1y decltype(auto)
     TST_auto_type,        // __auto_type extension
     TST_unknown_anytype,  // __unknown_anytype extension
     TST_atomic,           // C11 _Atomic
 #define GENERIC_IMAGE_TYPE(ImgType, Id) TST_##ImgType##_t, // OpenCL image types
 #include "clang/Basic/OpenCLImageTypes.def"
     TST_error // erroneous type

clang/include/clang/Basic/TokenKinds.def

 TYPE_TRAIT_1(__is_standard_layout, IsStandardLayout, KEYCXX)
 TYPE_TRAIT_1(__is_trivial, IsTrivial, KEYCXX)
 TYPE_TRAIT_2(__is_trivially_assignable, IsTriviallyAssignable, KEYCXX)
 TYPE_TRAIT_N(__is_trivially_constructible, IsTriviallyConstructible, KEYCXX)
 TYPE_TRAIT_1(__is_trivially_copyable, IsTriviallyCopyable, KEYCXX)
 TYPE_TRAIT_1(__is_union, IsUnion, KEYCXX)
 TYPE_TRAIT_1(__has_unique_object_representations,
              HasUniqueObjectRepresentations, KEYCXX)
-KEYWORD(__underlying_type           , KEYCXX)
+KEYWORD(__underlying_type                  , KEYCXX)
+KEYWORD(__add_lvalue_reference             , KEYCXX)
+KEYWORD(__add_rvalue_reference             , KEYCXX)
+KEYWORD(__remove_reference                 , KEYCXX)
 
 // Clang-only C++ Type Traits
 TYPE_TRAIT_2(__reference_binds_to_temporary, ReferenceBindsToTemporary, KEYCXX)
 
 // Embarcadero Expression Traits
 KEYWORD(__is_lvalue_expr            , KEYCXX)
 KEYWORD(__is_rvalue_expr            , KEYCXX)
 

clang/include/clang/Parse/Parser.h

ParseAttributeWithTypeArg(IdentifierInfo &AttrName,
                             SourceLocation ScopeLoc, ParsedAttr::Syntax Syntax);
 
   void ParseTypeofSpecifier(DeclSpec &DS);
   SourceLocation ParseDecltypeSpecifier(DeclSpec &DS);
   void AnnotateExistingDecltypeSpecifier(const DeclSpec &DS,
                                          SourceLocation StartLoc,
                                          SourceLocation EndLoc);
   void ParseUnderlyingTypeSpecifier(DeclSpec &DS);
+  DeclSpec::TST ReferenceTransformTokToDeclSpec();
+  void ParseAddReferenceTypeSpecifier(DeclSpec &DS);
   void ParseAtomicSpecifier(DeclSpec &DS);
 
   ExprResult ParseAlignArgument(SourceLocation Start,
                                 SourceLocation &EllipsisLoc);
   void ParseAlignmentSpecifier(ParsedAttributes &Attrs,
                                SourceLocation *endLoc = nullptr);
 
   VirtSpecifiers::Specifier isCXX11VirtSpecifier(const Token &Tok) const;

clang/include/clang/Sema/DeclSpec.h

public:
   static const TST TST_interface = clang::TST_interface;
   static const TST TST_class = clang::TST_class;
   static const TST TST_typename = clang::TST_typename;
   static const TST TST_typeofType = clang::TST_typeofType;
   static const TST TST_typeofExpr = clang::TST_typeofExpr;
   static const TST TST_decltype = clang::TST_decltype;
   static const TST TST_decltype_auto = clang::TST_decltype_auto;
   static const TST TST_underlyingType = clang::TST_underlyingType;
+  static const TST TST_addLValueReferenceType = clang::TST_addLValueReferenceType;
+  static const TST TST_addRValueReferenceType = clang::TST_addRValueReferenceType;
+  static const TST TST_removeReferenceType = clang::TST_removeReferenceType;
   static const TST TST_auto = clang::TST_auto;
   static const TST TST_auto_type = clang::TST_auto_type;
   static const TST TST_unknown_anytype = clang::TST_unknown_anytype;
   static const TST TST_atomic = clang::TST_atomic;
 #define GENERIC_IMAGE_TYPE(ImgType, Id) \
   static const TST TST_##ImgType##_t = clang::TST_##ImgType##_t;
 #include "clang/Basic/OpenCLImageTypes.def"
   static const TST TST_error = clang::TST_error;
private:
 
   WrittenBuiltinSpecs writtenBS;
   void SaveWrittenBuiltinSpecs();
 
   ObjCDeclSpec *ObjCQualifiers;
 
   static bool isTypeRep(TST T) {
     return (T == TST_typename || T == TST_typeofType ||
-            T == TST_underlyingType || T == TST_atomic);
+            T == TST_underlyingType || T == TST_atomic||
+            T == TST_addLValueReferenceType || T == TST_addRValueReferenceType ||
+            T == TST_removeReferenceType);
   }
   static bool isExprRep(TST T) {
     return (T == TST_typeofExpr || T == TST_decltype);
   }
 
   DeclSpec(const DeclSpec &) = delete;
   void operator=(const DeclSpec &) = delete;
 public:

clang/lib/Format/FormatToken.cpp

bool FormatToken::isSimpleTypeSpecifier() const {
   case tok::kw_half:
   case tok::kw_float:
   case tok::kw_double:
   case tok::kw__Float16:
   case tok::kw___float128:
   case tok::kw_wchar_t:
   case tok::kw_bool:
   case tok::kw___underlying_type:
+  case tok::kw___add_lvalue_reference:
+  case tok::kw___add_rvalue_reference:
+  case tok::kw___remove_reference:
   case tok::annot_typename:
   case tok::kw_char8_t:
   case tok::kw_char16_t:
   case tok::kw_char32_t:
   case tok::kw_typeof:
   case tok::kw_decltype:
     return true;
   default:

clang/lib/Lex/PPMacroExpansion.cpp

EvaluateFeatureLikeBuiltinMacro(OS, Tok, II, *this,
               II->getName().startswith("__is_") ||
               II->getName().startswith("__has_"))
             return true;
           return llvm::StringSwitch<bool>(II->getName())
               .Case("__array_rank", true)
               .Case("__array_extent", true)
               .Case("__reference_binds_to_temporary", true)
               .Case("__underlying_type", true)
+              .Case("__add_lvalue_reference", true)
+              .Case("__add_rvalue_reference", true)
+              .Case("__remove_reference", true)
               .Default(false);
         } else {
           return llvm::StringSwitch<bool>(II->getName())
               // Report builtin templates as being builtins.
               .Case("__make_integer_seq", LangOpts.CPlusPlus)
               .Case("__type_pack_element", LangOpts.CPlusPlus)
               // Likewise for some builtin preprocessor macros.
               // FIXME: This is inconsistent; we usually suggest detecting

clang/lib/Parse/ParseDecl.cpp

#include "clang/Basic/OpenCLImageTypes.def"
     case tok::annot_pragma_ms_pointers_to_members:
       HandlePragmaMSPointersToMembers();
       continue;
 
     case tok::kw___underlying_type:
       ParseUnderlyingTypeSpecifier(DS);
       continue;
 
+    case tok::kw___remove_reference:
+    case tok::kw___add_rvalue_reference:
+    case tok::kw___add_lvalue_reference:
+      ParseAddReferenceTypeSpecifier(DS);
+      continue;
+
     case tok::kw__Atomic:
       // C11 6.7.2.4/4:
       //   If the _Atomic keyword is immediately followed by a left parenthesis,
       //   it is interpreted as a type specifier (with a type name), not as a
       //   type qualifier.
       if (NextToken().is(tok::l_paren)) {
         ParseAtomicSpecifier(DS);
         continue;

clang/lib/Parse/ParseDeclCXX.cpp

void Parser::ParseUnderlyingTypeSpecifier(DeclSpec &DS) {
   unsigned DiagID;
   if (DS.SetTypeSpecType(DeclSpec::TST_underlyingType, StartLoc, PrevSpec,
                          DiagID, Result.get(),
                          Actions.getASTContext().getPrintingPolicy()))
     Diag(StartLoc, DiagID) << PrevSpec;
   DS.setTypeofParensRange(T.getRange());
 }
 
+DeclSpec::TST Parser::ReferenceTransformTokToDeclSpec() {
+  switch (Tok.getKind()) {
+  case tok::kw___add_lvalue_reference:
+    return DeclSpec::TST_addLValueReferenceType;
+  case tok::kw___add_rvalue_reference:
+    return DeclSpec::TST_addRValueReferenceType;
+  case tok::kw___remove_reference:
+    return DeclSpec::TST_removeReferenceType;
+  default:
+    assert(false && "Not a reference type specifier");
+  }
+}
+
+void Parser::ParseAddReferenceTypeSpecifier(DeclSpec &DS) {
+  DeclSpec::TST ReferenceTransformTST = ReferenceTransformTokToDeclSpec();
+
+  SourceLocation StartLoc = ConsumeToken();
+  BalancedDelimiterTracker T(*this, tok::l_paren);
+  if (T.expectAndConsume(diag::err_expected_lparen_after,
+                         "reference manipulation builtin",
+                         tok::r_paren)) return;
+
+  TypeResult BaseTyResult = ParseTypeName();
+  if (BaseTyResult.isInvalid()) {
+    SkipUntil(tok::r_paren, StopAtSemi);
+    return;
+  }
+
+  T.consumeClose();
+  if (T.getCloseLocation().isInvalid()) return;
+
+  const char *PrevSpec;
+  unsigned DiagID;
+  if (DS.SetTypeSpecType(ReferenceTransformTST,
+                         StartLoc, PrevSpec,
+                         DiagID, BaseTyResult.get(),
+                         Actions.getASTContext().getPrintingPolicy())) {
+    Diag(StartLoc, DiagID) << PrevSpec;
+  }
+  DS.setTypeofParensRange(T.getRange());
+}
+
 /// ParseBaseTypeSpecifier - Parse a C++ base-type-specifier which is either a
 /// class name or decltype-specifier. Note that we only check that the result
 /// names a type; semantic analysis will need to verify that the type names a
 /// class. The result is either a type or null, depending on whether a type
 /// name was found.
 ///
 ///       base-type-specifier: [C++11 class.derived]
 ///         class-or-decltype

clang/lib/Sema/DeclSpec.cpp

#include "clang/Basic/OpenCLImageTypes.def"
 
     case TST_decltype:
     case TST_typeofExpr:
       if (Expr *E = DS.getRepAsExpr())
         return E->getType()->isFunctionType();
       return false;
 
     case TST_underlyingType:
+    case TST_addLValueReferenceType:
+    case TST_addRValueReferenceType:
+    case TST_removeReferenceType:
     case TST_typename:
     case TST_typeofType: {
       QualType QT = DS.getRepAsType().get();
       if (QT.isNull())
         return false;
 
       if (const LocInfoType *LIT = dyn_cast<LocInfoType>(QT))
         QT = LIT->getType();
const char *DeclSpec::getSpecifierName(DeclSpec::TST T,
   case DeclSpec::TST_typename:    return "type-name";
   case DeclSpec::TST_typeofType:
   case DeclSpec::TST_typeofExpr:  return "typeof";
   case DeclSpec::TST_auto:        return "auto";
   case DeclSpec::TST_auto_type:   return "__auto_type";
   case DeclSpec::TST_decltype:    return "(decltype)";
   case DeclSpec::TST_decltype_auto: return "decltype(auto)";
   case DeclSpec::TST_underlyingType: return "__underlying_type";
+  case DeclSpec::TST_addLValueReferenceType: return "__add_lvalue_reference";
+  case DeclSpec::TST_addRValueReferenceType: return "__add_rvalue_reference";
+  case DeclSpec::TST_removeReferenceType: return "__remove_reference";
   case DeclSpec::TST_unknown_anytype: return "__unknown_anytype";
   case DeclSpec::TST_atomic: return "_Atomic";
 #define GENERIC_IMAGE_TYPE(ImgType, Id) \
   case DeclSpec::TST_##ImgType##_t: \
     return #ImgType "_t";
 #include "clang/Basic/OpenCLImageTypes.def"
   case DeclSpec::TST_error:       return "(error)";
   }

clang/lib/Sema/SemaDecl.cpp

bool Sema::isSimpleTypeSpecifier(tok::TokenKind Kind) const {
   case tok::kw_half:
   case tok::kw_float:
   case tok::kw_double:
   case tok::kw__Float16:
   case tok::kw___float128:
   case tok::kw_wchar_t:
   case tok::kw_bool:
   case tok::kw___underlying_type:
+  case tok::kw___add_lvalue_reference:
+  case tok::kw___add_rvalue_reference:
+  case tok::kw___remove_reference:
   case tok::kw___auto_type:
     return true;
 
   case tok::annot_typename:
   case tok::kw_char16_t:
   case tok::kw_char32_t:
   case tok::kw_typeof:
   case tok::annot_decltype:
static bool RebuildDeclaratorInCurrentInstantiation(Sema &S, Declarator &D,
   // type.  It's safest to just say "dependent", but we call out a
   // few cases here.
 
   DeclSpec &DS = D.getMutableDeclSpec();
   switch (DS.getTypeSpecType()) {
   case DeclSpec::TST_typename:
   case DeclSpec::TST_typeofType:
   case DeclSpec::TST_underlyingType:
+  case DeclSpec::TST_addLValueReferenceType:
+  case DeclSpec::TST_addRValueReferenceType:
+  case DeclSpec::TST_removeReferenceType:
   case DeclSpec::TST_atomic: {
     // Grab the type from the parser.
     TypeSourceInfo *TSI = nullptr;
     QualType T = S.GetTypeFromParser(DS.getRepAsType(), &TSI);
     if (T.isNull() || !T->isDependentType()) break;
 
     // Make sure there's a type source info.  This isn't really much
     // of a waste; most dependent types should have type source info

clang/lib/Sema/SemaTemplateVariadic.cpp

 }
 
 bool Sema::containsUnexpandedParameterPacks(Declarator &D) {
   const DeclSpec &DS = D.getDeclSpec();
   switch (DS.getTypeSpecType()) {
   case TST_typename:
   case TST_typeofType:
   case TST_underlyingType:
+  case TST_addLValueReferenceType:
+  case TST_addRValueReferenceType:
+  case TST_removeReferenceType:
   case TST_atomic: {
     QualType T = DS.getRepAsType().get();
     if (!T.isNull() && T->containsUnexpandedParameterPack())
       return true;
     break;
   }
 
   case TST_typeofExpr:

clang/lib/Sema/SemaType.cpp

 static OpenCLAccessAttr::Spelling
 getImageAccess(const ParsedAttributesView &Attrs) {
   for (const ParsedAttr &AL : Attrs)
     if (AL.getKind() == ParsedAttr::AT_OpenCLAccess)
       return static_cast<OpenCLAccessAttr::Spelling>(AL.getSemanticSpelling());
   return OpenCLAccessAttr::Keyword_read_only;
 }
 
+static UnaryTransformType::UTTKind
+TSTToUnaryTransformType(DeclSpec::TST SwitchTST) {
+  switch (SwitchTST) {
+  case TST_addLValueReferenceType:
+    return UnaryTransformType::AddLValueType;
+  case TST_addRValueReferenceType:
+    return UnaryTransformType::AddRValueType;
+  case TST_removeReferenceType:
+    return UnaryTransformType::RemoveReferenceType;
+  case TST_underlyingType:
+    return UnaryTransformType::EnumUnderlyingType;
+  default:
+    assert(false && "Cannot map TST to unary transform type");
+  }
+}
+
 /// Convert the specified declspec to the appropriate type
 /// object.
 /// \param state Specifies the declarator containing the declaration specifier
 /// to be converted, along with other associated processing state.
 /// \returns The type described by the declaration specifiers.  This function
 /// never returns null.
 static QualType ConvertDeclSpecToType(TypeProcessingState &state) {
   // FIXME: Should move the logic from DeclSpec::Finish to here for validity
case DeclSpec::TST_decltype: {
     // TypeQuals handled by caller.
     Result = S.BuildDecltypeType(E, DS.getTypeSpecTypeLoc());
     if (Result.isNull()) {
       Result = Context.IntTy;
       declarator.setInvalidType(true);
     }
     break;
   }
   case DeclSpec::TST_underlyingType:
     Result = S.GetTypeFromParser(DS.getRepAsType());
     assert(!Result.isNull() && "Didn't get a type for __underlying_type?");
     Result = S.BuildUnaryTransformType(Result,
                                        UnaryTransformType::EnumUnderlyingType,
                                        DS.getTypeSpecTypeLoc());
     if (Result.isNull()) {
       Result = Context.IntTy;
       declarator.setInvalidType(true);
     }
     break;
+  case DeclSpec::TST_addLValueReferenceType:
+  case DeclSpec::TST_addRValueReferenceType:
+  case DeclSpec::TST_removeReferenceType:
+    Result = S.GetTypeFromParser(DS.getRepAsType());
+    assert(!Result.isNull() &&
+           "Reference manipulation type transform may not have received a type.");
+    Result = S.BuildUnaryTransformType(Result,
+                                       TSTToUnaryTransformType(DS.getTypeSpecType()),
+                                       DS.getTypeSpecTypeLoc());
+    if (Result.isNull()) {
+      declarator.setInvalidType(true);
+    }
+    break;
+
 
   case DeclSpec::TST_auto:
     Result = Context.getAutoType(QualType(), AutoTypeKeyword::Auto, false);
     break;
 
   case DeclSpec::TST_auto_type:
     Result = Context.getAutoType(QualType(), AutoTypeKeyword::GNUAutoType, false);
     break;
void VisitTypeOfTypeLoc(TypeOfTypeLoc TL) {
       TL.setTypeofLoc(DS.getTypeSpecTypeLoc());
       TL.setParensRange(DS.getTypeofParensRange());
       assert(DS.getRepAsType());
       TypeSourceInfo *TInfo = nullptr;
       Sema::GetTypeFromParser(DS.getRepAsType(), &TInfo);
       TL.setUnderlyingTInfo(TInfo);
     }
     void VisitUnaryTransformTypeLoc(UnaryTransformTypeLoc TL) {
-      // FIXME: This holds only because we only have one unary transform.
-      assert(DS.getTypeSpecType() == DeclSpec::TST_underlyingType);
+      // TODO: Should we check something like "IsUnaryTypeTransfrom(DS.getTypeSpecTypeLoc())"?
+      // assert(DS.getTypeSpecType() == DeclSpec::TST_underlyingType);
       TL.setKWLoc(DS.getTypeSpecTypeLoc());
       TL.setParensRange(DS.getTypeofParensRange());
       assert(DS.getRepAsType());
       TypeSourceInfo *TInfo = nullptr;
       Sema::GetTypeFromParser(DS.getRepAsType(), &TInfo);
       TL.setUnderlyingTInfo(TInfo);
     }
     void VisitBuiltinTypeLoc(BuiltinTypeLoc TL) {
if (!BaseType->isDependentType() && !BaseType->isEnumeralType()) {
         DiagnoseUseOfDecl(ED, Loc);
 
         Underlying = ED->getIntegerType();
         assert(!Underlying.isNull());
       }
       return Context.getUnaryTransformType(BaseType, Underlying,
                                         UnaryTransformType::EnumUnderlyingType);
     }
+  case UnaryTransformType::AddLValueType: {
+    QualType Underlying = Context.getLValueReferenceType(BaseType.getCanonicalType());
+    return Context.getUnaryTransformType(BaseType, Underlying,
+                                         UnaryTransformType::AddLValueType);
+  }
+  case UnaryTransformType::AddRValueType: {
+    QualType Underlying = Context.getRValueReferenceType(BaseType.getCanonicalType());
+    return Context.getUnaryTransformType(BaseType, Underlying,
+                                         UnaryTransformType::AddRValueType);
+  }
+  case UnaryTransformType::RemoveReferenceType: {
+    QualType Underlying = BaseType.getNonReferenceType();
+    return Context.getUnaryTransformType(BaseType, Underlying,
+                                         UnaryTransformType::RemoveReferenceType);
+  }
   }
   llvm_unreachable("unknown unary transform type");
 }
 
 QualType Sema::BuildAtomicType(QualType T, SourceLocation Loc) {
   if (!T->isDependentType()) {
     // FIXME: It isn't entirely clear whether incomplete atomic types
     // are allowed or not; for simplicity, ban them for the moment.

clang/test/SemaCXX/add_reference.cpp

+// RUN: %clang_cc1 -std=c++11 %s
+
+template<class T>
+struct test_lval_trait
+{
+  typedef __add_lvalue_reference(T) type;
+};
+
+template<class T>
+struct test_lval
+{
+  static const bool value = __is_same(typename test_lval_trait<T>::type, T&)  &&
+                            __is_same(typename test_lval_trait<T&>::type, T&) &&
+                            __is_same(typename test_lval_trait<T&&>::type, T&);
+};
+
+template<class T>
+struct test_rval_trait
+{
+  typedef __add_rvalue_reference(T) type;
+};
+
+template<class T>
+struct test_rval
+{
+  static const bool value = __is_same(typename test_rval_trait<T>::type, T&&)  &&
+                            __is_same(typename test_rval_trait<T&>::type, T&&) &&
+                            __is_same(typename test_rval_trait<T&&>::type, T&&);
+};
+
+template<class T>
+struct test_remove_ref_trait
+{
+  typedef __remove_reference(T) type;
+};
+
+template<class T>
+struct test_remove_ref
+{
+  static const bool value = __is_same(typename test_remove_ref_trait<T>::type, T)  &&
+                            __is_same(typename test_remove_ref_trait<T&>::type, T) &&
+                            __is_same(typename test_remove_ref_trait<T&&>::type, T);
+};
+
+template<class T>
+struct test
+{
+  static const bool value = test_remove_ref<T>::value &&
+                            test_rval<T>::value &&
+                            test_lval<T>::value;
+};
+
+struct Foo { };
+
+template<class T>
+struct Bar { };
+
+template<class T>
+class Baz { };
+
+static_assert(test<int>::value, "");
+static_assert(test<int[]>::value, "");
+static_assert(test<int[8]>::value, "");
+static_assert(test<Foo>::value, "");
+static_assert(test<Bar<int>>::value, "");
+static_assert(test<Baz<int>>::value, "");

libcxx/test/libcxx/utilities/meta/stress_tests/lit.site.cfg

+config.cxx_under_test = "/Users/zoe/Developer/llvm-apple/mono-build/bin/clang++"
+lit_config.load_config(config, "/Users/zoe/Developer/llvm-apple/llvm-project/libcxx/test/lit.cfg")

libcxx/test/libcxx/utilities/meta/stress_tests/stress_test_add_lvalue_reference.sh.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This is a dummy feature that prevents this test from running by default.
+
+// The table below compares the compile time and object size for each of the
+// variants listed in the RUN script.
+//
+//  Impl                       Compile Time      Object Size
+// ----------------------------------------------------------
+// new_add_lvalue_reference:   23,359.202 ms     171 K
+// std::add_lvalue_reference:  73,160.138 ms     201 K
+//
+// RUN: %cxx %flags %compile_flags -c %s -o %S/orig.o -ggdb  -ggnu-pubnames -ftemplate-depth=5000 -ftime-trace -std=c++17
+// RUN: %cxx %flags %compile_flags -c %s -o %S/new.o -ggdb  -ggnu-pubnames -ftemplate-depth=5000 -ftime-trace -std=c++17 -DTEST_NEW
+
+#include <type_traits>
+#include <cassert>
+
+#include "test_macros.h"
+#include "template_cost_testing.h"
+
+template <int N> struct Arg { enum { value = 1 }; };
+
+#ifdef TEST_NEW
+
+template <class T>
+struct new_add_lvalue_reference
+{
+  typedef __add_lvalue_reference(T) type;
+};
+
+#define TEST_CASE_NOP()  new_add_lvalue_reference< Arg< __COUNTER__ > >{},
+#define TEST_CASE_TYPE() typename new_add_lvalue_reference< Arg< __COUNTER__ > >::type,
+
+#else
+
+#define TEST_CASE_NOP()  std::add_lvalue_reference< Arg< __COUNTER__ > >{},
+#define TEST_CASE_TYPE() typename std::add_lvalue_reference< Arg< __COUNTER__ > >::type,
+
+#endif
+
+int sink(...);
+
+int x = sink(
+  REPEAT_10000(TEST_CASE_NOP)
+  REPEAT_10000(TEST_CASE_NOP) 42
+);
+
+void Foo( REPEAT_10000(TEST_CASE_TYPE) int) { }
+
+void escape() {
+
+sink(&x);
+sink(&Foo);
+}
+
+

libcxx/test/libcxx/utilities/meta/stress_tests/stress_test_add_rvalue_reference.sh.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This is a dummy feature that prevents this test from running by default.
+
+// The table below compares the compile time and object size for each of the
+// variants listed in the RUN script.
+//
+//  Impl                       Compile Time      Object Size
+// ----------------------------------------------------------
+// new_add_rvalue_reference:   56.398 s          171 K
+// std::add_rvalue_reference:  114.59 s          271 K
+//
+// RUN: %cxx %flags %compile_flags -c %s -o %S/orig.o -ggdb  -ggnu-pubnames -ftemplate-depth=5000 -ftime-trace -std=c++17
+// RUN: %cxx %flags %compile_flags -c %s -o %S/new.o -ggdb  -ggnu-pubnames -ftemplate-depth=5000 -ftime-trace -std=c++17 -DTEST_NEW
+
+#include <type_traits>
+#include <cassert>
+
+#include "test_macros.h"
+#include "template_cost_testing.h"
+
+template <int N> struct Arg { enum { value = 1 }; };
+
+#ifdef TEST_NEW
+
+template <class T>
+struct new_add_rvalue_reference
+{
+  typedef __add_rvalue_reference(T) type;
+};
+
+#define TEST_CASE_NOP()  new_add_rvalue_reference< Arg< __COUNTER__ > >{},
+#define TEST_CASE_TYPE() typename new_add_rvalue_reference< Arg< __COUNTER__ > >::type,
+
+#else
+
+#define TEST_CASE_NOP()  std::add_rvalue_reference< Arg< __COUNTER__ > >{},
+#define TEST_CASE_TYPE() typename std::add_rvalue_reference< Arg< __COUNTER__ > >::type,
+
+#endif
+
+int sink(...);
+
+int x = sink(
+  REPEAT_10000(TEST_CASE_NOP)
+  REPEAT_10000(TEST_CASE_NOP) 42
+);
+
+void Foo( REPEAT_10000(TEST_CASE_TYPE) int) { }
+
+void escape() {
+
+sink(&x);
+sink(&Foo);
+}
+
+

libcxx/test/libcxx/utilities/meta/stress_tests/stress_test_remove_reference.sh.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This is a dummy feature that prevents this test from running by default.
+
+// The table below compares the compile time and object size for each of the
+// variants listed in the RUN script.
+//
+//  Impl                       Compile Time      Object Size
+// ----------------------------------------------------------
+// new_remove_reference:       22.849 s          121 K
+// std::remove_reference:      25.643 s          121 K
+//
+// RUN: %cxx %flags %compile_flags -c %s -o %S/orig.o -ggdb  -ggnu-pubnames -ftemplate-depth=5000 -ftime-trace -std=c++17
+// RUN: %cxx %flags %compile_flags -c %s -o %S/new.o -ggdb  -ggnu-pubnames -ftemplate-depth=5000 -ftime-trace -std=c++17 -DTEST_NEW
+
+#include <type_traits>
+#include <cassert>
+
+#include "test_macros.h"
+#include "template_cost_testing.h"
+
+template <int N> struct Arg { enum { value = 1 }; };
+
+#ifdef TEST_NEW
+
+template <class T>
+struct new_remove_reference
+{
+  typedef __remove_reference(T) type;
+};
+
+#define TEST_CASE_NOP()  new_remove_reference< Arg< __COUNTER__ > >{},
+#define TEST_CASE_TYPE() typename new_remove_reference< Arg< __COUNTER__ > >::type,
+
+#else
+
+#define TEST_CASE_NOP()  std::remove_reference< Arg< __COUNTER__ > >{},
+#define TEST_CASE_TYPE() typename std::remove_reference< Arg< __COUNTER__ > >::type,
+
+#endif
+
+int sink(...);
+
+int x = sink(
+  REPEAT_10000(TEST_CASE_NOP)
+  REPEAT_10000(TEST_CASE_NOP) 42
+);
+
+void Foo( REPEAT_10000(TEST_CASE_TYPE) int) { }
+
+void escape() {
+
+sink(&x);
+sink(&Foo);
+}
+
+