[Clang] Add __builtin_launder

[EricWF]

Summary

This patch adds __builtin_launder, which is required to implement std::launder. Additionally GCC provides __builtin_launder, so thing brings Clang in-line with GCC.

I'm not exactly sure what magic __builtin_launder requires, but based on previous discussions this patch applies a @llvm.invariant.group.barrier. As noted in previous discussions, this may not be enough to correctly handle vtables.

[EricWF]

• Remove incorrect FIXME comment.

[majnemer]

A test with restrict and __restrict might be interesting.

[jroelofs]

include/clang/Basic/Builtins.def

486

GCC's is type-generic:

#include <type_traits>

bool is_type_generic() {
  int v;
  return std::is_same<decltype(__builtin_launder(&v)),
                      decltype(&v)>::value;
}

However this BUILTIN line says that this one is not (yeah, I see it's being fixed up in Sema later in this patch). I think you need to use custom type checking here via t.

[EricWF]

include/clang/Basic/Builtins.def

486

The t is specified. It's in the third argument.

[jroelofs]

include/clang/Basic/Builtins.def

486

ohhh. duh. sorry.

[EricWF]

• Fix argument initialization.
• Make constexpr.

[EricWF]

• Improve quality of tests.
• Format code.

[rsmith]

lib/CodeGen/CGBuiltin.cpp

1674

It would be nice to avoid this for types that contain no const subobjects / reference subobjects / vptrs. I think we can also omit this entirely if -fstrict-vtable-ptrs is disabled, since in that case we don't generate any invariant.group metadata.

I'd be OK with the former being left to a future change, but the latter should be part of this change so we don't generate unnecessarily-inefficient code in the default mode for uses of std::launder.

lib/Sema/SemaChecking.cpp

861–865

Please also check that the pointee type is an object type -- per [ptr.launder]p3, "the program is ill-formed if T is a function type or cv void", and we don't want our builtin to need to deal with such cases.

[EricWF]

lib/CodeGen/CGBuiltin.cpp

1674

I'll put it in this change set.

[EricWF]

• Address inline comments about missing diagnostics for void pointers and function pointers.
• Address inline comments about only enabling when -fstrict-vtable-pointers is specified, and only on types with vtables.

[EricWF]

• Improve diagnostic handling.

[EricWF]

Ping.

[EricWF]

• Rebase off master.

[EricWF]

Ping.

[rsmith]

lib/CodeGen/CGBuiltin.cpp

1947–1948

I think you also need to catch class types that contain dynamic classes as subobjects.

lib/Sema/SemaChecking.cpp

861

Might be a bit clearer to use llvm::Optional<unsigned> as the type of this.

test/CodeGenCXX/builtin-launder.cpp

94–97

I would note that this means adding optimizations for those cases later is an LTO ABI break. That's probably OK, but just something we're going to need to remember.

[EricWF]

lib/CodeGen/CGBuiltin.cpp

1947–1948

Only by-value subobjects, or also reference subobjects?

[EricWF]

test/CodeGenCXX/builtin-launder.cpp

94–97

I added your note almost verbatim to the test case.

[EricWF]

• Launder types with subobjects of dynamic class type.
• Improve diagnostic selection using llvm::Optional<unsigned>.
• Add comment about LTO ABI concerns to test file.
• Merge with master.

[EricWF]

Ping.

[rsmith]

LGTM with a fix (and test) for pointer-to-array-of-dynamic-class-type handling.

lib/CodeGen/CGBuiltin.cpp

936–938

I think you also need to recurse through array types (use Ty->getBaseElementTypeUnsafe()->getAsCXXRecordDecl() or similar).

[rsmith]

Also we probably want to hold off on landing this until PR37458 is fixed, otherwise std::launder will start miscompiling code.

[EricWF]

• Handle array types as requested.

• Attempt to handle incomplete class types. They are always considered in need of laundering. Including class templates who's instantiation hasn't been forced yet. @rsmith is this the correct thing to do?

[erichkeane]

Bump! I don't see anything here that seems questionable (and this has extensive testing), but I presume @rsmith should be the final one to approve.

[Romain-Geissler-1A]

Hi,

Is there any news on this code review ? Is it ready to land ?

Cheers,
Romain

[EricWF]

Merge with upstream.

[EricWF]

In D40218#1298943, @Romain-Geissler-1A wrote:

Is there any news on this code review ? Is it ready to land ?

I think so, but there were some changes regarding incomplete types since Richard last looked at it.
I wanted him to sign off on the new behavior.

lib/CodeGen/CGBuiltin.cpp

1037

@rsmith I believe this changed since you approved the patch.

We talked about this offline, but could you give it a thumbs up?

[rsmith]

lib/CodeGen/CGBuiltin.cpp

1425–1426

I think it's a bug that launder doesn't require T to be a complete type. Can you file an LWG issue?

We should also decide whether we want to proactively fix this issue (require the type to be complete from the Sema checking of the builtin and assert that it's defined here) or not.

1437–1438

Delete this commented-out code.

1451

Would SmallPtrSet be a better choice here?

lib/Sema/SemaChecking.cpp

854

Did you mean to add this blank line?

859–861

Instead of performing some of the conversions here and some of them as part of initialization, I think it'd be more obvious to compute the builtin's parameter type here (which is the type of the argument if it's not of array [or function] type, and the decayed type of the argument otherwise), and do the decay and lvalue-to-rvalue conversion as part of the parameter initialization below.

The current code arrangement (and especially this comment) leaves a reader thinking "but you *need* an lvalue-to-rvalue conversion if the argument is an lvalue".

test/CodeGen/builtins.c

404–409

This test is not robust against minor IR differences such as variable or basic block names changing (some of these change in a release build), and is testing things that are not related to this builtin (eg, that we produce an alloca for a function parameter and its relative order to an alloca for a local variable).

I would remove everything here other than the load and the store, and add an explicit check that we don't generate a launder call:

// CHECK: [[TMP:%.*]] = load i32*,
// CHECK-NOT: @llvm.launder
// CHECK: store i32* [[TMP]],

test/CodeGenCXX/builtin-launder.cpp

17

This is likewise likely to fail with a release build of clang.

test/SemaCXX/builtins.cpp

120

&i doesn't what?

[EricWF]

lib/CodeGen/CGBuiltin.cpp

1425–1426

Apparently I misread the specification. The blanket wording in [res.on.functions] already prohibits this.

Though it's worth noting that GCC doesn't enforce this requirement (nor the one about function pointers).

1451

I have no idea. I'm ignorant to the internals of both containers. Since you asked the question I'll assume it is and make the change.
If you would like, I can do further investigation upon request.

lib/Sema/SemaChecking.cpp

854

Oh Richard, you're too kind. Of course I didn't mean to do that.

859–861

Ack.

I think I've implemented what you requested. Could you please verify?

test/CodeGen/builtins.c

404–409

If you're referring to the behavior of discarding value names, Clang CC1 only does that if -discard-value-names is explicitly passed, regardless of how the compiler is built, and so the names should be stable. (The Clang driver still conditionally passes -discard-value-names in release builds).

Are there other IR changes you have in mind?

None the less, I'll simply this test as requested.

test/CodeGenCXX/builtin-launder.cpp

17

I just verified that they pass in both release and debug builds. I previously changed Clang and LLVM to make the label names consistent between the two configurations for the purposes of testing.

I'll still attempt to clean unnecessary checks up.

test/SemaCXX/builtins.cpp

120

That's a great question. Not sure what I was up to when I wrote that. I tried a couple of things I though I might have been thinking, but they all passed.

Removing the comment.

[EricWF]

Address review comments.

@rsmith I think this is good to go. Just need your input on one change.

[EricWF]

Merging with upstream. Preparing to commit.

include/clang/Basic/Builtins.def

 BUILTIN(__builtin_setjmp, "iv**", "j")
 BUILTIN(__builtin_longjmp, "vv**i", "r")
 BUILTIN(__builtin_unwind_init, "v", "")
 BUILTIN(__builtin_eh_return_data_regno, "iIi", "nc")
 BUILTIN(__builtin_snprintf, "ic*zcC*.", "nFp:2:")
 BUILTIN(__builtin_vsprintf, "ic*cC*a", "nFP:1:")
 BUILTIN(__builtin_vsnprintf, "ic*zcC*a", "nFP:2:")
 BUILTIN(__builtin_thread_pointer, "v*", "nc")
+BUILTIN(__builtin_launder, "v*v*", "nt")
 
 // GCC exception builtins
 BUILTIN(__builtin_eh_return, "vzv*", "r") // FIXME: Takes intptr_t, not size_t!
 BUILTIN(__builtin_frob_return_addr, "v*v*", "n")
 BUILTIN(__builtin_dwarf_cfa, "v*", "n")
 BUILTIN(__builtin_init_dwarf_reg_size_table, "vv*", "n")
 BUILTIN(__builtin_dwarf_sp_column, "Ui", "n")
 BUILTIN(__builtin_extend_pointer, "ULLiv*", "n") // _Unwind_Word == uint64_t

include/clang/Basic/DiagnosticSemaKinds.td

   InGroup<GccCompat>;
 
 def warn_shadow_field :
   Warning<"non-static data member %0 of %1 shadows member inherited from "
   "type %2">,
   InGroup<ShadowField>, DefaultIgnore;
 def note_shadow_field : Note<"declared here">;
 
+def err_builtin_launder_invalid_arg : Error<
+  "%select{non-pointer|function pointer|void pointer}0 argument to "
+  "'__builtin_launder' is not allowed">;
+
 def err_target_required_in_redecl : Error<
   "function declaration is missing 'target' attribute in a multiversioned "
   "function">;
 def note_multiversioning_caused_here : Note<
   "function multiversioning caused by this declaration">;
 def err_multiversion_after_used : Error<
   "function declaration cannot become a multiversioned function after first "
   "usage">;

lib/AST/ExprConstant.cpp

                      diag::note_constexpr_baa_value_insufficient_alignment))
         << (int)OffsetResult.Offset.getQuantity()
         << (unsigned)Align.getQuantity();
       return false;
     }
 
     return true;
   }
-
+  case Builtin::BI__builtin_launder:
+    return evaluatePointer(E->getArg(0), Result);
   case Builtin::BIstrchr:
   case Builtin::BIwcschr:
   case Builtin::BImemchr:
   case Builtin::BIwmemchr:
     if (Info.getLangOpts().CPlusPlus11)
       Info.CCEDiag(E, diag::note_constexpr_invalid_function)
         << /*isConstexpr*/0 << /*isConstructor*/0
         << (std::string("'") + Info.Ctx.BuiltinInfo.getName(BuiltinOp) + "'");

lib/CodeGen/CGBuiltin.cpp

       ResultArg->getType()->getPointeeType().isVolatileQualified();
   CGF.Builder.CreateStore(CGF.EmitToMemory(Result, ResultQTy), ResultPtr,
                           isVolatile);
   return RValue::get(Overflow);
 }
 
 static llvm::Value *dumpRecord(CodeGenFunction &CGF, QualType RType,
                                Value *&RecordPtr, CharUnits Align, Value *Func,
                                int Lvl) {
   const auto *RT = RType->getAs<RecordType>();
   ASTContext &Context = CGF.getContext();
   RecordDecl *RD = RT->getDecl()->getDefinition();
   ASTContext &Ctx = RD->getASTContext();
   const ASTRecordLayout &RL = Ctx.getASTRecordLayout(RD);
   std::string Pad = std::string(Lvl * 4, ' ');
 
   Value *GString =
       CGF.Builder.CreateGlobalStringPtr(RType.getAsString() + " {\n");
   Value *Res = CGF.Builder.CreateCall(Func, {GString});

   }
 
   GString = CGF.Builder.CreateGlobalStringPtr(Pad + "}\n");
   Value *TmpRes = CGF.Builder.CreateCall(Func, {GString});
   Res = CGF.Builder.CreateAdd(Res, TmpRes);
   return Res;
 }
 
+static bool TypeRequiresBuiltinLaunderImp(const ASTContext &Ctx, QualType Ty,
+                                          llvm::DenseSet<const Decl *> &Seen) {
+  if (const auto *Arr = Ctx.getAsArrayType(Ty))
+    Ty = Ctx.getBaseElementType(Arr);
+
+  const auto *Record = Ty->getAsCXXRecordDecl();
+  if (!Record)
+    return false;
+
+  // We've already checked this type, or are in the process of checking it.
+  if (!Seen.insert(Record).second)
+    return false;
+
+  // FIXME: We either have an incomplete class type, or we have a class template
+  // whose instantiation has not been forced. Example:
+  //
+  //   template <class T> struct Foo { T value; };
+  //   Foo<int> *p = nullptr;
+  //   auto *d = __builtin_launder(p);
+  if (!Record->hasDefinition())
+    return true;
+
+  if (Record->isDynamicClass())
+    return true;
+
+  // if (!Seen.insert(Record).second)
+  //  return false;
+  for (FieldDecl *F : Record->fields()) {
+    if (TypeRequiresBuiltinLaunderImp(Ctx, F->getType(), Seen))
+      return true;
+  }
+  return false;
+}
+
+/// Determine if the specified type requires laundering by checking if it is a
+/// dynamic class type or contains a subobject which is a dynamic class type.
+static bool TypeRequiresBuiltinLaunder(CodeGenModule &CGM, QualType Ty) {
+  if (!CGM.getCodeGenOpts().StrictVTablePointers)
+    return false;
+  llvm::DenseSet<const Decl *> Seen;
+  return TypeRequiresBuiltinLaunderImp(CGM.getContext(), Ty, Seen);
+}
+
 RValue CodeGenFunction::EmitBuiltinExpr(const FunctionDecl *FD,
                                         unsigned BuiltinID, const CallExpr *E,
                                         ReturnValueSlot ReturnValue) {
   // See if we can constant fold this builtin.  If so, don't emit it at all.
   Expr::EvalResult Result;
   if (E->EvaluateAsRValue(Result, CGM.getContext()) &&
       !Result.hasSideEffects()) {
     if (Result.Val.isInt())

     Value *F = CGM.getIntrinsic(Intrinsic::ctpop, ArgType);
 
     llvm::Type *ResultType = ConvertType(E->getType());
     Value *Tmp = Builder.CreateCall(F, ArgValue);
     Value *Result = Builder.CreateAnd(Tmp, llvm::ConstantInt::get(ArgType, 1));
     if (Result->getType() != ResultType)
       Result = Builder.CreateIntCast(Result, ResultType, /*isSigned*/true,
                                      "cast");
     return RValue::get(Result);
   }
   case Builtin::BI__popcnt16:
   case Builtin::BI__popcnt:
   case Builtin::BI__popcnt64:
   case Builtin::BI__builtin_popcount:
   case Builtin::BI__builtin_popcountl:
   case Builtin::BI__builtin_popcountll: {
     Value *ArgValue = EmitScalarExpr(E->getArg(0));
 
     llvm::Type *ArgType = ArgValue->getType();
     Value *F = CGM.getIntrinsic(Intrinsic::ctpop, ArgType);
 
     llvm::Type *ResultType = ConvertType(E->getType());
     Value *Result = Builder.CreateCall(F, ArgValue);
     if (Result->getType() != ResultType)
       Result = Builder.CreateIntCast(Result, ResultType, /*isSigned*/true,
                                      "cast");
     return RValue::get(Result);
   }
   case Builtin::BI_rotr8:
   case Builtin::BI_rotr16:
   case Builtin::BI_rotr:
   case Builtin::BI_lrotr:
   case Builtin::BI_rotr64: {
     Value *Val = EmitScalarExpr(E->getArg(0));
     Value *Shift = EmitScalarExpr(E->getArg(1));
 
     llvm::Type *ArgType = Val->getType();
     Shift = Builder.CreateIntCast(Shift, ArgType, false);
     unsigned ArgWidth = ArgType->getIntegerBitWidth();
     Value *Mask = llvm::ConstantInt::get(ArgType, ArgWidth - 1);
 
     Value *RightShiftAmt = Builder.CreateAnd(Shift, Mask);
     Value *RightShifted = Builder.CreateLShr(Val, RightShiftAmt);

     Value *NegativeOne = ConstantInt::get(IntTy, -1);
     Value *SignResult = Builder.CreateSelect(IsNeg, NegativeOne, One);
     Value *Result = Builder.CreateSelect(IsInf, SignResult, Zero);
     return RValue::get(Result);
   }
 
   case Builtin::BI__builtin_isnormal: {
     // isnormal(x) --> x == x && fabsf(x) < infinity && fabsf(x) >= float_min
     Value *V = EmitScalarExpr(E->getArg(0));
     Value *Eq = Builder.CreateFCmpOEQ(V, V, "iseq");
 
     Value *Abs = EmitFAbs(*this, V);
     Value *IsLessThanInf =
       Builder.CreateFCmpULT(Abs, ConstantFP::getInfinity(V->getType()),"isinf");
     APFloat Smallest = APFloat::getSmallestNormalized(
                    getContext().getFloatTypeSemantics(E->getArg(0)->getType()));
     Value *IsNormal =

     // all that the backend can't reasonably determine this while
     // lowering llvm.eh.dwarf.cfa()?
     //
     // TODO: If there's a satisfactory reason, add a target hook for
     // this instead of hard-coding 0, which is correct for most targets.
     int32_t Offset = 0;
 
     Value *F = CGM.getIntrinsic(Intrinsic::eh_dwarf_cfa);
     return RValue::get(Builder.CreateCall(F,
                                       llvm::ConstantInt::get(Int32Ty, Offset)));
   }
   case Builtin::BI__builtin_return_address: {
     Value *Depth = ConstantEmitter(*this).emitAbstract(E->getArg(0),
                                                    getContext().UnsignedIntTy);
     Value *F = CGM.getIntrinsic(Intrinsic::returnaddress);
     return RValue::get(Builder.CreateCall(F, Depth));
   }
   case Builtin::BI_ReturnAddress: {

     // longjmp doesn't return; mark this as unreachable.
     Builder.CreateUnreachable();
 
     // We do need to preserve an insertion point.
     EmitBlock(createBasicBlock("longjmp.cont"));
 
     return RValue::get(nullptr);
   }
+  case Builtin::BI__builtin_launder: {
+    const Expr *Arg = E->getArg(0);
+    QualType ArgTy = Arg->getType()->getPointeeType();
+    Value *Ptr = EmitScalarExpr(Arg);
+    if (TypeRequiresBuiltinLaunder(CGM, ArgTy))
+      Ptr = Builder.CreateLaunderInvariantGroup(Ptr);
+
+    return RValue::get(Ptr);
+  }
   case Builtin::BI__sync_fetch_and_add:
   case Builtin::BI__sync_fetch_and_sub:
   case Builtin::BI__sync_fetch_and_or:
   case Builtin::BI__sync_fetch_and_and:
   case Builtin::BI__sync_fetch_and_xor:
   case Builtin::BI__sync_fetch_and_nand:
   case Builtin::BI__sync_add_and_fetch:
   case Builtin::BI__sync_sub_and_fetch:

lib/Sema/SemaChecking.cpp

default:
     llvm_unreachable("Invalid builtin function");
   }
   Call->setType(S.Context.getPointerType(S.Context.getQualifiedType(
       RT.getUnqualifiedType(), Qual)));
 
   return false;
 }
 
+static ExprResult SemaBuiltinLaunder(Sema &S, CallExpr *TheCall) {
+  if (checkArgCount(S, TheCall, 1))
+    return ExprError();
+
+  Expr *OrigArg = TheCall->getArg(0);
+  // Don't perform LValue conversions since they may strip things like the
+  // restrict qualifier
+  ExprResult Arg = S.DefaultFunctionArrayConversion(OrigArg);
+  if (Arg.isInvalid())
+    return ExprError();
+
+  TheCall->setArg(0, Arg.get());
+  QualType ArgTy = Arg.get()->getType();
+  TheCall->setType(ArgTy);
+
+  auto DiagSelect = [&]() -> llvm::Optional<unsigned> {
+    if (!ArgTy->isPointerType())
+      return 0;
+    if (ArgTy->isFunctionPointerType())
+      return 1;
+    if (ArgTy->isVoidPointerType())
+      return 2;
+    return llvm::Optional<unsigned>{};
+  }();
+  if (DiagSelect.hasValue()) {
+    S.Diag(TheCall->getLocStart(), diag::err_builtin_launder_invalid_arg)
+        << DiagSelect.getValue() << TheCall->getSourceRange();
+    return ExprError();
+  }
+
+  assert(ArgTy->getPointeeType()->isObjectType() && "Unhandled non-object pointer case");
+
+  InitializedEntity Entity =
+      InitializedEntity::InitializeParameter(S.Context, ArgTy, false);
+  Arg = TheCall->getArg(0);
+  Arg = S.PerformCopyInitialization(Entity, SourceLocation(), Arg);
+  if (Arg.isInvalid())
+    return ExprError();
+  TheCall->setArg(0, Arg.get());
+
+  return TheCall;
+}
+
 ExprResult
 Sema::CheckBuiltinFunctionCall(FunctionDecl *FDecl, unsigned BuiltinID,
                                CallExpr *TheCall) {
   ExprResult TheCallResult(TheCall);
 
   // Find out if any arguments are required to be integer constant expressions.
   unsigned ICEArguments = 0;
   ASTContext::GetBuiltinTypeError Error;
Sema::CheckBuiltinFunctionCall(FunctionDecl *FDecl, unsigned BuiltinID,
   case Builtin::BI__builtin_classify_type:
     if (checkArgCount(*this, TheCall, 1)) return true;
     TheCall->setType(Context.IntTy);
     break;
   case Builtin::BI__builtin_constant_p:
     if (checkArgCount(*this, TheCall, 1)) return true;
     TheCall->setType(Context.IntTy);
     break;
+  case Builtin::BI__builtin_launder:
+    return SemaBuiltinLaunder(*this, TheCall);
   case Builtin::BI__sync_fetch_and_add:
   case Builtin::BI__sync_fetch_and_add_1:
   case Builtin::BI__sync_fetch_and_add_2:
   case Builtin::BI__sync_fetch_and_add_4:
   case Builtin::BI__sync_fetch_and_add_8:
   case Builtin::BI__sync_fetch_and_add_16:
   case Builtin::BI__sync_fetch_and_sub:
   case Builtin::BI__sync_fetch_and_sub_1:

test/CodeGen/builtins.c

#define V(n,args) p(#n #args, (__builtin_##n args, 0))
   P(bitreverse64, (N));
 
   // FIXME
   // V(clear_cache, (&N, &N+1));
   V(trap, ());
   R(extract_return_addr, (&N));
   P(signbit, (1.0));
 
+  R(launder, (&N));
+
   return 0;
 }
 
 
 
 void foo() {
  __builtin_strcat(0, 0);
 }

 #endif
 
 // CHECK-LABEL: define i64 @test_builtin_readcyclecounter
 long long test_builtin_readcyclecounter() {
   // CHECK: call i64 @llvm.readcyclecounter()
   return __builtin_readcyclecounter();
 }
 
+/// __builtin_launder should be a NOP in C since there are no vtables.
+// CHECK-LABEL: define void @test_builtin_launder
+void test_builtin_launder(int *p) {
+  // CHECK: entry
+  // CHECK-NEXT: %p.addr = alloca i32*
+  // CHECK-NEXT: %d = alloca i32*
+  // CHECK-NEXT: store i32* %p, i32** %p.addr, align 8
+  // CHECK-NEXT: [[TMP:%.*]] = load i32*, i32** %p.addr
+  // CHECK-NEXT: store i32* [[TMP]], i32** %d
+  int *d = __builtin_launder(p);
+}
+
 // Behavior of __builtin_os_log differs between platforms, so only test on X86
 #ifdef __x86_64__
 
 // CHECK-LABEL: define void @test_builtin_os_log
 // CHECK: (i8* %[[BUF:.*]], i32 %[[I:.*]], i8* %[[DATA:.*]])
 void test_builtin_os_log(void *buf, int i, const char *data) {
   volatile int len;
   // CHECK: %[[BUF_ADDR:.*]] = alloca i8*, align 8

test/CodeGenCXX/builtin-launder.cpp

+// RUN: %clang_cc1 -triple=x86_64-linux-gnu -emit-llvm -fstrict-vtable-pointers -o - %s \
+// RUN: | FileCheck --check-prefixes=CHECK,CHECK-STRICT %s
+// RUN: %clang_cc1 -triple=x86_64-linux-gnu -emit-llvm -o - %s \
+// RUN: | FileCheck --check-prefixes=CHECK,CHECK-NONSTRICT %s
+
+//===----------------------------------------------------------------------===//
+//                            Positive Cases
+//===----------------------------------------------------------------------===//
+
+struct TestVirtualFn {
+  virtual void foo() {}
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_virtual_fn
+extern "C" void test_builtin_launder_virtual_fn(TestVirtualFn *p) {
+  // CHECK: entry
+  // CHECK-NEXT: %p.addr = alloca [[TYPE:%.*]], align 8
+  // CHECK-NEXT: %d = alloca [[TYPE]]
+  // CHECK-NEXT: store [[TYPE]] %p, [[TYPE]]* %p.addr
+  // CHECK-NEXT: [[TMP0:%.*]] = load [[TYPE]], [[TYPE]]* %p.addr
+
+  // CHECK-NONSTRICT-NEXT: store [[TYPE]] [[TMP0]], [[TYPE]]* %d
+
+  // CHECK-STRICT-NEXT: [[TMP1:%.*]] = bitcast [[TYPE]] [[TMP0]] to i8*
+  // CHECK-STRICT-NEXT: [[TMP2:%.*]] = call i8* @llvm.launder.invariant.group.p0i8(i8* [[TMP1]])
+  // CHECK-STRICT-NEXT: [[TMP3:%.*]] = bitcast i8* [[TMP2]] to [[TYPE]]
+  // CHECK-STRICT-NEXT: store [[TYPE]] [[TMP3]], [[TYPE]]* %d
+
+  // CHECK-NEXT: ret void
+  TestVirtualFn *d = __builtin_launder(p);
+}
+
+struct TestPolyBase : TestVirtualFn {
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_poly_base
+extern "C" void test_builtin_launder_poly_base(TestPolyBase *p) {
+  // CHECK-STRICT-NOT: ret void
+  // CHECK-STRICT: @llvm.launder.invariant.group
+
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+
+  // CHECK: ret void
+  TestPolyBase *d = __builtin_launder(p);
+}
+
+struct TestBase {};
+struct TestVirtualBase : virtual TestBase {};
+
+// CHECK-LABEL: define void @test_builtin_launder_virtual_base
+extern "C" void test_builtin_launder_virtual_base(TestVirtualBase *p) {
+  // CHECK-STRICT-NOT: ret void
+  // CHECK-STRICT: @llvm.launder.invariant.group
+
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+
+  // CHECK: ret void
+  TestVirtualBase *d = __builtin_launder(p);
+}
+
+//===----------------------------------------------------------------------===//
+//                            Negative Cases
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: define void @test_builtin_launder_ommitted_one
+extern "C" void test_builtin_launder_ommitted_one(int *p) {
+  // CHECK: entry
+  // CHECK-NEXT: %p.addr = alloca i32*
+  // CHECK-NEXT: %d = alloca i32*
+  // CHECK-NEXT: store i32* %p, i32** %p.addr, align 8
+  // CHECK-NEXT: [[TMP:%.*]] = load i32*, i32** %p.addr
+  // CHECK-NEXT: store i32* [[TMP]], i32** %d
+  // CHECK-NEXT: ret void
+  int *d = __builtin_launder(p);
+}
+
+struct TestNoInvariant {
+  int x;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_ommitted_two
+extern "C" void test_builtin_launder_ommitted_two(TestNoInvariant *p) {
+  // CHECK: entry
+  // CHECK-NOT: llvm.launder.invariant.group
+  // CHECK-NEXT: %p.addr = alloca [[TYPE:%.*]], align 8
+  // CHECK-NEXT: %d = alloca [[TYPE]]
+  // CHECK-NEXT: store [[TYPE]] %p, [[TYPE]]* %p.addr
+  // CHECK-NEXT: [[TMP:%.*]] = load [[TYPE]], [[TYPE]]* %p.addr
+  // CHECK-NEXT: store [[TYPE]] [[TMP]], [[TYPE]]* %d
+  // CHECK-NEXT: ret void
+  TestNoInvariant *d = __builtin_launder(p);
+}
+
+struct TestVirtualMember {
+  TestVirtualFn member;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_virtual_member
+extern "C" void test_builtin_launder_virtual_member(TestVirtualMember *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestVirtualMember *d = __builtin_launder(p);
+}
+
+struct TestVirtualMemberDepth2 {
+  TestVirtualMember member;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_virtual_member_depth_2
+extern "C" void test_builtin_launder_virtual_member_depth_2(TestVirtualMemberDepth2 *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestVirtualMemberDepth2 *d = __builtin_launder(p);
+}
+
+struct TestVirtualReferenceMember {
+  TestVirtualFn &member;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_virtual_reference_member
+extern "C" void test_builtin_launder_virtual_reference_member(TestVirtualReferenceMember *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestVirtualReferenceMember *d = __builtin_launder(p);
+}
+
+struct TestRecursiveMember {
+  TestRecursiveMember() : member(*this) {}
+  TestRecursiveMember &member;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_recursive_member
+extern "C" void test_builtin_launder_recursive_member(TestRecursiveMember *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestRecursiveMember *d = __builtin_launder(p);
+}
+
+struct TestVirtualRecursiveMember {
+  TestVirtualRecursiveMember() : member(*this) {}
+  TestVirtualRecursiveMember &member;
+  virtual void foo();
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_virtual_recursive_member
+extern "C" void test_builtin_launder_virtual_recursive_member(TestVirtualRecursiveMember *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestVirtualRecursiveMember *d = __builtin_launder(p);
+}
+
+// CHECK-LABEL: define void @test_builtin_launder_array(
+extern "C" void test_builtin_launder_array(TestVirtualFn (&Arr)[5]) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestVirtualFn *d = __builtin_launder(Arr);
+}
+
+// CHECK-LABEL: define void @test_builtin_launder_array_nested(
+extern "C" void test_builtin_launder_array_nested(TestVirtualFn (&Arr)[5][2]) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  using RetTy = TestVirtualFn(*)[2];
+  RetTy d = __builtin_launder(Arr);
+}
+
+// CHECK-LABEL: define void @test_builtin_launder_array_no_invariant(
+extern "C" void test_builtin_launder_array_no_invariant(TestNoInvariant (&Arr)[5]) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestNoInvariant *d = __builtin_launder(Arr);
+}
+
+// CHECK-LABEL: define void @test_builtin_launder_array_nested_no_invariant(
+extern "C" void test_builtin_launder_array_nested_no_invariant(TestNoInvariant (&Arr)[5][2]) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  using RetTy = TestNoInvariant(*)[2];
+  RetTy d = __builtin_launder(Arr);
+}
+
+template <class Member>
+struct WithMember {
+  Member mem;
+};
+
+template struct WithMember<TestVirtualFn[5]>;
+
+// CHECK-LABEL: define void @test_builtin_launder_member_array(
+extern "C" void test_builtin_launder_member_array(WithMember<TestVirtualFn[5]> *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  auto *d = __builtin_launder(p);
+}
+
+template struct WithMember<TestVirtualFn[5][2]>;
+
+// CHECK-LABEL: define void @test_builtin_launder_member_array_nested(
+extern "C" void test_builtin_launder_member_array_nested(WithMember<TestVirtualFn[5][2]> *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  auto *d = __builtin_launder(p);
+}
+
+template struct WithMember<TestNoInvariant[5]>;
+
+// CHECK-LABEL: define void @test_builtin_launder_member_array_no_invariant(
+extern "C" void test_builtin_launder_member_array_no_invariant(WithMember<TestNoInvariant[5]> *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  auto *d = __builtin_launder(p);
+}
+
+template struct WithMember<TestNoInvariant[5][2]>;
+
+// CHECK-LABEL: define void @test_builtin_launder_member_array_nested_no_invariant(
+extern "C" void test_builtin_launder_member_array_nested_no_invariant(WithMember<TestNoInvariant[5][2]> *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  auto *d = __builtin_launder(p);
+}
+
+// FIXME: Because WithMember<TestNoInvariant> is never instantiated, we have to assume
+// it requires laundering.
+// CHECK-LABEL: define void @test_builtin_launder_uninstantiated_template(
+extern "C" void test_builtin_launder_uninstantiated_template(WithMember<TestNoInvariant> *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  auto *d = __builtin_launder(p);
+}
+
+struct Incomplete;
+// CHECK-LABEL: define void @test_builtin_launder_incomplete(
+extern "C" void test_builtin_launder_incomplete(Incomplete *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  Incomplete *d = __builtin_launder(p);
+}
+
+struct LaterComplete;
+
+// FIXME: Can/Should we dig out the definition for LaterComplete when performing
+// CodeGen?
+
+// CHECK-LABEL: define void @test_builtin_launder_complete_later(
+extern "C" void test_builtin_launder_complete_later(LaterComplete *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  LaterComplete *d = __builtin_launder(p);
+}
+struct LaterComplete {};
+
+template <class T>
+struct WithBase : T {};
+
+template struct WithBase<TestNoInvariant>;
+
+// CHECK-LABEL: define void @test_builtin_launder_base_no_invariant(
+extern "C" void test_builtin_launder_base_no_invariant(WithBase<TestNoInvariant> *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  auto *d = __builtin_launder(p);
+}
+
+template struct WithBase<TestVirtualFn>;
+
+// CHECK-LABEL: define void @test_builtin_launder_base(
+extern "C" void test_builtin_launder_base(WithBase<TestVirtualFn> *p) {
+  // CHECK: entry
+  // CHECK-NONSTRICT-NOT: @llvm.launder.invariant.group
+  // CHECK-STRICT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  auto *d = __builtin_launder(p);
+}
+
+/// The test cases in this namespace technically need to be laundered according
+/// to the language in the standard (ie they have const or reference subobjects)
+/// but LLVM doesn't currently optimize on these cases -- so Clang emits
+/// __builtin_launder as a nop.
+///
+/// NOTE: Adding optimizations for these cases later is an LTO ABI break. That's
+/// probably OK for now -- but is something to keep in mind.
+namespace pessimizing_cases {
+
+struct TestConstMember {
+  const int x;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_const_member
+extern "C" void test_builtin_launder_const_member(TestConstMember *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestConstMember *d = __builtin_launder(p);
+}
+
+struct TestConstSubobject {
+  TestConstMember x;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_const_subobject
+extern "C" void test_builtin_launder_const_subobject(TestConstSubobject *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestConstSubobject *d = __builtin_launder(p);
+}
+
+struct TestConstObject {
+  const struct TestConstMember x;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_const_object
+extern "C" void test_builtin_launder_const_object(TestConstObject *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestConstObject *d = __builtin_launder(p);
+}
+
+struct TestReferenceMember {
+  int &x;
+};
+
+// CHECK-LABEL: define void @test_builtin_launder_reference_member
+extern "C" void test_builtin_launder_reference_member(TestReferenceMember *p) {
+  // CHECK: entry
+  // CHECK-NOT: @llvm.launder.invariant.group
+  // CHECK: ret void
+  TestReferenceMember *d = __builtin_launder(p);
+}
+
+} // namespace pessimizing_cases

test/Preprocessor/feature_tests.c

 // RUN: %clang_cc1 %s -triple=i686-apple-darwin9 -verify -DVERIFY
 // RUN: %clang_cc1 %s -E -triple=i686-apple-darwin9
 #ifndef __has_feature
 #error Should have __has_feature
 #endif
 
 
 #if __has_feature(something_we_dont_have)
 #error Bad
 #endif
 
 #if  !__has_builtin(__builtin_huge_val) || \
      !__has_builtin(__builtin_shufflevector) || \
      !__has_builtin(__builtin_convertvector) || \
      !__has_builtin(__builtin_trap) || \
      !__has_builtin(__c11_atomic_init) || \
+     !__has_builtin(__builtin_launder) || \
      !__has_feature(attribute_analyzer_noreturn) || \
      !__has_feature(attribute_overloadable)
 #error Clang should have these
 #endif
 
 #if __has_builtin(__builtin_insanity)
 #error Clang should not have this
 #endif

test/Sema/builtins.c

char * Test20(char *p, const char *in, unsigned n)
 
     __builtin___memcpy_chk (&buf[5], "abcde", n, __builtin_object_size (&buf[5], 0));
 
     __builtin___memcpy_chk (&buf[6], "abcde", 5, __builtin_object_size (&buf[6], 0)); // expected-warning {{'__builtin___memcpy_chk' will always overflow destination buffer}}
 
     return buf;
 }
 
+typedef void (fn_t)(int);
+
+void test_builtin_launder(char *p, void *vp, const void *cvp,
+                          const volatile int *ip, float *restrict fp,
+                          fn_t *fn) {
+  __builtin_launder(); // expected-error {{too few arguments to function call, expected 1, have 0}}
+  __builtin_launder(p, p); // expected-error {{too many arguments to function call, expected 1, have 2}}
+  int x;
+  __builtin_launder(x); // expected-error {{non-pointer argument to '__builtin_launder' is not allowed}}
+  char *d = __builtin_launder(p);
+  __builtin_launder(vp);  // expected-error {{void pointer argument to '__builtin_launder' is not allowed}}
+  __builtin_launder(cvp); // expected-error {{void pointer argument to '__builtin_launder' is not allowed}}
+  const volatile int *id = __builtin_launder(ip);
+  int *id2 = __builtin_launder(ip); // expected-warning {{discards qualifiers}}
+  float *fd = __builtin_launder(fp);
+  __builtin_launder(fn); // expected-error {{function pointer argument to '__builtin_launder' is not allowed}}
+}
+
 void test21(const int *ptr) {
   __sync_fetch_and_add(ptr, 1); // expected-error{{address argument to atomic builtin cannot be const-qualified ('const int *' invalid)}}
   __atomic_fetch_add(ptr, 1, 0);  // expected-error {{address argument to atomic operation must be a pointer to non-const type ('const int *' invalid)}}
 }

test/SemaCXX/builtins.cpp

 
 struct FILE;
 extern "C" int vfprintf(FILE *__restrict, const char *__restrict,
                         __builtin_va_list va);
 
 void synchronize_args() {
   __sync_synchronize(0); // expected-error {{too many arguments}}
 }
+
+namespace test_launder {
+
+struct Dummy {};
+
+using FnType = int(char);
+using MemFnType = int (Dummy::*)(char);
+using ConstMemFnType = int (Dummy::*)() const;
+
+void foo() {}
+
+void test_builtin_launder_diags(void *vp, const void *cvp, FnType *fnp,
+                                MemFnType mfp, ConstMemFnType cmfp, int (&Arr)[5]) {
+  __builtin_launder(vp);   // expected-error {{void pointer argument to '__builtin_launder' is not allowed}}
+  __builtin_launder(cvp);  // expected-error {{void pointer argument to '__builtin_launder' is not allowed}}
+  __builtin_launder(fnp);  // expected-error {{function pointer argument to '__builtin_launder' is not allowed}}
+  __builtin_launder(mfp);  // expected-error {{non-pointer argument to '__builtin_launder' is not allowed}}
+  __builtin_launder(cmfp); // expected-error {{non-pointer argument to '__builtin_launder' is not allowed}}
+  (void)__builtin_launder(&fnp);
+  __builtin_launder(42);      // expected-error {{non-pointer argument to '__builtin_launder' is not allowed}}
+  __builtin_launder(nullptr); // expected-error {{non-pointer argument to '__builtin_launder' is not allowed}}
+  __builtin_launder(foo);     // expected-error {{function pointer argument to '__builtin_launder' is not allowed}}
+  (void)__builtin_launder(Arr);
+}
+
+void test_builtin_launder(char *p, const volatile int *ip, const float *&fp,
+                          double *__restrict dp) {
+  int x;
+  __builtin_launder(x); // expected-error {{non-pointer argument to '__builtin_launder' is not allowed}}
+#define TEST_TYPE(Ptr, Type) \
+  static_assert(__is_same(decltype(__builtin_launder(Ptr)), Type), "expected same type")
+  TEST_TYPE(p, char*);
+  TEST_TYPE(ip, const volatile int*);
+  TEST_TYPE(fp, const float*);
+  TEST_TYPE(dp, double *__restrict);
+#undef TEST_TYPE
+  char *d = __builtin_launder(p);
+  const volatile int *id = __builtin_launder(ip);
+  int *id2 = __builtin_launder(ip); // expected-error {{cannot initialize a variable of type 'int *' with an rvalue of type 'const volatile int *'}}
+  const float* fd = __builtin_launder(fp);
+}
+
+template <class Tp>
+constexpr Tp *test_constexpr_launder(Tp *tp) {
+  return __builtin_launder(tp);
+}
+constexpr int const_int = 42;
+constexpr int const_int2 = 101;
+constexpr const int *const_ptr = test_constexpr_launder(&const_int);
+static_assert(&const_int == const_ptr, "");
+static_assert(const_ptr != test_constexpr_launder(&const_int2), "");
+
+void test_non_constexpr() {
+  constexpr int i = 42;                            // expected-note {{declared here}}
+  constexpr const int *ip = __builtin_launder(&i); // expected-error {{constexpr variable 'ip' must be initialized by a constant expression}}
+  // expected-note@-1 {{pointer to 'i' is not a constant expression}}
+}
+
+constexpr bool test_in_constexpr(const int &i) {
+  return (__builtin_launder(&i) == &i);
+}
+static_assert(test_in_constexpr(const_int), "");
+void f() {
+  constexpr int i = 42;
+  // FIXME: Should this work? Since `&i` doesn't.
+  static_assert(test_in_constexpr(i), "");
+}
+
+} // end namespace test_launder