Introduce per-callsite inline intrinsics

[kuhar]

Summary

Traditionally, to force some inlining decisions one has to annotate function declarations with __attribute__((always_inline)) or __attribute__((noinline)). One problem with these attributes is that they affect every call site. Always inlining or forbidding inlining may not be desirable in every context, and a workaround for that is creating a few copies of functions, each with a different inline attribute. Furthermore, it's not always feasible (in every project) to modify library code and introduce new function attributes there.

This patch introduces a new way of forcing inlining decisions on a per-callsite basis. This allows for more fine-grained control over inlining, without creating any duplicate functions. The two new intrinsics for controlling inlining are:

• __builtin_always_inline(Foo()) -- inlines the function Foo at the callsite, if possible. Internally, this applies the alwaysinline attribute to the generated call instruction.
• __builtin_no_inline(Foo()) -- forbids the function Foo to be inlined at the callsite. Internally, this applies the noinline attribute to the generated call instruction.

The inline intrinsics support function, function pointer, member function, member function pointer, virutal function, and operator calls. Support for constructor calls (CXXTemporaryExpr) should also be possible, but is not the part of this patch.

[kuhar]

Disclaimer: I'm a Clang newbie and I'm not sure if that's a good way to implement these intrinsics. I'm not sure about the following things:

• The new enum CallInlineKind may not be in the right place
• Not sure if adding the extra parameter to EmitSomething methods is the cleanest thing possible -- some functions have it now, some don't, and it makes argument lists longer
• Not sure if just adding an extra attribute at each callsite is enough -- I'm not sure what is supposed to happen when there are conflicting inline attributes at call sites and function declarations (e.g. __builtin_always_inline and __attribute__ ((always_inline)))
• I don't know how to handle constexpr functions
• I don't know how to implement it for constructor calls

I would really appreciate any tips here :).

[kuhar]

One extra question: do you think this deserves a separate RFC?

[• Quuxplusone]

include/clang/Basic/DiagnosticSemaKinds.td

8236

I'd expect to be able to write

__builtin_always_inline(sqrt(x))
__builtin_no_inline(sqrt(x))

without caring whether sqrt was a real function or just a macro around __builtin_sqrt. How important is it that calls to builtin functions be errors, instead of just being ignored for this purpose?

8238

Spelling: s/dotr/dtor/
Semantics: Does this trigger in generic contexts? I feel like I'd want to be able to write

__builtin_always_inline(p->~T());

without caring whether T was a primitive type or not. How important is it that pseudo-destructor calls be errors, instead of just being ignored for this purpose?

include/clang/CodeGen/CGFunctionInfo.h

713

Here and throughout, wouldn't it be more traditional to name the parameter variable CIK? (You have sometimes CIK, sometimes inlineKind, sometimes InlineKind.)

It also feels needlessly convoluted to have a value of type CallInlineKind stored in a member named InlineCall (note the reversal of the words), but I'm not sure how that's generally dealt with.

test/SemaCXX/inline-builtins.cpp

13

This raises a question for me about the semantics of "always inlining" a "call":

struct A { A(); A(A&&); };
struct B { B(A); }
void foo(B);

__builtin_always_inline(foo(A{}));

Does __builtin_always_inline apply to foo, B(A) (used to create the parameter to foo), A() (used to create the argument to A(A&&)), or all of the above?
You should add a test case something like this, maybe with multiple function arguments.

72

Really good you thought of this case! But shouldn't this *not* be an error? 1_s is an operator call.

[kuhar]

Fix naming and add more tests.

include/clang/Basic/DiagnosticSemaKinds.td

8236

Very good point. Initially I thought this should be an error, but I think that since compiler can reason about intrinsics anyway, it makes sense to treat inline intrinsics as NoOps in this case.

8238

I thought about it an initially disliked the idea of of silently accepting code that cannot be inlined (because there's no function call).

This case is more general, because you can have the same problem with operator calls.
Consider a function template that does __builtin_always_inline(t + t). It makes sense to inline it for some custom type that implements the + operator, but it could be also used when t is e.g. an int.

Given that I the intrinsics already works on the best-effort basis (and doesn't not strictly guarantee that inlining/no inlining will happen; e.g., indirect calls), and that in the future I also want to introduce a third intrinsic for transitively inlining everything, I think it makes sense to treat it as as NoOp in generic context.

When the context is not generic, maybe it still would make sense to emit an error/warning? I don't like the idea of being able to write __builtin_always_inline(2 + 3) and the compiler not complaining at all... What do you think?

include/clang/CodeGen/CGFunctionInfo.h

713

Good suggestion, thanks!

test/SemaCXX/inline-builtins.cpp

13

It should apply to the outermost call, e.g.:
__builtin_always_inline(A(B(C()))) -- applies to A(...)
A(__builtin_always_inline(B(C()))) -- applies to B(...)
__builtin_always_inline(a.foo().bar()) -- applies to .bar()
__builtin_always_inline(a.foo()).bar() -- applies to .foo()

I added extra testcases for this. Ultimately, I believe that these intrinsic deserve a solid documentation on the clang www for builtins.

13

I also plan to have a third inline intrinsic: __builtin_flatten_inline that will (try to) transitively inline everything in parentheses.

72

This is simply not implemented yet. I added a TODO.
(From what I see, UDL don't look like operators on the AST level.)

[kuhar]

Rename missed CallInlineKind parameters to CIK

[rsmith]

Support for constructor calls (CXXTemporaryExpr) should also be possible, but is not the part of this patch.

(You should handle the base class (CXXConstructExpr) that describes the semantics, rather than the derived class (CXXTemporaryObjectExpr) that describes a particular syntactic form for said semantics.)

Have you considered whether the builtin should apply to new expressions? (There are potentially three different top-level calls implied by a new expression -- an operator new, a constructor, and an operator delete -- so it's not completely obvious what effect this would have there.) And likewise for delete.

include/clang/Basic/DiagnosticSemaKinds.td

8236

I think it would be surprising if __builtin_no_inline(sqrt(x)) would use a target-specific sqrt instruction rather than making a library call. But given that these builtins are best-effort anyway, maybe it's acceptable.

8238

Nit: we generally use "cannot" rather than "must not" in diagnostics.

lib/CodeGen/CGBuiltin.cpp

3008–3013

Nit: please add braces around the outer if.

lib/Sema/SemaChecking.cpp

338–345

Use isa<CallExpr> rather than checking the exact class here; you should accept statements derived from these kinds too (eg, UserDefinedLiteralExpr). (You should explicitly disallow CUDAKernelCallExpr, though, since the builtins don't make any sense for a host -> device call.)

349

Nit: braces here.

356–360

Is this a necessary restriction? I would expect calls to blocks to be inlineable when the callee can be statically determined.

[rsmith]

+rjmccall for CodeGen changes

[kuhar]

In D51200#1223752, @rsmith wrote:

+rjmccall for CodeGen changes

@rsmith @rjmccall
I have one high-level question about the CodeGen part that I wasn't able to figure out: is it possible to bail out of custom CodeGen in CGBuiltin and somehow say: emit whatever is inside (IE treat the builtin call node as a no-op)?

[kuhar]

Thank you for the comments, Richard!

In D51200#1223745, @rsmith wrote:

Have you considered whether the builtin should apply to new expressions? (There are potentially three different top-level calls implied by a new expression -- an operator new, a constructor, and an operator delete -- so it's not completely obvious what effect this would have there.) And likewise for delete.

I haven't thought about it, but to be honest I'm not sure what should happen. Intuitively, I think these 3 options would make most sense, listed from the most to the least reasonable in my opionion:

1. apply the corresponding attribute to all of the implied calls
2. apply to constructor/destructor only
3. don't handle it at all -- emit an error

How would you suggest to handle it?

[rjmccall]

In D51200#1223768, @kuhar wrote:

In D51200#1223752, @rsmith wrote:

+rjmccall for CodeGen changes

@rsmith @rjmccall
I have one high-level question about the CodeGen part that I wasn't able to figure out: is it possible to bail out of custom CodeGen in CGBuiltin and somehow say: emit whatever is inside (IE treat the builtin call node as a no-op)?

You can just emit the sub-expression. Make sure you pass down the ReturnValueSlot, and make sure you test C++ calls that return references where the result is really treated as an l-value.

I've commented about the language design on your RFC thread.

include/clang/Basic/DiagnosticSemaKinds.td

8240

This seems pretty trivial to add.

include/clang/CodeGen/CGFunctionInfo.h

517

Please don't propagate this information through the CGFunctionInfo. I really want this type to be *less* site-specific, not *more*.

lib/CodeGen/CGBuiltin.cpp

2998

I wonder if it would make more sense to give this its own Expr subclass. We do that with several other "builtin" calls, like __builtin_choose_expr and __builtin_offsetof. That would avoid the problem of ending up in CGBuiltin, and it would make it easier to recognize and look through these annotation calls in Sema or IRGen.

lib/CodeGen/CodeGenFunction.h

3570

I feel like the type of this argument should definitely be generalized to not just be about inlining.

[rsmith]

Please also teach constant expression evaluation (lib/AST/ExprConstant.cpp) to look through these builtins when evaluating.

[xbolva00]

include/clang/CodeGen/CGFunctionInfo.h

517

I like this patch but sadly, it was abandoned.

What way of passing this info would you recommend?

[rjmccall]

include/clang/CodeGen/CGFunctionInfo.h

517

I don't remember much about this anymore, but CGCall is perfectly capable of adding attributes based on call-site-specific information.

include/clang/Basic/Builtins.def

 BUILTIN(__builtin_debugtrap, "v", "n")
 BUILTIN(__builtin_unreachable, "v", "nr")
 BUILTIN(__builtin_shufflevector, "v."   , "nc")
 BUILTIN(__builtin_convertvector, "v."   , "nct")
 BUILTIN(__builtin_alloca, "v*z"   , "Fn")
 BUILTIN(__builtin_alloca_with_align, "v*zIz", "Fn")
 BUILTIN(__builtin_call_with_static_chain, "v.", "nt")
 
+// Clang builtins (not available in GCC).
+BUILTIN(__builtin_always_inline, "v.", "nt")
+BUILTIN(__builtin_no_inline, "v.", "nt")
+
 // "Overloaded" Atomic operator builtins.  These are overloaded to support data
 // types of i8, i16, i32, i64, and i128.  The front-end sees calls to the
 // non-suffixed version of these (which has a bogus type) and transforms them to
 // the right overloaded version in Sema (plus casts).
 
 // FIXME: These assume that char -> i8, short -> i16, int -> i32,
 // long long -> i64.
 

include/clang/Basic/DiagnosticSemaKinds.td

   "first argument to __builtin_call_with_static_chain must not be a block call">;
 def err_first_argument_to_cwsc_builtin_call : Error<
   "first argument to __builtin_call_with_static_chain must not be a builtin call">;
 def err_first_argument_to_cwsc_pdtor_call : Error<
   "first argument to __builtin_call_with_static_chain must not be a pseudo-destructor call">;
 def err_second_argument_to_cwsc_not_pointer : Error<
   "second argument to __builtin_call_with_static_chain must be of pointer type">;
 
+def err_argument_to_inline_intrinsic_not_call : Error<
+  "argument to %0 must be a function, member function, or operator call">;
+def err_argument_to_inline_intrinsic_builtin_call : Error<
+  "argument to %0 must not be a builtin call">;
+def err_argument_to_inline_intrinsic_pdtor_call : Error<
+  "argument to %0 must not be a pseudo-destructor call">;
+def err_argument_to_inline_intrinsic_block_call : Error<
+  "argument to %0 must not be a block call">;
+
 def err_vector_incorrect_num_initializers : Error<
   "%select{too many|too few}0 elements in vector initialization (expected %1 elements, have %2)">;
 def err_altivec_empty_initializer : Error<"expected initializer">;
 
 def err_invalid_neon_type_code : Error<
   "incompatible constant for this __builtin_neon function">;
 def err_argument_invalid_range : Error<
   "argument value %0 is outside the valid range [%1, %2]">;

include/clang/CodeGen/CGFunctionInfo.h

class CGFunctionInfo final
   unsigned ASTCallingConvention : 6;
 
   /// Whether this is an instance method.
   unsigned InstanceMethod : 1;
 
   /// Whether this is a chain call.
   unsigned ChainCall : 1;
 
+  unsigned InlineCall : 2;
+
   /// Whether this function is noreturn.
   unsigned NoReturn : 1;
 
   /// Whether this function is returns-retained.
   unsigned ReturnsRetained : 1;
 
   /// Whether this function saved caller registers.
   unsigned NoCallerSavedRegs : 1;
class CGFunctionInfo final
   }
   const ExtParameterInfo *getExtParameterInfosBuffer() const{
     return getTrailingObjects<ExtParameterInfo>();
   }
 
   CGFunctionInfo() : Required(RequiredArgs::All) {}
 
 public:
-  static CGFunctionInfo *create(unsigned llvmCC,
-                                bool instanceMethod,
-                                bool chainCall,
-                                const FunctionType::ExtInfo &extInfo,
-                                ArrayRef<ExtParameterInfo> paramInfos,
-                                CanQualType resultType,
-                                ArrayRef<CanQualType> argTypes,
-                                RequiredArgs required);
+  enum class CallInlineKind : unsigned char {
+    DefaultInline = 0,
+    NoInline = 1,
+    AlwaysInline = 2
+  };
+  static CGFunctionInfo *
+  create(unsigned llvmCC, bool instanceMethod, bool chainCall,
+         ArrayRef<ExtParameterInfo> paramInfos, CanQualType resultType,
+         ArrayRef<CanQualType> argTypes, RequiredArgs required,
+         CallInlineKind inlineKind = CallInlineKind::DefaultInline);
   void operator delete(void *p) { ::operator delete(p); }
 
   // Friending class TrailingObjects is apparently not good enough for MSVC,
   // so these have to be public.
   friend class TrailingObjects;
   size_t numTrailingObjects(OverloadToken<ArgInfo>) const {
     return NumArgs + 1;
   }
public:
   RequiredArgs getRequiredArgs() const { return Required; }
   unsigned getNumRequiredArgs() const {
     return isVariadic() ? getRequiredArgs().getNumRequiredArgs() : arg_size();
   }
 
   bool isInstanceMethod() const { return InstanceMethod; }
 
   bool isChainCall() const { return ChainCall; }
+  CallInlineKind getInlineCallKind() const {
+    return static_cast<CallInlineKind>(InlineCall);
+  }
 
   bool isNoReturn() const { return NoReturn; }
 
   /// In ARC, whether this function retains its return value.  This
   /// is not always reliable for call sites.
   bool isReturnsRetained() const { return ReturnsRetained; }
 
   /// Whether this function no longer saves caller registers.
public:
   llvm::StructType *getArgStruct() const { return ArgStruct; }
   CharUnits getArgStructAlignment() const {
     return CharUnits::fromQuantity(ArgStructAlign);
   }
   void setArgStruct(llvm::StructType *Ty, CharUnits Align) {
     ArgStruct = Ty;
     ArgStructAlign = Align.getQuantity();
   }
+  void setInlineKind(CallInlineKind InlineKind) {
+    InlineCall = static_cast<unsigned>(InlineKind);
+  }
 
   void Profile(llvm::FoldingSetNodeID &ID) {
     ID.AddInteger(getASTCallingConvention());
     ID.AddBoolean(InstanceMethod);
     ID.AddBoolean(ChainCall);
+    ID.AddInteger(InlineCall);
     ID.AddBoolean(NoReturn);
     ID.AddBoolean(ReturnsRetained);
     ID.AddBoolean(NoCallerSavedRegs);
     ID.AddBoolean(HasRegParm);
     ID.AddInteger(RegParm);
     ID.AddBoolean(NoCfCheck);
     ID.AddInteger(Required.getOpaqueData());
     ID.AddBoolean(HasExtParameterInfos);
     if (HasExtParameterInfos) {
       for (auto paramInfo : getExtParameterInfos())
         ID.AddInteger(paramInfo.getOpaqueValue());
     }
     getReturnType().Profile(ID);
     for (const auto &I : arguments())
       I.type.Profile(ID);
   }
-  static void Profile(llvm::FoldingSetNodeID &ID,
-                      bool InstanceMethod,
-                      bool ChainCall,
-                      const FunctionType::ExtInfo &info,
-                      ArrayRef<ExtParameterInfo> paramInfos,
-                      RequiredArgs required,
-                      CanQualType resultType,
-                      ArrayRef<CanQualType> argTypes) {
+  static void
+  Profile(llvm::FoldingSetNodeID &ID, bool InstanceMethod, bool ChainCall,
+          ArrayRef<ExtParameterInfo> paramInfos, RequiredArgs required,
+          CanQualType resultType, ArrayRef<CanQualType> argTypes,
+          CallInlineKind inlineKind = CallInlineKind::DefaultInline) {
     ID.AddInteger(info.getCC());
     ID.AddBoolean(InstanceMethod);
     ID.AddBoolean(ChainCall);
+    ID.AddInteger(static_cast<unsigned>(inlineKind));
     ID.AddBoolean(info.getNoReturn());
     ID.AddBoolean(info.getProducesResult());
     ID.AddBoolean(info.getNoCallerSavedRegs());
     ID.AddBoolean(info.getHasRegParm());
     ID.AddInteger(info.getRegParm());
     ID.AddBoolean(info.getNoCfCheck());
     ID.AddInteger(required.getOpaqueData());
     ID.AddBoolean(!paramInfos.empty());

lib/CodeGen/CGBuiltin.cpp

     return RValue::get(ConstantInt::get(IntTy, 0));
   case Builtin::BI__builtin_call_with_static_chain: {
     const CallExpr *Call = cast<CallExpr>(E->getArg(0));
     const Expr *Chain = E->getArg(1);
     return EmitCall(Call->getCallee()->getType(),
                     EmitCallee(Call->getCallee()), Call, ReturnValue,
                     EmitScalarExpr(Chain));
   }
+
+  case Builtin::BI__builtin_no_inline:
+  case Builtin::BI__builtin_always_inline: {
+    CGFunctionInfo::CallInlineKind CIK =
+        BuiltinID == Builtin::BI__builtin_no_inline
+            ? CGFunctionInfo::CallInlineKind::NoInline
+            : CGFunctionInfo::CallInlineKind::AlwaysInline;
+
+    auto *Arg = E->getArg(0);
+    if (const CXXMemberCallExpr *MemberCall = dyn_cast<CXXMemberCallExpr>(Arg))
+      return EmitCXXMemberCallExpr(MemberCall, ReturnValue, CIK);
+
+    if (const CXXOperatorCallExpr *OperatorCall =
+            dyn_cast<CXXOperatorCallExpr>(Arg))
+      if (const CXXMethodDecl *MD =
+              dyn_cast_or_null<CXXMethodDecl>(OperatorCall->getCalleeDecl()))
+        return EmitCXXOperatorMemberCallExpr(OperatorCall, MD, ReturnValue,
+                                             CIK);
+
+    const CallExpr *Call = cast<CallExpr>(Arg);
+    return EmitCall(Call->getCallee()->getType(), EmitCallee(Call->getCallee()),
+                    Call, ReturnValue, nullptr, CIK);
+  }
+
   case Builtin::BI_InterlockedExchange8:
   case Builtin::BI_InterlockedExchange16:
   case Builtin::BI_InterlockedExchange:
   case Builtin::BI_InterlockedExchangePointer:
     return RValue::get(
         EmitMSVCBuiltinExpr(MSVCIntrin::_InterlockedExchange, E));
   case Builtin::BI_InterlockedCompareExchangePointer: {
     llvm::Type *RTy;

lib/CodeGen/CGCall.cpp

return arrangeLLVMFunctionInfo(Context.VoidTy, /*instanceMethod=*/true,
                                  /*chainCall=*/false, ArgTys,
                                  FunctionType::ExtInfo(CC), {},
                                  RequiredArgs::All);
 }
 
 /// Arrange a call as unto a free function, except possibly with an
 /// additional number of formal parameters considered required.
 static const CGFunctionInfo &
-arrangeFreeFunctionLikeCall(CodeGenTypes &CGT,
-                            CodeGenModule &CGM,
-                            const CallArgList &args,
-                            const FunctionType *fnType,
-                            unsigned numExtraRequiredArgs,
-                            bool chainCall) {
+arrangeFreeFunctionLikeCall(CodeGenTypes &CGT, CodeGenModule &CGM,
+                            const CallArgList &args, const FunctionType *fnType,
+                            unsigned numExtraRequiredArgs, bool chainCall,
+                            CGFunctionInfo::CallInlineKind CIK =
+                                CGFunctionInfo::CallInlineKind::DefaultInline) {
   assert(args.size() >= numExtraRequiredArgs);
 
   llvm::SmallVector<FunctionProtoType::ExtParameterInfo, 16> paramInfos;
 
   // In most cases, there are no optional arguments.
   RequiredArgs required = RequiredArgs::All;
 
   // If we have a variadic prototype, the required arguments are the
arrangeFreeFunctionLikeCall(CodeGenTypes &CGT, CodeGenModule &CGM,
 
   // FIXME: Kill copy.
   SmallVector<CanQualType, 16> argTypes;
   for (const auto &arg : args)
     argTypes.push_back(CGT.getContext().getCanonicalParamType(arg.Ty));
   return CGT.arrangeLLVMFunctionInfo(GetReturnType(fnType->getReturnType()),
                                      /*instanceMethod=*/false, chainCall,
                                      argTypes, fnType->getExtInfo(), paramInfos,
-                                     required);
+                                     required, CIK);
 }
 
 /// Figure out the rules for calling a function with the given formal
 /// type using the given arguments.  The arguments are necessary
 /// because the function might be unprototyped, in which case it's
 /// target-dependent in crazy ways.
-const CGFunctionInfo &
-CodeGenTypes::arrangeFreeFunctionCall(const CallArgList &args,
-                                      const FunctionType *fnType,
-                                      bool chainCall) {
+const CGFunctionInfo &CodeGenTypes::arrangeFreeFunctionCall(
+    const CallArgList &args, const FunctionType *fnType, bool chainCall,
+    CGFunctionInfo::CallInlineKind CIK) {
   return arrangeFreeFunctionLikeCall(*this, CGM, args, fnType,
-                                     chainCall ? 1 : 0, chainCall);
+                                     chainCall ? 1 : 0, chainCall, CIK);
 }
 
 /// A block function is essentially a free function with an
 /// extra implicit argument.
 const CGFunctionInfo &
 CodeGenTypes::arrangeBlockFunctionCall(const CallArgList &args,
                                        const FunctionType *fnType) {
   return arrangeFreeFunctionLikeCall(*this, CGM, args, fnType, 1,
return arrangeLLVMFunctionInfo(
       resultType, /*instanceMethod=*/false, /*chainCall=*/false,
       argTypes, FunctionType::ExtInfo(), {}, RequiredArgs::All);
 }
 
 /// Arrange a call to a C++ method, passing the given arguments.
 ///
 /// numPrefixArgs is the number of ABI-specific prefix arguments we have. It
 /// does not count `this`.
-const CGFunctionInfo &
-CodeGenTypes::arrangeCXXMethodCall(const CallArgList &args,
-                                   const FunctionProtoType *proto,
-                                   RequiredArgs required,
-                                   unsigned numPrefixArgs) {
+const CGFunctionInfo &CodeGenTypes::arrangeCXXMethodCall(
+    const CallArgList &args, const FunctionProtoType *proto,
+    RequiredArgs required, unsigned numPrefixArgs,
+    CGFunctionInfo::CallInlineKind CIK) {
   assert(numPrefixArgs + 1 <= args.size() &&
          "Emitting a call with less args than the required prefix?");
   // Add one to account for `this`. It's a bit awkward here, but we don't count
   // `this` in similar places elsewhere.
   auto paramInfos =
     getExtParameterInfosForCall(proto, numPrefixArgs + 1, args.size());
 
   // FIXME: Kill copy.
   auto argTypes = getArgTypesForCall(Context, args);
 
   FunctionType::ExtInfo info = proto->getExtInfo();
   return arrangeLLVMFunctionInfo(
       GetReturnType(proto->getReturnType()), /*instanceMethod=*/true,
-      /*chainCall=*/false, argTypes, info, paramInfos, required);
+      /*chainCall=*/false, argTypes, info, paramInfos, required, CIK);
 }
 
 const CGFunctionInfo &CodeGenTypes::arrangeNullaryFunction() {
   return arrangeLLVMFunctionInfo(
       getContext().VoidTy, /*instanceMethod=*/false, /*chainCall=*/false,
       None, FunctionType::ExtInfo(), {}, RequiredArgs::All);
 }
 

 namespace CodeGen {
 void computeSPIRKernelABIInfo(CodeGenModule &CGM, CGFunctionInfo &FI);
 }
 }
 
 /// Arrange the argument and result information for an abstract value
 /// of a given function type.  This is the method which all of the
 /// above functions ultimately defer to.
-const CGFunctionInfo &
-CodeGenTypes::arrangeLLVMFunctionInfo(CanQualType resultType,
-                                      bool instanceMethod,
-                                      bool chainCall,
-                                      ArrayRef<CanQualType> argTypes,
-                                      FunctionType::ExtInfo info,
-                     ArrayRef<FunctionProtoType::ExtParameterInfo> paramInfos,
-                                      RequiredArgs required) {
+const CGFunctionInfo &CodeGenTypes::arrangeLLVMFunctionInfo(
+    CanQualType resultType, bool instanceMethod, bool chainCall,
+    ArrayRef<CanQualType> argTypes, FunctionType::ExtInfo info,
+    RequiredArgs required, CGFunctionInfo::CallInlineKind CIK) {
   assert(std::all_of(argTypes.begin(), argTypes.end(),
                      [](CanQualType T) { return T.isCanonicalAsParam(); }));
 
   // Lookup or create unique function info.
   llvm::FoldingSetNodeID ID;
   CGFunctionInfo::Profile(ID, instanceMethod, chainCall, info, paramInfos,
-                          required, resultType, argTypes);
+                          required, resultType, argTypes, CIK);
 
   void *insertPos = nullptr;
   CGFunctionInfo *FI = FunctionInfos.FindNodeOrInsertPos(ID, insertPos);
   if (FI)
     return *FI;
 
   unsigned CC = ClangCallConvToLLVMCallConv(info.getCC());
 
   // Construct the function info.  We co-allocate the ArgInfos.
-  FI = CGFunctionInfo::create(CC, instanceMethod, chainCall, info,
-                              paramInfos, resultType, argTypes, required);
+  FI = CGFunctionInfo::create(CC, instanceMethod, chainCall, info, paramInfos,
+                              resultType, argTypes, required, CIK);
   FunctionInfos.InsertNode(FI, insertPos);
 
   bool inserted = FunctionsBeingProcessed.insert(FI).second;
   (void)inserted;
   assert(inserted && "Recursively being processed?");
 
   // Compute ABI information.
   if (CC == llvm::CallingConv::SPIR_KERNEL) {
if (I.info.canHaveCoerceToType() && I.info.getCoerceToType() == nullptr)
       I.info.setCoerceToType(ConvertType(I.type));
 
   bool erased = FunctionsBeingProcessed.erase(FI); (void)erased;
   assert(erased && "Not in set?");
 
   return *FI;
 }
 
-CGFunctionInfo *CGFunctionInfo::create(unsigned llvmCC,
-                                       bool instanceMethod,
-                                       bool chainCall,
-                                       const FunctionType::ExtInfo &info,
-                                       ArrayRef<ExtParameterInfo> paramInfos,
-                                       CanQualType resultType,
-                                       ArrayRef<CanQualType> argTypes,
-                                       RequiredArgs required) {
+CGFunctionInfo *
+CGFunctionInfo::create(unsigned llvmCC, bool instanceMethod, bool chainCall,
+                       CanQualType resultType, ArrayRef<CanQualType> argTypes,
+                       RequiredArgs required, CallInlineKind CIK) {
   assert(paramInfos.empty() || paramInfos.size() == argTypes.size());
 
   void *buffer =
     operator new(totalSizeToAlloc<ArgInfo,             ExtParameterInfo>(
                                   argTypes.size() + 1, paramInfos.size()));
 
   CGFunctionInfo *FI = new(buffer) CGFunctionInfo();
   FI->CallingConvention = llvmCC;
   FI->EffectiveCallingConvention = llvmCC;
   FI->ASTCallingConvention = info.getCC();
   FI->InstanceMethod = instanceMethod;
   FI->ChainCall = chainCall;
+  FI->InlineCall = static_cast<unsigned>(CIK);
   FI->NoReturn = info.getNoReturn();
   FI->ReturnsRetained = info.getProducesResult();
   FI->NoCallerSavedRegs = info.getNoCallerSavedRegs();
   FI->NoCfCheck = info.getNoCfCheck();
   FI->Required = required;
   FI->HasRegParm = info.getHasRegParm();
   FI->RegParm = info.getRegParm();
   FI->ArgStruct = nullptr;
void CodeGenModule::ConstructAttributeList(
   // Attach attributes to inalloca argument.
   if (IRFunctionArgs.hasInallocaArg()) {
     llvm::AttrBuilder Attrs;
     Attrs.addAttribute(llvm::Attribute::InAlloca);
     ArgAttrs[IRFunctionArgs.getInallocaArgNo()] =
         llvm::AttributeSet::get(getLLVMContext(), Attrs);
   }
 
+  if (FI.getInlineCallKind() == CGFunctionInfo::CallInlineKind::AlwaysInline)
+    FuncAttrs.addAttribute(llvm::Attribute::AlwaysInline);
+  else if (FI.getInlineCallKind() == CGFunctionInfo::CallInlineKind::NoInline)
+    FuncAttrs.addAttribute(llvm::Attribute::NoInline);
+
   unsigned ArgNo = 0;
   for (CGFunctionInfo::const_arg_iterator I = FI.arg_begin(),
                                           E = FI.arg_end();
        I != E; ++I, ++ArgNo) {
     QualType ParamType = I->type;
     const ABIArgInfo &AI = I->info;
     llvm::AttrBuilder Attrs;
 
void CodeGenFunction::deferPlaceholderReplacement(llvm::Instruction *Old,
   DeferredReplacements.push_back(std::make_pair(Old, New));
 }
 
 RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
                                  const CGCallee &Callee,
                                  ReturnValueSlot ReturnValue,
                                  const CallArgList &CallArgs,
                                  llvm::Instruction **callOrInvoke,
-                                 SourceLocation Loc) {
+                                 SourceLocation Loc,
+                                 CGFunctionInfo::CallInlineKind CIK) {
   // FIXME: We no longer need the types from CallArgs; lift up and simplify.
 
   assert(Callee.isOrdinary() || Callee.isVirtual());
 
   // Handle struct-return functions by passing a pointer to the
   // location that we would like to return into.
   QualType RetTy = CallInfo.getReturnType();
   const ABIArgInfo &RetAI = CallInfo.getReturnInfo();
#endif
   if (CurCodeDecl && CurCodeDecl->hasAttr<FlattenAttr>() &&
       !(Callee.getAbstractInfo().getCalleeDecl() &&
         Callee.getAbstractInfo().getCalleeDecl()->hasAttr<NoInlineAttr>())) {
     Attrs =
         Attrs.addAttribute(getLLVMContext(), llvm::AttributeList::FunctionIndex,
                            llvm::Attribute::AlwaysInline);
   }
 
+  if (CIK != CGFunctionInfo::CallInlineKind::DefaultInline)
+    Attrs =
+        Attrs.addAttribute(getLLVMContext(), llvm::AttributeList::FunctionIndex,
+                           CIK == CGFunctionInfo::CallInlineKind::NoInline
+                               ? llvm::Attribute::NoInline
+                               : llvm::Attribute::AlwaysInline);
+
   // Disable inlining inside SEH __try blocks.
   if (isSEHTryScope()) {
     Attrs =
         Attrs.addAttribute(getLLVMContext(), llvm::AttributeList::FunctionIndex,
                            llvm::Attribute::NoInline);
   }
 
   // Decide whether to use a call or an invoke.

lib/CodeGen/CGExpr.cpp

 
 LValue CodeGenFunction::EmitStmtExprLValue(const StmtExpr *E) {
   // Can only get l-value for message expression returning aggregate type
   RValue RV = EmitAnyExprToTemp(E);
   return MakeAddrLValue(RV.getAggregateAddress(), E->getType(),
                         AlignmentSource::Decl);
 }
 
-RValue CodeGenFunction::EmitCall(QualType CalleeType, const CGCallee &OrigCallee,
-                                 const CallExpr *E, ReturnValueSlot ReturnValue,
-                                 llvm::Value *Chain) {
+RValue CodeGenFunction::EmitCall(QualType CalleeType,
+                                 const CGCallee &OrigCallee, const CallExpr *E,
+                                 ReturnValueSlot ReturnValue,
+                                 llvm::Value *Chain,
+                                 CGFunctionInfo::CallInlineKind CIK) {
   // Get the actual function type. The callee type will always be a pointer to
   // function type or a block pointer type.
   assert(CalleeType->isFunctionPointerType() &&
          "Call must have function pointer type!");
 
   const Decl *TargetDecl = OrigCallee.getAbstractInfo().getCalleeDecl();
 
   if (const FunctionDecl *FD = dyn_cast_or_null<FunctionDecl>(TargetDecl))

         break;
       }
     }
   }
 
   EmitCallArgs(Args, dyn_cast<FunctionProtoType>(FnType), E->arguments(),
                E->getDirectCallee(), /*ParamsToSkip*/ 0, Order);
 
-  const CGFunctionInfo &FnInfo = CGM.getTypes().arrangeFreeFunctionCall(
-      Args, FnType, /*isChainCall=*/Chain);
+  const CGFunctionInfo *FnInfo = &CGM.getTypes().arrangeFreeFunctionCall(
+      Args, FnType, /*isChainCall=*/Chain, CIK);
 
   // C99 6.5.2.2p6:
   //   If the expression that denotes the called function has a type
   //   that does not include a prototype, [the default argument
   //   promotions are performed]. If the number of arguments does not
   //   equal the number of parameters, the behavior is undefined. If
   //   the function is defined with a type that includes a prototype,
   //   and either the prototype ends with an ellipsis (, ...) or the
   //   types of the arguments after promotion are not compatible with
   //   the types of the parameters, the behavior is undefined. If the
   //   function is defined with a type that does not include a
   //   prototype, and the types of the arguments after promotion are
   //   not compatible with those of the parameters after promotion,
   //   the behavior is undefined [except in some trivial cases].
   // That is, in the general case, we should assume that a call
   // through an unprototyped function type works like a *non-variadic*
   // call.  The way we make this work is to cast to the exact type
   // of the promoted arguments.
   //
   // Chain calls use this same code path to add the invisible chain parameter
   // to the function type.
   if (isa<FunctionNoProtoType>(FnType) || Chain) {
-    llvm::Type *CalleeTy = getTypes().GetFunctionType(FnInfo);
+    llvm::Type *CalleeTy = getTypes().GetFunctionType(*FnInfo);
     CalleeTy = CalleeTy->getPointerTo();
 
     llvm::Value *CalleePtr = Callee.getFunctionPointer();
     CalleePtr = Builder.CreateBitCast(CalleePtr, CalleeTy, "callee.knr.cast");
     Callee.setFunctionPointer(CalleePtr);
   }
 
-  return EmitCall(FnInfo, Callee, ReturnValue, Args, nullptr, E->getExprLoc());
+  return EmitCall(*FnInfo, Callee, ReturnValue, Args, nullptr, E->getExprLoc());
 }
 
 LValue CodeGenFunction::
 EmitPointerToDataMemberBinaryExpr(const BinaryOperator *E) {
   Address BaseAddr = Address::invalid();
   if (E->getOpcode() == BO_PtrMemI) {
     BaseAddr = EmitPointerWithAlignment(E->getLHS());
   } else {

lib/CodeGen/CGExprCXX.cpp

assert(
         FPT->getNumParams() == 0 &&
         "No CallExpr specified for function with non-zero number of arguments");
   }
   return {required, PrefixSize};
 }
 
 RValue CodeGenFunction::EmitCXXMemberOrOperatorCall(
     const CXXMethodDecl *MD, const CGCallee &Callee,
-    ReturnValueSlot ReturnValue,
-    llvm::Value *This, llvm::Value *ImplicitParam, QualType ImplicitParamTy,
-    const CallExpr *CE, CallArgList *RtlArgs) {
+    ReturnValueSlot ReturnValue, llvm::Value *This, llvm::Value *ImplicitParam,
+    QualType ImplicitParamTy, const CallExpr *CE, CallArgList *RtlArgs,
+    CGFunctionInfo::CallInlineKind CIK) {
   const FunctionProtoType *FPT = MD->getType()->castAs<FunctionProtoType>();
   CallArgList Args;
   MemberCallInfo CallInfo = commonEmitCXXMemberOrOperatorCall(
       *this, MD, This, ImplicitParam, ImplicitParamTy, CE, Args, RtlArgs);
   auto &FnInfo = CGM.getTypes().arrangeCXXMethodCall(
-      Args, FPT, CallInfo.ReqArgs, CallInfo.PrefixSize);
+      Args, FPT, CallInfo.ReqArgs, CallInfo.PrefixSize, CIK);
   return EmitCall(FnInfo, Callee, ReturnValue, Args, nullptr,
                   CE ? CE->getExprLoc() : SourceLocation());
 }
 
 RValue CodeGenFunction::EmitCXXDestructorCall(
     const CXXDestructorDecl *DD, const CGCallee &Callee, llvm::Value *This,
     llvm::Value *ImplicitParam, QualType ImplicitParamTy, const CallExpr *CE,
     StructorType Type) {
static CXXRecordDecl *getCXXRecord(const Expr *E) {
   if (const PointerType *PTy = T->getAs<PointerType>())
     T = PTy->getPointeeType();
   const RecordType *Ty = T->castAs<RecordType>();
   return cast<CXXRecordDecl>(Ty->getDecl());
 }
 
 // Note: This function also emit constructor calls to support a MSVC
 // extensions allowing explicit constructor function call.
-RValue CodeGenFunction::EmitCXXMemberCallExpr(const CXXMemberCallExpr *CE,
-                                              ReturnValueSlot ReturnValue) {
+RValue
+CodeGenFunction::EmitCXXMemberCallExpr(const CXXMemberCallExpr *CE,
+                                       ReturnValueSlot ReturnValue,
+                                       CGFunctionInfo::CallInlineKind CIK) {
   const Expr *callee = CE->getCallee()->IgnoreParens();
 
-  if (isa<BinaryOperator>(callee))
-    return EmitCXXMemberPointerCallExpr(CE, ReturnValue);
+  if (isa<BinaryOperator>(callee)) {
+    llvm::errs() << "lolol\n";
+    return EmitCXXMemberPointerCallExpr(CE, ReturnValue, CIK);
+  }
 
   const MemberExpr *ME = cast<MemberExpr>(callee);
   const CXXMethodDecl *MD = cast<CXXMethodDecl>(ME->getMemberDecl());
 
   if (MD->isStatic()) {
     // The method is static, emit it as we would a regular call.
     CGCallee callee = CGCallee::forDirect(CGM.GetAddrOfFunction(MD), MD);
     return EmitCall(getContext().getPointerType(MD->getType()), callee, CE,
-                    ReturnValue);
+                    ReturnValue, /*Chain = */ nullptr, CIK);
   }
 
   bool HasQualifier = ME->hasQualifier();
   NestedNameSpecifier *Qualifier = HasQualifier ? ME->getQualifier() : nullptr;
   bool IsArrow = ME->isArrow();
   const Expr *Base = ME->getBase();
 
   return EmitCXXMemberOrOperatorMemberCallExpr(
-      CE, MD, ReturnValue, HasQualifier, Qualifier, IsArrow, Base);
+      CE, MD, ReturnValue, HasQualifier, Qualifier, IsArrow, Base, CIK);
 }
 
 RValue CodeGenFunction::EmitCXXMemberOrOperatorMemberCallExpr(
     const CallExpr *CE, const CXXMethodDecl *MD, ReturnValueSlot ReturnValue,
     bool HasQualifier, NestedNameSpecifier *Qualifier, bool IsArrow,
-    const Expr *Base) {
+    const Expr *Base, CGFunctionInfo::CallInlineKind CIK) {
   assert(isa<CXXMemberCallExpr>(CE) || isa<CXXOperatorCallExpr>(CE));
 
   // Compute the object pointer.
   bool CanUseVirtualCall = MD->isVirtual() && !HasQualifier;
 
   const CXXMethodDecl *DevirtualizedMethod = nullptr;
   if (CanUseVirtualCall &&
       MD->getDevirtualizedMethod(Base, getLangOpts().AppleKext)) {
if (MD->isVirtual()) {
     Address NewThisAddr =
         CGM.getCXXABI().adjustThisArgumentForVirtualFunctionCall(
             *this, CalleeDecl, This.getAddress(), UseVirtualCall);
     This.setAddress(NewThisAddr);
   }
 
   return EmitCXXMemberOrOperatorCall(
       CalleeDecl, Callee, ReturnValue, This.getPointer(),
-      /*ImplicitParam=*/nullptr, QualType(), CE, RtlArgs);
+      /*ImplicitParam=*/nullptr, QualType(), CE, RtlArgs, CIK);
 }
 
-RValue
-CodeGenFunction::EmitCXXMemberPointerCallExpr(const CXXMemberCallExpr *E,
-                                              ReturnValueSlot ReturnValue) {
+RValue CodeGenFunction::EmitCXXMemberPointerCallExpr(
+    const CXXMemberCallExpr *E, ReturnValueSlot ReturnValue,
+    CGFunctionInfo::CallInlineKind CIK) {
   const BinaryOperator *BO =
       cast<BinaryOperator>(E->getCallee()->IgnoreParens());
   const Expr *BaseExpr = BO->getLHS();
   const Expr *MemFnExpr = BO->getRHS();
 
   const MemberPointerType *MPT =
     MemFnExpr->getType()->castAs<MemberPointerType>();
 
RValue CodeGenFunction::EmitCXXMemberPointerCallExpr(
 
   RequiredArgs required =
       RequiredArgs::forPrototypePlus(FPT, 1, /*FD=*/nullptr);
 
   // And the rest of the call args
   EmitCallArgs(Args, FPT, E->arguments());
   return EmitCall(CGM.getTypes().arrangeCXXMethodCall(Args, FPT, required,
                                                       /*PrefixSize=*/0),
-                  Callee, ReturnValue, Args, nullptr, E->getExprLoc());
+                  Callee, ReturnValue, Args, nullptr, E->getExprLoc(), CIK);
 }
 
-RValue
-CodeGenFunction::EmitCXXOperatorMemberCallExpr(const CXXOperatorCallExpr *E,
-                                               const CXXMethodDecl *MD,
-                                               ReturnValueSlot ReturnValue) {
+RValue CodeGenFunction::EmitCXXOperatorMemberCallExpr(
+    const CXXOperatorCallExpr *E, const CXXMethodDecl *MD,
+    ReturnValueSlot ReturnValue, CGFunctionInfo::CallInlineKind CIK) {
   assert(MD->isInstance() &&
          "Trying to emit a member call expr on a static method!");
   return EmitCXXMemberOrOperatorMemberCallExpr(
       E, MD, ReturnValue, /*HasQualifier=*/false, /*Qualifier=*/nullptr,
-      /*IsArrow=*/false, E->getArg(0));
+      /*IsArrow=*/false, E->getArg(0), CIK);
 }
 
 RValue CodeGenFunction::EmitCUDAKernelCallExpr(const CUDAKernelCallExpr *E,
                                                ReturnValueSlot ReturnValue) {
   return CGM.getCUDARuntime().EmitCUDAKernelCallExpr(*this, E, ReturnValue);
 }
 
 static void EmitNullBaseClassInitialization(CodeGenFunction &CGF,

lib/CodeGen/CodeGenFunction.h

   //                         Scalar Expression Emission
   //===--------------------------------------------------------------------===//
 
   /// EmitCall - Generate a call of the given function, expecting the given
   /// result type, and using the given argument list which specifies both the
   /// LLVM arguments and the types they were derived from.
   RValue EmitCall(const CGFunctionInfo &CallInfo, const CGCallee &Callee,
                   ReturnValueSlot ReturnValue, const CallArgList &Args,
-                  llvm::Instruction **callOrInvoke, SourceLocation Loc);
+                  llvm::Instruction **callOrInvoke, SourceLocation Loc,
+                  CGFunctionInfo::CallInlineKind CIK =
+                      CGFunctionInfo::CallInlineKind::DefaultInline);
   RValue EmitCall(const CGFunctionInfo &CallInfo, const CGCallee &Callee,
                   ReturnValueSlot ReturnValue, const CallArgList &Args,
-                  llvm::Instruction **callOrInvoke = nullptr) {
+                  llvm::Instruction **callOrInvoke = nullptr,
+                  CGFunctionInfo::CallInlineKind CIK =
+                      CGFunctionInfo::CallInlineKind::DefaultInline) {
     return EmitCall(CallInfo, Callee, ReturnValue, Args, callOrInvoke,
-                    SourceLocation());
+                    SourceLocation(), CIK);
   }
   RValue EmitCall(QualType FnType, const CGCallee &Callee, const CallExpr *E,
-                  ReturnValueSlot ReturnValue, llvm::Value *Chain = nullptr);
+                  ReturnValueSlot ReturnValue, llvm::Value *Chain = nullptr,
+                  CGFunctionInfo::CallInlineKind CIK =
+                      CGFunctionInfo::CallInlineKind::DefaultInline);
   RValue EmitCallExpr(const CallExpr *E,
                       ReturnValueSlot ReturnValue = ReturnValueSlot());
   RValue EmitSimpleCallExpr(const CallExpr *E, ReturnValueSlot ReturnValue);
   CGCallee EmitCallee(const Expr *E);
 
   void checkTargetFeatures(const CallExpr *E, const FunctionDecl *TargetDecl);
 
   llvm::CallInst *EmitRuntimeCall(llvm::Value *callee,

   void defaultInitNonTrivialCStructVar(LValue Dst);
   void callCStructDefaultConstructor(LValue Dst);
   void callCStructDestructor(LValue Dst);
   void callCStructCopyConstructor(LValue Dst, LValue Src);
   void callCStructMoveConstructor(LValue Dst, LValue Src);
   void callCStructCopyAssignmentOperator(LValue Dst, LValue Src);
   void callCStructMoveAssignmentOperator(LValue Dst, LValue Src);
 
-  RValue
-  EmitCXXMemberOrOperatorCall(const CXXMethodDecl *Method,
-                              const CGCallee &Callee,
-                              ReturnValueSlot ReturnValue, llvm::Value *This,
-                              llvm::Value *ImplicitParam,
-                              QualType ImplicitParamTy, const CallExpr *E,
-                              CallArgList *RtlArgs);
+  RValue EmitCXXMemberOrOperatorCall(
+      const CXXMethodDecl *Method, const CGCallee &Callee,
+      ReturnValueSlot ReturnValue, llvm::Value *This,
+      llvm::Value *ImplicitParam, QualType ImplicitParamTy, const CallExpr *E,
+      CallArgList *RtlArgs,
+      CGFunctionInfo::CallInlineKind CIK =
+          CGFunctionInfo::CallInlineKind::DefaultInline);
   RValue EmitCXXDestructorCall(const CXXDestructorDecl *DD,
                                const CGCallee &Callee,
                                llvm::Value *This, llvm::Value *ImplicitParam,
                                QualType ImplicitParamTy, const CallExpr *E,
                                StructorType Type);
-  RValue EmitCXXMemberCallExpr(const CXXMemberCallExpr *E,
-                               ReturnValueSlot ReturnValue);
-  RValue EmitCXXMemberOrOperatorMemberCallExpr(const CallExpr *CE,
-                                               const CXXMethodDecl *MD,
-                                               ReturnValueSlot ReturnValue,
-                                               bool HasQualifier,
-                                               NestedNameSpecifier *Qualifier,
-                                               bool IsArrow, const Expr *Base);
+  RValue
+  EmitCXXMemberCallExpr(const CXXMemberCallExpr *E, ReturnValueSlot ReturnValue,
+                        CGFunctionInfo::CallInlineKind CIK =
+                            CGFunctionInfo::CallInlineKind::DefaultInline);
+  RValue EmitCXXMemberOrOperatorMemberCallExpr(
+      const CallExpr *CE, const CXXMethodDecl *MD, ReturnValueSlot ReturnValue,
+      bool HasQualifier, NestedNameSpecifier *Qualifier, bool IsArrow,
+      const Expr *Base,
+      CGFunctionInfo::CallInlineKind CIK =
+          CGFunctionInfo::CallInlineKind::DefaultInline);
   // Compute the object pointer.
   Address EmitCXXMemberDataPointerAddress(const Expr *E, Address base,
                                           llvm::Value *memberPtr,
                                           const MemberPointerType *memberPtrType,
                                           LValueBaseInfo *BaseInfo = nullptr,
                                           TBAAAccessInfo *TBAAInfo = nullptr);
-  RValue EmitCXXMemberPointerCallExpr(const CXXMemberCallExpr *E,
-                                      ReturnValueSlot ReturnValue);
-
-  RValue EmitCXXOperatorMemberCallExpr(const CXXOperatorCallExpr *E,
-                                       const CXXMethodDecl *MD,
-                                       ReturnValueSlot ReturnValue);
+  RValue EmitCXXMemberPointerCallExpr(
+      const CXXMemberCallExpr *E, ReturnValueSlot ReturnValue,
+      CGFunctionInfo::CallInlineKind CIK =
+          CGFunctionInfo::CallInlineKind::DefaultInline);
+
+  RValue EmitCXXOperatorMemberCallExpr(
+      const CXXOperatorCallExpr *E, const CXXMethodDecl *MD,
+      ReturnValueSlot ReturnValue,
+      CGFunctionInfo::CallInlineKind CIK =
+          CGFunctionInfo::CallInlineKind::DefaultInline);
   RValue EmitCXXPseudoDestructorExpr(const CXXPseudoDestructorExpr *E);
 
   RValue EmitCUDAKernelCallExpr(const CUDAKernelCallExpr *E,
                                 ReturnValueSlot ReturnValue);
 
   RValue EmitNVPTXDevicePrintfCallExpr(const CallExpr *E,
                                        ReturnValueSlot ReturnValue);
 

lib/CodeGen/CodeGenTypes.h

public:
   ///
   /// Often this will be able to simply return the declaration info.
   const CGFunctionInfo &arrangeCall(const CGFunctionInfo &declFI,
                                     const CallArgList &args);
 
   /// Free functions are functions that are compatible with an ordinary
   /// C function pointer type.
   const CGFunctionInfo &arrangeFunctionDeclaration(const FunctionDecl *FD);
-  const CGFunctionInfo &arrangeFreeFunctionCall(const CallArgList &Args,
-                                                const FunctionType *Ty,
-                                                bool ChainCall);
+  const CGFunctionInfo &
+  arrangeFreeFunctionCall(const CallArgList &Args, const FunctionType *Ty,
+                          bool ChainCall,
+                          CGFunctionInfo::CallInlineKind InlineKind =
+                              CGFunctionInfo::CallInlineKind::DefaultInline);
   const CGFunctionInfo &arrangeFreeFunctionType(CanQual<FunctionProtoType> Ty,
                                                 const FunctionDecl *FD);
   const CGFunctionInfo &arrangeFreeFunctionType(CanQual<FunctionNoProtoType> Ty);
 
   /// A nullary function is a freestanding function of type 'void ()'.
   /// This method works for both calls and declarations.
   const CGFunctionInfo &arrangeNullaryFunction();
 
const CGFunctionInfo &arrangeCXXStructorDeclaration(const CXXMethodDecl *MD,
                                                       StructorType Type);
   const CGFunctionInfo &arrangeCXXConstructorCall(const CallArgList &Args,
                                                   const CXXConstructorDecl *D,
                                                   CXXCtorType CtorKind,
                                                   unsigned ExtraPrefixArgs,
                                                   unsigned ExtraSuffixArgs,
                                                   bool PassProtoArgs = true);
 
-  const CGFunctionInfo &arrangeCXXMethodCall(const CallArgList &args,
-                                             const FunctionProtoType *type,
-                                             RequiredArgs required,
-                                             unsigned numPrefixArgs);
+  const CGFunctionInfo &
+  arrangeCXXMethodCall(const CallArgList &args, const FunctionProtoType *type,
+                       RequiredArgs required, unsigned numPrefixArgs,
+                       CGFunctionInfo::CallInlineKind InlineKind =
+                           CGFunctionInfo::CallInlineKind::DefaultInline);
   const CGFunctionInfo &
   arrangeUnprototypedMustTailThunk(const CXXMethodDecl *MD);
   const CGFunctionInfo &arrangeMSCtorClosure(const CXXConstructorDecl *CD,
                                                  CXXCtorType CT);
   const CGFunctionInfo &arrangeCXXMethodType(const CXXRecordDecl *RD,
                                              const FunctionProtoType *FTP,
                                              const CXXMethodDecl *MD);
 
   /// "Arrange" the LLVM information for a call or type with the given
   /// signature.  This is largely an internal method; other clients
   /// should use one of the above routines, which ultimately defer to
   /// this.
   ///
   /// \param argTypes - must all actually be canonical as params
-  const CGFunctionInfo &arrangeLLVMFunctionInfo(CanQualType returnType,
-                                                bool instanceMethod,
-                                                bool chainCall,
-                                                ArrayRef<CanQualType> argTypes,
-                                                FunctionType::ExtInfo info,
-                    ArrayRef<FunctionProtoType::ExtParameterInfo> paramInfos,
-                                                RequiredArgs args);
+  const CGFunctionInfo &arrangeLLVMFunctionInfo(
+      CanQualType returnType, bool instanceMethod, bool chainCall,
+      ArrayRef<CanQualType> argTypes, FunctionType::ExtInfo info,
+      RequiredArgs args,
+      CGFunctionInfo::CallInlineKind InlineKind =
+          CGFunctionInfo::CallInlineKind::DefaultInline);
 
   /// Compute a new LLVM record layout object for the given record.
   CGRecordLayout *ComputeRecordLayout(const RecordDecl *D,
                                       llvm::StructType *Ty);
 
   /// addRecordTypeName - Compute a name from the given record decl with an
   /// optional suffix and name the given LLVM type using it.
   void addRecordTypeName(const RecordDecl *RD, llvm::StructType *Ty,

lib/Sema/SemaChecking.cpp

static bool SemaBuiltinCallWithStaticChain(Sema &S, CallExpr *BuiltinCall) {
   BuiltinCall->setValueKind(CE->getValueKind());
   BuiltinCall->setObjectKind(CE->getObjectKind());
   BuiltinCall->setCallee(Builtin);
   BuiltinCall->setArg(1, ChainResult.get());
 
   return false;
 }
 
+static bool SemaBuiltinCallWithInline(Sema &S, CallExpr *BuiltinCall) {
+  if (checkArgCount(S, BuiltinCall, 1))
+    return true;
+
+  SourceLocation BuiltinLoc = BuiltinCall->getBeginLoc();
+  Expr *Builtin = BuiltinCall->getCallee()->IgnoreImpCasts();
+  Expr *Call = BuiltinCall->getArg(0);
+
+  const auto CallStmtClass = Call->getStmtClass();
+  if (CallStmtClass != Stmt::CallExprClass &&
+      CallStmtClass != Stmt::CXXMemberCallExprClass &&
+      CallStmtClass != Stmt::CXXOperatorCallExprClass) {
+    S.Diag(BuiltinLoc, diag::err_argument_to_inline_intrinsic_not_call)
+        << Builtin << Call->getSourceRange();
+    return true;
+  }
+
+  auto *CE = cast<CallExpr>(Call);
+  const Decl *TargetDecl = CE->getCalleeDecl();
+  if (const auto *FD = dyn_cast_or_null<FunctionDecl>(TargetDecl))
+    if (FD->getBuiltinID()) {
+      S.Diag(BuiltinLoc, diag::err_argument_to_inline_intrinsic_builtin_call)
+          << Builtin << Call->getSourceRange();
+      return true;
+    }
+
+  if (CE->getCallee()->getType()->isBlockPointerType()) {
+    S.Diag(BuiltinLoc, diag::err_argument_to_inline_intrinsic_block_call)
+        << Builtin << Call->getSourceRange();
+    return true;
+  }
+
+  if (isa<CXXPseudoDestructorExpr>(CE->getCallee()->IgnoreParens())) {
+    S.Diag(BuiltinLoc, diag::err_argument_to_inline_intrinsic_pdtor_call)
+        << Builtin << Call->getSourceRange();
+    return true;
+  }
+
+  QualType ReturnTy = CE->getCallReturnType(S.Context);
+  QualType ArgTys[1] = {ReturnTy};
+  QualType BuiltinTy = S.Context.getFunctionType(
+      ReturnTy, ArgTys, FunctionProtoType::ExtProtoInfo());
+  QualType BuiltinPtrTy = S.Context.getPointerType(BuiltinTy);
+
+  Builtin =
+      S.ImpCastExprToType(Builtin, BuiltinPtrTy, CK_BuiltinFnToFnPtr).get();
+
+  BuiltinCall->setType(CE->getType());
+  BuiltinCall->setValueKind(CE->getValueKind());
+  BuiltinCall->setObjectKind(CE->getObjectKind());
+  BuiltinCall->setCallee(Builtin);
+
+  return false;
+}
+
 static bool SemaBuiltinSEHScopeCheck(Sema &SemaRef, CallExpr *TheCall,
                                      Scope::ScopeFlags NeededScopeFlags,
                                      unsigned DiagID) {
   // Scopes aren't available during instantiation. Fortunately, builtin
   // functions cannot be template args so they cannot be formed through template
   // instantiation. Therefore checking once during the parse is sufficient.
   if (SemaRef.inTemplateInstantiation())
     return false;
#include "clang/Basic/Builtins.def"
   case Builtin::BI__builtin___snprintf_chk:
   case Builtin::BI__builtin___vsnprintf_chk:
     SemaBuiltinMemChkCall(*this, FDecl, TheCall, 1, 3);
     break;
   case Builtin::BI__builtin_call_with_static_chain:
     if (SemaBuiltinCallWithStaticChain(*this, TheCall))
       return ExprError();
     break;
+  case Builtin::BI__builtin_always_inline:
+  case Builtin::BI__builtin_no_inline:
+    if (SemaBuiltinCallWithInline(*this, TheCall))
+      return ExprError();
+    break;
   case Builtin::BI__exception_code:
   case Builtin::BI_exception_code:
     if (SemaBuiltinSEHScopeCheck(*this, TheCall, Scope::SEHExceptScope,
                                  diag::err_seh___except_block))
       return ExprError();
     break;
   case Builtin::BI__exception_info:
   case Builtin::BI_exception_info:

test/CodeGenCXX/inline-builtin-call.cpp

+// RUN: %clang_cc1 -std=c++17 -O0 -disable-O0-optnone -triple x86_64-unknown-linux -emit-llvm -o - %s | FileCheck %s
+
+struct S {
+  S() {}
+  void foo();
+  void bar(int);
+  int baz(float, char);
+  static void lol();
+
+  virtual void virt(){};
+  void operator++();
+  friend S operator+(const S &, const S &);
+};
+
+void test_always() {
+  S s;
+
+  // CHECK: call void @_ZN1S3fooEv({{.*}}) [[ATTR_ALWAYS:#[0-9]+]]
+  __builtin_always_inline(s.foo());
+
+  // CHECK-NOT: call void @_ZN1S3fooEv({{.*}}) [[ATTR_ALWAYS]]
+  // CHECK: call void @_ZN1S3fooEv({{.*}})
+  s.foo();
+
+  // CHECK: call void @_ZN1S3barEi({{.*}}) [[ATTR_ALWAYS]]
+  __builtin_always_inline(s.bar(42));
+
+  // CHECK: call i32 @_ZN1S3bazEfc({{.*}}) [[ATTR_ALWAYS]]
+  __builtin_always_inline(s.baz(3.14f, 'x'));
+
+  // CHECK: call i32 @_ZN1S3bazEfc({{.*}}) [[ATTR_ALWAYS]]
+  // CHECK-NOT: call void @_ZN1S3barEi({{.*}}) [[ATTR_ALWAYS]]
+  // CHECK: call void @_ZN1S3barEi({{.*}})
+  s.bar(__builtin_always_inline(s.baz(3.14f, 'x')));
+
+  // CHECK-NOT: call i32 @_ZN1S3bazEfc({{.*}}) [[ATTR_ALWAYS]]
+  // CHECK: call i32 @_ZN1S3bazEfc({{.*}})
+  // CHECK: call void @_ZN1S3barEi({{.*}}) [[ATTR_ALWAYS]]
+  __builtin_always_inline(s.bar(s.baz(3.14f, 'x')));
+
+  S s2;
+
+  // CHECK-NOT: call void @_ZN1SppEv({{.*}}) [[ATTR_ALWAYS]]
+  // CHECK: call void @_ZN1SppEv({{.*}})
+  ++s2;
+  // CHECK: call void @_ZN1SppEv({{.*}}) [[ATTR_ALWAYS]]
+  __builtin_always_inline(++s2);
+
+  // CHECK: call void @_ZplRK1SS1_({{.*}}) [[ATTR_ALWAYS]]
+  __builtin_always_inline(s + s2);
+
+  // CHECK: call void @_ZN1S3lolEv({{.*}}) [[ATTR_ALWAYS]]
+  __builtin_always_inline(s.lol());
+
+  // CHECK: call void @_ZN1S3lolEv({{.*}}) [[ATTR_ALWAYS]]
+  __builtin_always_inline(S::lol());
+}
+
+void test_no_inline() {
+  S s;
+
+  // CHECK: call void @_ZN1S3fooEv({{.*}}) [[ATTR_NO:#[0-9]+]]
+  __builtin_no_inline(s.foo());
+
+  // CHECK-NOT: call void @_ZN1S3fooEv({{.*}}) [[ATTR_NO]]
+  // CHECK: call void @_ZN1S3fooEv({{.*}})
+  s.foo();
+
+  // CHECK: call void @_ZN1S3barEi({{.*}}) [[ATTR_NO]]
+  __builtin_no_inline(s.bar(42));
+
+  // CHECK: call i32 @_ZN1S3bazEfc({{.*}}) [[ATTR_NO]]
+  __builtin_no_inline(s.baz(3.14f, 'x'));
+
+  // CHECK: call i32 @_ZN1S3bazEfc({{.*}}) [[ATTR_NO]]
+  // CHECK-NOT: call void @_ZN1S3barEi({{.*}}) [[ATTR_NO]]
+  // CHECK: call void @_ZN1S3barEi({{.*}})
+  s.bar(__builtin_no_inline(s.baz(3.14f, 'x')));
+
+  // CHECK-NOT: call i32 @_ZN1S3bazEfc({{.*}}) [[ATTR_NO]]
+  // CHECK: call i32 @_ZN1S3bazEfc({{.*}})
+  // CHECK: call void @_ZN1S3barEi({{.*}}) [[ATTR_NO]]
+  __builtin_no_inline(s.bar(s.baz(3.14f, 'x')));
+
+  S s2;
+
+  // CHECK-NOT: call void @_ZN1SppEv({{.*}}) [[ATTR_NO]]
+  // CHECK: call void @_ZN1SppEv({{.*}})
+  ++s2;
+  // CHECK: call void @_ZN1SppEv({{.*}}) [[ATTR_NO]]
+  __builtin_no_inline(++s2);
+
+  // CHECK: call void @_ZplRK1SS1_({{.*}}) [[ATTR_NO]]
+  __builtin_no_inline(s + s2);
+
+  // CHECK: call void @_ZN1S3lolEv({{.*}}) [[ATTR_NO]]
+  __builtin_no_inline(s.lol());
+
+  // CHECK: call void @_ZN1S3lolEv({{.*}}) [[ATTR_NO]]
+  __builtin_no_inline(S::lol());
+}
+
+S &getS();
+
+void test_indirect() {
+  auto fptr = &test_always;
+  // CHECK: call void %{{.*}} [[ATTR_ALWAYS]]
+  __builtin_always_inline(fptr());
+  // CHECK: call void %{{.*}} [[ATTR_NO]]
+  __builtin_no_inline(fptr());
+
+  // CHECK: call dereferenceable(8) %struct.S* @_Z4getSv()
+  S &s = getS();
+  // CHECK: call void %{{.*}} [[ATTR_ALWAYS]]
+  __builtin_always_inline(s.virt());
+  // CHECK: call void %{{.*}} [[ATTR_NO]]
+  __builtin_no_inline(s.virt());
+
+  auto mptr = &S::foo;
+  // CHECK: call void %{{.*}} [[ATTR_ALWAYS]]
+  __builtin_always_inline((s.*mptr)());
+  // CHECK: call void %{{.*}} [[ATTR_NO]]
+  __builtin_no_inline((s.*mptr)());
+}
+
+// CHECK: attributes [[ATTR_ALWAYS]] = { alwaysinline }
+// CHECK: attributes [[ATTR_NO]] = { noinline }

test/Sema/inline-builtins.c

+// RUN: %clang_cc1 -fsyntax-only -fblocks -verify %s
+
+void foo();
+
+void test() {
+  void (*bar)() = &foo;
+  __builtin_always_inline(foo());
+  __builtin_no_inline(foo());
+
+  __builtin_always_inline(bar());
+  __builtin_no_inline(bar());
+  __builtin_always_inline((*bar)());
+  __builtin_no_inline((*bar)());
+
+  // clang-format off
+  __builtin_always_inline(^{}()); // expected-error {{argument to __builtin_always_inline must not be a block call}}
+  __builtin_no_inline(^{}()); // expected-error {{argument to __builtin_no_inline must not be a block call}}
+  // clang-format on
+
+  __builtin_always_inline(__builtin_always_inline(foo())); // expected-error {{argument to __builtin_always_inline must not be a builtin call}}
+  __builtin_always_inline(__builtin_no_inline(foo()));     // expected-error {{argument to __builtin_always_inline must not be a builtin call}}
+  __builtin_no_inline(__builtin_no_inline(foo()));         // expected-error {{argument to __builtin_no_inline must not be a builtin call}}
+  __builtin_no_inline(__builtin_always_inline(foo()));     // expected-error {{argument to __builtin_no_inline must not be a builtin call}}
+}

test/SemaCXX/inline-builtins.cpp

+// RUN: %clang_cc1 -std=c++17 -fsyntax-only -verify %s
+
+struct S {
+  S();
+  void foo();
+  void bar(int);
+  int baz(float, char);
+  static void lol();
+
+  virtual void virt();
+  void operator++();
+  friend S operator+(const S &, const S &);
+};
+
+S operator"" _s(unsigned long long);
+
+S &getS();
+
+void test_positive() {
+  S &s = getS();
+  __builtin_always_inline(s.foo());
+  __builtin_no_inline(s.foo());
+
+  __builtin_always_inline(getS().foo());
+  __builtin_no_inline(getS().foo());
+
+  __builtin_always_inline(getS()).foo();
+  __builtin_no_inline(getS()).foo();
+
+  __builtin_always_inline(s.bar(1));
+  __builtin_no_inline(s.bar(1));
+
+  int a = __builtin_always_inline(s.baz(3.14, 'a'));
+  int b = __builtin_no_inline(s.baz(3.14, 'a'));
+
+  __builtin_always_inline(s.bar(s.baz(3.14, 'a')));
+  __builtin_no_inline(s.bar(s.baz(3.14, 'a')));
+
+  s.bar(__builtin_always_inline(s.baz(3.14, 'a')));
+  s.bar(__builtin_no_inline(s.baz(3.14, 'a')));
+
+  __builtin_always_inline(s.lol());
+  __builtin_no_inline(s.lol());
+
+  __builtin_always_inline(S::lol());
+  __builtin_no_inline(S::lol());
+
+  __builtin_always_inline(s.virt());
+  __builtin_no_inline(s.virt());
+
+  __builtin_always_inline(++s);
+  __builtin_no_inline(++s);
+
+  __builtin_always_inline(s.operator++());
+  __builtin_no_inline(s.operator++());
+
+  S s1;
+  __builtin_always_inline(s + s1);
+  __builtin_no_inline(s + s1);
+
+  auto mptr = &S::foo;
+  __builtin_always_inline((s.*mptr)());
+  __builtin_no_inline((s.*mptr)());
+
+  __builtin_always_inline(s.~S());
+  __builtin_no_inline(s.~S());
+
+  __builtin_always_inline([] {}());
+  __builtin_no_inline([] {}());
+
+  auto lam = [&a, b](S &ss) { return ++a + b; };
+  int c = __builtin_always_inline(lam(s1));
+  int d = __builtin_no_inline(lam(s1));
+}
+
+template <typename T>
+void callDtorAlways(T &t) {
+  __builtin_always_inline(t.~T()); // expected-error {{argument to __builtin_always_inline must not be a pseudo-destructor call}}
+}
+template <typename T>
+void callDtorNo(T &t) {
+  __builtin_no_inline(t.~T()); // expected-error {{argument to __builtin_no_inline must not be a pseudo-destructor call}}
+}
+
+void test_errors() {
+  using I = int;
+  I e = 1;
+  __builtin_always_inline(e.~I()); // expected-error {{argument to __builtin_always_inline must not be a pseudo-destructor call}}
+  __builtin_no_inline(e.~I());     // expected-error {{argument to __builtin_no_inline must not be a pseudo-destructor call}}
+
+  S s;
+  callDtorAlways(s);
+  callDtorNo(s);
+
+  callDtorAlways(e); // expected-note {{in instantiation of function template specialization 'callDtorAlways<int>' requested here}}
+  callDtorNo(e);     // expected-note {{in instantiation of function template specialization 'callDtorNo<int>' requested here}}
+
+  // TODO: Support User Defined Literals.
+  S s2 = __builtin_always_inline(1_s); // expected-error {{argument to __builtin_always_inline must be a function, member function, or operator call}}
+  s2 = __builtin_no_inline(1_s);       // expected-error {{argument to __builtin_no_inline must be a function, member function, or operator call}}
+
+  // TODO: This should be handled as well.
+  S s3 = __builtin_always_inline(S()); // expected-error {{argument to __builtin_always_inline must be a function, member function, or operator call}}
+  S s4 = __builtin_no_inline(S());     // expected-error {{argument to __builtin_no_inline must be a function, member function, or operator call}}
+  S s5 = __builtin_always_inline(S{}); // expected-error {{argument to __builtin_always_inline must be a function, member function, or operator call}}
+  S s6 = __builtin_no_inline(S{});     // expected-error {{argument to __builtin_no_inline must be a function, member function, or operator call}}
+}
+
+constexpr int f1() { return 1; }
+
+// TODO: It should be possible to make the inline intrinsics constexpr-friendly.
+constexpr int f2(int x) {                   // expected-error {{constexpr function never produces a constant expression}}
+  return x + __builtin_always_inline(f1()); // expected-note {{subexpression not valid in a constant expression}}
+}
+
+template <typename T, typename U>
+struct IsSame { static constexpr bool value = false; };
+
+template <typename T>
+struct IsSame<T, T> { static constexpr bool value = true; };
+
+void test_unevaluated_context() {
+  static_assert(IsSame<decltype(getS()), decltype(__builtin_always_inline(getS()))>::value, "");
+  static_assert(IsSame<decltype(getS()), decltype(__builtin_no_inline(getS()))>::value, "");
+
+  static_assert(!IsSame<decltype(getS()), decltype(__builtin_always_inline(getS().foo()))>::value, "");
+  static_assert(!IsSame<decltype(getS()), decltype(__builtin_no_inline(getS().foo()))>::value, "");
+}