[Clang] always_inline statement attribute

[xbolva00]

Summary

Motivation:

int test(int x, int y) {
    int r = 0;
    [[clang::always_inline]] r += foo(x, y); // force compiler to inline this function here
    return r;
}

In 2018, @kuhar proposed "Introduce per-callsite inline intrinsics" in https://reviews.llvm.org/D51200 to solve this motivation case (and many others).

This patch solves this problem with call site attribute. "noinline" statement attribute already landed in D119061. Also, some LLVM Inliner fixes landed so call site attribute is stronger than function attribute.

[xbolva00]

Version for review.

[xbolva00]

ping

(I will fix clang format nits later)

[xbolva00]

Ping

[ChuanqiXu]

Cool. Given D119061 is landed, I think this one should be good.

[aaron.ballman]

clang/include/clang/Basic/Attr.td

714

I'm not certain we should add ErrorDiag here. That's likely to break existing code that's ignoring the attribute.

clang/include/clang/Basic/AttrDocs.td

6246

We should document more clearly (perhaps just with an example) what "contains calls" means. e.g.,

[[clang::always_inline]] return foo(bar(), baz(), bing() + 12);

Do we attempt to inline everything? Just the simple calls with no expressions? Just the call to foo()?

It's probably also worth reminding people that a declaration statement, which is a statement, is not a statement that can have an attribute associated with it (the attribute applies to the declaration, not the statement in that case). So this won't work, and it's not a bug:

[[clang::always_inline]] int i = bar();

(If this case matters to you, you could have the attribute also apply to VarDecl or some subset of that.)

clang/lib/CodeGen/CGCall.cpp

5252–5253

You should fix the clang-format issue here.

clang/lib/CodeGen/CGStmt.cpp

695

clang-format here as well.

clang/lib/Sema/SemaStmtAttr.cpp

247

Should we create this after the early return below so that we don't do the work of finding the call exprs just to ignore the results because the spelling was wrong?

[xbolva00]

Addressed review feedback.

[xbolva00]

clang/include/clang/Basic/Attr.td

714

Yeah. WarnDiag.

clang/include/clang/Basic/AttrDocs.td

6246

Right, doc updated.

(If this case matters to you, you could have the attribute also apply to VarDecl or some subset of that.)

Not so much currently.

[xbolva00]

ping

[aaron.ballman]

You should also add a release note for the new functionality, but other than that and a nit, this LGTM.

clang/include/clang/Basic/AttrDocs.td

6272

Declaration statement -> A declaration statement

[xbolva00]

fixed nit

clang/include/clang/Basic/Attr.td

 
 def AlignNatural : InheritableAttr {
   // This attribute has no spellings as it is only ever created implicitly.
   let Spellings = [];
   let SemaHandler = 0;
   let Documentation = [Undocumented];
 }
 
-def AlwaysInline : InheritableAttr {
-  let Spellings = [GCC<"always_inline">, Keyword<"__forceinline">];
-  let Subjects = SubjectList<[Function]>;
+def AlwaysInline : DeclOrStmtAttr {
+  let Spellings = [GCC<"always_inline">, CXX11<"clang", "always_inline">,
+                   C2x<"clang", "always_inline">, Keyword<"__forceinline">];
+  let Accessors = [Accessor<"isClangAlwaysInline", [CXX11<"clang", "always_inline">,
+                                                    C2x<"clang", "always_inline">]>];
+  let Subjects = SubjectList<[Function, Stmt], ErrorDiag,
+                             "functions and statements">;
   let Documentation = [AlwaysInlineDocs];
 }
 
 def Artificial : InheritableAttr {
   let Spellings = [GCC<"artificial">];
   let Subjects = SubjectList<[InlineFunction]>;
   let Documentation = [ArtificialDocs];
   let SimpleHandler = 1;

clang/include/clang/Basic/AttrDocs.td

 }
 
 def AlwaysInlineDocs : Documentation {
   let Category = DocCatFunction;
   let Content = [{
 Inlining heuristics are disabled and inlining is always attempted regardless of
 optimization level.
 
+``[[clang::always_inline]]`` spelling can be used as a statement attribute; other
+spellings of the attribute are not supported on statements. If a statement is
+marked ``[[clang::always_inline]]`` and contains calls, the compiler attempts
+to inline those calls.
+
+.. code-block:: c
+
+  int example(void) {
+    int r;
+    [[clang::always_inline]] foo();
+    [[clang::always_inline]] r = bar();
+    return r;
+  }
+
 Does not guarantee that inline substitution actually occurs.
 
 <ins>Note: applying this attribute to a coroutine at the `-O0` optimization level
 has no effect; other optimization levels may only partially inline and result in a
 diagnostic.</ins>
 
 See also `the Microsoft Docs on Inline Functions`_, `the GCC Common Function
 Attribute docs`_, and `the GCC Inline docs`_.
 
 .. _the Microsoft Docs on Inline Functions: https://docs.microsoft.com/en-us/cpp/cpp/inline-functions-cpp
 .. _the GCC Common Function Attribute docs: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html
 .. _the GCC Inline docs: https://gcc.gnu.org/onlinedocs/gcc/Inline.html
 
 }];
   let Heading = "always_inline, __force_inline";
 }
 
 def EnforceTCBDocs : Documentation {
   let Category = DocCatFunction;
   let Content = [{
   The ``enforce_tcb`` attribute can be placed on functions to enforce that a
   trusted compute base (TCB) does not call out of the TCB. This generates a
   warning every time a function not marked with an ``enforce_tcb`` attribute is

clang/include/clang/Basic/DiagnosticSemaKinds.td

 def warn_attribute_nonnull_no_pointers : Warning<
   "'nonnull' attribute applied to function with no pointer arguments">,
   InGroup<IgnoredAttributes>;
 def warn_attribute_nonnull_parm_no_args : Warning<
   "'nonnull' attribute when used on parameters takes no arguments">,
   InGroup<IgnoredAttributes>;
 def warn_function_stmt_attribute_precedence : Warning<
   "statement attribute %0 has higher precedence than function attribute "
-  "'%select{always_inline|flatten}1'">,
+  "'%select{always_inline|flatten|noinline}1'">,
   InGroup<IgnoredAttributes>;
 def note_declared_nonnull : Note<
   "declared %select{'returns_nonnull'|'nonnull'}0 here">;
 def warn_attribute_sentinel_named_arguments : Warning<
   "'sentinel' attribute requires named arguments">,
   InGroup<IgnoredAttributes>;
 def warn_attribute_sentinel_not_variadic : Warning<
   "'sentinel' attribute only supported for variadic %select{functions|blocks}0">,

clang/lib/CodeGen/CGCall.cpp

   // Add call-site nomerge attribute if exists.
   if (InNoMergeAttributedStmt)
     Attrs = Attrs.addFnAttribute(getLLVMContext(), llvm::Attribute::NoMerge);
 
   // Add call-site noinline attribute if exists.
   if (InNoInlineAttributedStmt)
     Attrs = Attrs.addFnAttribute(getLLVMContext(), llvm::Attribute::NoInline);
 
+  // Add call-site always_inline attribute if exists.
+  if (InAlwaysInlineAttributedStmt)
+    Attrs = Attrs.addFnAttribute(getLLVMContext(), llvm::Attribute::AlwaysInline);
+
   // Apply some call-site-specific attributes.
   // TODO: work this into building the attribute set.
 
   // Apply always_inline to all calls within flatten functions.
   // FIXME: should this really take priority over __try, below?
   if (CurCodeDecl && CurCodeDecl->hasAttr<FlattenAttr>() &&
       !InNoInlineAttributedStmt &&
       !(TargetDecl && TargetDecl->hasAttr<NoInlineAttr>())) {

clang/lib/CodeGen/CGStmt.cpp

if (getLangOpts().EHAsynch && S.isSideEntry())
     EmitSehCppScopeBegin();
 
   EmitStmt(S.getSubStmt());
 }
 
 void CodeGenFunction::EmitAttributedStmt(const AttributedStmt &S) {
   bool nomerge = false;
   bool noinline = false;
+  bool alwaysinline = false;
   const CallExpr *musttail = nullptr;
 
   for (const auto *A : S.getAttrs()) {
-    if (A->getKind() == attr::NoMerge) {
+    switch (A->getKind()) {
+    default:
+      break;
+    case attr::NoMerge:
       nomerge = true;
-    }
-    if (A->getKind() == attr::NoInline) {
+      break;
+    case attr::NoInline:
       noinline = true;
-    }
-    if (A->getKind() == attr::MustTail) {
+      break;
+    case attr::AlwaysInline:
+      alwaysinline = true;
+      break;
+    case attr::MustTail:
       const Stmt *Sub = S.getSubStmt();
       const ReturnStmt *R = cast<ReturnStmt>(Sub);
       musttail = cast<CallExpr>(R->getRetValue()->IgnoreParens());
+      break;
     }
   }
   SaveAndRestore<bool> save_nomerge(InNoMergeAttributedStmt, nomerge);
   SaveAndRestore<bool> save_noinline(InNoInlineAttributedStmt, noinline);
+  SaveAndRestore<bool> save_alwaysinline(InAlwaysInlineAttributedStmt, alwaysinline);
   SaveAndRestore<const CallExpr *> save_musttail(MustTailCall, musttail);
   EmitStmt(S.getSubStmt(), S.getAttrs());
 }
 
 void CodeGenFunction::EmitGotoStmt(const GotoStmt &S) {
   // If this code is reachable then emit a stop point (if generating
   // debug info). We have to do this ourselves because we are on the
   // "simple" statement path.

clang/lib/CodeGen/CodeGenFunction.h

public:
   bool IsInPreservedAIRegion = false;
 
   /// True if the current statement has nomerge attribute.
   bool InNoMergeAttributedStmt = false;
 
   /// True if the current statement has noinline attribute.
   bool InNoInlineAttributedStmt = false;
 
+  /// True if the current statement has always_inline attribute.
+  bool InAlwaysInlineAttributedStmt = false;
+
   // The CallExpr within the current statement that the musttail attribute
   // applies to.  nullptr if there is no 'musttail' on the current statement.
   const CallExpr *MustTailCall = nullptr;
 
   /// Returns true if a function must make progress, which means the
   /// mustprogress attribute can be added.
   bool checkIfFunctionMustProgress() {
     if (CGM.getCodeGenOpts().getFiniteLoops() ==

clang/lib/Sema/SemaStmtAttr.cpp

for (const auto *CallExpr : CEF.getCallExprs()) {
     if (Decl->hasAttr<AlwaysInlineAttr>() || Decl->hasAttr<FlattenAttr>())
       S.Diag(St->getBeginLoc(), diag::warn_function_stmt_attribute_precedence)
           << A << (Decl->hasAttr<AlwaysInlineAttr>() ? 0 : 1);
   }
 
   return ::new (S.Context) NoInlineAttr(S.Context, A);
 }
 
+static Attr *handleAlwaysInlineAttr(Sema &S, Stmt *St, const ParsedAttr &A,
+                                    SourceRange Range) {
+  AlwaysInlineAttr AIA(S.Context, A);
+  CallExprFinder CEF(S, St);
+
+  if (!AIA.isClangAlwaysInline()) {
+    S.Diag(St->getBeginLoc(), diag::warn_function_attribute_ignored_in_stmt)
+        << "[[clang::always_inline]]";
+    return nullptr;
+  }
+
+  if (!CEF.foundCallExpr()) {
+    S.Diag(St->getBeginLoc(), diag::warn_attribute_ignored_no_calls_in_stmt)
+        << A;
+    return nullptr;
+  }
+
+  for (const auto *CallExpr : CEF.getCallExprs()) {
+    const Decl *Decl = CallExpr->getCalleeDecl();
+    if (Decl->hasAttr<NoInlineAttr>() || Decl->hasAttr<FlattenAttr>())
+      S.Diag(St->getBeginLoc(), diag::warn_function_stmt_attribute_precedence)
+          << A << (Decl->hasAttr<NoInlineAttr>() ? 2 : 1);
+  }
+
+  return ::new (S.Context) AlwaysInlineAttr(S.Context, A);
+}
+
 static Attr *handleMustTailAttr(Sema &S, Stmt *St, const ParsedAttr &A,
                                 SourceRange Range) {
   // Validation is in Sema::ActOnAttributedStmt().
   return ::new (S.Context) MustTailAttr(S.Context, A);
 }
 
 static Attr *handleLikely(Sema &S, Stmt *St, const ParsedAttr &A,
                           SourceRange Range) {
S.Diag(A.getLoc(), A.isDeclspecAttribute()
         << A << A.getRange();
     return nullptr;
   }
 
   if (S.checkCommonAttributeFeatures(St, A))
     return nullptr;
 
   switch (A.getKind()) {
+  case ParsedAttr::AT_AlwaysInline:
+    return handleAlwaysInlineAttr(S, St, A, Range);
   case ParsedAttr::AT_FallThrough:
     return handleFallThroughAttr(S, St, A, Range);
   case ParsedAttr::AT_LoopHint:
     return handleLoopHintAttr(S, St, A, Range);
   case ParsedAttr::AT_OpenCLUnrollHint:
     return handleOpenCLUnrollHint(S, St, A, Range);
   case ParsedAttr::AT_Suppress:
     return handleSuppressAttr(S, St, A, Range);

clang/test/CodeGen/attr-alwaysinline.cpp

+// RUN: %clang_cc1 -S -emit-llvm %s -triple x86_64-unknown-linux-gnu -o - | FileCheck %s
+
+bool bar();
+void f(bool, bool);
+void g(bool);
+
+void foo(int i) {
+  [[clang::always_inline]] bar();
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[ALWAYSINLINEATTR:[0-9]+]]
+  [[clang::always_inline]] (i = 4, bar());
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[ALWAYSINLINEATTR]]
+  [[clang::always_inline]] (void)(bar());
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[ALWAYSINLINEATTR]]
+  [[clang::always_inline]] f(bar(), bar());
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[ALWAYSINLINEATTR]]
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[ALWAYSINLINEATTR]]
+// CHECK: call void @_Z1fbb({{.*}}) #[[ALWAYSINLINEATTR]]
+  [[clang::always_inline]] for (bar(); bar(); bar()) {}
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[ALWAYSINLINEATTR]]
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[ALWAYSINLINEATTR]]
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[ALWAYSINLINEATTR]]
+  bar();
+// CHECK: call noundef zeroext i1 @_Z3barv()
+  [[gnu::always_inline]] bar();
+// CHECK: call noundef zeroext i1 @_Z3barv()
+}
+
+struct S {
+  friend bool operator==(const S &LHS, const S &RHS);
+};
+
+void func(const S &s1, const S &s2) {
+  [[clang::always_inline]]g(s1 == s2);
+// CHECK: call noundef zeroext i1 @_ZeqRK1SS1_({{.*}}) #[[ALWAYSINLINEATTR]]
+// CHECK: call void @_Z1gb({{.*}}) #[[ALWAYSINLINEATTR]]
+  bool b;
+  [[clang::always_inline]] b = s1 == s2;
+// CHECK: call noundef zeroext i1 @_ZeqRK1SS1_({{.*}}) #[[ALWAYSINLINEATTR]]
+}
+
+// CHECK: attributes #[[ALWAYSINLINEATTR]] = { alwaysinline }

clang/test/Parser/objc-implementation-attrs.m

 // RUN: %clang_cc1 -triple x86_64-apple-macosx10.14.0 -fsyntax-only -Wno-objc-root-class -verify %s
 
 @interface I1 @end
 
-// expected-warning@+1 {{'always_inline' attribute only applies to functions}}
+// expected-error@+1 {{'always_inline' attribute only applies to functions and statements}}
 __attribute__((always_inline))
 @implementation I1 @end
 
-// expected-warning@+1 {{'always_inline' attribute only applies to functions}}
+// expected-error@+1 {{'always_inline' attribute only applies to functions and statements}}
 __attribute__((always_inline))
 @implementation I1 (MyCat) @end
 
-// expected-warning@+1 {{'always_inline' attribute only applies to functions}}
+// expected-error@+1 {{'always_inline' attribute only applies to functions and statements}}
 __attribute__((always_inline))
 // expected-warning@+1 {{cannot find interface declaration for 'I2'}}
 @implementation I2 @end
 
 // expected-error@+1 {{only applies to Objective-C interfaces}}
 __attribute__((objc_root_class))
 // expected-warning@+1 {{cannot find interface declaration for 'I3'}}
 @implementation I3 @end

clang/test/Sema/attr-alwaysinline.cpp

+// RUN: %clang_cc1 -verify -fsyntax-only %s
+
+int bar();
+
+[[gnu::always_inline]] void always_inline_fn(void) {}
+[[gnu::flatten]] void flatten_fn(void) {}
+
+[[gnu::noinline]] void noinline_fn(void) {}
+
+void foo() {
+  [[clang::always_inline]] bar();
+  [[clang::always_inline(0)]] bar(); // expected-error {{'always_inline' attribute takes no arguments}}
+  int x;
+  [[clang::always_inline]] x = 0;          // expected-warning {{'always_inline' attribute is ignored because there exists no call expression inside the statement}}
+  [[clang::always_inline]] { asm("nop"); } // expected-warning {{'always_inline' attribute is ignored because there exists no call expression inside the statement}}
+  [[clang::always_inline]] label : x = 1;  // expected-error {{'always_inline' attribute only applies to functions and statements}}
+
+  [[clang::always_inline]] always_inline_fn();
+  [[clang::always_inline]] noinline_fn(); // expected-warning {{statement attribute 'always_inline' has higher precedence than function attribute 'noinline'}}
+  [[clang::always_inline]] flatten_fn();  // expected-warning {{statement attribute 'always_inline' has higher precedence than function attribute 'flatten'}}
+
+  [[gnu::always_inline]] bar();         // expected-warning {{attribute is ignored on this statement as it only applies to functions; use '[[clang::always_inline]]' on statements}}
+  __attribute__((always_inline)) bar(); // expected-warning {{attribute is ignored on this statement as it only applies to functions; use '[[clang::always_inline]]' on statements}}
+}
+
+[[clang::always_inline]] static int i = bar(); // expected-error {{'always_inline' attribute only applies to functions and statements}}