[Clang] always_inline statement attribute

[xbolva00]

Summary

Motivation:

int test(int x, int y) {
    int r = 0;
    [[clang::always_inline]] r += foo(x, y); // force compiler to inline this function here
    return r;
}

In 2018, @kuhar proposed "Introduce per-callsite inline intrinsics" in https://reviews.llvm.org/D51200 to solve this motivation case (and many others).

This patch solves this problem with call site attribute. "noinline" statement attribute already landed in D119061. Also, some LLVM Inliner fixes landed so call site attribute is stronger than function attribute.

[xbolva00]

Version for review.

[xbolva00]

ping

(I will fix clang format nits later)

[xbolva00]

Ping

[ChuanqiXu]

Cool. Given D119061 is landed, I think this one should be good.

[aaron.ballman]

clang/include/clang/Basic/Attr.td

714

I'm not certain we should add ErrorDiag here. That's likely to break existing code that's ignoring the attribute.

clang/include/clang/Basic/AttrDocs.td

6246

We should document more clearly (perhaps just with an example) what "contains calls" means. e.g.,

[[clang::always_inline]] return foo(bar(), baz(), bing() + 12);

Do we attempt to inline everything? Just the simple calls with no expressions? Just the call to foo()?

It's probably also worth reminding people that a declaration statement, which is a statement, is not a statement that can have an attribute associated with it (the attribute applies to the declaration, not the statement in that case). So this won't work, and it's not a bug:

[[clang::always_inline]] int i = bar();

(If this case matters to you, you could have the attribute also apply to VarDecl or some subset of that.)

clang/lib/CodeGen/CGCall.cpp

5252–5253

You should fix the clang-format issue here.

clang/lib/CodeGen/CGStmt.cpp

686

clang-format here as well.

clang/lib/Sema/SemaStmtAttr.cpp

220

Should we create this after the early return below so that we don't do the work of finding the call exprs just to ignore the results because the spelling was wrong?

[xbolva00]

Addressed review feedback.

[xbolva00]

clang/include/clang/Basic/Attr.td

714

Yeah. WarnDiag.

clang/include/clang/Basic/AttrDocs.td

6246

Right, doc updated.

(If this case matters to you, you could have the attribute also apply to VarDecl or some subset of that.)

Not so much currently.

[xbolva00]

ping

[aaron.ballman]

You should also add a release note for the new functionality, but other than that and a nit, this LGTM.

clang/include/clang/Basic/AttrDocs.td

6258

Declaration statement -> A declaration statement

[xbolva00]

fixed nit

clang/include/clang/Basic/Attr.td

 
 def Convergent : InheritableAttr {
   let Spellings = [Clang<"convergent">];
   let Subjects = SubjectList<[Function]>;
   let Documentation = [ConvergentDocs];
   let SimpleHandler = 1;
 }
 
-def NoInline : InheritableAttr {
-  let Spellings = [GCC<"noinline">, Declspec<"noinline">];
-  let Subjects = SubjectList<[Function]>;
-  let Documentation = [Undocumented];
+def NoInline : DeclOrStmtAttr {
+  let Spellings = [GCC<"noinline">, CXX11<"clang", "noinline">,
+                   C2x<"clang", "noinline">, Declspec<"noinline">];
+  let Accessors = [Accessor<"isClangNoInline", [CXX11<"clang", "noinline">,
+                                                C2x<"clang", "noinline">]>];
+  let Documentation = [NoInlineDocs];
+  let Subjects = SubjectList<[Function, Stmt], ErrorDiag,
+                             "functions and statements">;
   let SimpleHandler = 1;
 }
 
 def NoMips16 : InheritableAttr, TargetSpecificAttr<TargetMips32> {
   let Spellings = [GCC<"nomips16">];
   let Subjects = SubjectList<[Function], ErrorDiag>;
   let Documentation = [Undocumented];
   let SimpleHandler = 1;

clang/include/clang/Basic/AttrDocs.td

 over the tradeoff between code size and debug information precision. 
 
 ``nomerge`` attribute can also be used as function attribute to prevent all 
 calls to the specified function from merging. It has no effect on indirect 
 calls.
   }];
 }
 
+def NoInlineDocs : Documentation {
+  let Category = DocCatFunction;
+  let Content = [{
+This function attribute suppresses the inlining of a function at the call sites
+of the function.
+
+``[[clang::noinline]]`` spelling can be used as a statement attribute; other
+spellings of the attribute are not supported on statements. If a statement is
+marked ``[[clang::noinline]]`` and contains calls, those calls inside the
+statement will not be inlined by the compiler.
+
+.. code-block:: c
+
+  int example(void) {
+    int r;
+    [[clang::noinline]] foo();
+    [[clang::noinline]] r = bar();
+    return r;
+  }
+
+  }];
+}
+
 def MustTailDocs : Documentation {
   let Category = DocCatStmt;
   let Content = [{
 If a ``return`` statement is marked ``musttail``, this indicates that the
 compiler must generate a tail call for the program to be correct, even when
 optimizations are disabled. This guarantees that the call will not cause
 unbounded stack growth if it is part of a recursive cycle in the call graph.
 
``macos``
   specified by the ``-mmacosx-version-min=*version*`` command-line argument.
   ``macosx`` is supported for backward-compatibility reasons, but it is
   deprecated.
 
 ``tvos``
   Apple's tvOS operating system. The minimum deployment target is specified by
   the ``-mtvos-version-min=*version*`` command-line argument.
 
 ``watchos``

clang/include/clang/Basic/DiagnosticSemaKinds.td

 // Objective-C++
 def err_objc_decls_may_only_appear_in_global_scope : Error<
   "Objective-C declarations may only appear in global scope">;
 def warn_auto_var_is_id : Warning<
   "'auto' deduced as 'id' in declaration of %0">,
   InGroup<DiagGroup<"auto-var-id">>;
 
 // Attributes
-def warn_nomerge_attribute_ignored_in_stmt: Warning<
+def warn_attribute_ignored_no_calls_in_stmt: Warning<
   "%0 attribute is ignored because there exists no call expression inside the "
   "statement">,
   InGroup<IgnoredAttributes>;
 
+def warn_function_attribute_ignored_in_stmt : Warning<
+  "attribute is ignored on this statement as it only applies to functions; "
+  "use '%0' on statements">,
+  InGroup<IgnoredAttributes>;
+
 def err_musttail_needs_trivial_args : Error<
   "tail call requires that the return value, all parameters, and any "
   "temporaries created by the expression are trivially destructible">;
 def err_musttail_needs_call : Error<
   "%0 attribute requires that the return value is the result of a function call"
   >;
 def err_musttail_needs_prototype : Error<
   "%0 attribute requires that both caller and callee functions have a "

 def err_enum_mode_vector_type : Error<
   "mode %0 is not supported for enumeration types">;
 def warn_attribute_nonnull_no_pointers : Warning<
   "'nonnull' attribute applied to function with no pointer arguments">,
   InGroup<IgnoredAttributes>;
 def warn_attribute_nonnull_parm_no_args : Warning<
   "'nonnull' attribute when used on parameters takes no arguments">,
   InGroup<IgnoredAttributes>;
+def warn_function_stmt_attribute_precedence : Warning<
+  "statement attribute %0 has higher precedence than function attribute "
+  "'%select{always_inline|flatten}1'">,
+  InGroup<IgnoredAttributes>;
 def note_declared_nonnull : Note<
   "declared %select{'returns_nonnull'|'nonnull'}0 here">;
 def warn_attribute_sentinel_named_arguments : Warning<
   "'sentinel' attribute requires named arguments">,
   InGroup<IgnoredAttributes>;
 def warn_attribute_sentinel_not_variadic : Warning<
   "'sentinel' attribute only supported for variadic %select{functions|blocks}0">,
   InGroup<IgnoredAttributes>;

clang/lib/CodeGen/CGCall.cpp

     if (FD->hasAttr<StrictFPAttr>())
       // All calls within a strictfp function are marked strictfp
       Attrs = Attrs.addFnAttribute(getLLVMContext(), llvm::Attribute::StrictFP);
 
   // Add call-site nomerge attribute if exists.
   if (InNoMergeAttributedStmt)
     Attrs = Attrs.addFnAttribute(getLLVMContext(), llvm::Attribute::NoMerge);
 
+  // Add call-site noinline attribute if exists.
+  if (InNoInlineAttributedStmt)
+    Attrs = Attrs.addFnAttribute(getLLVMContext(), llvm::Attribute::NoInline);
+
   // Apply some call-site-specific attributes.
   // TODO: work this into building the attribute set.
 
   // Apply always_inline to all calls within flatten functions.
   // FIXME: should this really take priority over __try, below?
   if (CurCodeDecl && CurCodeDecl->hasAttr<FlattenAttr>() &&
+      !InNoInlineAttributedStmt &&
       !(TargetDecl && TargetDecl->hasAttr<NoInlineAttr>())) {
     Attrs =
         Attrs.addFnAttribute(getLLVMContext(), llvm::Attribute::AlwaysInline);
   }
 
   // Disable inlining inside SEH __try blocks.
   if (isSEHTryScope()) {
     Attrs = Attrs.addFnAttribute(getLLVMContext(), llvm::Attribute::NoInline);

clang/lib/CodeGen/CGStmt.cpp

void CodeGenFunction::EmitLabelStmt(const LabelStmt &S) {
   if (getLangOpts().EHAsynch && S.isSideEntry())
     EmitSehCppScopeBegin();
 
   EmitStmt(S.getSubStmt());
 }
 
 void CodeGenFunction::EmitAttributedStmt(const AttributedStmt &S) {
   bool nomerge = false;
+  bool noinline = false;
   const CallExpr *musttail = nullptr;
 
   for (const auto *A : S.getAttrs()) {
     if (A->getKind() == attr::NoMerge) {
       nomerge = true;
     }
+    if (A->getKind() == attr::NoInline) {
+      noinline = true;
+    }
     if (A->getKind() == attr::MustTail) {
       const Stmt *Sub = S.getSubStmt();
       const ReturnStmt *R = cast<ReturnStmt>(Sub);
       musttail = cast<CallExpr>(R->getRetValue()->IgnoreParens());
     }
   }
   SaveAndRestore<bool> save_nomerge(InNoMergeAttributedStmt, nomerge);
+  SaveAndRestore<bool> save_noinline(InNoInlineAttributedStmt, noinline);
   SaveAndRestore<const CallExpr *> save_musttail(MustTailCall, musttail);
   EmitStmt(S.getSubStmt(), S.getAttrs());
 }
 
 void CodeGenFunction::EmitGotoStmt(const GotoStmt &S) {
   // If this code is reachable then emit a stop point (if generating
   // debug info). We have to do this ourselves because we are on the
   // "simple" statement path.

clang/lib/CodeGen/CodeGenFunction.h

public:
 
   /// True if CodeGen currently emits code inside presereved access index
   /// region.
   bool IsInPreservedAIRegion = false;
 
   /// True if the current statement has nomerge attribute.
   bool InNoMergeAttributedStmt = false;
 
+  /// True if the current statement has noinline attribute.
+  bool InNoInlineAttributedStmt = false;
+
   // The CallExpr within the current statement that the musttail attribute
   // applies to.  nullptr if there is no 'musttail' on the current statement.
   const CallExpr *MustTailCall = nullptr;
 
   /// Returns true if a function must make progress, which means the
   /// mustprogress attribute can be added.
   bool checkIfFunctionMustProgress() {
     if (CGM.getCodeGenOpts().getFiniteLoops() ==

clang/lib/Sema/SemaStmtAttr.cpp

if (Option == LoopHintAttr::VectorizeWidth) {
       llvm_unreachable("bad loop hint");
   }
 
   return LoopHintAttr::CreateImplicit(S.Context, Option, State, ValueExpr, A);
 }
 
 namespace {
 class CallExprFinder : public ConstEvaluatedExprVisitor<CallExprFinder> {
-  bool FoundCallExpr = false;
+  bool FoundAsmStmt = false;
+  std::vector<const CallExpr *> CallExprs;
 
 public:
   typedef ConstEvaluatedExprVisitor<CallExprFinder> Inherited;
 
   CallExprFinder(Sema &S, const Stmt *St) : Inherited(S.Context) { Visit(St); }
 
-  bool foundCallExpr() { return FoundCallExpr; }
+  bool foundCallExpr() { return !CallExprs.empty(); }
+  const std::vector<const CallExpr *> &getCallExprs() { return CallExprs; }
 
-  void VisitCallExpr(const CallExpr *E) { FoundCallExpr = true; }
-  void VisitAsmStmt(const AsmStmt *S) { FoundCallExpr = true; }
+  bool foundAsmStmt() { return FoundAsmStmt; }
+
+  void VisitCallExpr(const CallExpr *E) { CallExprs.push_back(E); }
+
+  void VisitAsmStmt(const AsmStmt *S) { FoundAsmStmt = true; }
 
   void Visit(const Stmt *St) {
     if (!St)
       return;
     ConstEvaluatedExprVisitor<CallExprFinder>::Visit(St);
   }
 };
 } // namespace
 
 static Attr *handleNoMergeAttr(Sema &S, Stmt *St, const ParsedAttr &A,
                                SourceRange Range) {
   NoMergeAttr NMA(S.Context, A);
   CallExprFinder CEF(S, St);
 
-  if (!CEF.foundCallExpr()) {
-    S.Diag(St->getBeginLoc(), diag::warn_nomerge_attribute_ignored_in_stmt)
+  if (!CEF.foundCallExpr() && !CEF.foundAsmStmt()) {
+    S.Diag(St->getBeginLoc(), diag::warn_attribute_ignored_no_calls_in_stmt)
         << A;
     return nullptr;
   }
 
   return ::new (S.Context) NoMergeAttr(S.Context, A);
 }
 
+static Attr *handleNoInlineAttr(Sema &S, Stmt *St, const ParsedAttr &A,
+                                SourceRange Range) {
+  NoInlineAttr NIA(S.Context, A);
+  CallExprFinder CEF(S, St);
+
+  if (!NIA.isClangNoInline()) {
+    S.Diag(St->getBeginLoc(), diag::warn_function_attribute_ignored_in_stmt)
+        << "[[clang::noinline]]";
+    return nullptr;
+  }
+
+  if (!CEF.foundCallExpr()) {
+    S.Diag(St->getBeginLoc(), diag::warn_attribute_ignored_no_calls_in_stmt)
+        << A;
+    return nullptr;
+  }
+
+  for (const auto *CallExpr : CEF.getCallExprs()) {
+    const Decl *Decl = CallExpr->getCalleeDecl();
+    if (Decl->hasAttr<AlwaysInlineAttr>() || Decl->hasAttr<FlattenAttr>())
+      S.Diag(St->getBeginLoc(), diag::warn_function_stmt_attribute_precedence)
+          << A << (Decl->hasAttr<AlwaysInlineAttr>() ? 0 : 1);
+  }
+
+  return ::new (S.Context) NoInlineAttr(S.Context, A);
+}
+
 static Attr *handleMustTailAttr(Sema &S, Stmt *St, const ParsedAttr &A,
                                 SourceRange Range) {
   // Validation is in Sema::ActOnAttributedStmt().
   return ::new (S.Context) MustTailAttr(S.Context, A);
 }
 
 static Attr *handleLikely(Sema &S, Stmt *St, const ParsedAttr &A,
                           SourceRange Range) {
static Attr *ProcessStmtAttribute(Sema &S, Stmt *St, const ParsedAttr &A,
   case ParsedAttr::AT_LoopHint:
     return handleLoopHintAttr(S, St, A, Range);
   case ParsedAttr::AT_OpenCLUnrollHint:
     return handleOpenCLUnrollHint(S, St, A, Range);
   case ParsedAttr::AT_Suppress:
     return handleSuppressAttr(S, St, A, Range);
   case ParsedAttr::AT_NoMerge:
     return handleNoMergeAttr(S, St, A, Range);
+  case ParsedAttr::AT_NoInline:
+    return handleNoInlineAttr(S, St, A, Range);
   case ParsedAttr::AT_MustTail:
     return handleMustTailAttr(S, St, A, Range);
   case ParsedAttr::AT_Likely:
     return handleLikely(S, St, A, Range);
   case ParsedAttr::AT_Unlikely:
     return handleUnlikely(S, St, A, Range);
   default:
     // N.B., ClangAttrEmitter.cpp emits a diagnostic helper that ensures a

clang/test/CodeGen/attr-noinline.cpp

+// RUN: %clang_cc1 -S -emit-llvm %s -triple x86_64-unknown-linux-gnu -o - | FileCheck %s
+
+bool bar();
+void f(bool, bool);
+void g(bool);
+
+static int baz(int x) {
+    return x * 10;
+}
+
+[[clang::noinline]] bool noi() { }
+
+void foo(int i) {
+  [[clang::noinline]] bar();
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[NOINLINEATTR:[0-9]+]]
+  [[clang::noinline]] i = baz(i);
+// CHECK: call noundef i32 @_ZL3bazi({{.*}}) #[[NOINLINEATTR]]
+  [[clang::noinline]] (i = 4, bar());
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[NOINLINEATTR]]
+  [[clang::noinline]] (void)(bar());
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[NOINLINEATTR]]
+  [[clang::noinline]] f(bar(), bar());
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[NOINLINEATTR]]
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[NOINLINEATTR]]
+// CHECK: call void @_Z1fbb({{.*}}) #[[NOINLINEATTR]]
+  [[clang::noinline]] [] { bar(); bar(); }(); // noinline only applies to the anonymous function call
+// CHECK: call void @"_ZZ3fooiENK3$_0clEv"(%class.anon* {{[^,]*}} %ref.tmp) #[[NOINLINEATTR]]
+  [[clang::noinline]] for (bar(); bar(); bar()) {}
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[NOINLINEATTR]]
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[NOINLINEATTR]]
+// CHECK: call noundef zeroext i1 @_Z3barv() #[[NOINLINEATTR]]
+  bar();
+// CHECK: call noundef zeroext i1 @_Z3barv()
+  [[clang::noinline]] noi();
+// CHECK: call noundef zeroext i1 @_Z3noiv()
+  noi();
+// CHECK: call noundef zeroext i1 @_Z3noiv()
+  [[gnu::noinline]] bar();
+// CHECK: call noundef zeroext i1 @_Z3barv()
+}
+
+struct S {
+  friend bool operator==(const S &LHS, const S &RHS);
+};
+
+void func(const S &s1, const S &s2) {
+  [[clang::noinline]]g(s1 == s2);
+// CHECK: call noundef zeroext i1 @_ZeqRK1SS1_({{.*}}) #[[NOINLINEATTR]]
+// CHECK: call void @_Z1gb({{.*}}) #[[NOINLINEATTR]]
+  bool b;
+  [[clang::noinline]] b = s1 == s2;
+// CHECK: call noundef zeroext i1 @_ZeqRK1SS1_({{.*}}) #[[NOINLINEATTR]]
+}
+
+// CHECK: attributes #[[NOINLINEATTR]] = { noinline }

clang/test/Parser/stmt-attributes.c

void foo(int i) {
 
   __attribute__((unused)) switch (i) {         // expected-error {{'unused' attribute cannot be applied to a statement}}
   __attribute__((uuid)) case 0:                // expected-warning {{unknown attribute 'uuid' ignored}}
   __attribute__((visibility(""))) default:         // expected-error {{'visibility' attribute cannot be applied to a statement}}
     __attribute__((carries_dependency)) break; // expected-error {{'carries_dependency' attribute cannot be applied to a statement}}
   }
 
   __attribute__((fastcall)) goto there; // expected-error {{'fastcall' attribute cannot be applied to a statement}}
-  __attribute__((noinline)) there :     // expected-warning {{'noinline' attribute only applies to functions}}
+  __attribute__((noinline)) there :     // expected-error {{'noinline' attribute only applies to functions and statements}}
 
                                     __attribute__((weakref)) return; // expected-error {{'weakref' attribute only applies to variables and functions}}
 
   __attribute__((carries_dependency));            // expected-error {{'carries_dependency' attribute cannot be applied to a statement}}
   __attribute__((carries_dependency)) {}          // expected-error {{'carries_dependency' attribute cannot be applied to a statement}}
   __attribute__((carries_dependency)) if (0) {}   // expected-error {{'carries_dependency' attribute cannot be applied to a statement}}
   __attribute__((carries_dependency)) for (;;);   // expected-error {{'carries_dependency' attribute cannot be applied to a statement}}
   __attribute__((carries_dependency)) do {        // expected-error {{'carries_dependency' attribute cannot be applied to a statement}}

clang/test/Sema/attr-noinline.c

 // RUN: %clang_cc1 %s -verify -fsyntax-only
 
-int a __attribute__((noinline)); // expected-warning {{'noinline' attribute only applies to functions}}
+int a __attribute__((noinline)); // expected-error {{'noinline' attribute only applies to functions and statements}}
 
 void t1(void) __attribute__((noinline));
 
 void t2(void) __attribute__((noinline(2))); // expected-error {{'noinline' attribute takes no arguments}}
 

clang/test/Sema/attr-noinline.cpp

+// RUN: %clang_cc1 -verify -fsyntax-only %s
+
+int bar();
+
+[[gnu::always_inline]] void always_inline_fn(void) { }
+[[gnu::flatten]] void flatten_fn(void) { }
+
+[[gnu::noinline]] void noinline_fn(void) { }
+
+void foo() {
+  [[clang::noinline]] bar();
+  [[clang::noinline(0)]] bar(); // expected-error {{'noinline' attribute takes no arguments}}
+  int x;
+  [[clang::noinline]] x = 0; // expected-warning {{'noinline' attribute is ignored because there exists no call expression inside the statement}}
+  [[clang::noinline]] { asm("nop"); } // expected-warning {{'noinline' attribute is ignored because there exists no call expression inside the statement}}
+  [[clang::noinline]] label: x = 1; // expected-error {{'noinline' attribute only applies to functions and statements}}
+
+
+  [[clang::noinline]] always_inline_fn(); // expected-warning {{statement attribute 'noinline' has higher precedence than function attribute 'always_inline'}}
+  [[clang::noinline]] flatten_fn(); // expected-warning {{statement attribute 'noinline' has higher precedence than function attribute 'flatten'}}
+  [[clang::noinline]] noinline_fn();
+
+  [[gnu::noinline]] bar(); // expected-warning {{attribute is ignored on this statement as it only applies to functions; use '[[clang::noinline]]' on statements}}
+  __attribute__((noinline)) bar(); // expected-warning {{attribute is ignored on this statement as it only applies to functions; use '[[clang::noinline]]' on statements}}
+}
+
+[[clang::noinline]] static int i = bar(); // expected-error {{'noinline' attribute only applies to functions and statements}}