always_inline codegen rewrite

[eugenis]

Summary

Currently always_inline definitions are emitted as (in most cases) an available_externally llvm function with an alwaysinline attribute. This is not exactly right: always_inline functions are NOT available externally, and, for example, libc++ uses this semantics to preserve ABI stability.

Emitting an undefined symbol for an always_inline function is always a bug. The current code can still do it in certain cases.
a. Inliner is an SCC pass. It traverses the graph starting from the roots, which are either main() function, or all externally-visible functions. Inlining does not happen in functions that are not reachable.
b. Dead code elimination is not perfect. There are cases where a function will become unreachable due to some late optimizations and will still be emitted into the binary.

This patch changes the way always_inline functions are emitted in the Clang codegen to ensure this never happens. A function F is emitted as a pair of
a. internal F.inlinefunction() alwaysinline { original function body }
and, depending on the function visibility, either
b1. declare F()
or
b2. define external F() { musttail call F.inlinefunction() }

Frontend ensures that all direct calls go to F.inlinefunction().

This provides a simple invariant that all alwaysinline functions are internal, which can be checked in the IR verifier. Another invariant would be that alwaysinline functions never reach the backend.

This patch is based on ideas by Chandler Carruth and Richard Smith.

[rsmith]

lib/CodeGen/CodeGenModule.cpp

462–463

Do you need to check which operand of the call or invoke you found? For instance:

void f(void (*)());
extern inline __attribute__((always_inline, gnu_inline)) void g() {}
void h() { f(g); } // should not use alwaysinline symbol

520–521

I think it would be better to make a list of always_inline functions rather than actually marking them as always_inline in the IR; that way, you don't need to walk all the functions here, and you don't need to temporarily violate the invariant that only internal/private functions are ever always_inline.

[chandlerc]

FYI, we should send an RFC to llvm-dev about the design change of always_inline and make sure folks generally like the IR-level direction as well. We can point at this review as an example.

I'm happy to write that up and send it if that's useful?

[probinson]

lib/CodeGen/CodeGenModule.cpp

511

What does the debug info look like for the wrapper?

[eugenis]

lib/CodeGen/CodeGenModule.cpp

462–463

Good catch.

511

I don't see any difference from the current behavior. All lines inlined from F.inlinefunction to F are attrubuted directly to F, and inspecting debug info in the binary and IR I don't find any mentions of "inlinefunction".

[rsmith]

lib/CodeGen/CodeGenModule.cpp

528

Does it matter that this traversal is nondeterministic? (Is the nondeterminism visible in the output IR?) Could you use a vector type for AlwaysInlineFunctions? (Does the same function ever actually get added to it twice?) If not, maybe a SetVector would be more appropriate.

[eugenis]

lib/CodeGen/CodeGenModule.cpp

528

Done. Should I make it a "SmallSetVector"? The number of always_inline functions seems very unpredictable and usually quite big, so it would not help, I guess.

[rnk]

lib/CodeGen/CodeGenModule.cpp

484–485

This is a lot of work to do for every always_inline function that got called. Can we do this like:

1. Build SmallVector<Use*> of all non-direct call uses of Fn
2. Return if there are no such uses
3. Build the stub function replacement
4. for (Use *U : IndirectUses) U->set(StubFn)

test/CodeGen/2008-05-19-AlwaysInline.c

1 ↗

(On Diff #32341)

FileCheck?

test/CodeGenCXX/dllimport.cpp

247–250 ↗

(On Diff #32341)

This change will go away if you only create the decl when its used indirectly.

[eugenis]

lib/CodeGen/CodeGenModule.cpp

484–485

This is a very good idea. It's not exactly as easy as that, but it works, see the new code.

test/CodeGen/2008-05-19-AlwaysInline.c

1 ↗

(On Diff #32341)

Actually, this chunk is not needed with your proposed change.
Reverted.

[rnk]

lgtm

lib/CodeGen/CodeGenModule.cpp

463

Commented out code?

479–480

Some debugging code maybe?

[rsmith]

lib/CodeGen/CodeGenModule.cpp

485

I have a slight preference for ".alwaysinline" over ".inlinefunction", since we don't do this for all inline functions.

491–492

Should we check this before we rename the function?

491–492

Should we also skip this if the function has local linkage, regardless of how it's used?

494

I would expect available_externally globals to be considered discardable if unused. Can you fix that in LLVM rather than here?

540–541

Swap over these two, so we never have a non-internal always_inline function.

lib/CodeGen/CodeGenModule.h

505

You only call AddAlwaysInlineFunction when creating a definition, which should happen at most once per function. Do you really need a SetVector here, or could you use a SmallVector instead?

test/CodeGen/alwaysinline-unused.c

1 ↗

(On Diff #33766)

Rename this to always_inline-unused.c for consistency with the existing always_inline.c.

test/CodeGen/alwaysinline.c

1 ↗

(On Diff #33766)

Seems weird to have both a test/CodeGen/always_inline.c and a test/CodeGen/alwaysinline.c. Either merge these together or rename this to test/CodeGen/always_inline-wrappers.c or similar.

47–53 ↗

(On Diff #33766)

Only the first of these should have the :[01-9]+, otherwise you'll overwrite AI rather than checking it's the same. Also, use [0-9]+ instead of [01-9]+?

55–59 ↗

(On Diff #33766)

I don't think we need to split f3 and f5 here.

55–67 ↗

(On Diff #33766)

Can you interleave the define and musttail lines so it's more obvious how they correspond?

61–67 ↗

(On Diff #33766)

Likewise.

[eugenis]

lib/CodeGen/CodeGenModule.cpp

491–492

I think having all alwaysinline functions called smth.alwaysinline is good for consistency.

491–492

Why skip local linkage functions? They may still need a stub for non-direct calls and we want to preserve the invariant that alwaysinline IR functions are always inlined.

494

D12576

test/CodeGen/alwaysinline.c

1 ↗

(On Diff #33766)

renamed

55–59 ↗

(On Diff #33766)

We need to split those to avoid a non-direct call to an alwaysinline function, which would not be possible to inline, and then an alwaysinline function would reach the backend and that's not good.

[yaron.keren]

In CGCXX.cpp, may be fixable after this commit:

// FIXME: An extern template instantiation will create functions with
// linkage "AvailableExternally". In libc++, some classes also define
// members with attribute "AlwaysInline" and expect no reference to
// be generated. It is desirable to reenable this optimisation after
// corresponding LLVM changes.

[eugenis]

In D12087#239243, @yaron.keren wrote:

In CGCXX.cpp, may be fixable after this commit:

// FIXME: An extern template instantiation will create functions with
// linkage "AvailableExternally". In libc++, some classes also define
// members with attribute "AlwaysInline" and expect no reference to
// be generated. It is desirable to reenable this optimisation after
// corresponding LLVM changes.

I've removed the checks for alwaysinline and available_externally. Tests + bootstrap do not show any new failures.

[rsmith]

lib/CodeGen/CodeGenModule.h

505

Did you try making this a vector? It'd be nice to avoid the set overhead here if we can.

[eugenis]

lib/CodeGen/CodeGenModule.h

505

Right. Sorry I missed the comment.
I've switched this to a vector.

[eugenis]

I'm going to commit this tomorrow unless someone speaks up.

[eugenis]

rebase, fix a merge conflict

[eugenis]

r247465, thanks for the review!

[eugenis]

Fixed the debug info problem.

[eugenis]

lib/CodeGen/CodeGenModule.cpp

539

As the comment says.
W/o this, the debug info for .alwaysinline instructions is attached to the .alwaysinline function, which gets deleted before codegen, and we end up with DI as if it is just a declaration (no low/high pc, etc).

[eugenis]

second attempt in r247494

lib/CodeGen/CGCXX.cpp

bool CodeGenModule::TryEmitBaseDestructorAsAlias(const CXXDestructorDecl *D) {
 
   // Give up if the calling conventions don't match. We could update the call,
   // but it is probably not worth it.
   const CXXDestructorDecl *BaseD = UniqueBase->getDestructor();
   if (BaseD->getType()->getAs<FunctionType>()->getCallConv() !=
       D->getType()->getAs<FunctionType>()->getCallConv())
     return true;
 
+  if (BaseD->hasAttr<AlwaysInlineAttr>())
+    return true;
+
   return TryEmitDefinitionAsAlias(GlobalDecl(D, Dtor_Base),
                                   GlobalDecl(BaseD, Dtor_Base),
                                   false);
 }
 
 /// Try to emit a definition as a global alias for another definition.
 /// If \p InEveryTU is true, we know that an equivalent alias can be produced
 /// in every translation unit.

lib/CodeGen/CGClass.cpp

else if (Body)
       EmitStmt(Body);
     else {
       assert(Dtor->isImplicit() && "bodyless dtor not implicit");
       // nothing to do besides what's in the epilogue
     }
     // -fapple-kext must inline any call to this dtor into
     // the caller's body.
     if (getLangOpts().AppleKext)
-      CurFn->addFnAttr(llvm::Attribute::AlwaysInline);
+      CGM.AddAlwaysInlineFunction(CurFn);
 
     // Insert memory-poisoning instrumentation, before final clean ups,
     // to ensure this class's members are protected from invalid access.
     if (CGM.getCodeGenOpts().SanitizeMemoryUseAfterDtor
         && SanOpts.has(SanitizerKind::Memory))
       EmitDtorSanitizerCallback(*this, Dtor);
 
     break;

lib/CodeGen/CGOpenMPRuntime.cpp

auto &TaskPrivatesMapFnInfo =
                                                     /*isVariadic=*/false);
   auto *TaskPrivatesMapTy =
       CGM.getTypes().GetFunctionType(TaskPrivatesMapFnInfo);
   auto *TaskPrivatesMap = llvm::Function::Create(
       TaskPrivatesMapTy, llvm::GlobalValue::InternalLinkage,
       ".omp_task_privates_map.", &CGM.getModule());
   CGM.SetLLVMFunctionAttributes(/*D=*/nullptr, TaskPrivatesMapFnInfo,
                                 TaskPrivatesMap);
-  TaskPrivatesMap->addFnAttr(llvm::Attribute::AlwaysInline);
+  CGM.AddAlwaysInlineFunction(TaskPrivatesMap);
   CodeGenFunction CGF(CGM);
   CGF.disableDebugInfo();
   CGF.StartFunction(GlobalDecl(), C.VoidTy, TaskPrivatesMap,
                     TaskPrivatesMapFnInfo, Args);
 
   // *privi = &.privates.privi;
   auto *TaskPrivatesArgAddr = CGF.Builder.CreateAlignedLoad(
       CGF.GetAddrOfLocalVar(&TaskPrivatesArg), CGM.PointerAlignInBytes);
if (CancelDest.isValid()) {
       // __kmpc_cancel_barrier();
       emitBarrierCall(CGF, Loc, OMPD_unknown, /*CheckForCancel=*/false);
       // exit from construct;
       CGF.EmitBranchThroughCleanup(CancelDest);
       CGF.EmitBlock(ContBB, /*IsFinished=*/true);
     }
   }
 }
-

lib/CodeGen/CodeGenModule.h

private:
 
   std::unique_ptr<SanitizerMetadata> SanitizerMD;
 
   /// @}
 
   llvm::DenseMap<const Decl *, bool> DeferredEmptyCoverageMappingDecls;
 
   std::unique_ptr<CoverageMappingModuleGen> CoverageMapping;
+
+  llvm::SetVector<llvm::Function*> AlwaysInlineFunctions;
+
 public:
   CodeGenModule(ASTContext &C, const HeaderSearchOptions &headersearchopts,
                 const PreprocessorOptions &ppopts,
                 const CodeGenOptions &CodeGenOpts, llvm::Module &M,
                 DiagnosticsEngine &Diags,
                 CoverageSourceInfo *CoverageInfo = nullptr);
 
   ~CodeGenModule();
public:
   /// Create a bitset entry for the given vtable.
   llvm::MDTuple *CreateVTableBitSetEntry(llvm::GlobalVariable *VTable,
                                          CharUnits Offset,
                                          const CXXRecordDecl *RD);
 
   /// \breif Get the declaration of std::terminate for the platform.
   llvm::Constant *getTerminateFn();
 
+  void AddAlwaysInlineFunction(llvm::Function *Fn);
+
 private:
   llvm::Constant *
   GetOrCreateLLVMFunction(StringRef MangledName, llvm::Type *Ty, GlobalDecl D,
                           bool ForVTable, bool DontDefer = false,
                           bool IsThunk = false,
                           llvm::AttributeSet ExtraAttrs = llvm::AttributeSet(),
                           bool IsForDefinition = false);
 
private:
   void EmitDeclMetadata();
 
   /// \brief Emit the Clang version as llvm.ident metadata.
   void EmitVersionIdentMetadata();
 
   /// Emits target specific Metadata for global declarations.
   void EmitTargetMetadata();
 
+  /// Replaces alwaysinline functions with a pair of internal xxx.inlinefunction
+  /// for direct calls, and a stub for indirect calls, and rewrites all uses of
+  /// those.
+  void RewriteAlwaysInlineFunctions();
+  void RewriteAlwaysInlineFunction(llvm::Function *Fn);
+
   /// Emit the llvm.gcov metadata used to tell LLVM where to emit the .gcno and
   /// .gcda files in a way that persists in .bc files.
   void EmitCoverageFile();
 
   /// Emits the initializer for a uuidof string.
   llvm::Constant *EmitUuidofInitializer(StringRef uuidstr);
 
   /// Determine whether the definition must be emitted; if this returns \c

lib/CodeGen/CodeGenModule.cpp

if (getCodeGenOpts().EmitGcovArcs || getCodeGenOpts().EmitGcovNotes)
     EmitCoverageFile();
 
   if (DebugInfo)
     DebugInfo->finalize();
 
   EmitVersionIdentMetadata();
 
   EmitTargetMetadata();
+
+  RewriteAlwaysInlineFunctions();
+}
+
+void CodeGenModule::AddAlwaysInlineFunction(llvm::Function *Fn) {
+  AlwaysInlineFunctions.insert(Fn);
+}
+
+/// Find all uses of GV that are not direct calls or invokes.
+static void FindNonDirectCallUses(llvm::GlobalValue *GV,
+                                  llvm::SmallVectorImpl<llvm::Use *> *Uses) {
+  llvm::GlobalValue::use_iterator UI = GV->use_begin(), E = GV->use_end();
+  for (; UI != E;) {
+    llvm::Use &U = *UI;
+    ++UI;
+
+    llvm::CallSite CS(U.getUser());
+    bool isDirectCall = (CS.isCall() || CS.isInvoke()) && CS.isCallee(&U);
+    if (!isDirectCall)
+      Uses->push_back(&U);
+  }
+}
+
+/// Replace a list of uses.
+static void ReplaceUsesWith(const llvm::SmallVectorImpl<llvm::Use *> &Uses,
+                            llvm::GlobalValue *V,
+                            llvm::GlobalValue *Replacement) {
+  for (llvm::Use *U : Uses) {
+    auto *C = dyn_cast<llvm::Constant>(U->getUser());
+    if (C && !isa<llvm::GlobalValue>(C))
+      C->handleOperandChange(V, Replacement, U);
+    else
+      U->set(Replacement);
+  }
+}
+
+void CodeGenModule::RewriteAlwaysInlineFunction(llvm::Function *Fn) {
+  std::string Name = Fn->getName();
+  std::string InlineName = Name + ".alwaysinline";
+  Fn->setName(InlineName);
+
+  llvm::SmallVector<llvm::Use *, 8> NonDirectCallUses;
+  Fn->removeDeadConstantUsers();
+  FindNonDirectCallUses(Fn, &NonDirectCallUses);
+  // Do not create the wrapper if there are no non-direct call uses, and we are
+  // not required to emit an external definition.
+  if (NonDirectCallUses.empty() && Fn->isDiscardableIfUnused())
+    return;
+
+  llvm::FunctionType *FT = Fn->getFunctionType();
+  llvm::LLVMContext &Ctx = getModule().getContext();
+  llvm::Function *StubFn =
+      llvm::Function::Create(FT, Fn->getLinkage(), Name, &getModule());
+  assert(StubFn->getName() == Name && "name was uniqued!");
+
+  // Insert the stub immediately after the original function. Helps with the
+  // fragile tests, among other things.
+  StubFn->removeFromParent();
+  TheModule.getFunctionList().insertAfter(Fn, StubFn);
+
+  StubFn->copyAttributesFrom(Fn);
+  StubFn->setPersonalityFn(nullptr);
+
+  // AvailableExternally functions are replaced with a declaration.
+  // Everyone else gets a wrapper that musttail-calls the original function.
+  if (Fn->hasAvailableExternallyLinkage()) {
+    StubFn->setLinkage(llvm::GlobalValue::ExternalLinkage);
+  } else {
+    llvm::BasicBlock *BB = llvm::BasicBlock::Create(Ctx, "entry", StubFn);
+    std::vector<llvm::Value *> Args;
+    for (llvm::Function::arg_iterator ai = StubFn->arg_begin();
+         ai != StubFn->arg_end(); ++ai)
+      Args.push_back(&*ai);
+    llvm::CallInst *CI = llvm::CallInst::Create(Fn, Args, "", BB);
+    CI->setCallingConv(Fn->getCallingConv());
+    CI->setTailCallKind(llvm::CallInst::TCK_MustTail);
+    CI->setAttributes(Fn->getAttributes());
+    if (FT->getReturnType()->isVoidTy())
+      llvm::ReturnInst::Create(Ctx, BB);
+    else
+      llvm::ReturnInst::Create(Ctx, CI, BB);
+  }
+
+  if (Fn->hasComdat())
+    StubFn->setComdat(Fn->getComdat());
+
+  ReplaceUsesWith(NonDirectCallUses, Fn, StubFn);
+}
+
+void CodeGenModule::RewriteAlwaysInlineFunctions() {
+  for (llvm::Function *Fn : AlwaysInlineFunctions) {
+    RewriteAlwaysInlineFunction(Fn);
+    Fn->setLinkage(llvm::GlobalValue::InternalLinkage);
+    Fn->addFnAttr(llvm::Attribute::AlwaysInline);
+    Fn->setDLLStorageClass(llvm::GlobalVariable::DefaultStorageClass);
+    Fn->setVisibility(llvm::GlobalValue::DefaultVisibility);
+  }
 }
 
 void CodeGenModule::UpdateCompletedType(const TagDecl *TD) {
   // Make sure that this type is translated.
   Types.UpdateCompletedType(TD);
 }
 
 llvm::MDNode *CodeGenModule::getTBAAInfo(QualType QTy) {
void CodeGenModule::SetLLVMFunctionAttributesForDefinition(const Decl *D,
   } else if (D->hasAttr<NoDuplicateAttr>()) {
     B.addAttribute(llvm::Attribute::NoDuplicate);
   } else if (D->hasAttr<NoInlineAttr>()) {
     B.addAttribute(llvm::Attribute::NoInline);
   } else if (D->hasAttr<AlwaysInlineAttr>() &&
              !F->getAttributes().hasAttribute(llvm::AttributeSet::FunctionIndex,
                                               llvm::Attribute::NoInline)) {
     // (noinline wins over always_inline, and we can't specify both in IR)
-    B.addAttribute(llvm::Attribute::AlwaysInline);
+    AddAlwaysInlineFunction(F);
   }
 
   if (D->hasAttr<ColdAttr>()) {
     if (!D->hasAttr<OptimizeNoneAttr>())
       B.addAttribute(llvm::Attribute::OptimizeForSize);
     B.addAttribute(llvm::Attribute::Cold);
   }
 

lib/CodeGen/ItaniumCXXABI.cpp

static StructorCodegen getCodegenToUse(CodeGenModule &CGM,
   if (!CGM.getCodeGenOpts().CXXCtorDtorAliases)
     return StructorCodegen::Emit;
 
   // The complete and base structors are not equivalent if there are any virtual
   // bases, so emit separate functions.
   if (MD->getParent()->getNumVBases())
     return StructorCodegen::Emit;
 
+  if (MD->hasAttr<AlwaysInlineAttr>())
+    return StructorCodegen::Emit;
+
   GlobalDecl AliasDecl;
   if (const auto *DD = dyn_cast<CXXDestructorDecl>(MD)) {
     AliasDecl = GlobalDecl(DD, Dtor_Complete);
   } else {
     const auto *CD = cast<CXXConstructorDecl>(MD);
     AliasDecl = GlobalDecl(CD, Ctor_Complete);
   }
   llvm::GlobalValue::LinkageTypes Linkage = CGM.getFunctionLinkage(AliasDecl);

test/CodeGen/always-inline.c

 // RUN: %clang_cc1 -emit-llvm %s -o - | FileCheck %s
 // RUN: %clang_cc1 -fno-inline -emit-llvm %s -o - | FileCheck %s
 
+// CHECK-LABEL: define void @i_want_bar()
 // CHECK-NOT: foo
+// CHECK: ret void
 
 void bar() {
 }
 
 inline void __attribute__((__always_inline__)) foo() {
   bar();
 }
 
 void i_want_bar() {
   foo();
 }

test/CodeGen/always_inline-unused.c

+// Test alwaysinline definitions w/o any non-direct-call uses.
+// None of the declarations are emitted. Stub are only emitted when the original
+// function can not be discarded.
+
+// RUN: %clang_cc1 -disable-llvm-optzns -emit-llvm %s -o - | FileCheck %s
+
+void __attribute__((__always_inline__)) f1() {}
+inline void __attribute__((__always_inline__)) f2() {}
+static inline void __attribute__((__always_inline__)) f3() {}
+inline void __attribute__((gnu_inline, __always_inline__)) f4() {}
+static inline void __attribute__((gnu_inline, __always_inline__)) f5() {}
+inline void __attribute__((visibility("hidden"), __always_inline__)) f6() {}
+inline void __attribute__((visibility("hidden"), gnu_inline, __always_inline__)) f7() {}
+
+void g() {
+  f1();
+  f2();
+  f3();
+  f4();
+  f5();
+  f6();
+  f7();
+}
+
+// CHECK: define void @f1()
+// CHECK-NOT: void @f2()
+// CHECK-NOT: void @f3()
+// CHECK: define void @f4()
+// CHECK-NOT: void @f5()
+// CHECK-NOT: void @f6()
+// CHECK: define hidden void @f7()

test/CodeGen/always_inline-wrappers.c

+// Test different kinds of alwaysinline definitions.
+
+// RUN: %clang_cc1 -disable-llvm-optzns -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-INLINE
+// RUN: %clang_cc1 -disable-llvm-optzns -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK-USE
+// RUN: %clang_cc1 -disable-llvm-optzns -fno-inline -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK
+// RUN: %clang_cc1 -disable-llvm-optzns -fno-inline -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK-USE
+
+void __attribute__((__always_inline__)) f1() {}
+inline void __attribute__((__always_inline__)) f2() {}
+static inline void __attribute__((__always_inline__)) f3() {}
+inline void __attribute__((gnu_inline, __always_inline__)) f4() {}
+static inline void __attribute__((gnu_inline, __always_inline__)) f5() {}
+inline void __attribute__((visibility("hidden"), __always_inline__)) f6() {}
+inline void __attribute__((visibility("hidden"), gnu_inline, __always_inline__)) f7() {}
+
+void g() {
+  f1();
+  f2();
+  f3();
+  f4();
+  f5();
+  f6();
+  f7();
+}
+
+void (*p)(void);
+void h() {
+  p = f1;
+  p = f2;
+  p = f3;
+  p = f4;
+  p = f5;
+  p = f6;
+  p = f7;
+}
+
+void (*const cp1)(void) = f1;
+void (*p1)(void) = f1;
+void (*p2)(int) = (void (*)(int))f1;
+
+void __attribute__((__always_inline__)) f8(void(*f)(void)) {}
+
+void call() {
+  f8(f1);
+}
+
+// CHECK-DAG: define internal void @f1.alwaysinline() #[[AI:[0-9]+]]
+// CHECK-DAG: define internal void @f2.alwaysinline() #[[AI_IH:[0-9]+]]
+// CHECK-DAG: define internal void @f3.alwaysinline() #[[AI_IH]]
+// CHECK-DAG: define internal void @f4.alwaysinline() #[[AI_IH]]
+// CHECK-DAG: define internal void @f5.alwaysinline() #[[AI_IH]]
+// CHECK-DAG: define internal void @f6.alwaysinline() #[[AI_IH]]
+// CHECK-DAG: define internal void @f7.alwaysinline() #[[AI_IH]]
+
+
+// CHECK-DAG: define void @f1() #[[NOAI:[01-9]+]]
+// CHECK-DAG: musttail call void @f1.alwaysinline()
+
+// CHECK-DAG: declare void @f2() #[[NOAI:[01-9]+]]
+
+// CHECK-DAG: define internal void @f3() #[[NOAI:[01-9]+]]
+// CHECK-DAG: musttail call void @f3.alwaysinline()
+
+// CHECK-DAG: define void @f4() #[[NOAI:[01-9]+]]
+// CHECK-DAG: musttail call void @f4.alwaysinline()
+
+// CHECK-DAG: define internal void @f5() #[[NOAI:[01-9]+]]
+// CHECK-DAG: musttail call void @f5.alwaysinline()
+
+// CHECK-DAG: declare hidden void @f6() #[[NOAI:[01-9]+]]
+
+// CHECK-DAG: define hidden void @f7() #[[NOAI:[01-9]+]]
+// CHECK-DAG: musttail call void @f7.alwaysinline()
+
+
+// CHECK-DAG: @cp1 = constant void ()* @f1, align
+// CHECK-DAG: @p1 = global void ()* @f1, align
+// CHECK-DAG: @p2 = global void (i32)* bitcast (void ()* @f1 to void (i32)*), align
+
+// CHECK: attributes #[[AI]] = {{.*alwaysinline.*}}
+// CHECK-INLINE: attributes #[[AI_IH]] = {{.*alwaysinline.*inlinehint.*}}
+// CHECK-NOT: attributes #[[NOAI]] = {{.*alwaysinline.*}}
+
+// CHECK-USE-LABEL: define void @g()
+// CHECK-USE-NEXT:   entry:
+// CHECK-USE-NEXT:     call void @f1.alwaysinline()
+// CHECK-USE-NEXT:     call void @f2.alwaysinline()
+// CHECK-USE-NEXT:     call void @f3.alwaysinline()
+// CHECK-USE-NEXT:     call void @f4.alwaysinline()
+// CHECK-USE-NEXT:     call void @f5.alwaysinline()
+// CHECK-USE-NEXT:     call void @f6.alwaysinline()
+// CHECK-USE-NEXT:     call void @f7.alwaysinline()
+// CHECK-USE-NEXT:     ret void
+
+// CHECK-USE-LABEL: define void @h()
+// CHECK-USE-NEXT:   entry:
+// CHECK-USE-NEXT:     store void ()* @f1,
+// CHECK-USE-NEXT:     store void ()* @f2,
+// CHECK-USE-NEXT:     store void ()* @f3,
+// CHECK-USE-NEXT:     store void ()* @f4,
+// CHECK-USE-NEXT:     store void ()* @f5,
+// CHECK-USE-NEXT:     store void ()* @f6,
+// CHECK-USE-NEXT:     store void ()* @f7,
+// CHECK-USE-NEXT:     ret void
+
+// CHECK-USE-LABEL:  define void @call()
+// CHECK-USE:           call void @f8.alwaysinline(void ()* @f1)
+// CHECK-USE:           ret void

test/CodeGen/always_inline.c

-// RUN: %clang -emit-llvm -S -o %t %s
-// RUN: not grep '@f0' %t
-// RUN: not grep 'call ' %t
-// RUN: %clang -mllvm -disable-llvm-optzns -emit-llvm -S -o %t %s
-// RUN: grep '@f0' %t | count 2
+// RUN: %clang -emit-llvm -S -o - %s | FileCheck %s
+// RUN: %clang -mllvm -disable-llvm-optzns -emit-llvm -S -o - %s | FileCheck %s --check-prefix=CHECK-NO-OPTZNS
 
 //static int f0() { 
 static int __attribute__((always_inline)) f0() { 
   return 1;
 }
 
 int f1() {
   return f0();
 }
 
 // PR4372
 inline int f2() __attribute__((always_inline));
 int f2() { return 7; }
 int f3(void) { return f2(); }
 
+// CHECK-LABEL: define i32 @f1()
+// CHECK: ret i32 1
+// CHECK-LABEL: define i32 @f2()
+// CHECK: ret i32 7
+// CHECK-LABEL: define i32 @f3()
+// CHECK: ret i32 7
+
+// CHECK-NO-OPTZNS: define i32 @f3()
+// CHECK-NO-OPTZNS-NEXT: entry:
+// CHECK-NO-OPTZNS-NEXT:   call i32 @f2.alwaysinline()
+// CHECK-NO-OPTZNS-NEXT:   ret i32

test/CodeGen/function-attributes.c

 signed short f4(int x) { return x; }
 
 unsigned short f5(int x) { return x; }
 
 void f6(signed short x) { }
 
 void f7(unsigned short x) { }
 
-// CHECK-LABEL: define void @f8()
-// CHECK: [[AI:#[0-9]+]]
+// CHECK: define void @f8()
+// CHECK: [[NUW:#[0-9]+]]
 // CHECK: {
 void __attribute__((always_inline)) f8(void) { }
 
 // CHECK: call void @f9_t()
 // CHECK: [[NR:#[0-9]+]]
 // CHECK: }
 void __attribute__((noreturn)) f9_t(void);
 void f9(void) { f9_t(); }

 // CHECK: [[RT_CALL]]
 // CHECK: ret void
 int _setjmp(jmp_buf);
 void f20(void) {
   _setjmp(0);
 }
 
 // CHECK: attributes [[NUW]] = { nounwind optsize readnone{{.*}} }
-// CHECK: attributes [[AI]] = { alwaysinline nounwind optsize readnone{{.*}} }
 // CHECK: attributes [[ALIGN]] = { nounwind optsize readnone alignstack=16{{.*}} }
 // CHECK: attributes [[RT]] = { nounwind optsize returns_twice{{.*}} }
 // CHECK: attributes [[NR]] = { noreturn nounwind optsize }
 // CHECK: attributes [[NUW_RN]] = { nounwind optsize readnone }
 // CHECK: attributes [[RT_CALL]] = { nounwind optsize returns_twice }

test/CodeGen/pr9614.c

-// RUN: %clang_cc1 -triple x86_64-pc-linux -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-pc-linux -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK0
+// RUN: %clang_cc1 -triple x86_64-pc-linux -O1 -disable-llvm-optzns -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK1
 
 extern void foo_alias (void) __asm ("foo");
 inline void foo (void) {
   return foo_alias ();
 }
 extern int abs_alias (int) __asm ("abs");
 inline __attribute__ ((__always_inline__)) int abs (int x) {
   return abs_alias(x);

 }
 
 extern inline __attribute__((__always_inline__, __gnu_inline__)) void *memchr(void *__s, int __c, __SIZE_TYPE__ __n) {
   return __builtin_memchr(__s, __c, __n);
 }
 
 void f(void) {
   foo();
-  abs(0);
+  int x = abs(0);
   strrchr_foo("", '.');
   prefetch();
   memchr("", '.', 0);
 }
 
 // CHECK-LABEL: define void @f()
 // CHECK: call void @foo()
-// CHECK: call i32 @abs(i32 0)
+// CHECK: call i32 @abs(
 // CHECK: call i8* @strrchr(
-// CHECK: call void @llvm.prefetch(
+// CHECK0: call void @llvm.prefetch(
+// CHECK1: call void @prefetch.alwaysinline(
 // CHECK: call i8* @memchr(
 // CHECK: ret void
 
 // CHECK: declare void @foo()
 // CHECK: declare i32 @abs(i32
 // CHECK: declare i8* @strrchr(i8*, i32)
 // CHECK: declare i8* @memchr(
 // CHECK: declare void @llvm.prefetch(

test/CodeGenCXX/alwaysinline.cpp

+// Test different kinds of alwaysinline *structor definitions.
+
+// RUN: %clang_cc1 -disable-llvm-optzns -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK
+// RUN: %clang_cc1 -disable-llvm-optzns -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK-CALL
+
+// RUN: %clang_cc1 -mconstructor-aliases -disable-llvm-optzns -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK
+// RUN: %clang_cc1 -mconstructor-aliases -disable-llvm-optzns -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECK-CALL
+
+struct A1 {
+  __attribute__((__always_inline__)) A1() {}
+  __attribute__((__always_inline__)) ~A1() {}
+};
+
+void g1() {
+  A1 a1;
+}
+
+struct A2 {
+  inline __attribute__((__always_inline__)) A2() {}
+  inline __attribute__((__always_inline__)) ~A2() {}
+};
+
+void g2() {
+  A2 a2;
+}
+
+struct A3 {
+  inline __attribute__((gnu_inline, __always_inline__)) A3() {}
+  inline __attribute__((gnu_inline, __always_inline__)) ~A3() {}
+};
+
+void g3() {
+  A3 a3;
+}
+
+// CHECK-DAG: define internal void @_ZN2A1C1Ev.alwaysinline(%struct.A1* %this) unnamed_addr #[[AI:[01-9]+]]
+// CHECK-DAG: define internal void @_ZN2A1C2Ev.alwaysinline(%struct.A1* %this) unnamed_addr #[[AI]]
+// CHECK-DAG: define internal void @_ZN2A1D1Ev.alwaysinline(%struct.A1* %this) unnamed_addr #[[AI]]
+// CHECK-DAG: define internal void @_ZN2A1D2Ev.alwaysinline(%struct.A1* %this) unnamed_addr #[[AI]]
+
+// CHECK-DAG: define internal void @_ZN2A2C1Ev.alwaysinline(%struct.A2* %this) unnamed_addr #[[AIIH:[01-9]+]]
+// CHECK-DAG: define internal void @_ZN2A2C2Ev.alwaysinline(%struct.A2* %this) unnamed_addr #[[AIIH]]
+// CHECK-DAG: define internal void @_ZN2A2D1Ev.alwaysinline(%struct.A2* %this) unnamed_addr #[[AIIH]]
+// CHECK-DAG: define internal void @_ZN2A2D2Ev.alwaysinline(%struct.A2* %this) unnamed_addr #[[AIIH]]
+
+// CHECK-DAG: define internal void @_ZN2A3C1Ev.alwaysinline(%struct.A3* %this) unnamed_addr #[[AIIH]]
+// CHECK-DAG: define internal void @_ZN2A3C2Ev.alwaysinline(%struct.A3* %this) unnamed_addr #[[AIIH]]
+// CHECK-DAG: define internal void @_ZN2A3D1Ev.alwaysinline(%struct.A3* %this) unnamed_addr #[[AIIH]]
+// CHECK-DAG: define internal void @_ZN2A3D2Ev.alwaysinline(%struct.A3* %this) unnamed_addr #[[AIIH]]
+
+// CHECK-DAG: attributes #[[AI]] = {{.*alwaysinline.*}}
+// CHECK-DAG: attributes #[[AIIH]] = {{.*alwaysinline.*inlinehint.*}}
+// CHECK-NOT: attributes #[[NOAI]] = {{.*alwaysinline.*}}
+
+// CHECK-CALL-LABEL: define void @_Z2g1v()
+// CHECK-CALL:       call void @_ZN2A1C1Ev.alwaysinline
+// CHECK-CALL:       call void @_ZN2A1D1Ev.alwaysinline
+// CHECK-CALL:       ret void
+
+// CHECK-CALL-LABEL: define void @_Z2g2v()
+// CHECK-CALL:       call void @_ZN2A2C1Ev.alwaysinline
+// CHECK-CALL:       call void @_ZN2A2D1Ev.alwaysinline
+// CHECK-CALL:       ret void
+
+// CHECK-CALL-LABEL: define void @_Z2g3v()
+// CHECK-CALL:       call void @_ZN2A3C1Ev.alwaysinline
+// CHECK-CALL:       call void @_ZN2A3D1Ev.alwaysinline
+// CHECK-CALL:       ret void

test/Frontend/optimization-remark-line-directive.c

 // This file tests -Rpass diagnostics together with #line
 // directives. We cannot map #line directives back to
 // a SourceLocation.
 
 // RUN: %clang_cc1 %s -Rpass=inline -gline-tables-only -dwarf-column-info -emit-llvm-only -verify
 
 int foo(int x, int y) __attribute__((always_inline));
+// expected-remark@+1 {{foo.alwaysinline inlined into foo}}
 int foo(int x, int y) { return x + y; }
 
-// expected-remark@+2 {{foo inlined into bar}} expected-note@+2 {{could not determine the original source location for /bad/path/to/original.c:1230:25}}
+// expected-remark@+2 {{foo.alwaysinline inlined into bar}} expected-note@+2 {{could not determine the original source location for /bad/path/to/original.c:1230:25}}
 #line 1230 "/bad/path/to/original.c"
 int bar(int j) { return foo(j, j - 2); }

test/Frontend/optimization-remark.c

 // CHECK: , !dbg !
 // CHECK-NOT: DW_TAG_base_type
 
 // But llvm.dbg.cu should be missing (to prevent writing debug info to
 // the final output).
 // CHECK-NOT: !llvm.dbg.cu = !{
 
 int foo(int x, int y) __attribute__((always_inline));
+// expected-remark@+2 {{foo.alwaysinline should always be inlined}}
+// expected-remark@+1 {{foo.alwaysinline inlined into foo}}
 int foo(int x, int y) { return x + y; }
 
 float foz(int x, int y) __attribute__((noinline));
 float foz(int x, int y) { return x * y; }
 
 // The negative diagnostics are emitted twice because the inliner runs
 // twice.
 //
 int bar(int j) {
 // expected-remark@+6 {{foz should never be inlined (cost=never)}}
 // expected-remark@+5 {{foz will not be inlined into bar}}
 // expected-remark@+4 {{foz should never be inlined}}
 // expected-remark@+3 {{foz will not be inlined into bar}}
-// expected-remark@+2 {{foo should always be inlined}}
-// expected-remark@+1 {{foo inlined into bar}}
+// expected-remark@+2 {{foo.alwaysinline should always be inlined}}
+// expected-remark@+1 {{foo.alwaysinline inlined into bar}}
   return foo(j, j - 2) * foz(j - 2, j);
 }

test/Modules/cxx-irgen.cpp

 namespace EmitInlineMethods {
   // CHECK-DAG: define linkonce_odr [[CC]]void @_ZN17EmitInlineMethods1D1fEPNS_1BE(
   // CHECK-DAG: define linkonce_odr [[CC]]void @_ZN17EmitInlineMethods1B1gEv(
   struct D {
     __attribute__((used)) void f(B *p) { p->g(); }
   };
 }
 
-// CHECK-DAG: define available_externally hidden {{.*}}{{signext i32|i32}} @_ZN1SIiE1gEv({{.*}} #[[ALWAYS_INLINE:.*]] align
+// CHECK-DAG: define internal i32 @_ZN1SIiE1gEv.alwaysinline() #[[ALWAYS_INLINE:.*]] align
 int a = S<int>::g();
 
 int b = h();
 
 // CHECK-DAG: define linkonce_odr {{.*}}{{signext i32|i32}} @_Z3minIiET_S0_S0_(i32
 int c = min(1, 2);
-// CHECK: define available_externally {{.*}}{{signext i32|i32}} @_ZN1SIiE1fEv({{.*}} #[[ALWAYS_INLINE]] align
+// CHECK-DAG: define internal {{.*}}{{signext i32|i32}} @_ZN1SIiE1fEv.alwaysinline() #[[ALWAYS_INLINE]] align
 
 namespace ImplicitSpecialMembers {
   // CHECK-LABEL: define {{.*}} @_ZN22ImplicitSpecialMembers1BC2ERKS0_(
   // CHECK: call {{.*}} @_ZN22ImplicitSpecialMembers1AC1ERKS0_(
   // CHECK-LABEL: define {{.*}} @_ZN22ImplicitSpecialMembers1BC2EOS0_(
   // CHECK: call {{.*}} @_ZN22ImplicitSpecialMembers1AC1ERKS0_(
   // CHECK-LABEL: define {{.*}} @_ZN22ImplicitSpecialMembers1CC2ERKS0_(
   // CHECK: call {{.*}} @_ZN22ImplicitSpecialMembers1AC1ERKS0_(