Add new 'flatten' LLVM attribute to fix clang's 'flatten' function attribute

[LevitatingLion]

Summary

This adds a new 'flatten' attribute, which works like 'always_inline' but applies recursively to inlined call sites. The addition was briefly discussed on the mailing list: http://lists.llvm.org/pipermail/llvm-dev/2019-November/136514.html

This patch also contains changes to clang, so that it uses the new LLVM attribute on functions marked with the clang attribute 'flatten'. Previously, clang marked all calls in such functions with 'always_inline'; in effect, only the first level of calls was inlined.

Currently this patch fails the '/llvm/test/Bitcode/highLevelStructure.3.2.ll' test. llvm-dis seems to be unable to correctly decode attributes stored in the bitcode when the new attribute is added, although other attributes don't seem to have required any handling of this problem, see https://reviews.llvm.org/D62766 or https://reviews.llvm.org/D49165. I speculated that's because this is the 65th attribute, so a bitmask indicating all attributes doesn't fit in 64 bit anymore.

[arsenm]

llvm/docs/LangRef.rst

1431

It's not obvious to me what the flatten name means. flatteninline? recursive_alwaysinline? Something else?

[lebedev.ri]

Thank you for working on this!

[LevitatingLion]

llvm/docs/LangRef.rst

1431

I agree. What about always_inline_recurse or always_inline_recursively?

[LevitatingLion]

Ping

[jdoerfert]

We need more tests here.

For one, the flatten attribute is not necessary to pass the test.
Second, we need to check the corner cases, e.g. reduction with different cycle lengths.

llvm/docs/LangRef.rst

1431

I'd prefer always_inline_recursively or recursive_alwaysinline so far. Though something shorter would be fine too.
always_inline_rec maybe?

[jdoerfert]

[reverse ping] Is this still active?

[LevitatingLion]

Thanks for the ping. I hadn't looked at this since, but I'll update the patch this weekend.

[LevitatingLion]

I rebased my changes onto 49d00824bbb, renamed the attribute to 'alwaysinline_recursively', and added some more tests. The testcase 'highLevelStructure.3.2.ll' does not fail anymore, all regression tests pass.

Are there any more places where changes are required? I looked at the changes when other attributes were introduced and grep'd for 'Attribute::AlwaysInline' to find places which need handling of the new attribute.

[jdoerfert]

I'm fine with this. I would hope a C/C++/Clang person will also take a look though.

llvm/docs/LangRef.rst

1398

Maybe mention the correspondence to the flatten C/C++ attribute here.

[lebedev.ri]

In D70366#1960300, @jdoerfert wrote:

I'm fine with this. I would hope a C/C++/Clang person will also take a look though.

This is missing clang codegen test[s].
Seems to look fine to me otherwise.

[erichkeane]

The semantics of this llvm attribute seem to better match 'flatten'. However, it is unfortunate that this doesn't change any clang tests. Can you add/alter a test in clang to validate the IR?

[LevitatingLion]

While adding tests to clang I realized the attribute is not working as intended when using an optimization level of zero, because clang adds the noinline attribute to all functions. In this case the optimizer cannot distinguish between functions originally marked noinline (where recursive always-inlining should stop) and those where clang added the attribute (where recursive always-inlining should continue).

Is this acceptable? I think we should fix this, and recursively inline at optimization level zero. GCC's documentation on the flatten attribute states that "every call inside this function is inlined, if possible", clang's that calls are "inlined unless it is impossible to do so".

Maybe we can add an additional string attribute when adding the noinline attribute to functions which are not marked noinline in the source code, something like "noinline-added-by-clang". I don't know if that's a legitimate use case for a string attribute, but it wouldn't be very invasive. What do you think?

[jdoerfert]

In D70366#1970299, @LevitatingLion wrote:

While adding tests to clang I realized the attribute is not working as intended when using an optimization level of zero, because clang adds the noinline attribute to all functions. In this case the optimizer cannot distinguish between functions originally marked noinline (where recursive always-inlining should stop) and those where clang added the attribute (where recursive always-inlining should continue).

Is this acceptable? I think we should fix this, and recursively inline at optimization level zero. GCC's documentation on the flatten attribute states that "every call inside this function is inlined, if possible", clang's that calls are "inlined unless it is impossible to do so".

Maybe we can add an additional string attribute when adding the noinline attribute to functions which are not marked noinline in the source code, something like "noinline-added-by-clang". I don't know if that's a legitimate use case for a string attribute, but it wouldn't be very invasive. What do you think?

TBH, I would issue a warning if we see flatten in O0 that says this will not work and be done with it.

[dexonsmith]

In D70366#1970375, @jdoerfert wrote:

TBH, I would issue a warning if we see flatten in O0 that says this will not work and be done with it.

I would argue against diagnostics that depend on optimization level, since that leads to an inconsistent developer experience. In practice developers tend to use Debug and Release configurations fairly interchangeably, with different optimization levels for each.

[dexonsmith]

In D70366#1970299, @LevitatingLion wrote:

Maybe we can add an additional string attribute when adding the noinline attribute to functions which are not marked noinline in the source code, something like "noinline-added-by-clang". I don't know if that's a legitimate use case for a string attribute, but it wouldn't be very invasive. What do you think?

Another option (not sure if it's better) would be to add a noopt LLVM attribute that Clang adds for -O0 instead of noinline. Two possibilities would be to update the inliner to pay attention to that as well (with special logic for flatten), or to change the always-inliner to add noinline to anything marked noopt.

[dexonsmith]

In D70366#1970526, @dexonsmith wrote:

In D70366#1970299, @LevitatingLion wrote:

Maybe we can add an additional string attribute when adding the noinline attribute to functions which are not marked noinline in the source code, something like "noinline-added-by-clang". I don't know if that's a legitimate use case for a string attribute, but it wouldn't be very invasive. What do you think?

Another option (not sure if it's better) would be to add a noopt LLVM attribute that Clang adds for -O0 instead of noinline. Two possibilities would be to update the inliner to pay attention to that as well (with special logic for flatten), or to change the always-inliner to add noinline to anything marked noopt.

Or, have clang add a new noflatten attribute when it sees __attribute__((noinline)).

[jdoerfert]

In D70366#1970526, @dexonsmith wrote:

In D70366#1970299, @LevitatingLion wrote:

Maybe we can add an additional string attribute when adding the noinline attribute to functions which are not marked noinline in the source code, something like "noinline-added-by-clang". I don't know if that's a legitimate use case for a string attribute, but it wouldn't be very invasive. What do you think?

Another option (not sure if it's better) would be to add a noopt LLVM attribute that Clang adds for -O0 instead of noinline. Two possibilities would be to update the inliner to pay attention to that as well (with special logic for flatten), or to change the always-inliner to add noinline to anything marked noopt.

noopt == optnone? Both optnone and noinline are set in O0, so we could just not place noinline (I think).

[dexonsmith]

In D70366#1970758, @jdoerfert wrote:

In D70366#1970526, @dexonsmith wrote:

In D70366#1970299, @LevitatingLion wrote:

Maybe we can add an additional string attribute when adding the noinline attribute to functions which are not marked noinline in the source code, something like "noinline-added-by-clang". I don't know if that's a legitimate use case for a string attribute, but it wouldn't be very invasive. What do you think?

Another option (not sure if it's better) would be to add a noopt LLVM attribute that Clang adds for -O0 instead of noinline. Two possibilities would be to update the inliner to pay attention to that as well (with special logic for flatten), or to change the always-inliner to add noinline to anything marked noopt.

noopt == optnone? Both optnone and noinline are set in O0, so we could just not place noinline (I think).

Sure, that could work. Or the noflatten idea is another possibility. It would be good to hear what others think.

[LevitatingLion]

In D70366#1971137, @dexonsmith wrote:

In D70366#1970758, @jdoerfert wrote:

In D70366#1970526, @dexonsmith wrote:

In D70366#1970299, @LevitatingLion wrote:

Maybe we can add an additional string attribute when adding the noinline attribute to functions which are not marked noinline in the source code, something like "noinline-added-by-clang". I don't know if that's a legitimate use case for a string attribute, but it wouldn't be very invasive. What do you think?

Another option (not sure if it's better) would be to add a noopt LLVM attribute that Clang adds for -O0 instead of noinline. Two possibilities would be to update the inliner to pay attention to that as well (with special logic for flatten), or to change the always-inliner to add noinline to anything marked noopt.

noopt == optnone? Both optnone and noinline are set in O0, so we could just not place noinline (I think).

Sure, that could work. Or the noflatten idea is another possibility. It would be good to hear what others think.

optnone currently requires noinline. Can we simply remove this requirement or would that need more changes?

If I understand the noflatten idea correctly, we would change the LLVM behaviour so that alwaysinline_recursively ignores noinline and stops inlining only when a callee has a dedicated "stop-marker" attribute (e.g. noflatten)? I think that would be counter-intuitive, noinline should prevent inlining.

[jdoerfert]

In D70366#1972880, @LevitatingLion wrote:

In D70366#1971137, @dexonsmith wrote:

In D70366#1970758, @jdoerfert wrote:

In D70366#1970526, @dexonsmith wrote:

In D70366#1970299, @LevitatingLion wrote:

Maybe we can add an additional string attribute when adding the noinline attribute to functions which are not marked noinline in the source code, something like "noinline-added-by-clang". I don't know if that's a legitimate use case for a string attribute, but it wouldn't be very invasive. What do you think?

Another option (not sure if it's better) would be to add a noopt LLVM attribute that Clang adds for -O0 instead of noinline. Two possibilities would be to update the inliner to pay attention to that as well (with special logic for flatten), or to change the always-inliner to add noinline to anything marked noopt.

noopt == optnone? Both optnone and noinline are set in O0, so we could just not place noinline (I think).

Sure, that could work. Or the noflatten idea is another possibility. It would be good to hear what others think.

optnone currently requires noinline. Can we simply remove this requirement or would that need more changes?

I don't see a reason why we couldn't.

If I understand the noflatten idea correctly, we would change the LLVM behaviour so that alwaysinline_recursively ignores noinline and stops inlining only when a callee has a dedicated "stop-marker" attribute (e.g. noflatten)? I think that would be counter-intuitive, noinline should prevent inlining.

I would prefer the above solution instead of another "stop token".

[jdoerfert]

Found the commit: dcbe35bad518

The way I see it we just have to teach the inliner about optnone so we can uncouple the two (optnone and noinline).
@probinson WDTY?

clang/lib/CodeGen/CGCall.cpp

if (FD->usesFPIntrin())
       // All calls within a strictfp function are marked strictfp
       Attrs =
         Attrs.addAttribute(getLLVMContext(), llvm::AttributeList::FunctionIndex,
                            llvm::Attribute::StrictFP);
 
   // Apply some call-site-specific attributes.
   // TODO: work this into building the attribute set.
 
-  // Apply always_inline to all calls within flatten functions.
+  // Apply alwaysinline_recursively to all calls within flatten functions.
   // FIXME: should this really take priority over __try, below?
   if (CurCodeDecl && CurCodeDecl->hasAttr<FlattenAttr>() &&
       !(TargetDecl && TargetDecl->hasAttr<NoInlineAttr>())) {
     Attrs =
         Attrs.addAttribute(getLLVMContext(), llvm::AttributeList::FunctionIndex,
-                           llvm::Attribute::AlwaysInline);
+                           llvm::Attribute::AlwaysInlineRecursively);
   }
 
   // Disable inlining inside SEH __try blocks.
   if (isSEHTryScope()) {
     Attrs =
         Attrs.addAttribute(getLLVMContext(), llvm::AttributeList::FunctionIndex,
                            llvm::Attribute::NoInline);
   }

llvm/docs/BitCodeFormat.rst

 * code 52: ``writeonly``
 * code 53: ``speculatable``
 * code 54: ``strictfp``
 * code 55: ``sanitize_hwaddress``
 * code 56: ``nocf_check``
 * code 57: ``optforfuzzing``
 * code 58: ``shadowcallstack``
 * code 64: ``sanitize_memtag``
+* code 65: ``alwaysinline_recursively``
 
 .. note::
   The ``allocsize`` attribute has a special encoding for its arguments. Its two
   arguments, which are 32-bit integers, are packed into one 64-bit integer value
   (i.e. ``(EltSizeParam << 32) | NumEltsParam``), with ``NumEltsParam`` taking on
   the sentinel value -1 if it is not specified.
 
 .. _TYPE_BLOCK:

llvm/docs/LangRef.rst

``allocsize(<EltSizeParam>[, <NumEltsParam>])``
     returned pointer. If two are provided, then it's assumed that
     ``CallSite.Args[EltSizeParam] * CallSite.Args[NumEltsParam]`` bytes are
     available. The referenced parameters must be integer types. No assumptions
     are made about the contents of the returned block of memory.
 ``alwaysinline``
     This attribute indicates that the inliner should attempt to inline
     this function into callers whenever possible, ignoring any active
     inlining size threshold for this caller.
+``alwaysinline_recursively``
+    This attribute is similar to ``alwaysinline``, but also applies recursively to
+    all inlined function calls.
 ``builtin``
     This indicates that the callee function at a call site should be
     recognized as a built-in function, even though the function's declaration
     uses the ``nobuiltin`` attribute. This is only valid at call sites for
     direct calls to functions that are declared with the ``nobuiltin``
     attribute.
 ``cold``
     This attribute indicates that this function is rarely called. When
``convergent``
     that we should treat the call as though we're calling a convergent
     function.  This is particularly useful on indirect calls; without this we
     may treat such calls as though the target is non-convergent.
 
     The optimizer may remove the ``convergent`` attribute on functions when it
     can prove that the function does not execute any convergent operations.
     Similarly, the optimizer may remove ``convergent`` on calls/invokes when it
     can prove that the call/invoke cannot call a convergent function.
 ``inaccessiblememonly``
     This attribute indicates that the function may only access memory that
     is not accessible by the module being compiled. This is a weaker form
     of ``readnone``. If the function reads or writes other memory, the
     behavior is undefined.
 ``inaccessiblemem_or_argmemonly``
     This attribute indicates that the function may only access memory that is
     either not accessible by the module being compiled, or is pointed to
     by its pointer arguments. This is a weaker form of  ``argmemonly``. If the

llvm/include/llvm/Bitcode/LLVMBitCodes.h

enum AttributeKindCodes {
   ATTR_KIND_OPT_FOR_FUZZING = 57,
   ATTR_KIND_SHADOWCALLSTACK = 58,
   ATTR_KIND_SPECULATIVE_LOAD_HARDENING = 59,
   ATTR_KIND_IMMARG = 60,
   ATTR_KIND_WILLRETURN = 61,
   ATTR_KIND_NOFREE = 62,
   ATTR_KIND_NOSYNC = 63,
   ATTR_KIND_SANITIZE_MEMTAG = 64,
+  ATTR_KIND_ALWAYS_INLINE_RECURSIVELY = 65,
 };
 
 enum ComdatSelectionKindCodes {
   COMDAT_SELECTION_KIND_ANY = 1,
   COMDAT_SELECTION_KIND_EXACT_MATCH = 2,
   COMDAT_SELECTION_KIND_LARGEST = 3,
   COMDAT_SELECTION_KIND_NO_DUPLICATES = 4,
   COMDAT_SELECTION_KIND_SAME_SIZE = 5,

llvm/include/llvm/IR/Attributes.td

 
 /// The result of the function is guaranteed to point to a number of bytes that
 /// we can determine if we know the value of the function's arguments.
 def AllocSize : EnumAttr<"allocsize">;
 
 /// inline=always.
 def AlwaysInline : EnumAttr<"alwaysinline">;
 
+/// Like AlwaysInline, but applies recursively
+def AlwaysInlineRecursively : EnumAttr<"alwaysinline_recursively">;
+
 /// Function can access memory only using pointers based on its arguments.
 def ArgMemOnly : EnumAttr<"argmemonly">;
 
 /// Callee is recognized as a builtin, despite nobuiltin attribute on its
 /// declaration.
 def Builtin : EnumAttr<"builtin">;
 
 /// Pass structure by value.

llvm/lib/Analysis/InlineCost.cpp

InlineCost llvm::getInlineCost(
   // whenever possible.
   if (Call.hasFnAttr(Attribute::AlwaysInline)) {
     auto IsViable = isInlineViable(*Callee);
     if (IsViable.isSuccess())
       return llvm::InlineCost::getAlways("always inline attribute");
     return llvm::InlineCost::getNever(IsViable.getFailureReason());
   }
 
+  // Inline call sites marked alwaysinline_recursively
+  if (Call.hasFnAttr(Attribute::AlwaysInlineRecursively)) {
+    auto IsViable = isInlineViable(*Callee);
+    if (IsViable.isSuccess())
+      return llvm::InlineCost::getAlways("alwaysinline_recursively attribute");
+    return llvm::InlineCost::getNever(IsViable.getFailureReason());
+  }
+
   // Never inline functions with conflicting attributes (unless callee has
   // always-inline attribute).
   Function *Caller = Call.getCaller();
   if (!functionsHaveCompatibleAttributes(Caller, Callee, CalleeTTI, GetTLI))
     return llvm::InlineCost::getNever("conflicting attributes");
 
   // Don't inline this call if the caller has the optnone attribute.
   if (Caller->hasOptNone())

llvm/lib/AsmParser/LLLexer.cpp

#define KEYWORD(STR)                                                           \
   KEYWORD(tailcc);
 
   KEYWORD(cc);
   KEYWORD(c);
 
   KEYWORD(attributes);
 
   KEYWORD(alwaysinline);
+  KEYWORD(alwaysinline_recursively);
   KEYWORD(allocsize);
   KEYWORD(argmemonly);
   KEYWORD(builtin);
   KEYWORD(byval);
   KEYWORD(inalloca);
   KEYWORD(cold);
   KEYWORD(convergent);
   KEYWORD(dereferenceable);

llvm/lib/AsmParser/LLParser.cpp

case lltok::kw_allocsize: {
       Optional<unsigned> NumElemsArg;
       // inAttrGrp doesn't matter; we only support allocsize(a[, b])
       if (parseAllocSizeArguments(ElemSizeArg, NumElemsArg))
         return true;
       B.addAllocSizeAttr(ElemSizeArg, NumElemsArg);
       continue;
     }
     case lltok::kw_alwaysinline: B.addAttribute(Attribute::AlwaysInline); break;
+    case lltok::kw_alwaysinline_recursively:
+      B.addAttribute(Attribute::AlwaysInlineRecursively);
+      break;
     case lltok::kw_argmemonly: B.addAttribute(Attribute::ArgMemOnly); break;
     case lltok::kw_builtin: B.addAttribute(Attribute::Builtin); break;
     case lltok::kw_cold: B.addAttribute(Attribute::Cold); break;
     case lltok::kw_convergent: B.addAttribute(Attribute::Convergent); break;
     case lltok::kw_inaccessiblememonly:
       B.addAttribute(Attribute::InaccessibleMemOnly); break;
     case lltok::kw_inaccessiblemem_or_argmemonly:
       B.addAttribute(Attribute::InaccessibleMemOrArgMemOnly); break;
while (true) {
     case lltok::kw_swifterror:      B.addAttribute(Attribute::SwiftError); break;
     case lltok::kw_swiftself:       B.addAttribute(Attribute::SwiftSelf); break;
     case lltok::kw_writeonly:       B.addAttribute(Attribute::WriteOnly); break;
     case lltok::kw_zeroext:         B.addAttribute(Attribute::ZExt); break;
     case lltok::kw_immarg:          B.addAttribute(Attribute::ImmArg); break;
 
     case lltok::kw_alignstack:
     case lltok::kw_alwaysinline:
+    case lltok::kw_alwaysinline_recursively:
     case lltok::kw_argmemonly:
     case lltok::kw_builtin:
     case lltok::kw_inlinehint:
     case lltok::kw_jumptable:
     case lltok::kw_minsize:
     case lltok::kw_naked:
     case lltok::kw_nobuiltin:
     case lltok::kw_noduplicate:
while (true) {
     case lltok::kw_swifterror:
     case lltok::kw_swiftself:
     case lltok::kw_immarg:
       HaveError |= Error(Lex.getLoc(), "invalid use of parameter-only attribute");
       break;
 
     case lltok::kw_alignstack:
     case lltok::kw_alwaysinline:
+    case lltok::kw_alwaysinline_recursively:
     case lltok::kw_argmemonly:
     case lltok::kw_builtin:
     case lltok::kw_cold:
     case lltok::kw_inlinehint:
     case lltok::kw_jumptable:
     case lltok::kw_minsize:
     case lltok::kw_naked:
     case lltok::kw_nobuiltin:

llvm/lib/AsmParser/LLToken.h

enum Kind {
   kw_amdgpu_cs,
   kw_amdgpu_kernel,
   kw_tailcc,
 
   // Attributes:
   kw_attributes,
   kw_allocsize,
   kw_alwaysinline,
+  kw_alwaysinline_recursively,
   kw_argmemonly,
   kw_sanitize_address,
   kw_sanitize_hwaddress,
   kw_sanitize_memtag,
   kw_builtin,
   kw_byval,
   kw_inalloca,
   kw_cold,

llvm/lib/Bitcode/Reader/BitcodeReader.cpp

case Attribute::ArgMemOnly:
     llvm_unreachable("argmemonly attribute not supported in raw format");
     break;
   case Attribute::AllocSize:
     llvm_unreachable("allocsize not supported in raw format");
     break;
   case Attribute::SanitizeMemTag:
     llvm_unreachable("sanitize_memtag attribute not supported in raw format");
     break;
+  case Attribute::AlwaysInlineRecursively:
+    llvm_unreachable(
+        "alwaysinline_recursively attribute not supported in raw format");
+    break;
   }
   llvm_unreachable("Unsupported attribute type");
 }
 
 static void addRawAttributeValue(AttrBuilder &B, uint64_t Val) {
   if (!Val) return;
 
   for (Attribute::AttrKind I = Attribute::None; I != Attribute::EndAttrKinds;
        I = Attribute::AttrKind(I + 1)) {
-    if (I == Attribute::SanitizeMemTag ||
-        I == Attribute::Dereferenceable ||
-        I == Attribute::DereferenceableOrNull ||
-        I == Attribute::ArgMemOnly ||
-        I == Attribute::AllocSize ||
-        I == Attribute::NoSync)
+    if (I == Attribute::SanitizeMemTag || I == Attribute::Dereferenceable ||
+        I == Attribute::DereferenceableOrNull || I == Attribute::ArgMemOnly ||
+        I == Attribute::AllocSize || I == Attribute::NoSync ||
+        I == Attribute::AlwaysInlineRecursively)
       continue;
     if (uint64_t A = (Val & getRawAttributeMask(I))) {
       if (I == Attribute::Alignment)
         B.addAlignmentAttr(1ULL << ((A >> 16) - 1));
       else if (I == Attribute::StackAlignment)
         B.addStackAlignmentAttr(1ULL << ((A >> 26)-1));
       else
         B.addAttribute(I);

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

 static uint64_t getAttrKindEncoding(Attribute::AttrKind Kind) {
   switch (Kind) {
   case Attribute::Alignment:
     return bitc::ATTR_KIND_ALIGNMENT;
   case Attribute::AllocSize:
     return bitc::ATTR_KIND_ALLOC_SIZE;
   case Attribute::AlwaysInline:
     return bitc::ATTR_KIND_ALWAYS_INLINE;
+  case Attribute::AlwaysInlineRecursively:
+    return bitc::ATTR_KIND_ALWAYS_INLINE_RECURSIVELY;
   case Attribute::ArgMemOnly:
     return bitc::ATTR_KIND_ARGMEMONLY;
   case Attribute::Builtin:
     return bitc::ATTR_KIND_BUILTIN;
   case Attribute::ByVal:
     return bitc::ATTR_KIND_BY_VAL;
   case Attribute::Convergent:
     return bitc::ATTR_KIND_CONVERGENT;

llvm/lib/CodeGen/SafeStack.cpp

for (inst_iterator It = inst_begin(&F), Ie = inst_end(&F); It != Ie;) {
         II->eraseFromParent();
       }
     }
   }
 }
 
 bool SafeStack::ShouldInlinePointerAddress(CallSite &CS) {
   Function *Callee = CS.getCalledFunction();
-  if (CS.hasFnAttr(Attribute::AlwaysInline) &&
+  if ((CS.hasFnAttr(Attribute::AlwaysInline) ||
+       CS.hasFnAttr(Attribute::AlwaysInlineRecursively)) &&
       isInlineViable(*Callee).isSuccess())
     return true;
   if (Callee->isInterposable() || Callee->hasFnAttribute(Attribute::NoInline) ||
       CS.isNoInline())
     return false;
   return true;
 }
 

llvm/lib/IR/Attributes.cpp

std::string Attribute::getAsString(bool InAttrGrp) const {
   if (hasAttribute(Attribute::SanitizeAddress))
     return "sanitize_address";
   if (hasAttribute(Attribute::SanitizeHWAddress))
     return "sanitize_hwaddress";
   if (hasAttribute(Attribute::SanitizeMemTag))
     return "sanitize_memtag";
   if (hasAttribute(Attribute::AlwaysInline))
     return "alwaysinline";
+  if (hasAttribute(Attribute::AlwaysInlineRecursively))
+    return "alwaysinline_recursively";
   if (hasAttribute(Attribute::ArgMemOnly))
     return "argmemonly";
   if (hasAttribute(Attribute::Builtin))
     return "builtin";
   if (hasAttribute(Attribute::Convergent))
     return "convergent";
   if (hasAttribute(Attribute::SwiftError))
     return "swifterror";

llvm/lib/IR/Verifier.cpp

static bool isFuncOnlyAttr(Attribute::AttrKind Kind) {
   case Attribute::ArgMemOnly:
   case Attribute::NoRecurse:
   case Attribute::InaccessibleMemOnly:
   case Attribute::InaccessibleMemOrArgMemOnly:
   case Attribute::AllocSize:
   case Attribute::SpeculativeLoadHardening:
   case Attribute::Speculatable:
   case Attribute::StrictFP:
+  case Attribute::AlwaysInlineRecursively:
     return true;
   default:
     break;
   }
   return false;
 }
 
 /// Return true if this is a function attribute that can also appear on
Assert(!(Attrs.hasAttribute(Attribute::ReadOnly) &&
          V);
 
   Assert(!(Attrs.hasAttribute(Attribute::NoInline) &&
            Attrs.hasAttribute(Attribute::AlwaysInline)),
          "Attributes "
          "'noinline and alwaysinline' are incompatible!",
          V);
 
+  Assert(!(Attrs.hasAttribute(Attribute::NoInline) &&
+           Attrs.hasAttribute(Attribute::AlwaysInlineRecursively)),
+         "Attributes "
+         "'noinline and alwaysinline_recursively' are incompatible!",
+         V);
+
   if (Attrs.hasAttribute(Attribute::ByVal) && Attrs.getByValType()) {
     Assert(Attrs.getByValType() == cast<PointerType>(Ty)->getElementType(),
            "Attribute 'byval' type does not match parameter!", V);
   }
 
   AttrBuilder IncompatibleAttrs = AttributeFuncs::typeIncompatible(Ty);
   Assert(!AttrBuilder(Attrs).overlaps(IncompatibleAttrs),
          "Wrong types for attribute: " +
void Verifier::verifyFunctionAttrs(FunctionType *FT, AttributeList Attrs,
   Assert(!(Attrs.hasFnAttribute(Attribute::ReadNone) &&
            Attrs.hasFnAttribute(Attribute::InaccessibleMemOnly)),
          "Attributes 'readnone and inaccessiblememonly' are incompatible!", V);
 
   Assert(!(Attrs.hasFnAttribute(Attribute::NoInline) &&
            Attrs.hasFnAttribute(Attribute::AlwaysInline)),
          "Attributes 'noinline and alwaysinline' are incompatible!", V);
 
+  Assert(!(Attrs.hasFnAttribute(Attribute::NoInline) &&
+           Attrs.hasFnAttribute(Attribute::AlwaysInlineRecursively)),
+         "Attributes 'noinline and alwaysinline_recursively' are incompatible!",
+         V);
+
   if (Attrs.hasFnAttribute(Attribute::OptimizeNone)) {
     Assert(Attrs.hasFnAttribute(Attribute::NoInline),
            "Attribute 'optnone' requires 'noinline'!", V);
 
     Assert(!Attrs.hasFnAttribute(Attribute::OptimizeForSize),
            "Attributes 'optsize and optnone' are incompatible!", V);
 
     Assert(!Attrs.hasFnAttribute(Attribute::MinSize),

llvm/lib/Target/AMDGPU/AMDGPUInline.cpp

InlineCost AMDGPUInliner::getInlineCost(CallSite CS) {
 
   if (CS.isNoInline())
     return llvm::InlineCost::getNever("noinline");
 
   TargetTransformInfo &TTI = TTIWP->getTTI(*Callee);
   if (!TTI.areInlineCompatible(Caller, Callee))
     return llvm::InlineCost::getNever("incompatible");
 
-  if (CS.hasFnAttr(Attribute::AlwaysInline)) {
+  if (CS.hasFnAttr(Attribute::AlwaysInline) ||
+      CS.hasFnAttr(Attribute::AlwaysInlineRecursively)) {
     auto IsViable = isInlineViable(*Callee);
     if (IsViable.isSuccess())
       return llvm::InlineCost::getAlways("alwaysinline viable");
     return llvm::InlineCost::getNever(IsViable.getFailureReason());
   }
 
   if (isWrapperOnlyCall(CS))
     return llvm::InlineCost::getAlways("wrapper-only call");

llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp

CleanupAndExit:
 
   if (!Overlap) {
     if (DisableMemcpyIdiom || !HasMemcpy)
       goto CleanupAndExit;
   } else {
     // Don't generate memmove if this function will be inlined. This is
     // because the caller will undergo this transformation after inlining.
     Function *Func = CurLoop->getHeader()->getParent();
-    if (Func->hasFnAttribute(Attribute::AlwaysInline))
+    if ((Func->hasFnAttribute(Attribute::AlwaysInline) ||
+         Func->hasFnAttribute(Attribute::AlwaysInlineRecursively)))
       goto CleanupAndExit;
 
     // In case of a memmove, the call to memmove will be executed instead
     // of the loop, so we need to make sure that there is nothing else in
     // the loop than the load, store and instructions that these two depend
     // on.
     SmallVector<Instruction*,2> Insts;
     Insts.push_back(SI);

llvm/lib/Transforms/IPO/AlwaysInliner.cpp

std::function<AssumptionCache &(Function &)> GetAssumptionCache =
     return FAM.getResult<AssumptionAnalysis>(F);
   };
   InlineFunctionInfo IFI(/*cg=*/nullptr, &GetAssumptionCache);
 
   SmallSetVector<CallSite, 16> Calls;
   bool Changed = false;
   SmallVector<Function *, 16> InlinedFunctions;
   for (Function &F : M)
-    if (!F.isDeclaration() && F.hasFnAttribute(Attribute::AlwaysInline) &&
+    if (!F.isDeclaration() &&
+        (F.hasFnAttribute(Attribute::AlwaysInline) ||
+         F.hasFnAttribute(Attribute::AlwaysInlineRecursively)) &&
         isInlineViable(F).isSuccess()) {
       Calls.clear();
 
       for (User *U : F.users())
         if (auto CS = CallSite(U))
           if (CS.getCalledFunction() == &F)
             Calls.insert(CS);
 
InlineCost AlwaysInlinerLegacyPass::getInlineCost(CallSite CS) {
   // that are viable for inlining.
   if (!Callee)
     return InlineCost::getNever("indirect call");
 
   // FIXME: We shouldn't even get here for declarations.
   if (Callee->isDeclaration())
     return InlineCost::getNever("no definition");
 
-  if (!CS.hasFnAttr(Attribute::AlwaysInline))
+  if (!CS.hasFnAttr(Attribute::AlwaysInline) &&
+      !CS.hasFnAttr(Attribute::AlwaysInlineRecursively))
     return InlineCost::getNever("no alwaysinline attribute");
 
   auto IsViable = isInlineViable(*Callee);
   if (!IsViable.isSuccess())
     return InlineCost::getNever(IsViable.getFailureReason());
 
   return InlineCost::getAlways("always inliner");
 }

llvm/lib/Transforms/IPO/Attributor.cpp

case Instruction::Ret:
       IsInterestingOpcode = true;
     }
     if (IsInterestingOpcode)
       InstOpcodeMap[I.getOpcode()].push_back(&I);
     if (I.mayReadOrWriteMemory())
       ReadOrWriteInsts.push_back(&I);
   }
 
-  if (F.hasFnAttribute(Attribute::AlwaysInline) &&
+  if ((F.hasFnAttribute(Attribute::AlwaysInline) ||
+       F.hasFnAttribute(Attribute::AlwaysInlineRecursively)) &&
       isInlineViable(F).isSuccess())
     InfoCache.InlineableFunctions.insert(&F);
 }
 
 void Attributor::recordDependence(const AbstractAttribute &FromAA,
                                   const AbstractAttribute &ToAA,
                                   DepClassTy DepClass) {
   if (FromAA.getState().isAtFixpoint())

llvm/lib/Transforms/IPO/ForceFunctionAttrs.cpp

ForceAttributes("force-attribute", cl::Hidden,
                     cl::desc("Add an attribute to a function. This should be a "
                              "pair of 'function-name:attribute-name', for "
                              "example -force-attribute=foo:noinline. This "
                              "option can be specified multiple times."));
 
 static Attribute::AttrKind parseAttrKind(StringRef Kind) {
   return StringSwitch<Attribute::AttrKind>(Kind)
       .Case("alwaysinline", Attribute::AlwaysInline)
+      .Case("alwaysinline_recursively", Attribute::AlwaysInlineRecursively)
       .Case("builtin", Attribute::Builtin)
       .Case("cold", Attribute::Cold)
       .Case("convergent", Attribute::Convergent)
       .Case("inlinehint", Attribute::InlineHint)
       .Case("jumptable", Attribute::JumpTable)
       .Case("minsize", Attribute::MinSize)
       .Case("naked", Attribute::Naked)
       .Case("nobuiltin", Attribute::NoBuiltin)

llvm/lib/Transforms/IPO/HotColdSplitting.cpp

bool HotColdSplitting::isFunctionCold(const Function &F) const {
   return false;
 }
 
 // Returns false if the function should not be considered for hot-cold split
 // optimization.
 bool HotColdSplitting::shouldOutlineFrom(const Function &F) const {
   if (F.hasFnAttribute(Attribute::AlwaysInline))
     return false;
+  if (F.hasFnAttribute(Attribute::AlwaysInlineRecursively))
+    return false;
 
   if (F.hasFnAttribute(Attribute::NoInline))
     return false;
 
   // A function marked `noreturn` may contain unreachable terminators: these
   // should not be considered cold, as the function may be a trampoline.
   if (F.hasFnAttribute(Attribute::NoReturn))
     return false;

llvm/lib/Transforms/IPO/Inliner.cpp

for (unsigned CSi = 0; CSi != CallSites.size(); ++CSi) {
         CG[Caller]->removeCallEdgeFor(*cast<CallBase>(CS.getInstruction()));
         Instr->eraseFromParent();
         ++NumCallsDeleted;
       } else {
         // Get DebugLoc to report. CS will be invalid after Inliner.
         DebugLoc DLoc = CS->getDebugLoc();
         BasicBlock *Block = CS.getParent();
 
+        bool AlwaysInlineRecursively =
+            CS.hasFnAttr(Attribute::AlwaysInlineRecursively);
+
         // Attempt to inline the function.
         using namespace ore;
 
         InlineResult IR = InlineCallIfPossible(
             CS, InlineInfo, InlinedArrayAllocas, InlineHistoryID,
             InsertLifetime, AARGetter, ImportedFunctionsStats);
         if (!IR.isSuccess()) {
           setInlineRemark(CS, std::string(IR.getFailureReason()) + "; " +
for (unsigned CSi = 0; CSi != CallSites.size(); ++CSi) {
         // If inlining this function gave us any new call sites, throw them
         // onto our worklist to process.  They are useful inline candidates.
         if (!InlineInfo.InlinedCalls.empty()) {
           // Create a new inline history entry for this, so that we remember
           // that these new callsites came about due to inlining Callee.
           int NewHistoryID = InlineHistory.size();
           InlineHistory.push_back(std::make_pair(Callee, InlineHistoryID));
 
-          for (Value *Ptr : InlineInfo.InlinedCalls)
-            CallSites.push_back(std::make_pair(CallSite(Ptr), NewHistoryID));
+          for (Value *Ptr : InlineInfo.InlinedCalls) {
+            CallSite NewCS(Ptr);
+            // Propagate alwaysinline_recursively attribute to all inlined call
+            // sites which are not marked noinline
+            if (AlwaysInlineRecursively &&
+                !NewCS.hasFnAttr(Attribute::NoInline))
+              NewCS.addAttribute(AttributeList::FunctionIndex,
+                                 Attribute::AlwaysInlineRecursively);
+            CallSites.push_back(std::make_pair(NewCS, NewHistoryID));
+          }
         }
       }
 
       // If we inlined or deleted the last possible call site to the function,
       // delete the function body now.
       if (Callee && Callee->use_empty() && Callee->hasLocalLinkage() &&
           // TODO: Can remove if in SCC now.
           !SCCFunctions.count(Callee) &&
for (const auto &I : CG) {
     CallGraphNode *CGN = I.second.get();
     Function *F = CGN->getFunction();
     if (!F || F->isDeclaration())
       continue;
 
     // Handle the case when this function is called and we only want to care
     // about always-inline functions. This is a bit of a hack to share code
     // between here and the InlineAlways pass.
-    if (AlwaysInlineOnly && !F->hasFnAttribute(Attribute::AlwaysInline))
+    if (AlwaysInlineOnly && !F->hasFnAttribute(Attribute::AlwaysInline) &&
+        !F->hasFnAttribute(Attribute::AlwaysInlineRecursively))
       continue;
 
     // If the only remaining users of the function are dead constants, remove
     // them.
     F->removeDeadConstantUsers();
 
     if (!F->isDefTriviallyDead())
       continue;
for (; i < (int)Calls.size() && Calls[i].first.getCaller() == &F; ++i) {
           /*cg=*/nullptr, &GetAssumptionCache, PSI,
           &FAM.getResult<BlockFrequencyAnalysis>(*(CS.getCaller())),
           &FAM.getResult<BlockFrequencyAnalysis>(Callee));
 
       // Get DebugLoc to report. CS will be invalid after Inliner.
       DebugLoc DLoc = CS->getDebugLoc();
       BasicBlock *Block = CS.getParent();
 
+      bool AlwaysInlineRecursively =
+          CS.hasFnAttr(Attribute::AlwaysInlineRecursively);
+
       using namespace ore;
 
       InlineResult IR = InlineFunction(CS, IFI);
       if (!IR.isSuccess()) {
         setInlineRemark(CS, std::string(IR.getFailureReason()) + "; " +
                                 inlineCostStr(*OIC));
         ORE.emit([&]() {
           return OptimizationRemarkMissed(DEBUG_TYPE, "NotInlined", DLoc, Block)
for (; i < (int)Calls.size() && Calls[i].first.getCaller() == &F; ++i) {
           if (!NewCallee) {
             // Try to promote an indirect (virtual) call without waiting for the
             // post-inline cleanup and the next DevirtSCCRepeatedPass iteration
             // because the next iteration may not happen and we may miss
             // inlining it.
             if (tryPromoteCall(CS))
               NewCallee = CS.getCalledFunction();
           }
-          if (NewCallee)
-            if (!NewCallee->isDeclaration())
+          if (NewCallee) {
+            if (!NewCallee->isDeclaration()) {
+              // Propagate alwaysinline_recursively attribute to all inlined
+              // call sites, which are not marked noinline
+              if (AlwaysInlineRecursively && !CS.hasFnAttr(Attribute::NoInline))
+                CS.addAttribute(AttributeList::FunctionIndex,
+                                Attribute::AlwaysInlineRecursively);
               Calls.push_back({CS, NewHistoryID});
             }
           }
+        }
+      }
 
       if (InlinerFunctionImportStats != InlinerFunctionImportStatsOpts::No)
         ImportedFunctionsStats->recordInline(F, Callee);
 
       // Merge the attributes based on the inlining.
       AttributeFuncs::mergeAttributesForInlining(F, Callee);
 
       // For local functions, check whether this makes the callee trivially

llvm/lib/Transforms/IPO/PartialInlining.cpp

std::pair<bool, Function *> PartialInlinerImpl::unswitchFunction(Function *F) {
 
   if (F->hasAddressTaken())
     return {false, nullptr};
 
   // Let inliner handle it
   if (F->hasFnAttribute(Attribute::AlwaysInline))
     return {false, nullptr};
 
+  if (F->hasFnAttribute(Attribute::AlwaysInlineRecursively))
+    return {false, nullptr};
+
   if (F->hasFnAttribute(Attribute::NoInline))
     return {false, nullptr};
 
   if (PSI->isFunctionEntryCold(F))
     return {false, nullptr};
 
   if (F->users().empty())
     return {false, nullptr};

llvm/lib/Transforms/IPO/SyntheticCountsPropagation.cpp

auto MayHaveIndirectCalls = [](Function &F) {
     return false;
   };
 
   for (Function &F : M) {
     uint64_t InitialCount = InitialSyntheticCount;
     if (F.isDeclaration())
       continue;
     if (F.hasFnAttribute(Attribute::AlwaysInline) ||
+        F.hasFnAttribute(Attribute::AlwaysInlineRecursively) ||
         F.hasFnAttribute(Attribute::InlineHint)) {
       // Use a higher value for inline functions to account for the fact that
       // these are usually beneficial to inline.
       InitialCount = InlineSyntheticCount;
     } else if (F.hasLocalLinkage() && !MayHaveIndirectCalls(F)) {
       // Local functions without inline hints get counts only through
       // propagation.
       InitialCount = 0;

llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp

static inline bool shouldRecordFunctionAddr(Function *F) {
   if (!F->hasLinkOnceLinkage() && !F->hasLocalLinkage() &&
       !HasAvailableExternallyLinkage)
     return true;
 
   // A function marked 'alwaysinline' with available_externally linkage can't
   // have its address taken. Doing so would create an undefined external ref to
   // the function, which would fail to link.
   if (HasAvailableExternallyLinkage &&
-      F->hasFnAttribute(Attribute::AlwaysInline))
+      (F->hasFnAttribute(Attribute::AlwaysInline) ||
+       F->hasFnAttribute(Attribute::AlwaysInlineRecursively)))
     return false;
 
   // Prohibit function address recording if the function is both internal and
   // COMDAT. This avoids the profile data variable referencing internal symbols
   // in COMDAT.
   if (F->hasLocalLinkage() && F->hasComdat())
     return false;
 

llvm/lib/Transforms/Utils/CodeExtractor.cpp

if (Attr.isStringAttribute()) {
       case Attribute::ZExt:
       case Attribute::ImmArg:
       case Attribute::EndAttrKinds:
       case Attribute::EmptyKey:
       case Attribute::TombstoneKey:
         continue;
       // Those attributes should be safe to propagate to the extracted function.
       case Attribute::AlwaysInline:
+      case Attribute::AlwaysInlineRecursively:
       case Attribute::Cold:
       case Attribute::NoRecurse:
       case Attribute::InlineHint:
       case Attribute::MinSize:
       case Attribute::NoDuplicate:
       case Attribute::NoFree:
       case Attribute::NoImplicitFloat:
       case Attribute::NoInline:

llvm/test/Transforms/Inline/always-inline-recursively.ll

+; RUN: opt < %s -inline-threshold=0 -inline -S | FileCheck %s
+; RUN: opt < %s -inline-threshold=0 -always-inline -S | FileCheck %s
+;
+; Ensure the threshold has no impact on these decisions.
+; RUN: opt < %s -inline-threshold=20000000 -inline -S | FileCheck %s
+; RUN: opt < %s -inline-threshold=20000000 -always-inline -S | FileCheck %s
+; RUN: opt < %s -inline-threshold=-20000000 -inline -S | FileCheck %s
+; RUN: opt < %s -inline-threshold=-20000000 -always-inline -S | FileCheck %s
+
+; In the tests involving recursive functions we call the external function and
+; continue recursion depending on the external variable, so that any recursion
+; conditions are opaque to the optimizer
+
+; Following tests are conducted, for annotating call-sites and function declarations:
+;   Test that a simple tree call-graph is inlined
+;   Test that functions marked noinline are not inlined
+;   Test that a recursive call is not inlined
+;   Test that an indirectly recursive call is inlined until a directly recursive call remains
+
+; External funcion not visible to the optimizer
+declare void @ext_func()
+; External variable not visible to the optimizer
+@ext_var = external global i32
+
+; Test that a simple tree call-graph is inlined
+; when annotating call-sites
+
+define void @test_calls_tree() {
+; CHECK-LABEL: @test_calls_tree() {
+  call void @test_calls_tree_1() alwaysinline_recursively
+; CHECK-NEXT: call void @ext_func() #1
+  call void @test_calls_tree_2() alwaysinline_recursively
+; CHECK-NEXT: call void @ext_func() #1
+; CHECK-NEXT: call void @ext_func() #1
+; CHECK-NEXT: call void @ext_func() #1
+  ret void
+; CHECK-NEXT: ret void
+}
+
+define void @test_calls_tree_1() {
+  call void @ext_func()
+  ret void
+}
+
+define void @test_calls_tree_2() {
+  call void @test_calls_tree_2_1()
+  call void @test_calls_tree_2_2()
+  call void @test_calls_tree_2_3()
+  ret void
+}
+
+define void @test_calls_tree_2_1() {
+    call void @ext_func()
+    ret void
+}
+
+define void @test_calls_tree_2_2() {
+    call void @ext_func()
+    ret void
+}
+
+define void @test_calls_tree_2_3() {
+    call void @ext_func()
+    ret void
+}
+
+; Test that functions marked noinline are not inlined
+; when annotating call-sites
+
+define void @test_calls_noinline() {
+; CHECK-LABEL: @test_calls_noinline() {
+  call void @test_calls_noinline_inlined() alwaysinline_recursively
+; CHECK-NEXT: call void @ext_func() #1
+  call void @test_calls_noinline_inlined2() alwaysinline_recursively
+; CHECK-NEXT: call void @test_calls_noinline_notinlined()
+  ret void
+; CHECK-NEXT: ret void
+}
+
+define void @test_calls_noinline_inlined() {
+  call void @ext_func()
+  ret void
+}
+
+define void @test_calls_noinline_inlined2() {
+  call void @test_calls_noinline_notinlined()
+  ret void
+}
+
+define void @test_calls_noinline_notinlined() noinline {
+    call void @ext_func()
+    ret void
+}
+
+; Test that a recursive call is not inlined
+; when annotating call-sites
+
+define void @test_calls_rec() {
+; CHECK-LABEL: @test_calls_rec() {
+  call void @test_calls_rec_func() alwaysinline_recursively
+; CHECK-NEXT: call void @test_calls_rec_func() #1
+  ret void
+; CHECK-NEXT: ret void
+}
+
+define void @test_calls_rec_func() {
+  call void @ext_func()
+  %1 = load i32, i32* @ext_var
+  %2 = icmp ne i32 %1, 0
+  br i1 %2, label %3, label %4
+
+3:
+  call void @test_calls_rec_func()
+  br label %4
+
+4:
+  ret void
+}
+
+; Test that an indirectly recursive call is inlined
+; until a directly recursive call remains
+; when annotating call-sites
+
+define void @test_calls_irec() {
+; CHECK-LABEL: @test_calls_irec() {
+  call void @test_calls_irec_func() alwaysinline_recursively
+; CHECK: call void @test_calls_irec() #1
+  ret void
+; CHECK: ret void
+}
+
+define void @test_calls_irec_func() {
+  call void @ext_func()
+  %1 = load i32, i32* @ext_var
+  %2 = icmp ne i32 %1, 0
+  br i1 %2, label %3, label %4
+
+3:
+  call void @test_calls_irec()
+  br label %4
+
+4:
+  ret void
+}
+
+; Test that a simple tree call-graph is inlined
+; when annotating function definitions
+
+define void @test_defs_tree() {
+; CHECK-LABEL: @test_defs_tree() {
+  call void @test_defs_tree_1()
+; CHECK-NEXT: call void @ext_func() #1
+  call void @test_defs_tree_2()
+; CHECK-NEXT: call void @ext_func() #1
+; CHECK-NEXT: call void @ext_func() #1
+; CHECK-NEXT: call void @ext_func() #1
+  ret void
+; CHECK-NEXT: ret void
+}
+
+define void @test_defs_tree_1() alwaysinline_recursively {
+  call void @ext_func()
+  ret void
+}
+
+define void @test_defs_tree_2() alwaysinline_recursively {
+  call void @test_defs_tree_2_1()
+  call void @test_defs_tree_2_2()
+  call void @test_defs_tree_2_3()
+  ret void
+}
+
+define void @test_defs_tree_2_1() {
+    call void @ext_func()
+    ret void
+}
+
+define void @test_defs_tree_2_2() {
+    call void @ext_func()
+    ret void
+}
+
+define void @test_defs_tree_2_3() {
+    call void @ext_func()
+    ret void
+}
+
+; Test that functions marked noinline are not inlined
+; when annotating function definitions
+
+define void @test_defs_noinline() {
+; CHECK-LABEL: @test_defs_noinline() {
+  call void @test_defs_noinline_inlined()
+; CHECK-NEXT: call void @ext_func() #1
+  call void @test_defs_noinline_inlined2()
+; CHECK-NEXT: call void @test_defs_noinline_notinlined()
+  ret void
+; CHECK-NEXT: ret void
+}
+
+define void @test_defs_noinline_inlined() alwaysinline_recursively {
+  call void @ext_func()
+  ret void
+}
+
+define void @test_defs_noinline_inlined2() alwaysinline_recursively {
+  call void @test_defs_noinline_notinlined()
+  ret void
+}
+
+define void @test_defs_noinline_notinlined() noinline {
+    call void @ext_func()
+    ret void
+}
+
+; Test that a recursive call is not inlined
+; when annotating function definitions
+
+define void @test_defs_rec() {
+; CHECK-LABEL: @test_defs_rec() {
+  call void @test_defs_rec_func()
+; CHECK-NEXT: call void @test_defs_rec_func()
+  ret void
+; CHECK-NEXT: ret void
+}
+
+define void @test_defs_rec_func() alwaysinline_recursively {
+  call void @ext_func()
+  %1 = load i32, i32* @ext_var
+  %2 = icmp ne i32 %1, 0
+  br i1 %2, label %3, label %4
+
+3:
+  call void @test_defs_rec_func()
+  br label %4
+
+4:
+  ret void
+}
+
+; Test that an indirectly recursive call is inlined
+; until a directly recursive call remains
+; when annotating function definitions
+
+define void @test_defs_irec() {
+; CHECK-LABEL: @test_defs_irec() {
+  call void @test_defs_irec_func()
+; CHECK: call void @test_defs_irec() #1
+  ret void
+; CHECK: ret void
+}
+
+define void @test_defs_irec_func() alwaysinline_recursively {
+  call void @ext_func()
+  %1 = load i32, i32* @ext_var
+  %2 = icmp ne i32 %1, 0
+  br i1 %2, label %3, label %4
+
+3:
+  call void @test_defs_irec()
+  br label %4
+
+4:
+  ret void
+}
+
+; CHECK: attributes #1 = { alwaysinline_recursively }