This is an archive of the discontinued LLVM Phabricator instance.

[libcxx] By default, do not use internal_linkage to hide symbols from the ABI
ClosedPublic

Authored by ldionne on Aug 13 2018, 11:25 AM.

Details

Summary

This led to symbol size problems in Chromium, and we expect this may be
the case in other projects built in debug mode too. Instead, unless users
explicitly ask for internal_linkage, we use always_inline like we used to.

In the future, when we have a solution that allows us to drop always_inline
without falling back on internal_linkage, we can replace always_inline by
that.

Note that this commit introduces a change in contract for existing libc++
users: by default, libc++ used to guarantee that TUs built with different
versions of libc++ could be linked together. With the introduction of the
_LIBCPP_HIDE_FROM_ABI_PER_TU macro, the default behavior is that TUs built
with different libc++ versions are not guaranteed to link. This is a change
in contract but not a change in behavior, since the current implementation
still allows linking TUs built with different libc++ versions together.

Diff Detail

Repository
rL LLVM

Event Timeline

ldionne created this revision.Aug 13 2018, 11:25 AM

The intent is for this patch to be cherry-picked onto the LLVM 7 release.

This is a straw man proposal to fix issues raised in https://reviews.llvm.org/D49240. The idea is that in the future, we would probably want the non-internal_linkage case to be the default. By introducing the _LIBCPP_HIDE_FROM_ABI_PER_TU setting, we're setting up libc++ users for the right default (i.e. no insane guarantee of being able to link TUs built with different libc++ versions), without actually changing any behavior for the LLVM 7 release. Once we have a solution that allows us to drop _LIBCPP_ALWAYS_INLINE from _LIBCPP_HIDE_FROM_ABI (in LLVM 8), we can just do it and everybody should see code size improvements, without a crazy increase in the number of symbols (as reported by Chromium). In LLVM8, the few projects that actually need to link TUs built with different libc++ versions can then just define _LIBCPP_HIDE_FROM_ABI_PER_TU to keep today's behavior.

ldionne updated this revision to Diff 160414.Aug 13 2018, 11:35 AM

Update documentation for _LIBCPP_HIDE_FROM_ABI

I suspect this might deserve a wider discussion. At least cfe-dev, perhaps?

rnk added a comment.Aug 13 2018, 12:21 PM

I'd prefer not to do this, since internal_linkage generates smaller, more debuggable code by default. I think the symbol table size increase may be specific to MachO, and it may be possible to work around this by changing ld64 to pool strings for symbols by default. I don't know enough about MachO to say if this is possible.

I also think the symbol table size problem may be limited to "large" users of C++: people with 500+ object files using libc++ in a DSO. I'm more comfortable imposing a cost on these users to ask them to opt in to some setting that disables internal_linkage and always_inline than I am leaving always_inline on by default, which adversely affects all users.

In D50652#1197775, @rnk wrote:

I'd prefer not to do this, since internal_linkage generates smaller, more debuggable code by default. I think the symbol table size increase may be specific to MachO, and it may be possible to work around this by changing ld64 to pool strings for symbols by default. I don't know enough about MachO to say if this is possible.

I also think the symbol table size problem may be limited to "large" users of C++: people with 500+ object files using libc++ in a DSO. I'm more comfortable imposing a cost on these users to ask them to opt in to some setting that disables internal_linkage and always_inline than I am leaving always_inline on by default, which adversely affects all users.

One thing to keep in mind is that we do not have a solution that allows removing both internal_linkage and always_inline. It's either internal_linkage or always_inline, but you can't get rid of both until we fix some problems with extern template declarations and visibility attributes. What I can do, however, is reverse the default to use internal_linkage, and then have a temporary hook that allows Chromium to stay on always_inline. In the future, we'd remove that hook and the choice would be between internal_linkage and nothing at all.

rnk added a comment.Aug 13 2018, 1:19 PM

One thing to keep in mind is that we do not have a solution that allows removing both internal_linkage and always_inline. It's either internal_linkage or always_inline, but you can't get rid of both until we fix some problems with extern template declarations and visibility attributes. What I can do, however, is reverse the default to use internal_linkage, and then have a temporary hook that allows Chromium to stay on always_inline. In the future, we'd remove that hook and the choice would be between internal_linkage and nothing at all.

It's probably worth it to me to debug and understand that problem. Is there a good explanation of it?

Also, if a user could define _LIBCPP_ABI_UNSTABLE, does this problem still apply? Could we remove the problematic visibility attributes if the ABI is unstable?

In D50652#1197830, @rnk wrote:

One thing to keep in mind is that we do not have a solution that allows removing both internal_linkage and always_inline. It's either internal_linkage or always_inline, but you can't get rid of both until we fix some problems with extern template declarations and visibility attributes. What I can do, however, is reverse the default to use internal_linkage, and then have a temporary hook that allows Chromium to stay on always_inline. In the future, we'd remove that hook and the choice would be between internal_linkage and nothing at all.

It's probably worth it to me to debug and understand that problem. Is there a good explanation of it?

http://lists.llvm.org/pipermail/llvm-dev/2018-July/124549.html

This is understood, it just needs more work than we can reasonably expect to get into LLVM 7.

Also, if a user could define _LIBCPP_ABI_UNSTABLE, does this problem still apply? Could we remove the problematic visibility attributes if the ABI is unstable?

I guess in that case, we could simply remove _all_ visibility attributes (and linkage attributes). libc++ may end up exporting more symbols than we want, but defining _LIBCPP_ABI_UNSTABLE would explicitly state that we don't care about that. Doing this is a possibility, but don't think this is the way we want to solve the problem at hand.

hans added a comment.Aug 14 2018, 6:12 AM
In D50652#1197775, @rnk wrote:

I'd prefer not to do this, since internal_linkage generates smaller, more debuggable code by default.

IIUC, this is what we had before though, so it's a conservative "revert to safety" approach until a better solution can be found.

I think the symbol table size increase may be specific to MachO, and it may be possible to work around this by changing ld64 to pool strings for symbols by default. I don't know enough about MachO to say if this is possible.

I looked some more at my local build. The sum of object file sizes didn't increase very much, the big growth is for the executable, and there it's mostly the string table as your analysis showed.

As far as I understand, string pooling should work fine for Mach-O; LLVM does it for object files already. And yes, ld64 doesn't do it. I looked at this symbol:

__ZNSt3__1eqERKNS_21__tree_const_iteratorIN7testing11ExpectationEPNS_11__tree_nodeIS2_PvEElEES9_

If I understand correctly that function would previously have been inlined everywhere, but now it occurs 20+ times in the symbol table of base_unittests, and yes it's repeated 20+ times in the string table too :-/

I also think the symbol table size problem may be limited to "large" users of C++: people with 500+ object files using libc++ in a DSO. I'm more comfortable imposing a cost on these users to ask them to opt in to some setting that disables internal_linkage and always_inline than I am leaving always_inline on by default, which adversely affects all users.

Sure, if there's a macro we can use to opt in to get the old inlining behaviour, or maybe even better getting linkonce_odr linkage, I'd be a happy camper. But the cost of just growing the executables this much seems like a real problem for our build infrastructure.

hans added a comment.EditedAug 14 2018, 6:37 AM

Oh, or could we do

-D_LIBCPP_HIDE_FROM_ABI=

and just get regular odr linkage for these functions?

Oh, or could we do

-D_LIBCPP_HIDE_FROM_ABI=

and just get regular odr linkage for these functions?

No, you do need to use _LIBCPP_HIDDEN _LIBCPP_ALWAYS_INLINE because of the issue described in http://lists.llvm.org/pipermail/llvm-dev/2018-July/124549.html. But yeah, Chromium could use this workaround.

Oh, or could we do

-D_LIBCPP_HIDE_FROM_ABI=

and just get regular odr linkage for these functions?

No, you do need to use _LIBCPP_HIDDEN _LIBCPP_ALWAYS_INLINE because of the issue described in http://lists.llvm.org/pipermail/llvm-dev/2018-July/124549.html. But yeah, Chromium could use this workaround.

Actually, scratch that, it does work. One can either use -D_LIBCPP_HIDE_FROM_ABI=_LIBCPP_HIDDEN _LIBCPP_ALWAYS_INLINE to restore the old behavior, or -D_LIBCPP_HIDE_FROM_ABI= to get odr linkage.

ldionne updated this revision to Diff 160680.Aug 14 2018, 1:39 PM

Add a way to change the default behavior of _LIBCPP_HIDE_FROM_ABI at build time. Also, rename the macro to _LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE to align with other ABI-related macros.

bcain added a subscriber: bcain.Aug 15 2018, 8:33 AM
dexonsmith added inline comments.Aug 15 2018, 9:25 AM
libcxx/include/__config
798–804 ↗(On Diff #160680)

This doesn't leave a user-friendly way of opting out of internal linkage if a vendor decides to turn it on by default. I suggest adding a layer of indirection:

// Elsewhere.
#cmakedefine _LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE_BY_DEFAULT

// Here.
#ifndef _LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE
#  ifdef _LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE_BY_DEFAULT
#    define _LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE 1
#  else
#    define _LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE 0
#  endif
#endif

#ifndef _LIBCPP_HIDE_FROM_ABI
#  if _LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE
#    define _LIBCPP_HIDE_FROM_ABI _LIBCPP_HIDDEN _LIBCPP_INTERNAL_LINKAGE
#  else
#    define _LIBCPP_HIDE_FROM_ABI _LIBCPP_HIDDEN _LIBCPP_ALWAYS_INLINE
#  endif
#endif

Then users can set the behaviour with -D_LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE=1 or -D_LIBCPP_ABI_HIDDEN_USE_INTERNAL_LINKAGE=0, overriding the vendor default.

ldionne updated this revision to Diff 160873.Aug 15 2018, 11:40 AM

Allow picking a custom default behavior for vendors, per Duncan's comment.

Also, revert to a better name for the macro.

ldionne marked an inline comment as done.Aug 15 2018, 11:40 AM
thakis added a subscriber: thakis.Aug 15 2018, 12:42 PM

I haven't read all the messages in these threads, forgive me if someone asked this already. It's a bit weird to me that we have to override this behavior in Chromium while the default is different. Why isn't the executable size blowup we see in chromium a problem for everyone else too? Is the plan to fix ld64's string pooling at the same time as rolling this change out, and this is just a workaround for people who have head libc++ but not head ld64?

I haven't read all the messages in these threads, forgive me if someone asked this already. It's a bit weird to me that we have to override this behavior in Chromium while the default is different. Why isn't the executable size blowup we see in chromium a problem for everyone else too? Is the plan to fix ld64's string pooling at the same time as rolling this change out, and this is just a workaround for people who have head libc++ but not head ld64?

It's the other way around -- the default behavior introduced in this patch is the one before the internal_linkage change. If you want to opt-in, you can define _LIBCPP_HIDE_FROM_ABI_PER_TU.

As a separate goal, we will (in the future) make things better for the default case, i.e. we will get rid of __always_inline__ too and allow ODR-deduplication.

Does that make sense?

dexonsmith accepted this revision.Aug 15 2018, 2:39 PM

LGTM if thakis is happy.

This revision is now accepted and ready to land.Aug 15 2018, 2:39 PM

If that's possible at all, I would like for the Chromium people to build with this patch applied. The expectation is that we'll cherry-pick this patch onto LLVM 7, and it would suck if that did not solve Chromium's problem for some stupid reason.

hans accepted this revision.Aug 16 2018, 1:21 AM

Oh, or could we do

-D_LIBCPP_HIDE_FROM_ABI=

and just get regular odr linkage for these functions?

No, you do need to use _LIBCPP_HIDDEN _LIBCPP_ALWAYS_INLINE because of the issue described in http://lists.llvm.org/pipermail/llvm-dev/2018-July/124549.html. But yeah, Chromium could use this workaround.

Actually, scratch that, it does work. One can either use -D_LIBCPP_HIDE_FROM_ABI=_LIBCPP_HIDDEN _LIBCPP_ALWAYS_INLINE to restore the old behavior, or -D_LIBCPP_HIDE_FROM_ABI= to get odr linkage.

I tried -D_LIBCPP_HIDE_FROM_ABI= but got lots of link errors (see https://bugs.chromium.org/p/chromium/issues/detail?id=872926#c21).

-D_LIBCPP_HIDE_FROM_ABI=_LIBCPP_HIDDEN _LIBCPP_ALWAYS_INLINE works nicely though.

I also tried applying your patch and verified that works for Chromium. If I understand correctly, without setting LIBCXX_HIDE_FROM_ABI_PER_TU_BY_DEFAULT, it restores the old behaviour of force inlining.

I think this makes sense for Chromium since it allows us to build without any special flags until we can get ODR linkage, and I think it also makes sense for the 7.0 branch to prevent regressing binary sizes for users of the release.

lgtm

(I defer to Hans)

This revision was automatically updated to reflect the committed changes.
hans added a comment.Aug 16 2018, 7:55 AM

Thanks! Merged to 7.0 in r339882.

Thanks! Merged to 7.0 in r339882.

Now that this has been done, I guess we need to document somewhere in the release notes that the default contract given by libc++ is changing in LLVM 7, right?

Yes. But in practice nothing really changed though, right? It's just
going forward that things might break:

"This is a change in contract but not a change in behavior, since the
current implementation still allows linking TUs built with different
libc++ versions together."