This is an archive of the discontinued LLVM Phabricator instance.

Rework interface for bitset-using features to use a notion of LTO visibility.
ClosedPublic

Authored by pcc on Mar 30 2016, 5:14 PM.

Details

Summary

Bitsets, and the compiler features they rely on (vtable opt, CFI),
only have visibility within the LTO'd part of the linkage unit. Therefore,
only enable these features for classes with hidden LTO visibility. This
notion is based on object file visibility or (on Windows)
dllimport/dllexport attributes.

We provide the [[clang::lto_visibility_default]] attribute to override the
compiler's LTO visibility inference in cases where the class is defined
in the non-LTO'd part of the linkage unit, or where the ABI supports
calling classes derived from abstract base classes with hidden visibility
in other linkage units (e.g. COM on Windows).

If the cross-DSO CFI mode is enabled, bitset checks are emitted even for
classes with default LTO visibility, as that mode uses a separate mechanism
to cause bitsets to be exported.

This mechanism replaces the whole-program-vtables blacklist, so remove the
-fwhole-program-vtables-blacklist flag.

Because __declspec(uuid()) now implies [[clang::lto_visibility_default]], the
support for the special attr:uuid blacklist entry is removed.

Diff Detail

Repository
rL LLVM

Event Timeline

pcc updated this revision to Diff 52160.Mar 30 2016, 5:14 PM
pcc retitled this revision from to Rework interface for bitset-using features to use a notion of class scope..
pcc updated this object.
pcc added subscribers: mehdi_amini, pete.
eugenis added inline comments.Mar 31 2016, 12:40 PM
docs/ClassScope.rst
23 ↗(On Diff #52160)

Maybe call it "default"? Attrs sounds too specific. Basically this setting lets clang figure out scope based on the source code.

28 ↗(On Diff #52160)

hidden visibility = linkage-unit scope, not global scope.

docs/ControlFlowIntegrity.rst
271 ↗(On Diff #52160)

Do we emit a fast non-cross-DSO check for classes with linkage-unit scope?

rsmith added inline comments.Mar 31 2016, 1:07 PM
docs/ClassScope.rst
2 ↗(On Diff #52160)

Can you use some word other than "scope" here? "Class scope" is already a term of art in C++, meaning something completely different. I think what you're referring to is exactly the visibility of the class (in the ELF sense).

13 ↗(On Diff #52160)

classes -> a class

Otherwise this reads like you're saying at most one linkage unit per program can declare classes with linkage-unit scope.

14 ↗(On Diff #52160)

Classes -> A class

pcc added inline comments.Mar 31 2016, 4:26 PM
docs/ClassScope.rst
2 ↗(On Diff #52160)

Yes, this is pretty much visibility. I wanted to avoid using the term "visibility" because I'm introducing flags and attributes which can make scope mean something different to object file visibility, so I wanted to avoid the overload to avoid confusion.

Maybe the overloading would help with understanding though if I add a qualifying adjective. This is all about whether all derived classes are visible, so maybe the right term is something like "derived visibility"?

13 ↗(On Diff #52160)

Will do

14 ↗(On Diff #52160)

Will do

23 ↗(On Diff #52160)

-fdefault-class-scope=default? Sounds a little stuttery. Although attrs isn't entirely accurate (it's also affected by flags), it's close enough I reckon.

28 ↗(On Diff #52160)

Yes, I somehow completely screwed up the sense of these, even in the Windows part. Will fix.

docs/ControlFlowIntegrity.rst
271 ↗(On Diff #52160)

We do not. I reckon we can start doing that in a follow up, this change is large enough as it is.

rsmith added inline comments.Mar 31 2016, 5:01 PM
docs/ClassScope.rst
2 ↗(On Diff #52160)

We already have attributes that can set the visibility of a class (which in turn affects the visibility of the vtable etc.) In what way is that different from what you're proposing? Is this a valuable difference, given the complexity of having two similar-but-different ways of describing the cross-DSO visibility of a class?

pcc added inline comments.Mar 31 2016, 5:30 PM
docs/ClassScope.rst
2 ↗(On Diff #52160)

Yes, ideally I'd like to just use visibility for this. There were a couple of things that motivated a design that separated the concepts, though:

  1. The part of the executable or DSO that we're LTOing may be a subset of the whole executable or DSO. If the user has prebuilt object files that they need to link into their executable or DSO, we need to exclude those classes from our analysis. One example of this would be the standard library. On most platforms this is marked up with default visibility attributes, so we're fine, but on Windows the platform ships an un-marked up standard library as a static library, and allows users to link to it. The idea is that Windows users who use the static standard library would be able to say
namespace [[clang::global_scope]] std {}

in a -included header and get global scope on the standard library.

  1. I wanted -fsanitize=cfi* to turn on CFI checks on classes by default. It would be surprising to users if that flag on its own did not enable CFI on classes declared in the normal way, and it could easily lead to users shipping unprotected binaries. I suppose an alternative thing we could do would be to make -fsanitize=cfi* imply -fvisibility=hidden. Then if things break, they would at least be likely to break loudly.
rsmith added inline comments.Mar 31 2016, 5:42 PM
docs/ClassScope.rst
2 ↗(On Diff #52160)

I'm really not happy about having a third granularity for entity uniqueness (LTO granularity) between DSO granularity (visibility) and module granularity (linkage). But if we need that to accurately represent the state of the world, then perhaps there's no way to escape it (although if that is the intent, then the documentation here should be talking about something like LTO units rather than linkage units).

  1. Would it be inappropriate for entities in namespace std on Windows to be given default visibility when people do this?
  2. I don't think it makes sense for the -fsanitize=cfi* flag to enable "types can only be declared within their own LTO unit by default" semantics, whether those semantics are related to devirtualization, CFI, or symbol visibility. Sanitizer flags are supposed to turn on checks for the ambient configuration, not change what the code means; I'd prefer to say that it's an error if you enable -fsanitize=cfi* and don't *also* enable fvisibility=hidden (or a hypothetical -fclass-visibility=hidden).
pcc added inline comments.Mar 31 2016, 6:22 PM
docs/ClassScope.rst
2 ↗(On Diff #52160)
  1. Deciding to treat std specially based on flags like /MT makes sense to me, but visibility isn't a thing on Windows, and dll*port affects the ABI, so we probably need the in-between state there.
  2. Okay, I prefer that over what I proposed. I suppose users could specify -fvisibility=default if that's really what they want.

Let's say we simplify this down to no new driver flags and one new attribute that disables vtable opt and CFI on hidden visibility and non-dll*port classes (no namespace attributes). Let me confirm that that's enough for the use cases I care about.

pcc updated this revision to Diff 52730.Apr 5 2016, 1:11 PM
  • Rewrite docs
pcc added a comment.Apr 5 2016, 1:12 PM

I have updated the documentation for the new design; PTAL before I proceed with implementation.

pcc updated this revision to Diff 52732.Apr 5 2016, 1:19 PM
  • Update command line flag docs
pcc updated this revision to Diff 52759.Apr 5 2016, 8:03 PM
  • New implementation
pcc retitled this revision from Rework interface for bitset-using features to use a notion of class scope. to Rework interface for bitset-using features to use a notion of LTO visibility..Apr 6 2016, 3:52 PM
pcc updated this object.
rsmith edited edge metadata.Apr 19 2016, 1:36 PM

Generally-speaking, I think this is looking really good. I think we need to be careful and precise in how we document this new visibility notion; most of my comments are on that.

I wonder if we'd benefit from a diagram in the documentation showing the nesting levels between object files (translated translation units in C / C++ parlance), LTO units (or whatever you want to call them), and DSOs (or linkage units or whatever you want to call executable / .so / .dylib / .dll files), and precise definitions of the terms we use for each. The two of us seem to be using different terms for the same entities, so a single set of terms we define and use consistently -- at least within Clang's documentation -- would be great.

docs/LTOVisibility.rst
5–8 ↗(On Diff #52759)

Please start with a definition of this that says what it means separate from how we're going to use it. Something like:

"""
LTO visibility is a property of an entity that specifies whether it can be referenced from outside the current LTO unit. An LTO unit is one or more translation units that are linked together using link-time optimization; in the case where LTO is not being used, each translation unit is a separate LTO unit.

The LTO visibility of a class is used by the compiler to determine whether certain virtual function call optimizations and control flow integrity features can be applied to the class. These features use whole-program information, so they require the entire class hierarchy to be visible in order to work correctly.
"""

10–12 ↗(On Diff #52759)

"[...] or to declare such a class in multiple translation units not built with LTO."

11 ↗(On Diff #52759)

What do you mean by linkage units here? Do you mean DSOs? (In which case this seems imprecise, since I can link multiple LTO units into a single DSO.)

28–30 ↗(On Diff #52759)

The LTO'd files may also need the attribute if the classes are referenced by the non-LTO'd files.

37–43 ↗(On Diff #52759)

"default" is a terrible name for this. It was a mistake for the GNU attribute to use "default" to mean "public"; let's not replicate this mistake for LTO visibility.

39–40 ↗(On Diff #52759)

It seems to me that __declspec(uuid(...)) should notionally imply that the class is visible to other DSOs, not just that it's visible outside the LTO unit within its DSO. I'm not sure we have a reasonable way to model that, though (dllexport and default visibility both imply other things that we don't really mean here).

40 ↗(On Diff #52759)

Don't say the attributes is implied here; instead say the semantics are implied -- "classes with the __declspec(uuid()) attribute receive public LTO visibility" or similar?

docs/UsersManual.rst
1059–1060 ↗(On Diff #52759)

I would think internal linkage should imply hidden LTO visibility.

include/clang/Driver/CC1Options.td
287 ↗(On Diff #52759)

Looks like stdext is an MS STL thing; can we limit the special-casing to just MS targets?

lib/CodeGen/CGVTables.cpp
925–926 ↗(On Diff #52759)

The namespace std declaration might be in an extern "C" or extern "C++" context; you should check the redecl context of DC here (cf DeclContext::isStdNamespace).

936–939 ↗(On Diff #52759)

Maybe perform these checks before the (likely more expensive) check for an enclosing std namespace?

pcc updated this revision to Diff 54439.Apr 20 2016, 4:48 PM
pcc marked 2 inline comments as done.
pcc edited edge metadata.
  • Address review comments
pcc added a comment.Apr 20 2016, 4:48 PM

Thanks for the comments. I've also taken another pass over the docs and changed them to specify that classes defined in translation units compiled without LTO receive public LTO visibility. This seems more precise to me, but please let me know what you think.

I have also added an ASCII art diagram with some examples.

I have also updated the implementation to match the documentation. We now correctly implement the case where only some translation units are compiled with CFI or vtable opt by emitting bitsets for translation units where only LTO is enabled. I considered the alternative of adding another type of "unit", but felt that that would add too much complexity.

docs/LTOVisibility.rst
5–8 ↗(On Diff #52759)

Done, except for:

in the case where LTO is not being used, each translation unit is a separate LTO unit

This part is inaccurate; the compiler features we're talking about require -flto at the moment. If we do decide to allow this, that should probably be a separate discussion.

10–12 ↗(On Diff #52759)

I don't think we need this part as long as we have the LTO requirement.

11 ↗(On Diff #52759)

I mean the contents of a DSO or executable. I have added a definition. I thought this term was defined in the standard, but I guess I must have picked it up from a draft proposal [1] or somewhere else.

I have updated the documentation to clarify that a linkage unit may have only a single LTO unit.

[1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2003/n1428.html

28–30 ↗(On Diff #52759)

The attribute can only have an effect on generated code if a virtual call is made using the class. Since virtual functions can only be called on classes with definitions, I think this restriction is sufficient. In any case, I've rewritten it to try to clarify it.

37–43 ↗(On Diff #52759)

Agreed, done.

39–40 ↗(On Diff #52759)

It seems to me that __declspec(uuid(...)) should notionally imply that the class is visible to other DSOs, not just that it's visible outside the LTO unit within its DSO

Yes. I have changed the docs to state that this is exactly what public LTO visibility means ("A class with public LTO visibility may be defined in multiple linkage units...").

Don't say the attributes is implied here; instead say the semantics are implied

Done.

docs/UsersManual.rst
1059–1060 ↗(On Diff #52759)

Agreed. I've updated LTOVisibility.rst.

include/clang/Driver/CC1Options.td
287 ↗(On Diff #52759)

It is already so limited. This is a -cc1 only flag and is enabled only if the clang-cl driver is being used.

rsmith accepted this revision.Apr 27 2016, 1:25 PM
rsmith edited edge metadata.
rsmith added inline comments.
docs/ControlFlowIntegrity.rst
268–269 ↗(On Diff #54439)

Maybe reverse the sense of this to avoid the double negative:

Normally, CFI checks will only be performed for classes that have hidden LTO visibility.

or similar?

This revision is now accepted and ready to land.Apr 27 2016, 1:25 PM
This revision was automatically updated to reflect the committed changes.
cfe/trunk/runtime/CMakeLists.txt