This is an archive of the discontinued LLVM Phabricator instance.

Merge some of the PCH object support with modular codegen
ClosedPublic

Authored by dblaikie on Jul 12 2020, 7:28 PM.

Details

Summary

I was trying to pick this up a bit when reviewing D48426 (& perhaps D69778) - in any case, looks like D48426 added a module level flag that might not be needed.

The D48426 implementation worked by setting a module level flag, then code generating contents from the PCH a special case in ASTContext::DeclMustBeEmitted would be used to delay emitting the definition of these functions if they came from a Module with this flag.

This strategy is similar to the one initially implemented for modular codegen that was removed in D29901 in favor of the modular decls list and a bit on each decl to specify whether it's homed to a module.

One major difference between PCH object support and modular code generation, other than the specific list of decls that are homed, is the compilation model: MSVC PCH modules are built into the object file for some other source file (when compiling that source file /Yc is specified to say "this compilation is where the PCH is homed"), whereas modular code generation invokes a separate compilation for the PCH alone. So the current modular code generation test of to decide if a decl should be emitted "is the module where this decl is serialized the current main file" has to be extended (as Lubos did in D69778) to also test the command line flag -building-pch-with-obj.

Otherwise the whole thing is basically streamlined down to the modular code generation path.

This even offers one extra material improvement compared to the existing divergent implementation: Homed functions are not emitted into object files that use the pch. Instead at -O0 they are not emitted into the IR at all, and at -O1 they are emitted using available_externally (existing functionality implemented for modular code generation). The pch-codegen test has been updated to reflect this new behavior.

[If possible: I'd love it if we could not have the extra MSVC-style way of accessing dllexport-pch-homing, and just do it the modular codegen way, but I understand that it might be a limitation of existing build systems. @hans / @thakis: Do either of you know if it'd be practical to move to something more similar to .pcm handling, where the pch itself is passed to the compilation, rather than homed as a side effect of compiling some other source file?]

Diff Detail

Event Timeline

dblaikie created this revision.Jul 12 2020, 7:28 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 12 2020, 7:28 PM
Herald added a subscriber: cfe-commits. · View Herald Transcript

The patch is incomplete, isn't it? It removes DeclIsFromPCHWithObjectFile(), but it's still called from ASTContext::DeclMustBeEmitted(). The description also mentions updating of the pch-codegen test, but that's not included.

But assuming this is intended to replace the D48426 functionality with modular codegen, I mentioned already in D69778 (starting with 'Mar 17 2020, 10:59 PM') that it's a question if this is possible. Modular codegen is a much bigger hammer than D48426 and it has also bigger possible problems: Possibly making compilation actually slower (even f63556d8b44b209e741833b0e6be3f8e72a1d91a mentions this), and possibly causing build problems because of referencing private symbols. D48426 is much smaller both in gains and in problems (=safer).

Do either of you know if it'd be practical to move to something more similar to .pcm handling, where the pch itself is passed to the compilation, rather than homed as a side effect of compiling some other source file?

Do you mean dropping compatibility with 'cl.exe /Yc'?

hans added a comment.Jul 13 2020, 5:35 AM

Do either of you know if it'd be practical to move to something more similar to .pcm handling, where the pch itself is passed to the compilation, rather than homed as a side effect of compiling some other source file?

Do you mean dropping compatibility with 'cl.exe /Yc'?

I don't think we can change the interface, because we want cl.exe compatibility.

dblaikie updated this revision to Diff 277449.Jul 13 2020, 8:58 AM

Include all the commits

The patch is incomplete, isn't it? It removes DeclIsFromPCHWithObjectFile(), but it's still called from ASTContext::DeclMustBeEmitted(). The description also mentions updating of the pch-codegen test, but that's not included.

Ah, right you are - arc diff only took the last commit on this branch. I've updated it/used arc correctly to get the whole branch worth of changes.

But assuming this is intended to replace the D48426 functionality with modular codegen,

Not quite - it's intended to implement the D48426 functionality using an implementation strategy that is closer to modular code generation. Removing the need for the module flag, using the MODULAR_CODEGEN_DECLS list, etc. But only putting the dllexported entities in there (when using only building-pch-with-obj without -fmodules-codegen).

The intent is not to change the behavior of D48426, but to simplify the implementation a bit by reducing the divergence between the two features.

Do either of you know if it'd be practical to move to something more similar to .pcm handling, where the pch itself is passed to the compilation, rather than homed as a side effect of compiling some other source file?

Do you mean dropping compatibility with 'cl.exe /Yc'?

Yep, that was essentially what I was wondering - since you were proposing a build mode/flags that would not be /Yc compatible, adding an explicit pch build and pch object build step - was wondering if the build system support for that was sufficiently available that it would be practical for users (like Chromium) to migrate to that kind of model and no longer need /Yc compatibility. That would further reduce some of the divergence between the two features (then it'd come down to only a "what entities do we want to home" question - the mechanics of homing them, building their object file, etc, would all be the same).

Do either of you know if it'd be practical to move to something more similar to .pcm handling, where the pch itself is passed to the compilation, rather than homed as a side effect of compiling some other source file?

Do you mean dropping compatibility with 'cl.exe /Yc'?

I don't think we can change the interface, because we want cl.exe compatibility.

Fair enough - I guess the only other way to get the cl.exe behavior from the Clang driver using modular codegen-type functionality would be to use 4 steps:

  • build PCH (with dllexport modular codegen)
  • build PCH object file using PCH
  • build source object using PCH
  • ld -r the two objects together

But I guess the added driver complexity isn't worth the simplification inside Clang proper, probably. I guess we already have the "build PCH then use PCH to build object" part so /maybe/ it'd be feasible to extend it in this way.

Not quite - it's intended to implement the D48426 functionality using an implementation strategy that is closer to modular code generation. Removing the need for the module flag, using the MODULAR_CODEGEN_DECLS list, etc. But only putting the dllexported entities in there (when using only building-pch-with-obj without -fmodules-codegen).

I see. I still suggest testing this on a real codebase for the reasons outlined above.

Yep, that was essentially what I was wondering - since you were proposing a build mode/flags that would not be /Yc compatible, adding an explicit pch build and pch object build step - was wondering if the build system support for that was sufficiently available that it would be practical for users (like Chromium) to migrate to that kind of model and no longer need /Yc compatibility.

Not that I know for sure, but I personally doubt there's build system support for D83716. The only build system I've tried is LibreOffice's custom gbuild, where I've added the support for -fpch-codegen/debuginfo myself, and even there's I'm going to stick with -building-pch-with-obj, since it's simpler to compile yet another .cxx file with an extra flag rather than have a build step with .pch as yet another kind of input file. But /Yc is presumably supported by pretty much anything on Windows.

@hans - could you perhaps give me a quick summary of commands I could use to test this feature in Chromium (or anything else you might suggest) on a Linux box? I don't have a Windows machine, or any projects that use PCH. (or if you'd be willing to test this, that'd be great - see if the PCH object file has the same symbols before/after this patch)

hans added a comment.Jul 20 2020, 5:50 AM

@hans - could you perhaps give me a quick summary of commands I could use to test this feature in Chromium (or anything else you might suggest) on a Linux box? I don't have a Windows machine, or any projects that use PCH. (or if you'd be willing to test this, that'd be great - see if the PCH object file has the same symbols before/after this patch)

I tried to do this today, and then realized Chromium's PCH .obj files are basically empty these days. Back when I did D48426, dllexported inline functions were emitted in the pch .obj file, but then Chromium started using /Zc:dllexportInlines- which means we don't dllexport inlines anymore, and even if I remove that flag we've moved to only including libc++ headers in our PCH, and nothing is dllexport there.

After applying and building with your patch, the PCH .obj files look the same.

I also verified that building one of Chromium's test targets succeeds with this patch.

@hans - could you perhaps give me a quick summary of commands I could use to test this feature in Chromium (or anything else you might suggest) on a Linux box? I don't have a Windows machine, or any projects that use PCH. (or if you'd be willing to test this, that'd be great - see if the PCH object file has the same symbols before/after this patch)

I tried to do this today, and then realized Chromium's PCH .obj files are basically empty these days. Back when I did D48426, dllexported inline functions were emitted in the pch .obj file, but then Chromium started using /Zc:dllexportInlines- which means we don't dllexport inlines anymore, and even if I remove that flag we've moved to only including libc++ headers in our PCH, and nothing is dllexport there.

After applying and building with your patch, the PCH .obj files look the same.

I also verified that building one of Chromium's test targets succeeds with this patch.

Thanks for checking! So, in summary: This seems to work fine for Chromium, but Chromium isn't really exercising this functionality (only insofar as the functionality is enabled, but essentially a no-op)?

@llunak Would you be able to test this on anything you've got?

@llunak Would you be able to test this on anything you've got?

No, but thinking more about this, I think dllexport specifically voids the possible problems I listed. If I'm getting it right, dllexport is used only for code in the current library, so codegen won't create code referencing private symbols in other libraries, and dllexport means the code will be emitted anyway, so no unnecessary code generating. So I'm actually fine with the patch as it is.

hans added a comment.Jul 21 2020, 7:03 AM

Thanks for checking! So, in summary: This seems to work fine for Chromium, but Chromium isn't really exercising this functionality (only insofar as the functionality is enabled, but essentially a no-op)?

Yup.

@hans @llunak - sounds like you're both fine with this? Either of you mind to give it a formal approval, if that's the case?

hans accepted this revision.Jul 22 2020, 6:49 AM
This revision is now accepted and ready to land.Jul 22 2020, 6:49 AM
This revision was automatically updated to reflect the committed changes.