This is an archive of the discontinued LLVM Phabricator instance.

Allow searching for prebuilt implicit modules.
ClosedPublic

Authored by arames on Oct 15 2019, 10:02 AM.

Details

Summary

The behavior is controlled by the -fprebuilt-implicit-modules option, and
allows searching for implicit modules in the prebuilt module cache paths.

The current command-line options for prebuilt modules do not allow to easily
maintain and use multiple versions of modules. Both the producer and users of
prebuilt modules are required to know the relationships between compilation
options and module file paths. Using a particular version of a prebuilt module
requires passing a particular option on the command line (e.g.
-fmodule-file=[<name>=]<file> or -fprebuilt-module-path=<directory>).

However the compiler already knows how to distinguish and automatically locate
implicit modules. Hence this proposal to introduce the
-fprebuilt-implicit-modules option. When set, it enables searching for
implicit modules in the prebuilt module paths (specified via
-fprebuilt-module-path). To not modify existing behavior, this search takes
place after the standard search for prebuilt modules. If not

Here is a workflow illustrating how both the producer and consumer of prebuilt
modules would need to know what versions of prebuilt modules are available and
where they are located.

clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules_v1 <config 1 options>
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules_v2 <config 2 options>
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules_v3 <config 3 options>

clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap -fprebuilt-module-path=prebuilt_modules_v1 <config 1 options>
clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap <non-prebuilt config options>

With prebuilt implicit modules, the producer can generate prebuilt modules as
usual, all in the same output directory. The same mechanisms as for implicit
modules take care of incorporating hashes in the path to distinguish between
module versions.

Note that we do not specify the output module filename, so -o implicit modules are generated in the cache path prebuilt_modules.

clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules <config 1 options>
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules <config 2 options>
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules <config 3 options>

The user can now simply enable prebuilt implicit modules and point to the
prebuilt modules cache. No need to "parse" command-line options to decide
what prebuilt modules (paths) to use.

clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap -fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules <config 1 options>
clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap -fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules <non-prebuilt config options>

This is for example particularly useful in a use-case where compilation is
expensive, and the configurations expected to be used are predictable, but not
controlled by the producer of prebuilt modules. Modules for the set of
predictable configurations can be prebuilt, and using them does not require
"parsing" the configuration (command-line options).

Diff Detail

Event Timeline

arames created this revision.Oct 15 2019, 10:02 AM
Herald added a project: Restricted Project. · View Herald TranscriptOct 15 2019, 10:02 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript
arames edited the summary of this revision. (Show Details)Oct 15 2019, 10:03 AM
arames added reviewers: bruno, rsmith.
arames updated this revision to Diff 228267.Nov 7 2019, 9:39 AM

Rebase.

Ping.
Still looking for someone to take a look. Happy to answer any questions.

arames updated this revision to Diff 264096.May 14 2020, 2:38 PM

Rebase on top of tree.

arames updated this revision to Diff 279263.Jul 20 2020, 8:39 AM

Rebase on top-of-tree.

Bigcheese added inline comments.Jul 30 2020, 1:31 PM
clang/docs/Modules.rst
231
295

I also think it's important to point out here that disabling the module hash means you are responsible for making sure the modules are compatible. How to do this is mentioned below, but it's never stated why you would want to do this.

clang/test/Modules/prebuilt-implicit-modules.m
10

Is this actually supposed to be module_e? I would expect this to be verifying that it didn't add module_a to %t1.

arames marked 2 inline comments as done.Jul 31 2020, 9:56 AM
arames added inline comments.
clang/docs/Modules.rst
295

Fixed the typo.

Good point.
I am refactoring the examples to make everything clearer.

303

Fixed a few other typos in the examples as well.

clang/test/Modules/prebuilt-implicit-modules.m
10

That was a typo. Fixed.

arames updated this revision to Diff 282335.Jul 31 2020, 3:29 PM
arames marked an inline comment as done.

Addressed review comments.

  • Fixed typos in the doc.
  • Added doc about module compatibility.
  • Cleaned and tested commands in the doc.
  • Reworked module hash code to not require -fmodules-cache-path when using -fprebuilt-implicit-modules.
arames updated this revision to Diff 282336.Jul 31 2020, 3:34 PM

Fix a typo.

arames added inline comments.Jul 31 2020, 3:36 PM
clang/include/clang/Lex/HeaderSearch.h
187

In the previous version of this patch, this value was derived from the stem of ModuleCachePath.
However we do not want to depend on ModuleCachePath being set to be able to retrieve the module hash. See the update in the test.

clang/test/Modules/prebuilt-implicit-modules.m
12

For this to be possible, the computation of the paths to (potential) implicit prebuilt modules cannot depend on the cache path being set.

This revision is now accepted and ready to land.Sep 23 2020, 4:30 PM
arames updated this revision to Diff 300780.Oct 26 2020, 1:38 PM

Rebased on ToT.

@arames, I see you've had a number of commits. I suggest you request commit access: https://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access

clang/docs/Modules.rst
358

Nit: please remove the stray space at the end of this line.

arames marked an inline comment as done.Nov 5 2020, 12:34 PM

Fixed the trailing whitespace.

I also just got commit rights, so I will commit it myself.

This revision was landed with ongoing or failed builds.Nov 5 2020, 1:11 PM
This revision was automatically updated to reflect the committed changes.
thakis added a subscriber: thakis.Nov 5 2020, 1:37 PM
thakis added inline comments.
clang/lib/Driver/ToolChains/Clang.cpp
3279

You need to use a fno_ flag as 2nd param if you want to use hasFlag, the default value is the third param: http://45.33.8.238/win/27354/step_4.txt

`
../../clang/lib/Driver/ToolChains/Clang.cpp(3382,63): error: conversion from 'bool' to 'llvm::opt::OptSpecifier' is ambiguous
    if (Args.hasFlag(options::OPT_fprebuilt_implicit_modules, false))
                                                              ^~~~~
../../llvm/include\llvm/Option/OptSpecifier.h(24,16): note: candidate constructor
  /*implicit*/ OptSpecifier(unsigned ID) : ID(ID) {}
               ^
../../llvm/include\llvm/Option/OptSpecifier.h(25,16): note: candidate constructor
  /*implicit*/ OptSpecifier(const Option *Opt);
               ^
../../llvm/include\llvm/Option/ArgList.h(295,47): note: passing argument to parameter 'Neg' here
  bool hasFlag(OptSpecifier Pos, OptSpecifier Neg, bool Default=true) const;
                                              ^
1 error generated.

This change broke the windows lldb bot:

http://lab.llvm.org:8011/#/builders/83/builds/570

Can you please fix this or revert?

arames added a comment.Nov 6 2020, 7:28 AM

This change broke the windows lldb bot:

http://lab.llvm.org:8011/#/builders/83/builds/570

Can you please fix this or revert?

Looking at this now.