[C++20][Modules][Driver][HU 1/N] Initial handling for -xc++-{system,user}-header.

[iains]

Summary

This adds file types and handling for three input types, representing a C++20
header unit source:

1. When provided with a complete pathname for the header.
2. For a header to be looked up (by the frontend) in the user search paths
3. For a header to be looked up in the system search paths.

We also add a pre-processed file type (although that is a single type, regardless
of the original input type).

These types may be specified with -xc++-{user,system,header-unit}-header xxxx.

These types allow us to disambiguate header unit jobs from PCH ones, and thus
we handle these differently from other header jobs in two ways:

1. The job construction is altered to build a C++20 header unit (rather than a PCH file, as would be the case for other headers).
2. When the type is "user" or "system" we defer checking for the file until the front end is run, since we need to look up the header in the relevant paths which are not known at this point.

[iains]

This is a set of 4 patches that adds the driver-side support for C++20 header modules.

We use the same user-facing options as the existing implementation in GCC (there seems no engineering reason to deviate from this and it makes things easier for users and build system alike).

1/4 - this patch introduces file types for header unit headers to be searched for in the user or system paths (or to be given an absolute pathname).
2/4 - adds the -fmoduke-header{.=} command line options
3/4 - is a user-convenience change that allows "foo.h" without complaining about C code used in C++ compilations (since we know that the user has already specified the intent).
4/4 - adds the -fdirectives-only command that produces pre-processor output which retains the macro definitions/undefs that will be present in the built header unit.

[ChuanqiXu]

LGTM. Although I don't like these option names, it is more important to keep consistent with GCC.

[iains]

note: I do not plan to fix the formatting issue in clang/lib/Driver/Types.cpp, since I am adding one line and the format change would mean ≈ 110 lines of changes.

[ChuanqiXu]

In D121588#3381872, @iains wrote:

note: I do not plan to fix the formatting issue in clang/lib/Driver/Types.cpp, since I am adding one line and the format change would mean ≈ 110 lines of changes.

It is odd. I thought all things could be handled by clang-format-diff. @MyDeveloperDay could you help to take a look?

[vsapsai]

Sorry, I'm pretty ignorant in this area but based on

The job construction is altered to build a C++20 header unit (rather than a PCH file, as would be the case for other headers).

I want to clarify. The goal is to allow mixing PCH and PCM files, right? I haven't double checked and maybe that have changed already but I think -x c++-header started building .pcm instead of .pch with -std=c++20. That's where my confusion about mixing PCH and PCM comes from.

clang/include/clang/Driver/Types.def

66–70

Sorry, it's not really related to your change but do you have a rule where "ii" should go? It's just we have both "mii" and "iim" and I want to make sure it should be "iih" and not "hii". I haven't tried to find a pattern here myself, asking you first.

clang/lib/Frontend/FrontendOptions.cpp

30–31

Given the other branches in this StringSwitch I would expect .Cases("iih", "iim", InputKind(Language::CXX).getPreprocessed()). Is there a reason not to do that (like differences between iih and iim) or is it accidental?

clang/test/Driver/cxx20-header-units-01.cpp

8

What should happen in case of inconsistencies like %clang++ -### -std=c++20 -xc++-system-header %S/Inputs/header-unit-01.hh? Or -xc++-system-header foo.h?

[iains]

In D121588#3384799, @vsapsai wrote:

Sorry, I'm pretty ignorant in this area but based on

The job construction is altered to build a C++20 header unit (rather than a PCH file, as would be the case for other headers).

I want to clarify. The goal is to allow mixing PCH and PCM files, right? I haven't double checked and maybe that have changed already but I think -x c++-header started building .pcm instead of .pch with -std=c++20. That's where my confusion about mixing PCH and PCM comes from.

From my understanding of the modules group objectives:

The (my at least understand of) objective is that when the user produces a C++20+ command line with no contradicting options, then the output would be standardised C++20 modules.

As of this moment, clang modules and C++20 modules have some divergence (I understand that the goal is to unify - but that will take some time, I expect).

If the user wants clang modules, then they should add "-fmodules" to the command line - which would switch off C++20 modules and produce a PCH header module as before 9perhaps built from several headers, where a C++20 HU only reflects one header),

Perhaps we cannot make this objective yet - but I can say that the current choices in this code do not produce any regressions in the testsuite - so that is some indication that it could be possible, right now.

There is a quite along discussion on this in https://reviews.llvm.org/D120540

clang/include/clang/Driver/Types.def

66–70

Sorry, it's not really related to your change but do you have a rule where "ii" should go? It's just we have both "mii" and "iim" and I want to make sure it should be "iih" and not "hii". I haven't tried to find a pattern here myself, asking you first.

My understanding is this:

ObjectiveC/C++ append "I" and "ii" (mi and mii)

C++ Modules-related code pre-pends "ii".

so that a pre-processed header unit ==> "iih"
and a pre-processed module ==> "iim".

clang/lib/Frontend/FrontendOptions.cpp

30–31

accidental on someone's part, I expect .. but pre-existing
(I have not [intentionally, at least] modified the behaviour of non-header unit module in this patch set)

clang/test/Driver/cxx20-header-units-01.cpp

8

If the user states that %S/Inputs/header-unit-01.hh is a system header, I do not think that the driver is in a position to argue?

-xc++-system-header foo.h again the user has made an explicit statement..

I suppose that we can always diagnose these things with warnings - but I do not think we can easily reject them (and we risk producing unhelpful output).

[iains]

clang/lib/Frontend/FrontendOptions.cpp

30–31

hmm.. sorry I misread .. I will update this to use Cases.

[iains]

adress review comments, adjust test case for windows compatibility.

[vsapsai]

Thanks for explaining the desired interplay between PCH and PCM. The problem I'm seeing is that it can be hard to find correct compiler flags to achieve the desired result (what is enabled and what is disabled). But that is out of scope for this change.

clang/include/clang/Driver/Types.def

66–70

Thanks for the explanation. Now I see it should be "iih" and not "hii". One could argue that modular-related stuff can be expressed by appending "m", like ".ii" -> ".iim". But I don't know what that would mean for PP_CXXHeaderUnit ("iihm"?) and I think "iih" is reasonable.

clang/test/Driver/cxx20-header-units-01.cpp

8

What is the intended use of -xc++-user-header and -xc++-system-header? Are these supposed to be commonly-used flags or as an escape hatch of some sort? Because if it is commonly-used, I have concerns about usability that forces users to track user and system headers correctly, and duplicates the information already encoded in header search paths.

Also I think it is out of scope but for usability it is important to know the consequences of using a wrong flag and how one can diagnose it. Have no idea what you've already planned in this area, just mentioning it because I'm scarred by inscrutable clang behavior.

[iains]

adjust testcase path separators

[iains]

clang/test/Driver/cxx20-header-units-01.cpp

8

These are "generally internal" marking to be specific about how files should be treated.

The usual user-facing commands would be -fmodule-header={user, system}

So the answer is no, they would not be commonly used - pretty much the same as -x c++ or so, only necessary if the file type is ambiguous to the driver in some way.

If the user wants a specific header to be searched for in the user or system paths, then it is reasonable that they have to indicate this. Actually, probably the largest benefit is that the user would not need to know where system headers were stored -- so when trying to build a header unit for, say, 'vector' it would greatly simplify the process,

I hope that the behaviour should not be in any way inscrutable - there are some things that the user (or build system) has to communicate to the driver, it cannot deduce them (and where to search for headers is in that category).

---

the consequences of using the wrong flag in this case would be a failure to find the header file (the diagnostic would be emitted by the FE rather than the driver, but that would be transparent to a regular user). The logic here is that using the lookup paths is not a job for the driver, because the FE already knows and has mechanisms for it.

[iains]

In D121588#3390956, @vsapsai wrote:

Thanks for explaining the desired interplay between PCH and PCM. The problem I'm seeing is that it can be hard to find correct compiler flags to achieve the desired result (what is enabled and what is disabled). But that is out of scope for this change.

IMHO, that is an unfortunate consequence of having ≈ 60+ module-related command line flags; we do need to rationalise this. This is being discussed elsewhere [D120540] (and probably it's better to leave that conversation in one place).

[vsapsai]

Thanks for your patience and for all the explanations, I have no other comments.

[iains]

rebased.

clang/include/clang/Driver/Types.def

 // C family input files to precompile.
 TYPE("c-header-cpp-output",      PP_CHeader,   INVALID,         "i",      phases::Precompile)
 TYPE("c-header",                 CHeader,      PP_CHeader,      "h",      phases::Preprocess, phases::Precompile)
 TYPE("cl-header",                CLHeader,     PP_CHeader,      "h",      phases::Preprocess, phases::Precompile)
 TYPE("objective-c-header-cpp-output", PP_ObjCHeader, INVALID,   "mi",     phases::Precompile)
 TYPE("objective-c-header",       ObjCHeader,   PP_ObjCHeader,   "h",      phases::Preprocess, phases::Precompile)
 TYPE("c++-header-cpp-output",    PP_CXXHeader, INVALID,         "ii",     phases::Precompile)
 TYPE("c++-header",               CXXHeader,    PP_CXXHeader,    "hh",     phases::Preprocess, phases::Precompile)
-TYPE("objective-c++-header-cpp-output", PP_ObjCXXHeader, INVALID, "mii",  phases::Precompile)
+TYPE("c++-header-unit-cpp-output", PP_CXXHeaderUnit,INVALID,    "iih",    phases::Precompile)
+TYPE("c++-header-unit-header",   CXXHUHeader,  PP_CXXHeaderUnit,"hh",     phases::Preprocess, phases::Precompile)
+TYPE("c++-system-header",        CXXSHeader,   PP_CXXHeaderUnit,"hh",     phases::Preprocess, phases::Precompile)
+TYPE("c++-user-header",          CXXUHeader,   PP_CXXHeaderUnit,"hh",     phases::Preprocess, phases::Precompile)
+TYPE("objective-c++-header-cpp-output", PP_ObjCXXHeader, INVALID,"mii",   phases::Precompile)
 TYPE("objective-c++-header",     ObjCXXHeader, PP_ObjCXXHeader, "h",      phases::Preprocess, phases::Precompile)
 TYPE("c++-module",               CXXModule,    PP_CXXModule,    "cppm",   phases::Preprocess, phases::Precompile, phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("c++-module-cpp-output",    PP_CXXModule, INVALID,         "iim",    phases::Precompile, phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 
 // Other languages.
 TYPE("ada",                      Ada,          INVALID,         nullptr,  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("assembler",                PP_Asm,       INVALID,         "s",      phases::Assemble, phases::Link)
 TYPE("assembler-with-cpp",       Asm,          PP_Asm,          "S",      phases::Preprocess, phases::Assemble, phases::Link)
 TYPE("f95",                      PP_Fortran,   INVALID,         nullptr,  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("f95-cpp-input",            Fortran,      PP_Fortran,      nullptr,  phases::Preprocess, phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("java",                     Java,         INVALID,         nullptr,  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 
 // LLVM IR/LTO types. We define separate types for IR and LTO because LTO
 // outputs should use the standard suffixes.
 TYPE("ir",                       LLVM_IR,      INVALID,         "ll",     phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("ir",                       LLVM_BC,      INVALID,         "bc",     phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("lto-ir",                   LTO_IR,       INVALID,         "s",      phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("lto-bc",                   LTO_BC,       INVALID,         "o",      phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 
 // Misc.
 TYPE("ast",                      AST,          INVALID,         "ast",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("ifs",                      IFS,          INVALID,         "ifs",    phases::IfsMerge)
 TYPE("ifs-cpp",                  IFS_CPP,      INVALID,         "ifs",    phases::Compile, phases::IfsMerge)
 TYPE("pcm",                      ModuleFile,   INVALID,         "pcm",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
+TYPE("header-unit",              HeaderUnit,   INVALID,         "pcm",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("plist",                    Plist,        INVALID,         "plist",  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("rewritten-objc",           RewrittenObjC,INVALID,         "cpp",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("rewritten-legacy-objc",    RewrittenLegacyObjC,INVALID,   "cpp",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("remap",                    Remap,        INVALID,         "remap",  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("precompiled-header",       PCH,          INVALID,         "gch",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("object",                   Object,       INVALID,         "o",      phases::Link)
 TYPE("treelang",                 Treelang,     INVALID,         nullptr,  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("image",                    Image,        INVALID,         "out",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("dSYM",                     dSYM,         INVALID,         "dSYM",   phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("dependencies",             Dependencies, INVALID,         "d",      phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("cuda-fatbin",              CUDA_FATBIN,  INVALID,         "fatbin", phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("hip-fatbin",               HIP_FATBIN,   INVALID,         "hipfb",  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("api-information",          API_INFO,     INVALID,         "json",   phases::Compile)
 TYPE("none",                     Nothing,      INVALID,         nullptr,  phases::Compile, phases::Backend, phases::Assemble, phases::Link)

clang/lib/Driver/Driver.cpp

bool Driver::DiagnoseInputExistence(const DerivedArgList &Args, StringRef Value,
                                     types::ID Ty, bool TypoCorrect) const {
   if (!getCheckInputsExist())
     return true;
 
   // stdin always exists.
   if (Value == "-")
     return true;
 
+  // If it's a header to be found in the system or user search path, then defer
+  // complaints about its absence until those searches can be done.
+  if (Ty == types::TY_CXXSHeader || Ty == types::TY_CXXUHeader)
+    return true;
+
   if (getVFS().exists(Value))
     return true;
 
   if (TypoCorrect) {
     // Check if the filename is a typo for an option flag. OptTable thinks
     // that all args that are not known options and that start with / are
     // filenames, but e.g. `/diagnostic:caret` is more likely a typo for
     // the option `/diagnostics:caret` than a reference to a file in the root

clang/lib/Driver/ToolChains/Clang.cpp

if (isa<AnalyzeJobAction>(JA)) {
     (void)Args.hasArg(options::OPT_force__cpusubtype__ALL);
   } else if (isa<PrecompileJobAction>(JA)) {
     if (JA.getType() == types::TY_Nothing)
       CmdArgs.push_back("-fsyntax-only");
     else if (JA.getType() == types::TY_ModuleFile)
       CmdArgs.push_back(IsHeaderModulePrecompile
                             ? "-emit-header-module"
                             : "-emit-module-interface");
+    else if (JA.getType() == types::TY_HeaderUnit)
+      CmdArgs.push_back("-emit-header-unit");
     else
       CmdArgs.push_back("-emit-pch");
   } else if (isa<VerifyPCHJobAction>(JA)) {
     CmdArgs.push_back("-verify-pch");
   } else {
     assert((isa<CompileJobAction>(JA) || isa<BackendJobAction>(JA)) &&
            "Invalid action for clang tool.");
     if (JA.getType() == types::TY_Nothing) {

clang/lib/Driver/Types.cpp

assert((getInfo(Id).Phases.contains(phases::Preprocess) !=
          "Unexpected Preprocess Type.");
   return PPT;
 }
 
 static bool isPreprocessedModuleType(ID Id) {
   return Id == TY_CXXModule || Id == TY_PP_CXXModule;
 }
 
+static bool isPreprocessedHeaderUnitType(ID Id) {
+  return Id == TY_CXXSHeader || Id == TY_CXXUHeader || Id == TY_CXXHUHeader ||
+         Id == TY_PP_CXXHeaderUnit;
+}
+
 types::ID types::getPrecompiledType(ID Id) {
   if (isPreprocessedModuleType(Id))
     return TY_ModuleFile;
+  if (isPreprocessedHeaderUnitType(Id))
+    return TY_HeaderUnit;
   if (onlyPrecompileType(Id))
     return TY_PCH;
   return TY_INVALID;
 }
 
 const char *types::getTypeTempSuffix(ID Id, bool CLMode) {
   if (CLMode) {
     switch (Id) {
bool types::isAcceptedByClang(ID Id) {
   case TY_HIP_DEVICE:
   case TY_ObjC: case TY_PP_ObjC: case TY_PP_ObjC_Alias:
   case TY_CXX: case TY_PP_CXX:
   case TY_ObjCXX: case TY_PP_ObjCXX: case TY_PP_ObjCXX_Alias:
   case TY_CHeader: case TY_PP_CHeader:
   case TY_CLHeader:
   case TY_ObjCHeader: case TY_PP_ObjCHeader:
   case TY_CXXHeader: case TY_PP_CXXHeader:
+  case TY_CXXSHeader:
+  case TY_CXXUHeader:
+  case TY_CXXHUHeader:
+  case TY_PP_CXXHeaderUnit:
   case TY_ObjCXXHeader: case TY_PP_ObjCXXHeader:
   case TY_CXXModule: case TY_PP_CXXModule:
   case TY_AST: case TY_ModuleFile: case TY_PCH:
   case TY_LLVM_IR: case TY_LLVM_BC:
   case TY_API_INFO:
     return true;
   }
 }

 bool types::isCXX(ID Id) {
   switch (Id) {
   default:
     return false;
 
   case TY_CXX: case TY_PP_CXX:
   case TY_ObjCXX: case TY_PP_ObjCXX: case TY_PP_ObjCXX_Alias:
   case TY_CXXHeader: case TY_PP_CXXHeader:
+  case TY_CXXSHeader:
+  case TY_CXXUHeader:
+  case TY_CXXHUHeader:
+  case TY_PP_CXXHeaderUnit:
   case TY_ObjCXXHeader: case TY_PP_ObjCXXHeader:
   case TY_CXXModule: case TY_PP_CXXModule:
   case TY_CUDA: case TY_PP_CUDA: case TY_CUDA_DEVICE:
   case TY_HIP:
   case TY_PP_HIP:
   case TY_HIP_DEVICE:
     return true;
   }

 }
 
 bool types::isSrcFile(ID Id) {
   return Id != TY_Object && getPreprocessedType(Id) != TY_INVALID;
 }
 
 types::ID types::lookupTypeForExtension(llvm::StringRef Ext) {
   return llvm::StringSwitch<types::ID>(Ext)
            .Case("c", TY_C)
            .Case("C", TY_CXX)
            .Case("F", TY_Fortran)
            .Case("f", TY_PP_Fortran)
            .Case("h", TY_CHeader)
            .Case("H", TY_CXXHeader)
            .Case("i", TY_PP_C)
            .Case("m", TY_ObjC)
            .Case("M", TY_ObjCXX)
return llvm::StringSwitch<types::ID>(Ext)
            .Case("FOR", TY_PP_Fortran)
            .Case("fpp", TY_Fortran)
            .Case("FPP", TY_Fortran)
            .Case("gch", TY_PCH)
            .Case("hip", TY_HIP)
            .Case("hpp", TY_CXXHeader)
            .Case("hxx", TY_CXXHeader)
            .Case("iim", TY_PP_CXXModule)
+           .Case("iih", TY_PP_CXXHeaderUnit)
            .Case("lib", TY_Object)
            .Case("mii", TY_PP_ObjCXX)
            .Case("obj", TY_Object)
            .Case("ifs", TY_IFS)
            .Case("pch", TY_PCH)
            .Case("pcm", TY_ModuleFile)
            .Case("c++m", TY_CXXModule)
            .Case("cppm", TY_CXXModule)

clang/lib/Frontend/FrontendOptions.cpp

return llvm::StringSwitch<InputKind>(Extension)
       .Case("cui", InputKind(Language::CUDA).getPreprocessed())
       .Case("m", Language::ObjC)
       .Case("mi", InputKind(Language::ObjC).getPreprocessed())
       .Cases("mm", "M", Language::ObjCXX)
       .Case("mii", InputKind(Language::ObjCXX).getPreprocessed())
       .Cases("C", "cc", "cp", Language::CXX)
       .Cases("cpp", "CPP", "c++", "cxx", "hpp", "hxx", Language::CXX)
       .Case("cppm", Language::CXX)
-      .Case("iim", InputKind(Language::CXX).getPreprocessed())
+      .Cases("iim", "iih", InputKind(Language::CXX).getPreprocessed())
       .Case("cl", Language::OpenCL)
       .Case("clcpp", Language::OpenCLCXX)
       .Cases("cu", "cuh", Language::CUDA)
       .Case("hip", Language::HIP)
       .Cases("ll", "bc", Language::LLVM_IR)
       .Default(Language::Unknown);
 }

clang/test/Driver/cxx20-header-units-01.cpp

+// RUN: %clang -### -std=c++20 -xc++-user-header foo.h 2>&1 | \
+// RUN:   FileCheck -check-prefix=CHECK-USER %s
+
+// RUN: %clang -### -std=c++20 -xc++-system-header vector 2>&1 | \
+// RUN:   FileCheck -check-prefix=CHECK-SYSTEM %s
+
+// RUN: %clang -### -std=c++20 \
+// RUN:   -xc++-header-unit-header %/S/Inputs/header-unit-01.hh 2>&1 | \
+// RUN:   FileCheck -check-prefix=CHECK-ABS %s -DTDIR=%/S/Inputs
+
+// CHECK-USER: "-emit-header-unit"
+// CHECK-USER-SAME: "-o" "foo.pcm"
+// CHECK-USER-SAME: "-x" "c++-user-header" "foo.h"
+// CHECK-SYSTEM: "-emit-header-unit"
+// CHECK-SYSTEM-SAME: "-o" "vector.pcm"
+// CHECK-SYSTEM-SAME: "-x" "c++-system-header" "vector"
+// CHECK-ABS: "-emit-header-unit"
+// CHECK-ABS-SAME: "-o" "header-unit-01.pcm"
+// CHECK-ABS-SAME: "-x" "c++-header-unit-header" "[[TDIR]]/header-unit-01.hh"