[C++20][Modules][HU 4/5] Handle pre-processed header units.

[iains]

Summary

We wish to support emitting a pre-processed output for an importable
header unit, that can be consumed to produce the same header units as
the original source.

This means that ee need to find the original filename used to produce
the re-preprocessed output, so that it can be assigned as the module
name. This is peeked from the first line of the pre-processed source
when the action sets up the files.

[iains]

rebased

[iains]

rebased.

[ChuanqiXu]

It lacks tests. This is not NFC, right?

---

I am a little bit confused for the intuition. Couldn't we just import the pre-processed header? What's the problem? Could you elaborate on this?

[iains]

In D121098#3378375, @ChuanqiXu wrote:

It lacks tests. This is not NFC, right?

Right, (there are tests with the next patch which introduces the mechanism for producing the pre-processed output)
but, I will find a suitable one for this...

---

I am a little bit confused for the intuition. Couldn't we just import the pre-processed header? What's the problem? Could you elaborate on this?

We cannot import a pre-processed file; the preprocessed file is still source code, not a CMI.

In addition, the pre-processor output for a header unit requires further pre-processing on read.
This is because a header unit actually preserves some of the pre-processor information [macro definitions] (which would normally be discarded after phase 4).

Header units are identifiable by a name (in common with GCC, we make the name == the path by which the header is specified) -- that name is intentionally not a legal named module name (neither is it, in general, a legal identifier).

[ChuanqiXu]

In D121098#3383835, @iains wrote:

I am a little bit confused for the intuition. Couldn't we just import the pre-processed header? What's the problem? Could you elaborate on this?

We cannot import a pre-processed file; the preprocessed file is still source code, not a CMI.

But if I understand correctly, we are able to import a pre-processed file as a header unit. Couldn't we?

For example:

// cc -E foo.h > foo.preprocessed
import "foo.preprocessed"
...

Is this not acceptable?

In addition, the pre-processor output for a header unit requires further pre-processing on read.
This is because a header unit actually preserves some of the pre-processor information [macro definitions] (which would normally be discarded after phase 4).

Header units are identifiable by a name (in common with GCC, we make the name == the path by which the header is specified) -- that name is intentionally not a legal named module name (neither is it, in general, a legal identifier).

So it doesn't eliminate my confusion. I guess there is some workflows that we need to import a preprocessed header as header unit (and we need to find the position of original file). Or in what circumstances, do we need to import a pre-processed header as header unit?

[iains]

In D121098#3384590, @ChuanqiXu wrote:

In D121098#3383835, @iains wrote:

I am a little bit confused for the intuition. Couldn't we just import the pre-processed header? What's the problem? Could you elaborate on this?

We cannot import a pre-processed file; the preprocessed file is still source code, not a CMI.

But if I understand correctly, we are able to import a pre-processed file as a header unit. Couldn't we?

For example:

// cc -E foo.h > foo.preprocessed
import "foo.preprocessed"
...

Is this not acceptable?

In addition, the pre-processor output for a header unit requires further pre-processing on read.
This is because a header unit actually preserves some of the pre-processor information [macro definitions] (which would normally be discarded after phase 4).

Header units are identifiable by a name (in common with GCC, we make the name == the path by which the header is specified) -- that name is intentionally not a legal named module name (neither is it, in general, a legal identifier).

So it doesn't eliminate my confusion. I guess there is some workflows that we need to import a preprocessed header as header unit (and we need to find the position of original file). Or in what circumstances, do we need to import a pre-processed header as header unit?

A pre-processed header unit can only be used to build that header unit - just like a pre-processed c++ source is used to build an object for the original source file.

In this code, we are not considering importing the HU into another module - that can only be done once the HU is built as a CMI.

The pre-processed output has two immediate uses I could see:
(1) traditionally, it can be helpful in debugging and problem reporting [since the included headers are flattened]
(2) because we will have processed the directives, it might be useful for scanners that want to determine remaining dependencies.

[ChuanqiXu]

OK, I feel good if this one contains tests.

[iains]

rebased, added test.

[urnathan]

yeah, gcc's preprocessor has peeking code to deal with this (more generally than modules)

[iains]

rebased

clang/lib/Frontend/FrontendAction.cpp

if (CI.getLangOpts().CPlusPlusModules && Input.getKind().isHeaderUnit() &&
     if (CI.getLangOpts().ModuleName.empty())
       CI.getLangOpts().ModuleName = std::string(FileName);
     CI.getLangOpts().CurrentModule = CI.getLangOpts().ModuleName;
   }
 
   if (!CI.InitializeSourceManager(Input))
     return false;
 
+  if (CI.getLangOpts().CPlusPlusModules && Input.getKind().isHeaderUnit() &&
+      Input.getKind().isPreprocessed() && !usesPreprocessorOnly()) {
+    // We have an input filename like foo.iih, but we want to find the right
+    // module name (and original file, to build the map entry).
+    // Check if the first line specifies the original source file name with a
+    // linemarker.
+    std::string PresumedInputFile = std::string(getCurrentFileOrBufferName());
+    ReadOriginalFileName(CI, PresumedInputFile);
+    // Unless the user overrides this, the module name is the name by which the
+    // original file was known.
+    if (CI.getLangOpts().ModuleName.empty())
+      CI.getLangOpts().ModuleName = std::string(PresumedInputFile);
+    CI.getLangOpts().CurrentModule = CI.getLangOpts().ModuleName;
+  }
+
   // For module map files, we first parse the module map and synthesize a
   // "<module-includes>" buffer before more conventional processing.
   if (Input.getKind().getFormat() == InputKind::ModuleMap) {
     CI.getLangOpts().setCompilingModule(LangOptions::CMK_ModuleMap);
 
     std::string PresumedModuleMapFile;
     unsigned OffsetToContents;
     if (loadModuleMapForModuleBuild(CI, Input.isSystem(),

clang/lib/Sema/SemaModule.cpp

SourceLocation StartOfTU =
       SourceMgr.getLocForStartOfFile(SourceMgr.getMainFileID());
 
   StringRef HUName = getLangOpts().CurrentModule;
   if (HUName.empty()) {
     HUName = SourceMgr.getFileEntryForID(SourceMgr.getMainFileID())->getName();
     const_cast<LangOptions &>(getLangOpts()).CurrentModule = HUName.str();
   }
 
-  auto &Map = PP.getHeaderSearchInfo().getModuleMap();
   // TODO: Make the C++20 header lookup independent.
-  Module::Header H{getLangOpts().CurrentModule, getLangOpts().CurrentModule,
-                   SourceMgr.getFileEntryForID(SourceMgr.getMainFileID())};
+  // When the input is pre-processed source, we need a file ref to the original
+  // file for the header map.
+  auto F = SourceMgr.getFileManager().getFile(HUName);
+  // For the sake of error recovery (if someone has moved the original header
+  // after creating the pre-processed output) fall back to obtaining the file
+  // ref for the input file, which must be present.
+  if (!F)
+    F = SourceMgr.getFileEntryForID(SourceMgr.getMainFileID());
+  assert(F && "failed to find the header unit source?");
+  Module::Header H{HUName.str(), HUName.str(), *F};
+  auto &Map = PP.getHeaderSearchInfo().getModuleMap();
   Module *Mod = Map.createHeaderUnit(StartOfTU, HUName, H);
   assert(Mod && "module creation should not fail");
   ModuleScopes.push_back({}); // No GMF
   ModuleScopes.back().BeginLoc = StartOfTU;
   ModuleScopes.back().Module = Mod;
   ModuleScopes.back().ModuleInterface = true;
   ModuleScopes.back().IsPartition = false;
   VisibleModules.setVisible(Mod, StartOfTU);

clang/test/Modules/cxx20-hu-05.cpp

+// Test macro preservation in C++20 Header Units.
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: split-file %s %t
+// RUN: cd %t
+
+// Produce a pre-processed file.
+// RUN: %clang_cc1 -std=c++20 -E -xc++-user-header hu-01.h -o hu-01.iih
+
+// consume that to produce the heder unit.
+// RUN: %clang_cc1 -std=c++20 -emit-header-unit \
+// RUN: -xc++-header-unit-header-cpp-output hu-01.iih -o hu-01.pcm
+
+// check that the header unit is named for the original file, not the .iih.
+// RUN: %clang_cc1 -std=c++20 -module-file-info hu-01.pcm | \
+// RUN: FileCheck --check-prefix=CHECK-HU %s -DTDIR=%t
+
+//--- hu-01.h
+#ifndef __GUARD
+#define __GUARD
+
+int baz(int);
+#define FORTYTWO 42
+
+#define SHOULD_NOT_BE_DEFINED -1
+#undef SHOULD_NOT_BE_DEFINED
+
+#endif // __GUARD
+
+// CHECK-HU:  ====== C++20 Module structure ======
+// CHECK-HU-NEXT:  Header Unit './hu-01.h' is the Primary Module at index #1