[clang][extract-api] Use relative includes

[zixuw]

Summary

This patch transforms the given input headers to relative (angled)
include names using header search entries and some heuritics.
For example: /Path/To/Header.h will be included as <Header.h> with a
search path of -I /Path/To/; and
/Path/To/Framework.framework/Headers/Header.h will be included as
<Framework/Header.h>, given a search path of -F /Path/To.
Headermaps will also be queried in reverse to find a spelled name to
include headers.

---

Original Summary:

Proof-of-concept patch. Needs more work.
Transform input headers to relative (angle) includes.
getRelativeIncludeName walks HeaderSearchOptions::UserEntries to try
to find a relative include stem. Problems:

• do we want the exact logic of HeaderSearch::suggestPathToFileForDiagnostics?
• in ExtractAPIAction::PrepareToExecuteAction where the includes are transformed, we don't have a pre-processor yet, so we don't have access to a lot of useful information like HeaderSearch, headermaps, DirectoryLookup etc.
• we might not always want to transform an absolute path because the resulting relative include name might get remapped in a headermap, for example in test known_files_only_hmap.c. But how does it work with modules where we need relative includes? Is the setup in known_files_only_hmap even valid?
• reverse lookup in headermap when deciding if the symbol is coming from a knwon file: I think my current implementation is correct and necessary to solve the headermap problem, but might have problems due to the above point: an unknown file might be mapped to the transformed relative name of a known input.

I think the key problem is that, by passing in an input
/absolute/path/header.h, do we mean to process the actual content of
that particular header, or do we mean to use it just as a "lable" and
the thing we really want to process is whatever this lable points to
with the given search paths and headermaps?
In the example of frameworks, I think the latter is true: we want to
use DSTROOT to pass in the headers because DSTROOT has a defined
framework layout so that we can transform the paths to framework-style
relative includes by matching the layout pattern *.framework/Header/
and rely on headermaps/VFS/module/search paths to find the intended
headers.
In that sense, the known_files_only_hmap test is a really strange
setup with ambiguious intention and context.

[zixuw]

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

87

I still need to dig into this logic and see what kinds of special cases it's trying to handle, and do we really need these instead of simply File.startswith(Dir).

[zixuw]

Add test case to demonstrate the framework case.

[dang]

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

65

This does rely on the path separator being "/". If stick with this regex, do we need to convert all paths to POSIX format? I think the best thing to do is to iterate through the given path components and match for just "(.+)\\.framework" to match just the framework name and the use the base name of the file directly instead of matching it via the regex as well...

200–201

Not sure that just accounting for the header map case is the correct thing to do here. Ideally we would like to know what the include string was, e.g. <MyFramework/MyHeader.h> and match that with how we included one of the original files. This would account for all remapping functionality.

[cishida]

we might not always want to transform an absolute path because the resulting relative include name might get remapped in a headermap, for example in test known_files_only_hmap.c. But how does it work with modules where we need relative includes? Is the setup in known_files_only_hmap even valid?

I think, in most cases, this shouldn't matter because if the header path input doesn't match the location stored in the header map, they should still have the same source content. The same should be true with header search resolution with modules & vfsoverlay

[dang]

In D123831#3455048, @cishida wrote:

we might not always want to transform an absolute path because the resulting relative include name might get remapped in a headermap, for example in test known_files_only_hmap.c. But how does it work with modules where we need relative includes? Is the setup in known_files_only_hmap even valid?

I think, in most cases, this shouldn't matter because if the header path input doesn't match the location stored in the header map, they should still have the same source content. The same should be true with header search resolution with modules & vfsoverlay

Agreed, I think it would be classified as a user error to remap to different source content via headermap.

[ributzka]

In D123831#3458774, @dang wrote:

In D123831#3455048, @cishida wrote:

we might not always want to transform an absolute path because the resulting relative include name might get remapped in a headermap, for example in test known_files_only_hmap.c. But how does it work with modules where we need relative includes? Is the setup in known_files_only_hmap even valid?

I think, in most cases, this shouldn't matter because if the header path input doesn't match the location stored in the header map, they should still have the same source content. The same should be true with header search resolution with modules & vfsoverlay

Agreed, I think it would be classified as a user error to remap to different source content via headermap.

This is happening today in Xcode. Headers that are mapped from the DSTROOT back to the SRCROOT can be different, because they are not simply copied. Xcode also runs unifdef on them.

[dang]

In D123831#3459368, @ributzka wrote:

In D123831#3458774, @dang wrote:

In D123831#3455048, @cishida wrote:

we might not always want to transform an absolute path because the resulting relative include name might get remapped in a headermap, for example in test known_files_only_hmap.c. But how does it work with modules where we need relative includes? Is the setup in known_files_only_hmap even valid?

I think, in most cases, this shouldn't matter because if the header path input doesn't match the location stored in the header map, they should still have the same source content. The same should be true with header search resolution with modules & vfsoverlay

Agreed, I think it would be classified as a user error to remap to different source content via headermap.

This is happening today in Xcode. Headers that are mapped from the DSTROOT back to the SRCROOT can be different, because they are not simply copied. Xcode also runs unifdef on them.

What do you mean? The headers in the DSTROOT don't have the ifdef? Either way, this shouldn't matter because the declarations we are processing would be the same across both versions of the header?

[zixuw]

• Use first match in getRelativeIncludeName
• Try to get Preprocessor and reverse lookup headermaps in getRelativeIncludeName
• Use getRelativeIncludeName to check for known files in LocationFileChecker
• Misc fixes & adjustments

[zixuw]

The problem is that we are trying to shorten the input file paths in PrepareToExecuteAction, where the CompilerInstance is still primal and doesn't even have a FileManager that we could use. That makes it hard (if possible at all) to reverse lookup headermaps and use the spelled names if we failed to find a search path prefix.
The impact is that we won't be able to get an angled include name for cases where an input doesn't reside in any of the normal search paths, but a headermap entry maps some name to that path. Or we might get the wrong include name where there are both headermap entries and search paths that match the input path and the ordering matters.

[zixuw]

• Create FileManager in PrepareToExecuteAction
• Use FileManager to load headermaps and reverse lookup mappings
• Use FileSystem to correctly get working directory and make absolute paths

[zixuw]

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

134

One problem I can see in this right now is that there might be multiple headermaps that together construct a chain of mapping, for example

// A.hmap
Header.h -> Product/Header.h

// B.hmap
Product/Header.h -> /Absolute/path/to/Header.h

Then this iteration would conclude on using Product/Header.h and missed the first mapping (if the command line goes clang -extract-api -I A.hmap -I B.hmap ...).

It's also possible that a headermap and a search path together resolve the header. For example:

//A.hmap
Header.h -> Product/Header.h

// Invocation:
clang -extract-api /Some/Path/to/Product/Header.h -I A.hmap -I /Some/path/to/

I think we need to keep records and iterate backwards on the search paths again to get the full reverse lookup

[zixuw]

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

134

Or, iterate once in reverse and find the last match? 🤔

[zixuw]

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

134

Another thing just came to me: with multiple headermaps chaining remaps, which is the "correct" way to include a header?

// A.hmap
Header.h -> Product/Header.h

// B.hmap
Product/Header.h -> /Absolute/path/to/Header.h

In the context of Darwin frameworks, we'd use <Framework/Header.h> so we don't want to follow the chain all the way backwards.
But without any context or build system rationales of why multiple headermaps with remap chains are generated in the first place, I'm feeling that we don't nearly have enough information for this if we're only talking about headermap as it is and refraining from adopting heuristics.

[zixuw]

• Rewrite commit message for preparation
• Remove shortening paths based on the current working directory: it does not work with angled includes, and unnecessary for our use
• XFAIL test known_files_only_hmap.c, as it is not a valid setup with a questionable headermap. Need to determine how to fix that test case or just discard it, as the new relative_include.m test also checks that symbols from external files are dropped

[dang]

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

134

Two things:

• If we want an exhaustive search, then it would make sense to do what would actually happen in a compilation. Iterate forward and find all matches, then iterate forwards again with each of the matches from the previous step until we find all terminal matches and then heuristically pick the "best one" probably the shortest one.
• The "correct" way of including a header would certainly be #include <Product/Header.h> for a Darwin framework. Nonetheless if the search paths and headermaps setup means that #include "Header.h" or #include <Header.h> is a possible way of getting to the same file I see no harm in doing that. As long as shortening a given file results in deterministic results it should work fine. I think it would be a user error if shortening a file path to say <Header.h> meant that including it would lead to a completely different file (that is with different declarations/definitions).

922

I think getRelativeIncludeName is slightly wrong. It doesn't seem like it accounts for matches specifically made with -iquote arguments. My understanding is that we should record that information at that point and include the file using the appropriate form #include "RelativeName" or #include <RelativeName>. It would probably be best if getRelativeIncludeName included the quote type in the string directly.

clang/test/ExtractAPI/known_files_only_hmap.c

-175

I think you can safely delete this test.

[zixuw]

• Delete test known_files_only_hmap
• Handle quoted includes
• Attempt to fix Windows fails by converting backslashes before matching the framework regex
• Update test relative_include

[zixuw]

Convert file path to use slashes for headermap reverse lookup

[dang]

Minor comment LGTM otherwise

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

203

Should we be tracking if the include was quoted or not? In principle, that could mean a different redirect in header search and therefore potentially mapping to different content.

[zixuw]

• Capture whether the include is quoted in KnownFiles
• Misc: use getInputBufferName() instead of string literal

[dang]

LGTM

[dang]

Since this is a new test can we use the approach in https://reviews.llvm.org/D124634 to check for diagnostics output.

[zixuw]

In D123831#3490684, @dang wrote:

Since this is a new test can we use the approach in https://reviews.llvm.org/D124634 to check for diagnostics output.

I used FileCheck to check the input buffer dump in this test, but I'll try to move to cc1 and fix the diagnostics check.

[zixuw]

Update test case to use cc1 instead of driver

[zixuw]

Accidentally added another test case in my local workspace. Removed in 5f841c71fc2cc77c92f526791cd7a938bcac69aa

clang/include/clang/ExtractAPI/FrontendActions.h

private:
   /// The product this action is extracting API information for.
   std::string ProductName;
 
   /// The synthesized input buffer that contains all the provided input header
   /// files.
   std::unique_ptr<llvm::MemoryBuffer> Buffer;
 
   /// The input file originally provided on the command line.
-  std::vector<std::string> KnownInputFiles;
+  ///
+  /// This captures the spelling used to include the file and whether the
+  /// include is quoted or not.
+  SmallVector<std::pair<SmallString<32>, bool>> KnownInputFiles;
 
   /// Prepare to execute the action on the given CompilerInstance.
   ///
   /// This is called before executing the action on any inputs. This generates a
   /// single header that includes all of CI's inputs and replaces CI's input
   /// list with it before actually executing the action.
   bool PrepareToExecuteAction(CompilerInstance &CI) override;
 

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

 #include "clang/Frontend/FrontendOptions.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/PPCallbacks.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Regex.h"
 #include "llvm/Support/raw_ostream.h"
 #include <memory>
 #include <utility>
 
 using namespace clang;
 using namespace extractapi;
 
 namespace {
 
 StringRef getTypedefName(const TagDecl *Decl) {
   if (const auto *TypedefDecl = Decl->getTypedefNameForAnonDecl())
     return TypedefDecl->getName();
 
   return {};
 }
 
+Optional<std::string> getRelativeIncludeName(const CompilerInstance &CI,
+                                             StringRef File,
+                                             bool *IsQuoted = nullptr) {
+  assert(CI.hasFileManager() &&
+         "CompilerInstance does not have a FileNamager!");
+
+  using namespace llvm::sys;
+  // Matches framework include patterns
+  const llvm::Regex Rule("/(.+)\\.framework/(.+)?Headers/(.+)");
+
+  const auto &FS = CI.getVirtualFileSystem();
+
+  SmallString<128> FilePath(File.begin(), File.end());
+  FS.makeAbsolute(FilePath);
+  path::remove_dots(FilePath, true);
+  File = FilePath;
+
+  // Checks whether `Dir` is a strict path prefix of `File`. If so returns
+  // the prefix length. Otherwise return 0.
+  auto CheckDir = [&](llvm::StringRef Dir) -> unsigned {
+    llvm::SmallString<32> DirPath(Dir.begin(), Dir.end());
+    FS.makeAbsolute(DirPath);
+    path::remove_dots(DirPath, true);
+    Dir = DirPath;
+    for (auto NI = path::begin(File), NE = path::end(File),
+              DI = path::begin(Dir), DE = path::end(Dir);
+         /*termination condition in loop*/; ++NI, ++DI) {
+      // '.' components in File are ignored.
+      while (NI != NE && *NI == ".")
+        ++NI;
+      if (NI == NE)
+        break;
+
+      // '.' components in Dir are ignored.
+      while (DI != DE && *DI == ".")
+        ++DI;
+
+      // Dir is a prefix of File, up to '.' components and choice of path
+      // separators.
+      if (DI == DE)
+        return NI - path::begin(File);
+
+      // Consider all path separators equal.
+      if (NI->size() == 1 && DI->size() == 1 &&
+          path::is_separator(NI->front()) && path::is_separator(DI->front()))
+        continue;
+
+      // Special case Apple .sdk folders since the search path is typically a
+      // symlink like `iPhoneSimulator14.5.sdk` while the file is instead
+      // located in `iPhoneSimulator.sdk` (the real folder).
+      if (NI->endswith(".sdk") && DI->endswith(".sdk")) {
+        StringRef NBasename = path::stem(*NI);
+        StringRef DBasename = path::stem(*DI);
+        if (DBasename.startswith(NBasename))
+          continue;
+      }
+
+      if (*NI != *DI)
+        break;
+    }
+    return 0;
+  };
+
+  unsigned PrefixLength = 0;
+
+  // Go through the search paths and find the first one that is a prefix of
+  // the header.
+  for (const auto &Entry : CI.getHeaderSearchOpts().UserEntries) {
+    // Note whether the match is found in a quoted entry.
+    if (IsQuoted)
+      *IsQuoted = Entry.Group == frontend::Quoted;
+
+    if (auto EntryFile = CI.getFileManager().getOptionalFileRef(Entry.Path)) {
+      if (auto HMap = HeaderMap::Create(*EntryFile, CI.getFileManager())) {
+        // If this is a headermap entry, try to reverse lookup the full path
+        // for a spelled name before mapping.
+        StringRef SpelledFilename =
+            HMap->reverseLookupFilename(path::convert_to_slash(File));
+        if (!SpelledFilename.empty())
+          return SpelledFilename.str();
+
+        // No matching mapping in this headermap, try next search entry.
+        continue;
+      }
+    }
+
+    // Entry is a directory search entry, try to check if it's a prefix of File.
+    PrefixLength = CheckDir(Entry.Path);
+    if (PrefixLength > 0) {
+      // The header is found in a framework path, construct the framework-style
+      // include name `<Framework/Header.h>`
+      if (Entry.IsFramework) {
+        SmallVector<StringRef, 4> Matches;
+        // Convert the file path to forward slashes for matching the framework
+        // regex.
+        Rule.match(path::convert_to_slash(File), &Matches);
+        // Returned matches are always in stable order.
+        if (Matches.size() != 4)
+          return None;
+
+        return path::convert_to_slash(
+            (Matches[1].drop_front(Matches[1].rfind('/') + 1) + "/" +
+             Matches[3])
+                .str());
+
+      }
+
+      // The header is found in a normal search path, strip the search path
+      // prefix to get an include name.
+      return path::convert_to_slash(File.drop_front(PrefixLength));
+    }
+  }
+
+  // Couldn't determine a include name, use full path instead.
+  return None;
+}
+
 struct LocationFileChecker {
   bool isLocationInKnownFile(SourceLocation Loc) {
     // If the loc refers to a macro expansion we need to first get the file
     // location of the expansion.
+    auto &SM = CI.getSourceManager();
     auto FileLoc = SM.getFileLoc(Loc);
     FileID FID = SM.getFileID(FileLoc);
     if (FID.isInvalid())
       return false;
 
     const auto *File = SM.getFileEntryForID(FID);
     if (!File)
       return false;
 
     if (KnownFileEntries.count(File))
       return true;
 
+    if (ExternalFileEntries.count(File))
       return false;
+
+    StringRef FileName = File->tryGetRealPathName().empty()
+                             ? File->getName()
+                             : File->tryGetRealPathName();
+
+    // Try to reduce the include name the same way we tried to include it.
+    bool IsQuoted = false;
+    if (auto IncludeName = getRelativeIncludeName(CI, FileName, &IsQuoted))
+      if (llvm::find_if(KnownFiles,
+                        [&IsQuoted, &IncludeName](const auto &KnownFile) {
+                          return KnownFile.first.equals(*IncludeName) &&
+                                 KnownFile.second == IsQuoted;
+                        }) != KnownFiles.end()) {
+        KnownFileEntries.insert(File);
+        return true;
       }
 
-  LocationFileChecker(const SourceManager &SM,
-                      const std::vector<std::string> &KnownFiles)
-      : SM(SM) {
-    for (const auto &KnownFilePath : KnownFiles)
-      if (auto FileEntry = SM.getFileManager().getFile(KnownFilePath))
+    // Record that the file was not found to avoid future reverse lookup for
+    // the same file.
+    ExternalFileEntries.insert(File);
+    return false;
+  }
+
+  LocationFileChecker(const CompilerInstance &CI,
+                      SmallVector<std::pair<SmallString<32>, bool>> &KnownFiles)
+      : CI(CI), KnownFiles(KnownFiles), ExternalFileEntries() {
+    for (const auto &KnownFile : KnownFiles)
+      if (auto FileEntry = CI.getFileManager().getFile(KnownFile.first))
         KnownFileEntries.insert(*FileEntry);
   }
 
 private:
-  const SourceManager &SM;
+  const CompilerInstance &CI;
+  SmallVector<std::pair<SmallString<32>, bool>> &KnownFiles;
   llvm::DenseSet<const FileEntry *> KnownFileEntries;
+  llvm::DenseSet<const FileEntry *> ExternalFileEntries;
 };
 
 /// The RecursiveASTVisitor to traverse symbol declarations and collect API
 /// information.
 class ExtractAPIVisitor : public RecursiveASTVisitor<ExtractAPIVisitor> {
 public:
   ExtractAPIVisitor(ASTContext &Context, LocationFileChecker &LCF, APISet &API)
       : Context(Context), API(API), LCF(LCF) {}
ExtractAPIAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
   ProductName = CI.getFrontendOpts().ProductName;
 
   // Now that we have enough information about the language options and the
   // target triple, let's create the APISet before anyone uses it.
   API = std::make_unique<APISet>(
       CI.getTarget().getTriple(),
       CI.getFrontendOpts().Inputs.back().getKind().getLanguage());
 
-  auto LCF = std::make_unique<LocationFileChecker>(CI.getSourceManager(),
-                                                   KnownInputFiles);
+  auto LCF = std::make_unique<LocationFileChecker>(CI, KnownInputFiles);
 
   CI.getPreprocessor().addPPCallbacks(std::make_unique<MacroCallback>(
       CI.getSourceManager(), *LCF, *API, CI.getPreprocessor()));
 
   return std::make_unique<ExtractAPIConsumer>(CI.getASTContext(),
                                               std::move(LCF), *API);
 }
 
 bool ExtractAPIAction::PrepareToExecuteAction(CompilerInstance &CI) {
   auto &Inputs = CI.getFrontendOpts().Inputs;
   if (Inputs.empty())
     return true;
 
+  if (!CI.hasFileManager())
+    if (!CI.createFileManager())
+      return false;
+
   auto Kind = Inputs[0].getKind();
 
   // Convert the header file inputs into a single input buffer.
   SmallString<256> HeaderContents;
+  bool IsQuoted = false;
   for (const FrontendInputFile &FIF : Inputs) {
     if (Kind.isObjectiveC())
       HeaderContents += "#import";
     else
       HeaderContents += "#include";
+
+    StringRef FilePath = FIF.getFile();
+    if (auto RelativeName = getRelativeIncludeName(CI, FilePath, &IsQuoted)) {
+      if (IsQuoted)
         HeaderContents += " \"";
-    HeaderContents += FIF.getFile();
-    HeaderContents += "\"\n";
+      else
+        HeaderContents += " <";
 
-    KnownInputFiles.emplace_back(FIF.getFile());
+      HeaderContents += *RelativeName;
+
+      if (IsQuoted)
+        HeaderContents += "\"\n";
+      else
+        HeaderContents += ">\n";
+      KnownInputFiles.emplace_back(*RelativeName, IsQuoted);
+    } else {
+      HeaderContents += " \"";
+      HeaderContents += FilePath;
+      HeaderContents += "\"\n";
+      KnownInputFiles.emplace_back(FilePath, true);
     }
+  }
+
+  if (CI.getHeaderSearchOpts().Verbose)
+    CI.getVerboseOutputStream() << getInputBufferName() << ":\n"
+                                << HeaderContents << "\n";
 
   Buffer = llvm::MemoryBuffer::getMemBufferCopy(HeaderContents,
                                                 getInputBufferName());
 
   // Set that buffer up as our "real" input in the CompilerInstance.
   Inputs.clear();
   Inputs.emplace_back(Buffer->getMemBufferRef(), Kind, /*IsSystem*/ false);
 

clang/test/ExtractAPI/known_files_only_hmap.c

-// RUN: rm -rf %t
-// RUN: split-file %s %t
-// RUN: sed -e "s@INPUT_DIR@%{/t:regex_replacement}@g" \
-// RUN: %t/reference.output.json.in >> %t/reference.output.json
-// RUN: sed -e "s@INPUT_DIR@%{/t:regex_replacement}@g" \
-// RUN: %t/known_files_only.hmap.json.in >> %t/known_files_only.hmap.json
-// RUN: %hmaptool write %t/known_files_only.hmap.json %t/known_files_only.hmap
-// RUN: %clang -extract-api --product-name=KnownFilesOnlyHmap -target arm64-apple-macosx \
-// RUN: -I%t/known_files_only.hmap -I%t/subdir %t/subdir/subdir1/input.h \
-// RUN: %t/subdir/subdir2/known_file.h -o %t/output.json | FileCheck -allow-empty %s
-
-// Generator version is not consistent across test runs, normalize it.
-// RUN: sed -e "s@\"generator\": \".*\"@\"generator\": \"?\"@g" \
-// RUN: %t/output.json >> %t/output-normalized.json
-// RUN: diff %t/reference.output.json %t/output-normalized.json
-
-// CHECK-NOT: error:
-// CHECK-NOT: warning:
-//--- known_files_only.hmap.json.in
-{
-  "mappings" :
-    {
-     "subdir2/known_file.h" : "INPUT_DIR/subdir/subdir3/unknown.h"
-    }
-}
-
-//--- subdir/subdir1/input.h
-int num;
-#include "subdir2/known_file.h"
-
-//--- subdir/subdir2/known_file.h
-int known_num;
-
-//--- subdir/subdir3/unknown.h
-// Ensure that these symbols are not emitted in the Symbol Graph.
-#ifndef INPUT4_H
-#define INPUT4_H
-
-#define HELLO 1
-char not_emitted;
-void foo(int);
-struct Foo { int a; };
-
-#endif
-
-//--- reference.output.json.in
-{
-  "metadata": {
-    "formatVersion": {
-      "major": 0,
-      "minor": 5,
-      "patch": 3
-    },
-    "generator": "?"
-  },
-  "module": {
-    "name": "KnownFilesOnlyHmap",
-    "platform": {
-      "architecture": "arm64",
-      "operatingSystem": {
-        "minimumVersion": {
-          "major": 11,
-          "minor": 0,
-          "patch": 0
-        },
-        "name": "macosx"
-      },
-      "vendor": "apple"
-    }
-  },
-  "relationships": [],
-  "symbols": [
-    {
-      "accessLevel": "public",
-      "declarationFragments": [
-        {
-          "kind": "typeIdentifier",
-          "preciseIdentifier": "c:I",
-          "spelling": "int"
-        },
-        {
-          "kind": "text",
-          "spelling": " "
-        },
-        {
-          "kind": "identifier",
-          "spelling": "num"
-        }
-      ],
-      "identifier": {
-        "interfaceLanguage": "c",
-        "precise": "c:@num"
-      },
-      "kind": {
-        "displayName": "Global Variable",
-        "identifier": "c.var"
-      },
-      "location": {
-        "position": {
-          "character": 5,
-          "line": 1
-        },
-        "uri": "file://INPUT_DIR/subdir/subdir1/input.h"
-      },
-      "names": {
-        "navigator": [
-          {
-            "kind": "identifier",
-            "spelling": "num"
-          }
-        ],
-        "subHeading": [
-          {
-            "kind": "identifier",
-            "spelling": "num"
-          }
-        ],
-        "title": "num"
-      },
-      "pathComponents": [
-        "num"
-      ]
-    },
-    {
-      "accessLevel": "public",
-      "declarationFragments": [
-        {
-          "kind": "typeIdentifier",
-          "preciseIdentifier": "c:I",
-          "spelling": "int"
-        },
-        {
-          "kind": "text",
-          "spelling": " "
-        },
-        {
-          "kind": "identifier",
-          "spelling": "known_num"
-        }
-      ],
-      "identifier": {
-        "interfaceLanguage": "c",
-        "precise": "c:@known_num"
-      },
-      "kind": {
-        "displayName": "Global Variable",
-        "identifier": "c.var"
-      },
-      "location": {
-        "position": {
-          "character": 5,
-          "line": 1
-        },
-        "uri": "file://INPUT_DIR/subdir/subdir2/known_file.h"
-      },
-      "names": {
-        "navigator": [
-          {
-            "kind": "identifier",
-            "spelling": "known_num"
-          }
-        ],
-        "subHeading": [
-          {
-            "kind": "identifier",
-            "spelling": "known_num"
-          }
-        ],
-        "title": "known_num"
-      },
-      "pathComponents": [
-        "known_num"
-      ]
-    }
-  ]
-}

clang/test/ExtractAPI/relative_include.m

+// RUN: rm -rf %t
+// RUN: split-file %s %t
+
+// Setup framework root
+// RUN: mkdir -p %t/Frameworks/MyFramework.framework/Headers
+// RUN: cp %t/MyFramework.h %t/Frameworks/MyFramework.framework/Headers/
+// RUN: cp %t/MyHeader.h %t/Frameworks/MyFramework.framework/Headers/
+
+// RUN: sed -e "s@SRCROOT@%{/t:regex_replacement}@g" \
+// RUN: %t/reference.output.json.in >> %t/reference.output.json
+
+// Headermap maps headers to the source root SRCROOT
+// RUN: sed -e "s@SRCROOT@%{/t:regex_replacement}@g" \
+// RUN: %t/headermap.hmap.json.in >> %t/headermap.hmap.json
+// RUN: %hmaptool write %t/headermap.hmap.json %t/headermap.hmap
+
+// Input headers use paths to the framework root/DSTROOT
+// RUN: %clang_cc1 -extract-api -v --product-name=MyFramework \
+// RUN: -triple arm64-apple-macosx \
+// RUN: -iquote%t -I%t/headermap.hmap -F%t/Frameworks \
+// RUN: -x objective-c-header \
+// RUN: %t/Frameworks/MyFramework.framework/Headers/MyFramework.h \
+// RUN: %t/Frameworks/MyFramework.framework/Headers/MyHeader.h \
+// RUN: %t/QuotedHeader.h \
+// RUN: -o %t/output.json 2>&1 -verify | FileCheck -allow-empty %s
+
+// Generator version is not consistent across test runs, normalize it.
+// RUN: sed -e "s@\"generator\": \".*\"@\"generator\": \"?\"@g" \
+// RUN: %t/output.json >> %t/output-normalized.json
+// RUN: diff %t/reference.output.json %t/output-normalized.json
+
+// CHECK:      <extract-api-includes>:
+// CHECK-NEXT: #import <MyFramework/MyFramework.h>
+// CHECK-NEXT: #import <MyFramework/MyHeader.h>
+// CHECK-NEXT: #import "QuotedHeader.h"
+
+//--- headermap.hmap.json.in
+{
+  "mappings" :
+    {
+     "MyFramework/MyHeader.h" : "SRCROOT/MyHeader.h"
+    }
+}
+
+//--- MyFramework.h
+// Umbrella for MyFramework
+#import <MyFramework/MyHeader.h>
+// expected-no-diagnostics
+
+//--- MyHeader.h
+#import <OtherFramework/OtherHeader.h>
+int MyInt;
+// expected-no-diagnostics
+
+//--- QuotedHeader.h
+char MyChar;
+// expected-no-diagnostics
+
+//--- Frameworks/OtherFramework.framework/Headers/OtherHeader.h
+int OtherInt;
+// expected-no-diagnostics
+
+//--- reference.output.json.in
+{
+  "metadata": {
+    "formatVersion": {
+      "major": 0,
+      "minor": 5,
+      "patch": 3
+    },
+    "generator": "?"
+  },
+  "module": {
+    "name": "MyFramework",
+    "platform": {
+      "architecture": "arm64",
+      "operatingSystem": {
+        "minimumVersion": {
+          "major": 11,
+          "minor": 0,
+          "patch": 0
+        },
+        "name": "macosx"
+      },
+      "vendor": "apple"
+    }
+  },
+  "relationships": [],
+  "symbols": [
+    {
+      "accessLevel": "public",
+      "declarationFragments": [
+        {
+          "kind": "typeIdentifier",
+          "preciseIdentifier": "c:I",
+          "spelling": "int"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "MyInt"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "objective-c",
+        "precise": "c:@MyInt"
+      },
+      "kind": {
+        "displayName": "Global Variable",
+        "identifier": "objective-c.var"
+      },
+      "location": {
+        "position": {
+          "character": 5,
+          "line": 2
+        },
+        "uri": "file://SRCROOT/MyHeader.h"
+      },
+      "names": {
+        "navigator": [
+          {
+            "kind": "identifier",
+            "spelling": "MyInt"
+          }
+        ],
+        "subHeading": [
+          {
+            "kind": "identifier",
+            "spelling": "MyInt"
+          }
+        ],
+        "title": "MyInt"
+      },
+      "pathComponents": [
+        "MyInt"
+      ]
+    },
+    {
+      "accessLevel": "public",
+      "declarationFragments": [
+        {
+          "kind": "typeIdentifier",
+          "preciseIdentifier": "c:C",
+          "spelling": "char"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "MyChar"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "objective-c",
+        "precise": "c:@MyChar"
+      },
+      "kind": {
+        "displayName": "Global Variable",
+        "identifier": "objective-c.var"
+      },
+      "location": {
+        "position": {
+          "character": 6,
+          "line": 1
+        },
+        "uri": "file://SRCROOT/QuotedHeader.h"
+      },
+      "names": {
+        "navigator": [
+          {
+            "kind": "identifier",
+            "spelling": "MyChar"
+          }
+        ],
+        "subHeading": [
+          {
+            "kind": "identifier",
+            "spelling": "MyChar"
+          }
+        ],
+        "title": "MyChar"
+      },
+      "pathComponents": [
+        "MyChar"
+      ]
+    }
+  ]
+}

clang/test/Index/annotate-comments-enum-constant.c

+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: c-index-test -test-load-source all -comments-xml-schema=%S/../../bindings/xml/comment-xml-schema.rng %s > %t/out
+// RUN: FileCheck %s --dump-input always < %t/out
+
+enum {
+  /// Documentation for Foo
+  Foo,
+  Bar, // No documentation for Bar
+  /// Documentation for Baz
+  Baz,
+};
+// CHECK: EnumConstantDecl=Foo:[[@LINE-5]]:3 (Definition) {{.*}} BriefComment=[Documentation for Foo] FullCommentAsHTML=[<p class="para-brief"> Documentation for Foo</p>] FullCommentAsXML=[<Variable file="{{[^"]+}}annotate-comments-enum-constant.c" line="[[@LINE-5]]" column="3"><Name>Foo</Name><USR>c:@Ea@Foo@Foo</USR><Declaration>Foo</Declaration><Abstract><Para> Documentation for Foo</Para></Abstract></Variable>]
+// CHECK: EnumConstantDecl=Bar:[[@LINE-5]]:3 (Definition)
+// CHECK-NOT: BriefComment=[Documentation for Foo]
+// CHECK: EnumConstantDecl=Baz:[[@LINE-5]]:3 (Definition) {{.*}} BriefComment=[Documentation for Baz] FullCommentAsHTML=[<p class="para-brief"> Documentation for Baz</p>] FullCommentAsXML=[<Variable file="{{[^"]+}}annotate-comments-enum-constant.c" line="[[@LINE-5]]" column="3"><Name>Baz</Name><USR>c:@Ea@Foo@Baz</USR><Declaration>Baz</Declaration><Abstract><Para> Documentation for Baz</Para></Abstract></Variable>]
+