[clang-doc] Move file layout to the generators.

[brettw]

Summary

Previously file naming and directory layout was handled on a per Info object basis by ClangDocMain and the generators blindly wrote to the files given. This means all generators must use the same file layout and caused problems where multiple objects mapped to the same file. The object collision problem happens most easily with template specializations because the template parameters are not part of the "name".

This patch moves the responsibility for output file organization to the generators. Currently HTML and MD use the same structure as before. But they now collect all objects that map to a given file and combine them, avoiding the corruption problems.

Converts the YAML generator to naming files based on USR in one directory. This is easier for downstream tools to manage and avoids the naming problems with template specializations. Since this change requires backward-incompatible output changes to referenced files anyway (since each one is now an array), this is a good time to introduce this change.

[brettw]

If you're curious, you can see the simplification the YAML output format change makes in the consuming code:
https://fuchsia-review.git.corp.google.com/c/fuchsia/+/760570/2/tools/cppdocgen/clangdoc/clangdoc.go
Since it allows us to remove some special-casing code for loading things in the global namespace and for anonymous records.

[brettw]

Updated the lit tests.

[paulkirth]

Can you document the new directory structure with a small example? It would also be good to document in the code that some of these changes were driven by file corruption, and if possible point to a bug on github.

clang-tools-extra/clang-doc/HTMLGenerator.cpp

860

Nit: necessary

890–892

So does this patch make HTML unusable? Currently, if there is a conflict, the file gets replaced, right?

I read this as we're going from a situation where we lost data, but still had valid HTML pages to one where we have complete, but incorrect HTML, is that correct?

Also, why does that happen under template specialization? USRs are SHA1 hashes of the symbol. I would expect that hashing the symbol would be unambiguous and unique, given C++'s ODR.

That last point made me wonder if an alternate way to handle this without using the hash would be to use the mangled names?

893

Nit: separately

clang-tools-extra/clang-doc/MDGenerator.cpp

382

nit: necessary

422–435

Why did this move? Nothing has changed in its implementation, right?

clang-tools-extra/test/clang-doc/single-file-public.cpp

6–9

Its OK to use ls and grep in the runline, but you cannot use the backtick syntax. It will cause problems on other platforms and I can't find any other usage of that in our codebase. TBH I'm a bit surprised this works, since lit has some interesting behavior around RUN.

I think you can just use cat < ls %t/docs | grep -v index.yaml instead. I haven't tried it, but that is certainly a more typical method of doing this in a lit test. Using find may also work, but I only found one use in llvm/test/ExecutionEngine/MCJIT/load-object-a.ll.

[brettw]

clang-tools-extra/clang-doc/HTMLGenerator.cpp

890–892

I think this makes HTML strictly better (though not ideal).

Previously if there was a collision, they got written over the top of each other without truncation (not sure why). So if the longer page was written second, you got lucky and you get a complete valid page and just lost the shorter struct's definition. If you got unlucky the shorter page was written second and you get a corrupted file with the shorter content, followed by the end of the longer content peeking out from the end.

With this change, you get both content concatenated without corruption, you just have duplicate doctype and title tags in between them. Browsers should generally handle this so the page should be usable.

Fixing this requires a refactor of the HTML generator such that the concept of writing a struct is separate from writing a page. Even if I was going to work on the HTML generator, I would probably want to do this in a separate patch anyway.

The markdown generator does the same, but markdown doesn't have all of the headers stuff that shouldn't be in the middle of the file, you just get a new # ... for the second section and it should be reasonable.

The HTML and MD generators name files according to the Name of the structure. USR is not involved. So any time to things in the same namespace have the same name, you'll get a collision. This happens most commonly with template specialization because the names are identical (the name doesn't account for the template parameters).

All of this complexity is why I changed the YAML output to use USR which avoids the problem. I didn't want to make this change for the HTML and MD generators because it will make ugly file names that the user will see, and you probably want the template specializations to be in the same file anyway.

clang-tools-extra/test/clang-doc/single-file-public.cpp

6–9

Your example doesn't work (it's trying to load a file named "ls") but I found another solution using xargs (which is hopefully supported on all of the platforms).

[paulkirth]

clang-tools-extra/clang-doc/HTMLGenerator.cpp

890–892

I think this makes HTML strictly better (though not ideal).

Previously if there was a collision, they got written over the top of each other without truncation (not sure why). So if the longer page was written second, you got lucky and you get a complete valid page and just lost the shorter struct's definition. If you got unlucky the shorter page was written second and you get a corrupted file with the shorter content, followed by the end of the longer content peeking out from the end.

With this change, you get both content concatenated without corruption, you just have duplicate doctype and title tags in between them. Browsers should generally handle this so the page should be usable.

Thanks for the explanation. My concern was that this change made HTML unusable, if it is still valid (if not ideal) then I don't think there is an issue.

Fixing this requires a refactor of the HTML generator such that the concept of writing a struct is separate from writing a page. Even if I was going to work on the HTML generator, I would probably want to do this in a separate patch anyway.

No arguments re splitting up work. Can you file an issue to track this on github and reference it in the TODO? That should help make sure this gets addressed.

The markdown generator does the same, but markdown doesn't have all of the headers stuff that shouldn't be in the middle of the file, you just get a new # ... for the second section and it should be reasonable.

The HTML and MD generators name files according to the Name of the structure. USR is not involved. So any time to things in the same namespace have the same name, you'll get a collision. This happens most commonly with template specialization because the names are identical (the name doesn't account for the template parameters).

I think there should be a way to obtain the mangled name from whatever you're looking at. Those should be available, even from the AST, so if you've found a template specialization, it should be possible to get the mangled name (even for partial specializations). This shouldn't necessarily block you, but it seems like that would solve the issue nicely for all three generators.

clang-tools-extra/test/clang-doc/single-file-public.cpp

6–9

I don't think we require xargs, so I'm not sure you can rely on that. We do require find, so that may be a better choice, and you can even have it invoke cat directly.

See: https://llvm.org/docs/GettingStarted.html#software

[brettw]

clang-tools-extra/clang-doc/HTMLGenerator.cpp

890–892

TODO added.

I don't think the mangled names helps both because I think you want the file names to be readable (the mangled names are barely better than the USR) and because I think you also want the specializations to all end up in the same file for end-user documentation. I discussed this in the bug.

clang-tools-extra/test/clang-doc/single-file-public.cpp

6–9

I found a find invocation that worked. I couldn't get it to recognize the "repeats 40 times" regex syntax so I copy-pasted [0-9A-Z] 40 times like in the body of the test.

[paulkirth]

This is mostly LGTM, modulo the small change to the test that I've suggested. Assuming the presubmit tests still pass with that small edit, you're good on my end.

clang-tools-extra/clang-doc/HTMLGenerator.cpp

890–892

TODO added.

I don't think the mangled names helps both because I think you want the file names to be readable (the mangled names are barely better than the USR) and because I think you also want the specializations to all end up in the same file for end-user documentation. I discussed this in the bug.

Name mangling less than ideal, but it's an established part of C++. I don't know how important it is that file names be completely readable. Lots of web URLs are not, and have little impact on users. Mangled names are also commonly used in doxygen's HTML generation, so there is a precedent.

Also, I'm not sure I follow your argument that file names should be readable, but you're willing to use a SHA1 as a file name? Mangled names may not be completely intuitive, but you can quickly figure out how to read them, where as you cannot do so with a SHA.

My questions about this pertain to a more unified solution that could work across generators. Do you think that if we just substituted the mangled names, we would no longer have the file corruption? From your description of the underlying problem, I would assume "yes", but I'd like your opinion. We can ignore the question about whether they should be grouped under a single file for now, as I view that as a different, but related issue. I haven't' had time to read the issue on github yet, but will soon.

clang-tools-extra/test/clang-doc/single-file-public.cpp

9

clang-tools-extra/test/clang-doc/single-file.cpp

14

I don't expect to fix these, since they're not part of your change, but this is a more concise way to do the same check for a USR using the FileCheck regular expressions. I doubt you need it, but just in case.

[brettw]

clang-tools-extra/clang-doc/HTMLGenerator.cpp

890–892

I would assume mangled names would resolve the template override issue in my particular use-case but I don't think this is a good way to go:

• I assume you could apply the name mangling algorithm to the record name, but mangling type names is weird to me because this doesn't correspond to anything the compiler generates since type names aren't linked, only their contribution to function and static variable names.
• I'm not using clang-doc this way, but the design seems to be with the libtooling stuff that you would apply it to your repository. By default, this will cross executable and shared library boundaries which can cause name collisions even in the absence of ODR violations in the linked binaries.
• Although you're not supposed to write ODR violations, these do exist in real code and I think clang-doc should not generate corrupt output if it can be easily avoided.

clang-tools-extra/test/clang-doc/single-file.cpp

14

I fixed it anyway, I actually found this confusing because I thought there must be some reason why it was expressed this way.

[thakis]

This breaks tests on mac:
http://45.33.8.238/macm1/49211/step_8.txt

(and possibly win; that's still cycling.)

Please take a look and revert for now if it takes a while to fix.

(Also, 👋 Mr W!)

[paulkirth]

@thakis thanks for the report. I've reverted for now. @brettw can you take a look? I'm a bit surprised that passed on the Linux and windows bot, but failed on Mac.

[thakis]

In D138073#3944730, @paulkirth wrote:

@thakis thanks for the report. I've reverted for now. @brettw can you take a look? I'm a bit surprised that passed on the Linux and windows bot, but failed on Mac.

Probably a BSD find vs GNU find issue.

[paulkirth]

clang-tools-extra/test/clang-doc/single-file-public.cpp

9

since this failed, maybe:

// RUN: find %t/docs -regex ".*/[0-9A-F]*.yaml" -exec cat {} \; | FileCheck %s

That works locally for me on both mac and linux. you lose the nice check for 40 chars, but at least it functions. if its very important, you can always have a single run line that does ls and grep.

FYI: the BSD find accepts the failing regex w/o -regextype sed. It's unfortunate that GNU find doesn't have the same default regex, and doesn't accept that version w/o the -regextype being set.

clang-tools-extra/clang-doc/Generators.h

 namespace doc {
 
 // Abstract base class for generators.
 // This is expected to be implemented and exposed via the GeneratorRegistry.
 class Generator {
 public:
   virtual ~Generator() = default;
 
-  // Write out the decl info in the specified format.
-  virtual llvm::Error generateDocForInfo(Info *I, llvm::raw_ostream &OS,
-                                         const ClangDocContext &CDCtx) = 0;
+  // Write out the decl info for the objects in the given map in the specified
+  // format.
+  virtual llvm::Error
+  generateDocs(StringRef RootDir,
+               llvm::StringMap<std::unique_ptr<doc::Info>> Infos,
+
   // This function writes a file with the index previously constructed.
   // It can be overwritten by any of the inherited generators.
   // If the override method wants to run this it should call
   // Generator::createResources(CDCtx);
   virtual llvm::Error createResources(ClangDocContext &CDCtx);
 
+  // Write out one specific decl info to the destination stream.
+  virtual llvm::Error generateDocForInfo(Info *I, llvm::raw_ostream &OS,
+                                         const ClangDocContext &CDCtx) = 0;
+
   static void addInfoToIndex(Index &Idx, const doc::Info *Info);
 };
 
 typedef llvm::Registry<Generator> GeneratorRegistry;
 
 llvm::Expected<std::unique_ptr<Generator>>
 findGeneratorByName(llvm::StringRef Format);
 
 std::string getTagType(TagTypeKind AS);
 
 } // namespace doc
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANG_DOC_GENERATOR_H

clang-tools-extra/clang-doc/HTMLGenerator.cpp

 //===-- HTMLGenerator.cpp - HTML Generator ----------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Generators.h"
 #include "Representation.h"
 #include "clang/Basic/Version.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/StringSet.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/JSON.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
 #include <string>
 
 using namespace llvm;
 
genHTML(const TypedefInfo &I, const ClangDocContext &CDCtx,
   return {};
 }
 
 /// Generator for HTML documentation.
 class HTMLGenerator : public Generator {
 public:
   static const char *Format;
 
-  llvm::Error generateDocForInfo(Info *I, llvm::raw_ostream &OS,
-                                 const ClangDocContext &CDCtx) override;
+  llvm::Error generateDocs(StringRef RootDir,
+                           llvm::StringMap<std::unique_ptr<doc::Info>> Infos,
   llvm::Error createResources(ClangDocContext &CDCtx) override;
+  llvm::Error generateDocForInfo(Info *I, llvm::raw_ostream &OS,
+                                 const ClangDocContext &CDCtx) override;
 };
 
 const char *HTMLGenerator::Format = "html";
 
+llvm::Error
+HTMLGenerator::generateDocs(StringRef RootDir,
+                            llvm::StringMap<std::unique_ptr<doc::Info>> Infos,
+                            const ClangDocContext &CDCtx) {
+  // Track which directories we already tried to create.
+  llvm::StringSet<> CreatedDirs;
+
+  // Collect all output by file name and create the nexessary directories.
+  llvm::StringMap<std::vector<doc::Info *>> FileToInfos;
+  for (const auto &Group : Infos) {
+    doc::Info *Info = Group.getValue().get();
+
+    llvm::SmallString<128> Path;
+    llvm::sys::path::native(RootDir, Path);
+    llvm::sys::path::append(Path, Info->getRelativeFilePath(""));
+    if (CreatedDirs.find(Path) == CreatedDirs.end()) {
+      if (std::error_code Err = llvm::sys::fs::create_directories(Path);
+          Err != std::error_code()) {
+        return llvm::createStringError(Err, "Failed to create directory '%s'.",
+                                       Path.c_str());
+      }
+      CreatedDirs.insert(Path);
+    }
+
+    llvm::sys::path::append(Path, Info->getFileBaseName() + ".html");
+    FileToInfos[Path].push_back(Info);
+  }
+
+  for (const auto &Group : FileToInfos) {
+    std::error_code FileErr;
+    llvm::raw_fd_ostream InfoOS(Group.getKey(), FileErr,
+                                llvm::sys::fs::OF_None);
+    if (FileErr) {
+      return llvm::createStringError(FileErr, "Error opening file '%s'",
+                                     Group.getKey().str().c_str());
+    }
+
+    // TODO: https://github.com/llvm/llvm-project/issues/59073
+    // If there are multiple Infos for this file name (for example, template
+    // specializations), this will generate multiple complete web pages (with
+    // <DOCTYPE> and <title>, etc.) concatenated together. This generator needs
+    // some refactoring to be able to output the headers separately from the
+    // contents.
+    for (const auto &Info : Group.getValue()) {
+      if (llvm::Error Err = generateDocForInfo(Info, InfoOS, CDCtx)) {
+        return Err;
+      }
+    }
+  }
+
+  return llvm::Error::success();
+}
+
 llvm::Error HTMLGenerator::generateDocForInfo(Info *I, llvm::raw_ostream &OS,
                                               const ClangDocContext &CDCtx) {
   std::string InfoTitle;
   std::vector<std::unique_ptr<TagNode>> MainContentNodes;
   Index InfoIndex;
   switch (I->IT) {
   case InfoType::IT_namespace:
     MainContentNodes = genHTML(*static_cast<clang::doc::NamespaceInfo *>(I),

clang-tools-extra/clang-doc/MDGenerator.cpp

if (!C.Children.empty()) {
       OS << "* " << Type << ": [" << C.Name << "](";
       if (!C.Path.empty())
         OS << C.Path << "/";
       OS << C.Name << ")\n";
     }
   }
   return llvm::Error::success();
 }
+
 /// Generator for Markdown documentation.
 class MDGenerator : public Generator {
 public:
   static const char *Format;
 
-  llvm::Error generateDocForInfo(Info *I, llvm::raw_ostream &OS,
-                                 const ClangDocContext &CDCtx) override;
+  llvm::Error generateDocs(StringRef RootDir,
+                           llvm::StringMap<std::unique_ptr<doc::Info>> Infos,
   llvm::Error createResources(ClangDocContext &CDCtx) override;
+  llvm::Error generateDocForInfo(Info *I, llvm::raw_ostream &OS,
+                                 const ClangDocContext &CDCtx) override;
 };
 
 const char *MDGenerator::Format = "md";
 
+llvm::Error
+MDGenerator::generateDocs(StringRef RootDir,
+                          llvm::StringMap<std::unique_ptr<doc::Info>> Infos,
+                          const ClangDocContext &CDCtx) {
+  // Track which directories we already tried to create.
+  llvm::StringSet<> CreatedDirs;
+
+  // Collect all output by file name and create the necessary directories.
+  llvm::StringMap<std::vector<doc::Info *>> FileToInfos;
+  for (const auto &Group : Infos) {
+    doc::Info *Info = Group.getValue().get();
+
+    llvm::SmallString<128> Path;
+    llvm::sys::path::native(RootDir, Path);
+    llvm::sys::path::append(Path, Info->getRelativeFilePath(""));
+    if (CreatedDirs.find(Path) == CreatedDirs.end()) {
+      if (std::error_code Err = llvm::sys::fs::create_directories(Path);
+          Err != std::error_code()) {
+        return llvm::createStringError(Err, "Failed to create directory '%s'.",
+                                       Path.c_str());
+      }
+      CreatedDirs.insert(Path);
+    }
+
+    llvm::sys::path::append(Path, Info->getFileBaseName() + ".md");
+    FileToInfos[Path].push_back(Info);
+  }
+
+  for (const auto &Group : FileToInfos) {
+    std::error_code FileErr;
+    llvm::raw_fd_ostream InfoOS(Group.getKey(), FileErr,
+                                llvm::sys::fs::OF_None);
+    if (FileErr) {
+      return llvm::createStringError(FileErr, "Error opening file '%s'",
+                                     Group.getKey().str().c_str());
+    }
+
+    for (const auto &Info : Group.getValue()) {
+      if (llvm::Error Err = generateDocForInfo(Info, InfoOS, CDCtx)) {
+        return Err;
+      }
+    }
+  }
+
+  return llvm::Error::success();
+}
+
 llvm::Error MDGenerator::generateDocForInfo(Info *I, llvm::raw_ostream &OS,
                                             const ClangDocContext &CDCtx) {
   switch (I->IT) {
   case InfoType::IT_namespace:
     genMarkdown(CDCtx, *static_cast<clang::doc::NamespaceInfo *>(I), OS);
     break;
   case InfoType::IT_record:
     genMarkdown(CDCtx, *static_cast<clang::doc::RecordInfo *>(I), OS);
     break;
   case InfoType::IT_enum:
     genMarkdown(CDCtx, *static_cast<clang::doc::EnumInfo *>(I), OS);
     break;
   case InfoType::IT_function:
     genMarkdown(CDCtx, *static_cast<clang::doc::FunctionInfo *>(I), OS);
     break;
   case InfoType::IT_typedef:
     genMarkdown(CDCtx, *static_cast<clang::doc::TypedefInfo *>(I), OS);
     break;
   case InfoType::IT_default:
     return createStringError(llvm::inconvertibleErrorCode(),
                              "unexpected InfoType");
   }

clang-tools-extra/clang-doc/YAMLGenerator.cpp

 namespace clang {
 namespace doc {
 
 /// Generator for YAML documentation.
 class YAMLGenerator : public Generator {
 public:
   static const char *Format;
 
+  llvm::Error generateDocs(StringRef RootDir,
+                           llvm::StringMap<std::unique_ptr<doc::Info>> Infos,
+                           const ClangDocContext &CDCtx) override;
   llvm::Error generateDocForInfo(Info *I, llvm::raw_ostream &OS,
                                  const ClangDocContext &CDCtx) override;
 };
 
 const char *YAMLGenerator::Format = "yaml";
 
+llvm::Error
+YAMLGenerator::generateDocs(StringRef RootDir,
+                            llvm::StringMap<std::unique_ptr<doc::Info>> Infos,
+                            const ClangDocContext &CDCtx) {
+  for (const auto &Group : Infos) {
+    doc::Info *Info = Group.getValue().get();
+
+    // Output file names according to the USR except the global namesapce.
+    // Anonymous namespaces are taken care of in serialization, so here we can
+    // safely assume an unnamed namespace is the global one.
+    llvm::SmallString<128> Path;
+    llvm::sys::path::native(RootDir, Path);
+    if (Info->IT == InfoType::IT_namespace && Info->Name.empty()) {
+      llvm::sys::path::append(Path, "index.yaml");
+    } else {
+      llvm::sys::path::append(Path, Group.getKey() + ".yaml");
+    }
+
+    std::error_code FileErr;
+    llvm::raw_fd_ostream InfoOS(Path, FileErr, llvm::sys::fs::OF_None);
+    if (FileErr) {
+      return llvm::createStringError(FileErr, "Error opening file '%s'",
+                                     Path.c_str());
+    }
+
+    if (llvm::Error Err = generateDocForInfo(Info, InfoOS, CDCtx)) {
+      return Err;
+    }
+  }
+
+  return llvm::Error::success();
+}
+
 llvm::Error YAMLGenerator::generateDocForInfo(Info *I, llvm::raw_ostream &OS,
                                               const ClangDocContext &CDCtx) {
   llvm::yaml::Output InfoYAML(OS);
   switch (I->IT) {
   case InfoType::IT_namespace:
     InfoYAML << *static_cast<clang::doc::NamespaceInfo *>(I);
     break;
   case InfoType::IT_record:

clang-tools-extra/clang-doc/tool/ClangDocMain.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Mutex.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Process.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Support/raw_ostream.h"
 #include <atomic>
+#include <mutex>
 #include <string>
 
 using namespace clang::ast_matchers;
 using namespace clang::tooling;
 using namespace clang;
 
 static llvm::cl::extrahelp CommonHelp(CommonOptionsParser::HelpMessage);
 static llvm::cl::OptionCategory ClangDocCategory("clang-doc options");

 // can't use the "static" keyword because its address is used for
 // GetMainExecutable (since some platforms don't support taking the
 // address of main, and some platforms can't implement GetMainExecutable
 // without being given the address of a function in the main executable).
 std::string GetExecutablePath(const char *Argv0, void *MainAddr) {
   return llvm::sys::fs::getMainExecutable(Argv0, MainAddr);
 }
 
-bool CreateDirectory(const Twine &DirName, bool ClearDirectory = false) {
-  std::error_code OK;
-  llvm::SmallString<128> DocsRootPath;
-  if (ClearDirectory) {
-    std::error_code RemoveStatus = llvm::sys::fs::remove_directories(DirName);
-    if (RemoveStatus != OK) {
-      llvm::errs() << "Unable to remove existing documentation directory for "
-                   << DirName << ".\n";
-      return true;
-    }
-  }
-  std::error_code DirectoryStatus = llvm::sys::fs::create_directories(DirName);
-  if (DirectoryStatus != OK) {
-    llvm::errs() << "Unable to create documentation directories.\n";
-    return true;
-  }
-  return false;
-}
-
-// A function to extract the appropriate file name for a given info's
-// documentation. The path returned is a composite of the output directory, the
-// info's relative path and name and the extension. The relative path should
-// have been constructed in the serialization phase.
-//
-// Example: Given the below, the <ext> path for class C will be
-// <root>/A/B/C.<ext>
-//
-// namespace A {
-// namespace B {
-//
-// class C {};
-//
-// }
-// }
-llvm::Expected<llvm::SmallString<128>> getInfoOutputFile(StringRef Root,
-                                                         StringRef RelativePath,
-                                                         StringRef Name,
-                                                         StringRef Ext) {
-  llvm::SmallString<128> Path;
-  llvm::sys::path::native(Root, Path);
-  llvm::sys::path::append(Path, RelativePath);
-  if (CreateDirectory(Path))
-    return llvm::createStringError(llvm::inconvertibleErrorCode(),
-                                   "failed to create directory");
-  llvm::sys::path::append(Path, Name + Ext);
-  return Path;
-}
-
 int main(int argc, const char **argv) {
   llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
   std::error_code OK;
 
   const char *Overview =
     R"(Generates documentation from source code and comments.
 
 Example usage for files without flags (default):
)";
   llvm::outs() << "Collecting infos...\n";
   llvm::StringMap<std::vector<StringRef>> USRToBitcode;
   Executor->get()->getToolResults()->forEachResult(
       [&](StringRef Key, StringRef Value) {
         auto R = USRToBitcode.try_emplace(Key, std::vector<StringRef>());
         R.first->second.emplace_back(Value);
       });
 
+  // Collects all Infos according to their unique USR value. This map is added
+  // to from the thread pool below and is protected by the USRToInfoMutex.
+  llvm::sys::Mutex USRToInfoMutex;
+  llvm::StringMap<std::unique_ptr<doc::Info>> USRToInfo;
+
   // First reducing phase (reduce all decls into one info per decl).
   llvm::outs() << "Reducing " << USRToBitcode.size() << " infos...\n";
   std::atomic<bool> Error;
   Error = false;
   llvm::sys::Mutex IndexMutex;
   // ExecutorConcurrency is a flag exposed by AllTUsExecution.h
   llvm::ThreadPool Pool(llvm::hardware_concurrency(ExecutorConcurrency));
   for (auto &Group : USRToBitcode) {
Pool.async([&]() {
       }
 
       auto Reduced = doc::mergeInfos(Infos);
       if (!Reduced) {
         llvm::errs() << llvm::toString(Reduced.takeError());
         return;
       }
 
-      doc::Info *I = Reduced.get().get();
-      auto InfoPath =
-          getInfoOutputFile(OutDirectory, I->getRelativeFilePath(""),
-                            I->getFileBaseName(), "." + Format);
-      if (!InfoPath) {
-        llvm::errs() << toString(InfoPath.takeError()) << "\n";
-        Error = true;
-        return;
-      }
-      std::error_code FileErr;
-      llvm::raw_fd_ostream InfoOS(InfoPath.get(), FileErr,
-                                  llvm::sys::fs::OF_None);
-      if (FileErr) {
-        llvm::errs() << "Error opening info file " << InfoPath.get() << ": "
-                     << FileErr.message() << "\n";
-        return;
-      }
-
-      IndexMutex.lock();
       // Add a reference to this Info in the Index
-      clang::doc::Generator::addInfoToIndex(CDCtx.Idx, I);
-      IndexMutex.unlock();
+      {
+        std::lock_guard Guard(IndexMutex);
+        clang::doc::Generator::addInfoToIndex(CDCtx.Idx, Reduced.get().get());
+      }
 
-      if (auto Err = G->get()->generateDocForInfo(I, InfoOS, CDCtx))
-        llvm::errs() << toString(std::move(Err)) << "\n";
+      // Save in the result map (needs a lock due to threaded access).
+      {
+        std::lock_guard Guard(USRToInfoMutex);
+        USRToInfo[Group.getKey()] = std::move(Reduced.get());
+      }
     });
   }
 
   Pool.wait();
 
   if (Error)
     return 1;
 
+  // Ensure the root output directory exists.
+  if (std::error_code Err = llvm::sys::fs::create_directories(OutDirectory);
+      Err != std::error_code()) {
+    llvm::errs() << "Failed to create directory '" << OutDirectory << "'\n";
+    return 1;
+  }
+
+  // Run the generator.
+  llvm::outs() << "Generating docs...\n";
+  if (auto Err =
+          G->get()->generateDocs(OutDirectory, std::move(USRToInfo), CDCtx)) {
+    llvm::errs() << toString(std::move(Err)) << "\n";
+    return 1;
+  }
+
   llvm::outs() << "Generating assets for docs...\n";
   Err = G->get()->createResources(CDCtx);
   if (Err) {
     llvm::errs() << toString(std::move(Err)) << "\n";
     return 1;
   }
 
   return 0;
 }

clang-tools-extra/test/clang-doc/single-file-public.cpp

 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: echo "" > %t/compile_flags.txt
 // RUN: cp "%s" "%t/test.cpp"
 // RUN: clang-doc --doxygen --public --executor=standalone -p %t %t/test.cpp -output=%t/docs
-// RUN: cat %t/docs/GlobalNamespace/Record.yaml | FileCheck %s --check-prefix=CHECK
+//   This produces two files, index.yaml and one for the record named by its USR
+//   (which we don't know in advance). This checks the record file by searching
+//   for a name with a 40-char USR name.
+// RUN: find %t/docs -regex ".*/[0-9A-F]*.yaml" -exec cat {} ";" | FileCheck %s --check-prefix=CHECK
 // RUN: rm -rf %t
 
 class Record {
 private:
   void function_private();
 
 public:
   void function_public();
 };
 
 void Record::function_private() {}
 
 void Record::function_public() {}
 
 // CHECK: ---
-// CHECK-NEXT: USR:             '{{[0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z]}}'
+// CHECK-NEXT: USR:             '{{([0-9A-F]{40})}}'
 // CHECK-NEXT: Name:            'Record'
 // CHECK-NEXT: Path:            'GlobalNamespace'
 // CHECK-NEXT: Namespace:
 // CHECK-NEXT:   - Type:             Namespace
 // CHECK-NEXT:     Name:             'GlobalNamespace'
 // CHECK-NEXT: DefLocation:
 // CHECK-NEXT:   LineNumber:      [[@LINE-20]]
 // CHECK-NEXT:   Filename:        '{{.*}}'
 // CHECK-NEXT: TagType:         Class
 // CHECK-NEXT: ChildFunctions:
-// CHECK-NEXT:   - USR:             '{{[0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z]}}'
+// CHECK-NEXT:   - USR:             '{{([0-9A-F]{40})}}'
 // CHECK-NEXT:     Name:            'function_public'
 // CHECK-NEXT:     Namespace:
 // CHECK-NEXT:       - Type:            Record
 // CHECK-NEXT:         Name:            'Record'
-// CHECK-NEXT:         USR:             '{{[0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z]}}'
+// CHECK-NEXT:         USR:             '{{([0-9A-F]{40})}}'
 // CHECK-NEXT:       - Type:            Namespace
 // CHECK-NEXT:         Name:            'GlobalNamespace'
 // CHECK-NEXT:     DefLocation:
 // CHECK-NEXT:         LineNumber:      [[@LINE-23]]
 // CHECK-NEXT:         Filename:        '{{.*}}'
 // CHECK-NEXT:     Location:
 // CHECK-NEXT:       - LineNumber:      [[@LINE-31]]
 // CHECK-NEXT:         Filename:        '{{.*}}'
 // CHECK-NEXT:     IsMethod:        true
 // CHECK-NEXT:     Parent:
 // CHECK-NEXT:         Type:            Record
 // CHECK-NEXT:         Name:            'Record'
-// CHECK-NEXT:         USR:             '{{[0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z]}}'
+// CHECK-NEXT:         USR:             '{{([0-9A-F]{40})}}'
 // CHECK-NEXT:     ReturnType:
 // CHECK-NEXT:       Type:
 // CHECK-NEXT:         Name:            'void'
 // CHECK-NEXT:     Access:      Public
 // CHECK-NEXT: ...

clang-tools-extra/test/clang-doc/single-file.cpp

 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: echo "" > %t/compile_flags.txt
 // RUN: cp "%s" "%t/test.cpp"
 // RUN: clang-doc --doxygen --executor=standalone -p %t %t/test.cpp -output=%t/docs
-// RUN: cat %t/docs/GlobalNamespace/index.yaml | FileCheck %s --check-prefix=CHECK
+// RUN: cat %t/docs/index.yaml | FileCheck %s --check-prefix=CHECK
 // RUN: rm -rf %t
 
 void function(int x);
 
 void function(int x) {}
 
 // CHECK: ---
 // CHECK-NEXT: USR:             '{{[0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z]}}'
 // CHECK-NEXT: ChildFunctions:
 // CHECK-NEXT:   - USR:             '{{[0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z]}}'
 // CHECK-NEXT:    Name:            'function'
 // CHECK-NEXT:    DefLocation:
 // CHECK-NEXT:      LineNumber:      [[@LINE-8]]
 // CHECK-NEXT:      Filename:        '{{.*}}
 // CHECK-NEXT:    Location:
 // CHECK-NEXT:      - LineNumber:      [[@LINE-13]]