[docs] Add documentation on using the new pass manager

[aeubanks]

Summary

And clarify in the "writing a pass" docs that both the legacy and new
PMs are being used for the codegen/optimization pipelines.

[aeubanks]

change writing a pass docs

[ychen]

All looks very good to me. NewPassManager.rst is a great place to add more new pass manager details in the future, especially the trickier parts like analysis lifetime/invalidation, etc. Please let @asbirlea ack also.

[Matt]

llvm/docs/NewPassManager.rst

24

Typo: *a* function pass.

[aeubanks]

typo

[asbirlea]

Thank you for adding this, it was long overdue and it is well written!
LGTM.

llvm/docs/NewPassManager.rst

+==========================
+Using the New Pass Manager
+==========================
+
+.. contents::
+    :local:
+
+Adding Passes to a Pass Manager
+===============================
+
+For how to write a new PM pass, see :doc:`this page <WritingAnLLVMNewPMPass>`.
+
+To add a pass to a new PM pass manager, the important thing is to match the
+pass type and the pass manager type. For example, a ``FunctionPassManager``
+can only contain function passes:
+
+.. code-block:: c++
+
+  FunctionPassManager FPM;
+  // InstSimplifyPass is a function pass
+  FPM.addPass(InstSimplifyPass());
+
+If you want add a loop pass that runs on all loops in a function to a
+``FunctionPassManager``, the loop pass must be wrapped in an function pass
+adaptor that goes through all the loops in the function and runs the loop
+pass on each one.
+
+.. code-block:: c++
+
+  FunctionPassManager FPM;
+  // LoopRotatePass is a loop pass
+  FPM.addPass(createFunctionToLoopPassAdaptor(LoopRotatePass()));
+
+The IR hierarchy in terms of the new PM is Module -> (CGSCC ->) Function ->
+Loop, where going through a CGSCC is optional.
+
+.. code-block:: c++
+
+  FunctionPassManager FPM;
+  // loop -> function
+  FPM.addPass(createFunctionToLoopPassAdaptor(LoopFooPass()));
+
+  CGSCCPassManager CGPM;
+  // loop -> function -> cgscc
+  CGPM.addPass(createCGSCCToFunctionPassAdaptor(createFunctionToLoopPassAdaptor(LoopFooPass())));
+  // function -> cgscc
+  CGPM.addPass(createCGSCCToFunctionPassAdaptor(FunctionFooPass()));
+
+  ModulePassManager MPM;
+  // loop -> function -> module
+  MPM.addPass(createModuleToFunctionPassAdaptor(createFunctionToLoopPassAdaptor(LoopFooPass())));
+  // function -> module
+  MPM.addPass(createModuleToFunctionPassAdaptor(FunctionFooPass()));
+
+  // loop -> function -> cgscc -> module
+  MPM.addPass(createModuleToCGSCCPassAdaptor(createCGSCCToFunctionPassAdaptor(createFunctionToLoopPassAdaptor(LoopFooPass()))));
+  // function -> cgscc -> module
+  MPM.addPass(createModuleToCGSCCPassAdaptor(createCGSCCToFunctionPassAdaptor(FunctionFooPass())));
+
+
+A pass manager of a specific IR unit is also a pass of that kind. For
+example, a ``FunctionPassManager`` is a function pass, meaning it can be
+added to a ``ModulePassManager``:
+
+.. code-block:: c++
+
+  ModulePassManager MPM;
+
+  FunctionPassManager FPM;
+  // InstSimplifyPass is a function pass
+  FPM.addPass(InstSimplifyPass());
+
+  MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
+
+Generally you want to group CGSCC/function/loop passes together in a pass
+manager, as opposed to adding adaptors for each pass to the containing upper
+level pass manager. For example,
+
+.. code-block:: c++
+
+  ModulePassManager MPM;
+  MPM.addPass(createModuleToFunctionPassAdaptor(FunctionPass1()));
+  MPM.addPass(createModuleToFunctionPassAdaptor(FunctionPass2()));
+  MPM.run();
+
+will run ``FunctionPass1`` on each function in a module, then run
+``FunctionPass2`` on each function in the module. In contrast,
+
+.. code-block:: c++
+
+  ModulePassManager MPM;
+
+  FunctionPassManager FPM;
+  FPM.addPass(FunctionPass1());
+  FPM.addPass(FunctionPass2());
+
+  MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
+
+will run ``FunctionPass1`` and ``FunctionPass2`` on the first function in a
+module, then run both passes on the second function in the module, and so on.
+This is better for cache locality around LLVM data structures. This similarly
+applies for the other IR types, and in some cases can even affect the quality
+of optimization. For example, running all loop passes on a loop may cause a
+later loop to be able to be optimized more than if each loop pass were run
+separately.
+
+Inserting Passes into Default Pipelines
+=======================================
+
+Rather than manually adding passes to a pass manager, the typical way of
+creating a pass manager is to use a ``PassBuilder`` and call something like
+``PassBuilder::buildPerModuleDefaultPipeline()`` which creates a typical
+pipeline for a given optimization level.
+
+Sometimes either frontends or backends will want to inject passes into the
+pipeline. For example, frontends may want to add instrumentation, and target
+backends may want to add passes that lower custom intrinsics. For these
+cases, ``PassBuilder`` exposes callbacks that allow injecting passes into
+certain parts of the pipeline. For example,
+
+.. code-block:: c++
+
+  PassBuilder PB;
+  PB.registerPipelineStartEPCallback([&](ModulePassManager &MPM,
+                                         PassBuilder::OptimizationLevel Level) {
+      MPM.addPass(FooPass());
+  };
+
+will add ``FooPass`` near the very beginning of the pipeline for pass
+managers created by that ``PassBuilder``. See the documentation for
+``PassBuilder`` for the various places that passes can be added.
+
+If a ``PassBuilder`` has a corresponding ``TargetMachine`` for a backend, it
+will call ``TargetMachine::registerPassBuilderCallbacks()`` to allow the
+backend to inject passes into the pipeline. This is equivalent to the legacy
+PM's ``TargetMachine::adjustPassManager()``.
+
+Clang's ``BackendUtil.cpp`` shows examples of a frontend adding (mostly
+sanitizer) passes to various parts of the pipeline.
+``AMDGPUTargetMachine::registerPassBuilderCallbacks()`` is an example of a
+backend adding passes to various parts of the pipeline.
+
+Status of the New and Legacy Pass Managers
+==========================================
+
+LLVM currently contains two pass managers, the legacy PM and the new PM. The
+optimization pipeline (aka the middle-end) works with both the legacy PM and
+the new PM, whereas the backend target-dependent code generation only works
+with the legacy PM.
+
+For the optimization pipeline, the new PM is the default PM. The legacy PM is
+available for the optimization pipeline either by setting the CMake flag
+``-DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=OFF`` when building LLVM, or by
+various compiler/linker flags, e.g. ``-flegacy-pass-manager`` for ``clang``.
+
+There will be efforts to deprecate and remove the legacy PM for the
+optimization pipeline in the future.
+
+Some IR passes are considered part of the backend codegen pipeline even if
+they are LLVM IR passes (whereas all MIR passes are codegen passes). This
+includes anything added via ``TargetPassConfig`` hooks, e.g.
+``TargetPassConfig::addCodeGenPrepare()``. As mentioned before, passes added
+in ``TargetMachine::adjustPassManager()`` are part of the optimization
+pipeline, and should have a corresponding line in
+``TargetMachine::registerPassBuilderCallbacks()``.
+
+Currently there are efforts to make the codegen pipeline work with the new
+PM.

llvm/docs/UserGuides.rst

.. toctree::
    LinkTimeOptimization
    LoopTerminology
    MarkdownQuickstartTemplate
    MemorySSA
    MergeFunctions
    MCJITDesignAndImplementation
    ORCv2
    JITLink
+   NewPassManager
    NVPTXUsage
    Phabricator
    Passes
    ReportingGuide
    Remarks
    SourceLevelDebugging
    StackSafetyAnalysis
    SupportLibrary

llvm/docs/WritingAnLLVMNewPMPass.rst

 method which returns a ``PreservedAnalyses`` and takes in some unit of IR
 along with an analysis manager. For example, a function pass would have a
 ``PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);`` method.
 
 We start by showing you how to construct a pass, from setting up the build,
 creating the pass, to executing and testing it. Looking at existing passes is
 always a great way to learn details.
 
+.. warning::
+  This document deals with the new pass manager. LLVM uses the legacy pass
+  manager for the codegen pipeline. For more details, see
+  :doc:`WritingAnLLVMPass` and :doc:`NewPassManager`.
+
 Quick Start --- Writing hello world
 ===================================
 
 Here we describe how to write the "hello world" of passes. The "HelloWorld"
 pass is designed to simply print out the name of non-external functions that
 exist in the program being compiled. It does not modify the program at all,
 it just inspects it.
 

llvm/docs/WritingAnLLVMPass.rst

 schedules passes to run in an efficient way based on the constraints that your
 pass meets (which are indicated by which class they derive from).
 
 We start by showing you how to construct a pass, everything from setting up the
 code, to compiling, loading, and executing it.  After the basics are down, more
 advanced features are discussed.
 
 .. warning::
-  This document deals with the legacy pass manager. LLVM now uses the new
-  pass manager by default, which has its own way of defining passes. For more
-  details, see :doc:`WritingAnLLVMNewPMPass`. To use the legacy pass manager
-  with ``opt``, pass the ``-enable-new-pm=0`` flag to all ``opt``
-  invocations.
+  This document deals with the legacy pass manager. LLVM uses the new pass
+  manager by default for the optimization pipeline (the codegen pipeline is
+  still using the legacy pass manager), which has its own way of defining
+  passes. For more details, see :doc:`WritingAnLLVMNewPMPass` and
+  :doc:`NewPassManager`. To use the legacy pass manager with ``opt``, pass
+  the ``-enable-new-pm=0`` flag to all ``opt`` invocations.
 
 Quick Start --- Writing hello world
 ===================================
 
 Here we describe how to write the "hello world" of passes.  The "Hello" pass is
 designed to simply print out the name of non-external functions that exist in
 the program being compiled.  It does not modify the program at all, it just
 inspects it.  The source code and files for this pass are available in the LLVM