[PGO] clang part of change for context-sensitive PGO.

[xur]

Summary

Add new options for context-sensitive PGO.

Refer to https://reviews.llvm.org/D54175

[xur]

update the patch to work with new version in
https://reviews.llvm.org/D54175

[davidxl]

Can you add a summary of this change, i.e., the driver option processing flow for csPGO.

Test cases are also missing.

[xur]

Here is the summary of CSPGO options workflow:

(1) CSPGO instrumentation.
We add new driver options of fcs-profile-generate fcs-profile-generate=. They are typically used with -fprofile-use (but not required).
They will not work together with -fprofile-generate (They could, but not do not make sense. There is a hard error for this).

We also add a new instrument kind (csllvm) for cc1 option fprofile-instrument. This is corresponding to a new CodeGen option: CodeGenOptions::ProfileCSIRInstr.

The driver option is parsed and translated to cc1 option in addPGOAndCoverageFlags().

cc1 option is converted to CodeGenOpts in setPGOInstrumentor() (CompilerInvocation.cpp).

CodeGenOpts is passed into llvm pass manager in BackendUtil.cpp:
(1) PMBuilder options for legacy passmanager in EmitAssemblyHelper::CreatePasses().
(2) PGOOptions struct for new passmanager in EmitAssemblyHelper::EmitAssemblyWithNewPassManager().

(2) PGO use
We will set CodeGenOptions::ProfileCSIRInstr in setPGOUseInstrumentor() (CompilerInvocation.cpp) when processing -fprofile-use option.

CodeGenOpts is handled similarly as in CSPGO instrumentation.

(3) lto/thinlto
We need to pass CSPGO instrumentation and PGO profile to the back-end compilation. This is through some new options to -plugin-opt (in tools::AddGoldPlugin()). Note:

• For CSPGO instrumentation, it is guarded by CSPGOGenerateArg.
• For PGO use, I unconditionally add -plugin-opt=cs-profile-path=<> because I don't want to open profile. I could read the profile and check if there is CS counters. But that needs to open the profile file.

The change to lib/CodeGen/CodeGenModule.cpp is for llvm profile summary change.

[xur]

Add a testcase to test the options

[davidxl]

include/clang/Frontend/CodeGenOptions.h

330 ↗

(On Diff #175926)

The comment is not precise.

lib/CodeGen/BackendUtil.cpp

692

what is going to happen if user uses a merged data with CS profile in profile-use and also specifies CS prof-gen?

959

Can you explain the change at this line?

965

Explain the change at this line.

978

should emit error message instead.

1260

please add +tejohnson for reviewing thinlto related pieces.

lib/Frontend/CompilerInvocation.cpp

615

CSIRInstr and IRInstr are not exclusive to each other, so why use if-then-else here?

[xur]

include/clang/Frontend/CodeGenOptions.h

330 ↗

(On Diff #175926)

Fixed

lib/CodeGen/BackendUtil.cpp

692

Good question. As of this patch, the behavior is different for the legacy pass manager and the new pass manager.
For the legacy pass manager (the code here), we will call normal PGOUsePass, CSPGOGenPass, and CSPGOUsePass.
For the new pass manager, we will report error of
Assertion `!CodeGenOpts.hasProfileCSIRUse() && "Cannot have both CSProfileUse and CSProfileGen at the same time"' failed.

The behavior for legacy pass manager is wrong. I will fix.

959

My earlier version will have two new booleans (bool RunCSProfileGen and bool RunCSProfileUse).
Here I combined the following 3 field in PGOOptions struct

bool RunProfileGen;
bool RunCSProfileGen;
bool RunCSProfileUse;

into an enum. There three are mutually exclusive.

lib/Frontend/CompilerInvocation.cpp

615

For CodeGenOptions kind, CSIRInstr and IRinstr are exclusive -- there is just one enum for the kind.

[xur]

Integrated David's review comments. Also add a test to test the error when both CSInstrumentGen and CSInstrumentUse on.

[tejohnson]

lib/CodeGen/BackendUtil.cpp

1260

This part looks fine (haven't examined the rest in detail). Please add a test. There are some existing ones for the ThinLTO backend via clang in test/CodeGen/thinlto-* that you can look at for examples.

[xur]

Added a thinlto test case and fixed a few issues.

[xur]

Updated the patch to sync with the updates in https://reviews.llvm.org/D54175

[lebedev.ri]

Clang reviews should mainly go to cfe-commits, not llvm-commits. (all that will happen automagically if one sets the correct repo for the differential..)

[xur]

Update the patch to sync with https://reviews.llvm.org/D54175

[xur]

Updated the patch to sync with D54175

[tejohnson]

LGTM except for place noted below where I disagree with a change made earlier. Will let @davidxl chime in if he disagrees with me or has any other comments.

lib/CodeGen/BackendUtil.cpp

978

I disagree with David's comment here. All the error checking should (and is) being done in Clang.cpp where the options are being processed. These and the places above for the old PM should just be asserts.

[davidxl]

Please add changes to option documentation.

[xur]

Integrated Teresa's suggestion to change the err() to assert.
Added documentation change suggested by David.

[davidxl]

docs/UsersManual.rst

1792 ↗

(On Diff #188955)

Please add a use example to show the typical flow with this option (both gen and use phases -- including llvm-profdata step).

[xur]

Added usage example to UserManual.rst suggested by David.

[davidxl]

lgtm

include/clang/Basic/CodeGenOptions.h

enum StructReturnConventionKind {
     SRCK_InRegs    // Small structs in registers (-freg-struct-return).
   };
 
   enum ProfileInstrKind {
     ProfileNone,       // Profile instrumentation is turned off.
     ProfileClangInstr, // Clang instrumentation to generate execution counts
                        // to use with PGO.
     ProfileIRInstr,    // IR level PGO instrumentation in LLVM.
+    ProfileCSIRInstr, // IR level PGO context sensitive instrumentation in LLVM.
   };
 
   enum EmbedBitcodeKind {
     Embed_Off,      // No embedded bitcode.
     Embed_All,      // Embed both bitcode and commandline in the output.
     Embed_Bitcode,  // Embed just the bitcode in the output.
     Embed_Marker    // Embed a marker as a placeholder for bitcode.
   };
public:
   std::string TrapFuncName;
 
   /// A list of dependent libraries.
   std::vector<std::string> DependentLibraries;
 
   /// A list of linker options to embed in the object file.
   std::vector<std::string> LinkerOptions;
 
-  /// Name of the profile file to use as output for -fprofile-instr-generate
-  /// and -fprofile-generate.
+  /// Name of the profile file to use as output for -fprofile-instr-generate,
+  /// -fprofile-generate, and -fcs-profile-generate.
   std::string InstrProfileOutput;
 
   /// Name of the profile file to use with -fprofile-sample-use.
   std::string SampleProfileFile;
 
   /// Name of the profile file to use as input for -fprofile-instr-use
   std::string ProfileInstrumentUsePath;
 
bool hasProfileClangInstr() const {
     return getProfileInstr() == ProfileClangInstr;
   }
 
   /// Check if IR level profile instrumentation is on.
   bool hasProfileIRInstr() const {
     return getProfileInstr() == ProfileIRInstr;
   }
 
+  /// Check if CS IR level profile instrumentation is on.
+  bool hasProfileCSIRInstr() const {
+    return getProfileInstr() == ProfileCSIRInstr;
+  }
+
   /// Check if Clang profile use is on.
   bool hasProfileClangUse() const {
     return getProfileUse() == ProfileClangInstr;
   }
 
   /// Check if IR level profile use is on.
   bool hasProfileIRUse() const {
-    return getProfileUse() == ProfileIRInstr;
+    return getProfileUse() == ProfileIRInstr ||
+           getProfileUse() == ProfileCSIRInstr;
   }
 
+  /// Check if CSIR profile use is on.
+  bool hasProfileCSIRUse() const { return getProfileUse() == ProfileCSIRInstr; }
 };
 
 }  // end namespace clang
 
 #endif

include/clang/Driver/Options.td

def fno_coverage_mapping : Flag<["-"], "fno-coverage-mapping">,
     Group<f_Group>, Flags<[DriverOption, CoreOption]>,
     HelpText<"Disable code coverage analysis">;
 def fprofile_generate : Flag<["-"], "fprofile-generate">,
     Group<f_Group>, Flags<[DriverOption]>,
     HelpText<"Generate instrumented code to collect execution counts into default.profraw (overridden by LLVM_PROFILE_FILE env var)">;
 def fprofile_generate_EQ : Joined<["-"], "fprofile-generate=">,
     Group<f_Group>, Flags<[DriverOption]>, MetaVarName<"<directory>">,
     HelpText<"Generate instrumented code to collect execution counts into <directory>/default.profraw (overridden by LLVM_PROFILE_FILE env var)">;
+def fcs_profile_generate : Flag<["-"], "fcs-profile-generate">,
+    Group<f_Group>, Flags<[DriverOption]>,
+    HelpText<"Generate instrumented code to collect context sensitive execution counts into default.profraw (overridden by LLVM_PROFILE_FILE env var)">;
+def fcs_profile_generate_EQ : Joined<["-"], "fcs-profile-generate=">,
+    Group<f_Group>, Flags<[DriverOption]>, MetaVarName<"<directory>">,
+    HelpText<"Generate instrumented code to collect context sensitive execution counts into <directory>/default.profraw (overridden by LLVM_PROFILE_FILE env var)">;
 def fprofile_use : Flag<["-"], "fprofile-use">, Group<f_Group>,
     Alias<fprofile_instr_use>;
 def fprofile_use_EQ : Joined<["-"], "fprofile-use=">,
     Group<f_Group>, Flags<[DriverOption]>, MetaVarName<"<pathname>">,
     HelpText<"Use instrumentation data for profile-guided optimization. If pathname is a directory, it reads from <pathname>/default.profdata. Otherwise, it reads from file <pathname>.">;
 def fno_profile_instr_generate : Flag<["-"], "fno-profile-instr-generate">,
     Group<f_Group>, Flags<[DriverOption]>,
     HelpText<"Disable generation of profile instrumentation.">;

lib/CodeGen/BackendUtil.cpp

if (CodeGenOpts.hasProfileClangInstr()) {
     Options.InstrProfileOutput = CodeGenOpts.InstrProfileOutput;
 
     // TODO: Surface the option to emit atomic profile counter increments at
     // the driver level.
     Options.Atomic = LangOpts.Sanitize.has(SanitizerKind::Thread);
 
     MPM.add(createInstrProfilingLegacyPass(Options));
   }
+  bool hasIRInstr = false;
   if (CodeGenOpts.hasProfileIRInstr()) {
     PMBuilder.EnablePGOInstrGen = true;
+    hasIRInstr = true;
+  }
+  if (CodeGenOpts.hasProfileCSIRInstr()) {
+    if (CodeGenOpts.hasProfileCSIRUse()) {
+      errs() << "Cannot have both CSProfileUse pass and CSProfileGen pass at "
+                "the same time";
+      return;
+    }
+    if (hasIRInstr) {
+      errs() << "Cannot have both ProfileGen pass and CSProfileGen pass at the "
+                "same time";
+      return;
+    }
+    PMBuilder.EnablePGOCSInstrGen = true;
+    hasIRInstr = true;
+  }
+  if (hasIRInstr) {
     if (!CodeGenOpts.InstrProfileOutput.empty())
       PMBuilder.PGOInstrGen = CodeGenOpts.InstrProfileOutput;
     else
       PMBuilder.PGOInstrGen = DefaultProfileGenName;
   }
-  if (CodeGenOpts.hasProfileIRUse())
+  if (CodeGenOpts.hasProfileIRUse()) {
     PMBuilder.PGOInstrUse = CodeGenOpts.ProfileInstrumentUsePath;
+    PMBuilder.EnablePGOCSInstrUse = CodeGenOpts.hasProfileCSIRUse();
+  }
 
   if (!CodeGenOpts.SampleProfileFile.empty())
     PMBuilder.PGOSampleUse = CodeGenOpts.SampleProfileFile;
 
   PMBuilder.populateFunctionPassManager(FPM);
   PMBuilder.populateModulePassManager(MPM);
 }
 
if (!TM)
     // This will already be diagnosed, just bail.
     return;
   TheModule->setDataLayout(TM->createDataLayout());
 
   Optional<PGOOptions> PGOOpt;
 
   if (CodeGenOpts.hasProfileIRInstr())
     // -fprofile-generate.
-    PGOOpt = PGOOptions(CodeGenOpts.InstrProfileOutput.empty()
-                            ? DefaultProfileGenName
+    PGOOpt = PGOOptions(
+        CodeGenOpts.InstrProfileOutput.empty() ? DefaultProfileGenName
                                                : CodeGenOpts.InstrProfileOutput,
-                        "", "", "", true,
-                        CodeGenOpts.DebugInfoForProfiling);
-  else if (CodeGenOpts.hasProfileIRUse())
+        "", "", "", PGOOptions::IRInstr, CodeGenOpts.DebugInfoForProfiling);
+  else if (CodeGenOpts.hasProfileIRUse()) {
     // -fprofile-use.
+    auto Action = CodeGenOpts.hasProfileCSIRUse() ? PGOOptions::CSIRUse
+                                                  : PGOOptions::NoAction;
     PGOOpt = PGOOptions("", CodeGenOpts.ProfileInstrumentUsePath, "",
-                        CodeGenOpts.ProfileRemappingFile, false,
+                        CodeGenOpts.ProfileRemappingFile, Action,
                         CodeGenOpts.DebugInfoForProfiling);
-  else if (!CodeGenOpts.SampleProfileFile.empty())
+  } else if (!CodeGenOpts.SampleProfileFile.empty())
     // -fprofile-sample-use
     PGOOpt = PGOOptions("", "", CodeGenOpts.SampleProfileFile,
-                        CodeGenOpts.ProfileRemappingFile, false,
+                        CodeGenOpts.ProfileRemappingFile, PGOOptions::NoAction,
                         CodeGenOpts.DebugInfoForProfiling);
   else if (CodeGenOpts.DebugInfoForProfiling)
     // -fdebug-info-for-profiling
-    PGOOpt = PGOOptions("", "", "", "", false, true);
+    PGOOpt = PGOOptions("", "", "", "", PGOOptions::NoAction, true);
+
+  // Check to see if we want to generate a CS profile.
+  if (CodeGenOpts.hasProfileCSIRInstr()) {
+    if (CodeGenOpts.hasProfileCSIRUse()) {
+      errs() << "Cannot have both CSProfileUse pass and CSProfileGen pass at "
+                "the same time";
+      return;
+    }
+    if (PGOOpt.hasValue()) {
+      if (!PGOOpt->ProfileGenFile.empty()) {
+        errs() << "Cannot have both ProfileGen pass and CSProfileGen pass at "
+                  "the same time";
+        return;
+      }
+      PGOOpt->ProfileGenFile = CodeGenOpts.InstrProfileOutput;
+      PGOOpt->Action = PGOOptions::CSIRInstr;
+    } else
+      PGOOpt = PGOOptions(CodeGenOpts.InstrProfileOutput.empty()
+                              ? DefaultProfileGenName
+                              : CodeGenOpts.InstrProfileOutput,
+                          "", "", "", PGOOptions::CSIRInstr,
+                          CodeGenOpts.DebugInfoForProfiling);
+  }
 
   PassBuilder PB(TM.get(), PGOOpt);
 
   LoopAnalysisManager LAM(CodeGenOpts.DebugPassManager);
   FunctionAnalysisManager FAM(CodeGenOpts.DebugPassManager);
   CGSCCAnalysisManager CGAM(CodeGenOpts.DebugPassManager);
   ModuleAnalysisManager MAM(CodeGenOpts.DebugPassManager);
 
static void runThinLTOBackend(ModuleSummaryIndex *CombinedIndex, Module *M,
   }
   Conf.CPU = TOpts.CPU;
   Conf.CodeModel = getCodeModel(CGOpts);
   Conf.MAttrs = TOpts.Features;
   Conf.RelocModel = CGOpts.RelocationModel;
   Conf.CGOptLevel = getCGOptLevel(CGOpts);
   initTargetOptions(Conf.Options, CGOpts, TOpts, LOpts, HeaderOpts);
   Conf.SampleProfile = std::move(SampleProfile);
+
+  // Context sensitive profile.
+  if (CGOpts.hasProfileCSIRInstr()) {
+    Conf.RunCSIRInstr = true;
+    Conf.CSIRProfile = std::move(CGOpts.InstrProfileOutput);
+  } else if (CGOpts.hasProfileCSIRUse()) {
+    Conf.RunCSIRInstr = false;
+    Conf.CSIRProfile = std::move(CGOpts.ProfileInstrumentUsePath);
+  }
+
   Conf.ProfileRemapping = std::move(ProfileRemapping);
   Conf.UseNewPM = CGOpts.ExperimentalNewPassManager;
   Conf.DebugPassManager = CGOpts.DebugPassManager;
   Conf.RemarksWithHotness = CGOpts.DiagnosticsWithHotness;
   Conf.RemarksFilename = CGOpts.OptRecordFile;
   Conf.DwoPath = CGOpts.SplitDwarfFile;
   switch (Action) {
   case Backend_EmitNothing:

lib/CodeGen/CodeGenModule.cpp

 #include "llvm/ADT/Triple.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Module.h"
+#include "llvm/IR/ProfileSummary.h"
 #include "llvm/ProfileData/InstrProfReader.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/ConvertUTF.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MD5.h"
 
 using namespace clang;
 using namespace CodeGen;
if (llvm::Function *OpenMPRegistrationFunction =
             OpenMPRuntime->emitRegistrationFunction()) {
       auto ComdatKey = OpenMPRegistrationFunction->hasComdat() ?
         OpenMPRegistrationFunction : nullptr;
       AddGlobalCtor(OpenMPRegistrationFunction, 0, ComdatKey);
     }
     OpenMPRuntime->clear();
   }
   if (PGOReader) {
-    getModule().setProfileSummary(PGOReader->getSummary().getMD(VMContext));
+    getModule().setProfileSummary(PGOReader->getSummary(false).getMD(VMContext),
+                                  llvm::ProfileSummary::PSK_Instr);
     if (PGOStats.hasDiagnostics())
       PGOStats.reportDiagnostics(getDiags(), getCodeGenOpts().MainFileName);
   }
   EmitCtorList(GlobalCtors, "llvm.global_ctors");
   EmitCtorList(GlobalDtors, "llvm.global_dtors");
   EmitGlobalAnnotations();
   EmitStaticExternCAliases();
   EmitDeferredUnusedCoverageMappings();

lib/Driver/ToolChain.cpp

llvm::sys::path::append(Path, "lib", getOSLibName(),
                           llvm::Triple::getArchTypeName(getArch()));
   return Path.str();
 }
 
 bool ToolChain::needsProfileRT(const ArgList &Args) {
   if (needsGCovInstrumentation(Args) ||
       Args.hasArg(options::OPT_fprofile_generate) ||
       Args.hasArg(options::OPT_fprofile_generate_EQ) ||
+      Args.hasArg(options::OPT_fcs_profile_generate) ||
+      Args.hasArg(options::OPT_fcs_profile_generate_EQ) ||
       Args.hasArg(options::OPT_fprofile_instr_generate) ||
       Args.hasArg(options::OPT_fprofile_instr_generate_EQ) ||
       Args.hasArg(options::OPT_fcreate_profile))
     return true;
 
   return false;
 }
 

lib/Driver/ToolChains/Clang.cpp

static void addPGOAndCoverageFlags(Compilation &C, const Driver &D,
 
   auto *PGOGenerateArg = Args.getLastArg(options::OPT_fprofile_generate,
                                          options::OPT_fprofile_generate_EQ,
                                          options::OPT_fno_profile_generate);
   if (PGOGenerateArg &&
       PGOGenerateArg->getOption().matches(options::OPT_fno_profile_generate))
     PGOGenerateArg = nullptr;
 
+  auto *CSPGOGenerateArg = Args.getLastArg(options::OPT_fcs_profile_generate,
+                                           options::OPT_fcs_profile_generate_EQ,
+                                           options::OPT_fno_profile_generate);
+  if (CSPGOGenerateArg &&
+      CSPGOGenerateArg->getOption().matches(options::OPT_fno_profile_generate))
+    CSPGOGenerateArg = nullptr;
+
   auto *ProfileGenerateArg = Args.getLastArg(
       options::OPT_fprofile_instr_generate,
       options::OPT_fprofile_instr_generate_EQ,
       options::OPT_fno_profile_instr_generate);
   if (ProfileGenerateArg &&
       ProfileGenerateArg->getOption().matches(
           options::OPT_fno_profile_instr_generate))
     ProfileGenerateArg = nullptr;
if (ProfileGenerateArg) {
     if (ProfileGenerateArg->getOption().matches(
             options::OPT_fprofile_instr_generate_EQ))
       CmdArgs.push_back(Args.MakeArgString(Twine("-fprofile-instrument-path=") +
                                            ProfileGenerateArg->getValue()));
     // The default is to use Clang Instrumentation.
     CmdArgs.push_back("-fprofile-instrument=clang");
   }
 
+  Arg *PGOGenArg = nullptr;
   if (PGOGenerateArg) {
+    assert(!CSPGOGenerateArg);
+    PGOGenArg = PGOGenerateArg;
     CmdArgs.push_back("-fprofile-instrument=llvm");
-    if (PGOGenerateArg->getOption().matches(
-            options::OPT_fprofile_generate_EQ)) {
-      SmallString<128> Path(PGOGenerateArg->getValue());
+  }
+  if (CSPGOGenerateArg) {
+    assert(!PGOGenerateArg);
+    PGOGenArg = CSPGOGenerateArg;
+    CmdArgs.push_back("-fprofile-instrument=csllvm");
+  }
+  if (PGOGenArg) {
+    if (PGOGenArg->getOption().matches(
+            PGOGenerateArg ? options::OPT_fprofile_generate_EQ
+                           : options::OPT_fcs_profile_generate_EQ)) {
+      SmallString<128> Path(PGOGenArg->getValue());
       llvm::sys::path::append(Path, "default_%m.profraw");
       CmdArgs.push_back(
           Args.MakeArgString(Twine("-fprofile-instrument-path=") + Path));
     }
   }
 
   if (ProfileUseArg) {
     if (ProfileUseArg->getOption().matches(options::OPT_fprofile_instr_use_EQ))

lib/Driver/ToolChains/CommonArgs.cpp

if (Arg *A = getLastProfileSampleUseArg(Args)) {
     StringRef FName = A->getValue();
     if (!llvm::sys::fs::exists(FName))
       ToolChain.getDriver().Diag(diag::err_drv_no_such_file) << FName;
     else
       CmdArgs.push_back(
           Args.MakeArgString(Twine("-plugin-opt=sample-profile=") + FName));
   }
 
+  auto *CSPGOGenerateArg = Args.getLastArg(options::OPT_fcs_profile_generate,
+                                           options::OPT_fcs_profile_generate_EQ,
+                                           options::OPT_fno_profile_generate);
+  if (CSPGOGenerateArg &&
+      CSPGOGenerateArg->getOption().matches(options::OPT_fno_profile_generate))
+    CSPGOGenerateArg = nullptr;
+
+  auto *ProfileUseArg = getLastProfileUseArg(Args);
+
+  if (CSPGOGenerateArg) {
+    CmdArgs.push_back(Args.MakeArgString("-plugin-opt=cs-profile-generate"));
+    if (CSPGOGenerateArg->getOption().matches(
+            options::OPT_fcs_profile_generate_EQ)) {
+      SmallString<128> Path(CSPGOGenerateArg->getValue());
+      llvm::sys::path::append(Path, "default_%m.profraw");
+      CmdArgs.push_back(
+          Args.MakeArgString(Twine("-plugin-opt=cs-profile-path=") + Path));
+    } else
+      CmdArgs.push_back(
+          Args.MakeArgString("-plugin-opt=cs-profile-path=default_%m.profraw"));
+  } else if (ProfileUseArg) {
+    CmdArgs.push_back(Args.MakeArgString(Twine("-plugin-opt=cs-profile-path=") +
+                                         ProfileUseArg->getValue()));
+  }
+
   // Need this flag to turn on new pass manager via Gold plugin.
   if (Args.hasFlag(options::OPT_fexperimental_new_pass_manager,
                    options::OPT_fno_experimental_new_pass_manager,
                    /* Default */ ENABLE_EXPERIMENTAL_NEW_PASS_MANAGER)) {
     CmdArgs.push_back("-plugin-opt=new-pass-manager");
   }
 
   // Setup statistics file output.

lib/Frontend/CompilerInvocation.cpp

static void setPGOInstrumentor(CodeGenOptions &Opts, ArgList &Args,
   Arg *A = Args.getLastArg(OPT_fprofile_instrument_EQ);
   if (A == nullptr)
     return;
   StringRef S = A->getValue();
   unsigned I = llvm::StringSwitch<unsigned>(S)
                    .Case("none", CodeGenOptions::ProfileNone)
                    .Case("clang", CodeGenOptions::ProfileClangInstr)
                    .Case("llvm", CodeGenOptions::ProfileIRInstr)
+                   .Case("csllvm", CodeGenOptions::ProfileCSIRInstr)
                    .Default(~0U);
   if (I == ~0U) {
     Diags.Report(diag::err_drv_invalid_pgo_instrumentor) << A->getAsString(Args)
                                                          << S;
     return;
   }
   auto Instrumentor = static_cast<CodeGenOptions::ProfileInstrKind>(I);
   Opts.setProfileInstr(Instrumentor);
 }
 
 // Set the profile kind using fprofile-instrument-use-path.
 static void setPGOUseInstrumentor(CodeGenOptions &Opts,
                                   const Twine &ProfileName) {
   auto ReaderOrErr = llvm::IndexedInstrProfReader::create(ProfileName);
   // In error, return silently and let Clang PGOUse report the error message.
   if (auto E = ReaderOrErr.takeError()) {
     llvm::consumeError(std::move(E));
     Opts.setProfileUse(CodeGenOptions::ProfileClangInstr);
     return;
   }
   std::unique_ptr<llvm::IndexedInstrProfReader> PGOReader =
     std::move(ReaderOrErr.get());
-  if (PGOReader->isIRLevelProfile())
-    Opts.setProfileUse(CodeGenOptions::ProfileIRInstr);
+  if (PGOReader->isIRLevelProfile()) {
+    if (PGOReader->hasCSIRLevelProfile())
+      Opts.setProfileUse(CodeGenOptions::ProfileCSIRInstr);
     else
+      Opts.setProfileUse(CodeGenOptions::ProfileIRInstr);
+  } else
     Opts.setProfileUse(CodeGenOptions::ProfileClangInstr);
 }
 
 static bool ParseCodeGenArgs(CodeGenOptions &Opts, ArgList &Args, InputKind IK,
                              DiagnosticsEngine &Diags,
                              const TargetOptions &TargetOpts,
                              const FrontendOptions &FrontendOpts) {
   bool Success = true;

test/CodeGen/Inputs/pgotestir.proftext

+# IR level Instrumentation Flag
+:ir

test/CodeGen/Inputs/pgotestir_cs.proftext

+# IR level Instrumentation Flag with CS
+:csir

test/CodeGen/cspgo-instrumentation.c

+// Test if CSPGO instrumentation and use pass are invoked.
+//
+// Ensure Pass PGOInstrumentationGenPass is invoked.
+// RUN: %clang_cc1 -O2 -fprofile-instrument=csllvm %s -mllvm -debug-pass=Structure -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN
+// RUN: %clang_cc1 -O2 -fprofile-instrument=csllvm %s -fexperimental-new-pass-manager -fdebug-pass-manager -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-NEWPM
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN: PGOInstrumentationGenPass
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-NEWPM: Running pass: PGOInstrumentationGen
+//
+// RUN: llvm-profdata merge -o %t.profdata %S/Inputs/pgotestir.proftext
+//
+// Ensure Pass PGOInstrumentationUsePass and PGOInstrumentationGenPass are invoked.
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t.profdata -fprofile-instrument=csllvm %s -mllvm -debug-pass=Structure -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN2
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t.profdata -fprofile-instrument=csllvm %s -fexperimental-new-pass-manager -fdebug-pass-manager  -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN2-NEWPM
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN2: PGOInstrumentationUsePass
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN2: PGOInstrumentationGenPass
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN2-NEWPM: Running pass: PGOInstrumentationUse
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN2-NEWPM: Running pass: PGOInstrumentationGen
+
+// Ensure Pass PGOInstrumentationUsePass is invoked only once.
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t.profdata %s -mllvm -debug-pass=Structure -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-PGOUSEPASS-INVOKED-USE
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t.profdata %s -fexperimental-new-pass-manager -fdebug-pass-manager -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-PGOUSEPASS-INVOKED-USE-NEWPM
+// CHECK-PGOUSEPASS-INVOKED-USE: PGOInstrumentationUsePass
+// CHECK-PGOUSEPASS-INVOKED-USE-NOT: PGOInstrumentationUsePass
+// CHECK-PGOUSEPASS-INVOKED-USE-NEWPM: Running pass: PGOInstrumentationUse
+// CHECK-PGOUSEPASS-INVOKED-USE-NEWPM-NOT: Running pass: PGOInstrumentationUse
+//
+// Ensure Pass PGOInstrumentationUsePass is invoked twice.
+// RUN: llvm-profdata merge -o %t_cs.profdata %S/Inputs/pgotestir_cs.proftext
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t_cs.profdata %s -mllvm -debug-pass=Structure -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-PGOUSEPASS-INVOKED-USE2
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t_cs.profdata %s -fexperimental-new-pass-manager -fdebug-pass-manager  -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-PGOUSEPASS-INVOKED-USE2-NEWPM
+// CHECK-PGOUSEPASS-INVOKED-USE2: PGOInstrumentationUsePass
+// CHECK-PGOUSEPASS-INVOKED-USE2: PGOInstrumentationUsePass
+// CHECK-PGOUSEPASS-INVOKED-USE2-NEWPM: Running pass: PGOInstrumentationUse
+// CHECK-PGOUSEPASS-INVOKED-USE2-NEWPM: Running pass: PGOInstrumentationUse
+
+// Ensure Pass CSPGOInstrumentationGenPass and CSPGOInstrumentationUsePass will
+// not be on at the same time.
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t_cs.profdata %s -fprofile-instrument=csllvm -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOGENUSEPASS-ERROR
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t_cs.profdata %s -fprofile-instrument=csllvm -fexperimental-new-pass-manager -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOGENUSEPASS-ERROR
+// CHECK-CSPGOGENUSEPASS-ERROR: Cannot have both CSProfileUse pass and CSProfileGen pass at the same time

test/CodeGen/cspgo-instrumentation_thinlto.c

+// Test if CSPGO instrumentation and use pass are invoked in thinlto.
+//
+// RUN: llvm-profdata merge -o %t.profdata %S/Inputs/pgotestir.proftext
+//
+// Ensure Pass PGOInstrumentationGenPass is not invoked in PreLink.
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t.profdata -fprofile-instrument=csllvm %s -flto=thin -mllvm -debug-pass=Structure -emit-llvm-bc -o %S/foo_fe.bc 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-PRE
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t.profdata -fprofile-instrument=csllvm %s -flto=thin -fexperimental-new-pass-manager -fdebug-pass-manager -emit-llvm-bc -o %S/foo_fe_pm.bc 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-PRE-NEWPM
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-PRE: PGOInstrumentationUsePass
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-PRE-NOT: PGOInstrumentationGenPass
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-PRE-NEWPM: Running pass: PGOInstrumentationUse
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-PRE-NEWPM-NOT: Running pass: PGOInstrumentationGen
+//
+// RUN: llvm-lto -thinlto -o %S/foo %S/foo_fe.bc
+// RUN: llvm-lto -thinlto -o %S/foo_pm %S/foo_fe_pm.bc
+// Ensure Pass PGOInstrumentationGenPass is invoked in PostLink.
+// RUN: %clang_cc1 -O2 -x ir %S/foo_fe.bc -fthinlto-index=%S/foo.thinlto.bc -fprofile-instrument=csllvm -flto=thin -emit-llvm -mllvm -debug-pass=Structure -o - |& FileCheck %s -check-prefix=CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-POST
+// RUN: %clang_cc1 -O2 -x ir %S/foo_fe_pm.bc -fthinlto-index=%S/foo_pm.thinlto.bc -fexperimental-new-pass-manager -fdebug-pass-manager  -fprofile-instrument=csllvm -flto=thin -emit-llvm -o - |& FileCheck %s -check-prefix=CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-POST-NEWPM
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-POST-NOT: PGOInstrumentationUsePass
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-POST: PGOInstrumentationGenPass
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-POST-NEWPM-NOT: Running pass: PGOInstrumentationUse
+// CHECK-CSPGOGENPASS-INVOKED-INSTR-GEN-POST-NEWPM: Running pass: PGOInstrumentationGen
+//
+// RUN: llvm-profdata merge -o %t_cs.profdata %S/Inputs/pgotestir_cs.proftext
+//
+// Ensure Pass PGOInstrumentationUsePass is invoked Once in PreLink.
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t_cs.profdata %s -flto=thin -mllvm -debug-pass=Structure -emit-llvm-bc -o %S/foo_fe.bc 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-PRE
+// RUN: %clang_cc1 -O2 -fprofile-instrument-use-path=%t_cs.profdata %s -flto=thin -fexperimental-new-pass-manager -fdebug-pass-manager -emit-llvm-bc -o %S/foo_fe_pm.bc 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-PRE-NEWPM
+// CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-PRE: PGOInstrumentationUsePass
+// CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-PRE-NOT: PGOInstrumentationUsePass
+// CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-PRE-NEWPM: Running pass: PGOInstrumentationUse
+// CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-PRE-NEWPM-NOT: Running pass: PGOInstrumentationUse
+//
+// RUN: llvm-lto -thinlto -o %S/foo %S/foo_fe.bc
+// RUN: llvm-lto -thinlto -o %S/foo_pm %S/foo_fe_pm.bc
+// Ensure Pass PGOInstrumentationUSEPass is invoked in PostLink.
+// RUN: %clang_cc1 -O2 -x ir %S/foo_fe.bc -fthinlto-index=%S/foo.thinlto.bc -fprofile-instrument-use-path=%t_cs.profdata -flto=thin -emit-llvm -mllvm -debug-pass=Structure -o - 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-POST
+// RUN: %clang_cc1 -O2 -x ir %S/foo_fe_pm.bc -fthinlto-index=%S/foo_pm.thinlto.bc -fexperimental-new-pass-manager -fdebug-pass-manager -fprofile-instrument-use-path=%t_cs.profdata -flto=thin -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-POST-NEWPM
+// CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-POST: PGOInstrumentationUsePass
+// CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-POST-NOT: PGOInstrumentationUsePass
+// CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-POST-NEWPM: Running pass: PGOInstrumentationUse
+// CHECK-CSPGOUSEPASS-INVOKED-INSTR-USE-POST-NEWPM-NOT: Running pass: PGOInstrumentationUse
+//
+// Finally, test if a non-cs profile is passed to postline, PGO UsePass is not invoked.
+// RUN: %clang_cc1 -O2 -x ir %S/foo_fe.bc -fthinlto-index=%S/foo.thinlto.bc -fprofile-instrument-use-path=%t.profdata -flto=thin -emit-llvm -mllvm -debug-pass=Structure -o - 2>&1 | FileCheck %s -check-prefix=CHECK-PGOUSEPASS-INVOKED-INSTR-USE-POST
+// RUN: %clang_cc1 -O2 -x ir %S/foo_fe_pm.bc -fthinlto-index=%S/foo_pm.thinlto.bc -fexperimental-new-pass-manager -fdebug-pass-manager -fprofile-instrument-use-path=%t.profdata -flto=thin -emit-llvm -o - 2>&1 | FileCheck %s -check-prefix=CHECK-PGOUSEPASS-INVOKED-INSTR-USE-POST-NEWPM
+// CHECK-PGOUSEPASS-INVOKED-INSTR-USE-POST-NOT: PGOInstrumentationUsePass
+// CHECK-PGOUSEPASS-INVOKED-INSTR-USE-POST-NEWPM-NOT: Running pass: PGOInstrumentationUse