[Remarks][1/2] Expand remarks hotness threshold option support in more tools

[weiwang]

Summary

This is the #1 of 2 changes that make remarks hotness threshold option
available in more tools. The changes also allow the threshold to sync with
hotness threshold from profile summary with special value 'auto'.

This change modifies the interface of lto::setupLLVMOptimizationRemarks() to
accept remarks hotness threshold. Update all the tools that use it with remarks
hotness threshold options:

• lld: '--opt-remarks-hotness-threshold='
• llvm-lto2: '--pass-remarks-hotness-threshold='
• llvm-lto: '--lto-pass-remarks-hotness-threshold='
• gold plugin: '-plugin-opt=opt-remarks-hotness-threshold='

NOTE: Internal remarks hotness threshold type is updated to accommodate the next change. The 'auto' value is allowed, but has no effect.

[weiwang]

This is the 1st of 3 dependent patches:

1. [lld] Enable remarks hotness filtering in lld: https://reviews.llvm.org/D85809
2. [clang] Pass-through remarks options to lld: https://reviews.llvm.org/D85810
3. [remarks] Optimization remarks hotness filtering from profile summary: https://reviews.llvm.org/D85808

[weiwang]

clang-format

[thegameg]

LGTM, thanks for plugging all these options through!

[tejohnson]

The changes to *LTOCodeGenerator and the test using llvm-lto are unrelated to lld (they are the old LTO API used by ld64). So they should be moved to a different patch or the patch name changed. Would be good to add llvm-lto2 tests which exercise the new LTO API changed in LTO.cpp, LTOBackend.cpp etc. Note those changes are broader than just lld (also used by gold-plugin).

[weiwang]

changes:

1. revert the change made to llvm-lto.
2. add the new option to llvm-lto2 wit test cases.

ld.gold's behavior is not affected by the change: No threshold is enforced.

[weiwang]

In D85809#2214246, @tejohnson wrote:

The changes to *LTOCodeGenerator and the test using llvm-lto are unrelated to lld (they are the old LTO API used by ld64). So they should be moved to a different patch or the patch name changed. Would be good to add llvm-lto2 tests which exercise the new LTO API changed in LTO.cpp, LTOBackend.cpp etc. Note those changes are broader than just lld (also used by gold-plugin).

I have removed the change to llvm-lto and added support in llvm-lto2. The new option is not exposed to ld.gold, so it is not affected.

[MaskRay]

lld/test/ELF/lto/opt-remarks.ll

10

For continuation lines, we usually indent by 2 columns.

This file was created in 2017, so some lines do not obey this rule. For new RUN lines you probably need to follow the rule.

15

Replace cat ... | cmd with cmd < ...

Actually FileCheck supports --input-file, but since it is long, some people don't use it.

llvm/test/LTO/Resolution/X86/diagnostic-handler-remarks-with-hotness.ll

20

cat ... | cmd => cmd < ...

[MaskRay]

llvm/include/llvm/LTO/Config.h

124

Add "0 means no threshold".

[tejohnson]

In D85809#2217341, @weiwang wrote:

In D85809#2214246, @tejohnson wrote:

The changes to *LTOCodeGenerator and the test using llvm-lto are unrelated to lld (they are the old LTO API used by ld64). So they should be moved to a different patch or the patch name changed. Would be good to add llvm-lto2 tests which exercise the new LTO API changed in LTO.cpp, LTOBackend.cpp etc. Note those changes are broader than just lld (also used by gold-plugin).

I have removed the change to llvm-lto and added support in llvm-lto2. The new option is not exposed to ld.gold, so it is not affected.

You might as well have the legacy LTO and llvm-lto support, presumably it could be useful for ld64 etc that use it. Since you already implemented it.

And I suggest adding the support to gold-plugin.cpp as well (added a comment to one of your follow on patches to this effect as well).

[thegameg]

In D85809#2217341, @weiwang wrote:

I have removed the change to llvm-lto

It would be nice if you can bring that back (maybe in a separate patch as Teresa suggested). I can see it being useful through ld64/libLTO, and for consistency it doesn't sound like it would cause any harm, since these options are passed as "debug options" through -mllvm.

[tejohnson]

In D85809#2218725, @thegameg wrote:

In D85809#2217341, @weiwang wrote:

I have removed the change to llvm-lto

It would be nice if you can bring that back (maybe in a separate patch as Teresa suggested). I can see it being useful through ld64/libLTO, and for consistency it doesn't sound like it would cause any harm, since these options are passed as "debug options" through -mllvm.

Just to clarify, I wasn't suggesting putting it in another patch, but just bringing it back here (plus adding in the gold-plugin support). The other patch I was referring to was the follow on patch D85810 where I also suggested adding support to gold-plugin.cpp.

[thegameg]

In D85809#2218742, @tejohnson wrote:

Just to clarify, I wasn't suggesting putting it in another patch, but just bringing it back here (plus adding in the gold-plugin support). The other patch I was referring to was the follow on patch D85810 where I also suggested adding support to gold-plugin.cpp.

Ah, looks like I missed your comment from this morning. I agree with that.

[weiwang]

update:
add remarks hotness threshold to all the tools uses lto::setupLLVMOptimizationRemarks

[weiwang]

format update

[weiwang]

format update

[MaskRay]

lld/ELF/Config.h

266

This should be moved above

llvm::StringRef optRemarksFilename;
llvm::StringRef optRemarksPasses;
llvm::StringRef optRemarksFormat;

lld/ELF/Options.td

546

<integer> might be preciser.

llvm/tools/gold/gold-plugin.cpp

302

Drop unneeded inner braces

940

stray ;

[weiwang]

lld/ELF/Config.h

266

Thanks for the comments! I will upload another revision to fix this, and also address Teresa's comment in https://reviews.llvm.org/D85808.

[weiwang]

update:

1. move internal remarks hotness threshold type change and its related handling code from child diff into this one. With this move, remarks hotness threshold now accepts 'auto' value, but its handling is currently disabled.
2. format change.

[weiwang]

Try to fix a windows build error. No msvc available locally, so even with this change, the build could still fail.

Suspected a naming conflict between llvm::Optional class and llvm::Optional enum defined in CommandLine.h.

[weiwang]

try to fix windows build error. Use fully qualified name for enum value Optional.

[weiwang]

try to fix windows build linking error.

[weiwang]

Really appreciate the feedbacks. If there is no more, could anyone approve 3 changes? Thanks.

[weiwang]

@tejohnson @MaskRay Do you have other comments to the change?

[MaskRay]

lld/ELF/Config.h

115

Don't repeat the include/llvm/LTO/Config.h comment here

lld just forwards these options. Their help messages should be in llvm/LTO/

lld/ELF/Driver.cpp

991

delete trailing period

llvm/include/llvm/Remarks/HotnessThresholdParser.h

20 ↗

(On Diff #286588)

empty line before namespace

llvm/include/llvm/Support/CommandLine.h

1484 ↗

(On Diff #286588)

Unrelated change

[weiwang]

Address Comments.

[weiwang]

Thanks @MaskRay ! I am really sorry I didn't catch up your comments earlier. My email somehow stops giving me notifications on diff updates.

llvm/include/llvm/Support/CommandLine.h

1484 ↗

(On Diff #286588)

This change is need for window msvc build test to pass. Previously the msvc build failed during template instantiation at cl::opt<Optional<uint64_t>, false, remarks::HotnessThresholdParser>. The first argument for the base class ctor is an enum type llvm::cl::Optional, but msvc somehow tries to map it to llvm::Optional. I am not sure why the compiler thinks they are the same. I've tried several ways to fix this, but it only works by explicitly giving its fully qualified name. Linux build does not have this issue.

[weiwang]

@MaskRay @tejohnson, do you have other comments regarding this change and its dependent?

[tejohnson]

LGTM, but please wait for @MaskRay since he had the most recent comments on this patch.

[weiwang]

Thanks! I will wait for his input.

[MaskRay]

lld/ELF/Options.td

546

This is not done

llvm/include/llvm/LTO/Config.h

128

Does swapping the meaning of 0 and None make more sense? None looks like a better default value.

llvm/include/llvm/Remarks/HotnessThresholdParser.h

47 ↗

(On Diff #296593)

Regarding the MSVC compiliation issue. If you use llvm::Optional here, does it work?

llvm/test/LTO/Resolution/X86/diagnostic-handler-remarks-with-hotness.ll

12

Full stop.

30

We don't use not FileCheck. If you want to check a pattern does not exist, use https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-not-directive

53

Please fix the not FIleCheck

[weiwang]

lld/ELF/Options.td

546

Since the value can be integer or 'auto', maybe <value> is more appropriate here?

[weiwang]

llvm/test/LTO/Resolution/X86/diagnostic-handler-remarks-with-hotness.ll

30

Sorry, I am not sure how to apply "CHECK-EMPTY" in this case. The input is empty, and I think "CHECK-EMPTY" assumes a non-empty input (requires a "CHECK" line above it).

[weiwang]

llvm/include/llvm/Remarks/HotnessThresholdParser.h

47 ↗

(On Diff #296593)

Good point! I completed overlooked this place. I will give it a try in the next revision.

[MaskRay]

llvm/test/LTO/Resolution/X86/diagnostic-handler-remarks-with-hotness.ll

30

Use | count 0 to check empty output

[weiwang]

llvm/include/llvm/LTO/Config.h

128

None means unknown, more of a transient value. It would be updated by the actual hotness value from PSI later. Also currently 0 is treated as disabled, I figured keeping its current meaning requires less change.

[weiwang]

update test case

[weiwang]

update to address MSVC build error

[weiwang]

llvm/include/llvm/Remarks/HotnessThresholdParser.h

47 ↗

(On Diff #296593)

Well, just tried it, looks like the windows build failed at the same place:
https://buildkite.com/llvm-project/premerge-checks/builds/17124

FAILED: lib/LTO/CMakeFiles/LLVMLTO.dir/LTOCodeGenerator.cpp.obj 
sccache C:\BuildTools\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe  /nologo /TP -DBUILD_EXAMPLES -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib\LTO -IC:\ws\w64\llvm-project\premerge-checks\llvm\lib\LTO -Iinclude -IC:\ws\w64\llvm-project\premerge-checks\llvm\include /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Zc:strictStrings /Oi /Zc:rvalueCast /bigobj /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4324 -w14062 -we4238 /Gw /MD /O2 /Ob2    /EHs-c- /GR- -UNDEBUG -std:c++14 /showIncludes /Folib\LTO\CMakeFiles\LLVMLTO.dir\LTOCodeGenerator.cpp.obj /Fdlib\LTO\CMakeFiles\LLVMLTO.dir\LLVMLTO.pdb /FS -c C:\ws\w64\llvm-project\premerge-checks\llvm\lib\LTO\LTOCodeGenerator.cpp
C:\ws\w64\llvm-project\premerge-checks\llvm\include\llvm/Support/CommandLine.h(1484): error C2275: 'llvm::Optional<uint64_t>': illegal use of this type as an expression
C:\ws\w64\llvm-project\premerge-checks\llvm\lib\LTO\LTOCodeGenerator.cpp(97): note: see reference to function template instantiation 'llvm::cl::opt<llvm::Optional<uint64_t>,false,llvm::remarks::HotnessThresholdParser>::opt<char[35],llvm::cl::desc,llvm::cl::value_desc,llvm::cl::initializer<int>,llvm::cl::OptionHidden>(const char (&)[35],const llvm::cl::desc &,const llvm::cl::value_desc &,const llvm::cl::initializer<int> &,const llvm::cl::OptionHidden &)' being compiled
C:\ws\w64\llvm-project\premerge-checks\llvm\lib\LTO\LTOCodeGenerator.cpp(92): note: see reference to function template instantiation 'llvm::cl::opt<llvm::Optional<uint64_t>,false,llvm::remarks::HotnessThresholdParser>::opt<char[35],llvm::cl::desc,llvm::cl::value_desc,llvm::cl::initializer<int>,llvm::cl::OptionHidden>(const char (&)[35],const llvm::cl::desc &,const llvm::cl::value_desc &,const llvm::cl::initializer<int> &,const llvm::cl::OptionHidden &)' being compiled

I am going to revert it back.

[weiwang]

revert back MSVC fix

[weiwang]

@tejohnson @MaskRay Do you have other comments?

[weiwang]

rebase

[tejohnson]

lgtm

lld/ELF/Config.h

struct Configuration {
   llvm::StringRef ltoCSProfileFile;
   llvm::StringRef ltoNewPmPasses;
   llvm::StringRef ltoObjPath;
   llvm::StringRef ltoSampleProfile;
   llvm::StringRef mapFile;
   llvm::StringRef outputFile;
   llvm::StringRef optRemarksFilename;
   llvm::StringRef optRemarksPasses;
   llvm::StringRef optRemarksFormat;
   llvm::StringRef progName;
   llvm::StringRef printArchiveStats;
   llvm::StringRef printSymbolOrder;
   llvm::StringRef soName;
   llvm::StringRef sysroot;
   llvm::StringRef thinLTOCacheDir;
   llvm::StringRef thinLTOIndexOnlyArg;
   llvm::StringRef ltoBasicBlockSections;
struct Configuration {
   uint64_t mipsGotSize;
   uint64_t zStackSize;
   unsigned ltoPartitions;
   unsigned ltoo;
   unsigned optimize;
   StringRef thinLTOJobs;
   unsigned timeTraceGranularity;
   int32_t splitStackAdjustSize;
+  unsigned optRemarksHotnessThreshold;
 
   // The following config options do not directly correspond to any
   // particular command line options.
 
   // True if we need to pass through relocations in input files to the
   // output file. Usually false because we consume relocations.
   bool copyRelocs;
 

lld/ELF/Driver.cpp

static void readConfigs(opt::InputArgList &args) {
   config->mmapOutputFile =
       args.hasFlag(OPT_mmap_output_file, OPT_no_mmap_output_file, true);
   config->nmagic = args.hasFlag(OPT_nmagic, OPT_no_nmagic, false);
   config->noinhibitExec = args.hasArg(OPT_noinhibit_exec);
   config->nostdlib = args.hasArg(OPT_nostdlib);
   config->oFormatBinary = isOutputFormatBinary(args);
   config->omagic = args.hasFlag(OPT_omagic, OPT_no_omagic, false);
   config->optRemarksFilename = args.getLastArgValue(OPT_opt_remarks_filename);
+  config->optRemarksHotnessThreshold =
+      args::getInteger(args, OPT_opt_remarks_hotness_threshold, 0);
   config->optRemarksPasses = args.getLastArgValue(OPT_opt_remarks_passes);
   config->optRemarksWithHotness = args.hasArg(OPT_opt_remarks_with_hotness);
   config->optRemarksFormat = args.getLastArgValue(OPT_opt_remarks_format);
   config->optimize = args::getInteger(args, OPT_O, 1);
   config->orphanHandling = getOrphanHandling(args);
   config->outputFile = args.getLastArgValue(OPT_o);
   config->pie = args.hasFlag(OPT_pie, OPT_no_pie, false);
   config->printIcfSections =
       args.hasFlag(OPT_print_icf_sections, OPT_no_print_icf_sections, false);
   config->printGcSections =
       args.hasFlag(OPT_print_gc_sections, OPT_no_print_gc_sections, false);
   config->printArchiveStats = args.getLastArgValue(OPT_print_archive_stats);
   config->printSymbolOrder =

lld/ELF/LTO.cpp

static lto::Config createConfig() {
   // Set up a custom pipeline if we've been asked to.
   c.OptPipeline = std::string(config->ltoNewPmPasses);
   c.AAPipeline = std::string(config->ltoAAPipeline);
 
   // Set up optimization remarks if we've been asked to.
   c.RemarksFilename = std::string(config->optRemarksFilename);
   c.RemarksPasses = std::string(config->optRemarksPasses);
   c.RemarksWithHotness = config->optRemarksWithHotness;
+  c.RemarksHotnessThreshold = config->optRemarksHotnessThreshold;
   c.RemarksFormat = std::string(config->optRemarksFormat);
 
   c.SampleProfile = std::string(config->ltoSampleProfile);
   c.UseNewPM = config->ltoNewPassManager;
   c.DebugPassManager = config->ltoDebugPassManager;
   c.DwoDir = std::string(config->dwoDir);
 
   c.HasWholeProgramVisibility = config->ltoWholeProgramVisibility;

lld/ELF/Options.td

 def lto_sample_profile: JJ<"lto-sample-profile=">,
   HelpText<"Sample profile file path">;
 def lto_whole_program_visibility: FF<"lto-whole-program-visibility">,
   HelpText<"Asserts that the LTO link has whole program visibility">;
 def disable_verify: F<"disable-verify">;
 defm mllvm: Eq<"mllvm", "Additional arguments to forward to LLVM's option processing">;
 def opt_remarks_filename: Separate<["--"], "opt-remarks-filename">,
   HelpText<"YAML output file for optimization remarks">;
+defm opt_remarks_hotness_threshold: EEq<"opt-remarks-hotness-threshold",
+  "Minimum profile count required for an optimization remark to be output">,
+  MetaVarName<"<number>">;
 def opt_remarks_passes: Separate<["--"], "opt-remarks-passes">,
   HelpText<"Regex for the passes that need to be serialized to the output file">;
 def opt_remarks_with_hotness: FF<"opt-remarks-with-hotness">,
   HelpText<"Include hotness information in the optimization remarks file">;
 def opt_remarks_format: Separate<["--"], "opt-remarks-format">,
   HelpText<"The format used for serializing remarks (default: YAML)">;
 def save_temps: F<"save-temps">;
 def lto_basic_block_sections: JJ<"lto-basic-block-sections=">,

lld/test/ELF/lto/opt-remarks.ll

 ; REQUIRES: x86
 ; RUN: llvm-as %s -o %t.o
 
-; RUN: rm -f %t.yaml
+; RUN: rm -f %t.yaml %t1.yaml %t.hot.yaml %t.t300.yaml %t.t301.yaml
 ; RUN: ld.lld --opt-remarks-filename %t.yaml %t.o -o %t -shared -save-temps
 ; RUN: llvm-dis %t.0.4.opt.bc -o - | FileCheck %s
 ; RUN: ld.lld --opt-remarks-with-hotness --opt-remarks-filename %t.hot.yaml \
 ; RUN:  %t.o -o %t -shared
+; RUN: ld.lld --opt-remarks-with-hotness --opt-remarks-hotness-threshold=300 \
+; RUN:   --opt-remarks-filename %t.t300.yaml %t.o -o %t -shared
+; RUN: ld.lld --opt-remarks-with-hotness --opt-remarks-hotness-threshold=301 \
+; RUN:   --opt-remarks-filename %t.t301.yaml %t.o -o %t -shared
 ; RUN: cat %t.yaml | FileCheck %s -check-prefix=YAML
 ; RUN: cat %t.hot.yaml | FileCheck %s -check-prefix=YAML-HOT
+; RUN: FileCheck %s -check-prefix=YAML-HOT < %t.t300.yaml
+; RUN: count 0 < %t.t301.yaml
 ; RUN: ld.lld --opt-remarks-filename %t1.yaml --opt-remarks-passes inline %t.o \
 ; RUN: -o /dev/null -shared
 ; RUN: cat %t1.yaml | FileCheck %s -check-prefix=YAML-PASSES
 ; RUN: ld.lld --opt-remarks-filename %t1.yaml --opt-remarks-format yaml %t.o \
 ; RUN: -o /dev/null -shared
-; RUN: cat %t.yaml | FileCheck %s -check-prefix=YAML
+; RUN: cat %t1.yaml | FileCheck %s -check-prefix=YAML
 
 ; Check that @tinkywinky is inlined after optimizations.
 ; CHECK-LABEL: define i32 @main
 ; CHECK-NEXT:  %a.i = {{.*}}call i32 @patatino()
 ; CHECK-NEXT:  ret i32 %a.i
 ; CHECK-NEXT: }
 
 ; YAML:      --- !Passed

llvm/include/llvm/LTO/Config.h

struct Config {
   std::string RemarksFilename = "";
 
   /// Optimization remarks pass filter.
   std::string RemarksPasses = "";
 
   /// Whether to emit optimization remarks with hotness informations.
   bool RemarksWithHotness = false;
 
+  /// Minimal number of hotness number for remarks to report.
+  /// "0 means no threshold".
+  unsigned RemarksHotnessThreshold = 0;
+
   /// The format used for serializing remarks (default: YAML).
   std::string RemarksFormat = "";
 
   /// Whether to emit the pass manager debuggging informations.
   bool DebugPassManager = false;
 
   /// Statistics output file path.
   std::string StatsFile;
 

llvm/include/llvm/LTO/LTO.h

 /// Given the original \p Path to an output file, replace any path
 /// prefix matching \p OldPrefix with \p NewPrefix. Also, create the
 /// resulting directory if it does not yet exist.
 std::string getThinLTOOutputFile(const std::string &Path,
                                  const std::string &OldPrefix,
                                  const std::string &NewPrefix);
 
 /// Setup optimization remarks.
-Expected<std::unique_ptr<ToolOutputFile>>
-setupLLVMOptimizationRemarks(LLVMContext &Context, StringRef RemarksFilename,
-                             StringRef RemarksPasses, StringRef RemarksFormat,
-                             bool RemarksWithHotness, int Count = -1);
+Expected<std::unique_ptr<ToolOutputFile>> setupLLVMOptimizationRemarks(
+    LLVMContext &Context, StringRef RemarksFilename, StringRef RemarksPasses,
+    StringRef RemarksFormat, bool RemarksWithHotness,
+    unsigned RemarksHotnessThreshold = 0, int Count = -1);
 
 /// Setups the output file for saving statistics.
 Expected<std::unique_ptr<ToolOutputFile>>
 setupStatsFile(StringRef StatsFilename);
 
 class LTO;
 struct SymbolResolution;
 class ThinBackendProc;

llvm/lib/LTO/LTO.cpp

Error LTO::run(AddStreamFn AddStream, NativeObjectCache Cache) {
 
   return Result;
 }
 
 Error LTO::runRegularLTO(AddStreamFn AddStream) {
   // Setup optimization remarks.
   auto DiagFileOrErr = lto::setupLLVMOptimizationRemarks(
       RegularLTO.CombinedModule->getContext(), Conf.RemarksFilename,
-      Conf.RemarksPasses, Conf.RemarksFormat, Conf.RemarksWithHotness);
+      Conf.RemarksPasses, Conf.RemarksFormat, Conf.RemarksWithHotness,
+      Conf.RemarksHotnessThreshold);
   if (!DiagFileOrErr)
     return DiagFileOrErr.takeError();
 
   // Finalize linking of regular LTO modules containing summaries now that
   // we have computed liveness information.
   for (auto &M : RegularLTO.ModsWithSummaries)
     if (Error Err = linkRegularLTO(std::move(M),
                                    /*LivenessFromIndex=*/true))
for (auto &Mod : ModuleMap) {
     ++Task;
   }
 
   return BackendProc->wait();
 }
 
 Expected<std::unique_ptr<ToolOutputFile>> lto::setupLLVMOptimizationRemarks(
     LLVMContext &Context, StringRef RemarksFilename, StringRef RemarksPasses,
-    StringRef RemarksFormat, bool RemarksWithHotness, int Count) {
+    StringRef RemarksFormat, bool RemarksWithHotness,
+    unsigned RemarksHotnessThreshold, int Count) {
   std::string Filename = std::string(RemarksFilename);
   // For ThinLTO, file.opt.<format> becomes
   // file.opt.<format>.thin.<num>.<format>.
   if (!Filename.empty() && Count != -1)
     Filename =
         (Twine(Filename) + ".thin." + llvm::utostr(Count) + "." + RemarksFormat)
             .str();
 
   auto ResultOrErr = llvm::setupLLVMOptimizationRemarks(
-      Context, Filename, RemarksPasses, RemarksFormat, RemarksWithHotness);
+      Context, Filename, RemarksPasses, RemarksFormat, RemarksWithHotness,
+      RemarksHotnessThreshold);
   if (Error E = ResultOrErr.takeError())
     return std::move(E);
 
   if (*ResultOrErr)
     (*ResultOrErr)->keep();
 
   return ResultOrErr;
 }

llvm/lib/LTO/LTOBackend.cpp

Error lto::thinBackend(const Config &Conf, unsigned Task, AddStreamFn AddStream,
   if (!TOrErr)
     return TOrErr.takeError();
 
   std::unique_ptr<TargetMachine> TM = createTargetMachine(Conf, *TOrErr, Mod);
 
   // Setup optimization remarks.
   auto DiagFileOrErr = lto::setupLLVMOptimizationRemarks(
       Mod.getContext(), Conf.RemarksFilename, Conf.RemarksPasses,
-      Conf.RemarksFormat, Conf.RemarksWithHotness, Task);
+      Conf.RemarksFormat, Conf.RemarksWithHotness, Conf.RemarksHotnessThreshold,
+      Task);
   if (!DiagFileOrErr)
     return DiagFileOrErr.takeError();
   auto DiagnosticOutputFile = std::move(*DiagFileOrErr);
 
   // Set the partial sample profile ratio in the profile summary module flag of
   // the module, if applicable.
   Mod.setPartialSampleProfileRatio(CombinedIndex);
 

llvm/lib/LTO/LTOCodeGenerator.cpp

 #endif
     cl::Hidden);
 
 cl::opt<bool> RemarksWithHotness(
     "lto-pass-remarks-with-hotness",
     cl::desc("With PGO, include profile count in optimization remarks"),
     cl::Hidden);
 
+cl::opt<unsigned>
+    RemarksHotnessThreshold("lto-pass-remarks-hotness-threshold",
+                            cl::desc("Minimum profile count required for an "
+                                     "optimization remark to be output"),
+                            cl::init(0), cl::Hidden);
+
 cl::opt<std::string>
     RemarksFilename("lto-pass-remarks-output",
                     cl::desc("Output filename for pass remarks"),
                     cl::value_desc("filename"));
 
 cl::opt<std::string>
     RemarksPasses("lto-pass-remarks-filter",
                   cl::desc("Only record optimization remarks from passes whose "

 
 /// Optimize merged modules using various IPO passes
 bool LTOCodeGenerator::optimize(bool DisableVerify, bool DisableInline,
                                 bool DisableGVNLoadPRE,
                                 bool DisableVectorization) {
   if (!this->determineTarget())
     return false;
 
-  auto DiagFileOrErr =
-      lto::setupLLVMOptimizationRemarks(Context, RemarksFilename, RemarksPasses,
-                                        RemarksFormat, RemarksWithHotness);
+  auto DiagFileOrErr = lto::setupLLVMOptimizationRemarks(
+      Context, RemarksFilename, RemarksPasses, RemarksFormat,
+      RemarksWithHotness, RemarksHotnessThreshold);
   if (!DiagFileOrErr) {
     errs() << "Error: " << toString(DiagFileOrErr.takeError()) << "\n";
     report_fatal_error("Can't get an output file for the remarks");
   }
   DiagnosticOutputFile = std::move(*DiagFileOrErr);
 
   // Setup output file to emit statistics.
   auto StatsFileOrErr = lto::setupStatsFile(LTOStatsFile);

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

 #define DEBUG_TYPE "thinlto"
 
 namespace llvm {
 // Flags -discard-value-names, defined in LTOCodeGenerator.cpp
 extern cl::opt<bool> LTODiscardValueNames;
 extern cl::opt<std::string> RemarksFilename;
 extern cl::opt<std::string> RemarksPasses;
 extern cl::opt<bool> RemarksWithHotness;
+extern cl::opt<unsigned> RemarksHotnessThreshold;
 extern cl::opt<std::string> RemarksFormat;
 }
 
 namespace {
 
 // Default to using all available threads in the system, but using only one
 // thred per core, as indicated by the usage of
 // heavyweight_hardware_concurrency() below.
for (auto IndexCount : ModulesOrdering) {
           }
         }
 
         LLVMContext Context;
         Context.setDiscardValueNames(LTODiscardValueNames);
         Context.enableDebugTypeODRUniquing();
         auto DiagFileOrErr = lto::setupLLVMOptimizationRemarks(
             Context, RemarksFilename, RemarksPasses, RemarksFormat,
-            RemarksWithHotness, count);
+            RemarksWithHotness, RemarksHotnessThreshold, count);
         if (!DiagFileOrErr) {
           errs() << "Error: " << toString(DiagFileOrErr.takeError()) << "\n";
           report_fatal_error("ThinLTO: Can't get an output file for the "
                              "remarks");
         }
 
         // Parse module now
         auto TheModule = loadModuleFromInput(Mod.get(), Context, false,

llvm/test/LTO/Resolution/X86/diagnostic-handler-remarks-with-hotness.ll

 ; RUN: llvm-as < %s >%t.bc
 
 ; Check pass remarks emitted to YAML file
 ; RUN: rm -f %t.yaml
 ; RUN: llvm-lto2 run -pass-remarks-output=%t.yaml \
 ; RUN:           -pass-remarks-with-hotness \
 ; RUN:           -r %t.bc,tinkywinky,p \
 ; RUN:           -r %t.bc,patatino,px \
 ; RUN:           -r %t.bc,main,px -o %t.o %t.bc
 ; RUN: cat %t.yaml | FileCheck %s -check-prefix=YAML
 
+; Check low threshold allows remarks to emit
+; RUN: rm -f %t.t300.yaml
+; RUN: llvm-lto2 run -pass-remarks-output=%t.t300.yaml \
+; RUN:           -pass-remarks-with-hotness \
+; RUN:           -pass-remarks-hotness-threshold=300 \
+; RUN:           -r %t.bc,tinkywinky,p \
+; RUN:           -r %t.bc,patatino,px \
+; RUN:           -r %t.bc,main,px -o %t.o %t.bc
+; RUN: FileCheck %s -check-prefix=YAML < %t.t300.yaml
+
+; Check high threshold disallows remarks to emit
+; RUN: rm -f %t.t301.yaml
+; RUN: llvm-lto2 run -pass-remarks-output=%t.t301.yaml \
+; RUN:           -pass-remarks-with-hotness \
+; RUN:           -pass-remarks-hotness-threshold=301 \
+; RUN:           -r %t.bc,tinkywinky,p \
+; RUN:           -r %t.bc,patatino,px \
+; RUN:           -r %t.bc,main,px -o %t.o %t.bc
+; RUN: not FileCheck %s -check-prefix=YAML < %t.t301.yaml
+
 ; Check pass remarks emitted to stderr
 ; RUN: llvm-lto2 run -pass-remarks=inline \
 ; RUN:           -pass-remarks-with-hotness \
 ; RUN:           -r %t.bc,tinkywinky,p \
 ; RUN:           -r %t.bc,patatino,px \
 ; RUN:           -r %t.bc,main,px -o %t.o %t.bc 2>&1 | FileCheck %s
 
+; Check low threshold allows remarks to emit
+; RUN: llvm-lto2 run -pass-remarks=inline \
+; RUN:           -pass-remarks-with-hotness \
+; RUN:           -pass-remarks-hotness-threshold=300 \
+; RUN:           -r %t.bc,tinkywinky,p \
+; RUN:           -r %t.bc,patatino,px \
+; RUN:           -r %t.bc,main,px -o %t.o %t.bc 2>&1 | FileCheck %s
+
+; Check high threshold disallows remarks to emit
+; RUN: llvm-lto2 run -pass-remarks=inline \
+; RUN:           -pass-remarks-with-hotness \
+; RUN:           -pass-remarks-hotness-threshold=301 \
+; RUN:           -r %t.bc,tinkywinky,p \
+; RUN:           -r %t.bc,patatino,px \
+; RUN:           -r %t.bc,main,px -o %t.o %t.bc 2>&1 | not FileCheck %s
+
 ; YAML: --- !Passed
 ; YAML-NEXT: Pass:            inline
 ; YAML-NEXT: Name:            Inlined
 ; YAML-NEXT: Function:        main
 ; YAML-NEXT: Hotness:         300
 ; YAML-NEXT: Args:
 ; YAML-NEXT:   - Callee:          tinkywinky
 ; YAML-NEXT:   - String:          ' inlined into '

llvm/test/LTO/X86/diagnostic-handler-remarks-with-hotness.ll

 ; Check that the hotness attribute is included in the optimization record file
 ; with -lto-pass-remarks-with-hotness.
 
 ; RUN: llvm-as < %s >%t.bc
-; RUN: rm -f %t.yaml
+; RUN: rm -f %t.yaml %t.t300.yaml %t.t301.yaml
 ; RUN: llvm-lto -lto-pass-remarks-output=%t.yaml \
 ; RUN:          -lto-pass-remarks-with-hotness \
 ; RUN:          -exported-symbol _main -o %t.o %t.bc
 ; RUN: cat %t.yaml | FileCheck -check-prefix=YAML %s
 
+; RUN: llvm-lto -lto-pass-remarks-output=%t.t300.yaml \
+; RUN:          -lto-pass-remarks-with-hotness \
+; RUN:          -lto-pass-remarks-hotness-threshold=300 \
+; RUN:          -exported-symbol _main -o %t.o %t.bc
+; RUN: FileCheck -check-prefix=YAML %s < %t.t300.yaml
+
+; RUN: llvm-lto -lto-pass-remarks-output=%t.t301.yaml \
+; RUN:          -lto-pass-remarks-with-hotness \
+; RUN:          -lto-pass-remarks-hotness-threshold=301 \
+; RUN:          -exported-symbol _main -o %t.o %t.bc
+; RUN: not FileCheck -check-prefix=YAML %s < %t.t301.yaml
+
 ; YAML:      --- !Passed
 ; YAML-NEXT: Pass:            inline
 ; YAML-NEXT: Name:            Inlined
 ; YAML-NEXT: Function:        main
 ; YAML-NEXT: Hotness:         300
 ; YAML-NEXT: Args:
 ; YAML-NEXT:   - Callee:          foo
 ; YAML-NEXT:   - String:          ' inlined into '

llvm/test/tools/gold/X86/opt-remarks.ll

 ; Test plugin options for opt-remarks.
 ; RUN: llvm-as %s -o %t.o
 ; RUN: %gold -m elf_x86_64 -plugin %llvmshlibdir/LLVMgold%shlibext -shared \
 ; RUN:    -plugin-opt=save-temps \
 ; RUN:    -plugin-opt=opt-remarks-passes=inline \
 ; RUN:    -plugin-opt=opt-remarks-format=yaml \
 ; RUN:    -plugin-opt=opt-remarks-filename=%t.yaml %t.o -o %t2.o 2>&1
 ; RUN: llvm-dis %t2.o.0.4.opt.bc -o - | FileCheck %s
 ; RUN: %gold -m elf_x86_64 -plugin %llvmshlibdir/LLVMgold%shlibext -shared \
 ; RUN:    -plugin-opt=opt-remarks-passes=inline \
 ; RUN:    -plugin-opt=opt-remarks-format=yaml \
 ; RUN:    -plugin-opt=opt-remarks-with-hotness \
 ; RUN:    -plugin-opt=opt-remarks-filename=%t.hot.yaml %t.o -o %t2.o 2>&1
+; RUN: %gold -m elf_x86_64 -plugin %llvmshlibdir/LLVMgold%shlibext -shared \
+; RUN:    -plugin-opt=opt-remarks-passes=inline \
+; RUN:    -plugin-opt=opt-remarks-format=yaml \
+; RUN:    -plugin-opt=opt-remarks-with-hotness \
+; RUN:    -plugin-opt=opt-remarks-hotness-threshold=300 \
+; RUN:    -plugin-opt=opt-remarks-filename=%t.t300.yaml %t.o -o %t2.o 2>&1
+; RUN: %gold -m elf_x86_64 -plugin %llvmshlibdir/LLVMgold%shlibext -shared \
+; RUN:    -plugin-opt=opt-remarks-passes=inline \
+; RUN:    -plugin-opt=opt-remarks-format=yaml \
+; RUN:    -plugin-opt=opt-remarks-with-hotness \
+; RUN:    -plugin-opt=opt-remarks-hotness-threshold=301 \
+; RUN:    -plugin-opt=opt-remarks-filename=%t.t301.yaml %t.o -o %t2.o 2>&1
 ; RUN: cat %t.yaml | FileCheck %s -check-prefix=YAML
 ; RUN: cat %t.hot.yaml | FileCheck %s -check-prefix=YAML-HOT
+; RUN: FileCheck %s -check-prefix=YAML-HOT < %t.t300.yaml
+; RUN: count 0 < %t.t301.yaml
 
 ; Check that @f is inlined after optimizations.
 ; CHECK-LABEL: define i32 @_start
 ; CHECK-NEXT:  %a.i = tail call i32 @bar()
 ; CHECK-NEXT:  ret i32 %a.i
 ; CHECK-NEXT: }
 
 ; YAML: --- !Missed

 ; YAML-NEXT:   - String:          ' with '
 ; YAML-NEXT:   - String:          '(cost='
 ; YAML-NEXT:   - Cost:            '0'
 ; YAML-NEXT:   - String:          ', threshold='
 ; YAML-NEXT:   - Threshold:       '337'
 ; YAML-NEXT:   - String:          ')'
 ; YAML-NEXT: ...
 
-; YAML-HOT: ...
 ; YAML-HOT: --- !Passed
 ; YAML-HOT: Pass:            inline
 ; YAML-HOT-NEXT: Name:            Inlined
 ; YAML-HOT-NEXT: Function:        _start
 ; YAML-HOT-NEXT: Hotness:         300
 ; YAML-HOT-NEXT: Args:
 ; YAML-HOT-NEXT:   - Callee:          f
 ; YAML-HOT-NEXT:   - String:          ' inlined into '

llvm/tools/gold/gold-plugin.cpp

 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/Threading.h"
 #include "llvm/Support/raw_ostream.h"
 #include <list>
 #include <map>
 #include <plugin-api.h>
 #include <string>
 #include <system_error>
 #include <utility>
 #include <vector>
 
 // FIXME: remove this declaration when we stop maintaining Ubuntu Quantal and
 // Precise and Debian Wheezy (binutils 2.23 is required)
 #define LDPO_PIE 3
#endif
   static std::string stats_file;
   // Asserts that LTO link has whole program visibility
   static bool whole_program_visibility = false;
 
   // Optimization remarks filename, accepted passes and hotness options
   static std::string RemarksFilename;
   static std::string RemarksPasses;
   static bool RemarksWithHotness = false;
+  static unsigned RemarksHotnessThreshold = 0;
   static std::string RemarksFormat;
 
   // Context sensitive PGO options.
   static std::string cs_profile_path;
   static bool cs_pgo_gen = false;
 
   static void process_plugin_option(const char *opt_)
   {
static void process_plugin_option(const char *opt_)
     } else if (opt.consume_front("dwo_dir=")) {
       dwo_dir = std::string(opt);
     } else if (opt.consume_front("opt-remarks-filename=")) {
       RemarksFilename = std::string(opt);
     } else if (opt.consume_front("opt-remarks-passes=")) {
       RemarksPasses = std::string(opt);
     } else if (opt == "opt-remarks-with-hotness") {
       RemarksWithHotness = true;
+    } else if (opt.consume_front("opt-remarks-hotness-threshold=")) {
+      if (opt.getAsInteger(10, RemarksHotnessThreshold)) {
+        message(LDPL_FATAL, "Invalid remarks hotness threshold: %s", opt);
+      }
     } else if (opt.consume_front("opt-remarks-format=")) {
       RemarksFormat = std::string(opt);
     } else if (opt.consume_front("stats-file=")) {
       stats_file = std::string(opt);
     } else {
       // Save this option to pass to the code generator.
       // ParseCommandLineOptions() expects argv[0] to be program name. Lazily
       // add that.
static std::unique_ptr<LTO> createLTO(IndexWriteCallback OnIndexWrite,
   Conf.RunCSIRInstr = options::cs_pgo_gen;
 
   Conf.DwoDir = options::dwo_dir;
 
   // Set up optimization remarks handling.
   Conf.RemarksFilename = options::RemarksFilename;
   Conf.RemarksPasses = options::RemarksPasses;
   Conf.RemarksWithHotness = options::RemarksWithHotness;
+  Conf.RemarksHotnessThreshold = options::RemarksHotnessThreshold;
+  ;
   Conf.RemarksFormat = options::RemarksFormat;
 
   // Use new pass manager if set in driver
   Conf.UseNewPM = options::new_pass_manager;
   // Debug new pass manager if requested
   Conf.DebugPassManager = options::debug_pass_manager;
 
   Conf.HasWholeProgramVisibility = options::whole_program_visibility;

llvm/tools/llvm-lto2/llvm-lto2.cpp

static cl::opt<std::string> DefaultTriple(
     cl::desc(
         "Replace unspecified target triples in input files with this triple"));
 
 static cl::opt<bool> RemarksWithHotness(
     "pass-remarks-with-hotness",
     cl::desc("With PGO, include profile count in optimization remarks"),
     cl::Hidden);
 
+static cl::opt<unsigned>
+    RemarksHotnessThreshold("pass-remarks-hotness-threshold",
+                            cl::desc("Minimum profile count required for an "
+                                     "optimization remark to be output"),
+                            cl::init(0), cl::Hidden);
+
 static cl::opt<std::string>
     RemarksFilename("pass-remarks-output",
                     cl::desc("Output filename for pass remarks"),
                     cl::value_desc("filename"));
 
 static cl::opt<std::string>
     RemarksPasses("pass-remarks-filter",
                   cl::desc("Only record optimization remarks from passes whose "
static int run(int argc, char **argv) {
   if (SaveTemps)
     check(Conf.addSaveTemps(OutputFilename + "."),
           "Config::addSaveTemps failed");
 
   // Optimization remarks.
   Conf.RemarksFilename = RemarksFilename;
   Conf.RemarksPasses = RemarksPasses;
   Conf.RemarksWithHotness = RemarksWithHotness;
+  Conf.RemarksHotnessThreshold = RemarksHotnessThreshold;
   Conf.RemarksFormat = RemarksFormat;
 
   Conf.SampleProfile = SamplePGOFile;
   Conf.CSIRProfile = CSPGOFile;
   Conf.RunCSIRInstr = RunCSIRInstr;
 
   // Run a custom pipeline, if asked for.
   Conf.OptPipeline = OptPipeline;