This is an archive of the discontinued LLVM Phabricator instance.

[CSSPGO] Pseudo probe encoding and emission.
ClosedPublic

Authored by hoy on Nov 20 2020, 9:26 AM.

Details

Summary

This change implements pseudo probe encoding and emission for CSSPGO. Please see RFC here for more context: https://groups.google.com/g/llvm-dev/c/1p1rdYbL93s

Pseudo probes are in the form of intrinsic calls on IR/MIR but they do not turn into any machine instructions. Instead they are emitted into the binary as a piece of data in standalone sections. The probe-specific sections are not needed to be loaded into memory at execution time, thus they do not incur a runtime overhead. 

ELF object emission

The binary data to emit are organized as two ELF sections, i.e, the .pseudo_probe_desc section and the .pseudo_probe section. The .pseudo_probe_desc section stores a function descriptor for each function and the .pseudo_probe section stores the actual probes, each fo which corresponds to an IR basic block or an IR function callsite. A function descriptor is stored as a module-level metadata during the compilation and is serialized into the object file during object emission.

Both the probe descriptors and pseudo probes can be emitted into a separate ELF section per function to leverage the linker for deduplication. A .pseudo_probe section shares the same COMDAT group with the function code so that when the function is dead, the probes are dead and disposed too. On the contrary, a .pseudo_probe_desc section has its own COMDAT group. This is because even if a function is dead, its probes may be inlined into other functions and its descriptor is still needed by the profile generation tool.

The format of .pseudo_probe_desc section looks like:

.section   .pseudo_probe_desc,"",@progbits
.quad   6309742469962978389  // Func GUID
.quad   4294967295           // Func Hash
.byte   9                    // Length of func name
.ascii  "_Z5funcAi"          // Func name
.quad   7102633082150537521
.quad   138828622701
.byte   12
.ascii  "_Z8funcLeafi"
.quad   446061515086924981
.quad   4294967295
.byte   9
.ascii  "_Z5funcBi"
.quad   -2016976694713209516
.quad   72617220756
.byte   7
.ascii  "_Z3fibi"

For each .pseudoprobe section, the encoded binary data consists of a single function record corresponding to an outlined function (i.e, a function with a code entry in the .text section). A function record has the following format :

FUNCTION BODY (one for each outlined function present in the text section)
    GUID (uint64)
        GUID of the function
    NPROBES (ULEB128)
        Number of probes originating from this function.
    NUM_INLINED_FUNCTIONS (ULEB128)
        Number of callees inlined into this function, aka number of
        first-level inlinees
    PROBE RECORDS
        A list of NPROBES entries. Each entry contains:
          INDEX (ULEB128)
          TYPE (uint4)
            0 - block probe, 1 - indirect call, 2 - direct call
          ATTRIBUTE (uint3)
            reserved
          ADDRESS_TYPE (uint1)
            0 - code address, 1 - address delta
          CODE_ADDRESS (uint64 or ULEB128)
            code address or address delta, depending on ADDRESS_TYPE
    INLINED FUNCTION RECORDS
        A list of NUM_INLINED_FUNCTIONS entries describing each of the inlined
        callees.  Each record contains:
          INLINE SITE
            GUID of the inlinee (uint64)
            ID of the callsite probe (ULEB128)
          FUNCTION BODY
            A FUNCTION BODY entry describing the inlined function.

To support building a context-sensitive profile, probes from inlinees are grouped by their inline contexts. An inline context is logically a call path through which a callee function lands in a caller function. The probe emitter builds an inline tree based on the debug metadata for each outlined function in the form of a trie tree. A tree root is the outlined function. Each tree edge stands for a callsite where inlining happens. Pseudo probes originating from an inlinee function are stored in a tree node and the tree path starting from the root all the way down to the tree node is the inline context of the probes. The emission happens on the whole tree top-down recursively. Probes of a tree node will be emitted altogether with their direct parent edge. Since a pseudo probe corresponds to a real code address, for size savings, the address is encoded as a delta from the previous probe except for the first probe. Variant-sized integer encoding, aka LEB128, is used for address delta and probe index.

Assembling

Pseudo probes can be printed as assembly directives alternatively. This allows for good assembly code readability and also provides a view of how optimizations and pseudo probes affect each other, especially helpful for diff time assembly analysis.

A pseudo probe directive has the following operands in order: function GUID, probe index, probe type, probe attributes and inline context. The directive is generated by the compiler and can be parsed by the assembler to form an encoded .pseudoprobe section in the object file.

A example assembly looks like:

foo2: # @foo2
# %bb.0: # %bb0
pushq %rax
testl %edi, %edi
.pseudoprobe 837061429793323041 1 0 0
je .LBB1_1
# %bb.2: # %bb2
.pseudoprobe 837061429793323041 6 2 0
callq foo
.pseudoprobe 837061429793323041 3 0 0
.pseudoprobe 837061429793323041 4 0 0
popq %rax
retq
.LBB1_1: # %bb1
.pseudoprobe 837061429793323041 5 1 0
callq *%rsi
.pseudoprobe 837061429793323041 2 0 0
.pseudoprobe 837061429793323041 4 0 0
popq %rax
retq
# -- End function
.section .pseudo_probe_desc,"",@progbits
.quad 6699318081062747564
.quad 72617220756
.byte 3
.ascii "foo"
.quad 837061429793323041
.quad 281547593931412
.byte 4
.ascii "foo2"

With inlining turned on, the assembly may look different around %bb2 with an inlined probe:

# %bb.2:                                # %bb2
.pseudoprobe    837061429793323041 3 0
.pseudoprobe    6699318081062747564 1 0 @ 837061429793323041:6
.pseudoprobe    837061429793323041 4 0
popq    %rax
retq

Disassembling

We have a disassembling tool (llvm-profgen) that can display disassembly alongside with pseudo probes. So far it only supports ELF executable file.

An example disassembly looks like:

00000000002011a0 <foo2>:
  2011a0: 50                    push   rax
  2011a1: 85 ff                 test   edi,edi
  [Probe]:  FUNC: foo2  Index: 1  Type: Block
  2011a3: 74 02                 je     2011a7 <foo2+0x7>
  [Probe]:  FUNC: foo2  Index: 3  Type: Block
  [Probe]:  FUNC: foo2  Index: 4  Type: Block
  [Probe]:  FUNC: foo   Index: 1  Type: Block  Inlined: @ foo2:6
  2011a5: 58                    pop    rax
  2011a6: c3                    ret
  [Probe]:  FUNC: foo2  Index: 2  Type: Block
  2011a7: bf 01 00 00 00        mov    edi,0x1
  [Probe]:  FUNC: foo2  Index: 5  Type: IndirectCall
  2011ac: ff d6                 call   rsi
  [Probe]:  FUNC: foo2  Index: 4  Type: Block
  2011ae: 58                    pop    rax
  2011af: c3                    ret

Diff Detail

Event Timeline

hoy created this revision.Nov 20 2020, 9:26 AM
Herald added a project: Restricted Project. · View Herald TranscriptNov 20 2020, 9:26 AM
hoy requested review of this revision.Nov 20 2020, 9:26 AM
hoy edited the summary of this revision. (Show Details)Nov 20 2020, 9:35 AM
hoy edited the summary of this revision. (Show Details)Nov 20 2020, 9:42 AM
hoy added reviewers: wmi, davidxl, wenlei, wlei.
wmi added a comment.Dec 8 2020, 12:42 PM

Sorry for the delay in review. Some general questions.

I remember Wenlei mentioned in the RFC discussion that the overall binary size increase from pseudo probe is ~12% if CFG encoding is added. Is the size increase for object file from pseudo probe about the same as binary? I am asking because the total size of linker input matters for us. Previously we may use fission to separate the debug information from object file so it won't be accounted as linker input.

According to the description, The address range will be split into multiple ranges by the address of pseudo probe. If some BB somehow doesn't have block type pseudo probe, the addresses in the BB will be mistakenly associated with the pseudo probe located before it in the binary. I don't know whether you saw cases like that. A random thought is we may generate some fake pseudo probes for the BBs which don't have them, as a mark of the BB boundary.

llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp
52

Is the comment up-to-date? Looks like the InlineStack only contains guid and probeid?

llvm/lib/MC/MCAsmStreamer.cpp
2135–2136

It is a reverse of sequence as the context string in the profile. Is it intentional?

2137

Is it unnecessary?

llvm/lib/MC/MCPseudoProbe.cpp
93–96

I think inline stack only contains guids and pseudo probe ids but here the example mentions line numbers in the stack. I understand probe id is extracted from the discriminator in the debug Location but it will be confusing if people don't know about it.

hoy added a comment.Dec 8 2020, 3:23 PM
In D91878#2440875, @wmi wrote:

Sorry for the delay in review. Some general questions.

I remember Wenlei mentioned in the RFC discussion that the overall binary size increase from pseudo probe is ~12% if CFG encoding is added. Is the size increase for object file from pseudo probe about the same as binary? I am asking because the total size of linker input matters for us. Previously we may use fission to separate the debug information from object file so it won't be accounted as linker input.

According to the description, The address range will be split into multiple ranges by the address of pseudo probe. If some BB somehow doesn't have block type pseudo probe, the addresses in the BB will be mistakenly associated with the pseudo probe located before it in the binary. I don't know whether you saw cases like that. A random thought is we may generate some fake pseudo probes for the BBs which don't have them, as a mark of the BB boundary.

Thanks for reviewing this patch!

Regarding the binary size, an object file will likely have the same size inflation with an executable since the size of the pseudo probe sections are propositional to the text section. If that is an issue to the linker, we can also perform fission to the pseudo probe data. Pseudo probe encoding is currently done on function (comdat) level (see MCPseudoProbeSection::Emit in MCPseudoProbe.cpp`), which means a function always gets a separate real absolute probe encoded in the first place followed by relative address delta encoded for subsequent probes. Therefore fission on function level should be fine. Fission on block level won't work, as you pointed out. Would function-level fission work for you?

llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp
52

Good catch. The numbers should refer to probe id.

llvm/lib/MC/MCAsmStreamer.cpp
2135–2136

Yes, it's intentional. We feel that it's easier to read when the inline context gets long. The decoding tool (https://reviews.llvm.org/D92334) also prints in the reverse order. What do you think? Would you perfer the profile context order?

2137

Good catch. Will remove it.

llvm/lib/MC/MCPseudoProbe.cpp
93–96

Right, the comment is out-dated. Will replace line numbers with probe ids.

hoy updated this revision to Diff 310371.Dec 8 2020, 3:24 PM
hoy edited the summary of this revision. (Show Details)

Addressing Wei's feedbacks.

hoy updated this revision to Diff 310376.Dec 8 2020, 3:47 PM

Fixing one more outdated comment.

wmi added a comment.Dec 8 2020, 5:00 PM
In D91878#2441363, @hoy wrote:
In D91878#2440875, @wmi wrote:

Sorry for the delay in review. Some general questions.

I remember Wenlei mentioned in the RFC discussion that the overall binary size increase from pseudo probe is ~12% if CFG encoding is added. Is the size increase for object file from pseudo probe about the same as binary? I am asking because the total size of linker input matters for us. Previously we may use fission to separate the debug information from object file so it won't be accounted as linker input.

According to the description, The address range will be split into multiple ranges by the address of pseudo probe. If some BB somehow doesn't have block type pseudo probe, the addresses in the BB will be mistakenly associated with the pseudo probe located before it in the binary. I don't know whether you saw cases like that. A random thought is we may generate some fake pseudo probes for the BBs which don't have them, as a mark of the BB boundary.

Thanks for reviewing this patch!

Regarding the binary size, an object file will likely have the same size inflation with an executable since the size of the pseudo probe sections are propositional to the text section. If that is an issue to the linker, we can also perform fission to the pseudo probe data. Pseudo probe encoding is currently done on function (comdat) level (see MCPseudoProbeSection::Emit in MCPseudoProbe.cpp`), which means a function always gets a separate real absolute probe encoded in the first place followed by relative address delta encoded for subsequent probes. Therefore fission on function level should be fine. Fission on block level won't work, as you pointed out. Would function-level fission work for you?

Yes, function-level fission works for us. Adding that support later will be helpful.

hoy added a comment.Dec 8 2020, 5:27 PM
In D91878#2441512, @wmi wrote:
In D91878#2441363, @hoy wrote:
In D91878#2440875, @wmi wrote:

Sorry for the delay in review. Some general questions.

I remember Wenlei mentioned in the RFC discussion that the overall binary size increase from pseudo probe is ~12% if CFG encoding is added. Is the size increase for object file from pseudo probe about the same as binary? I am asking because the total size of linker input matters for us. Previously we may use fission to separate the debug information from object file so it won't be accounted as linker input.

According to the description, The address range will be split into multiple ranges by the address of pseudo probe. If some BB somehow doesn't have block type pseudo probe, the addresses in the BB will be mistakenly associated with the pseudo probe located before it in the binary. I don't know whether you saw cases like that. A random thought is we may generate some fake pseudo probes for the BBs which don't have them, as a mark of the BB boundary.

Thanks for reviewing this patch!

Regarding the binary size, an object file will likely have the same size inflation with an executable since the size of the pseudo probe sections are propositional to the text section. If that is an issue to the linker, we can also perform fission to the pseudo probe data. Pseudo probe encoding is currently done on function (comdat) level (see MCPseudoProbeSection::Emit in MCPseudoProbe.cpp`), which means a function always gets a separate real absolute probe encoded in the first place followed by relative address delta encoded for subsequent probes. Therefore fission on function level should be fine. Fission on block level won't work, as you pointed out. Would function-level fission work for you?

Yes, function-level fission works for us. Adding that support later will be helpful.

Cool. It might already be there depending on how fission will be done. The current implementation always emit a relocation from a function's pseudo probe section to the function's text section. The rest of the function's probe will be encoded as a delta from that relocation. We will add more support if this is not sufficient.

hoy added inline comments.Dec 8 2020, 9:53 PM
llvm/lib/MC/MCAsmStreamer.cpp
2135–2136

I might be wrong. The decoding tool prints in the profile context (caller-callee) order. We should do the same here.

hoy updated this revision to Diff 310560.Dec 9 2020, 9:12 AM

Print inline context in caller-callee order during asm printing

wmi accepted this revision.Dec 9 2020, 8:29 PM

Please add a message for all the assertions. Other than that, LGTM.

This revision is now accepted and ready to land.Dec 9 2020, 8:29 PM
hoy updated this revision to Diff 310921.Dec 10 2020, 8:53 AM

Adding messages for assertions.

This revision was landed with ongoing or failed builds.Dec 10 2020, 9:50 AM
This revision was automatically updated to reflect the committed changes.

This breaks the build with gcc5:

llvm/lib/MC/MCParser/AsmParser.cpp:5836:49: error: converting to 'llvm::InlineSite {aka std::tuple<long unsigned int, unsigned int>}' from initializer list would use explicit constructor 'constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = long int&; _U2 = long int&; <template-parameter-2-3> = void; _T1 = long unsigned int; _T2 = unsigned int]'
     InlineSite Site = {CallerGuid, CallerProbeId};
hoy added a comment.Dec 10 2020, 11:17 AM

This breaks the build with gcc5:

llvm/lib/MC/MCParser/AsmParser.cpp:5836:49: error: converting to 'llvm::InlineSite {aka std::tuple<long unsigned int, unsigned int>}' from initializer list would use explicit constructor 'constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = long int&; _U2 = long int&; <template-parameter-2-3> = void; _T1 = long unsigned int; _T2 = unsigned int]'
     InlineSite Site = {CallerGuid, CallerProbeId};

Thanks for reporting the issue. I will change to use an explicit constructor.

hoy added a comment.Dec 10 2020, 11:24 AM
In D91878#2446557, @hoy wrote:

This breaks the build with gcc5:

llvm/lib/MC/MCParser/AsmParser.cpp:5836:49: error: converting to 'llvm::InlineSite {aka std::tuple<long unsigned int, unsigned int>}' from initializer list would use explicit constructor 'constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = long int&; _U2 = long int&; <template-parameter-2-3> = void; _T1 = long unsigned int; _T2 = unsigned int]'
     InlineSite Site = {CallerGuid, CallerProbeId};

Thanks for reporting the issue. I will change to use an explicit constructor.

Fixed by changing to InlineSite Site(CallerGuid, CallerProbeId);

craig.topper added inline comments.
llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp
65

This line doesn't compile with gcc 5.4

llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp:65:62: error: converting to ‘std::tuple<long unsigned int, unsigned int>’ from initializer list would use explicit constructor ‘constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = long unsigned int&; _U2 = long unsigned int&; <template-parameter-2-3> = void; _T1 = long unsigned int; _T2 = unsigned int]’
     ReversedInlineStack.push_back({CallerGuid, CallerProbeId})

Changing it to emplace_back and dropping the curly braces seemed to work.

hoy marked an inline comment as done.Dec 10 2020, 11:56 AM
hoy added inline comments.
llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp
65

Thanks for the suggestion! Fix pushed in just now.

hoy marked an inline comment as done.Dec 10 2020, 4:00 PM
hoy added a subscriber: hctim.

@hctim Can you please let me know why this is reverted? Is it breaking any build? Thanks.

hctim added a comment.Dec 10 2020, 5:00 PM

Hi - yep, sorry for taking a sec to comment here. It broke the sanitizer buildbots here: http://lab.llvm.org:8011/#/builders/5/builds/2269

You can learn more about how to repro the exact bot here: https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild

I've copied the ASan errors below just for your convenience :)

FAIL: LLVM :: Transforms/SampleProfile/pseudo-probe-emit-inline.ll (65962 of 72157)
******************** TEST 'LLVM :: Transforms/SampleProfile/pseudo-probe-emit-inline.ll' FAILED ********************
Script:
--
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/opt < /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll -passes='pseudo-probe,cgscc(inline)' -function-sections -mtriple=x86_64-unknown-linux-gnu -S -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp
: 'RUN: at line 3';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp --check-prefix=CHECK-IL
: 'RUN: at line 4';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc -pseudo-probe-for-profiling -function-sections </b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp -filetype=asm -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp1
: 'RUN: at line 5';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp1 --check-prefix=CHECK-ASM
: 'RUN: at line 6';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc -pseudo-probe-for-profiling -function-sections </b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp -filetype=obj -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp2
: 'RUN: at line 7';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-objdump --section-headers  /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp2 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll --check-prefix=CHECK-OBJ
: 'RUN: at line 8';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-mc -filetype=asm </b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp1 -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp3
: 'RUN: at line 9';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp3 --check-prefix=CHECK-ASM
: 'RUN: at line 10';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-mc -filetype=obj </b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp1 -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp4
: 'RUN: at line 11';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-objdump --section-headers  /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp4 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll --check-prefix=CHECK-OBJ
--
Exit Code: 1
Command Output (stderr):
--
=================================================================
==13817==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x462a9cd in operator new(unsigned long) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
    #1 0x8727eac in llvm::AsmPrinter::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:339:10
    #2 0x9b380d5 in llvm::FPPassManager::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1495:41
    #3 0x9b222a8 in runOnModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1529:41
    #4 0x9b222a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
    #5 0x463416f in compileModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #6 0x463416f in main /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:363:22
    #7 0x7fa76d5fa09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
Indirect leak of 1536 byte(s) in 1 object(s) allocated from:
    #0 0x462a9cd in operator new(unsigned long) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
    #1 0x88de0b6 in allocateBuckets /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:871:9
    #2 0x88de0b6 in llvm::DenseMap<llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >::grow(unsigned int) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:805:5
    #3 0x88ddea4 in grow /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:536:36
    #4 0x88ddea4 in llvm::detail::DenseMapPair<llvm::StringRef, unsigned long>* llvm::DenseMapBase<llvm::DenseMap<llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >, llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >::InsertIntoBucketImpl<llvm::StringRef>(llvm::StringRef const&, llvm::StringRef const&, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long>*) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:580:13
    #5 0x88dbb41 in InsertIntoBucket<const llvm::StringRef &> /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:546:17
    #6 0x88dbb41 in FindAndConstruct /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:325:13
    #7 0x88dbb41 in operator[] /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:329:12
    #8 0x88dbb41 in llvm::PseudoProbeHandler::PseudoProbeHandler(llvm::AsmPrinter*, llvm::Module*) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp:40:5
    #9 0x8727ec1 in llvm::AsmPrinter::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:339:14
    #10 0x9b380d5 in llvm::FPPassManager::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1495:41
    #11 0x9b222a8 in runOnModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1529:41
    #12 0x9b222a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
    #13 0x463416f in compileModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #14 0x463416f in main /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:363:22
    #15 0x7fa76d5fa09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
SUMMARY: AddressSanitizer: 1576 byte(s) leaked in 2 allocation(s).
--
********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 
FAIL: LLVM :: Transforms/SampleProfile/pseudo-probe-emit.ll (66047 of 72157)
******************** TEST 'LLVM :: Transforms/SampleProfile/pseudo-probe-emit.ll' FAILED ********************
Script:
--
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/opt < /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll -passes=pseudo-probe -function-sections -S -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp
: 'RUN: at line 3';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp --check-prefix=CHECK-IL
: 'RUN: at line 4';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp -pseudo-probe-for-profiling -stop-after=pseudo-probe-inserter -o - | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll --check-prefix=CHECK-MIR
: 'RUN: at line 5';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp -pseudo-probe-for-profiling -function-sections -filetype=asm -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp1
: 'RUN: at line 6';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp1 --check-prefix=CHECK-ASM
: 'RUN: at line 7';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp -pseudo-probe-for-profiling -function-sections -filetype=obj -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp2
: 'RUN: at line 8';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-objdump --section-headers  /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp2 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll --check-prefix=CHECK-OBJ
: 'RUN: at line 9';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-mc /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp1 -filetype=obj -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp3
: 'RUN: at line 10';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-objdump --section-headers  /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp3 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll --check-prefix=CHECK-OBJ
--
Exit Code: 1
Command Output (stderr):
--
=================================================================
==15119==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x462a9cd in operator new(unsigned long) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
    #1 0x8727eac in llvm::AsmPrinter::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:339:10
    #2 0x9b380d5 in llvm::FPPassManager::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1495:41
    #3 0x9b222a8 in runOnModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1529:41
    #4 0x9b222a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
    #5 0x463416f in compileModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #6 0x463416f in main /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:363:22
    #7 0x7f642eb0909a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
Indirect leak of 1536 byte(s) in 1 object(s) allocated from:
    #0 0x462a9cd in operator new(unsigned long) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
    #1 0x88de0b6 in allocateBuckets /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:871:9
    #2 0x88de0b6 in llvm::DenseMap<llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >::grow(unsigned int) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:805:5
    #3 0x88ddea4 in grow /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:536:36
    #4 0x88ddea4 in llvm::detail::DenseMapPair<llvm::StringRef, unsigned long>* llvm::DenseMapBase<llvm::DenseMap<llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >, llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >::InsertIntoBucketImpl<llvm::StringRef>(llvm::StringRef const&, llvm::StringRef const&, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long>*) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:580:13
    #5 0x88dbb41 in InsertIntoBucket<const llvm::StringRef &> /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:546:17
    #6 0x88dbb41 in FindAndConstruct /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:325:13
    #7 0x88dbb41 in operator[] /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:329:12
    #8 0x88dbb41 in llvm::PseudoProbeHandler::PseudoProbeHandler(llvm::AsmPrinter*, llvm::Module*) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp:40:5
    #9 0x8727ec1 in llvm::AsmPrinter::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:339:14
    #10 0x9b380d5 in llvm::FPPassManager::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1495:41
    #11 0x9b222a8 in runOnModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1529:41
    #12 0x9b222a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
    #13 0x463416f in compileModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #14 0x463416f in main /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:363:22
    #15 0x7f642eb0909a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
SUMMARY: AddressSanitizer: 1576 byte(s) leaked in 2 allocation(s).
--
********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
********************
Failed Tests (2):
  LLVM :: Transforms/SampleProfile/pseudo-probe-emit-inline.ll
  LLVM :: Transforms/SampleProfile/pseudo-probe-emit.ll
hoy added a comment.Dec 10 2020, 5:32 PM

Hi - yep, sorry for taking a sec to comment here. It broke the sanitizer buildbots here: http://lab.llvm.org:8011/#/builders/5/builds/2269

You can learn more about how to repro the exact bot here: https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild

I've copied the ASan errors below just for your convenience :)

FAIL: LLVM :: Transforms/SampleProfile/pseudo-probe-emit-inline.ll (65962 of 72157)
******************** TEST 'LLVM :: Transforms/SampleProfile/pseudo-probe-emit-inline.ll' FAILED ********************
Script:
--
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/opt < /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll -passes='pseudo-probe,cgscc(inline)' -function-sections -mtriple=x86_64-unknown-linux-gnu -S -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp
: 'RUN: at line 3';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp --check-prefix=CHECK-IL
: 'RUN: at line 4';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc -pseudo-probe-for-profiling -function-sections </b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp -filetype=asm -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp1
: 'RUN: at line 5';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp1 --check-prefix=CHECK-ASM
: 'RUN: at line 6';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc -pseudo-probe-for-profiling -function-sections </b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp -filetype=obj -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp2
: 'RUN: at line 7';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-objdump --section-headers  /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp2 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll --check-prefix=CHECK-OBJ
: 'RUN: at line 8';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-mc -filetype=asm </b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp1 -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp3
: 'RUN: at line 9';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp3 --check-prefix=CHECK-ASM
: 'RUN: at line 10';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-mc -filetype=obj </b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp1 -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp4
: 'RUN: at line 11';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-objdump --section-headers  /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit-inline.ll.tmp4 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll --check-prefix=CHECK-OBJ
--
Exit Code: 1
Command Output (stderr):
--
=================================================================
==13817==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x462a9cd in operator new(unsigned long) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
    #1 0x8727eac in llvm::AsmPrinter::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:339:10
    #2 0x9b380d5 in llvm::FPPassManager::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1495:41
    #3 0x9b222a8 in runOnModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1529:41
    #4 0x9b222a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
    #5 0x463416f in compileModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #6 0x463416f in main /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:363:22
    #7 0x7fa76d5fa09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
Indirect leak of 1536 byte(s) in 1 object(s) allocated from:
    #0 0x462a9cd in operator new(unsigned long) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
    #1 0x88de0b6 in allocateBuckets /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:871:9
    #2 0x88de0b6 in llvm::DenseMap<llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >::grow(unsigned int) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:805:5
    #3 0x88ddea4 in grow /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:536:36
    #4 0x88ddea4 in llvm::detail::DenseMapPair<llvm::StringRef, unsigned long>* llvm::DenseMapBase<llvm::DenseMap<llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >, llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >::InsertIntoBucketImpl<llvm::StringRef>(llvm::StringRef const&, llvm::StringRef const&, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long>*) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:580:13
    #5 0x88dbb41 in InsertIntoBucket<const llvm::StringRef &> /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:546:17
    #6 0x88dbb41 in FindAndConstruct /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:325:13
    #7 0x88dbb41 in operator[] /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:329:12
    #8 0x88dbb41 in llvm::PseudoProbeHandler::PseudoProbeHandler(llvm::AsmPrinter*, llvm::Module*) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp:40:5
    #9 0x8727ec1 in llvm::AsmPrinter::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:339:14
    #10 0x9b380d5 in llvm::FPPassManager::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1495:41
    #11 0x9b222a8 in runOnModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1529:41
    #12 0x9b222a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
    #13 0x463416f in compileModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #14 0x463416f in main /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:363:22
    #15 0x7fa76d5fa09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
SUMMARY: AddressSanitizer: 1576 byte(s) leaked in 2 allocation(s).
--
********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 
FAIL: LLVM :: Transforms/SampleProfile/pseudo-probe-emit.ll (66047 of 72157)
******************** TEST 'LLVM :: Transforms/SampleProfile/pseudo-probe-emit.ll' FAILED ********************
Script:
--
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/opt < /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll -passes=pseudo-probe -function-sections -S -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp
: 'RUN: at line 3';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp --check-prefix=CHECK-IL
: 'RUN: at line 4';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp -pseudo-probe-for-profiling -stop-after=pseudo-probe-inserter -o - | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll --check-prefix=CHECK-MIR
: 'RUN: at line 5';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp -pseudo-probe-for-profiling -function-sections -filetype=asm -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp1
: 'RUN: at line 6';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll < /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp1 --check-prefix=CHECK-ASM
: 'RUN: at line 7';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llc /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp -pseudo-probe-for-profiling -function-sections -filetype=obj -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp2
: 'RUN: at line 8';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-objdump --section-headers  /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp2 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll --check-prefix=CHECK-OBJ
: 'RUN: at line 9';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-mc /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp1 -filetype=obj -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp3
: 'RUN: at line 10';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/llvm-objdump --section-headers  /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/test/Transforms/SampleProfile/Output/pseudo-probe-emit.ll.tmp3 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_asan/bin/FileCheck --allow-unused-prefixes=false /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll --check-prefix=CHECK-OBJ
--
Exit Code: 1
Command Output (stderr):
--
=================================================================
==15119==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x462a9cd in operator new(unsigned long) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
    #1 0x8727eac in llvm::AsmPrinter::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:339:10
    #2 0x9b380d5 in llvm::FPPassManager::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1495:41
    #3 0x9b222a8 in runOnModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1529:41
    #4 0x9b222a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
    #5 0x463416f in compileModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #6 0x463416f in main /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:363:22
    #7 0x7f642eb0909a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
Indirect leak of 1536 byte(s) in 1 object(s) allocated from:
    #0 0x462a9cd in operator new(unsigned long) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
    #1 0x88de0b6 in allocateBuckets /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:871:9
    #2 0x88de0b6 in llvm::DenseMap<llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >::grow(unsigned int) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:805:5
    #3 0x88ddea4 in grow /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:536:36
    #4 0x88ddea4 in llvm::detail::DenseMapPair<llvm::StringRef, unsigned long>* llvm::DenseMapBase<llvm::DenseMap<llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >, llvm::StringRef, unsigned long, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long> >::InsertIntoBucketImpl<llvm::StringRef>(llvm::StringRef const&, llvm::StringRef const&, llvm::detail::DenseMapPair<llvm::StringRef, unsigned long>*) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:580:13
    #5 0x88dbb41 in InsertIntoBucket<const llvm::StringRef &> /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:546:17
    #6 0x88dbb41 in FindAndConstruct /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:325:13
    #7 0x88dbb41 in operator[] /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/include/llvm/ADT/DenseMap.h:329:12
    #8 0x88dbb41 in llvm::PseudoProbeHandler::PseudoProbeHandler(llvm::AsmPrinter*, llvm::Module*) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp:40:5
    #9 0x8727ec1 in llvm::AsmPrinter::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:339:14
    #10 0x9b380d5 in llvm::FPPassManager::doInitialization(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1495:41
    #11 0x9b222a8 in runOnModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1529:41
    #12 0x9b222a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
    #13 0x463416f in compileModule /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #14 0x463416f in main /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/tools/llc/llc.cpp:363:22
    #15 0x7f642eb0909a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
SUMMARY: AddressSanitizer: 1576 byte(s) leaked in 2 allocation(s).
--
********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
********************
Failed Tests (2):
  LLVM :: Transforms/SampleProfile/pseudo-probe-emit-inline.ll
  LLVM :: Transforms/SampleProfile/pseudo-probe-emit.ll

Thanks for the pointers! I have reproduced and fixed the failure. Will re-push the patch in.