This is an archive of the discontinued LLVM Phabricator instance.

[llvm-profdata] Refactoring Sample Profile Reader to increase FDO build speed using MD5 as key to Sample Profile map
ClosedPublic

Authored by huangjd on Apr 6 2023, 1:53 PM.

Details

Summary

This is phase 1 of multiple planned improvements on the sample profile loader. The major change is to use MD5 hash code ((instead of the function itself) as the key to look up the function offset table and the profiles, which significantly reduce the time it takes to construct the map.

The optimization is based on the fact that many practical sample profiles are using MD5 values for function names to reduce profile size, so we shouldn't need to convert the MD5 to a string and then to a SampleContext and use it as the map's key, because it's extremely slow.

Several changes to note:

(1) For non-CS SampleContext, if it is already MD5 string, the hash value will be its integral value, instead of hashing the MD5 again. In phase 2 this is going to be optimized further using a union to represent MD5 function (without converting it to string) and regular function names.

(2) The SampleProfileMap is a wrapper to *map<uint64_t, FunctionSamples>, while providing interface allowing using SampleContext as key, so that existing code still work. It will check for MD5 collision (unlikely but not too unlikely, since we only takes the lower 64 bits) and handle it to at least guarantee compilation correctness (conflicting old profile is dropped, instead of returning an old profile with inconsistent context). Other code should not try to use MD5 as key to access the map directly, because it will not be able to handle MD5 collision at all. (see exception at (5) )

(3) Any SampleProfileMap::emplace() followed by SampleContext assignment if newly inserted, should be replaced with SampleProfileMap::Create(), which does the same thing.

(4) Previously we ensure an invariant that in SampleProfileMap, the key is equal to the Context of the value, for profile map that is eventually being used for output (as in llvm-profdata/llvm-profgen). Since the key became MD5 hash, only the value keeps the context now, in several places where an intermediate SampleProfileMap is created, each new FunctionSample's context is set immediately after insertion, which is necessary to "remember" the context otherwise irretrievable.

(5) When reading a profile, we cache the MD5 values of all functions, because they are used at least twice (one to index into FuncOffsetTable, the other into SampleProfileMap, more if there are additional sections), in this case the SampleProfileMap is directly accessed with MD5 value so that we don't recalculate it each time (expensive)

Performance impact:
When reading a ~1GB extbinary profile (fixed length MD5, not compressed) with 10 million function names and 2.5 million top level functions (non CS functions, each function has varying nesting level from 0 to 20), this patch improves the function offset table loading time by 20%, and improves full profile read by 5%.

Diff Detail

Unit TestsFailed

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
huangjd added a comment.EditedJul 6 2023, 5:09 PM
In D147740#4452405, @ro wrote:

...
i.e. the code tries an unaligned access, which is a no-no on strict-alignment targets like SPARC.

It looks like the error happens at line 577 of SampleProfReader.cpp. I changed `hash_code Hash = MD5SampleContextStart[Idx]`
into

hash_code Hash =
      support::endian::read<hash_code, support::little, support::unaligned>(
          MD5SampleContextStart + Idx);

Since MD5SampleContextStart can point to an unaligned position in the actual profile, if I understand it correctly, although I am not sure if this is a bug on SPARC code gen by itself since MD5SampleContextStart[Idx] is a valid C++ expression even if MD5SampleContextStart is not aligned, so the compiler should generate the correct code.

I don't have a sparc machine, so I would like to find someone to test it.

huangjd updated this revision to Diff 537945.Jul 6 2023, 6:04 PM

Update SampleProfReader for unaligned access on SPARC

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

I built it with -DLLVM_USE_SANITIZER=Memory and didn't see any (new) issue on X86, but I don't have a SPARC machine available to test on it. Is there another flag I need to use?

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

I built it with -DLLVM_USE_SANITIZER=Memory and didn't see any (new) issue on X86, but I don't have a SPARC machine available to test on it. Is there another flag I need to use?

alignment checking is a ubsan thing, not a msan thing, so it should be -DLLVM_USE_SANITIZER=Undefined. I believe that should enable alignment checking: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#available-checks

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

I built it with -DLLVM_USE_SANITIZER=Memory and didn't see any (new) issue on X86, but I don't have a SPARC machine available to test on it. Is there another flag I need to use?

alignment checking is a ubsan thing, not a msan thing, so it should be -DLLVM_USE_SANITIZER=Undefined. I believe that should enable alignment checking: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#available-checks

That option does not work for me. Note that -DLLVM_USE_SANITIZER is an option in CMake configuration, but the list you gave is compiler options. I got the error cc: error: unrecognized argument to ‘-fno-sanitize=’ option: ‘function’ when DLLVM_USE_SANITIZER is set to anything other than Memory. Could you show the exact command to build and run all tests with said sanitizer enabled?

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

I built it with -DLLVM_USE_SANITIZER=Memory and didn't see any (new) issue on X86, but I don't have a SPARC machine available to test on it. Is there another flag I need to use?

alignment checking is a ubsan thing, not a msan thing, so it should be -DLLVM_USE_SANITIZER=Undefined. I believe that should enable alignment checking: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#available-checks

That option does not work for me. Note that -DLLVM_USE_SANITIZER is an option in CMake configuration, but the list you gave is compiler options. I got the error cc: error: unrecognized argument to ‘-fno-sanitize=’ option: ‘function’ when DLLVM_USE_SANITIZER is set to anything other than Memory. Could you show the exact command to build and run all tests with said sanitizer enabled?

cmake -S llvm -B build/cmake -GNinja -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Undefined -DLLVM_ENABLE_LLD=ON -DCMAKE_C_COMPILER=$HOME/repos/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang -DCMAKE_CXX_COMPILER=$HOME/repos/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang++ works for me. I'm using Chrome's toolchain which is a very close to ToT clang package, that error message seems to imply that the host compiler you're using isn't a recent clang.

huangjd updated this revision to Diff 539339.Jul 11 2023, 4:44 PM

updated test to fix sign comparison warning

@ro I do not have access to SPARC machine, is there a way I can get a definite testing result on it, instead of adding more alignment checks and testing it on X86?

This seems to have broken https://lab.llvm.org/buildbot/#/builders/245/builds/10311

Kindly revert or address the issues.

I tested the latest diff on ARM and could not replicate the errors.

ro added a comment.Jul 12 2023, 5:19 AM

@ro I do not have access to SPARC machine, is there a way I can get a definite testing result on it, instead of adding more alignment checks and testing it on X86?

You actually do have access to SPARC machines, both Solaris and Linux: the GCC farm provides just that.

That said, I've tested the previous version of your patch on Solaris/sparcv9. While 4 of the failures I'd reported are gone, LLVM :: Transforms/SampleProfile/profile-format.ll still FAILs, now with

Assertion failed: Hash == hash_value(Key), file /vol/llvm/src/llvm-project/local/llvm/include/llvm/ProfileData/SampleProf.h, line 1352
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /var/llvm/local-sparcv9-release-stage2-A-flang/tools/clang/stage2-bins/bin/opt -passes=sample-profile -sample-profile-file=/vol/llvm/src/llvm-project/local/llvm/test/Transforms/SampleProfile/Inputs/inline.extbinary.afdo -S
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  opt       0x00000001063a8648 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 36
1  opt       0x00000001063a8f34 SignalHandler(int) + 896
2  libc.so.1 0xffffffff7eec4f00 __sighndlr + 12
3  libc.so.1 0xffffffff7eeb77a8 call_user_handler + 1024
4  libc.so.1 0xffffffff7eeb7b98 sigacthandler + 208
5  libc.so.1 0xffffffff7eec9fc0 __lwp_sigqueue + 8
6  libc.so.1 0xffffffff7ede484c abort + 180
7  libc.so.1 0xffffffff7ede5680 _assert + 96
8  opt       0x0000000106dac008 std::pair<llvm::DenseMapIterator<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>, false>, bool> llvm::sampleprof::HashKeyMap<llvm::DenseMap, llvm::sampleprof::SampleContext, llvm::sampleprof::FunctionSamples>::try_emplace<llvm::sampleprof::FunctionSamples>(llvm::hash_code const&, llvm::sampleprof::SampleContext const&, llvm::sampleprof::FunctionSamples&&) + 396
9  opt       0x00000001072dfe40 llvm::sampleprof::SampleProfileReaderBinary::readFuncProfile(unsigned char const*) + 284
10 opt       0x00000001072e25d0 llvm::sampleprof::SampleProfileReaderExtBinaryBase::readFuncProfiles() + 3476
11 opt       0x00000001072e02f4 llvm::sampleprof::SampleProfileReaderExtBinaryBase::readOneSection(unsigned char const*, unsigned long, llvm::sampleprof::SecHdrTableEntry const&) + 384
12 opt       0x00000001072e3994 llvm::sampleprof::SampleProfileReaderExtBinaryBase::readImpl() + 212
13 opt       0x0000000106da3210 llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) + 2172
4 opt       0x0000000105cf7ca8 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) + 480
15 opt       0x0000000103769858 llvm::runPassPipeline(llvm::StringRef, llvm::Module&, llvm::TargetMachine*, llvm::TargetLibraryInfoImpl*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::StringRef, llvm::ArrayRef<llvm::PassPlugin>, llvm::opt_tool::OutputKind, llvm::opt_tool::VerifierKind, bool, bool, bool, bool, bool, bool, bool) + 11060
16 opt       0x0000000103777f5c main + 8692
17 opt       0x0000000103766744 _start + 100

In addition, quite a number of other tests now FAIL with the same assertiion failure:

LLVM :: Transforms/SampleProfile/compressed-profile-symbol-list.ll
LLVM :: Transforms/SampleProfile/csspgo-import-list.ll
LLVM :: Transforms/SampleProfile/csspgo-inline-icall.ll
LLVM :: Transforms/SampleProfile/csspgo-inline.ll
LLVM :: Transforms/SampleProfile/fsafdo_test.ll
LLVM :: Transforms/SampleProfile/inline-mergeprof.ll
LLVM :: Transforms/SampleProfile/profile-context-tracker.ll
LLVM :: Transforms/SampleProfile/profile-format-compress.ll
LLVM :: Transforms/SampleProfile/profile-format.ll
LLVM :: Transforms/SampleProfile/profile-sample-accurate.ll
LLVM :: Transforms/SampleProfile/pseudo-probe-inline.ll
LLVM :: Transforms/SampleProfile/remap.ll
LLVM :: Transforms/SampleProfile/uncompressed-profile-symbol-list.ll
LLVM :: Transforms/SampleProfile/uniqname.ll

I've since retried with the latest version of the patch, but the failures remain.

Another question regarding ARM and SPARC, what is size_t on these machines? I suspect the error has something to do with that, since I couldn't replicate them on X64.

ro added a comment.Jul 12 2023, 3:01 PM

Another question regarding ARM and SPARC, what is size_t on these machines? I suspect the error has something to do with that, since I couldn't replicate them on X64.

Solaris <iso/stddef_iso.h> has

#if defined(_LP64) || defined(_I32LPx) 
typedef unsigned long   size_t;         /* size of something in bytes */
#else
typedef unsigned int    size_t;         /* (historical version) */ 
#endif

Nothing unusual here.

I still suspect that this is rather an endianess issue

huangjd updated this revision to Diff 539808.Jul 12 2023, 5:59 PM

In SampleProfileWriter, prevent already hashed function being hashed again when writing the profile

ro added a comment.Jul 13 2023, 3:36 AM

In SampleProfileWriter, prevent already hashed function being hashed again when writing the profile

This didn't make difference, unfortunately.

In D147740#4495250, @ro wrote:

Another question regarding ARM and SPARC, what is size_t on these machines? I suspect the error has something to do with that, since I couldn't replicate them on X64.

Solaris <iso/stddef_iso.h> has

#if defined(_LP64) || defined(_I32LPx) 
typedef unsigned long   size_t;         /* size of something in bytes */
#else
typedef unsigned int    size_t;         /* (historical version) */ 
#endif

Nothing unusual here.

I still suspect that this is rather an endianess issue

I am waiting for the approval to use GCC farm machine so that I can debug it. Do you know any person I can ping to expedite the process?

ro added a comment.Jul 13 2023, 12:20 PM

I am waiting for the approval to use GCC farm machine so that I can debug it. Do you know any person I can ping to expedite the process?

Unfortunately not: sometimes they're very quick, at others a response takes days (or even weeks).

huangjd updated this revision to Diff 540214.Jul 13 2023, 4:14 PM

Fix MD5 hash table write on big endian

Always use little endian write on Line 586 in SampleProfileReader.cpp, this should make it consistent on different systems

@huangjd any follow up on simplifying the implantation based on the assumption that collision is non-issue? Just want to make sure the comments on D153692 don't fall through the cracks.

ro added a comment.Jul 14 2023, 12:58 AM

Always use little endian write on Line 586 in SampleProfileReader.cpp, this should make it consistent on different systems

With the last revision, the assertion failures are gone. However, two tailures do remain:

+  LLVM :: Transforms/SampleProfile/inline-mergeprof.ll

Command Output (stderr):
--
/vol/llvm/src/llvm-project/local/llvm/test/Transforms/SampleProfile/profile-format.ll:27:10: error: CHECK: expected string not found in input
; CHECK: br i1 %cmp, label %while.body, label %while.end{{.*}} !prof ![[IDX1:[0-9]*]]
         ^
<stdin>:1:1: note: scanning from here
; ModuleID = '<stdin>'
^
<stdin>:32:2: note: possible intended match here
 br i1 %cmp, label %while.body, label %while.end, !dbg !42
 ^

+  LLVM :: Transforms/SampleProfile/profile-format.ll

Command Output (stderr):
--
/vol/llvm/src/llvm-project/local/llvm/test/Transforms/SampleProfile/profile-format.ll:27:10: error: CHECK: expected string not found in input
; CHECK: br i1 %cmp, label %while.body, label %while.end{{.*}} !prof ![[IDX1:[0-9]*]]
         ^
<stdin>:1:1: note: scanning from here
; ModuleID = '<stdin>'
^
<stdin>:32:2: note: possible intended match here
 br i1 %cmp, label %while.body, label %while.end, !dbg !42
 ^

@huangjd any follow up on simplifying the implantation based on the assumption that collision is non-issue? Just want to make sure the comments on D153692 don't fall through the cracks.

I am actually going to change that patch to removing the collision check

huangjd added a comment.EditedJul 24 2023, 11:59 AM

@ro
I am using the SPARC machine on GCC farm but it seems to hang on linking the unit tests. Is this normal? Is there an easy option to cross build all the binaries from a X86 machine targeting SPARC and send the file over?

What command did you use to build and test on SPARC?

ro added a comment.Jul 25 2023, 1:40 AM

@ro
I am using the SPARC machine on GCC farm but it seems to hang on linking the unit tests. Is this normal? Is there an easy option to cross build all the binaries from a X86 machine targeting SPARC and send the file over?

Which cfarm system did you use for the builds? There are several, both Solaris and Linux. I've avoided the Linux ones since they could be unreliable at times. I'd suggest going for gcc106, the only Solaris 11.4 box which is required for LLVM. There are two issues to be wary about:

  • Be careful to control the parallellism: the link steps can be quite memory intensive (for comparison's sake, my Solaris 11.4/SPARC box used to have 256 GB RAM and is now at 512 GB which helped tremendously).
  • If you are doing a debug build, the linking is painfully slow indeed (on the order of hours in some cases). Nothing to do about that but being patient.

I've never done cross builds and would expect them to be a real PITA, especially since clang only supports the Solaris linker right now, and GNU ld is not command-line compatible.

What command did you use to build and test on SPARC?

I'm using a script of my own to do the cmake invocation, for a considerable part handling the 2-stage build. Nothing really special in there, though.

huangjd updated this revision to Diff 544567.Jul 26 2023, 6:06 PM

Fix endianess issue on SPARC

This revision was landed with ongoing or failed builds.Jul 27 2023, 4:08 PM
This revision was automatically updated to reflect the committed changes.
antmo added a subscriber: antmo.Jul 28 2023, 12:03 AM

Hi, I think this patch broke clang-armv8-quick bot : https://lab.llvm.org/buildbot/#/builders/245/builds/11732
Could you please take a look ?

ro added a comment.Jul 28 2023, 12:53 AM

Fix endianess issue on SPARC

Thanks: Solaris/sparcv9 results with that revision are fine now indeed.

I think this patch broke clang-armv8-quick bot

Some context here, that's 32 bit Armv8. Our Armv7 32 bit bots aren't happy with it either.

Hi, I think this patch broke clang-armv8-quick bot : https://lab.llvm.org/buildbot/#/builders/245/builds/11732
Could you please take a look ?

This broke quite a few of the ARM bots, so I've reverted in https://github.com/llvm/llvm-project/commit/1a53b5c367b5ebf7d7f34afaa653ea337982f1d6 to hopefully get them back to green while you investigate. Sorry for any troubles!

MatzeB added a subscriber: MatzeB.Jul 28 2023, 9:24 AM

FWIW This was also crashing our x86 builds (using -flto=thin and -fprofile-sample-use). Asan reports a use-after-free: https://reviews.llvm.org/F28484911

In case its relevant: Those crashing builds use profile files created with slightly older versions of LLVM. I assume backwards compatibility is provided with this patch or at least some versioning check and proper error message if file format is incompatible?

In case its relevant: Those crashing builds use profile files created with slightly older versions of LLVM. I assume backwards compatibility is provided with this patch or at least some versioning check and proper error message if file format is incompatible?

Do you have the profile available for me to test?

Do you have the profile available for me to test?

The profiles are hundreds of megabytes and I am not sure whether I am even allowed to make them public. Though I can test changes or inspect the profiles if it helps. Otherwise I can try whether I can manage to create a smaller reproducer on monday.

huangjd reopened this revision.Aug 1 2023, 2:38 PM
This revision is now accepted and ready to land.Aug 1 2023, 2:38 PM
huangjd updated this revision to Diff 546232.Aug 1 2023, 2:38 PM
This comment was removed by huangjd.
huangjd updated this revision to Diff 546296.Aug 1 2023, 7:29 PM

Fixed more errors on 32-bit platform, always use uint64_t for function hash value

Hi, I think this patch broke clang-armv8-quick bot : https://lab.llvm.org/buildbot/#/builders/245/builds/11732
Could you please take a look ?

This broke quite a few of the ARM bots, so I've reverted in https://github.com/llvm/llvm-project/commit/1a53b5c367b5ebf7d7f34afaa653ea337982f1d6 to hopefully get them back to green while you investigate. Sorry for any troubles!

Are you able to verify that the current patch has resolved the problem? I do not have a 32-bit ARM machine, and GCC farm does not provide one either. I am not sure how to test it on buildbot without actually submitting it

@MatzeB

After some investigation it looks like it's an independent bug in Transforms/IPO/SampleProfile.cpp that has never been discovered.

In SampleProfile.cpp, non-inlined callees are added to the Profiles as new functions, which can trigger a rehashing and invalidates all iterator, including SampleProfileLoader::SampleProfileLoaderBaseImpl::Samples which is a pointer to the current function's FunctionSamples inside the profile. This pointer is later used so it is already undefined behavior.

LLVM's standard library implementation was able to expand the profile (originaly unordered_map) in place without relocating objects, so the code would work. In my patch I changed the container to llvm::DenseMap, which will always relocate objects on rehashing, causing the pointer to be invalid and crashing the program. Since std::unordered_map does not guarantee that either, this function is UB and needs to be rewritten

I am going to submit a patch to that first, although I am not able to create a small test case for that bug since std::unordered_map is doing well to avoid relocating objects as long as there is enough memory, so it would require more careful code review.

Thanks for looking into this! Sounds like we may need unreasonably large inputs so going without a test should be okay. Either as a separate patch or merged with this one, whatever works best for you.

Thanks for looking into this! Sounds like we may need unreasonably large inputs so going without a test should be okay. Either as a separate patch or merged with this one, whatever works best for you.

The fix is in D157061

@antmo @DavidSpickett @aaron.ballman
Are you able to check if the latest version fixed the issues on ARM-32? I do not have access to ARM-32 machine but was able to reproduce the same failing test cases in Win32 and fixed it. I would like to confirm it's working before landing the patch again

@antmo @DavidSpickett @aaron.ballman
Are you able to check if the latest version fixed the issues on ARM-32? I do not have access to ARM-32 machine but was able to reproduce the same failing test cases in Win32 and fixed it. I would like to confirm it's working before landing the patch again

I don't have access to an ARM machine myself; I only noticed the issue through the build farm. If you think the code is working, I think it's fine to land it again to see if the bots agree -- it's pretty normal to not have access to all the same hardware as the bots, so speculative commits are somewhat routine.

antmo added a comment.Aug 8 2023, 3:01 PM

Hi @huangjd, the 8 failures shown by the bot clang-armv8-quick look fixed in the latest version.

I think it's fine to land it again to see if the bots agree

Yes, totally fine to do this if you're around to revert promptly.

Thanks Antoine for testing the patch!

This revision was landed with ongoing or failed builds.Aug 17 2023, 1:11 PM
This revision was automatically updated to reflect the committed changes.

This diff is causing clang to crash on one of our builds, when clang is linked with jemalloc.
-DCMAKE_EXE_LINKER_FLAGS="-L /usr/lib64 -Wl,-rpath,/usr/lib64 -ljemalloc" \

I am trying to come up with small repro, but in meantime can this diff be reverted?

Full stack trace:

 #0 0x000055b18add6a58 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:723:13
 #1 0x000055b18add4950 llvm::sys::RunSignalHandlers() /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Support/Signals.cpp:106:18
 #2 0x000055b18add722d SignalHandler(int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:413:1
 #3 0x00007ffaa7812cf0 __restore_rt (/usr/lib64/libpthread.so.0+0x12cf0)
 #4 0x000055b18c0c9033 llvm::sampleprof::LineLocation::operator<(llvm::sampleprof::LineLocation const&) const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:296:23
 #5 0x000055b18c0c9033 std::less<llvm::sampleprof::LineLocation>::operator()(llvm::sampleprof::LineLocation const&, llvm::sampleprof::LineLocation const&) const /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_function.h:408:20
 #6 0x000055b18c0c9033 std::_Rb_tree<llvm::sampleprof::LineLocation, std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>, std::_Select1st<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>, std::less<llvm::sampleprof::LineLocation>, std::allocator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_iterator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>, llvm::sampleprof::LineLocation const&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_tree.h:2220:11
 #7 0x000055b18c0caf2d std::_Rb_tree_iterator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>> std::_Rb_tree<llvm::sampleprof::LineLocation, std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>, std::_Select1st<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>, std::less<llvm::sampleprof::LineLocation>, std::allocator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>>::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<llvm::sampleprof::LineLocation&&>, std::tuple<>>(std::_Rb_tree_const_iterator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>, std::piecewise_construct_t const&, std::tuple<llvm::sampleprof::LineLocation&&>&&, std::tuple<>&&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_tree.h:2462:15
 #8 0x000055b18c0c9bfb llvm::sampleprof::FunctionSamples::addBodySamples(unsigned int, unsigned int, unsigned long, unsigned long) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_map.h:0:15
 #9 0x000055b18c0c6d27 llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1599:21
#10 0x000055b18c0c659f llvm::sampleprof::ProfileConverter::flattenProfile(llvm::sampleprof::SampleProfileMap const&, llvm::sampleprof::SampleProfileMap&, bool) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:0:9
#11 0x000055b18c0c0e28 std::__uniq_ptr_impl<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>::reset((anonymous namespace)::SampleProfileMatcher*) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:200:26
#12 0x000055b18c0c0e28 std::__uniq_ptr_impl<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>::operator=(std::__uniq_ptr_impl<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>&&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:183:2
#13 0x000055b18c0c0e28 std::__uniq_ptr_data<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>, true, true>::operator=(std::__uniq_ptr_data<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>, true, true>&&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:235:61
#14 0x000055b18c0c0e28 std::unique_ptr<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>::operator=(std::unique_ptr<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>&&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:406:51
#15 0x000055b18c0c0e28 (anonymous namespace)::SampleProfileLoader::doInitialization(llvm::Module&, llvm::AnalysisManager<llvm::Function>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2104:21
#16 0x000055b18c0c0e28 llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2632:21
#17 0x000055b18bfcfb0d llvm::detail::PassModel<llvm::Module, llvm::SampleProfileLoaderPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:89:5
#18 0x000055b18a95e3b9 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:521:10
#19 0x000055b18b4f33ff llvm::SmallPtrSetImplBase::isSmall() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/SmallPtrSet.h:195:33
#20 0x000055b18b4f33ff llvm::SmallPtrSetImplBase::~SmallPtrSetImplBase() /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/SmallPtrSet.h:83:10
#21 0x000055b18b4f33ff llvm::PreservedAnalyses::~PreservedAnalyses() /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:152:7
#22 0x000055b18b4f33ff (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>&, std::unique_ptr<llvm::ToolOutputFile, std::default_delete<llvm::ToolOutputFile>>&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1101:5
#23 0x000055b18b4eaa22 (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:0:3
#24 0x000055b18b4eaa22 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1321:13
#25 0x000055b18b95a8f5 std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>::~unique_ptr() /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:395:6
#26 0x000055b18b95a8f5 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:386:7
#27 0x000055b18cf852c6 __gnu_cxx::__normal_iterator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>*, std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>>>>::__normal_iterator(std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>* const&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_iterator.h:1073:20
#28 0x000055b18cf852c6 std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>>>::begin() /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:869:16
#29 0x000055b18cf852c6 void clang::finalize<std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>>>>(std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>>>&, clang::Sema const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/include/clang/Sema/TemplateInstCallback.h:54:16
#30 0x000055b18cf852c6 clang::ParseAST(clang::Sema&, bool, bool) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:183:3
#31 0x000055b18b881b10 clang::FrontendAction::Execute() /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1067:10
#32 0x000055b18b7fdc3d llvm::Error::getPtr() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/Support/Error.h:270:42
#33 0x000055b18b7fdc3d llvm::Error::operator bool() /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/Support/Error.h:233:16
#34 0x000055b18b7fdc3d clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1054:23
#35 0x000055b18b953335 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:272:25
#36 0x000055b189dc769c cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/cc1_main.cpp:249:15
#37 0x000055b189dc479a ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:366:12
#38 0x000055b189dc3732 clang_main(int, char**, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:407:12
#39 0x000055b189dd2d71 main /home/ayermolo/local/llvm-build-upstream-release/tools/clang/tools/driver/clang-driver.cpp:15:3
#40 0x00007ffaa583ad85 __libc_start_main (/usr/lib64/libc.so.6+0x3ad85)
#41 0x000055b189dc102e _start (/home/ayermolo/local/llvm-build-upstream-release/bin/clang+++0x2a6a02e)

@huangjd

MSAN output. Does this ring any bells?

==3359553==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564b193d7f4e in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30
    #1 0x564b193ce93a in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1612:36
    #2 0x564b193ce582 in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1607:9
    #3 0x564b193ca9cc in llvm::sampleprof::ProfileConverter::flattenProfile(llvm::sampleprof::SampleProfileMap const&, llvm::sampleprof::SampleProfileMap&, bool) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1558:9
    #4 0x564b1936a0f6 in (anonymous namespace)::SampleProfileMatcher::SampleProfileMatcher(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager const*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:466:7
    #5 0x564b1936a0f6 in std::__1::__unique_if<(anonymous namespace)::SampleProfileMatcher>::__unique_single std::__1::make_unique[abi:v180000]<(anonymous namespace)::SampleProfileMatcher, llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*>(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*&&) /home/ayermolo/local/upstream-llvm/llvm-project/build_libcxx/include/c++/v1/__memory/unique_ptr.h:685:30
    #6 0x564b1936a0f6 in (anonymous namespace)::SampleProfileLoader::doInitialization(llvm::Module&, llvm::AnalysisManager<llvm::Function>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2105:9
    #7 0x564b1936a0f6 in llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2632:21
    #8 0x564b18f338ac in llvm::detail::PassModel<llvm::Module, llvm::SampleProfileLoaderPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:89:17
    #9 0x564b12a1306b in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #10 0x564b15b5e1a7 in (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>&, std::__1::unique_ptr<llvm::ToolOutputFile, std::__1::default_delete<llvm::ToolOutputFile>>&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1101:9
    #11 0x564b15b447fd in (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1158:3
    #12 0x564b15b447fd in clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1321:13
    #13 0x564b16f72ca1 in clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:386:7
    #14 0x564b1d01a29b in clang::ParseAST(clang::Sema&, bool, bool) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:176:13
    #15 0x564b16be4f7e in clang::FrontendAction::Execute() /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1063:8
    #16 0x564b1695cd8d in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1054:33
    #17 0x564b16f50383 in clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:272:25
    #18 0x564b0f3bb128 in cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/cc1_main.cpp:249:15
    #19 0x564b0f3b0d6a in ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:366:12
    #20 0x564b0f3abb78 in clang_main(int, char**, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:407:12
    #21 0x564b0f3ec39c in main /home/ayermolo/local/llvm-build-upstream-msan-release/tools/clang/tools/driver/clang-driver.cpp:15:10
    #22 0x7f73e6a3ad84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: 1356e140fb964a20b0d2838960ee69ca6faeb034)
    #23 0x564b0f31642d in _start (/data/users/ayermolo/llvm-build-upstream-msan-release/bin/clang-18+0x315042d)

  Uninitialized value was created by a heap deallocation
    #0 0x564b0f3a2269 in operator delete(void*, std::align_val_t) /home/ayermolo/local/upstream-llvm/llvm-project/compiler-rt/lib/msan/msan_new_delete.cpp:90:3
    #1 0x564b193d94e5 in llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::grow(unsigned int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:564:36
    #2 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucketImpl<llvm::hash_code>(llvm::hash_code const&, llvm::hash_code const&, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h
    #3 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucket<llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&>(llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*, llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:574:17

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30 in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const
Exiting

What were you building when this error happened?

@huangjd

MSAN output. Does this ring any bells?

==3359553==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564b193d7f4e in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30
    #1 0x564b193ce93a in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1612:36
    #2 0x564b193ce582 in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1607:9
    #3 0x564b193ca9cc in llvm::sampleprof::ProfileConverter::flattenProfile(llvm::sampleprof::SampleProfileMap const&, llvm::sampleprof::SampleProfileMap&, bool) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1558:9
    #4 0x564b1936a0f6 in (anonymous namespace)::SampleProfileMatcher::SampleProfileMatcher(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager const*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:466:7
    #5 0x564b1936a0f6 in std::__1::__unique_if<(anonymous namespace)::SampleProfileMatcher>::__unique_single std::__1::make_unique[abi:v180000]<(anonymous namespace)::SampleProfileMatcher, llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*>(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*&&) /home/ayermolo/local/upstream-llvm/llvm-project/build_libcxx/include/c++/v1/__memory/unique_ptr.h:685:30
    #6 0x564b1936a0f6 in (anonymous namespace)::SampleProfileLoader::doInitialization(llvm::Module&, llvm::AnalysisManager<llvm::Function>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2105:9
    #7 0x564b1936a0f6 in llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2632:21
    #8 0x564b18f338ac in llvm::detail::PassModel<llvm::Module, llvm::SampleProfileLoaderPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:89:17
    #9 0x564b12a1306b in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #10 0x564b15b5e1a7 in (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>&, std::__1::unique_ptr<llvm::ToolOutputFile, std::__1::default_delete<llvm::ToolOutputFile>>&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1101:9
    #11 0x564b15b447fd in (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1158:3
    #12 0x564b15b447fd in clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1321:13
    #13 0x564b16f72ca1 in clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:386:7
    #14 0x564b1d01a29b in clang::ParseAST(clang::Sema&, bool, bool) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:176:13
    #15 0x564b16be4f7e in clang::FrontendAction::Execute() /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1063:8
    #16 0x564b1695cd8d in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1054:33
    #17 0x564b16f50383 in clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:272:25
    #18 0x564b0f3bb128 in cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/cc1_main.cpp:249:15
    #19 0x564b0f3b0d6a in ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:366:12
    #20 0x564b0f3abb78 in clang_main(int, char**, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:407:12
    #21 0x564b0f3ec39c in main /home/ayermolo/local/llvm-build-upstream-msan-release/tools/clang/tools/driver/clang-driver.cpp:15:10
    #22 0x7f73e6a3ad84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: 1356e140fb964a20b0d2838960ee69ca6faeb034)
    #23 0x564b0f31642d in _start (/data/users/ayermolo/llvm-build-upstream-msan-release/bin/clang-18+0x315042d)

  Uninitialized value was created by a heap deallocation
    #0 0x564b0f3a2269 in operator delete(void*, std::align_val_t) /home/ayermolo/local/upstream-llvm/llvm-project/compiler-rt/lib/msan/msan_new_delete.cpp:90:3
    #1 0x564b193d94e5 in llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::grow(unsigned int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:564:36
    #2 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucketImpl<llvm::hash_code>(llvm::hash_code const&, llvm::hash_code const&, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h
    #3 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucket<llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&>(llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*, llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:574:17

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30 in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const
Exiting
ayermolo added a comment.EditedAug 23 2023, 2:28 PM

What were you building when this error happened?

@huangjd

MSAN output. Does this ring any bells?

==3359553==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564b193d7f4e in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30
    #1 0x564b193ce93a in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1612:36
    #2 0x564b193ce582 in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1607:9
    #3 0x564b193ca9cc in llvm::sampleprof::ProfileConverter::flattenProfile(llvm::sampleprof::SampleProfileMap const&, llvm::sampleprof::SampleProfileMap&, bool) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1558:9
    #4 0x564b1936a0f6 in (anonymous namespace)::SampleProfileMatcher::SampleProfileMatcher(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager const*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:466:7
    #5 0x564b1936a0f6 in std::__1::__unique_if<(anonymous namespace)::SampleProfileMatcher>::__unique_single std::__1::make_unique[abi:v180000]<(anonymous namespace)::SampleProfileMatcher, llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*>(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*&&) /home/ayermolo/local/upstream-llvm/llvm-project/build_libcxx/include/c++/v1/__memory/unique_ptr.h:685:30
    #6 0x564b1936a0f6 in (anonymous namespace)::SampleProfileLoader::doInitialization(llvm::Module&, llvm::AnalysisManager<llvm::Function>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2105:9
    #7 0x564b1936a0f6 in llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2632:21
    #8 0x564b18f338ac in llvm::detail::PassModel<llvm::Module, llvm::SampleProfileLoaderPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:89:17
    #9 0x564b12a1306b in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #10 0x564b15b5e1a7 in (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>&, std::__1::unique_ptr<llvm::ToolOutputFile, std::__1::default_delete<llvm::ToolOutputFile>>&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1101:9
    #11 0x564b15b447fd in (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1158:3
    #12 0x564b15b447fd in clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1321:13
    #13 0x564b16f72ca1 in clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:386:7
    #14 0x564b1d01a29b in clang::ParseAST(clang::Sema&, bool, bool) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:176:13
    #15 0x564b16be4f7e in clang::FrontendAction::Execute() /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1063:8
    #16 0x564b1695cd8d in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1054:33
    #17 0x564b16f50383 in clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:272:25
    #18 0x564b0f3bb128 in cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/cc1_main.cpp:249:15
    #19 0x564b0f3b0d6a in ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:366:12
    #20 0x564b0f3abb78 in clang_main(int, char**, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:407:12
    #21 0x564b0f3ec39c in main /home/ayermolo/local/llvm-build-upstream-msan-release/tools/clang/tools/driver/clang-driver.cpp:15:10
    #22 0x7f73e6a3ad84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: 1356e140fb964a20b0d2838960ee69ca6faeb034)
    #23 0x564b0f31642d in _start (/data/users/ayermolo/llvm-build-upstream-msan-release/bin/clang-18+0x315042d)

  Uninitialized value was created by a heap deallocation
    #0 0x564b0f3a2269 in operator delete(void*, std::align_val_t) /home/ayermolo/local/upstream-llvm/llvm-project/compiler-rt/lib/msan/msan_new_delete.cpp:90:3
    #1 0x564b193d94e5 in llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::grow(unsigned int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:564:36
    #2 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucketImpl<llvm::hash_code>(llvm::hash_code const&, llvm::hash_code const&, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h
    #3 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucket<llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&>(llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*, llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:574:17

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30 in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const
Exiting

It's c++ code
"-triple" "x86_64-redhat-linux-gnu" "-emit-llvm-bc" "-flto=thin" "-flto-unit" ... "-fprofile-sample-use=<path>/__autofdo-bolt-compatible__/out/profile" "-fpseudo-probe-for-profiling"
Some of the more relevant options I think.
it's a csspgo profile.

I think I figured out what's the problem, writing a patch now

huangjd reopened this revision.Aug 28 2023, 12:18 PM
This revision is now accepted and ready to land.Aug 28 2023, 12:18 PM
huangjd closed this revision.Sep 14 2023, 2:22 PM