This is an archive of the discontinued LLVM Phabricator instance.

[llvm-profdata] Refactoring Sample Profile Reader to increase FDO build speed using MD5 as key to Sample Profile map
ClosedPublic

Authored by huangjd on Apr 6 2023, 1:53 PM.

Details

Summary

This is phase 1 of multiple planned improvements on the sample profile loader. The major change is to use MD5 hash code ((instead of the function itself) as the key to look up the function offset table and the profiles, which significantly reduce the time it takes to construct the map.

The optimization is based on the fact that many practical sample profiles are using MD5 values for function names to reduce profile size, so we shouldn't need to convert the MD5 to a string and then to a SampleContext and use it as the map's key, because it's extremely slow.

Several changes to note:

(1) For non-CS SampleContext, if it is already MD5 string, the hash value will be its integral value, instead of hashing the MD5 again. In phase 2 this is going to be optimized further using a union to represent MD5 function (without converting it to string) and regular function names.

(2) The SampleProfileMap is a wrapper to *map<uint64_t, FunctionSamples>, while providing interface allowing using SampleContext as key, so that existing code still work. It will check for MD5 collision (unlikely but not too unlikely, since we only takes the lower 64 bits) and handle it to at least guarantee compilation correctness (conflicting old profile is dropped, instead of returning an old profile with inconsistent context). Other code should not try to use MD5 as key to access the map directly, because it will not be able to handle MD5 collision at all. (see exception at (5) )

(3) Any SampleProfileMap::emplace() followed by SampleContext assignment if newly inserted, should be replaced with SampleProfileMap::Create(), which does the same thing.

(4) Previously we ensure an invariant that in SampleProfileMap, the key is equal to the Context of the value, for profile map that is eventually being used for output (as in llvm-profdata/llvm-profgen). Since the key became MD5 hash, only the value keeps the context now, in several places where an intermediate SampleProfileMap is created, each new FunctionSample's context is set immediately after insertion, which is necessary to "remember" the context otherwise irretrievable.

(5) When reading a profile, we cache the MD5 values of all functions, because they are used at least twice (one to index into FuncOffsetTable, the other into SampleProfileMap, more if there are additional sections), in this case the SampleProfileMap is directly accessed with MD5 value so that we don't recalculate it each time (expensive)

Performance impact:
When reading a ~1GB extbinary profile (fixed length MD5, not compressed) with 10 million function names and 2.5 million top level functions (non CS functions, each function has varying nesting level from 0 to 20), this patch improves the function offset table loading time by 20%, and improves full profile read by 5%.

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
davidxl added inline comments.May 31 2023, 9:06 AM
llvm/include/llvm/ProfileData/SampleProf.h
1327

Since this forces insert in case of conflict, should it be called just 'emplace'?

1336

potential memory leak? Also why inserting an empty FunctionSamples instead of the one passed in?

1350

what is the purpose of this wrapper?

1354

This interface is confusing. User expect it to return existing entry, but here it erases it. Should this interface be hidden (and not allowed with assert)?

1362

is this always true as currently implemented?

huangjd added inline comments.Jun 1 2023, 1:56 PM
llvm/include/llvm/ProfileData/SampleProf.h
1327

try_emplace functionally is same as emplace (https://en.cppreference.com/w/cpp/container/unordered_map/try_emplace), only difference is that try_emplace does not move the arguments to construct a mapped_type if key exists.

1350

existing code compatibility

huangjd added a comment.EditedJun 1 2023, 2:25 PM

Explaining the behavior of MD5-key FunctionSampleMap

Previously the map is {SampleContext : FunctionSamples}, where FunctionSamples holds a copy of the SampleContext (but not always enforced, for example when the map is an intermediate product of a merge, or during IPO passes). Currently the map is {Hash(SampleContext) : FunctionSamples}, where FunctionSamples holds the SampleContext immediately after insertion (otherwise the context is lost forever)

When inserting to the map, the caller must provide a SampleContext, so that when an existing entry with the same hash value is found, its SampleContext is compared with the caller's new SampleContext.

If they are the same, then it's actually a match, so the existing FunctionSamples is returned and no insertion happens.

If they are different, then there's a MD5 collision, and in this case, we have to decide which FunctionSamples to keep. The decision is to use the new FunctionSamples and erase the old one, because IPO calls SampleProfileReader:;getOrCreateSamplesFor(), and we want to return a sample with the requested function name, rather than a sample with a *different* function name/SampleContext. In this case since a new entry is inserted with a new FunctionSamples provided by the caller, return.second should be true to indicate a new value is inserted, so that the caller can perform other necessary logic as if a new entry is inserted.

There's not much we can do to keep both entries in case of a collision, as it is very rare (and probably only happens theoretically), and using a multi-map slows down too much. Note that the profile should not affect compilation correctness, so even when collision happens and profiles are dropped, it should only affect the optimization applied to related functions

huangjd updated this revision to Diff 527624.Jun 1 2023, 2:28 PM

Fix bug in emplace on MD5 collision

Added unit test

huangjd marked 3 inline comments as done.Jun 1 2023, 2:39 PM
huangjd added inline comments.
llvm/include/llvm/ProfileData/SampleProf.h
1354

In C++, [] is same as try_emplace with default constructed mapped_type, so I am keeping the behavior consistent. Keeping this function because many places use it.

1362

This returns false if the existing entry actually has the same context, which indicates a match, rather than a MD5 collision, so no need to set the context again (And setting the context actually erase the flags in the context, which is not used for equality comparison or hasing)

huangjd edited the summary of this revision. (Show Details)Jun 2 2023, 7:35 PM
huangjd marked 2 inline comments as done.Jun 6 2023, 11:42 AM
huangjd added inline comments.
llvm/include/llvm/ProfileData/SampleProf.h
1336

Fixed. FunctionSamples is copy assignable

huangjd updated this revision to Diff 529099.Jun 6 2023, 5:11 PM

Use llvm:DenseMap for profiles since now the key is uint64

@davidxl @wlei @wenlei @aeubanks
Could you please review the revised patch? Note that it is very different from the original one

huangjd updated this revision to Diff 529756.Jun 8 2023, 3:51 PM

Change test case to avoid reserved key value in Dense Map (~0ULL)

huangjd added inline comments.Jun 8 2023, 3:57 PM
llvm/test/tools/llvm-profdata/sample-nametable.test
11

Note: 0xFFFFFFFFFFFFFFFF and 0xFFFFFFFFFFFFFFFE are reserved in llvm::DenseMap and cannot be used as key. Changing it to 0xFFFFFFFFFFFFFFFD. I am not adding a check in SampleProfileMap to check for them because the assumption that a hash value is never equal them is made in so many places throughout LLVM, any check should be done inside DenseMap if actually needed.

davidxl added inline comments.Jun 9 2023, 7:45 AM
llvm/lib/ProfileData/SampleProf.cpp
403

is this a dead function? should it be removed in a separate patch?

llvm/lib/ProfileData/SampleProfReader.cpp
589

add comment explaining the benefit of lazy hash computing.

snehasish added inline comments.Jun 9 2023, 4:06 PM
llvm/include/llvm/ProfileData/SampleProf.h
324

Is there a chance that the base 10 encoding may change? Is this the only place where we generate the hashes?

1323–1324

Can this lead to non-deterministic builds?

1386

typo "function"

llvm/lib/ProfileData/SampleProfReader.cpp
526–527

Can we use a more descriptive name for the output parameters (here and elsewhere)?

llvm/unittests/tools/llvm-profdata/MD5CollisionTest.cpp
85

Can we use the text format (with some additional helper functions) here instead of the binary data? It would be hard to update in case of changes in the future.

117

This should be assert since if it doesn't hold the following lines which deref NameTable[0] and NameTable[1] will segfault.

141

Capture with structured binding here with clearer variable names to make it easier to read?

huangjd marked an inline comment as done.Jun 13 2023, 11:57 AM
huangjd added inline comments.
llvm/include/llvm/ProfileData/SampleProf.h
324

No, and doesn't matter. Base-10 encoding is used only because the existing implementation wanted to make it compatible to represent both strings (function names) and integers (MD5), and the solution was to convert MD5 into a base-10 string. This seems inefficient and I am working on a subsequent patch to deal with it.

1323–1324

Why? There's no non-determinism here, the existing entry always gets erased first. SampleProfWriter does sort the profile before writing so it's ok

llvm/lib/ProfileData/SampleProf.cpp
403

After refactoring, the invariant key == value.getContext() is moot, so this function is dead

llvm/unittests/tools/llvm-profdata/MD5CollisionTest.cpp
85

See comments

...  A unit test is required because the function
/// names are not printable ASCII characters.
85

This test case is very theoretical, as I can't find two printable ASCII strings with colliding MD5 (there exists for sure, but none is known yet) In this case I have to use a unit test because I cannot validate the output using llvm-lit which requires printable characters.

huangjd updated this revision to Diff 531076.Jun 13 2023, 2:49 PM

Clarify some functions & comments
Rewrite unit test

huangjd marked 6 inline comments as done.Jun 13 2023, 4:18 PM
huangjd updated this revision to Diff 531529.Jun 14 2023, 3:08 PM

Use hash_code for SampleContext.getHashValue() so that DenseMap does not hash the MD5 again inside the map (uint64_t get hashed again which is unnecessary here because MD5 is sufficiently sparse)

huangjd updated this revision to Diff 531964.Jun 15 2023, 6:41 PM

Extracted HashKeyMap wrapper class instead of binding it to SampleProfileMap

davidxl accepted this revision.Jun 20 2023, 1:21 PM

lgtm . Please run more extensive testing including testing with sanitizer on. Please also wait a little for further comments from other reviewers.

This revision is now accepted and ready to land.Jun 20 2023, 1:21 PM
snehasish accepted this revision.Jun 21 2023, 12:56 PM

lgtm

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp
207

Document the parameter name /*ProfileIsCS=*/?

llvm/lib/ProfileData/SampleProf.cpp
205

nit: drop the make_pair in favour or an initializer list?

llvm/unittests/tools/llvm-profdata/MD5CollisionTest.cpp
85

test is required because the function
names below (i.e. String1 and String2) are not printable ASCII characters.

I see, I got confused because I thought "String1" is used in a literal sense. Perhaps enhance the comments above to note this?

Actually do we really care about MD5 collision? ExtBinary format already ignored MD5 collision for regular string names (and therefore regular function profiles), as only one of two functions with colliding MD5 get written to the name table (and the other is therefore lost). If we are using CS profiles, since different CS profiles have different serializations, their hashes are distributed as expected. The most important thing is that, even if we detect a hash collision, we can't do anything about it except logging it (using a multi-map makes the reader much slower), so I think the MD5 collision check should be marked as LLVM_DEBUG. This does reduce 0.5 second out of ~30 seconds (1.67%) over the 1 GB profile read .

Actually do we really care about MD5 collision? ExtBinary format already ignored MD5 collision for regular string names (and therefore regular function profiles), as only one of two functions with colliding MD5 get written to the name table (and the other is therefore lost). If we are using CS profiles, since different CS profiles have different serializations, their hashes are distributed as expected. The most important thing is that, even if we detect a hash collision, we can't do anything about it except logging it (using a multi-map makes the reader much slower), so I think the MD5 collision check should be marked as LLVM_DEBUG. This does reduce 0.5 second out of ~30 seconds (1.67%) over the 1 GB profile read .

I don't think we care. Is the new type HashKeyMap and SampleProfileMap all for detecting and reporting collision? I'd avoid all that complexity and prefer a simple DenseMap + a SampleContext->hash_code converter and not even bother with debug prints for collision...

huangjd updated this revision to Diff 534078.Jun 23 2023, 2:48 PM

Fix a few comments

This revision was landed with ongoing or failed builds.Jun 23 2023, 2:50 PM
This revision was automatically updated to reflect the committed changes.

The error messages (VC)
https://lab.llvm.org/buildbot/#/builders/119/builds/13870/steps/7/logs/stdio

63.990 [3705/51/830] Building CXX object lib\CodeGen\CMakeFiles\LLVMCodeGen.dir\MIRSampleProfile.cpp.obj
FAILED: lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/MIRSampleProfile.cpp.obj 
C:\PROGRA~2\MICROS~1\2019\COMMUN~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\cl.exe  /nologo /TP -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_GLIBCXX_ASSERTIONS -D_HAS_EXCEPTIONS=0 -D_LIBCPP_ENABLE_ASSERTIONS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\buildbot\as-builder-2\x-aarch64\build\lib\CodeGen -IC:\buildbot\as-builder-2\x-aarch64\llvm-project\llvm\lib\CodeGen -IC:\buildbot\as-builder-2\x-aarch64\build\include -IC:\buildbot\as-builder-2\x-aarch64\llvm-project\llvm\include -external:IC:\buildbot\.zlib-win32\include -external:W0 -D__OPTIMIZE__ /Zc:inline /Zc:preprocessor /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd5105 -wd4324 -w14062 -we4238 /Gw /MT /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Folib\CodeGen\CMakeFiles\LLVMCodeGen.dir\MIRSampleProfile.cpp.obj /Fdlib\CodeGen\CMakeFiles\LLVMCodeGen.dir\LLVMCodeGen.pdb /FS -c C:\buildbot\as-builder-2\x-aarch64\llvm-project\llvm\lib\CodeGen\MIRSampleProfile.cpp
C:\buildbot\as-builder-2\x-aarch64\llvm-project\llvm\include\llvm/ProfileData/SampleProf.h(1423): error C3200: 'llvm::DenseMap<llvm::hash_code,ValueT,llvm::DenseMapInfo<llvm::hash_code,void>,llvm::detail::DenseMapPair<KeyT,ValueT>>': invalid template argument for template parameter 'MapT', expected a class template
        with
        [
            ValueT=llvm::sampleprof::FunctionSamples,
            KeyT=llvm::hash_code
        ]
C:\buildbot\as-builder-2\x-aarch64\llvm-project\llvm\include\llvm/ProfileData/SampleProf.h(1427): error C3200: 'llvm::DenseMap<llvm::hash_code,ValueT,llvm::DenseMapInfo<llvm::hash_code,void>,llvm::detail::DenseMapPair<KeyT,ValueT>>': invalid template argument for template parameter 'MapT', expected a class template
        with
        [
            ValueT=llvm::sampleprof::FunctionSamples,
            KeyT=llvm::hash_code
        ]
C:\buildbot\as-builder-2\x-aarch64\llvm-project\llvm\include\llvm/ProfileData/SampleProf.h(1446): error C3200: 'llvm::DenseMap<llvm::hash_code,ValueT,llvm::DenseMapInfo<llvm::hash_code,void>,llvm::detail::DenseMapPair<KeyT,ValueT>>': invalid template argument for template parameter 'MapT', expected a class template
        with
        [
            ValueT=llvm::sampleprof::FunctionSamples,
            KeyT=llvm::hash_code
        ]
huangjd reopened this revision.Jun 26 2023, 3:24 PM
This revision is now accepted and ready to land.Jun 26 2023, 3:24 PM
huangjd updated this revision to Diff 534765.Jun 26 2023, 3:24 PM

Fixed build error in MSVC

This revision was landed with ongoing or failed builds.Jun 26 2023, 5:06 PM
This revision was automatically updated to reflect the committed changes.
omjavaid reopened this revision.Jun 27 2023, 1:47 AM
omjavaid added a subscriber: omjavaid.

This seems to have broken https://lab.llvm.org/buildbot/#/builders/245/builds/10311

Kindly revert or address the issues.

This revision is now accepted and ready to land.Jun 27 2023, 1:47 AM
ro added a subscriber: ro.Jun 27 2023, 6:59 AM

This patch causes regressions on Solaris/sparcv9:

+  LLVM :: Transforms/SampleProfile/ctxsplit.ll
+  LLVM :: Transforms/SampleProfile/indirect-call.ll
+  LLVM :: Transforms/SampleProfile/profile-format.ll
+  LLVM :: tools/llvm-profdata/sample-nametable.test
+  LLVM-Unit :: ProfileData/./ProfileDataTests/26/194

All of the fail in the same way (using indirect-call.ll as an example):

Stack dump:
0.      Program arguments: /var/llvm/dist-sparcv9-release-stage2-A-flang/tools/clang/stage2-bins/bin/opt -S /vol/llvm/src/llvm-project/dist/llvm/test/Transforms/SampleProfile/indirect-call.ll -passes=sample-profile -sample-profile-file=/vol/llvm/src/llvm-project/dist/llvm/test/Transforms/SampleProfile/Inputs/indirect-call.extbinary.afdo
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  opt       0x00000001063410ec llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 36
1  opt       0x00000001063419d8 SignalHandler(int) + 896
2  libc.so.1 0xffffffff7eec4f00 __sighndlr + 12
3  libc.so.1 0xffffffff7eeb77a8 call_user_handler + 1024
4  libc.so.1 0xffffffff7eeb7b68 sigacthandler + 160
5  opt       0x000000010726a738 llvm::sampleprof::SampleProfileReaderBinary::readSampleContextFromTable() + 556
6  opt       0x000000010726e370 llvm::sampleprof::SampleProfileReaderExtBinaryBase::readFuncOffsetTable() + 680
7  opt       0x000000010726bb88 llvm::sampleprof::SampleProfileReaderExtBinaryBase::readOneSection(unsigned char const*, unsigned long, llvm::sampleprof::SecHdrTableEntry const&) + 484
8  opt       0x000000010726f1bc llvm::sampleprof::SampleProfileReaderExtBinaryBase::readImpl() + 212
9  opt       0x0000000106d31128 llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) + 2172
10 opt       0x0000000105c94770 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) + 480
11 opt       0x000000010373b650 llvm::runPassPipeline(llvm::StringRef, llvm::Module&, llvm::TargetMachine*, llvm::TargetLibraryInfoImpl*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::StringRef, llvm::ArrayRef<llvm::PassPlugin>, llvm::opt_tool::OutputKind, llvm::opt_tool::VerifierKind, bool, bool, bool, bool, bool, bool) + 11052
12 opt       0x0000000103749cf0 main + 8592
13 opt       0x0000000103738544 _start + 100
FileCheck error: '<stdin>' is empty.

truss (the Solaris syscall tracer) shows

22092:	    Incurred fault #5, FLTACCESS  %pc = 0x10726A738
22092:	      siginfo: SIGBUS BUS_ADRALN addr=0x1082ED3D3
22092:	    Received signal #10, SIGBUS [caught]
22092:	      siginfo: SIGBUS BUS_ADRALN addr=0x1082ED3D3

i.e. the code tries an unaligned access, which is a no-no on strict-alignment targets like SPARC.

@ro It seems the error you described does not match the error reported in https://lab.llvm.org/buildbot/#/builders/245/builds/10311. The build bot showed a few tests triggering assert on ARM, which I am investigating. However you said the error is about unaligned access on readSampleContextFromTable.

huangjd added a comment.EditedJul 6 2023, 5:09 PM
In D147740#4452405, @ro wrote:

...
i.e. the code tries an unaligned access, which is a no-no on strict-alignment targets like SPARC.

It looks like the error happens at line 577 of SampleProfReader.cpp. I changed `hash_code Hash = MD5SampleContextStart[Idx]`
into

hash_code Hash =
      support::endian::read<hash_code, support::little, support::unaligned>(
          MD5SampleContextStart + Idx);

Since MD5SampleContextStart can point to an unaligned position in the actual profile, if I understand it correctly, although I am not sure if this is a bug on SPARC code gen by itself since MD5SampleContextStart[Idx] is a valid C++ expression even if MD5SampleContextStart is not aligned, so the compiler should generate the correct code.

I don't have a sparc machine, so I would like to find someone to test it.

huangjd updated this revision to Diff 537945.Jul 6 2023, 6:04 PM

Update SampleProfReader for unaligned access on SPARC

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

I built it with -DLLVM_USE_SANITIZER=Memory and didn't see any (new) issue on X86, but I don't have a SPARC machine available to test on it. Is there another flag I need to use?

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

I built it with -DLLVM_USE_SANITIZER=Memory and didn't see any (new) issue on X86, but I don't have a SPARC machine available to test on it. Is there another flag I need to use?

alignment checking is a ubsan thing, not a msan thing, so it should be -DLLVM_USE_SANITIZER=Undefined. I believe that should enable alignment checking: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#available-checks

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

I built it with -DLLVM_USE_SANITIZER=Memory and didn't see any (new) issue on X86, but I don't have a SPARC machine available to test on it. Is there another flag I need to use?

alignment checking is a ubsan thing, not a msan thing, so it should be -DLLVM_USE_SANITIZER=Undefined. I believe that should enable alignment checking: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#available-checks

That option does not work for me. Note that -DLLVM_USE_SANITIZER is an option in CMake configuration, but the list you gave is compiler options. I got the error cc: error: unrecognized argument to ‘-fno-sanitize=’ option: ‘function’ when DLLVM_USE_SANITIZER is set to anything other than Memory. Could you show the exact command to build and run all tests with said sanitizer enabled?

can you repro the previous failure with the alignment sanitizer? then if that goes away with your fix it should be good

I built it with -DLLVM_USE_SANITIZER=Memory and didn't see any (new) issue on X86, but I don't have a SPARC machine available to test on it. Is there another flag I need to use?

alignment checking is a ubsan thing, not a msan thing, so it should be -DLLVM_USE_SANITIZER=Undefined. I believe that should enable alignment checking: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#available-checks

That option does not work for me. Note that -DLLVM_USE_SANITIZER is an option in CMake configuration, but the list you gave is compiler options. I got the error cc: error: unrecognized argument to ‘-fno-sanitize=’ option: ‘function’ when DLLVM_USE_SANITIZER is set to anything other than Memory. Could you show the exact command to build and run all tests with said sanitizer enabled?

cmake -S llvm -B build/cmake -GNinja -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Undefined -DLLVM_ENABLE_LLD=ON -DCMAKE_C_COMPILER=$HOME/repos/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang -DCMAKE_CXX_COMPILER=$HOME/repos/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang++ works for me. I'm using Chrome's toolchain which is a very close to ToT clang package, that error message seems to imply that the host compiler you're using isn't a recent clang.

huangjd updated this revision to Diff 539339.Jul 11 2023, 4:44 PM

updated test to fix sign comparison warning

@ro I do not have access to SPARC machine, is there a way I can get a definite testing result on it, instead of adding more alignment checks and testing it on X86?

This seems to have broken https://lab.llvm.org/buildbot/#/builders/245/builds/10311

Kindly revert or address the issues.

I tested the latest diff on ARM and could not replicate the errors.

ro added a comment.Jul 12 2023, 5:19 AM

@ro I do not have access to SPARC machine, is there a way I can get a definite testing result on it, instead of adding more alignment checks and testing it on X86?

You actually do have access to SPARC machines, both Solaris and Linux: the GCC farm provides just that.

That said, I've tested the previous version of your patch on Solaris/sparcv9. While 4 of the failures I'd reported are gone, LLVM :: Transforms/SampleProfile/profile-format.ll still FAILs, now with

Assertion failed: Hash == hash_value(Key), file /vol/llvm/src/llvm-project/local/llvm/include/llvm/ProfileData/SampleProf.h, line 1352
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /var/llvm/local-sparcv9-release-stage2-A-flang/tools/clang/stage2-bins/bin/opt -passes=sample-profile -sample-profile-file=/vol/llvm/src/llvm-project/local/llvm/test/Transforms/SampleProfile/Inputs/inline.extbinary.afdo -S
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  opt       0x00000001063a8648 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 36
1  opt       0x00000001063a8f34 SignalHandler(int) + 896
2  libc.so.1 0xffffffff7eec4f00 __sighndlr + 12
3  libc.so.1 0xffffffff7eeb77a8 call_user_handler + 1024
4  libc.so.1 0xffffffff7eeb7b98 sigacthandler + 208
5  libc.so.1 0xffffffff7eec9fc0 __lwp_sigqueue + 8
6  libc.so.1 0xffffffff7ede484c abort + 180
7  libc.so.1 0xffffffff7ede5680 _assert + 96
8  opt       0x0000000106dac008 std::pair<llvm::DenseMapIterator<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>, false>, bool> llvm::sampleprof::HashKeyMap<llvm::DenseMap, llvm::sampleprof::SampleContext, llvm::sampleprof::FunctionSamples>::try_emplace<llvm::sampleprof::FunctionSamples>(llvm::hash_code const&, llvm::sampleprof::SampleContext const&, llvm::sampleprof::FunctionSamples&&) + 396
9  opt       0x00000001072dfe40 llvm::sampleprof::SampleProfileReaderBinary::readFuncProfile(unsigned char const*) + 284
10 opt       0x00000001072e25d0 llvm::sampleprof::SampleProfileReaderExtBinaryBase::readFuncProfiles() + 3476
11 opt       0x00000001072e02f4 llvm::sampleprof::SampleProfileReaderExtBinaryBase::readOneSection(unsigned char const*, unsigned long, llvm::sampleprof::SecHdrTableEntry const&) + 384
12 opt       0x00000001072e3994 llvm::sampleprof::SampleProfileReaderExtBinaryBase::readImpl() + 212
13 opt       0x0000000106da3210 llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) + 2172
4 opt       0x0000000105cf7ca8 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) + 480
15 opt       0x0000000103769858 llvm::runPassPipeline(llvm::StringRef, llvm::Module&, llvm::TargetMachine*, llvm::TargetLibraryInfoImpl*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::StringRef, llvm::ArrayRef<llvm::PassPlugin>, llvm::opt_tool::OutputKind, llvm::opt_tool::VerifierKind, bool, bool, bool, bool, bool, bool, bool) + 11060
16 opt       0x0000000103777f5c main + 8692
17 opt       0x0000000103766744 _start + 100

In addition, quite a number of other tests now FAIL with the same assertiion failure:

LLVM :: Transforms/SampleProfile/compressed-profile-symbol-list.ll
LLVM :: Transforms/SampleProfile/csspgo-import-list.ll
LLVM :: Transforms/SampleProfile/csspgo-inline-icall.ll
LLVM :: Transforms/SampleProfile/csspgo-inline.ll
LLVM :: Transforms/SampleProfile/fsafdo_test.ll
LLVM :: Transforms/SampleProfile/inline-mergeprof.ll
LLVM :: Transforms/SampleProfile/profile-context-tracker.ll
LLVM :: Transforms/SampleProfile/profile-format-compress.ll
LLVM :: Transforms/SampleProfile/profile-format.ll
LLVM :: Transforms/SampleProfile/profile-sample-accurate.ll
LLVM :: Transforms/SampleProfile/pseudo-probe-inline.ll
LLVM :: Transforms/SampleProfile/remap.ll
LLVM :: Transforms/SampleProfile/uncompressed-profile-symbol-list.ll
LLVM :: Transforms/SampleProfile/uniqname.ll

I've since retried with the latest version of the patch, but the failures remain.

Another question regarding ARM and SPARC, what is size_t on these machines? I suspect the error has something to do with that, since I couldn't replicate them on X64.

ro added a comment.Jul 12 2023, 3:01 PM

Another question regarding ARM and SPARC, what is size_t on these machines? I suspect the error has something to do with that, since I couldn't replicate them on X64.

Solaris <iso/stddef_iso.h> has

#if defined(_LP64) || defined(_I32LPx) 
typedef unsigned long   size_t;         /* size of something in bytes */
#else
typedef unsigned int    size_t;         /* (historical version) */ 
#endif

Nothing unusual here.

I still suspect that this is rather an endianess issue

huangjd updated this revision to Diff 539808.Jul 12 2023, 5:59 PM

In SampleProfileWriter, prevent already hashed function being hashed again when writing the profile

ro added a comment.Jul 13 2023, 3:36 AM

In SampleProfileWriter, prevent already hashed function being hashed again when writing the profile

This didn't make difference, unfortunately.

In D147740#4495250, @ro wrote:

Another question regarding ARM and SPARC, what is size_t on these machines? I suspect the error has something to do with that, since I couldn't replicate them on X64.

Solaris <iso/stddef_iso.h> has

#if defined(_LP64) || defined(_I32LPx) 
typedef unsigned long   size_t;         /* size of something in bytes */
#else
typedef unsigned int    size_t;         /* (historical version) */ 
#endif

Nothing unusual here.

I still suspect that this is rather an endianess issue

I am waiting for the approval to use GCC farm machine so that I can debug it. Do you know any person I can ping to expedite the process?

ro added a comment.Jul 13 2023, 12:20 PM

I am waiting for the approval to use GCC farm machine so that I can debug it. Do you know any person I can ping to expedite the process?

Unfortunately not: sometimes they're very quick, at others a response takes days (or even weeks).

huangjd updated this revision to Diff 540214.Jul 13 2023, 4:14 PM

Fix MD5 hash table write on big endian

Always use little endian write on Line 586 in SampleProfileReader.cpp, this should make it consistent on different systems

@huangjd any follow up on simplifying the implantation based on the assumption that collision is non-issue? Just want to make sure the comments on D153692 don't fall through the cracks.

ro added a comment.Jul 14 2023, 12:58 AM

Always use little endian write on Line 586 in SampleProfileReader.cpp, this should make it consistent on different systems

With the last revision, the assertion failures are gone. However, two tailures do remain:

+  LLVM :: Transforms/SampleProfile/inline-mergeprof.ll

Command Output (stderr):
--
/vol/llvm/src/llvm-project/local/llvm/test/Transforms/SampleProfile/profile-format.ll:27:10: error: CHECK: expected string not found in input
; CHECK: br i1 %cmp, label %while.body, label %while.end{{.*}} !prof ![[IDX1:[0-9]*]]
         ^
<stdin>:1:1: note: scanning from here
; ModuleID = '<stdin>'
^
<stdin>:32:2: note: possible intended match here
 br i1 %cmp, label %while.body, label %while.end, !dbg !42
 ^

+  LLVM :: Transforms/SampleProfile/profile-format.ll

Command Output (stderr):
--
/vol/llvm/src/llvm-project/local/llvm/test/Transforms/SampleProfile/profile-format.ll:27:10: error: CHECK: expected string not found in input
; CHECK: br i1 %cmp, label %while.body, label %while.end{{.*}} !prof ![[IDX1:[0-9]*]]
         ^
<stdin>:1:1: note: scanning from here
; ModuleID = '<stdin>'
^
<stdin>:32:2: note: possible intended match here
 br i1 %cmp, label %while.body, label %while.end, !dbg !42
 ^

@huangjd any follow up on simplifying the implantation based on the assumption that collision is non-issue? Just want to make sure the comments on D153692 don't fall through the cracks.

I am actually going to change that patch to removing the collision check

huangjd added a comment.EditedJul 24 2023, 11:59 AM

@ro
I am using the SPARC machine on GCC farm but it seems to hang on linking the unit tests. Is this normal? Is there an easy option to cross build all the binaries from a X86 machine targeting SPARC and send the file over?

What command did you use to build and test on SPARC?

ro added a comment.Jul 25 2023, 1:40 AM

@ro
I am using the SPARC machine on GCC farm but it seems to hang on linking the unit tests. Is this normal? Is there an easy option to cross build all the binaries from a X86 machine targeting SPARC and send the file over?

Which cfarm system did you use for the builds? There are several, both Solaris and Linux. I've avoided the Linux ones since they could be unreliable at times. I'd suggest going for gcc106, the only Solaris 11.4 box which is required for LLVM. There are two issues to be wary about:

  • Be careful to control the parallellism: the link steps can be quite memory intensive (for comparison's sake, my Solaris 11.4/SPARC box used to have 256 GB RAM and is now at 512 GB which helped tremendously).
  • If you are doing a debug build, the linking is painfully slow indeed (on the order of hours in some cases). Nothing to do about that but being patient.

I've never done cross builds and would expect them to be a real PITA, especially since clang only supports the Solaris linker right now, and GNU ld is not command-line compatible.

What command did you use to build and test on SPARC?

I'm using a script of my own to do the cmake invocation, for a considerable part handling the 2-stage build. Nothing really special in there, though.

huangjd updated this revision to Diff 544567.Jul 26 2023, 6:06 PM

Fix endianess issue on SPARC

This revision was landed with ongoing or failed builds.Jul 27 2023, 4:08 PM
This revision was automatically updated to reflect the committed changes.
antmo added a subscriber: antmo.Jul 28 2023, 12:03 AM

Hi, I think this patch broke clang-armv8-quick bot : https://lab.llvm.org/buildbot/#/builders/245/builds/11732
Could you please take a look ?

ro added a comment.Jul 28 2023, 12:53 AM

Fix endianess issue on SPARC

Thanks: Solaris/sparcv9 results with that revision are fine now indeed.

I think this patch broke clang-armv8-quick bot

Some context here, that's 32 bit Armv8. Our Armv7 32 bit bots aren't happy with it either.

Hi, I think this patch broke clang-armv8-quick bot : https://lab.llvm.org/buildbot/#/builders/245/builds/11732
Could you please take a look ?

This broke quite a few of the ARM bots, so I've reverted in https://github.com/llvm/llvm-project/commit/1a53b5c367b5ebf7d7f34afaa653ea337982f1d6 to hopefully get them back to green while you investigate. Sorry for any troubles!

MatzeB added a subscriber: MatzeB.Jul 28 2023, 9:24 AM

FWIW This was also crashing our x86 builds (using -flto=thin and -fprofile-sample-use). Asan reports a use-after-free: https://reviews.llvm.org/F28484911

In case its relevant: Those crashing builds use profile files created with slightly older versions of LLVM. I assume backwards compatibility is provided with this patch or at least some versioning check and proper error message if file format is incompatible?

In case its relevant: Those crashing builds use profile files created with slightly older versions of LLVM. I assume backwards compatibility is provided with this patch or at least some versioning check and proper error message if file format is incompatible?

Do you have the profile available for me to test?

Do you have the profile available for me to test?

The profiles are hundreds of megabytes and I am not sure whether I am even allowed to make them public. Though I can test changes or inspect the profiles if it helps. Otherwise I can try whether I can manage to create a smaller reproducer on monday.

huangjd reopened this revision.Aug 1 2023, 2:38 PM
This revision is now accepted and ready to land.Aug 1 2023, 2:38 PM
huangjd updated this revision to Diff 546232.Aug 1 2023, 2:38 PM
This comment was removed by huangjd.
huangjd updated this revision to Diff 546296.Aug 1 2023, 7:29 PM

Fixed more errors on 32-bit platform, always use uint64_t for function hash value

Hi, I think this patch broke clang-armv8-quick bot : https://lab.llvm.org/buildbot/#/builders/245/builds/11732
Could you please take a look ?

This broke quite a few of the ARM bots, so I've reverted in https://github.com/llvm/llvm-project/commit/1a53b5c367b5ebf7d7f34afaa653ea337982f1d6 to hopefully get them back to green while you investigate. Sorry for any troubles!

Are you able to verify that the current patch has resolved the problem? I do not have a 32-bit ARM machine, and GCC farm does not provide one either. I am not sure how to test it on buildbot without actually submitting it

@MatzeB

After some investigation it looks like it's an independent bug in Transforms/IPO/SampleProfile.cpp that has never been discovered.

In SampleProfile.cpp, non-inlined callees are added to the Profiles as new functions, which can trigger a rehashing and invalidates all iterator, including SampleProfileLoader::SampleProfileLoaderBaseImpl::Samples which is a pointer to the current function's FunctionSamples inside the profile. This pointer is later used so it is already undefined behavior.

LLVM's standard library implementation was able to expand the profile (originaly unordered_map) in place without relocating objects, so the code would work. In my patch I changed the container to llvm::DenseMap, which will always relocate objects on rehashing, causing the pointer to be invalid and crashing the program. Since std::unordered_map does not guarantee that either, this function is UB and needs to be rewritten

I am going to submit a patch to that first, although I am not able to create a small test case for that bug since std::unordered_map is doing well to avoid relocating objects as long as there is enough memory, so it would require more careful code review.

Thanks for looking into this! Sounds like we may need unreasonably large inputs so going without a test should be okay. Either as a separate patch or merged with this one, whatever works best for you.

Thanks for looking into this! Sounds like we may need unreasonably large inputs so going without a test should be okay. Either as a separate patch or merged with this one, whatever works best for you.

The fix is in D157061

@antmo @DavidSpickett @aaron.ballman
Are you able to check if the latest version fixed the issues on ARM-32? I do not have access to ARM-32 machine but was able to reproduce the same failing test cases in Win32 and fixed it. I would like to confirm it's working before landing the patch again

@antmo @DavidSpickett @aaron.ballman
Are you able to check if the latest version fixed the issues on ARM-32? I do not have access to ARM-32 machine but was able to reproduce the same failing test cases in Win32 and fixed it. I would like to confirm it's working before landing the patch again

I don't have access to an ARM machine myself; I only noticed the issue through the build farm. If you think the code is working, I think it's fine to land it again to see if the bots agree -- it's pretty normal to not have access to all the same hardware as the bots, so speculative commits are somewhat routine.

antmo added a comment.Aug 8 2023, 3:01 PM

Hi @huangjd, the 8 failures shown by the bot clang-armv8-quick look fixed in the latest version.

I think it's fine to land it again to see if the bots agree

Yes, totally fine to do this if you're around to revert promptly.

Thanks Antoine for testing the patch!

This revision was landed with ongoing or failed builds.Aug 17 2023, 1:11 PM
This revision was automatically updated to reflect the committed changes.

This diff is causing clang to crash on one of our builds, when clang is linked with jemalloc.
-DCMAKE_EXE_LINKER_FLAGS="-L /usr/lib64 -Wl,-rpath,/usr/lib64 -ljemalloc" \

I am trying to come up with small repro, but in meantime can this diff be reverted?

Full stack trace:

 #0 0x000055b18add6a58 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:723:13
 #1 0x000055b18add4950 llvm::sys::RunSignalHandlers() /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Support/Signals.cpp:106:18
 #2 0x000055b18add722d SignalHandler(int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:413:1
 #3 0x00007ffaa7812cf0 __restore_rt (/usr/lib64/libpthread.so.0+0x12cf0)
 #4 0x000055b18c0c9033 llvm::sampleprof::LineLocation::operator<(llvm::sampleprof::LineLocation const&) const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:296:23
 #5 0x000055b18c0c9033 std::less<llvm::sampleprof::LineLocation>::operator()(llvm::sampleprof::LineLocation const&, llvm::sampleprof::LineLocation const&) const /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_function.h:408:20
 #6 0x000055b18c0c9033 std::_Rb_tree<llvm::sampleprof::LineLocation, std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>, std::_Select1st<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>, std::less<llvm::sampleprof::LineLocation>, std::allocator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_iterator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>, llvm::sampleprof::LineLocation const&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_tree.h:2220:11
 #7 0x000055b18c0caf2d std::_Rb_tree_iterator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>> std::_Rb_tree<llvm::sampleprof::LineLocation, std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>, std::_Select1st<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>, std::less<llvm::sampleprof::LineLocation>, std::allocator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>>::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<llvm::sampleprof::LineLocation&&>, std::tuple<>>(std::_Rb_tree_const_iterator<std::pair<llvm::sampleprof::LineLocation const, llvm::sampleprof::SampleRecord>>, std::piecewise_construct_t const&, std::tuple<llvm::sampleprof::LineLocation&&>&&, std::tuple<>&&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_tree.h:2462:15
 #8 0x000055b18c0c9bfb llvm::sampleprof::FunctionSamples::addBodySamples(unsigned int, unsigned int, unsigned long, unsigned long) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_map.h:0:15
 #9 0x000055b18c0c6d27 llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1599:21
#10 0x000055b18c0c659f llvm::sampleprof::ProfileConverter::flattenProfile(llvm::sampleprof::SampleProfileMap const&, llvm::sampleprof::SampleProfileMap&, bool) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:0:9
#11 0x000055b18c0c0e28 std::__uniq_ptr_impl<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>::reset((anonymous namespace)::SampleProfileMatcher*) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:200:26
#12 0x000055b18c0c0e28 std::__uniq_ptr_impl<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>::operator=(std::__uniq_ptr_impl<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>&&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:183:2
#13 0x000055b18c0c0e28 std::__uniq_ptr_data<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>, true, true>::operator=(std::__uniq_ptr_data<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>, true, true>&&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:235:61
#14 0x000055b18c0c0e28 std::unique_ptr<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>::operator=(std::unique_ptr<(anonymous namespace)::SampleProfileMatcher, std::default_delete<(anonymous namespace)::SampleProfileMatcher>>&&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:406:51
#15 0x000055b18c0c0e28 (anonymous namespace)::SampleProfileLoader::doInitialization(llvm::Module&, llvm::AnalysisManager<llvm::Function>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2104:21
#16 0x000055b18c0c0e28 llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2632:21
#17 0x000055b18bfcfb0d llvm::detail::PassModel<llvm::Module, llvm::SampleProfileLoaderPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:89:5
#18 0x000055b18a95e3b9 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:521:10
#19 0x000055b18b4f33ff llvm::SmallPtrSetImplBase::isSmall() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/SmallPtrSet.h:195:33
#20 0x000055b18b4f33ff llvm::SmallPtrSetImplBase::~SmallPtrSetImplBase() /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/SmallPtrSet.h:83:10
#21 0x000055b18b4f33ff llvm::PreservedAnalyses::~PreservedAnalyses() /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:152:7
#22 0x000055b18b4f33ff (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>&, std::unique_ptr<llvm::ToolOutputFile, std::default_delete<llvm::ToolOutputFile>>&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1101:5
#23 0x000055b18b4eaa22 (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:0:3
#24 0x000055b18b4eaa22 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1321:13
#25 0x000055b18b95a8f5 std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>::~unique_ptr() /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unique_ptr.h:395:6
#26 0x000055b18b95a8f5 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:386:7
#27 0x000055b18cf852c6 __gnu_cxx::__normal_iterator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>*, std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>>>>::__normal_iterator(std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>* const&) /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_iterator.h:1073:20
#28 0x000055b18cf852c6 std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>>>::begin() /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:869:16
#29 0x000055b18cf852c6 void clang::finalize<std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>>>>(std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback>>>>&, clang::Sema const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/include/clang/Sema/TemplateInstCallback.h:54:16
#30 0x000055b18cf852c6 clang::ParseAST(clang::Sema&, bool, bool) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:183:3
#31 0x000055b18b881b10 clang::FrontendAction::Execute() /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1067:10
#32 0x000055b18b7fdc3d llvm::Error::getPtr() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/Support/Error.h:270:42
#33 0x000055b18b7fdc3d llvm::Error::operator bool() /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/Support/Error.h:233:16
#34 0x000055b18b7fdc3d clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1054:23
#35 0x000055b18b953335 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:272:25
#36 0x000055b189dc769c cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/cc1_main.cpp:249:15
#37 0x000055b189dc479a ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:366:12
#38 0x000055b189dc3732 clang_main(int, char**, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:407:12
#39 0x000055b189dd2d71 main /home/ayermolo/local/llvm-build-upstream-release/tools/clang/tools/driver/clang-driver.cpp:15:3
#40 0x00007ffaa583ad85 __libc_start_main (/usr/lib64/libc.so.6+0x3ad85)
#41 0x000055b189dc102e _start (/home/ayermolo/local/llvm-build-upstream-release/bin/clang+++0x2a6a02e)

@huangjd

MSAN output. Does this ring any bells?

==3359553==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564b193d7f4e in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30
    #1 0x564b193ce93a in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1612:36
    #2 0x564b193ce582 in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1607:9
    #3 0x564b193ca9cc in llvm::sampleprof::ProfileConverter::flattenProfile(llvm::sampleprof::SampleProfileMap const&, llvm::sampleprof::SampleProfileMap&, bool) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1558:9
    #4 0x564b1936a0f6 in (anonymous namespace)::SampleProfileMatcher::SampleProfileMatcher(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager const*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:466:7
    #5 0x564b1936a0f6 in std::__1::__unique_if<(anonymous namespace)::SampleProfileMatcher>::__unique_single std::__1::make_unique[abi:v180000]<(anonymous namespace)::SampleProfileMatcher, llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*>(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*&&) /home/ayermolo/local/upstream-llvm/llvm-project/build_libcxx/include/c++/v1/__memory/unique_ptr.h:685:30
    #6 0x564b1936a0f6 in (anonymous namespace)::SampleProfileLoader::doInitialization(llvm::Module&, llvm::AnalysisManager<llvm::Function>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2105:9
    #7 0x564b1936a0f6 in llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2632:21
    #8 0x564b18f338ac in llvm::detail::PassModel<llvm::Module, llvm::SampleProfileLoaderPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:89:17
    #9 0x564b12a1306b in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #10 0x564b15b5e1a7 in (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>&, std::__1::unique_ptr<llvm::ToolOutputFile, std::__1::default_delete<llvm::ToolOutputFile>>&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1101:9
    #11 0x564b15b447fd in (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1158:3
    #12 0x564b15b447fd in clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1321:13
    #13 0x564b16f72ca1 in clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:386:7
    #14 0x564b1d01a29b in clang::ParseAST(clang::Sema&, bool, bool) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:176:13
    #15 0x564b16be4f7e in clang::FrontendAction::Execute() /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1063:8
    #16 0x564b1695cd8d in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1054:33
    #17 0x564b16f50383 in clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:272:25
    #18 0x564b0f3bb128 in cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/cc1_main.cpp:249:15
    #19 0x564b0f3b0d6a in ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:366:12
    #20 0x564b0f3abb78 in clang_main(int, char**, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:407:12
    #21 0x564b0f3ec39c in main /home/ayermolo/local/llvm-build-upstream-msan-release/tools/clang/tools/driver/clang-driver.cpp:15:10
    #22 0x7f73e6a3ad84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: 1356e140fb964a20b0d2838960ee69ca6faeb034)
    #23 0x564b0f31642d in _start (/data/users/ayermolo/llvm-build-upstream-msan-release/bin/clang-18+0x315042d)

  Uninitialized value was created by a heap deallocation
    #0 0x564b0f3a2269 in operator delete(void*, std::align_val_t) /home/ayermolo/local/upstream-llvm/llvm-project/compiler-rt/lib/msan/msan_new_delete.cpp:90:3
    #1 0x564b193d94e5 in llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::grow(unsigned int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:564:36
    #2 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucketImpl<llvm::hash_code>(llvm::hash_code const&, llvm::hash_code const&, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h
    #3 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucket<llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&>(llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*, llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:574:17

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30 in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const
Exiting

What were you building when this error happened?

@huangjd

MSAN output. Does this ring any bells?

==3359553==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564b193d7f4e in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30
    #1 0x564b193ce93a in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1612:36
    #2 0x564b193ce582 in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1607:9
    #3 0x564b193ca9cc in llvm::sampleprof::ProfileConverter::flattenProfile(llvm::sampleprof::SampleProfileMap const&, llvm::sampleprof::SampleProfileMap&, bool) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1558:9
    #4 0x564b1936a0f6 in (anonymous namespace)::SampleProfileMatcher::SampleProfileMatcher(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager const*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:466:7
    #5 0x564b1936a0f6 in std::__1::__unique_if<(anonymous namespace)::SampleProfileMatcher>::__unique_single std::__1::make_unique[abi:v180000]<(anonymous namespace)::SampleProfileMatcher, llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*>(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*&&) /home/ayermolo/local/upstream-llvm/llvm-project/build_libcxx/include/c++/v1/__memory/unique_ptr.h:685:30
    #6 0x564b1936a0f6 in (anonymous namespace)::SampleProfileLoader::doInitialization(llvm::Module&, llvm::AnalysisManager<llvm::Function>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2105:9
    #7 0x564b1936a0f6 in llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2632:21
    #8 0x564b18f338ac in llvm::detail::PassModel<llvm::Module, llvm::SampleProfileLoaderPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:89:17
    #9 0x564b12a1306b in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #10 0x564b15b5e1a7 in (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>&, std::__1::unique_ptr<llvm::ToolOutputFile, std::__1::default_delete<llvm::ToolOutputFile>>&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1101:9
    #11 0x564b15b447fd in (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1158:3
    #12 0x564b15b447fd in clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1321:13
    #13 0x564b16f72ca1 in clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:386:7
    #14 0x564b1d01a29b in clang::ParseAST(clang::Sema&, bool, bool) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:176:13
    #15 0x564b16be4f7e in clang::FrontendAction::Execute() /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1063:8
    #16 0x564b1695cd8d in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1054:33
    #17 0x564b16f50383 in clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:272:25
    #18 0x564b0f3bb128 in cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/cc1_main.cpp:249:15
    #19 0x564b0f3b0d6a in ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:366:12
    #20 0x564b0f3abb78 in clang_main(int, char**, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:407:12
    #21 0x564b0f3ec39c in main /home/ayermolo/local/llvm-build-upstream-msan-release/tools/clang/tools/driver/clang-driver.cpp:15:10
    #22 0x7f73e6a3ad84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: 1356e140fb964a20b0d2838960ee69ca6faeb034)
    #23 0x564b0f31642d in _start (/data/users/ayermolo/llvm-build-upstream-msan-release/bin/clang-18+0x315042d)

  Uninitialized value was created by a heap deallocation
    #0 0x564b0f3a2269 in operator delete(void*, std::align_val_t) /home/ayermolo/local/upstream-llvm/llvm-project/compiler-rt/lib/msan/msan_new_delete.cpp:90:3
    #1 0x564b193d94e5 in llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::grow(unsigned int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:564:36
    #2 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucketImpl<llvm::hash_code>(llvm::hash_code const&, llvm::hash_code const&, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h
    #3 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucket<llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&>(llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*, llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:574:17

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30 in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const
Exiting
ayermolo added a comment.EditedAug 23 2023, 2:28 PM

What were you building when this error happened?

@huangjd

MSAN output. Does this ring any bells?

==3359553==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564b193d7f4e in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30
    #1 0x564b193ce93a in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1612:36
    #2 0x564b193ce582 in llvm::sampleprof::ProfileConverter::flattenNestedProfile(llvm::sampleprof::SampleProfileMap&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1607:9
    #3 0x564b193ca9cc in llvm::sampleprof::ProfileConverter::flattenProfile(llvm::sampleprof::SampleProfileMap const&, llvm::sampleprof::SampleProfileMap&, bool) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:1558:9
    #4 0x564b1936a0f6 in (anonymous namespace)::SampleProfileMatcher::SampleProfileMatcher(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager const*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:466:7
    #5 0x564b1936a0f6 in std::__1::__unique_if<(anonymous namespace)::SampleProfileMatcher>::__unique_single std::__1::make_unique[abi:v180000]<(anonymous namespace)::SampleProfileMatcher, llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*>(llvm::Module&, llvm::sampleprof::SampleProfileReader&, llvm::PseudoProbeManager*&&) /home/ayermolo/local/upstream-llvm/llvm-project/build_libcxx/include/c++/v1/__memory/unique_ptr.h:685:30
    #6 0x564b1936a0f6 in (anonymous namespace)::SampleProfileLoader::doInitialization(llvm::Module&, llvm::AnalysisManager<llvm::Function>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2105:9
    #7 0x564b1936a0f6 in llvm::SampleProfileLoaderPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/lib/Transforms/IPO/SampleProfile.cpp:2632:21
    #8 0x564b18f338ac in llvm::detail::PassModel<llvm::Module, llvm::SampleProfileLoaderPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:89:17
    #9 0x564b12a1306b in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #10 0x564b15b5e1a7 in (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>&, std::__1::unique_ptr<llvm::ToolOutputFile, std::__1::default_delete<llvm::ToolOutputFile>>&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1101:9
    #11 0x564b15b447fd in (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1158:3
    #12 0x564b15b447fd in clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1321:13
    #13 0x564b16f72ca1 in clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:386:7
    #14 0x564b1d01a29b in clang::ParseAST(clang::Sema&, bool, bool) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:176:13
    #15 0x564b16be4f7e in clang::FrontendAction::Execute() /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1063:8
    #16 0x564b1695cd8d in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1054:33
    #17 0x564b16f50383 in clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:272:25
    #18 0x564b0f3bb128 in cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/cc1_main.cpp:249:15
    #19 0x564b0f3b0d6a in ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:366:12
    #20 0x564b0f3abb78 in clang_main(int, char**, llvm::ToolContext const&) /home/ayermolo/local/upstream-llvm/llvm-project/clang/tools/driver/driver.cpp:407:12
    #21 0x564b0f3ec39c in main /home/ayermolo/local/llvm-build-upstream-msan-release/tools/clang/tools/driver/clang-driver.cpp:15:10
    #22 0x7f73e6a3ad84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: 1356e140fb964a20b0d2838960ee69ca6faeb034)
    #23 0x564b0f31642d in _start (/data/users/ayermolo/llvm-build-upstream-msan-release/bin/clang-18+0x315042d)

  Uninitialized value was created by a heap deallocation
    #0 0x564b0f3a2269 in operator delete(void*, std::align_val_t) /home/ayermolo/local/upstream-llvm/llvm-project/compiler-rt/lib/msan/msan_new_delete.cpp:90:3
    #1 0x564b193d94e5 in llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::grow(unsigned int) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:564:36
    #2 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucketImpl<llvm::hash_code>(llvm::hash_code const&, llvm::hash_code const&, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h
    #3 0x564b193d94e5 in llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>* llvm::DenseMapBase<llvm::DenseMap<llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>, llvm::hash_code, llvm::sampleprof::FunctionSamples, llvm::DenseMapInfo<llvm::hash_code, void>, llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>>::InsertIntoBucket<llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&>(llvm::detail::DenseMapPair<llvm::hash_code, llvm::sampleprof::FunctionSamples>*, llvm::hash_code const&, llvm::sampleprof::FunctionSamples const&) /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ADT/DenseMap.h:574:17

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/ayermolo/local/upstream-llvm/llvm-project/llvm/include/llvm/ProfileData/SampleProf.h:959:30 in llvm::sampleprof::FunctionSamples::getHeadSamplesEstimate() const
Exiting

It's c++ code
"-triple" "x86_64-redhat-linux-gnu" "-emit-llvm-bc" "-flto=thin" "-flto-unit" ... "-fprofile-sample-use=<path>/__autofdo-bolt-compatible__/out/profile" "-fpseudo-probe-for-profiling"
Some of the more relevant options I think.
it's a csspgo profile.

I think I figured out what's the problem, writing a patch now

huangjd reopened this revision.Aug 28 2023, 12:18 PM
This revision is now accepted and ready to land.Aug 28 2023, 12:18 PM
huangjd closed this revision.Sep 14 2023, 2:22 PM