Reduce memory consumption of coverage dumps

[serge-sans-paille]

Summary

Avoiding an intermediate join operation, which in turns removes the need for an
intermediate buffer that may be quite large, as showcased by

https://bugs.llvm.org/show_bug.cgi?id=41965

[kwk]

@serge-sans-paille testing it now.

[kwk]

In D62623#1521999, @kwk wrote:

@serge-sans-paille testing it now.

Unfortunately, the error for me is similar (or the same) to before;

[692/2860] Building CXX object lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.o
FAILED: lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.o 
/usr/bin/clang++  -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Target/AMDGPU/Utils -I/home/kkleine/dev/llvm-project/llvm/lib/Target/AMDGPU/Utils -I/home/kkleine/dev/llvm-project/llvm/lib/Target/AMDGPU -Ilib/Target/AMDGPU -I/usr/include/libxml2 -Iinclude -I/home/kkleine/dev/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -fprofile-instr-generate='/home/kkleine/llvm-builds/coverage-ninja-clang-lld/profiles/%4m.profraw' -fcoverage-mapping -O2 -DNDEBUG    -fno-exceptions -fno-rtti -MD -MT lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.o -MF lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.o.d -o lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.o -c /home/kkleine/dev/llvm-project/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
LLVM ERROR: out of memory
Stack dump:
0.	Program arguments: /usr/bin/clang-7 -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name AMDGPUBaseInfo.cpp -mrelocation-model pic -pic-level 2 -mthread-model posix -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -dwarf-column-info -debugger-tuning=gdb -momit-leaf-frame-pointer -ffunction-sections -fdata-sections -fprofile-instrument-path=/home/kkleine/llvm-builds/coverage-ninja-clang-lld/profiles/%4m.profraw -fprofile-instrument=clang -fcoverage-mapping -coverage-notes-file /home/kkleine/llvm-builds/coverage-ninja-clang-lld/lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.gcno -resource-dir /usr/lib64/clang/7.0.1 -dependency-file lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.o.d -sys-header-deps -MT lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.o -D GTEST_HAS_RTTI=0 -D _GNU_SOURCE -D __STDC_CONSTANT_MACROS -D __STDC_FORMAT_MACROS -D __STDC_LIMIT_MACROS -I lib/Target/AMDGPU/Utils -I /home/kkleine/dev/llvm-project/llvm/lib/Target/AMDGPU/Utils -I /home/kkleine/dev/llvm-project/llvm/lib/Target/AMDGPU -I lib/Target/AMDGPU -I /usr/include/libxml2 -I include -I /home/kkleine/dev/llvm-project/llvm/include -D NDEBUG -internal-isystem /usr/bin/../lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8 -internal-isystem /usr/bin/../lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/x86_64-redhat-linux -internal-isystem /usr/bin/../lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/backward -internal-isystem /usr/local/include -internal-isystem /usr/lib64/clang/7.0.1/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O2 -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -pedantic -std=c++11 -fdeprecated-macro -fdebug-compilation-dir /home/kkleine/llvm-builds/coverage-ninja-clang-lld -ferror-limit 19 -fmessage-length 0 -fvisibility-inlines-hidden -fno-rtti -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -vectorize-loops -vectorize-slp -o lib/Target/AMDGPU/Utils/CMakeFiles/LLVMAMDGPUUtils.dir/AMDGPUBaseInfo.cpp.o -x c++ /home/kkleine/dev/llvm-project/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp -faddrsig 
1.	<eof> parser at end of file
2.	Per-file LLVM IR generation
#0 0x00007f814657b09e llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/lib64/libLLVM-7.so+0x9b409e)
#1 0x00007f8146579554 llvm::sys::RunSignalHandlers() (/lib64/libLLVM-7.so+0x9b2554)
#2 0x00007f81465796d5 (/lib64/libLLVM-7.so+0x9b26d5)
#3 0x00007f8145bb8070 __restore_rt (/lib64/libpthread.so.0+0x13070)
#4 0x00007f8144b6457f __GI_raise (/lib64/libc.so.6+0x3857f)
#5 0x00007f8144b4e895 __GI_abort (/lib64/libc.so.6+0x22895)
#6 0x00007f81464fde34 llvm::report_bad_alloc_error(char const*, bool) (/lib64/libLLVM-7.so+0x936e34)
#7 0x00007f8144f28b34 operator new(unsigned long) (/lib64/libstdc++.so.6+0x97b34)
#8 0x00007f814573c8f7 clang::CodeGen::CoverageMappingModuleGen::emit() (/lib64/libclangCodeGen.so.7+0x3958f7)
#9 0x00007f81456ffbe0 clang::CodeGen::CodeGenModule::Release() (/lib64/libclangCodeGen.so.7+0x358be0)
#10 0x00007f814576af1a (/lib64/libclangCodeGen.so.7+0x3c3f1a)
#11 0x00007f81456a9a2f (/lib64/libclangCodeGen.so.7+0x302a2f)
#12 0x00007f8144437417 clang::ParseAST(clang::Sema&, bool, bool) (/lib64/libclangParse.so.7+0x3c417)
#13 0x00007f81456a70ca clang::CodeGenAction::ExecuteAction() (/lib64/libclangCodeGen.so.7+0x3000ca)
#14 0x00007f81451237a6 clang::FrontendAction::Execute() (/lib64/libclangFrontend.so.7+0xee7a6)
#15 0x00007f81450de2fe clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/lib64/libclangFrontend.so.7+0xa92fe)
#16 0x00007f814502eaa8 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/lib64/libclangFrontendTool.so.7+0x5aa8)
#17 0x00005644c3b2df98 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/usr/bin/clang-7+0x15f98)
#18 0x00005644c3b29ab8 main (/usr/bin/clang-7+0x11ab8)
#19 0x00007f8144b50413 __libc_start_main (/lib64/libc.so.6+0x24413)
#20 0x00005644c3b2a63e _start (/usr/bin/clang-7+0x1263e)
clang-7: error: unable to execute command: Aborted (core dumped)
clang-7: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 7.0.1 (Fedora 7.0.1-6.fc29)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
clang-7: note: diagnostic msg: PLEASE submit a bug report to  and include the crash backtrace, preprocessed source, and associated run script.
clang-7: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-7: note: diagnostic msg: /tmp/AMDGPUBaseInfo-d40c2b.cpp
clang-7: note: diagnostic msg: /tmp/AMDGPUBaseInfo-d40c2b.sh
clang-7: note: diagnostic msg: 

********************
ninja: build stopped: subcommand failed.

[serge-sans-paille]

@kwk: looks like you're still compiling with clang-7, this patch is to be applied on the master version of LLVM/clang, then you can install it and use it to recompile llvm/lld with coverage info.

Does it make sense to you?

[kwk]

In D62623#1522575, @serge-sans-paille wrote:

@kwk: looks like you're still compiling with clang-7, this patch is to be applied on the master version of LLVM/clang, then you can install it and use it to recompile llvm/lld with coverage info.

Does it make sense to you?

@serge-sans-paille oh yes. Absolutely. Thank you! I will try that out right away.

[vsk]

clang/lib/CodeGen/CoverageMappingGen.cpp

1393

It doesn't look like the CoverageMappings std::vector is needed at all. Consider moving FilenamesAndCoverageMappings into CoverageMappingModuleGen?

That should reduce the memory requirements a bit more.

[kwk]

In D62623#1524666, @kwk wrote:

In D62623#1522575, @serge-sans-paille wrote:

@kwk: looks like you're still compiling with clang-7, this patch is to be applied on the master version of LLVM/clang, then you can install it and use it to recompile llvm/lld with coverage info.

Does it make sense to you?

@serge-sans-paille oh yes. Absolutely. Thank you! I will try that out right away.

I no longer have a crash during compilation of ninja lldb. Thank you so much @serge-sans-paille for addressing this and helping me on IRC.

[serge-sans-paille]

Update comment + force reduced memory consumption.

[serge-sans-paille]

clang/lib/CodeGen/CoverageMappingGen.cpp

1393

It's not possible to directly store into FilenamesAndCoverageMappings because filenames need to be stored before all mappings. It's possible to use a std::string instead of a std::vector but it means that at some point, memory usage will be | CoverageMappings| x 2 while if we empty the vector elements as we push them to the stream, we should keep |CoverageMappings|. I've updated the diff to enforce this stream behavior.

[vsk]

LGTM.

clang/lib/CodeGen/CoverageMappingGen.cpp

1393

Ah, thanks for explaining.

clang/lib/CodeGen/CoverageMappingGen.cpp

for (const auto &Entry : FileEntries) {
     auto I = Entry.second;
     FilenameStrs[I] = normalizeFilename(Entry.first->getName());
     FilenameRefs[I] = FilenameStrs[I];
   }
 
   std::string FilenamesAndCoverageMappings;
   llvm::raw_string_ostream OS(FilenamesAndCoverageMappings);
   CoverageFilenamesSectionWriter(FilenameRefs).write(OS);
-  std::string RawCoverageMappings =
-      llvm::join(CoverageMappings.begin(), CoverageMappings.end(), "");
-  OS << RawCoverageMappings;
-  size_t CoverageMappingSize = RawCoverageMappings.size();
+
+  // Stream the content of CoverageMappings to OS while keeping
+  // memory consumption under control.
+  size_t CoverageMappingSize = 0;
+  for (auto &S : CoverageMappings) {
+    CoverageMappingSize += S.size();
+    OS << S;
+    S.clear();
+    S.shrink_to_fit();
+  }
+  CoverageMappings.clear();
+  CoverageMappings.shrink_to_fit();
+
   size_t FilenamesSize = OS.str().size() - CoverageMappingSize;
   // Append extra zeroes if necessary to ensure that the size of the filenames
   // and coverage mappings is a multiple of 8.
   if (size_t Rem = OS.str().size() % 8) {
     CoverageMappingSize += 8 - Rem;
     OS.write_zeros(8 - Rem);
   }
   auto *FilenamesAndMappingsVal =