Fix GSYM tests to run the yaml files and fix test failures on some machines.

[clayborg]

Summary

YAML files were not being run during lit testing as there was no lit.local.cfg file. Once this was fixed, some buildbots would fail due to a StringRef that pointed to a std::string inside of a temporary llvm::Triple object. These issues are fixed here by making a local triple object that stays around long enough so the StringRef points to valid data.

[clayborg]

These changes can easily be tested with:

./bin/llvm-lit -v ../llvm-project/llvm/test/tools/llvm-gsymutil

[thakis]

This breaks tests on Windows: http://45.33.8.238/win/9449/step_11.txt

Please take a look, and revert for now if the fix isn't obvious.

[clayborg]

Reverted with:

commit 5d11e7f81cb60abc94deb0c4e41c48f39ea1da1c (HEAD -> master, origin/master, origin/HEAD)
Author: Greg Clayton <[e-mail]>
Date: Fri Feb 28 21:19:05 2020 -0800

Revert "Fix GSYM tests to run the yaml files and fix test failures on some machines."

This reverts commit d334ce0b5acb945d6202d0ab6a17bdca530f50c1.

[clayborg]

I figured out what was going on the build servers: function calls ordering in parameters lists is not defined which caused file table entries, which split string into directory and filename, to come out directory first, then basename, or basename then directory name. Should be an easy fix.

[clayborg]

Fixed an issue causing buildbots to fail where strings from files in the file table could be added in an undefined order.

[clayborg]

Trying again with:

commit 57688350adea307e7bccb83b68a5b7333de31fd7 (HEAD -> master, origin/master, origin/HEAD)
Author: Greg Clayton <[e-mail]>
Date: Mon Mar 2 12:40:46 2020 -0800

Fix GSYM tests to run the yaml files and fix test failures on some machines.

YAML files were not being run during lit testing as there was no lit.local.cfg file. Once this was fixed, some buildbots would fail due to a StringRef that pointed to a std::string inside of a temporary llvm::Triple object. These issues are fixed here by making a local triple object that stays around long enough so the StringRef points to valid data. Also fixed an issue where strings for files in the file table could be added in opposite order due to parameters to function calls not having a strong ordering, which caused tests to fail.

[clayborg]

Need to conditionalize for ARM targets, this is failing on machines that don't have ARM targets.

commit e3afe5952dfa9851738887eec0a84fa5cafe07ce (HEAD -> master, origin/master, origin/HEAD)
Author: Greg Clayton <[e-mail]>
Date: Mon Mar 2 13:07:58 2020 -0800

Revert "Fix GSYM tests to run the yaml files and fix test failures on some machines."

This reverts commit 57688350adea307e7bccb83b68a5b7333de31fd7.

Need to conditionalize for ARM targets, this is failing on machines that don't have ARM targets.

[clayborg]

Added architecture specific directories so when targets are not enabled, tests don't run.

[hctim]

This patch broke the MSan buildbot. The MSan bot is fickle to work (as you need a sanitized libcxx), the easiest way to reproduce the issue is following these instructions.

Full MSan report with -fsanitize-memory-track-origins=2:

Uninitialized bytes in __interceptor_write at offset 24 inside [0x720000002000, 118)
==146908==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5d94bc7 in llvm::raw_fd_ostream::write_impl(char const*, unsigned long) /usr/local/google/home/mitchp/llvm/llvm/lib/Support/raw_ostream.cpp:712:19
    #1 0x5d94fc2 in flush_nonempty /usr/local/google/home/mitchp/llvm/llvm/lib/Support/raw_ostream.cpp:219:3
    #2 0x5d94fc2 in flush /usr/local/google/home/mitchp/llvm/llvm/include/llvm/Support/raw_ostream.h:157:7
    #3 0x5d94fc2 in llvm::raw_fd_ostream::seek(unsigned long) /usr/local/google/home/mitchp/llvm/llvm/lib/Support/raw_ostream.cpp:754:3
    #4 0x5d95493 in llvm::raw_fd_ostream::pwrite_impl(char const*, unsigned long, unsigned long) /usr/local/google/home/mitchp/llvm/llvm/lib/Support/raw_ostream.cpp:770:3
    #5 0x567ee45 in pwrite /usr/local/google/home/mitchp/llvm/llvm/include/llvm/Support/raw_ostream.h:387:5
    #6 0x567ee45 in llvm::gsym::FileWriter::fixup32(unsigned int, unsigned long) /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/FileWriter.cpp:55:6
    #7 0x56847a4 in llvm::gsym::FunctionInfo::encode(llvm::gsym::FileWriter&) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/FunctionInfo.cpp:126:7
    #8 0x566321a in llvm::gsym::GsymCreator::encode(llvm::gsym::FileWriter&) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp:146:51
    #9 0x56619f9 in llvm::gsym::GsymCreator::save(llvm::StringRef, llvm::support::endianness) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp:56:10
    #10 0x1736921 in handleObjectFile(llvm::object::ObjectFile&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:335:23
    #11 0x1726afa in handleBuffer /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:391:22
    #12 0x1726afa in handleFileConversionToGSYM /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:404:10
    #13 0x1726afa in convertFileToGSYM /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:422:20
    #14 0x1726afa in main /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:463:20
    #15 0x7f483a67bbba in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26bba)
    #16 0x16ac059 in _start (/usr/local/google/home/mitchp/llvm-build/msan/sanitized-clang/bin/llvm-gsymutil+0x16ac059)

  Uninitialized value was stored to memory at
    #0 0x5d8eb94 in copy_to_buffer /usr/local/google/home/mitchp/llvm/llvm/lib/Support/raw_ostream.cpp:294:24
    #1 0x5d8eb94 in llvm::raw_ostream::write(char const*, unsigned long) /usr/local/google/home/mitchp/llvm/llvm/lib/Support/raw_ostream.cpp:280:3
    #2 0x567ea18 in llvm::gsym::FileWriter::writeU32(unsigned int) /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/FileWriter.cpp:45:6
    #3 0x568dd27 in llvm::gsym::Header::encode(llvm::gsym::FileWriter&) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/Header.cpp:96:5
    #4 0x5662275 in llvm::gsym::GsymCreator::encode(llvm::gsym::FileWriter&) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp:101:25
    #5 0x56619f9 in llvm::gsym::GsymCreator::save(llvm::StringRef, llvm::support::endianness) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp:56:10
    #6 0x1736921 in handleObjectFile(llvm::object::ObjectFile&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:335:23
    #7 0x1726afa in handleBuffer /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:391:22
    #8 0x1726afa in handleFileConversionToGSYM /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:404:10
    #9 0x1726afa in convertFileToGSYM /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:422:20
    #10 0x1726afa in main /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:463:20
    #11 0x7f483a67bbba in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26bba)

  Uninitialized value was stored to memory at
    #0 0x567ea3a in llvm::gsym::FileWriter::writeU32(unsigned int) /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/FileWriter.cpp:44:18
    #1 0x568dd27 in llvm::gsym::Header::encode(llvm::gsym::FileWriter&) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/Header.cpp:96:5
    #2 0x5662275 in llvm::gsym::GsymCreator::encode(llvm::gsym::FileWriter&) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp:101:25
    #3 0x56619f9 in llvm::gsym::GsymCreator::save(llvm::StringRef, llvm::support::endianness) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp:56:10
    #4 0x1736921 in handleObjectFile(llvm::object::ObjectFile&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:335:23
    #5 0x1726afa in handleBuffer /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:391:22
    #6 0x1726afa in handleFileConversionToGSYM /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:404:10
    #7 0x1726afa in convertFileToGSYM /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:422:20
    #8 0x1726afa in main /usr/local/google/home/mitchp/llvm/llvm/tools/llvm-gsymutil/llvm-gsymutil.cpp:463:20
    #9 0x7f483a67bbba in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26bba)

  Uninitialized value was created by an allocation of 'Hdr' in the stack frame of function '_ZNK4llvm4gsym11GsymCreator6encodeERNS0_10FileWriterE'
    #0 0x5661a80 in llvm::gsym::GsymCreator::encode(llvm::gsym::FileWriter&) const /usr/local/google/home/mitchp/llvm/llvm/lib/DebugInfo/GSYM/GsymCreator.cpp:59

SUMMARY: MemorySanitizer: use-of-uninitialized-value /usr/local/google/home/mitchp/llvm/llvm/lib/Support/raw_ostream.cpp:712:19 in llvm::raw_fd_ostream::write_impl(char const*, unsigned long)
Exiting

From the bot:

-- Testing: 36342 tests, 96 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
FAIL: LLVM :: tools/llvm-gsymutil/X86/elf-dwarf.yaml (35798 of 36342)
******************** TEST 'LLVM :: tools/llvm-gsymutil/X86/elf-dwarf.yaml' FAILED ********************
Script:
--
: 'RUN: at line 5';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/yaml2obj /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/elf-dwarf.yaml -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/elf-dwarf.yaml.tmp
: 'RUN: at line 6';   llvm-gsymutil --convert /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/elf-dwarf.yaml.tmp -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/elf-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/elf-dwarf.yaml --check-prefix=CONVERT
: 'RUN: at line 7';   llvm-gsymutil --address=0x400391 --address=0x4004cd /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/elf-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/elf-dwarf.yaml --check-prefix=ADDR
: 'RUN: at line 8';   llvm-gsymutil --address=0x400391 --address=0x4004cd --verbose /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/elf-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/elf-dwarf.yaml --check-prefix=ADDRV --dump-input=always
: 'RUN: at line 9';   llvm-gsymutil /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/elf-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/elf-dwarf.yaml --check-prefix=DUMP
--
Exit Code: 1

Command Output (stderr):
--
/b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/elf-dwarf.yaml:29:12: error: CONVERT: expected string not found in input
# CONVERT: Input file: {{.*\.yaml\.tmp}}
           ^
<stdin>:1:1: note: scanning from here
Uninitialized bytes in __interceptor_write at offset 24 inside [0x720000003000, 395)
^
<stdin>:18:61: note: possible intended match here
 #15 0x7fcccffc82e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)
                                                            ^

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
FAIL: LLVM :: tools/llvm-gsymutil/ARM_AArch64/fat-macho-dwarf.yaml (35814 of 36342)
******************** TEST 'LLVM :: tools/llvm-gsymutil/ARM_AArch64/fat-macho-dwarf.yaml' FAILED ********************
Script:
--
: 'RUN: at line 9';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/yaml2obj /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/ARM_AArch64/fat-macho-dwarf.yaml -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/ARM_AArch64/Output/fat-macho-dwarf.yaml.tmp
: 'RUN: at line 10';   llvm-gsymutil --convert /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/ARM_AArch64/Output/fat-macho-dwarf.yaml.tmp --arch armv7 -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/ARM_AArch64/Output/fat-macho-dwarf.yaml.tmp.armv7.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/ARM_AArch64/fat-macho-dwarf.yaml --check-prefix=ARMV7
: 'RUN: at line 11';   llvm-gsymutil --convert /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/ARM_AArch64/Output/fat-macho-dwarf.yaml.tmp --arch arm64 -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/ARM_AArch64/Output/fat-macho-dwarf.yaml.tmp.arm64.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/ARM_AArch64/fat-macho-dwarf.yaml --check-prefix=ARM64
: 'RUN: at line 12';   llvm-gsymutil --convert /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/ARM_AArch64/Output/fat-macho-dwarf.yaml.tmp -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/ARM_AArch64/Output/fat-macho-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/ARM_AArch64/fat-macho-dwarf.yaml --check-prefix=FAT
--
Exit Code: 1

Command Output (stderr):
--
/b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/ARM_AArch64/fat-macho-dwarf.yaml:14:10: error: ARMV7: expected string not found in input
# ARMV7: Input file: {{.*\.yaml\.tmp}}
         ^
<stdin>:1:1: note: scanning from here
Uninitialized bytes in __interceptor_write at offset 24 inside [0x720000002000, 118)
^
<stdin>:18:61: note: possible intended match here
 #15 0x7f561e2c92e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)
                                                            ^

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
FAIL: LLVM :: tools/llvm-gsymutil/X86/mach-dwarf.yaml (35815 of 36342)
******************** TEST 'LLVM :: tools/llvm-gsymutil/X86/mach-dwarf.yaml' FAILED ********************
Script:
--
: 'RUN: at line 5';   /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/yaml2obj /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/mach-dwarf.yaml -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/mach-dwarf.yaml.tmp
: 'RUN: at line 6';   llvm-gsymutil --convert /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/mach-dwarf.yaml.tmp -o /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/mach-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/mach-dwarf.yaml --check-prefix=CONVERT
: 'RUN: at line 7';   llvm-gsymutil --address=0 --address=0x100000000 --address=0x100000f90 --address=0x100000faa --address=0x200000000 /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/mach-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/mach-dwarf.yaml --check-prefix=ADDR
: 'RUN: at line 8';   llvm-gsymutil --verbose --address=0x100000000 --address=0x100000f90 --address=0x100000faa /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/mach-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/mach-dwarf.yaml --check-prefix=ADDRV
: 'RUN: at line 9';   llvm-gsymutil /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/test/tools/llvm-gsymutil/X86/Output/mach-dwarf.yaml.tmp.gsym 2>&1 | /b/sanitizer-x86_64-linux-fast/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/mach-dwarf.yaml --check-prefix=DUMP
--
Exit Code: 1

Command Output (stderr):
--
/b/sanitizer-x86_64-linux-fast/build/llvm-project/llvm/test/tools/llvm-gsymutil/X86/mach-dwarf.yaml:11:12: error: CONVERT: expected string not found in input
# CONVERT: Input file: {{.*\.yaml\.tmp}}
           ^
<stdin>:1:1: note: scanning from here
Uninitialized bytes in __interceptor_write at offset 24 inside [0x720000003000, 130)
^
<stdin>:18:61: note: possible intended match here
 #15 0x7fc3869db2e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)
                                                            ^

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 

Testing Time: 124.12s
********************
Failing Tests (3):
    LLVM :: tools/llvm-gsymutil/ARM_AArch64/fat-macho-dwarf.yaml
    LLVM :: tools/llvm-gsymutil/X86/elf-dwarf.yaml
    LLVM :: tools/llvm-gsymutil/X86/mach-dwarf.yaml

[clayborg]

I just need to be able to build my binary with -fsanitize=memory. Most clangs that we have installed do not support this. Is our solution to this really to create your own buildbot and run the test just like the buildbot would? Is there a toolchain I can download from somewhere? This really makes it hard to fix issues that come up.

[clayborg]

The full report you attached was enough to see what was going on. Is there a way we can enable always reporting the full output with FileCheck on the msan bots? That would help to see the full report, as we just got the first few lines in the buildbot error message.

[clayborg]

Recommitted with

commit ffe6695acf1f011a2acda31624beee826fb68956 (HEAD -> master, origin/master, origin/HEAD)
Author: Greg Clayton <[e-mail]>
Date: Wed Mar 4 19:28:24 2020 -0800

Fix buildbots with merge that didn't happen for 4050b01ba9ece02721ec496383baee219ca8cc2b.

commit 4050b01ba9ece02721ec496383baee219ca8cc2b
Author: Greg Clayton <[e-mail]>
Date: Wed Mar 4 19:12:08 2020 -0800

Fix GSYM tests to run the yaml files and fix test failures on some machines.

YAML files were not being run during lit testing as there was no lit.local.cfg file. Once this was fixed, some buildbots would fail due to a StringRef that pointed to a std::string inside of a temporary llvm::Triple object. These issues are fixed here by making a local triple object that stays around long enough so the StringRef points to valid data. Fixed memory sanitizer bot bugs as well.

Differential Revision: https://reviews.llvm.org/D75390

[hctim]

In D75390#1906642, @clayborg wrote:

I just need to be able to build my binary with -fsanitize=memory. Most clangs that we have installed do not support this. Is our solution to this really to create your own buildbot and run the test just like the buildbot would? Is there a toolchain I can download from somewhere? This really makes it hard to fix issues that come up.

Building the binary with -fsanitize=memory is basically it, yes. The tricky thing about MSan is that it requires an-MSan built libcxx/libcxxabi, which which isn't trivial to bootstrap for some configurations, hence I generally just point to the "how to repro exactly the bot". :)

In D75390#1906732, @clayborg wrote:

The full report you attached was enough to see what was going on. Is there a way we can enable always reporting the full output with FileCheck on the msan bots? That would help to see the full report, as we just got the first few lines in the buildbot error message.

I can take a look at this, but last time I looked into it there was a FileCheck limitation.

[clayborg]

In D75390#1907713, @hctim wrote:

In D75390#1906642, @clayborg wrote:

I just need to be able to build my binary with -fsanitize=memory. Most clangs that we have installed do not support this. Is our solution to this really to create your own buildbot and run the test just like the buildbot would? Is there a toolchain I can download from somewhere? This really makes it hard to fix issues that come up.

Building the binary with -fsanitize=memory is basically it, yes. The tricky thing about MSan is that it requires an-MSan built libcxx/libcxxabi, which which isn't trivial to bootstrap for some configurations, hence I generally just point to the "how to repro exactly the bot". :)

It would be great if we maybe built some toolchains to popular targets (linux and Darwin) so people to "sudo yum install clang-msan" or "brew install clang-msan" to allow debugging of these issues.

In D75390#1906732, @clayborg wrote:

The full report you attached was enough to see what was going on. Is there a way we can enable always reporting the full output with FileCheck on the msan bots? That would help to see the full report, as we just got the first few lines in the buildbot error message.

I can take a look at this, but last time I looked into it there was a FileCheck limitation.

All it would take is to set the default for --dump-input to be "fail" by default for memory sanitizer:

--dump-input=fail

After looking at this bug further, I am wondering if the memory sanitizer should have triggered in my case. I had a structure where I didn't init all of the values in one function, and then in another function I wrote a value to this location and this is what triggered the memory sanitizer? Pseudo code:

void write32(void *ptr, uint32_t value) {
   *(uint32_t *)ptr = value; // MSAN triggered here
}
struct Header = { int a, b; };
Header H;
H.a = 0;
write32(&H.b, 12);

And this code caused msan to trigger? I would expect read from uninitialized memory to trigger, but writes?

[eugenis]

In D75390#1908166, @clayborg wrote:

In D75390#1907713, @hctim wrote:

In D75390#1906642, @clayborg wrote:

I just need to be able to build my binary with -fsanitize=memory. Most clangs that we have installed do not support this. Is our solution to this really to create your own buildbot and run the test just like the buildbot would? Is there a toolchain I can download from somewhere? This really makes it hard to fix issues that come up.

Building the binary with -fsanitize=memory is basically it, yes. The tricky thing about MSan is that it requires an-MSan built libcxx/libcxxabi, which which isn't trivial to bootstrap for some configurations, hence I generally just point to the "how to repro exactly the bot". :)

It would be great if we maybe built some toolchains to popular targets (linux and Darwin) so people to "sudo yum install clang-msan" or "brew install clang-msan" to allow debugging of these issues.

Yeah... simply adding msan-instrumented libcxx + libcxxabi to the default build of the toolchain and teaching clang driver to detect/link them with msan would solve ~90% of the cases.

In D75390#1906732, @clayborg wrote:

The full report you attached was enough to see what was going on. Is there a way we can enable always reporting the full output with FileCheck on the msan bots? That would help to see the full report, as we just got the first few lines in the buildbot error message.

I can take a look at this, but last time I looked into it there was a FileCheck limitation.

All it would take is to set the default for --dump-input to be "fail" by default for memory sanitizer:

--dump-input=fail

This sounds like a very good idea. We'd want this for ASan and other tools that may add interesting information to the binary output, as well.

After looking at this bug further, I am wondering if the memory sanitizer should have triggered in my case. I had a structure where I didn't init all of the values in one function, and then in another function I wrote a value to this location and this is what triggered the memory sanitizer? Pseudo code:

void write32(void *ptr, uint32_t value) {
   *(uint32_t *)ptr = value; // MSAN triggered here
}
struct Header = { int a, b; };
Header H;
H.a = 0;
write32(&H.b, 12);

And this code caused msan to trigger? I would expect read from uninitialized memory to trigger, but writes?

No, writes should not trigger msan. Unless it's a bitfield (then that would be read-modify-write).
Perhaps ptr or value are *derived* from a something that was read from uninitialized memory?

[eugenis]

Oh, is is possible that you've misunderstood the wording of the report?

In this report:

MemorySanitizer: use-of-uninitialized-value
...
  Uninitialized value was stored to memory at
...
  Uninitialized value was stored to memory at
...
  Uninitialized value was created by an allocation of 'Hdr' in the stack frame of function '_ZNK4llvm4gsym11GsymCreator6encodeERNS0_10FileWriterE'

only the first stack trace is the error. The rest ("stored to memory at") explain how the uninitialized data reached that code, and should be read from the bottom:

• allocated as "Hdr" on stack
• then a value derived from the uninitialized contents of Hdr (possibly a straight copy) was written to memory
• then a value derived from that value was written to memory
• then a value derived from the previous value was used as a side-effect

llvm/lib/DebugInfo/GSYM/GsymCreator.cpp

 GsymCreator::GsymCreator() : StrTab(StringTableBuilder::ELF) {
   insertFile(StringRef());
 }
 
 uint32_t GsymCreator::insertFile(StringRef Path,
                                  llvm::sys::path::Style Style) {
   llvm::StringRef directory = llvm::sys::path::parent_path(Path, Style);
   llvm::StringRef filename = llvm::sys::path::filename(Path, Style);
-  FileEntry FE(insertString(directory), insertString(filename));
+  // We must insert the strings first, then call the FileEntry constructor.
+  // If we inline the insertString() function call into the constructor, the
+  // call order is undefined due to parameter lists not having any ordering
+  // requirements.
+  const uint32_t Dir = insertString(directory);
+  const uint32_t Base = insertString(filename);
+  FileEntry FE(Dir, Base);
 
   std::lock_guard<std::recursive_mutex> Guard(Mutex);
   const auto NextIndex = Files.size();
   // Find FE in hash map and insert if not present.
   auto R = FileEntryToIndex.insert(std::make_pair(FE, NextIndex));
   if (R.second)
     Files.emplace_back(FE);
   return R.first->second;

llvm/test/tools/llvm-gsymutil/ARM_AArch64/lit.local.cfg

+if not ('ARM' in config.root.targets and 'AArch64' in config.root.targets):
+    config.unsupported = True
+
+config.suffixes = ['.test', '.yaml']

llvm/test/tools/llvm-gsymutil/X86/lit.local.cfg

+if not 'X86' in config.root.targets:
+    config.unsupported = True
+
+config.suffixes = ['.test', '.yaml']

llvm/tools/llvm-gsym/llvm-gsymutil.cpp

else
     return MachO.getHeader().cputype;
 }
 
 /// Return true if the object file has not been filtered by an --arch option.
 static bool filterArch(MachOObjectFile &Obj) {
   if (ArchFilters.empty())
     return true;
 
-  StringRef ObjArch = Obj.getArchTriple().getArchName();
+  Triple ObjTriple(Obj.getArchTriple());
+  StringRef ObjArch = ObjTriple.getArchName();
 
   for (auto Arch : ArchFilters) {
     // Match name.
     if (Arch == ObjArch)
       return true;
 
     // Match architecture number.
     unsigned Value;

 }
 
 static llvm::Error handleBuffer(StringRef Filename, MemoryBufferRef Buffer,
                                 const std::string &OutFile) {
   Expected<std::unique_ptr<Binary>> BinOrErr = object::createBinary(Buffer);
   error(Filename, errorToErrorCode(BinOrErr.takeError()));
 
   if (auto *Obj = dyn_cast<ObjectFile>(BinOrErr->get())) {
-    auto ArchName = Obj->makeTriple().getArchName();
+    Triple ObjTriple(Obj->makeTriple());
+    auto ArchName = ObjTriple.getArchName();
     outs() << "Output file (" << ArchName << "): " << OutFile << "\n";
     if (auto Err = handleObjectFile(*Obj, OutFile.c_str()))
       return Err;
   } else if (auto *Fat = dyn_cast<MachOUniversalBinary>(BinOrErr->get())) {
     // Iterate over all contained architectures and filter out any that were
     // not specified with the "--arch <arch>" option. If the --arch option was
     // not specified on the command line, we will process all architectures.
     std::vector< std::unique_ptr<MachOObjectFile> > FilterObjs;
     for (auto &ObjForArch : Fat->objects()) {
       if (auto MachOOrErr = ObjForArch.getAsObjectFile()) {
         auto &Obj = **MachOOrErr;
         if (filterArch(Obj))
           FilterObjs.emplace_back(MachOOrErr->release());
       } else {
         error(Filename, MachOOrErr.takeError());
       }
     }
     if (FilterObjs.empty())
       error(Filename, createStringError(std::errc::invalid_argument,
                                         "no matching architectures found"));
 
     // Now handle each architecture we need to convert.
     for (auto &Obj: FilterObjs) {
-      auto ArchName = Obj->getArchTriple().getArchName();
+      Triple ObjTriple(Obj->getArchTriple());
+      auto ArchName = ObjTriple.getArchName();
       std::string ArchOutFile(OutFile);
       // If we are only handling a single architecture, then we will use the
       // normal output file. If we are handling multiple architectures append
       // the architecture name to the end of the out file path so that we
       // don't overwrite the previous architecture's gsym file.
       if (FilterObjs.size() > 1) {
         ArchOutFile.append(1, '.');
         ArchOutFile.append(ArchName.str());