Avoid ar_to_bc.sh script in symbolizer build; add -ignore-non-bitcode flag to llvm-link

[MatzeB]

Summary

Remove the ar_to_bc.sh helper script in the compiler-rt symbolizer
build. Instead use llvm-link directly as D81109 introduced the
capability to read archives. I just needed to add a new flag
-ignore-non-bitcode to avoid llvm-link aborting when it found a
non-bitcode file in the archive.

Removing ar_to_bc.sh avoids build problems I had with inconsistent output between different versions of the file utility.

[vitalybuka]

Do we need a test for llvm-link?

[MatzeB]

In D128729#3615885, @vitalybuka wrote:

Do we need a test for llvm-link?

I guess. So much trouble to keep these strange build scripts working...

[MatzeB]

Added test for new llvm-link option.

[vitalybuka]

In D128729#3615964, @MatzeB wrote:

In D128729#3615885, @vitalybuka wrote:

Do we need a test for llvm-link?

I guess. So much trouble to keep these strange build scripts working...

It's not clear from description that it's done to fix those scripts.

Scripts probably can be converted into CMake, which will simplify reproducing build errors, but i guess a step like this will be needed anyway.

[MaskRay]

build_symbolizer.sh is used by our internal Bazel build system. (It will be needed when sb contributes utils/bazel/llvm-project-overlay/compiler-rt) In llvm-project it seems not referenced. Curious how you noticed this script...

[MatzeB]

It's not clear from description that it's done to fix those scripts.

I originally added a note to D126728. I added a sentence to the summary now.

In D128729#3616085, @MaskRay wrote:

build_symbolizer.sh is used by our internal Bazel build system. In llvm-project it seems not referenced. Curious how you noticed this script...

Our internal build system is somehow using it too... CC @sugak

[MatzeB]

Judging from reaction from folks to older diffs, I have a feeling the script may also be running within apple internal builds...

[sugak]

Curious how you noticed this script...

I stumbled on this while exploring the compiler-rt codebase. We've being using it for for over 5 years now! This way we didn't have to teach people how to switch from add2line to llvm-symbolizer and didn't need to deploy llvm-symbolizer anywhere.

compiler-rt/lib/sanitizer_common/symbolizer/scripts/ar_to_bc.sh

-#!/usr/bin/env bash
-
-function usage() {
-  echo "Usage: $0 INPUT... OUTPUT"
-  exit 1
-}
-
-if [ "$#" -le 1 ]; then
-  usage
-fi
-
-[[ $AR == /* ]] || AR=$PWD/$AR
-[[ $LINK == /* ]] || LINK=$PWD/$LINK
-
-INPUTS=
-OUTPUT=
-for ARG in $@; do
-  INPUTS="$INPUTS $OUTPUT"
-  OUTPUT=$(readlink -f $ARG)
-done
-
-echo Inputs: $INPUTS
-echo Output: $OUTPUT
-
-SCRATCH_DIR=$(mktemp -d)
-ln -s $INPUTS $SCRATCH_DIR/
-
-pushd $SCRATCH_DIR
-
-for INPUT in *; do
-  for OBJ in $($AR t $INPUT); do
-    $AR x $INPUT $OBJ
-    if [[ $(file $OBJ) =~ 'LLVM IR bitcode' ]]; then
-      mv -f $OBJ $(basename $INPUT).$OBJ
-    else
-      # Skip $OBJ which may come from an assembly file (e.g. Support/BLAKE3/*.S).
-      rm -f $OBJ
-    fi
-  done
-done
-
-$LINK *.o -o $OUTPUT
-
-rm -rf $SCRATCH_DIR

compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh

 BUILD_DIR=$(readlink -f ./symbolizer)
 mkdir -p $BUILD_DIR
 cd $BUILD_DIR
 
 CC=$CLANG_DIR/clang
 CXX=$CLANG_DIR/clang++
 TBLGEN=$CLANG_DIR/llvm-tblgen
 OPT=$CLANG_DIR/opt
-export AR=$CLANG_DIR/llvm-ar
-export LINK=$CLANG_DIR/llvm-link
+AR=$CLANG_DIR/llvm-ar
+LINK=$CLANG_DIR/llvm-link
 TARGET_TRIPLE=$($CC -print-target-triple)
 
 for F in $CC $CXX $TBLGEN $LINK $OPT $AR; do
   if [[ ! -x "$F" ]]; then
     echo "Missing $F"
      exit 1
   fi
 done

 SYMBOLIZER_API_LIST+=,__sanitizer_symbolize_flush
 SYMBOLIZER_API_LIST+=,__sanitizer_symbolize_demangle
 SYMBOLIZER_API_LIST+=,__sanitizer_symbolize_set_demangle
 SYMBOLIZER_API_LIST+=,__sanitizer_symbolize_set_inline_frames
 
 LIBCXX_ARCHIVE_DIR=$(dirname $(find $LIBCXX_BUILD -name libc++.a | head -n1))
 
 # Merge all the object files together and copy the resulting library back.
-$SCRIPT_DIR/ar_to_bc.sh $LIBCXX_ARCHIVE_DIR/libc++.a \
-                        $LIBCXX_ARCHIVE_DIR/libc++abi.a \
-                        $LLVM_BUILD/lib/libLLVMSymbolize.a \
-                        $LLVM_BUILD/lib/libLLVMObject.a \
-                        $LLVM_BUILD/lib/libLLVMBinaryFormat.a \
-                        $LLVM_BUILD/lib/libLLVMDebugInfoDWARF.a \
-                        $LLVM_BUILD/lib/libLLVMSupport.a \
-                        $LLVM_BUILD/lib/libLLVMDebugInfoPDB.a \
-                        $LLVM_BUILD/lib/libLLVMDebugInfoMSF.a \
-                        $LLVM_BUILD/lib/libLLVMDebugInfoCodeView.a \
-                        $LLVM_BUILD/lib/libLLVMDebuginfod.a \
-                        $LLVM_BUILD/lib/libLLVMDemangle.a \
-                        $LLVM_BUILD/lib/libLLVMMC.a \
-                        $LLVM_BUILD/lib/libLLVMTextAPI.a \
-                        $ZLIB_BUILD/libz.a \
-                        symbolizer.a \
-                        all.bc
+$LINK $LIBCXX_ARCHIVE_DIR/libc++.a \
+      -ignore-non-bitcode -o all.bc
 
 echo "Optimizing..."
 $OPT -internalize -internalize-public-api-list=${SYMBOLIZER_API_LIST} all.bc -o opt.bc
 $CC $FLAGS -fno-lto -c opt.bc -o symbolizer.o
 
 echo "Checking undefined symbols..."
 nm -f posix -g symbolizer.o | cut -f 1,2 -d \  | LC_COLLATE=C sort -u > undefined.new
 (diff -u $SCRIPT_DIR/global_symbols.txt undefined.new | grep -E "^\+[^+]") && \

llvm/test/tools/llvm-link/archive-non-bitcode.ll

+# RUN: llvm-as %S/Inputs/f.ll -o %t.f.bc
+# RUN: llvm-as %S/Inputs/g.ll -o %t.g.bc
+# RUN: echo "This is not a bitcode file" > %t.not_bitcode.txt
+# RUN: llvm-ar cr %t.a %t.f.bc %t.not_bitcode.txt %t.g.bc
+# RUN: llvm-ar cr --format=gnu %t.empty.lib
+# RUN: llvm-link -ignore-non-bitcode %t.a %t.empty.lib -o %t.linked.bc 2>&1 | FileCheck --check-prefix CHECK_IGNORE_NON_BITCODE %s
+# RUN: not llvm-link %t.a %t.empty.lib -o %t.linked2.bc 2>&1 | FileCheck --check-prefix CHECK_ERROR_BITCODE %s
+
+# CHECK_ERROR_BITCODE: error: member of archive is not a bitcode file
+# CHECK_IGNORE_NON_BITCODE-NOT: is not a bitcode file

llvm/tools/llvm-link/llvm-link.cpp

static cl::opt<bool> PreserveAssemblyUseListOrder(
     "preserve-ll-uselistorder",
     cl::desc("Preserve use-list order when writing LLVM assembly."),
     cl::init(false), cl::Hidden, cl::cat(LinkCategory));
 
 static cl::opt<bool> NoVerify("disable-verify",
                               cl::desc("Do not run the verifier"), cl::Hidden,
                               cl::cat(LinkCategory));
 
+static cl::opt<bool> IgnoreNonBitcode(
+    "ignore-non-bitcode",
+    cl::desc("Do not report an error for non-bitcode files in archives"),
+    cl::Hidden);
+
 static ExitOnError ExitOnErr;
 
 // Read the specified bitcode file in and return it. This routine searches the
 // link path for the specified file to try to find it...
 //
 static std::unique_ptr<Module> loadFile(const char *argv0,
                                         std::unique_ptr<MemoryBuffer> Buffer,
                                         LLVMContext &Context,
if (Error E = MemBuf.takeError()) {
                          << "'\n";
       return nullptr;
     };
 
     if (!isBitcode(reinterpret_cast<const unsigned char *>(
                        MemBuf.get().getBufferStart()),
                    reinterpret_cast<const unsigned char *>(
                        MemBuf.get().getBufferEnd()))) {
+      if (IgnoreNonBitcode)
+        continue;
       errs() << Argv0 << ": ";
       WithColor::error() << "  member of archive is not a bitcode file: '"
                          << ChildName << "'\n";
       return nullptr;
     }
 
     std::unique_ptr<Module> M;
     if (DisableLazyLoad)