[Proof of concept] Add an opt-in way to statically link a hermetic libxml2
Needs ReviewPublic
Actions

Authored by thakis on Jul 13 2022, 9:44 AM.

Details

Reviewers

Summary

When LLVM_ENABLE_LIBXML2 is set to HERMETIC, we now download libxml2
at cmake time, build it at build time, and statically link to it
in llvm-mt.

This reverts https://reviews.llvm.org/D129571 and uses
LLVM_ENABLE_LIBXML2=HERMETIC instead for the Windows installer.

This works for llvm-mt, but c-index-test and lldb also pick up the hermetic libxml2, and since the built libxml2 is so minimal, fail to link.

But even if that worked, I'm not sure if this is worth it. It looks kind of janky: The external project build step runs on every build; every target depending on libxml needs some extra code to force the install step to run, cmake is a bit temperamental about this, and so on. Maybe this isn't all that much easier than what D129571 did.

Or maybe there's a better way for doing this that I can't see.

Maybe this should be WindowsManifest-specific and WindowsManifest should optionally depend on the hermetic stuff, and HermeticLibXML2.cmake shouldn't try to pretend its find_package()-compatible. Then lldb and c-index-test wouldn't pick up the hermetic lib.

I've spent more time on this than I should've, so uploading as-is. Maybe someone has an opinion on this.

Diff Detail

Unit TestsFailed
View All

	Time	Test
	60,040 ms	x64 debian > MLIR.Examples/standalone::test.toy Script: -- : 'RUN: at line 1'; /usr/bin/cmake /var/lib/buildkite-agent/builds/llvm-project/mlir/examples/standalone -G "Ninja" -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DCMAKE_C_COMPILER=/usr/bin/clang -DLLVM_ENABLE_LIBCXX=OFF -DMLIR_DIR=/var/lib/buildkite-agent/builds/llvm-project/build/lib/cmake/mlir -DLLVM_USE_LINKER=lld
	60,040 ms	x64 debian > ThreadSanitizer-x86_64.ThreadSanitizer-x86_64::restore_stack.cpp Script: -- : 'RUN: at line 1'; /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang --driver-mode=g++ -fsanitize=thread -Wall -m64 -msse4.2 -gline-tables-only -I/var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/tsan/../ -std=c++11 -I/var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/tsan/../ -O1 /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/tsan/restore_stack.cpp -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/tsan/X86_64Config/Output/restore_stack.cpp.tmp && not /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/tsan/X86_64Config/Output/restore_stack.cpp.tmp 2>&1 \| FileCheck /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/tsan/restore_stack.cpp
	60,050 ms	x64 debian > libFuzzer.libFuzzer::fuzzer-leak.test Script: -- : 'RUN: at line 3'; /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang --driver-mode=g++ -O2 -gline-tables-only -fsanitize=address,fuzzer -I/var/lib/buildkite-agent/builds/llvm-project/compiler-rt/lib/fuzzer -m64 /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/fuzzer/LeakTest.cpp -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/fuzzer/X86_64DefaultLinuxConfig/Output/fuzzer-leak.test.tmp-LeakTest
	60,040 ms	x64 debian > libFuzzer.libFuzzer::minimize_crash.test Script: -- : 'RUN: at line 1'; /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang --driver-mode=g++ -O2 -gline-tables-only -fsanitize=address,fuzzer -I/var/lib/buildkite-agent/builds/llvm-project/compiler-rt/lib/fuzzer -m64 /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/fuzzer/NullDerefTest.cpp -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/fuzzer/X86_64DefaultLinuxConfig/Output/minimize_crash.test.tmp-NullDerefTest
	60,020 ms	x64 debian > libFuzzer.libFuzzer::out-of-process-fuzz.test Script: -- : 'RUN: at line 2'; rm -rf /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/fuzzer/X86_64DefaultLinuxConfig/Output/out-of-process-fuzz.test.tmp && mkdir /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/fuzzer/X86_64DefaultLinuxConfig/Output/out-of-process-fuzz.test.tmp
		View Full Test Results (6 Failed)

Event Timeline

thakis created this revision.Jul 13 2022, 9:44 AM

Herald added a project: Restricted Project. · View Herald TranscriptJul 13 2022, 9:44 AM

Herald added subscribers: hiraditya, mgorny. · View Herald Transcript

thakis requested review of this revision.Jul 13 2022, 9:44 AM

Herald added a project: Restricted Project. · View Herald TranscriptJul 13 2022, 9:44 AM

Harbormaster completed remote builds in B175158: Diff 444310.Jul 13 2022, 12:01 PM

It certainly simplifies build_llvm_release.bat a lot :-)

nit: Calling this hermetic seems counter-intuitive here, since it's actually pulling in stuff from way outside the build -- the opposite of hermetic really. Maybe the flag value should be "DOWNLOAD" or something like that?

But yeah, also not sure whether this is worth it. I kind of wish llvm-mt didn't have this dependency and could use something simpler, but I don't know whether that's realistic.