build: reduce CMake handling for zlib

[compnerd]

Summary

Rather than handling zlib handling manually, use find_package from CMake to find zlib properly. Use this to normalize the LLVM_ENABLE_ZLIB, HAVE_ZLIB, HAVE_ZLIB_H. Furthermore, require zlib if LLVM_ENABLE_ZLIB is set to YES, which requires the distributor to explicitly select whether zlib is enabled or not. This simplifies the CMake handling and usage in the rest of the tooling.

[labath]

Please take a look at D70519 for the issues with this approach.

Also, while I do agree with you that we should not auto-select dependencies, I think this runs contrary to the llvm philosophy that the default built should "just work" (I don't know if that's formalized anywhere, but I know it's certainly what (some?) people expect).

[compnerd]

@labath I think you are misunderstanding the patch. This is not autoselecting the dependencies. It is simply doing that based on an existing option that we have - LLVM_ENABLE_ZLIB. We could always search for zlib and override the results with LLVM_ENABLE_ZLIB as well. The current build will continue to just work - zlib is used only for the compressed debug sections (which requires the user to opt-in).

[JDevlieghere]

Having one canonical variable controlling zlib support seems indeed desirable.

In D70519#1754618, @labath wrote:

With this patch, what is the output of llvm-config --system-libs ?

@compnerd What's the answer to this for this patch?

[compnerd]

In D70764#1767395, @JDevlieghere wrote:

Having one canonical variable controlling zlib support seems indeed desirable.

In D70519#1754618, @labath wrote:

With this patch, what is the output of llvm-config --system-libs ?

@compnerd What's the answer to this for this patch?

Sorry, didn't see the question. From my local build:

 $ ./bin/llvm-config --system-libs
/usr/lib/x86_64-linux-gnu/libz.so -lrt -ldl -ltinfo -lpthread -lm /usr/lib/x86_64-linux-gnu/libxml2.so

[JDevlieghere]

In D70764#1767413, @compnerd wrote:

In D70764#1767395, @JDevlieghere wrote:

Having one canonical variable controlling zlib support seems indeed desirable.

In D70519#1754618, @labath wrote:

With this patch, what is the output of llvm-config --system-libs ?

@compnerd What's the answer to this for this patch?

Sorry, didn't see the question. From my local build:

 $ ./bin/llvm-config --system-libs
/usr/lib/x86_64-linux-gnu/libz.so -lrt -ldl -ltinfo -lpthread -lm /usr/lib/x86_64-linux-gnu/libxml2.so

So that's the issue Pavel raised in the other review. Can we make this return -lz with the current approach?

[labath]

/usr/lib/x86_64-linux-gnu/libz.so is definitely better than -l/usr/lib/x86_64-linux-gnu/libz.so, as it's at least a valid link line. It's not completely equivalent, and may not work in all cases -- last time I accidentally changed this, I got about a dozen emails from people I broke, and I wouldn't be surprised if one of them depended on the subtle differences here. However, the fact that libxml is also spelled out this way gives me some hope...

That said, this still leaves us with the "problem" that the default cmake configuration will be broken for people who don't have zlib installed (pretty much everyone on windows, at least). You can try that out by deleting/renaming/uninstalling zlib.h from your system. With this patch you get a configuration error whereas previously zlib support was automatically disabled. Whether this "problem" is a problem or "working as intended" depends on what do you expect out of your build system, but it's definitely not consistent with how other llvm dependencies are managed.

Anyway, I don't really have a horse in this race, but I figured I should at least warn you about the problems you're likely to run into. (and I really wish @beanz would say something here, as he's the one who's complaining about build system inconsistencies...)

[beanz]

@labath sorry for not replying on-list. I've actually been discussing offline with @compnerd. If llvm-config is printing an absolute path, I'm concerned about how that will interact with binary package distributions for the llvm-dev packages. Not sure if @tstellar has any thoughts on that.

[beanz]

I conferred off-list with @compnerd. He and I talked through my concerns, and I believe this patch is fine as-is, so LGTM.

[beanz]

I think this is good, and it has been sitting a while.

[compnerd]

GIT 68a235d07f9

[raj.khem]

this is now in master, and I am seeing build failures in cross-building clang, e.g. when building clang for arm on a x86_64 host. its resorting to finding, libz from buildhost instead of target sysroot ( using --sysroot) and failing in link step. e.g.

FAILED: bin/llvm-config
...
.... -o bin/llvm-config -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a /usr/lib/libz.so -lrt -ldl -ltinfo -lm lib/libLLVMDemangle.a
...

aarch64-yoe-linux-musl/aarch64-yoe-linux-musl-ld: /usr/lib/libz.so: error adding symbols: file in wrong format
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)

you can see that its adding /usr/lib/libz.so to linker cmdline while cross linking.

[smeenai]

In D70764#1803559, @raj.khem wrote:

this is now in master, and I am seeing build failures in cross-building clang, e.g. when building clang for arm on a x86_64 host. its resorting to finding, libz from buildhost instead of target sysroot ( using --sysroot) and failing in link step. e.g.

FAILED: bin/llvm-config
...
.... -o bin/llvm-config -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a /usr/lib/libz.so -lrt -ldl -ltinfo -lm lib/libLLVMDemangle.a
...

aarch64-yoe-linux-musl/aarch64-yoe-linux-musl-ld: /usr/lib/libz.so: error adding symbols: file in wrong format
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)

you can see that its adding /usr/lib/libz.so to linker cmdline while cross linking.

Have you set CMAKE_SYSROOT appropriately?

[raj.khem]

In D70764#1807405, @smeenai wrote:

In D70764#1803559, @raj.khem wrote:

this is now in master, and I am seeing build failures in cross-building clang, e.g. when building clang for arm on a x86_64 host. its resorting to finding, libz from buildhost instead of target sysroot ( using --sysroot) and failing in link step. e.g.

FAILED: bin/llvm-config
...
.... -o bin/llvm-config -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a /usr/lib/libz.so -lrt -ldl -ltinfo -lm lib/libLLVMDemangle.a
...

aarch64-yoe-linux-musl/aarch64-yoe-linux-musl-ld: /usr/lib/libz.so: error adding symbols: file in wrong format
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)

you can see that its adding /usr/lib/libz.so to linker cmdline while cross linking.

Have you set CMAKE_SYSROOT appropriately?

That is not the problem. Since in cases of target its finding is properly.
there are multiple pieces where some are being built for native( build host) some for target so it thinks its building for
buildhost (native) llvm-config e.g. is one such case. Replacing it with -lz fixes it becasue compile environment is providing
right --sysroot for native or cross cases and it always links with correct libs.

[hans]

There were some concerns raised on this patch, and also in PR44780.

I think at this point it's safer to revert these changes and start over. I've pushed the revert in 916be8fd6a0a0feea4cefcbeb0c22c65848d7a2e and will merge that to 10.x after it's baked a little.

clang/test/CMakeLists.txt

 
 string(REPLACE ${CMAKE_CFG_INTDIR} ${LLVM_BUILD_MODE} CLANG_TOOLS_DIR ${LLVM_RUNTIME_OUTPUT_INTDIR})
 
-if(CLANG_BUILT_STANDALONE)
-  # Set HAVE_LIBZ according to recorded LLVM_ENABLE_ZLIB value. This
-  # value is forced to 0 if zlib was not found, so it is fine to use it
-  # instead of HAVE_LIBZ (not recorded).
-  if(LLVM_ENABLE_ZLIB)
-    set(HAVE_LIBZ 1)
-  endif()
-endif()
-
 llvm_canonicalize_cmake_booleans(
   CLANG_BUILD_EXAMPLES
   CLANG_ENABLE_ARCMT
   CLANG_ENABLE_STATIC_ANALYZER
   ENABLE_BACKTRACES
   ENABLE_EXPERIMENTAL_NEW_PASS_MANAGER
-  HAVE_LIBZ
+  LLVM_ENABLE_ZLIB
   LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
   LLVM_ENABLE_PLUGINS
   LLVM_ENABLE_THREADS)

clang/test/lit.site.cfg.py.in

 config.target_triple = "@TARGET_TRIPLE@"
 config.host_cxx = "@CMAKE_CXX_COMPILER@"
 config.llvm_use_sanitizer = "@LLVM_USE_SANITIZER@"
-config.have_zlib = @HAVE_LIBZ@
+config.have_zlib = @LLVM_ENABLE_ZLIB@
 config.clang_arcmt = @CLANG_ENABLE_ARCMT@
 config.clang_default_cxx_stdlib = "@CLANG_DEFAULT_CXX_STDLIB@"
 config.clang_staticanalyzer = @CLANG_ENABLE_STATIC_ANALYZER@

compiler-rt/test/lit.common.configured.in

 else:
   set_default("target_suffix", "-%s" % config.target_arch)
 
-set_default("have_zlib", "@HAVE_LIBZ@")
+set_default("have_zlib", "@LLVM_ENABLE_ZLIB@")
 set_default("libcxx_used", "@LLVM_LIBCXX_USED@")
 
 # LLVM tools dir can be passed in lit parameters, so try to

lld/test/CMakeLists.txt

 set(LLVM_TOOLS_DIR "${LLVM_TOOLS_BINARY_DIR}/%(build_config)s")
 set(LLVM_LIBS_DIR "${LLVM_BINARY_DIR}/lib${LLVM_LIBDIR_SUFFIX}/%(build_config)s")
 
-if(LLD_BUILT_STANDALONE)
-  # Set HAVE_LIBZ according to recorded LLVM_ENABLE_ZLIB value. This
-  # value is forced to 0 if zlib was not found, so it is fine to use it
-  # instead of HAVE_LIBZ (not recorded).
-  if(LLVM_ENABLE_ZLIB)
-    set(HAVE_LIBZ 1)
-  endif()
-endif()
-
 llvm_canonicalize_cmake_booleans(
-  HAVE_LIBZ
+  LLVM_ENABLE_ZLIB
   LLVM_LIBXML2_ENABLED
   )
 

lld/test/lit.site.cfg.py.in

 config.lld_tools_dir = "@LLVM_RUNTIME_OUTPUT_INTDIR@"
 config.target_triple = "@TARGET_TRIPLE@"
 config.python_executable = "@PYTHON_EXECUTABLE@"
-config.have_zlib = @HAVE_LIBZ@
+config.have_zlib = @LLVM_ENABLE_ZLIB@
 config.sizeof_void_p = @CMAKE_SIZEOF_VOID_P@
 
 # Support substitution of the tools and libs dirs with user parameters. This is

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp

 #include <compression.h>
 #endif
 
-#if defined(HAVE_LIBZ)
+#if LLVM_ENABLE_ZLIB
 #include <zlib.h>
 #endif
 

   }
 #endif
 
-#if defined(HAVE_LIBZ)
+#if LLVM_ENABLE_ZLIB
   if (decompressed_bytes == 0 && decompressed_bufsize != ULONG_MAX &&
       decompressed_buffer != nullptr &&
       m_compression_type == CompressionType::ZlibDeflate) {

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

   }
 #endif
 
-#if defined(HAVE_LIBZ)
+#if LLVM_ENABLE_ZLIB
   if (avail_type == CompressionType::None) {
     for (auto compression : supported_compressions) {
       if (compression == "zlib-deflate") {

llvm/cmake/config-ix.cmake

 check_include_file(termios.h HAVE_TERMIOS_H)
 check_include_file(unistd.h HAVE_UNISTD_H)
 check_include_file(valgrind/valgrind.h HAVE_VALGRIND_VALGRIND_H)
-check_include_file(zlib.h HAVE_ZLIB_H)
 check_include_file(fenv.h HAVE_FENV_H)
 check_symbol_exists(FE_ALL_EXCEPT "fenv.h" HAVE_DECL_FE_ALL_EXCEPT)
 check_symbol_exists(FE_INEXACT "fenv.h" HAVE_DECL_FE_INEXACT)

 # Don't look for these libraries if we're using MSan, since uninstrumented third
 # party code may call MSan interceptors like strlen, leading to false positives.
 if(NOT LLVM_USE_SANITIZER MATCHES "Memory.*")
-  set(HAVE_LIBZ 0)
   if(LLVM_ENABLE_ZLIB)
-    foreach(library z zlib_static zlib)
-      string(TOUPPER ${library} library_suffix)
-      check_library_exists(${library} compress2 "" HAVE_LIBZ_${library_suffix})
-      if(HAVE_LIBZ_${library_suffix})
-        set(HAVE_LIBZ 1)
-        set(ZLIB_LIBRARIES "${library}")
-        break()
-      endif()
-    endforeach()
+    find_package(ZLIB REQUIRED)
   endif()
 
   # Don't look for these libraries on Windows.

   message(STATUS "Threads disabled.")
 endif()
 
-if (LLVM_ENABLE_ZLIB )
-  # Check if zlib is available in the system.
-  if ( NOT HAVE_ZLIB_H OR NOT HAVE_LIBZ )
-    set(LLVM_ENABLE_ZLIB 0)
-  endif()
+if(LLVM_ENABLE_ZLIB)
+  find_package(ZLIB REQUIRED)
 endif()
 
 if (LLVM_ENABLE_DOXYGEN)

llvm/include/llvm/Config/config.h.cmake

 /* Define to 1 if you have the `pthread_setname_np' function. */
 #cmakedefine HAVE_PTHREAD_SETNAME_NP ${HAVE_PTHREAD_SETNAME_NP}
 
-/* Define to 1 if you have the `z' library (-lz). */
-#cmakedefine HAVE_LIBZ ${HAVE_LIBZ}
-
 /* Define to 1 if you have the <link.h> header file. */
 #cmakedefine HAVE_LINK_H ${HAVE_LINK_H}
 

 /* Define to 1 if you have the <valgrind/valgrind.h> header file. */
 #cmakedefine HAVE_VALGRIND_VALGRIND_H ${HAVE_VALGRIND_VALGRIND_H}
 
-/* Define to 1 if you have the <zlib.h> header file. */
-#cmakedefine HAVE_ZLIB_H ${HAVE_ZLIB_H}
-
 /* Have host's _alloca */
 #cmakedefine HAVE__ALLOCA ${HAVE__ALLOCA}
 

llvm/lib/Support/CMakeLists.txt

-set(system_libs)
-if ( LLVM_ENABLE_ZLIB AND HAVE_LIBZ )
-  set(system_libs ${system_libs} ${ZLIB_LIBRARIES})
-endif()
+set(system_libs ${ZLIB_LIBRARY})
 if( MSVC OR MINGW )
   # libuuid required for FOLDERID_Profile usage in lib/Support/Windows/Path.inc.
   # advapi32 required for CryptAcquireContextW in lib/Support/Windows/Path.inc.

llvm/lib/Support/CRC.cpp

 
 using namespace llvm;
 
-#if LLVM_ENABLE_ZLIB == 0 || !HAVE_ZLIB_H
+#if !LLVM_ENABLE_ZLIB
 
 static const uint32_t CRCTable[256] = {
     0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f,

llvm/lib/Support/Compression.cpp

 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
-#if LLVM_ENABLE_ZLIB == 1 && HAVE_ZLIB_H
+#if LLVM_ENABLE_ZLIB
 #include <zlib.h>
 #endif
 
 using namespace llvm;
 
-#if LLVM_ENABLE_ZLIB == 1 && HAVE_LIBZ
+#if LLVM_ENABLE_ZLIB
 static Error createError(StringRef Err) {
   return make_error<StringError>(Err, inconvertibleErrorCode());
 }

llvm/test/CMakeLists.txt

 llvm_canonicalize_cmake_booleans(
   BUILD_SHARED_LIBS
   HAVE_LIBXAR
-  HAVE_LIBZ
   HAVE_OCAMLOPT
   HAVE_OCAML_OUNIT
   LLVM_ENABLE_DIA_SDK
   LLVM_ENABLE_FFI
   LLVM_ENABLE_THREADS
+  LLVM_ENABLE_ZLIB
   LLVM_INCLUDE_GO_TESTS
   LLVM_LIBXML2_ENABLED
   LLVM_LINK_LLVM_DYLIB

llvm/test/lit.site.cfg.py.in

 config.host_ldflags = '@HOST_LDFLAGS@'
 config.llvm_use_intel_jitevents = @LLVM_USE_INTEL_JITEVENTS@
 config.llvm_use_sanitizer = "@LLVM_USE_SANITIZER@"
-config.have_zlib = @HAVE_LIBZ@
+config.have_zlib = @LLVM_ENABLE_ZLIB@
 config.have_libxar = @HAVE_LIBXAR@
 config.have_dia_sdk = @LLVM_ENABLE_DIA_SDK@
 config.enable_ffi = @LLVM_ENABLE_FFI@

llvm/unittests/Support/CompressionTest.cpp

 
 namespace {
 
-#if LLVM_ENABLE_ZLIB == 1 && HAVE_LIBZ
+#if LLVM_ENABLE_ZLIB
 
 void TestZlibCompression(StringRef Input, int Level) {
   SmallString<32> Compressed;