[llvm] add zstd to `llvm::compression` namespace

[ckissane]

Summary

• add zstd to llvm::compression namespace
• add a CMake option LLVM_ENABLE_ZSTD with behavior mirroring that of LLVM_ENABLE_ZLIB
• add tests for zstd to llvm/unittests/Support/CompressionTest.cpp
• debian users should install libzstd when using LLVM_ENABLE_ZSTD=FORCE_ON from source due to this bug https://bugs.launchpad.net/ubuntu/+source/libzstd/+bug/1941956

[ckissane]

Fix missing cmake findZstd, remove unused crc32 from compression namespaces and simplify some code

[ckissane]

fix type for FindZstd cmake usage

[phosek]

I think this patch should be broken into at least two:

1. Refactor llvm/include/llvm/Support/Compression.h and llvm/lib/Support/Compression.cpp to introduce a generic interface and use it throughout the codebase.
2. zstd support in llvm/include/llvm/Support/Compression.h including the CMake bits.

When uploading future changes, please also make sure to include full context.

llvm/include/llvm/Support/Compression.h

67–87

I think think that this should be a runtime rather than compile time selection. Many (if not most) toolchain vendors would likely want to enable both zstd and zlib and choose the algorithm based an option or file content.

For example, in the case of profiles, we want to be able to read existing profiles that use zlib, but for newly created profiles we may want to use zstd if available.

To make it possible, I suspect we may want to wrap the existing functions into a class that implements a common interface. I expect we would also need a function to detect the compression type based on the buffer content.

[ckissane]

more context

[MaskRay]

In D128465#3612948, @phosek wrote:

I think this patch should be broken into at least two:

1. Refactor llvm/include/llvm/Support/Compression.h and llvm/lib/Support/Compression.cpp to introduce a generic interface and use it throughout the codebase.
2. zstd support in llvm/include/llvm/Support/Compression.h including the CMake bits.

When uploading future changes, please also make sure to include full context.

Agree. As soon as the namespace refactoring is in a good enough shape, I think you may land the refactoring part before the rest of zstd patches.

Note: if you have a deep stacked patches, it may be useful to have a branch somewhere (e.g. your llvm-project fork on Github) so that interested folks can get the whole picture more easily.
(arc patch Dxxxxx can technically apply a deep stack, but it often fails to apply changes cleanly.)

[ckissane]

format

[ckissane]

make it the second stage

[ckissane]

add AlgorithmName to compression namespaces

[ckissane]

• feat: add zstd compression
• Merge branch 'ckissane.refactor-compression-namespace' into ckissane.add-zstd-compression
• Merge branch 'ckissane.refactor-compression-namespace' into ckissane.add-zstd-compression

[MaskRay]

llvm/cmake/modules/FindZSTD.cmake

1 ↗

(On Diff #441114)

How did you derive this?

The file seems contributed by you (I don't think facebook/zstd has such a file). It should not have a Meta Platforms, Inc notice.

3 ↗

(On Diff #441114)

Search SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception from other .cmake files

[ckissane]

start breaking this up further, this is now the add cmake config patch

[ckissane]

mark outdated comments done

[leonardchan]

LGTM although I think I'd be more comfortable with Petr's +2 for cmake changes.

Would it be better instead to expose LLVM_ENABLE_ZSTD to subrepos like clangd or flang until they're needed there? Not sure in this case if this will need +2s from their respective owners.

[MaskRay]

Please check both LLVM_ENABLE_ZSTD={on,off} work. Ideally, check that when zstd' cmake (libzstd-dev on Debian) is unavailable, LLVM_ENABLE_ZSTD=on works (there is no zstd functionality but the cmake invocation should succeed).

llvm/cmake/modules/FindZSTD.cmake

3 ↗

(On Diff #441114)

Delete //

[MaskRay]

Consider adding have_zstd to compiler-rt/test/lit.common.cfg.py and several other lit.site.cfg.py.in files

[phosek]

llvm/cmake/modules/FindZSTD.cmake

1 ↗

(On Diff #441114)

We usually don't include the copyright header in .cmake files at all.

[ckissane]

• Merge remote-tracking branch 'origin/main' into ckissane.add-zstd.0-cmake
• added have_zstd to compiler-rt/test/lit.common.cfg.py, clang-tools-extra/clangd/test/lit.cfg.py and several lit.site.cfg.py.in files

[ckissane]

marked fixed comments as done

[ckissane]

• Merge remote-tracking branch 'origin/main' into ckissane.add-zstd.0-cmake

[ldionne]

llvm/CMakeLists.txt

441

This broke builds that do not specify LLVM_ENABLE_ZSTD=OFF explicitly on macOS, and I suspect on other platforms as well:

ld: library not found for -lzstd
clang: error: linker command failed with exit code 1 (use -v to see invocation)

It looks to me like the build should not require zstd to be installed by default.

[MaskRay]

lld/test/lit.site.cfg.py.in

21 ↗

(On Diff #443316)

This needs a change in lld/test/CMakeLists.txt

Looks like you haven't run ninja check-lld as I mentioned.

21 ↗

(On Diff #443316)

Actually I am preparing a lld patch and you probably should drop have_zstd from lld/ for this patch.

[MaskRay]

This patch has changed a lot from what I have reviewed. The CMake change should be added along with llvm::compression::zstd::* functions.
Otherwise the change just introduces some CMake variables which cannot be tested.

Since you haven't touched flang, compiler-rt, etc. The patch should not updated their CMake files.

For lld/ELF, I've created D129406 to add the support. It will need to wait until you have landed these zstd changes.

[ckissane]

clean a space and make accurate

[ckissane]

lld/test/lit.site.cfg.py.in

21 ↗

(On Diff #443316)

I get this when I try to run that:
ninja: error: unknown target 'check-lld', did you mean 'check-lit'?

21 ↗

(On Diff #443316)

Got it

[MaskRay]

As I mentioned, the proper approach is to add zstd functionality along with the CMake change, instead of adding CMake to all llvm-project components without a way to test them.

lld/test/lit.site.cfg.py.in

21 ↗

(On Diff #443316)

During cmake, ensure that LLVM_ENABLE_PROJECTS contains lld

I posted the patch link to another patch of yours: https://reviews.llvm.org/D129406

[ckissane]

• make sure LLVM_ENABLE_ZSTD OFF turns into 0
• add zstd compression namespace

[ckissane]

In D128465#3642997, @MaskRay wrote:

As I mentioned, the proper approach is to add zstd functionality along with the CMake change, instead of adding CMake to all llvm-project components without a way to test them.

@MaskRay, I have now done this and ran the ldd tests as requested:

With LLVM_ENABLE_ZSTD=ON
$ ninja check-lld
Testing Time: 8.98s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1
With LLVM_ENABLE_ZSTD=OFF
$ ninja check-lld
Testing Time: 8.95s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1

[MaskRay]

In D128465#3646203, @ckissane wrote:

In D128465#3642997, @MaskRay wrote:

As I mentioned, the proper approach is to add zstd functionality along with the CMake change, instead of adding CMake to all llvm-project components without a way to test them.

@MaskRay, I have now done this and ran the ldd tests as requested:

With LLVM_ENABLE_ZSTD=ON
$ ninja check-lld
Testing Time: 8.98s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1
With LLVM_ENABLE_ZSTD=OFF
$ ninja check-lld
Testing Time: 8.95s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1

I request that you abandon this patch and incorporate the llvm cmake part into the llvm patch which you actually use zstd.
It is not appropriate to add zstd cmake to llvm-project components which don't use zstd.

[ckissane]

• add zstd tests to CompressionTest.cpp

[ckissane]

In D128465#3646258, @MaskRay wrote:

In D128465#3646203, @ckissane wrote:

In D128465#3642997, @MaskRay wrote:

As I mentioned, the proper approach is to add zstd functionality along with the CMake change, instead of adding CMake to all llvm-project components without a way to test them.

@MaskRay, I have now done this and ran the ldd tests as requested:

With LLVM_ENABLE_ZSTD=ON
$ ninja check-lld
Testing Time: 8.98s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1
With LLVM_ENABLE_ZSTD=OFF
$ ninja check-lld
Testing Time: 8.95s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1

I request that you abandon this patch and incorporate the llvm cmake part into the llvm patch which you actually use zstd.
It is not appropriate to add zstd cmake to llvm-project components which don't use zstd.

Let me see if I understand this correctly:
Are you saying that I should abandon this patch, and create a new patch that:

• adds findZSTD.cmake
• adds zstd to compression namespace
• adds tests to CompressionTest.cpp
• and does the minimal amount of cmake work to have the flags and test work
  • thus differing per component cmake config to patches like how you are doing in https://reviews.llvm.org/D129406

Is this correct?

I realize though that then https://reviews.llvm.org/D129406 or similar would be "the [first] llvm patch which actually use[s] zstd"

[MaskRay]

In D128465#3646561, @ckissane wrote:

In D128465#3646258, @MaskRay wrote:

In D128465#3646203, @ckissane wrote:

In D128465#3642997, @MaskRay wrote:

As I mentioned, the proper approach is to add zstd functionality along with the CMake change, instead of adding CMake to all llvm-project components without a way to test them.

@MaskRay, I have now done this and ran the ldd tests as requested:

With LLVM_ENABLE_ZSTD=ON
$ ninja check-lld
Testing Time: 8.98s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1
With LLVM_ENABLE_ZSTD=OFF
$ ninja check-lld
Testing Time: 8.95s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1

I request that you abandon this patch and incorporate the llvm cmake part into the llvm patch which you actually use zstd.
It is not appropriate to add zstd cmake to llvm-project components which don't use zstd.

Let me see if I understand this correctly:
Are you saying that I should abandon this patch, and create a new patch that:

• adds findZSTD.cmake
• adds zstd to compression namespace
• adds tests to CompressionTest.cpp
• and does the minimal amount of cmake work to have the flags and test work
  • thus differing per component cmake config to patches like how you are doing in https://reviews.llvm.org/D129406

Is this correct?

I realize though that then https://reviews.llvm.org/D129406 or similar would be "the [first] llvm patch which actually use[s] zstd"

You can find a component in llvm which uses zstd (e.g. lib/Support). Add the functionality with tests (e.g. a CompressionTest.cpp unittest) along with the CMake/(optional Bazel, gn) changes.
Keep clang, clang-tools-extra, lld, etc unchanged. These components don't yet use zstd and the CMake changes in this patch will be untestable.

[ckissane]

In D128465#3646601, @MaskRay wrote:

In D128465#3646561, @ckissane wrote:

In D128465#3646258, @MaskRay wrote:

In D128465#3646203, @ckissane wrote:

In D128465#3642997, @MaskRay wrote:

As I mentioned, the proper approach is to add zstd functionality along with the CMake change, instead of adding CMake to all llvm-project components without a way to test them.

@MaskRay, I have now done this and ran the ldd tests as requested:

With LLVM_ENABLE_ZSTD=ON
$ ninja check-lld
Testing Time: 8.98s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1
With LLVM_ENABLE_ZSTD=OFF
$ ninja check-lld
Testing Time: 8.95s
  Unsupported      :   17
  Passed           : 2638
  Expectedly Failed:    1

I request that you abandon this patch and incorporate the llvm cmake part into the llvm patch which you actually use zstd.
It is not appropriate to add zstd cmake to llvm-project components which don't use zstd.

Let me see if I understand this correctly:
Are you saying that I should abandon this patch, and create a new patch that:

• adds findZSTD.cmake
• adds zstd to compression namespace
• adds tests to CompressionTest.cpp
• and does the minimal amount of cmake work to have the flags and test work
  • thus differing per component cmake config to patches like how you are doing in https://reviews.llvm.org/D129406

Is this correct?

I realize though that then https://reviews.llvm.org/D129406 or similar would be "the [first] llvm patch which actually use[s] zstd"

You can find a component in llvm which uses zstd (e.g. lib/Support). Add the functionality with tests (e.g. a CompressionTest.cpp unittest) along with the CMake/(optional Bazel, gn) changes.
Keep clang, clang-tools-extra, lld, etc unchanged. These components don't yet use zstd and the CMake changes in this patch will be untestable.

Got it!, this patch already has such tests (CompressionTest.cpp unittest) in lib/Support, so I should just remove the untestable cmake config.

[ckissane]

• Trim CMAKE modifications to minimum needed

[MaskRay]

The patch looks mostly good but please fix the subject and the summary.

Note: if you fix the local commit message, you can use arc diff --head=HEAD 'HEAD^' --verbatim to update the subject and the summary.

llvm/lib/Support/Compression.cpp

88

https://llvm.org/docs/CodingStandards.html#don-t-use-braces-on-simple-single-statement-bodies-of-if-else-loop-statements

ditto below

[MaskRay]

llvm/lib/Support/Compression.cpp

139

const size_t

llvm/unittests/Support/CompressionTest.cpp

75

delete blank line

76

static void testZstd...

79

delete blank line

105

https://llvm.org/docs/CodingStandards.html#don-t-use-braces-on-simple-single-statement-bodies-of-if-else-loop-statements

[ckissane]

tidy some code

[ckissane]

Mark handled comments from @MaskRay as done

[phosek]

llvm/lib/Support/Compression.cpp

68

I'd make changes to zlib separately.

88

Ditto here.

[ckissane]

llvm/lib/Support/Compression.cpp

68

Here https://reviews.llvm.org/D129698

[ckissane]

• Merge branch main

[MaskRay]

llvm/include/llvm/Support/Compression.h

55

I changed the zlib functions to use uint8_t * and ArrayRef<uint8_t> instead. This patch needs to switch to uint8_t too.

llvm/lib/Support/Compression.cpp

163

uint8_t

[ckissane]

• Merge remote-tracking branch 'origin/main' into ckissane.add-zstd
• [Support] update zstd interface to use uint8_t * and ArrayRef<uint8_t>

[MaskRay]

Why was this reverted? Please make extensive tests especially for something dealing with the build system. When reverting a commit, briefly describe what happened.

[sebastian-ne]

This breaks builds when LLVM is included with CMake’s add_subdirectory.
I think the zstd target needs to be marked as imported.
The following patch fixes the problem, although I’m not familiar enough with CMake to know if this is the right way to go:

diff --git a/llvm/cmake/modules/FindZSTD.cmake b/llvm/cmake/modules/FindZSTD.cmake
index 43ccf7232138..8e6ec6e8a988 100644
--- a/llvm/cmake/modules/FindZSTD.cmake
+++ b/llvm/cmake/modules/FindZSTD.cmake
@@ -12,6 +12,10 @@ find_package_handle_standard_args(
     ZSTD_LIBRARY ZSTD_INCLUDE_DIR)

 if(ZSTD_FOUND)
+    if(NOT TARGET Zstd::zstd)
+      add_library(Zstd::zstd UNKNOWN IMPORTED)
+      set_target_properties(Zstd::zstd PROPERTIES IMPORTED_LOCATION "${ZSTD_LIBRARY}")
+    endif()
     set(ZSTD_LIBRARIES ${ZSTD_LIBRARY})
     set(ZSTD_INCLUDE_DIRS ${ZSTD_INCLUDE_DIR})
 endif()
diff --git a/llvm/lib/Support/CMakeLists.txt b/llvm/lib/Support/CMakeLists.txt
index 52b95c5377d3..031adfa33ba8 100644
--- a/llvm/lib/Support/CMakeLists.txt
+++ b/llvm/lib/Support/CMakeLists.txt
@@ -26,7 +26,7 @@ if(LLVM_ENABLE_ZLIB)
 endif()

 if(LLVM_ENABLE_ZSTD)
-  list(APPEND imported_libs zstd)
+  list(APPEND imported_libs Zstd::zstd)
 endif()

 if( MSVC OR MINGW )

[aemerson]

I just reverted this in 6e6be5f9504d because it seems to have broken macOS builds:

llvm/lib/Support/Compression.cpp:24:10: fatal error: 'zstd.h' file not found
#include <zstd.h>
         ^~~~~~~~

[ckissane]

In D128465#3651025, @aemerson wrote:

I just reverted this in 6e6be5f9504d because it seems to have broken macOS builds:

llvm/lib/Support/Compression.cpp:24:10: fatal error: 'zstd.h' file not found
#include <zstd.h>
         ^~~~~~~~

@aemerson Could you provide the output of your cmake command?
(I can't easily reproduce because I don't have a mac on hand)

[ckissane]

@aemerson can you let me know if https://reviews.llvm.org/D129786 works on MacOS?
I am continuing work on that revision because this one has a lot of outdated unrelated history.

[jasonmolenda]

In D128465#3652525, @ckissane wrote:

In D128465#3651025, @aemerson wrote:

I just reverted this in 6e6be5f9504d because it seems to have broken macOS builds:

llvm/lib/Support/Compression.cpp:24:10: fatal error: 'zstd.h' file not found
#include <zstd.h>
         ^~~~~~~~

@aemerson Could you provide the output of your cmake command?
(I can't easily reproduce because I don't have a mac on hand)

FWIW I hit this last night on my mac desktop - it looked like I had zstd installed by homebrew (as a dependency on something I installed), and cmake was able to find it in /opt/homebrew (somehow) but when Compression.cpp was built, nothing had added -I/opt/homebrew/include so the header wasn't found. I hacked the FindZSTD.cmake to not find it, to finish what I was working on before bedtime. This might not be related to what @aemerson saw though.

[ckissane]

In D128465#3653051, @jasonmolenda wrote:

In D128465#3652525, @ckissane wrote:

In D128465#3651025, @aemerson wrote:

I just reverted this in 6e6be5f9504d because it seems to have broken macOS builds:

llvm/lib/Support/Compression.cpp:24:10: fatal error: 'zstd.h' file not found
#include <zstd.h>
         ^~~~~~~~

@aemerson Could you provide the output of your cmake command?
(I can't easily reproduce because I don't have a mac on hand)

FWIW I hit this last night on my mac desktop - it looked like I had zstd installed by homebrew (as a dependency on something I installed), and cmake was able to find it in /opt/homebrew (somehow) but when Compression.cpp was built, nothing had added -I/opt/homebrew/include so the header wasn't found. I hacked the FindZSTD.cmake to not find it, to finish what I was working on before bedtime. This might not be related to what @aemerson saw though.

@jasonmolenda Thank you for the info, I believe it is similar, could you try: https://reviews.llvm.org/D129786 for me?

[jasonmolenda]

In D128465#3653216, @ckissane wrote:

In D128465#3653051, @jasonmolenda wrote:

In D128465#3652525, @ckissane wrote:

In D128465#3651025, @aemerson wrote:

I just reverted this in 6e6be5f9504d because it seems to have broken macOS builds:

llvm/lib/Support/Compression.cpp:24:10: fatal error: 'zstd.h' file not found
#include <zstd.h>
         ^~~~~~~~

@aemerson Could you provide the output of your cmake command?
(I can't easily reproduce because I don't have a mac on hand)

FWIW I hit this last night on my mac desktop - it looked like I had zstd installed by homebrew (as a dependency on something I installed), and cmake was able to find it in /opt/homebrew (somehow) but when Compression.cpp was built, nothing had added -I/opt/homebrew/include so the header wasn't found. I hacked the FindZSTD.cmake to not find it, to finish what I was working on before bedtime. This might not be related to what @aemerson saw though.

@jasonmolenda Thank you for the info, I believe it is similar, could you try: https://reviews.llvm.org/D129786 for me?

I applied the patch from https://reviews.llvm.org/D129786 on a clean checkout, cmake ../llvm -G Ninja -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS=1 -DLLVM_ENABLE_PROJECTS='llvm;lldb;clang' '-DLLVM_ENABLE_RUNTIMES=libcxx;libcxxabi' -DLLDB_ENABLE_PYTHON=1 (my normal cmake line) and got

-- Found ZSTD: /opt/homebrew/lib/libzstd.dylib  
-- Looking for ZSTD_compress
-- Looking for ZSTD_compress - found

[...]

FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/Compression.cpp.o 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -DGTEST_HAS_RTTI=0 -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/Users/jason/k/llvm/llvm-project/build/lib/Support -I/Users/jason/k/llvm/llvm-project/llvm/lib/Support -I/Users/jason/k/llvm/llvm-project/build/include -I/Users/jason/k/llvm/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color  -g -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -std=c++14  -fno-exceptions -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/Compression.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/Compression.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/Compression.cpp.o -c /Users/jason/k/llvm/llvm-project/llvm/lib/Support/Compression.cpp
/Users/jason/k/llvm/llvm-project/llvm/lib/Support/Compression.cpp:24:10: fatal error: 'zstd.h' file not found
#include <zstd.h>
         ^~~~~~~~
1 error generated.

% file /opt/homebrew/lib/libzstd.dylib 
/opt/homebrew/lib/libzstd.dylib: Mach-O 64-bit dynamically linked shared library arm64
% head -2 /opt/homebrew/include/zstd.h 
/*
 * Copyright (c) Yann Collet, Facebook, Inc.
%

% grep -i zstd CMakeCache.txt 
//Use zstd for compression/decompression if available. Can be ON,
LLVM_ENABLE_ZSTD:STRING=ON
ZSTD_INCLUDE_DIR:PATH=/opt/homebrew/include
ZSTD_LIBRARY:FILEPATH=/opt/homebrew/lib/libzstd.dylib
//Details about finding ZSTD
FIND_PACKAGE_MESSAGE_DETAILS_ZSTD:INTERNAL=[/opt/homebrew/lib/libzstd.dylib][/opt/homebrew/include][v()]
//Have symbol ZSTD_compress
HAVE_ZSTD:INTERNAL=1
//ADVANCED property for variable: ZSTD_INCLUDE_DIR
ZSTD_INCLUDE_DIR-ADVANCED:INTERNAL=1
//ADVANCED property for variable: ZSTD_LIBRARY
ZSTD_LIBRARY-ADVANCED:INTERNAL=1

[MaskRay]

Instead of marking a differential [OLD] (which may confuse readers whether this is to be abandoned), you can mark a differential Request Reviews/Request Changes. Then it will not be in the green state.

I clicked "Reopen" to make it clear the patch hasn't relanded.

[ckissane]

In D128465#3660714, @MaskRay wrote:

Instead of marking a differential [OLD] (which may confuse readers whether this is to be abandoned), you can mark a differential Request Reviews/Request Changes. Then it will not be in the green state.

I clicked "Reopen" to make it clear the patch hasn't relanded.

Oh, sorry about that!
I had actually created https://reviews.llvm.org/D129786 instead of reopening this one because at the time I tried, reopening didn't seem to work.

[MaskRay]

In D128465#3660733, @ckissane wrote:

In D128465#3660714, @MaskRay wrote:

Instead of marking a differential [OLD] (which may confuse readers whether this is to be abandoned), you can mark a differential Request Reviews/Request Changes. Then it will not be in the green state.

I clicked "Reopen" to make it clear the patch hasn't relanded.

Oh, sorry about that!

That's fine :) We need to learn the web UI.

I had actually created https://reviews.llvm.org/D129786 instead of reopening this one because at the time I tried, reopening didn't seem to work.

A reverted patch by default is still in the "Closed" state. You can "reopen" it so that people know it hasn't relanded.
When repairing a differential, just reuse the existing patch. It keeps all discussions (including past suggestions and issues a previous commit did not address) in one thread.
If the new patch is very similar to this one, you probably need to abandon it in favor of this one.

---

If you revert B because you haven't done A, and A and B are somewhat not so related (e.g. B addresses an existing brittle build system issue which happens to affect some platform when A is added), then having A as a separate differential is fine.

[ckissane]

• move try_find_dependency into it's own file
• add newline to end of TryFindDependencyMacro.cmake
• clarify comment in TryFindDependencyMacro.cmake

[ckissane]

• remove extra cmake info

[MaskRay]

Does this address the macOS build failure?

[ckissane]

In D128465#3661049, @MaskRay wrote:

Does this address the macOS build failure?

I believe so! @leonardchan is double checking

[leonardchan]

LGTM tested on mac on my end

[barannikov88]

@ckissane
I took fresh sources of LLVM and when I run cmake I get the following warning:

-- Looking for compress2
-- Looking for compress2 - found
CMake Warning at cmake/config-ix.cmake:149 (find_package):
  By not providing "Findzstd.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "zstd", but
  CMake did not find one.

  Could not find a package configuration file provided by "zstd" with any of
  the following names:

    zstdConfig.cmake
    zstd-config.cmake

  Add the installation prefix of "zstd" to CMAKE_PREFIX_PATH or set
  "zstd_DIR" to a directory containing one of the above files.  If "zstd"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  CMakeLists.txt:768 (include)

Ubuntu 22.04 LTS
cmake version 3.22.3

[phosek]

llvm/cmake/config-ix.cmake

146

Since there's no Findzstd.cmake module shipped with CMake, and we don't provide in LLVM, we should use the config mode.

[MaskRay]

I am still seeing the

zstdConfig.cmake
zstd-config.cmake

warning. @ckissane @phosek will you fix it? :)

[abrachet]

In D128465#3672515, @MaskRay wrote:

I am still seeing the

zstdConfig.cmake
zstd-config.cmake

warning. @ckissane @phosek will you fix it? :)

I've quieted the warnings in https://github.com/llvm/llvm-project/commit/a71a01bc10d509bd4e746e7d277c4613ef886875, though I'm just now seeing @phosek's suggestion to use CONFIG I'm not sure what that does

[MaskRay]

llvm/include/llvm/Support/Compression.h

48

I missed the values here. Why is -5 picked for NoCompression? What does it mean?

zstd.h says ZSTD_maxCLevel() is currently 22. The CLI program mentions 19. Why is BestSizeCompression 12?

ZSTD_CLEVEL_DEFAULT/the CLI CLI uses 3 for the default level. Why is DefaultCompression 5?

[MaskRay]

llvm/include/llvm/Support/Compression.h

48

Ping @ckissane about the choice.

[andrewrk]

if(LLVM_ENABLE_ZSTD)
  list(APPEND imported_libs zstd::libzstd_shared)
endif()

This hard codes shared linking which breaks the use case of static linking LLVM.

Also LLVM needs to now include a Findzstd.cmake file or else we get this error:

CMake Error at cmake/config-ix.cmake:144 (find_package):
  By not providing "Findzstd.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "zstd", but
  CMake did not find one.

  Could not find a package configuration file provided by "zstd" with any of
  the following names:

    zstdConfig.cmake
    zstd-config.cmake

  Add the installation prefix of "zstd" to CMAKE_PREFIX_PATH or set
  "zstd_DIR" to a directory containing one of the above files.  If "zstd"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  CMakeLists.txt:774 (include)

It is impossible to satisfy this dependency when bootstrapping a static build of zig without patching LLVM.

[MaskRay]

zstd provides GNU Makefile, CMake, and Meson. The CMake files are only installed when configured with CMake. Debian and Ubuntu lack them.
The pkg-config file libzstd.pc probably provides the best portability on at least *NIX systems. (I have also used it for a binutils-gdb patch.)

I think we can then avoid the

zstdConfig.cmake
zstd-config.cmake

issue.

[andrewrk]

Compiler infrastructure should not assume the existence of a Linux distribution. The portable way is simple and can easily be supported. One should be able to install $prefix/lib/libzstd.a and $prefix/include/zstd.h, then have that prefix searched as part of the standard CMAKE_PREFIX_PATH.

If LLVM will not carry Findzstd.cmake then it should not use FIND_PACKAGE in the case of -DLLVM_ENABLE_ZSTD=FORCE_ON; it should just throw -lzstd on the linker line instead of throwing rakes in my path for no reason.

[phosek]

In D128465#3800479, @andrewrk wrote:

if(LLVM_ENABLE_ZSTD)
  list(APPEND imported_libs zstd::libzstd_shared)
endif()

This hard codes shared linking which breaks the use case of static linking LLVM.

This was addressed by D132870 and D133222.

Also LLVM needs to now include a Findzstd.cmake file or else we get this error:

CMake Error at cmake/config-ix.cmake:144 (find_package):
  By not providing "Findzstd.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "zstd", but
  CMake did not find one.

  Could not find a package configuration file provided by "zstd" with any of
  the following names:

    zstdConfig.cmake
    zstd-config.cmake

  Add the installation prefix of "zstd" to CMAKE_PREFIX_PATH or set
  "zstd_DIR" to a directory containing one of the above files.  If "zstd"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  CMakeLists.txt:774 (include)

It is impossible to satisfy this dependency when bootstrapping a static build of zig without patching LLVM.

It should be possible to use CMAKE_FIND_PACKAGE_PREFER_CONFIG=ON to pick up config file that's part of the zstd installation, but I'd be also fine including Findzstd.cmake in LLVM to avoid needing that.

[mgorny]

Is anyone working on a solution that would work for people using a zstd install method that's actually supported upstream?

[phosek]

In D128465#3826150, @mgorny wrote:

Is anyone working on a solution that would work for people using a zstd install method that's actually supported upstream?

I have created D134990, it'd be great if others could test it on their system.

[aaronmondal]

Porting to bazel in https://reviews.llvm.org/D143344

clang-tools-extra/clangd/CMakeLists.txt

 
 llvm_canonicalize_cmake_booleans(
   CLANGD_BUILD_XPC
   CLANGD_ENABLE_REMOTE
   ENABLE_GRPC_REFLECTION
   CLANGD_MALLOC_TRIM
   CLANGD_TIDY_CHECKS
   LLVM_ENABLE_ZLIB
+  LLVM_ENABLE_ZSTD
 )
 
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/Features.inc.in
   ${CMAKE_CURRENT_BINARY_DIR}/Features.inc
 )
 
 set(LLVM_LINK_COMPONENTS

clang-tools-extra/clangd/index/Serialization.cpp

void finalize(llvm::raw_ostream &OS) {
     for (unsigned I = 0; I < Sorted.size(); ++I)
       Index.try_emplace({Sorted[I].data(), Sorted[I].size()}, I);
 
     std::string RawTable;
     for (llvm::StringRef S : Sorted) {
       RawTable.append(std::string(S));
       RawTable.push_back(0);
     }
-    if (llvm::zlib::isAvailable()) {
+    if (llvm::compression::serialize::isAvailable()) {
       llvm::SmallString<1> Compressed;
-      llvm::zlib::compress(RawTable, Compressed);
+      llvm::compression::serialize::compress(RawTable, Compressed);
       write32(RawTable.size(), OS);
       OS << Compressed;
     } else {
       write32(0, OS); // No compression.
       OS << RawTable;
     }
   }
   // Get the ID of an string, which must be interned. Table must be finalized.
llvm::Expected<StringTableIn> readStringTable(llvm::StringRef Data) {
   size_t UncompressedSize = R.consume32();
   if (R.err())
     return error("Truncated string table");
 
   llvm::StringRef Uncompressed;
   llvm::SmallString<1> UncompressedStorage;
   if (UncompressedSize == 0) // No compression
     Uncompressed = R.rest();
-  else if (llvm::zlib::isAvailable()) {
+  else if (llvm::compression::serialize::isAvailable()) {
     // Don't allocate a massive buffer if UncompressedSize was corrupted
     // This is effective for sharded index, but not big monolithic ones, as
     // once compressed size reaches 4MB nothing can be ruled out.
     // Theoretical max ratio from https://zlib.net/zlib_tech.html
     constexpr int MaxCompressionRatio = 1032;
     if (UncompressedSize / MaxCompressionRatio > R.rest().size())
       return error("Bad stri table: uncompress {0} -> {1} bytes is implausible",
                    R.rest().size(), UncompressedSize);
 
-    if (llvm::Error E = llvm::zlib::uncompress(R.rest(), UncompressedStorage,
+    if (llvm::Error E = llvm::compression::serialize::uncompress(R.rest(), UncompressedStorage,
                                                UncompressedSize))
       return std::move(E);
     Uncompressed = UncompressedStorage;
   } else
     return error("Compressed string table, but zlib is unavailable");
 
   StringTableIn Table;
   llvm::StringSaver Saver(Table.Arena);

clang-tools-extra/clangd/test/lit.site.cfg.py.in

 config.llvm_libs_dir = lit_config.substitute("@LLVM_LIBS_DIR@")
 
 config.clangd_source_dir = "@CMAKE_CURRENT_SOURCE_DIR@/.."
 config.clangd_binary_dir = "@CMAKE_CURRENT_BINARY_DIR@/.."
 config.clangd_build_xpc = @CLANGD_BUILD_XPC@
 config.clangd_enable_remote = @CLANGD_ENABLE_REMOTE@
 config.clangd_tidy_checks = @CLANGD_TIDY_CHECKS@
 config.have_zlib = @LLVM_ENABLE_ZLIB@
+config.have_zstd = @LLVM_ENABLE_ZSTD@
 
 # Delegate logic to lit.cfg.py.
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg.py")

clang-tools-extra/clangd/unittests/SerializationTests.cpp

TEST(SerializationTest, NoCrashOnBadArraySize) {
   ASSERT_TRUE(!CorruptParsed);
   EXPECT_EQ(llvm::toString(CorruptParsed.takeError()),
             "malformed or truncated include uri");
 }
 
 // Check we detect invalid string table size size without allocating it first.
 // If this detection fails, the test should allocate a huge array and crash.
 TEST(SerializationTest, NoCrashOnBadStringTableSize) {
-  if (!llvm::zlib::isAvailable()) {
-    log("skipping test, no zlib");
+  if (!llvm::compression::serialize::isAvailable()) {
+    log("skipping test, no llvm::compression");
     return;
   }
 
   // First, create a valid serialized file.
   auto In = readIndexFile(YAML);
   ASSERT_FALSE(!In) << In.takeError();
   IndexFileOut Out(*In);
   Out.Format = IndexFileFormat::RIFF;

clang/cmake/caches/Apple-stage1.cmake

 set(LLVM_INCLUDE_EXAMPLES OFF CACHE BOOL "")
 set(LLVM_INCLUDE_UTILS OFF CACHE BOOL "")
 set(LLVM_INCLUDE_DOCS OFF CACHE BOOL "")
 set(CLANG_INCLUDE_TESTS OFF CACHE BOOL "")
 set(COMPILER_RT_INCLUDE_TESTS OFF CACHE BOOL "")
 set(COMPILER_RT_BUILD_SANITIZERS OFF CACHE BOOL "")
 set(CMAKE_MACOSX_RPATH ON CACHE BOOL "")
 set(LLVM_ENABLE_ZLIB OFF CACHE BOOL "")
+set(LLVM_ENABLE_ZSTD OFF CACHE BOOL "")
 set(LLVM_ENABLE_BACKTRACES OFF CACHE BOOL "")
 set(CLANG_PLUGIN_SUPPORT OFF CACHE BOOL "")
 set(CLANG_SPAWN_CC1 ON CACHE BOOL "")
 set(CLANG_BOOTSTRAP_PASSTHROUGH
   CMAKE_OSX_ARCHITECTURES
   CACHE STRING "")
 
 # Disabling embedded darwin compiler-rt on stage1 builds is required because we

clang/cmake/caches/Apple-stage2.cmake

 # This file sets up a CMakeCache for Apple-style stage2 bootstrap. It is
 # specified by the stage1 build.
 
 set(LLVM_TARGETS_TO_BUILD X86 ARM AArch64 CACHE STRING "") 
 set(PACKAGE_VENDOR Apple CACHE STRING "")
 set(CLANG_VENDOR_UTI com.apple.clang CACHE STRING "")
 set(LLVM_INCLUDE_EXAMPLES OFF CACHE BOOL "")
 set(LLVM_INCLUDE_DOCS OFF CACHE BOOL "")
 set(LLVM_TOOL_CLANG_TOOLS_EXTRA_BUILD OFF CACHE BOOL "")
 set(CLANG_TOOL_SCAN_BUILD_BUILD OFF CACHE BOOL "")
 set(CLANG_TOOL_SCAN_VIEW_BUILD OFF CACHE BOOL "")
 set(CLANG_LINKS_TO_CREATE clang++ cc c++ CACHE STRING "")
 set(CMAKE_MACOSX_RPATH ON CACHE BOOL "")
 set(LLVM_ENABLE_ZLIB ON CACHE BOOL "")
+set(LLVM_ENABLE_ZSTD ON CACHE BOOL "")
 set(LLVM_ENABLE_BACKTRACES OFF CACHE BOOL "")
 set(LLVM_ENABLE_MODULES ON CACHE BOOL "")
 set(LLVM_EXTERNALIZE_DEBUGINFO ON CACHE BOOL "")
 set(CLANG_PLUGIN_SUPPORT OFF CACHE BOOL "")
 set(CLANG_SPAWN_CC1 ON CACHE BOOL "")
 set(BUG_REPORT_URL "http://developer.apple.com/bugreporter/" CACHE STRING "")
 
 set(LLVM_BUILD_EXTERNAL_COMPILER_RT ON CACHE BOOL "Build Compiler-RT with just-built clang")

clang/cmake/caches/Fuchsia-stage2.cmake

 set(LLVM_ENABLE_LIBEDIT OFF CACHE BOOL "")
 set(LLVM_ENABLE_LLD ON CACHE BOOL "")
 set(LLVM_ENABLE_LTO ON CACHE BOOL "")
 set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ON CACHE BOOL "")
 set(LLVM_ENABLE_TERMINFO OFF CACHE BOOL "")
 set(LLVM_ENABLE_UNWIND_TABLES OFF CACHE BOOL "")
 set(LLVM_ENABLE_Z3_SOLVER OFF CACHE BOOL "")
 set(LLVM_ENABLE_ZLIB ON CACHE BOOL "")
+set(LLVM_ENABLE_ZSTD ON CACHE BOOL "")
 set(LLVM_INCLUDE_DOCS OFF CACHE BOOL "")
 set(LLVM_INCLUDE_EXAMPLES OFF CACHE BOOL "")
 set(LLVM_INCLUDE_GO_TESTS OFF CACHE BOOL "")
 set(LLVM_STATIC_LINK_CXX_STDLIB ON CACHE BOOL "")
 set(LLVM_USE_RELATIVE_PATHS_IN_FILES ON CACHE BOOL "")
 
 if(WIN32)
   set(LLVM_USE_CRT_RELEASE "MT" CACHE STRING "")

clang/cmake/caches/Fuchsia.cmake

 # This file sets up a CMakeCache for a Fuchsia toolchain build.
 
 set(LLVM_TARGETS_TO_BUILD X86;ARM;AArch64;RISCV CACHE STRING "")
 
 set(PACKAGE_VENDOR Fuchsia CACHE STRING "")
 
 set(LLVM_ENABLE_PROJECTS "bolt;clang;clang-tools-extra;lld;llvm;polly" CACHE STRING "")
 
 set(LLVM_ENABLE_DIA_SDK OFF CACHE BOOL "")
 set(LLVM_ENABLE_LIBEDIT OFF CACHE BOOL "")
 set(LLVM_ENABLE_LIBXML2 OFF CACHE BOOL "")
 set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ON CACHE BOOL "")
 set(LLVM_ENABLE_TERMINFO OFF CACHE BOOL "")
 set(LLVM_ENABLE_UNWIND_TABLES OFF CACHE BOOL "")
 set(LLVM_ENABLE_Z3_SOLVER OFF CACHE BOOL "")
 set(LLVM_ENABLE_ZLIB OFF CACHE BOOL "")
+set(LLVM_ENABLE_ZSTD OFF CACHE BOOL "")
 set(LLVM_INCLUDE_DOCS OFF CACHE BOOL "")
 set(LLVM_INCLUDE_EXAMPLES OFF CACHE BOOL "")
 set(LLVM_INCLUDE_GO_TESTS OFF CACHE BOOL "")
 
 if(WIN32)
   set(LLVM_USE_CRT_RELEASE "MT" CACHE STRING "")
 endif()
 

clang/lib/Driver/ToolChains/Clang.cpp

static void RenderDebugInfoCompressionArgs(const ArgList &Args,
   const Arg *A = Args.getLastArg(options::OPT_gz_EQ);
   if (!A)
     return;
   if (checkDebugInfoOption(A, Args, D, TC)) {
     StringRef Value = A->getValue();
     if (Value == "none") {
       CmdArgs.push_back("--compress-debug-sections=none");
     } else if (Value == "zlib") {
-      if (llvm::zlib::isAvailable()) {
+      if (llvm::compression::elf::isAvailable()) {
         CmdArgs.push_back(
             Args.MakeArgString("--compress-debug-sections=" + Twine(Value)));
       } else {
         D.Diag(diag::warn_debug_compression_unavailable);
       }
     } else {
       D.Diag(diag::err_drv_unsupported_option_argument)
           << A->getOption().getName() << Value;

clang/lib/Serialization/ASTReader.cpp

Expected<unsigned> MaybeRecCode =
         SLocEntryCursor.readRecord(Code, Record, &Blob);
     if (!MaybeRecCode) {
       Error(MaybeRecCode.takeError());
       return nullptr;
     }
     unsigned RecCode = MaybeRecCode.get();
 
     if (RecCode == SM_SLOC_BUFFER_BLOB_COMPRESSED) {
-      if (!llvm::zlib::isAvailable()) {
+      if (!llvm::compression::serialize::isAvailable()) {
         Error("zlib is not available");
         return nullptr;
       }
       SmallString<0> Uncompressed;
       if (llvm::Error E =
-              llvm::zlib::uncompress(Blob, Uncompressed, Record[0])) {
+              llvm::compression::serialize::uncompress(Blob, Uncompressed, Record[0])) {
         Error("could not decompress embedded file contents: " +
               llvm::toString(std::move(E)));
         return nullptr;
       }
       return llvm::MemoryBuffer::getMemBufferCopy(Uncompressed, Name);
     } else if (RecCode == SM_SLOC_BUFFER_BLOB) {
       return llvm::MemoryBuffer::getMemBuffer(Blob.drop_back(1), Name, true);
     } else {

clang/lib/Serialization/ASTWriter.cpp

 static void emitBlob(llvm::BitstreamWriter &Stream, StringRef Blob,
                      unsigned SLocBufferBlobCompressedAbbrv,
                      unsigned SLocBufferBlobAbbrv) {
   using RecordDataType = ASTWriter::RecordData::value_type;
 
   // Compress the buffer if possible. We expect that almost all PCM
   // consumers will not want its contents.
   SmallString<0> CompressedBuffer;
-  if (llvm::zlib::isAvailable()) {
-    llvm::zlib::compress(Blob.drop_back(1), CompressedBuffer);
+  if (llvm::compression::serialize::isAvailable()) {
+    llvm::compression::serialize::compress(Blob.drop_back(1), CompressedBuffer);
     RecordDataType Record[] = {SM_SLOC_BUFFER_BLOB_COMPRESSED, Blob.size() - 1};
     Stream.EmitRecordWithBlob(SLocBufferBlobCompressedAbbrv, Record,
                               CompressedBuffer);
     return;
   }
 
   RecordDataType Record[] = {SM_SLOC_BUFFER_BLOB};
   Stream.EmitRecordWithBlob(SLocBufferBlobAbbrv, Record, Blob);

clang/test/CMakeLists.txt

 # Test runner infrastructure for Clang. This configures the Clang test trees
 # for use by Lit, and delegates to LLVM's lit test handlers.
 
 llvm_canonicalize_cmake_booleans(
   CLANG_BUILD_EXAMPLES
   CLANG_DEFAULT_PIE_ON_LINUX
   CLANG_ENABLE_OPAQUE_POINTERS_INTERNAL
   CLANG_ENABLE_ARCMT
   CLANG_ENABLE_STATIC_ANALYZER
   CLANG_PLUGIN_SUPPORT
   CLANG_SPAWN_CC1
   ENABLE_BACKTRACES
   LLVM_ENABLE_ZLIB
+  LLVM_ENABLE_ZSTD
   LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
   LLVM_ENABLE_THREADS
   LLVM_WITH_Z3
   )
 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py

clang/test/lit.site.cfg.py.in

 config.clang_tools_dir = lit_config.substitute(path(r"@CURRENT_TOOLS_DIR@"))
 config.clang_lib_dir = path(r"@CMAKE_LIBRARY_OUTPUT_DIRECTORY@")
 config.host_triple = "@LLVM_HOST_TRIPLE@"
 config.target_triple = "@LLVM_TARGET_TRIPLE@"
 config.host_cc = "@CMAKE_C_COMPILER@"
 config.host_cxx = "@CMAKE_CXX_COMPILER@"
 config.llvm_use_sanitizer = "@LLVM_USE_SANITIZER@"
 config.have_zlib = @LLVM_ENABLE_ZLIB@
+config.have_zstd = @LLVM_ENABLE_ZSTD@
 config.clang_arcmt = @CLANG_ENABLE_ARCMT@
 config.clang_default_pie_on_linux = @CLANG_DEFAULT_PIE_ON_LINUX@
 config.clang_enable_opaque_pointers = @CLANG_ENABLE_OPAQUE_POINTERS_INTERNAL@
 config.clang_default_cxx_stdlib = "@CLANG_DEFAULT_CXX_STDLIB@"
 config.clang_staticanalyzer = @CLANG_ENABLE_STATIC_ANALYZER@
 config.clang_staticanalyzer_z3 = @LLVM_WITH_Z3@
 config.clang_examples = @CLANG_BUILD_EXAMPLES@
 config.enable_shared = @ENABLE_SHARED@

compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh

if [[ ! -d ${LLVM_BUILD} ]]; then
   cmake -GNinja \
     -DCMAKE_BUILD_TYPE=Release \
     -DCMAKE_C_COMPILER=$CC \
     -DCMAKE_CXX_COMPILER=$CXX \
     -DCMAKE_C_FLAGS_RELEASE="${LLVM_FLAGS}" \
     -DCMAKE_CXX_FLAGS_RELEASE="${LLVM_FLAGS}" \
     -DLLVM_TABLEGEN=$TBLGEN \
     -DLLVM_ENABLE_ZLIB=ON \
+    -DLLVM_ENABLE_ZSTD=ON \
     -DLLVM_ENABLE_TERMINFO=OFF \
     -DLLVM_ENABLE_THREADS=OFF \
   $LLVM_SRC
 fi
 cd ${LLVM_BUILD}
 ninja LLVMSymbolize LLVMObject LLVMBinaryFormat LLVMDebugInfoDWARF LLVMSupport LLVMDebugInfoPDB LLVMDebuginfod LLVMMC LLVMDemangle LLVMTextAPI
 
 cd ${BUILD_DIR}

compiler-rt/test/lit.common.configured.in

elif config.android:
   if config.target_arch == "i386":
     set_default("target_suffix", "-i686-android")
   else:
     set_default("target_suffix", "-%s-android" % config.target_arch)
 else:
   set_default("target_suffix", "-%s" % config.target_arch)
 
 set_default("have_zlib", "@LLVM_ENABLE_ZLIB@")
+set_default("have_zstd", "@LLVM_ENABLE_ZSTD@")
 set_default("libcxx_used", "@LLVM_LIBCXX_USED@")
 
 # LLVM tools dir can be passed in lit parameters, so try to
 # apply substitution.
 config.llvm_tools_dir = lit_config.substitute(config.llvm_tools_dir)
 config.clang = lit_config.substitute(config.clang)
 config.compiler_rt_libdir = lit_config.substitute(config.compiler_rt_libdir)
 
 if not os.path.exists(config.clang):
   lit_config.fatal("Can't find compiler on path %r" % config.clang)
 
 # Setup attributes common for all compiler-rt projects.
 lit_config.load_config(config, "@COMPILER_RT_SOURCE_DIR@/test/lit.common.cfg.py")

flang/CMakeLists.txt

if (FLANG_STANDALONE_BUILD)
   if (NOT Clang_FOUND)
     message(FATAL_ERROR "Failed to find Clang")
   endif()
 
   # If LLVM links to zlib we need the imported targets so we can too.
   if(LLVM_ENABLE_ZLIB)
     find_package(ZLIB REQUIRED)
   endif()
+  # If LLVM links to zstd we need the imported targets so we can too.
+  if(LLVM_ENABLE_ZSTD)
+    find_package(ZSTD REQUIRED)
+  endif()
   option(LLVM_ENABLE_PEDANTIC "Compile with pedantic enabled." ON)
   if(CMAKE_COMPILER_IS_GNUCXX)
     set(USE_NO_MAYBE_UNINITIALIZED 1)
   endif()
 
   include(CMakeParseArguments)
   include(AddLLVM)
   include(HandleLLVMOptions)

lld/ELF/CMakeLists.txt

 set(LLVM_TARGET_DEFINITIONS Options.td)
 tablegen(LLVM Options.inc -gen-opt-parser-defs)
 add_public_tablegen_target(ELFOptionsTableGen)
 
 if(LLVM_ENABLE_ZLIB)
-  set(imported_libs ZLIB::ZLIB)
+  list(APPEND imported_libs ZLIB::ZLIB)
+endif()
+
+if(LLVM_ENABLE_ZSTD)
+  list(APPEND imported_libs zstd)
 endif()
 
 add_lld_library(lldELF
   AArch64ErrataFix.cpp
   Arch/AArch64.cpp
   Arch/AMDGPU.cpp
   Arch/ARM.cpp
   Arch/AVR.cpp

lld/ELF/Driver.cpp

 }
 
 static bool getCompressDebugSections(opt::InputArgList &args) {
   StringRef s = args.getLastArgValue(OPT_compress_debug_sections, "none");
   if (s == "none")
     return false;
   if (s != "zlib")
     error("unknown --compress-debug-sections value: " + s);
-  if (!zlib::isAvailable())
+  if (!compression::elf::isAvailable())
     error("--compress-debug-sections: zlib is not available");
   return true;
 }
 
 static StringRef getAliasSpelling(opt::Arg *arg) {
   if (const opt::Arg *alias = arg->getAlias())
     return alias->getSpelling();
   return arg->getSpelling();

lld/ELF/InputSection.cpp

InputSectionBase::InputSectionBase(InputFile *file, uint64_t flags,
   uint32_t v = std::max<uint32_t>(alignment, 1);
   if (!isPowerOf2_64(v))
     fatal(toString(this) + ": sh_addralign is not a power of 2");
   this->alignment = v;
 
   // If SHF_COMPRESSED is set, parse the header. The legacy .zdebug format is no
   // longer supported.
   if (flags & SHF_COMPRESSED) {
-    if (!zlib::isAvailable())
+    if (!compression::elf::isAvailable())
       error(toString(file) + ": contains a compressed section, " +
             "but zlib is not available");
     invokeELFT(parseCompressedHeader);
   }
 }
 
 // Drop SHF_GROUP bit unless we are producing a re-linkable object file.
 // SHF_GROUP is a marker that a section belongs to some comdat group.
void InputSectionBase::uncompress() const {
   size_t size = uncompressedSize;
   char *uncompressedBuf;
   {
     static std::mutex mu;
     std::lock_guard<std::mutex> lock(mu);
     uncompressedBuf = bAlloc().Allocate<char>(size);
   }
 
-  if (Error e = zlib::uncompress(toStringRef(rawData), uncompressedBuf, size))
+  if (Error e = compression::elf::uncompress(toStringRef(rawData), uncompressedBuf, size))
     fatal(toString(this) +
           ": uncompress failed: " + llvm::toString(std::move(e)));
   rawData = makeArrayRef((uint8_t *)uncompressedBuf, size);
   uncompressedSize = -1;
 }
 
 template <class ELFT> RelsOrRelas<ELFT> InputSectionBase::relsOrRelas() const {
   if (relSecIdx == 0)
if (LLVM_UNLIKELY(type == SHT_GROUP)) {
     copyShtGroup<ELFT>(buf);
     return;
   }
 
   // If this is a compressed section, uncompress section contents directly
   // to the buffer.
   if (uncompressedSize >= 0) {
     size_t size = uncompressedSize;
-    if (Error e = zlib::uncompress(toStringRef(rawData), (char *)buf, size))
+    if (Error e = compression::elf::uncompress(toStringRef(rawData), (char *)buf, size))
       fatal(toString(this) +
             ": uncompress failed: " + llvm::toString(std::move(e)));
     uint8_t *bufEnd = buf + size;
     relocate<ELFT>(buf, bufEnd);
     return;
   }
 
   // Copy section contents from source object file to output file

lldb/source/Plugins/Process/gdb-remote/CMakeLists.txt

 if(HAVE_LIBCOMPRESSION)
   set(LIBCOMPRESSION compression)
 endif()
 
 if(LLVM_ENABLE_ZLIB)
   list(APPEND LLDB_SYSTEM_LIBS ZLIB::ZLIB)
 endif()
 
+if(LLVM_ENABLE_ZSTD)
+  list(APPEND LLDB_SYSTEM_LIBS zstd)
+endif()
+
 add_lldb_library(lldbPluginProcessGDBRemote PLUGIN
   GDBRemoteClientBase.cpp
   GDBRemoteCommunication.cpp
   GDBRemoteCommunicationClient.cpp
   GDBRemoteCommunicationHistory.cpp
   GDBRemoteCommunicationServer.cpp
   GDBRemoteCommunicationServerCommon.cpp
   GDBRemoteCommunicationServerLLGS.cpp

llvm/CMakeLists.txt

 if (CMAKE_SYSTEM_NAME MATCHES "OS390")
   option(LLVM_ENABLE_THREADS "Use threads if available." OFF)
 else()
   option(LLVM_ENABLE_THREADS "Use threads if available." ON)
 endif()
 
 set(LLVM_ENABLE_ZLIB "ON" CACHE STRING "Use zlib for compression/decompression if available. Can be ON, OFF, or FORCE_ON")
 
+set(LLVM_ENABLE_ZSTD "ON" CACHE STRING "Use zstd for compression/decompression if available. Can be ON, OFF, or FORCE_ON")
+
 set(LLVM_ENABLE_CURL "OFF" CACHE STRING "Use libcurl for the HTTP client if available. Can be ON, OFF, or FORCE_ON")
 
 set(LLVM_Z3_INSTALL_DIR "" CACHE STRING "Install directory of the Z3 solver.")
 
 option(LLVM_ENABLE_Z3_SOLVER
   "Enable Support for the Z3 constraint solver in LLVM."
   ${LLVM_ENABLE_Z3_SOLVER_DEFAULT}
 )

llvm/cmake/config-ix.cmake

if(ZLIB_FOUND)
     cmake_pop_check_state()
     if(LLVM_ENABLE_ZLIB STREQUAL FORCE_ON AND NOT HAVE_ZLIB)
       message(FATAL_ERROR "Failed to configure zlib")
     endif()
   endif()
   set(LLVM_ENABLE_ZLIB "${HAVE_ZLIB}")
 endif()
 
+if(LLVM_ENABLE_ZSTD)
+  if(LLVM_ENABLE_ZSTD STREQUAL FORCE_ON)
+    find_package(ZSTD REQUIRED)
+  elseif(NOT LLVM_USE_SANITIZER MATCHES "Memory.*")
+    find_package(ZSTD)
+  endif()
+  if(ZSTD_FOUND)
+    # Check if zstd we found is usable; for example, we may have found a 32-bit
+    # library on a 64-bit system which would result in a link-time failure.
+    cmake_push_check_state()
+    list(APPEND CMAKE_REQUIRED_INCLUDES ${ZSTD_INCLUDE_DIRS})
+    list(APPEND CMAKE_REQUIRED_LIBRARIES ${ZSTD_LIBRARY})
+    check_symbol_exists(ZSTD_compress zstd.h HAVE_ZSTD)
+    cmake_pop_check_state()
+    if(LLVM_ENABLE_ZSTD STREQUAL FORCE_ON AND NOT HAVE_ZSTD)
+      message(FATAL_ERROR "Failed to configure zstd")
+    endif()
+  endif()
+  set(LLVM_ENABLE_ZSTD "${HAVE_ZSTD}")
+endif()
+
 if(LLVM_ENABLE_LIBXML2)
   if(LLVM_ENABLE_LIBXML2 STREQUAL FORCE_ON)
     find_package(LibXml2 REQUIRED)
   elseif(NOT LLVM_USE_SANITIZER MATCHES "Memory.*")
     find_package(LibXml2)
   endif()
   if(LibXml2_FOUND)
     # Check if libxml2 we found is usable; for example, we may have found a 32-bit

llvm/cmake/modules/LLVMConfig.cmake.in

 set(LLVM_ENABLE_UNWIND_TABLES @LLVM_ENABLE_UNWIND_TABLES@)
 
 set(LLVM_ENABLE_ZLIB @LLVM_ENABLE_ZLIB@)
 if(LLVM_ENABLE_ZLIB)
   set(ZLIB_ROOT @ZLIB_ROOT@)
   find_package(ZLIB)
 endif()
 
+set(LLVM_ENABLE_ZSTD @LLVM_ENABLE_ZSTD@)
+if(LLVM_ENABLE_ZSTD)
+  set(ZSTD_ROOT @ZSTD_ROOT@)
+  find_package(ZSTD)
+endif()
+
 set(LLVM_ENABLE_LIBXML2 @LLVM_ENABLE_LIBXML2@)
 if(LLVM_ENABLE_LIBXML2)
   find_package(LibXml2)
 endif()
 
 set(LLVM_WITH_Z3 @LLVM_WITH_Z3@)
 
 set(LLVM_ENABLE_DIA_SDK @LLVM_ENABLE_DIA_SDK@)

llvm/include/llvm/Config/llvm-config.h.cmake

 #cmakedefine LLVM_WITH_Z3 ${LLVM_WITH_Z3}
 
 /* Define if we have curl and want to use it */
 #cmakedefine LLVM_ENABLE_CURL ${LLVM_ENABLE_CURL}
 
 /* Define if zlib compression is available */
 #cmakedefine01 LLVM_ENABLE_ZLIB
 
+/* Define if zstd compression is available */
+#cmakedefine01 LLVM_ENABLE_ZSTD
+
 /* Define if LLVM was built with a dependency to the libtensorflow dynamic library */
 #cmakedefine LLVM_HAVE_TF_API
 
 /* Define to 1 if you have the <sysexits.h> header file. */
 #cmakedefine HAVE_SYSEXITS_H ${HAVE_SYSEXITS_H}
 
 /* Define if the xar_open() function is supported on this platform. */
 #cmakedefine LLVM_HAVE_LIBXAR ${LLVM_HAVE_LIBXAR}

llvm/include/llvm/Support/Compression.h

 
 #include "llvm/Support/DataTypes.h"
 
 namespace llvm {
 template <typename T> class SmallVectorImpl;
 class Error;
 class StringRef;
 
+namespace compression {
+
 namespace zlib {
 
 static constexpr int NoCompression = 0;
 static constexpr int BestSpeedCompression = 1;
 static constexpr int DefaultCompression = 6;
 static constexpr int BestSizeCompression = 9;
 
 bool isAvailable();
 
 void compress(StringRef InputBuffer, SmallVectorImpl<char> &CompressedBuffer,
               int Level = DefaultCompression);
 
 Error uncompress(StringRef InputBuffer, char *UncompressedBuffer,
                  size_t &UncompressedSize);
 
 Error uncompress(StringRef InputBuffer,
                  SmallVectorImpl<char> &UncompressedBuffer,
                  size_t UncompressedSize);
 
 uint32_t crc32(StringRef Buffer);
 
-}  // End of namespace zlib
+} // End of namespace zlib
 
+namespace zstd {
+
+static constexpr int NoCompression = -5;
+static constexpr int BestSpeedCompression = 1;
+static constexpr int DefaultCompression = 2;
+static constexpr int BestSizeCompression = 12;
+
+bool isAvailable();
+
+void compress(StringRef InputBuffer, SmallVectorImpl<char> &CompressedBuffer,
+              int Level = DefaultCompression);
+
+Error uncompress(StringRef InputBuffer, char *UncompressedBuffer,
+                 size_t &UncompressedSize);
+
+Error uncompress(StringRef InputBuffer,
+                 SmallVectorImpl<char> &UncompressedBuffer,
+                 size_t UncompressedSize);
+
+uint32_t crc32(StringRef Buffer);
+
+} // End of namespace zstd
+
+#if LLVM_ENABLE_ZSTD
+namespace internal = llvm::compression::zstd;
+#else
+namespace internal = llvm::compression::zlib;
+#endif
+
+namespace elf = llvm::compression::zlib;
+
+// TODO: COLE: should this be always zlib? if not make profile just internal, if so make profile zlib
+// #if
+// namespace profile = llvm::compression::zlib;
+// #else
+namespace profile = llvm::compression::internal;
+// #endif
+
+// TODO: COLE: should this be always zlib? if not make serialize just internal, if so make serialize zlib
+// #if
+// namespace serialize = llvm::compression::zlib;
+// #else
+namespace serialize = llvm::compression::internal;
+// #endif
+
+} // End of namespace compression
+
 } // End of namespace llvm
 
 #endif

llvm/lib/MC/ELFObjectWriter.cpp

assert((MAI->compressDebugSections() == DebugCompressionType::Z ||
           MAI->compressDebugSections() == DebugCompressionType::GNU) &&
          "expected zlib or zlib-gnu style compression");
 
   SmallVector<char, 128> UncompressedData;
   raw_svector_ostream VecOS(UncompressedData);
   Asm.writeSectionData(VecOS, &Section, Layout);
 
   SmallVector<char, 128> CompressedContents;
-  zlib::compress(StringRef(UncompressedData.data(), UncompressedData.size()),
+  compression::elf::compress(StringRef(UncompressedData.data(), UncompressedData.size()),
                  CompressedContents);
 
   bool ZlibStyle = MAI->compressDebugSections() == DebugCompressionType::Z;
   if (!maybeWriteCompression(UncompressedData.size(), CompressedContents,
                              ZlibStyle, Sec.getAlignment())) {
     W.OS << UncompressedData;
     return;
   }

llvm/lib/ObjCopy/ELF/ELFObject.cpp

const size_t DataOffset = isDataGnuCompressed(Sec.OriginalData)
                                 ? (ZlibGnuMagic.size() + sizeof(Sec.Size))
                                 : sizeof(Elf_Chdr_Impl<ELFT>);
 
   StringRef CompressedContent(
       reinterpret_cast<const char *>(Sec.OriginalData.data()) + DataOffset,
       Sec.OriginalData.size() - DataOffset);
 
   SmallVector<char, 128> DecompressedContent;
-  if (Error Err = zlib::uncompress(CompressedContent, DecompressedContent,
+  if (Error Err = compression::elf::uncompress(CompressedContent, DecompressedContent,
                                    static_cast<size_t>(Sec.Size)))
     return createStringError(errc::invalid_argument,
                              "'" + Sec.Name + "': " + toString(std::move(Err)));
 
   uint8_t *Buf = reinterpret_cast<uint8_t *>(Out.getBufferStart()) + Sec.Offset;
   std::copy(DecompressedContent.begin(), DecompressedContent.end(), Buf);
 
   return Error::success();
Error ELFSectionWriter<ELFT>::visit(const CompressedSection &Sec) {
   std::copy(Sec.CompressedData.begin(), Sec.CompressedData.end(), Buf);
   return Error::success();
 }
 
 CompressedSection::CompressedSection(const SectionBase &Sec,
                                      DebugCompressionType CompressionType)
     : SectionBase(Sec), CompressionType(CompressionType),
       DecompressedSize(Sec.OriginalData.size()), DecompressedAlign(Sec.Align) {
-  zlib::compress(StringRef(reinterpret_cast<const char *>(OriginalData.data()),
+  compression::elf::compress(StringRef(reinterpret_cast<const char *>(OriginalData.data()),
                            OriginalData.size()),
                  CompressedData);
 
   size_t ChdrSize;
   if (CompressionType == DebugCompressionType::GNU) {
     Name = ".z" + Sec.Name.substr(1);
     ChdrSize = sizeof("ZLIB") - 1 + sizeof(uint64_t);
   } else {

llvm/lib/Object/Decompressor.cpp

 #include "llvm/Support/Endian.h"
 
 using namespace llvm;
 using namespace llvm::support::endian;
 using namespace object;
 
 Expected<Decompressor> Decompressor::create(StringRef Name, StringRef Data,
                                             bool IsLE, bool Is64Bit) {
-  if (!zlib::isAvailable())
+  if (!compression::elf::isAvailable())
     return createError("zlib is not available");
 
   Decompressor D(Data);
   Error Err = isGnuStyle(Name) ? D.consumeCompressedGnuHeader()
                                : D.consumeCompressedZLibHeader(Is64Bit, IsLE);
   if (Err)
     return std::move(Err);
   return D;

 }
 
 bool Decompressor::isCompressedELFSection(uint64_t Flags, StringRef Name) {
   return (Flags & ELF::SHF_COMPRESSED) || isGnuStyle(Name);
 }
 
 Error Decompressor::decompress(MutableArrayRef<char> Buffer) {
   size_t Size = Buffer.size();
-  return zlib::uncompress(SectionData, Buffer.data(), Size);
+  return compression::elf::uncompress(SectionData, Buffer.data(), Size);
 }

llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

Error RawCoverageFilenamesReader::read(CovMapVersion Version) {
   if (auto Err = readULEB128(UncompressedLen))
     return Err;
 
   uint64_t CompressedLen;
   if (auto Err = readSize(CompressedLen))
     return Err;
 
   if (CompressedLen > 0) {
-    if (!zlib::isAvailable())
+    if (!compression::internal::isAvailable())
       return make_error<CoverageMapError>(
           coveragemap_error::decompression_failed);
 
     // Allocate memory for the decompressed filenames.
     SmallVector<char, 0> StorageBuf;
 
     // Read compressed filenames.
     StringRef CompressedFilenames = Data.substr(0, CompressedLen);
     Data = Data.substr(CompressedLen);
     auto Err =
-        zlib::uncompress(CompressedFilenames, StorageBuf, UncompressedLen);
+        compression::internal::uncompress(CompressedFilenames, StorageBuf, UncompressedLen);
     if (Err) {
       consumeError(std::move(Err));
       return make_error<CoverageMapError>(
           coveragemap_error::decompression_failed);
     }
 
     StringRef UncompressedFilenames(StorageBuf.data(), StorageBuf.size());
     RawCoverageFilenamesReader Delegate(UncompressedFilenames, Filenames,

llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp

std::string FilenamesStr;
     for (const auto &Filename : Filenames) {
       encodeULEB128(Filename.size(), FilenamesOS);
       FilenamesOS << Filename;
     }
   }
 
   SmallString<128> CompressedStr;
   bool doCompression =
-      Compress && zlib::isAvailable() && DoInstrProfNameCompression;
+      Compress && compression::internal::isAvailable() && DoInstrProfNameCompression;
   if (doCompression)
-    zlib::compress(FilenamesStr, CompressedStr, zlib::BestSizeCompression);
+    compression::internal::compress(FilenamesStr, CompressedStr, compression::internal::BestSizeCompression);
 
   // ::= <num-filenames>
   //     <uncompressed-len>
   //     <compressed-len-or-zero>
   //     (<compressed-filenames> | <uncompressed-filenames>)
   encodeULEB128(Filenames.size(), OS);
   encodeULEB128(FilenamesStr.size(), OS);
   encodeULEB128(doCompression ? CompressedStr.size() : 0U, OS);

llvm/lib/ProfileData/InstrProf.cpp

auto WriteStringToResult = [&](size_t CompressedLen, StringRef InputStr) {
     return Error::success();
   };
 
   if (!doCompression) {
     return WriteStringToResult(0, UncompressedNameStrings);
   }
 
   SmallString<128> CompressedNameStrings;
-  zlib::compress(StringRef(UncompressedNameStrings), CompressedNameStrings,
-                 zlib::BestSizeCompression);
+  compression::profile::compress(StringRef(UncompressedNameStrings), CompressedNameStrings,
+                 compression::profile::BestSizeCompression);
 
   return WriteStringToResult(CompressedNameStrings.size(),
                              CompressedNameStrings);
 }
 
 StringRef getPGOFuncNameVarInitializer(GlobalVariable *NameVar) {
   auto *Arr = cast<ConstantDataArray>(NameVar->getInitializer());
   StringRef NameStr =
       Arr->isCString() ? Arr->getAsCString() : Arr->getAsString();
   return NameStr;
 }
 
 Error collectPGOFuncNameStrings(ArrayRef<GlobalVariable *> NameVars,
                                 std::string &Result, bool doCompression) {
   std::vector<std::string> NameStrs;
   for (auto *NameVar : NameVars) {
     NameStrs.push_back(std::string(getPGOFuncNameVarInitializer(NameVar)));
   }
   return collectPGOFuncNameStrings(
-      NameStrs, zlib::isAvailable() && doCompression, Result);
+      NameStrs, compression::profile::isAvailable() && doCompression, Result);
 }
 
 Error readPGOFuncNameStrings(StringRef NameStrings, InstrProfSymtab &Symtab) {
   const uint8_t *P = NameStrings.bytes_begin();
   const uint8_t *EndP = NameStrings.bytes_end();
   while (P < EndP) {
     uint32_t N;
     uint64_t UncompressedSize = decodeULEB128(P, &N);
     P += N;
     uint64_t CompressedSize = decodeULEB128(P, &N);
     P += N;
     bool isCompressed = (CompressedSize != 0);
     SmallString<128> UncompressedNameStrings;
     StringRef NameStrings;
     if (isCompressed) {
-      if (!llvm::zlib::isAvailable())
+      if (!llvm::compression::profile::isAvailable())
         return make_error<InstrProfError>(instrprof_error::zlib_unavailable);
 
       StringRef CompressedNameStrings(reinterpret_cast<const char *>(P),
                                       CompressedSize);
       if (Error E =
-              zlib::uncompress(CompressedNameStrings, UncompressedNameStrings,
+              compression::profile::uncompress(CompressedNameStrings, UncompressedNameStrings,
                                UncompressedSize)) {
         consumeError(std::move(E));
         return make_error<InstrProfError>(instrprof_error::uncompress_failed);
       }
       P += CompressedSize;
       NameStrings = StringRef(UncompressedNameStrings.data(),
                               UncompressedNameStrings.size());
     } else {

llvm/lib/ProfileData/SampleProfReader.cpp

std::error_code SampleProfileReaderExtBinaryBase::decompressSection(
   if (std::error_code EC = DecompressSize.getError())
     return EC;
   DecompressBufSize = *DecompressSize;
 
   auto CompressSize = readNumber<uint64_t>();
   if (std::error_code EC = CompressSize.getError())
     return EC;
 
-  if (!llvm::zlib::isAvailable())
+  if (!llvm::compression::profile::isAvailable())
     return sampleprof_error::zlib_unavailable;
 
   StringRef CompressedStrings(reinterpret_cast<const char *>(Data),
                               *CompressSize);
   char *Buffer = Allocator.Allocate<char>(DecompressBufSize);
   size_t UCSize = DecompressBufSize;
   llvm::Error E =
-      zlib::uncompress(CompressedStrings, Buffer, UCSize);
+      compression::profile::uncompress(CompressedStrings, Buffer, UCSize);
   if (E)
     return sampleprof_error::uncompress_failed;
   DecompressBuf = reinterpret_cast<const uint8_t *>(Buffer);
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileReaderExtBinaryBase::readImpl() {
   const uint8_t *BufStart =

llvm/lib/ProfileData/SampleProfWriter.cpp

SampleProfileWriterExtBinaryBase::markSectionStart(SecType Type,
   assert(Entry.Type == Type && "Unexpected section type");
   // Use LocalBuf as a temporary output for writting data.
   if (hasSecFlag(Entry, SecCommonFlags::SecFlagCompress))
     LocalBufStream.swap(OutputStream);
   return SectionStart;
 }
 
 std::error_code SampleProfileWriterExtBinaryBase::compressAndOutput() {
-  if (!llvm::zlib::isAvailable())
+  if (!llvm::compression::internal::isAvailable())
     return sampleprof_error::zlib_unavailable;
   std::string &UncompressedStrings =
       static_cast<raw_string_ostream *>(LocalBufStream.get())->str();
   if (UncompressedStrings.size() == 0)
     return sampleprof_error::success;
   auto &OS = *OutputStream;
   SmallString<128> CompressedStrings;
-  zlib::compress(UncompressedStrings, CompressedStrings,
-                 zlib::BestSizeCompression);
+  compression::internal::compress(UncompressedStrings, CompressedStrings,
+                 compression::internal::BestSizeCompression);
   encodeULEB128(UncompressedStrings.size(), OS);
   encodeULEB128(CompressedStrings.size(), OS);
   OS << CompressedStrings.str();
   UncompressedStrings.clear();
   return sampleprof_error::success;
 }
 
 /// Add a new section into section header table given the section type

llvm/lib/Support/CMakeLists.txt

if (HAS_WERROR_GLOBAL_CTORS)
   int main() { (void)TestGlobalCtorDtor; (void)TestGlobalCtorDtor2; return 0;}
   " LLVM_HAS_NOGLOBAL_CTOR_MUTEX)
   if (NOT LLVM_HAS_NOGLOBAL_CTOR_MUTEX)
     string(REPLACE "-Werror=global-constructors" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
   endif()
 endif()
 
 if(LLVM_ENABLE_ZLIB)
-  set(imported_libs ZLIB::ZLIB)
+  list(APPEND imported_libs ZLIB::ZLIB)
+endif()
+
+if(LLVM_ENABLE_ZSTD)
+  list(APPEND imported_libs zstd)
 endif()
 
 if( MSVC OR MINGW )
   # libuuid required for FOLDERID_Profile usage in lib/Support/Windows/Path.inc.
   # advapi32 required for CryptAcquireContextW in lib/Support/Windows/Path.inc.
   set(system_libs ${system_libs} psapi shell32 ole32 uuid advapi32)
 elseif( CMAKE_HOST_UNIX )
   if( HAVE_LIBRT )

llvm/lib/Support/Compression.cpp

 #include "llvm/ADT/StringRef.h"
 #include "llvm/Config/config.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #if LLVM_ENABLE_ZLIB
 #include <zlib.h>
 #endif
+#if LLVM_ENABLE_ZSTD
+#include <zstd.h>
+#endif
 
 using namespace llvm;
 
-#if LLVM_ENABLE_ZLIB
+using namespace compression;
+
 static Error createError(StringRef Err) {
   return make_error<StringError>(Err, inconvertibleErrorCode());
 }
 
+#if LLVM_ENABLE_ZLIB
+
 static StringRef convertZlibCodeToString(int Code) {
   switch (Code) {
   case Z_MEM_ERROR:
     return "zlib error: Z_MEM_ERROR";
   case Z_BUF_ERROR:
     return "zlib error: Z_BUF_ERROR";
   case Z_STREAM_ERROR:
     return "zlib error: Z_STREAM_ERROR";
int Res =
       ::compress2((Bytef *)CompressedBuffer.data(), &CompressedSize,
                   (const Bytef *)InputBuffer.data(), InputBuffer.size(), Level);
   if (Res == Z_MEM_ERROR)
     report_bad_alloc_error("Allocation failed");
   assert(Res == Z_OK);
   // Tell MemorySanitizer that zlib output buffer is fully initialized.
   // This avoids a false report when running LLVM with uninstrumented ZLib.
   __msan_unpoison(CompressedBuffer.data(), CompressedSize);
   CompressedBuffer.truncate(CompressedSize);
 }
 
 Error zlib::uncompress(StringRef InputBuffer, char *UncompressedBuffer,
                        size_t &UncompressedSize) {
   int Res =
       ::uncompress((Bytef *)UncompressedBuffer, (uLongf *)&UncompressedSize,
                    (const Bytef *)InputBuffer.data(), InputBuffer.size());
   // Tell MemorySanitizer that zlib output buffer is fully initialized.
   // This avoids a false report when running LLVM with uninstrumented ZLib.
   __msan_unpoison(UncompressedBuffer, UncompressedSize);
   return Res ? createError(convertZlibCodeToString(Res)) : Error::success();
 }
 
 Error zlib::uncompress(StringRef InputBuffer,
                        SmallVectorImpl<char> &UncompressedBuffer,
                        size_t UncompressedSize) {
   UncompressedBuffer.resize_for_overwrite(UncompressedSize);
   Error E =
-      uncompress(InputBuffer, UncompressedBuffer.data(), UncompressedSize);
+      zlib::uncompress(InputBuffer, UncompressedBuffer.data(), UncompressedSize);
   UncompressedBuffer.truncate(UncompressedSize);
   return E;
 }
 
 uint32_t zlib::crc32(StringRef Buffer) {
   return ::crc32(0, (const Bytef *)Buffer.data(), Buffer.size());
 }
 
 #else
Error zlib::uncompress(StringRef InputBuffer,
                        SmallVectorImpl<char> &UncompressedBuffer,
                        size_t UncompressedSize) {
   llvm_unreachable("zlib::uncompress is unavailable");
 }
 uint32_t zlib::crc32(StringRef Buffer) {
   llvm_unreachable("zlib::crc32 is unavailable");
 }
 #endif
+
+#if LLVM_ENABLE_ZSTD
+
+bool zstd::isAvailable() { return true; }
+
+void zstd::compress(StringRef InputBuffer,
+                    SmallVectorImpl<char> &CompressedBuffer, int Level) {
+  unsigned long CompressedBufferSize = ::ZSTD_compressBound(InputBuffer.size());
+  CompressedBuffer.resize_for_overwrite(CompressedBufferSize);
+  unsigned long CompressedSize =
+      ::ZSTD_compress((Bytef *)CompressedBuffer.data(), CompressedBufferSize,
+                  (const Bytef *)InputBuffer.data(), InputBuffer.size(), Level);
+  if (ZSTD_isError(CompressedSize))
+    report_bad_alloc_error("Allocation failed");
+  // assert(Res == Z_OK);
+  // Tell MemorySanitizer that zstd output buffer is fully initialized.
+  // This avoids a false report when running LLVM with uninstrumented ZLib.
+  __msan_unpoison(CompressedBuffer.data(), CompressedSize);
+  CompressedBuffer.truncate(CompressedSize);
+}
+
+Error zstd::uncompress(StringRef InputBuffer, char *UncompressedBuffer,
+                       size_t &UncompressedSize) {
+  unsigned long long const rSize = ZSTD_getFrameContentSize((const Bytef *)InputBuffer.data(),  InputBuffer.size());
+  size_t const Res =
+      ::ZSTD_decompress((Bytef *)UncompressedBuffer, rSize,
+                   (const Bytef *)InputBuffer.data(), InputBuffer.size());
+  UncompressedSize=Res;
+  // Tell MemorySanitizer that zstd output buffer is fully initialized.
+  // This avoids a false report when running LLVM with uninstrumented ZLib.
+  __msan_unpoison(UncompressedBuffer, UncompressedSize);
+  return Res!=rSize ? createError(ZSTD_getErrorName(Res)) : Error::success();
+}
+
+Error zstd::uncompress(StringRef InputBuffer,
+                       SmallVectorImpl<char> &UncompressedBuffer,
+                       size_t UncompressedSize) {
+  UncompressedBuffer.resize_for_overwrite(UncompressedSize);
+  Error E =
+      zstd::uncompress(InputBuffer, UncompressedBuffer.data(), UncompressedSize);
+  UncompressedBuffer.truncate(UncompressedSize);
+  return E;
+}
+
+uint32_t zstd::crc32(StringRef Buffer) {
+  return ::crc32(0, (const Bytef *)Buffer.data(), Buffer.size());
+}
+
+#else
+bool zstd::isAvailable() { return false; }
+void zstd::compress(StringRef InputBuffer,
+                    SmallVectorImpl<char> &CompressedBuffer, int Level) {
+  llvm_unreachable("zstd::compress is unavailable");
+}
+Error zstd::uncompress(StringRef InputBuffer, char *UncompressedBuffer,
+                       size_t &UncompressedSize) {
+  llvm_unreachable("zstd::uncompress is unavailable");
+}
+Error zstd::uncompress(StringRef InputBuffer,
+                       SmallVectorImpl<char> &UncompressedBuffer,
+                       size_t UncompressedSize) {
+  llvm_unreachable("zstd::uncompress is unavailable");
+}
+uint32_t zstd::crc32(StringRef Buffer) {
+  llvm_unreachable("zstd::crc32 is unavailable");
+}
+#endif

llvm/tools/llvm-mc/llvm-mc.cpp

int main(int argc, char **argv) {
 
   std::unique_ptr<MCAsmInfo> MAI(
       TheTarget->createMCAsmInfo(*MRI, TripleName, MCOptions));
   assert(MAI && "Unable to create target asm info!");
 
   MAI->setRelaxELFRelocations(RelaxELFRel);
 
   if (CompressDebugSections != DebugCompressionType::None) {
-    if (!zlib::isAvailable()) {
+    if (!compression::zlib::isAvailable()) {
       WithColor::error(errs(), ProgName)
           << "build tools with zlib to enable -compress-debug-sections";
       return 1;
     }
     MAI->setCompressDebugSections(CompressDebugSections);
   }
   MAI->setPreserveAsmComments(PreserveComments);
 

llvm/tools/llvm-objcopy/ObjcopyOptions.cpp

if (Arg->getOption().getID() == OBJCOPY_compress_debug_sections_eq) {
       if (Config.CompressionType == DebugCompressionType::None)
         return createStringError(
             errc::invalid_argument,
             "invalid or unsupported --compress-debug-sections format: %s",
             InputArgs.getLastArgValue(OBJCOPY_compress_debug_sections_eq)
                 .str()
                 .c_str());
     }
-    if (!zlib::isAvailable())
+    if (!compression::elf::isAvailable())
       return createStringError(
           errc::invalid_argument,
           "LLVM was not compiled with LLVM_ENABLE_ZLIB: can not compress");
   }
 
   Config.AddGnuDebugLink = InputArgs.getLastArgValue(OBJCOPY_add_gnu_debuglink);
   // The gnu_debuglink's target is expected to not change or else its CRC would
   // become invalidated and get rejected. We can avoid recalculating the
objcopy::parseObjcopyOptions(ArrayRef<const char *> RawArgsArr,
   if (Config.DecompressDebugSections &&
       Config.CompressionType != DebugCompressionType::None) {
     return createStringError(
         errc::invalid_argument,
         "cannot specify both --compress-debug-sections and "
         "--decompress-debug-sections");
   }
 
-  if (Config.DecompressDebugSections && !zlib::isAvailable())
+  if (Config.DecompressDebugSections && !compression::elf::isAvailable())
     return createStringError(
         errc::invalid_argument,
         "LLVM was not compiled with LLVM_ENABLE_ZLIB: cannot decompress");
 
   if (Config.ExtractPartition && Config.ExtractMainPartition)
     return createStringError(errc::invalid_argument,
                              "cannot specify --extract-partition together with "
                              "--extract-main-partition");

llvm/unittests/ProfileData/InstrProfTest.cpp

for (int I = 0; I < 3; I++) {
     OS << "BlahblahBlahblahBar_" << I;
     FuncNames2.push_back(OS.str());
   }
 
   for (bool DoCompression : {false, true}) {
     // Compressing:
     std::string FuncNameStrings1;
     EXPECT_THAT_ERROR(collectPGOFuncNameStrings(
-                          FuncNames1, (DoCompression && zlib::isAvailable()),
+                          FuncNames1, (DoCompression && compression::internal::isAvailable()),
                           FuncNameStrings1),
                       Succeeded());
 
     // Compressing:
     std::string FuncNameStrings2;
     EXPECT_THAT_ERROR(collectPGOFuncNameStrings(
-                          FuncNames2, (DoCompression && zlib::isAvailable()),
+                          FuncNames2, (DoCompression && compression::internal::isAvailable()),
                           FuncNameStrings2),
                       Succeeded());
 
     for (int Padding = 0; Padding < 2; Padding++) {
       // Join with paddings :
       std::string FuncNameStrings = FuncNameStrings1;
       for (int P = 0; P < Padding; P++) {
         FuncNameStrings.push_back('\0');

llvm/unittests/Support/CompressionTest.cpp

 #include "llvm/Support/Compression.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Config/config.h"
 #include "llvm/Support/Error.h"
 #include "gtest/gtest.h"
 
 using namespace llvm;
+using namespace compression;
 
 namespace {
 
 #if LLVM_ENABLE_ZLIB
 
 void TestZlibCompression(StringRef Input, int Level) {
   SmallString<32> Compressed;
   SmallString<32> Uncompressed;

 TEST(CompressionTest, ZlibCRC32) {
   EXPECT_EQ(
       0x414FA339U,
       zlib::crc32(StringRef("The quick brown fox jumps over the lazy dog")));
 }
 
 #endif
 
+#if LLVM_ENABLE_ZSTD
+
+void TestZstdCompression(StringRef Input, int Level) {
+  SmallString<32> Compressed;
+  SmallString<32> Uncompressed;
+
+  zstd::compress(Input, Compressed, Level);
+
+  // Check that uncompressed buffer is the same as original.
+  Error E = zstd::uncompress(Compressed, Uncompressed, Input.size());
+  consumeError(std::move(E));
+
+  EXPECT_EQ(Input, Uncompressed);
+  if (Input.size() > 0) {
+    // Uncompression fails if expected length is too short.
+    E = zstd::uncompress(Compressed, Uncompressed, Input.size() - 1);
+    EXPECT_EQ("zstd error: Z_BUF_ERROR", llvm::toString(std::move(E)));
+  }
+}
+
+TEST(CompressionTest, Zstd) {
+  TestZstdCompression("", zstd::DefaultCompression);
+
+  TestZstdCompression("hello, world!", zstd::NoCompression);
+  TestZstdCompression("hello, world!", zstd::BestSizeCompression);
+  TestZstdCompression("hello, world!", zstd::BestSpeedCompression);
+  TestZstdCompression("hello, world!", zstd::DefaultCompression);
+
+  const size_t kSize = 1024;
+  char BinaryData[kSize];
+  for (size_t i = 0; i < kSize; ++i) {
+    BinaryData[i] = i & 255;
+  }
+  StringRef BinaryDataStr(BinaryData, kSize);
+
+  TestZstdCompression(BinaryDataStr, zstd::NoCompression);
+  TestZstdCompression(BinaryDataStr, zstd::BestSizeCompression);
+  TestZstdCompression(BinaryDataStr, zstd::BestSpeedCompression);
+  TestZstdCompression(BinaryDataStr, zstd::DefaultCompression);
+}
+
+TEST(CompressionTest, ZstdCRC32) {
+  EXPECT_EQ(
+      0x414FA339U,
+      zstd::crc32(StringRef("The quick brown fox jumps over the lazy dog")));
+}
+
+#endif
+
 }

utils/bazel/llvm_configs/llvm-config.h.cmake

 #cmakedefine LLVM_WITH_Z3 ${LLVM_WITH_Z3}
 
 /* Define if we have curl and want to use it */
 #cmakedefine LLVM_ENABLE_CURL ${LLVM_ENABLE_CURL}
 
 /* Define if zlib compression is available */
 #cmakedefine01 LLVM_ENABLE_ZLIB
 
+/* Define if zstd compression is available */
+#cmakedefine01 LLVM_ENABLE_ZSTD
+
 /* Define if LLVM was built with a dependency to the libtensorflow dynamic library */
 #cmakedefine LLVM_HAVE_TF_API
 
 /* Define to 1 if you have the <sysexits.h> header file. */
 #cmakedefine HAVE_SYSEXITS_H ${HAVE_SYSEXITS_H}
 
 /* Define if the xar_open() function is supported on this platform. */
 #cmakedefine LLVM_HAVE_LIBXAR ${LLVM_HAVE_LIBXAR}