This is an archive of the discontinued LLVM Phabricator instance.

cmake: Enable 64bit off_t on 32bit glibc systems
ClosedPublic

Authored by raj.khem on Dec 9 2022, 4:53 PM.

Details

Summary

Pass -D_FILE_OFFSET_BITS=64 to compiler flags on 32bit glibc based
systems. This will make sure that 64bit versions of LFS functions are
used e.g. seek will behave same as lseek64. Also revert [1] partially
because this added a cmake test to detect lseek64 but then forgot to
pass the needed macro to actual compile, this test was incomplete too
since libc implementations like musl has 64bit off_t by default on 32bit
systems and does not bundle[2] -D_LARGEFILE64_SOURCE under -D_GNU_SOURCE
like glibc, which means the compile now fails on musl because the cmake
check passes but we do not have _LARGEFILE64_SOURCE defined. Using the
*64 function was transitional anyways so use -D_FILE_OFFSET_BITS=64
instead

[1] https://github.com/llvm/llvm-project/commit/8db7e5e4eed4c4e697dc3164f2c9351d8c3e942b
[2] https://git.musl-libc.org/cgit/musl/commit/?id=25e6fee27f4a293728dd15b659170e7b9c7db9bc

Diff Detail

Event Timeline

raj.khem created this revision.Dec 9 2022, 4:53 PM
Herald added a project: Restricted Project. · View Herald TranscriptDec 9 2022, 4:53 PM
raj.khem requested review of this revision.Dec 9 2022, 4:53 PM
MaskRay accepted this revision.Dec 17 2022, 12:49 PM
MaskRay added subscribers: brad, dim.

LGTM

(FYI @dim @brad)

This revision is now accepted and ready to land.Dec 17 2022, 12:49 PM
This revision was automatically updated to reflect the committed changes.
dim added a comment.Dec 17 2022, 4:55 PM

FWIW the BSDs never had lseek64; off_t was 64 bit from the original BSD 4.4 Lite sources... :)

srj added a subscriber: srj.Dec 19 2022, 4:34 PM

This change seems to have injected a failure into Halide builds on 32-bit Linux: we now fail with <command-line>:0:70: error: token "=" is not valid in preprocessor expressions; examination of our CMake-based build command line shows that it is something like:

cmake -E env CCACHE_CPP2=yes CCACHE_HASHDIR=yes CCACHE_SLOPPINESS=pch_defines /usr/bin/ccache /usr/bin/g++-7  -m32 -DHALIDE_ENABLE_RTTI -DHALIDE_WITH_EXCEPTIONS -DHalide_EXPORTS -DLLVM_VERSION=160 -DWITH_AARCH64 -DWITH_ARM -DWITH_D3D12 -DWITH_HEXAGON -DWITH_INTROSPECTION -DWITH_METAL -DWITH_MIPS -DWITH_NVPTX -DWITH_OPENCL -DWITH_OPENGLCOMPUTE -DWITH_POWERPC -DWITH_RISCV -DWITH_WABT -DWITH_WEBASSEMBLY -DWITH_X86 -D_GNU_SOURCE -D_FILE_OFFSET_BITS="64 -D_DEBUG -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS" -I/home/halidenightly/build_bot/worker/llvm-16-x86-32-linux/llvm-install/include -isystem /home/halidenightly/build_bot/worker/halide-testbranch-main-llvm16-x86-32-linux-cmake/halide-build/_deps/wabt-src/include -isystem /home/halidenightly/build_bot/worker/halide-testbranch-main-llvm16-x86-32-linux-cmake/halide-build/_deps/wabt-build/include -O3 -DNDEBUG -fPIC -Wall -Wcast-qual -Wignored-qualifiers -Woverloaded-virtual -Wsuggest-override -Wdeprecated-declarations -Wno-double-promotion -Wno-float-conversion -Wno-float-equal -Wno-missing-field-initializers -Wno-old-style-cast -Wno-shadow -Wno-sign-conversion -Wno-switch-enum -Wno-undef -Wno-unused-function -Wno-unused-macros -Wno-unused-parameter -pthread -std=c++1z -MD -MT src/CMakeFiles/Halide.dir/AddParameterChecks.cpp.o -MF src/CMakeFiles/Halide.dir/AddParameterChecks.cpp.o.d -o src/CMakeFiles/Halide.dir/AddParameterChecks.cpp.o -c /home/halidenightly/build_bot/worker/halide-testbranch-main-llvm16-x86-32-linux-cmake/halide-source/src/AddParameterChecks.cpp

Look closely at the _FILE_OFFSET_BITS section:

-D_FILE_OFFSET_BITS="64 -D_DEBUG -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DSTDC_CONSTANT_MACROS -DSTDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS"

...I'm not sure how or why we're getting this bizarre failure, but, uh, this is pretty bizarre. Has this been tested with a project that also uses CMake?

(FYI, I can repeat this running cmake 3.24.2 on x86-64 linux)

alexreinking added inline comments.
llvm/cmake/config-ix.cmake
342

I believe the problem @srj encountered is caused by this line. The add_definitions command is bug-prone and was superseded by add_compile_definitions (correctly used a few lines up) in CMake 3.12.

srj added a comment.Dec 20 2022, 9:23 AM

This change continues to break all 32-bit builds of Halide (and probably anything else that consumes LLVM via CMake); please revert ASAP until a fix can be found.

This change continues to break all 32-bit builds of Halide (and probably anything else that consumes LLVM via CMake); please revert ASAP until a fix can be found.

I applied @alexreinking's suggestion. Feel free to revert if errors still continue.

srj added a comment.Dec 20 2022, 9:39 AM

This change continues to break all 32-bit builds of Halide (and probably anything else that consumes LLVM via CMake); please revert ASAP until a fix can be found.

I applied @alexreinking's suggestion. Feel free to revert if errors still continue.

I tried @alexreinking's suggestion locally, and it remains broken in the same way. This change should be reverted for now.

MaskRay reopened this revision.Dec 20 2022, 12:54 PM
This revision is now accepted and ready to land.Dec 20 2022, 12:54 PM
MaskRay requested changes to this revision.Dec 20 2022, 12:55 PM
This revision now requires changes to proceed.Dec 20 2022, 12:55 PM
MaskRay updated this revision to Diff 484363.Dec 20 2022, 12:56 PM
MaskRay edited the summary of this revision. (Show Details)

include gn and bazel changes

MaskRay added a comment.EditedJan 6 2023, 11:46 PM

This change continues to break all 32-bit builds of Halide (and probably anything else that consumes LLVM via CMake); please revert ASAP until a fix can be found.

I applied @alexreinking's suggestion. Feel free to revert if errors still continue.

I tried @alexreinking's suggestion locally, and it remains broken in the same way. This change should be reverted for now.

I installed =sys-libs/musl-9999 (git master) on my x86_64-gentoo-linux-musl machine today and noticed that llvm/lib/Support/raw_ostream.cpp fails to compile which can be fixed by this patch.
If you could help figure out the 32-bit Halide issue to unblock this patch it will be great....

https://www.openwall.com/lists/musl/2022/09/26/1 "Revisiting LFS64 removal"

@srj Will you be able to help figure out the 32-bit Halide issue? Thanks in advance:)

srj added a comment.Jan 11 2023, 4:10 PM

@srj Will you be able to help figure out the 32-bit Halide issue? Thanks in advance:)

Yes, I can take a look; what is the current status of this -- is there an updated patch or should I just reapply this?

llvm/cmake/config-ix.cmake
342

I tried replacing that line with add_compile_definitions(_FILE_OFFSET_BITS=64) and I still fail locally in exactly the same way.

MaskRay added a comment.EditedJan 11 2023, 4:13 PM

@srj Will you be able to help figure out the 32-bit Halide issue? Thanks in advance:)

Yes, I can take a look; what is the current status of this -- is there an updated patch or should I just reapply this?

This webpage shows the reverse of the revert.
You can use curl -L 'https://reviews.llvm.org/D139752?download=1' | patch -p1 to apply the changes.

Applying this patch fixes my llvm-project build on my Gentoo machine using =sys-libs/musl-9999 (git master). We probably want to fix these issues so that the next release llvm-project 16.0.0 will be buildable with the next release of musl (1.2.4).

srj added a comment.Jan 12 2023, 12:36 PM

FYI, before I could start looking into this, it looks like a more-recent change (https://reviews.llvm.org/D141051) has injected another 32-bit-specific breakage (at least when building under gcc-7) -- I'm investigating that first :-/

srj added a comment.EditedJan 17 2023, 11:16 AM

You can use curl -L 'https://reviews.llvm.org/D139752?download=1' | patch -p1 to apply the changes.

Applying this patch fixes my llvm-project build on my Gentoo machine using =sys-libs/musl-9999 (git master). We probably want to fix these issues so that the next release llvm-project 16.0.0 will be buildable with the next release of musl (1.2.4).

Sorry for the long delay. Trying this at today's top of tree fails for me with <command-line>: error: token "=" is not valid in preprocessor expressions... looking at the commandline, I see that we now have -D_FILE_OFFSET_BITS="64 (yes, missing a trailing quote) on the command line. Not sure how/why. (Note that this is build using CMake with Ninja as the Generator, not Make.)

EDIT: this definition actually appears *twice* -- once unquoted, once half-quoted:

-D_FILE_OFFSET_BITS="64 -D_DEBUG -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

srj added a comment.Jan 17 2023, 11:39 AM

-D_FILE_OFFSET_BITS="64 -D_DEBUG -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

From experimentation, it seems that its add_compile_definitions(_FILE_OFFSET_BITS=64) that is the culprit; commenting it out allows us to build and run correctly. (Why it's failing in this way is not at all clear to me, but then, CMake has a few quirks here and there...)

Why do we need both these definitions, BTW?

srj added a comment.Jan 17 2023, 11:46 AM

From experimentation, it seems that its add_compile_definitions(_FILE_OFFSET_BITS=64) that is the culprit; commenting it out allows us to build and run correctly. (Why it's failing in this way is not at all clear to me, but then, CMake has a few quirks here and there...)

This is even stranger, actually... the full cmake commandline I'm getting is

/usr/bin/cmake -E env CCACHE_CPP2=yes CCACHE_HASHDIR=yes CCACHE_SLOPPINESS=pch_defines /usr/bin/ccache /usr/bin/g++ -m32 -DHALIDE_ENABLE_RTTI -DHALIDE_WITH_EXCEPTIONS -DHalide_EXPORTS -DLLVM_VERSION=160 -DWITH_AARCH64 -DWITH_ARM -DWITH_D3D12 -DWITH_HEXAGON -DWITH_INTROSPECTION -DWITH_METAL -DWITH_MIPS -DWITH_NVPTX -DWITH_OPENCL -DWITH_OPENGLCOMPUTE -DWITH_POWERPC -DWITH_WABT -DWITH_WEBASSEMBLY -DWITH_X86 -D_GNU_SOURCE -D_FILE_OFFSET_BITS="64 -D_DEBUG -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS" -I/usr/local/google/home/srj/llvm-16-install-32/include -isystem /usr/local/google/home/srj/GitHub/Halide/build-32/_deps/wabt-src/include -isystem /usr/local/google/home/srj/GitHub/Halide/build-32/_deps/wabt-build/include -O3 -DNDEBUG -fPIC -Wall -Wcast-qual -Wignored-qualifiers -Woverloaded-virtual -Wsuggest-override -Wdeprecated-declarations -Wno-double-promotion -Wno-float-conversion -Wno-float-equal -Wno-missing-field-initializers -Wno-old-style-cast -Wno-shadow -Wno-sign-conversion -Wno-switch-enum -Wno-undef -Wno-unused-function -Wno-unused-macros -Wno-unused-parameter -std=c++17 -MD -MT src/CMakeFiles/Halide.dir/BoundsInference.cpp.o -MF src/CMakeFiles/Halide.dir/BoundsInference.cpp.o.d -o src/CMakeFiles/Halide.dir/BoundsInference.cpp.o -c /usr/local/google/home/srj/GitHub/Halide/src/BoundsInference.cpp

Note this section: -D_FILE_OFFSET_BITS="64 -D_DEBUG -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS"

... that is, for some reason, the quoted part is expanding to contain other definitions as well. Never seen this before in CMake.

@raj.khem would you be able to figure out the issue? :) https://llvm.org/ release/16.x will be created soon...

@raj.khem would you be able to figure out the issue? :) https://llvm.org/ release/16.x will be created soon...

I have been trying to reproduce it on my end but have not succeeded, I don't have same setup as @srj, I would also look for some alternative ways to do it so we can avoid the issue at hand which seems some weird cmake thing.

MaskRay added a comment.EditedJan 29 2023, 12:32 PM

@raj.khem would you be able to figure out the issue? :) https://llvm.org/ release/16.x will be created soon...

I have been trying to reproduce it on my end but have not succeeded, I don't have same setup as @srj, I would also look for some alternative ways to do it so we can avoid the issue at hand which seems some weird cmake thing.

@srj It will be nice to figure out the 32-bit glibc Halide issue and fix this for just just-branched-release/16.x so that llvm-project 16.0 can be built with next musl release, otherwise many musl based distributions will have to carry this local patch...

srj added a comment.Jan 30 2023, 9:26 AM

@raj.khem would you be able to figure out the issue? :) https://llvm.org/ release/16.x will be created soon...

I have been trying to reproduce it on my end but have not succeeded, I don't have same setup as @srj, I would also look for some alternative ways to do it so we can avoid the issue at hand which seems some weird cmake thing.

@srj It will be nice to figure out the 32-bit glibc Halide issue and fix this for just just-branched-release/16.x so that llvm-project 16.0 can be built with next musl release, otherwise many musl based distributions will have to carry this local patch...

I wonder if this could be a bug in a specific version(s) of CMake? What version(s) have you tried replicating with? Our buildbots are using v3.22.6 (Halide requires a minimum of 3.22).

@raj.khem would you be able to figure out the issue? :) https://llvm.org/ release/16.x will be created soon...

I have been trying to reproduce it on my end but have not succeeded, I don't have same setup as @srj, I would also look for some alternative ways to do it so we can avoid the issue at hand which seems some weird cmake thing.

@srj It will be nice to figure out the 32-bit glibc Halide issue and fix this for just just-branched-release/16.x so that llvm-project 16.0 can be built with next musl release, otherwise many musl based distributions will have to carry this local patch...

I wonder if this could be a bug in a specific version(s) of CMake? What version(s) have you tried replicating with? Our buildbots are using v3.22.6 (Halide requires a minimum of 3.22).

I am using cmake 3.25.1

srj added a comment.EditedJan 30 2023, 2:52 PM

I wonder if this could be a bug in a specific version(s) of CMake? What version(s) have you tried replicating with? Our buildbots are using v3.22.6 (Halide requires a minimum of 3.22).

I am using cmake 3.25.1

I get the same failure when using CMake 25.1 as well (linux x86-64). I can't explain the failure, but it is 100% reproducible for me on x86-64 Linux using CMake 3.25.1. Try these steps to repeat when building from scratch:

$ export LLVM_INSTALL_DIR=path/to/llvm/install/directory

# Build a local 32-bit build of LLVM (with this patch in place)
$ CC='gcc -m32' \
  CXX='g++ -m32' \
  LD='ld -melf_i386' \
  cmake \
    -D LLVM_BUILD_32_BITS=ON \
    -D CMAKE_FIND_ROOT_PATH=/usr/lib/i386-linux-gnu \
    -D CMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY \
    -D CMAKE_BUILD_TYPE=Release \
    -D CMAKE_INSTALL_PREFIX=${LLVM_INSTALL_DIR} \
    -D LLVM_CCACHE_BUILD=ON \
    -D LLVM_CCACHE_MAXSIZE=20G \
    -D LLVM_ENABLE_ASSERTIONS=ON \
    -D LLVM_ENABLE_CURL=OFF \
    -D LLVM_ENABLE_HTTPLIB=OFF \
    -D LLVM_ENABLE_LIBXML2=OFF \
    -D LLVM_ENABLE_PROJECTS="clang;lld" \
    -D LLVM_ENABLE_RTTI=ON \
    -D LLVM_ENABLE_TERMINFO=OFF \
    -D LLVM_ENABLE_ZLIB=OFF \
    -D LLVM_ENABLE_ZSTD=OFF \
    -D LLVM_ENABLE_OCAMLDOC=OFF \
    -D LLVM_ENABLE_BINDINGS=OFF \
    -D LLVM_ENABLE_IDE=OFF \
    -D LLVM_TARGETS_TO_BUILD="X86;ARM;NVPTX;AArch64;PowerPC;Hexagon;WebAssembly;RISCV" \
    -S path/to/llvm/source/tree \
    -B path/to/llvm/build/directory \
    -G Ninja

$  ninja install

# Clone Halide
$ git clone https://github.com/halide/Halide path/to/halide/source

# Build Halide
$ CC='gcc -m32' \
  CXX='g++ -m32' \
  LD='ld -melf_i386' \
  cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DHalide_CCACHE_BUILD=ON \
    -DLLVM_DIR=${LLVM_INSTALL_DIR}/lib/cmake/llvm \
    -DWITH_PYTHON_BINDINGS=OFF \
    -DHalide_TARGET=x86-32-linux\
    -S path/to/halide/source \
    -B path/to/halide/build-dir \
    -G Ninja

$ ninja Halide
mgorny added a subscriber: mgorny.EditedFeb 4 2023, 9:17 AM

I haven't been able to get Halide to configure successfully yet but I already spotted something suspicious. dependencies/llvm/CMakeLists.txt does:

set(Halide_LLVM_DEFS ${LLVM_DEFINITIONS} $<BUILD_INTERFACE:LLVM_VERSION=${LLVM_VERSION_MAJOR}${LLVM_VERSION_MINOR}>)

then:

list(APPEND Halide_LLVM_DEFS $<BUILD_INTERFACE:${DEFINE}>)

So Halide_LLVM_DEFS becomes a weird mix of semicolon separated entries with space separated entries inside:

-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS
-D__STDC_LIMIT_MACROS;$<BUILD_INTERFACE:LLVM_VERSION=170>;$<BUILD_INTERFACE:WITH_AARCH64>;$<BUILD_INTERFACE:WITH_AMDGPU>;$<BUILD_INTERFACE:WITH_ARM>;$<BUILD_INTERFACE:WITH_HEXAGON>;$<BUILD_INTERFACE:WITH_NVPTX>;$<BUILD_INTERFACE:WITH_POWERPC>;$<BUILD_INTERFACE:WITH_RISCV>;$<BUILD_INTERFACE:WITH_X86>

My suggestion would be to start by trying to replace spaces with semicolons there, so it's a proper list and see if that helps, i.e. after the first set:

string(REPLACE " " ";" Halide_LLVM_DEFS ${Halide_LLVM_DEFS})
MaskRay added a comment.EditedFeb 4 2023, 11:48 AM

I haven't been able to get Halide to configure successfully yet but I already spotted something suspicious. dependencies/llvm/CMakeLists.txt does:

Thanks for the investigation. If it's Halide's issue we should land this change despite breakage of Halide.
Considering the approaching 16.0.0 release, we should not wait too long.

Filed https://github.com/halide/Halide/issues/7319 for the Halide issue. We should proceed with this patch. llvm-project has already waited too long. (Downstream problems are their own. llvm-project has no obligation waiting for a downstream. Sometimes llvm-project developers may wait a bit but that is more of a courtesy not an obligation)

MaskRay accepted this revision.Feb 4 2023, 1:38 PM
This revision is now accepted and ready to land.Feb 4 2023, 1:38 PM
This revision was automatically updated to reflect the committed changes.

With this patch, the following line appears in the generated LLVMConfig.cmake file:

set(LLVM_DEFINITIONS "-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_DEBUG -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS")

Notice that -D_FILE_OFFSET_BITS=64 appears twice. Removing either copy fixes Halide's build. So does converting LLVM_DEFINITIONS to a typical CMake list via separate_arguments (which we now have an open PR to do). Somehow, there is a bad interaction between the space/semicolon mixing and CMake's argument de-duplication. I haven't minimized this yet, but at least this explains why this patch causes this problem.

This patch might cause problems for other downstreams that (reasonably, but incorrectly) assume that LLVM_DEFINITIONS is a typical CMake list rather than a space-separated list.

srj added a comment.Feb 6 2023, 11:11 AM

This patch might cause problems for other downstreams that (reasonably, but incorrectly) assume that LLVM_DEFINITIONS is a typical CMake list rather than a space-separated list.

In general, trying to have CMake lists separated by spaces (or really, anything other than a semicolon) is a recipe for flakiness and pain; this is regrettable, but a fact of life with CMake, unfortunately. I'm tempted to suggest that LLVM considers converting it to a 'traditional' CMake list, but that might cause even more dislocation/problems from code that assumes otherwise...

Somehow, there is a bad interaction between the space/semicolon mixing and CMake's argument de-duplication.

Actually, upon closer inspection, it has to do with the presence of an = in the list. When there's an =, it seems that CMake does not split on spaces automatically anymore. We've never built Halide on AIX or SunOS, which are the only other build scenarios where an = can appear in LLVM_DEFINITIONS.

I'm tempted to suggest that LLVM considers converting it to a 'traditional' CMake list, but that might cause even more dislocation/problems from code that assumes otherwise...

This would definitely be the Right Thing to do, given that every other CMake package I've seen (including all the first-party find modules) use semicolon separators in their ${PACKAGE_NAME}_DEFINITIONS variables. Whether or not it's practical is a separate question.