Page MenuHomePhabricator

Add clang headers that fix machine-dependent definitions on FreeBSD 9.2
ClosedPublic

Authored by kutuzov.viktor.84 on May 26 2014, 3:52 AM.

Details

Summary

This patch fixes some types and macros on x86-64 FreeBSD 9.2. These types and macros are defined incorrectly in 32-bit mode and thus prevent i386 sanitizers tests from passing.

Diff Detail

Repository
rL LLVM

Event Timeline

kutuzov.viktor.84 retitled this revision from to Add clang headers that fix machine-dependent definitions on FreeBSD 9.2.
kutuzov.viktor.84 updated this object.
kutuzov.viktor.84 edited the test plan for this revision. (Show Details)
kutuzov.viktor.84 added reviewers: kcc, samsonov.
kutuzov.viktor.84 added subscribers: Unknown Object (MLST), emaste.
kcc edited edge metadata.May 26 2014, 4:22 AM
kcc added a subscriber: klimek.

You need to send this review to someone in the Clang land

klimek added a subscriber: rsmith.May 26 2014, 10:43 AM

I'm definitely the wrong person to review this...

+rsmith

Is there a FreeBSD bug filed for this?

emaste added a comment.Jun 9 2014, 8:55 AM

http://bugs.freebsd.org/186247 has been fixed in stable/9 and will be in 9.3

One issue reported in http://bugs.freebsd.org/186958 is still open but the others are now not reproducible.

The remaining issue in http://bugs.freebsd.org/186958 is that id_t is an int64_t. This is not an -m32 bug; id_t is indeed 64-bit on all of our platforms.

Thanks, Ed. This diff doesn't affect id_t so it should be OK as is in this regard.

theraven edited edge metadata.Jul 1 2014, 5:44 AM

Looks fine to me (with my FreeBSD hat on).

OK, I'm going to commit the patch, if there are no objections.

kutuzov.viktor.84 updated this revision to Diff 11244.

Closed by commit rL212689 (authored by vkutuzov).

Committed with a typo fixed in the title of _types.h .

This commit seems to break compilation with clang on Mac OS X.

My environment:
Mac OS X 10.9.4
CMake 3.0.0
clang/llvm/libcxx/extras all trunk

Build using:

cmake -G "Ninja" -DCMAKE_BUILD_TYPE:STRING="RelWithDebInfo" -DCMAKE_INSTALL_PREFIX:PATH="/Users/tobias/llvm" -DCMAKE_CXX_FLAGS:STRING="-DLLVM_ENABLE_DUMP" LLVM_BUILD_DOCS:BOOL=ON ../llvm
ninja 
ninja install

Test case:

#include <vector>

int main() { return 0; }

Test build (using clang trunk, preprocessed code https://dl.dropboxusercontent.com/u/14672346/tmp/clang-include-test_trunk.s ):

tobias@ruediger(2) /tmp> ~/llvm/bin/clang++ -v -o clang-include-test clang-include-text.cpp 
clang version 3.5.0 
Target: x86_64-apple-darwin13.3.0
Thread model: posix
 "/Users/tobias/llvm/bin/clang-3.5" -cc1 -triple x86_64-apple-macosx10.9.0 -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -main-file-name clang-include-text.cpp -mrelocation-model pic -pic-level 2 -mdisable-fp-elim -masm-verbose -munwind-tables -target-cpu core2 -v -dwarf-column-info -resource-dir /Users/tobias/llvm/bin/../lib/clang/3.5.0 -stdlib=libc++ -fdeprecated-macro -fdebug-compilation-dir /private/tmp -ferror-limit 19 -fmessage-length 204 -stack-protector 1 -mstackrealign -fblocks -fobjc-runtime=macosx-10.9.0 -fencode-extended-block-signature -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -o /var/folders/qm/7pz79pjd5gg0h1w3dny4yl540000gn/T/clang-include-text-f12daf.o -x c++ clang-include-text.cpp
clang -cc1 version 3.5.0 based upon LLVM 3.5.0svn default target x86_64-apple-darwin13.3.0
ignoring nonexistent directory "/usr/include/c++/v1"
#include "..." search starts here:
#include <...> search starts here:
 /Users/tobias/llvm/bin/../include/c++/v1
 /usr/local/include
 /Users/tobias/llvm/bin/../lib/clang/3.5.0/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
In file included from clang-include-text.cpp:1:
In file included from /Users/tobias/llvm/bin/../include/c++/v1/vector:265:
In file included from /Users/tobias/llvm/bin/../include/c++/v1/__bit_reference:15:
In file included from /Users/tobias/llvm/bin/../include/c++/v1/algorithm:628:
In file included from /Users/tobias/llvm/bin/../include/c++/v1/memory:604:
In file included from /Users/tobias/llvm/bin/../include/c++/v1/iterator:335:
In file included from /Users/tobias/llvm/bin/../include/c++/v1/iosfwd:90:
In file included from /usr/include/wchar.h:92:
In file included from /usr/include/_wctype.h:47:
/usr/include/_types/_wctype_t.h:31:9: error: unknown type name '__darwin_wctype_t'
typedef __darwin_wctype_t wctype_t;
        ^
1 error generated.

Test build (using system clang, preprocessed code https://dl.dropboxusercontent.com/u/14672346/tmp/clang-include-test_system.s ):

tobias@ruediger(2) /tmp> clang++ -v -o clang-include-test clang-include-text.cpp 
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.3.0
Thread model: posix
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.9.0 -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -main-file-name clang-include-text.cpp -mrelocation-model pic -pic-level 2 -mdisable-fp-elim -masm-verbose -munwind-tables -target-cpu core2 -target-linker-version 236.3 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/5.1 -stdlib=libc++ -fdeprecated-macro -fdebug-compilation-dir /private/tmp -ferror-limit 19 -fmessage-length 204 -stack-protector 1 -mstackrealign -fblocks -fobjc-runtime=macosx-10.9.0 -fencode-extended-block-signature -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -vectorize-slp -o /var/folders/qm/7pz79pjd5gg0h1w3dny4yl540000gn/T/clang-include-text-04db85.o -x c++ clang-include-text.cpp
clang -cc1 version 5.1 based upon LLVM 3.4svn default target x86_64-apple-darwin13.3.0
ignoring nonexistent directory "/usr/include/c++/v1"
#include "..." search starts here:
#include <...> search starts here:
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/5.1/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -dynamic -arch x86_64 -macosx_version_min 10.9.0 -o clang-include-test /var/folders/qm/7pz79pjd5gg0h1w3dny4yl540000gn/T/clang-include-text-04db85.o -lc++ -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/5.1/lib/darwin/libclang_rt.osx.a

Just search for __darwin_wctype_t in the preprocessed code. It's defined for system clang but not for clang trunk.

Tobias, can you please provide the preprocessed output for which the compilation fails? Thanks.

Tobias, can you please provide the preprocessed output for which the compilation fails? Thanks.

Already did. See above.

Test build (using clang trunk, preprocessed code https://dl.dropboxusercontent.com/u/14672346/tmp/clang-include-test_trunk.s ):

The link is for the preprocessed output. Searching for __darwin_wctype_t shows it being used in a typedef but never itself being defined.

Right, somehow missed it.

The trunk output reads:

...
# 1 "/Users/tobias/llvm/bin/../lib/clang/3.5.0/include/_types.h" 1 3 4
# 62 "/Users/tobias/llvm/bin/../lib/clang/3.5.0/include/_types.h" 3 4
# 1 "/usr/include/machine/_types.h" 1 3 4
# 32 "/usr/include/machine/_types.h" 3 4
...

what means 'machine/_types.h' has been copied to 'include/_types.h' and not 'include/machine/_types.h' as it should be.

Does your test source compile successfully if you move _types.h under /machine manually?

The code that copies the headers is in clang/lib/Headers/CMakeLists.txt:

set(out_files)
foreach( f ${files} )
  set( src ${CMAKE_CURRENT_SOURCE_DIR}/${f} )
  set( dst ${output_dir}/${f} )
  add_custom_command(OUTPUT ${dst}
    DEPENDS ${src}
    COMMAND ${CMAKE_COMMAND} -E copy_if_different ${src} ${dst}
    COMMENT "Copying clang's ${f}...")
  list(APPEND out_files ${dst})
endforeach( f )

So it seems like cmake's copy_if_different doesn't create new directories on Darwin. Can you please confirm that?

Does your test source compile successfully if you move _types.h under /machine manually?

Yes. I created the 'machine' directory at /Users/tobias/llvm/lib/clang/3.5.0/include and moved _stdint.h and _types.h into that new directory. Then my test source compiles just fine.

So it seems like cmake's copy_if_different doesn't create new directories on Darwin. Can you please confirm that?

Well, apparently it does not keep the hierarchy of your specified files when copying to the target dir.

Here some pointers from #cmake:

[23:00:51] <+ngladitz> hm I wouldn't expect it to append the source path to the target path
[23:01:16] <TobiasFar> apparently it does on linux..at least that's what other llvm devs have told me
[23:01:36] <+ngladitz> I am on linux
[23:01:41] <TobiasFar> ph
[23:01:43] <TobiasFar> *oh
[23:04:16] <+ngladitz> as far as I can see in the implementation it specifically appends just the file name to the destination path (if it is a directory)
[23:05:18] <TobiasFar> so the hierarchy is basically flattend? if one has source file src/a/b/c/filename.txt it'll copy it to target/filename.txt
[23:07:20] <+ngladitz> just like every other command line copy command I know
[23:07:24] <+ngladitz> http://cmake.org/gitweb?p=cmake.git;a=blob;f=Source/kwsys/SystemTools.cxx;h=db94510730aa7c0e8259d8fcc3ed6e23065b562c;hb=refs/heads/master#l2013

This is what a cmake trace during install showed:

/Users/tobias/dev/rep/llvm_build/tools/clang/lib/cmake_install.cmake(32): include(/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake )
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(4): if(NOT DEFINED CMAKE_INSTALL_PREFIX )
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(7): string(REGEX REPLACE /$ CMAKE_INSTALL_PREFIX ${CMAKE_INSTALL_PREFIX} )
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(10): if(NOT DEFINED CMAKE_INSTALL_CONFIG_NAME )
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(21): if(NOT CMAKE_INSTALL_COMPONENT )
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(22): if(COMPONENT )
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(25): else()
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(26): set(CMAKE_INSTALL_COMPONENT )
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(30): if(NOT CMAKE_INSTALL_COMPONENT OR ${CMAKE_INSTALL_COMPONENT} STREQUAL Unspecified )
/Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/cmake_install.cmake(31): file(INSTALL DESTINATION ${CMAKE_INSTALL_PREFIX}/lib/clang/3.5.0/include TYPE FILE PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ WORLD_READ FILES /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/machine/_stdint.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/machine/_types.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/altivec.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/ammintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/arm_acle.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/avxintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/avx2intrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/bmiintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/bmi2intrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/emmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/f16cintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/float.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/fma4intrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/fmaintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/ia32intrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/immintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/iso646.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/Intrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/limits.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/lzcntintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/mm3dnow.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/mmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/mm_malloc.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/nmmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/pmmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/popcntintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/prfchwintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/rdseedintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/rtmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/shaintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/smmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/stdalign.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/stdarg.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/stdbool.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/stddef.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/stdint.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/stdnoreturn.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/tbmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/tgmath.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/tmmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/varargs.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/wmmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/__wmmintrin_aes.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/__wmmintrin_pclmul.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/x86intrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/xmmintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/xopintrin.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/cpuid.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/unwind.h /Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/module.modulemap /Users/tobias/dev/rep/llvm_build/tools/clang/lib/Headers/arm_neon.h )

  • Installing: /Users/tobias/llvm/lib/clang/3.5.0/include/_stdint.h
  • Installing: /Users/tobias/llvm/lib/clang/3.5.0/include/_types.h

This SO page could also be of interest:

http://stackoverflow.com/questions/11096471/how-can-i-install-a-hierarchy-of-files-using-cmake

[23:05:18] <TobiasFar> so the hierarchy is basically flattend? if one has source file src/a/b/c/filename.txt it'll copy it to target/filename.txt
[23:07:20] <+ngladitz> just like every other command line copy command I know
[23:07:24] <+ngladitz> http://cmake.org/gitweb?p=cmake.git;a=blob;f=Source/kwsys/SystemTools.cxx;h=db94510730aa7c0e8259d8fcc3ed6e23065b562c;hb=refs/heads/master#l2013

According to cmake 3.0.0 sources and the HEAD version referenced above, SystemTools::CopyFileIfDifferent() leverages SystemTools::CopyFileAlways() which relies on SystemTools::MakeDirectory() which, in turn, attempts to create each of the specified subdirectories explicitly:

while((pos = dir.find('/', pos)) != kwsys_stl::string::npos)
  {
  topdir = dir.substr(0, pos);
  Mkdir(topdir.c_str());
  pos++;
  }
topdir = dir;
if(Mkdir(topdir.c_str()) != 0)

http://cmake.org/gitweb?p=cmake.git;a=blob;f=Source/kwsys/SystemTools.cxx;h=db94510730aa7c0e8259d8fcc3ed6e23065b562c;hb=refs/heads/master#l655

I tested it with Linux cmake 2.8.12.2, FreeBSD cmake 2.8.12.1, Linux cmake 3.0.0 and Darwin cmake 3.0.0--all work the same way creating all the specified subdirectories.

Well, apparently it does not keep the hierarchy of your specified files when copying to the target dir.

In this specific case the target is a (non-existent) file; not a directory.

Can you get the sources of the cmake version you're dealing with? Alternatively, can you try to build the original v3.0.0 sources on your machine and see if it behaves differently for copy_if_different?

What I do to test the behavior is this:

touch x
rm -rf a && ./cmake -E copy_if_different x a/b/c/x
ls a

One more thing to check. If you add a tracing message() line to the copying loop, like this:

set(out_files)
foreach( f ${files} )
  set( src ${CMAKE_CURRENT_SOURCE_DIR}/${f} )
  set( dst ${output_dir}/${f} )
  message( "'${src}' -> '${dst}'" )
  add_custom_command(OUTPUT ${dst}
    DEPENDS ${src}
    COMMAND ${CMAKE_COMMAND} -E copy_if_different ${src} ${dst}
    COMMENT "Copying clang's ${f}...")
  list(APPEND out_files ${dst})
endforeach( f )

what does it print for your config?

Can you get the sources of the cmake version you're dealing with? Alternatively, can you try to build the original v3.0.0 sources on your machine and see if it behaves differently for copy_if_different?

So I freshly downloaded cmake 3.0 source, build it and installed it to ~/cmake3.

bash-3.2$ touch x
bash-3.2$ rm -rf a && ~/cmake3/bin/cmake -E copy_if_different x a/b/c/x
bash-3.2$ tree .
.
├── a
│   └── b
│       └── c
│           └── x
└── x

3 directories, 2 files

what does it print for your config?

'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/machine/_stdint.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/machine/_stdint.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/machine/_types.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/machine/_types.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/altivec.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/altivec.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/ammintrin.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/ammintrin.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/arm_acle.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/arm_acle.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/avxintrin.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/avxintrin.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/avx2intrin.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/avx2intrin.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/bmiintrin.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/bmiintrin.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/bmi2intrin.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/bmi2intrin.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/emmintrin.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/emmintrin.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/f16cintrin.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/f16cintrin.h'
'/Users/tobias/dev/rep/llvm/tools/clang/lib/Headers/float.h' -> '/Users/tobias/dev/rep/llvm_build/./lib/clang/3.5.0/include/float.h'

Don't know if that helps. Let me know if you need further info or could you reproduce this issue?

The problem is that install() does not preserve the directory structure so we need another install() invocation for the machine headers.

The patch is updated and put on review as D4623.

Thanks a lot, Tobias.