[windows-itanium] Add Windows Itanium How-To Guide

[bd1976llvm]

Summary

Adds a basic How-To describing what Windows Itanium is and how to assemble a build environment.

[mstorsjo]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

40

Did you look into why it produces such comdats? Because for msvc triple targets, it should work with link.exe.

54

Actually - the code for patching it isn't emitted by either compiler or linker, it's done by the _pei386_runtime_relocator function in the mingw runtime. The linker just generates a list of fixups that the runtime later will need to process. If the runtime that processes this list isn't linked in, it won't work. Unfortunately, this can happen silently...

Have you checked this aspect, that it actually works as intended?

See https://github.com/mstorsjo/llvm-project/commit/lld-pseudo-relocs for a patch (not recently rebased, unfortunately) to lld that makes it force a reference to this function, if runtime fixups actually are needed. That would make it clear if it's needed, but missing.

Finally, the _pei386_runtime_relocator function also needs to be called. Currently in the mingw-w64 runtime, it's always called unconditionally by other startup code, but with a change like https://github.com/mstorsjo/mingw-w64/commit/pseudo-reloc-ctor, that object file would have a ctor that forces it to be called autonomously, if the object file is linked in. (Unfortunately, that change doesn't work with ld.bfd as it is right now, so it isn't upstreamed.)

58

Fwiw, since release 11, you can enable autoimporting alone without enabling the rest of the mingw specific quirks, with the -autoimport flag. Also see D89006 for a recent related fixup.

119

Typo, libc++abi

177

Why is that needed? If it's the same thing I've run into at some point, you can add -DLIBCXXABI_ENABLE_NEW_DELETE_DEFINITIONS=OFF to avoid the redundant symbols.

[bd1976llvm]

Hi @mstorsjo - I have been a bit short of time recently; but, I am going to look into everything that you have pointed out here. I should be able to make a reply by Monday.

Anyway, the point of this comment is just to say that I very much appreciate the helpful comments you have made - thanks!

[mstorsjo]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

117

For my builds, as long as I'm using LIBUNWIND_USE_COMPILER_RT, it links clang-rt.builtins instead of libgcc, but I guess there's a need for more options there.

128

I hadn't noticed this wording here before, but this editing of cmake files to tweak visibility macros pretty much amounts to https://reviews.llvm.org/D90021 I guess? Hopefully that one (in whichever form I tweak it to) works for your case as well, to avoid any need for manual modifications.

[bd1976llvm]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

40

I think I conflated several issues here, as I was trying to get the build set up, and got myself confused. Originally, I think I was somehow generating the "leaderless comdats" that you mentioned in another thread. I had thought that the leaderless comdats were related to multiply defined symbols being present. Happily, I no longer see the leaderless comdats and the remaining multiply defined symbol problems have now been solved and were unrelated. I can now link with MS's link.exe and I have adjusted the doc accordingly.

There is a remaining comdat issue that I see when building libc++ when not using the internal assembler which occurs if I try to use the ' -save-temps=obj' option for debugging. Errors are emitted such as:

1. It seems that source -> assembly -> obj may report comdat
2. errors (when assembling) that are not reported doing source-> obj.
3. Example:
4. src/CMakeFiles/cxx_shared.dir\locale.s:107958:2: error: section '.xdata$' is already linkonce
5. .linkonce discard

I doubt this is related to Windows Itanium and I haven't looked into it more as it isn't currently blocking me.

54

It does not work as intended. Thanks for providing details here. I have adjusted the doc to add a more detailed explanation. As I understand it one of the goal of Windows Itanium is to use the Itanium C++ ABI on windows without relying on extra dependencies such as a custom runtime. Therefore, this must remain as a limitation for now. I spoke to @compnerd briefly about this problem and he though that the solution would be in the compiler emitting code that is called by the MS runtime at start-up and initialises these fields correct. I haven't yet investigated whether this is feasible or not.

58

Thanks. This is a nice improvement!

117

I'll look into upstreaming something for this. Currently, not adding these libraries is an easy edit to make tothe CMake files so I'm not worried about it.

128

Great - thanks for putting that up!

177

Thanks - it's not needed. There was a simple fix to the build that I had missed.

[bd1976llvm]

Updated the diff following helpful review comments.

[mstorsjo]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

40

Ok, great to hear that those bits are sorted out.

Regarding roundtripping through assembly, it's possible that it ends up misgenerated for these cases, but it doesn't seem like I can reproduce it for a trivial case at least. For the gnu assembly syntax, GCC (which doesn't properly use all the aspects of COFF comdats) uses one syntax, while llvm has extended the syntax to support indicating the leader symbol.

A COFF comdat generated by GCC looks like this:

.section        .text$_Z6myfuncv,"x"
.linkonce discard

while generated by clang it looks like this:

.section        .text$_Z6myfuncv,"xr",discard,_Z6myfuncv

(For comdats without a leader symbol, LLVM also generates the GCC-like syntax.)

It sounds to me like there's some mixup between these two happening?

With a test file like this:

inline void myfunc(void) {}
void (*ptr)(void) = myfunc;

I get the expected syntax for a x86_64-windows-itanium target, but maybe it happens with cases that are a bit harder to trigger, like jump tables or something like that...

54

Oh, ok, that explains it.

Yeah it might be possible to make clang generate something like that, but IIRC I tried to do that at some point, and it ended up rather messy - see https://reviews.llvm.org/D43184 for an old attempt.

You can check how it works if you build this test file:

__declspec(dllimport) extern int dllVar;
int *myptr = &dllVar;

If you build this as C, it errors out (initializer element is not a compile-time constant), but if you build it as C++, it generates a constructor that initializes myptr at runtime. (And if you make it int *const myptr, the whole symbol vanishes!)

58

It seems nicer, yes, but it also gives the false impression that you can enable and use it without the corresponding mingw runtime, so that's not ideal either. I'm pondering if I should spend more effort on making it harder to fall into that trap.

[bd1976llvm]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

58

Interestingly, If I try to use -auto-import in LLD it does diagnose the problem. I see errors like:

"lld-link: error: automatic dllimport of _ZTVN10cxxabiv117class_type_infoE in plugin-f2d5e0.o requires pseudo relocations"

This doesn't seem to happen with -lldmingw.

[mstorsjo]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

58

Ah, that's right.

Yes, the feature is split up into two separate feature flags. -autoimport handles the aspect of looking for __imp_<symbol> if <symbol> is needed but missing. An autoimported variable may or may not need a pseudo relocation.

If the reference to the autoimported variable is in a pointer-sized section chunk, with the relocation being storing the absolute address of the variable, then lld can skip doing the pseudo relocation, and redirect all accesses from that section chunk towards the __imp_<symbol> symbol instead (which is a plain pointer that will contain the real address of the symbol at runtime). When compiling code for a mingw target, llvm generates indirection via such stubs (if you read code, COFFSTUB is used to refer to this), e.g. like this:

$ cat test.c
extern int maybeImported;
int getter(void) {
  return maybeImported;
}
$ clang -target x86_64-windows-gnu -S -o - test.c
[...]
getter:
        movq    .refptr.maybeImported(%rip), %rax
        movl    (%rax), %eax
        retq

        .section        .rdata$.refptr.maybeImported,"dr",discard,.refptr.maybeImported
        .globl  .refptr.maybeImported
.refptr.maybeImported:
        .quad   maybeImported

This is done for all accesses to global variables that aren't defined in the same translation unit. If the variable turns out to exist in the same DLL, it produces one pointer sized data block that contains a pointer to the function - if the variable turns out to need to be imported, the whole section ends up thrown away, and all accesses go to the __imp_<symbol> pointer instead, which behaves exactly like that one.

That works fine for cases where code accesses variables, but if you have a pointer in a structure next to other things, lld can't do that:

$ cat test2.c
extern int maybeImported;
struct {
    int foo;
    int *ptr;
    int bar;
} mystruct = {
    42,
    &maybeImported,
    43
};
$ clang -target x86_64-windows-gnu -S -o - test2.c
        .data
        .globl  mystruct
mystruct:
        .long   42
        .zero   4
        .quad   maybeImported
        .long   43
        .zero   4

So the cases with pointers in vtables, you really need psuedo relocations for that.

[bd1976llvm]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

58

Thanks! It would be great to add a test-case to show this limitation with the current implementation. I suppose that anything that relies on a type comparison is broken. I will have a play around and see if I can come up with some testcases.

[bd1976llvm]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

58

I gave this a go and couldn't make a testcase that fails. I think that this is because the type_info class only has one virtual method - the destructor - and the destructors of type_infos are not called (the standard does seem to permit the destructors not to be called). Therefore, the fact that the vtable references are not fixed up correct in type_info objects is not functionally important.

I think that the "correct" Windows Itanium fix for this is to adopt a change very much like your https://reviews.llvm.org/D43184. However, I wonder if, as a work-around, we couldn't implement a constructor (in the sense of some code that runs as program initialisation time) for Windows Itanium that patches the mingw "pseudo relocations". This might be preferable as I would like to keep the patchset for Windows Itanium as minimal as possible.

[bd1976llvm]

@compnerd @smeenai are you basically happy with this doc now?

[mstorsjo]

Looks mostly ok to me now in general, just one minor note.

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

40

The doc still talks about generating invalid comdats that MS link.exe rejects - that's no longer the case right?

[bd1976llvm]

Removed COMDAT limitation text.

[bd1976llvm]

Added the cmake defines from https://reviews.llvm.org/D90021 for the libcxxabi build.

[bd1976llvm]

@mstorsjo are you happy with this now?

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

40

Correct. Thanks!

[mstorsjo]

I think this one looks fine to me now - do @compnerd or @smeenai want to comment on it still?

[bd1976llvm]

ping :)

[smeenai]

Sorry for the delay here. This looks good; thank you!

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

46

Why this instead of directly using -target x86_64-unknown-windows-itanium?

[bd1976llvm]

In D89518#2598986, @smeenai wrote:

Sorry for the delay here. This looks good; thank you!

Thanks! I also have https://reviews.llvm.org/D93203 open - would be great if you could have a look at it.

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

46

There isn't a useable driver for Windows Itanium on Windows OS yet AFAIU. -target x86_64-unknown-windows-itanium uses the CrossWindowsToolChain driver - which is somewhat out of date. Currently, it makes sense (made my build script smaller) to use the windows-msvc driver - which knows how to locate the MS sdk components - and then force that driver to invoke the Windows Itanium code with "-Xclang -triple -Xclang x86_64-unknown-windows-itanium". If there is a better way please let me know :)

[mstorsjo]

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

46

--target=x86_64-windows-itanium should work; there's two syntaxes for that option in the regular clang driver, -target <triple> and --target=<triple>, the latter also works in clang-cl.

llvm/docs/HowToBuildWindowsItaniumPrograms.rst

+==========================================
+How to build Windows Itanium applications.
+==========================================
+
+Introduction
+============
+
+This document contains information describing how to create a Windows Itanium toolchain.
+
+Windows Itanium allows you to deploy Itanium C++ ABI applications on top of the MS VS CRT.
+This environment can use the Windows SDK headers directly and does not required additional
+headers or additional runtime machinery (such as is used by mingw).
+
+Windows Itanium Stack:
+
+* Uses the Itanium C++ abi.
+* libc++.
+* libc++-abi.
+* libunwind.
+* The MS VS CRT.
+* Is compatible with MS Windows SDK include headers.
+* COFF/PE file format.
+* LLD
+
+Note: compiler-rt is not used. This functionality is supplied by the MS VCRT.
+
+Prerequisites
+=============
+
+* The MS SDK is installed as part of MS Visual Studio.
+* Clang with support for the windows-itanium triple.
+* COFF LLD with support for the -autoimport switch.
+
+Known issues:
+=============
+
+SJLJ exceptions, "-fsjlj-exceptions", are the only currently supported model.
+
+link.exe (the MS linker) is unsuitable as it doesn't support auto-importing which
+is currently required to link correctly. However, if that limitation is removed
+then there are no other known issues with using link.exe.
+
+Currently, there is a lack of a usable Windows compiler driver for Windows Itanium.
+A reasonable work-around is to build clang with a windows-msvc default target and
+then override the triple with e.g. "-Xclang -triple -Xclang x86_64-unknown-windows-itanium".
+The linker can be specified with: "-fuse-ld=lld".
+
+In the Itanium C++ ABI the first member of an object is a pointer to the vtable
+for its class. The vtable is often emitted into the object file with the key function
+and must be imported for classes marked dllimport. The pointers must be globally
+unique. Unfortunately, the COFF/PE file format does not provide a mechanism to 
+store a runtime address from another DLL into this pointer (although runtime
+addresses are patched into the IAT). Therefore, the compiler must emit some code,
+that runs after IAT patching but before anything that might use the vtable pointers,
+and sets the vtable pointer to the address from the IAT. For the special case of
+the references to vtables for __cxxabiv1::__class_type_info from typeinto objects
+there is no declaration available to the compiler so this can't be done. To allow
+programs to link we currently rely on the -auto-import switch in LLD to auto-import
+references to __cxxabiv1::__class_type_info pointers (see: https://reviews.llvm.org/D43184
+for a related discussion). This allows for linking; but, code that actually uses
+such fields will not work as they these will not be fixed up at runtime. See 
+_pei386_runtime_relocator which handles the runtime component of the autoimporting
+scheme used for mingw and comments in https://reviews.llvm.org/D43184 and
+https://reviews.llvm.org/D89518 for more.
+
+Assembling a Toolchain:
+=======================
+
+The procedure is:
+
+# Build an LLVM toolchain with support for Windows Itanium.
+# Use the toolchain from step 1. to build libc++, libc++abi, and libunwind.
+
+It is also possible to cross-compile from Linux.
+
+One method of building the libraries in step 2. is to build them "stand-alone".
+A stand-alone build doesn't involve the rest of the LLVM tree. The steps are:
+
+* ``cd build-dir``
+* ``cmake -DLLVM_PATH=<path to llvm checkout e.g. /llvm-project/> -DCMAKE_INSTALL_PREFIX=<install path> <other options> <path to project e.g. /llvm-project/libcxxabi>``
+* ``<make program e.g. ninja>``
+* ``<make program> install``
+
+More information on standalone builds can be found in the build documentation for
+the respective libraries. The next section discuss the salient options and modifications
+required for building and installing the libraries using standalone builds. This assumes
+that we are building libunwind and ibc++ as DLLs and statically linking libc++abi into
+libc++. Other build configurations are possible, but they are not discussed here.
+
+Common CMake configuration options:
+-----------------------------------
+
+* ``-D_LIBCPP_ABI_FORCE_ITANIUM'``
+
+Tell the libc++ headers that the Itanium C++ ABI is being used.
+
+* ``-DCMAKE_C_FLAGS="-lmsvcrt -llegacy_stdio_definitions -D_NO_CRT_STDIO_INLINE"``
+
+Supply CRT definitions including stdio definitions that have been removed from the MS VS CRT.
+We don't want the stdio functions decalred inline as they will casuse multiple defintiion
+errors when the same symbols are pulled in from legacy_stdio_definitions.ib.
+
+* ``-DCMAKE_INSTALL_PREFIX=<install path>``
+
+Where to install the library and headers.
+
+Building libunwind:
+-------------------
+
+* ``-DLIBUNWIND_ENABLE_SHARED=ON``
+* ``-DLIBUNWIND_ENABLE_STATIC=OFF``
+
+libunwind can be built as a DLL. It is not dependent on other projects.
+
+* ``-DLIBUNWIND_USE_COMPILER_RT=OFF``
+
+We use the MS runtime.
+
+The CMake files will need to be edited to prevent them adding GNU specific libraries to the link line.
+
+Building libc++abi:
+------------------
+
+* ``-DLIBCXXABI_ENABLE_SHARED=OFF``
+* ``-DLIBCXXABI_ENABLE_STATIC=ON``
+* ``-DLIBCXX_ENABLE_SHARED=ON'``
+* ``-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON``
+
+To break the symbol dependency between libc++abi and libc++ we
+build libc++abi as a static library and then statically link it
+into the libc++ DLL. This necessitates setting the CMake file
+to ensure that the visibility macros (which expand to dllexport/import)
+are expanded as they will be needed when creating the final libc++
+DLL later, see: https://reviews.llvm.org/D90021.
+
+* ``-DLIBCXXABI_LIBCXX_INCLUDES=<path to libcxx>/include``
+
+Where to find the libc++ headers
+
+Building libc++:
+----------------
+
+* ``-DLIBCXX_ENABLE_SHARED=ON``
+* ``-DLIBCXX_ENABLE_STATIC=OFF``
+
+We build libc++ as a DLL and statically link libc++abi into it.
+
+* ``-DLIBCXX_INSTALL_HEADERS=ON``
+
+Install the headers.
+
+* ``-DLIBCXX_USE_COMPILER_RT=OFF``
+
+We use the MS runtime.
+
+* ``-DLIBCXX_HAS_WIN32_THREAD_API=ON``
+
+Windows Itanium does not offer a POSIX-like layer over WIN32.
+
+* ``-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON``
+* ``-DLIBCXX_CXX_ABI=libcxxabi``
+* ``-DLIBCXX_CXX_ABI_INCLUDE_PATHS=<libcxxabi src path>/include``
+* ``-DLIBCXX_CXX_ABI_LIBRARY_PATH=<libcxxabi build path>/lib``
+
+Use the static libc++abi library built earlier.
+
+* ``-DLIBCXX_NO_VCRUNTIME=ON``
+
+Remove any dependency on the VC runtime - we need libc++abi to supply the C++ runtime.
+
+* ``-DCMAKE_C_FLAGS=<path to installed unwind.lib>``
+
+As we are statically linking against libcxxabi we need to link
+against the unwind import library to resolve unwind references
+from the libcxxabi objects.
+
+* ``-DCMAKE_C_FLAGS+=' -UCLOCK_REALTIME'``
+
+Prevent the inclusion of sys/time that MS doesn't provide.
+
+Notes:
+------
+
+An example build recipe is available here: https://reviews.llvm.org/D88124

llvm/docs/UserGuides.rst

.. toctree::
    CompileCudaWithLLVM
    CoverageMappingFormat
    DebuggingJITedCode
    Docker
    ExtendingLLVM
    GoldPlugin
    HowToBuildOnARM
    HowToBuildWithPGO
+   HowToBuildWindowsItaniumPrograms
    HowToCrossCompileBuiltinsOnArm
    HowToCrossCompileLLVM
    HowToUpdateDebugInfo
    LinkTimeOptimization
    LoopTerminology
    MarkdownQuickstartTemplate
    MemorySSA
    MergeFunctions

 
 :doc:`ExtendingLLVM`
   Look here to see how to add instructions and intrinsics to LLVM.
 
 :doc:`AddingConstrainedIntrinsics`
    Gives the steps necessary when adding a new constrained math intrinsic
    to LLVM.
 
+:doc:`HowToBuildWindowsItaniumPrograms`
+   Notes on assembling a Windows Itanium enviroment.
+
 :doc:`HowToCrossCompileBuiltinsOnArm`
    Notes on cross-building and testing the compiler-rt builtins for Arm.
 
 :doc:`BigEndianNEON`
   LLVM's support for generating NEON instructions on big endian ARM targets is
   somewhat nonintuitive. This document explains the implementation and rationale.
 
 :doc:`CompileCudaWithLLVM`