[lldb] Reduce the stack alignment requirements for the Windows x86_64 ABI

[mstorsjo]

Summary

This fixes https://github.com/llvm/llvm-project/issues/56095.

@labath, FWIW while making the testcase, I realized why most tests on
Windows need the %build script instead of plain %clang_host.

For this testcase, I do need to have at least symbols available
(full debug info isn't needed). On unix-like systems, a command like
"clang source.c -o exec" will produce an executable with symbols.
With MSVC and clang in MSVC mode, the default linking command won't
produce any debug info at all, no symbols, nothing. We can fix it by
passing "-fuse-ld=lld -Wl,-debug:symtab", but that breaks mingw builds
(where symbols are kept by default).

The %build script handles all the options that are needed for creating
PDB debug info with clang-cl/lld-link, and produces symbols when building
in mingw mode too.

[labath]

Looks good. The patch is really trivial so I don't want to make big deal out of the test.

However, I would like to continue the discussion started in the patch description, as I think it would be very useful if we could figure out a way to make this test (and others) cross-platform, so we don't have to have two versions of all of them. I'm going to start by asking a couple of questions, just so I understand the problem better.

You say that the issue is the lack of symtab in the "msvc" mode. What makes this test work then? Do some of the assembler directives (.def?) produce some sort of debug info entries?

What determines the modes that clang is operating in? Is it the default target triple?

What does -Wl,-debug:symtab actually produce? Would it make sense to make it a part of the %clang_host substitution (only for the msvc mode) to make the output of %clang_host more uniform? Or maybe achieve equivalence in some other way (add -g1, or similar, to the cmd line)?

[mstorsjo]

In D129455#3641967, @labath wrote:

You say that the issue is the lack of symtab in the "msvc" mode. What makes this test work then?

When invoked via the %build python script (lldb/test/Shell/helper/build.py), clang is invoked with extra options (/Z7) that generate codeview debug info, and then later the linker is invoked with extra options (/DEBUG:FULL and '/PDB:' + self._pdb_file_name())) to write that codeview debug info into a separate standalone PDB file.

In the MSVC ecosystem, executables/DLLs never have embedded debug info - it's always in a separate PDB file. Contrary to the GCC-compatible ecosystem, debug info is opt-in here, not opt-out (e.g. if linking with a unix linker, you get the symbol table and debug info included automatically unless you pass -Wl,-s or strip things separately afterwards). In the MSVC ecosystem, the only way to have a symbol table (without actual dwarf debug info) is with a PDB file.

In the mingw ecosystem, things behave as on unix - you get any embedded DWARF debug info included in the executable by default, and a symbol table - both which can be stripped out afterwards.

Do some of the assembler directives (.def?) produce some sort of debug info entries?

Those just set the symbol type to "function" - it could be omitted from this test for clarity.

What determines the modes that clang is operating in? Is it the default target triple?

Exactly.

Additionally, there's a couple more tooling differences that makes things a bit more complicated:

In the MSVC ecosystem, you normally would execute cl.exe (the MSVC compiler) or clang-cl (the cl.exe compatible clang frontend) - which has got an option syntax that is distinct and mostly incompatible from the gcc-compatible normal clang interface.

Despite this, you can also build code in MSVC mode with the regular gcc-compatible clang interface (either by passing e.g. --target=x86_64-windows-msvc or if such a triple is the deafult target triple). You can do most things in this setup too, but some MSVC-isms are tricky to achieve. This is what the %clang_host lit test macro does. Regular compiling, e.g. %clang_host input.c -o executable -luser32 works fine for both MSVC and mingw mode.

In the MSVC ecosystem, you very rarely use the compiler driver to run linking - you'd normally execute link.exe or lld-link directly. If linking via the gcc-compatible clang interface, options passed to the linked with -Wl, need to be link.exe/lld-link options, which a unix-style linker obviously doesn't recognize.

What does -Wl,-debug:symtab actually produce?

It produces an executable embedded symbol table, like in mingw mode. This is a lld specific option, MS's upstream link.exe doesn't support this option (it supports other parameters to -debug: but symtab is lld's invention). LLDB happily picks up and uses that (for mingw built executables it's the default case anyway).

Would it make sense to make it a part of the %clang_host substitution (only for the msvc mode) to make the output of %clang_host more uniform? Or maybe achieve equivalence in some other way (add -g1, or similar, to the cmd line)?

I think that might be a sensible path forward in general! But as noted, I'd rather not hold up this trivial patch with that. (Also I'm a bit more short on time at the moment than usual, but I'm trying to get low hanging fruit merged before the llvm 15 branch deadline.)

as I think it would be very useful if we could figure out a way to make this test (and others) cross-platform, so we don't have to have two versions of all of them.

A bunch of tests might be doable crossplatform that way, but any tests that involve assembly might be tricky. On i386, arm and aarch64, Windows has got mostly the same calling conventions as other OSes, but on x86_64, it's radically different - arguments are passed in different registers than on x86_64 unix. Additionally, the caller is required to allocate shadow space on the stack for all arguments passed in registers (the extra subq $32, %rsp) so that the callee can dump them there if wanted. And finally, this particular testcase generates SEH unwind instructions with the .seh_* directives. (I didn't test whether the testcase would work without the SEH unwind instructions or not - I made it to look mostly like what a regular Windows function would look like.)

[mstorsjo]

In D129455#3642015, @mstorsjo wrote:

In D129455#3641967, @labath wrote:

What does -Wl,-debug:symtab actually produce? Would it make sense to make it a part of the %clang_host substitution (only for the msvc mode) to make the output of %clang_host more uniform? Or maybe achieve equivalence in some other way (add -g1, or similar, to the cmd line)?

I think that might be a sensible path forward in general!

If we'd want to take that even further, we could consider to have %clang_host add options for generating DWARF debug info even in MSVC mode, and pass -Wl,-debug:dwarf to include it embedded in the exe like in mingw mode. (I've understood that some people even do this in proper production setups.) It's not what you'd normally have in MSVC mode, but could be useful for making testcases more portable, for cases where such details don't matter.

[labath]

In D129455#3642015, @mstorsjo wrote:

In D129455#3641967, @labath wrote:

You say that the issue is the lack of symtab in the "msvc" mode. What makes this test work then?

When invoked via the %build python script (lldb/test/Shell/helper/build.py), clang is invoked with extra options (/Z7) that generate codeview debug info, and then later the linker is invoked with extra options (/DEBUG:FULL and '/PDB:' + self._pdb_file_name())) to write that codeview debug info into a separate standalone PDB file.

Ok. So I take it this means the resulting PDB will contain debug info describing the .s file (the location of call_func)? Similar to how clang w/dwarf will emit a DW_TAG_label when compiling an .s file with -g (?)

Do some of the assembler directives (.def?) produce some sort of debug info entries?

Those just set the symbol type to "function" - it could be omitted from this test for clarity.

What determines the modes that clang is operating in? Is it the default target triple?

Exactly.

Got it. Thanks.

Additionally, there's a couple more tooling differences that makes things a bit more complicated:

In the MSVC ecosystem, you normally would execute cl.exe (the MSVC compiler) or clang-cl (the cl.exe compatible clang frontend) - which has got an option syntax that is distinct and mostly incompatible from the gcc-compatible normal clang interface.

Despite this, you can also build code in MSVC mode with the regular gcc-compatible clang interface (either by passing e.g. --target=x86_64-windows-msvc or if such a triple is the deafult target triple). You can do most things in this setup too, but some MSVC-isms are tricky to achieve. This is what the %clang_host lit test macro does. Regular compiling, e.g. %clang_host input.c -o executable -luser32 works fine for both MSVC and mingw mode.

In the MSVC ecosystem, you very rarely use the compiler driver to run linking - you'd normally execute link.exe or lld-link directly. If linking via the gcc-compatible clang interface, options passed to the linked with -Wl, need to be link.exe/lld-link options, which a unix-style linker obviously doesn't recognize.

Ok, makes sense. If some test requires a specific "msvc-ism", then I think that test can stay windows-only, and invoke clang-cl (or whatever) directly. What I'd like to figure out is how to write simple tests that don't depend on any architecture specifics in an cross-platform manner. Or (since we already kind of have that), I guess I should say, expand the scope of what can be done that way.

A bunch of tests might be doable crossplatform that way, but any tests that involve assembly might be tricky. On i386, arm and aarch64, Windows has got mostly the same calling conventions as other OSes, but on x86_64, it's radically different - arguments are passed in different registers than on x86_64 unix. Additionally, the caller is required to allocate shadow space on the stack for all arguments passed in registers (the extra subq $32, %rsp) so that the callee can dump them there if wanted. And finally, this particular testcase generates SEH unwind instructions with the .seh_* directives. (I didn't test whether the testcase would work without the SEH unwind instructions or not - I made it to look mostly like what a regular Windows function would look like.)

Yes, asm is tricky even without cross-platform considerations. I don't imagine we will be having many of these tests, but some things are hard to test differently. It's either this or core files, and both options have a lot of disadvantages. The good thing about asm and calling conventions is that, as long as you're staying with the asm code, the exact calling convention does not really matter. And a lot of these asm tests are for situations where the code uses some kind of nonstandard calling convention (yes, the spec says you should allocate shadow space, but it also says you should align the stack, yet here we are..)

I'm also wondering if we could not use inline asm (either in __attribute__((naked)) functions, or as top-level asm blocks) instead of actual asm files. I think that could insulate us from some of the target asm specifics, and (in the first case) take care of the function declaration boilerplate.

lldb/source/Plugins/ABI/X86/ABIWindows_x86_64.h

public:
 
   bool
   CreateFunctionEntryUnwindPlan(lldb_private::UnwindPlan &unwind_plan) override;
 
   bool CreateDefaultUnwindPlan(lldb_private::UnwindPlan &unwind_plan) override;
 
   bool RegisterIsVolatile(const lldb_private::RegisterInfo *reg_info) override;
 
-  // In Windows_x86_64 ABI, stack will always be maintained 16-byte aligned
+  // In Windows_x86_64 ABI requires that the stack will be maintained 16-byte
+  // aligned.
+  // When ntdll invokes callbacks such as KiUserExceptionDispatcher or
+  // KiUserCallbackDispatcher, those functions won't have a properly 16-byte
+  // aligned stack - but tolerate unwinding through them by relaxing the
+  // requirement to 8 bytes.
   bool CallFrameAddressIsValid(lldb::addr_t cfa) override {
-    if (cfa & (16ull - 1ull))
-      return false; // Not 16 byte aligned
+    if (cfa & (8ull - 1ull))
+      return false; // Not 8 byte aligned
     if (cfa == 0)
       return false; // Zero is not a valid stack address
     return true;
   }
 
   bool CodeAddressIsValid(lldb::addr_t pc) override {
     // We have a 64 bit address space, so anything is valid as opcodes
     // aren't fixed width...

lldb/test/Shell/Unwind/Inputs/windows-unaligned-x86_64-asm.s

+        .globl    call_func
+        .def      call_func;   .scl    2;      .type   32;     .endef
+        .seh_proc call_func
+call_func:
+        subq    $32, %rsp
+        .seh_stackalloc 32
+        .seh_endprologue
+        call    realign_stack
+        addq    $32, %rsp
+        ret
+        .seh_endproc
+
+        .globl    realign_stack
+        .def      realign_stack;   .scl    2;      .type   32;     .endef
+        .seh_proc realign_stack
+realign_stack:
+        subq    $32, %rsp
+        .seh_stackalloc 32
+        .seh_endprologue
+        movq    %rcx, %rax
+        movl    %edx, %ecx
+        call    *%rax
+        addq    $32, %rsp
+        ret
+        .seh_endproc

lldb/test/Shell/Unwind/Inputs/windows-unaligned-x86_64.cpp

+extern "C" void call_func(void (*ptr)(int a), int a);
+
+extern "C" void func(int arg) { }
+
+int main(int argc, char **argv) {
+  call_func(func, 42);
+  return 0;
+}

lldb/test/Shell/Unwind/windows-unaligned-x86_64.test

+# Test unwinding through stack frames that aren't aligned to 16 bytes.
+# (In real world code, this happens when unwinding through
+# KiUserExceptionDispatcher and KiUserCallbackDispatcher in ntdll.dll.)
+
+# REQUIRES: target-x86_64, native, system-windows
+
+# RUN: %build %p/Inputs/windows-unaligned-x86_64.cpp %p/Inputs/windows-unaligned-x86_64-asm.s -o %t
+# RUN: %lldb %t -s %s -o exit | FileCheck %s
+
+# Future TODO: If %build could compile the source file in C mode, the symbol
+# name handling would be easier across msvc and mingw build configurations.
+# (In mingw mode, the extern C symbol "func" is printed as "::func" while
+# it's plain "func" in msvc mode. Without the extern C, it's "func(..." in
+# mingw mode, but "void __cdecl func(..." in msvc mode.)
+
+breakpoint set -n func
+# CHECK: Breakpoint 1: where = {{.*}}`{{(::)?}}func
+
+process launch
+# CHECK: stop reason = breakpoint 1.1
+
+thread backtrace
+# CHECK: frame #0: {{.*}}`{{(::)?}}func
+# CHECK: frame #1: {{.*}}`realign_stack
+# CHECK: frame #2: {{.*}}`call_func
+# CHECK: frame #3: {{.*}}`main