Reland [sanitizer] Support Intel CET

[hjl.tools]

Summary

1. Include <cet.h> in sanitizer_common/sanitizer_asm.h, if it exists, to mark Intel CET support when Intel CET is enabled.
2. Define _CET_ENDBR as empty if it isn't defined.
3. Add _CET_ENDBR to function entries in assembly codes so that ENDBR instruction will be generated when Intel CET is enabled.

Tested on Intel Tiger Lake with Intel CET enabled.

[MaskRay]

Thanks!

Instead of creating a new diff, you may change the local git description, then run arc diff --verbatim --head=head 'HEAD^' to update the Differential's subject.

Some intro to arc (https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-command-line)

[hjl.tools]

In D111185#3045884, @MaskRay wrote:

Thanks!

Instead of creating a new diff, you may change the local git description, then run arc diff --verbatim --head=head 'HEAD^' to update the Differential's subject.

Some intro to arc (https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-command-line)

Thanks for the tip.

[wolfgangp]

This is causing problems with gcc versions that don't have cet.h yet. I think cet.h wasn't added until gcc 8.

[MaskRay]

In D111185#3046891, @wolfgangp wrote:

This is causing problems with gcc versions that don't have cet.h yet. I think cet.h wasn't added until gcc 8.

Use -DLLVM_ENABLE_RUNTIMES=compiler-rt instead of -DLLVM_ENABLE_PROJECTS=compiler-rt to build with freshly built Clang with cet.h in its resource directory.

I think the trend is that older GCC will be unsupported when building runtime (including libc++/libc++abi/libunwind/compiler-rt; "building" != "using", "using" needs to support a wide range of compilers), even if most other toplevel projects of llvm-project target GCC>=5.

E.g.
https://lists.llvm.org/pipermail/llvm-dev/2021-March/148881.html "Compiler support in libc++" says "We support the latest major release of GCC, as advertised on https://gcc.gnu.org/releases.html."
So it seems fairly reasonable to me that a quite old GCC (7.x) cannot build in -DLLVM_ENABLE_PROJECTS=compiler-rt mode. @phosek

[hjl.tools]

In D111185#3046891, @wolfgangp wrote:

This is causing problems with gcc versions that don't have cet.h yet. I think cet.h wasn't added until gcc 8.

Please try this:

0001-sanitizer-Include-cet.h-only-if-it-exists.patch3 KBDownload

[wolfgangp]

In D111185#3046902, @hjl.tools wrote:

In D111185#3046891, @wolfgangp wrote:

This is causing problems with gcc versions that don't have cet.h yet. I think cet.h wasn't added until gcc 8.

Please try this:

0001-sanitizer-Include-cet.h-only-if-it-exists.patch3 KBDownload

Thank you. Are you planning to push this patch to main? This is affecting one of our internal bots that's building the main branch, so either maskray's solution or yours would work if it goes into main.

[hjl.tools]

In D111185#3046919, @wolfgangp wrote:

In D111185#3046902, @hjl.tools wrote:

In D111185#3046891, @wolfgangp wrote:

This is causing problems with gcc versions that don't have cet.h yet. I think cet.h wasn't added until gcc 8.

Please try this:

0001-sanitizer-Include-cet.h-only-if-it-exists.patch3 KBDownload

Thank you. Are you planning to push this patch to main? This is affecting one of our internal bots that's building the main branch, so either maskray's solution or yours would work if it goes into main.

I can submit it if it works for you.

[MaskRay]

In D111185#3046919, @wolfgangp wrote:

In D111185#3046902, @hjl.tools wrote:

In D111185#3046891, @wolfgangp wrote:

This is causing problems with gcc versions that don't have cet.h yet. I think cet.h wasn't added until gcc 8.

Please try this:

0001-sanitizer-Include-cet.h-only-if-it-exists.patch3 KBDownload

Thank you. Are you planning to push this patch to main? This is affecting one of our internal bots that's building the main branch, so either maskray's solution or yours would work if it goes into main.

I'd prefer we don't work around such usage which is unsupported. This is an opportunity for users to migrate to -DLLVM_ENABLE_RUNTIMES=compiler-rt

[wolfgangp]

In D111185#3046921, @MaskRay wrote:

I'd prefer we don't work around such usage which is unsupported. This is an opportunity for users to migrate to -DLLVM_ENABLE_RUNTIMES=compiler-rt

I agree, it would be awkward. We'll give your solution a try.

[thakis]

This breaks building on macOS: https://bugs.chromium.org/p/chromium/issues/detail?id=1257863

Please take a look.

[xiongji90]

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

Hi, @hjl.tools
We had better guard the cet.h header using "linux" macro, since CET is only implemented on x86.x86_64 linux platforms currently.

Thanks very much.

[MaskRay]

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

Probably use __ELF__ so that other ELF OSes can pick up CET easily.

[xiongji90]

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

Hi, @MaskRay
If CET is not implemented on all ELF based OS, I am afraid using "ELF" may introduce risk, we may encounter " 'cet.h' is not found" error.
Thanks very much.

[MaskRay]

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

FreeBSD 13.0 has /usr/lib/clang/11.0.1/include/cet.h

Clang resource headers generally include files for all arches, I don't understand why @thakis's macOS platform doesn't have cet.h, though.

[aeubanks]

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

Other places that include <cet.h> only do so on Linux. I'm reverting this for now while we figure out the non-Linux story.

[hjl.tools]

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

There is a patch here

0001-sanitizer-Include-cet.h-only-if-it-exists.patch3 KBDownload

. Does it work for you?

[aeubanks]

In D111185#3046921, @MaskRay wrote:

In D111185#3046919, @wolfgangp wrote:

In D111185#3046902, @hjl.tools wrote:

In D111185#3046891, @wolfgangp wrote:

This is causing problems with gcc versions that don't have cet.h yet. I think cet.h wasn't added until gcc 8.

Please try this:

0001-sanitizer-Include-cet.h-only-if-it-exists.patch3 KBDownload

Thank you. Are you planning to push this patch to main? This is affecting one of our internal bots that's building the main branch, so either maskray's solution or yours would work if it goes into main.

I'd prefer we don't work around such usage which is unsupported. This is an opportunity for users to migrate to -DLLVM_ENABLE_RUNTIMES=compiler-rt

Is this actually unsupported? Do we currently claim to only support building sanitizers via the runtime build? (I remember something like that for libc++, but not for sanitizers)
https://llvm.org/docs/GettingStarted.html#host-c-toolchain-both-compiler-and-standard-library says we allow gcc 5.1.

I agree we should ideally migrate to using the runtimes build, but that's not trivial (e.g. in our case it also turns on LLVM_ENABLE_PER_TARGET_RUNTIME_DIR, which we've turned off).

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

That would probably work, but that's a solution to a slightly different problem of too old of a host compiler. I'm fine with relanding this change either with your patch or a check for Linux.

[MaskRay]

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

Use && to pack two #if together.

[hjl.tools]

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

70

Use && to pack two #if together.

Oops. I have pushed my original patch. Should I create an incremental patch?

compiler-rt/lib/hwasan/hwasan_setjmp_x86_64.S

 
 .section .text
 .file "hwasan_setjmp_x86_64.S"
 
 .global __interceptor_setjmp
 ASM_TYPE_FUNCTION(__interceptor_setjmp)
 __interceptor_setjmp:
   CFI_STARTPROC
+  _CET_ENDBR
   xorl %esi, %esi
   jmp __interceptor_sigsetjmp
   CFI_ENDPROC
 ASM_SIZE(__interceptor_setjmp)
 
 .global __interceptor_sigsetjmp
 ASM_TYPE_FUNCTION(__interceptor_sigsetjmp)
 __interceptor_sigsetjmp:
   CFI_STARTPROC
+  _CET_ENDBR
 
   // Save callee save registers.
   mov %rbx, (0*8)(%rdi)
   mov %rbp, (1*8)(%rdi)
   mov %r12, (2*8)(%rdi)
   mov %r13, (3*8)(%rdi)
   mov %r14, (4*8)(%rdi)
   mov %r15, (5*8)(%rdi)

compiler-rt/lib/sanitizer_common/sanitizer_asm.h

 #if defined(__ELF__) && (defined(__GNU__) || defined(__FreeBSD__) || \
                          defined(__Fuchsia__) || defined(__linux__))
 // clang-format off
 #define NO_EXEC_STACK_DIRECTIVE .section .note.GNU-stack,"",%progbits
 // clang-format on
 #else
 #define NO_EXEC_STACK_DIRECTIVE
 #endif
+
+#if defined(__x86_64__) || defined(__i386__)
+#if defined(__has_include)
+#if __has_include(<cet.h>)
+#include <cet.h>
+#endif
+#endif
+#ifndef _CET_ENDBR
+#define _CET_ENDBR
+#endif
+#endif

compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_vfork_i386.inc.S

 #if defined(__i386__) && defined(__linux__)
 
 #include "sanitizer_common/sanitizer_asm.h"
 
 .comm _ZN14__interception10real_vforkE,4,4
 .globl ASM_WRAPPER_NAME(vfork)
 ASM_TYPE_FUNCTION(ASM_WRAPPER_NAME(vfork))
 ASM_WRAPPER_NAME(vfork):
+        _CET_ENDBR
         // Store return address in the spill area and tear down the stack frame.
         sub     $12, %esp
         call    COMMON_INTERCEPTOR_SPILL_AREA
         mov     12(%esp), %ecx
         mov     %ecx, (%eax)
         add     $16, %esp
 
         call    .L0$pb

compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_vfork_x86_64.inc.S

 #if defined(__x86_64__) && defined(__linux__)
 
 #include "sanitizer_common/sanitizer_asm.h"
 
 .comm _ZN14__interception10real_vforkE,8,8
 .globl ASM_WRAPPER_NAME(vfork)
 ASM_TYPE_FUNCTION(ASM_WRAPPER_NAME(vfork))
 ASM_WRAPPER_NAME(vfork):
+        _CET_ENDBR
         // Store return address in the spill area and tear down the stack frame.
         push    %rcx
         call    COMMON_INTERCEPTOR_SPILL_AREA
         pop     %rcx
         pop     %rdi
         mov     %rdi, (%rax)
 
         call    *_ZN14__interception10real_vforkE(%rip)

compiler-rt/lib/tsan/rtl/tsan_rtl_amd64.S

 // The content of this file is x86_64-only:
 #if defined(__x86_64__)
 
 #include "sanitizer_common/sanitizer_asm.h"
 
 #if !defined(__APPLE__)
 .section .text
 #else
 .section __TEXT,__text
 #endif
 
 ASM_HIDDEN(__tsan_trace_switch)
 .globl ASM_SYMBOL(__tsan_trace_switch_thunk)
 ASM_SYMBOL(__tsan_trace_switch_thunk):
   CFI_STARTPROC
+  _CET_ENDBR
   # Save scratch registers.
   push %rax
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rax, 0)
   push %rcx
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rcx, 0)
   push %rdx
ASM_SYMBOL(__tsan_trace_switch_thunk):
   CFI_RESTORE(%r11)
   ret
   CFI_ENDPROC
 
 ASM_HIDDEN(__tsan_report_race)
 .globl ASM_SYMBOL(__tsan_report_race_thunk)
 ASM_SYMBOL(__tsan_report_race_thunk):
   CFI_STARTPROC
+  _CET_ENDBR
   # Save scratch registers.
   push %rax
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rax, 0)
   push %rcx
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rcx, 0)
   push %rdx

 ASM_TYPE_FUNCTION(ASM_SYMBOL_INTERCEPTOR(__setjmp14))
 ASM_SYMBOL_INTERCEPTOR(__setjmp14):
 #else
 .globl ASM_SYMBOL_INTERCEPTOR(setjmp)
 ASM_TYPE_FUNCTION(ASM_SYMBOL_INTERCEPTOR(setjmp))
 ASM_SYMBOL_INTERCEPTOR(setjmp):
 #endif
   CFI_STARTPROC
+  _CET_ENDBR
   // save env parameter
   push %rdi
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rdi, 0)
   // obtain SP, store in %rdi, first argument to `void __tsan_setjmp(uptr sp)`
 #if defined(__FreeBSD__) || defined(__NetBSD__)
   lea 8(%rsp), %rdi
 #elif defined(__linux__) || defined(__APPLE__)

 ASM_SIZE(ASM_SYMBOL_INTERCEPTOR(setjmp))
 #endif
 
 .comm _ZN14__interception12real__setjmpE,8,8
 .globl ASM_SYMBOL_INTERCEPTOR(_setjmp)
 ASM_TYPE_FUNCTION(ASM_SYMBOL_INTERCEPTOR(_setjmp))
 ASM_SYMBOL_INTERCEPTOR(_setjmp):
   CFI_STARTPROC
+  _CET_ENDBR
   // save env parameter
   push %rdi
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rdi, 0)
   // obtain SP, store in %rdi, first argument to `void __tsan_setjmp(uptr sp)`
 #if defined(__FreeBSD__) || defined(__NetBSD__)
   lea 8(%rsp), %rdi
 #elif defined(__linux__) || defined(__APPLE__)

 ASM_SYMBOL_INTERCEPTOR(__sigsetjmp14):
 #else
 .comm _ZN14__interception14real_sigsetjmpE,8,8
 .globl ASM_SYMBOL_INTERCEPTOR(sigsetjmp)
 ASM_TYPE_FUNCTION(ASM_SYMBOL_INTERCEPTOR(sigsetjmp))
 ASM_SYMBOL_INTERCEPTOR(sigsetjmp):
 #endif
   CFI_STARTPROC
+  _CET_ENDBR
   // save env parameter
   push %rdi
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rdi, 0)
   // save savesigs parameter
   push %rsi
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rsi, 0)

 #endif
 
 #if !defined(__APPLE__) && !defined(__NetBSD__)
 .comm _ZN14__interception16real___sigsetjmpE,8,8
 .globl ASM_SYMBOL_INTERCEPTOR(__sigsetjmp)
 ASM_TYPE_FUNCTION(ASM_SYMBOL_INTERCEPTOR(__sigsetjmp))
 ASM_SYMBOL_INTERCEPTOR(__sigsetjmp):
   CFI_STARTPROC
+  _CET_ENDBR
   // save env parameter
   push %rdi
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rdi, 0)
   // save savesigs parameter
   push %rsi
   CFI_ADJUST_CFA_OFFSET(8)
   CFI_REL_OFFSET(%rsi, 0)