CFI runtime library (cross-DSO support)

[eugenis]

Summary

This is an initial version of the runtime cross-DSO CFI support library.
It contains a number of FIXMEs, ex. it does not support the diagnostic mode nor dlopen/dlclose, but it works and can be tested.
Diagnostic mode, in particular, would require some refactoring (we'd like to gather all CFI hooks in the UBSan library into one function so that we could easier pass the diagnostic information down to CFI_Check). It will be implemented later.
Once the diagnostic mode is in, I plan to create a second test configuration to run all existing tests in both modes. For now, this patch includes only a couple of new, cross-DSO tests.

[pcc]

lib/cfi/cfi.cc

39

This can be a function rather than a macro.

60

I think the code would be easier to read if this function were inlined into its caller, mainly because of the magic numbers you are using here.

79

Yes, but there's also a reason why we might prefer to use this function all the time: an attacker may have corrupted the dynamic loader's internal data structures so as to make dlsym return some value of its choosing.

111

This is reasonable if all major linkers place strtab immediately after symtab. Have you verified that this is the case?

113

We should probably not use this as the function name, as it intrudes on the user's namespace. Instead, please use a C/C++ implementation reserved identifier (e.g. __CFI_Check).

133

I believe that this is controlled by the RELRO program header.

150

Implementation reserved identifier please.

159

It's a little confusing that this function can return both function addresses and magic numbers. I would inline find_cfi_check_for_address into here.

176

Can we link against ubsan dependent on whether diagnostics are enabled? That would save us from having to link ubsan into production builds.

test/cfi/cross-dso/icall/icall-from-dso.cpp

12

Was this intended to be a positive test? This isn't a positive test because it is a direct call.

test/cfi/cross-dso/icall/icall.cpp

15

Same here.

test/cfi/cross-dso/lit.local.cfg

1 ↗

(On Diff #42238)

It would be less confusing to add a new substitution instead of rewriting the existing one.

test/cfi/cross-dso/simple-fail.cpp

73

We should make the other test cases do this as well. It doesn't have to be done now though.

[eugenis]

lib/cfi/cfi.cc

60

Does it look better now?
I've split fill_shadow in 2: for function pointers, and for (0) and (-1) constants.
I've also refactored find_cfi_check_for_address to a class with, hopefully, cleaner interface.

111

Yes, looks that way.
Changed the code to bail out if symtab and strtab point to different ELF segments. Now, if they are inside one segment, nothing really bad can happen for over-scanning. The probability of misinterpreting a random sequence of bytes as __cfi_check symbol is super low.

113

Renamed to __cfi_check.

133

keep as FIXME?

159

PTAL

176

So, we build libclang_rt.cfi and libclangrt.cfi_diag (the latter includes ubsan)?
And teach the driver to include the latter when needed.
Sounds doable.

[pcc]

lib/cfi/cfi.cc

42

Looks like most callers of this function would be better off with a uint16_t * so maybe should return that?

58

assert(!is_invalid() && !is_unchecked()); ?

61

Yes, this looks nicer.

72–73

Is this necessary? Looks like you could just pass these directly to mem_to_shadow.

76

This would become mem_to_shadow(end - 1) + 1.

84–85

Same as above.

134

Yes.

160

Looks good.

177

Maybe.

Why don't we build a libclang_rt.sanitizer_common library and have the sanitizers use that instead of each having its own copy of sanitizer_common? Then we could just have a cfi library and link the regular ubsan library conditionally.

200

Remove commented line.

211

You don't need this cast.

[eugenis]

lib/cfi/cfi.cc

177

The way it's done now, the "main" sanitizer calls Ubsan::InitAsPlugin and it also handles flags for ubsan. Replacing that with a runtime check could be brittle and would need to handle static vs shared linking, different initialization order (between the two runtime libraries), etc. Anyway, this is a bigger refactoring that would also affect asan+ubsan mode, I don't want to do it now.

[pcc]

LGTM with nit

lib/cfi/cfi.cc

178

Okay, let's not make things too complicated to start with. I'm happy with how things stand at the moment, but I think we should eventually resolve this by doing the refactoring.

249

Won't this request one fewer byte than needed? This might not actually matter, as MmapNoReserveOrDie rounds up to the page size, but I think we should at least leave a comment here.

[eugenis]

lib/cfi/cfi.cc

249

fixed

[eugenis]

r255695, thanks for the review!

cmake/config-ix.cmake

 set(ALL_LSAN_SUPPORTED_ARCH ${X86_64} ${MIPS64} ${ARM64})
 set(ALL_MSAN_SUPPORTED_ARCH ${X86_64} ${MIPS64} ${ARM64})
 set(ALL_PROFILE_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64} ${PPC64}
     ${MIPS32} ${MIPS64})
 set(ALL_TSAN_SUPPORTED_ARCH ${X86_64} ${MIPS64} ${ARM64} ${PPC64})
 set(ALL_UBSAN_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64}
     ${MIPS32} ${MIPS64} ${PPC64})
 set(ALL_SAFESTACK_SUPPORTED_ARCH ${X86} ${X86_64})
+set(ALL_CFI_SUPPORTED_ARCH ${X86} ${X86_64})
 
 if(APPLE)
   include(CompilerRTDarwinUtils)
 
   # On Darwin if /usr/include doesn't exist, the user probably has Xcode but not
   # the command line tools. If this is the case, we need to find the OS X
   # sysroot to pass to clang.
   if(NOT EXISTS /usr/include)
list_union(TSAN_SUPPORTED_ARCH
     ALL_TSAN_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
   list_union(UBSAN_SUPPORTED_ARCH
     ALL_UBSAN_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
   list_union(SAFESTACK_SUPPORTED_ARCH
     ALL_SAFESTACK_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
+  list_union(CFI_SUPPORTED_ARCH
+    ALL_CFI_SUPPORTED_ARCH
+    SANITIZER_COMMON_SUPPORTED_ARCH)
 else()
   # Architectures supported by compiler-rt libraries.
   filter_available_targets(BUILTIN_SUPPORTED_ARCH
     ${ALL_BUILTIN_SUPPORTED_ARCH})
   filter_available_targets(SANITIZER_COMMON_SUPPORTED_ARCH
     ${ALL_SANITIZER_COMMON_SUPPORTED_ARCH})
   # LSan and UBSan common files should be available on all architectures
   # supported by other sanitizers (even if they build into dummy object files).
   filter_available_targets(LSAN_COMMON_SUPPORTED_ARCH
     ${SANITIZER_COMMON_SUPPORTED_ARCH})
   filter_available_targets(UBSAN_COMMON_SUPPORTED_ARCH
     ${SANITIZER_COMMON_SUPPORTED_ARCH})
   filter_available_targets(ASAN_SUPPORTED_ARCH ${ALL_ASAN_SUPPORTED_ARCH})
   filter_available_targets(DFSAN_SUPPORTED_ARCH ${ALL_DFSAN_SUPPORTED_ARCH})
   filter_available_targets(LSAN_SUPPORTED_ARCH ${ALL_LSAN_SUPPORTED_ARCH})
   filter_available_targets(MSAN_SUPPORTED_ARCH ${ALL_MSAN_SUPPORTED_ARCH})
   filter_available_targets(PROFILE_SUPPORTED_ARCH ${ALL_PROFILE_SUPPORTED_ARCH})
   filter_available_targets(TSAN_SUPPORTED_ARCH ${ALL_TSAN_SUPPORTED_ARCH})
   filter_available_targets(UBSAN_SUPPORTED_ARCH ${ALL_UBSAN_SUPPORTED_ARCH})
   filter_available_targets(SAFESTACK_SUPPORTED_ARCH
     ${ALL_SAFESTACK_SUPPORTED_ARCH})
+  filter_available_targets(CFI_SUPPORTED_ARCH ${ALL_CFI_SUPPORTED_ARCH})
 endif()
 
 message(STATUS "Compiler-RT supported architectures: ${COMPILER_RT_SUPPORTED_ARCH}")
 
 if(ANDROID)
   set(OS_NAME "Android")
 else()
   set(OS_NAME "${CMAKE_SYSTEM_NAME}")

 endif()
 
 if (COMPILER_RT_HAS_SANITIZER_COMMON AND SAFESTACK_SUPPORTED_ARCH AND
     OS_NAME MATCHES "Darwin|Linux|FreeBSD")
   set(COMPILER_RT_HAS_SAFESTACK TRUE)
 else()
   set(COMPILER_RT_HAS_SAFESTACK FALSE)
 endif()
+
+if (COMPILER_RT_HAS_SANITIZER_COMMON AND CFI_SUPPORTED_ARCH AND
+    OS_NAME MATCHES "Linux")
+  set(COMPILER_RT_HAS_CFI TRUE)
+else()
+  set(COMPILER_RT_HAS_CFI FALSE)
+endif()

lib/cfi/CMakeLists.txt

 add_custom_target(cfi)
+
+set(CFI_SOURCES cfi.cc)
+
+include_directories(..)
+
+set(CFI_CFLAGS
+  ${SANITIZER_COMMON_CFLAGS}
+)
+
+foreach(arch ${CFI_SUPPORTED_ARCH})
+  add_compiler_rt_runtime(clang_rt.cfi
+    STATIC
+    ARCHS ${arch}
+    SOURCES ${CFI_SOURCES}
+    OBJECT_LIBS RTInterception
+                RTSanitizerCommon
+                RTSanitizerCommonLibc
+    RTUbsan
+    RTUbsan_cxx
+    CFLAGS ${CFI_CFLAGS}
+    PARENT_TARGET cfi)
+endforeach()
+
 add_compiler_rt_resource_file(cfi_blacklist cfi_blacklist.txt)
 add_dependencies(cfi cfi_blacklist)
 add_dependencies(compiler-rt cfi)

lib/cfi/cfi.cc

+//===-------- cfi.cc ------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the runtime support for the cross-DSO CFI.
+//
+//===----------------------------------------------------------------------===//
+
+// FIXME: Intercept dlopen/dlclose.
+// FIXME: Support diagnostic mode.
+// FIXME: Harden:
+//  * mprotect shadow, use mremap for updates
+//  * something else equally important
+
+#include <assert.h>
+#include <elf.h>
+#include <link.h>
+#include <string.h>
+
+typedef ElfW(Phdr) Elf_Phdr;
+typedef ElfW(Ehdr) Elf_Ehdr;
+
+#include "interception/interception.h"
+#include "sanitizer_common/sanitizer_common.h"
+#include "sanitizer_common/sanitizer_flag_parser.h"
+#include "ubsan/ubsan_init.h"
+#include "ubsan/ubsan_flags.h"
+
+static uptr __cfi_shadow;
+static constexpr uptr kShadowGranularity = 12;
+static constexpr uptr kShadowAlign = 1UL << kShadowGranularity; // 4096
+
+static constexpr uint16_t kInvalidShadow = 0;
+static constexpr uint16_t kUncheckedShadow = 0xFFFFU;
+
+static uint16_t *mem_to_shadow(uptr x) {
+  return (uint16_t *)(__cfi_shadow + ((x >> kShadowGranularity) << 1));
+}
+
+typedef int (*CFICheckFn)(uptr, void *);
+
+class ShadowValue {
+  uptr addr;
+  uint16_t v;
+  explicit ShadowValue(uptr addr, uint16_t v) : addr(addr), v(v) {}
+
+public:
+  bool is_invalid() const { return v == kInvalidShadow; }
+
+  bool is_unchecked() const { return v == kUncheckedShadow; }
+
+  CFICheckFn get_cfi_check() const {
+    assert(!is_invalid() && !is_unchecked());
+    uptr aligned_addr = addr & ~(kShadowAlign - 1);
+    uptr p = aligned_addr - (((uptr)v - 1) << kShadowGranularity);
+    return reinterpret_cast<CFICheckFn>(p);
+  }
+
+  // Load a shadow valud for the given application memory address.
+  static const ShadowValue load(uptr addr) {
+    return ShadowValue(addr, *mem_to_shadow(addr));
+  }
+};
+
+static void fill_shadow_constant(uptr begin, uptr end, uint16_t v) {
+  assert(v == kInvalidShadow || v == kUncheckedShadow);
+  uint16_t *shadow_begin = mem_to_shadow(begin);
+  uint16_t *shadow_end = mem_to_shadow(end - 1) + 1;
+  memset(shadow_begin, v, (shadow_end - shadow_begin) * sizeof(*shadow_begin));
+}
+
+static void fill_shadow(uptr begin, uptr end, uptr cfi_check) {
+  assert((cfi_check & (kShadowAlign - 1)) == 0);
+
+  // Don't fill anything below cfi_check. We can not represent those addresses
+  // in the shadow, and must make sure at codegen to place all valid call
+  // targets above cfi_check.
+  uptr p = Max(begin, cfi_check);
+  uint16_t *s = mem_to_shadow(p);
+  uint16_t *s_end = mem_to_shadow(end - 1) + 1;
+  uint16_t sv = ((p - cfi_check) >> kShadowGranularity) + 1;
+  for (; s < s_end; s++, sv++)
+    *s = sv;
+
+  // Sanity checks.
+  for (; p < end; p += kShadowAlign) {
+    assert((uptr)ShadowValue::load(p).get_cfi_check() == cfi_check);
+    assert((uptr)ShadowValue::load(p + kShadowAlign / 2).get_cfi_check() ==
+           cfi_check);
+    assert((uptr)ShadowValue::load(p + kShadowAlign - 1).get_cfi_check() ==
+           cfi_check);
+  }
+}
+
+// This is a workaround for a glibc bug:
+// https://sourceware.org/bugzilla/show_bug.cgi?id=15199
+// Other platforms can, hopefully, just do
+//    dlopen(RTLD_NOLOAD | RTLD_LAZY)
+//    dlsym("__cfi_check").
+static uptr find_cfi_check_in_dso(dl_phdr_info *info) {
+  const ElfW(Dyn) *dynamic = nullptr;
+  for (int i = 0; i < info->dlpi_phnum; ++i) {
+    if (info->dlpi_phdr[i].p_type == PT_DYNAMIC) {
+      dynamic =
+          (const ElfW(Dyn) *)(info->dlpi_addr + info->dlpi_phdr[i].p_vaddr);
+      break;
+    }
+  }
+  if (!dynamic) return 0;
+  uptr strtab = 0, symtab = 0;
+  for (const ElfW(Dyn) *p = dynamic; p->d_tag != PT_NULL; ++p) {
+    if (p->d_tag == DT_SYMTAB)
+      symtab = p->d_un.d_ptr;
+    else if (p->d_tag == DT_STRTAB)
+      strtab = p->d_un.d_ptr;
+  }
+
+  if (symtab > strtab) {
+    VReport(1, "Can not handle: symtab > strtab (%p > %zx)\n", symtab, strtab);
+    return 0;
+  }
+
+  // Verify that strtab and symtab are inside of the same LOAD segment.
+  // This excludes VDSO, which has (very high) bogus strtab and symtab pointers.
+  int phdr_idx;
+  for (phdr_idx = 0; phdr_idx < info->dlpi_phnum; phdr_idx++) {
+    const Elf_Phdr *phdr = &info->dlpi_phdr[phdr_idx];
+    if (phdr->p_type == PT_LOAD) {
+      uptr beg = info->dlpi_addr + phdr->p_vaddr;
+      uptr end = beg + phdr->p_memsz;
+      if (strtab >= beg && strtab < end && symtab >= beg && symtab < end)
+        break;
+    }
+  }
+  if (phdr_idx == info->dlpi_phnum) {
+    // Nope, either different segments or just bogus pointers.
+    // Can not handle this.
+    VReport(1, "Can not handle: symtab %p, strtab %zx\n", symtab, strtab);
+    return 0;
+  }
+
+  for (const ElfW(Sym) *p = (const ElfW(Sym) *)symtab; (ElfW(Addr))p < strtab;
+       ++p) {
+    char *name = (char*)(strtab + p->st_name);
+    if (strcmp(name, "__cfi_check") == 0) {
+      assert(p->st_info == ELF32_ST_INFO(STB_GLOBAL, STT_FUNC));
+      uptr addr = info->dlpi_addr + p->st_value;
+      return addr;
+    }
+  }
+  return 0;
+}
+
+static int dl_iterate_phdr_cb(dl_phdr_info *info, size_t size, void *data) {
+  uptr cfi_check = find_cfi_check_in_dso(info);
+  if (cfi_check)
+    VReport(1, "Module '%s' __cfi_check %zx\n", info->dlpi_name, cfi_check);
+
+  for (int i = 0; i < info->dlpi_phnum; i++) {
+    const Elf_Phdr *phdr = &info->dlpi_phdr[i];
+    if (phdr->p_type == PT_LOAD) {
+      // Jump tables are in the executable segment.
+      // VTables are in the non-executable one.
+      // Need to fill shadow for both.
+      // FIXME: reject writable if vtables are in the r/o segment. Depend on
+      // PT_RELRO?
+      uptr cur_beg = info->dlpi_addr + phdr->p_vaddr;
+      uptr cur_end = cur_beg + phdr->p_memsz;
+      if (cfi_check) {
+        VReport(1, "   %zx .. %zx\n", cur_beg, cur_end);
+        fill_shadow(cur_beg, cur_end, cfi_check ? cfi_check : (uptr)(-1));
+      } else {
+        fill_shadow_constant(cur_beg, cur_end, kInvalidShadow);
+      }
+    }
+  }
+  return 0;
+}
+
+// Fill shadow for the initial libraries.
+static void init_shadow() {
+  dl_iterate_phdr(dl_iterate_phdr_cb, nullptr);
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE extern "C"
+void __cfi_slowpath(uptr CallSiteTypeId, void *Ptr) {
+  uptr Addr = (uptr)Ptr;
+  VReport(3, "__cfi_slowpath: %zx, %p\n", CallSiteTypeId, Ptr);
+  ShadowValue sv = ShadowValue::load(Addr);
+  if (sv.is_invalid()) {
+    VReport(2, "CFI: invalid memory region for a function pointer (shadow==0): %p\n", Ptr);
+    Die();
+  }
+  if (sv.is_unchecked()) {
+    VReport(2, "CFI: unchecked call (shadow=FFFF): %p\n", Ptr);
+    return;
+  }
+  CFICheckFn cfi_check = sv.get_cfi_check();
+  VReport(2, "__cfi_check at %p\n", cfi_check);
+  cfi_check(CallSiteTypeId, Ptr);
+}
+
+static void InitializeFlags() {
+  SetCommonFlagsDefaults();
+  __ubsan::Flags *uf = __ubsan::flags();
+  uf->SetDefaults();
+
+  FlagParser cfi_parser;
+  RegisterCommonFlags(&cfi_parser);
+
+  FlagParser ubsan_parser;
+  __ubsan::RegisterUbsanFlags(&ubsan_parser, uf);
+  RegisterCommonFlags(&ubsan_parser);
+
+  const char *ubsan_default_options = __ubsan::MaybeCallUbsanDefaultOptions();
+  ubsan_parser.ParseString(ubsan_default_options);
+
+  cfi_parser.ParseString(GetEnv("CFI_OPTIONS"));
+  ubsan_parser.ParseString(GetEnv("UBSAN_OPTIONS"));
+
+  SetVerbosity(common_flags()->verbosity);
+
+  if (Verbosity()) ReportUnrecognizedFlags();
+
+  if (common_flags()->help) {
+    cfi_parser.PrintFlagDescriptions();
+  }
+}
+
+extern "C" __attribute__((visibility("default")))
+#if !SANITIZER_CAN_USE_PREINIT_ARRAY
+// On ELF platforms, the constructor is invoked using .preinit_array (see below)
+__attribute__((constructor(0)))
+#endif
+void __cfi_init() {
+  SanitizerToolName = "CFI";
+  InitializeFlags();
+
+  uptr vma = GetMaxVirtualAddress();
+  // Shadow is 2 -> 2**kShadowGranularity.
+  uptr shadow_size = (vma >> (kShadowGranularity - 1)) + 1;
+  VReport(1, "CFI: VMA size %zx, shadow size %zx\n", vma, shadow_size);
+  void *shadow = MmapNoReserveOrDie(shadow_size, "CFI shadow");
+  VReport(1, "CFI: shadow at %zx .. %zx\n", shadow,
+          reinterpret_cast<uptr>(shadow) + shadow_size);
+  __cfi_shadow = (uptr)shadow;
+  init_shadow();
+
+  __ubsan::InitAsPlugin();
+}
+
+#if SANITIZER_CAN_USE_PREINIT_ARRAY
+// On ELF platforms, run cfi initialization before any other constructors.
+// On other platforms we use the constructor attribute to arrange to run our
+// initialization early.
+extern "C" {
+__attribute__((section(".preinit_array"),
+               used)) void (*__cfi_preinit)(void) = __cfi_init;
+}
+#endif

test/cfi/CMakeLists.txt

 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
   )
 
 set(CFI_TEST_DEPS ${SANITIZER_COMMON_LIT_TEST_DEPS})
 if(NOT COMPILER_RT_STANDALONE_BUILD)
   list(APPEND CFI_TEST_DEPS
+    cfi
     opt
     ubsan
   )
   if(LLVM_ENABLE_PIC AND LLVM_BINUTILS_INCDIR)
     list(APPEND CFI_TEST_DEPS
       LLVMgold
     )
   endif()

test/cfi/cross-dso/icall/icall-from-dso.cpp

+// RUN: %clangxx_cfi_dso -DSHARED_LIB %s -fPIC -shared -o %t-so.so
+// RUN: %clangxx_cfi_dso %s -o %t %t-so.so && %expect_crash %t 2>&1 | FileCheck %s
+
+#include <stdio.h>
+
+#ifdef SHARED_LIB
+void g();
+void f() {
+  // CHECK: =1=
+  fprintf(stderr, "=1=\n");
+  ((void (*)(void))g)();
+  // CHECK: =2=
+  fprintf(stderr, "=2=\n");
+  ((void (*)(int))g)(42); // UB here
+  // CHECK-NOT: =3=
+  fprintf(stderr, "=3=\n");
+}
+#else
+void f();
+void g() {
+}
+
+int main() {
+  f();
+}
+#endif

test/cfi/cross-dso/icall/icall.cpp

+// RUN: %clangxx_cfi_dso -DSHARED_LIB %s -fPIC -shared -o %t-so.so
+// RUN: %clangxx_cfi_dso %s -o %t %t-so.so && %expect_crash %t 2>&1 | FileCheck %s
+
+#include <stdio.h>
+
+#ifdef SHARED_LIB
+void f() {
+}
+#else
+void f();
+int main() {
+  // CHECK: =1=
+  fprintf(stderr, "=1=\n");
+  ((void (*)(void))f)();
+  // CHECK: =2=
+  fprintf(stderr, "=2=\n");
+  ((void (*)(int))f)(42); // UB here
+  // CHECK-NOT: =3=
+  fprintf(stderr, "=3=\n");
+}
+#endif

test/cfi/cross-dso/icall/lit.local.cfg

+# The cfi-icall checker is only supported on x86 and x86_64 for now.
+if config.root.host_arch not in ['x86', 'x86_64']:
+  config.unsupported = True

test/cfi/cross-dso/simple-fail.cpp

+// RUN: %clangxx_cfi_dso -DSHARED_LIB %s -fPIC -shared -o %t1-so.so
+// RUN: %clangxx_cfi_dso %s -o %t1 %t1-so.so
+// RUN: %expect_crash %t1 2>&1 | FileCheck --check-prefix=CFI %s
+// RUN: %expect_crash %t1 x 2>&1 | FileCheck --check-prefix=CFI-CAST %s
+
+// RUN: %clangxx_cfi_dso -DB32 -DSHARED_LIB %s -fPIC -shared -o %t2-so.so
+// RUN: %clangxx_cfi_dso -DB32 %s -o %t2 %t2-so.so
+// RUN: %expect_crash %t2 2>&1 | FileCheck --check-prefix=CFI %s
+// RUN: %expect_crash %t2 x 2>&1 | FileCheck --check-prefix=CFI-CAST %s
+
+// RUN: %clangxx_cfi_dso -DB64 -DSHARED_LIB %s -fPIC -shared -o %t3-so.so
+// RUN: %clangxx_cfi_dso -DB64 %s -o %t3 %t3-so.so
+// RUN: %expect_crash %t3 2>&1 | FileCheck --check-prefix=CFI %s
+// RUN: %expect_crash %t3 x 2>&1 | FileCheck --check-prefix=CFI-CAST %s
+
+// RUN: %clangxx_cfi_dso -DBM -DSHARED_LIB %s -fPIC -shared -o %t4-so.so
+// RUN: %clangxx_cfi_dso -DBM %s -o %t4 %t4-so.so
+// RUN: %expect_crash %t4 2>&1 | FileCheck --check-prefix=CFI %s
+// RUN: %expect_crash %t4 x 2>&1 | FileCheck --check-prefix=CFI-CAST %s
+
+// RUN: %clangxx -DBM -DSHARED_LIB %s -fPIC -shared -o %t5-so.so
+// RUN: %clangxx -DBM %s -o %t5 %t5-so.so
+// RUN: %t5 2>&1 | FileCheck --check-prefix=NCFI %s
+// RUN: %t5 x 2>&1 | FileCheck --check-prefix=NCFI %s
+
+// Tests that the CFI mechanism crashes the program when making a virtual call
+// to an object of the wrong class but with a compatible vtable, by casting a
+// pointer to such an object and attempting to make a call through it.
+
+// REQUIRES: cxxabi
+
+#include <stdio.h>
+#include <string.h>
+
+struct A {
+  virtual void f();
+};
+
+void *create_B();
+
+#ifdef SHARED_LIB
+
+#include "../utils.h"
+struct B {
+  virtual void f();
+};
+void B::f() {}
+
+void *create_B() {
+  create_derivers<B>();
+  return (void *)(new B());
+}
+
+#else
+
+void A::f() {}
+
+int main(int argc, char *argv[]) {
+  void *p = create_B();
+  A *a;
+
+  // CFI: =0=
+  // CFI-CAST: =0=
+  // NCFI: =0=
+  fprintf(stderr, "=0=\n");
+
+  if (argc > 1 && argv[1][0] == 'x') {
+    // Test cast. BOOM.
+    a = (A*)p;
+  } else {
+    // Invisible to CFI. Test virtual call later.
+    memcpy(&a, &p, sizeof(a));
+  }
+
+  // CFI: =1=
+  // CFI-CAST-NOT: =1=
+  // NCFI: =1=
+  fprintf(stderr, "=1=\n");
+
+  a->f(); // UB here
+
+  // CFI-NOT: =2=
+  // CFI-CAST-NOT: =2=
+  // NCFI: =2=
+  fprintf(stderr, "=2=\n");
+}
+#endif

test/cfi/cross-dso/simple-pass.cpp

+// RUN: %clangxx_cfi_dso -DSHARED_LIB %s -fPIC -shared -o %t1-so.so
+// RUN: %clangxx_cfi_dso %s -o %t1 %t1-so.so
+// RUN: %t1 2>&1 | FileCheck --check-prefix=CFI %s
+
+// RUN: %clangxx_cfi_dso -DB32 -DSHARED_LIB %s -fPIC -shared -o %t2-so.so
+// RUN: %clangxx_cfi_dso -DB32 %s -o %t2 %t2-so.so
+// RUN: %t2 2>&1 | FileCheck --check-prefix=CFI %s
+
+// RUN: %clangxx_cfi_dso -DB64 -DSHARED_LIB %s -fPIC -shared -o %t3-so.so
+// RUN: %clangxx_cfi_dso -DB64 %s -o %t3 %t3-so.so
+// RUN: %t3 2>&1 | FileCheck --check-prefix=CFI %s
+
+// RUN: %clangxx_cfi_dso -DBM -DSHARED_LIB %s -fPIC -shared -o %t4-so.so
+// RUN: %clangxx_cfi_dso -DBM %s -o %t4 %t4-so.so
+// RUN: %t4 2>&1 | FileCheck --check-prefix=CFI %s
+
+// RUN: %clangxx -DBM -DSHARED_LIB %s -fPIC -shared -o %t5-so.so
+// RUN: %clangxx -DBM %s -o %t5 %t5-so.so
+// RUN: %t5 2>&1 | FileCheck --check-prefix=NCFI %s
+// RUN: %t5 x 2>&1 | FileCheck --check-prefix=NCFI %s
+
+// Tests that the CFI mechanism crashes the program when making a virtual call
+// to an object of the wrong class but with a compatible vtable, by casting a
+// pointer to such an object and attempting to make a call through it.
+
+// REQUIRES: cxxabi
+
+#include <stdio.h>
+#include <string.h>
+
+struct A {
+  virtual void f();
+};
+
+A *create_B();
+
+#ifdef SHARED_LIB
+
+#include "../utils.h"
+struct B : public A {
+  virtual void f();
+};
+void B::f() {}
+
+A *create_B() {
+  create_derivers<B>();
+  return new B();
+}
+
+#else
+
+void A::f() {}
+
+int main(int argc, char *argv[]) {
+  A *a = create_B();
+
+  // CFI: =1=
+  // NCFI: =1=
+  fprintf(stderr, "=1=\n");
+  a->f(); // OK
+  // CFI: =2=
+  // NCFI: =2=
+  fprintf(stderr, "=2=\n");
+}
+#endif

test/cfi/lit.cfg

 import lit.formats
 import os
 
 config.name = 'cfi'
 config.suffixes = ['.c', '.cpp', '.test']
 config.test_source_root = os.path.dirname(__file__)
 
 clangxx = ' '.join([config.clang] + config.cxx_mode_flags)
 
 config.substitutions.append((r"%clangxx ", clangxx + ' '))
 if config.lto_supported:
   clangxx_cfi = ' '.join(config.lto_launch + [clangxx] + config.lto_flags + ['-flto -fsanitize=cfi '])
+  clangxx_cfi_diag = clangxx_cfi + '-fno-sanitize-trap=cfi -fsanitize-recover=cfi '
   config.substitutions.append((r"%clangxx_cfi ", clangxx_cfi))
-  config.substitutions.append((r"%clangxx_cfi_diag ", clangxx_cfi + '-fno-sanitize-trap=cfi -fsanitize-recover=cfi '))
+  config.substitutions.append((r"%clangxx_cfi_diag ", clangxx_cfi_diag))
+  config.substitutions.append((r"%clangxx_cfi_dso ", clangxx_cfi + '-fsanitize-cfi-cross-dso '))
+  config.substitutions.append((r"%clangxx_cfi_dso_diag ", clangxx_cfi_diag + '-fsanitize-cfi-cross-dso '))
 else:
   config.unsupported = True
 
 if lit_config.params.get('check_supported', None) and config.unsupported:
   raise BaseException("Tests unsupported")