[cfi] Support for dlopen and dlclose

[eugenis]

Summary

Add dlopen/dlclose interceptors to update CFI shadow for loaded/unloaded libraries.

[pcc]

lib/cfi/cfi.cc

297

As you mentioned offline, this won't correctly handle dependencies.

Perhaps the code should be building the shadow from scratch using dl_iterate_phdr every time a rebuild is needed? That way, you save needing to worry about dependencies. This may require a redesign of D16098 to avoid the copy.

[eugenis]

A new approach.
dlclose may unmap (and dlopen may map) multiple libraries, so we actually need to do the entire dl_iterate_phdr run and update all libraries. For dlclose, we also need to detect libraries that have disappeared from the address space, so we do dl_iterate_phdr before and after dlclose and compare the results.

[pcc]

lib/cfi/cfi.cc

41

Looks like we already have a kMaxNumberOfModules constant defined.

228

I changed this interface in r257858.

279–305

Can you avoid needing to take the set difference by rebuilding from scratch?

lib/sanitizer_common/sanitizer_common.cc

341 ↗

(On Diff #44918)

Should there be a single init() function that combines set and set_phdr?

lib/sanitizer_common/sanitizer_common.h

649

Is this function needed? Looks like you're passing a comparator to your sort function above.

[eugenis]

lib/cfi/cfi.cc

279–305

Do you mean unmap or clean the entire shadow and then add the existing modules? That would be racy, we can not ensure that there are no concurrent virtual/indirect calls.

lib/sanitizer_common/sanitizer_common.h

649

removed

[pcc]

lib/cfi/cfi.cc

279–305

Yes, that's why I was thinking this would need to be done after D16098. If it's too painful to re-order the patches I'd be fine with combining them.

[eugenis]

Yet another new approach, way simpler than before.
Deprecates D16098.

[eugenis]

The previous approach contained a data race on the shadow pointer, which can not be updated without a lock in __cfi_slowpath. That would not be acceptable performance-wise. Probably.

This new version is linux-only as it requires mremap().

[eugenis]

how does it look?

[pcc]

lib/cfi/cfi.cc

29–35

Maybe assert somewhere that this is equal to GetPageSize()?

33

This name is unclear. It should reflect that the shadow pointer is stored here, not the shadow itself.

48

Please inline into ShadowBuilder::Install, as the pre-conditions of this function aren't clear without context.

126

Do you need to set the memory protection for shadow_ first?

292–299

You can use it now I guess.

lib/sanitizer_common/sanitizer_posix.cc

178

This function is unused.

[eugenis]

lib/cfi/cfi.cc

126

Of course.
Good catch.

[pcc]

lib/cfi/cfi.cc

13–14

This is done now, right?

126

Thanks. Is it possible to construct a test case that makes sure we get the memory protection right? I was thinking that the test case would try to write to the shadow at various points (e.g. start of main, after dlopen, after dlclose), and use %expect_crash to test that the write fails.

[eugenis]

lib/cfi/cfi.cc

126

But we don't really want there to be an easy way to find the cfi shadow.

[eugenis]

I'll move most of the implementation under a namespace in a separate commit.

[pcc]

LGTM

[eugenis]

Thanks, http://reviews.llvm.org/rL258857

lib/cfi/cfi.cc

 //===-------- cfi.cc ------------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the runtime support for the cross-DSO CFI.
 //
 //===----------------------------------------------------------------------===//
 
-// FIXME: Intercept dlopen/dlclose.
-// FIXME: Support diagnostic mode.
-// FIXME: Harden:
-//  * mprotect shadow, use mremap for updates
-//  * something else equally important
-
 #include <assert.h>
 #include <elf.h>
 #include <link.h>
 #include <string.h>
+#include <sys/mman.h>
 
 typedef ElfW(Phdr) Elf_Phdr;
 typedef ElfW(Ehdr) Elf_Ehdr;
 
 #include "interception/interception.h"
 #include "sanitizer_common/sanitizer_common.h"
 #include "sanitizer_common/sanitizer_flag_parser.h"
 #include "ubsan/ubsan_init.h"
 #include "ubsan/ubsan_flags.h"
 
-static uptr __cfi_shadow;
+static constexpr uptr kCfiShadowPointerStorageSize = 4096; // 1 page
+// Lets hope that the data segment is mapped with 4K pages.
+// The pointer to the cfi shadow region is stored at the start of this page.
+// The rest of the page is unused, re-mapped read-only and updated via mremap.
+static char __cfi_shadow_pointer_storage[kCfiShadowPointerStorageSize]
+    __attribute__((aligned(kCfiShadowPointerStorageSize)));
+static uptr __cfi_shadow_size;
 static constexpr uptr kShadowGranularity = 12;
 static constexpr uptr kShadowAlign = 1UL << kShadowGranularity; // 4096
 
 static constexpr uint16_t kInvalidShadow = 0;
 static constexpr uint16_t kUncheckedShadow = 0xFFFFU;
 
-static uint16_t *mem_to_shadow(uptr x) {
-  return (uint16_t *)(__cfi_shadow + ((x >> kShadowGranularity) << 1));
+// Get the start address of the CFI shadow region.
+uptr GetShadow() {
+  return *reinterpret_cast<uptr *>(&__cfi_shadow_pointer_storage);
+}
+
+static uint16_t *MemToShadow(uptr x, uptr shadow_base) {
+  return (uint16_t *)(shadow_base + ((x >> kShadowGranularity) << 1));
 }
 
 typedef int (*CFICheckFn)(u64, void *, void *);
 
+// This class reads and decodes the shadow contents.
 class ShadowValue {
   uptr addr;
   uint16_t v;
   explicit ShadowValue(uptr addr, uint16_t v) : addr(addr), v(v) {}
 
 public:
   bool is_invalid() const { return v == kInvalidShadow; }
 
   bool is_unchecked() const { return v == kUncheckedShadow; }
 
   CFICheckFn get_cfi_check() const {
     assert(!is_invalid() && !is_unchecked());
     uptr aligned_addr = addr & ~(kShadowAlign - 1);
     uptr p = aligned_addr - (((uptr)v - 1) << kShadowGranularity);
     return reinterpret_cast<CFICheckFn>(p);
   }
 
   // Load a shadow valud for the given application memory address.
   static const ShadowValue load(uptr addr) {
-    return ShadowValue(addr, *mem_to_shadow(addr));
+    return ShadowValue(addr, *MemToShadow(addr, GetShadow()));
   }
 };
 
-static void fill_shadow_constant(uptr begin, uptr end, uint16_t v) {
-  assert(v == kInvalidShadow || v == kUncheckedShadow);
-  uint16_t *shadow_begin = mem_to_shadow(begin);
-  uint16_t *shadow_end = mem_to_shadow(end - 1) + 1;
-  memset(shadow_begin, v, (shadow_end - shadow_begin) * sizeof(*shadow_begin));
+class ShadowBuilder {
+  uptr shadow_;
+
+public:
+  // Allocate a new empty shadow (for the entire address space) on the side.
+  void Start();
+  // Mark the given address range as unchecked.
+  // This is used for uninstrumented libraries like libc.
+  // Any CFI check with a target in that range will pass.
+  void AddUnchecked(uptr begin, uptr end);
+  // Mark the given address range as belonging to a library with the given
+  // cfi_check function.
+  void Add(uptr begin, uptr end, uptr cfi_check);
+  // Finish shadow construction. Atomically switch the current active shadow
+  // region with the newly constructed one and deallocate the former.
+  void Install();
+};
+
+void ShadowBuilder::Start() {
+  shadow_ = (uptr)MmapNoReserveOrDie(__cfi_shadow_size, "CFI shadow");
+  VReport(1, "CFI: shadow at %zx .. %zx\n", shadow_,
+          shadow_ + __cfi_shadow_size);
+}
+
+void ShadowBuilder::AddUnchecked(uptr begin, uptr end) {
+  uint16_t *shadow_begin = MemToShadow(begin, shadow_);
+  uint16_t *shadow_end = MemToShadow(end - 1, shadow_) + 1;
+  memset(shadow_begin, kUncheckedShadow,
+         (shadow_end - shadow_begin) * sizeof(*shadow_begin));
 }
 
-static void fill_shadow(uptr begin, uptr end, uptr cfi_check) {
+void ShadowBuilder::Add(uptr begin, uptr end, uptr cfi_check) {
   assert((cfi_check & (kShadowAlign - 1)) == 0);
 
   // Don't fill anything below cfi_check. We can not represent those addresses
   // in the shadow, and must make sure at codegen to place all valid call
   // targets above cfi_check.
-  uptr p = Max(begin, cfi_check);
-  uint16_t *s = mem_to_shadow(p);
-  uint16_t *s_end = mem_to_shadow(end - 1) + 1;
-  uint16_t sv = ((p - cfi_check) >> kShadowGranularity) + 1;
+  begin = Max(begin, cfi_check);
+  uint16_t *s = MemToShadow(begin, shadow_);
+  uint16_t *s_end = MemToShadow(end - 1, shadow_) + 1;
+  uint16_t sv = ((begin - cfi_check) >> kShadowGranularity) + 1;
   for (; s < s_end; s++, sv++)
     *s = sv;
+}
 
-  // Sanity checks.
-  uptr q = p & ~(kShadowAlign - 1);
-  for (; q < end; q += kShadowAlign) {
-    assert((uptr)ShadowValue::load(q).get_cfi_check() == cfi_check);
-    assert((uptr)ShadowValue::load(q + kShadowAlign / 2).get_cfi_check() ==
-           cfi_check);
-    assert((uptr)ShadowValue::load(q + kShadowAlign - 1).get_cfi_check() ==
-           cfi_check);
+#if SANITIZER_LINUX
+void ShadowBuilder::Install() {
+  MprotectReadOnly(shadow_, __cfi_shadow_size);
+  uptr main_shadow = GetShadow();
+  if (main_shadow) {
+    // Update.
+    void *res = mremap((void *)shadow_, __cfi_shadow_size, __cfi_shadow_size,
+                       MREMAP_MAYMOVE | MREMAP_FIXED, main_shadow);
+    CHECK(res != MAP_FAILED);
+  } else {
+    // Initial setup.
+    CHECK_EQ(kCfiShadowPointerStorageSize, GetPageSizeCached());
+    CHECK_EQ(0, GetShadow());
+    *reinterpret_cast<uptr *>(&__cfi_shadow_pointer_storage) = shadow_;
+    MprotectReadOnly((uptr)&__cfi_shadow_pointer_storage,
+                     kCfiShadowPointerStorageSize);
+    CHECK_EQ(shadow_, GetShadow());
   }
 }
+#else
+#error not implemented
+#endif
 
 // This is a workaround for a glibc bug:
 // https://sourceware.org/bugzilla/show_bug.cgi?id=15199
 // Other platforms can, hopefully, just do
 //    dlopen(RTLD_NOLOAD | RTLD_LAZY)
 //    dlsym("__cfi_check").
 static uptr find_cfi_check_in_dso(dl_phdr_info *info) {
   const ElfW(Dyn) *dynamic = nullptr;
static uptr find_cfi_check_in_dso(dl_phdr_info *info) {
   return 0;
 }
 
 static int dl_iterate_phdr_cb(dl_phdr_info *info, size_t size, void *data) {
   uptr cfi_check = find_cfi_check_in_dso(info);
   if (cfi_check)
     VReport(1, "Module '%s' __cfi_check %zx\n", info->dlpi_name, cfi_check);
 
+  ShadowBuilder *b = reinterpret_cast<ShadowBuilder *>(data);
+
   for (int i = 0; i < info->dlpi_phnum; i++) {
     const Elf_Phdr *phdr = &info->dlpi_phdr[i];
     if (phdr->p_type == PT_LOAD) {
       // Jump tables are in the executable segment.
       // VTables are in the non-executable one.
       // Need to fill shadow for both.
       // FIXME: reject writable if vtables are in the r/o segment. Depend on
       // PT_RELRO?
       uptr cur_beg = info->dlpi_addr + phdr->p_vaddr;
       uptr cur_end = cur_beg + phdr->p_memsz;
       if (cfi_check) {
         VReport(1, "   %zx .. %zx\n", cur_beg, cur_end);
-        fill_shadow(cur_beg, cur_end, cfi_check ? cfi_check : (uptr)(-1));
+        b->Add(cur_beg, cur_end, cfi_check);
       } else {
-        fill_shadow_constant(cur_beg, cur_end, kUncheckedShadow);
+        b->AddUnchecked(cur_beg, cur_end);
       }
     }
   }
   return 0;
 }
 
-// Fill shadow for the initial libraries.
-static void init_shadow() {
-  dl_iterate_phdr(dl_iterate_phdr_cb, nullptr);
+// Init or update shadow for the current set of loaded libraries.
+static void UpdateShadow() {
+  ShadowBuilder b;
+  b.Start();
+  dl_iterate_phdr(dl_iterate_phdr_cb, &b);
+  b.Install();
+}
+
+static void InitShadow() {
+  // No difference, really.
+  UpdateShadow();
+}
+
+static THREADLOCAL int in_loader;
+static BlockingMutex shadow_update_lock(LINKER_INITIALIZED);
+
+static void EnterLoader() {
+  if (in_loader == 0) {
+    shadow_update_lock.Lock();
+  }
+  ++in_loader;
+}
+
+static void ExitLoader() {
+  CHECK(in_loader > 0);
+  --in_loader;
+  UpdateShadow();
+  if (in_loader == 0) {
+    shadow_update_lock.Unlock();
+  }
+}
+
+// Setup shadow for dlopen()ed libraries.
+// The actual shadow setup happens after dlopen() returns, which means that
+// a library can not be a target of any CFI checks while its constructors are
+// running. It's unclear how to fix this without some extra help from libc.
+// In glibc, mmap inside dlopen is not interceptable.
+// Maybe a seccomp-bpf filter?
+// We could insert a high-priority constructor into the library, but that would
+// not help with the uninstrumented libraries.
+INTERCEPTOR(void*, dlopen, const char *filename, int flag) {
+  EnterLoader();
+  void *handle = REAL(dlopen)(filename, flag);
+  ExitLoader();
+  return handle;
+}
+
+INTERCEPTOR(int, dlclose, void *handle) {
+  EnterLoader();
+  int res = REAL(dlclose)(handle);
+  ExitLoader();
+  return res;
 }
 
 static ALWAYS_INLINE void CfiSlowPathCommon(u64 CallSiteTypeId, void *Ptr,
                                             void *DiagData) {
   uptr Addr = (uptr)Ptr;
   VReport(3, "__cfi_slowpath: %llx, %p\n", CallSiteTypeId, Ptr);
   ShadowValue sv = ShadowValue::load(Addr);
   if (sv.is_invalid()) {
     // FIXME: call the ubsan handler if DiagData != nullptr?
     Report(
         "CFI: invalid memory region for a function pointer (shadow==0): %p\n",
         Ptr);
     if (DiagData)
       return;
     else
-      Die();
+      Trap();
   }
   if (sv.is_unchecked()) {
     VReport(2, "CFI: unchecked call (shadow=FFFF): %p\n", Ptr);
     return;
   }
   CFICheckFn cfi_check = sv.get_cfi_check();
   VReport(2, "__cfi_check at %p\n", cfi_check);
   cfi_check(CallSiteTypeId, Ptr, DiagData);
 }
 
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE void
 __cfi_slowpath(u64 CallSiteTypeId, void *Ptr) {
   CfiSlowPathCommon(CallSiteTypeId, Ptr, nullptr);
 }

 __attribute__((constructor(0)))
 #endif
 void __cfi_init() {
   SanitizerToolName = "CFI";
   InitializeFlags();
 
   uptr vma = GetMaxVirtualAddress();
   // Shadow is 2 -> 2**kShadowGranularity.
-  uptr shadow_size = (vma >> (kShadowGranularity - 1)) + 1;
-  VReport(1, "CFI: VMA size %zx, shadow size %zx\n", vma, shadow_size);
-  void *shadow = MmapNoReserveOrDie(shadow_size, "CFI shadow");
-  VReport(1, "CFI: shadow at %zx .. %zx\n", shadow,
-          reinterpret_cast<uptr>(shadow) + shadow_size);
-  __cfi_shadow = (uptr)shadow;
-  init_shadow();
+  __cfi_shadow_size = (vma >> (kShadowGranularity - 1)) + 1;
+  VReport(1, "CFI: VMA size %zx, shadow size %zx\n", vma, __cfi_shadow_size);
+
+  InitShadow();
+
+  INTERCEPT_FUNCTION(dlopen);
+  INTERCEPT_FUNCTION(dlclose);
 
 #ifdef CFI_ENABLE_DIAG
   __ubsan::InitAsPlugin();
 #endif
 }
 
 #if SANITIZER_CAN_USE_PREINIT_ARRAY
 // On ELF platforms, run cfi initialization before any other constructors.
 // On other platforms we use the constructor attribute to arrange to run our
 // initialization early.
 extern "C" {
 __attribute__((section(".preinit_array"),
                used)) void (*__cfi_preinit)(void) = __cfi_init;
 }
 #endif

lib/sanitizer_common/sanitizer_common.h

 void *MmapNoReserveOrDie(uptr size, const char *mem_type);
 void *MmapFixedOrDie(uptr fixed_addr, uptr size);
 void *MmapNoAccess(uptr fixed_addr, uptr size, const char *name = nullptr);
 // Map aligned chunk of address space; size and alignment are powers of two.
 void *MmapAlignedOrDie(uptr size, uptr alignment, const char *mem_type);
 // Disallow access to a memory range.  Use MmapNoAccess to allocate an
 // unaccessible memory.
 bool MprotectNoAccess(uptr addr, uptr size);
+bool MprotectReadOnly(uptr addr, uptr size);
 
 // Used to check if we can map shadow memory to a fixed location.
 bool MemoryRangeIsAvailable(uptr range_start, uptr range_end);
 void FlushUnneededShadowMemory(uptr addr, uptr size);
 void IncreaseTotalMmap(uptr size);
 void DecreaseTotalMmap(uptr size);
 uptr GetRSS();
 void NoHugePagesInRegion(uptr addr, uptr length);
struct AddressRange {
     bool executable;
 
     AddressRange(uptr beg, uptr end, bool executable)
         : next(nullptr), beg(beg), end(end), executable(executable) {}
   };
 
   const IntrusiveList<AddressRange> &ranges() const { return ranges_; }
 
  private:
   char *full_name_;  // Owned.
   uptr base_address_;
   IntrusiveList<AddressRange> ranges_;
 };
 
 // OS-dependent function that fills array with descriptions of at most
 // "max_modules" currently loaded modules. Returns the number of
 // initialized modules. If filter is nonzero, ignores modules for which

lib/sanitizer_common/sanitizer_posix.cc

void *MmapFixedOrDie(uptr fixed_addr, uptr size) {
   IncreaseTotalMmap(size);
   return (void *)p;
 }
 
 bool MprotectNoAccess(uptr addr, uptr size) {
   return 0 == internal_mprotect((void*)addr, size, PROT_NONE);
 }
 
+bool MprotectReadOnly(uptr addr, uptr size) {
+  return 0 == internal_mprotect((void *)addr, size, PROT_READ);
+}
+
 fd_t OpenFile(const char *filename, FileAccessMode mode, error_t *errno_p) {
   int flags;
   switch (mode) {
     case RdOnly: flags = O_RDONLY; break;
     case WrOnly: flags = O_WRONLY | O_CREAT; break;
     case RdWr: flags = O_RDWR | O_CREAT; break;
   }
   fd_t res = internal_open(filename, flags, 0660);
   if (internal_iserror(res, errno_p))

test/cfi/cross-dso/dlopen.cpp

+// RUN: %clangxx_cfi_dso -DSHARED_LIB %s -fPIC -shared -o %t1-so.so
+// RUN: %clangxx_cfi_dso %s -o %t1
+// RUN: %expect_crash %t1 2>&1 | FileCheck --check-prefix=CFI %s
+// RUN: %expect_crash %t1 cast 2>&1 | FileCheck --check-prefix=CFI-CAST %s
+// RUN: %expect_crash %t1 dlclose 2>&1 | FileCheck --check-prefix=CFI %s
+
+// RUN: %clangxx_cfi_dso -DB32 -DSHARED_LIB %s -fPIC -shared -o %t2-so.so
+// RUN: %clangxx_cfi_dso -DB32 %s -o %t2
+// RUN: %expect_crash %t2 2>&1 | FileCheck --check-prefix=CFI %s
+// RUN: %expect_crash %t2 cast 2>&1 | FileCheck --check-prefix=CFI-CAST %s
+// RUN: %expect_crash %t2 dlclose 2>&1 | FileCheck --check-prefix=CFI %s
+
+// RUN: %clangxx_cfi_dso -DB64 -DSHARED_LIB %s -fPIC -shared -o %t3-so.so
+// RUN: %clangxx_cfi_dso -DB64 %s -o %t3
+// RUN: %expect_crash %t3 2>&1 | FileCheck --check-prefix=CFI %s
+// RUN: %expect_crash %t3 cast 2>&1 | FileCheck --check-prefix=CFI-CAST %s
+// RUN: %expect_crash %t3 dlclose 2>&1 | FileCheck --check-prefix=CFI %s
+
+// RUN: %clangxx_cfi_dso -DBM -DSHARED_LIB %s -fPIC -shared -o %t4-so.so
+// RUN: %clangxx_cfi_dso -DBM %s -o %t4
+// RUN: %expect_crash %t4 2>&1 | FileCheck --check-prefix=CFI %s
+// RUN: %expect_crash %t4 cast 2>&1 | FileCheck --check-prefix=CFI-CAST %s
+// RUN: %expect_crash %t4 dlclose 2>&1 | FileCheck --check-prefix=CFI %s
+
+// RUN: %clangxx -g -DBM -DSHARED_LIB -DNOCFI %s -fPIC -shared -o %t5-so.so
+// RUN: %clangxx -g -DBM -DNOCFI %s -ldl -o %t5
+// RUN: %t5 2>&1 | FileCheck --check-prefix=NCFI %s
+// RUN: %t5 cast 2>&1 | FileCheck --check-prefix=NCFI %s
+// RUN: %t5 dlclose 2>&1 | FileCheck --check-prefix=NCFI %s
+
+// Test that calls to uninstrumented library are unchecked.
+// RUN: %clangxx -DBM -DSHARED_LIB %s -fPIC -shared -o %t6-so.so
+// RUN: %clangxx_cfi_dso -DBM %s -o %t6
+// RUN: %t6 2>&1 | FileCheck --check-prefix=NCFI %s
+// RUN: %t6 cast 2>&1 | FileCheck --check-prefix=NCFI %s
+
+// Call-after-dlclose is checked on the caller side.
+// RUN: %expect_crash %t6 dlclose 2>&1 | FileCheck --check-prefix=CFI %s
+
+// Tests calls into dlopen-ed library.
+// REQUIRES: cxxabi
+
+#include <assert.h>
+#include <dlfcn.h>
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/mman.h>
+
+struct A {
+  virtual void f();
+};
+
+#ifdef SHARED_LIB
+
+#include "../utils.h"
+struct B {
+  virtual void f();
+};
+void B::f() {}
+
+extern "C" void *create_B() {
+  create_derivers<B>();
+  return (void *)(new B());
+}
+
+extern "C" void do_nothing() __attribute__((aligned(4096))) {}
+
+#else
+
+void A::f() {}
+
+static const int kCodeAlign = 4096;
+static const int kCodeSize = 4096;
+static char saved_code[kCodeSize];
+static char *real_start;
+
+static void save_code(char *p) {
+  real_start = (char *)(((uintptr_t)p) & ~(kCodeAlign - 1));
+  memcpy(saved_code, real_start, kCodeSize);
+}
+
+static void restore_code() {
+  char *code = (char *)mmap(real_start, kCodeSize, PROT_WRITE | PROT_EXEC,
+                            MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, 0, 0);
+  assert(code == real_start);
+  memcpy(code, saved_code, kCodeSize);
+}
+
+int main(int argc, char *argv[]) {
+  const bool test_cast = argc > 1 && strcmp(argv[1], "cast") == 0;
+  const bool test_dlclose = argc > 1 && strcmp(argv[1], "dlclose") == 0;
+
+  char name[100];
+  snprintf(name, sizeof(name), "%s-so.so", argv[0]);
+  void *handle = dlopen(name, RTLD_NOW);
+  assert(handle);
+  void *(*create_B)() = (void *(*)())dlsym(handle, "create_B");
+  assert(create_B);
+
+  void *p = create_B();
+  A *a;
+
+  // CFI: =0=
+  // CFI-CAST: =0=
+  // NCFI: =0=
+  fprintf(stderr, "=0=\n");
+
+  if (test_cast) {
+    // Test cast. BOOM.
+    a = (A*)p;
+  } else {
+    // Invisible to CFI. Test virtual call later.
+    memcpy(&a, &p, sizeof(a));
+  }
+
+  // CFI: =1=
+  // CFI-CAST-NOT: =1=
+  // NCFI: =1=
+  fprintf(stderr, "=1=\n");
+
+  if (test_dlclose) {
+    // Imitate an attacker sneaking in an executable page where a dlclose()d
+    // library was loaded. This needs to pass w/o CFI, so for the testing
+    // purpose, we just copy the bytes of a "void f() {}" function back and
+    // forth.
+    void (*do_nothing)() = (void (*)())dlsym(handle, "do_nothing");
+    assert(do_nothing);
+    save_code((char *)do_nothing);
+
+    int res = dlclose(handle);
+    assert(res == 0);
+
+    restore_code();
+
+    do_nothing(); // UB here
+  } else {
+    a->f(); // UB here
+  }
+
+  // CFI-NOT: =2=
+  // CFI-CAST-NOT: =2=
+  // NCFI: =2=
+  fprintf(stderr, "=2=\n");
+}
+#endif

test/cfi/cross-dso/shadow_is_read_only.cpp

+// RUN: %clangxx_cfi_dso -std=c++11 -g -DSHARED_LIB %s -fPIC -shared -o %t-cfi-so.so
+// RUN: %clangxx -std=c++11 -g -DSHARED_LIB %s -fPIC -shared -o %t-nocfi-so.so
+// RUN: %clangxx_cfi_dso -std=c++11 -g %s -o %t
+
+// RUN: %expect_crash %t start 2>&1 | FileCheck %s
+// RUN: %expect_crash %t mmap 2>&1 | FileCheck %s
+// RUN: %expect_crash %t dlopen %t-cfi-so.so 2>&1 | FileCheck %s
+// RUN: %expect_crash %t dlclose %t-cfi-so.so 2>&1 | FileCheck %s
+// RUN: %expect_crash %t dlopen %t-nocfi-so.so 2>&1 | FileCheck %s
+// RUN: %expect_crash %t dlclose %t-nocfi-so.so 2>&1 | FileCheck %s
+
+// Tests that shadow is read-only most of the time.
+// REQUIRES: cxxabi
+
+#include <assert.h>
+#include <dlfcn.h>
+#include <stdio.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+
+struct A {
+  virtual void f();
+};
+
+#ifdef SHARED_LIB
+
+void A::f() {}
+
+extern "C" A *create_A() { return new A(); }
+
+#else
+
+constexpr unsigned kShadowGranularity = 12;
+uintptr_t GetShadow();
+
+void write_shadow(void *ptr) {
+  uintptr_t base = GetShadow();
+  uint16_t *s =
+      (uint16_t *)(base + (((uintptr_t)ptr >> kShadowGranularity) << 1));
+  fprintf(stderr, "going to crash\n");
+  // CHECK: going to crash
+  *s = 42;
+  fprintf(stderr, "did not crash\n");
+  // CHECK-NOT: did not crash
+  exit(1);
+}
+
+int main(int argc, char *argv[]) {
+  assert(argc > 1);
+  const bool test_mmap = strcmp(argv[1], "mmap") == 0;
+  const bool test_start = strcmp(argv[1], "start") == 0;
+  const bool test_dlopen = strcmp(argv[1], "dlopen") == 0;
+  const bool test_dlclose = strcmp(argv[1], "dlclose") == 0;
+  const char *lib = argc > 2 ? argv[2] : nullptr;
+
+  if (test_start)
+    write_shadow((void *)&main);
+
+  if (test_mmap) {
+    void *p = mmap(nullptr, 1 << 20, PROT_READ | PROT_WRITE | PROT_EXEC,
+                   MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
+    assert(p != MAP_FAILED);
+    write_shadow((char *)p + 100);
+  } else {
+    void *handle = dlopen(lib, RTLD_NOW);
+    assert(handle);
+    void *create_A = dlsym(handle, "create_A");
+    assert(create_A);
+
+    if (test_dlopen)
+      write_shadow(create_A);
+
+    int res = dlclose(handle);
+    assert(res == 0);
+
+    if (test_dlclose)
+      write_shadow(create_A);
+  }
+}
+#endif