[Asan, Tsan] Generalize means the sanitizers work with memory

[kosarev]

Summary

This diff is here to give an overview of the patch that suggests moving toward adding support for MMU-less target platforms. This diff is not to be committed.

Kostya, thanks for your reply. Re: the "platforms that lack such support": we believe that this proposal affects many custom platforms based on modern IP cores. In some cases such platforms intentionally lack the support for virtual memory, but in all other respects are capable of and can benefit from using sanitizers. In our case we are targeting an ARM-based platform, which is well supported by other parts of LLVM already, so it's not about rare and/or exotic targets.

---

Here's the link to the original RFC message:
https://groups.google.com/d/msg/llvm-dev/Xi5Jl4s_Ofc/oboR4zkgAAAJ

[dvyukov]

lib/tsan/rtl/tsan_platform.h

636

I would suggest to keep the existing name as is.

Why do we need the second MemToPShadow? It is used in 2 places in tsan code which looks somewhat strange. Can't we have a single MemToShadow which maps user address to the address where shadow for that address is stored?

[kosarev]

lib/tsan/rtl/tsan_platform.h

636

Hello Dmitry. Thanks for reviewing. We keep both the P (for physical addresses) and V (for virtual addresses) variants of this function as on some platforms the backward P-to-V translation may be nontrivial/expensive or not available at all.

I had some success with elimination of ShadowToMem() for Asan, but for Tsan it doesn't seem to be that simple. So whenever we want to be able to restore application memory addresses by given shadow addresses, we probably have to maintain both the concepts of virtual and physical shadow addresses.

[kosarev]

This update implements a simple software memory manager that allocates physical shadow pages with a usual heap malloc()/free()-like API. Thanks to eliminated calls to msync() and mmap() that we used to do on every shadow access, with this patch we pass Asan and Tsan tests about 12 times faster comparing to the previous version and just about 3 times slower than we do with the original code.

[dvyukov]

Few suggestions if you want to upstream this:

• remove unchanged files, unnecessary ifdefs and todo's
• start with just asan (tsan is more complex)
• use the existing asan compiler instrumentation that inserts function calls, and do the rest in runtime (at least in the first version, this will greatly simplify the change)

lib/tsan/rtl/tsan_platform.h

636

I am still not following completely.
Both asan and tsan runtime has only 2 notions: applications memory (let's call it A) and shadow memory (let's call it S). Consequently they care only about 2 transitions: A->S and S->A.
Now a platform can do these transitions in an arbitrary complex way, potentially involving a third concept (say, a virtual memory V), or maybe a fourth or whatever. But the point is that is an implementation detail of a particular platform, the rest of the runtimes can still manipulate with only A and S and know nothing about V/P/whatever.
Now the question is: why do you want to disclose these implementation details about P/V to the rest of runtime? Do you want to store V addresses somewhere? Where?
Note that tsan seems to use ShadowToMem only in 1 place -- during race reporting. And you could just remember the application (A) address in thr->racy_shadow_addr in the new mode.

[kosarev]

Will do. Thanks Dmitry.

lib/tsan/rtl/tsan_platform.h

636

Well, we cannot rely exclusively on virtual shadow addresses as sanitizers do not compute the virtual address of every shadow cell they access. Instead, they often translate the base address of a shadow memory block that corresponds to a given application memory address range and call a memory-block function like memset(). This means sanitizers expect adjacent application addresses to be mapped to the same or adjacent shadow memory addresses, which is not a quality that physical shadow addresses hold. So every time an RTL function takes a base shadow address and a size, that base address cannot be a physical one.

Furthermore, there are functions like FastPoisonShadowPartialRightRedzone() and GetStackFrameAccessByAddr() that potentially access lots of shadow cells, but do this in ways that are more complex than what we can do with the usual memory-block functions. Still, in terms of performance it would be expensive to translate the address of every shadow cell these functions access, hence the necessity to introduce the concept of the physical shadow page size.

[dvyukov]

lib/tsan/rtl/tsan_platform.h

636

Still, in terms of performance it would be expensive to translate the address of every shadow cell these functions access, hence the necessity to introduce the concept of the physical shadow page size.

This is the part that I did not understand.
So this is required to preserve performance of the current MMU mode and support NOMMU mode at the same time, right? Initially I read it as it is a performance optimization for the new NOMMU mode. But later I realized that it's hardly the case because VShadowToPShadow is the expensive part.

There is a trick that we use in kernel sanitizers (KMSAN/KTSAN) to solve a similar problem (inability to do continuous pre-allocated shadow). I am not completely sure if it works here, but I think we should try hard to employ it because (1) it will eliminate these pervasive changes throughout the runtimes, (2) it will give you way better performance.
The insight is that when runtime accesses a range of memory (whether it's a 16-byte access, memcpy, stack poisoning or whatever), the range always belongs to a single memory region (either a heap block, a thread stack, a mmap-ed region, etc). If such range crosses boundary of a single memory block then this a bug in user code, so crashing is OK. So we intercept all memory allocations (heap allocator requesting a new superblock, mmap, thread creation, etc) and map a continuous region of shadow for that user memory block. This gives us: (1) shadow memory is always continuous for ranges runtime accesses (so current code will work as is), (2) we don't need lazy init checks in MemToShadow as shadow is always pre-allocated (faster).

What do you think? Will it work in your case? Please give it a try.
Maybe we will need to change something in this approach or adapt it otherwise, but I see great value in making something like this work rather than introducing 2-phase shadow (great burden on all sanitizer developers).

[kosarev]

lib/tsan/rtl/tsan_platform.h

636

So this is required to preserve performance of the current MMU mode and support NOMMU mode at the same time, right? Initially I read it as it is a performance optimization for the new NOMMU mode. But later I realized that it's hardly the case because VShadowToPShadow is the expensive part.

Right, VShadowToPShadow() is expensive, since it's responsible for allocation of physical shadow memory. This is why we want to call it as little as possible. In addition to this, a VShadowToPShadow(vs) call only guarantees that the physical page that corresponds to the vs address is in place. So if we do not know where physical shadow pages begin and end, then we cannot say if accesses to shadow cells at (vs - 1) and (vs + 1) hit the same physical page and so we have to call VShadowToPShadow() for each of them, resulting in an unacceptable slowdown.

So we intercept all memory allocations (heap allocator requesting a new superblock, mmap, thread creation, etc) and map a continuous region of shadow for that user memory block.

Right, we actually do try to get things working this way. As of right now, it's not completely clear if this approach is viable in our situation, but we certainly will give it a try.

However, chances are we will need to proceed with per-access allocations, so I'm looking for ways to support it in mainline without making too much stress to sanitizer developers. One thing I'm thinking of is wrapping virtual shadow addresses that are currently presented as uptr variables with a class that hides the V-to-P translation and physical pages mechanics. It might be somewhat less efficient comparing to the originally proposed patch set, though still worth to try. Will let you know how it goes. Thanks.

lib/Transforms/Instrumentation/AddressSanitizer.cpp

   "__asan_register_image_globals";
 static const char *const kAsanUnregisterImageGlobalsName =
   "__asan_unregister_image_globals";
+static const char *const kAsanMemToVShadowName = "__asan_mem_to_vshadow";
+static const char *const kAsanMemToPShadowName = "__asan_mem_to_pshadow";
 static const char *const kAsanPoisonGlobalsName = "__asan_before_dynamic_init";
 static const char *const kAsanUnpoisonGlobalsName = "__asan_after_dynamic_init";
 static const char *const kAsanInitName = "__asan_init";

 static const char *const kAsanShadowMemoryDynamicAddress =
     "__asan_shadow_memory_dynamic_address";
 
+static const char *const kAsanVShadowToPShadow = "__asan_vshadow_to_pshadow";
+
 static const char *const kAsanAllocaPoison = "__asan_alloca_poison";
 static const char *const kAsanAllocasUnpoison = "__asan_allocas_unpoison";
 

                                  bool IsWrite, size_t AccessSizeIndex,
                                  Value *SizeArgument, uint32_t Exp);
   void instrumentMemIntrinsic(MemIntrinsic *MI);
+#if 0  // TODO(ikosarev)
   Value *memToShadow(Value *Shadow, IRBuilder<> &IRB);
+#endif
+  Value *memToVShadow(Value *Mem, IRBuilder<> &IRB);
+  Value *memToPShadow(Value *Mem, IRBuilder<> &IRB);
   bool runOnFunction(Function &F) override;
   bool maybeInsertAsanInitAtFunctionEntry(Function &F);
   void maybeInsertDynamicShadowAtFunctionEntry(Function &F);

   // This array is indexed by AccessIsWrite and Experiment.
   Function *AsanErrorCallbackSized[2][2];
   Function *AsanMemoryAccessCallbackSized[2][2];
+  Function *AsanMemToVShadow;
+  Function *AsanMemToPShadow;
   Function *AsanMemmove, *AsanMemcpy, *AsanMemset;
   InlineAsm *EmptyAsm;
   Value *LocalDynamicShadow;

   Function *AsanSetShadowFunc[0x100] = {};
   Function *AsanPoisonStackMemoryFunc, *AsanUnpoisonStackMemoryFunc;
   Function *AsanAllocaPoisonFunc, *AsanAllocasUnpoisonFunc;
+  Function *AsanVShadowToPShadowFunc;
 
   // Stores a place and arguments of poisoning/unpoisoning call for alloca.
   struct AllocaPoisonCall {

   /// Finds alloca where the value comes from.
   AllocaInst *findAllocaForValue(Value *V);
 
+  Value *vshadowToPShadow(Value *VShadow, IRBuilder<> &IRB);
+
   // Copies bytes from ShadowBytes into shadow memory for indexes where
   // ShadowMask is not zero. If ShadowMask[i] is zero, we assume that
   // ShadowBytes[i] is constantly zero and doesn't need to be overwritten.
   void copyToShadow(ArrayRef<uint8_t> ShadowMask, ArrayRef<uint8_t> ShadowBytes,
-                    IRBuilder<> &IRB, Value *ShadowBase);
+                    IRBuilder<> &IRB, Value *VShadowBase);
   void copyToShadow(ArrayRef<uint8_t> ShadowMask, ArrayRef<uint8_t> ShadowBytes,
                     size_t Begin, size_t End, IRBuilder<> &IRB,
-                    Value *ShadowBase);
+                    Value *VShadowBase);
   void copyToShadowInline(ArrayRef<uint8_t> ShadowMask,
                           ArrayRef<uint8_t> ShadowBytes, size_t Begin,
-                          size_t End, IRBuilder<> &IRB, Value *ShadowBase);
+                          size_t End, IRBuilder<> &IRB, Value *VShadowBase);
 
   void poisonAlloca(Value *V, uint64_t Size, IRBuilder<> &IRB, bool DoPoison);
 

   return false;
 }
 
+#if 0  // TODO(ikosarev)
 Value *AddressSanitizer::memToShadow(Value *Shadow, IRBuilder<> &IRB) {
   // Shadow >> scale
   Shadow = IRB.CreateLShr(Shadow, Mapping.Scale);

   else
     return IRB.CreateAdd(Shadow, ShadowBase);
 }
+#endif
 
+Value *AddressSanitizer::memToVShadow(Value *Mem, IRBuilder<> &IRB) {
+    return IRB.CreateCall(AsanMemToVShadow, Mem);
+}
+
+Value *AddressSanitizer::memToPShadow(Value *Mem, IRBuilder<> &IRB) {
+    return IRB.CreateCall(AsanMemToPShadow, Mem);
+}
+
 // Instrument memset/memmove/memcpy
 void AddressSanitizer::instrumentMemIntrinsic(MemIntrinsic *MI) {
   IRBuilder<> IRB(MI);

   Type *ShadowTy =
       IntegerType::get(*C, std::max(8U, TypeSize >> Mapping.Scale));
   Type *ShadowPtrTy = PointerType::get(ShadowTy, 0);
-  Value *ShadowPtr = memToShadow(AddrLong, IRB);
+  Value *ShadowPtr = memToPShadow(AddrLong, IRB);
   Value *CmpVal = Constant::getNullValue(ShadowTy);
   Value *ShadowValue =
       IRB.CreateLoad(IRB.CreateIntToPtr(ShadowPtr, ShadowPtrTy));

     }
   }
 
+  // Declare the functions that translate application memory addresses to
+  // virtual and physical shadow addresses.
+  AsanMemToVShadow = checkSanitizerInterfaceFunction(M.getOrInsertFunction(
+      kAsanMemToVShadowName, IntptrTy, IntptrTy, nullptr));
+  AsanMemToPShadow = checkSanitizerInterfaceFunction(M.getOrInsertFunction(
+      kAsanMemToPShadowName, IntptrTy, IntptrTy, nullptr));
+
   const std::string MemIntrinCallbackPrefix =
       CompileKernel ? std::string("") : ClMemoryAccessCallbackPrefix;
   AsanMemmove = checkSanitizerInterfaceFunction(M.getOrInsertFunction(

   AsanAllocasUnpoisonFunc =
       checkSanitizerInterfaceFunction(M.getOrInsertFunction(
           kAsanAllocasUnpoison, IRB.getVoidTy(), IntptrTy, IntptrTy, nullptr));
+
+  AsanVShadowToPShadowFunc = checkSanitizerInterfaceFunction(M.getOrInsertFunction(
+      kAsanVShadowToPShadow, IntptrTy, IntptrTy, nullptr));
 }
 
+Value *FunctionStackPoisoner::vshadowToPShadow(Value *VShadow,
+                                               IRBuilder<> &IRB) {
+    return IRB.CreateCall(AsanVShadowToPShadowFunc, VShadow);
+}
+
 void FunctionStackPoisoner::copyToShadowInline(ArrayRef<uint8_t> ShadowMask,
                                                ArrayRef<uint8_t> ShadowBytes,
                                                size_t Begin, size_t End,
                                                IRBuilder<> &IRB,
-                                               Value *ShadowBase) {
+                                               Value *VShadowBase) {
   if (Begin >= End)
     return;
 

         Val = (Val << 8) | ShadowBytes[i + j];
     }
 
-    Value *Ptr = IRB.CreateAdd(ShadowBase, ConstantInt::get(IntptrTy, i));
+    Value *VPtr = IRB.CreateAdd(VShadowBase, ConstantInt::get(IntptrTy, i));
+    Value *PPtr = vshadowToPShadow(VPtr, IRB);
     Value *Poison = IRB.getIntN(StoreSizeInBytes * 8, Val);
     IRB.CreateAlignedStore(
-        Poison, IRB.CreateIntToPtr(Ptr, Poison->getType()->getPointerTo()), 1);
+        Poison, IRB.CreateIntToPtr(PPtr, Poison->getType()->getPointerTo()), 1);
 
     i += StoreSizeInBytes;
   }

 
 void FunctionStackPoisoner::copyToShadow(ArrayRef<uint8_t> ShadowMask,
                                          ArrayRef<uint8_t> ShadowBytes,
-                                         IRBuilder<> &IRB, Value *ShadowBase) {
-  copyToShadow(ShadowMask, ShadowBytes, 0, ShadowMask.size(), IRB, ShadowBase);
+                                         IRBuilder<> &IRB,
+                                         Value *VShadowBase) {
+  copyToShadow(ShadowMask, ShadowBytes, 0, ShadowMask.size(), IRB,
+               VShadowBase);
 }
 
 void FunctionStackPoisoner::copyToShadow(ArrayRef<uint8_t> ShadowMask,
                                          ArrayRef<uint8_t> ShadowBytes,
                                          size_t Begin, size_t End,
-                                         IRBuilder<> &IRB, Value *ShadowBase) {
+                                         IRBuilder<> &IRB,
+                                         Value *VShadowBase) {
   assert(ShadowMask.size() == ShadowBytes.size());
   size_t Done = Begin;
   for (size_t i = Begin, j = Begin + 1; i < End; i = j++) {

     }
 
     if (j - i >= ClMaxInlinePoisoningSize) {
-      copyToShadowInline(ShadowMask, ShadowBytes, Done, i, IRB, ShadowBase);
+      copyToShadowInline(ShadowMask, ShadowBytes, Done, i, IRB, VShadowBase);
       IRB.CreateCall(AsanSetShadowFunc[Val],
-                     {IRB.CreateAdd(ShadowBase, ConstantInt::get(IntptrTy, i)),
+                     {IRB.CreateAdd(VShadowBase, ConstantInt::get(IntptrTy, i)),
                       ConstantInt::get(IntptrTy, j - i)});
       Done = j;
     }
   }
 
-  copyToShadowInline(ShadowMask, ShadowBytes, Done, End, IRB, ShadowBase);
+  copyToShadowInline(ShadowMask, ShadowBytes, Done, End, IRB, VShadowBase);
 }
 
 // Fake stack allocator (asan_fake_stack.h) has 11 size classes

   const auto &ShadowAfterScope = GetShadowBytesAfterScope(SVD, L);
 
   // Poison the stack red zones at the entry.
-  Value *ShadowBase = ASan.memToShadow(LocalStackBase, IRB);
+  Value *VShadowBase = ASan.memToVShadow(LocalStackBase, IRB);
   // As mask we must use most poisoned case: red zones and after scope.
   // As bytes we can use either the same or just red zones only.
-  copyToShadow(ShadowAfterScope, ShadowAfterScope, IRB, ShadowBase);
+  copyToShadow(ShadowAfterScope, ShadowAfterScope, IRB, VShadowBase);
 
   if (!StaticAllocaPoisonCallVec.empty()) {
     const auto &ShadowInScope = GetShadowBytes(SVD, L);

       IRBuilder<> IRB(APC.InsBefore);
       copyToShadow(ShadowAfterScope,
                    APC.DoPoison ? ShadowAfterScope : ShadowInScope, Begin, End,
-                   IRB, ShadowBase);
+                   IRB, VShadowBase);
     }
   }
 

         ShadowAfterReturn.resize(ClassSize / L.Granularity,
                                  kAsanStackUseAfterReturnMagic);
         copyToShadow(ShadowAfterReturn, ShadowAfterReturn, IRBPoison,
-                     ShadowBase);
+                     VShadowBase);
         Value *SavedFlagPtrPtr = IRBPoison.CreateAdd(
             FakeStack,
             ConstantInt::get(IntptrTy, ClassSize - ASan.LongSize / 8));

       }
 
       IRBuilder<> IRBElse(ElseTerm);
-      copyToShadow(ShadowAfterScope, ShadowClean, IRBElse, ShadowBase);
+      copyToShadow(ShadowAfterScope, ShadowClean, IRBElse, VShadowBase);
     } else {
-      copyToShadow(ShadowAfterScope, ShadowClean, IRBRet, ShadowBase);
+      copyToShadow(ShadowAfterScope, ShadowClean, IRBRet, VShadowBase);
     }
   }
 

lib/asan/asan_allocator.cc

 
     if (!allocated) return allocator.ReturnNullOrDieOnOOM();
 
-    if (*(u8 *)MEM_TO_SHADOW((uptr)allocated) == 0 && CanPoisonMemory()) {
+    if (*MEM_TO_PSHADOW((uptr)allocated) == 0 && CanPoisonMemory()) {
       // Heap poisoning is enabled, but the allocator provides an unpoisoned
       // chunk. This is possible if CanPoisonMemory() was false for some
       // time, for example, due to flags()->start_disabled.

       PoisonShadow(user_beg, size_rounded_down_to_granularity, 0);
     // Deal with the end of the region if size is not aligned to granularity.
     if (size != size_rounded_down_to_granularity && CanPoisonMemory()) {
-      u8 *shadow =
-          (u8 *)MemToShadow(user_beg + size_rounded_down_to_granularity);
-      *shadow = fl.poison_partial ? (size & (SHADOW_GRANULARITY - 1)) : 0;
+      u8 sval = fl.poison_partial ? (size & (SHADOW_GRANULARITY - 1)) : 0;
+      *MemToPShadow(user_beg + size_rounded_down_to_granularity) = sval;
     }
 
     AsanStats &thread_stats = GetCurrentThreadStats();

lib/asan/asan_errors.cc

   ReportErrorSummary(bug_type, &stack);
 }
 
-static bool AdjacentShadowValuesAreFullyPoisoned(u8 *s) {
-  return s[-1] > 127 && s[1] > 127;
+static bool AdjacentShadowValuesAreFullyPoisoned(uptr vs) {
+  // TODO(ikosarev): Use ReadVShadow() to avoid unnecessary allocations.
+  return *VSHADOW_TO_PSHADOW(vs - 1) > 127 &&
+         *VSHADOW_TO_PSHADOW(vs + 1) > 127;
 }
 
 ErrorGeneric::ErrorGeneric(u32 tid, uptr pc_, uptr bp_, uptr sp_, uptr addr,

     // Determine the error type.
     bug_descr = "unknown-crash";
     if (AddrIsInMem(addr)) {
-      u8 *shadow_addr = (u8 *)MemToShadow(addr);
+      uptr shadow_addr = MemToVShadow(addr);
+      // TODO(ikosarev): Use ReadVShadow() here and below to avoid unnecessary
+      //                 allocations.
+      shadow_val = *VSHADOW_TO_PSHADOW(shadow_addr);
       // If we are accessing 16 bytes, look at the second shadow byte.
-      if (*shadow_addr == 0 && access_size > SHADOW_GRANULARITY) shadow_addr++;
+      if (shadow_val == 0 && access_size > SHADOW_GRANULARITY) {
+        shadow_addr++;
+        shadow_val = *VSHADOW_TO_PSHADOW(shadow_addr);
+      }
       // If we are in the partial right redzone, look at the next shadow byte.
-      if (*shadow_addr > 0 && *shadow_addr < 128) shadow_addr++;
+      if (shadow_val > 0 && shadow_val < 128) {
+        shadow_addr++;
+        shadow_val = *VSHADOW_TO_PSHADOW(shadow_addr);
+      }
       bool far_from_bounds = false;
-      shadow_val = *shadow_addr;
       int bug_type_score = 0;
       // For use-after-frees reads are almost as bad as writes.
       int read_after_free_bonus = 0;

 }
 
 static void PrintShadowBytes(InternalScopedString *str, const char *before,
-                             u8 *bytes, u8 *guilty, uptr n) {
+                             uptr bytes, uptr guilty, uptr n) {
   Decorator d;
-  if (before) str->append("%s%p:", before, bytes);
+  if (before) str->append("%s%p:", before, (void*)bytes);
   for (uptr i = 0; i < n; i++) {
-    u8 *p = bytes + i;
+    uptr p = bytes + i;
     const char *before =
         p == guilty ? "[" : (p - 1 == guilty && i != 0) ? "" : " ";
     const char *after = p == guilty ? "]" : "";
-    PrintShadowByte(str, before, *p, after);
+    // TODO(ikosarev): We better use ReadVShadow() that relies on
+    //                 VShadowToPShadowOrNull() so we do not allocate any new
+    //                 pages during this printing.
+    PrintShadowByte(str, before, *VSHADOW_TO_PSHADOW(p), after);
   }
   str->append("\n");
 }

 
 static void PrintShadowMemoryForAddress(uptr addr) {
   if (!AddrIsInMem(addr)) return;
-  uptr shadow_addr = MemToShadow(addr);
+  uptr shadow_addr = MemToVShadow(addr);
   const uptr n_bytes_per_row = 16;
   uptr aligned_shadow = shadow_addr & ~(n_bytes_per_row - 1);
   InternalScopedString str(4096 * 8);

   str.append("Shadow bytes around the buggy address:\n");
   for (int i = -5; i <= 5; i++) {
     const char *prefix = (i == 0) ? "=>" : "  ";
-    PrintShadowBytes(&str, prefix, (u8 *)(aligned_shadow + i * n_bytes_per_row),
-                     (u8 *)shadow_addr, n_bytes_per_row);
+    PrintShadowBytes(&str, prefix, aligned_shadow + i * n_bytes_per_row,
+                     shadow_addr, n_bytes_per_row);
   }
   if (flags()->print_legend) PrintLegend(&str);
   Printf("%s", str.data());

lib/asan/asan_fake_stack.cc

 // For small size classes inline PoisonShadow for better performance.
 ALWAYS_INLINE void SetShadow(uptr ptr, uptr size, uptr class_id, u64 magic) {
   CHECK_EQ(SHADOW_SCALE, 3);  // This code expects SHADOW_SCALE=3.
-  u64 *shadow = reinterpret_cast<u64*>(MemToShadow(ptr));
   if (class_id <= 6) {
+    uptr vshadow = MemToVShadow(ptr);
+    if (SANITIZER_USE_SHADOW_MEMORY_MANAGER) {
+      VShadowMemset(vshadow, static_cast<u8>(magic),
+                    (((uptr)1) << class_id) * 8);
+      return;
+    }
+    u64 *shadow = reinterpret_cast<u64*>(vshadow);
     for (uptr i = 0; i < (((uptr)1) << class_id); i++) {
       shadow[i] = magic;
       // Make sure this does not become memset.

 SANITIZER_INTERFACE_ATTRIBUTE
 void __asan_allocas_unpoison(uptr top, uptr bottom) {
   if ((!top) || (top > bottom)) return;
-  REAL(memset)(reinterpret_cast<void*>(MemToShadow(top)), 0,
-               (bottom - top) / SHADOW_GRANULARITY);
+  VShadowMemset(MemToVShadow(top), 0, (bottom - top) / SHADOW_GRANULARITY);
 }
 } // extern "C"

lib/asan/asan_interface_internal.h

   SANITIZER_INTERFACE_ATTRIBUTE
   void __asan_unregister_globals(__asan_global *globals, uptr n);
 
+  // These functions should be called by the instrumented code to translate
+  // apllication memory addresses to virtual and physical shadow addresses.
+  SANITIZER_INTERFACE_ATTRIBUTE uptr __asan_mem_to_vshadow(uptr mem);
+  SANITIZER_INTERFACE_ATTRIBUTE uptr __asan_mem_to_pshadow(uptr mem);
+  SANITIZER_INTERFACE_ATTRIBUTE uptr __asan_vshadow_to_pshadow(uptr vs);
+
   // These two functions should be called before and after dynamic initializers
   // of a single module run, respectively.
   SANITIZER_INTERFACE_ATTRIBUTE

lib/asan/asan_mapping.h

 #endif
 
 #define SHADOW_GRANULARITY (1ULL << SHADOW_SCALE)
-#define MEM_TO_SHADOW(mem) (((mem) >> SHADOW_SCALE) + (SHADOW_OFFSET))
-#define SHADOW_TO_MEM(shadow) (((shadow) - SHADOW_OFFSET) << SHADOW_SCALE)
+#define MEM_TO_VSHADOW(mem) (((mem) >> SHADOW_SCALE) + SHADOW_OFFSET)
 
+#if 0  // TODO(ikosarev)
+#if !SANITIZER_USE_SHADOW_MEMORY_MANAGER
+// Shadow pages come in the order of increasing virtual addresses:
+// 0, 1, 2, 3, ...
+#define VSHADOW_TO_PSHADOW_OR_NULL(vs) (reinterpret_cast<u8*>((vs)))
+#else
+// Shuffle shadow pages so that in every pair of adjancent pages the odd page
+// comes before the even one:
+// 1, 0, 3, 2, ...
+#define VSHADOW_TO_PSHADOW_OR_NULL(vs) \
+          (reinterpret_cast<u8*>((vs) ^ kShadowPageSize))
+#endif
+#else
+
+namespace __asan {
+namespace shadow_map {
+u8 *VShadowToPShadowOrNull(uptr vs);
+}
+}
+
+#define VSHADOW_TO_PSHADOW_OR_NULL(vs) \
+    (__asan::shadow_map::VShadowToPShadowOrNull((vs)))
+
+#endif
+
+#define VSHADOW_TO_PSHADOW(vs) (VSHADOW_TO_PSHADOW_OR_NULL((vs)))
+#define MEM_TO_PSHADOW_OR_NULL(a) \
+          (VSHADOW_TO_PSHADOW_OR_NULL(MEM_TO_VSHADOW((a))))
+#define MEM_TO_PSHADOW(a) (MEM_TO_PSHADOW_OR_NULL((a)))
+#define VSHADOW_TO_PAGE_OFFSET(vs) ((vs) & kShadowPageMask)
+
 #define kLowMemBeg      0
 #define kLowMemEnd      (SHADOW_OFFSET ? SHADOW_OFFSET - 1 : 0)
 
 #define kLowShadowBeg   SHADOW_OFFSET
-#define kLowShadowEnd   MEM_TO_SHADOW(kLowMemEnd)
+#define kLowShadowEnd   MEM_TO_VSHADOW(kLowMemEnd)
 
-#define kHighMemBeg     (MEM_TO_SHADOW(kHighMemEnd) + 1)
+#define kHighMemBeg     (MEM_TO_VSHADOW(kHighMemEnd) + 1)
 
-#define kHighShadowBeg  MEM_TO_SHADOW(kHighMemBeg)
-#define kHighShadowEnd  MEM_TO_SHADOW(kHighMemEnd)
+#define kHighShadowBeg  MEM_TO_VSHADOW(kHighMemBeg)
+#define kHighShadowEnd  MEM_TO_VSHADOW(kHighMemEnd)
 
-# define kMidShadowBeg MEM_TO_SHADOW(kMidMemBeg)
-# define kMidShadowEnd MEM_TO_SHADOW(kMidMemEnd)
+# define kMidShadowBeg MEM_TO_VSHADOW(kMidMemBeg)
+# define kMidShadowEnd MEM_TO_VSHADOW(kMidMemEnd)
 
 // With the zero shadow base we can not actually map pages starting from 0.
 // This constant is somewhat arbitrary.

 
 namespace __asan {
 
+const unsigned kShadowPageScale = 16;  // 64K bytes
+const uptr kShadowPageSize = ((uptr)1) << kShadowPageScale;
+const uptr kShadowPageMask = kShadowPageSize - 1;
+
 extern uptr AsanMappingProfile[];
 
 #if ASAN_FIXED_MAPPING

       (flags()->protect_shadow_gap == 0 && AddrIsInShadowGap(a));
 }
 
-static inline uptr MemToShadow(uptr p) {
+static inline uptr MemToVShadow(uptr p) {
   PROFILE_ASAN_MAPPING();
   CHECK(AddrIsInMem(p));
-  return MEM_TO_SHADOW(p);
+  return MEM_TO_VSHADOW(p);
 }
 
 static inline bool AddrIsInHighShadow(uptr a) {

   return (a & (SHADOW_GRANULARITY - 1)) == 0;
 }
 
+static inline u8 *VShadowToPShadowOrNull(uptr vs) {
+  PROFILE_ASAN_MAPPING();
+  CHECK(AddrIsInShadow(vs));
+  return VSHADOW_TO_PSHADOW_OR_NULL(vs);
+}
+
+static inline u8 *VShadowToPShadow(uptr vs) {
+  PROFILE_ASAN_MAPPING();
+  CHECK(AddrIsInShadow(vs));
+  return VSHADOW_TO_PSHADOW(vs);
+}
+
+// TODO(ikosarev): For trivial implementation this function can be declared as
+//                 one that never returns null.
+static inline u8 *MemToPShadowOrNull(uptr a) {
+  return VShadowToPShadowOrNull(MemToVShadow(a));
+}
+
+static inline u8 *MemToPShadow(uptr a) {
+  return VShadowToPShadow(MemToVShadow(a));
+}
+
+// TODO(ikosarev): May/should cause releasing of previously allocated pages.
+static inline void VShadowMemset(uptr vs, u8 value, uptr size) {
+#if !SANITIZER_USE_SHADOW_MEMORY_MANAGER
+  u8 *ps = VSHADOW_TO_PSHADOW(vs);
+  REAL(memset)(ps, value, size);
+#else
+  // TODO(ikosarev): Optimize.
+  for (uptr i = vs, e = vs + size; i != e; ++i) {
+    *VSHADOW_TO_PSHADOW(i) = value;
+    // Make sure this does not become memset.
+    SanitizerBreakOptimization(nullptr);
+  }
+#endif
+}
+
+static inline bool VShadowIsZero(uptr vs, uptr size) {
+#if !SANITIZER_USE_SHADOW_MEMORY_MANAGER
+  u8 *ps = VSHADOW_TO_PSHADOW(vs);
+  return __sanitizer::mem_is_zero(reinterpret_cast<char*>(ps), size);
+#else
+  // TODO(ikosarev): Optimize.
+  for (uptr i = vs, e = vs + size; i != e; ++i) {
+    if (*VSHADOW_TO_PSHADOW(i))
+      return false;
+    // Make sure this does not become memset.
+    SanitizerBreakOptimization(nullptr);
+  }
+  return true;
+#endif
+}
+
+static inline void ReleaseVShadow(uptr vs, uptr size) {
+#if !SANITIZER_USE_SHADOW_MEMORY_MANAGER
+  // Since asan's mapping is compacting, the shadow chunk may be
+  // not page-aligned, so we only flush the page-aligned portion.
+  uptr page_size = GetPageSizeCached();
+  uptr shadow_beg = RoundUpTo(vs, page_size);
+  uptr shadow_end = RoundDownTo(vs + size, page_size);
+  // TODO(ikosarev): if (shadow_end > shadow_beg) ?
+  ReleaseMemoryToOS(shadow_beg, shadow_end - shadow_beg);
+#else
+  // TODO(ikosarev): Support releasing of shadow pages.
+#endif
+}
+
 static inline bool AddressIsPoisoned(uptr a) {
   PROFILE_ASAN_MAPPING();
   const uptr kAccessSize = 1;
-  u8 *shadow_address = (u8*)MEM_TO_SHADOW(a);
+  u8 *shadow_address = MEM_TO_PSHADOW_OR_NULL(a);
+  if (!shadow_address)
+    return true;
   s8 shadow_value = *shadow_address;
   if (shadow_value) {
     u8 last_accessed_byte = (a & (SHADOW_GRANULARITY - 1))

lib/asan/asan_poisoning.h

 ALWAYS_INLINE void FastPoisonShadow(uptr aligned_beg, uptr aligned_size,
                                     u8 value) {
   DCHECK(CanPoisonMemory());
-  uptr shadow_beg = MEM_TO_SHADOW(aligned_beg);
-  uptr shadow_end = MEM_TO_SHADOW(
+  uptr shadow_beg = MEM_TO_VSHADOW(aligned_beg);
+  uptr shadow_end = MEM_TO_VSHADOW(
       aligned_beg + aligned_size - SHADOW_GRANULARITY) + 1;
+  if (SANITIZER_USE_SHADOW_MEMORY_MANAGER) {
+    VShadowMemset(shadow_beg, value, shadow_end - shadow_beg);
+    return;
+  }
   // FIXME: Page states are different on Windows, so using the same interface
   // for mapping shadow and zeroing out pages doesn't "just work", so we should
   // probably provide higher-level interface for these operations.

     uptr aligned_addr, uptr size, uptr redzone_size, u8 value) {
   DCHECK(CanPoisonMemory());
   bool poison_partial = flags()->poison_partial;
-  u8 *shadow = (u8*)MEM_TO_SHADOW(aligned_addr);
-  for (uptr i = 0; i < redzone_size; i += SHADOW_GRANULARITY, shadow++) {
+
+  uptr vshadow = MEM_TO_VSHADOW(aligned_addr);
+  uptr off = VSHADOW_TO_PAGE_OFFSET(vshadow);
+  uptr vshadow_page = vshadow - off;
+  u8 *page = VSHADOW_TO_PSHADOW(vshadow_page);
+
+  for (uptr i = 0; i < redzone_size; i += SHADOW_GRANULARITY, off++) {
+    if (SANITIZER_USE_SHADOW_MEMORY_MANAGER && off == kShadowPageSize) {
+        vshadow_page += kShadowPageSize;
+        page = VSHADOW_TO_PSHADOW(vshadow_page);
+        off = 0;
+    }
+
+    CHECK_LT(off, kShadowPageSize);
+
     if (i + SHADOW_GRANULARITY <= size) {
-      *shadow = 0;  // fully addressable
+      page[off] = 0;  // fully addressable
     } else if (i >= size) {
-      *shadow = (SHADOW_GRANULARITY == 128) ? 0xff : value;  // unaddressable
+      page[off] = (SHADOW_GRANULARITY == 128) ? 0xff : value;  // unaddressable
     } else {
       // first size-i bytes are addressable
-      *shadow = poison_partial ? static_cast<u8>(size - i) : 0;
+      page[off] = poison_partial ? static_cast<u8>(size - i) : 0;
     }
   }
 }
 
 // Calls __sanitizer::ReleaseMemoryToOS() on
-// [MemToShadow(p), MemToShadow(p+size)] with proper rounding.
+// [MemToVShadow(p), MemToVShadow(p+size)] with proper rounding.
 void FlushUnneededASanShadowMemory(uptr p, uptr size);
 
 }  // namespace __asan

lib/asan/asan_poisoning.cc

 }
 
 struct ShadowSegmentEndpoint {
-  u8 *chunk;
+  uptr x_chunk;  // TODO(ikosarev): Rename.
   s8 offset;  // in [0, SHADOW_GRANULARITY)
   s8 value;  // = *chunk;
 
   explicit ShadowSegmentEndpoint(uptr address) {
-    chunk = (u8*)MemToShadow(address);
+    x_chunk = MemToVShadow(address);
     offset = address & (SHADOW_GRANULARITY - 1);
-    value = *chunk;
+    value = *VSHADOW_TO_PSHADOW(x_chunk);
   }
 };
 
 void FlushUnneededASanShadowMemory(uptr p, uptr size) {
-    // Since asan's mapping is compacting, the shadow chunk may be
-    // not page-aligned, so we only flush the page-aligned portion.
-    uptr page_size = GetPageSizeCached();
-    uptr shadow_beg = RoundUpTo(MemToShadow(p), page_size);
-    uptr shadow_end = RoundDownTo(MemToShadow(p + size), page_size);
-    ReleaseMemoryToOS(shadow_beg, shadow_end - shadow_beg);
+  uptr shadow_beg = MemToVShadow(p);
+  uptr shadow_end = MemToVShadow(p + size);
+  ReleaseVShadow(shadow_beg, shadow_end - shadow_beg);
 }
 
 void AsanPoisonOrUnpoisonIntraObjectRedzone(uptr ptr, uptr size, bool poison) {

   CHECK_LE(size, 4096);
   CHECK(IsAligned(end, SHADOW_GRANULARITY));
   if (!IsAligned(ptr, SHADOW_GRANULARITY)) {
-    *(u8 *)MemToShadow(ptr) =
+    *MemToPShadow(ptr) =
         poison ? static_cast<u8>(ptr % SHADOW_GRANULARITY) : 0;
     ptr |= SHADOW_GRANULARITY - 1;
     ptr++;
   }
+  // TODO(ikosarev): Can we do this more efficiently?
   for (; ptr < end; ptr += SHADOW_GRANULARITY)
-    *(u8*)MemToShadow(ptr) = poison ? kAsanIntraObjectRedzone : 0;
+    *MemToPShadow(ptr) = poison ? kAsanIntraObjectRedzone : 0;
 }
 
 }  // namespace __asan

           (void *)end_addr);
   ShadowSegmentEndpoint beg(beg_addr);
   ShadowSegmentEndpoint end(end_addr);
-  if (beg.chunk == end.chunk) {
+  if (beg.x_chunk == end.x_chunk) {
     CHECK_LT(beg.offset, end.offset);
     s8 value = beg.value;
     CHECK_EQ(value, end.value);

     // We can only poison memory if the byte in end.offset is unaddressable.
     // No need to re-poison memory if it is poisoned already.
     if (value > 0 && value <= end.offset) {
+      u8 *sptr = VSHADOW_TO_PSHADOW(beg.x_chunk);
       if (beg.offset > 0) {
-        *beg.chunk = Min(value, beg.offset);
+        *sptr = Min(value, beg.offset);
       } else {
-        *beg.chunk = kAsanUserPoisonedMemoryMagic;
+        *sptr = kAsanUserPoisonedMemoryMagic;
       }
     }
     return;
   }
-  CHECK_LT(beg.chunk, end.chunk);
+  CHECK_LT(beg.x_chunk, end.x_chunk);
   if (beg.offset > 0) {
     // Mark bytes from beg.offset as unaddressable.
+    u8 *sptr = VSHADOW_TO_PSHADOW(beg.x_chunk);
     if (beg.value == 0) {
-      *beg.chunk = beg.offset;
+      *sptr = beg.offset;
     } else {
-      *beg.chunk = Min(beg.value, beg.offset);
+      *sptr = Min(beg.value, beg.offset);
     }
-    beg.chunk++;
+    beg.x_chunk++;
   }
-  REAL(memset)(beg.chunk, kAsanUserPoisonedMemoryMagic, end.chunk - beg.chunk);
+  VShadowMemset(beg.x_chunk, kAsanUserPoisonedMemoryMagic,
+                end.x_chunk - beg.x_chunk);
   // Poison if byte in end.offset is unaddressable.
   if (end.value > 0 && end.value <= end.offset) {
-    *end.chunk = kAsanUserPoisonedMemoryMagic;
+    *VSHADOW_TO_PSHADOW(end.x_chunk) = kAsanUserPoisonedMemoryMagic;
   }
 }
 

           (void *)end_addr);
   ShadowSegmentEndpoint beg(beg_addr);
   ShadowSegmentEndpoint end(end_addr);
-  if (beg.chunk == end.chunk) {
+  if (beg.x_chunk == end.x_chunk) {
     CHECK_LT(beg.offset, end.offset);
     s8 value = beg.value;
     CHECK_EQ(value, end.value);

     // We unpoison memory bytes up to enbytes up to end.offset if it is not
     // unpoisoned already.
     if (value != 0) {
-      *beg.chunk = Max(value, end.offset);
+      *VSHADOW_TO_PSHADOW(beg.x_chunk) = Max(value, end.offset);
     }
     return;
   }
-  CHECK_LT(beg.chunk, end.chunk);
+  CHECK_LT(beg.x_chunk, end.x_chunk);
   if (beg.offset > 0) {
-    *beg.chunk = 0;
-    beg.chunk++;
+    *VSHADOW_TO_PSHADOW(beg.x_chunk) = 0;
+    beg.x_chunk++;
   }
-  REAL(memset)(beg.chunk, 0, end.chunk - beg.chunk);
+  VShadowMemset(beg.x_chunk, 0, end.x_chunk - beg.x_chunk);
   if (end.offset > 0 && end.value != 0) {
-    *end.chunk = Max(end.value, end.offset);
+    *VSHADOW_TO_PSHADOW(end.x_chunk) = Max(end.value, end.offset);
   }
 }
 

   CHECK_LT(beg, end);
   uptr aligned_b = RoundUpTo(beg, SHADOW_GRANULARITY);
   uptr aligned_e = RoundDownTo(end, SHADOW_GRANULARITY);
-  uptr shadow_beg = MemToShadow(aligned_b);
-  uptr shadow_end = MemToShadow(aligned_e);
+  uptr shadow_beg = MemToVShadow(aligned_b);
+  uptr shadow_end = MemToVShadow(aligned_e);
   // First check the first and the last application bytes,
   // then check the SHADOW_GRANULARITY-aligned region by calling
   // mem_is_zero on the corresponding shadow.

   if (!__asan::AddressIsPoisoned(beg) &&
       !__asan::AddressIsPoisoned(end - 1) &&
       (shadow_end <= shadow_beg ||
-       __sanitizer::mem_is_zero((const char *)shadow_beg,
-                                shadow_end - shadow_beg)))
+       VShadowIsZero(shadow_beg, shadow_end - shadow_beg)))
     return 0;
   // The fast check failed, so we have a poisoned byte somewhere.
   // Find it slowly.

 void __asan_poison_cxx_array_cookie(uptr p) {
   if (SANITIZER_WORDSIZE != 64) return;
   if (!flags()->poison_array_cookie) return;
-  uptr s = MEM_TO_SHADOW(p);
-  *reinterpret_cast<u8*>(s) = kAsanArrayCookieMagic;
+  *MEM_TO_PSHADOW(p) = kAsanArrayCookieMagic;
 }
 
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE

 uptr __asan_load_cxx_array_cookie(uptr *p) {
   if (SANITIZER_WORDSIZE != 64) return *p;
   if (!flags()->poison_array_cookie) return *p;
-  uptr s = MEM_TO_SHADOW(reinterpret_cast<uptr>(p));
-  u8 sval = *reinterpret_cast<u8*>(s);
+  u8 sval = *MEM_TO_PSHADOW(reinterpret_cast<uptr>(p));
   if (sval == kAsanArrayCookieMagic) return *p;
   // If sval is not kAsanArrayCookieMagic it can only be freed memory,
   // which means that we are going to get double-free. So, return 0 to avoid

   if (size == aligned_size)
     return;
   s8 end_offset = (s8)(size - aligned_size);
-  s8* shadow_end = (s8*)MemToShadow(addr + aligned_size);
+  s8* shadow_end = (s8*)MemToPShadow(addr + aligned_size);
   s8 end_value = *shadow_end;
   if (do_poison) {
     // If possible, mark all the bytes mapping to last shadow byte as

 }
 
 void __asan_set_shadow_00(uptr addr, uptr size) {
-  REAL(memset)((void *)addr, 0, size);
+  // TODO(ikosarev):
+  // REAL(memset)((void *)addr, 0, size);
+  VShadowMemset(addr, 0, size);
 }
 
 void __asan_set_shadow_f1(uptr addr, uptr size) {
-  REAL(memset)((void *)addr, 0xf1, size);
+  // TODO(ikosarev):
+  // REAL(memset)((void *)addr, 0xf1, size);
+  VShadowMemset(addr, 0xf1, size);
 }
 
 void __asan_set_shadow_f2(uptr addr, uptr size) {
-  REAL(memset)((void *)addr, 0xf2, size);
+  // TODO(ikosarev):
+  // REAL(memset)((void *)addr, 0xf2, size);
+  VShadowMemset(addr, 0xf2, size);
 }
 
 void __asan_set_shadow_f3(uptr addr, uptr size) {
-  REAL(memset)((void *)addr, 0xf3, size);
+  // TODO(ikosarev):
+  // REAL(memset)((void *)addr, 0xf3, size);
+  VShadowMemset(addr, 0xf3, size);
 }
 
 void __asan_set_shadow_f5(uptr addr, uptr size) {
-  REAL(memset)((void *)addr, 0xf5, size);
+  // TODO(ikosarev):
+  // REAL(memset)((void *)addr, 0xf5, size);
+  VShadowMemset(addr, 0xf5, size);
 }
 
 void __asan_set_shadow_f8(uptr addr, uptr size) {
-  REAL(memset)((void *)addr, 0xf8, size);
+  // TODO(ikosarev):
+  // REAL(memset)((void *)addr, 0xf8, size);
+  VShadowMemset(addr, 0xf8, size);
 }
 
 void __asan_poison_stack_memory(uptr addr, uptr size) {

   // FIXME: Two of these three checks are disabled until we fix
   // https://github.com/google/sanitizers/issues/258.
   // if (d1 != d2)
-  //  CHECK_EQ(*(u8*)MemToShadow(d1), old_mid - d1);
+  //  CHECK_EQ(*MemToPShadow(d1), old_mid - d1);
   if (a + granularity <= d1)
-    CHECK_EQ(*(u8*)MemToShadow(a), 0);
+    CHECK_EQ(*MemToPShadow(a), 0);
   // if (d2 + granularity <= c && c <= end)
-  //   CHECK_EQ(*(u8 *)MemToShadow(c - granularity),
+  //   CHECK_EQ(*MemToPShadow(c - granularity),
   //            kAsanContiguousContainerOOBMagic);
 
   uptr b1 = RoundDownTo(new_mid, granularity);

   PoisonShadow(b2, c - b2, kAsanContiguousContainerOOBMagic);
   if (b1 != b2) {
     CHECK_EQ(b2 - b1, granularity);
-    *(u8*)MemToShadow(b1) = static_cast<u8>(new_mid - b1);
+    *MemToPShadow(b1) = static_cast<u8>(new_mid - b1);
   }
 }
 

lib/asan/asan_rtl.cc

 #include "sanitizer_common/sanitizer_atomic.h"
 #include "sanitizer_common/sanitizer_flags.h"
 #include "sanitizer_common/sanitizer_libc.h"
+#include "sanitizer_common/sanitizer_posix.h"
 #include "sanitizer_common/sanitizer_symbolizer.h"
 #include "lsan/lsan_common.h"
 #include "ubsan/ubsan_init.h"

 uptr kHighMemEnd, kMidMemBeg, kMidMemEnd;
 #endif
 
+// -------------------------- Shadow Memory Manager --------------------- {{{1
+
+#if SANITIZER_USE_SHADOW_MEMORY_MANAGER
+namespace shadow_map {
+
+#if 1  // TODO(ikosarev)
+const unsigned kLinkSizeScale = kShadowPageScale;
+const uptr kLinkSize = ((uptr)1) << kLinkSizeScale;
+const uptr kLinkSizeMask = kLinkSize - 1;
+
+// The first free physical shadow address.
+uptr PShadowEnd;
+
+typedef u8 Level0[kLinkSize];       // A 64Kb page.
+typedef Level0 *Level1[kLinkSize];  // 4G bytes.
+typedef Level1 *Level2[kLinkSize];  // 2^48 bytes.
+typedef Level2 *Level3[kLinkSize];  // 2^64 bytes.
+Level3 *PShadowMapRoot;
+
+void Init() {
+  PShadowEnd = kHighShadowBeg;
+  PShadowMapRoot = nullptr;
+}
+
+void *pshadow_alloc(uptr size) {
+  uptr sys_page_size = GetPageSizeCached();
+  size = RoundUpTo(size, sys_page_size);
+
+  CHECK_EQ(PShadowEnd % sys_page_size, 0);
+  uptr beg = PShadowEnd;
+  PShadowEnd += size;
+
+  void *ptr = MmapFixedNoReserve(beg, size, /* name= */ nullptr);
+  if (ptr != reinterpret_cast<void*>(beg)) {
+    Report("Cannot allocate physical shadow memory.\n");
+    Abort();
+  }
+
+  return ptr;
+}
+
+template<typename T>
+T *pshadow_alloc() {
+  return static_cast<T*>(pshadow_alloc(sizeof(T)));
+}
+
+u8 *VShadowToPShadowOrNull(uptr vs) {
+  if (flags()->protect_shadow_gap)
+    CHECK(!AddrIsInShadowGap(vs));
+
+  Level3 *&p3 = PShadowMapRoot;
+  if (!p3)
+    p3 = pshadow_alloc<Level3>();
+
+  uptr i3 = (vs >> (kLinkSizeScale * 3)) & kLinkSizeMask;
+  Level2 *&p2 = (*p3)[i3];
+  if (!p2)
+    p2 = pshadow_alloc<Level2>();
+
+  uptr i2 = (vs >> (kLinkSizeScale * 2)) & kLinkSizeMask;
+  Level1 *&p1 = (*p2)[i2];
+  if (!p1)
+    p1 = pshadow_alloc<Level1>();
+
+  uptr i1 = (vs >> (kLinkSizeScale * 1)) & kLinkSizeMask;
+  Level0 *&p0 = (*p1)[i1];
+  if (!p0)
+    p0 = pshadow_alloc<Level0>();
+
+  uptr i0 = (vs >> (kLinkSizeScale * 0)) & kLinkSizeMask;
+  u8 *p = &(*p0)[i0];
+
+  return p;
+}
+#else
+
+void Init()
+{}
+
+u8 *VShadowToPShadowOrNull(uptr vs) {
+#if 1
+  // TODO(ikosarev): Refine.
+  uptr sys_page_size = GetPageSizeCached();
+  CHECK_EQ(kShadowPageSize % sys_page_size, 0);
+  uptr page_size = kShadowPageSize;
+
+  uptr beg = RoundDownTo(vs, page_size);
+  uptr end = beg + page_size - 1;
+
+  int res = internal_msync((void*)beg, /* len= */ page_size, /* flags= */ 0);
+  int reserrno;
+  if (internal_iserror(res, &reserrno) && reserrno == 12 /* ENOMEM */) {
+    // Printf("VShadowToPShadowOrNull(%zx): Reserve %zx - %zx.\n",
+    //        vs, beg, end);
+    ReserveShadowMemoryRange(beg, end, /* name= */ nullptr);
+  }
+#endif
+  return reinterpret_cast<u8*>(vs);
+}
+
+#endif
+
+}  // namespace shadow_map
+#endif  // SANITIZER_USE_SHADOW_MEMORY_MANAGER
+
 // -------------------------- Misc ---------------- {{{1
 void ShowStatsAndAbort() {
   __asan_print_accumulated_stats();

 // ---------------------- mmap -------------------- {{{1
 // Reserve memory range [beg, end].
 // We need to use inclusive range because end+1 may not be representable.
+// TODO(ikosarev): Do not reserve these ranges if we allocate shadow pages
+//                 dynamically.
 void ReserveShadowMemoryRange(uptr beg, uptr end, const char *name) {
   CHECK_EQ((beg % GetMmapGranularity()), 0);
   CHECK_EQ(((end + 1) % GetMmapGranularity()), 0);

 ASAN_REPORT_ERROR_N(store, true)
 
 #define ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, exp_arg, fatal) \
-    uptr sp = MEM_TO_SHADOW(addr);                                             \
-    uptr s = size <= SHADOW_GRANULARITY ? *reinterpret_cast<u8 *>(sp)          \
-                                        : *reinterpret_cast<u16 *>(sp);        \
+    u8 *sp = MEM_TO_PSHADOW(addr);                                             \
+    uptr s = size <= SHADOW_GRANULARITY ? *sp : *reinterpret_cast<u16 *>(sp);  \
     if (UNLIKELY(s)) {                                                         \
       if (UNLIKELY(size >= SHADOW_GRANULARITY ||                               \
                    ((s8)((addr & (SHADOW_GRANULARITY - 1)) + size - 1)) >=     \

   }
 }
 
+extern "C"
+SANITIZER_INTERFACE_ATTRIBUTE
+uptr __asan_mem_to_vshadow(uptr mem) {
+    return MemToVShadow(mem);
+}
+
+extern "C"
+SANITIZER_INTERFACE_ATTRIBUTE
+uptr __asan_mem_to_pshadow(uptr mem) {
+    // TODO(ikosarev): return reinterpret_cast<uptr>(MemToPShadow(mem));
+    return reinterpret_cast<uptr>(MEM_TO_PSHADOW(mem));
+}
+
+extern "C"
+SANITIZER_INTERFACE_ATTRIBUTE
+uptr __asan_vshadow_to_pshadow(uptr vs) {
+    return reinterpret_cast<uptr>(VShadowToPShadow(vs));
+}
+
 // Force the linker to keep the symbols for various ASan interface functions.
 // We want to keep those in the executable in order to let the instrumented
 // dynamic libraries access the symbol even if it is not used by the executable

   CHECK_EQ((kHighMemBeg % GetMmapGranularity()), 0);
 }
 
-static void ProtectGap(uptr addr, uptr size) {
+/* TODO(ikosarev): static */ void ProtectGap(uptr addr, uptr size) {
   if (!flags()->protect_shadow_gap) {
     // The shadow gap is unprotected, so there is a chance that someone
     // is actually using this memory. Which means it needs a shadow...
-    uptr GapShadowBeg = RoundDownTo(MEM_TO_SHADOW(addr), GetPageSizeCached());
+    uptr GapShadowBeg = RoundDownTo(MEM_TO_VSHADOW(addr), GetPageSizeCached());
     uptr GapShadowEnd =
-        RoundUpTo(MEM_TO_SHADOW(addr + size), GetPageSizeCached()) - 1;
+        RoundUpTo(MEM_TO_VSHADOW(addr + size), GetPageSizeCached()) - 1;
     if (Verbosity())
       Printf("protect_shadow_gap=0:"
              " not protecting shadow gap, allocating gap's shadow\n"

     Printf("|| `[%p, %p]` || LowMem     ||\n",
            (void*)kLowMemBeg, (void*)kLowMemEnd);
   }
-  Printf("MemToShadow(shadow): %p %p %p %p",
-         (void*)MEM_TO_SHADOW(kLowShadowBeg),
-         (void*)MEM_TO_SHADOW(kLowShadowEnd),
-         (void*)MEM_TO_SHADOW(kHighShadowBeg),
-         (void*)MEM_TO_SHADOW(kHighShadowEnd));
+  Printf("MemToVShadow(shadow): %p %p %p %p",
+         (void*)MEM_TO_VSHADOW(kLowShadowBeg),
+         (void*)MEM_TO_VSHADOW(kLowShadowEnd),
+         (void*)MEM_TO_VSHADOW(kHighShadowBeg),
+         (void*)MEM_TO_VSHADOW(kHighShadowEnd));
   if (kMidMemBeg) {
     Printf(" %p %p",
-           (void*)MEM_TO_SHADOW(kMidShadowBeg),
-           (void*)MEM_TO_SHADOW(kMidShadowEnd));
+           (void*)MEM_TO_VSHADOW(kMidShadowBeg),
+           (void*)MEM_TO_VSHADOW(kMidShadowEnd));
   }
   Printf("\n");
   Printf("redzone=%zu\n", (uptr)flags()->redzone);

   // initialization steps look at flags().
   InitializeFlags();
 
+#if SANITIZER_USE_SHADOW_MEMORY_MANAGER
+  shadow_map::Init();
+#endif
+
   AsanCheckIncompatibleRT();
   AsanCheckDynamicRTPrereqs();
   AvoidCVE_2016_2143();

 
   DisableCoreDumperIfNecessary();
 
+#if 0  // TODO(ikosarev)
   if (full_shadow_is_available) {
     // mmap the low shadow plus at least one page at the left.
     if (kLowShadowBeg)

     DumpProcessMap();
     Die();
   }
+#endif
 
   AsanTSDInit(PlatformTSDDtor);
   InstallDeadlySignalHandlers(AsanOnDeadlySignal);

lib/asan/asan_thread.cc

     return true;
   }
   uptr aligned_addr = addr & ~(SANITIZER_WORDSIZE/8 - 1);  // align addr.
-  u8 *shadow_ptr = (u8*)MemToShadow(aligned_addr);
-  u8 *shadow_bottom = (u8*)MemToShadow(bottom);
+  uptr mem_ptr = RoundDownTo(aligned_addr, SHADOW_GRANULARITY);
+  uptr shadow_ptr = MemToVShadow(aligned_addr);
+  uptr shadow_bottom = MemToVShadow(bottom);
 
+  // TODO(ikosarev): Performance?
   while (shadow_ptr >= shadow_bottom &&
-         *shadow_ptr != kAsanStackLeftRedzoneMagic) {
+         *VSHADOW_TO_PSHADOW(shadow_ptr) != kAsanStackLeftRedzoneMagic) {
     shadow_ptr--;
+    mem_ptr -= SHADOW_GRANULARITY;
   }
 
+  // TODO(ikosarev): Performance?
   while (shadow_ptr >= shadow_bottom &&
-         *shadow_ptr == kAsanStackLeftRedzoneMagic) {
+         *VSHADOW_TO_PSHADOW(shadow_ptr) == kAsanStackLeftRedzoneMagic) {
     shadow_ptr--;
+    mem_ptr -= SHADOW_GRANULARITY;
   }
 
   if (shadow_ptr < shadow_bottom) {

     return false;
   }
 
-  uptr* ptr = (uptr*)SHADOW_TO_MEM((uptr)(shadow_ptr + 1));
+  uptr* ptr = (uptr*)(mem_ptr + SHADOW_GRANULARITY);
   CHECK(ptr[0] == kCurrentStackFrameMagic);
   access->offset = addr - (uptr)ptr;
   access->frame_pc = ptr[2];

lib/asan/tests/asan_asm_test.cc

 
 #if defined(__linux__)
 
+#if 0  // TODO(ikosarev): Won't work with the software memory manager.
+
 // Assembly instrumentation is broken on x86 Android (x86 + PIC + shared runtime
 // library). See https://github.com/google/sanitizers/issues/353
 #if defined(__x86_64__) ||                                                     \

 
 } // End of anonymous namespace
 
+#if 0  // TODO(ikosarev): Won't work with the software memory manager.
 TEST(AddressSanitizer, asm_load_store) {
   U4* buf = new U4[2];
   EXPECT_DEATH(AsmLoad(&buf[3]), "READ of size 4");

   EXPECT_DEATH(AsmStore(0x1234, &buf[3]), "WRITE of size 4");
   delete [] buf;
 }
+#endif
 
+#if 0  // TODO(ikosarev): Won't work with the software memory manager.
 TEST(AddressSanitizer, asm_rw) {
   TestAsmWrite<U1>("WRITE of size 1");
   TestAsmWrite<U2>("WRITE of size 2");

 #endif // defined(__x86_64__)
   TestAsmRead<__m128i>("READ of size 16");
 }
+#endif
 
+#if 0  // TODO(ikosarev): Won't work with the software memory manager.
 TEST(AddressSanitizer, asm_flags) {
   long magic = 0x1234;
   long r = 0x0;

 
   ASSERT_EQ(0x1, r);
 }
+#endif
 
+#if 0  // TODO(ikosarev): Won't work with the software memory manager.
 TEST(AddressSanitizer, asm_rep_movs) {
   TestAsmRepMovs<U1>("READ of size 1", "WRITE of size 1");
   TestAsmRepMovs<U2>("READ of size 2", "WRITE of size 2");

   TestAsmRepMovs<U8>("READ of size 8", "WRITE of size 8");
 #endif  // defined(__x86_64__)
 }
+#endif
 
 #endif // defined(__x86_64__) || (defined(__i386__) && defined(__SSE2__))
 
+#endif
+
 #endif // defined(__linux__)

lib/asan/tests/asan_internal_interface_test.cc

 #include "asan_interface_internal.h"
 #include "asan_test_utils.h"
 
+#if 0  // TODO(ikosarev): This test expects that physical shadow memory
+       //                 addresses are the same as corresponding virtual
+       //                 ones, so it won't work with the software memory
+       //                 manager enabled.
 TEST(AddressSanitizerInternalInterface, SetShadow) {
   std::vector<char> buffer(17, 0xff);
 

   __asan_set_shadow_f8((uptr)buffer.data(), buffer.size());
   EXPECT_EQ(std::vector<char>(buffer.size(), 0xf8), buffer);
 }
+#endif

lib/asan/tests/asan_noinst_test.cc

   fprintf(stderr, "%s shadow: %lx size % 3ld: ", tag, (long)ptr, (long)size);
   uptr prev_shadow = 0;
   for (sptr i = -32; i < (sptr)size + 32; i++) {
-    uptr shadow = __asan::MemToShadow(ptr + i);
+    uptr shadow = __asan::MemToVShadow(ptr + i);
     if (i == 0 || i == (sptr)size)
       fprintf(stderr, ".");
     if (shadow != prev_shadow) {
       prev_shadow = shadow;
-      fprintf(stderr, "%02x", (int)*(u8*)shadow);
+      fprintf(stderr, "%02x", (unsigned)*__asan::VShadowToPShadow(shadow));
     }
   }
   fprintf(stderr, "\n");

lib/asan/tests/asan_test.cc

   PTHREAD_JOIN(t, 0);
 }
 
+#if 0  // TODO(ikosarev): Accesses to protected shadow memory are not allowed
+       //                 with the software memory manager enabled.
 #if ASAN_NEEDS_SEGV
 TEST(AddressSanitizer, ShadowGapTest) {
 #if SANITIZER_WORDSIZE == 32

   EXPECT_DEATH(*addr = 1, "AddressSanitizer: SEGV on unknown");
 }
 #endif  // ASAN_NEEDS_SEGV
+#endif
 
 extern "C" {
 NOINLINE static void UseThenFreeThenUse() {

lib/sanitizer_common/sanitizer_linux.cc

 }
 #endif // !SANITIZER_S390
 
+// TODO(ikosarev)
+int internal_msync(void *addr, uptr length, int flags) {
+  return internal_syscall(SYSCALL(msync), (uptr)addr, length, flags);
+}
+
 uptr internal_munmap(void *addr, uptr length) {
   return internal_syscall(SYSCALL(munmap), (uptr)addr, length);
 }

lib/sanitizer_common/sanitizer_platform.h

 #define SANITIZER_NLDBL_VERSION "GLIBC_2.4"
 #endif
 
+// Whether to use the shadow memory manager. Useful for plaforms that do not
+// support virtual memory.
+#define SANITIZER_USE_SHADOW_MEMORY_MANAGER 1
+
 #endif // SANITIZER_PLATFORM_H

lib/sanitizer_common/sanitizer_posix.h

 // Memory
 uptr internal_mmap(void *addr, uptr length, int prot, int flags,
                    int fd, OFF_T offset);
+int internal_msync(void *addr, uptr length, int flags);
 uptr internal_munmap(void *addr, uptr length);
 int internal_mprotect(void *addr, uptr length, int prot);
 

lib/tsan/rtl/tsan_interceptors.cc

 static ThreadSignalContext *SigCtx(ThreadState *thr) {
   ThreadSignalContext *ctx = (ThreadSignalContext*)thr->signal_ctx;
   if (ctx == 0 && !thr->is_dead) {
+    DPrintf("MmapOrDie() in SigCtx()\n");
     ctx = (ThreadSignalContext*)MmapOrDie(sizeof(*ctx), "ThreadSignalContext");
     MemoryResetRange(thr, (uptr)&SigCtx, (uptr)ctx, sizeof(*ctx));
     thr->signal_ctx = ctx;

 
 TSAN_INTERCEPTOR(int, pthread_create,
     void *th, void *attr, void *(*callback)(void*), void * param) {
+  DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
   SCOPED_INTERCEPTOR_RAW(pthread_create, th, attr, callback, param);
   if (ctx->after_multithreaded_fork) {
     if (flags()->die_after_fork) {

     // 2. ThreadStart must finish before this thread continues.
     //    Otherwise, this thread can call pthread_detach and reset thr->sync
     //    before the new thread got a chance to acquire from it in ThreadStart.
+    DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
     atomic_store(&p.tid, tid, memory_order_release);
     while (atomic_load(&p.tid, memory_order_acquire) != 0)
       internal_sched_yield();
+    DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
   }
   if (attr == &myattr)
     pthread_attr_destroy(&myattr);
+  DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
   return res;
 }
 

 };
 
 static bool IsAppNotRodata(uptr addr) {
-  return IsAppMem(addr) && *(u64*)MemToShadow(addr) != kShadowRodata;
+  return IsAppMem(addr) && *MemToPShadow(addr) != kShadowRodata;
 }
 
 static int dl_iterate_phdr_cb(__sanitizer_dl_phdr_info *info, SIZE_T size,

 
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE
 void __tsan_testonly_barrier_wait(u64 *barrier) {
+  DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
   unsigned old = __atomic_fetch_add(barrier, 1 << 8, __ATOMIC_RELAXED);
   unsigned old_epoch = (old >> 8) / (old & 0xff);
   for (;;) {
     unsigned cur = __atomic_load_n(barrier, __ATOMIC_RELAXED);
     unsigned cur_epoch = (cur >> 8) / (cur & 0xff);
-    if (cur_epoch != old_epoch)
+    if (cur_epoch != old_epoch) {
+      DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
       return;
+    }
     internal_sched_yield();
   }
 }

lib/tsan/rtl/tsan_interface_java.cc

   ctx->metamap.MoveMemory(src, dst, size);
 
   // Move shadow.
-  u64 *s = (u64*)MemToShadow(src);
-  u64 *d = (u64*)MemToShadow(dst);
-  u64 *send = (u64*)MemToShadow(src + size);
-  uptr inc = 1;
+  uptr s = MemToVShadow(src);
+  uptr d = MemToVShadow(dst);
+  uptr s_end = MemToVShadow(src + size);
+  uptr inc = 8;
   if (dst > src) {
-    s = (u64*)MemToShadow(src + size) - 1;
-    d = (u64*)MemToShadow(dst + size) - 1;
-    send = (u64*)MemToShadow(src) - 1;
-    inc = -1;
+    s = MemToVShadow(src + size) - 8;
+    d = MemToVShadow(dst + size) - 8;
+    s_end = MemToVShadow(src) - 8;
+    inc = -8;
   }
-  for (; s != send; s += inc, d += inc) {
-    *d = *s;
-    *s = 0;
+  for (; s != s_end; s += inc, d += inc) {
+    u64 *ps = VShadowToPShadow(s);
+    *VShadowToPShadow(d) = *ps;
+    *ps = 0;
   }
 }
 

lib/tsan/rtl/tsan_mman.cc

     diff = p + size - RoundDown(p + size, kPageSize);
     if (diff != 0)
       size -= diff;
-    ReleaseMemoryToOS((uptr)MemToMeta(p), size / kMetaRatio);
+    ReleaseVMeta(MemToVMeta(p), size / kMetaRatio);
   }
 };
 

 
 void user_free(ThreadState *thr, uptr pc, void *p, bool signal) {
   ScopedGlobalProcessor sgp;
+  DPrintf("user_free()\n");
   if (ctx && ctx->initialized)
     OnUserFree(thr, pc, (uptr)p, true);
   allocator()->Deallocate(&thr->proc()->alloc_cache, p);

 }
 
 void OnUserFree(ThreadState *thr, uptr pc, uptr p, bool write) {
+  DPrintf("OnUserFree()\n");
   CHECK_NE(p, (void*)0);
   uptr sz = ctx->metamap.FreeBlock(thr->proc(), p);
-  DPrintf("#%d: free(%p, %zu)\n", thr->tid, p, sz);
+  DPrintf("#%d: free(%zx, %zu)\n", thr->tid, p, sz);
   if (write && thr->ignore_reads_and_writes == 0)
     MemoryRangeFreed(thr, pc, (uptr)p, sz);
 }

lib/tsan/rtl/tsan_new_delete.cc

 SANITIZER_INTERFACE_ATTRIBUTE
 void operator delete(void *ptr) NOEXCEPT;
 void operator delete(void *ptr) NOEXCEPT {
+  DPrintf("operator delete ()\n");
   OPERATOR_DELETE_BODY(_ZdlPv);
 }
 

lib/tsan/rtl/tsan_platform.h

 # error "Only 64-bit is supported"
 #endif
 
+#include "sanitizer_common/sanitizer_common.h"
 #include "tsan_defs.h"
 #include "tsan_trace.h"
 

 
 
 template<typename Mapping>
-bool IsShadowMemImpl(uptr mem) {
+bool IsVShadowMemImpl(uptr mem) {
   return mem >= Mapping::kShadowBeg && mem <= Mapping::kShadowEnd;
 }
 
 ALWAYS_INLINE
-bool IsShadowMem(uptr mem) {
+bool IsVShadowMem(uptr mem) {
 #ifdef __aarch64__
   switch (vmaSize) {
-    case 39: return IsShadowMemImpl<Mapping39>(mem);
-    case 42: return IsShadowMemImpl<Mapping42>(mem);
-    case 48: return IsShadowMemImpl<Mapping48>(mem);
+    case 39: return IsVShadowMemImpl<Mapping39>(mem);
+    case 42: return IsVShadowMemImpl<Mapping42>(mem);
+    case 48: return IsVShadowMemImpl<Mapping48>(mem);
   }
   DCHECK(0);
   return false;
 #elif defined(__powerpc64__)
   if (vmaSize == 44)
-    return IsShadowMemImpl<Mapping44>(mem);
+    return IsVShadowMemImpl<Mapping44>(mem);
   else
-    return IsShadowMemImpl<Mapping46>(mem);
+    return IsVShadowMemImpl<Mapping46>(mem);
   DCHECK(0);
 #else
-  return IsShadowMemImpl<Mapping>(mem);
+  return IsVShadowMemImpl<Mapping>(mem);
 #endif
 }
 

 
 
 template<typename Mapping>
-uptr MemToShadowImpl(uptr x) {
+uptr MemToVShadowImpl(uptr x) {
   DCHECK(IsAppMem(x));
 #ifndef SANITIZER_GO
   return (((x) & ~(Mapping::kAppMemMsk | (kShadowCell - 1)))

 }
 
 ALWAYS_INLINE
-uptr MemToShadow(uptr x) {
+uptr MemToVShadow(uptr x) {
 #ifdef __aarch64__
   switch (vmaSize) {
-    case 39: return MemToShadowImpl<Mapping39>(x);
-    case 42: return MemToShadowImpl<Mapping42>(x);
-    case 48: return MemToShadowImpl<Mapping48>(x);
+    case 39: return MemToVShadowImpl<Mapping39>(x);
+    case 42: return MemToVShadowImpl<Mapping42>(x);
+    case 48: return MemToVShadowImpl<Mapping48>(x);
   }
   DCHECK(0);
   return 0;
 #elif defined(__powerpc64__)
   if (vmaSize == 44)
-    return MemToShadowImpl<Mapping44>(x);
+    return MemToVShadowImpl<Mapping44>(x);
   else
-    return MemToShadowImpl<Mapping46>(x);
+    return MemToVShadowImpl<Mapping46>(x);
   DCHECK(0);
 #else
-  return MemToShadowImpl<Mapping>(x);
+  return MemToVShadowImpl<Mapping>(x);
 #endif
 }
 
+#ifndef SANITIZER_GO
+namespace shadow_map {
+u64 *VShadowToPShadow(uptr vs);
+u32 *VMetaToPMeta(uptr vs);
+}
+#endif
 
+ALWAYS_INLINE
+u64 *VShadowToPShadow(uptr vs) {
+#ifndef SANITIZER_GO
+  return shadow_map::VShadowToPShadow(vs);
+#else
+  return reinterpret_cast<u64*>(vs);
+#endif
+}
+
+ALWAYS_INLINE
+u64 *MemToPShadow(uptr mem) {
+  return VShadowToPShadow(MemToVShadow(mem));
+}
+
+ALWAYS_INLINE
+void ZeroVShadow(uptr vs, uptr size) {
+#if 1  // TODO(ikosarev)
+  for (uptr i = 0; i != size; ++i)
+    *reinterpret_cast<u8*>(VShadowToPShadow(vs + i)) = 0;
+#else
+  u64 *ps = VShadowToPShadow(vs);
+  UnmapOrDie((void*)ps, size);
+
+  // TODO(ikosarev): Don't do this when we allocate meta info on-demand.
+  MmapFixedNoReserve((uptr)ps, size);
+#endif
+}
+
+ALWAYS_INLINE
+void ReleaseVShadow(uptr addr, uptr size) {
+#if 1  // TODO(ikosarev)
+  ZeroVShadow(addr, size);
+#else
+  ReleaseMemoryToOS(addr, size);
+#endif
+}
+
 template<typename Mapping>
-u32 *MemToMetaImpl(uptr x) {
+uptr MemToVMetaImpl(uptr x) {
   DCHECK(IsAppMem(x));
 #ifndef SANITIZER_GO
-  return (u32*)(((((x) & ~(Mapping::kAppMemMsk | (kMetaShadowCell - 1)))) /
+  return (((((x) & ~(Mapping::kAppMemMsk | (kMetaShadowCell - 1)))) /
       kMetaShadowCell * kMetaShadowSize) | Mapping::kMetaShadowBeg);
 #else
-  return (u32*)(((x & ~(kMetaShadowCell - 1)) / \
+  return (((x & ~(kMetaShadowCell - 1)) / \
       kMetaShadowCell * kMetaShadowSize) | Mapping::kMetaShadowBeg);
 #endif
 }
 
 ALWAYS_INLINE
-u32 *MemToMeta(uptr x) {
+uptr MemToVMeta(uptr x) {
 #ifdef __aarch64__
   switch (vmaSize) {
-    case 39: return MemToMetaImpl<Mapping39>(x);
-    case 42: return MemToMetaImpl<Mapping42>(x);
-    case 48: return MemToMetaImpl<Mapping48>(x);
+    case 39: return MemToVMetaImpl<Mapping39>(x);
+    case 42: return MemToVMetaImpl<Mapping42>(x);
+    case 48: return MemToVMetaImpl<Mapping48>(x);
   }
   DCHECK(0);
   return 0;
 #elif defined(__powerpc64__)
   if (vmaSize == 44)
-    return MemToMetaImpl<Mapping44>(x);
+    return MemToVMetaImpl<Mapping44>(x);
   else
-    return MemToMetaImpl<Mapping46>(x);
+    return MemToVMetaImpl<Mapping46>(x);
   DCHECK(0);
 #else
-  return MemToMetaImpl<Mapping>(x);
+  return MemToVMetaImpl<Mapping>(x);
 #endif
 }
 
+ALWAYS_INLINE
+u32 *VMetaToPMeta(uptr vm) {
+#ifndef SANITIZER_GO
+  return shadow_map::VMetaToPMeta(vm);
+#else
+  return reinterpret_cast<u32*>(vm);
+#endif
+}
 
+
+ALWAYS_INLINE
+u32 *MemToPMeta(uptr mem) {
+  return VMetaToPMeta(MemToVMeta(mem));
+}
+
+ALWAYS_INLINE
+void ZeroVMeta(uptr vm, uptr size) {
+  u32 *pm = VMetaToPMeta(vm);
+  UnmapOrDie((void*)pm, size);
+
+  // TODO(ikosarev): Don't do this when we allocate meta info on-demand.
+  MmapFixedNoReserve((uptr)pm, size);
+}
+
+ALWAYS_INLINE
+void ReleaseVMeta(uptr vm, uptr size) {
+  u32 *pm = VMetaToPMeta(vm);
+  ReleaseMemoryToOS((uptr)pm, size);
+}
+
 template<typename Mapping>
-uptr ShadowToMemImpl(uptr s) {
-  DCHECK(IsShadowMem(s));
+uptr VShadowToMemImpl(uptr s) {
+  DCHECK(IsVShadowMem(s));
 #ifndef SANITIZER_GO
   // The shadow mapping is non-linear and we've lost some bits, so we don't have
   // an easy way to restore the original app address. But the mapping is a

   // same address.
   uptr p = (s / kShadowCnt) ^ Mapping::kAppMemXor;
   if (p >= Mapping::kLoAppMemBeg && p < Mapping::kLoAppMemEnd &&
-      MemToShadow(p) == s)
+      MemToVShadow(p) == s)
     return p;
 # ifdef TSAN_MID_APP_RANGE
   p = ((s / kShadowCnt) ^ Mapping::kAppMemXor) +
       (Mapping::kMidAppMemBeg & Mapping::kAppMemMsk);
   if (p >= Mapping::kMidAppMemBeg && p < Mapping::kMidAppMemEnd &&
-      MemToShadow(p) == s)
+      MemToVShadow(p) == s)
     return p;
 # endif
   return ((s / kShadowCnt) ^ Mapping::kAppMemXor) | Mapping::kAppMemMsk;

 }
 
 ALWAYS_INLINE
-uptr ShadowToMem(uptr s) {
+uptr VShadowToMem(uptr s) {
 #ifdef __aarch64__
   switch (vmaSize) {
-    case 39: return ShadowToMemImpl<Mapping39>(s);
-    case 42: return ShadowToMemImpl<Mapping42>(s);
-    case 48: return ShadowToMemImpl<Mapping48>(s);
+    case 39: return VShadowToMemImpl<Mapping39>(s);
+    case 42: return VShadowToMemImpl<Mapping42>(s);
+    case 48: return VShadowToMemImpl<Mapping48>(s);
   }
   DCHECK(0);
   return 0;
 #elif defined(__powerpc64__)
   if (vmaSize == 44)
-    return ShadowToMemImpl<Mapping44>(s);
+    return VShadowToMemImpl<Mapping44>(s);
   else
-    return ShadowToMemImpl<Mapping46>(s);
+    return VShadowToMemImpl<Mapping46>(s);
   DCHECK(0);
 #else
-  return ShadowToMemImpl<Mapping>(s);
+  return VShadowToMemImpl<Mapping>(s);
 #endif
 }
 

lib/tsan/rtl/tsan_platform_linux.cc

 #endif
 
 void FlushShadowMemory() {
+#if 0  // TODO(ikosarev)
 #if SANITIZER_LINUX
   StopTheWorld(FlushShadowMemoryCallback, 0);
 #endif
+#endif
 }
 
 #ifndef SANITIZER_GO

 // Mark shadow for .rodata sections with the special kShadowRodata marker.
 // Accesses to .rodata can't race, so this saves time, memory and trace space.
 static void MapRodata() {
+  return;  // TODO(ikosarev): Do not handle .rodata in a special way for now.
+
   // First create temp file.
   const char *tmpdir = GetEnv("TMPDIR");
   if (tmpdir == 0)

         && !(prot & MemoryMappingLayout::kProtectionWrite)
         && IsAppMem(start)) {
       // Assume it's .rodata
-      char *shadow_start = (char*)MemToShadow(start);
-      char *shadow_end = (char*)MemToShadow(end);
+      char *shadow_start = (char*)MemToVShadow(start);
+      char *shadow_end = (char*)MemToVShadow(end);
       for (char *p = shadow_start; p < shadow_end; p += marker.size()) {
         internal_mmap(p, Min<uptr>(marker.size(), shadow_end - p),
                       PROT_READ, MAP_PRIVATE | MAP_FIXED, fd, 0);

     CHECK_EQ(0, internal_sigprocmask(SIG_SETMASK, &emptyset, &oldset));
     thr = reinterpret_cast<ThreadState*>(__get_tls()[TLS_SLOT_TSAN]);
     if (thr == nullptr) {
+      Printf("MmapOrDie() in cur_thread()\n");
       thr = reinterpret_cast<ThreadState*>(MmapOrDie(sizeof(ThreadState),
                                                      "ThreadState"));
       __get_tls()[TLS_SLOT_TSAN] = thr;
       if (dead_thread_state == nullptr) {
+        Printf("MmapOrDie() in cur_thread() #2\n");
         dead_thread_state = reinterpret_cast<ThreadState*>(
             MmapOrDie(sizeof(ThreadState), "ThreadState"));
         dead_thread_state->fast_state.SetIgnoreBit();

lib/tsan/rtl/tsan_platform_mac.cc

   if (thread_identity == main_thread_identity || main_thread_identity == 0) {
     return (ThreadState *)&main_thread_state;
   }
-  ThreadState **fake_tls = (ThreadState **)MemToShadow(thread_identity);
+  ThreadState **fake_tls = (ThreadState **)MemToVShadow(thread_identity);
   ThreadState *thr = (ThreadState *)SignalSafeGetOrAllocate(
       (uptr *)fake_tls, sizeof(ThreadState));
   return thr;

     // exit the main thread. Let's keep the main thread's ThreadState.
     return;
   }
-  ThreadState **fake_tls = (ThreadState **)MemToShadow(thread_identity);
+  ThreadState **fake_tls = (ThreadState **)MemToVShadow(thread_identity);
   internal_munmap(*fake_tls, sizeof(ThreadState));
   *fake_tls = nullptr;
 }

lib/tsan/rtl/tsan_platform_posix.cc

 
 #include "sanitizer_common/sanitizer_common.h"
 #include "sanitizer_common/sanitizer_libc.h"
+#include "sanitizer_common/sanitizer_posix.h"
 #include "sanitizer_common/sanitizer_procmaps.h"
 #include "tsan_platform.h"
 #include "tsan_rtl.h"

 
 namespace __tsan {
 
+namespace shadow_map {
+
+const unsigned kShadowPageScale = 16;  // 64K bytes
+
+#define USE_SOFTWARE_MEMORY_MANAGER 1  // TODO(ikosarev)
+
+#if USE_SOFTWARE_MEMORY_MANAGER
+
+const unsigned kLinkSizeScale = kShadowPageScale;
+const uptr kLinkSize = ((uptr)1) << kLinkSizeScale;
+const uptr kLinkSizeMask = kLinkSize - 1;
+
+// The first free physical shadow address.
+uptr PShadowEnd;
+
+typedef u8 Level0[kLinkSize];       // A 64Kb page.
+typedef Level0 *Level1[kLinkSize];  // 4G bytes.
+typedef Level1 *Level2[kLinkSize];  // 2^48 bytes.
+typedef Level2 *Level3[kLinkSize];  // 2^64 bytes.
+Level3 *PShadowMapRoot;
+
+void Init() {
+  PShadowEnd = ShadowBeg();
+  PShadowMapRoot = nullptr;
+  DPrintf("PShadow initialized.\n");
+}
+
+void *pshadow_alloc(uptr size) {
+  // Printf("Allocate %zx.\n", size);
+
+  uptr sys_page_size = GetPageSizeCached();
+  size = RoundUpTo(size, sys_page_size);
+
+  CHECK_EQ(PShadowEnd % sys_page_size, 0);
+  uptr beg = PShadowEnd;
+  PShadowEnd += size;
+
+  void *ptr = MmapFixedNoReserve(beg, size, /* name= */ nullptr);
+  if (ptr != reinterpret_cast<void*>(beg)) {
+    Report("Cannot allocate physical shadow memory.\n");
+    Abort();
+  }
+
+  return ptr;
+}
+
+template<typename T>
+T *pshadow_alloc() {
+  return static_cast<T*>(pshadow_alloc(sizeof(T)));
+}
+
+u64 *VShadowToPShadow(uptr vs) {
+  Level3 *&p3 = PShadowMapRoot;
+  if (!p3)
+    p3 = pshadow_alloc<Level3>();
+
+  uptr i3 = (vs >> (kLinkSizeScale * 3)) & kLinkSizeMask;
+  Level2 *&p2 = (*p3)[i3];
+  if (!p2)
+    p2 = pshadow_alloc<Level2>();
+
+  uptr i2 = (vs >> (kLinkSizeScale * 2)) & kLinkSizeMask;
+  Level1 *&p1 = (*p2)[i2];
+  if (!p1)
+    p1 = pshadow_alloc<Level1>();
+
+  uptr i1 = (vs >> (kLinkSizeScale * 1)) & kLinkSizeMask;
+  Level0 *&p0 = (*p1)[i1];
+  if (!p0)
+    p0 = pshadow_alloc<Level0>();
+
+  uptr i0 = (vs >> (kLinkSizeScale * 0)) & kLinkSizeMask;
+  u8 *p = &(*p0)[i0];
+
+  return reinterpret_cast<u64*>(p);
+}
+
+u32 *VMetaToPMeta(uptr vm) {
+    return reinterpret_cast<u32*>(VShadowToPShadow(vm));
+}
+
+#else
+
+const uptr kShadowPageSize = ((uptr)1) << kShadowPageScale;
+
+static void MmapShadowMemoryRange(uptr beg, uptr end) {
+  uptr shadow = (uptr)MmapFixedNoReserve(beg, end - beg, "shadow");
+  if (shadow != beg) {
+    Printf("FATAL: ThreadSanitizer can not mmap the shadow memory\n");
+    Printf("FATAL: Make sure to compile with -fPIE and "
+           "to link with -pie (%zx, %zx).\n", shadow, beg);
+    Die();
+  }
+}
+
+u64 *VShadowToPShadow(uptr vs) {
+  // TODO(ikosarev): Refine.
+  uptr sys_page_size = GetPageSizeCached();
+  CHECK_EQ(kShadowPageSize % sys_page_size, 0);
+  uptr page_size = kShadowPageSize;
+
+  uptr beg = RoundDownTo(vs, page_size);
+  uptr end = beg + page_size - 1;
+
+  int res = internal_msync((void*)beg, /* len= */ page_size, /* flags= */ 0);
+  int reserrno;
+  if (internal_iserror(res, &reserrno) && reserrno == 12 /* ENOMEM */) {
+    // Printf("VShadowToPShadowOrNull(%zx): Reserve %zx - %zx.\n",
+    //        vs, beg, end);
+    MmapShadowMemoryRange(beg, end);
+  }
+  return reinterpret_cast<u64*>(vs);
+}
+
+static void MmapMetaInfoMemoryRange(uptr beg, uptr end) {
+  uptr meta_size = end - beg;
+  uptr meta = (uptr)MmapFixedNoReserve(beg, meta_size, "meta shadow");
+  if (meta != beg) {
+    Printf("FATAL: ThreadSanitizer can not mmap the shadow memory\n");
+    Printf("FATAL: Make sure to compile with -fPIE and "
+               "to link with -pie (%zx, %zx).\n", meta, beg);
+    Die();
+  }
+}
+
+u32 *VMetaToPMeta(uptr vm) {
+  // TODO(ikosarev): Refine.
+  uptr sys_page_size = GetPageSizeCached();
+  CHECK_EQ(kShadowPageSize % sys_page_size, 0);
+  uptr page_size = kShadowPageSize;
+
+  uptr beg = RoundDownTo(vm, page_size);
+  uptr end = beg + page_size - 1;
+
+  int res = internal_msync((void*)beg, /* len= */ page_size, /* flags= */ 0);
+  int reserrno;
+  if (internal_iserror(res, &reserrno) && reserrno == 12 /* ENOMEM */) {
+    // Printf("VShadowToPShadowOrNull(%zx): Reserve %zx - %zx.\n",
+    //        vm, beg, end);
+    MmapMetaInfoMemoryRange(beg, end);
+  }
+  return reinterpret_cast<u32*>(vm);
+}
+#endif
+
+}  // namespace shadow_map
+
 #ifndef SANITIZER_GO
 void InitializeShadowMemory() {
+#if 0  // TODO(ikosarev)
   // Map memory shadow.
   uptr shadow =
       (uptr)MmapFixedNoReserve(ShadowBeg(), ShadowEnd() - ShadowBeg(),

                "to link with -pie (%p, %p).\n", shadow, ShadowBeg());
     Die();
   }
+#endif
   // This memory range is used for thread stacks and large user mmaps.
   // Frequently a thread uses only a small part of stack and similarly
   // a program uses a small part of large mmap. On some programs

     DCHECK(0);
   }
 #endif
-  NoHugePagesInRegion(MemToShadow(kMadviseRangeBeg),
+  NoHugePagesInRegion(MemToVShadow(kMadviseRangeBeg),
                       kMadviseRangeSize * kShadowMultiplier);
   // Meta shadow is compressing and we don't flush it,
   // so it makes sense to mark it as NOHUGEPAGE to not over-allocate memory.

       (ShadowEnd() - ShadowBeg()) >> 30);
 
   // Map meta shadow.
+#if 0  // TODO(ikosarev)
   uptr meta_size = MetaShadowEnd() - MetaShadowBeg();
   uptr meta =
       (uptr)MmapFixedNoReserve(MetaShadowBeg(), meta_size, "meta shadow");

     DontDumpShadowMemory(meta, meta_size);
   DPrintf("meta shadow: %zx-%zx (%zuGB)\n",
       meta, meta + meta_size, meta_size >> 30);
+#endif
 
+#if USE_SOFTWARE_MEMORY_MANAGER
+  shadow_map::Init();
+#endif
+
   InitializeShadowMemoryPlatform();
 }
 

   ProtectRange(TraceMemEnd(), HeapMemBeg());
   ProtectRange(HeapEnd(), HiAppMemBeg());
 }
-#endif
 
+#endif  // !SANITIZER_GO
+
 }  // namespace __tsan
 
 #endif  // SANITIZER_POSIX

lib/tsan/rtl/tsan_rtl.h

   uptr *shadow_stack;
   uptr *shadow_stack_end;
   uptr *shadow_stack_pos;
-  u64 *racy_shadow_addr;
+  uptr racy_vshadow_addr;
   u64 racy_state[2];
   MutexSet mset;
   ThreadClock clock;

 bool IsExpectedReport(uptr addr, uptr size);
 void PrintMatchedBenignRaces();
 
+// TODO(ikosarev)
+// #define TSAN_DEBUG_OUTPUT 1
+
 #if defined(TSAN_DEBUG_OUTPUT) && TSAN_DEBUG_OUTPUT >= 1
 # define DPrintf Printf
 #else

     int kAccessSizeLog, bool kAccessIsWrite, bool kIsAtomic);
 void MemoryAccessImpl(ThreadState *thr, uptr addr,
     int kAccessSizeLog, bool kAccessIsWrite, bool kIsAtomic,
-    u64 *shadow_mem, Shadow cur);
+    uptr shadow_mem, Shadow cur);
 void MemoryAccessRange(ThreadState *thr, uptr pc, uptr addr,
     uptr size, bool is_write);
 void MemoryAccessRangeStep(ThreadState *thr, uptr pc, uptr addr,

lib/tsan/rtl/tsan_rtl.cc

 #endif
 
 void DontNeedShadowFor(uptr addr, uptr size) {
-  uptr shadow_beg = MemToShadow(addr);
-  uptr shadow_end = MemToShadow(addr + size);
-  ReleaseMemoryToOS(shadow_beg, shadow_end - shadow_beg);
+  uptr shadow_beg = MemToVShadow(addr);
+  uptr shadow_end = MemToVShadow(addr + size);
+  ReleaseVShadow(shadow_beg, shadow_end - shadow_beg);
 }
 
 void MapShadow(uptr addr, uptr size) {

   // Global data is not 64K aligned, but there are no adjacent mappings,
   // so we can get away with unaligned mapping.
   // CHECK_EQ(addr, addr & ~((64 << 10) - 1));  // windows wants 64K alignment
-  MmapFixedNoReserve(MemToShadow(addr), size * kShadowMultiplier, "shadow");
+  // TODO(ikosarev): Allocate shadow memory with a special function.
+  CHECK(0);
+  MmapFixedNoReserve((uptr)MemToPShadow(addr), size * kShadowMultiplier,
+                     "shadow");
 
   // Meta shadow is 2:1, so tread carefully.
   static bool data_mapped = false;
   static uptr mapped_meta_end = 0;
-  uptr meta_begin = (uptr)MemToMeta(addr);
-  uptr meta_end = (uptr)MemToMeta(addr + size);
+  // TODO(ikosarev): Allocate meta info memory with a special function.
+  uptr meta_begin = (uptr)MemToPMeta(addr);
+  uptr meta_end = (uptr)MemToPMeta(addr + size);
   meta_begin = RoundDownTo(meta_begin, 64 << 10);
   meta_end = RoundUpTo(meta_end, 64 << 10);
   if (!data_mapped) {

         const uptr p = RoundDown(p0 + x, kShadowCell);
         if (p < beg || p >= end)
           continue;
-        const uptr s = MemToShadow(p);
-        const uptr m = (uptr)MemToMeta(p);
-        VPrintf(3, "  checking pointer %p: shadow=%p meta=%p\n", p, s, m);
+        const uptr vs = MemToVShadow(p);
+        const uptr vm = MemToVMeta(p);
+        VPrintf(3, "  checking pointer %p: shadow=%zx meta=%xz\n", p, vs, vm);
         CHECK(IsAppMem(p));
-        CHECK(IsShadowMem(s));
-        CHECK_EQ(p, ShadowToMem(s));
-        CHECK(IsMetaMem(m));
+        CHECK(IsVShadowMem(vs));
+        CHECK_EQ(p, VShadowToMem(vs));
+        CHECK(IsMetaMem(vm));
         if (prev) {
           // Ensure that shadow and meta mappings are linear within a single
           // user range. Lots of code that processes memory ranges assumes it.
-          const uptr prev_s = MemToShadow(prev);
-          const uptr prev_m = (uptr)MemToMeta(prev);
-          CHECK_EQ(s - prev_s, (p - prev) * kShadowMultiplier);
-          CHECK_EQ((m - prev_m) / kMetaShadowSize,
+          const uptr prev_vs = MemToVShadow(prev);
+          const uptr prev_vm = MemToVMeta(prev);
+          CHECK_EQ(vs - prev_vs, (p - prev) * kShadowMultiplier);
+          CHECK_EQ((vm - prev_vm) / kMetaShadowSize,
                    (p - prev) / kMetaShadowCell);
         }
         prev = p;

 #endif
 
 ALWAYS_INLINE
-Shadow LoadShadow(u64 *p) {
-  u64 raw = atomic_load((atomic_uint64_t*)p, memory_order_relaxed);
+Shadow LoadVShadow(uptr vs) {
+  u64 *ps = VShadowToPShadow(vs);
+  u64 raw = atomic_load(reinterpret_cast<atomic_uint64_t*>(ps),
+                        memory_order_relaxed);
   return Shadow(raw);
 }
 
 ALWAYS_INLINE
-void StoreShadow(u64 *sp, u64 s) {
-  atomic_store((atomic_uint64_t*)sp, s, memory_order_relaxed);
+void StoreVShadow(uptr vs, u64 s) {
+  u64 *ps = VShadowToPShadow(vs);
+  atomic_store(reinterpret_cast<atomic_uint64_t*>(ps), s,
+               memory_order_relaxed);
 }
 
 ALWAYS_INLINE
-void StoreIfNotYetStored(u64 *sp, u64 *s) {
-  StoreShadow(sp, *s);
+void StoreIfNotYetStored(uptr vs, u64 *s) {
+  StoreVShadow(vs, *s);
   *s = 0;
 }
 
 ALWAYS_INLINE
-void HandleRace(ThreadState *thr, u64 *shadow_mem,
-                              Shadow cur, Shadow old) {
+void HandleRace(ThreadState *thr, uptr vshadow_mem,
   thr->racy_state[0] = cur.raw();
   thr->racy_state[1] = old.raw();
-  thr->racy_shadow_addr = shadow_mem;
+  thr->racy_vshadow_addr = vshadow_mem;
 #ifndef SANITIZER_GO
   HACKY_CALL(__tsan_report_race);
 #else

 ALWAYS_INLINE
 void MemoryAccessImpl1(ThreadState *thr, uptr addr,
     int kAccessSizeLog, bool kAccessIsWrite, bool kIsAtomic,
-    u64 *shadow_mem, Shadow cur) {
+    uptr vshadow_mem, Shadow cur) {
   StatInc(thr, StatMop);
   StatInc(thr, kAccessIsWrite ? StatMopWrite : StatMopRead);
   StatInc(thr, (StatType)(StatMop1 + kAccessSizeLog));

   if (LIKELY(store_word == 0))
     return;
   // choose a random candidate slot and replace it
-  StoreShadow(shadow_mem + (cur.epoch() % kShadowCnt), store_word);
+  StoreVShadow(vshadow_mem + (cur.epoch() % kShadowCnt) * 8, store_word);
   StatInc(thr, StatShadowReplace);
   return;
  RACE:
-  HandleRace(thr, shadow_mem, cur, old);
+  HandleRace(thr, vshadow_mem, cur, old);
   return;
 }
 

 }
 
 ALWAYS_INLINE
-bool ContainsSameAccessSlow(u64 *s, u64 a, u64 sync_epoch, bool is_write) {
+bool ContainsSameAccessSlow(uptr vs, u64 a, u64 sync_epoch, bool is_write) {
   Shadow cur(a);
   for (uptr i = 0; i < kShadowCnt; i++) {
-    Shadow old(LoadShadow(&s[i]));
+    Shadow old(LoadVShadow(vs + i * 8));
     if (Shadow::Addr0AndSizeAreEqual(cur, old) &&
         old.TidWithIgnore() == cur.TidWithIgnore() &&
         old.epoch() > sync_epoch &&

   return false;
 }
 
-#if defined(__SSE3__)
+#if 0  // TODO(ikosarev): defined(__SSE3__)
 #define SHUF(v0, v1, i0, i1, i2, i3) _mm_castps_si128(_mm_shuffle_ps( \
     _mm_castsi128_ps(v0), _mm_castsi128_ps(v1), \
     (i0)*1 + (i1)*4 + (i2)*16 + (i3)*64))

 #endif
 
 ALWAYS_INLINE
-bool ContainsSameAccess(u64 *s, u64 a, u64 sync_epoch, bool is_write) {
-#if defined(__SSE3__)
-  bool res = ContainsSameAccessFast(s, a, sync_epoch, is_write);
+bool ContainsSameAccess(uptr vs, u64 a, u64 sync_epoch, bool is_write) {
+#if 0  // TODO(ikosarev): defined(__SSE3__)
+  bool res = ContainsSameAccessFast(vs, a, sync_epoch, is_write);
   // NOTE: this check can fail if the shadow is concurrently mutated
   // by other threads. But it still can be useful if you modify
   // ContainsSameAccessFast and want to ensure that it's not completely broken.

   // DCHECK_EQ(res, ContainsSameAccessSlow(s, a, sync_epoch, is_write));
   return res;
 #else
-  return ContainsSameAccessSlow(s, a, sync_epoch, is_write);
+  return ContainsSameAccessSlow(vs, a, sync_epoch, is_write);
 #endif
 }
 

 ALWAYS_INLINE USED
 void MemoryAccess(ThreadState *thr, uptr pc, uptr addr,
     int kAccessSizeLog, bool kAccessIsWrite, bool kIsAtomic) {
-  u64 *shadow_mem = (u64*)MemToShadow(addr);
+  uptr vshadow_mem = MemToVShadow(addr);
+  u64 *pshadow_mem = VShadowToPShadow(vshadow_mem);
   DPrintf2("#%d: MemoryAccess: @%p %p size=%d"
-      " is_write=%d shadow_mem=%p {%zx, %zx, %zx, %zx}\n",
+      " is_write=%d shadow_mem=%zx {%zx, %zx, %zx, %zx}\n",
       (int)thr->fast_state.tid(), (void*)pc, (void*)addr,
-      (int)(1 << kAccessSizeLog), kAccessIsWrite, shadow_mem,
-      (uptr)shadow_mem[0], (uptr)shadow_mem[1],
-      (uptr)shadow_mem[2], (uptr)shadow_mem[3]);
+      (int)(1 << kAccessSizeLog), kAccessIsWrite, vshadow_mem,
+      // TODO(ikosarev): Do VShadowToPShadow() for each of these:
+      (uptr)pshadow_mem[0], (uptr)pshadow_mem[1],
+      (uptr)pshadow_mem[2], (uptr)pshadow_mem[3]);
 #if SANITIZER_DEBUG
   if (!IsAppMem(addr)) {
     Printf("Access to non app mem %zx\n", addr);
     DCHECK(IsAppMem(addr));
   }
-  if (!IsShadowMem((uptr)shadow_mem)) {
-    Printf("Bad shadow addr %p (%zx)\n", shadow_mem, addr);
-    DCHECK(IsShadowMem((uptr)shadow_mem));
+  if (!IsVShadowMem(vshadow_mem)) {
+    Printf("Bad shadow addr %zx (%zx)\n", vshadow_mem, addr);
+    DCHECK(IsVShadowMem(vshadow_mem));
   }
 #endif
 
-  if (kCppMode && *shadow_mem == kShadowRodata) {
+  if (kCppMode && *pshadow_mem == kShadowRodata) {
     // Access to .rodata section, no races here.
     // Measurements show that it can be 10-20% of all memory accesses.
     StatInc(thr, StatMop);

   cur.SetWrite(kAccessIsWrite);
   cur.SetAtomic(kIsAtomic);
 
-  if (LIKELY(ContainsSameAccess(shadow_mem, cur.raw(),
+  if (LIKELY(ContainsSameAccess(vshadow_mem, cur.raw(),
                                 thr->fast_synch_epoch, kAccessIsWrite))) {
     StatInc(thr, StatMop);
     StatInc(thr, kAccessIsWrite ? StatMopWrite : StatMopRead);
     StatInc(thr, (StatType)(StatMop1 + kAccessSizeLog));

   }
 
   MemoryAccessImpl1(thr, addr, kAccessSizeLog, kAccessIsWrite, kIsAtomic,
-      shadow_mem, cur);
+                    vshadow_mem, cur);
 }
 
 // Called by MemoryAccessRange in tsan_rtl_thread.cc

 ALWAYS_INLINE USED
 void MemoryAccessImpl(ThreadState *thr, uptr addr,
     int kAccessSizeLog, bool kAccessIsWrite, bool kIsAtomic,
-    u64 *shadow_mem, Shadow cur) {
-  if (LIKELY(ContainsSameAccess(shadow_mem, cur.raw(),
+    uptr vshadow_mem, Shadow cur) {
+  if (LIKELY(ContainsSameAccess(vshadow_mem, cur.raw(),
       thr->fast_synch_epoch, kAccessIsWrite))) {
     StatInc(thr, StatMop);
     StatInc(thr, kAccessIsWrite ? StatMopWrite : StatMopRead);

   }
 
   MemoryAccessImpl1(thr, addr, kAccessSizeLog, kAccessIsWrite, kIsAtomic,
-      shadow_mem, cur);
+                    vshadow_mem, cur);
 }
 
 static void MemoryRangeSet(ThreadState *thr, uptr pc, uptr addr, uptr size,

   // UnmapOrDie/MmapFixedNoReserve does not work on Windows,
   // so we do it only for C/C++.
   if (kGoMode || size < common_flags()->clear_shadow_mmap_threshold) {
-    u64 *p = (u64*)MemToShadow(addr);
-    CHECK(IsShadowMem((uptr)p));
-    CHECK(IsShadowMem((uptr)(p + size * kShadowCnt / kShadowCell - 1)));
+    uptr vs = MemToVShadow(addr);
+    CHECK(IsVShadowMem(vs));
+    CHECK(IsVShadowMem(vs + (size * kShadowCnt / kShadowCell - 1) * 8));
     // FIXME: may overwrite a part outside the region
     for (uptr i = 0; i < size / kShadowCell * kShadowCnt;) {
-      p[i++] = val;
+      *VShadowToPShadow(vs + i++ * 8) = val;  // TODO(ikosarev): Optimize.
       for (uptr j = 1; j < kShadowCnt; j++)
-        p[i++] = 0;
+        *VShadowToPShadow(vs + i++ * 8) = 0;  // TODO(ikosarev): Optimize.
     }
   } else {
     // The region is big, reset only beginning and end.
     const uptr kPageSize = GetPageSizeCached();
-    u64 *begin = (u64*)MemToShadow(addr);
-    u64 *end = begin + size / kShadowCell * kShadowCnt;
-    u64 *p = begin;
+    uptr begin = MemToVShadow(addr);
+    uptr end = begin + size / kShadowCell * kShadowCnt * 8;
+    uptr vs = begin;
     // Set at least first kPageSize/2 to page boundary.
-    while ((p < begin + kPageSize / kShadowSize / 2) || ((uptr)p % kPageSize)) {
-      *p++ = val;
-      for (uptr j = 1; j < kShadowCnt; j++)
-        *p++ = 0;
+    while (vs < begin + kPageSize / kShadowSize / 2 * 8 ||
+           (vs % kPageSize)) {
+      *VShadowToPShadow(vs) = val;  // TODO(ikosarev): Optimize.
+      vs += 8;
+      for (uptr j = 1; j < kShadowCnt; j++) {
+        *VShadowToPShadow(vs) = 0;  // TODO(ikosarev): Optimize.
+        vs += 8;
+      }
     }
     // Reset middle part.
-    u64 *p1 = p;
-    p = RoundDown(end, kPageSize);
-    UnmapOrDie((void*)p1, (uptr)p - (uptr)p1);
-    MmapFixedNoReserve((uptr)p1, (uptr)p - (uptr)p1);
+    uptr vs1 = vs;
+    vs = RoundDown(end, kPageSize);
+    // TODO(ikosarev): We allocate shadow memory somewhere else.
+
+    // UnmapOrDie((void*)VShadowToPShadow(vs1), vs - vs1);
+    // MmapFixedNoReserve((uptr)VShadowToPShadow(vs1), vs - vs1);
+    ZeroVShadow(vs1, vs - vs1);
+
     // Set the ending.
-    while (p < end) {
-      *p++ = val;
-      for (uptr j = 1; j < kShadowCnt; j++)
-        *p++ = 0;
+    while (vs < end) {
+      *VShadowToPShadow(vs) = val;  // TODO(ikosarev): Optimize.
+      vs += 8;
+      for (uptr j = 1; j < kShadowCnt; j++) {
+        *VShadowToPShadow(vs) = 0;  // TODO(ikosarev): Optimize.
+        vs += 8;
+      }
     }
   }
 }

   // Processing more than 1k (4k of shadow) is expensive,
   // can cause excessive memory consumption (user does not necessary touch
   // the whole range) and most likely unnecessary.
+  DPrintf("MemoryRangeFreed()\n");
   if (size > 1024)
     size = 1024;
   CHECK_EQ(thr->is_freeing, false);
   thr->is_freeing = true;
+  DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
   MemoryAccessRange(thr, pc, addr, size, true);
+  DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
   thr->is_freeing = false;
   if (kCollectHistory) {
     thr->fast_state.IncrementEpoch();
     TraceAddEvent(thr, thr->fast_state, EventTypeMop, pc);
   }
+  DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
   Shadow s(thr->fast_state);
   s.ClearIgnoreBit();
   s.MarkAsFreed();
   s.SetWrite(true);
   s.SetAddr0AndSizeLog(0, 3);
+  DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
   MemoryRangeSet(thr, pc, addr, size, s.raw());
+  DPrintf("%s, line %d.\n", __FILE__, (int)__LINE__);
 }
 
 void MemoryRangeImitateWrite(ThreadState *thr, uptr pc, uptr addr, uptr size) {

lib/tsan/rtl/tsan_rtl_report.cc

 }
 
 void ReportRace(ThreadState *thr) {
+  DPrintf("ReportRace()\n");
+
   CheckNoLocks(thr);
 
   // Symbolizer makes lots of intercepted calls. If we try to process them,

     thr->racy_state[1] = s.raw();
   }
 
-  uptr addr = ShadowToMem((uptr)thr->racy_shadow_addr);
+  uptr addr = VShadowToMem(thr->racy_vshadow_addr);
   uptr addr_min = 0;
   uptr addr_max = 0;
   {

lib/tsan/rtl/tsan_rtl_thread.cc

   if (size == 0)
     return;
 
-  u64 *shadow_mem = (u64*)MemToShadow(addr);
+  uptr vshadow_mem = MemToVShadow(addr);
   DPrintf2("#%d: MemoryAccessRange: @%p %p size=%d is_write=%d\n",
       thr->tid, (void*)pc, (void*)addr,
       (int)size, is_write);

     Printf("Access to non app mem %zx\n", addr + size - 1);
     DCHECK(IsAppMem(addr + size - 1));
   }
-  if (!IsShadowMem((uptr)shadow_mem)) {
-    Printf("Bad shadow addr %p (%zx)\n", shadow_mem, addr);
-    DCHECK(IsShadowMem((uptr)shadow_mem));
+  if (!IsVShadowMem(vshadow_mem)) {
+    Printf("Bad shadow addr %zx (%zx)\n", vshadow_mem, addr);
+    DCHECK(IsVShadowMem(vshadow_mem));
   }
-  if (!IsShadowMem((uptr)(shadow_mem + size * kShadowCnt / 8 - 1))) {
-    Printf("Bad shadow addr %p (%zx)\n",
-               shadow_mem + size * kShadowCnt / 8 - 1, addr + size - 1);
-    DCHECK(IsShadowMem((uptr)(shadow_mem + size * kShadowCnt / 8 - 1)));
+  if (!IsVShadowMem(vshadow_mem + (size * kShadowCnt / 8 - 1) * 8)) {
+    Printf("Bad shadow addr %zx (%zx)\n",
+               shadow_mem + (size * kShadowCnt / 8 - 1) * 8, addr + size - 1);
+    DCHECK(IsVShadowMem(vshadow_mem + (size * kShadowCnt / 8 - 1) * 8));
   }
 #endif
 
   StatInc(thr, StatMopRange);
 
-  if (*shadow_mem == kShadowRodata) {
+  u64 *pshadow_mem = VShadowToPShadow(vshadow_mem);
+  if (*pshadow_mem == kShadowRodata) {
     // Access to .rodata section, no races here.
     // Measurements show that it can be 10-20% of all memory accesses.
     StatInc(thr, StatMopRangeRodata);

     cur.SetWrite(is_write);
     cur.SetAddr0AndSizeLog(addr & (kShadowCell - 1), kAccessSizeLog);
     MemoryAccessImpl(thr, addr, kAccessSizeLog, is_write, false,
-        shadow_mem, cur);
+                     vshadow_mem, cur);
   }
   if (unaligned)
-    shadow_mem += kShadowCnt;
+    vshadow_mem += kShadowCnt * 8;
   // Handle middle part, if any.
   for (; size >= kShadowCell; addr += kShadowCell, size -= kShadowCell) {
     int const kAccessSizeLog = 3;

     cur.SetWrite(is_write);
     cur.SetAddr0AndSizeLog(0, kAccessSizeLog);
     MemoryAccessImpl(thr, addr, kAccessSizeLog, is_write, false,
-        shadow_mem, cur);
-    shadow_mem += kShadowCnt;
+                     vshadow_mem, cur);
+    vshadow_mem += kShadowCnt * 8;
   }
   // Handle ending, if any.
   for (; size; addr++, size--) {

     cur.SetWrite(is_write);
     cur.SetAddr0AndSizeLog(addr & (kShadowCell - 1), kAccessSizeLog);
     MemoryAccessImpl(thr, addr, kAccessSizeLog, is_write, false,
-        shadow_mem, cur);
+                     vshadow_mem, cur);
   }
 }
 

lib/tsan/rtl/tsan_sync.cc

 }
 
 void MetaMap::AllocBlock(ThreadState *thr, uptr pc, uptr p, uptr sz) {
+  DPrintf("Pointer = %p.\n", (void*)p);
   u32 idx = block_alloc_.Alloc(&thr->proc()->block_cache);
   MBlock *b = block_alloc_.Map(idx);
+  DPrintf("Block %p, index %u.\n", (void*)b, (unsigned)idx);
   b->siz = sz;
   b->tid = thr->tid;
   b->stk = CurrentStackId(thr, pc);
-  u32 *meta = MemToMeta(p);
-  DCHECK_EQ(*meta, 0);
-  *meta = idx | kFlagBlock;
+  u32 *pmeta = MemToPMeta(p);
+  DPrintf("pmeta = %p.\n", (void*)pmeta);
+  DCHECK_EQ(*pmeta, 0);
+  *pmeta = idx | kFlagBlock;
 }
 
 uptr MetaMap::FreeBlock(Processor *proc, uptr p) {
+  DPrintf("Pointer = %p.\n", (void*)p);
   MBlock* b = GetBlock(p);
+  DPrintf("Block %p.\n", (void*)b);
   if (b == 0)
     return 0;
   uptr sz = RoundUpTo(b->siz, kMetaShadowCell);

 
 bool MetaMap::FreeRange(Processor *proc, uptr p, uptr sz) {
   bool has_something = false;
-  u32 *meta = MemToMeta(p);
-  u32 *end = MemToMeta(p + sz);
-  if (end == meta)
-    end++;
-  for (; meta < end; meta++) {
-    u32 idx = *meta;
+  uptr vmeta = MemToVMeta(p);
+  uptr vend = MemToVMeta(p + sz);
+  if (vend == vmeta)
+    vend += 4;
+  for (; vmeta < vend; vmeta += 4) {
+    u32 *pmeta = VMetaToPMeta(vmeta);
+    u32 idx = *pmeta;
     if (idx == 0) {
       // Note: don't write to meta in this case -- the block can be huge.
       continue;
     }
-    *meta = 0;
+    *pmeta = 0;
     has_something = true;
     while (idx != 0) {
       if (idx & kFlagBlock) {

   // freed). Note: we can't simply madvise, because we need to leave a zeroed
   // range (otherwise __tsan_java_move can crash if it encounters a left-over
   // meta objects in java heap).
-  uptr metap = (uptr)MemToMeta(p0);
-  uptr metasz = sz0 / kMetaRatio;
-  UnmapOrDie((void*)metap, metasz);
-  MmapFixedNoReserve(metap, metasz);
+  uptr vmeta = MemToVMeta(p0);
+  uptr vmeta_sz = sz0 / kMetaRatio;
+  ZeroVMeta(vmeta, vmeta_sz);
 }
 
 MBlock* MetaMap::GetBlock(uptr p) {
-  u32 *meta = MemToMeta(p);
-  u32 idx = *meta;
+  u32 *pmeta = MemToPMeta(p);
+  DPrintf("pmeta = %p.\n", (void*)pmeta);
+  u32 idx = *pmeta;
+  DPrintf("Index %u.\n", (unsigned)idx);
   for (;;) {
     if (idx == 0)
       return 0;

 
 SyncVar* MetaMap::GetAndLock(ThreadState *thr, uptr pc,
                              uptr addr, bool write_lock, bool create) {
-  u32 *meta = MemToMeta(addr);
-  u32 idx0 = *meta;
+  u32 *pmeta = MemToPMeta(addr);
+  u32 idx0 = *pmeta;
   u32 myidx = 0;
   SyncVar *mys = 0;
   for (;;) {

     }
     if (!create)
       return 0;
-    if (*meta != idx0) {
-      idx0 = *meta;
+    if (*pmeta != idx0) {
+      idx0 = *pmeta;
       continue;
     }
 

       mys->Init(thr, pc, addr, uid);
     }
     mys->next = idx0;
-    if (atomic_compare_exchange_strong((atomic_uint32_t*)meta, &idx0,
+    if (atomic_compare_exchange_strong((atomic_uint32_t*)pmeta, &idx0,
         myidx | kFlagSync, memory_order_release)) {
       if (write_lock)
         mys->mtx.Lock();

   CHECK_NE(src, dst);
   CHECK_NE(sz, 0);
   uptr diff = dst - src;
-  u32 *src_meta = MemToMeta(src);
-  u32 *dst_meta = MemToMeta(dst);
-  u32 *src_meta_end = MemToMeta(src + sz);
-  uptr inc = 1;
+  uptr src_vmeta = MemToVMeta(src);
+  uptr dst_vmeta = MemToVMeta(dst);
+  uptr src_vmeta_end = MemToVMeta(src + sz);
+  uptr inc = 4;
   if (dst > src) {
-    src_meta = MemToMeta(src + sz) - 1;
-    dst_meta = MemToMeta(dst + sz) - 1;
-    src_meta_end = MemToMeta(src) - 1;
-    inc = -1;
+    src_vmeta = MemToVMeta(src + sz) - 4;
+    dst_vmeta = MemToVMeta(dst + sz) - 4;
+    src_vmeta_end = MemToVMeta(src) - 4;
+    inc = -4;
   }
-  for (; src_meta != src_meta_end; src_meta += inc, dst_meta += inc) {
-    CHECK_EQ(*dst_meta, 0);
-    u32 idx = *src_meta;
-    *src_meta = 0;
-    *dst_meta = idx;
+  for (; src_vmeta != src_vmeta_end; src_vmeta += inc, dst_vmeta += inc) {
+    u32 *dst_pmeta = VMetaToPMeta(dst_vmeta);
+    CHECK_EQ(*dst_pmeta, 0);
+    u32 *src_pmeta = VMetaToPMeta(src_vmeta);
+    u32 idx = *src_pmeta;
+    *src_pmeta = 0;
+    *dst_pmeta = idx;
     // Patch the addresses in sync objects.
     while (idx != 0) {
       if (idx & kFlagBlock)

lib/tsan/rtl/tsan_update_shadow_word_inl.h

 do {
   StatInc(thr, StatShadowProcessed);
   const unsigned kAccessSize = 1 << kAccessSizeLog;
-  u64 *sp = &shadow_mem[idx];
-  old = LoadShadow(sp);
+  uptr vs = vshadow_mem + idx * 8;
+  old = LoadVShadow(vs);
   if (old.IsZero()) {
     StatInc(thr, StatShadowZero);
     if (store_word)
-      StoreIfNotYetStored(sp, &store_word);
+      StoreIfNotYetStored(vs, &store_word);
     // The above StoreIfNotYetStored could be done unconditionally
     // and it even shows 4% gain on synthetic benchmarks (r4307).
     break;

     if (Shadow::TidsAreEqual(old, cur)) {
       StatInc(thr, StatShadowSameThread);
       if (old.IsRWWeakerOrEqual(kAccessIsWrite, kIsAtomic))
-        StoreIfNotYetStored(sp, &store_word);
+        StoreIfNotYetStored(vs, &store_word);
       break;
     }
     StatInc(thr, StatShadowAnotherThread);
     if (HappensBefore(old, thr)) {
       if (old.IsRWWeakerOrEqual(kAccessIsWrite, kIsAtomic))
-        StoreIfNotYetStored(sp, &store_word);
+        StoreIfNotYetStored(vs, &store_word);
       break;
     }
     if (old.IsBothReadsOrAtomic(kAccessIsWrite, kIsAtomic))
       break;
+    DPrintf("RACE #1\n");
     goto RACE;
   }
   // Do the memory access intersect?

       break;
     if (HappensBefore(old, thr))
       break;
+    DPrintf("RACE #2\n");
     goto RACE;
   }
   // The accesses do not intersect.

lib/tsan/tests/unit/tsan_shadow_test.cc

   CHECK(IsAppMem((uptr)&stack));
   CHECK(IsAppMem((uptr)heap));
 
-  CHECK(IsShadowMem(MemToShadow((uptr)&global)));
-  CHECK(IsShadowMem(MemToShadow((uptr)&stack)));
-  CHECK(IsShadowMem(MemToShadow((uptr)heap)));
+  CHECK(IsVShadowMem(MemToVShadow((uptr)&global)));
+  CHECK(IsVShadowMem(MemToVShadow((uptr)&stack)));
+  CHECK(IsVShadowMem(MemToVShadow((uptr)heap)));
 }
 
 TEST(Shadow, Celling) {

   u64 aligned_data[4];
   char *data = (char*)aligned_data;
   CHECK_EQ((uptr)data % kShadowSize, 0);
-  uptr s0 = MemToShadow((uptr)&data[0]);
+  uptr s0 = MemToVShadow((uptr)&data[0]);
   CHECK_EQ(s0 % kShadowSize, 0);
   for (unsigned i = 1; i < kShadowCell; i++)
-    CHECK_EQ(s0, MemToShadow((uptr)&data[i]));
+    CHECK_EQ(s0, MemToVShadow((uptr)&data[i]));
   for (unsigned i = kShadowCell; i < 2*kShadowCell; i++)
-    CHECK_EQ(s0 + kShadowSize*kShadowCnt, MemToShadow((uptr)&data[i]));
+    CHECK_EQ(s0 + kShadowSize*kShadowCnt, MemToVShadow((uptr)&data[i]));
   for (unsigned i = 2*kShadowCell; i < 3*kShadowCell; i++)
-    CHECK_EQ(s0 + 2*kShadowSize*kShadowCnt, MemToShadow((uptr)&data[i]));
+    CHECK_EQ(s0 + 2*kShadowSize*kShadowCnt, MemToVShadow((uptr)&data[i]));
 }
 
 }  // namespace __tsan

test/asan/TestCases/Linux/asan-asm-stacktrace-test.cc

-// Check that a stack unwinding algorithm works corretly even with the assembly
-// instrumentation.
-
-// REQUIRES: x86_64-target-arch
-// RUN: %clangxx_asan -g -O1 %s -fno-inline-functions -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -mllvm -asan-instrument-assembly -o %t && not %run %t 2>&1 | FileCheck %s
-// RUN: %clangxx_asan -g -O1 %s -fno-inline-functions -fomit-frame-pointer -momit-leaf-frame-pointer -mllvm -asan-instrument-assembly -o %t && not %run %t 2>&1 | FileCheck %s
-// RUN: %clangxx_asan -g0 -O1 %s -fno-unwind-tables -fno-asynchronous-unwind-tables -fno-exceptions -fno-inline-functions -fomit-frame-pointer -momit-leaf-frame-pointer -mllvm -asan-instrument-assembly -o %t && not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-nounwind
-
-#include <cstddef>
-
-// CHECK: READ of size 4
-// CHECK-NEXT: {{#0 0x[0-9a-fA-F]+ in foo}}
-// CHECK-NEXT: {{#1 0x[0-9a-fA-F]+ in main}}
-
-// CHECK-nounwind: READ of size 4
-// CHECK-nounwind-NEXT: {{#0 0x[0-9a-fA-F]+ in foo}}
-
-__attribute__((noinline)) int foo(size_t n, int *buffer) {
-  int r;
-  __asm__("movl (%[buffer], %[n], 4), %[r]  \n\t"
-          : [r] "=r"(r)
-          : [buffer] "r"(buffer), [n] "r"(n)
-          : "memory");
-  return r;
-}
-
-int main() {
-  const size_t n = 16;
-  int *buffer = new int[n];
-  foo(n, buffer);
-  delete[] buffer;
-  return 0;
-}

test/asan/TestCases/Linux/auto_memory_profile_test.cc

-// Tests heap_profile=1.
-// Printing memory profiling only works in the configuration where we can
-// detect leaks.
-// REQUIRES: leak-detection
-//
-// RUN: %clangxx_asan %s -o %t
-// RUN: %env_asan_opts=heap_profile=1 %run %t 2>&1 | FileCheck %s
-#include <sanitizer/common_interface_defs.h>
-
-#include <stdio.h>
-#include <string.h>
-#include <unistd.h>
-
-char *sink[1000];
-
-int main() {
-
-  for (int i = 0; i < 3; i++) {
-    const size_t  kSize = 13000000;
-    char *x = new char[kSize];
-    memset(x, 0, kSize);
-    sink[i] = x;
-    sleep(1);
-  }
-}
-
-// CHECK: HEAP PROFILE at RSS
-// CHECK: 13000000 byte(s)
-// CHECK: HEAP PROFILE at RSS
-// CHECK: 26000000 byte(s)
-// CHECK: HEAP PROFILE at RSS
-// CHECK: 39000000 byte(s)

test/asan/TestCases/Linux/cuda_test.cc

-// Emulate the behavior of the NVIDIA CUDA driver
-// that mmaps memory inside the asan's shadow gap.
-//
-// REQUIRES: x86_64-target-arch
-//
-// RUN: %clangxx_asan %s -o %t
-// RUN: not %env_asan_opts=protect_shadow_gap=1 %t 2>&1 | FileCheck %s  --check-prefix=CHECK-PROTECT1
-// RUN: not                                     %t 2>&1 | FileCheck %s  --check-prefix=CHECK-PROTECT1
-// RUN: not %env_asan_opts=protect_shadow_gap=0 %t 2>&1 | FileCheck %s  --check-prefix=CHECK-PROTECT0
-#include <assert.h>
-#include <unistd.h>
-#include <sys/mman.h>
-#include <stdint.h>
-
-#include "sanitizer/asan_interface.h"
-
-int main(void) {
-  uintptr_t Base = 0x200000000;
-  uintptr_t Size = 0x1100000000;
-  void *addr =
-      mmap((void *)Base, Size, PROT_READ | PROT_WRITE,
-           MAP_NORESERVE | MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, 0, 0);
-  assert(addr == (void*)Base);
-  // Make sure we can access memory in shadow gap.
-  // W/o protect_shadow_gap=0 we should fail here.
-  for (uintptr_t Addr = Base; Addr < Base + Size; Addr += Size / 100)
-    *(char*)Addr = 1;
-  // CHECK-PROTECT1: AddressSanitizer: SEGV on unknown address 0x0000bfff8000
-
-  // Poison a part of gap's shadow:
-  __asan_poison_memory_region((void*)Base, 4096);
-  // Now we should fail with use-after-poison.
-  *(char*)(Base + 1234) = 1;
-  // CHECK-PROTECT0: AddressSanitizer: use-after-poison on address 0x0002000004d2
-}
-
-

test/asan/TestCases/Linux/nohugepage_test.cc

-// Regression test for
-// https://code.google.com/p/chromium/issues/detail?id=446692
-// where asan consumed too much RAM due to transparent hugetables.
-//
-// RUN: %clangxx_asan -g %s -o %t
-// RUN: %env_asan_opts=no_huge_pages_for_shadow=1 %run %t 2>&1 | FileCheck %s
-// RUN: %run %t 2>&1 | FileCheck %s
-//
-// Would be great to run the test with no_huge_pages_for_shadow=0, but
-// the result will depend on the OS version and settings...
-//
-// REQUIRES: x86_64-target-arch
-//
-// WARNING: this test is very subtle and may nto work on some systems.
-// If this is the case we'll need to futher improve it or disable it.
-#include <assert.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/mman.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <fcntl.h>
-#include <unistd.h>
-#include <errno.h>
-#include <sanitizer/asan_interface.h>
-
-char FileContents[1 << 16];
-
-void FileToString(const char *path) {
-  FileContents[0] = 0;
-  int fd = open(path, 0);
-  if (fd < 0) return;
-  char *p = FileContents;
-  ssize_t size = sizeof(FileContents) - 1;
-  ssize_t res = 0;
-  do {
-    ssize_t got = read (fd, p, size);
-    if (got == 0)
-      break;
-    else if (got > 0)
-      {
-        p += got;
-        res += got;
-        size -= got;
-      }
-    else if (errno != EINTR)
-      break;
-  } while (size > 0 && res < sizeof(FileContents));
-  if (res >= 0)
-    FileContents[res] = 0;
-}
-
-long ReadShadowRss() {
-  const char *path = "/proc/self/smaps";
-  FileToString(path);
-  char *s = strstr(FileContents, "2008fff7000-10007fff8000");
-  if (!s) return 0;
-
-  s = strstr(s, "Rss:");
-  if (!s) return 0;
-  s = s + 4;
-  return atol(s);
-}
-
-const int kAllocSize = 1 << 28;  // 256Mb
-const int kTwoMb = 1 << 21;
-const int kAsanShadowGranularity = 8;
-
-char *x;
-
-__attribute__((no_sanitize_address)) void TouchNoAsan(size_t i) { x[i] = 0; }
-
-int main() {
-  long rss[5];
-  rss[0] = ReadShadowRss();
-  // use mmap directly to avoid asan touching the shadow.
-  x = (char *)mmap(0, kAllocSize, PROT_READ | PROT_WRITE,
-                   MAP_PRIVATE | MAP_ANON, 0, 0);
-  fprintf(stderr, "X: %p-%p\n", x, x + kAllocSize);
-  rss[1] = ReadShadowRss();
-
-  // Touch the allocated region, but not the shadow.
-  for (size_t i = 0; i < kAllocSize; i += kTwoMb * kAsanShadowGranularity)
-    TouchNoAsan(i);
-  rss[2] = ReadShadowRss();
-
-  // Touch the shadow just a bit, in 2Mb*Granularity steps.
-  for (size_t i = 0; i < kAllocSize; i += kTwoMb * kAsanShadowGranularity)
-    __asan_poison_memory_region(x + i, kAsanShadowGranularity);
-  rss[3] = ReadShadowRss();
-
-  // Touch all the shadow.
-  __asan_poison_memory_region(x, kAllocSize);
-  rss[4] = ReadShadowRss();
-
-  // Print the differences.
-  for (int i = 0; i < 4; i++) {
-    assert(rss[i] > 0);
-    assert(rss[i+1] >= rss[i]);
-    long diff = rss[i+1] / rss[i];
-    fprintf(stderr, "RSS CHANGE IS %d => %d: %s (%ld vs %ld)\n", i, i + 1,
-            diff < 10 ? "SMALL" : "LARGE", rss[i], rss[i + 1]);
-  }
-}
-// CHECK: RSS CHANGE IS 2 => 3: SMALL
-// CHECK: RSS CHANGE IS 3 => 4: LARGE

test/asan/TestCases/Linux/swapcontext_annotation.cc

-// Check that ASan plays well with annotated makecontext/swapcontext.
-
-// RUN: %clangxx_asan -std=c++11 -lpthread -O0 %s -o %t && %run %t 2>&1 | FileCheck %s
-// RUN: %clangxx_asan -std=c++11 -lpthread -O1 %s -o %t && %run %t 2>&1 | FileCheck %s
-// RUN: %clangxx_asan -std=c++11 -lpthread -O2 %s -o %t && %run %t 2>&1 | FileCheck %s
-// RUN: %clangxx_asan -std=c++11 -lpthread -O3 %s -o %t && %run %t 2>&1 | FileCheck %s
-// RUN: %clangxx_asan -std=c++11 -lpthread -O0 %s -o %t && %run %t 2>&1 | FileCheck <( seq 60 | xargs -i -- grep LOOPCHECK %s ) --check-prefix LOOPCHECK
-// RUN: %clangxx_asan -std=c++11 -lpthread -O1 %s -o %t && %run %t 2>&1 | FileCheck <( seq 60 | xargs -i -- grep LOOPCHECK %s ) --check-prefix LOOPCHECK
-// RUN: %clangxx_asan -std=c++11 -lpthread -O2 %s -o %t && %run %t 2>&1 | FileCheck <( seq 60 | xargs -i -- grep LOOPCHECK %s ) --check-prefix LOOPCHECK
-// RUN: %clangxx_asan -std=c++11 -lpthread -O3 %s -o %t && %run %t 2>&1 | FileCheck <( seq 60 | xargs -i -- grep LOOPCHECK %s ) --check-prefix LOOPCHECK
-
-//
-// This test is too subtle to try on non-x86 arch for now.
-// REQUIRES: x86-target-arch
-
-#include <pthread.h>
-#include <setjmp.h>
-#include <stdio.h>
-#include <sys/time.h>
-#include <ucontext.h>
-#include <unistd.h>
-
-#include <sanitizer/common_interface_defs.h>
-
-ucontext_t orig_context;
-ucontext_t child_context;
-ucontext_t next_child_context;
-
-char *next_child_stack;
-
-const int kStackSize = 1 << 20;
-
-const void *main_thread_stack;
-size_t main_thread_stacksize;
-
-const void *from_stack;
-size_t from_stacksize;
-
-__attribute__((noinline, noreturn)) void LongJump(jmp_buf env) {
-  longjmp(env, 1);
-  _exit(1);
-}
-
-// Simulate __asan_handle_no_return().
-__attribute__((noinline)) void CallNoReturn() {
-  jmp_buf env;
-  if (setjmp(env) != 0) return;
-
-  LongJump(env);
-  _exit(1);
-}
-
-void NextChild() {
-  CallNoReturn();
-  __sanitizer_finish_switch_fiber(nullptr, &from_stack, &from_stacksize);
-
-  printf("NextChild from: %p %zu\n", from_stack, from_stacksize);
-
-  char x[32] = {0};  // Stack gets poisoned.
-  printf("NextChild: %p\n", x);
-
-  CallNoReturn();
-
-  __sanitizer_start_switch_fiber(nullptr,
-                                 main_thread_stack,
-                                 main_thread_stacksize);
-  CallNoReturn();
-  if (swapcontext(&next_child_context, &orig_context) < 0) {
-    perror("swapcontext");
-    _exit(1);
-  }
-}
-
-void Child(int mode) {
-  CallNoReturn();
-  __sanitizer_finish_switch_fiber(nullptr,
-                                  &main_thread_stack,
-                                  &main_thread_stacksize);
-  char x[32] = {0};  // Stack gets poisoned.
-  printf("Child: %p\n", x);
-  CallNoReturn();
-  // (a) Do nothing, just return to parent function.
-  // (b) Jump into the original function. Stack remains poisoned unless we do
-  //     something.
-  // (c) Jump to another function which will then jump back to the main function
-  if (mode == 0) {
-    __sanitizer_start_switch_fiber(nullptr,
-                                   main_thread_stack,
-                                   main_thread_stacksize);
-    CallNoReturn();
-  } else if (mode == 1) {
-    __sanitizer_start_switch_fiber(nullptr,
-                                   main_thread_stack,
-                                   main_thread_stacksize);
-    CallNoReturn();
-    if (swapcontext(&child_context, &orig_context) < 0) {
-      perror("swapcontext");
-      _exit(1);
-    }
-  } else if (mode == 2) {
-    printf("NextChild stack: %p\n", next_child_stack);
-
-    getcontext(&next_child_context);
-    next_child_context.uc_stack.ss_sp = next_child_stack;
-    next_child_context.uc_stack.ss_size = kStackSize / 2;
-    makecontext(&next_child_context, (void (*)())NextChild, 0);
-    __sanitizer_start_switch_fiber(nullptr,
-                                   next_child_context.uc_stack.ss_sp,
-                                   next_child_context.uc_stack.ss_size);
-    CallNoReturn();
-    if (swapcontext(&child_context, &next_child_context) < 0) {
-      perror("swapcontext");
-      _exit(1);
-    }
-  }
-}
-
-int Run(int arg, int mode, char *child_stack) {
-  printf("Child stack: %p\n", child_stack);
-  // Setup child context.
-  getcontext(&child_context);
-  child_context.uc_stack.ss_sp = child_stack;
-  child_context.uc_stack.ss_size = kStackSize / 2;
-  if (mode == 0) {
-    child_context.uc_link = &orig_context;
-  }
-  makecontext(&child_context, (void (*)())Child, 1, mode);
-  CallNoReturn();
-  void* fake_stack_save;
-  __sanitizer_start_switch_fiber(&fake_stack_save,
-                                 child_context.uc_stack.ss_sp,
-                                 child_context.uc_stack.ss_size);
-  CallNoReturn();
-  if (swapcontext(&orig_context, &child_context) < 0) {
-    perror("swapcontext");
-    _exit(1);
-  }
-  CallNoReturn();
-  __sanitizer_finish_switch_fiber(fake_stack_save,
-                                  &from_stack,
-                                  &from_stacksize);
-  CallNoReturn();
-  printf("Main context from: %p %zu\n", from_stack, from_stacksize);
-
-  // Touch childs's stack to make sure it's unpoisoned.
-  for (int i = 0; i < kStackSize; i++) {
-    child_stack[i] = i;
-  }
-  return child_stack[arg];
-}
-
-void handler(int sig) { CallNoReturn(); }
-
-int main(int argc, char **argv) {
-  // set up a signal that will spam and trigger __asan_handle_no_return at
-  // tricky moments
-  struct sigaction act = {};
-  act.sa_handler = &handler;
-  if (sigaction(SIGPROF, &act, 0)) {
-    perror("sigaction");
-    _exit(1);
-  }
-
-  itimerval t;
-  t.it_interval.tv_sec = 0;
-  t.it_interval.tv_usec = 10;
-  t.it_value = t.it_interval;
-  if (setitimer(ITIMER_PROF, &t, 0)) {
-    perror("setitimer");
-    _exit(1);
-  }
-
-  char *heap = new char[kStackSize + 1];
-  next_child_stack = new char[kStackSize + 1];
-  char stack[kStackSize + 1];
-  // CHECK: WARNING: ASan doesn't fully support makecontext/swapcontext
-  int ret = 0;
-  // CHECK-NOT: ASan is ignoring requested __asan_handle_no_return
-  for (unsigned int i = 0; i < 30; ++i) {
-    ret += Run(argc - 1, 0, stack);
-    // LOOPCHECK: Child stack: [[CHILD_STACK:0x[0-9a-f]*]]
-    // LOOPCHECK: Main context from: [[CHILD_STACK]] 524288
-    ret += Run(argc - 1, 1, stack);
-    // LOOPCHECK: Child stack: [[CHILD_STACK:0x[0-9a-f]*]]
-    // LOOPCHECK: Main context from: [[CHILD_STACK]] 524288
-    ret += Run(argc - 1, 2, stack);
-    // LOOPCHECK: Child stack: [[CHILD_STACK:0x[0-9a-f]*]]
-    // LOOPCHECK: NextChild stack: [[NEXT_CHILD_STACK:0x[0-9a-f]*]]
-    // LOOPCHECK: NextChild from: [[CHILD_STACK]] 524288
-    // LOOPCHECK: Main context from: [[NEXT_CHILD_STACK]] 524288
-    ret += Run(argc - 1, 0, heap);
-    ret += Run(argc - 1, 1, heap);
-    ret += Run(argc - 1, 2, heap);
-    printf("Iteration %d passed\n", i);
-  }
-
-  // CHECK: Test passed
-  printf("Test passed\n");
-
-  delete[] heap;
-  delete[] next_child_stack;
-
-  return ret;
-}

test/asan/TestCases/debug_locate.cc

-// Checks the ASan memory address type debugging API, makes sure it returns
-// the correct memory type for heap, stack, global and shadow addresses and
-// that it correctly finds out which region (and name and size) the address
-// belongs to.
-// RUN: %clangxx_asan -O0 %s -o %t && %run %t 2>&1
-
-#include <assert.h>
-#include <sanitizer/asan_interface.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-int global_var;
-
-int main() {
-  int local_var;
-  char *heap_ptr = (char *)malloc(10);
-
-  char name[100];
-  void *region_address;
-  size_t region_size;
-  const char *type;
-
-  type = __asan_locate_address(&global_var, name, 100,
-                               &region_address, &region_size);
-  assert(0 == strcmp(name, "global_var"));
-  assert(0 == strcmp(type, "global"));
-  assert(region_address == &global_var);
-  assert(region_size == sizeof(global_var));
-
-  type = __asan_locate_address((char *)(&global_var)+1, name, 100,
-                               &region_address, &region_size);
-  assert(0 == strcmp(name, "global_var"));
-  assert(0 == strcmp(type, "global"));
-  assert(region_address == &global_var);
-  assert(region_size == sizeof(global_var));
-
-  type = __asan_locate_address(&local_var, name, 100,
-                               &region_address, &region_size);
-  assert(0 == strcmp(name, "local_var"));
-  assert(0 == strcmp(type, "stack"));
-  assert(region_address == &local_var);
-  assert(region_size == sizeof(local_var));
-
-  type = __asan_locate_address((char *)(&local_var)+1, name, 100,
-                               &region_address, &region_size);
-  assert(0 == strcmp(name, "local_var"));
-  assert(0 == strcmp(type, "stack"));
-  assert(region_address == &local_var);
-  assert(region_size == sizeof(local_var));
-
-  type = __asan_locate_address(heap_ptr, name, 100,
-                               &region_address, &region_size);
-  assert(0 == strcmp(type, "heap"));
-  assert(region_address == heap_ptr);
-  assert(10 == region_size);
-
-  type = __asan_locate_address(heap_ptr+1, name, 100,
-                               &region_address, &region_size);
-  assert(0 == strcmp(type, "heap"));
-  assert(region_address == heap_ptr);
-  assert(10 == region_size);
-
-  size_t shadow_scale;
-  size_t shadow_offset;
-  __asan_get_shadow_mapping(&shadow_scale, &shadow_offset);
-
-  uintptr_t shadow_ptr = (((uintptr_t)heap_ptr) >> shadow_scale)
-                         + shadow_offset;
-  type = __asan_locate_address((void *)shadow_ptr, NULL, 0, NULL, NULL);
-  assert((0 == strcmp(type, "high shadow")) || 0 == strcmp(type, "low shadow"));
-
-  uintptr_t shadow_gap = (shadow_ptr >> shadow_scale) + shadow_offset;
-  type = __asan_locate_address((void *)shadow_gap, NULL, 0, NULL, NULL);
-  assert(0 == strcmp(type, "shadow gap"));
-
-  free(heap_ptr);
-
-  return 0;
-}

test/sanitizer_common/TestCases/Linux/decorate_proc_maps.cc

-// RUN: %clangxx -g %s -o %t
-// RUN: %env_tool_opts=decorate_proc_maps=1 %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-%tool_name
-// REQUIRES: stable-runtime
-#include <errno.h>
-#include <fcntl.h>
-#include <pthread.h>
-#include <stdio.h>
-#include <sys/stat.h>
-#include <sys/types.h>
-#include <unistd.h>
-
-bool CopyFdToFd(int in_fd, int out_fd) {
-  const size_t kBufSize = 0x10000;
-  static char buf[kBufSize];
-  while (true) {
-    ssize_t got = read(in_fd, buf, kBufSize);
-    if (got > 0) {
-      write(out_fd, buf, got);
-    } else if (got == 0) {
-      break;
-    } else if (errno != EAGAIN || errno != EWOULDBLOCK || errno != EINTR) {
-      fprintf(stderr, "error reading file, errno %d\n", errno);
-      return false;
-    }
-  }
-  return true;
-}
-
-void *ThreadFn(void *arg) {
-  (void)arg;
-  int fd = open("/proc/self/maps", O_RDONLY);
-  bool res = CopyFdToFd(fd, 2);
-  close(fd);
-  return (void *)!res;
-}
-
-int main(void) {
-  pthread_t t;
-  void *res;
-  pthread_create(&t, 0, ThreadFn, 0);
-  pthread_join(t, &res);
-  return (int)(size_t)res;
-}
-
-// CHECK-asan: rw-p {{.*}} [low shadow]
-// CHECK-asan: ---p {{.*}} [shadow gap]
-// CHECK-asan: rw-p {{.*}} [high shadow]
-
-// CHECK-msan: ---p {{.*}} [invalid]
-// CHECK-msan: rw-p {{.*}} [shadow{{.*}}]
-// CHECK-msan: ---p {{.*}} [origin{{.*}}]
-
-// CHECK-tsan: rw-p {{.*}} [shadow]
-// CHECK-tsan: rw-p {{.*}} [meta shadow]
-// CHECK-tsan: rw-p {{.*}} [trace 0]
-// CHECK-tsan: rw-p {{.*}} [trace header 0]
-// CHECK-tsan: rw-p {{.*}} [trace 1]
-// CHECK-tsan: rw-p {{.*}} [trace header 1]
-
-// Nothing interesting with standalone LSan.
-// CHECK-lsan: decorate_proc_maps

test/tsan/mmap_large.cc

-// RUN: %clang_tsan -O1 %s -o %t && %run %t 2>&1 | FileCheck %s
-#include <stdint.h>
-#include <stdio.h>
-#include <errno.h>
-#include <sys/mman.h>
-
-#if defined(__FreeBSD__)
-// The MAP_NORESERVE define has been removed in FreeBSD 11.x, and even before
-// that, it was never implemented.  So just define it to zero.
-#undef  MAP_NORESERVE
-#define MAP_NORESERVE 0
-#endif
-
-int main() {
-#ifdef __x86_64__
-  const size_t kLog2Size = 39;
-#elif defined(__mips64) || defined(__aarch64__)
-  const size_t kLog2Size = 32;
-#elif defined(__powerpc64__)
-  const size_t kLog2Size = 39;
-#endif
-  const uintptr_t kLocation = 0x40ULL << kLog2Size;
-  void *p = mmap(
-      reinterpret_cast<void*>(kLocation),
-      1ULL << kLog2Size,
-      PROT_READ|PROT_WRITE,
-      MAP_PRIVATE|MAP_ANON|MAP_NORESERVE,
-      -1, 0);
-  fprintf(stderr, "DONE %p %d\n", p, errno);
-  return p == MAP_FAILED;
-}
-
-// CHECK: DONE

test/tsan/signal_thread.cc

-// RUN: %clangxx_tsan -O1 %s -o %t && %run %t 2>&1 | FileCheck %s
-// UNSUPPORTED: darwin
-#include <pthread.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <signal.h>
-#include <sys/types.h>
-#include <sys/time.h>
-#include <unistd.h>
-#include <errno.h>
-
-volatile int X;
-
-static void handler(int sig) {
-  (void)sig;
-  if (X != 0)
-    printf("bad");
-}
-
-static void* thr(void *p) {
-  return 0;
-}
-
-int main() {
-  struct sigaction act = {};
-  act.sa_handler = &handler;
-  if (sigaction(SIGPROF, &act, 0)) {
-    perror("sigaction");
-    exit(1);
-  }
-
-  itimerval t;
-  t.it_value.tv_sec = 0;
-  t.it_value.tv_usec = 10;
-  t.it_interval = t.it_value;
-  if (setitimer(ITIMER_PROF, &t, 0)) {
-    perror("setitimer");
-    exit(1);
-  }
-
-  for (int i = 0; i < 10000; i++) {
-    pthread_t th;
-    pthread_create(&th, 0, thr, 0);
-    pthread_join(th, 0);
-  }
-
-  fprintf(stderr, "DONE\n");
-  return 0;
-}
-
-// CHECK-NOT: WARNING: ThreadSanitizer:
-// CHECK: DONE
-// CHECK-NOT: WARNING: ThreadSanitizer: