[DFSan] Change shadow and origin memory layouts to match MSan.

[browneee]

Summary

Previously on x86_64:

+--------------------+ 0x800000000000 (top of memory)
| application memory |
+--------------------+ 0x700000008000 (kAppAddr)
|                    |
|       unused       |
|                    |
+--------------------+ 0x300000000000 (kUnusedAddr)
|       origin       |
+--------------------+ 0x200000008000 (kOriginAddr)
|       unused       |
+--------------------+ 0x200000000000
|   shadow memory    |
+--------------------+ 0x100000008000 (kShadowAddr)
|       unused       |
+--------------------+ 0x000000010000
| reserved by kernel |
+--------------------+ 0x000000000000

MEM_TO_SHADOW(mem) = mem & ~0x600000000000
SHADOW_TO_ORIGIN(shadow) = kOriginAddr - kShadowAddr + shadow

Now for x86_64:

+--------------------+ 0x800000000000 (top of memory)
|    application 3   |
+--------------------+ 0x700000000000
|      invalid       |
+--------------------+ 0x610000000000
|      origin 1      |
+--------------------+ 0x600000000000
|    application 2   |
+--------------------+ 0x510000000000
|      shadow 1      |
+--------------------+ 0x500000000000
|      invalid       |
+--------------------+ 0x400000000000
|      origin 3      |
+--------------------+ 0x300000000000
|      shadow 3      |
+--------------------+ 0x200000000000
|      origin 2      |
+--------------------+ 0x110000000000
|      invalid       |
+--------------------+ 0x100000000000
|      shadow 2      |
+--------------------+ 0x010000000000
|    application 1   |
+--------------------+ 0x000000000000

MEM_TO_SHADOW(mem) = mem ^ 0x500000000000
SHADOW_TO_ORIGIN(shadow) = shadow + 0x100000000000

[stephan.yichao.zhao]

compiler-rt/lib/dfsan/dfsan.cpp

168

Based on the recent issue about using a replaced memmove.
It seems safe to always keep this CHECK and update the comments to reflect this.

169

Please remove the commented code.

176

This branch seems redundant with the one below.

322–327

Please update the comments to be consistent with other MEM_IS_SHADOW checks.

[stephan.yichao.zhao]

Thank you for making this work!

compiler-rt/lib/dfsan/dfsan.cpp

861

Please add a comment about that these CheckMemoryLayoutSanity, ... and InitShadow are possible to be shared with MSan (like the TODO in dfsan_platform.h), by moving them and those platform mapping definitions/macros compiler-rt/lib/msan/msan.h to sanitizer_common because it is highly likely that MSan and DFSan always have the same layouts...
Since the change does not branch from MSan's code, w/o comments, it is not easy for others to know they are similar.

892

I suggest moving CheckMemoryRangeAvailability and ProtectMemoryRange to sanitizer_common (if this does not break any sanitizer convention).
They are checking some general things and also some corner cases like "protect address 0...".
If shared, it is more likely that others can help DFSan to improve them.

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

257

Does this suggest LinuxX8664MemoryMapParams? Not sure if there is a workaround to suppress this warning.

461

in mapping application to shadow and origin....

[browneee]

Addressed comments.

clang/docs/DataFlowSanitizerDesign.rst

 Memory layout and label management
 ----------------------------------
 
 The following is the memory layout for Linux/x86\_64:
 
 +---------------+---------------+--------------------+
 |    Start      |    End        |        Use         |
 +===============+===============+====================+
-| 0x700000008000|0x800000000000 | application memory |
+| 0x700000000000|0x800000000000 |    application 3   |
 +---------------+---------------+--------------------+
-| 0x300000000000|0x700000008000 |       unused       |
+| 0x610000000000|0x700000000000 |       unused       |
 +---------------+---------------+--------------------+
-| 0x200000008000|0x300000000000 |       origin       |
+| 0x600000000000|0x610000000000 |      origin 1      |
 +---------------+---------------+--------------------+
-| 0x200000000000|0x200000008000 |       unused       |
+| 0x510000000000|0x600000000000 |    application 2   |
 +---------------+---------------+--------------------+
-| 0x100000008000|0x200000000000 |   shadow memory    |
+| 0x500000000000|0x510000000000 |      shadow 1      |
 +---------------+---------------+--------------------+
-| 0x000000010000|0x100000008000 |       unused       |
+| 0x400000000000|0x500000000000 |       unused       |
 +---------------+---------------+--------------------+
-| 0x000000000000|0x000000010000 | reserved by kernel |
+| 0x300000000000|0x400000000000 |      origin 3      |
++---------------+---------------+--------------------+
+| 0x200000000000|0x300000000000 |      shadow 3      |
++---------------+---------------+--------------------+
+| 0x110000000000|0x200000000000 |      origin 2      |
++---------------+---------------+--------------------+
+| 0x100000000000|0x110000000000 |       unused       |
++---------------+---------------+--------------------+
+| 0x010000000000|0x100000000000 |      shadow 2      |
++---------------+---------------+--------------------+
+| 0x000000000000|0x010000000000 |    application 1   |
 +---------------+---------------+--------------------+
 
 Each byte of application memory corresponds to a single byte of shadow
-memory, which is used to store its taint label. As for LLVM SSA
-registers, we have not found it necessary to associate a label with
-each byte or bit of data, as some other tools do. Instead, labels are
+memory, which is used to store its taint label. We map memory, shadow, and
+origin regions to each other with these masks and offsets:
+
+* shadow_addr = memory_addr ^ 0x500000000000
+
+* origin_addr = shadow_addr + 0x100000000000
+
+As for LLVM SSA registers, we have not found it necessary to associate a label
+with each byte or bit of data, as some other tools do. Instead, labels are
 associated directly with registers.  Loads will result in a union of
 all shadow labels corresponding to bytes loaded, and stores will
 result in a copy of the label of the stored value to the shadow of all
 bytes stored to.
 
 Propagating labels through arguments
 ------------------------------------
 

compiler-rt/lib/dfsan/dfsan.h

 namespace __dfsan {
 
 extern bool dfsan_inited;
 extern bool dfsan_init_is_running;
 
 void initialize_interceptors();
 
 inline dfsan_label *shadow_for(void *ptr) {
-  return (dfsan_label *)(((uptr)ptr) & ShadowMask());
+  return (dfsan_label *)MEM_TO_SHADOW(ptr);
 }
 
 inline const dfsan_label *shadow_for(const void *ptr) {
   return shadow_for(const_cast<void *>(ptr));
 }
 
-inline uptr unaligned_origin_for(uptr ptr) {
-  return OriginAddr() - ShadowAddr() + (ptr & ShadowMask());
-}
+inline uptr unaligned_origin_for(uptr ptr) { return MEM_TO_ORIGIN(ptr); }
 
 inline dfsan_origin *origin_for(void *ptr) {
   auto aligned_addr = unaligned_origin_for(reinterpret_cast<uptr>(ptr)) &
                       ~(sizeof(dfsan_origin) - 1);
   return reinterpret_cast<dfsan_origin *>(aligned_addr);
 }
 
 inline const dfsan_origin *origin_for(const void *ptr) {
   return origin_for(const_cast<void *>(ptr));
 }
 
-inline bool is_shadow_addr_valid(uptr shadow_addr) {
-  return (uptr)shadow_addr >= ShadowAddr() && (uptr)shadow_addr < OriginAddr();
-}
-
-inline bool has_valid_shadow_addr(const void *ptr) {
-  const dfsan_label *ptr_s = shadow_for(ptr);
-  return is_shadow_addr_valid((uptr)ptr_s);
-}
-
-inline bool is_origin_addr_valid(uptr origin_addr) {
-  return (uptr)origin_addr >= OriginAddr() && (uptr)origin_addr < UnusedAddr();
-}
-
-inline bool has_valid_origin_addr(const void *ptr) {
-  const dfsan_origin *ptr_orig = origin_for(ptr);
-  return is_origin_addr_valid((uptr)ptr_orig);
-}
-
 void dfsan_copy_memory(void *dst, const void *src, uptr size);
 
 void dfsan_allocator_init();
 void dfsan_deallocate(void *ptr);
 
 void *dfsan_malloc(uptr size);
 void *dfsan_calloc(uptr nmemb, uptr size);
 void *dfsan_realloc(void *ptr, uptr size);

compiler-rt/lib/dfsan/dfsan.cpp

 extern "C" SANITIZER_WEAK_ATTRIBUTE const int __dfsan_track_origins;
 
 int __dfsan_get_track_origins() {
   return &__dfsan_track_origins ? __dfsan_track_origins : 0;
 }
 
 // On Linux/x86_64, memory is laid out as follows:
 //
-// +--------------------+ 0x800000000000 (top of memory)
-// | application memory |
-// +--------------------+ 0x700000008000 (kAppAddr)
-// |                    |
-// |       unused       |
-// |                    |
-// +--------------------+ 0x300000000000 (kUnusedAddr)
-// |       origin       |
-// +--------------------+ 0x200000008000 (kOriginAddr)
-// |       unused       |
-// +--------------------+ 0x200000000000
-// |   shadow memory    |
-// +--------------------+ 0x100000008000 (kShadowAddr)
-// |       unused       |
-// +--------------------+ 0x000000010000
-// | reserved by kernel |
-// +--------------------+ 0x000000000000
+//  +--------------------+ 0x800000000000 (top of memory)
+//  |    application 3   |
+//  +--------------------+ 0x700000000000
+//  |      invalid       |
+//  +--------------------+ 0x610000000000
+//  |      origin 1      |
+//  +--------------------+ 0x600000000000
+//  |    application 2   |
+//  +--------------------+ 0x510000000000
+//  |      shadow 1      |
+//  +--------------------+ 0x500000000000
+//  |      invalid       |
+//  +--------------------+ 0x400000000000
+//  |      origin 3      |
+//  +--------------------+ 0x300000000000
+//  |      shadow 3      |
+//  +--------------------+ 0x200000000000
+//  |      origin 2      |
+//  +--------------------+ 0x110000000000
+//  |      invalid       |
+//  +--------------------+ 0x100000000000
+//  |      shadow 2      |
+//  +--------------------+ 0x010000000000
+//  |    application 1   |
+//  +--------------------+ 0x000000000000
 //
-// To derive a shadow memory address from an application memory address, bits
-// 45-46 are cleared to bring the address into the range
-// [0x100000008000,0x200000000000).  See the function shadow_for below.
-//
-//
-
+//  MEM_TO_SHADOW(mem) = mem ^ 0x500000000000
+//  SHADOW_TO_ORIGIN(shadow) = shadow + 0x100000000000
 
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE
 dfsan_label __dfsan_union_load(const dfsan_label *ls, uptr n) {
   dfsan_label label = ls[0];
   for (uptr i = 1; i != n; ++i)
     label |= ls[i];
   return label;
 }

 
 static uptr OriginAlignDown(uptr u) { return u & kOriginAlignMask; }
 
 // Return the origin of the first taint byte in the size bytes from the address
 // addr.
 static dfsan_origin GetOriginIfTainted(uptr addr, uptr size) {
   for (uptr i = 0; i < size; ++i, ++addr) {
     dfsan_label *s = shadow_for((void *)addr);
-    if (!is_shadow_addr_valid((uptr)s)) {
-      // The current DFSan memory layout is not always correct. For example,
-      // addresses (0, 0x10000) are mapped to (0, 0x10000). Before fixing the
-      // issue, we ignore such addresses.
-      continue;
-    }
-    if (*s)
+
+    if (*s) {
+      // Validate address region.
+      CHECK(MEM_IS_SHADOW(s));
       return *(dfsan_origin *)origin_for((void *)addr);
     }
+  }
   return 0;
 }
 
 // For platforms which support slow unwinder only, we need to restrict the store
 // context size to 1, basically only storing the current pc, because the slow
 // unwinder which is based on libunwind is not async signal safe and causes
 // random freezes in forking applications as well as in signal handlers.
 // DFSan supports only Linux. So we do not restrict the store context size.
 #define GET_STORE_STACK_TRACE_PC_BP(pc, bp) \
   BufferedStackTrace stack;                 \
   stack.Unwind(pc, bp, nullptr, true, flags().store_context_size);

 }
 
 // Copy or move the origins of the len bytes from src to dst. The source and
 // target memory ranges may or may not be overlapped. This is used by memory
 // transfer operations. stack records the stack trace of the memory transfer
 // operation.
 static void MoveOrigin(const void *dst, const void *src, uptr size,
                        StackTrace *stack) {
-  if (!has_valid_shadow_addr(dst) ||
-      !has_valid_shadow_addr((void *)((uptr)dst + size)) ||
-      !has_valid_shadow_addr(src) ||
-      !has_valid_shadow_addr((void *)((uptr)src + size))) {
+  // Validate address regions.
+  if (!MEM_IS_SHADOW(shadow_for(dst)) ||
+      !MEM_IS_SHADOW(shadow_for((void *)((uptr)dst + size))) ||
+      !MEM_IS_SHADOW(shadow_for(src)) ||
+      !MEM_IS_SHADOW(shadow_for((void *)((uptr)src + size)))) {
+    CHECK(false);
     return;
   }
   // If destination origin range overlaps with source origin range, move
   // origins by copying origins in a reverse order; otherwise, copy origins in
   // a normal order. The orders of origin transfer are consistent with the
   // orders of how memcpy and memmove transfer user data.
   uptr src_aligned_beg = reinterpret_cast<uptr>(src) & ~3UL;
   uptr src_aligned_end = (reinterpret_cast<uptr>(src) + size) & ~3UL;
void dfsan_clear_thread_local_state() {
   if (__dfsan_get_track_origins()) {
     internal_memset(__dfsan_arg_origin_tls, 0, sizeof(__dfsan_arg_origin_tls));
     internal_memset(&__dfsan_retval_origin_tls, 0,
                     sizeof(__dfsan_retval_origin_tls));
   }
 }
 
 extern "C" void dfsan_flush() {
-  if (!MmapFixedSuperNoReserve(ShadowAddr(), UnusedAddr() - ShadowAddr()))
+  const uptr maxVirtualAddress = GetMaxUserVirtualAddress();
+  for (unsigned i = 0; i < kMemoryLayoutSize; ++i) {
+    uptr start = kMemoryLayout[i].start;
+    uptr end = kMemoryLayout[i].end;
+    uptr size = end - start;
+    MappingDesc::Type type = kMemoryLayout[i].type;
+
+    if (type != MappingDesc::SHADOW && type != MappingDesc::ORIGIN)
+      continue;
+
+    // Check if the segment should be mapped based on platform constraints.
+    if (start >= maxVirtualAddress)
+      continue;
+
+    if (!MmapFixedSuperNoReserve(start, size, kMemoryLayout[i].name)) {
+      Printf("FATAL: DataFlowSanitizer: failed to clear memory region\n");
       Die();
     }
+  }
+}
+
+// TODO: CheckMemoryLayoutSanity is based on msan.
+// Consider refactoring these into a shared implementation.
+static void CheckMemoryLayoutSanity() {
+  uptr prev_end = 0;
+  for (unsigned i = 0; i < kMemoryLayoutSize; ++i) {
+    uptr start = kMemoryLayout[i].start;
+    uptr end = kMemoryLayout[i].end;
+    MappingDesc::Type type = kMemoryLayout[i].type;
+    CHECK_LT(start, end);
+    CHECK_EQ(prev_end, start);
+    CHECK(addr_is_type(start, type));
+    CHECK(addr_is_type((start + end) / 2, type));
+    CHECK(addr_is_type(end - 1, type));
+    if (type == MappingDesc::APP) {
+      uptr addr = start;
+      CHECK(MEM_IS_SHADOW(MEM_TO_SHADOW(addr)));
+      CHECK(MEM_IS_ORIGIN(MEM_TO_ORIGIN(addr)));
+      CHECK_EQ(MEM_TO_ORIGIN(addr), SHADOW_TO_ORIGIN(MEM_TO_SHADOW(addr)));
+
+      addr = (start + end) / 2;
+      CHECK(MEM_IS_SHADOW(MEM_TO_SHADOW(addr)));
+      CHECK(MEM_IS_ORIGIN(MEM_TO_ORIGIN(addr)));
+      CHECK_EQ(MEM_TO_ORIGIN(addr), SHADOW_TO_ORIGIN(MEM_TO_SHADOW(addr)));
+
+      addr = end - 1;
+      CHECK(MEM_IS_SHADOW(MEM_TO_SHADOW(addr)));
+      CHECK(MEM_IS_ORIGIN(MEM_TO_ORIGIN(addr)));
+      CHECK_EQ(MEM_TO_ORIGIN(addr), SHADOW_TO_ORIGIN(MEM_TO_SHADOW(addr)));
+    }
+    prev_end = end;
+  }
+}
+
+// TODO: CheckMemoryRangeAvailability is based on msan.
+// Consider refactoring these into a shared implementation.
+static bool CheckMemoryRangeAvailability(uptr beg, uptr size) {
+  if (size > 0) {
+    uptr end = beg + size - 1;
+    if (!MemoryRangeIsAvailable(beg, end)) {
+      Printf("FATAL: Memory range %p - %p is not available.\n", beg, end);
+      return false;
+    }
+  }
+  return true;
+}
+
+// TODO: ProtectMemoryRange is based on msan.
+// Consider refactoring these into a shared implementation.
+static bool ProtectMemoryRange(uptr beg, uptr size, const char *name) {
+  if (size > 0) {
+    void *addr = MmapFixedNoAccess(beg, size, name);
+    if (beg == 0 && addr) {
+      // Depending on the kernel configuration, we may not be able to protect
+      // the page at address zero.
+      uptr gap = 16 * GetPageSizeCached();
+      beg += gap;
+      size -= gap;
+      addr = MmapFixedNoAccess(beg, size, name);
+    }
+    if ((uptr)addr != beg) {
+      uptr end = beg + size - 1;
+      Printf("FATAL: Cannot protect memory range %p - %p (%s).\n", beg, end,
+             name);
+      return false;
+    }
+  }
+  return true;
+}
+
+// TODO: InitShadow is based on msan.
+// Consider refactoring these into a shared implementation.
+bool InitShadow(bool init_origins) {
+  // Let user know mapping parameters first.
+  VPrintf(1, "dfsan_init %p\n", &__dfsan::dfsan_init);
+  for (unsigned i = 0; i < kMemoryLayoutSize; ++i)
+    VPrintf(1, "%s: %zx - %zx\n", kMemoryLayout[i].name, kMemoryLayout[i].start,
+            kMemoryLayout[i].end - 1);
+
+  CheckMemoryLayoutSanity();
+
+  if (!MEM_IS_APP(&__dfsan::dfsan_init)) {
+    Printf("FATAL: Code %p is out of application range. Non-PIE build?\n",
+           (uptr)&__dfsan::dfsan_init);
+    return false;
+  }
+
+  const uptr maxVirtualAddress = GetMaxUserVirtualAddress();
+
+  for (unsigned i = 0; i < kMemoryLayoutSize; ++i) {
+    uptr start = kMemoryLayout[i].start;
+    uptr end = kMemoryLayout[i].end;
+    uptr size = end - start;
+    MappingDesc::Type type = kMemoryLayout[i].type;
+
+    // Check if the segment should be mapped based on platform constraints.
+    if (start >= maxVirtualAddress)
+      continue;
+
+    bool map = type == MappingDesc::SHADOW ||
+               (init_origins && type == MappingDesc::ORIGIN);
+    bool protect = type == MappingDesc::INVALID ||
+                   (!init_origins && type == MappingDesc::ORIGIN);
+    CHECK(!(map && protect));
+    if (!map && !protect)
+      CHECK(type == MappingDesc::APP);
+    if (map) {
+      if (!CheckMemoryRangeAvailability(start, size))
+        return false;
+      if (!MmapFixedSuperNoReserve(start, size, kMemoryLayout[i].name))
+        return false;
+      if (common_flags()->use_madv_dontdump)
+        DontDumpShadowMemory(start, size);
+    }
+    if (protect) {
+      if (!CheckMemoryRangeAvailability(start, size))
+        return false;
+      if (!ProtectMemoryRange(start, size, kMemoryLayout[i].name))
+        return false;
+    }
+  }
+
+  return true;
+}
 
 static void DFsanInit(int argc, char **argv, char **envp) {
   CHECK(!dfsan_init_is_running);
   if (dfsan_inited)
     return;
   dfsan_init_is_running = true;
   SanitizerToolName = "DataflowSanitizer";
 
   AvoidCVE_2016_2143();
 
   InitializeFlags();
 
-  dfsan_flush();
-  if (common_flags()->use_madv_dontdump)
-    DontDumpShadowMemory(ShadowAddr(), UnusedAddr() - ShadowAddr());
+  CheckASLR();
 
-  // Protect the region of memory we don't use, to preserve the one-to-one
-  // mapping from application to shadow memory. But if ASLR is disabled, Linux
-  // will load our executable in the middle of our unused region. This mostly
-  // works so long as the program doesn't use too much memory. We support this
-  // case by disabling memory protection when ASLR is disabled.
-  uptr init_addr = (uptr)&DFsanInit;
-  if (!(init_addr >= UnusedAddr() && init_addr < AppAddr()))
-    MmapFixedNoAccess(UnusedAddr(), AppAddr() - UnusedAddr());
+  InitShadow(__dfsan_get_track_origins());
 
   initialize_interceptors();
 
   // Set up threads
   DFsanTSDInit(DFsanTSDDtor);
 
   dfsan_allocator_init();
 

compiler-rt/lib/dfsan/dfsan_allocator.cpp

struct Metadata {
   uptr requested_size;
 };
 
 struct DFsanMapUnmapCallback {
   void OnMap(uptr p, uptr size) const { dfsan_set_label(0, (void *)p, size); }
   void OnUnmap(uptr p, uptr size) const { dfsan_set_label(0, (void *)p, size); }
 };
 
+static const uptr kAllocatorSpace = 0x700000000000ULL;
 static const uptr kMaxAllowedMallocSize = 8UL << 30;
 
 struct AP64 {  // Allocator64 parameters. Deliberately using a short name.
-  // TODO: DFSan assumes application memory starts from 0x700000008000. For
-  // unknown reason, the sanitizer allocator does not support any start address
-  // between 0x701000000000 and 0x700000008000. After switching to fast8labels
-  // mode, DFSan memory layout will be changed to the same to MSan's. Then we
-  // set the start address to 0x700000000000 as MSan.
-  static const uptr kSpaceBeg = 0x701000000000ULL;
+  static const uptr kSpaceBeg = kAllocatorSpace;
   static const uptr kSpaceSize = 0x40000000000;  // 4T.
   static const uptr kMetadataSize = sizeof(Metadata);
   typedef DefaultSizeClassMap SizeClassMap;
   typedef DFsanMapUnmapCallback MapUnmapCallback;
   static const uptr kFlags = 0;
   using AddressSpaceView = LocalAddressSpaceView;
 };
 

compiler-rt/lib/dfsan/dfsan_platform.h

 //
 // Platform specific information for DFSan.
 //===----------------------------------------------------------------------===//
 
 #ifndef DFSAN_PLATFORM_H
 #define DFSAN_PLATFORM_H
 
 #include "sanitizer_common/sanitizer_common.h"
+#include "sanitizer_common/sanitizer_platform.h"
 
 namespace __dfsan {
 
 using __sanitizer::uptr;
 
-struct Mapping {
-  static const uptr kShadowAddr = 0x100000008000;
-  static const uptr kOriginAddr = 0x200000008000;
-  static const uptr kUnusedAddr = 0x300000000000;
-  static const uptr kAppAddr = 0x700000008000;
-  static const uptr kShadowMask = ~0x600000000000;
-};
+// TODO: The memory mapping code to setup a 1:1 shadow is based on msan.
+// Consider refactoring these into a shared implementation.
 
-enum MappingType {
-  MAPPING_SHADOW_ADDR,
-  MAPPING_ORIGIN_ADDR,
-  MAPPING_UNUSED_ADDR,
-  MAPPING_APP_ADDR,
-  MAPPING_SHADOW_MASK
+struct MappingDesc {
+  uptr start;
+  uptr end;
+  enum Type { INVALID, APP, SHADOW, ORIGIN } type;
+  const char *name;
 };
 
-template<typename Mapping, int Type>
-uptr MappingImpl(void) {
-  switch (Type) {
-    case MAPPING_SHADOW_ADDR: return Mapping::kShadowAddr;
-#if defined(__x86_64__)
-    case MAPPING_ORIGIN_ADDR:
-      return Mapping::kOriginAddr;
-#endif
-    case MAPPING_UNUSED_ADDR:
-      return Mapping::kUnusedAddr;
-    case MAPPING_APP_ADDR: return Mapping::kAppAddr;
-    case MAPPING_SHADOW_MASK: return Mapping::kShadowMask;
-  }
-}
+#if SANITIZER_LINUX && SANITIZER_WORDSIZE == 64
 
-template<int Type>
-uptr MappingArchImpl(void) {
-  return MappingImpl<Mapping, Type>();
-}
+// All of the following configurations are supported.
+// ASLR disabled: main executable and DSOs at 0x555550000000
+// PIE and ASLR: main executable and DSOs at 0x7f0000000000
+// non-PIE: main executable below 0x100000000, DSOs at 0x7f0000000000
+// Heap at 0x700000000000.
+const MappingDesc kMemoryLayout[] = {
+    {0x000000000000ULL, 0x010000000000ULL, MappingDesc::APP, "app-1"},
+    {0x010000000000ULL, 0x100000000000ULL, MappingDesc::SHADOW, "shadow-2"},
+    {0x100000000000ULL, 0x110000000000ULL, MappingDesc::INVALID, "invalid"},
+    {0x110000000000ULL, 0x200000000000ULL, MappingDesc::ORIGIN, "origin-2"},
+    {0x200000000000ULL, 0x300000000000ULL, MappingDesc::SHADOW, "shadow-3"},
+    {0x300000000000ULL, 0x400000000000ULL, MappingDesc::ORIGIN, "origin-3"},
+    {0x400000000000ULL, 0x500000000000ULL, MappingDesc::INVALID, "invalid"},
+    {0x500000000000ULL, 0x510000000000ULL, MappingDesc::SHADOW, "shadow-1"},
+    {0x510000000000ULL, 0x600000000000ULL, MappingDesc::APP, "app-2"},
+    {0x600000000000ULL, 0x610000000000ULL, MappingDesc::ORIGIN, "origin-1"},
+    {0x610000000000ULL, 0x700000000000ULL, MappingDesc::INVALID, "invalid"},
+    {0x700000000000ULL, 0x800000000000ULL, MappingDesc::APP, "app-3"}};
+#  define MEM_TO_SHADOW(mem) (((uptr)(mem)) ^ 0x500000000000ULL)
+#  define SHADOW_TO_ORIGIN(mem) (((uptr)(mem)) + 0x100000000000ULL)
 
-ALWAYS_INLINE
-uptr ShadowAddr() {
-  return MappingArchImpl<MAPPING_SHADOW_ADDR>();
-}
+#else
+#  error "Unsupported platform"
+#endif
 
-ALWAYS_INLINE
-uptr OriginAddr() {
-  return MappingArchImpl<MAPPING_ORIGIN_ADDR>();
-}
+const uptr kMemoryLayoutSize = sizeof(kMemoryLayout) / sizeof(kMemoryLayout[0]);
 
-ALWAYS_INLINE
-uptr UnusedAddr() { return MappingArchImpl<MAPPING_UNUSED_ADDR>(); }
+#define MEM_TO_ORIGIN(mem) (SHADOW_TO_ORIGIN(MEM_TO_SHADOW((mem))))
 
-ALWAYS_INLINE
-uptr AppAddr() {
-  return MappingArchImpl<MAPPING_APP_ADDR>();
+#ifndef __clang__
+__attribute__((optimize("unroll-loops")))
+#endif
+inline bool
+addr_is_type(uptr addr, MappingDesc::Type mapping_type) {
+// It is critical for performance that this loop is unrolled (because then it is
+// simplified into just a few constant comparisons).
+#ifdef __clang__
+#  pragma unroll
+#endif
+  for (unsigned i = 0; i < kMemoryLayoutSize; ++i)
+    if (kMemoryLayout[i].type == mapping_type &&
+        addr >= kMemoryLayout[i].start && addr < kMemoryLayout[i].end)
+      return true;
+  return false;
 }
 
-ALWAYS_INLINE
-uptr ShadowMask() {
-  return MappingArchImpl<MAPPING_SHADOW_MASK>();
-}
+#define MEM_IS_APP(mem) addr_is_type((uptr)(mem), MappingDesc::APP)
+#define MEM_IS_SHADOW(mem) addr_is_type((uptr)(mem), MappingDesc::SHADOW)
+#define MEM_IS_ORIGIN(mem) addr_is_type((uptr)(mem), MappingDesc::ORIGIN)
 
 }  // namespace __dfsan
 
 #endif

compiler-rt/test/dfsan/origin_invalid.c

 // RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 // 
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 int main(int argc, char *argv[]) {
   uint64_t a = 10;
   dfsan_set_label(1, &a, sizeof(a));
-  size_t origin_addr =
-      (((size_t)&a & ~0x600000000000LL + 0x100000000000LL) & ~0x3UL);
+
+  // Manually compute origin address for &a.
+  // See x86 MEM_TO_ORIGIN macro for logic to replicate here.
+  // Alignment is also needed after to MEM_TO_ORIGIN.
+  uint64_t origin_addr =
+      (((uint64_t)&a ^ 0x500000000000ULL) + 0x100000000000ULL) & ~0x3ULL;
+
+  // Take the address we computed, and store 0 in it to mess it up.
   asm("mov %0, %%rax": :"r"(origin_addr));
   asm("movq $0, (%rax)");
   dfsan_print_origin_trace(&a, "invalid");
 }
 
 // CHECK: Taint value 0x1 (at {{.*}}) origin tracking (invalid)
 // CHECK: Taint value 0x1 (at {{.*}}) has invalid origin tracking. This can be a DFSan bug.

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

 /// The analysis is based on automatic propagation of data flow labels (also
 /// known as taint labels) through a program as it performs computation.
 ///
 /// Each byte of application memory is backed by a shadow memory byte. The
 /// shadow byte can represent up to 8 labels. On Linux/x86_64, memory is then
 /// laid out as follows:
 ///
 /// +--------------------+ 0x800000000000 (top of memory)
-/// | application memory |
-/// +--------------------+ 0x700000008000 (kAppAddr)
-/// |                    |
-/// |       unused       |
-/// |                    |
-/// +--------------------+ 0x300000000000 (kUnusedAddr)
-/// |       origin       |
-/// +--------------------+ 0x200000008000 (kOriginAddr)
-/// |       unused       |
+/// |    application 3   |
+/// +--------------------+ 0x700000000000
+/// |      invalid       |
+/// +--------------------+ 0x610000000000
+/// |      origin 1      |
+/// +--------------------+ 0x600000000000
+/// |    application 2   |
+/// +--------------------+ 0x510000000000
+/// |      shadow 1      |
+/// +--------------------+ 0x500000000000
+/// |      invalid       |
+/// +--------------------+ 0x400000000000
+/// |      origin 3      |
+/// +--------------------+ 0x300000000000
+/// |      shadow 3      |
 /// +--------------------+ 0x200000000000
-/// |   shadow memory    |
-/// +--------------------+ 0x100000008000 (kShadowAddr)
-/// |       unused       |
-/// +--------------------+ 0x000000010000
-/// | reserved by kernel |
+/// |      origin 2      |
+/// +--------------------+ 0x110000000000
+/// |      invalid       |
+/// +--------------------+ 0x100000000000
+/// |      shadow 2      |
+/// +--------------------+ 0x010000000000
+/// |    application 1   |
 /// +--------------------+ 0x000000000000
 ///
-///
-/// To derive a shadow memory address from an application memory address, bits
-/// 45-46 are cleared to bring the address into the range
-/// [0x100000008000,0x200000000000). See the function
-/// DataFlowSanitizer::getShadowAddress below.
+/// MEM_TO_SHADOW(mem) = mem ^ 0x500000000000
+/// SHADOW_TO_ORIGIN(shadow) = shadow + 0x100000000000
 ///
 /// For more information, please refer to the design document:
 /// http://clang.llvm.org/docs/DataFlowSanitizerDesign.html
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Transforms/Instrumentation/DataFlowSanitizer.h"
 #include "llvm/ADT/DenseMap.h"
if (StructType *SGType = dyn_cast<StructType>(GType)) {
     if (!SGType->isLiteral())
       return SGType->getName();
   }
   return "<unknown type>";
 }
 
 namespace {
 
+// Memory map parameters used in application-to-shadow address calculation.
+// Offset = (Addr & ~AndMask) ^ XorMask
+// Shadow = ShadowBase + Offset
+// Origin = (OriginBase + Offset) & ~3ULL
+struct MemoryMapParams {
+  uint64_t AndMask;
+  uint64_t XorMask;
+  uint64_t ShadowBase;
+  uint64_t OriginBase;
+};
+
+} // end anonymous namespace
+
+// x86_64 Linux
+// NOLINTNEXTLINE(readability-identifier-naming)
+static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
+    0,              // AndMask (not used)
+    0x500000000000, // XorMask
+    0,              // ShadowBase (not used)
+    0x100000000000, // OriginBase
+};
+
+namespace {
+
 class DFSanABIList {
   std::unique_ptr<SpecialCaseList> SCL;
 
 public:
   DFSanABIList() = default;
 
   void set(std::unique_ptr<SpecialCaseList> List) { SCL = std::move(List); }
 
class DataFlowSanitizer {
   IntegerType *OriginTy;
   PointerType *OriginPtrTy;
   ConstantInt *ZeroOrigin;
   /// The shadow type for all primitive types and vector types.
   IntegerType *PrimitiveShadowTy;
   PointerType *PrimitiveShadowPtrTy;
   IntegerType *IntptrTy;
   ConstantInt *ZeroPrimitiveShadow;
-  ConstantInt *ShadowPtrMask;
-  ConstantInt *ShadowBase;
-  ConstantInt *OriginBase;
   Constant *ArgTLS;
   ArrayType *ArgOriginTLSTy;
   Constant *ArgOriginTLS;
   Constant *RetvalTLS;
   Constant *RetvalOriginTLS;
   FunctionType *DFSanUnionLoadFnTy;
   FunctionType *DFSanLoadLabelAndOriginFnTy;
   FunctionType *DFSanUnimplementedFnTy;
class DataFlowSanitizer {
   FunctionCallee DFSanMaybeStoreOriginFn;
   SmallPtrSet<Value *, 16> DFSanRuntimeFunctions;
   MDNode *ColdCallWeights;
   MDNode *OriginStoreWeights;
   DFSanABIList ABIList;
   DenseMap<Value *, Function *> UnwrappedFnMap;
   AttrBuilder ReadOnlyNoneAttrs;
 
+  /// Memory map parameters used in calculation mapping application addresses
+  /// to shadow addresses and origin addresses.
+  const MemoryMapParams *MapParams;
+
   Value *getShadowOffset(Value *Addr, IRBuilder<> &IRB);
   Value *getShadowAddress(Value *Addr, Instruction *Pos);
   Value *getShadowAddress(Value *Addr, Instruction *Pos, Value *ShadowOffset);
   std::pair<Value *, Value *>
   getShadowOriginAddress(Value *Addr, Align InstAlignment, Instruction *Pos);
   bool isInstrumented(const Function *F);
   bool isInstrumented(const GlobalAlias *GA);
   FunctionType *getArgsFunctionType(FunctionType *T);
   FunctionType *getTrampolineFunctionType(FunctionType *T);
   TransformedFunction getCustomFunctionType(FunctionType *T);
   InstrumentedABI getInstrumentedABI();
   WrapperKind getWrapperKind(Function *F);
   void addGlobalNameSuffix(GlobalValue *GV);
   Function *buildWrapperFunction(Function *F, StringRef NewFName,
                                  GlobalValue::LinkageTypes NewFLink,
                                  FunctionType *NewFT);
   Constant *getOrBuildTrampolineFunction(FunctionType *FT, StringRef FName);
   void initializeCallbackFunctions(Module &M);
   void initializeRuntimeFunctions(Module &M);
   void injectMetadataGlobals(Module &M);
-  bool init(Module &M);
+  bool initializeModule(Module &M);
 
   /// Advances \p OriginAddr to point to the next 32-bit origin and then loads
   /// from it. Returns the origin's loaded value.
   Value *loadNextOrigin(Instruction *Pos, Align OriginAlign,
                         Value **OriginAddr);
 
   /// Returns whether the given load byte size is amenable to inlined
   /// optimization patterns.
Type *DataFlowSanitizer::getShadowTy(Type *OrigTy) {
   }
   return PrimitiveShadowTy;
 }
 
 Type *DataFlowSanitizer::getShadowTy(Value *V) {
   return getShadowTy(V->getType());
 }
 
-bool DataFlowSanitizer::init(Module &M) {
+bool DataFlowSanitizer::initializeModule(Module &M) {
   Triple TargetTriple(M.getTargetTriple());
   const DataLayout &DL = M.getDataLayout();
 
   if (TargetTriple.getOS() != Triple::Linux)
     report_fatal_error("unsupported operating system");
   if (TargetTriple.getArch() != Triple::x86_64)
     report_fatal_error("unsupported architecture");
+  MapParams = &Linux_X86_64_MemoryMapParams;
 
   Mod = &M;
   Ctx = &M.getContext();
   Int8Ptr = Type::getInt8PtrTy(*Ctx);
   OriginTy = IntegerType::get(*Ctx, OriginWidthBits);
   OriginPtrTy = PointerType::getUnqual(OriginTy);
   PrimitiveShadowTy = IntegerType::get(*Ctx, ShadowWidthBits);
   PrimitiveShadowPtrTy = PointerType::getUnqual(PrimitiveShadowTy);
   IntptrTy = DL.getIntPtrType(*Ctx);
   ZeroPrimitiveShadow = ConstantInt::getSigned(PrimitiveShadowTy, 0);
   ZeroOrigin = ConstantInt::getSigned(OriginTy, 0);
 
-  ShadowBase = ConstantInt::get(IntptrTy, 0x100000008000LL);
-  OriginBase = ConstantInt::get(IntptrTy, 0x200000008000LL);
-  ShadowPtrMask = ConstantInt::getSigned(IntptrTy, ~0x600000000000LL);
   Type *DFSanUnionLoadArgs[2] = {PrimitiveShadowPtrTy, IntptrTy};
   DFSanUnionLoadFnTy = FunctionType::get(PrimitiveShadowTy, DFSanUnionLoadArgs,
                                          /*isVarArg=*/false);
   Type *DFSanLoadLabelAndOriginArgs[2] = {Int8Ptr, IntptrTy};
   DFSanLoadLabelAndOriginFnTy =
       FunctionType::get(IntegerType::get(*Ctx, 64), DFSanLoadLabelAndOriginArgs,
                         /*isVarArg=*/false);
   DFSanUnimplementedFnTy = FunctionType::get(
(void)Mod->getOrInsertGlobal("__dfsan_shadow_width_bytes", IntTy, [&] {
     return new GlobalVariable(M, IntTy, /*isConstant=*/true,
                               GlobalValue::WeakODRLinkage,
                               ConstantInt::get(IntTy, ShadowWidthBytes),
                               "__dfsan_shadow_width_bytes");
   });
 }
 
 bool DataFlowSanitizer::runImpl(Module &M) {
-  init(M);
+  initializeModule(M);
 
   if (ABIList.isIn(M, "skip"))
     return false;
 
   const unsigned InitialGlobalSize = M.global_size();
   const unsigned InitialModuleSize = M.size();
 
   bool Changed = false;

 
 void DFSanFunction::setShadow(Instruction *I, Value *Shadow) {
   assert(!ValShadowMap.count(I));
   assert(DFS.shouldTrackFieldsAndIndices() ||
          Shadow->getType() == DFS.PrimitiveShadowTy);
   ValShadowMap[I] = Shadow;
 }
 
+/// Compute the integer shadow offset that corresponds to a given
+/// application address.
+///
+/// Offset = (Addr & ~AndMask) ^ XorMask
 Value *DataFlowSanitizer::getShadowOffset(Value *Addr, IRBuilder<> &IRB) {
-  // Returns Addr & shadow_mask
   assert(Addr != RetvalTLS && "Reinstrumenting?");
-  return IRB.CreateAnd(IRB.CreatePtrToInt(Addr, IntptrTy),
-                       IRB.CreatePtrToInt(ShadowPtrMask, IntptrTy));
+  Value *OffsetLong = IRB.CreatePointerCast(Addr, IntptrTy);
+
+  uint64_t AndMask = MapParams->AndMask;
+  if (AndMask)
+    OffsetLong =
+        IRB.CreateAnd(OffsetLong, ConstantInt::get(IntptrTy, ~AndMask));
+
+  uint64_t XorMask = MapParams->XorMask;
+  if (XorMask)
+    OffsetLong = IRB.CreateXor(OffsetLong, ConstantInt::get(IntptrTy, XorMask));
+  return OffsetLong;
 }
 
 std::pair<Value *, Value *>
 DataFlowSanitizer::getShadowOriginAddress(Value *Addr, Align InstAlignment,
                                           Instruction *Pos) {
   // Returns ((Addr & shadow_mask) + origin_base - shadow_base) & ~4UL
   IRBuilder<> IRB(Pos);
   Value *ShadowOffset = getShadowOffset(Addr, IRB);
-  Value *ShadowPtr = getShadowAddress(Addr, Pos, ShadowOffset);
+  Value *ShadowLong = ShadowOffset;
+  uint64_t ShadowBase = MapParams->ShadowBase;
+  if (ShadowBase != 0) {
+    ShadowLong =
+        IRB.CreateAdd(ShadowLong, ConstantInt::get(IntptrTy, ShadowBase));
+  }
+  IntegerType *ShadowTy = IntegerType::get(*Ctx, ShadowWidthBits);
+  Value *ShadowPtr =
+      IRB.CreateIntToPtr(ShadowLong, PointerType::get(ShadowTy, 0));
   Value *OriginPtr = nullptr;
   if (shouldTrackOrigins()) {
-    static Value *OriginByShadowOffset = ConstantInt::get(
-        IntptrTy, OriginBase->getZExtValue() - ShadowBase->getZExtValue());
-
-    Value *OriginLong = IRB.CreateAdd(ShadowOffset, OriginByShadowOffset);
+    Value *OriginLong = ShadowOffset;
+    uint64_t OriginBase = MapParams->OriginBase;
+    if (OriginBase != 0)
+      OriginLong =
+          IRB.CreateAdd(OriginLong, ConstantInt::get(IntptrTy, OriginBase));
     const Align Alignment = llvm::assumeAligned(InstAlignment.value());
     // When alignment is >= 4, Addr must be aligned to 4, otherwise it is UB.
     // So Mask is unnecessary.
     if (Alignment < MinOriginAlignment) {
       uint64_t Mask = MinOriginAlignment.value() - 1;
       OriginLong = IRB.CreateAnd(OriginLong, ConstantInt::get(IntptrTy, ~Mask));
     }
     OriginPtr = IRB.CreateIntToPtr(OriginLong, OriginPtrTy);
   }
-  return {ShadowPtr, OriginPtr};
+  return std::make_pair(ShadowPtr, OriginPtr);
 }
 
 Value *DataFlowSanitizer::getShadowAddress(Value *Addr, Instruction *Pos,
                                            Value *ShadowOffset) {
   IRBuilder<> IRB(Pos);
   return IRB.CreateIntToPtr(ShadowOffset, PrimitiveShadowPtrTy);
 }
 
 Value *DataFlowSanitizer::getShadowAddress(Value *Addr, Instruction *Pos) {
-  // Returns (Addr & shadow_mask)
   IRBuilder<> IRB(Pos);
   Value *ShadowOffset = getShadowOffset(Addr, IRB);
   return getShadowAddress(Addr, Pos, ShadowOffset);
 }
 
 Value *DFSanFunction::combineShadowsThenConvert(Type *T, Value *V1, Value *V2,
                                                 Instruction *Pos) {
   Value *PrimitiveValue = combineShadows(V1, V2, Pos);

llvm/test/Instrumentation/DataFlowSanitizer/atomics.ll

 define i32 @AtomicRmwXchg(i32* %p, i32 %x) {
 entry:
   ; COMM: atomicrmw xchg: store clean shadow/origin, return clean shadow/origin
 
   ; CHECK-LABEL:       @AtomicRmwXchg.dfsan
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:        %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK-NEXT:        atomicrmw xchg i32* %p, i32 %x seq_cst
   ; CHECK-NEXT:        store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ; CHECK_ORIGIN-NEXT: store i32 0, i32* @__dfsan_retval_origin_tls, align 4
   ; CHECK-NEXT:        ret i32
 
   %0 = atomicrmw xchg i32* %p, i32 %x seq_cst
   ret i32 %0
 }
 
 define i32 @AtomicRmwMax(i32* %p, i32 %x) {
   ; COMM: atomicrmw max: exactly the same as above
 
   ; CHECK-LABEL:       @AtomicRmwMax.dfsan
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:        %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK-NEXT:        atomicrmw max i32* %p, i32 %x seq_cst
   ; CHECK-NEXT:        store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ; CHECK_ORIGIN-NEXT: store i32 0, i32* @__dfsan_retval_origin_tls, align 4
   ; CHECK-NEXT:        ret i32
 
 entry:
   %0 = atomicrmw max i32* %p, i32 %x seq_cst
   ret i32 %0
 }
 
 
 define i32 @Cmpxchg(i32* %p, i32 %a, i32 %b) {
   ; COMM: cmpxchg: store clean shadow/origin, return clean shadow/origin
 
   ; CHECK-LABEL:       @Cmpxchg.dfsan
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:        %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK-NEXT:        %pair = cmpxchg i32* %p, i32 %a, i32 %b seq_cst seq_cst
   ; CHECK:             store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ; CHECK_ORIGIN-NEXT: store i32 0, i32* @__dfsan_retval_origin_tls, align 4
   ; CHECK-NEXT:        ret i32
 
 entry:
   %pair = cmpxchg i32* %p, i32 %a, i32 %b seq_cst seq_cst
   %0 = extractvalue { i32, i1 } %pair, 0
   ret i32 %0
 }
 
 
 define i32 @CmpxchgMonotonic(i32* %p, i32 %a, i32 %b) {
   ; COMM: relaxed cmpxchg: bump up to "release monotonic"
 
   ; CHECK-LABEL:       @CmpxchgMonotonic.dfsan
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:        %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK-NEXT:        %pair = cmpxchg i32* %p, i32 %a, i32 %b release monotonic
   ; CHECK:             store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ; CHECK_ORIGIN-NEXT: store i32 0, i32* @__dfsan_retval_origin_tls, align 4
   ; CHECK-NEXT:        ret i32
 
 entry:

 define void @AtomicStore(i32* %p, i32 %x) {
   ; COMM: atomic store: store clean shadow value before app value
 
   ; CHECK-LABEL:       @AtomicStore.dfsan
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK_ORIGIN-NOT:  35184372088832
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:        %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK:             store atomic i32 %x, i32* %p seq_cst, align 16
   ; CHECK:             ret void
 
 entry:
   store atomic i32 %x, i32* %p seq_cst, align 16
   ret void
 }
 
 define void @AtomicStoreRelease(i32* %p, i32 %x) {
   ; COMM: atomic store: store clean shadow value before app value
 
   ; CHECK-LABEL:       @AtomicStoreRelease.dfsan
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK_ORIGIN-NOT:  35184372088832
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:        %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK:             store atomic i32 %x, i32* %p release, align 16
   ; CHECK:             ret void
 
 entry:
   store atomic i32 %x, i32* %p release, align 16
   ret void
 }
 
 define void @AtomicStoreMonotonic(i32* %p, i32 %x) {
   ; COMM: atomic store monotonic: bumped up to store release
 
   ; CHECK-LABEL:       @AtomicStoreMonotonic.dfsan
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK_ORIGIN-NOT:  35184372088832
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:        %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK:             store atomic i32 %x, i32* %p release, align 16
   ; CHECK:             ret void
 
 entry:
   store atomic i32 %x, i32* %p monotonic, align 16
   ret void
 }
 
 define void @AtomicStoreUnordered(i32* %p, i32 %x) {
   ; COMM: atomic store unordered: bumped up to store release
 
   ; CHECK-LABEL: @AtomicStoreUnordered.dfsan
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK_ORIGIN-NOT:  35184372088832
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:        %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK:             store atomic i32 %x, i32* %p release, align 16
   ; CHECK:             ret void
 
 entry:
   store atomic i32 %x, i32* %p unordered, align 16
   ret void
 }

llvm/test/Instrumentation/DataFlowSanitizer/basic.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,CHECK_NO_ORIGIN -DSHADOW_MASK=-105553116266497   --dump-input-context=100
-; RUN: opt < %s -dfsan -dfsan-track-origins=1  -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN -DSHADOW_MASK=-105553116266497  --dump-input-context=100
+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,CHECK_NO_ORIGIN -DSHADOW_XOR_MASK=87960930222080 --dump-input-context=100
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN -DSHADOW_XOR_MASK=87960930222080 --dump-input-context=100
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [100 x i64]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [100 x i64]
 ; CHECK: @__dfsan_arg_origin_tls = external thread_local(initialexec) global [200 x i32]
 ; CHECK: @__dfsan_retval_origin_tls = external thread_local(initialexec) global i32
 ; CHECK_NO_ORIGIN: @__dfsan_track_origins = weak_odr constant i32 0
 ; CHECK_ORIGIN: @__dfsan_track_origins = weak_odr constant i32 1
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define i8 @load(i8* %p) {
   ; CHECK-LABEL: define i8 @load.dfsan
-  ; CHECK: and i64 {{.*}}, [[SHADOW_MASK]]
+  ; CHECK: xor i64 {{.*}}, [[SHADOW_XOR_MASK]]
   ; CHECK: ret i8 %a
   %a = load i8, i8* %p
   ret i8 %a
 }
 
 define void @store(i8* %p) {
   ; CHECK-LABEL: define void @store.dfsan
-  ; CHECK: and i64 {{.*}}, [[SHADOW_MASK]]
+  ; CHECK: xor i64 {{.*}}, [[SHADOW_XOR_MASK]]
   ; CHECK: ret void
   store i8 0, i8* %p
   ret void
 }
 
 ; CHECK: declare void @__dfsan_load_callback(i[[#SBITS]], i8*)
 ; CHECK: declare void @__dfsan_store_callback(i[[#SBITS]], i8*)
 ; CHECK: declare void @__dfsan_mem_transfer_callback(i[[#SBITS]]*, i64)

llvm/test/Instrumentation/DataFlowSanitizer/load.ll

define {} @load0({}* %p) {
   %a = load {}, {}* %p
   ret {} %a
 }
 
 define i8 @load8(i8* %p) {
   ; CHECK-LABEL:           @load8.dfsan
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i8* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#SHADOW:]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR]]
   ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
   ; CHECK-NEXT:            %a = load i8, i8* %p
   ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            ret i8 %a
 
   %a = load i8, i8* %p
   ret i8 %a
 }
 
 define i16 @load16(i16* %p) {
   ; CHECK-LABEL:           @load16.dfsan
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i16* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#SHADOW_PTR+1]] = getelementptr i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR]], i64 1
   ; CHECK-NEXT:            %[[#SHADOW:]]  = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR]]
   ; CHECK-NEXT:            %[[#SHADOW+1]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR+1]]
 
   ; CHECK-NEXT:            %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#SHADOW+1]]
   ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
   ; CHECK-NEXT:            %a = load i16, i16* %p
   ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            ret i16 %a
 
   %a = load i16, i16* %p
   ret i16 %a
 }
 
 define i32 @load32(i32* %p) {
   ; CHECK-LABEL:           @load32.dfsan
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#WSBITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i[[#WSBITS]], i[[#WSBITS]]* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i[[#WSBITS]] %[[#WIDE_SHADOW]], [[#mul(SBITS,2)]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i[[#WSBITS]] %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i[[#WSBITS]] %[[#WIDE_SHADOW]], [[#SBITS]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i[[#WSBITS]] %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
   ; CHECK-NEXT:            %[[#SHADOW:]] = trunc i[[#WSBITS]] %[[#WIDE_SHADOW]] to i[[#SBITS]]
   ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
   ; CHECK-NEXT:            %a = load i32, i32* %p, align 4
   ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            ret i32 %a
 
   %a = load i32, i32* %p
   ret i32 %a
 }
 
 define i64 @load64(i64* %p) {
   ; CHECK-LABEL:           @load64.dfsan
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i64* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i64, i64* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 8
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
   ; CHECK-NEXT:            %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
   ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
   ; CHECK-NEXT:            %a = load i64, i64* %p, align 8
   ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            ret i64 %a
 
   %a = load i64, i64* %p
   ret i64 %a
 }
 
 define i128 @load128(i128* %p) {
   ; CHECK-LABEL:           @load128.dfsan
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i128* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i64, i64* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR2:]] = getelementptr i64, i64* %[[#WIDE_SHADOW_PTR]], i64 1
   ; CHECK-NEXT:            %[[#WIDE_SHADOW2:]] = load i64, i64* %[[#WIDE_SHADOW_PTR2]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW2]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
define i128 @load128(i128* %p) {
   ret i128 %a
 }
 
 
 define i17 @load17(i17* %p) {
   ; CHECK-LABEL:           @load17.dfsan
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i17* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#SHADOW:]] = call zeroext i8 @__dfsan_union_load(i[[#SBITS]]* %[[#SHADOW_PTR]], i64 3)
   ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
   ; CHECK-NEXT:            %a = load i17, i17* %p
   ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            ret i17 %a
 
   %a = load i17, i17* %p
   ret i17 %a

llvm/test/Instrumentation/DataFlowSanitizer/origin_load.ll

define i16 @load_non_escaped_alloca() {
   %p = alloca i16
   %a = load i16, i16* %p
   ret i16 %a
 }
 
 define i16* @load_escaped_alloca() {
   ; CHECK-LABEL:  @load_escaped_alloca.dfsan
   ; CHECK:        %[[#INTP:]] = ptrtoint i16* %p to i64
-  ; CHECK-NEXT:   %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:   %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
-  ; CHECK-NEXT:   %[[#ORIGIN_OFFSET:]] = add i64 %[[#INTP+1]], [[#%.10d,ORIGIN_MASK:]]
+  ; CHECK-NEXT:   %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:   %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
+  ; CHECK-NEXT:   %[[#ORIGIN_OFFSET:]] = add i64 %[[#SHADOW_OFFSET]], [[#%.10d,ORIGIN_BASE:]]
   ; CHECK-NEXT:   %[[#ORIGIN_ADDR:]] = and i64 %[[#ORIGIN_OFFSET]], -4
   ; CHECK-NEXT:   %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:   {{%.*}} = load i32, i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:   %[[#SHADOW_PTR1:]] = getelementptr i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR0]], i64 1
   ; CHECK-NEXT:   %[[#SHADOW:]]  = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR0]], align [[#SBYTES]]
   ; CHECK-NEXT:   %[[#SHADOW+1]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR1]], align [[#SBYTES]]
   ; CHECK-NEXT:   {{%.*}} = or i[[#SBITS]] %[[#SHADOW]], %[[#SHADOW+1]]
   ; CHECK-NEXT:   %a = load i16, i16* %p, align 2

 
 define i1 @load1(i1* %p) {
   ; CHECK-LABEL:             @load1.dfsan
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
 
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i1* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
-  ; CHECK-NEXT:            %[[#ORIGIN_OFFSET:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#ORIGIN_OFFSET:]] = add i64 %[[#SHADOW_OFFSET]], [[#ORIGIN_BASE]]
   ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = and i64 %[[#ORIGIN_OFFSET]], -4
   ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#AO:]] = load i32, i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:            %[[#AS:]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR]], align [[#SBYTES]]
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#AS:]] = or i[[#SBITS]] %[[#AS]], %[[#PS]]
   ; COMBINE_LOAD_PTR-NEXT: %[[#NZ:]] = icmp ne i[[#SBITS]] %[[#PS]], 0
   ; COMBINE_LOAD_PTR-NEXT: %[[#AO:]] = select i1 %[[#NZ]], i32 %[[#PO]], i32 %[[#AO]]
 
   ; CHECK-NEXT:            %a = load i1, i1* %p, align 1
   ; CHECK-NEXT:            store i[[#SBITS]] %[[#AS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            store i32 %[[#AO]], i32* @__dfsan_retval_origin_tls, align 4
 
   %a = load i1, i1* %p
   ret i1 %a
 }
 
 define i16 @load16(i1 %i, i16* %p) {
   ; CHECK-LABEL: @load16.dfsan
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 1), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
 
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i16* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
-  ; CHECK-NEXT:            %[[#ORIGIN_OFFSET:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#ORIGIN_OFFSET:]] = add i64 %[[#SHADOW_OFFSET]], [[#ORIGIN_BASE]]
   ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = and i64 %[[#ORIGIN_OFFSET]], -4
   ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#AO:]] = load i32, i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:            %[[#SHADOW_PTR1:]] = getelementptr i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR0]], i64 1
   ; CHECK-NEXT:            %[[#SHADOW:]]  = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR0]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#SHADOW+1]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR1]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#AS:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#SHADOW+1]]
 

 
 define i32 @load32(i32* %p) {
   ; CHECK-LABEL: @load32.dfsan
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
 
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i32* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
-  ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#SHADOW_OFFSET]], [[#ORIGIN_BASE]]
   ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#AO:]] = load i32, i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#WSBITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i[[#WSBITS]], i[[#WSBITS]]* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW+1]] = lshr i[[#WSBITS]] %[[#WIDE_SHADOW]], [[#mul(SBITS,2)]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW+2]] = or i[[#WSBITS]] %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW+1]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW+3]] = lshr i[[#WSBITS]] %[[#WIDE_SHADOW+2]], [[#SBITS]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW+4]] = or i[[#WSBITS]] %[[#WIDE_SHADOW+2]], %[[#WIDE_SHADOW+3]]

 
 define i64 @load64(i64* %p) {
   ; CHECK-LABEL: @load64.dfsan
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
 
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i64* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
-  ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#SHADOW_OFFSET]], [[#ORIGIN_BASE]]
   ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#ORIGIN:]] = load i32, i32* %[[#ORIGIN_PTR]], align 8
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i64, i64* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_LO:]] = shl i64 %[[#WIDE_SHADOW]], 32
   ; CHECK-NEXT:            %[[#ORIGIN2_PTR:]] = getelementptr i32, i32* %[[#ORIGIN_PTR]], i64 1
   ; CHECK-NEXT:            %[[#ORIGIN2:]] = load i32, i32* %[[#ORIGIN2_PTR]], align 8
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32

 
 define i128 @load128(i128* %p) {
   ; CHECK-LABEL: @load128.dfsan
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
 
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i128* %p to i64
-  ; CHECK-NEXT:            %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
-  ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#SHADOW_ADDR]], [[#ORIGIN_MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#SHADOW_OFFSET]], [[#ORIGIN_BASE]]
   ; CHECK-NEXT:            %[[#ORIGIN1_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#ORIGIN1:]] = load i32, i32* %[[#ORIGIN1_PTR]], align 8
   ; CHECK-NEXT:            %[[#WIDE_SHADOW1_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW1:]] = load i64, i64* %[[#WIDE_SHADOW1_PTR]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW1_LO:]] = shl i64 %[[#WIDE_SHADOW1]], 32
   ; CHECK-NEXT:            %[[#ORIGIN2_PTR:]] = getelementptr i32, i32* %[[#ORIGIN1_PTR]], i64 1
   ; CHECK-NEXT:            %[[#ORIGIN2:]] = load i32, i32* %[[#ORIGIN2_PTR]], align 8
   ; CHECK-NEXT:            %[[#WIDE_SHADOW2_PTR:]] = getelementptr i64, i64* %[[#WIDE_SHADOW1_PTR]], i64 1

llvm/test/Instrumentation/DataFlowSanitizer/origin_store.ll

define void @store_zero_to_escaped_alloca() {
   ret void
 }
 
 define void @store_nonzero_to_escaped_alloca(i16 %a) {
   ; CHECK-LABEL:  @store_nonzero_to_escaped_alloca.dfsan
   ; CHECK-NEXT:   %[[#AO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; CHECK-NEXT:   %[[#AS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK:        %[[#INTP:]] = ptrtoint i16* %p to i64
-  ; CHECK-NEXT:   %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK-NEXT:   %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
-  ; CHECK-NEXT:   %[[#ORIGIN_OFFSET:]] = add i64 %[[#INTP+1]], [[#%.10d,ORIGIN_MASK:]]
+  ; CHECK-NEXT:   %[[#SHADOW_OFFSET:]] = xor i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:   %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_OFFSET]] to i[[#SBITS]]*
+  ; CHECK-NEXT:   %[[#ORIGIN_OFFSET:]] = add i64 %[[#SHADOW_OFFSET]], [[#%.10d,ORIGIN_BASE:]]
   ; CHECK-NEXT:   %[[#ORIGIN_ADDR:]] = and i64 %[[#ORIGIN_OFFSET]], -4
   ; CHECK-NEXT:   %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK:        %_dfscmp = icmp ne i[[#SBITS]] %[[#AS]], 0
   ; CHECK-NEXT:   br i1 %_dfscmp, label %[[L1:.*]], label %[[L2:.*]],
   ; CHECK:       [[L1]]:
   ; CHECK-NEXT:   %[[#NO:]] = call i32 @__dfsan_chain_origin(i32 %[[#AO]])
   ; CHECK-NEXT:   store i32 %[[#NO]], i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:   br label %[[L2]]

llvm/test/Instrumentation/DataFlowSanitizer/store.ll

 }
 
 define void @store8(i8 %v, i8* %p) {
   ; CHECK-LABEL:       @store8.dfsan
   ; CHECK:             load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
   ; COMBINE_PTR_LABEL: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
   ; COMBINE_PTR_LABEL: or i[[#SBITS]]
   ; CHECK:             ptrtoint i8* {{.*}} i64
-  ; CHECK-NEXT:        and i64
+  ; CHECK-NEXT:        xor i64
   ; CHECK-NEXT:        inttoptr i64 {{.*}} i[[#SBITS]]*
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        store i8 %v, i8* %p
   ; CHECK-NEXT:        ret void
 
   store i8 %v, i8* %p
   ret void
 }
 
 define void @store16(i16 %v, i16* %p) {
   ; CHECK-LABEL:       @store16.dfsan
   ; CHECK:             load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
   ; COMBINE_PTR_LABEL: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
   ; COMBINE_PTR_LABEL: or i[[#SBITS]]
   ; CHECK:             ptrtoint i16* {{.*}} i64
-  ; CHECK-NEXT:        and i64
+  ; CHECK-NEXT:        xor i64
   ; CHECK-NEXT:        inttoptr i64 {{.*}} i[[#SBITS]]*
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        store i16 %v, i16* %p
   ; CHECK-NEXT:        ret void
 
   store i16 %v, i16* %p
   ret void
 }
 
 define void @store32(i32 %v, i32* %p) {
   ; CHECK-LABEL:       @store32.dfsan
   ; CHECK:             load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
   ; COMBINE_PTR_LABEL: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
   ; COMBINE_PTR_LABEL: or i[[#SBITS]]
   ; CHECK:             ptrtoint i32* {{.*}} i64
-  ; CHECK-NEXT:        and i64
+  ; CHECK-NEXT:        xor i64
   ; CHECK-NEXT:        inttoptr i64 {{.*}} i[[#SBITS]]*
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        store i32 %v, i32* %p
   ; CHECK-NEXT:        ret void
 
   store i32 %v, i32* %p
   ret void
 }
 
 define void @store64(i64 %v, i64* %p) {
   ; CHECK-LABEL:       @store64.dfsan
   ; CHECK:             load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
   ; COMBINE_PTR_LABEL: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
   ; COMBINE_PTR_LABEL: or i[[#SBITS]]
   ; CHECK:             ptrtoint i64* {{.*}} i64
-  ; CHECK-NEXT:        and i64
+  ; CHECK-NEXT:        xor i64
   ; CHECK-NEXT:        inttoptr i64 {{.*}} i[[#SBITS]]*
   ; CHECK-COUNT-8:     insertelement {{.*}} i[[#SBITS]]
   ; CHECK-NEXT:        bitcast i[[#SBITS]]* {{.*}} <8 x i[[#SBITS]]>*
   ; CHECK-NEXT:        getelementptr <8 x i[[#SBITS]]>
   ; CHECK-NEXT:        store <8 x i[[#SBITS]]>
   ; CHECK-NEXT:        store i64 %v, i64* %p
   ; CHECK-NEXT:        ret void