[dfsan] Add full fast8 support

[gbalats]

Summary

Complete support for fast8:

• amend shadow size and mapping in runtime
• remove fast16 mode and -dfsan-fast-16-labels flag
• remove legacy mode and make fast8 mode the default
• remove dfsan-fast-8-labels flag
• remove functions in dfsan interface only applicable to legacy
• remove legacy-related instrumentation code and tests
• update documentation.

[stephan.yichao.zhao]

The failed test cases from x64 debian > libFuzzer.libFuzzer::* seem related. They still use -dfsan-fast-16-labels.

[stephan.yichao.zhao]

clang/docs/DataFlowSanitizer.rst

172–178

If we swap assert(ij_label == 3) with the 3 dfsan_has_label, the two equivalent blocks are close to each other.

clang/docs/DataFlowSanitizerDesign.rst

60

integer

65

the labels, and each OR is in O(1).

68

picking

74

memory layout

99

, and

100

the label of a stored value

compiler-rt/lib/dfsan/dfsan.cpp

209

After removing legacy mode. if our code does not check l1 != l2 in IR, the comments can be updated.

[stephan.yichao.zhao]

How did we fix that alignment error from compiler-rt/test/dfsan/origin_ldst.c?

[gbalats]

Update docs to resolve review comments.

[gbalats]

clang/docs/DataFlowSanitizer.rst

172–178

I think this way is better to demonstrate the differences:

• the first block exposes dfsan internals (the integer representation) and makes explicit statements about label values
• the second block uses dfsan_has_label to abstract the internals, not exposing the integer representation (or the fact that labels are OR'd).

clang/docs/DataFlowSanitizerDesign.rst

60

Done.

65

Not sure how this changes the meaning, since the two labels would need one OR instruction.

68

Done.

74

Done.

99

Done

100

Done.

compiler-rt/lib/dfsan/dfsan.cpp

209

Done.

[gbalats]

Update dfsan-related tests under compiler-rt/test/fuzzer/.

[gbalats]

Update interceptors.c test after allocator patch.

[browneee]

llvm/test/Instrumentation/DataFlowSanitizer/external_mask.ll

6–9

Fix whitespace

llvm/test/Instrumentation/DataFlowSanitizer/load.ll

99

Fix whitespace

[gbalats]

Remove whitespace in empty lines.

[gbalats]

llvm/test/Instrumentation/DataFlowSanitizer/external_mask.ll

6–9

Acknowledged. Not a tab; just how phabricator indicates whitespace changes.

[twoh]

@gbalats @stephan.yichao.zhao Hello sorry for the late comment but I wonder what was the reason behind this change (changing taint label representation from 16-bit to 8-bit-fast only). Do we have any discussion thread from llvm-dev regarding this? Thanks!

[browneee]

In D103745#3094831, @twoh wrote:

@gbalats @stephan.yichao.zhao Hello sorry for the late comment but I wonder what was the reason behind this change (changing taint label representation from 16-bit to 8-bit-fast only). Do we have any discussion thread from llvm-dev regarding this? Thanks!

Hi twoh,

The reasons are:

• simplify code (smaller codesize overhead)
• share more code and shadow/origin memory layout with MSan (which also has a 1:1 shadow)
• reduce memory overhead

The only disadvantage is you need 2x executions to cover the same number of taint labels, but we already need do multiple executions.

There was no discussion thread, but we discussed with @kcc internally.

[twoh]

@browneee Thanks for the reply! IIUC, with non-fast mode and 16-bit shadow data, it could support 2^16 labels with a single run, so the coverage reduction is 2^16 -> 8. Do I miss something?

[browneee]

Sorry, I was thinking/describing fast-16 to fast-8.

The same reasons apply for legacy mode (2^16 labels) to fast-8.

• simplify code (smaller codesize overhead)
• share more code and shadow/origin memory layout with MSan (which also has a 1:1 shadow)
• reduce memory overhead

In this case it also reduces runtime overhead.

Our experience was legacy mode did not scale to large programs. The choice we saw was do multiple runs with fast-8, or be unable to run with legacy.
Using fast-8 we can successfully run dfsan with large programs.

Having said this, we also fixed many other issues (e.g. memory leaks) to make fast-8 scale. So our experience comparing legacy and fast-8 mode is not quite fair.

[twoh]

@browneee Thanks! Makes sense. Is there a chance that you have an idea of the ballpark overhead of the legacy mode? I'm curious if we track N-labels, the total CPU time is generally much higher with "legacy-mode single-run with N labels" than "fast-8 mode (N/8) runs". I think there are still reasons to prefer multiple runs of fast-8 mode over the legacy mode even when the total CPU times are similar, but I wonder how bad is the overhead of expensive union operations performed by the legacy mode.

clang/docs/DataFlowSanitizer.rst

.. code-block:: none
   # memcpy needs to copy the shadow from the source to the destination region.
   # This is done in a custom function.
   fun:memcpy=uninstrumented
   fun:memcpy=custom
 
 Example
 =======
 
+DataFlowSanitizer supports up to 8 labels, to achieve low CPU and code
+size overhead. Base labels are simply 8-bit unsigned integers that are
+powers of 2 (i.e. 1, 2, 4, 8, ..., 128), and union labels are created
+by ORing base labels.
+
 The following program demonstrates label propagation by checking that
 the correct labels are propagated.
 
 .. code-block:: c++
 
   #include <sanitizer/dfsan_interface.h>
   #include <assert.h>
 
   int main(void) {
-    int i = 1;
-    dfsan_label i_label = dfsan_create_label("i", 0);
-    dfsan_set_label(i_label, &i, sizeof(i));
-
-    int j = 2;
-    dfsan_label j_label = dfsan_create_label("j", 0);
-    dfsan_set_label(j_label, &j, sizeof(j));
-
-    int k = 3;
-    dfsan_label k_label = dfsan_create_label("k", 0);
-    dfsan_set_label(k_label, &k, sizeof(k));
-
-    dfsan_label ij_label = dfsan_get_label(i + j);
-    assert(dfsan_has_label(ij_label, i_label));
-    assert(dfsan_has_label(ij_label, j_label));
-    assert(!dfsan_has_label(ij_label, k_label));
-
-    dfsan_label ijk_label = dfsan_get_label(i + j + k);
-    assert(dfsan_has_label(ijk_label, i_label));
-    assert(dfsan_has_label(ijk_label, j_label));
-    assert(dfsan_has_label(ijk_label, k_label));
-
-    return 0;
-  }
-
-fast16labels mode
-=================
-
-If you need 16 or fewer labels, you can use fast16labels instrumentation for
-less CPU and code size overhead.  To use fast16labels instrumentation, you'll
-need to specify `-fsanitize=dataflow -mllvm -dfsan-fast-16-labels` in your
-compile and link commands and use a modified API for creating and managing
-labels.
-
-In fast16labels mode, base labels are simply 16-bit unsigned integers that are
-powers of 2 (i.e. 1, 2, 4, 8, ..., 32768), and union labels are created by ORing
-base labels.  In this mode DFSan does not manage any label metadata, so the
-functions `dfsan_create_label`, `dfsan_union`, `dfsan_get_label_info`,
-`dfsan_has_label`, `dfsan_has_label_with_desc`, `dfsan_get_label_count`, and
-`dfsan_dump_labels` are unsupported.  Instead of using them, the user should
-maintain any necessary metadata about base labels themselves.
-
-For example:
-
-.. code-block:: c++
-
-  #include <sanitizer/dfsan_interface.h>
-  #include <assert.h>
-
-  int main(void) {
     int i = 100;
     int j = 200;
     int k = 300;
     dfsan_label i_label = 1;
     dfsan_label j_label = 2;
     dfsan_label k_label = 4;
     dfsan_set_label(i_label, &i, sizeof(i));
     dfsan_set_label(j_label, &j, sizeof(j));
     dfsan_set_label(k_label, &k, sizeof(k));
 
     dfsan_label ij_label = dfsan_get_label(i + j);
 
     assert(ij_label & i_label);  // ij_label has i_label
     assert(ij_label & j_label);  // ij_label has j_label
     assert(!(ij_label & k_label));  // ij_label doesn't have k_label
     assert(ij_label == 3);  // Verifies all of the above
 
+    // Or, equivalently:
+    assert(dfsan_has_label(ij_label, i_label));
+    assert(dfsan_has_label(ij_label, j_label));
+    assert(!dfsan_has_label(ij_label, k_label));
+
     dfsan_label ijk_label = dfsan_get_label(i + j + k);
 
     assert(ijk_label & i_label);  // ijk_label has i_label
     assert(ijk_label & j_label);  // ijk_label has j_label
     assert(ijk_label & k_label);  // ijk_label has k_label
     assert(ijk_label == 7);  // Verifies all of the above
 
+    // Or, equivalently:
+    assert(dfsan_has_label(ijk_label, i_label));
+    assert(dfsan_has_label(ijk_label, j_label));
+    assert(dfsan_has_label(ijk_label, k_label));
+
     return 0;
   }
 
 Current status
 ==============
 
 DataFlowSanitizer is a work in progress, currently under development for
 x86\_64 Linux.
 
 Design
 ======
 
 Please refer to the :doc:`design document<DataFlowSanitizerDesign>`.

clang/docs/DataFlowSanitizerDesign.rst

 DataFlowSanitizer Design Document
 =================================
 
 This document sets out the design for DataFlowSanitizer, a general
 dynamic data flow analysis.  Unlike other Sanitizer tools, this tool is
 not designed to detect a specific class of bugs on its own. Instead,
 it provides a generic dynamic data flow analysis framework to be used
 by clients to help detect application-specific issues within their
 own code.
 
 DataFlowSanitizer is a program instrumentation which can associate
 a number of taint labels with any data stored in any memory region
 accessible by the program. The analysis is dynamic, which means that
 it operates on a running program, and tracks how the labels propagate
-through that program. The tool shall support a large (>100) number
-of labels, such that programs which operate on large numbers of data
-items may be analysed with each data item being tracked separately.
+through that program.
 
 Use Cases
 ---------
 
 This instrumentation can be used as a tool to help monitor how data
 flows from a program's inputs (sources) to its outputs (sinks).
 This has applications from a privacy/security perspective in that
 one can audit how a sensitive data item is used within a program and
 ensure it isn't exiting the program anywhere it shouldn't be.
 
 Interface
 ---------
 
-A number of functions are provided which will create taint labels,
-attach labels to memory regions and extract the set of labels
-associated with a specific memory region. These functions are declared
-in the header file ``sanitizer/dfsan_interface.h``.
+A number of functions are provided which will attach taint labels to
+memory regions and extract the set of labels associated with a
+specific memory region. These functions are declared in the header
+file ``sanitizer/dfsan_interface.h``.
 
 .. code-block:: c
 
-  /// Creates and returns a base label with the given description and user data.
-  dfsan_label dfsan_create_label(const char *desc, void *userdata);
-
   /// Sets the label for each address in [addr,addr+size) to \c label.
   void dfsan_set_label(dfsan_label label, void *addr, size_t size);
 
   /// Sets the label for each address in [addr,addr+size) to the union of the
   /// current label for that address and \c label.
   void dfsan_add_label(dfsan_label label, void *addr, size_t size);
 
   /// Retrieves the label associated with the given data.
   ///
   /// The type of 'data' is arbitrary.  The function accepts a value of any type,
   /// which can be truncated or extended (implicitly or explicitly) as necessary.
   /// The truncation/extension operations will preserve the label of the original
   /// value.
   dfsan_label dfsan_get_label(long data);
 
-  /// Retrieves a pointer to the dfsan_label_info struct for the given label.
-  const struct dfsan_label_info *dfsan_get_label_info(dfsan_label label);
-
   /// Returns whether the given label label contains the label elem.
   int dfsan_has_label(dfsan_label label, dfsan_label elem);
 
-  /// If the given label label contains a label with the description desc, returns
-  /// that label, else returns 0.
-  dfsan_label dfsan_has_label_with_desc(dfsan_label label, const char *desc);
+  /// Computes the union of \c l1 and \c l2, resulting in a union label.
+  dfsan_label dfsan_union(dfsan_label l1, dfsan_label l2);
 
 Taint label representation
 --------------------------
 
-As stated above, the tool must track a large number of taint
-labels. This poses an implementation challenge, as most multiple-label
-tainting systems assign one label per bit to shadow storage, and
-union taint labels using a bitwise or operation. This will not scale
-to clients which use hundreds or thousands of taint labels, as the
-label union operation becomes O(n) in the number of supported labels,
-and data associated with it will quickly dominate the live variable
-set, causing register spills and hampering performance.
-
-Instead, a low overhead approach is proposed which is best-case O(log\
-:sub:`2` n) during execution. The underlying assumption is that
-the required space of label unions is sparse, which is a reasonable
-assumption to make given that we are optimizing for the case where
-applications mostly copy data from one place to another, without often
-invoking the need for an actual union operation. The representation
-of a taint label is a 16-bit integer, and new labels are allocated
-sequentially from a pool. The label identifier 0 is special, and means
-that the data item is unlabelled.
-
-When a label union operation is requested at a join point (any
-arithmetic or logical operation with two or more operands, such as
-addition), the code checks whether a union is required, whether the
-same union has been requested before, and whether one union label
-subsumes the other. If so, it returns the previously allocated union
-label. If not, it allocates a new union label from the same pool used
-for new labels.
-
-Specifically, the instrumentation pass will insert code like this
-to decide the union label ``lu`` for a pair of labels ``l1``
-and ``l2``:
-
-.. code-block:: c
-
-  if (l1 == l2)
-    lu = l1;
-  else
-    lu = __dfsan_union(l1, l2);
-
-The equality comparison is outlined, to provide an early exit in
-the common cases where the program is processing unlabelled data, or
-where the two data items have the same label.  ``__dfsan_union`` is
-a runtime library function which performs all other union computation.
-
-Further optimizations are possible, for example if ``l1`` is known
-at compile time to be zero (e.g. it is derived from a constant),
-``l2`` can be used for ``lu``, and vice versa.
+We use an 8-bit unsigned integers for the representation of a
+label. The label identifier 0 is special, and means that the data item
+is unlabelled. This is optimizing for low CPU and code size overhead
+of the instrumentation. When a label union operation is requested at a
+join point (any arithmetic or logical operation with two or more
+operands, such as addition), we can simply OR the two labels in O(1).
+
+Users are responsible for managing the 8 integer labels (i.e., keeping
+track of what labels they have used so far, pick one that is yet
+unused, etc).
 
 Memory layout and label management
 ----------------------------------
 
 The following is the current memory layout for Linux/x86\_64:
 
 +---------------+---------------+--------------------+
 |    Start      |    End        |        Use         |
 +===============+===============+====================+
 | 0x700000008000|0x800000000000 | application memory |
 +---------------+---------------+--------------------+
-| 0x200200000000|0x700000008000 |       unused       |
+| 0x300000000000|0x700000008000 |       unused       |
++---------------+---------------+--------------------+
+| 0x200000008000|0x300000000000 |       origin       |
++---------------+---------------+--------------------+
+| 0x200000000000|0x200000008000 |       unused       |
 +---------------+---------------+--------------------+
-| 0x200000000000|0x200200000000 |    union table     |
+| 0x100000008000|0x200000000000 |   shadow memory    |
 +---------------+---------------+--------------------+
-| 0x000000010000|0x200000000000 |   shadow memory    |
+| 0x000000010000|0x100000008000 |       unused       |
 +---------------+---------------+--------------------+
 | 0x000000000000|0x000000010000 | reserved by kernel |
 +---------------+---------------+--------------------+
 
-Each byte of application memory corresponds to two bytes of shadow
-memory, which are used to store its taint label. As for LLVM SSA
+Each byte of application memory corresponds to a single byte of shadow
+memory, which is used to store its taint label. As for LLVM SSA
 registers, we have not found it necessary to associate a label with
 each byte or bit of data, as some other tools do. Instead, labels are
 associated directly with registers.  Loads will result in a union of
-all shadow labels corresponding to bytes loaded (which most of the
-time will be short circuited by the initial comparison) and stores will
-result in a copy of the label to the shadow of all bytes stored to.
+all shadow labels corresponding to bytes loaded and stores will result
+in a copy of the label to the shadow of all bytes stored to.
 
 Propagating labels through arguments
 ------------------------------------
 
 In order to propagate labels through function arguments and return values,
 DataFlowSanitizer changes the ABI of each function in the translation unit.
 There are currently two supported ABIs:
 

compiler-rt/include/sanitizer/dfsan_interface.h

 #include <stddef.h>
 #include <stdint.h>
 #include <sanitizer/common_interface_defs.h>
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-typedef uint16_t dfsan_label;
+typedef uint8_t dfsan_label;
 typedef uint32_t dfsan_origin;
 
-/// Stores information associated with a specific label identifier.  A label
-/// may be a base label created using dfsan_create_label, with associated
-/// text description and user data, or an automatically created union label,
-/// which represents the union of two label identifiers (which may themselves
-/// be base or union labels).
-struct dfsan_label_info {
-  // Fields for union labels, set to 0 for base labels.
-  dfsan_label l1;
-  dfsan_label l2;
-
-  // Fields for base labels.
-  const char *desc;
-  void *userdata;
-};
-
 /// Signature of the callback argument to dfsan_set_write_callback().
 typedef void (*dfsan_write_callback_t)(int fd, const void *buf, size_t count);
 
-/// Computes the union of \c l1 and \c l2, possibly creating a union label in
-/// the process.
+/// Computes the union of \c l1 and \c l2, resulting in a union label.
 dfsan_label dfsan_union(dfsan_label l1, dfsan_label l2);
 
-/// Creates and returns a base label with the given description and user data.
-dfsan_label dfsan_create_label(const char *desc, void *userdata);
-
 /// Sets the label for each address in [addr,addr+size) to \c label.
 void dfsan_set_label(dfsan_label label, void *addr, size_t size);
 
 /// Sets the label for each address in [addr,addr+size) to the union of the
 /// current label for that address and \c label.
 void dfsan_add_label(dfsan_label label, void *addr, size_t size);
 
 /// Retrieves the label associated with the given data.
 ///
 /// The type of 'data' is arbitrary.  The function accepts a value of any type,
 /// which can be truncated or extended (implicitly or explicitly) as necessary.
 /// The truncation/extension operations will preserve the label of the original
 /// value.
 dfsan_label dfsan_get_label(long data);
 
 /// Retrieves the immediate origin associated with the given data. The returned
 /// origin may point to another origin.
 ///
 /// The type of 'data' is arbitrary.
 dfsan_origin dfsan_get_origin(long data);
 
 /// Retrieves the label associated with the data at the given address.
 dfsan_label dfsan_read_label(const void *addr, size_t size);
 
-/// Retrieves a pointer to the dfsan_label_info struct for the given label.
-const struct dfsan_label_info *dfsan_get_label_info(dfsan_label label);
-
 /// Returns whether the given label label contains the label elem.
 int dfsan_has_label(dfsan_label label, dfsan_label elem);
 
-/// If the given label label contains a label with the description desc, returns
-/// that label, else returns 0.
-dfsan_label dfsan_has_label_with_desc(dfsan_label label, const char *desc);
-
-/// Returns the number of labels allocated.
-size_t dfsan_get_label_count(void);
-
 /// Flushes the DFSan shadow, i.e. forgets about all labels currently associated
 /// with the application memory.  Use this call to start over the taint tracking
 /// within the same process.
 ///
 /// Note: If another thread is working with tainted data during the flush, that
 /// taint could still be written to shadow after the flush.
 void dfsan_flush(void);
 
 /// Sets a callback to be invoked on calls to write().  The callback is invoked
 /// before the write is done.  The write is not guaranteed to succeed when the
 /// callback executes.  Pass in NULL to remove any callback.
 void dfsan_set_write_callback(dfsan_write_callback_t labeled_write_callback);
 
-/// Writes the labels currently used by the program to the given file
-/// descriptor. The lines of the output have the following format:
-///
-/// <label> <parent label 1> <parent label 2> <label description if any>
-void dfsan_dump_labels(int fd);
-
 /// Interceptor hooks.
 /// Whenever a dfsan's custom function is called the corresponding
 /// hook is called it non-zero. The hooks should be defined by the user.
 /// The primary use case is taint-guided fuzzing, where the fuzzer
 /// needs to see the parameters of the function and the labels.
 /// FIXME: implement more hooks.
 void dfsan_weak_hook_memcmp(void *caller_pc, const void *s1, const void *s2,
                             size_t n, dfsan_label s1_label,

compiler-rt/lib/dfsan/dfsan.h

 
 #ifndef DFSAN_H
 #define DFSAN_H
 
 #include "sanitizer_common/sanitizer_internal_defs.h"
 
 #include "dfsan_platform.h"
 
-using __sanitizer::u16;
 using __sanitizer::u32;
+using __sanitizer::u8;
 using __sanitizer::uptr;
 
 // Copy declarations from public sanitizer/dfsan_interface.h header here.
-typedef u16 dfsan_label;
+typedef u8 dfsan_label;
 typedef u32 dfsan_origin;
 
-struct dfsan_label_info {
-  dfsan_label l1;
-  dfsan_label l2;
-  const char *desc;
-  void *userdata;
-};
-
 extern "C" {
 void dfsan_add_label(dfsan_label label, void *addr, uptr size);
 void dfsan_set_label(dfsan_label label, void *addr, uptr size);
 dfsan_label dfsan_read_label(const void *addr, uptr size);
 dfsan_label dfsan_union(dfsan_label l1, dfsan_label l2);
 // Zero out [offset, offset+size) from __dfsan_arg_tls.
 void dfsan_clear_arg_tls(uptr offset, uptr size);
 // Zero out the TLS storage.

 namespace __dfsan {
 
 extern bool dfsan_inited;
 extern bool dfsan_init_is_running;
 
 void InitializeInterceptors();
 
 inline dfsan_label *shadow_for(void *ptr) {
-  return (dfsan_label *) ((((uptr) ptr) & ShadowMask()) << 1);
+  return (dfsan_label *)(((uptr)ptr) & ShadowMask());
 }
 
 inline const dfsan_label *shadow_for(const void *ptr) {
   return shadow_for(const_cast<void *>(ptr));
 }
 
 inline uptr unaligned_origin_for(uptr ptr) {
-  return OriginAddr() + (ptr & ShadowMask());
+  return OriginAddr() - ShadowAddr() + (ptr & ShadowMask());
 }
 
 inline dfsan_origin *origin_for(void *ptr) {
   auto aligned_addr = unaligned_origin_for(reinterpret_cast<uptr>(ptr)) &
                       ~(sizeof(dfsan_origin) - 1);
   return reinterpret_cast<dfsan_origin *>(aligned_addr);
 }
 
 inline const dfsan_origin *origin_for(const void *ptr) {
   return origin_for(const_cast<void *>(ptr));
 }
 
 inline bool is_shadow_addr_valid(uptr shadow_addr) {
   return (uptr)shadow_addr >= ShadowAddr() && (uptr)shadow_addr < OriginAddr();
 }
 
 inline bool has_valid_shadow_addr(const void *ptr) {
   const dfsan_label *ptr_s = shadow_for(ptr);
   return is_shadow_addr_valid((uptr)ptr_s);
 }
 
+inline bool is_origin_addr_valid(uptr origin_addr) {
+  return (uptr)origin_addr >= OriginAddr() && (uptr)origin_addr < UnusedAddr();
+}
+
+inline bool has_valid_origin_addr(const void *ptr) {
+  const dfsan_origin *ptr_orig = origin_for(ptr);
+  return is_origin_addr_valid((uptr)ptr_orig);
+}
+
 void dfsan_copy_memory(void *dst, const void *src, uptr size);
 
 void dfsan_allocator_init();
 void dfsan_deallocate(void *ptr);
 
 void *dfsan_malloc(uptr size);
 void *dfsan_calloc(uptr nmemb, uptr size);
 void *dfsan_realloc(void *ptr, uptr size);

compiler-rt/lib/dfsan/dfsan.cpp

 #include "sanitizer_common/sanitizer_flags.h"
 #include "sanitizer_common/sanitizer_internal_defs.h"
 #include "sanitizer_common/sanitizer_libc.h"
 #include "sanitizer_common/sanitizer_report_decorator.h"
 #include "sanitizer_common/sanitizer_stacktrace.h"
 
 using namespace __dfsan;
 
-typedef atomic_uint16_t atomic_dfsan_label;
-static const dfsan_label kInitializingLabel = -1;
-
-static const uptr kNumLabels = 1 << (sizeof(dfsan_label) * 8);
-
-static atomic_dfsan_label __dfsan_last_label;
-static dfsan_label_info __dfsan_label_info[kNumLabels];
-
 Flags __dfsan::flags_data;
 
 // The size of TLS variables. These constants must be kept in sync with the ones
 // in DataFlowSanitizer.cpp.
 static const int kDFsanArgTlsSize = 800;
 static const int kDFsanRetvalTlsSize = 800;
 static const int kDFsanArgOriginTlsSize = 800;
 

 // On Linux/x86_64, memory is laid out as follows:
 //
 // +--------------------+ 0x800000000000 (top of memory)
 // | application memory |
 // +--------------------+ 0x700000008000 (kAppAddr)
 // |                    |
 // |       unused       |
 // |                    |
-// +--------------------+ 0x300200000000 (kUnusedAddr)
-// |    union table     |
-// +--------------------+ 0x300000000000 (kUnionTableAddr)
+// +--------------------+ 0x300000000000 (kUnusedAddr)
 // |       origin       |
-// +--------------------+ 0x200000000000 (kOriginAddr)
+// +--------------------+ 0x200000008000 (kOriginAddr)
+// |       unused       |
+// +--------------------+ 0x200000000000
 // |   shadow memory    |
-// +--------------------+ 0x000000010000 (kShadowAddr)
+// +--------------------+ 0x100000008000 (kShadowAddr)
+// |       unused       |
+// +--------------------+ 0x000000010000
 // | reserved by kernel |
 // +--------------------+ 0x000000000000
 //
-// To derive a shadow memory address from an application memory address,
-// bits 44-46 are cleared to bring the address into the range
-// [0x000000008000,0x100000000000).  Then the address is shifted left by 1 to
-// account for the double byte representation of shadow labels and move the
-// address into the shadow memory range.  See the function shadow_for below.
-
+// To derive a shadow memory address from an application memory address, bits
+// 45-46 are cleared to bring the address into the range
+// [0x100000008000,0x200000000000).  See the function shadow_for below.
+//
 // On Linux/MIPS64, memory is laid out as follows:
 //
 // +--------------------+ 0x10000000000 (top of memory)
 // | application memory |
 // +--------------------+ 0xF000008000 (kAppAddr)
 // |                    |
 // |       unused       |
 // |                    |
-// +--------------------+ 0x2200000000 (kUnusedAddr)
-// |    union table     |
-// +--------------------+ 0x2000000000 (kUnionTableAddr)
+// +--------------------+ 0x2000000000 (kUnusedAddr)
 // |   shadow memory    |
-// +--------------------+ 0x0000010000 (kShadowAddr)
+// +--------------------+ 0x1000008000 (kShadowAddr)
+// |       unused       |
+// +--------------------+ 0x0000010000
 // | reserved by kernel |
 // +--------------------+ 0x0000000000
 
 // On Linux/AArch64 (39-bit VMA), memory is laid out as follow:
 //
 // +--------------------+ 0x8000000000 (top of memory)
 // | application memory |
 // +--------------------+ 0x7000008000 (kAppAddr)
 // |                    |
 // |       unused       |
 // |                    |
-// +--------------------+ 0x1200000000 (kUnusedAddr)
-// |    union table     |
-// +--------------------+ 0x1000000000 (kUnionTableAddr)
+// +--------------------+ 0x1000000000 (kUnusedAddr)
 // |   shadow memory    |
 // +--------------------+ 0x0000010000 (kShadowAddr)
 // | reserved by kernel |
 // +--------------------+ 0x0000000000
 
 // On Linux/AArch64 (42-bit VMA), memory is laid out as follow:
 //
 // +--------------------+ 0x40000000000 (top of memory)
 // | application memory |
 // +--------------------+ 0x3ff00008000 (kAppAddr)
 // |                    |
 // |       unused       |
 // |                    |
-// +--------------------+ 0x1200000000 (kUnusedAddr)
-// |    union table     |
-// +--------------------+ 0x8000000000 (kUnionTableAddr)
+// +--------------------+ 0x8000000000 (kUnusedAddr)
 // |   shadow memory    |
 // +--------------------+ 0x0000010000 (kShadowAddr)
 // | reserved by kernel |
 // +--------------------+ 0x0000000000
 
 // On Linux/AArch64 (48-bit VMA), memory is laid out as follow:
 //
 // +--------------------+ 0x1000000000000 (top of memory)
 // | application memory |
 // +--------------------+ 0xffff00008000 (kAppAddr)
 // |       unused       |
 // +--------------------+ 0xaaaab0000000 (top of PIE address)
 // | application PIE    |
 // +--------------------+ 0xaaaaa0000000 (top of PIE address)
 // |                    |
 // |       unused       |
 // |                    |
-// +--------------------+ 0x1200000000 (kUnusedAddr)
-// |    union table     |
-// +--------------------+ 0x8000000000 (kUnionTableAddr)
+// +--------------------+ 0x8000000000 (kUnusedAddr)
 // |   shadow memory    |
 // +--------------------+ 0x0000010000 (kShadowAddr)
 // | reserved by kernel |
 // +--------------------+ 0x0000000000
 
-typedef atomic_dfsan_label dfsan_union_table_t[kNumLabels][kNumLabels];
-
 #ifdef DFSAN_RUNTIME_VMA
 // Runtime detected VMA size.
 int __dfsan::vmaSize;
 #endif
 
-static uptr UnusedAddr() {
-  return UnionTableAddr() + sizeof(dfsan_union_table_t);
-}
-
-static atomic_dfsan_label *union_table(dfsan_label l1, dfsan_label l2) {
-  return &(*(dfsan_union_table_t *) UnionTableAddr())[l1][l2];
-}
-
-// Checks we do not run out of labels.
-static void dfsan_check_label(dfsan_label label) {
-  if (label == kInitializingLabel) {
-    Report("FATAL: DataFlowSanitizer: out of labels\n");
-    Die();
-  }
-}
-
-// Resolves the union of two unequal labels.  Nonequality is a precondition for
-// this function (the instrumentation pass inlines the equality test).
-extern "C" SANITIZER_INTERFACE_ATTRIBUTE
-dfsan_label __dfsan_union(dfsan_label l1, dfsan_label l2) {
-  DCHECK_NE(l1, l2);
-
-  if (l1 == 0)
-    return l2;
-  if (l2 == 0)
-    return l1;
-
-  // If no labels have been created, yet l1 and l2 are non-zero, we are using
-  // fast16labels mode.
-  if (atomic_load(&__dfsan_last_label, memory_order_relaxed) == 0)
-    return l1 | l2;
-
-  if (l1 > l2)
-    Swap(l1, l2);
-
-  atomic_dfsan_label *table_ent = union_table(l1, l2);
-  // We need to deal with the case where two threads concurrently request
-  // a union of the same pair of labels.  If the table entry is uninitialized,
-  // (i.e. 0) use a compare-exchange to set the entry to kInitializingLabel
-  // (i.e. -1) to mark that we are initializing it.
-  dfsan_label label = 0;
-  if (atomic_compare_exchange_strong(table_ent, &label, kInitializingLabel,
-                                     memory_order_acquire)) {
-    // Check whether l2 subsumes l1.  We don't need to check whether l1
-    // subsumes l2 because we are guaranteed here that l1 < l2, and (at least
-    // in the cases we are interested in) a label may only subsume labels
-    // created earlier (i.e. with a lower numerical value).
-    if (__dfsan_label_info[l2].l1 == l1 ||
-        __dfsan_label_info[l2].l2 == l1) {
-      label = l2;
-    } else {
-      label =
-        atomic_fetch_add(&__dfsan_last_label, 1, memory_order_relaxed) + 1;
-      dfsan_check_label(label);
-      __dfsan_label_info[label].l1 = l1;
-      __dfsan_label_info[label].l2 = l2;
-    }
-    atomic_store(table_ent, label, memory_order_release);
-  } else if (label == kInitializingLabel) {
-    // Another thread is initializing the entry.  Wait until it is finished.
-    do {
-      internal_sched_yield();
-      label = atomic_load(table_ent, memory_order_acquire);
-    } while (label == kInitializingLabel);
-  }
-  return label;
-}
-
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE
 dfsan_label __dfsan_union_load(const dfsan_label *ls, uptr n) {
   dfsan_label label = ls[0];
-  for (uptr i = 1; i != n; ++i) {
-    dfsan_label next_label = ls[i];
-    if (label != next_label)
-      label = __dfsan_union(label, next_label);
-  }
-  return label;
-}
-
-extern "C" SANITIZER_INTERFACE_ATTRIBUTE
-dfsan_label __dfsan_union_load_fast16labels(const dfsan_label *ls, uptr n) {
-  dfsan_label label = ls[0];
   for (uptr i = 1; i != n; ++i)
     label |= ls[i];
   return label;
 }
 
 // Return the union of all the n labels from addr at the high 32 bit, and the
 // origin of the first taint byte at the low 32 bit.
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE u64

 // handling these at the moment.
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE void
 __dfsan_vararg_wrapper(const char *fname) {
   Report("FATAL: DataFlowSanitizer: unsupported indirect call to vararg "
          "function %s\n", fname);
   Die();
 }
 
-// Like __dfsan_union, but for use from the client or custom functions.  Hence
-// the equality comparison is done here before calling __dfsan_union.
+// Resolves the union of two unequal labels.
 SANITIZER_INTERFACE_ATTRIBUTE dfsan_label
 dfsan_union(dfsan_label l1, dfsan_label l2) {
-  if (l1 == l2)
-    return l1;
-  return __dfsan_union(l1, l2);
-}
-
-extern "C" SANITIZER_INTERFACE_ATTRIBUTE
-dfsan_label dfsan_create_label(const char *desc, void *userdata) {
-  dfsan_label label =
-      atomic_fetch_add(&__dfsan_last_label, 1, memory_order_relaxed) + 1;
-  dfsan_check_label(label);
-  __dfsan_label_info[label].l1 = __dfsan_label_info[label].l2 = 0;
-  __dfsan_label_info[label].desc = desc;
-  __dfsan_label_info[label].userdata = userdata;
-  return label;
+  return l1 | l2;
 }
 
 // Return the origin of the first taint byte in the size bytes from the address
 // addr.
 static dfsan_origin GetOriginIfTainted(uptr addr, uptr size) {
   for (uptr i = 0; i < size; ++i, ++addr) {
     dfsan_label *s = shadow_for((void *)addr);
     if (!is_shadow_addr_valid((uptr)s)) {
ChainAndWriteOriginIfTainted((uptr)src + (end - d), (d + size) - end, end,
                                  stack);
 
   if (beg >= end)
     return;
 
   // Align src up.
   uptr s = AlignUp((uptr)src);
   dfsan_origin *src_o = (dfsan_origin *)origin_for((void *)s);
-  u64 *src_s = (u64 *)shadow_for((void *)s);
+  u32 *src_s = (u32 *)shadow_for((void *)s);
   dfsan_origin *src_end = (dfsan_origin *)origin_for((void *)(s + (end - beg)));
   dfsan_origin *dst_o = (dfsan_origin *)origin_for((void *)beg);
   dfsan_origin last_src_o = 0;
   dfsan_origin last_dst_o = 0;
   for (; src_o < src_end; ++src_o, ++src_s, ++dst_o) {
     if (!*src_s)
       continue;
     if (*src_o != last_src_o) {
static void ReverseCopyOrigin(const void *dst, const void *src, uptr size,
 
   uptr beg = AlignDown(d);
 
   if (beg + kOriginAlign < end) {
     // Align src up.
     uptr s = AlignUp((uptr)src);
     dfsan_origin *src =
         (dfsan_origin *)origin_for((void *)(s + end - beg - kOriginAlign));
-    u64 *src_s = (u64 *)shadow_for((void *)(s + end - beg - kOriginAlign));
+    u32 *src_s = (u32 *)shadow_for((void *)(s + end - beg - kOriginAlign));
     dfsan_origin *src_begin = (dfsan_origin *)origin_for((void *)s);
     dfsan_origin *dst =
         (dfsan_origin *)origin_for((void *)(end - kOriginAlign));
     dfsan_origin src_o = 0;
     dfsan_origin dst_o = 0;
     for (; src >= src_begin; --src, --src_s, --dst) {
       if (!*src_s)
         continue;

 }
 
 }  // namespace __dfsan
 
 // If the label s is tainted, set the size bytes from the address p to be a new
 // origin chain with the previous ID o and the current stack trace. This is
 // used by instrumentation to reduce code size when too much code is inserted.
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE void __dfsan_maybe_store_origin(
-    u16 s, void *p, uptr size, dfsan_origin o) {
+    dfsan_label s, void *p, uptr size, dfsan_origin o) {
   if (UNLIKELY(s)) {
     GET_CALLER_PC_BP_SP;
     (void)sp;
     GET_STORE_STACK_TRACE_PC_BP(pc, bp);
     SetOrigin(p, size, ChainOrigin(o, &stack));
   }
 }
 
void dfsan_add_label(dfsan_label label, void *addr, uptr size) {
   if (__dfsan_get_track_origins()) {
     GET_CALLER_PC_BP;
     GET_STORE_STACK_TRACE_PC_BP(pc, bp);
     dfsan_origin init_origin = ChainOrigin(0, &stack, true);
     SetOrigin(addr, size, init_origin);
   }
 
   for (dfsan_label *labelp = shadow_for(addr); size != 0; --size, ++labelp)
-    if (*labelp != label)
-      *labelp = __dfsan_union(*labelp, label);
+    *labelp |= label;
 }
 
 // Unlike the other dfsan interface functions the behavior of this function
 // depends on the label of one of its arguments.  Hence it is implemented as a
 // custom function.
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE dfsan_label
 __dfsw_dfsan_get_label(long data, dfsan_label data_label,
                        dfsan_label *ret_label) {

 
 SANITIZER_INTERFACE_ATTRIBUTE void dfsan_set_label_origin(dfsan_label label,
                                                           dfsan_origin origin,
                                                           void *addr,
                                                           uptr size) {
   __dfsan_set_label(label, origin, addr, size);
 }
 
-extern "C" SANITIZER_INTERFACE_ATTRIBUTE
-const struct dfsan_label_info *dfsan_get_label_info(dfsan_label label) {
-  return &__dfsan_label_info[label];
-}
-
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE int
 dfsan_has_label(dfsan_label label, dfsan_label elem) {
-  if (label == elem)
-    return true;
-  const dfsan_label_info *info = dfsan_get_label_info(label);
-  if (info->l1 != 0) {
-    return dfsan_has_label(info->l1, elem) || dfsan_has_label(info->l2, elem);
-  } else {
-    return false;
-  }
-}
-
-extern "C" SANITIZER_INTERFACE_ATTRIBUTE dfsan_label
-dfsan_has_label_with_desc(dfsan_label label, const char *desc) {
-  const dfsan_label_info *info = dfsan_get_label_info(label);
-  if (info->l1 != 0) {
-    return dfsan_has_label_with_desc(info->l1, desc) ||
-           dfsan_has_label_with_desc(info->l2, desc);
-  } else {
-    return internal_strcmp(desc, info->desc) == 0;
-  }
-}
-
-extern "C" SANITIZER_INTERFACE_ATTRIBUTE uptr
-dfsan_get_label_count(void) {
-  dfsan_label max_label_allocated =
-      atomic_load(&__dfsan_last_label, memory_order_relaxed);
-
-  return static_cast<uptr>(max_label_allocated);
-}
-
-extern "C" SANITIZER_INTERFACE_ATTRIBUTE void
-dfsan_dump_labels(int fd) {
-  dfsan_label last_label =
-      atomic_load(&__dfsan_last_label, memory_order_relaxed);
-  for (uptr l = 1; l <= last_label; ++l) {
-    char buf[64];
-    internal_snprintf(buf, sizeof(buf), "%u %u %u ", l,
-                      __dfsan_label_info[l].l1, __dfsan_label_info[l].l2);
-    WriteToFile(fd, buf, internal_strlen(buf));
-    if (__dfsan_label_info[l].l1 == 0 && __dfsan_label_info[l].desc) {
-      WriteToFile(fd, __dfsan_label_info[l].desc,
-                  internal_strlen(__dfsan_label_info[l].desc));
-    }
-    WriteToFile(fd, "\n", 1);
-  }
+  return (label & elem) == elem;
 }
 
 class Decorator : public __sanitizer::SanitizerCommonDecorator {
  public:
   Decorator() : SanitizerCommonDecorator() {}
   const char *Origin() const { return Magenta(); }
 };
 
#ifdef DFSAN_RUNTIME_VMA
   } else {
     Printf("FATAL: DataFlowSanitizer: unsupported VMA range\n");
     Printf("FATAL: Found %d - Supported 39, 42, and 48\n", __dfsan::vmaSize);
     Die();
   }
 #endif
 }
 
-static void dfsan_fini() {
-  if (internal_strcmp(flags().dump_labels_at_exit, "") != 0) {
-    fd_t fd = OpenFile(flags().dump_labels_at_exit, WrOnly);
-    if (fd == kInvalidFd) {
-      Report("WARNING: DataFlowSanitizer: unable to open output file %s\n",
-             flags().dump_labels_at_exit);
-      return;
-    }
-
-    Report("INFO: DataFlowSanitizer: dumping labels to %s\n",
-           flags().dump_labels_at_exit);
-    dfsan_dump_labels(fd);
-    CloseFile(fd);
-  }
-}
-
 extern "C" void dfsan_flush() {
   if (!MmapFixedSuperNoReserve(ShadowAddr(), UnusedAddr() - ShadowAddr()))
     Die();
 }
 
 static void DFsanInit(int argc, char **argv, char **envp) {
   CHECK(!dfsan_init_is_running);
   if (dfsan_inited)
static void DFsanInit(int argc, char **argv, char **envp) {
   // works so long as the program doesn't use too much memory. We support this
   // case by disabling memory protection when ASLR is disabled.
   uptr init_addr = (uptr)&DFsanInit;
   if (!(init_addr >= UnusedAddr() && init_addr < AppAddr()))
     MmapFixedNoAccess(UnusedAddr(), AppAddr() - UnusedAddr());
 
   InitializeInterceptors();
 
-  // Register the fini callback to run when the program terminates successfully
-  // or it is killed by the runtime.
-  Atexit(dfsan_fini);
-  AddDieCallback(dfsan_fini);
-
   // Set up threads
   DFsanTSDInit(DFsanTSDDtor);
 
   dfsan_allocator_init();
 
   DFsanThread *main_thread = DFsanThread::Create(nullptr, nullptr, nullptr);
   SetCurrentThread(main_thread);
   main_thread->ThreadStart();
 
-  __dfsan_label_info[kInitializingLabel].desc = "<init label>";
-
   dfsan_init_is_running = false;
   dfsan_inited = true;
 }
 
 namespace __dfsan {
 
 void dfsan_init() { DFsanInit(0, nullptr, nullptr); }
 
 }  // namespace __dfsan
 
 #if SANITIZER_CAN_USE_PREINIT_ARRAY
 __attribute__((section(".preinit_array"),
                used)) static void (*dfsan_init_ptr)(int, char **,
                                                     char **) = DFsanInit;
 #endif

compiler-rt/lib/dfsan/dfsan_flags.inc

 DFSAN_FLAG(bool, warn_nonzero_labels, false,
            "Whether to warn on unimplemented functions.")
 DFSAN_FLAG(
     bool, strict_data_dependencies, true,
     "Whether to propagate labels only when there is an obvious data dependency"
     "(e.g., when comparing strings, ignore the fact that the output of the"
     "comparison might be data-dependent on the content of the strings). This"
     "applies only to the custom functions defined in 'custom.c'.")
-DFSAN_FLAG(const char *, dump_labels_at_exit, "", "The path of the file where "
-                                                  "to dump the labels when the "
-                                                  "program terminates.")
 DFSAN_FLAG(
      int, origin_history_size, Origin::kMaxDepth,
     "The limit of origin chain length. Non-positive values mean unlimited.")
 DFSAN_FLAG(
      int, origin_history_per_stack_limit, 20000,
     "The limit of origin node's references count. "
     "Non-positive values mean unlimited.")
 DFSAN_FLAG(int, store_context_size, 20,
            "The depth limit of origin tracking stack traces.")
 DFSAN_FLAG(bool, check_origin_invariant, false,
            "Whether to check if the origin invariant holds.")
 DFSAN_FLAG(bool, zero_in_malloc, true,
            "Whether to zero shadow space of new allocated memory.")
 DFSAN_FLAG(bool, zero_in_free, true,
            "Whether to zero shadow space of deallocated memory.")

compiler-rt/lib/dfsan/dfsan_platform.h

 //===-- dfsan_platform.h ----------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is a part of DataFlowSanitizer.
 //
 // Platform specific information for DFSan.
 //===----------------------------------------------------------------------===//
 
 #ifndef DFSAN_PLATFORM_H
 #define DFSAN_PLATFORM_H
 
+#include "sanitizer_common/sanitizer_common.h"
+
 namespace __dfsan {
 
+using __sanitizer::uptr;
+
 #if defined(__x86_64__)
 struct Mapping {
-  static const uptr kShadowAddr = 0x10000;
-  static const uptr kOriginAddr = 0x200000000000;
-  static const uptr kUnionTableAddr = 0x300000000000;
+  static const uptr kShadowAddr = 0x100000008000;
+  static const uptr kOriginAddr = 0x200000008000;
+  static const uptr kUnusedAddr = 0x300000000000;
   static const uptr kAppAddr = 0x700000008000;
-  static const uptr kShadowMask = ~0x700000000000;
+  static const uptr kShadowMask = ~0x600000000000;
 };
 #elif defined(__mips64)
 struct Mapping {
-  static const uptr kShadowAddr = 0x10000;
-  static const uptr kUnionTableAddr = 0x2000000000;
+  static const uptr kShadowAddr = 0x1000008000;
+  static const uptr kUnusedAddr = 0x2000000000;
   static const uptr kAppAddr = 0xF000008000;
-  static const uptr kShadowMask = ~0xF000000000;
+  static const uptr kShadowMask = ~0xE000000000;
 };
 #elif defined(__aarch64__)
 struct Mapping39 {
   static const uptr kShadowAddr = 0x10000;
-  static const uptr kUnionTableAddr = 0x1000000000;
+  static const uptr kUnusedAddr = 0x1000000000;
   static const uptr kAppAddr = 0x7000008000;
   static const uptr kShadowMask = ~0x7800000000;
 };
 
 struct Mapping42 {
   static const uptr kShadowAddr = 0x10000;
-  static const uptr kUnionTableAddr = 0x8000000000;
+  static const uptr kUnusedAddr = 0x8000000000;
   static const uptr kAppAddr = 0x3ff00008000;
   static const uptr kShadowMask = ~0x3c000000000;
 };
 
 struct Mapping48 {
   static const uptr kShadowAddr = 0x10000;
-  static const uptr kUnionTableAddr = 0x8000000000;
+  static const uptr kUnusedAddr = 0x8000000000;
   static const uptr kAppAddr = 0xffff00008000;
   static const uptr kShadowMask = ~0xfffff0000000;
 };
 
 extern int vmaSize;
 # define DFSAN_RUNTIME_VMA 1
 #else
 # error "DFSan not supported for this platform!"
 #endif
 
 enum MappingType {
   MAPPING_SHADOW_ADDR,
 #if defined(__x86_64__)
   MAPPING_ORIGIN_ADDR,
 #endif
-  MAPPING_UNION_TABLE_ADDR,
+  MAPPING_UNUSED_ADDR,
   MAPPING_APP_ADDR,
   MAPPING_SHADOW_MASK
 };
 
 template<typename Mapping, int Type>
 uptr MappingImpl(void) {
   switch (Type) {
     case MAPPING_SHADOW_ADDR: return Mapping::kShadowAddr;
 #if defined(__x86_64__)
     case MAPPING_ORIGIN_ADDR:
       return Mapping::kOriginAddr;
 #endif
-    case MAPPING_UNION_TABLE_ADDR: return Mapping::kUnionTableAddr;
+    case MAPPING_UNUSED_ADDR:
+      return Mapping::kUnusedAddr;
     case MAPPING_APP_ADDR: return Mapping::kAppAddr;
     case MAPPING_SHADOW_MASK: return Mapping::kShadowMask;
   }
 }
 
 template<int Type>
 uptr MappingArchImpl(void) {
 #ifdef __aarch64__

 #if defined(__x86_64__)
   return MappingArchImpl<MAPPING_ORIGIN_ADDR>();
 #else
   return 0;
 #endif
 }
 
 ALWAYS_INLINE
-uptr UnionTableAddr() {
-  return MappingArchImpl<MAPPING_UNION_TABLE_ADDR>();
-}
+uptr UnusedAddr() { return MappingArchImpl<MAPPING_UNUSED_ADDR>(); }
 
 ALWAYS_INLINE
 uptr AppAddr() {
   return MappingArchImpl<MAPPING_APP_ADDR>();
 }
 
 ALWAYS_INLINE
 uptr ShadowMask() {
   return MappingArchImpl<MAPPING_SHADOW_MASK>();
 }
 
 }  // namespace __dfsan
 
 #endif

compiler-rt/lib/fuzzer/dataflow/DataFlow.cpp

 static size_t InputLen;
 static size_t NumIterations;
 static dfsan_label **FuncLabelsPerIter;  // NumIterations x NumFuncs;
 
 static inline bool BlockIsEntry(size_t BlockIdx) {
   return __dft.PCsBeg[BlockIdx * 2 + 1] & PCFLAG_FUNC_ENTRY;
 }
 
-const int kNumLabels = 16;
+const int kNumLabels = 8;
 
 // Prints all instrumented functions.
 static int PrintFunctions() {
   // We don't have the symbolizer integrated with dfsan yet.
   // So use backtrace_symbols_fd and pipe it through llvm-symbolizer.
   // TODO(kcc): this is pretty ugly and may break in lots of ways.
   //      We'll need to make a proper in-process symbolizer work with DFSan.
   FILE *Pipe = popen("sed 's/(+/ /g; s/).*//g' "

compiler-rt/test/dfsan/atomic.cpp

-// RUN: %clangxx_dfsan -mllvm -dfsan-fast-16-labels=true %s -fno-exceptions -o %t && %run %t
-// RUN: %clangxx_dfsan -DORIGIN_TRACKING -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -fno-exceptions -o %t && %run %t
+// RUN: %clangxx_dfsan %s -fno-exceptions -o %t && %run %t
+// RUN: %clangxx_dfsan -DORIGIN_TRACKING -mllvm -dfsan-track-origins=1 %s -fno-exceptions -o %t && %run %t
 //
 // REQUIRES: x86_64-target-arch
 //
 // Use -fno-exceptions to turn off exceptions to avoid instrumenting
 // __cxa_begin_catch, std::terminate and __gxx_personality_v0.
 //
 // TODO: Support builtin atomics. For example, https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
 // DFSan instrumentation pass cannot identify builtin callsites yet.

compiler-rt/test/dfsan/basic.c

 // RUN: %clang_dfsan %s -o %t && %run %t
 // RUN: %clang_dfsan -mllvm -dfsan-args-abi %s -o %t && %run %t
 //
 // REQUIRES: x86_64-target-arch
 
 // Tests that labels are propagated through loads and stores.
 
 #include <sanitizer/dfsan_interface.h>
 #include <assert.h>
 
 int main(void) {
   int i = 1;
-  dfsan_label i_label = dfsan_create_label("i", 0);
+  dfsan_label i_label = 1;
   dfsan_set_label(i_label, &i, sizeof(i));
 
   dfsan_label new_label = dfsan_get_label(i);
   assert(i_label == new_label);
 
   dfsan_label read_label = dfsan_read_label(&i, sizeof(i));
   assert(i_label == read_label);
 
-  dfsan_label j_label = dfsan_create_label("j", 0);
+  dfsan_label j_label = 2;
   dfsan_add_label(j_label, &i, sizeof(i));
 
   read_label = dfsan_read_label(&i, sizeof(i));
   assert(dfsan_has_label(read_label, i_label));
   assert(dfsan_has_label(read_label, j_label));
 
   return 0;
 }

compiler-rt/test/dfsan/custom.cpp

 // RUN: %clang_dfsan %s -o %t && DFSAN_OPTIONS="strict_data_dependencies=0" %run %t
 // RUN: %clang_dfsan -mllvm -dfsan-args-abi %s -o %t && DFSAN_OPTIONS="strict_data_dependencies=0" %run %t
-// RUN: %clang_dfsan -DFAST_16_LABELS -mllvm -dfsan-fast-16-labels %s -o %t && DFSAN_OPTIONS="strict_data_dependencies=0" %run %t
+// RUN: %clang_dfsan %s -o %t && DFSAN_OPTIONS="strict_data_dependencies=0" %run %t
 // RUN: %clang_dfsan -DSTRICT_DATA_DEPENDENCIES %s -o %t && %run %t
 // RUN: %clang_dfsan -DSTRICT_DATA_DEPENDENCIES -mllvm -dfsan-args-abi %s -o %t && %run %t
-// RUN: %clang_dfsan -DFAST_16_LABELS -DORIGIN_TRACKING -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-origins=1 -mllvm -dfsan-combine-pointer-labels-on-load=false -DSTRICT_DATA_DEPENDENCIES %s -o %t && %run %t
-// RUN: %clang_dfsan -DFAST_16_LABELS -DORIGIN_TRACKING -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-origins=1 -mllvm -dfsan-combine-pointer-labels-on-load=false %s -o %t && DFSAN_OPTIONS="strict_data_dependencies=0" %run %t
+// RUN: %clang_dfsan -DORIGIN_TRACKING -mllvm -dfsan-track-origins=1 -mllvm -dfsan-combine-pointer-labels-on-load=false -DSTRICT_DATA_DEPENDENCIES %s -o %t && %run %t
+// RUN: %clang_dfsan -DORIGIN_TRACKING -mllvm -dfsan-track-origins=1 -mllvm -dfsan-combine-pointer-labels-on-load=false %s -o %t && DFSAN_OPTIONS="strict_data_dependencies=0" %run %t
 //
 // Tests custom implementations of various glibc functions.
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <arpa/inet.h>
void test_snprintf() {
   ASSERT_LABEL(r, 0);
 }
 
 // Tested by a seperate source file.  This empty function is here to appease the
 // check-wrappers script.
 void test_fork() {}
 
 int main(void) {
-#ifdef FAST_16_LABELS
   i_label = 1;
   j_label = 2;
   k_label = 4;
   m_label = 8;
   n_label = 16;
-#else
-  i_label = dfsan_create_label("i", 0);
-  j_label = dfsan_create_label("j", 0);
-  k_label = dfsan_create_label("k", 0);
-  m_label = dfsan_create_label("m", 0);
-  n_label = dfsan_create_label("n", 0);
-#endif
   i_j_label = dfsan_union(i_label, j_label);
   assert(i_j_label != i_label);
   assert(i_j_label != j_label);
   assert(i_j_label != k_label);
 
   test__dl_get_tls_static_info();
   test_bcmp();
   test_calloc();

compiler-rt/test/dfsan/dump_labels.c

-// RUN: %clang_dfsan %s -o %t
-// RUN: DFSAN_OPTIONS=dump_labels_at_exit=/dev/stdout %run %t 2>&1 | FileCheck %s
-// RUN: DFSAN_OPTIONS=dump_labels_at_exit=/dev/stdout not %run %t c 2>&1 | FileCheck %s --check-prefix=CHECK-OOL
-// RUN: DFSAN_OPTIONS=dump_labels_at_exit=/dev/stdout not %run %t u 2>&1 | FileCheck %s --check-prefix=CHECK-OOL
-//
-// REQUIRES: x86_64-target-arch
-
-// Tests that labels are properly dumped at program termination.
-
-#include <sanitizer/dfsan_interface.h>
-#include <assert.h>
-#include <stdio.h>
-
-int main(int argc, char** argv) {
-  int i = 1;
-  dfsan_label i_label = dfsan_create_label("i", 0);
-  dfsan_set_label(i_label, &i, sizeof(i));
-
-  int j = 2;
-  dfsan_label j_label = dfsan_create_label("j", 0);
-  dfsan_set_label(j_label, &j, sizeof(j));
-
-  int k = 3;
-  dfsan_label k_label = dfsan_create_label("k", 0);
-  dfsan_set_label(k_label, &k, sizeof(k));
-
-  dfsan_label ij_label = dfsan_get_label(i + j);
-  dfsan_label ijk_label = dfsan_get_label(i + j + k);
-
-  fprintf(stderr, "i %d j %d k %d ij %d ijk %d\n", i_label, j_label, k_label,
-          ij_label, ijk_label);
-
-  // CHECK: 1 0 0 i
-  // CHECK: 2 0 0 j
-  // CHECK: 3 0 0 k
-  // CHECK: 4 1 2
-  // CHECK: 5 3 4
-
-  if (argc > 1) {
-    // Exhaust the labels.
-    unsigned long num_labels = 1 << (sizeof(dfsan_label) * 8);
-    for (unsigned long i =  ijk_label + 1; i < num_labels - 2; ++i) {
-      dfsan_label l = dfsan_create_label("l", 0);
-      assert(l == i);
-    }
-
-    // Consume the last available label.
-    dfsan_label l = dfsan_union(5, 6);
-    assert(l == num_labels - 2);
-
-    // Try to allocate another label (either explicitly or by unioning two
-    // existing labels), but expect a crash.
-    if (argv[1][0] == 'c') {
-      l = dfsan_create_label("l", 0);
-    } else {
-      l = dfsan_union(6, 7);
-    }
-
-    // CHECK-OOL: FATAL: DataFlowSanitizer: out of labels
-    // CHECK-OOL: 1 0 0 i
-    // CHECK-OOL: 2 0 0 j
-    // CHECK-OOL: 3 0 0 k
-    // CHECK-OOL: 4 1 2
-    // CHECK-OOL: 5 3 4
-    // CHECK-OOL: 6 0 0
-    // CHECK-OOL: 65534 5 6
-    // CHECK-OOL: 65535 0 0 <init label>
-  }
-
-  return 0;
-}

compiler-rt/test/dfsan/event_callbacks.c

 dfsan_label LabelArgv;
 
 size_t LenArgv;
 
 int main(int Argc, char *Argv[]) {
   assert(Argc == 2);
 
   int I = 1, J = 2;
-  LabelI = dfsan_create_label("I", 0);
+  LabelI = 1;
   dfsan_set_label(LabelI, &I, sizeof(I));
-  LabelJ = dfsan_create_label("J", 0);
+  LabelJ = 2;
   dfsan_set_label(LabelJ, &J, sizeof(J));
   LabelIJ = dfsan_union(LabelI, LabelJ);
 
   // CHECK: Label 1 stored to memory
   volatile int Sink = I;
 
   // CHECK: Label 1 loaded from memory
   // CHECK: Label 1 used for branching
int main(int Argc, char *Argv[]) {
   // CHECK: Label 3 loaded from memory
   // CHECK: Label 3 used for branching
   assert(Sink == 3);
 
   // CHECK: Label 3 used for branching
   assert(I != J);
 
   LenArgv = strlen(Argv[1]);
-  LabelArgv = dfsan_create_label("Argv", 0);
+  LabelArgv = 4;
   dfsan_set_label(LabelArgv, Argv[1], LenArgv);
 
   char Buf[64];
   assert(LenArgv < sizeof(Buf) - 1);
 
   // CHECK: Label 4 copied to memory
   void *volatile SinkPtr = Buf;
   memcpy(SinkPtr, Argv[1], LenArgv);

compiler-rt/test/dfsan/fast8labels.c

-// RUN: %clang_dfsan %s -mllvm -dfsan-fast-16-labels -o %t
+// RUN: %clang_dfsan %s -o %t
 // RUN: %run %t
 //
 // REQUIRES: x86_64-target-arch
 //
-// Tests fast16labels mode.
-
 #include <sanitizer/dfsan_interface.h>
 
 #include <assert.h>
 #include <stdio.h>
 #include <string.h>
 
 int foo(int a, int b) {
   return a + b;
 }
 
 int main(int argc, char *argv[]) {
   int a = 10;
   int b = 20;
   dfsan_set_label(8, &a, sizeof(a));
-  dfsan_set_label(512, &b, sizeof(b));
+  dfsan_set_label(128, &b, sizeof(b));
   int c = foo(a, b);
   printf("A: 0x%x\n", dfsan_get_label(a));
   printf("B: 0x%x\n", dfsan_get_label(b));
   dfsan_label l = dfsan_get_label(c);
   printf("C: 0x%x\n", l);
-  assert(l == 520);  // OR of the other two labels.
+  assert(l == 136);  // OR of the other two labels.
 }

compiler-rt/test/dfsan/flags.c

 // RUN: %clang_dfsan %s -fsanitize-ignorelist=%S/Inputs/flags_abilist.txt -mllvm -dfsan-debug-nonzero-labels -o %t && %run %t 2>&1 | FileCheck %s
 // RUN: %clang_dfsan %s -fsanitize-ignorelist=%S/Inputs/flags_abilist.txt -mllvm -dfsan-debug-nonzero-labels -o %t && DFSAN_OPTIONS=warn_unimplemented=0 %run %t 2>&1 | count 0
 // RUN: %clang_dfsan %s -fsanitize-ignorelist=%S/Inputs/flags_abilist.txt -mllvm -dfsan-debug-nonzero-labels -o %t && DFSAN_OPTIONS=warn_nonzero_labels=1 %run %t 2>&1 | FileCheck --check-prefix=CHECK-NONZERO %s
 //
 // REQUIRES: x86_64-target-arch
 
 // Tests that flags work correctly.
 
 #include <sanitizer/dfsan_interface.h>
 
 int f(int i) {
   return i;
 }
 
 int main(void) {
   int i = 1;
-  dfsan_label i_label = dfsan_create_label("i", 0);
+  dfsan_label i_label = 2;
   dfsan_set_label(i_label, &i, sizeof(i));
 
   // CHECK: WARNING: DataFlowSanitizer: call to uninstrumented function f
   // CHECK-NOT: WARNING: DataFlowSanitizer: saw nonzero label
   // CHECK-NONZERO: WARNING: DataFlowSanitizer: saw nonzero label
   f(i);
 
   return 0;
 }

compiler-rt/test/dfsan/flush.c

 // Tests dfsan_flush().
 // RUN: %clang_dfsan %s -o %t && %run %t
-// RUN: %clang_dfsan -DORIGIN_TRACKING -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && %run %t
+// RUN: %clang_dfsan -DORIGIN_TRACKING -mllvm -dfsan-track-origins=1 %s -o %t && %run %t
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 #include <assert.h>
 #include <stdlib.h>
 
 int global;

compiler-rt/test/dfsan/fncall.c

 // RUN: %clang_dfsan %s -o %t && %run %t
 // RUN: %clang_dfsan -mllvm -dfsan-args-abi %s -o %t && %run %t
 //
 // REQUIRES: x86_64-target-arch
 //
 // Tests that labels are propagated through function calls.
 
 #include <sanitizer/dfsan_interface.h>
 #include <assert.h>
 
 int f(int x) {
   int j = 2;
-  dfsan_label j_label = dfsan_create_label("j", 0);
+  dfsan_label j_label = 2;
   dfsan_set_label(j_label, &j, sizeof(j));
   return x + j;
 }
 
 int main(void) {
   int i = 1;
-  dfsan_label i_label = dfsan_create_label("i", 0);
+  dfsan_label i_label = 4;
   dfsan_set_label(i_label, &i, sizeof(i));
 
   dfsan_label ij_label = dfsan_get_label(f(i));
   assert(dfsan_has_label(ij_label, i_label));
-  assert(dfsan_has_label_with_desc(ij_label, "j"));
+
+  /* Must be consistent with the one in f(). */
+  dfsan_label j_label = 2;
+  assert(dfsan_has_label(ij_label, 2));
 
   return 0;
 }

compiler-rt/test/dfsan/fork.cpp

 // Test that chained origins are fork-safe.
 // Run a number of threads that create new chained origins, then fork
 // and verify that origin reads do not deadlock in the child process.
 //
-// RUN: %clangxx_dfsan -mllvm -dfsan-fast-16-labels=true %s -o %t
+// RUN: %clangxx_dfsan %s -o %t
 // RUN: %run %t 2>&1 | FileCheck %s
 //
-// RUN: %clangxx_dfsan -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t
+// RUN: %clangxx_dfsan -mllvm -dfsan-track-origins=1 %s -o %t
 // RUN: DFSAN_OPTIONS=store_context_size=1000,origin_history_size=0,origin_history_per_stack_limit=0 %run %t 2>&1 | FileCheck %s
 //
 // REQUIRES: x86_64-target-arch
 
 #include <assert.h>
 #include <errno.h>
 #include <pthread.h>
 #include <signal.h>

compiler-rt/test/dfsan/label_count.c

-// RUN: %clang_dfsan -DLIB -c %s -o %t.lib.o && \
-// RUN: %clang_dfsan       -c %s -o %t.o && \
-// RUN: %clang_dfsan %t.lib.o %t.o -o %t.bin && \
-// RUN: %run %t.bin
-
-// RUN: %clang_dfsan -mllvm -dfsan-args-abi -DLIB -c %s -o %t.lib.o && \
-// RUN: %clang_dfsan -mllvm -dfsan-args-abi -c %s -o %t.o && \
-// RUN: %clang_dfsan -mllvm -dfsan-args-abi %t.o %t.lib.o -o %t.bin && \
-// RUN: %run %t.bin
-//
-// REQUIRES: x86_64-target-arch
-
-#include <sanitizer/dfsan_interface.h>
-#include <assert.h>
-
-#ifdef LIB
-// Compiling this file with and without LIB defined allows this file to be
-// built as two separate translation units.  This ensures that the code
-// can not be optimized in a way that removes behavior we wish to test.  For
-// example, computing a value should cause labels to be allocated only if
-// the computation is actually done.  Putting the computation here prevents
-// the compiler from optimizing away the computation (and labeling) that
-// tests wish to verify.
-
-int add_in_separate_translation_unit(int a, int b) {
-  return a + b;
-}
-
-int multiply_in_separate_translation_unit(int a, int b) {
-  return a * b;
-}
-
-#else
-
-int add_in_separate_translation_unit(int i, int j);
-int multiply_in_separate_translation_unit(int i, int j);
-
-int main(void) {
-  size_t label_count;
-
-  // No labels allocated yet.
-  label_count = dfsan_get_label_count();
-  assert(0 == label_count);
-
-  int i = 1;
-  dfsan_label i_label = dfsan_create_label("i", 0);
-  dfsan_set_label(i_label, &i, sizeof(i));
-
-  // One label allocated for i.
-  label_count = dfsan_get_label_count();
-  assert(1u == label_count);
-
-  int j = 2;
-  dfsan_label j_label = dfsan_create_label("j", 0);
-  dfsan_set_label(j_label, &j, sizeof(j));
-
-  // Check that a new label was allocated for j.
-  label_count = dfsan_get_label_count();
-  assert(2u == label_count);
-
-  // Create a value that combines i and j.
-  int i_plus_j = add_in_separate_translation_unit(i, j);
-
-  // Check that a label was created for the union of i and j.
-  label_count = dfsan_get_label_count();
-  assert(3u == label_count);
-
-  // Combine i and j in a different way.  Check that the existing label is
-  // reused, and a new label is not created.
-  int j_times_i = multiply_in_separate_translation_unit(j, i);
-  label_count = dfsan_get_label_count();
-  assert(3u == label_count);
-  assert(dfsan_get_label(i_plus_j) == dfsan_get_label(j_times_i));
-
-  return 0;
-}
-#endif  // #ifdef LIB

compiler-rt/test/dfsan/origin_add_label.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 __attribute__((noinline)) uint64_t foo(uint64_t a, uint64_t b) { return a + b; }
 
 int main(int argc, char *argv[]) {

compiler-rt/test/dfsan/origin_branch.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 __attribute__((noinline)) uint64_t foo(uint64_t x, uint64_t y) { return x + y; }
 
 int main(int argc, char *argv[]) {

compiler-rt/test/dfsan/origin_disabled.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-fast-16-labels=true %s -o %t && \
-// RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: %clang_dfsan -gmlt %s -o %t && %run %t >%t.out 2>&1
+// RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 int main(int argc, char *argv[]) {
   uint64_t a = 10;
   dfsan_set_label(8, &a, sizeof(a));
   dfsan_print_origin_trace(&a, NULL);
 }
 
 // CHECK: DFSan: origin tracking is not enabled. Did you specify the -dfsan-track-origins=1 option?

compiler-rt/test/dfsan/origin_invalid.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-fast-16-labels=true -mllvm -dfsan-track-origins=1 %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 // 
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 int main(int argc, char *argv[]) {
   uint64_t a = 10;
-  dfsan_set_label(8, &a, sizeof(a));
+  dfsan_set_label(1, &a, sizeof(a));
   size_t origin_addr =
-      (((size_t)&a & ~0x700000000000LL + 0x200000000000LL) & ~0x3UL);
+      (((size_t)&a & ~0x600000000000LL + 0x100000000000LL) & ~0x3UL);
   asm("mov %0, %%rax": :"r"(origin_addr));
   asm("movq $0, (%rax)");
   dfsan_print_origin_trace(&a, "invalid");
 }
 
-// CHECK: Taint value 0x8 (at {{.*}}) origin tracking (invalid)
-// CHECK: Taint value 0x8 (at {{.*}}) has invalid origin tracking. This can be a DFSan bug.
+// CHECK: Taint value 0x1 (at {{.*}}) origin tracking (invalid)
+// CHECK: Taint value 0x1 (at {{.*}}) has invalid origin tracking. This can be a DFSan bug.

compiler-rt/test/dfsan/origin_ld_lost.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 //
 // Test origin tracking can lost origins at 2-byte load with addr % 4 == 3.
 
 #include <sanitizer/dfsan_interface.h>
 
 __attribute__((noinline)) uint16_t foo(uint16_t a, uint16_t b) { return a + b; }

compiler-rt/test/dfsan/origin_ldst.c

-// RUN: %clang_dfsan -gmlt -DTEST64 -DALIGN=8 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DTEST64 -DALIGN=8 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -DTEST32 -DALIGN=4 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DTEST32 -DALIGN=4 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -DALIGN=2 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DALIGN=2 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
-// rUN: %clang_dfsan -DTEST64 -DALIGN=5 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
-// rUN:     %run %t >%t.out 2>&1
-// rUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: %clang_dfsan -gmlt -DTEST64 -DALIGN=4 -mllvm -dfsan-track-origins=1 %s -o %t && \
+// RUN:     %run %t >%t.out 2>&1
+// RUN: FileCheck %s < %t.out
 //
-// rUN: %clang_dfsan -DTEST32 -DALIGN=3 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
-// rUN:     %run %t >%t.out 2>&1
-// rUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: %clang_dfsan -gmlt -DTEST32 -DALIGN=2 -mllvm -dfsan-track-origins=1 %s -o %t && \
+// RUN:     %run %t >%t.out 2>&1
+// RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -DALIGN=1 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DALIGN=1 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 //
 // Test origin tracking is accurate in terms of partial store/load, and
 // different aligments. Do not test alignments that are not power of 2.
 // Compilers do not always allow this.
 
 #include <sanitizer/dfsan_interface.h>

compiler-rt/test/dfsan/origin_limit.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t
 //
 // RUN: %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
 // RUN: DFSAN_OPTIONS=origin_history_size=2 %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK2 < %t.out
 //
 // RUN: DFSAN_OPTIONS=origin_history_size=0 %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK0 < %t.out
 //
 // REQUIRES: x86_64-target-arch

compiler-rt/test/dfsan/origin_memcpy.c

-// RUN: %clang_dfsan -gmlt -DOFFSET=0 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DOFFSET=0 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK0 < %t.out
 // 
-// RUN: %clang_dfsan -gmlt -DOFFSET=10 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DOFFSET=10 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK10 < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <string.h>

compiler-rt/test/dfsan/origin_memmove.c

-// RUN: %clang_dfsan -gmlt -DOFFSET=0 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DOFFSET=0 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK0 < %t.out
 //
-// RUN: %clang_dfsan -gmlt -DOFFSET=10 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DOFFSET=10 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK10 < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <string.h>

compiler-rt/test/dfsan/origin_memset.c

-// RUN: %clang_dfsan -gmlt -DOFFSET=0 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DOFFSET=0 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK0 < %t.out
 // 
-// RUN: %clang_dfsan -gmlt -DOFFSET=10 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DOFFSET=10 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK10 < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <string.h>

compiler-rt/test/dfsan/origin_overlapped.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 int main(int argc, char *argv[]) {
   char volatile z1;
   char volatile z2;

compiler-rt/test/dfsan/origin_set_label.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 // 
-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 __attribute__((noinline)) uint64_t foo(uint64_t a, uint64_t b) { return a + b; }
 
 int main(int argc, char *argv[]) {

compiler-rt/test/dfsan/origin_stack_trace.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 #include <stdio.h>
 #include <string.h>

compiler-rt/test/dfsan/origin_track_ld.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=2 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=2 %s -o %t && \
 // RUN:     %run %t > %t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 __attribute__((noinline)) uint64_t foo(uint64_t a, uint64_t b) { return a + b; }

compiler-rt/test/dfsan/origin_unaligned_memtrans.c

-// RUN: %clang_dfsan -gmlt -DOFFSET=0 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DOFFSET=0 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK0 < %t.out
 // 
-// RUN: %clang_dfsan -gmlt -DOFFSET=10 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DOFFSET=10 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s --check-prefix=CHECK10 < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <string.h>

compiler-rt/test/dfsan/origin_untainted.c

-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-fast-16-labels=true -mllvm -dfsan-track-origins=1 %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
-// RUN: FileCheck %s --check-prefix=CHECK < %t.out
+// RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 int main(int argc, char *argv[]) {
   uint64_t a = 10;
   dfsan_print_origin_trace(&a, NULL);
 }
 
 // CHECK: DFSan: no tainted value at {{.*}}

compiler-rt/test/dfsan/origin_with_sigactions.c

 // Check that stores in signal handlers are not recorded in origin history.
 //
 // Origin tracking uses ChainedOriginDepot that is not async signal safe, so we
 // do not track origins inside signal handlers.
 //
-// RUN: %clang_dfsan -gmlt -DUSE_SIGNAL_ACTION -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DUSE_SIGNAL_ACTION -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:      %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -DUSE_SIGNAL_ACTION -mllvm -dfsan-instrument-with-call-threshold=0 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -DUSE_SIGNAL_ACTION -mllvm -dfsan-instrument-with-call-threshold=0 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-instrument-with-call-threshold=0 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-instrument-with-call-threshold=0 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <assert.h>

compiler-rt/test/dfsan/origin_with_signals.cpp

 // Check that stores in signal handlers are not recorded in origin history.
 //
 // Origin tracking uses ChainedOriginDepot that is not async signal safe, so we
 // do not track origins inside signal handlers.
 //
-// RUN: %clangxx_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clangxx_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
-// RUN: %clangxx_dfsan -gmlt -mllvm -dfsan-instrument-with-call-threshold=0 -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clangxx_dfsan -gmlt -mllvm -dfsan-instrument-with-call-threshold=0 -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <signal.h>

compiler-rt/test/dfsan/pair.cpp

-// RUN: %clangxx_dfsan %s -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-select-control-flow=false -mllvm -dfsan-combine-pointer-labels-on-load=false -O0 -DO0 -o %t && %run %t
-// RUN: %clangxx_dfsan %s -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-select-control-flow=false -mllvm -dfsan-combine-pointer-labels-on-load=false -O1 -o %t && %run %t
+// RUN: %clangxx_dfsan %s -mllvm -dfsan-track-select-control-flow=false -mllvm -dfsan-combine-pointer-labels-on-load=false -O0 -DO0 -o %t && %run %t
+// RUN: %clangxx_dfsan %s -mllvm -dfsan-track-select-control-flow=false -mllvm -dfsan-combine-pointer-labels-on-load=false -O1 -o %t && %run %t
 //
 // REQUIRES: x86_64-target-arch
 
 #include <algorithm>
 #include <assert.h>
 #include <sanitizer/dfsan_interface.h>
 #include <utility>
 

compiler-rt/test/dfsan/propagate.c

 // RUN: %clang_dfsan %s -o %t && %run %t
 // RUN: %clang_dfsan -mllvm -dfsan-args-abi %s -o %t && %run %t
 //
 // REQUIRES: x86_64-target-arch
 
 // Tests that labels are propagated through computation and that union labels
 // are properly created.
 
 #include <sanitizer/dfsan_interface.h>
 #include <assert.h>
 
 int main(void) {
   assert(dfsan_union(0, 0) == 0);
 
   int i = 1;
-  dfsan_label i_label = dfsan_create_label("i", 0);
+  dfsan_label i_label = 1;
   dfsan_set_label(i_label, &i, sizeof(i));
 
   int j = 2;
-  dfsan_label j_label = dfsan_create_label("j", 0);
+  dfsan_label j_label = 2;
   dfsan_set_label(j_label, &j, sizeof(j));
 
   int k = 3;
-  dfsan_label k_label = dfsan_create_label("k", 0);
+  dfsan_label k_label = 4;
   dfsan_set_label(k_label, &k, sizeof(k));
 
   int k2 = 4;
   dfsan_set_label(k_label, &k2, sizeof(k2));
 
   dfsan_label ij_label = dfsan_get_label(i + j);
   assert(dfsan_has_label(ij_label, i_label));
   assert(dfsan_has_label(ij_label, j_label));

compiler-rt/test/dfsan/pthread.c

-// RUN: %clang_dfsan -mllvm -dfsan-fast-16-labels=true %s -o %t && %run %t
+// RUN: %clang_dfsan %s -o %t && %run %t
 //
-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
-// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && \
+// RUN: %clang_dfsan -gmlt -mllvm -dfsan-track-origins=1 -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && \
 // RUN:     %run %t >%t.out 2>&1
 // RUN: FileCheck %s < %t.out
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <assert.h>

compiler-rt/test/dfsan/release_shadow_space.c

 // DFSAN_OPTIONS=no_huge_pages_for_shadow=false RUN: %clang_dfsan %s -o %t && setarch `uname -m` -R %run %t
 // DFSAN_OPTIONS=no_huge_pages_for_shadow=true RUN: %clang_dfsan %s -o %t && setarch `uname -m` -R %run %t
-// DFSAN_OPTIONS=no_huge_pages_for_shadow=false RUN: %clang_dfsan %s -DORIGIN_TRACKING -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-origins=1 -o %t && setarch `uname -m` -R %run %t
-// DFSAN_OPTIONS=no_huge_pages_for_shadow=true RUN: %clang_dfsan %s -DORIGIN_TRACKING -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-origins=1 -o %t && setarch `uname -m` -R %run %t
+// DFSAN_OPTIONS=no_huge_pages_for_shadow=false RUN: %clang_dfsan %s -DORIGIN_TRACKING -mllvm -dfsan-track-origins=1 -o %t && setarch `uname -m` -R %run %t
+// DFSAN_OPTIONS=no_huge_pages_for_shadow=true RUN: %clang_dfsan %s -DORIGIN_TRACKING -mllvm -dfsan-track-origins=1 -o %t && setarch `uname -m` -R %run %t
 //
 // REQUIRES: x86_64-target-arch
 
 #include <assert.h>
 #include <sanitizer/dfsan_interface.h>
 #include <stdbool.h>
 #include <stdio.h>
 #include <string.h>
int main(int argc, char **argv) {
 
   // mmap and touch all addresses. The overhead is 1x.
   char *p = mmap(NULL, map_size, PROT_READ | PROT_WRITE,
                  MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
   memset(p, 0xff, map_size);
   size_t after_mmap = get_rss_kb();
 
   // store labels to all addresses. The overhead is 2x.
-  const dfsan_label label = dfsan_create_label("l", 0);
+  const dfsan_label label = 8;
   char val = 0xff;
   dfsan_set_label(label, &val, sizeof(val));
   memset(p, val, map_size);
   size_t after_mmap_and_set_label = get_rss_kb();
 
   // fixed-mmap the same address. OS recyles pages and reinitializes data at the
   // address. This should be the same to calling munmap.
   p = mmap(p, map_size, PROT_READ | PROT_WRITE,

compiler-rt/test/dfsan/sigaction.c

-// RUN: %clang_dfsan -DUSE_SIGNAL_ACTION -mllvm -dfsan-fast-16-labels=true %s -o %t && \
-// RUN:     %run %t
-// RUN: %clang_dfsan -mllvm -dfsan-fast-16-labels=true %s -o %t && %run %t
+// RUN: %clang_dfsan -DUSE_SIGNAL_ACTION %s -o %t && %run %t
+// RUN: %clang_dfsan %s -o %t && %run %t
 //
 // REQUIRES: x86_64-target-arch
 
 #include <sanitizer/dfsan_interface.h>
 
 #include <assert.h>
 #include <signal.h>
 #include <string.h>

compiler-rt/test/dfsan/sigaction_stress_test.c

-// RUN: %clangxx_dfsan -mllvm -dfsan-fast-16-labels=true %s -o %t && %run %t
-// RUN: %clangxx_dfsan -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true %s -o %t && %run %t
-// RUN: %clangxx_dfsan -mllvm -dfsan-track-origins=1 -mllvm -dfsan-fast-16-labels=true -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && %run %t
+// RUN: %clangxx_dfsan %s -o %t && %run %t
+// RUN: %clangxx_dfsan -mllvm -dfsan-track-origins=1 %s -o %t && %run %t
+// RUN: %clangxx_dfsan -mllvm -dfsan-track-origins=1 -mllvm -dfsan-instrument-with-call-threshold=0 %s -o %t && %run %t
 //
 // Test that the state of shadows from a sigaction handler are consistent.
 //
 // REQUIRES: x86_64-target-arch
 
 #include <signal.h>
 #include <stdarg.h>
 #include <sanitizer/dfsan_interface.h>

compiler-rt/test/dfsan/struct.c

-// RUN: %clang_dfsan %s -O1 -mllvm -dfsan-fast-16-labels=true -DFAST16_O1 -o %t && %run %t
-// RUN: %clang_dfsan %s -O1 -DO1 -o %t && %run %t
-// RUN: %clang_dfsan %s -O0 -mllvm -dfsan-fast-16-labels=true -DFAST16_O0 -o %t && %run %t
+// RUN: %clang_dfsan %s -O1 -o %t && %run %t
 // RUN: %clang_dfsan %s -O0 -DO0 -o %t && %run %t
 //
 // REQUIRES: x86_64-target-arch
 
 #include <assert.h>
 #include <sanitizer/dfsan_interface.h>
 
 typedef struct Pair {
Pair copy_pair2(const Pair pair0) {
   pair.i = pair0.i;
   pair.ptr = pair0.ptr;
   return pair;
 }
 
 int main(void) {
   int i = 1;
   char *ptr = NULL;
-#if defined(FAST16_O1) || defined(FAST16_O0)
   dfsan_label i_label = 1;
   dfsan_label ptr_label = 2;
-#else
-  dfsan_label i_label = dfsan_create_label("i", 0);
-  dfsan_label ptr_label = dfsan_create_label("ptr", 0);
-#endif
   dfsan_set_label(i_label, &i, sizeof(i));
   dfsan_set_label(ptr_label, &ptr, sizeof(ptr));
 
   Pair pair1 = make_pair(i, ptr);
   int i1 = pair1.i;
   char *ptr1 = pair1.ptr;
 
   dfsan_label i1_label = dfsan_read_label(&i1, sizeof(i1));
   dfsan_label ptr1_label = dfsan_read_label(&ptr1, sizeof(ptr1));
-#if defined(O0) || defined(O1)
-  assert(dfsan_has_label(i1_label, i_label));
-  assert(dfsan_has_label(i1_label, ptr_label));
-  assert(dfsan_has_label(ptr1_label, i_label));
-  assert(dfsan_has_label(ptr1_label, ptr_label));
-#elif defined(FAST16_O0)
+#if defined(O0)
   assert(i1_label == (i_label | ptr_label));
   assert(ptr1_label == (i_label | ptr_label));
 #else
   assert(i1_label == i_label);
   assert(ptr1_label == ptr_label);
 #endif
 
   Pair pair2 = copy_pair1(&pair1);
   int i2 = pair2.i;
   char *ptr2 = pair2.ptr;
 
   dfsan_label i2_label = dfsan_read_label(&i2, sizeof(i2));
   dfsan_label ptr2_label = dfsan_read_label(&ptr2, sizeof(ptr2));
-#if defined(O0) || defined(O1)
-  assert(dfsan_has_label(i2_label, i_label));
-  assert(dfsan_has_label(i2_label, ptr_label));
-  assert(dfsan_has_label(ptr2_label, i_label));
-  assert(dfsan_has_label(ptr2_label, ptr_label));
-#elif defined(FAST16_O0)
+#if defined(O0)
   assert(i2_label == (i_label | ptr_label));
   assert(ptr2_label == (i_label | ptr_label));
 #else
   assert(i2_label == i_label);
   assert(ptr2_label == ptr_label);
 #endif
 
   Pair pair3 = copy_pair2(pair1);
   int i3 = pair3.i;
   char *ptr3 = pair3.ptr;
 
   dfsan_label i3_label = dfsan_read_label(&i3, sizeof(i3));
   dfsan_label ptr3_label = dfsan_read_label(&ptr3, sizeof(ptr3));
-#if defined(O0) || defined(O1)
-  assert(dfsan_has_label(i3_label, i_label));
-  assert(dfsan_has_label(i3_label, ptr_label));
-  assert(dfsan_has_label(ptr3_label, i_label));
-  assert(dfsan_has_label(ptr3_label, ptr_label));
-#elif defined(FAST16_O0)
+#if defined(O0)
   assert(i3_label == (i_label | ptr_label));
   assert(ptr3_label == (i_label | ptr_label));
 #else
   assert(i3_label == i_label);
   assert(ptr3_label == ptr_label);
 #endif
 
 
   return 0;
 }

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

 /// Unlike other Sanitizer tools, this tool is not designed to detect a specific
 /// class of bugs on its own.  Instead, it provides a generic dynamic data flow
 /// analysis framework to be used by clients to help detect application-specific
 /// issues within their own code.
 ///
 /// The analysis is based on automatic propagation of data flow labels (also
 /// known as taint labels) through a program as it performs computation.
 ///
-/// There are two possible memory layouts. In the first one, each byte of
-/// application memory is backed by a shadow memory byte. The shadow byte can
-/// represent up to 8 labels. To enable this you must specify the
-/// -dfsan-fast-8-labels flag. On Linux/x86_64, memory is then laid out as
-/// follows:
+/// Each byte of application memory is backed by a shadow memory byte. The
+/// shadow byte can represent up to 8 labels. On Linux/x86_64, memory is then
+/// laid out as follows:
 ///
 /// +--------------------+ 0x800000000000 (top of memory)
 /// | application memory |
 /// +--------------------+ 0x700000008000 (kAppAddr)
 /// |                    |
 /// |       unused       |
 /// |                    |
-/// +--------------------+ 0x300200000000 (kUnusedAddr)
-/// |    union table     |
-/// +--------------------+ 0x300000000000 (kUnionTableAddr)
+/// +--------------------+ 0x300000000000 (kUnusedAddr)
 /// |       origin       |
 /// +--------------------+ 0x200000008000 (kOriginAddr)
+/// |       unused       |
+/// +--------------------+ 0x200000000000
 /// |   shadow memory    |
 /// +--------------------+ 0x100000008000 (kShadowAddr)
 /// |       unused       |
 /// +--------------------+ 0x000000010000
 /// | reserved by kernel |
 /// +--------------------+ 0x000000000000
 ///
 ///
-/// In the second memory layout, each byte of application memory is backed by
-/// two bytes of shadow memory which hold the label. That means we can represent
-/// either 16 labels (with -dfsan-fast-16-labels flag) or 2^16 labels (on the
-/// default legacy mode) per byte. On Linux/x86_64, memory is then laid out as
-/// follows:
-///
-/// +--------------------+ 0x800000000000 (top of memory)
-/// | application memory |
-/// +--------------------+ 0x700000008000 (kAppAddr)
-/// |                    |
-/// |       unused       |
-/// |                    |
-/// +--------------------+ 0x300200000000 (kUnusedAddr)
-/// |    union table     |
-/// +--------------------+ 0x300000000000 (kUnionTableAddr)
-/// |       origin       |
-/// +--------------------+ 0x200000008000 (kOriginAddr)
-/// |   shadow memory    |
-/// +--------------------+ 0x000000010000 (kShadowAddr)
-/// | reserved by kernel |
-/// +--------------------+ 0x000000000000
-///
-///
-/// To derive a shadow memory address from an application memory address,
-/// bits 44-46 are cleared to bring the address into the range
-/// [0x000000008000,0x100000000000).  Then the address is shifted left by 1 to
-/// account for the double byte representation of shadow labels and move the
-/// address into the shadow memory range.  See the function
+/// To derive a shadow memory address from an application memory address, bits
+/// 45-46 are cleared to bring the address into the range
+/// [0x100000008000,0x200000000000). See the function
 /// DataFlowSanitizer::getShadowAddress below.
 ///
 /// For more information, please refer to the design document:
 /// http://clang.llvm.org/docs/DataFlowSanitizerDesign.html
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Transforms/Instrumentation/DataFlowSanitizer.h"

 //   void __dfsan_store_callback(dfsan_label Label, void* addr);
 //   void __dfsan_mem_transfer_callback(dfsan_label *Start, size_t Len);
 //   void __dfsan_cmp_callback(dfsan_label CombinedLabel);
 static cl::opt<bool> ClEventCallbacks(
     "dfsan-event-callbacks",
     cl::desc("Insert calls to __dfsan_*_callback functions on data events."),
     cl::Hidden, cl::init(false));
 
-// Use a distinct bit for each base label, enabling faster unions with less
-// instrumentation.  Limits the max number of base labels to 16.
-static cl::opt<bool> ClFast16Labels(
-    "dfsan-fast-16-labels",
-    cl::desc("Use more efficient instrumentation, limiting the number of "
-             "labels to 16."),
-    cl::Hidden, cl::init(false));
-
-// Use a distinct bit for each base label, enabling faster unions with less
-// instrumentation.  Limits the max number of base labels to 8.
-static cl::opt<bool> ClFast8Labels(
-    "dfsan-fast-8-labels",
-    cl::desc("Use more efficient instrumentation, limiting the number of "
-             "labels to 8."),
-    cl::Hidden, cl::init(false));
-
 // Controls whether the pass tracks the control flow of select instructions.
 static cl::opt<bool> ClTrackSelectControlFlow(
     "dfsan-track-select-control-flow",
     cl::desc("Propagate labels from condition values of select instructions "
              "to results."),
     cl::Hidden, cl::init(true));
 
 // TODO: This default value follows MSan. DFSan may use a different value.
return AttributeList::get(Ctx, CallSiteAttrs.getFnAttributes(),
                             CallSiteAttrs.getRetAttributes(),
                             llvm::makeArrayRef(ArgumentAttributes));
 }
 
 class DataFlowSanitizer {
   friend struct DFSanFunction;
   friend class DFSanVisitor;
 
+  enum { ShadowWidthBits = 8, ShadowWidthBytes = ShadowWidthBits / 8 };
+
   enum {
     OriginWidthBits = 32,
     OriginWidthBytes = OriginWidthBits / 8
   };
 
   /// Which ABI should be used for instrumented functions?
   enum InstrumentedABI {
     /// Argument and return value labels are passed through additional
enum WrapperKind {
     /// original function or provide its own implementation.  This is similar to
     /// the IA_Args ABI, except that IA_Args uses a struct return type to
     /// pass the return value shadow in a register, while WK_Custom uses an
     /// extra pointer argument to return the shadow.  This allows the wrapped
     /// form of the function type to be expressed in C.
     WK_Custom
   };
 
-  unsigned ShadowWidthBits;
-  unsigned ShadowWidthBytes;
-
   Module *Mod;
   LLVMContext *Ctx;
   Type *Int8Ptr;
   IntegerType *OriginTy;
   PointerType *OriginPtrTy;
-  ConstantInt *OriginBase;
   ConstantInt *ZeroOrigin;
   /// The shadow type for all primitive types and vector types.
   IntegerType *PrimitiveShadowTy;
   PointerType *PrimitiveShadowPtrTy;
   IntegerType *IntptrTy;
   ConstantInt *ZeroPrimitiveShadow;
   ConstantInt *ShadowPtrMask;
-  ConstantInt *ShadowPtrMul;
+  ConstantInt *ShadowBase;
+  ConstantInt *OriginBase;
   Constant *ArgTLS;
   ArrayType *ArgOriginTLSTy;
   Constant *ArgOriginTLS;
   Constant *RetvalTLS;
   Constant *RetvalOriginTLS;
   Constant *ExternalShadowMask;
-  FunctionType *DFSanUnionFnTy;
   FunctionType *DFSanUnionLoadFnTy;
   FunctionType *DFSanLoadLabelAndOriginFnTy;
   FunctionType *DFSanUnimplementedFnTy;
   FunctionType *DFSanSetLabelFnTy;
   FunctionType *DFSanNonzeroLabelFnTy;
   FunctionType *DFSanVarargWrapperFnTy;
   FunctionType *DFSanCmpCallbackFnTy;
   FunctionType *DFSanLoadStoreCallbackFnTy;
   FunctionType *DFSanMemTransferCallbackFnTy;
   FunctionType *DFSanChainOriginFnTy;
   FunctionType *DFSanChainOriginIfTaintedFnTy;
   FunctionType *DFSanMemOriginTransferFnTy;
   FunctionType *DFSanMaybeStoreOriginFnTy;
-  FunctionCallee DFSanUnionFn;
-  FunctionCallee DFSanCheckedUnionFn;
   FunctionCallee DFSanUnionLoadFn;
-  FunctionCallee DFSanUnionLoadFastLabelsFn;
   FunctionCallee DFSanLoadLabelAndOriginFn;
   FunctionCallee DFSanUnimplementedFn;
   FunctionCallee DFSanSetLabelFn;
   FunctionCallee DFSanNonzeroLabelFn;
   FunctionCallee DFSanVarargWrapperFn;
   FunctionCallee DFSanLoadCallbackFn;
   FunctionCallee DFSanStoreCallbackFn;
   FunctionCallee DFSanMemTransferCallbackFn;
class DataFlowSanitizer {
   void addGlobalNamePrefix(GlobalValue *GV);
   Function *buildWrapperFunction(Function *F, StringRef NewFName,
                                  GlobalValue::LinkageTypes NewFLink,
                                  FunctionType *NewFT);
   Constant *getOrBuildTrampolineFunction(FunctionType *FT, StringRef FName);
   void initializeCallbackFunctions(Module &M);
   void initializeRuntimeFunctions(Module &M);
   void injectMetadataGlobals(Module &M);
-
   bool init(Module &M);
 
   /// Advances \p OriginAddr to point to the next 32-bit origin and then loads
   /// from it. Returns the origin's loaded value.
   Value *loadNextOrigin(Instruction *Pos, Align OriginAlign,
                         Value **OriginAddr);
 
-  /// Returns whether fast8 or fast16 mode has been specified.
-  bool hasFastLabelsEnabled();
-
   /// Returns whether the given load byte size is amenable to inlined
   /// optimization patterns.
   bool hasLoadSizeForFastPath(uint64_t Size);
 
-  /// Returns whether the pass tracks origins. Support only fast16 mode in TLS
-  /// ABI mode.
+  /// Returns whether the pass tracks origins. Supports only TLS ABI mode.
   bool shouldTrackOrigins();
 
   /// Returns whether the pass tracks labels for struct fields and array
-  /// indices. Support only fast16 mode in TLS ABI mode.
+  /// indices. Supports only TLS ABI mode.
   bool shouldTrackFieldsAndIndices();
 
   /// Returns a zero constant with the shadow type of OrigTy.
   ///
   /// getZeroShadow({T1,T2,...}) = {getZeroShadow(T1),getZeroShadow(T2,...}
   /// getZeroShadow([n x T]) = [n x getZeroShadow(T)]
   /// getZeroShadow(other type) = i16(0)
   ///
struct PHIFixupElement {
     PHINode *Phi;
     PHINode *ShadowPhi;
     PHINode *OriginPhi;
   };
   std::vector<PHIFixupElement> PHIFixups;
 
   DenseSet<Instruction *> SkipInsts;
   std::vector<Value *> NonZeroChecks;
-  bool AvoidNewBlocks;
 
   struct CachedShadow {
     BasicBlock *Block; // The block where Shadow is defined.
     Value *Shadow;
   };
   /// Maps a value to its latest shadow value in terms of domination tree.
   DenseMap<std::pair<Value *, Value *>, CachedShadow> CachedShadows;
   /// Maps a value to its latest collapsed shadow value it was converted to in
   /// terms of domination tree. When ClDebugNonzeroLabels is on, this cache is
   /// used at a post process where CFG blocks are split. So it does not cache
   /// BasicBlock like CachedShadows, but uses domination between values.
   DenseMap<Value *, Value *> CachedCollapsedShadows;
   DenseMap<Value *, std::set<Value *>> ShadowElements;
 
   DFSanFunction(DataFlowSanitizer &DFS, Function *F, bool IsNativeABI)
       : DFS(DFS), F(F), IA(DFS.getInstrumentedABI()), IsNativeABI(IsNativeABI) {
     DT.recalculate(*F);
-    // FIXME: Need to track down the register allocator issue which causes poor
-    // performance in pathological cases with large numbers of basic blocks.
-    AvoidNewBlocks = F->size() > 1000;
   }
 
   /// Computes the shadow address for a given function argument.
   ///
   /// Shadow = ArgTLS+ArgOffset.
   Value *getArgTLS(Type *T, unsigned ArgOffset, IRBuilder<> &IRB);
 
   /// Computes the shadow address for a return value.
private:
   Value *collapseAggregateShadow(AggregateType *AT, Value *Shadow,
                                  IRBuilder<> &IRB);
 
   Value *collapseToPrimitiveShadow(Value *Shadow, IRBuilder<> &IRB);
 
   /// Returns the shadow value of an argument A.
   Value *getShadowForTLSArgument(Argument *A);
 
-  /// The fast path of loading shadow in legacy mode.
-  Value *loadLegacyShadowFast(Value *ShadowAddr, uint64_t Size,
-                              Align ShadowAlign, Instruction *Pos);
-
-  /// The fast path of loading shadow in fast-16-label mode.
+  /// The fast path of loading shadows.
   std::pair<Value *, Value *>
-  loadFast16ShadowFast(Value *ShadowAddr, Value *OriginAddr, uint64_t Size,
-                       Align ShadowAlign, Align OriginAlign, Value *FirstOrigin,
-                       Instruction *Pos);
+  loadShadowFast(Value *ShadowAddr, Value *OriginAddr, uint64_t Size,
 
   Align getOriginAlign(Align InstAlignment);
 
   /// Because 4 contiguous bytes share one 4-byte origin, the most accurate load
   /// is __dfsan_load_label_and_origin. This function returns the union of all
   /// labels and the origin of the first taint label. However this is an
   /// additional call with many instructions. To ensure common cases are fast,
   /// checks if it is possible to load labels and origins without using the
private:
   void addOriginArguments(Function &F, CallBase &CB, std::vector<Value *> &Args,
                           IRBuilder<> &IRB);
 };
 
 } // end anonymous namespace
 
 DataFlowSanitizer::DataFlowSanitizer(
     const std::vector<std::string> &ABIListFiles) {
-  if (ClFast8Labels && ClFast16Labels) {
-    report_fatal_error(
-        "cannot set both -dfsan-fast-8-labels and -dfsan-fast-16-labels");
-  }
-
-  ShadowWidthBits = ClFast8Labels ? 8 : 16;
-  ShadowWidthBytes = ShadowWidthBits / 8;
-
   std::vector<std::string> AllABIListFiles(std::move(ABIListFiles));
   llvm::append_range(AllABIListFiles, ClABIListFiles);
   // FIXME: should we propagate vfs::FileSystem to this constructor?
   ABIList.set(
       SpecialCaseList::createOrDie(AllABIListFiles, *vfs::getRealFileSystem()));
 }
 
 FunctionType *DataFlowSanitizer::getArgsFunctionType(FunctionType *T) {
if (!isa<ArrayType>(T) && !isa<StructType>(T)) {
     if (const ConstantInt *CI = dyn_cast<ConstantInt>(V))
       return CI->isZero();
     return false;
   }
 
   return isa<ConstantAggregateZero>(V);
 }
 
-bool DataFlowSanitizer::hasFastLabelsEnabled() {
-  static const bool HasFastLabelsEnabled = ClFast8Labels || ClFast16Labels;
-  return HasFastLabelsEnabled;
-}
-
 bool DataFlowSanitizer::hasLoadSizeForFastPath(uint64_t Size) {
   uint64_t ShadowSize = Size * ShadowWidthBytes;
   return ShadowSize % 8 == 0 || ShadowSize == 4;
 }
 
 bool DataFlowSanitizer::shouldTrackOrigins() {
   static const bool ShouldTrackOrigins =
-      ClTrackOrigins && getInstrumentedABI() == DataFlowSanitizer::IA_TLS &&
-      hasFastLabelsEnabled();
+      ClTrackOrigins && getInstrumentedABI() == DataFlowSanitizer::IA_TLS;
   return ShouldTrackOrigins;
 }
 
 bool DataFlowSanitizer::shouldTrackFieldsAndIndices() {
-  return getInstrumentedABI() == DataFlowSanitizer::IA_TLS &&
-         hasFastLabelsEnabled();
+  return getInstrumentedABI() == DataFlowSanitizer::IA_TLS;
 }
 
 Constant *DataFlowSanitizer::getZeroShadow(Type *OrigTy) {
   if (!shouldTrackFieldsAndIndices())
     return ZeroPrimitiveShadow;
 
   if (!isa<ArrayType>(OrigTy) && !isa<StructType>(OrigTy))
     return ZeroPrimitiveShadow;
bool DataFlowSanitizer::init(Module &M) {
   Ctx = &M.getContext();
   Int8Ptr = Type::getInt8PtrTy(*Ctx);
   OriginTy = IntegerType::get(*Ctx, OriginWidthBits);
   OriginPtrTy = PointerType::getUnqual(OriginTy);
   PrimitiveShadowTy = IntegerType::get(*Ctx, ShadowWidthBits);
   PrimitiveShadowPtrTy = PointerType::getUnqual(PrimitiveShadowTy);
   IntptrTy = DL.getIntPtrType(*Ctx);
   ZeroPrimitiveShadow = ConstantInt::getSigned(PrimitiveShadowTy, 0);
-  ShadowPtrMul = ConstantInt::getSigned(IntptrTy, ShadowWidthBytes);
-  OriginBase = ConstantInt::get(IntptrTy, 0x200000000000LL);
   ZeroOrigin = ConstantInt::getSigned(OriginTy, 0);
 
+  // TODO: these should be platform-specific and set in the switch-stmt below.
+  ShadowBase = ConstantInt::get(IntptrTy, 0x100000008000LL);
+  OriginBase = ConstantInt::get(IntptrTy, 0x200000008000LL);
+
   switch (TargetTriple.getArch()) {
   case Triple::x86_64:
-    ShadowPtrMask = ClFast8Labels
-                        ? ConstantInt::getSigned(IntptrTy, ~0x600000000000LL)
-                        : ConstantInt::getSigned(IntptrTy, ~0x700000000000LL);
+    ShadowPtrMask = ConstantInt::getSigned(IntptrTy, ~0x600000000000LL);
     break;
   case Triple::mips64:
   case Triple::mips64el:
-    ShadowPtrMask = ClFast8Labels
-                        ? ConstantInt::getSigned(IntptrTy, ~0xE000000000LL)
-                        : ConstantInt::getSigned(IntptrTy, ~0xF000000000LL);
+    ShadowPtrMask = ConstantInt::getSigned(IntptrTy, ~0xE000000000LL);
     break;
   case Triple::aarch64:
   case Triple::aarch64_be:
     // AArch64 supports multiple VMAs and the shadow mask is set at runtime.
     DFSanRuntimeShadowMask = true;
     break;
   default:
     report_fatal_error("unsupported triple");
   }
 
-  Type *DFSanUnionArgs[2] = {PrimitiveShadowTy, PrimitiveShadowTy};
-  DFSanUnionFnTy =
-      FunctionType::get(PrimitiveShadowTy, DFSanUnionArgs, /*isVarArg=*/false);
   Type *DFSanUnionLoadArgs[2] = {PrimitiveShadowPtrTy, IntptrTy};
   DFSanUnionLoadFnTy = FunctionType::get(PrimitiveShadowTy, DFSanUnionLoadArgs,
                                          /*isVarArg=*/false);
   Type *DFSanLoadLabelAndOriginArgs[2] = {Int8Ptr, IntptrTy};
   DFSanLoadLabelAndOriginFnTy =
       FunctionType::get(IntegerType::get(*Ctx, 64), DFSanLoadLabelAndOriginArgs,
                         /*isVarArg=*/false);
   DFSanUnimplementedFnTy = FunctionType::get(

 
 // Initialize DataFlowSanitizer runtime functions and declare them in the module
 void DataFlowSanitizer::initializeRuntimeFunctions(Module &M) {
   {
     AttributeList AL;
     AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
                          Attribute::NoUnwind);
     AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
-                         Attribute::ReadNone);
-    AL = AL.addAttribute(M.getContext(), AttributeList::ReturnIndex,
-                         Attribute::ZExt);
-    AL = AL.addParamAttribute(M.getContext(), 0, Attribute::ZExt);
-    AL = AL.addParamAttribute(M.getContext(), 1, Attribute::ZExt);
-    DFSanUnionFn =
-        Mod->getOrInsertFunction("__dfsan_union", DFSanUnionFnTy, AL);
-  }
-  {
-    AttributeList AL;
-    AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
-                         Attribute::NoUnwind);
-    AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
-                         Attribute::ReadNone);
-    AL = AL.addAttribute(M.getContext(), AttributeList::ReturnIndex,
-                         Attribute::ZExt);
-    AL = AL.addParamAttribute(M.getContext(), 0, Attribute::ZExt);
-    AL = AL.addParamAttribute(M.getContext(), 1, Attribute::ZExt);
-    DFSanCheckedUnionFn =
-        Mod->getOrInsertFunction("dfsan_union", DFSanUnionFnTy, AL);
-  }
-  {
-    AttributeList AL;
-    AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
-                         Attribute::NoUnwind);
-    AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
                          Attribute::ReadOnly);
     AL = AL.addAttribute(M.getContext(), AttributeList::ReturnIndex,
                          Attribute::ZExt);
     DFSanUnionLoadFn =
         Mod->getOrInsertFunction("__dfsan_union_load", DFSanUnionLoadFnTy, AL);
   }
   {
     AttributeList AL;
     AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
                          Attribute::NoUnwind);
     AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
                          Attribute::ReadOnly);
     AL = AL.addAttribute(M.getContext(), AttributeList::ReturnIndex,
                          Attribute::ZExt);
-    DFSanUnionLoadFastLabelsFn = Mod->getOrInsertFunction(
-        "__dfsan_union_load_fast16labels", DFSanUnionLoadFnTy, AL);
-  }
-  {
-    AttributeList AL;
-    AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
-                         Attribute::NoUnwind);
-    AL = AL.addAttribute(M.getContext(), AttributeList::FunctionIndex,
-                         Attribute::ReadOnly);
-    AL = AL.addAttribute(M.getContext(), AttributeList::ReturnIndex,
-                         Attribute::ZExt);
     DFSanLoadLabelAndOriginFn = Mod->getOrInsertFunction(
         "__dfsan_load_label_and_origin", DFSanLoadLabelAndOriginFnTy, AL);
   }
   DFSanUnimplementedFn =
       Mod->getOrInsertFunction("__dfsan_unimplemented", DFSanUnimplementedFnTy);
   {
     AttributeList AL;
     AL = AL.addParamAttribute(M.getContext(), 0, Attribute::ZExt);
void DataFlowSanitizer::initializeRuntimeFunctions(Module &M) {
   {
     AttributeList AL;
     AL = AL.addParamAttribute(M.getContext(), 0, Attribute::ZExt);
     AL = AL.addParamAttribute(M.getContext(), 3, Attribute::ZExt);
     DFSanMaybeStoreOriginFn = Mod->getOrInsertFunction(
         "__dfsan_maybe_store_origin", DFSanMaybeStoreOriginFnTy, AL);
   }
 
-  DFSanRuntimeFunctions.insert(DFSanUnionFn.getCallee()->stripPointerCasts());
-  DFSanRuntimeFunctions.insert(
-      DFSanCheckedUnionFn.getCallee()->stripPointerCasts());
   DFSanRuntimeFunctions.insert(
       DFSanUnionLoadFn.getCallee()->stripPointerCasts());
   DFSanRuntimeFunctions.insert(
-      DFSanUnionLoadFastLabelsFn.getCallee()->stripPointerCasts());
-  DFSanRuntimeFunctions.insert(
       DFSanLoadLabelAndOriginFn.getCallee()->stripPointerCasts());
   DFSanRuntimeFunctions.insert(
       DFSanUnimplementedFn.getCallee()->stripPointerCasts());
   DFSanRuntimeFunctions.insert(
       DFSanSetLabelFn.getCallee()->stripPointerCasts());
   DFSanRuntimeFunctions.insert(
       DFSanNonzeroLabelFn.getCallee()->stripPointerCasts());
   DFSanRuntimeFunctions.insert(
else
     ShadowPtrMaskValue = ShadowPtrMask;
   return IRB.CreateAnd(IRB.CreatePtrToInt(Addr, IntptrTy),
                        IRB.CreatePtrToInt(ShadowPtrMaskValue, IntptrTy));
 }
 
 std::pair<Value *, Value *>
 DataFlowSanitizer::getShadowOriginAddress(Value *Addr, Align InstAlignment,
                                           Instruction *Pos) {
-  // Returns ((Addr & shadow_mask) + origin_base) & ~4UL
+  // Returns ((Addr & shadow_mask) + origin_base - shadow_base) & ~4UL
   IRBuilder<> IRB(Pos);
   Value *ShadowOffset = getShadowOffset(Addr, IRB);
   Value *ShadowPtr = getShadowAddress(Addr, Pos, ShadowOffset);
   Value *OriginPtr = nullptr;
   if (shouldTrackOrigins()) {
-    Value *OriginLong = IRB.CreateAdd(ShadowOffset, OriginBase);
+    static Value *OriginByShadowOffset = ConstantInt::get(
+        IntptrTy, OriginBase->getZExtValue() - ShadowBase->getZExtValue());
+
+    Value *OriginLong = IRB.CreateAdd(ShadowOffset, OriginByShadowOffset);
     const Align Alignment = llvm::assumeAligned(InstAlignment.value());
     // When alignment is >= 4, Addr must be aligned to 4, otherwise it is UB.
     // So Mask is unnecessary.
     if (Alignment < MinOriginAlignment) {
       uint64_t Mask = MinOriginAlignment.value() - 1;
       OriginLong = IRB.CreateAnd(OriginLong, ConstantInt::get(IntptrTy, ~Mask));
     }
     OriginPtr = IRB.CreateIntToPtr(OriginLong, OriginPtrTy);
   }
   return {ShadowPtr, OriginPtr};
 }
 
 Value *DataFlowSanitizer::getShadowAddress(Value *Addr, Instruction *Pos,
                                            Value *ShadowOffset) {
   IRBuilder<> IRB(Pos);
-
-  if (!ShadowPtrMul->isOne())
-    ShadowOffset = IRB.CreateMul(ShadowOffset, ShadowPtrMul);
-
   return IRB.CreateIntToPtr(ShadowOffset, PrimitiveShadowPtrTy);
 }
 
 Value *DataFlowSanitizer::getShadowAddress(Value *Addr, Instruction *Pos) {
-  // Returns (Addr & shadow_mask) x 2
+  // Returns (Addr & shadow_mask)
   IRBuilder<> IRB(Pos);
   Value *ShadowOffset = getShadowOffset(Addr, IRB);
   return getShadowAddress(Addr, Pos, ShadowOffset);
 }
 
 Value *DFSanFunction::combineShadowsThenConvert(Type *T, Value *V1, Value *V2,
                                                 Instruction *Pos) {
   Value *PrimitiveValue = combineShadows(V1, V2, Pos);
Value *DFSanFunction::combineShadows(Value *V1, Value *V2, Instruction *Pos) {
   if (CCS.Block && DT.dominates(CCS.Block, Pos->getParent()))
     return CCS.Shadow;
 
   // Converts inputs shadows to shadows with primitive types.
   Value *PV1 = collapseToPrimitiveShadow(V1, Pos);
   Value *PV2 = collapseToPrimitiveShadow(V2, Pos);
 
   IRBuilder<> IRB(Pos);
-  if (DFS.hasFastLabelsEnabled()) {
-    CCS.Block = Pos->getParent();
-    CCS.Shadow = IRB.CreateOr(PV1, PV2);
-  } else if (AvoidNewBlocks) {
-    CallInst *Call = IRB.CreateCall(DFS.DFSanCheckedUnionFn, {PV1, PV2});
-    Call->addAttribute(AttributeList::ReturnIndex, Attribute::ZExt);
-    Call->addParamAttr(0, Attribute::ZExt);
-    Call->addParamAttr(1, Attribute::ZExt);
-
-    CCS.Block = Pos->getParent();
-    CCS.Shadow = Call;
-  } else {
-    BasicBlock *Head = Pos->getParent();
-    Value *Ne = IRB.CreateICmpNE(PV1, PV2);
-    BranchInst *BI = cast<BranchInst>(SplitBlockAndInsertIfThen(
-        Ne, Pos, /*Unreachable=*/false, DFS.ColdCallWeights, &DT));
-    IRBuilder<> ThenIRB(BI);
-    CallInst *Call = ThenIRB.CreateCall(DFS.DFSanUnionFn, {PV1, PV2});
-    Call->addAttribute(AttributeList::ReturnIndex, Attribute::ZExt);
-    Call->addParamAttr(0, Attribute::ZExt);
-    Call->addParamAttr(1, Attribute::ZExt);
-
-    BasicBlock *Tail = BI->getSuccessor(0);
-    PHINode *Phi =
-        PHINode::Create(DFS.PrimitiveShadowTy, 2, "", &Tail->front());
-    Phi->addIncoming(Call, Call->getParent());
-    Phi->addIncoming(PV1, Head);
-
-    CCS.Block = Tail;
-    CCS.Shadow = Phi;
-  }
 
   std::set<Value *> UnionElems;
   if (V1Elems != ShadowElements.end()) {
     UnionElems = V1Elems->second;
   } else {
     UnionElems.insert(V1);
   }
   if (V2Elems != ShadowElements.end()) {

 Value *DataFlowSanitizer::loadNextOrigin(Instruction *Pos, Align OriginAlign,
                                          Value **OriginAddr) {
   IRBuilder<> IRB(Pos);
   *OriginAddr =
       IRB.CreateGEP(OriginTy, *OriginAddr, ConstantInt::get(IntptrTy, 1));
   return IRB.CreateAlignedLoad(OriginTy, *OriginAddr, OriginAlign);
 }
 
-std::pair<Value *, Value *> DFSanFunction::loadFast16ShadowFast(
+std::pair<Value *, Value *> DFSanFunction::loadShadowFast(
     Value *ShadowAddr, Value *OriginAddr, uint64_t Size, Align ShadowAlign,
     Align OriginAlign, Value *FirstOrigin, Instruction *Pos) {
   const bool ShouldTrackOrigins = DFS.shouldTrackOrigins();
   const uint64_t ShadowSize = Size * DFS.ShadowWidthBytes;
 
   assert(Size >= 4 && "Not large enough load size for fast path!");
 
   // Used for origin tracking.
   std::vector<Value *> Shadows;
   std::vector<Value *> Origins;
 
   // Load instructions in LLVM can have arbitrary byte sizes (e.g., 3, 12, 20)
   // but this function is only used in a subset of cases that make it possible
   // to optimize the instrumentation.
   //
   // Specifically, when the shadow size in bytes (i.e., loaded bytes x shadow
   // per byte) is either:
   // - a multiple of 8  (common)
-  // - equal to 4       (only for load32 in fast-8 mode)
+  // - equal to 4       (only for load32)
   //
   // For the second case, we can fit the wide shadow in a 32-bit integer. In all
   // other cases, we use a 64-bit integer to hold the wide shadow.
   Type *WideShadowTy =
       ShadowSize == 4 ? Type::getInt32Ty(*DFS.Ctx) : Type::getInt64Ty(*DFS.Ctx);
 
   IRBuilder<> IRB(Pos);
   Value *WideAddr = IRB.CreateBitCast(ShadowAddr, WideShadowTy->getPointerTo());
std::pair<Value *, Value *> DFSanFunction::loadShadowFast(
   }
   return {IRB.CreateTrunc(CombinedWideShadow, DFS.PrimitiveShadowTy),
           ShouldTrackOrigins
               ? combineOrigins(Shadows, Origins, Pos,
                                ConstantInt::getSigned(IRB.getInt64Ty(), 0))
               : DFS.ZeroOrigin};
 }
 
-Value *DFSanFunction::loadLegacyShadowFast(Value *ShadowAddr, uint64_t Size,
-                                           Align ShadowAlign,
-                                           Instruction *Pos) {
-  // Fast path for the common case where each byte has identical shadow: load
-  // shadow 64 (or 32) bits at a time, fall out to a __dfsan_union_load call if
-  // any shadow is non-equal.
-  BasicBlock *FallbackBB = BasicBlock::Create(*DFS.Ctx, "", F);
-  IRBuilder<> FallbackIRB(FallbackBB);
-  CallInst *FallbackCall = FallbackIRB.CreateCall(
-      DFS.DFSanUnionLoadFn, {ShadowAddr, ConstantInt::get(DFS.IntptrTy, Size)});
-  FallbackCall->addAttribute(AttributeList::ReturnIndex, Attribute::ZExt);
-
-  const uint64_t ShadowSize = Size * DFS.ShadowWidthBytes;
-  assert(Size >= 4 && "Not large enough load size for fast path!");
-
-  // Same as in loadFast16AShadowsFast. In the case of load32, we can fit the
-  // wide shadow in a 32-bit integer instead.
-  Type *WideShadowTy =
-      ShadowSize == 4 ? Type::getInt32Ty(*DFS.Ctx) : Type::getInt64Ty(*DFS.Ctx);
-
-  // Compare each of the shadows stored in the loaded 64 bits to each other,
-  // by computing (WideShadow rotl ShadowWidthBits) == WideShadow.
-  IRBuilder<> IRB(Pos);
-  unsigned WideShadowBitWidth = WideShadowTy->getIntegerBitWidth();
-  Value *WideAddr = IRB.CreateBitCast(ShadowAddr, WideShadowTy->getPointerTo());
-  Value *WideShadow =
-      IRB.CreateAlignedLoad(WideShadowTy, WideAddr, ShadowAlign);
-  Value *TruncShadow = IRB.CreateTrunc(WideShadow, DFS.PrimitiveShadowTy);
-  Value *ShlShadow = IRB.CreateShl(WideShadow, DFS.ShadowWidthBits);
-  Value *ShrShadow =
-      IRB.CreateLShr(WideShadow, WideShadowBitWidth - DFS.ShadowWidthBits);
-  Value *RotShadow = IRB.CreateOr(ShlShadow, ShrShadow);
-  Value *ShadowsEq = IRB.CreateICmpEQ(WideShadow, RotShadow);
-
-  BasicBlock *Head = Pos->getParent();
-  BasicBlock *Tail = Head->splitBasicBlock(Pos->getIterator());
-
-  if (DomTreeNode *OldNode = DT.getNode(Head)) {
-    std::vector<DomTreeNode *> Children(OldNode->begin(), OldNode->end());
-
-    DomTreeNode *NewNode = DT.addNewBlock(Tail, Head);
-    for (auto *Child : Children)
-      DT.changeImmediateDominator(Child, NewNode);
-  }
-
-  // In the following code LastBr will refer to the previous basic block's
-  // conditional branch instruction, whose true successor is fixed up to point
-  // to the next block during the loop below or to the tail after the final
-  // iteration.
-  BranchInst *LastBr = BranchInst::Create(FallbackBB, FallbackBB, ShadowsEq);
-  ReplaceInstWithInst(Head->getTerminator(), LastBr);
-  DT.addNewBlock(FallbackBB, Head);
-
-  const uint64_t BytesPerWideShadow = WideShadowBitWidth / DFS.ShadowWidthBits;
-
-  for (uint64_t ByteOfs = BytesPerWideShadow; ByteOfs < Size;
-       ByteOfs += BytesPerWideShadow) {
-    BasicBlock *NextBB = BasicBlock::Create(*DFS.Ctx, "", F);
-    DT.addNewBlock(NextBB, LastBr->getParent());
-    IRBuilder<> NextIRB(NextBB);
-    WideAddr = NextIRB.CreateGEP(WideShadowTy, WideAddr,
-                                 ConstantInt::get(DFS.IntptrTy, 1));
-    Value *NextWideShadow =
-        NextIRB.CreateAlignedLoad(WideShadowTy, WideAddr, ShadowAlign);
-    ShadowsEq = NextIRB.CreateICmpEQ(WideShadow, NextWideShadow);
-    LastBr->setSuccessor(0, NextBB);
-    LastBr = NextIRB.CreateCondBr(ShadowsEq, FallbackBB, FallbackBB);
-  }
-
-  LastBr->setSuccessor(0, Tail);
-  FallbackIRB.CreateBr(Tail);
-  PHINode *Shadow =
-      PHINode::Create(DFS.PrimitiveShadowTy, 2, "", &Tail->front());
-  Shadow->addIncoming(FallbackCall, FallbackBB);
-  Shadow->addIncoming(TruncShadow, LastBr->getParent());
-  return Shadow;
-}
-
 std::pair<Value *, Value *> DFSanFunction::loadShadowOriginSansLoadTracking(
     Value *Addr, uint64_t Size, Align InstAlignment, Instruction *Pos) {
   const bool ShouldTrackOrigins = DFS.shouldTrackOrigins();
 
   // Non-escaped loads.
   if (AllocaInst *AI = dyn_cast<AllocaInst>(Addr)) {
     const auto SI = AllocaShadowMap.find(AI);
     if (SI != AllocaShadowMap.end()) {
case 2: {
     Value *Load =
         IRB.CreateAlignedLoad(DFS.PrimitiveShadowTy, ShadowAddr, ShadowAlign);
     Value *Load1 =
         IRB.CreateAlignedLoad(DFS.PrimitiveShadowTy, ShadowAddr1, ShadowAlign);
     return {combineShadows(Load, Load1, Pos), Origin};
   }
   }
   bool HasSizeForFastPath = DFS.hasLoadSizeForFastPath(Size);
-  bool HasFastLabelsEnabled = DFS.hasFastLabelsEnabled();
 
-  if (HasFastLabelsEnabled && HasSizeForFastPath)
-    return loadFast16ShadowFast(ShadowAddr, OriginAddr, Size, ShadowAlign,
-                                OriginAlign, Origin, Pos);
+  if (HasSizeForFastPath)
+    return loadShadowFast(ShadowAddr, OriginAddr, Size, ShadowAlign,
 
-  if (!AvoidNewBlocks && HasSizeForFastPath)
-    return {loadLegacyShadowFast(ShadowAddr, Size, ShadowAlign, Pos), Origin};
-
   IRBuilder<> IRB(Pos);
-  FunctionCallee &UnionLoadFn = HasFastLabelsEnabled
-                                    ? DFS.DFSanUnionLoadFastLabelsFn
-                                    : DFS.DFSanUnionLoadFn;
   CallInst *FallbackCall = IRB.CreateCall(
-      UnionLoadFn, {ShadowAddr, ConstantInt::get(DFS.IntptrTy, Size)});
+      DFS.DFSanUnionLoadFn, {ShadowAddr, ConstantInt::get(DFS.IntptrTy, Size)});
   FallbackCall->addAttribute(AttributeList::ReturnIndex, Attribute::ZExt);
   return {FallbackCall, Origin};
 }
 
 std::pair<Value *, Value *> DFSanFunction::loadShadowOrigin(Value *Addr,
                                                             uint64_t Size,
                                                             Align InstAlignment,
                                                             Instruction *Pos) {

llvm/test/Instrumentation/DataFlowSanitizer/abilist.ll

 ; RUN: opt < %s -dfsan -dfsan-args-abi -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-args-abi -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-args-abi -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 ; CHECK: i32 @discard(i32 %a, i32 %b)
 define i32 @discard(i32 %a, i32 %b) {

llvm/test/Instrumentation/DataFlowSanitizer/abilist_aggregate.ll

-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s --check-prefixes=CHECK,TLS_ABI
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s --check-prefixes=CHECK,TLS_ABI
-; RUN: opt < %s -dfsan -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s --check-prefixes=CHECK,LEGACY
+; RUN: opt < %s -dfsan -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s --check-prefixes=CHECK,TLS_ABI
 ; RUN: opt < %s -dfsan -dfsan-args-abi -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s --check-prefixes=CHECK,ARGS_ABI
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 ; TLS_ABI: define { i1, i7 } @functional({ i32, i1 } %a, [2 x i7] %b)
define {i1, i7} @call_functional({i32, i1} %a, [2 x i7] %b) {
   ; TLS_ABI-NEXT: %[[#REG+5]] = extractvalue [2 x i[[#SBITS]]] %[[#REG]], 0
   ; TLS_ABI-NEXT: %[[#REG+6]] = extractvalue [2 x i[[#SBITS]]] %[[#REG]], 1
   ; TLS_ABI-NEXT: %[[#REG+7]] = or i[[#SBITS]] %[[#REG+5]], %[[#REG+6]]
   ; TLS_ABI-NEXT: %[[#REG+8]] = or i[[#SBITS]] %[[#REG+4]], %[[#REG+7]]
   ; TLS_ABI-NEXT: %[[#REG+9]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } undef, i[[#SBITS]] %[[#REG+8]], 0
   ; TLS_ABI-NEXT: %[[#REG+10]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } %[[#REG+9]], i[[#SBITS]] %[[#REG+8]], 1
   ; TLS_ABI: store { i[[#SBITS]], i[[#SBITS]] } %[[#REG+10]], { i[[#SBITS]], i[[#SBITS]] }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN]]
 
-  ; LEGACY: @"dfs$call_functional"
-  ; LEGACY: [[B:%.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; LEGACY: [[A:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; LEGACY: [[U:%.*]] = call zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext [[A]], i[[#SBITS]] zeroext [[B]])
-  ; LEGACY: [[PH:%.*]] = phi i[[#SBITS]] [ [[U]], {{.*}} ], [ [[A]], {{.*}} ]
-  ; LEGACY: store i[[#SBITS]] [[PH]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-
-  ; ARGS_ABI: @"dfs$call_functional"
-  ; ARGS_ABI: [[U:%.*]]  = call zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext %2, i[[#SBITS]] zeroext %3)
-  ; ARGS_ABI: [[PH:%.*]] = phi i[[#SBITS]] [ %7, {{.*}} ], [ %2, {{.*}} ]
-  ; ARGS_ABI: [[R0:%.*]] = insertvalue { { i1, i7 }, i[[#SBITS]] } undef, { i1, i7 } %r, 0
-  ; ARGS_ABI: [[R1:%.*]] = insertvalue { { i1, i7 }, i[[#SBITS]] } [[R0]], i[[#SBITS]] [[PH]], 1
-  ; ARGS_ABI: ret { { i1, i7 }, i[[#SBITS]] } [[R1]]
+  ; ARGS_ABI: @"dfs$call_functional"({ i32, i1 } %0, [2 x i7] %1, i[[#SBITS]] %2, i[[#SBITS]] %3)
+  ; ARGS_ABI: %[[#U:]]  = or i[[#SBITS]] %2, %3
+  ; ARGS_ABI: %r = call { i1, i7 } @functional({ i32, i1 } %0, [2 x i7] %1)
+  ; ARGS_ABI: %[[#R:]] = insertvalue { { i1, i7 }, i[[#SBITS]] } undef, { i1, i7 } %r, 0
+  ; ARGS_ABI: %[[#R+1]] = insertvalue { { i1, i7 }, i[[#SBITS]] } %[[#R]], i[[#SBITS]] %[[#U]], 1
+  ; ARGS_ABI: ret { { i1, i7 }, i[[#SBITS]] } %[[#R+1]]
 
   %r = call {i1, i7} @functional({i32, i1} %a, [2 x i7] %b)
   ret {i1, i7} %r
 }
 
 ; TLS_ABI: define { i1, i7 } @discard({ i32, i1 } %a, [2 x i7] %b)
 define {i1, i7} @discard({i32, i1} %a, [2 x i7] %b) {
   %a1 = extractvalue {i32, i1} %a, 1

llvm/test/Instrumentation/DataFlowSanitizer/arith.ll

 ; RUN: opt < %s -dfsan -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define i8 @add(i8 %a, i8 %b) {
   ; CHECK: @"dfs$add"
-  ; CHECK-DAG: %[[ALABEL:.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[ARGTLSTYPE:\[100 x i64\]]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; CHECK-DAG: %[[BLABEL:.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[ARGTLSTYPE]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK: %[[UNION:.*]] = call zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext %[[ALABEL]], i[[#SBITS]] zeroext %[[BLABEL]])
-  ; CHECK: %[[ADDLABEL:.*]] = phi i[[#SBITS]] [ %[[UNION]], {{.*}} ], [ %[[ALABEL]], {{.*}} ]
-  ; CHECK: add i8
-  ; CHECK: store i[[#SBITS]] %[[ADDLABEL]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK: ret i8
+  ; CHECK-DAG: %[[#ALABEL:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[ARGTLSTYPE:\[100 x i64\]]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
+  ; CHECK-DAG: %[[#BLABEL:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[ARGTLSTYPE]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK: %[[#UNION:]] = or i[[#SBITS]] %[[#ALABEL]], %[[#BLABEL]]
+  ; CHECK: %c = add i8 %a, %b
+  ; CHECK: store i[[#SBITS]] %[[#UNION]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK: ret i8 %c
   %c = add i8 %a, %b
   ret i8 %c
 }
 
 define i8 @sub(i8 %a, i8 %b) {
   ; CHECK: @"dfs$sub"
   ; CHECK: load{{.*}}__dfsan_arg_tls
   ; CHECK: load{{.*}}__dfsan_arg_tls
-  ; CHECK: call{{.*}}__dfsan_union
-  ; CHECK: sub i8
+  ; CHECK: or i[[#SBITS]]
+  ; CHECK: %c = sub i8 %a, %b
   ; CHECK: store{{.*}}__dfsan_retval_tls
-  ; CHECK: ret i8
+  ; CHECK: ret i8 %c
   %c = sub i8 %a, %b
   ret i8 %c
 }
 
 define i8 @mul(i8 %a, i8 %b) {
   ; CHECK: @"dfs$mul"
   ; CHECK: load{{.*}}__dfsan_arg_tls
   ; CHECK: load{{.*}}__dfsan_arg_tls
-  ; CHECK: call{{.*}}__dfsan_union
-  ; CHECK: mul i8
+  ; CHECK: or i[[#SBITS]]
+  ; CHECK: %c = mul i8 %a, %b
   ; CHECK: store{{.*}}__dfsan_retval_tls
-  ; CHECK: ret i8
+  ; CHECK: ret i8 %c
   %c = mul i8 %a, %b
   ret i8 %c
 }
 
 define i8 @sdiv(i8 %a, i8 %b) {
   ; CHECK: @"dfs$sdiv"
   ; CHECK: load{{.*}}__dfsan_arg_tls
   ; CHECK: load{{.*}}__dfsan_arg_tls
-  ; CHECK: call{{.*}}__dfsan_union
-  ; CHECK: sdiv i8
+  ; CHECK: or i[[#SBITS]]
+  ; CHECK: %c = sdiv i8 %a, %b
   ; CHECK: store{{.*}}__dfsan_retval_tls
-  ; CHECK: ret i8
+  ; CHECK: ret i8 %c
   %c = sdiv i8 %a, %b
   ret i8 %c
 }
 
 define i8 @udiv(i8 %a, i8 %b) {
   ; CHECK: @"dfs$udiv"
   ; CHECK: load{{.*}}__dfsan_arg_tls
   ; CHECK: load{{.*}}__dfsan_arg_tls
-  ; CHECK: call{{.*}}__dfsan_union
-  ; CHECK: udiv i8
+  ; CHECK: or i[[#SBITS]]
+  ; CHECK: %c = udiv i8 %a, %b
   ; CHECK: store{{.*}}__dfsan_retval_tls
-  ; CHECK: ret i8
+  ; CHECK: ret i8 %c
   %c = udiv i8 %a, %b
   ret i8 %c
 }
 
 define double @fneg(double %a) {
   ; CHECK: @"dfs$fneg"
   ; CHECK: load{{.*}}__dfsan_arg_tls
-  ; CHECK: fneg double
+  ; CHECK: %c = fneg double %a
   ; CHECK: store{{.*}}__dfsan_retval_tls
-  ; CHECK: ret double
+  ; CHECK: ret double %c
   %c = fneg double %a
   ret double %c
 }

llvm/test/Instrumentation/DataFlowSanitizer/array.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,LEGACY
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-event-callbacks=true -S | FileCheck %s --check-prefixes=CHECK,EVENT_CALLBACKS
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-event-callbacks=true -S | FileCheck %s --check-prefixes=CHECK,EVENT_CALLBACKS
+; RUN: opt < %s -dfsan -dfsan-event-callbacks=true -S | FileCheck %s --check-prefixes=CHECK,EVENT_CALLBACKS
 ; RUN: opt < %s -dfsan -dfsan-args-abi -S | FileCheck %s --check-prefixes=CHECK,ARGS_ABI
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,FAST
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefixes=CHECK,NO_COMBINE_LOAD_PTR
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-combine-pointer-labels-on-store=true -S | FileCheck %s --check-prefixes=CHECK,COMBINE_STORE_PTR
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-debug-nonzero-labels -S | FileCheck %s --check-prefixes=CHECK,DEBUG_NONZERO_LABELS
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -S | FileCheck %s --check-prefixes=CHECK,FAST
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefixes=CHECK,NO_COMBINE_LOAD_PTR
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-combine-pointer-labels-on-store=true -S | FileCheck %s --check-prefixes=CHECK,COMBINE_STORE_PTR
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-debug-nonzero-labels -S | FileCheck %s --check-prefixes=CHECK,DEBUG_NONZERO_LABELS
+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,FAST
+; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefixes=CHECK,NO_COMBINE_LOAD_PTR
+; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-store=true -S | FileCheck %s --check-prefixes=CHECK,COMBINE_STORE_PTR
+; RUN: opt < %s -dfsan -dfsan-debug-nonzero-labels -S | FileCheck %s --check-prefixes=CHECK,DEBUG_NONZERO_LABELS
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
define [1 x i1] @load_array1([1 x i1]* %p) {
 
   ; FAST: @"dfs$load_array1"
   ; FAST: [[P:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
   ; FAST: [[L:%.*]] = load i[[#SBITS]], i[[#SBITS]]* {{.*}}, align [[#SBYTES]]
   ; FAST: [[U:%.*]] = or i[[#SBITS]] [[L]], [[P]]
   ; FAST: [[S1:%.*]] = insertvalue [1 x i[[#SBITS]]] undef, i[[#SBITS]] [[U]], 0
   ; FAST: store [1 x i[[#SBITS]]] [[S1]], [1 x i[[#SBITS]]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to [1 x i[[#SBITS]]]*), align [[ALIGN]]
 
-  ; LEGACY: @"dfs$load_array1"
-  ; LEGACY: [[P:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; LEGACY: [[L:%.*]] = load i[[#SBITS]], i[[#SBITS]]* {{.*}}, align [[#SBYTES]]
-  ; LEGACY: [[U:%.*]] = call zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext [[L]], i[[#SBITS]] zeroext [[P]])
-  ; LEGACY: [[PH:%.*]] = phi i[[#SBITS]] [ [[U]], {{.*}} ], [ [[L]], {{.*}} ]
-  ; LEGACY: store i[[#SBITS]] [[PH]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-
   %a = load [1 x i1], [1 x i1]* %p
   ret [1 x i1] %a
 }
 
 define [2 x i1] @load_array2([2 x i1]* %p) {
   ; NO_COMBINE_LOAD_PTR: @"dfs$load_array2"
   ; NO_COMBINE_LOAD_PTR: [[P1:%.*]] = getelementptr i[[#SBITS]], i[[#SBITS]]* [[P0:%.*]], i64 1
   ; NO_COMBINE_LOAD_PTR-DAG: [[E1:%.*]] = load i[[#SBITS]], i[[#SBITS]]* [[P1]], align [[#SBYTES]]
define [4 x i1] @load_array4([4 x i1]* %p) {
   ; FAST: [[T:%.*]] = trunc i[[#mul(4, SBITS)]] {{.*}} to i[[#SBITS]]
   ; FAST: [[O:%.*]] = or i[[#SBITS]] [[T]]
   ; FAST: [[S1:%.*]] = insertvalue [4 x i[[#SBITS]]] undef, i[[#SBITS]] [[O]], 0
   ; FAST: [[S2:%.*]] = insertvalue [4 x i[[#SBITS]]] [[S1]], i[[#SBITS]] [[O]], 1
   ; FAST: [[S3:%.*]] = insertvalue [4 x i[[#SBITS]]] [[S2]], i[[#SBITS]] [[O]], 2
   ; FAST: [[S4:%.*]] = insertvalue [4 x i[[#SBITS]]] [[S3]], i[[#SBITS]] [[O]], 3
   ; FAST: store [4 x i[[#SBITS]]] [[S4]], [4 x i[[#SBITS]]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to [4 x i[[#SBITS]]]*), align 2
 
-  ; LEGACY: @"dfs$load_array4"
-  ; LEGACY: [[P:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; LEGACY: [[PH1:%.*]] = phi i[[#SBITS]]
-  ; LEGACY: [[U:%.*]] = call zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext [[PH1]], i[[#SBITS]] zeroext [[P]])
-  ; LEGACY: [[PH:%.*]] = phi i[[#SBITS]] [ [[U]], {{.*}} ], [ [[PH1]], {{.*}} ]
-  ; LEGACY: store i[[#SBITS]] [[PH]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-
   %a = load [4 x i1], [4 x i1]* %p
   ret [4 x i1] %a
 }
 
 define i1 @extract_array([4 x i1] %a) {
   ; NO_COMBINE_LOAD_PTR: @"dfs$extract_array"
   ; NO_COMBINE_LOAD_PTR: [[AM:%.*]] = load [4 x i[[#SBITS]]], [4 x i[[#SBITS]]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to [4 x i[[#SBITS]]]*), align [[ALIGN:2]]
   ; NO_COMBINE_LOAD_PTR: [[EM:%.*]] = extractvalue [4 x i[[#SBITS]]] [[AM]], 2

 define void @store_zero_array([4 x i1]* %p) {
   ; FAST: @"dfs$store_zero_array"
   ; FAST: store i[[#mul(4, SBITS)]] 0, i[[#mul(4, SBITS)]]* {{.*}}
   store [4 x i1] zeroinitializer, [4 x i1]* %p
   ret void
 }
 
 define void @store_array2([2 x i1] %a, [2 x i1]* %p) {
-  ; LEGACY: @"dfs$store_array2"
-  ; LEGACY: [[S:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; LEGACY: [[SP0:%.*]] = getelementptr i[[#SBITS]], i[[#SBITS]]* [[SP:%.*]], i32 0
-  ; LEGACY: store i[[#SBITS]] [[S]], i[[#SBITS]]* [[SP0]], align [[#SBYTES]]
-  ; LEGACY: [[SP1:%.*]] = getelementptr i[[#SBITS]], i[[#SBITS]]* [[SP]], i32 1
-  ; LEGACY: store i[[#SBITS]] [[S]], i[[#SBITS]]* [[SP1]], align [[#SBYTES]]
-
   ; EVENT_CALLBACKS: @"dfs$store_array2"
   ; EVENT_CALLBACKS: [[E12:%.*]] = or i[[#SBITS]]
   ; EVENT_CALLBACKS: [[P:%.*]] = bitcast [2 x i1]* %p to i8*
   ; EVENT_CALLBACKS: call void @__dfsan_store_callback(i[[#SBITS]] [[E12]], i8* [[P]])
 
   ; FAST: @"dfs$store_array2"
   ; FAST: [[S:%.*]] = load [2 x i[[#SBITS]]], [2 x i[[#SBITS]]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to [2 x i[[#SBITS]]]*), align [[ALIGN:2]]
   ; FAST: [[E1:%.*]] = extractvalue [2 x i[[#SBITS]]] [[S]], 0

llvm/test/Instrumentation/DataFlowSanitizer/atomics.ll

-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,CHECK16
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true -dfsan-instrument-with-call-threshold=0 -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,CHECK16,CHECK_ORIGIN
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -dfsan-instrument-with-call-threshold=0 -S | FileCheck %s --check-prefixes=CHECK,CHECK16,CHECK_ORIGIN
+; RUN: opt < %s -dfsan -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1 -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN
+; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-instrument-with-call-threshold=0 -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define i32 @AtomicRmwXchg(i32* %p, i32 %x) {
 entry:
   ; COMM: atomicrmw xchg: store clean shadow/origin, return clean shadow/origin
 
   ; CHECK-LABEL:       @"dfs$AtomicRmwXchg"
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT:      %[[#SHADOW_ADDR:INTP+2]] = mul i64 %[[#INTP+1]], 2
   ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK-NEXT:        atomicrmw xchg i32* %p, i32 %x seq_cst
   ; CHECK-NEXT:        store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ; CHECK_ORIGIN-NEXT: store i32 0, i32* @__dfsan_retval_origin_tls, align 4
   ; CHECK-NEXT:        ret i32
 
   %0 = atomicrmw xchg i32* %p, i32 %x seq_cst
   ret i32 %0
 }
 
 define i32 @AtomicRmwMax(i32* %p, i32 %x) {
   ; COMM: atomicrmw max: exactly the same as above
 
   ; CHECK-LABEL:       @"dfs$AtomicRmwMax"
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT:      %[[#SHADOW_ADDR:INTP+2]] = mul i64 %[[#INTP+1]], 2
   ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK-NEXT:        atomicrmw max i32* %p, i32 %x seq_cst
   ; CHECK-NEXT:        store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ; CHECK_ORIGIN-NEXT: store i32 0, i32* @__dfsan_retval_origin_tls, align 4
   ; CHECK-NEXT:        ret i32
 
 entry:
   %0 = atomicrmw max i32* %p, i32 %x seq_cst
   ret i32 %0
 }
 
 
 define i32 @Cmpxchg(i32* %p, i32 %a, i32 %b) {
   ; COMM: cmpxchg: store clean shadow/origin, return clean shadow/origin
 
   ; CHECK-LABEL:       @"dfs$Cmpxchg"
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT:      %[[#SHADOW_ADDR:INTP+2]] = mul i64 %[[#INTP+1]], 2
   ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK-NEXT:        %pair = cmpxchg i32* %p, i32 %a, i32 %b seq_cst seq_cst
   ; CHECK:             store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ; CHECK_ORIGIN-NEXT: store i32 0, i32* @__dfsan_retval_origin_tls, align 4
   ; CHECK-NEXT:        ret i32
 
 entry:
   %pair = cmpxchg i32* %p, i32 %a, i32 %b seq_cst seq_cst
   %0 = extractvalue { i32, i1 } %pair, 0
   ret i32 %0
 }
 
 
 define i32 @CmpxchgMonotonic(i32* %p, i32 %a, i32 %b) {
   ; COMM: relaxed cmpxchg: bump up to "release monotonic"
 
   ; CHECK-LABEL:       @"dfs$CmpxchgMonotonic"
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT:      %[[#SHADOW_ADDR:INTP+2]] = mul i64 %[[#INTP+1]], 2
   ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK-NEXT:        %pair = cmpxchg i32* %p, i32 %a, i32 %b release monotonic
   ; CHECK:             store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ; CHECK_ORIGIN-NEXT: store i32 0, i32* @__dfsan_retval_origin_tls, align 4
   ; CHECK-NEXT:        ret i32
 
define void @AtomicStore(i32* %p, i32 %x) {
   ; COMM: atomic store: store clean shadow value before app value
 
   ; CHECK-LABEL:       @"dfs$AtomicStore"
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK_ORIGIN-NOT:  35184372088832
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT:      %[[#SHADOW_ADDR:INTP+2]] = mul i64 %[[#INTP+1]], 2
   ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK:             store atomic i32 %x, i32* %p seq_cst, align 16
   ; CHECK:             ret void
 
 entry:
   store atomic i32 %x, i32* %p seq_cst, align 16
   ret void
 }
 
 define void @AtomicStoreRelease(i32* %p, i32 %x) {
   ; COMM: atomic store: store clean shadow value before app value
 
   ; CHECK-LABEL:       @"dfs$AtomicStoreRelease"
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK_ORIGIN-NOT:  35184372088832
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT:      %[[#SHADOW_ADDR:INTP+2]] = mul i64 %[[#INTP+1]], 2
   ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK:             store atomic i32 %x, i32* %p release, align 16
   ; CHECK:             ret void
 
 entry:
   store atomic i32 %x, i32* %p release, align 16
   ret void
 }
 
 define void @AtomicStoreMonotonic(i32* %p, i32 %x) {
   ; COMM: atomic store monotonic: bumped up to store release
 
   ; CHECK-LABEL:       @"dfs$AtomicStoreMonotonic"
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK_ORIGIN-NOT:  35184372088832
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT:      %[[#SHADOW_ADDR:INTP+2]] = mul i64 %[[#INTP+1]], 2
   ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK:             store atomic i32 %x, i32* %p release, align 16
   ; CHECK:             ret void
 
 entry:
   store atomic i32 %x, i32* %p monotonic, align 16
   ret void
 }
 
 define void @AtomicStoreUnordered(i32* %p, i32 %x) {
   ; COMM: atomic store unordered: bumped up to store release
 
   ; CHECK-LABEL: @"dfs$AtomicStoreUnordered"
   ; CHECK-NOT:         @__dfsan_arg_origin_tls
   ; CHECK-NOT:         @__dfsan_arg_tls
   ; CHECK_ORIGIN-NOT:  35184372088832
   ; CHECK:             %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:        %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT:      %[[#SHADOW_ADDR:INTP+2]] = mul i64 %[[#INTP+1]], 2
   ; CHECK-NEXT:        %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:        %[[#SHADOW_PTR64:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#NUM_BITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:        store i[[#NUM_BITS]] 0, i[[#NUM_BITS]]* %[[#SHADOW_PTR64]], align [[#SBYTES]]
   ; CHECK:             store atomic i32 %x, i32* %p release, align 16
   ; CHECK:             ret void
 
 entry:
   store atomic i32 %x, i32* %p unordered, align 16
   ret void
 }

llvm/test/Instrumentation/DataFlowSanitizer/basic.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,CHECK_NO_ORIGIN -DSHADOW_MASK=-123145302310913
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN -DSHADOW_MASK=-123145302310913
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN -DSHADOW_MASK=-105553116266497
+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,CHECK_NO_ORIGIN -DSHADOW_MASK=-105553116266497   --dump-input-context=100
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN -DSHADOW_MASK=-105553116266497  --dump-input-context=100
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [100 x i64]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [100 x i64]
 ; CHECK: @__dfsan_arg_origin_tls = external thread_local(initialexec) global [200 x i32]
 ; CHECK: @__dfsan_retval_origin_tls = external thread_local(initialexec) global i32
 ; CHECK_NO_ORIGIN: @__dfsan_track_origins = weak_odr constant i32 0
define void @store(i8* %p) {
   ret void
 }
 
 ; CHECK: declare void @__dfsan_load_callback(i[[#SBITS]], i8*)
 ; CHECK: declare void @__dfsan_store_callback(i[[#SBITS]], i8*)
 ; CHECK: declare void @__dfsan_mem_transfer_callback(i[[#SBITS]]*, i64)
 ; CHECK: declare void @__dfsan_cmp_callback(i[[#SBITS]])
 
-; CHECK: ; Function Attrs: nounwind readnone
-; CHECK-NEXT: declare zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext, i[[#SBITS]] zeroext) #0
-
-; CHECK: ; Function Attrs: nounwind readnone
-; CHECK-NEXT: declare zeroext i[[#SBITS]] @dfsan_union(i[[#SBITS]] zeroext, i[[#SBITS]] zeroext) #0
-
-; CHECK: ; Function Attrs: nounwind readonly
-; CHECK-NEXT: declare zeroext i[[#SBITS]] @__dfsan_union_load(i[[#SBITS]]*, i64) #1
-
 ; CHECK: ; Function Attrs: nounwind readonly
-; CHECK-NEXT: declare zeroext i[[#SBITS]] @__dfsan_union_load_fast16labels(i[[#SBITS]]*, i64) #1
+; CHECK-NEXT: declare zeroext i[[#SBITS]] @__dfsan_union_load(i[[#SBITS]]*, i64)
 
 ; CHECK: ; Function Attrs: nounwind readonly
-; CHECK-NEXT: declare zeroext i64 @__dfsan_load_label_and_origin(i8*, i64) #1
+; CHECK-NEXT: declare zeroext i64 @__dfsan_load_label_and_origin(i8*, i64)
 
 ; CHECK: declare void @__dfsan_unimplemented(i8*)
 ; CHECK: declare void @__dfsan_set_label(i[[#SBITS]] zeroext, i32 zeroext, i8*, i64)
 ; CHECK: declare void @__dfsan_nonzero_label()
 ; CHECK: declare void @__dfsan_vararg_wrapper(i8*)
 ; CHECK: declare zeroext i32 @__dfsan_chain_origin(i32 zeroext)
 ; CHECK: declare zeroext i32 @__dfsan_chain_origin_if_tainted(i[[#SBITS]] zeroext, i32 zeroext)
 ; CHECK: declare void @__dfsan_mem_origin_transfer(i8*, i8*, i64)
 ; CHECK: declare void @__dfsan_maybe_store_origin(i[[#SBITS]] zeroext, i8*, i64, i32 zeroext)

llvm/test/Instrumentation/DataFlowSanitizer/call.ll

 ; RUN: opt < %s -dfsan -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels -S | FileCheck %s
 ; RUN: opt < %s -passes=dfsan -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK-LABEL: @__dfsan_arg_tls
 ; CHECK: = external thread_local(initialexec) global [100 x i64]
 
 ; CHECK-LABEL: @__dfsan_retval_tls

llvm/test/Instrumentation/DataFlowSanitizer/dont_combine_offset_labels_on_gep.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-combine-offset-labels-on-gep=false -dfsan-fast-8-labels=true -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN
-; RUN: opt < %s -dfsan -dfsan-combine-offset-labels-on-gep=false -dfsan-fast-8-labels=true -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-combine-offset-labels-on-gep=false -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-combine-offset-labels-on-gep=false -dfsan-track-origins=1 -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 
 define i32* @gepop([10 x [20 x i32]]* %p, i32 %a, i32 %b, i32 %c) {

llvm/test/Instrumentation/DataFlowSanitizer/external_mask.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,CHECK16
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,CHECK16
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -S | FileCheck %s --check-prefixes=CHECK
+; RUN: opt < %s -dfsan -S | FileCheck %s
 target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-linux-gnu"
 
 define i32 @test(i32 %a, i32* nocapture readonly %b) #0 {
   ; CHECK: @"dfs$test"
   ; CHECK: %[[RV:.*]] load{{.*}}__dfsan_shadow_ptr_mask
   ; CHECK: ptrtoint i32* {{.*}} to i64
   ; CHECK: and {{.*}}%[[RV:.*]]
-; CHECK16: mul i64
   %1 = load i32, i32* %b, align 4
   %2 = add nsw i32 %1, %a
   ret i32 %2
 }

llvm/test/Instrumentation/DataFlowSanitizer/fast16labels.ll

-; Test that -dfsan-fast-16-labels mode uses inline ORs rather than calling
-; __dfsan_union or __dfsan_union_load.
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels -S | FileCheck %s --implicit-check-not="call{{.*}}__dfsan_union" --check-prefixes=CHECK,CHECK16
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels -S | FileCheck %s --implicit-check-not="call{{.*}}__dfsan_union" --check-prefixes=CHECK,CHECK8
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
-; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
-; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
-; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
-
-define i8 @add(i8 %a, i8 %b) {
-  ; CHECK-LABEL: define i8 @"dfs$add"
-  ; CHECK-DAG: %[[ALABEL:.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; CHECK-DAG: %[[BLABEL:.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK: %[[ADDLABEL:.*]] = or i[[#SBITS]] %[[ALABEL]], %[[BLABEL]]
-  ; CHECK: %c = add i8 %a, %b
-  ; CHECK: store i[[#SBITS]] %[[ADDLABEL]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK: ret i8 %c
-  %c = add i8 %a, %b
-  ret i8 %c
-}
-
-define i8 @load8(i8* %p) {
-  ; CHECK-LABEL:  define i8 @"dfs$load8"
-  ; CHECK-SAME:   (i8* %[[PADDR:.*]])
-  ; CHECK-NEXT:   %[[#ARG:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   %[[#R:]] = ptrtoint i8* %[[PADDR]] to i64
-  ; CHECK-NEXT:   %[[#PS:R+1]] = and i64 %[[#R]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT: %[[#PS:R+2]] = mul i64 %[[#R+1]], 2
-  ; CHECK-NEXT:   %[[#SADDR:]] = inttoptr i64 %[[#PS]] to i[[#SBITS]]*
-  ; CHECK-NEXT:   %[[#S:]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SADDR]]
-  ; CHECK-NEXT:   %[[#S_OUT:S+1]] = or i[[#SBITS]] %[[#S]], %[[#ARG]]
-  ; CHECK-NEXT:   %a = load i8, i8* %p
-  ; CHECK-NEXT:   store i[[#SBITS]] %[[#S_OUT]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   ret i8 %a
-
-  %a = load i8, i8* %p
-  ret i8 %a
-}
-
-define i16 @load16(i16* %p) {
-  ; CHECK-LABEL:  define i16 @"dfs$load16"
-  ; CHECK-SAME:   (i16* %[[PADDR:.*]])
-  ; CHECK-NEXT:   %[[#ARG:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   %[[#R:]] = ptrtoint i16* %[[PADDR]] to i64
-  ; CHECK-NEXT:   %[[#PS:R+1]] = and i64 %[[#R]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT: %[[#PS:R+2]] = mul i64 %[[#R+1]], 2
-  ; CHECK-NEXT:   %[[#SADDR:]]  = inttoptr i64 %[[#PS]] to i[[#SBITS]]*
-  ; CHECK-NEXT:   %[[#SADDR+1]] = getelementptr i[[#SBITS]], i[[#SBITS]]* %[[#SADDR]], i64 1
-  ; CHECK-NEXT:   %[[#S:]]  = load i[[#SBITS]], i[[#SBITS]]* %[[#SADDR]]
-  ; CHECK-NEXT:   %[[#S+1]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SADDR+1]]
-  ; CHECK-NEXT:   %[[#S+2]] = or i[[#SBITS]] %[[#S]], %[[#S+1]]
-  ; CHECK-NEXT:   %[[#S_OUT:S+3]] = or i[[#SBITS]] %[[#S+2]], %[[#ARG]]
-  ; CHECK-NEXT:   %a = load i16, i16* %p
-  ; CHECK-NEXT:   store i[[#SBITS]] %[[#S_OUT]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   ret i16 %a
-
-  %a = load i16, i16* %p
-  ret i16 %a
-}
-
-define i32 @load32(i32* %p) {
-  ; CHECK-LABEL: define i32 @"dfs$load32"
-  ; CHECK-SAME:   (i32* %[[PADDR:.*]])
-  ; CHECK-NEXT:   %[[#ARG:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   %[[#R:]] = ptrtoint i32* %[[PADDR]] to i64
-  ; CHECK-NEXT:   %[[#PS:R+1]] = and i64 %[[#R]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT: %[[#PS:R+2]] = mul i64 %[[#R+1]], 2
-  ; CHECK-NEXT:   %[[#SADDR:]] = inttoptr i64 %[[#PS]] to i[[#SBITS]]*
-  ; CHECK-NEXT:   %[[#SADDR+1]] = bitcast i[[#SBITS]]* %[[#SADDR]] to i[[#WBITS:mul(SBITS,4)]]*
-  ; CHECK-NEXT:   %[[#WS:]]  = load i[[#WBITS]], i[[#WBITS]]* %[[#SADDR+1]]
-  ; CHECK-NEXT:   %[[#WS+1]] = lshr i[[#WBITS]] %[[#WS]], [[#mul(SBITS,2)]]
-  ; CHECK-NEXT:   %[[#WS+2]] = or i[[#WBITS]] %[[#WS]], %[[#WS+1]]
-  ; CHECK-NEXT:   %[[#WS+3]] = lshr i[[#WBITS]] %[[#WS+2]], [[#SBITS]]
-  ; CHECK-NEXT:   %[[#WS+4]] = or i[[#WBITS]] %[[#WS+2]], %[[#WS+3]]
-  ; CHECK-NEXT:   %[[#WS+5]] = trunc i[[#WBITS]] %[[#WS+4]] to i[[#SBITS]]
-  ; CHECK-NEXT:   %[[#S_OUT:WS+6]] = or i[[#SBITS]] %[[#WS+5]], %[[#ARG]]
-  ; CHECK-NEXT:   %a = load i32, i32* %p
-  ; CHECK-NEXT:   store i[[#SBITS]] %[[#S_OUT]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   ret i32 %a
-
-  %a = load i32, i32* %p
-  ret i32 %a
-}
-
-define i64 @load64(i64* %p) {
-  ; CHECK-LABEL:  define i64 @"dfs$load64"
-  ; CHECK-SAME:   (i64* %[[PADDR:.*]])
-  ; CHECK-NEXT:   %[[#ARG:]]        = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   %[[#R:]]          = ptrtoint i64* %[[PADDR]] to i64
-  ; CHECK-NEXT:   %[[#PS:R+1]]      = and i64 %[[#R]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT: %[[#PS:R+2]]      = mul i64 %[[#R+1]], 2
-  ; CHECK-NEXT:   %[[#SADDR:]]      = inttoptr i64 %[[#PS]] to i[[#SBITS]]*
-  ; CHECK-NEXT:   %[[#SADDR+1]]     = bitcast i[[#SBITS]]* %[[#SADDR]] to i64*
-  ; CHECK-NEXT:   %[[#WS:]]         = load i64, i64* %[[#SADDR+1]]
-
-  ; COMM: On fast16, the 2x64 shadow bits need to be ORed first.
-  ; CHECK16-NEXT: %[[#SADDR_NEXT:]] = getelementptr i64, i64* %[[#SADDR+1]], i64 1
-  ; CHECK16-NEXT: %[[#WS_NEXT:]]    = load i64, i64* %[[#SADDR_NEXT]]
-  ; CHECK16-NEXT: %[[#WS:]]         = or i64 %[[#WS]], %[[#WS_NEXT]]
-  ; CHECK16-NEXT: %[[#WS+1]]        = lshr i64 %[[#WS]], 32
-  ; CHECK16-NEXT: %[[#WS+2]]        = or i64 %[[#WS]], %[[#WS+1]]
-  ; CHECK16-NEXT: %[[#WS+3]]        = lshr i64 %[[#WS+2]], 16
-  ; CHECK16-NEXT: %[[#WS+4]]        = or i64 %[[#WS+2]], %[[#WS+3]]
-  ; CHECK16-NEXT: %[[#WS+5]]        = trunc i64 %[[#WS+4]] to i[[#SBITS]]
-  ; CHECK16-NEXT: %[[#S_OUT:]]      = or i[[#SBITS]] %[[#WS+5]], %[[#ARG]]
-
-  ; COMM: On fast8, no need to OR the wide shadow but one more shift is needed.
-  ; CHECK8-NEXT: %[[#WS+1]]         = lshr i64 %[[#WS]], 32
-  ; CHECK8-NEXT: %[[#WS+2]]         = or i64 %[[#WS]], %[[#WS+1]]
-  ; CHECK8-NEXT: %[[#WS+3]]         = lshr i64 %[[#WS+2]], 16
-  ; CHECK8-NEXT: %[[#WS+4]]         = or i64 %[[#WS+2]], %[[#WS+3]]
-  ; CHECK8-NEXT: %[[#WS+5]]         = lshr i64 %[[#WS+4]], 8
-  ; CHECK8-NEXT: %[[#WS+6]]         = or i64 %[[#WS+4]], %[[#WS+5]]
-  ; CHECK8-NEXT: %[[#WS+7]]         = trunc i64 %[[#WS+6]] to i[[#SBITS]]
-  ; CHECK8-NEXT: %[[#S_OUT:]]       = or i[[#SBITS]] %[[#WS+7]], %[[#ARG]]
-
-  ; CHECK-NEXT:   %a = load i64, i64* %p
-  ; CHECK-NEXT:   store i[[#SBITS]] %[[#S_OUT]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   ret i64 %a
-
-  %a = load i64, i64* %p
-  ret i64 %a
-}
-
-define i128 @load128(i128* %p) {
-  ; CHECK-LABEL:  define i128 @"dfs$load128"
-  ; CHECK-SAME:   (i128* %[[PADDR:.*]])
-  ; CHECK-NEXT:   %[[#ARG:]]    = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT:   %[[#R:]]      = ptrtoint i128* %[[PADDR]] to i64
-  ; CHECK-NEXT:   %[[#PS:R+1]]  = and i64 %[[#R]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT: %[[#PS:R+2]]  = mul i64 %[[#R+1]], 2
-  ; CHECK-NEXT:   %[[#SADDR:]]  = inttoptr i64 %[[#PS]] to i[[#SBITS]]*
-  ; CHECK-NEXT:   %[[#SADDR+1]] = bitcast i[[#SBITS]]* %[[#SADDR]] to i64*
-  ; CHECK-NEXT:   %[[#S:]]      = load i64, i64* %[[#SADDR+1]]
-  ; CHECK-NEXT:   %[[#S+1]]     = getelementptr i64, i64* %[[#SADDR+1]], i64 1
-  ; CHECK-NEXT:   %[[#S+2]]     = load i64, i64* %[[#S+1]]
-  ; CHECK-NEXT:   %[[#WS:S+3]]  = or i64 %[[#S]], %[[#S+2]]
-
-  ; COMM: On fast16, we need to OR 4x64bits for the wide shadow, before ORing its bytes.
-  ; CHECK16-NEXT: %[[#S+4]]     = getelementptr i64, i64* %[[#S+1]], i64 1
-  ; CHECK16-NEXT: %[[#S+5]]     = load i64, i64* %[[#S+4]]
-  ; CHECK16-NEXT: %[[#S+6]]     = or i64 %[[#S+3]], %[[#S+5]]
-  ; CHECK16-NEXT: %[[#S+7]]     = getelementptr i64, i64* %[[#S+4]], i64 1
-  ; CHECK16-NEXT: %[[#S+8]]     = load i64, i64* %[[#S+7]]
-  ; CHECK16-NEXT: %[[#WS:S+9]]  = or i64 %[[#S+6]], %[[#S+8]]
-  ; CHECK16-NEXT: %[[#WS+1]]    = lshr i64 %[[#WS]], 32
-  ; CHECK16-NEXT: %[[#WS+2]]    = or i64 %[[#WS]], %[[#WS+1]]
-  ; CHECK16-NEXT: %[[#WS+3]]    = lshr i64 %[[#WS+2]], 16
-  ; CHECK16-NEXT: %[[#WS+4]]    = or i64 %[[#WS+2]], %[[#WS+3]]
-  ; CHECK16-NEXT: %[[#WS+5]]    = trunc i64 %[[#WS+4]] to i[[#SBITS]]
-  ; CHECK16-NEXT: %[[#S_OUT:]]  = or i[[#SBITS]] %[[#WS+5]], %[[#ARG]]
-
-  ; COMM: On fast8, we need to OR 2x64bits for the wide shadow, before ORing its bytes (one more shift).
-  ; CHECK8-NEXT: %[[#WS+1]]     = lshr i64 %[[#WS]], 32
-  ; CHECK8-NEXT: %[[#WS+2]]     = or i64 %[[#WS]], %[[#WS+1]]
-  ; CHECK8-NEXT: %[[#WS+3]]     = lshr i64 %[[#WS+2]], 16
-  ; CHECK8-NEXT: %[[#WS+4]]     = or i64 %[[#WS+2]], %[[#WS+3]]
-  ; CHECK8-NEXT: %[[#WS+5]]     = lshr i64 %[[#WS+4]], 8
-  ; CHECK8-NEXT: %[[#WS+6]]     = or i64 %[[#WS+4]], %[[#WS+5]]
-  ; CHECK8-NEXT: %[[#WS+7]]     = trunc i64 %[[#WS+6]] to i[[#SBITS]]
-  ; CHECK8-NEXT: %[[#S_OUT:]]   = or i[[#SBITS]] %[[#WS+7]], %[[#ARG]]
-
-  ; CHECK-NEXT: %a = load i128, i128* %p
-  ; CHECK-NEXT: store i[[#SBITS]] %[[#S_OUT]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; CHECK-NEXT: ret i128 %a
-
-  %a = load i128, i128* %p
-  ret i128 %a
-}

llvm/test/Instrumentation/DataFlowSanitizer/load.ll

-; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-load=1 -S | FileCheck %s --check-prefix=COMBINE_PTR_LABEL
-; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-load=0 -S | FileCheck %s --check-prefix=NO_COMBINE_PTR_LABEL
+; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-load=true -S | FileCheck %s --check-prefixes=CHECK,COMBINE_LOAD_PTR
+; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
+; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
+; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
+; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
+
+
 define {} @load0({}* %p) {
-  ; COMBINE_PTR_LABEL: @"dfs$load0"
-  ; COMBINE_PTR_LABEL: load
-  ; COMBINE_PTR_LABEL-NOT: load
-
-  ; NO_COMBINE_PTR_LABEL: @"dfs$load0"
-  ; NO_COMBINE_PTR_LABEL: load
-  ; NO_COMBINE_PTR_LABEL-NOT: load
+  ; CHECK-LABEL:           @"dfs$load0"
+  ; CHECK-NEXT:            %a = load {}, {}* %p, align 1
+  ; CHECK-NEXT:            store {} zeroinitializer, {}* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to {}*), align [[ALIGN:2]]
+  ; CHECK-NEXT:            ret {} %a
+
   %a = load {}, {}* %p
   ret {} %a
 }
 
 define i8 @load8(i8* %p) {
-  ; COMBINE_PTR_LABEL: @"dfs$load8"
-  ; COMBINE_PTR_LABEL: load i16, i16*
-  ; COMBINE_PTR_LABEL: ptrtoint i8* {{.*}} to i64
-  ; COMBINE_PTR_LABEL: and i64
-  ; COMBINE_PTR_LABEL: mul i64
-  ; COMBINE_PTR_LABEL: inttoptr i64
-  ; COMBINE_PTR_LABEL: load i16, i16*
-  ; COMBINE_PTR_LABEL: icmp ne i16
-  ; COMBINE_PTR_LABEL: call zeroext i16 @__dfsan_union
-  ; COMBINE_PTR_LABEL: load i8, i8*
-  ; COMBINE_PTR_LABEL: store i16 {{.*}} @__dfsan_retval_tls
-  ; COMBINE_PTR_LABEL: ret i8
-
-  ; NO_COMBINE_PTR_LABEL: @"dfs$load8"
-  ; NO_COMBINE_PTR_LABEL: ptrtoint i8*
-  ; NO_COMBINE_PTR_LABEL: and i64
-  ; NO_COMBINE_PTR_LABEL: mul i64
-  ; NO_COMBINE_PTR_LABEL: inttoptr i64 {{.*}} to i16*
-  ; NO_COMBINE_PTR_LABEL: load i16, i16*
-  ; NO_COMBINE_PTR_LABEL: load i8, i8*
-  ; NO_COMBINE_PTR_LABEL: store i16 {{.*}} @__dfsan_retval_tls
-  ; NO_COMBINE_PTR_LABEL: ret i8
+  ; CHECK-LABEL:           @"dfs$load8"
+  ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i8* %p to i64
+  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW:]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR]]
+  ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
+  ; CHECK-NEXT:            %a = load i8, i8* %p
+  ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            ret i8 %a
 
   %a = load i8, i8* %p
   ret i8 %a
 }
 
 define i16 @load16(i16* %p) {
-  ; COMBINE_PTR_LABEL: @"dfs$load16"
-  ; COMBINE_PTR_LABEL: ptrtoint i16*
-  ; COMBINE_PTR_LABEL: and i64
-  ; COMBINE_PTR_LABEL: mul i64
-  ; COMBINE_PTR_LABEL: inttoptr i64 {{.*}} i16*
-  ; COMBINE_PTR_LABEL: getelementptr i16
-  ; COMBINE_PTR_LABEL: load i16, i16*
-  ; COMBINE_PTR_LABEL: load i16, i16*
-  ; COMBINE_PTR_LABEL: icmp ne
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-  ; COMBINE_PTR_LABEL: icmp ne i16
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-  ; COMBINE_PTR_LABEL: load i16, i16*
-  ; COMBINE_PTR_LABEL: store {{.*}} @__dfsan_retval_tls
-  ; COMBINE_PTR_LABEL: ret i16
-
-  ; NO_COMBINE_PTR_LABEL: @"dfs$load16"
-  ; NO_COMBINE_PTR_LABEL: ptrtoint i16*
-  ; NO_COMBINE_PTR_LABEL: and i64
-  ; NO_COMBINE_PTR_LABEL: mul i64
-  ; NO_COMBINE_PTR_LABEL: inttoptr i64 {{.*}} i16*
-  ; NO_COMBINE_PTR_LABEL: getelementptr i16, i16*
-  ; NO_COMBINE_PTR_LABEL: load i16, i16*
-  ; NO_COMBINE_PTR_LABEL: load i16, i16*
-  ; NO_COMBINE_PTR_LABEL: icmp ne i16
-  ; NO_COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-  ; NO_COMBINE_PTR_LABEL: load i16, i16*
-  ; NO_COMBINE_PTR_LABEL: store i16 {{.*}} @__dfsan_retval_tls
-  ; NO_COMBINE_PTR_LABEL: ret i16
+  ; CHECK-LABEL:           @"dfs$load16"
+  ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i16* %p to i64
+  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW_PTR+1]] = getelementptr i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR]], i64 1
+  ; CHECK-NEXT:            %[[#SHADOW:]]  = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR]]
+  ; CHECK-NEXT:            %[[#SHADOW+1]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR+1]]
+
+  ; CHECK-NEXT:            %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#SHADOW+1]]
+  ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
+  ; CHECK-NEXT:            %a = load i16, i16* %p
+  ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            ret i16 %a
 
   %a = load i16, i16* %p
   ret i16 %a
 }
 
 define i32 @load32(i32* %p) {
-  ; COMBINE_PTR_LABEL: @"dfs$load32"
-  ; COMBINE_PTR_LABEL: ptrtoint i32*
-  ; COMBINE_PTR_LABEL: and i64
-  ; COMBINE_PTR_LABEL: mul i64
-  ; COMBINE_PTR_LABEL: inttoptr i64 {{.*}} i16*
-  ; COMBINE_PTR_LABEL: bitcast i16* {{.*}} i64*
-  ; COMBINE_PTR_LABEL: load i64, i64*
-  ; COMBINE_PTR_LABEL: trunc i64 {{.*}} i16
-  ; COMBINE_PTR_LABEL: shl i64
-  ; COMBINE_PTR_LABEL: lshr i64
-  ; COMBINE_PTR_LABEL: or i64
-  ; COMBINE_PTR_LABEL: icmp eq i64
-  ; COMBINE_PTR_LABEL: icmp ne i16
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-  ; COMBINE_PTR_LABEL: load i32, i32*
-  ; COMBINE_PTR_LABEL: store i16 {{.*}} @__dfsan_retval_tls
-  ; COMBINE_PTR_LABEL: ret i32
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union_load
-
-  ; NO_COMBINE_PTR_LABEL: @"dfs$load32"
-  ; NO_COMBINE_PTR_LABEL: ptrtoint i32*
-  ; NO_COMBINE_PTR_LABEL: and i64
-  ; NO_COMBINE_PTR_LABEL: mul i64
-  ; NO_COMBINE_PTR_LABEL: inttoptr i64 {{.*}} i16*
-  ; NO_COMBINE_PTR_LABEL: bitcast i16* {{.*}} i64*
-  ; NO_COMBINE_PTR_LABEL: load i64, i64*
-  ; NO_COMBINE_PTR_LABEL: trunc i64 {{.*}} i16
-  ; NO_COMBINE_PTR_LABEL: shl i64
-  ; NO_COMBINE_PTR_LABEL: lshr i64
-  ; NO_COMBINE_PTR_LABEL: or i64
-  ; NO_COMBINE_PTR_LABEL: icmp eq i64
-  ; NO_COMBINE_PTR_LABEL: load i32, i32*
-  ; NO_COMBINE_PTR_LABEL: store i16 {{.*}} @__dfsan_retval_tls
-  ; NO_COMBINE_PTR_LABEL: ret i32
-  ; NO_COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union_load
-  
+  ; CHECK-LABEL:           @"dfs$load32"
+  ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i32* %p to i64
+  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#WSBITS:mul(SBITS,4)]]*
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i[[#WSBITS]], i[[#WSBITS]]* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i[[#WSBITS]] %[[#WIDE_SHADOW]], [[#mul(SBITS,2)]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i[[#WSBITS]] %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i[[#WSBITS]] %[[#WIDE_SHADOW]], [[#SBITS]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i[[#WSBITS]] %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#SHADOW:]] = trunc i[[#WSBITS]] %[[#WIDE_SHADOW]] to i[[#SBITS]]
+  ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
+  ; CHECK-NEXT:            %a = load i32, i32* %p, align 4
+  ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            ret i32 %a
 
   %a = load i32, i32* %p
   ret i32 %a
 }
 
 define i64 @load64(i64* %p) {
-  ; COMBINE_PTR_LABEL: @"dfs$load64"
-  ; COMBINE_PTR_LABEL: ptrtoint i64*
-  ; COMBINE_PTR_LABEL: and i64
-  ; COMBINE_PTR_LABEL: mul i64
-  ; COMBINE_PTR_LABEL: inttoptr i64 {{.*}} i16*
-  ; COMBINE_PTR_LABEL: bitcast i16* {{.*}} i64*
-  ; COMBINE_PTR_LABEL: load i64, i64*
-  ; COMBINE_PTR_LABEL: trunc i64 {{.*}} i16
-  ; COMBINE_PTR_LABEL: shl i64
-  ; COMBINE_PTR_LABEL: lshr i64
-  ; COMBINE_PTR_LABEL: or i64
-  ; COMBINE_PTR_LABEL: icmp eq i64
-  ; COMBINE_PTR_LABEL: icmp ne i16
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-  ; COMBINE_PTR_LABEL: load i64, i64*
-  ; COMBINE_PTR_LABEL: store i16 {{.*}} @__dfsan_retval_tls
-  ; COMBINE_PTR_LABEL: ret i64
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union_load
-  ; COMBINE_PTR_LABEL: getelementptr i64, i64* {{.*}} i64
-  ; COMBINE_PTR_LABEL: load i64, i64*
-  ; COMBINE_PTR_LABEL: icmp eq i64
-
-  ; NO_COMBINE_PTR_LABEL: @"dfs$load64"
-  ; NO_COMBINE_PTR_LABEL: ptrtoint i64*
-  ; NO_COMBINE_PTR_LABEL: and i64
-  ; NO_COMBINE_PTR_LABEL: mul i64
-  ; NO_COMBINE_PTR_LABEL: inttoptr i64 {{.*}} i16*
-  ; NO_COMBINE_PTR_LABEL: bitcast i16* {{.*}} i64*
-  ; NO_COMBINE_PTR_LABEL: load i64, i64*
-  ; NO_COMBINE_PTR_LABEL: trunc i64 {{.*}} i16
-  ; NO_COMBINE_PTR_LABEL: shl i64
-  ; NO_COMBINE_PTR_LABEL: lshr i64
-  ; NO_COMBINE_PTR_LABEL: or i64
-  ; NO_COMBINE_PTR_LABEL: icmp eq i64
-  ; NO_COMBINE_PTR_LABEL: load i64, i64*
-  ; NO_COMBINE_PTR_LABEL: store i16 {{.*}} @__dfsan_retval_tls
-  ; NO_COMBINE_PTR_LABEL: ret i64
-  ; NO_COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union_load
-  ; NO_COMBINE_PTR_LABEL: getelementptr i64, i64* {{.*}} i64
-  ; NO_COMBINE_PTR_LABEL: load i64, i64*
-  ; NO_COMBINE_PTR_LABEL: icmp eq i64
+  ; CHECK-LABEL:           @"dfs$load64"
+  ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i64* %p to i64
+  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i64, i64* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
+  ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
+  ; CHECK-NEXT:            %a = load i64, i64* %p, align 8
+  ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            ret i64 %a
   
   %a = load i64, i64* %p
   ret i64 %a
 }
 
+define i128 @load128(i128* %p) {
+  ; CHECK-LABEL:           @"dfs$load128"
+  ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i128* %p to i64
+  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i64, i64* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR2:]] = getelementptr i64, i64* %[[#WIDE_SHADOW_PTR]], i64 1
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW2:]] = load i64, i64* %[[#WIDE_SHADOW_PTR2]], align [[#SBYTES]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW2]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
+  ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
+  ; CHECK-NEXT:            %a = load i128, i128* %p, align 8
+  ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            ret i128 %a
+  
+  %a = load i128, i128* %p
+  ret i128 %a
+}
+
+
+define i17 @load17(i17* %p) {
+  ; CHECK-LABEL:           @"dfs$load17"
+  ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i17* %p to i64
+  ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
+  ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
+  ; CHECK-NEXT:            %[[#SHADOW:]] = call zeroext i8 @__dfsan_union_load(i[[#SBITS]]* %[[#SHADOW_PTR]], i64 3)
+  ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
+  ; CHECK-NEXT:            %a = load i17, i17* %p
+  ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            ret i17 %a
+
+  %a = load i17, i17* %p
+  ret i17 %a
+}
+
 @X = constant i1 1
 define i1 @load_global() {
-  ; NO_COMBINE_PTR_LABEL: @"dfs$load_global"
-  ; NO_COMBINE_PTR_LABEL: store i16 0, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  ; CHECK-LABEL:           @"dfs$load_global"
+  ; CHECK-NEXT:            %a = load i1, i1* @X
+  ; CHECK-NEXT:            store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
+  ; CHECK-NEXT:            ret i1 %a
 
   %a = load i1, i1* @X
   ret i1 %a
 }

llvm/test/Instrumentation/DataFlowSanitizer/musttailcall.ll

 ; RUN: opt < %s -dfsan -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels -dfsan-track-origins=1 -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN
+; RUN: opt < %s -dfsan -dfsan-track-origins=1 -S | FileCheck %s --check-prefixes=CHECK,CHECK_ORIGIN
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 declare i32 @f(i32)
 
 ; CHECK-LABEL: @"dfs$inner_callee"
 define i32 @inner_callee(i32) {
   %r = call i32 @f(i32 %0)

llvm/test/Instrumentation/DataFlowSanitizer/origin_abilist.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define i32 @discard(i32 %a, i32 %b) {

 ; CHECK-NEXT:  store i32 %6, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 1), align 4
 ; CHECK-NEXT:  store i[[#SBITS]] %4, i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align 2
 ; CHECK-NEXT:  call void %0(i32 %1, i32 %2)
 ; CHECK-NEXT:  ret void
 
 ; CHECK: declare void @__dfso_custom_varg_without_ret(i32, i32, i[[#SBITS]], i[[#SBITS]], i[[#SBITS]]*, i32, i32, i32*, ...)
 
 ; CHECK: declare i32 @__dfso_custom_varg_with_ret(i32, i32, i[[#SBITS]], i[[#SBITS]], i[[#SBITS]]*, i[[#SBITS]]*, i32, i32, i32*, i32*, ...)
 No newline at end of file

llvm/test/Instrumentation/DataFlowSanitizer/origin_cached_shadows.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -S | FileCheck %s
 ;
 ; %15 and %17 have the same key in shadow cache. They should not reuse the same
 ; shadow because their blocks do not dominate each other. Origin tracking
 ; splt blocks. This test ensures DT is updated correctly, and cached shadows
 ; are not mis-used.
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
18:                                               ; preds = %16, %14, %12
   %19 = phi double [ %8, %14 ], [ %7, %12 ], [ %8, %16 ]
   %20 = phi double [ %15, %14 ], [ %8, %12 ], [ %17, %16 ]
   %21 = fcmp olt double %19, 9.900000e+01
   br i1 %21, label %6, label %22
 
 22:                                               ; preds = %18
   ret void
 }
 No newline at end of file

llvm/test/Instrumentation/DataFlowSanitizer/origin_call.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define i1 @arg_overflow(

llvm/test/Instrumentation/DataFlowSanitizer/origin_load.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels -S | FileCheck %s --check-prefixes=CHECK,CHECK8,COMBINE_LOAD_PTR
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefixes=CHECK,CHECK8
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels -S | FileCheck %s --check-prefixes=CHECK,CHECK16,COMBINE_LOAD_PTR
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefixes=CHECK,CHECK16
+; RUN: opt < %s -dfsan -dfsan-track-origins=1 -S | FileCheck %s --check-prefixes=CHECK,COMBINE_LOAD_PTR
+; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define {} @load0({}* %p) {
   ; CHECK-LABEL: @"dfs$load0"
define i16 @load_non_escaped_alloca() {
   %a = load i16, i16* %p
   ret i16 %a
 }
 
 define i16* @load_escaped_alloca() {
   ; CHECK-LABEL:  @"dfs$load_escaped_alloca"
   ; CHECK:        %[[#INTP:]] = ptrtoint i16* %p to i64
   ; CHECK-NEXT:   %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT: %[[#SHADOW_ADDR:]] = mul i64 %[[#SHADOW_ADDR]], 2
   ; CHECK-NEXT:   %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:   %[[#ORIGIN_OFFSET:]] = add i64 %[[#INTP+1]], [[#%.10d,ORIGIN_MASK:]]
   ; CHECK-NEXT:   %[[#ORIGIN_ADDR:]] = and i64 %[[#ORIGIN_OFFSET]], -4
   ; CHECK-NEXT:   %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:   {{%.*}} = load i32, i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:   %[[#SHADOW_PTR1:]] = getelementptr i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR0]], i64 1
   ; CHECK-NEXT:   %[[#SHADOW:]]  = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR0]], align [[#SBYTES]]
   ; CHECK-NEXT:   %[[#SHADOW+1]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR1]], align [[#SBYTES]]

 define i1 @load1(i1* %p) {
   ; CHECK-LABEL:             @"dfs$load1"
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
 
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i1* %p to i64
   ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK16-NEXT:          %[[#SHADOW_ADDR:]] = mul i64 %[[#SHADOW_ADDR]], 2
   ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#ORIGIN_OFFSET:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
   ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = and i64 %[[#ORIGIN_OFFSET]], -4
   ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#AO:]] = load i32, i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:            %[[#AS:]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR]], align [[#SBYTES]]
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#AS:]] = or i[[#SBITS]] %[[#AS]], %[[#PS]]

 define i16 @load16(i1 %i, i16* %p) {
   ; CHECK-LABEL: @"dfs$load16"
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 1), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
 
   ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i16* %p to i64
   ; CHECK-NEXT:            %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK16-NEXT:          %[[#SHADOW_ADDR:]] = mul i64 %[[#SHADOW_ADDR]], 2
   ; CHECK-NEXT:            %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#ORIGIN_OFFSET:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
   ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = and i64 %[[#ORIGIN_OFFSET]], -4
   ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#AO:]] = load i32, i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:            %[[#SHADOW_PTR1:]] = getelementptr i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR0]], i64 1
   ; CHECK-NEXT:            %[[#SHADOW:]]  = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR0]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#SHADOW+1]] = load i[[#SBITS]], i[[#SBITS]]* %[[#SHADOW_PTR1]], align [[#SBYTES]]

 }
 
 define i32 @load32(i32* %p) {
   ; CHECK-LABEL: @"dfs$load32"
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
 
-  ; CHECK:                 %[[#INTP:]] = ptrtoint i32* %p to i64
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i32* %p to i64
   ; CHECK-NEXT:            %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK16-NEXT:          %[[#SHADOW_ADDR:]] = mul i64 %[[#SHADOW_ADDR]], 2
   ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
   ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#AO:]] = load i32, i32* %[[#ORIGIN_PTR]], align 4
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i[[#WSBITS:mul(SBITS,4)]]*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i[[#WSBITS]], i[[#WSBITS]]* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW+1]] = lshr i[[#WSBITS]] %[[#WIDE_SHADOW]], [[#mul(SBITS,2)]]
   ; CHECK-NEXT:            %[[#WIDE_SHADOW+2]] = or i[[#WSBITS]] %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW+1]]

 }
 
 define i64 @load64(i64* %p) {
   ; CHECK-LABEL: @"dfs$load64"
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
 
-  ; CHECK:                 %[[#INTP:]] = ptrtoint i64* %p to i64
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i64* %p to i64
   ; CHECK-NEXT:            %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK16-NEXT:          %[[#SHADOW_ADDR:]] = mul i64 %[[#SHADOW_ADDR]], 2
   ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
   ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK-NEXT:            %[[#ORIGIN:]] = load i32, i32* %[[#ORIGIN_PTR]], align 8
   ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
   ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i64, i64* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
-
-  ; COMM: On fast16, the 2x64 shadow bits need to be ORed first.
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW_PTR2:]] = getelementptr i64, i64* %[[#WIDE_SHADOW_PTR]], i64 1
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW2:]] = load i64, i64* %[[#WIDE_SHADOW_PTR2]], align [[#SBYTES]]
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW2]]
-  ; CHECK16-NEXT:          %[[#ORIGIN_PTR2:]] = getelementptr i32, i32* %[[#ORIGIN_PTR]], i64 1
-  ; CHECK16-NEXT:          %[[#ORIGIN2:]] = load i32, i32* %[[#ORIGIN_PTR2]], align 8
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK16-NEXT:          %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
-  ; CHECK16-NEXT:          %[[#SHADOW_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW2]], 0
-  ; CHECK16-NEXT:          %[[#ORIGIN:]] = select i1 %[[#SHADOW_NZ]], i32 %[[#ORIGIN2]], i32 %[[#ORIGIN]]
-
-  ; COMM: On fast8, no need to OR the wide shadow but one more shift is needed.
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW_LO:]] = shl i64 %[[#WIDE_SHADOW]], 32
-  ; CHECK8-NEXT:           %[[#ORIGIN_PTR2:]] = getelementptr i32, i32* %[[#ORIGIN_PTR]], i64 1
-  ; CHECK8-NEXT:           %[[#ORIGIN2:]] = load i32, i32* %[[#ORIGIN_PTR2]], align 8
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 8
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK8-NEXT:           %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_LO:]] = shl i64 %[[#WIDE_SHADOW]], 32
+  ; CHECK-NEXT:            %[[#ORIGIN2_PTR:]] = getelementptr i32, i32* %[[#ORIGIN_PTR]], i64 1
+  ; CHECK-NEXT:            %[[#ORIGIN2:]] = load i32, i32* %[[#ORIGIN2_PTR]], align 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
+  ; CHECK-NEXT:            %[[#SHADOW_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW_LO]], 0
+  ; CHECK-NEXT:            %[[#ORIGIN:]] = select i1 %[[#SHADOW_NZ]], i32 %[[#ORIGIN]], i32 %[[#ORIGIN2]]
   ; CHECK8-NEXT:           %[[#SHADOW_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW_LO]], 0
   ; CHECK8-NEXT:           %[[#ORIGIN:]] = select i1 %[[#SHADOW_NZ]], i32 %[[#ORIGIN]], i32 %[[#ORIGIN2]]
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
   ; COMBINE_LOAD_PTR-NEXT: %[[#NZ:]] = icmp ne i[[#SBITS]] %[[#PS]], 0
   ; COMBINE_LOAD_PTR-NEXT: %[[#ORIGIN:]] = select i1 %[[#NZ]], i32 %[[#PO]], i32 %[[#ORIGIN]]
 
   ; CHECK-NEXT:            %a = load i64, i64* %p, align 8

 }
 
 define i128 @load128(i128* %p) {
   ; CHECK-LABEL: @"dfs$load128"
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#PO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; COMBINE_LOAD_PTR-NEXT: %[[#PS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
 
-  ; CHECK:                 %[[#INTP:]] = ptrtoint i128* %p to i64
+  ; CHECK-NEXT:            %[[#INTP:]] = ptrtoint i128* %p to i64
   ; CHECK-NEXT:            %[[#SHADOW_ADDR:INTP+1]] = and i64 %[[#INTP]], [[#MASK]]
-  ; CHECK16-NEXT:          %[[#SHADOW_ADDR:]] = mul i64 %[[#SHADOW_ADDR]], 2
   ; CHECK-NEXT:            %[[#SHADOW_PTR:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
-  ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#INTP+1]], [[#ORIGIN_MASK]]
-  ; CHECK-NEXT:            %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
-  ; CHECK-NEXT:            %[[#ORIGIN:]] = load i32, i32* %[[#ORIGIN_PTR]], align 8
-  ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
-  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = load i64, i64* %[[#WIDE_SHADOW_PTR]], align [[#SBYTES]]
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW_LO:]] = shl i64 %[[#WIDE_SHADOW]], 32
-  ; CHECK8-NEXT:           %[[#ORIGIN_PTR2:]] = getelementptr i32, i32* %[[#ORIGIN_PTR]], i64 1
-  ; CHECK8-NEXT:           %[[#ORIGIN2:]] = load i32, i32* %[[#ORIGIN_PTR2]], align 8
-  ; CHECK-NEXT:            %[[#WIDE_SHADOW_PTR2:]] = getelementptr i64, i64* %[[#WIDE_SHADOW_PTR]], i64 1
-  ; CHECK-NEXT:            %[[#WIDE_SHADOW2:]] = load i64, i64* %[[#WIDE_SHADOW_PTR2]], align [[#SBYTES]]
-  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW2]]
-
-  ; COMM: On fast16, we need to OR 4x64bits for the wide shadow, before ORing its bytes.
-  ; CHECK16-NEXT:          %[[#ORIGIN_PTR2:]] = getelementptr i32, i32* %[[#ORIGIN_PTR]], i64 1
-  ; CHECK16-NEXT:          %[[#ORIGIN2:]] = load i32, i32* %[[#ORIGIN_PTR2]], align 8
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW_PTR3:]] = getelementptr i64, i64* %[[#WIDE_SHADOW_PTR2]], i64 1
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW3:]] = load i64, i64* %[[#WIDE_SHADOW_PTR3]], align [[#SBYTES]]
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW3]]
-  ; CHECK16-NEXT:          %[[#ORIGIN_PTR3:]] = getelementptr i32, i32* %[[#ORIGIN_PTR2]], i64 1
-  ; CHECK16-NEXT:          %[[#ORIGIN3:]] = load i32, i32* %[[#ORIGIN_PTR3]], align 8
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW_PTR4:]] = getelementptr i64, i64* %[[#WIDE_SHADOW_PTR3]], i64 1
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW4:]] = load i64, i64* %[[#WIDE_SHADOW_PTR4]], align [[#SBYTES]]
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW4]]
-  ; CHECK16-NEXT:          %[[#ORIGIN_PTR4:]] = getelementptr i32, i32* %[[#ORIGIN_PTR3]], i64 1
-  ; CHECK16-NEXT:          %[[#ORIGIN4:]] = load i32, i32* %[[#ORIGIN_PTR4]], align 8
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
-  ; CHECK16-NEXT:          %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK16-NEXT:          %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
-  ; CHECK16-NEXT:          %[[#SHADOW2_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW2]], 0
-  ; CHECK16-NEXT:          %[[#ORIGIN:]] = select i1 %[[#SHADOW2_NZ]], i32 %[[#ORIGIN2]], i32 %[[#ORIGIN]]
-  ; CHECK16-NEXT:          %[[#SHADOW3_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW3]], 0
-  ; CHECK16-NEXT:          %[[#ORIGIN:]] = select i1 %[[#SHADOW3_NZ]], i32 %[[#ORIGIN3]], i32 %[[#ORIGIN]]
-  ; CHECK16-NEXT:          %[[#SHADOW4_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW4]], 0
-  ; CHECK16-NEXT:          %[[#ORIGIN:]] = select i1 %[[#SHADOW4_NZ]], i32 %[[#ORIGIN4]], i32 %[[#ORIGIN]]
-  
-  ; COMM: On fast8, we need to OR 2x64bits for the wide shadow, before ORing its bytes (one more shift).
-  ; CHECK8-NEXT:           %[[#ORIGIN_PTR3:]] = getelementptr i32, i32* %[[#ORIGIN_PTR2]], i64 1
-  ; CHECK8-NEXT:           %[[#ORIGIN3:]] = load i32, i32* %[[#ORIGIN_PTR3]], align 8
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW2_LO:]] = shl i64 %[[#WIDE_SHADOW2]], 32
-  ; CHECK8-NEXT:           %[[#ORIGIN_PTR4:]] = getelementptr i32, i32* %[[#ORIGIN_PTR3]], i64 1
-  ; CHECK8-NEXT:           %[[#ORIGIN4:]] = load i32, i32* %[[#ORIGIN_PTR4]], align 8
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 8
-  ; CHECK8-NEXT:           %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
-  ; CHECK8-NEXT:           %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
-  ; CHECK8-NEXT:           %[[#SHADOW_LO_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW_LO]], 0
-  ; CHECK8-NEXT:           %[[#ORIGIN:]] = select i1 %[[#SHADOW_LO_NZ]], i32 %[[#ORIGIN]], i32 %[[#ORIGIN2]]
-  ; CHECK8-NEXT:           %[[#SHADOW2_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW2]], 0
-  ; CHECK8-NEXT:           %[[#ORIGIN:]] = select i1 %[[#SHADOW2_NZ]], i32 %[[#ORIGIN4]], i32 %[[#ORIGIN]]
-  ; CHECK8-NEXT:           %[[#SHADOW2_LO_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW2_LO]], 0
-  ; CHECK8-NEXT:           %[[#ORIGIN:]] = select i1 %[[#SHADOW2_LO_NZ]], i32 %[[#ORIGIN3]], i32 %[[#ORIGIN]]
+  ; CHECK-NEXT:            %[[#ORIGIN_ADDR:]] = add i64 %[[#SHADOW_ADDR]], [[#ORIGIN_MASK]]
+  ; CHECK-NEXT:            %[[#ORIGIN1_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
+  ; CHECK-NEXT:            %[[#ORIGIN1:]] = load i32, i32* %[[#ORIGIN1_PTR]], align 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW1_PTR:]] = bitcast i[[#SBITS]]* %[[#SHADOW_PTR]] to i64*
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW1:]] = load i64, i64* %[[#WIDE_SHADOW1_PTR]], align [[#SBYTES]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW1_LO:]] = shl i64 %[[#WIDE_SHADOW1]], 32
+  ; CHECK-NEXT:            %[[#ORIGIN2_PTR:]] = getelementptr i32, i32* %[[#ORIGIN1_PTR]], i64 1
+  ; CHECK-NEXT:            %[[#ORIGIN2:]] = load i32, i32* %[[#ORIGIN2_PTR]], align 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW2_PTR:]] = getelementptr i64, i64* %[[#WIDE_SHADOW1_PTR]], i64 1
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW2:]] = load i64, i64* %[[#WIDE_SHADOW2_PTR]], align [[#SBYTES]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW1]], %[[#WIDE_SHADOW2]]
+  ; CHECK-NEXT:            %[[#ORIGIN3_PTR:]] = getelementptr i32, i32* %[[#ORIGIN2_PTR]], i64 1
+  ; CHECK-NEXT:            %[[#ORIGIN3:]] = load i32, i32* %[[#ORIGIN3_PTR]], align 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW2_LO:]] = shl i64 %[[#WIDE_SHADOW2]], 32
+  ; CHECK-NEXT:            %[[#ORIGIN4_PTR:]] = getelementptr i32, i32* %[[#ORIGIN3_PTR]], i64 1
+  ; CHECK-NEXT:            %[[#ORIGIN4:]] = load i32, i32* %[[#ORIGIN4_PTR]], align 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 32
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 16
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW_SHIFTED:]] = lshr i64 %[[#WIDE_SHADOW]], 8
+  ; CHECK-NEXT:            %[[#WIDE_SHADOW:]] = or i64 %[[#WIDE_SHADOW]], %[[#WIDE_SHADOW_SHIFTED]]
+  ; CHECK-NEXT:            %[[#SHADOW:]] = trunc i64 %[[#WIDE_SHADOW]] to i[[#SBITS]]
+  ; CHECK-NEXT:            %[[#SHADOW1_LO_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW1_LO]], 0
+  ; CHECK-NEXT:            %[[#ORIGIN12:]] = select i1 %[[#SHADOW1_LO_NZ]], i32 %[[#ORIGIN1]], i32 %[[#ORIGIN2]]
+  ; CHECK-NEXT:            %[[#SHADOW2_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW2]], 0
+  ; CHECK-NEXT:            %[[#ORIGIN124:]] = select i1 %[[#SHADOW2_NZ]], i32 %[[#ORIGIN4]], i32 %[[#ORIGIN12]]
+  ; CHECK-NEXT:            %[[#SHADOW2_LO_NZ:]] = icmp ne i64 %[[#WIDE_SHADOW2_LO]], 0
+  ; CHECK-NEXT:            %[[#ORIGIN:]] = select i1 %[[#SHADOW2_LO_NZ]], i32 %[[#ORIGIN3]], i32 %[[#ORIGIN124]]
 
   ; COMBINE_LOAD_PTR-NEXT: %[[#SHADOW:]] = or i[[#SBITS]] %[[#SHADOW]], %[[#PS]]
   ; COMBINE_LOAD_PTR-NEXT: %[[#NZ:]] = icmp ne i[[#SBITS]] %[[#PS]], 0
   ; COMBINE_LOAD_PTR-NEXT: %[[#ORIGIN:]] = select i1 %[[#NZ]], i32 %[[#PO]], i32 %[[#ORIGIN]]
 
   ; CHECK-NEXT:            %a = load i128, i128* %p, align 8
   ; CHECK-NEXT:            store i[[#SBITS]] %[[#SHADOW]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK-NEXT:            store i32 %[[#ORIGIN]], i32* @__dfsan_retval_origin_tls, align 4

llvm/test/Instrumentation/DataFlowSanitizer/origin_mem_intrinsic.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 declare void @llvm.memcpy.p0i8.p0i8.i32(i8*, i8*, i32, i1)
 declare void @llvm.memmove.p0i8.p0i8.i32(i8*, i8*, i32, i1)

llvm/test/Instrumentation/DataFlowSanitizer/origin_other_ops.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
define {i32, {float, float}} @ivop({i32, {float, float}} %a, {float, float} %b) {
   ; CHECK: [[BS01:%.*]] = or i[[#SBITS]] [[BS0]], [[BS1]]
   ; CHECK: [[NE:%.*]] = icmp ne i[[#SBITS]] [[BS01]], 0
   ; CHECK: [[MO:%.*]] = select i1 [[NE]], i32 [[BO]], i32 [[AO]]
   ; CHECK: store i32 [[MO]], i32* @__dfsan_retval_origin_tls, align 4
   
   %e = insertvalue {i32, {float, float}} %a, {float, float} %b, 1
   ret {i32, {float, float}} %e
 }
 No newline at end of file

llvm/test/Instrumentation/DataFlowSanitizer/origin_phi.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define i32 @phiop(i32 %a, i32 %b, i1 %c) {
T:
   br label %done
 F:
   %diff = sub i32 %b, %a 
   br label %done
 done:
   %r = phi i32 [%sum, %T], [%diff, %F], [%a, %entry]
   ret i32 %r
 }
 No newline at end of file

llvm/test/Instrumentation/DataFlowSanitizer/origin_select.ll

-; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=1 -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -S | FileCheck %s --check-prefixes=CHECK,TRACK_CONTROL_FLOW
-; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=0 -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -S | FileCheck %s --check-prefixes=CHECK,NO_TRACK_CONTROL_FLOW
-; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=1 -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,TRACK_CONTROL_FLOW
-; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=0 -dfsan-track-origins=1 -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,NO_TRACK_CONTROL_FLOW
+; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=1 -dfsan-track-origins=1  -S | FileCheck %s --check-prefixes=CHECK,TRACK_CONTROL_FLOW
+; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=0 -dfsan-track-origins=1  -S | FileCheck %s --check-prefixes=CHECK,NO_TRACK_CONTROL_FLOW
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
define <4 x i8> @select8v(<4 x i1> %c, <4 x i8> %t, <4 x i8> %f) {
   ; NO_TRACK_CONTROL_FLOW: [[FS:%.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 4) to i[[#SBITS]]*), align 2
   ; NO_TRACK_CONTROL_FLOW: [[FS_NE:%.*]] = icmp ne i[[#SBITS]] [[FS]], 0
   ; NO_TRACK_CONTROL_FLOW: [[FTO:%.*]] = select i1 [[FS_NE]], i32 [[FO]], i32 [[TO]]
   ; NO_TRACK_CONTROL_FLOW: store i32 [[FTO]], i32* @__dfsan_retval_origin_tls, align 4
 
   %a = select <4 x i1> %c, <4 x i8> %t, <4 x i8> %f
   ret <4 x i8> %a
 }
 No newline at end of file

llvm/test/Instrumentation/DataFlowSanitizer/origin_store.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels -dfsan-combine-pointer-labels-on-store -S | FileCheck %s --check-prefixes=CHECK,COMBINE_STORE_PTR
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels -S | FileCheck %s --check-prefixes=CHECK,CHECK16
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels -dfsan-combine-pointer-labels-on-store -S | FileCheck %s --check-prefixes=CHECK,CHECK16,COMBINE_STORE_PTR
+; RUN: opt < %s -dfsan -dfsan-track-origins=1 -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-combine-pointer-labels-on-store -S | FileCheck %s --check-prefixes=CHECK,COMBINE_STORE_PTR
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define void @store_zero_to_non_escaped_alloca() {
   ; CHECK-LABEL: @"dfs$store_zero_to_non_escaped_alloca"

 }
 
 define void @store_nonzero_to_escaped_alloca(i16 %a) {
   ; CHECK-LABEL:  @"dfs$store_nonzero_to_escaped_alloca"
   ; CHECK-NEXT:   %[[#AO:]] = load i32, i32* getelementptr inbounds ([200 x i32], [200 x i32]* @__dfsan_arg_origin_tls, i64 0, i64 0), align 4
   ; CHECK-NEXT:   %[[#AS:]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; CHECK:        %[[#INTP:]] = ptrtoint i16* %p to i64
   ; CHECK-NEXT:   %[[#SHADOW_ADDR:]] = and i64 %[[#INTP]], [[#%.10d,MASK:]]
-  ; CHECK16-NEXT: %[[#SHADOW_ADDR:]] = mul i64 %[[#SHADOW_ADDR]], 2
   ; CHECK-NEXT:   %[[#SHADOW_PTR0:]] = inttoptr i64 %[[#SHADOW_ADDR]] to i[[#SBITS]]*
   ; CHECK-NEXT:   %[[#ORIGIN_OFFSET:]] = add i64 %[[#INTP+1]], [[#%.10d,ORIGIN_MASK:]]
   ; CHECK-NEXT:   %[[#ORIGIN_ADDR:]] = and i64 %[[#ORIGIN_OFFSET]], -4
   ; CHECK-NEXT:   %[[#ORIGIN_PTR:]] = inttoptr i64 %[[#ORIGIN_ADDR]] to i32*
   ; CHECK:        %_dfscmp = icmp ne i[[#SBITS]] %[[#AS]], 0
   ; CHECK-NEXT:   br i1 %_dfscmp, label %[[L1:.*]], label %[[L2:.*]],
   ; CHECK:       [[L1]]:
   ; CHECK-NEXT:   %[[#NO:]] = call zeroext i32 @__dfsan_chain_origin(i32 zeroext %[[#AO]])

llvm/test/Instrumentation/DataFlowSanitizer/origin_store_threshold.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-8-labels=true  -dfsan-instrument-with-call-threshold=0 -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=1 -dfsan-fast-16-labels=true -dfsan-instrument-with-call-threshold=0 -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=1  -dfsan-instrument-with-call-threshold=0 -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define void @store_threshold([2 x i64]* %p, [2 x i64] %a) {
   ; CHECK: @"dfs$store_threshold"

llvm/test/Instrumentation/DataFlowSanitizer/origin_track_load.ll

-; RUN: opt < %s -dfsan -dfsan-track-origins=2 -dfsan-fast-8-labels -S | FileCheck %s
-; RUN: opt < %s -dfsan -dfsan-track-origins=2 -dfsan-fast-16-labels -S | FileCheck %s
+; RUN: opt < %s -dfsan -dfsan-track-origins=2 -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define i64 @load64(i64* %p) {
   ; CHECK-LABEL: @"dfs$load64"

llvm/test/Instrumentation/DataFlowSanitizer/phi.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,LEGACY
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,FAST
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -S | FileCheck %s --check-prefixes=CHECK,FAST
+; RUN: opt < %s -dfsan -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define {i32, i32} @test({i32, i32} %a, i1 %c) {
-  ; LEGACY: [[AL:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; LEGACY: [[PL:%.*]] = phi i[[#SBITS]] [ [[AL]], %T ], [ [[AL]], %F ]
-  ; LEGACY: store i[[#SBITS]] [[PL]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-
-  ; FAST: [[AL:%.*]] = load { i[[#SBITS]], i[[#SBITS]] }, { i[[#SBITS]], i[[#SBITS]] }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN:2]]
-  ; FAST: [[AL0:%.*]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } [[AL]], i[[#SBITS]] 0, 0
-  ; FAST: [[AL1:%.*]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } [[AL]], i[[#SBITS]] 0, 1
-  ; FAST: [[PL:%.*]] = phi { i[[#SBITS]], i[[#SBITS]] } [ [[AL0]], %T ], [ [[AL1]], %F ]
-  ; FAST: store { i[[#SBITS]], i[[#SBITS]] } [[PL]], { i[[#SBITS]], i[[#SBITS]] }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN]]
+  ; CHECK: %[[#AL:]] = load { i[[#SBITS]], i[[#SBITS]] }, { i[[#SBITS]], i[[#SBITS]] }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN:2]]
+  ; CHECK: %[[#AL0:]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } %[[#AL]], i[[#SBITS]] 0, 0
+  ; CHECK: %[[#AL1:]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } %[[#AL]], i[[#SBITS]] 0, 1
+  ; CHECK: %[[#PL:]] = phi { i[[#SBITS]], i[[#SBITS]] } [ %[[#AL0]], %T ], [ %[[#AL1]], %F ]
+  ; CHECK: store { i[[#SBITS]], i[[#SBITS]] } %[[#PL]], { i[[#SBITS]], i[[#SBITS]] }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN]]
 
 entry:
   br i1 %c, label %T, label %F
   
 T:
   %at = insertvalue {i32, i32} %a, i32 1, 0
   br label %done
   
 F:
   %af = insertvalue {i32, i32} %a, i32 1, 1
   br label %done
   
 done:
   %b = phi {i32, i32} [%at, %T], [%af, %F]
   ret {i32, i32} %b  
 }

llvm/test/Instrumentation/DataFlowSanitizer/select.ll

-; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=1 -S | FileCheck %s --check-prefixes=CHECK,TRACK_CF,TRACK_CF_LEGACY
-; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=0 -S | FileCheck %s --check-prefixes=CHECK,NO_TRACK_CF,NO_TRACK_CF_LEGACY
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels -dfsan-track-select-control-flow=1 -S | FileCheck %s --check-prefixes=CHECK,TRACK_CF,TRACK_CF_FAST
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels -dfsan-track-select-control-flow=0 -S | FileCheck %s --check-prefixes=CHECK,NO_TRACK_CF,NO_TRACK_CF_FAST
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels -dfsan-track-select-control-flow=1 -S | FileCheck %s --check-prefixes=CHECK,TRACK_CF,TRACK_CF_FAST
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels -dfsan-track-select-control-flow=0 -S | FileCheck %s --check-prefixes=CHECK,NO_TRACK_CF,NO_TRACK_CF_FAST
+; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=true -S | FileCheck %s --check-prefixes=CHECK,TRACK_CF
+; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=false -S | FileCheck %s --check-prefixes=CHECK,NO_TRACK_CF
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define i8 @select8(i1 %c, i8 %t, i8 %f) {
   ; TRACK_CF: @"dfs$select8"
   ; TRACK_CF: %[[#R:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 4) to i[[#SBITS]]*), align [[ALIGN:2]]
   ; TRACK_CF: %[[#R+1]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
   ; TRACK_CF: %[[#R+2]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; TRACK_CF: %[[#R+3]] = select i1 %c, i[[#SBITS]] %[[#R+1]], i[[#SBITS]] %[[#R]]
-  ; TRACK_CF_LEGACY: %[[#R+4]] = icmp ne i[[#SBITS]] %[[#R+2]], %[[#R+3]]
-  ; TRACK_CF_LEGACY: %[[#R+6]] = call {{.*}} i[[#SBITS]] @__dfsan_union(i[[#SBITS]] {{.*}} %[[#R+2]], i[[#SBITS]] {{.*}} %[[#R+3]])
-  ; TRACK_CF_LEGACY: %[[#RO:]] = phi i[[#SBITS]] [ %[[#R+6]], {{.*}} ], [ %[[#R+2]], {{.*}} ]
-  ; COMM: The union is simply an OR when fast labels are used.
-  ; TRACK_CF_FAST: %[[#RO:]] = or i[[#SBITS]] %[[#R+2]], %[[#R+3]]
+  ; TRACK_CF: %[[#RO:]] = or i[[#SBITS]] %[[#R+2]], %[[#R+3]]
   ; TRACK_CF: %a = select i1 %c, i8 %t, i8 %f
   ; TRACK_CF: store i[[#SBITS]] %[[#RO]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; TRACK_CF: ret i8 %a
 
   ; NO_TRACK_CF: @"dfs$select8"
   ; NO_TRACK_CF: %[[#R:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 4) to i[[#SBITS]]*), align [[ALIGN:2]]
   ; NO_TRACK_CF: %[[#R+1]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
   ; NO_TRACK_CF: %[[#R+2]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; NO_TRACK_CF: %[[#R+3]] = select i1 %c, i[[#SBITS]] %[[#R+1]], i[[#SBITS]] %[[#R]]
   ; NO_TRACK_CF: %a = select i1 %c, i8 %t, i8 %f
   ; NO_TRACK_CF: store i[[#SBITS]] %[[#R+3]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; NO_TRACK_CF: ret i8 %a
 
   %a = select i1 %c, i8 %t, i8 %f
   ret i8 %a
 }
 
 define i8 @select8e(i1 %c, i8 %tf) {
   ; TRACK_CF: @"dfs$select8e"
   ; TRACK_CF: %[[#R:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
   ; TRACK_CF: %[[#R+1]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; TRACK_CF_LEGACY: %[[#R+2]] = icmp ne i[[#SBITS]] %[[#R+1]], %[[#R]]
-  ; TRACK_CF_LEGACY: %[[#R+4]] = call {{.*}} i[[#SBITS]] @__dfsan_union(i[[#SBITS]] {{.*}} %[[#R+1]], i[[#SBITS]] {{.*}} %[[#R]])
-  ; TRACK_CF_LEGACY: %[[#RO:]] = phi i[[#SBITS]] [ %[[#R+4]], {{.*}} ], [ %[[#R+1]], {{.*}} ]
-  ; COMM: The union is simply an OR when fast labels are used.
-  ; TRACK_CF_FAST: %[[#RO:]] = or i[[#SBITS]] %[[#R+1]], %[[#R]]
+  ; TRACK_CF: %[[#RO:]] = or i[[#SBITS]] %[[#R+1]], %[[#R]]
   ; TRACK_CF: %a = select i1 %c, i8 %tf, i8 %tf
   ; TRACK_CF: store i[[#SBITS]] %[[#RO]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; TRACK_CF: ret i8 %a
 
   ; NO_TRACK_CF: @"dfs$select8e"
   ; NO_TRACK_CF: %[[#R:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
   ; NO_TRACK_CF: %[[#R+1]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; NO_TRACK_CF: %a = select i1 %c, i8 %tf, i8 %tf
   ; NO_TRACK_CF: store i[[#SBITS]] %[[#R]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; NO_TRACK_CF: ret i8 %a
 
   %a = select i1 %c, i8 %tf, i8 %tf
   ret i8 %a
 }
 
 define <4 x i8> @select8v(<4 x i1> %c, <4 x i8> %t, <4 x i8> %f) {
   ; TRACK_CF: @"dfs$select8v"
   ; TRACK_CF: %[[#R:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 4) to i[[#SBITS]]*), align [[ALIGN:2]]
   ; TRACK_CF: %[[#R+1]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
   ; TRACK_CF: %[[#R+2]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; TRACK_CF_LEGACY: %[[#R+3]] = icmp ne i[[#SBITS]] %[[#R+1]], %[[#R]]
-  ; TRACK_CF_LEGACY: %[[#R+5]] = call {{.*}} i[[#SBITS]] @__dfsan_union(i[[#SBITS]] {{.*}} %[[#R+1]], i[[#SBITS]] zeroext %[[#R]])
-  ; TRACK_CF_LEGACY: %[[#R+7]] = phi i[[#SBITS]] [ %[[#R+5]], {{.*}} ], [ %[[#R+1]], {{.*}} ]
-  ; TRACK_CF_LEGACY: %[[#R+8]] = icmp ne i[[#SBITS]] %[[#R+2]], %[[#R+7]]
-  ; TRACK_CF_LEGACY: %[[#R+10]] = call {{.*}} i[[#SBITS]] @__dfsan_union(i[[#SBITS]] {{.*}} %[[#R+2]], i[[#SBITS]] zeroext %[[#R+7]])
-  ; TRACK_CF_LEGACY: %[[#RO:]] = phi i[[#SBITS]] [ %[[#R+10]], {{.*}} ], [ %[[#R+2]], {{.*}} ]
-  ; COMM: The union is simply an OR when fast labels are used.
-  ; TRACK_CF_FAST: %[[#R+3]] = or i[[#SBITS]] %[[#R+1]], %[[#R]]
-  ; TRACK_CF_FAST: %[[#RO:]] = or i[[#SBITS]] %[[#R+2]], %[[#R+3]]
+  ; TRACK_CF: %[[#R+3]] = or i[[#SBITS]] %[[#R+1]], %[[#R]]
+  ; TRACK_CF: %[[#RO:]] = or i[[#SBITS]] %[[#R+2]], %[[#R+3]]
   ; TRACK_CF: %a = select <4 x i1> %c, <4 x i8> %t, <4 x i8> %f
   ; TRACK_CF: store i[[#SBITS]] %[[#RO]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; TRACK_CF: ret <4 x i8> %a
 
   ; NO_TRACK_CF: @"dfs$select8v"
   ; NO_TRACK_CF: %[[#R:]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 4) to i[[#SBITS]]*), align [[ALIGN:2]]
   ; NO_TRACK_CF: %[[#R+1]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN]]
   ; NO_TRACK_CF: %[[#R+2]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; NO_TRACK_CF_LEGACY: %[[#R+3]] = icmp ne i[[#SBITS]] %[[#R+1]], %[[#R]]
-  ; NO_TRACK_CF_LEGACY: %[[#R+5]] = call {{.*}} i[[#SBITS]] @__dfsan_union(i[[#SBITS]] {{.*}} %[[#R+1]], i[[#SBITS]] {{.*}} %[[#R]])
-  ; NO_TRACK_CF_LEGACY: %[[#RO:]] = phi i[[#SBITS]] [ %6, {{.*}} ], [ %2, {{.*}} ]
-  ; COMM: The union is simply an OR when fast labels are used.
-  ; NO_TRACK_CF_FAST: %[[#RO:]] = or i[[#SBITS]] %[[#R+1]], %[[#R]]
+  ; NO_TRACK_CF: %[[#RO:]] = or i[[#SBITS]] %[[#R+1]], %[[#R]]
   ; NO_TRACK_CF: %a = select <4 x i1> %c, <4 x i8> %t, <4 x i8> %f
   ; NO_TRACK_CF: store i[[#SBITS]] %[[#RO]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; NO_TRACK_CF: ret <4 x i8> %a
 
   %a = select <4 x i1> %c, <4 x i8> %t, <4 x i8> %f
   ret <4 x i8> %a
 }

llvm/test/Instrumentation/DataFlowSanitizer/shadow-args-zext.ll

 ; RUN: opt -mtriple=x86_64-unknown-linux-gnu < %s -dfsan -S --dfsan-abilist=%S/Inputs/shadow-args-abilist.txt | FileCheck %s
-; RUN: opt -mtriple=x86_64-unknown-linux-gnu < %s -dfsan -S --dfsan-abilist=%S/Inputs/shadow-args-abilist.txt -dfsan-fast-16-labels | FileCheck %s
-; RUN: opt -mtriple=x86_64-unknown-linux-gnu < %s -dfsan -S --dfsan-abilist=%S/Inputs/shadow-args-abilist.txt -dfsan-fast-8-labels | FileCheck %s
 
 ; REQUIRES: x86-registered-target
 
 ; Test that the custom abi marks shadow parameters as zero extended.
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 

llvm/test/Instrumentation/DataFlowSanitizer/store.ll

-; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-store=1 -S | FileCheck %s --check-prefixes=CHECK,CHECK16,COMBINE_PTR_LABEL
-; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-store=0 -S | FileCheck %s --check-prefixes=CHECK,CHECK16,NO_COMBINE_PTR_LABEL
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels -dfsan-combine-pointer-labels-on-store=1 -S | FileCheck %s --check-prefixes=CHECK,CHECK16,COMBINE_PTR_LABEL_FAST
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels -dfsan-combine-pointer-labels-on-store=0 -S | FileCheck %s --check-prefixes=CHECK,CHECK16,NO_COMBINE_PTR_LABEL
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels -dfsan-combine-pointer-labels-on-store=1 -S | FileCheck %s --check-prefixes=CHECK,COMBINE_PTR_LABEL_FAST
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels -dfsan-combine-pointer-labels-on-store=0 -S | FileCheck %s --check-prefixes=CHECK,NO_COMBINE_PTR_LABEL
+; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-store=1 -S | FileCheck %s --check-prefixes=CHECK,COMBINE_PTR_LABEL
+; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-store=0 -S | FileCheck %s --check-prefixes=CHECK,NO_COMBINE_PTR_LABEL
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define void @store0({} %v, {}* %p) {
   ; CHECK-LABEL: @"dfs$store0"
   ; CHECK:       store {} %v, {}* %p
   ; CHECK-NOT:   store
   ; CHECK:       ret void
 
   store {} %v, {}* %p
   ret void
 }
 
 define void @store8(i8 %v, i8* %p) {
   ; CHECK-LABEL:       @"dfs$store8"
   ; CHECK:             load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-
   ; COMBINE_PTR_LABEL: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-  ; COMBINE_PTR_LABEL: icmp ne i[[#SBITS]]
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-
-  ; COMM: When not in legacy mode, the three instructions above will
-  ;        be replaced with the following:
-  ; COMBINE_PTR_LABEL_FAST: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-  ; COMBINE_PTR_LABEL_FAST: or i[[#SBITS]]
-
+  ; COMBINE_PTR_LABEL: or i[[#SBITS]]
   ; CHECK:             ptrtoint i8* {{.*}} i64
   ; CHECK-NEXT:        and i64
-  ; CHECK16-NEXT:      mul i64
   ; CHECK-NEXT:        inttoptr i64 {{.*}} i[[#SBITS]]*
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        store i8 %v, i8* %p
   ; CHECK-NEXT:        ret void
 
   store i8 %v, i8* %p
   ret void
 }
 
 define void @store16(i16 %v, i16* %p) {
   ; CHECK-LABEL:       @"dfs$store16"
   ; CHECK:             load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-
   ; COMBINE_PTR_LABEL: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-  ; COMBINE_PTR_LABEL: icmp ne i[[#SBITS]]
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-
-  ; COMM: When not in legacy mode, the three instructions above will
-  ;        be replaced with the following:
-  ; COMBINE_PTR_LABEL_FAST: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-  ; COMBINE_PTR_LABEL_FAST: or i[[#SBITS]]
-
+  ; COMBINE_PTR_LABEL: or i[[#SBITS]]
   ; CHECK:             ptrtoint i16* {{.*}} i64
   ; CHECK-NEXT:        and i64
-  ; CHECK16-NEXT:      mul i64
   ; CHECK-NEXT:        inttoptr i64 {{.*}} i[[#SBITS]]*
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        store i16 %v, i16* %p
   ; CHECK-NEXT:        ret void
 
   store i16 %v, i16* %p
   ret void
 }
 
 define void @store32(i32 %v, i32* %p) {
   ; CHECK-LABEL:       @"dfs$store32"
   ; CHECK:             load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-
   ; COMBINE_PTR_LABEL: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-  ; COMBINE_PTR_LABEL: icmp ne i[[#SBITS]]
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-
-  ; COMM: When not in legacy mode, the three instructions above will
-  ;        be replaced with the following:
-  ; COMBINE_PTR_LABEL_FAST: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-  ; COMBINE_PTR_LABEL_FAST: or i[[#SBITS]]
-
+  ; COMBINE_PTR_LABEL: or i[[#SBITS]]
   ; CHECK:             ptrtoint i32* {{.*}} i64
   ; CHECK-NEXT:        and i64
-  ; CHECK16-NEXT:      mul i64
   ; CHECK-NEXT:        inttoptr i64 {{.*}} i[[#SBITS]]*
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        getelementptr i[[#SBITS]], i[[#SBITS]]*
   ; CHECK-NEXT:        store i[[#SBITS]]
   ; CHECK-NEXT:        store i32 %v, i32* %p
   ; CHECK-NEXT:        ret void
 
   store i32 %v, i32* %p
   ret void
 }
 
 define void @store64(i64 %v, i64* %p) {
   ; CHECK-LABEL:       @"dfs$store64"
   ; CHECK:             load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-
   ; COMBINE_PTR_LABEL: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-  ; COMBINE_PTR_LABEL: icmp ne i[[#SBITS]]
-  ; COMBINE_PTR_LABEL: call {{.*}} @__dfsan_union
-
-  ; COMM: When not in legacy mode, the three instructions above will
-  ;       be replaced with the following:
-  ; COMBINE_PTR_LABEL_FAST: load i[[#SBITS]], i[[#SBITS]]* {{.*}} @__dfsan_arg_tls
-  ; COMBINE_PTR_LABEL_FAST: or i[[#SBITS]]
-
+  ; COMBINE_PTR_LABEL: or i[[#SBITS]]
   ; CHECK:             ptrtoint i64* {{.*}} i64
   ; CHECK-NEXT:        and i64
-  ; CHECK16-NEXT:      mul i64
   ; CHECK-NEXT:        inttoptr i64 {{.*}} i[[#SBITS]]*
   ; CHECK-COUNT-8:     insertelement {{.*}} i[[#SBITS]]
   ; CHECK-NEXT:        bitcast i[[#SBITS]]* {{.*}} <8 x i[[#SBITS]]>*
   ; CHECK-NEXT:        getelementptr <8 x i[[#SBITS]]>
   ; CHECK-NEXT:        store <8 x i[[#SBITS]]>
   ; CHECK-NEXT:        store i64 %v, i64* %p
   ; CHECK-NEXT:        ret void
 

llvm/test/Instrumentation/DataFlowSanitizer/struct.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,LEGACY
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-event-callbacks=true -S | FileCheck %s --check-prefixes=CHECK,EVENT_CALLBACKS
+; RUN: opt < %s -dfsan -dfsan-event-callbacks=true -S | FileCheck %s --check-prefixes=CHECK,EVENT_CALLBACKS
 ; RUN: opt < %s -dfsan -dfsan-args-abi -S | FileCheck %s --check-prefixes=CHECK,ARGS_ABI
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,FAST
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefixes=CHECK,NO_COMBINE_LOAD_PTR
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-combine-pointer-labels-on-store=true -S | FileCheck %s --check-prefixes=CHECK,COMBINE_STORE_PTR
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-track-select-control-flow=false -S | FileCheck %s --check-prefixes=CHECK,NO_SELECT_CONTROL
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-debug-nonzero-labels -S | FileCheck %s --check-prefixes=CHECK,DEBUG_NONZERO_LABELS
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -S | FileCheck %s --check-prefixes=CHECK,FAST
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefixes=CHECK,NO_COMBINE_LOAD_PTR
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-combine-pointer-labels-on-store=true -S | FileCheck %s --check-prefixes=CHECK,COMBINE_STORE_PTR
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-track-select-control-flow=false -S | FileCheck %s --check-prefixes=CHECK,NO_SELECT_CONTROL
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -dfsan-debug-nonzero-labels -S | FileCheck %s --check-prefixes=CHECK,DEBUG_NONZERO_LABELS
+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,FAST
+; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefixes=CHECK,NO_COMBINE_LOAD_PTR
+; RUN: opt < %s -dfsan -dfsan-combine-pointer-labels-on-store=true -S | FileCheck %s --check-prefixes=CHECK,COMBINE_STORE_PTR
+; RUN: opt < %s -dfsan -dfsan-track-select-control-flow=false -S | FileCheck %s --check-prefixes=CHECK,NO_SELECT_CONTROL
+; RUN: opt < %s -dfsan -dfsan-debug-nonzero-labels -S | FileCheck %s --check-prefixes=CHECK,DEBUG_NONZERO_LABELS
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_arg_tls = external thread_local(initialexec) global [[TLS_ARR:\[100 x i64\]]]
 ; CHECK: @__dfsan_retval_tls = external thread_local(initialexec) global [[TLS_ARR]]
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
define {i1, i32} @select_struct(i1 %c, {i1, i32} %a, {i1, i32} %b) {
   ; FAST: %[[#R+4]] = extractvalue { i[[#SBITS]], i[[#SBITS]] } %[[#R+3]], 0
   ; FAST: %[[#R+5]] = extractvalue { i[[#SBITS]], i[[#SBITS]] } %[[#R+3]], 1
   ; FAST: %[[#R+6]] = or i[[#SBITS]] %[[#R+4]], %[[#R+5]]
   ; FAST: %[[#R+7]] = or i[[#SBITS]] %[[#R+2]], %[[#R+6]]
   ; FAST: %[[#R+8]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } undef, i[[#SBITS]] %[[#R+7]], 0
   ; FAST: %[[#R+9]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } %[[#R+8]], i[[#SBITS]] %[[#R+7]], 1
   ; FAST: store { i[[#SBITS]], i[[#SBITS]] } %[[#R+9]], { i[[#SBITS]], i[[#SBITS]] }* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN]]
 
-  ; LEGACY: @"dfs$select_struct"
-  ; LEGACY: [[U:%.*]] = call zeroext i[[#SBITS]] @__dfsan_union
-  ; LEGACY: [[P:%.*]] = phi i[[#SBITS]] [ [[U]],
-  ; LEGACY: store i[[#SBITS]] [[P]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
-
   %s = select i1 %c, {i1, i32} %a, {i1, i32} %b
   ret {i1, i32} %s
 }
 
 define { i32, i32 } @asm_struct(i32 %0, i32 %1) {
   ; FAST: @"dfs$asm_struct"
   ; FAST: [[E1:%.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN:2]]
   ; FAST: [[E0:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; FAST: [[E01:%.*]] = or i[[#SBITS]] [[E0]], [[E1]]
   ; FAST: [[S0:%.*]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } undef, i[[#SBITS]] [[E01]], 0
   ; FAST: [[S1:%.*]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } [[S0]], i[[#SBITS]] [[E01]], 1
   ; FAST: store { i[[#SBITS]], i[[#SBITS]] } [[S1]], { i[[#SBITS]], i[[#SBITS]] }* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN]]
 
-  ; LEGACY: @"dfs$asm_struct"
-  ; LEGACY: [[E1:%.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; LEGACY: [[E0:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; LEGACY: [[E01:%.*]] = call zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext [[E0]], i[[#SBITS]] zeroext [[E1]])
-  ; LEGACY: [[P:%.*]] = phi i[[#SBITS]] [ [[E01]], {{.*}} ], [ [[E0]], {{.*}} ]
-  ; LEGACY: store i[[#SBITS]] [[P]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
-
 entry:
   %a = call { i32, i32 } asm "", "=r,=r,r,r,~{dirflag},~{fpsr},~{flags}"(i32 %0, i32 %1)
   ret { i32, i32 } %a
 }
 
 define {i32, i32} @const_struct() {
   ; FAST: @"dfs$const_struct"
   ; FAST: store { i[[#SBITS]], i[[#SBITS]] } zeroinitializer, { i[[#SBITS]], i[[#SBITS]] }* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to { i[[#SBITS]], i[[#SBITS]] }*), align 2
-
-  ; LEGACY: @"dfs$const_struct"
-  ; LEGACY: store i[[#SBITS]] 0, i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align 2
   ret {i32, i32} { i32 42, i32 11 }
 }
 
 define i1 @extract_struct({i1, i5} %s) {
   ; FAST: @"dfs$extract_struct"
   ; FAST: [[SM:%.*]] = load { i[[#SBITS]], i[[#SBITS]] }, { i[[#SBITS]], i[[#SBITS]] }* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN:2]]
   ; FAST: [[EM:%.*]] = extractvalue { i[[#SBITS]], i[[#SBITS]] } [[SM]], 0
   ; FAST: store i[[#SBITS]] [[EM]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
 
-  ; LEGACY: @"dfs$extract_struct"
-  ; LEGACY: [[SM:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; LEGACY: store i[[#SBITS]] [[SM]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   %e2 = extractvalue {i1, i5} %s, 0
   ret i1 %e2
 }
 
 define {i1, i5} @insert_struct({i1, i5} %s, i5 %e1) {
   ; FAST: @"dfs$insert_struct"
   ; FAST: [[EM:%.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 [[#mul(2, SBYTES)]]) to i[[#SBITS]]*), align [[ALIGN:2]]
   ; FAST: [[SM:%.*]] = load { i[[#SBITS]], i[[#SBITS]] }, { i[[#SBITS]], i[[#SBITS]] }* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN]]
   ; FAST: [[SM1:%.*]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } [[SM]], i[[#SBITS]] [[EM]], 1
   ; FAST: store { i[[#SBITS]], i[[#SBITS]] } [[SM1]], { i[[#SBITS]], i[[#SBITS]] }* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to { i[[#SBITS]], i[[#SBITS]] }*), align [[ALIGN]]
-
-  ; LEGACY: @"dfs$insert_struct"
-  ; LEGACY: [[EM:%.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([[TLS_ARR]]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN:2]]
-  ; LEGACY: [[SM:%.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-  ; LEGACY: [[U:%.*]] = call zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext [[SM]], i[[#SBITS]] zeroext [[EM]])
-  ; LEGACY: [[P:%.*]] = phi i[[#SBITS]] [ [[U]], {{.*}} ], [ [[SM]], {{.*}} ]
-  ; LEGACY: store i[[#SBITS]] [[P]], i[[#SBITS]]* bitcast ([[TLS_ARR]]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   %s1 = insertvalue {i1, i5} %s, i5 %e1, 1
   ret {i1, i5} %s1
 }
 
 define {i1, i1} @load_struct({i1, i1}* %p) {
   ; NO_COMBINE_LOAD_PTR: @"dfs$load_struct"
   ; NO_COMBINE_LOAD_PTR: [[OL:%.*]] = or i[[#SBITS]]
   ; NO_COMBINE_LOAD_PTR: [[S0:%.*]] = insertvalue { i[[#SBITS]], i[[#SBITS]] } undef, i[[#SBITS]] [[OL]], 0

llvm/test/Instrumentation/DataFlowSanitizer/union-large.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s
-target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
-target triple = "x86_64-unknown-linux-gnu"
-
-; Check that we use dfsan_union in large functions instead of __dfsan_union.
-
-; CHECK-LABEL: @"dfs$foo"
-define i32 @foo(i32 %a, i32 %b) {
-bb0:
-  br label %bb1
-
-bb1:
-  br label %bb2
-
-bb2:
-  br label %bb3
-
-bb3:
-  br label %bb4
-
-bb4:
-  br label %bb5
-
-bb5:
-  br label %bb6
-
-bb6:
-  br label %bb7
-
-bb7:
-  br label %bb8
-
-bb8:
-  br label %bb9
-
-bb9:
-  br label %bb10
-
-bb10:
-  br label %bb11
-
-bb11:
-  br label %bb12
-
-bb12:
-  br label %bb13
-
-bb13:
-  br label %bb14
-
-bb14:
-  br label %bb15
-
-bb15:
-  br label %bb16
-
-bb16:
-  br label %bb17
-
-bb17:
-  br label %bb18
-
-bb18:
-  br label %bb19
-
-bb19:
-  br label %bb20
-
-bb20:
-  br label %bb21
-
-bb21:
-  br label %bb22
-
-bb22:
-  br label %bb23
-
-bb23:
-  br label %bb24
-
-bb24:
-  br label %bb25
-
-bb25:
-  br label %bb26
-
-bb26:
-  br label %bb27
-
-bb27:
-  br label %bb28
-
-bb28:
-  br label %bb29
-
-bb29:
-  br label %bb30
-
-bb30:
-  br label %bb31
-
-bb31:
-  br label %bb32
-
-bb32:
-  br label %bb33
-
-bb33:
-  br label %bb34
-
-bb34:
-  br label %bb35
-
-bb35:
-  br label %bb36
-
-bb36:
-  br label %bb37
-
-bb37:
-  br label %bb38
-
-bb38:
-  br label %bb39
-
-bb39:
-  br label %bb40
-
-bb40:
-  br label %bb41
-
-bb41:
-  br label %bb42
-
-bb42:
-  br label %bb43
-
-bb43:
-  br label %bb44
-
-bb44:
-  br label %bb45
-
-bb45:
-  br label %bb46
-
-bb46:
-  br label %bb47
-
-bb47:
-  br label %bb48
-
-bb48:
-  br label %bb49
-
-bb49:
-  br label %bb50
-
-bb50:
-  br label %bb51
-
-bb51:
-  br label %bb52
-
-bb52:
-  br label %bb53
-
-bb53:
-  br label %bb54
-
-bb54:
-  br label %bb55
-
-bb55:
-  br label %bb56
-
-bb56:
-  br label %bb57
-
-bb57:
-  br label %bb58
-
-bb58:
-  br label %bb59
-
-bb59:
-  br label %bb60
-
-bb60:
-  br label %bb61
-
-bb61:
-  br label %bb62
-
-bb62:
-  br label %bb63
-
-bb63:
-  br label %bb64
-
-bb64:
-  br label %bb65
-
-bb65:
-  br label %bb66
-
-bb66:
-  br label %bb67
-
-bb67:
-  br label %bb68
-
-bb68:
-  br label %bb69
-
-bb69:
-  br label %bb70
-
-bb70:
-  br label %bb71
-
-bb71:
-  br label %bb72
-
-bb72:
-  br label %bb73
-
-bb73:
-  br label %bb74
-
-bb74:
-  br label %bb75
-
-bb75:
-  br label %bb76
-
-bb76:
-  br label %bb77
-
-bb77:
-  br label %bb78
-
-bb78:
-  br label %bb79
-
-bb79:
-  br label %bb80
-
-bb80:
-  br label %bb81
-
-bb81:
-  br label %bb82
-
-bb82:
-  br label %bb83
-
-bb83:
-  br label %bb84
-
-bb84:
-  br label %bb85
-
-bb85:
-  br label %bb86
-
-bb86:
-  br label %bb87
-
-bb87:
-  br label %bb88
-
-bb88:
-  br label %bb89
-
-bb89:
-  br label %bb90
-
-bb90:
-  br label %bb91
-
-bb91:
-  br label %bb92
-
-bb92:
-  br label %bb93
-
-bb93:
-  br label %bb94
-
-bb94:
-  br label %bb95
-
-bb95:
-  br label %bb96
-
-bb96:
-  br label %bb97
-
-bb97:
-  br label %bb98
-
-bb98:
-  br label %bb99
-
-bb99:
-  br label %bb100
-
-bb100:
-  br label %bb101
-
-bb101:
-  br label %bb102
-
-bb102:
-  br label %bb103
-
-bb103:
-  br label %bb104
-
-bb104:
-  br label %bb105
-
-bb105:
-  br label %bb106
-
-bb106:
-  br label %bb107
-
-bb107:
-  br label %bb108
-
-bb108:
-  br label %bb109
-
-bb109:
-  br label %bb110
-
-bb110:
-  br label %bb111
-
-bb111:
-  br label %bb112
-
-bb112:
-  br label %bb113
-
-bb113:
-  br label %bb114
-
-bb114:
-  br label %bb115
-
-bb115:
-  br label %bb116
-
-bb116:
-  br label %bb117
-
-bb117:
-  br label %bb118
-
-bb118:
-  br label %bb119
-
-bb119:
-  br label %bb120
-
-bb120:
-  br label %bb121
-
-bb121:
-  br label %bb122
-
-bb122:
-  br label %bb123
-
-bb123:
-  br label %bb124
-
-bb124:
-  br label %bb125
-
-bb125:
-  br label %bb126
-
-bb126:
-  br label %bb127
-
-bb127:
-  br label %bb128
-
-bb128:
-  br label %bb129
-
-bb129:
-  br label %bb130
-
-bb130:
-  br label %bb131
-
-bb131:
-  br label %bb132
-
-bb132:
-  br label %bb133
-
-bb133:
-  br label %bb134
-
-bb134:
-  br label %bb135
-
-bb135:
-  br label %bb136
-
-bb136:
-  br label %bb137
-
-bb137:
-  br label %bb138
-
-bb138:
-  br label %bb139
-
-bb139:
-  br label %bb140
-
-bb140:
-  br label %bb141
-
-bb141:
-  br label %bb142
-
-bb142:
-  br label %bb143
-
-bb143:
-  br label %bb144
-
-bb144:
-  br label %bb145
-
-bb145:
-  br label %bb146
-
-bb146:
-  br label %bb147
-
-bb147:
-  br label %bb148
-
-bb148:
-  br label %bb149
-
-bb149:
-  br label %bb150
-
-bb150:
-  br label %bb151
-
-bb151:
-  br label %bb152
-
-bb152:
-  br label %bb153
-
-bb153:
-  br label %bb154
-
-bb154:
-  br label %bb155
-
-bb155:
-  br label %bb156
-
-bb156:
-  br label %bb157
-
-bb157:
-  br label %bb158
-
-bb158:
-  br label %bb159
-
-bb159:
-  br label %bb160
-
-bb160:
-  br label %bb161
-
-bb161:
-  br label %bb162
-
-bb162:
-  br label %bb163
-
-bb163:
-  br label %bb164
-
-bb164:
-  br label %bb165
-
-bb165:
-  br label %bb166
-
-bb166:
-  br label %bb167
-
-bb167:
-  br label %bb168
-
-bb168:
-  br label %bb169
-
-bb169:
-  br label %bb170
-
-bb170:
-  br label %bb171
-
-bb171:
-  br label %bb172
-
-bb172:
-  br label %bb173
-
-bb173:
-  br label %bb174
-
-bb174:
-  br label %bb175
-
-bb175:
-  br label %bb176
-
-bb176:
-  br label %bb177
-
-bb177:
-  br label %bb178
-
-bb178:
-  br label %bb179
-
-bb179:
-  br label %bb180
-
-bb180:
-  br label %bb181
-
-bb181:
-  br label %bb182
-
-bb182:
-  br label %bb183
-
-bb183:
-  br label %bb184
-
-bb184:
-  br label %bb185
-
-bb185:
-  br label %bb186
-
-bb186:
-  br label %bb187
-
-bb187:
-  br label %bb188
-
-bb188:
-  br label %bb189
-
-bb189:
-  br label %bb190
-
-bb190:
-  br label %bb191
-
-bb191:
-  br label %bb192
-
-bb192:
-  br label %bb193
-
-bb193:
-  br label %bb194
-
-bb194:
-  br label %bb195
-
-bb195:
-  br label %bb196
-
-bb196:
-  br label %bb197
-
-bb197:
-  br label %bb198
-
-bb198:
-  br label %bb199
-
-bb199:
-  br label %bb200
-
-bb200:
-  br label %bb201
-
-bb201:
-  br label %bb202
-
-bb202:
-  br label %bb203
-
-bb203:
-  br label %bb204
-
-bb204:
-  br label %bb205
-
-bb205:
-  br label %bb206
-
-bb206:
-  br label %bb207
-
-bb207:
-  br label %bb208
-
-bb208:
-  br label %bb209
-
-bb209:
-  br label %bb210
-
-bb210:
-  br label %bb211
-
-bb211:
-  br label %bb212
-
-bb212:
-  br label %bb213
-
-bb213:
-  br label %bb214
-
-bb214:
-  br label %bb215
-
-bb215:
-  br label %bb216
-
-bb216:
-  br label %bb217
-
-bb217:
-  br label %bb218
-
-bb218:
-  br label %bb219
-
-bb219:
-  br label %bb220
-
-bb220:
-  br label %bb221
-
-bb221:
-  br label %bb222
-
-bb222:
-  br label %bb223
-
-bb223:
-  br label %bb224
-
-bb224:
-  br label %bb225
-
-bb225:
-  br label %bb226
-
-bb226:
-  br label %bb227
-
-bb227:
-  br label %bb228
-
-bb228:
-  br label %bb229
-
-bb229:
-  br label %bb230
-
-bb230:
-  br label %bb231
-
-bb231:
-  br label %bb232
-
-bb232:
-  br label %bb233
-
-bb233:
-  br label %bb234
-
-bb234:
-  br label %bb235
-
-bb235:
-  br label %bb236
-
-bb236:
-  br label %bb237
-
-bb237:
-  br label %bb238
-
-bb238:
-  br label %bb239
-
-bb239:
-  br label %bb240
-
-bb240:
-  br label %bb241
-
-bb241:
-  br label %bb242
-
-bb242:
-  br label %bb243
-
-bb243:
-  br label %bb244
-
-bb244:
-  br label %bb245
-
-bb245:
-  br label %bb246
-
-bb246:
-  br label %bb247
-
-bb247:
-  br label %bb248
-
-bb248:
-  br label %bb249
-
-bb249:
-  br label %bb250
-
-bb250:
-  br label %bb251
-
-bb251:
-  br label %bb252
-
-bb252:
-  br label %bb253
-
-bb253:
-  br label %bb254
-
-bb254:
-  br label %bb255
-
-bb255:
-  br label %bb256
-
-bb256:
-  br label %bb257
-
-bb257:
-  br label %bb258
-
-bb258:
-  br label %bb259
-
-bb259:
-  br label %bb260
-
-bb260:
-  br label %bb261
-
-bb261:
-  br label %bb262
-
-bb262:
-  br label %bb263
-
-bb263:
-  br label %bb264
-
-bb264:
-  br label %bb265
-
-bb265:
-  br label %bb266
-
-bb266:
-  br label %bb267
-
-bb267:
-  br label %bb268
-
-bb268:
-  br label %bb269
-
-bb269:
-  br label %bb270
-
-bb270:
-  br label %bb271
-
-bb271:
-  br label %bb272
-
-bb272:
-  br label %bb273
-
-bb273:
-  br label %bb274
-
-bb274:
-  br label %bb275
-
-bb275:
-  br label %bb276
-
-bb276:
-  br label %bb277
-
-bb277:
-  br label %bb278
-
-bb278:
-  br label %bb279
-
-bb279:
-  br label %bb280
-
-bb280:
-  br label %bb281
-
-bb281:
-  br label %bb282
-
-bb282:
-  br label %bb283
-
-bb283:
-  br label %bb284
-
-bb284:
-  br label %bb285
-
-bb285:
-  br label %bb286
-
-bb286:
-  br label %bb287
-
-bb287:
-  br label %bb288
-
-bb288:
-  br label %bb289
-
-bb289:
-  br label %bb290
-
-bb290:
-  br label %bb291
-
-bb291:
-  br label %bb292
-
-bb292:
-  br label %bb293
-
-bb293:
-  br label %bb294
-
-bb294:
-  br label %bb295
-
-bb295:
-  br label %bb296
-
-bb296:
-  br label %bb297
-
-bb297:
-  br label %bb298
-
-bb298:
-  br label %bb299
-
-bb299:
-  br label %bb300
-
-bb300:
-  br label %bb301
-
-bb301:
-  br label %bb302
-
-bb302:
-  br label %bb303
-
-bb303:
-  br label %bb304
-
-bb304:
-  br label %bb305
-
-bb305:
-  br label %bb306
-
-bb306:
-  br label %bb307
-
-bb307:
-  br label %bb308
-
-bb308:
-  br label %bb309
-
-bb309:
-  br label %bb310
-
-bb310:
-  br label %bb311
-
-bb311:
-  br label %bb312
-
-bb312:
-  br label %bb313
-
-bb313:
-  br label %bb314
-
-bb314:
-  br label %bb315
-
-bb315:
-  br label %bb316
-
-bb316:
-  br label %bb317
-
-bb317:
-  br label %bb318
-
-bb318:
-  br label %bb319
-
-bb319:
-  br label %bb320
-
-bb320:
-  br label %bb321
-
-bb321:
-  br label %bb322
-
-bb322:
-  br label %bb323
-
-bb323:
-  br label %bb324
-
-bb324:
-  br label %bb325
-
-bb325:
-  br label %bb326
-
-bb326:
-  br label %bb327
-
-bb327:
-  br label %bb328
-
-bb328:
-  br label %bb329
-
-bb329:
-  br label %bb330
-
-bb330:
-  br label %bb331
-
-bb331:
-  br label %bb332
-
-bb332:
-  br label %bb333
-
-bb333:
-  br label %bb334
-
-bb334:
-  br label %bb335
-
-bb335:
-  br label %bb336
-
-bb336:
-  br label %bb337
-
-bb337:
-  br label %bb338
-
-bb338:
-  br label %bb339
-
-bb339:
-  br label %bb340
-
-bb340:
-  br label %bb341
-
-bb341:
-  br label %bb342
-
-bb342:
-  br label %bb343
-
-bb343:
-  br label %bb344
-
-bb344:
-  br label %bb345
-
-bb345:
-  br label %bb346
-
-bb346:
-  br label %bb347
-
-bb347:
-  br label %bb348
-
-bb348:
-  br label %bb349
-
-bb349:
-  br label %bb350
-
-bb350:
-  br label %bb351
-
-bb351:
-  br label %bb352
-
-bb352:
-  br label %bb353
-
-bb353:
-  br label %bb354
-
-bb354:
-  br label %bb355
-
-bb355:
-  br label %bb356
-
-bb356:
-  br label %bb357
-
-bb357:
-  br label %bb358
-
-bb358:
-  br label %bb359
-
-bb359:
-  br label %bb360
-
-bb360:
-  br label %bb361
-
-bb361:
-  br label %bb362
-
-bb362:
-  br label %bb363
-
-bb363:
-  br label %bb364
-
-bb364:
-  br label %bb365
-
-bb365:
-  br label %bb366
-
-bb366:
-  br label %bb367
-
-bb367:
-  br label %bb368
-
-bb368:
-  br label %bb369
-
-bb369:
-  br label %bb370
-
-bb370:
-  br label %bb371
-
-bb371:
-  br label %bb372
-
-bb372:
-  br label %bb373
-
-bb373:
-  br label %bb374
-
-bb374:
-  br label %bb375
-
-bb375:
-  br label %bb376
-
-bb376:
-  br label %bb377
-
-bb377:
-  br label %bb378
-
-bb378:
-  br label %bb379
-
-bb379:
-  br label %bb380
-
-bb380:
-  br label %bb381
-
-bb381:
-  br label %bb382
-
-bb382:
-  br label %bb383
-
-bb383:
-  br label %bb384
-
-bb384:
-  br label %bb385
-
-bb385:
-  br label %bb386
-
-bb386:
-  br label %bb387
-
-bb387:
-  br label %bb388
-
-bb388:
-  br label %bb389
-
-bb389:
-  br label %bb390
-
-bb390:
-  br label %bb391
-
-bb391:
-  br label %bb392
-
-bb392:
-  br label %bb393
-
-bb393:
-  br label %bb394
-
-bb394:
-  br label %bb395
-
-bb395:
-  br label %bb396
-
-bb396:
-  br label %bb397
-
-bb397:
-  br label %bb398
-
-bb398:
-  br label %bb399
-
-bb399:
-  br label %bb400
-
-bb400:
-  br label %bb401
-
-bb401:
-  br label %bb402
-
-bb402:
-  br label %bb403
-
-bb403:
-  br label %bb404
-
-bb404:
-  br label %bb405
-
-bb405:
-  br label %bb406
-
-bb406:
-  br label %bb407
-
-bb407:
-  br label %bb408
-
-bb408:
-  br label %bb409
-
-bb409:
-  br label %bb410
-
-bb410:
-  br label %bb411
-
-bb411:
-  br label %bb412
-
-bb412:
-  br label %bb413
-
-bb413:
-  br label %bb414
-
-bb414:
-  br label %bb415
-
-bb415:
-  br label %bb416
-
-bb416:
-  br label %bb417
-
-bb417:
-  br label %bb418
-
-bb418:
-  br label %bb419
-
-bb419:
-  br label %bb420
-
-bb420:
-  br label %bb421
-
-bb421:
-  br label %bb422
-
-bb422:
-  br label %bb423
-
-bb423:
-  br label %bb424
-
-bb424:
-  br label %bb425
-
-bb425:
-  br label %bb426
-
-bb426:
-  br label %bb427
-
-bb427:
-  br label %bb428
-
-bb428:
-  br label %bb429
-
-bb429:
-  br label %bb430
-
-bb430:
-  br label %bb431
-
-bb431:
-  br label %bb432
-
-bb432:
-  br label %bb433
-
-bb433:
-  br label %bb434
-
-bb434:
-  br label %bb435
-
-bb435:
-  br label %bb436
-
-bb436:
-  br label %bb437
-
-bb437:
-  br label %bb438
-
-bb438:
-  br label %bb439
-
-bb439:
-  br label %bb440
-
-bb440:
-  br label %bb441
-
-bb441:
-  br label %bb442
-
-bb442:
-  br label %bb443
-
-bb443:
-  br label %bb444
-
-bb444:
-  br label %bb445
-
-bb445:
-  br label %bb446
-
-bb446:
-  br label %bb447
-
-bb447:
-  br label %bb448
-
-bb448:
-  br label %bb449
-
-bb449:
-  br label %bb450
-
-bb450:
-  br label %bb451
-
-bb451:
-  br label %bb452
-
-bb452:
-  br label %bb453
-
-bb453:
-  br label %bb454
-
-bb454:
-  br label %bb455
-
-bb455:
-  br label %bb456
-
-bb456:
-  br label %bb457
-
-bb457:
-  br label %bb458
-
-bb458:
-  br label %bb459
-
-bb459:
-  br label %bb460
-
-bb460:
-  br label %bb461
-
-bb461:
-  br label %bb462
-
-bb462:
-  br label %bb463
-
-bb463:
-  br label %bb464
-
-bb464:
-  br label %bb465
-
-bb465:
-  br label %bb466
-
-bb466:
-  br label %bb467
-
-bb467:
-  br label %bb468
-
-bb468:
-  br label %bb469
-
-bb469:
-  br label %bb470
-
-bb470:
-  br label %bb471
-
-bb471:
-  br label %bb472
-
-bb472:
-  br label %bb473
-
-bb473:
-  br label %bb474
-
-bb474:
-  br label %bb475
-
-bb475:
-  br label %bb476
-
-bb476:
-  br label %bb477
-
-bb477:
-  br label %bb478
-
-bb478:
-  br label %bb479
-
-bb479:
-  br label %bb480
-
-bb480:
-  br label %bb481
-
-bb481:
-  br label %bb482
-
-bb482:
-  br label %bb483
-
-bb483:
-  br label %bb484
-
-bb484:
-  br label %bb485
-
-bb485:
-  br label %bb486
-
-bb486:
-  br label %bb487
-
-bb487:
-  br label %bb488
-
-bb488:
-  br label %bb489
-
-bb489:
-  br label %bb490
-
-bb490:
-  br label %bb491
-
-bb491:
-  br label %bb492
-
-bb492:
-  br label %bb493
-
-bb493:
-  br label %bb494
-
-bb494:
-  br label %bb495
-
-bb495:
-  br label %bb496
-
-bb496:
-  br label %bb497
-
-bb497:
-  br label %bb498
-
-bb498:
-  br label %bb499
-
-bb499:
-  br label %bb500
-
-bb500:
-  br label %bb501
-
-bb501:
-  br label %bb502
-
-bb502:
-  br label %bb503
-
-bb503:
-  br label %bb504
-
-bb504:
-  br label %bb505
-
-bb505:
-  br label %bb506
-
-bb506:
-  br label %bb507
-
-bb507:
-  br label %bb508
-
-bb508:
-  br label %bb509
-
-bb509:
-  br label %bb510
-
-bb510:
-  br label %bb511
-
-bb511:
-  br label %bb512
-
-bb512:
-  br label %bb513
-
-bb513:
-  br label %bb514
-
-bb514:
-  br label %bb515
-
-bb515:
-  br label %bb516
-
-bb516:
-  br label %bb517
-
-bb517:
-  br label %bb518
-
-bb518:
-  br label %bb519
-
-bb519:
-  br label %bb520
-
-bb520:
-  br label %bb521
-
-bb521:
-  br label %bb522
-
-bb522:
-  br label %bb523
-
-bb523:
-  br label %bb524
-
-bb524:
-  br label %bb525
-
-bb525:
-  br label %bb526
-
-bb526:
-  br label %bb527
-
-bb527:
-  br label %bb528
-
-bb528:
-  br label %bb529
-
-bb529:
-  br label %bb530
-
-bb530:
-  br label %bb531
-
-bb531:
-  br label %bb532
-
-bb532:
-  br label %bb533
-
-bb533:
-  br label %bb534
-
-bb534:
-  br label %bb535
-
-bb535:
-  br label %bb536
-
-bb536:
-  br label %bb537
-
-bb537:
-  br label %bb538
-
-bb538:
-  br label %bb539
-
-bb539:
-  br label %bb540
-
-bb540:
-  br label %bb541
-
-bb541:
-  br label %bb542
-
-bb542:
-  br label %bb543
-
-bb543:
-  br label %bb544
-
-bb544:
-  br label %bb545
-
-bb545:
-  br label %bb546
-
-bb546:
-  br label %bb547
-
-bb547:
-  br label %bb548
-
-bb548:
-  br label %bb549
-
-bb549:
-  br label %bb550
-
-bb550:
-  br label %bb551
-
-bb551:
-  br label %bb552
-
-bb552:
-  br label %bb553
-
-bb553:
-  br label %bb554
-
-bb554:
-  br label %bb555
-
-bb555:
-  br label %bb556
-
-bb556:
-  br label %bb557
-
-bb557:
-  br label %bb558
-
-bb558:
-  br label %bb559
-
-bb559:
-  br label %bb560
-
-bb560:
-  br label %bb561
-
-bb561:
-  br label %bb562
-
-bb562:
-  br label %bb563
-
-bb563:
-  br label %bb564
-
-bb564:
-  br label %bb565
-
-bb565:
-  br label %bb566
-
-bb566:
-  br label %bb567
-
-bb567:
-  br label %bb568
-
-bb568:
-  br label %bb569
-
-bb569:
-  br label %bb570
-
-bb570:
-  br label %bb571
-
-bb571:
-  br label %bb572
-
-bb572:
-  br label %bb573
-
-bb573:
-  br label %bb574
-
-bb574:
-  br label %bb575
-
-bb575:
-  br label %bb576
-
-bb576:
-  br label %bb577
-
-bb577:
-  br label %bb578
-
-bb578:
-  br label %bb579
-
-bb579:
-  br label %bb580
-
-bb580:
-  br label %bb581
-
-bb581:
-  br label %bb582
-
-bb582:
-  br label %bb583
-
-bb583:
-  br label %bb584
-
-bb584:
-  br label %bb585
-
-bb585:
-  br label %bb586
-
-bb586:
-  br label %bb587
-
-bb587:
-  br label %bb588
-
-bb588:
-  br label %bb589
-
-bb589:
-  br label %bb590
-
-bb590:
-  br label %bb591
-
-bb591:
-  br label %bb592
-
-bb592:
-  br label %bb593
-
-bb593:
-  br label %bb594
-
-bb594:
-  br label %bb595
-
-bb595:
-  br label %bb596
-
-bb596:
-  br label %bb597
-
-bb597:
-  br label %bb598
-
-bb598:
-  br label %bb599
-
-bb599:
-  br label %bb600
-
-bb600:
-  br label %bb601
-
-bb601:
-  br label %bb602
-
-bb602:
-  br label %bb603
-
-bb603:
-  br label %bb604
-
-bb604:
-  br label %bb605
-
-bb605:
-  br label %bb606
-
-bb606:
-  br label %bb607
-
-bb607:
-  br label %bb608
-
-bb608:
-  br label %bb609
-
-bb609:
-  br label %bb610
-
-bb610:
-  br label %bb611
-
-bb611:
-  br label %bb612
-
-bb612:
-  br label %bb613
-
-bb613:
-  br label %bb614
-
-bb614:
-  br label %bb615
-
-bb615:
-  br label %bb616
-
-bb616:
-  br label %bb617
-
-bb617:
-  br label %bb618
-
-bb618:
-  br label %bb619
-
-bb619:
-  br label %bb620
-
-bb620:
-  br label %bb621
-
-bb621:
-  br label %bb622
-
-bb622:
-  br label %bb623
-
-bb623:
-  br label %bb624
-
-bb624:
-  br label %bb625
-
-bb625:
-  br label %bb626
-
-bb626:
-  br label %bb627
-
-bb627:
-  br label %bb628
-
-bb628:
-  br label %bb629
-
-bb629:
-  br label %bb630
-
-bb630:
-  br label %bb631
-
-bb631:
-  br label %bb632
-
-bb632:
-  br label %bb633
-
-bb633:
-  br label %bb634
-
-bb634:
-  br label %bb635
-
-bb635:
-  br label %bb636
-
-bb636:
-  br label %bb637
-
-bb637:
-  br label %bb638
-
-bb638:
-  br label %bb639
-
-bb639:
-  br label %bb640
-
-bb640:
-  br label %bb641
-
-bb641:
-  br label %bb642
-
-bb642:
-  br label %bb643
-
-bb643:
-  br label %bb644
-
-bb644:
-  br label %bb645
-
-bb645:
-  br label %bb646
-
-bb646:
-  br label %bb647
-
-bb647:
-  br label %bb648
-
-bb648:
-  br label %bb649
-
-bb649:
-  br label %bb650
-
-bb650:
-  br label %bb651
-
-bb651:
-  br label %bb652
-
-bb652:
-  br label %bb653
-
-bb653:
-  br label %bb654
-
-bb654:
-  br label %bb655
-
-bb655:
-  br label %bb656
-
-bb656:
-  br label %bb657
-
-bb657:
-  br label %bb658
-
-bb658:
-  br label %bb659
-
-bb659:
-  br label %bb660
-
-bb660:
-  br label %bb661
-
-bb661:
-  br label %bb662
-
-bb662:
-  br label %bb663
-
-bb663:
-  br label %bb664
-
-bb664:
-  br label %bb665
-
-bb665:
-  br label %bb666
-
-bb666:
-  br label %bb667
-
-bb667:
-  br label %bb668
-
-bb668:
-  br label %bb669
-
-bb669:
-  br label %bb670
-
-bb670:
-  br label %bb671
-
-bb671:
-  br label %bb672
-
-bb672:
-  br label %bb673
-
-bb673:
-  br label %bb674
-
-bb674:
-  br label %bb675
-
-bb675:
-  br label %bb676
-
-bb676:
-  br label %bb677
-
-bb677:
-  br label %bb678
-
-bb678:
-  br label %bb679
-
-bb679:
-  br label %bb680
-
-bb680:
-  br label %bb681
-
-bb681:
-  br label %bb682
-
-bb682:
-  br label %bb683
-
-bb683:
-  br label %bb684
-
-bb684:
-  br label %bb685
-
-bb685:
-  br label %bb686
-
-bb686:
-  br label %bb687
-
-bb687:
-  br label %bb688
-
-bb688:
-  br label %bb689
-
-bb689:
-  br label %bb690
-
-bb690:
-  br label %bb691
-
-bb691:
-  br label %bb692
-
-bb692:
-  br label %bb693
-
-bb693:
-  br label %bb694
-
-bb694:
-  br label %bb695
-
-bb695:
-  br label %bb696
-
-bb696:
-  br label %bb697
-
-bb697:
-  br label %bb698
-
-bb698:
-  br label %bb699
-
-bb699:
-  br label %bb700
-
-bb700:
-  br label %bb701
-
-bb701:
-  br label %bb702
-
-bb702:
-  br label %bb703
-
-bb703:
-  br label %bb704
-
-bb704:
-  br label %bb705
-
-bb705:
-  br label %bb706
-
-bb706:
-  br label %bb707
-
-bb707:
-  br label %bb708
-
-bb708:
-  br label %bb709
-
-bb709:
-  br label %bb710
-
-bb710:
-  br label %bb711
-
-bb711:
-  br label %bb712
-
-bb712:
-  br label %bb713
-
-bb713:
-  br label %bb714
-
-bb714:
-  br label %bb715
-
-bb715:
-  br label %bb716
-
-bb716:
-  br label %bb717
-
-bb717:
-  br label %bb718
-
-bb718:
-  br label %bb719
-
-bb719:
-  br label %bb720
-
-bb720:
-  br label %bb721
-
-bb721:
-  br label %bb722
-
-bb722:
-  br label %bb723
-
-bb723:
-  br label %bb724
-
-bb724:
-  br label %bb725
-
-bb725:
-  br label %bb726
-
-bb726:
-  br label %bb727
-
-bb727:
-  br label %bb728
-
-bb728:
-  br label %bb729
-
-bb729:
-  br label %bb730
-
-bb730:
-  br label %bb731
-
-bb731:
-  br label %bb732
-
-bb732:
-  br label %bb733
-
-bb733:
-  br label %bb734
-
-bb734:
-  br label %bb735
-
-bb735:
-  br label %bb736
-
-bb736:
-  br label %bb737
-
-bb737:
-  br label %bb738
-
-bb738:
-  br label %bb739
-
-bb739:
-  br label %bb740
-
-bb740:
-  br label %bb741
-
-bb741:
-  br label %bb742
-
-bb742:
-  br label %bb743
-
-bb743:
-  br label %bb744
-
-bb744:
-  br label %bb745
-
-bb745:
-  br label %bb746
-
-bb746:
-  br label %bb747
-
-bb747:
-  br label %bb748
-
-bb748:
-  br label %bb749
-
-bb749:
-  br label %bb750
-
-bb750:
-  br label %bb751
-
-bb751:
-  br label %bb752
-
-bb752:
-  br label %bb753
-
-bb753:
-  br label %bb754
-
-bb754:
-  br label %bb755
-
-bb755:
-  br label %bb756
-
-bb756:
-  br label %bb757
-
-bb757:
-  br label %bb758
-
-bb758:
-  br label %bb759
-
-bb759:
-  br label %bb760
-
-bb760:
-  br label %bb761
-
-bb761:
-  br label %bb762
-
-bb762:
-  br label %bb763
-
-bb763:
-  br label %bb764
-
-bb764:
-  br label %bb765
-
-bb765:
-  br label %bb766
-
-bb766:
-  br label %bb767
-
-bb767:
-  br label %bb768
-
-bb768:
-  br label %bb769
-
-bb769:
-  br label %bb770
-
-bb770:
-  br label %bb771
-
-bb771:
-  br label %bb772
-
-bb772:
-  br label %bb773
-
-bb773:
-  br label %bb774
-
-bb774:
-  br label %bb775
-
-bb775:
-  br label %bb776
-
-bb776:
-  br label %bb777
-
-bb777:
-  br label %bb778
-
-bb778:
-  br label %bb779
-
-bb779:
-  br label %bb780
-
-bb780:
-  br label %bb781
-
-bb781:
-  br label %bb782
-
-bb782:
-  br label %bb783
-
-bb783:
-  br label %bb784
-
-bb784:
-  br label %bb785
-
-bb785:
-  br label %bb786
-
-bb786:
-  br label %bb787
-
-bb787:
-  br label %bb788
-
-bb788:
-  br label %bb789
-
-bb789:
-  br label %bb790
-
-bb790:
-  br label %bb791
-
-bb791:
-  br label %bb792
-
-bb792:
-  br label %bb793
-
-bb793:
-  br label %bb794
-
-bb794:
-  br label %bb795
-
-bb795:
-  br label %bb796
-
-bb796:
-  br label %bb797
-
-bb797:
-  br label %bb798
-
-bb798:
-  br label %bb799
-
-bb799:
-  br label %bb800
-
-bb800:
-  br label %bb801
-
-bb801:
-  br label %bb802
-
-bb802:
-  br label %bb803
-
-bb803:
-  br label %bb804
-
-bb804:
-  br label %bb805
-
-bb805:
-  br label %bb806
-
-bb806:
-  br label %bb807
-
-bb807:
-  br label %bb808
-
-bb808:
-  br label %bb809
-
-bb809:
-  br label %bb810
-
-bb810:
-  br label %bb811
-
-bb811:
-  br label %bb812
-
-bb812:
-  br label %bb813
-
-bb813:
-  br label %bb814
-
-bb814:
-  br label %bb815
-
-bb815:
-  br label %bb816
-
-bb816:
-  br label %bb817
-
-bb817:
-  br label %bb818
-
-bb818:
-  br label %bb819
-
-bb819:
-  br label %bb820
-
-bb820:
-  br label %bb821
-
-bb821:
-  br label %bb822
-
-bb822:
-  br label %bb823
-
-bb823:
-  br label %bb824
-
-bb824:
-  br label %bb825
-
-bb825:
-  br label %bb826
-
-bb826:
-  br label %bb827
-
-bb827:
-  br label %bb828
-
-bb828:
-  br label %bb829
-
-bb829:
-  br label %bb830
-
-bb830:
-  br label %bb831
-
-bb831:
-  br label %bb832
-
-bb832:
-  br label %bb833
-
-bb833:
-  br label %bb834
-
-bb834:
-  br label %bb835
-
-bb835:
-  br label %bb836
-
-bb836:
-  br label %bb837
-
-bb837:
-  br label %bb838
-
-bb838:
-  br label %bb839
-
-bb839:
-  br label %bb840
-
-bb840:
-  br label %bb841
-
-bb841:
-  br label %bb842
-
-bb842:
-  br label %bb843
-
-bb843:
-  br label %bb844
-
-bb844:
-  br label %bb845
-
-bb845:
-  br label %bb846
-
-bb846:
-  br label %bb847
-
-bb847:
-  br label %bb848
-
-bb848:
-  br label %bb849
-
-bb849:
-  br label %bb850
-
-bb850:
-  br label %bb851
-
-bb851:
-  br label %bb852
-
-bb852:
-  br label %bb853
-
-bb853:
-  br label %bb854
-
-bb854:
-  br label %bb855
-
-bb855:
-  br label %bb856
-
-bb856:
-  br label %bb857
-
-bb857:
-  br label %bb858
-
-bb858:
-  br label %bb859
-
-bb859:
-  br label %bb860
-
-bb860:
-  br label %bb861
-
-bb861:
-  br label %bb862
-
-bb862:
-  br label %bb863
-
-bb863:
-  br label %bb864
-
-bb864:
-  br label %bb865
-
-bb865:
-  br label %bb866
-
-bb866:
-  br label %bb867
-
-bb867:
-  br label %bb868
-
-bb868:
-  br label %bb869
-
-bb869:
-  br label %bb870
-
-bb870:
-  br label %bb871
-
-bb871:
-  br label %bb872
-
-bb872:
-  br label %bb873
-
-bb873:
-  br label %bb874
-
-bb874:
-  br label %bb875
-
-bb875:
-  br label %bb876
-
-bb876:
-  br label %bb877
-
-bb877:
-  br label %bb878
-
-bb878:
-  br label %bb879
-
-bb879:
-  br label %bb880
-
-bb880:
-  br label %bb881
-
-bb881:
-  br label %bb882
-
-bb882:
-  br label %bb883
-
-bb883:
-  br label %bb884
-
-bb884:
-  br label %bb885
-
-bb885:
-  br label %bb886
-
-bb886:
-  br label %bb887
-
-bb887:
-  br label %bb888
-
-bb888:
-  br label %bb889
-
-bb889:
-  br label %bb890
-
-bb890:
-  br label %bb891
-
-bb891:
-  br label %bb892
-
-bb892:
-  br label %bb893
-
-bb893:
-  br label %bb894
-
-bb894:
-  br label %bb895
-
-bb895:
-  br label %bb896
-
-bb896:
-  br label %bb897
-
-bb897:
-  br label %bb898
-
-bb898:
-  br label %bb899
-
-bb899:
-  br label %bb900
-
-bb900:
-  br label %bb901
-
-bb901:
-  br label %bb902
-
-bb902:
-  br label %bb903
-
-bb903:
-  br label %bb904
-
-bb904:
-  br label %bb905
-
-bb905:
-  br label %bb906
-
-bb906:
-  br label %bb907
-
-bb907:
-  br label %bb908
-
-bb908:
-  br label %bb909
-
-bb909:
-  br label %bb910
-
-bb910:
-  br label %bb911
-
-bb911:
-  br label %bb912
-
-bb912:
-  br label %bb913
-
-bb913:
-  br label %bb914
-
-bb914:
-  br label %bb915
-
-bb915:
-  br label %bb916
-
-bb916:
-  br label %bb917
-
-bb917:
-  br label %bb918
-
-bb918:
-  br label %bb919
-
-bb919:
-  br label %bb920
-
-bb920:
-  br label %bb921
-
-bb921:
-  br label %bb922
-
-bb922:
-  br label %bb923
-
-bb923:
-  br label %bb924
-
-bb924:
-  br label %bb925
-
-bb925:
-  br label %bb926
-
-bb926:
-  br label %bb927
-
-bb927:
-  br label %bb928
-
-bb928:
-  br label %bb929
-
-bb929:
-  br label %bb930
-
-bb930:
-  br label %bb931
-
-bb931:
-  br label %bb932
-
-bb932:
-  br label %bb933
-
-bb933:
-  br label %bb934
-
-bb934:
-  br label %bb935
-
-bb935:
-  br label %bb936
-
-bb936:
-  br label %bb937
-
-bb937:
-  br label %bb938
-
-bb938:
-  br label %bb939
-
-bb939:
-  br label %bb940
-
-bb940:
-  br label %bb941
-
-bb941:
-  br label %bb942
-
-bb942:
-  br label %bb943
-
-bb943:
-  br label %bb944
-
-bb944:
-  br label %bb945
-
-bb945:
-  br label %bb946
-
-bb946:
-  br label %bb947
-
-bb947:
-  br label %bb948
-
-bb948:
-  br label %bb949
-
-bb949:
-  br label %bb950
-
-bb950:
-  br label %bb951
-
-bb951:
-  br label %bb952
-
-bb952:
-  br label %bb953
-
-bb953:
-  br label %bb954
-
-bb954:
-  br label %bb955
-
-bb955:
-  br label %bb956
-
-bb956:
-  br label %bb957
-
-bb957:
-  br label %bb958
-
-bb958:
-  br label %bb959
-
-bb959:
-  br label %bb960
-
-bb960:
-  br label %bb961
-
-bb961:
-  br label %bb962
-
-bb962:
-  br label %bb963
-
-bb963:
-  br label %bb964
-
-bb964:
-  br label %bb965
-
-bb965:
-  br label %bb966
-
-bb966:
-  br label %bb967
-
-bb967:
-  br label %bb968
-
-bb968:
-  br label %bb969
-
-bb969:
-  br label %bb970
-
-bb970:
-  br label %bb971
-
-bb971:
-  br label %bb972
-
-bb972:
-  br label %bb973
-
-bb973:
-  br label %bb974
-
-bb974:
-  br label %bb975
-
-bb975:
-  br label %bb976
-
-bb976:
-  br label %bb977
-
-bb977:
-  br label %bb978
-
-bb978:
-  br label %bb979
-
-bb979:
-  br label %bb980
-
-bb980:
-  br label %bb981
-
-bb981:
-  br label %bb982
-
-bb982:
-  br label %bb983
-
-bb983:
-  br label %bb984
-
-bb984:
-  br label %bb985
-
-bb985:
-  br label %bb986
-
-bb986:
-  br label %bb987
-
-bb987:
-  br label %bb988
-
-bb988:
-  br label %bb989
-
-bb989:
-  br label %bb990
-
-bb990:
-  br label %bb991
-
-bb991:
-  br label %bb992
-
-bb992:
-  br label %bb993
-
-bb993:
-  br label %bb994
-
-bb994:
-  br label %bb995
-
-bb995:
-  br label %bb996
-
-bb996:
-  br label %bb997
-
-bb997:
-  br label %bb998
-
-bb998:
-  br label %bb999
-
-bb999:
-  br label %bb1000
-
-bb1000:
-  ; CHECK: call{{.*}}@dfsan_union
-  ; CHECK-NOT: phi
-  %ab = mul i32 %a, %b
-  ret i32 %ab
-}

llvm/test/Instrumentation/DataFlowSanitizer/union.ll

 ; RUN: opt < %s -dfsan -S | FileCheck %s
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
+; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
+
 @a = common global i32 0
 @b = common global i32 0
 
 ; Check that we reuse unions where possible.
 
 ; CHECK-LABEL: @"dfs$f"
 define void @f(i32 %x, i32 %y) {
-  ; CHECK: call{{.*}}__dfsan_union
+  ; CHECK: or i[[#SBITS]]
   %xay = add i32 %x, %y
   store i32 %xay, i32* @a
-  ; CHECK-NOT: call{{.*}}__dfsan_union
+  ; CHECK-NOT: or i[[#SBITS]]
   %xmy = mul i32 %x, %y
   store i32 %xmy, i32* @b
   ret void
 }
 
 ; In this case, we compute the unions on both sides because neither block
 ; dominates the other.
 
 ; CHECK-LABEL: @"dfs$g"
 define void @g(i1 %p, i32 %x, i32 %y) {
   br i1 %p, label %l1, label %l2
 
 l1:
-  ; CHECK: call{{.*}}__dfsan_union
+  ; CHECK: or i[[#SBITS]]
   %xay = add i32 %x, %y
   store i32 %xay, i32* @a
   br label %l3
 
 l2:
-  ; CHECK: call{{.*}}__dfsan_union
+  ; CHECK: or i[[#SBITS]]
   %xmy = mul i32 %x, %y
   store i32 %xmy, i32* @b
   br label %l3
 
 l3:
   ret void
 }
 
 ; In this case, we know that the label for %xayax subsumes the label for %xay.
 
 ; CHECK-LABEL: @"dfs$h"
 define i32 @h(i32 %x, i32 %y) {
-  ; CHECK: call{{.*}}__dfsan_union
+  ; CHECK: or i[[#SBITS]]
   %xay = add i32 %x, %y
-  ; CHECK-NOT: call{{.*}}__dfsan_union
+  ; CHECK-NOT: or i[[#SBITS]]
   %xayax = add i32 %xay, %x
   ret i32 %xayax
 }

llvm/test/Instrumentation/DataFlowSanitizer/vector.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,TLS_ABI,TLS_ABI_LEGACY
 ; RUN: opt < %s -dfsan -dfsan-args-abi -S | FileCheck %s --check-prefixes=CHECK,ARGS_ABI
-; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefixes=CHECK,TLS_ABI,TLS_ABI_FAST
-; RUN: opt < %s -dfsan -dfsan-fast-8-labels=true -S | FileCheck %s --check-prefixes=CHECK,TLS_ABI,TLS_ABI_FAST
+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefixes=CHECK,TLS_ABI
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK: @__dfsan_shadow_width_bits = weak_odr constant i32 [[#SBITS:]]
 ; CHECK: @__dfsan_shadow_width_bytes = weak_odr constant i32 [[#SBYTES:]]
 
 define <4 x i4> @pass_vector(<4 x i4> %v) {
   ; ARGS_ABI-LABEL: @"dfs$pass_vector"
define void @load_update_store_vector(<4 x i4>* %p) {
   store <4 x i4> %v1, <4 x i4>* %p
   ret void
 }
 
 define <4 x i1> @icmp_vector(<4 x i8> %a, <4 x i8> %b) {
   ; TLS_ABI-LABEL: @"dfs$icmp_vector"
   ; TLS_ABI-NEXT: %[[B:.*]] = load i[[#SBITS]], i[[#SBITS]]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i[[#SBITS]]*), align [[ALIGN:2]]
   ; TLS_ABI-NEXT: %[[A:.*]] = load i[[#SBITS]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_arg_tls to i[[#SBITS]]*), align [[ALIGN]]
-
-  ; TLS_ABI_LEGACY: %[[U:.*]] = call zeroext i[[#SBITS]] @__dfsan_union(i[[#SBITS]] zeroext %[[A]], i[[#SBITS]] zeroext %[[B]])
-  ; TLS_ABI_LEGACY: %[[L:.*]] = phi i[[#SBITS]] [ %[[U]], {{.*}} ], [ %[[A]], {{.*}} ]
-
-  ; COM: With fast labels enabled, union is just an OR.
-  ; TLS_ABI_FAST: %[[L:.*]] = or i[[#SBITS]] %[[A]], %[[B]]
+  ; TLS_ABI:       %[[L:.*]] = or i[[#SBITS]] %[[A]], %[[B]]
 
   ; TLS_ABI: %r = icmp eq <4 x i8> %a, %b
   ; TLS_ABI: store i[[#SBITS]] %[[L]], i[[#SBITS]]* bitcast ([100 x i64]* @__dfsan_retval_tls to i[[#SBITS]]*), align [[ALIGN]]
   ; TLS_ABI: ret <4 x i1> %r
 
   %r = icmp eq <4 x i8> %a, %b
   ret <4 x i1> %r
 }