[dfsan] Track field/index-level shadow values in variables

[stephan.yichao.zhao]

Summary

---

• The problem *****

See motivation examples in compiler-rt/test/dfsan/pair.cpp. The current
DFSan always uses a 16bit shadow value for a variable with any type by
combining all shadow values of all bytes of the variable. So it cannot
distinguish two fields of a struct: each field's shadow value equals the
combined shadow value of all fields. This introduces an overtaint issue.

Consider a parsing function

std::pair<char*, int> get_token(char* p);

where p points to a buffer to parse, the returned pair includes the next
token and the pointer to the position in the buffer after the token.

If the token is tainted, then both the returned pointer and int ar
tainted. If the parser keeps on using get_token for the rest parsing,
all the following outputs are tainted because of the tainted pointer.

The CL is the first change to address the issue.

---

• The proposed improvement ******

Eventually all fields and indices have their own shadow values in
variables and memory.

For example, variables with type {i1, i3}, [2 x i1], {[2 x i4], i8},
[2 x {i1, i1}] have shadow values with type {i16, i16}, [2 x i16],
{[2 x i16], i16}, [2 x {i16, i16}] correspondingly; variables with
primary type still have shadow values i16.

---

• An potential implementation plan *******

The idea is to adopt the change incrementially.

1. This CL
   
   Support field-level accuracy at variables/args/ret in TLS/Fast16 mode, load/store/alloca still use combined shadow values.
   
   After the alloca promotion and SSA construction phases (>=-O1), we assume alloca and memory operations are reduced. So if struct variables do not relate to memory, their tracking is accurate at field level.

2. Support field-level accuracy at alloca
3. Support field-level accuracy at load/store
   
   These two should make O0 and real memory access work.

4. Support vector if necessary.
5. Support Args mode if necessary.
6. Support passing more accurate shadow values via custom functions if necessary.
7. Support legacy non-fast16 mode if necessary.
   
   ***
   • About this CL. ***

The CL did the following

1. extended TLS arg/ret to work with aggregate types. This is similar to what MSan does.
2. implemented how to map between an original type/value/zero-const to its shadow type/value/zero-const.
3. extended (insert|extract)value to use field/index-level progagation.
4. for other instructions, propagation rules are combining inputs by or. The CL converts between aggragate and primary shadow values at the cases.
5. Custom function interfaces also need such a conversion because all existing custom functions use i16. It is unclear whether custom functions need more accurate shadow propagation yet.
6. Added test cases for aggregate type related cases.

[morehouse]

Thanks for your work on this, Jianzhou. Overall I think this is a reasonable approach to improving label precision.

This is a fairly large patch; would it be possible to split it into a series of dependent patches? For example, the renaming/refactoring stuff could go in its own patch and probably the changes to select.ll could as well. If there's a way to also split the functional changes, that would be even better. This would make it easier for me to review and give better feedback.

[stephan.yichao.zhao]

This is a fairly large patch; would it be possible to split it into a series of dependent patches? For example, the renaming/refactoring stuff could go in its own patch and probably the changes to select.ll could as well. If there's a way to also split the functional changes, that would be even better. This would make it easier for me to review and give better feedback.

Is the following split better?

1. the extension of dfsan_arg_tls and dfsan_ret_tls. This also updates test cases that check them. For example, select.ll
2. rename ShadowTy/ZeroShadow to PrimaryShadowTy/ZeroShadowTy
3. modification of test cases phi.ll and load.ll since they can test the existing work too.
4. add dfsan e2e test cases about struct to lock down the existing behavior.
5. the rest functional change that supports field-level accuracy. This part is hard to split because we may have to make all instruction propagation use the same shadow value to make it work.

[morehouse]

SGTM

[stephan.yichao.zhao]

updated after rebasing those children diffs

[stephan.yichao.zhao]

update

[stephan.yichao.zhao]

updated

[morehouse]

How does this change affect instrumented binary size?

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

422

426

441

445

479

Nit: This is only used for mapping expanded shadow to collapsed shadow. Maybe CachedCollapsedShadows is more descriptive, while being consistent with the naming of CachedShadows?

513–514

521–522

527

Nit: I think expandFromPrimitiveShadow and collapseToPrimitiveShadow are more intuitive names.

538–540

543–545

648–673

663–665

679

Nit: This is really just a recursive helper function. Maybe a name like expandFromPrimitiveShadowRecursive is more descriptive? Also since it doesn't access any member variables, it could just be a local static function.

708

This function is small enough, I think we should just inline it into convertFromPrimitiveShadow below.

745–746

746

OR only works for fast16 mode.

765

collapseStructShadow and collapseArrayShadow are practically identical. Can we use a templates or polymorphism to share their implementation?

768

Since this function is small, I think we should inline it into convertToPrimitiveShadow below.

788

This additional check isn't used for the CCS cache. Why do we need it here?

959

1409

Nit: This function is tiny. Maybe we should just inline it everywhere instead?

1415

Do you plan to make this work without converting everything to primitive shadow?

1521

1764

What's the reason for expanding the shadow when we're about to collapse it again in storeShadow?

[stephan.yichao.zhao]

update

[stephan.yichao.zhao]

In D92261#2437862, @morehouse wrote:

How does this change affect instrumented binary size?

For the same large application used by D92440, this diff added 0.004% code size overhead.
Because the change of D92440 reduces 0.1%, so after the diff, the code size is still smaller.
I think this is because returning or passing struct/array is rare in C++ code, although if this happens at critical code path, it introduces overtaint as the motivation example used in the description.

We also looked into a small code to see how this changes code gen. Interestingly, somehow at TLS part, the diff's code is slightly smaller.
For example, for a C code

typedef struct Pair {
  int i;
  char *ptr;
} Pair;
Pair make_pair(int i, char *ptr) {
  Pair pair;
  pair.i = i;
  pair.ptr = ptr;
  return pair;
}

At ret. the old dfsan does

mov    %rsi,%rdx
mov    %edi,%eax
mov    0x0(%rip),%rcx 
movzwl %fs:(%rcx),%esi
or     %fs:0x2(%rcx),%si
mov    0x0(%rip),%rcx 
mov    %si,%fs:(%rcx)
retq

the new dfsan does

mov    %rsi,%rdx
mov    %edi,%eax
mov    0x0(%rip),%rcx 
mov    %fs:(%rcx),%ecx
mov    0x0(%rip),%rsi  
mov    %ecx,%fs:(%rsi)
retq

We can see although Pair has two fields, it is represented by one register.
Because the old dfsan needs to union the two fields before returning, this seems a more complicated register-level code because of unioning a register itself.., while the new dfsan does not do such a union.

I feel there could be cases where the new dfsan may generate more code.
But because in a large application, passing/returning complicated struct is not common, it seems fine.

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

527

renamed to collapseToPrimitiveShadow

679

moved it to a static function expandFromPrimitiveShadowRecursive.

746

Thank you for catching this! My original test case worked for non-fast16 accidentially because it used only labels 1 and 2..., and our large applications use only fast16 mode..

The or operation of the legacy mode is more complicated; plus, introducing more 'or' can consume legacy labels more quickly. So it is not clear if it is a good trade-off between accuracy and running-out-of-labels...

I added a shouldTrackFieldsAndIndices method to ensure this diff only enables field-level shadow for fast16 mode. none-fast16 mode needs more evaluation in the next step.

768

This convertToPrimitiveShadow and the above collapseArrayShadow make a mutual recursion.

The convertToPrimitiveShadow below is like the main entry point.

788

The use of the CCS cache assumes that in the same block instructions are visited sequentially. So it does not need to check domination inside the same block.

CachedShadow2PrimitiveShadow has a case here.
Somehow it may insert a new instruction in a reversed order because of phi node insertion and because this ClDebugNonzeroLabels feature is a post-process.
So we added this to ensure in the same block CS.Shadow dominates Pos.

1409

It was inlined before. I found it is used many times, so made it a function.

1415

Yes.

The conversion or collapsing happens in mainly the following cases

1. memory operations

The next step is making load/store/alloca use non-collapsed values. This is required for O0-compiled code. O0 does not promote alloca to variables. So most struct operations go through memory...

2. ClCombinePointerLabelsOnStore/ClCombinePointerLabelsOnLoad/ClTrackSelectControlFlow

When ClCombinePointerLabelsOnStore/ClCombinePointerLabelsOnLoad/ClTrackSelectControlFlow are true, we can also do field-level combining rather than combining-then-union.

3. custom function wrapper

It is not clear if any wrapper needs struct parameters or ret values yet

4. when combineShadows is used by combineOperandShadows

I feel in this case most operands are not aggregate types except select, insert/extract, which are already considered separately. If any op can use aggregate types, we could treat them specially.

In the above 4 cases, probably we wanted to see how non-collapsed propagation changes code size to get a better trade-off.

1764

Good catch! I renamed storeShadow to be storePrimitiveShadow to indicate it reads primitive values, and we renamed unnecessary conversions.

W/o the change, it indeed generates dead-code, although the following pass may remove them.

[morehouse]

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

215

Nit: Moving this helper function closer to where it's used would improve readability.

520

WDYT about renaming to expandFromPrimitiveShadow?

788

Please add a comment explaining this.

If CS.Shadow dominates Pos, do we still need to check that CS.Block dominates Pos->getParent()?

1702

Please delete commented-out code here and elsewhere.

llvm/test/Instrumentation/DataFlowSanitizer/abilist_aggregate.ll

182

I'm not familiar with the custom wrapper logic. Why does this function store 0 to arg TLS?

192

This function should store {a1, b0} shadow to retval TLS, right? Should we verify that?

llvm/test/Instrumentation/DataFlowSanitizer/array.ll

214

Why are there two stores to SP? 2 x i1 is less than 1 byte, so wouldn't a single i16 shadow be enough?

Or is there a hidden 1 byte alignment in the array?

246

Shouldn't there be more ORs for each element in %a?

261

What does this last store to "P3" do?

llvm/test/Instrumentation/DataFlowSanitizer/struct.ll

21

What's the reason for having DEBUG_NONZERO_LABELS here when it tests nothing interesting?

261

274

llvm/test/Instrumentation/DataFlowSanitizer/union-large.ll

3013 ↗

(On Diff #310104)

What's the reason for changing this test?

[stephan.yichao.zhao]

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

788

Added the comments at CachedCollapsedShadows. Actually we only need to catch values since we do not need to check Block dominations if we have to check dominations between values.

llvm/test/Instrumentation/DataFlowSanitizer/abilist_aggregate.ll

182

This seems the effect of the existing design.

dfs$call_custom_cb calls the customized @__dfsw_custom_cb.
The user-provided @__dfsw_custom_cb calls @call_custom_cb.

And the instrumentation around "call %cb" inside @call_custom_cb is by this visitCallInst.

This visitor is from

DFSanFunction DFSF(*this, F, /*IsNativeABI=*/true);

When IsNativeABI is one, getShadow always returns 0 for arguments, and
@custom_cb does not return shadow.

@__dfsw_custom_cb receives dfst0$custom_cb and @"dfs$cb" instead of @cb.
@"dfs$cb" is a dfsan @cb that uses TLS to pass shadows at args/ret, while dfst0$custom_cb is a wrapper of @"dfs$cb", and dfst0$custom_cb passes shadows by additional arguments.

I think in @__dfsw_custom_cb, users' code is like

def @__dfsw_custom_cb(@"dfst0$custom_cb, @"dfs$cb", arg1, arg2, ..., arg_shadow1, arg_shadow2, ..., ret_shadow) {
    auto cb_ret_shadow;
    auto my_cb = [&] (...) {
      ...
      ret @"dfst0$custom_cb"(@"dfs$cb", cb_arg1, cb_arg2, ..., cb_arg_shadow1, cb_arg_shadow2, ..., cb_ret_shadow);
    }
    auto r = @custom_cb(my_cb, ...)
    // set ret_shadow in terms of all shadows...
}

The puzzle is that all arguments shadow are assigned to this DFSF, although they should never be used. It is from the first version.

I am not sure this is supposed to be the correct custom wrapper of a function with callbacks.

llvm/test/Instrumentation/DataFlowSanitizer/array.ll

214

This is defined by the above data layout. It defines i1:8:8. So each i1 takes 1 byte, and we have 2-byte shadow for each 1 byte.

246

This is where the current diff loses accuracy.
When saving an aggregate value into memory, we call that collapse function to convert an accurate shadow to a i16 label.
So this diff only increases accuracy for variables, arguments and ret.

This works for O1-compiled targets, because alloca premotion removes lots of memory operations, and practice code does not save aggregate types to memory.
If we build by O0, it does not work as those pair.cc and struct.c test.

We need to address this in the next change.

261

This is testing the loop from here to here.

when a data to store is large, it first saves vectors, then saves the rest as primitive types.
The instructions before P3 are about vector saving, the rest are for primitive-data saving.

This is another reason this diff collapses aggregate shadow to i16. It makes the code still reuse the same code for storing/loading.

The next change that preserves aggregate accuracy needs to redesign this logic.

llvm/test/Instrumentation/DataFlowSanitizer/struct.ll

21

Thank you for catching this. Added.

llvm/test/Instrumentation/DataFlowSanitizer/union-large.ll

3013 ↗

(On Diff #310104)

this is not necessary. removed.

[stephan.yichao.zhao]

updated

[morehouse]

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

1772

Please remove this commented-out code.

llvm/test/Instrumentation/DataFlowSanitizer/abilist_aggregate.ll

182

I think I understand... Normally custom_cb would not be instrumented by DFSan (hence why we added it to the ABI list). So probably whatever happens here is unimportant. Maybe we should remove all these checks except the first line:

; TLS_ABI: define { i1, i7 } @custom_cb({ i1, i7 } ({ i32, i1 }, [2 x i7])* %cb, { i32, i1 } %a, [2 x i7] %b)

llvm/test/Instrumentation/DataFlowSanitizer/array.ll

246

So there *should* be more ORs in the current diff, right? But the plan is to fix this, so that's why they aren't listed here?

[stephan.yichao.zhao]

addressed comments

llvm/test/Instrumentation/DataFlowSanitizer/array.ll

246

added.

[morehouse]

LGTM. Thanks for the nice test coverage too!

compiler-rt/test/dfsan/pair.cpp

-// RUN: %clangxx_dfsan %s -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-select-control-flow=false -mllvm -dfsan-combine-pointer-labels-on-load=false -o %t && %run %t
+// RUN: %clangxx_dfsan %s -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-select-control-flow=false -mllvm -dfsan-combine-pointer-labels-on-load=false -O0 -DO0 -o %t && %run %t
+// RUN: %clangxx_dfsan %s -mllvm -dfsan-fast-16-labels -mllvm -dfsan-track-select-control-flow=false -mllvm -dfsan-combine-pointer-labels-on-load=false -O1 -o %t && %run %t
 
 #include <algorithm>
 #include <assert.h>
 #include <sanitizer/dfsan_interface.h>
 #include <utility>
 
 __attribute__((noinline))
 std::pair<int *, int>
void test_simple_constructors() {
   int *ptr = NULL;
   dfsan_set_label(8, &i, sizeof(i));
   dfsan_set_label(2, &ptr, sizeof(ptr));
 
   std::pair<int *, int> pair1 = make_pair(ptr, i);
   int i1 = pair1.second;
   int *ptr1 = pair1.first;
 
+#ifdef O0
   assert(dfsan_read_label(&i1, sizeof(i1)) == 10);
   assert(dfsan_read_label(&ptr1, sizeof(ptr1)) == 10);
+#else
+  assert(dfsan_read_label(&i1, sizeof(i1)) == 8);
+  assert(dfsan_read_label(&ptr1, sizeof(ptr1)) == 2);
+#endif
 
   std::pair<int *, int> pair2 = copy_pair1(pair1);
   int i2 = pair2.second;
   int *ptr2 = pair2.first;
 
+#ifdef O0
   assert(dfsan_read_label(&i2, sizeof(i2)) == 10);
   assert(dfsan_read_label(&ptr2, sizeof(ptr2)) == 10);
+#else
+  assert(dfsan_read_label(&i2, sizeof(i2)) == 8);
+  assert(dfsan_read_label(&ptr2, sizeof(ptr2)) == 2);
+#endif
 
   std::pair<int *, int> pair3 = copy_pair2(&pair1);
   int i3 = pair3.second;
   int *ptr3 = pair3.first;
 
+#ifdef O0
   assert(dfsan_read_label(&i3, sizeof(i3)) == 10);
   assert(dfsan_read_label(&ptr3, sizeof(ptr3)) == 10);
+#else
+  assert(dfsan_read_label(&i3, sizeof(i3)) == 8);
+  assert(dfsan_read_label(&ptr3, sizeof(ptr3)) == 2);
+#endif
 
   std::pair<int *, int> pair4 = copy_pair3(std::move(pair1));
   int i4 = pair4.second;
   int *ptr4 = pair4.first;
 
+#ifdef O0
   assert(dfsan_read_label(&i4, sizeof(i4)) == 10);
   assert(dfsan_read_label(&ptr4, sizeof(ptr4)) == 10);
+#else
+  assert(dfsan_read_label(&i4, sizeof(i4)) == 8);
+  assert(dfsan_read_label(&ptr4, sizeof(ptr4)) == 2);
+#endif
 }
 
 void test_branches() {
   uint32_t res = 4;
   dfsan_set_label(8, &res, sizeof(res));
 
   char p[100];
   const char *q = p;
dfsan_set_label(2, &q, sizeof(q));
     }
   }
 
   {
     std::fill_n(p, 100, 0);
 
     {
       std::pair<const char *, uint32_t> r = return_ptr_and_i32(q, res);
+#ifdef O0
       assert(dfsan_read_label(&r.first, sizeof(r.first)) == 10);
       assert(dfsan_read_label(&r.second, sizeof(r.second)) == 10);
+#else
+      assert(dfsan_read_label(&r.first, sizeof(r.first)) == 2);
+      assert(dfsan_read_label(&r.second, sizeof(r.second)) == 8);
+#endif
     }
 
     {
       std::pair<const char *, uint64_t> r = return_ptr_and_i64(q, res);
+#ifdef O0
       assert(dfsan_read_label(&r.first, sizeof(r.first)) == 10);
       assert(dfsan_read_label(&r.second, sizeof(r.second)) == 10);
+#else
+      assert(dfsan_read_label(&r.first, sizeof(r.first)) == 2);
+      assert(dfsan_read_label(&r.second, sizeof(r.second)) == 8);
+#endif
     }
   }
 }
 
 int main(void) {
   test_simple_constructors();
   test_branches();
 
   return 0;
 }

compiler-rt/test/dfsan/struct.c

-// RUN: %clang_dfsan %s -o %t && %run %t
+// RUN: %clang_dfsan %s -O1 -mllvm -dfsan-fast-16-labels=true -DFAST16_O1 -o %t && %run %t
+// RUN: %clang_dfsan %s -O1 -DO1 -o %t && %run %t
+// RUN: %clang_dfsan %s -O0 -mllvm -dfsan-fast-16-labels=true -DFAST16_O0 -o %t && %run %t
+// RUN: %clang_dfsan %s -O0 -DO0 -o %t && %run %t
 
 #include <assert.h>
 #include <sanitizer/dfsan_interface.h>
 
 typedef struct Pair {
   int i;
   char *ptr;
 } Pair;
Pair copy_pair2(const Pair pair0) {
   pair.i = pair0.i;
   pair.ptr = pair0.ptr;
   return pair;
 }
 
 int main(void) {
   int i = 1;
   char *ptr = NULL;
+#if defined(FAST16_O1) || defined(FAST16_O0)
+  dfsan_label i_label = 1;
+  dfsan_label ptr_label = 2;
+#else
   dfsan_label i_label = dfsan_create_label("i", 0);
-  dfsan_set_label(i_label, &i, sizeof(i));
   dfsan_label ptr_label = dfsan_create_label("ptr", 0);
+#endif
+  dfsan_set_label(i_label, &i, sizeof(i));
   dfsan_set_label(ptr_label, &ptr, sizeof(ptr));
 
   Pair pair1 = make_pair(i, ptr);
   int i1 = pair1.i;
   char *ptr1 = pair1.ptr;
 
   dfsan_label i1_label = dfsan_read_label(&i1, sizeof(i1));
   dfsan_label ptr1_label = dfsan_read_label(&ptr1, sizeof(ptr1));
+#if defined(O0) || defined(O1)
   assert(dfsan_has_label(i1_label, i_label));
   assert(dfsan_has_label(i1_label, ptr_label));
   assert(dfsan_has_label(ptr1_label, i_label));
   assert(dfsan_has_label(ptr1_label, ptr_label));
+#elif defined(FAST16_O0)
+  assert(i1_label == (i_label | ptr_label));
+  assert(ptr1_label == (i_label | ptr_label));
+#else
+  assert(i1_label == i_label);
+  assert(ptr1_label == ptr_label);
+#endif
 
   Pair pair2 = copy_pair1(&pair1);
   int i2 = pair2.i;
   char *ptr2 = pair2.ptr;
 
   dfsan_label i2_label = dfsan_read_label(&i2, sizeof(i2));
   dfsan_label ptr2_label = dfsan_read_label(&ptr2, sizeof(ptr2));
+#if defined(O0) || defined(O1)
   assert(dfsan_has_label(i2_label, i_label));
   assert(dfsan_has_label(i2_label, ptr_label));
   assert(dfsan_has_label(ptr2_label, i_label));
   assert(dfsan_has_label(ptr2_label, ptr_label));
+#elif defined(FAST16_O0)
+  assert(i2_label == (i_label | ptr_label));
+  assert(ptr2_label == (i_label | ptr_label));
+#else
+  assert(i2_label == i_label);
+  assert(ptr2_label == ptr_label);
+#endif
 
   Pair pair3 = copy_pair2(pair1);
   int i3 = pair3.i;
   char *ptr3 = pair3.ptr;
 
   dfsan_label i3_label = dfsan_read_label(&i3, sizeof(i3));
   dfsan_label ptr3_label = dfsan_read_label(&ptr3, sizeof(ptr3));
+#if defined(O0) || defined(O1)
   assert(dfsan_has_label(i3_label, i_label));
   assert(dfsan_has_label(i3_label, ptr_label));
   assert(dfsan_has_label(ptr3_label, i_label));
   assert(dfsan_has_label(ptr3_label, ptr_label));
+#elif defined(FAST16_O0)
+  assert(i3_label == (i_label | ptr_label));
+  assert(ptr3_label == (i_label | ptr_label));
+#else
+  assert(i3_label == i_label);
+  assert(ptr3_label == ptr_label);
+#endif
+
 
   return 0;
 }

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

static StringRef GetGlobalTypeString(const GlobalValue &G) {
   // For now we support excluding struct types only.
   if (StructType *SGType = dyn_cast<StructType>(GType)) {
     if (!SGType->isLiteral())
       return SGType->getName();
   }
   return "<unknown type>";
 }
 
 namespace {
 
 class DFSanABIList {
   std::unique_ptr<SpecialCaseList> SCL;
 
  public:
   DFSanABIList() = default;
 
   void set(std::unique_ptr<SpecialCaseList> List) { SCL = std::move(List); }
enum WrapperKind {
     /// extra pointer argument to return the shadow.  This allows the wrapped
     /// form of the function type to be expressed in C.
     WK_Custom
   };
 
   Module *Mod;
   LLVMContext *Ctx;
   Type *Int8Ptr;
-  /// The shadow type for all primitive types. Until we support field/index
-  /// level shadow values, aggregate and vector types also use this shadow
-  /// type.
+  /// The shadow type for all primitive types and vector types.
   IntegerType *PrimitiveShadowTy;
   PointerType *PrimitiveShadowPtrTy;
   IntegerType *IntptrTy;
   ConstantInt *ZeroPrimitiveShadow;
   ConstantInt *ShadowPtrMask;
   ConstantInt *ShadowPtrMul;
   Constant *ArgTLS;
   Constant *RetvalTLS;
Function *buildWrapperFunction(Function *F, StringRef NewFName,
                                  GlobalValue::LinkageTypes NewFLink,
                                  FunctionType *NewFT);
   Constant *getOrBuildTrampolineFunction(FunctionType *FT, StringRef FName);
   void initializeCallbackFunctions(Module &M);
   void initializeRuntimeFunctions(Module &M);
 
   bool init(Module &M);
 
-  /// Returns a zero constant with the shadow type of V's type. Until we support
-  /// field/index level shadow values, the following methods always return
-  /// primitive types, values or zero constants.
+  /// Returns whether the pass tracks labels for struct fields and array
+  /// indices. Support only fast16 mode in TLS ABI mode.
+  bool shouldTrackFieldsAndIndices();
+
+  /// Returns a zero constant with the shadow type of OrigTy.
+  ///
+  /// getZeroShadow({T1,T2,...}) = {getZeroShadow(T1),getZeroShadow(T2,...}
+  /// getZeroShadow([n x T]) = [n x getZeroShadow(T)]
+  /// getZeroShadow(other type) = i16(0)
+  ///
+  /// Note that a zero shadow is always i16(0) when shouldTrackFieldsAndIndices
+  /// returns false.
+  Constant *getZeroShadow(Type *OrigTy);
+  /// Returns a zero constant with the shadow type of V's type.
   Constant *getZeroShadow(Value *V);
+
   /// Checks if V is a zero shadow.
   bool isZeroShadow(Value *V);
+
   /// Returns the shadow type of OrigTy.
+  ///
+  /// getShadowTy({T1,T2,...}) = {getShadowTy(T1),getShadowTy(T2),...}
+  /// getShadowTy([n x T]) = [n x getShadowTy(T)]
+  /// getShadowTy(other type) = i16
+  ///
+  /// Note that a shadow type is always i16 when shouldTrackFieldsAndIndices
+  /// returns false.
   Type *getShadowTy(Type *OrigTy);
   /// Returns the shadow type of of V's type.
   Type *getShadowTy(Value *V);
 
 public:
   DataFlowSanitizer(const std::vector<std::string> &ABIListFiles);
 
   bool runImpl(Module &M);
struct DFSanFunction {
   bool AvoidNewBlocks;
 
   struct CachedShadow {
     BasicBlock *Block; // The block where Shadow is defined.
     Value *Shadow;
   };
   /// Maps a value to its latest shadow value in terms of domination tree.
   DenseMap<std::pair<Value *, Value *>, CachedShadow> CachedShadows;
+  /// Maps a value to its latest collapsed shadow value it was converted to in
+  /// terms of domination tree. When ClDebugNonzeroLabels is on, this cache is
+  /// used at a post process where CFG blocks are split. So it does not cache
+  /// BasicBlock like CachedShadows, but uses domination between values.
+  DenseMap<Value *, Value *> CachedCollapsedShadows;
   DenseMap<Value *, std::set<Value *>> ShadowElements;
 
   DFSanFunction(DataFlowSanitizer &DFS, Function *F, bool IsNativeABI)
       : DFS(DFS), F(F), IA(DFS.getInstrumentedABI()), IsNativeABI(IsNativeABI) {
     DT.recalculate(*F);
     // FIXME: Need to track down the register allocator issue which causes poor
     // performance in pathological cases with large numbers of basic blocks.
     AvoidNewBlocks = F->size() > 1000;
   }
 
   /// Computes the shadow address for a given function argument.
   ///
   /// Shadow = ArgTLS+ArgOffset.
   Value *getArgTLS(Type *T, unsigned ArgOffset, IRBuilder<> &IRB);
 
   /// Computes the shadow address for a retval.
   Value *getRetvalTLS(Type *T, IRBuilder<> &IRB);
 
   Value *getShadow(Value *V);
   void setShadow(Instruction *I, Value *Shadow);
+  /// Generates IR to compute the union of the two given shadows, inserting it
+  /// before Pos. The combined value is with primitive type.
   Value *combineShadows(Value *V1, Value *V2, Instruction *Pos);
+  /// Combines the shadow values of V1 and V2, then converts the combined value
+  /// with primitive type into a shadow value with the original type T.
+  Value *combineShadowsThenConvert(Type *T, Value *V1, Value *V2,
+                                   Instruction *Pos);
   Value *combineOperandShadows(Instruction *Inst);
   Value *loadShadow(Value *ShadowAddr, uint64_t Size, uint64_t Align,
                     Instruction *Pos);
-  void storeShadow(Value *Addr, uint64_t Size, Align Alignment, Value *Shadow,
+  void storePrimitiveShadow(Value *Addr, uint64_t Size, Align Alignment,
+                            Value *PrimitiveShadow, Instruction *Pos);
+  /// Applies PrimitiveShadow to all primitive subtypes of T, returning
+  /// the expanded shadow value.
+  ///
+  /// EFP({T1,T2, ...}, PS) = {EFP(T1,PS),EFP(T2,PS),...}
+  /// EFP([n x T], PS) = [n x EFP(T,PS)]
+  /// EFP(other types, PS) = PS
+  Value *expandFromPrimitiveShadow(Type *T, Value *PrimitiveShadow,
                                    Instruction *Pos);
+  /// Collapses Shadow into a single primitive shadow value, unioning all
+  /// primitive shadow values in the process. Returns the final primitive
+  /// shadow value.
+  ///
+  /// CTP({V1,V2, ...}) = UNION(CFP(V1,PS),CFP(V2,PS),...)
+  /// CTP([V1,V2,...]) = UNION(CFP(V1,PS),CFP(V2,PS),...)
+  /// CTP(other types, PS) = PS
+  Value *collapseToPrimitiveShadow(Value *Shadow, Instruction *Pos);
 
 private:
+  /// Collapses the shadow with aggregate type into a single primitive shadow
+  /// value.
+  template <class AggregateType>
+  Value *collapseAggregateShadow(AggregateType *AT, Value *Shadow,
+                                 IRBuilder<> &IRB);
+
+  Value *collapseToPrimitiveShadow(Value *Shadow, IRBuilder<> &IRB);
+
   /// Returns the shadow value of an argument A.
   Value *getShadowForTLSArgument(Argument *A);
 };
 
 class DFSanVisitor : public InstVisitor<DFSanVisitor> {
 public:
   DFSanFunction &DFSF;
 
   DFSanVisitor(DFSanFunction &DFSF) : DFSF(DFSF) {}
 
   const DataLayout &getDataLayout() const {
     return DFSF.F->getParent()->getDataLayout();
   }
 
TransformedFunction DataFlowSanitizer::getCustomFunctionType(FunctionType *T) {
   if (!RetType->isVoidTy())
     ArgTypes.push_back(PrimitiveShadowPtrTy);
   return TransformedFunction(
       T, FunctionType::get(T->getReturnType(), ArgTypes, T->isVarArg()),
       ArgumentIndexMapping);
 }
 
 bool DataFlowSanitizer::isZeroShadow(Value *V) {
+  if (!shouldTrackFieldsAndIndices())
     return ZeroPrimitiveShadow == V;
+
+  Type *T = V->getType();
+  if (!isa<ArrayType>(T) && !isa<StructType>(T)) {
+    if (const ConstantInt *CI = dyn_cast<ConstantInt>(V))
+      return CI->isZero();
+    return false;
   }
 
-Constant *DataFlowSanitizer::getZeroShadow(Value *V) {
+  return isa<ConstantAggregateZero>(V);
+}
+
+bool DataFlowSanitizer::shouldTrackFieldsAndIndices() {
+  return getInstrumentedABI() == DataFlowSanitizer::IA_TLS && ClFast16Labels;
+}
+
+Constant *DataFlowSanitizer::getZeroShadow(Type *OrigTy) {
+  if (!shouldTrackFieldsAndIndices())
+    return ZeroPrimitiveShadow;
+
+  if (!isa<ArrayType>(OrigTy) && !isa<StructType>(OrigTy))
     return ZeroPrimitiveShadow;
+  Type *ShadowTy = getShadowTy(OrigTy);
+  return ConstantAggregateZero::get(ShadowTy);
+}
+
+Constant *DataFlowSanitizer::getZeroShadow(Value *V) {
+  return getZeroShadow(V->getType());
+}
+
+static Value *expandFromPrimitiveShadowRecursive(
+    Value *Shadow, SmallVector<unsigned, 4> &Indices, Type *SubShadowTy,
+    Value *PrimitiveShadow, IRBuilder<> &IRB) {
+  if (!isa<ArrayType>(SubShadowTy) && !isa<StructType>(SubShadowTy))
+    return IRB.CreateInsertValue(Shadow, PrimitiveShadow, Indices);
+
+  if (ArrayType *AT = dyn_cast<ArrayType>(SubShadowTy)) {
+    for (unsigned Idx = 0; Idx < AT->getNumElements(); Idx++) {
+      Indices.push_back(Idx);
+      Shadow = expandFromPrimitiveShadowRecursive(
+          Shadow, Indices, AT->getElementType(), PrimitiveShadow, IRB);
+      Indices.pop_back();
+    }
+    return Shadow;
+  }
+
+  if (StructType *ST = dyn_cast<StructType>(SubShadowTy)) {
+    for (unsigned Idx = 0; Idx < ST->getNumElements(); Idx++) {
+      Indices.push_back(Idx);
+      Shadow = expandFromPrimitiveShadowRecursive(
+          Shadow, Indices, ST->getElementType(Idx), PrimitiveShadow, IRB);
+      Indices.pop_back();
+    }
+    return Shadow;
+  }
+  llvm_unreachable("Unexpected shadow type");
+}
+
+Value *DFSanFunction::expandFromPrimitiveShadow(Type *T, Value *PrimitiveShadow,
+                                                Instruction *Pos) {
+  Type *ShadowTy = DFS.getShadowTy(T);
+
+  if (!isa<ArrayType>(ShadowTy) && !isa<StructType>(ShadowTy))
+    return PrimitiveShadow;
+
+  if (DFS.isZeroShadow(PrimitiveShadow))
+    return DFS.getZeroShadow(ShadowTy);
+
+  IRBuilder<> IRB(Pos);
+  SmallVector<unsigned, 4> Indices;
+  Value *Shadow = UndefValue::get(ShadowTy);
+  Shadow = expandFromPrimitiveShadowRecursive(Shadow, Indices, ShadowTy,
+                                              PrimitiveShadow, IRB);
+
+  // Caches the primitive shadow value that built the shadow value.
+  CachedCollapsedShadows[Shadow] = PrimitiveShadow;
+  return Shadow;
 }
 
-Type *DataFlowSanitizer::getShadowTy(Type *OrigTy) { return PrimitiveShadowTy; }
+template <class AggregateType>
+Value *DFSanFunction::collapseAggregateShadow(AggregateType *AT, Value *Shadow,
+                                              IRBuilder<> &IRB) {
+  if (!AT->getNumElements())
+    return DFS.ZeroPrimitiveShadow;
+
+  Value *FirstItem = IRB.CreateExtractValue(Shadow, 0);
+  Value *Aggregator = collapseToPrimitiveShadow(FirstItem, IRB);
+
+  for (unsigned Idx = 1; Idx < AT->getNumElements(); Idx++) {
+    Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
+    Value *ShadowInner = collapseToPrimitiveShadow(ShadowItem, IRB);
+    Aggregator = IRB.CreateOr(Aggregator, ShadowInner);
+  }
+  return Aggregator;
+}
+
+Value *DFSanFunction::collapseToPrimitiveShadow(Value *Shadow,
+                                                IRBuilder<> &IRB) {
+  Type *ShadowTy = Shadow->getType();
+  if (!isa<ArrayType>(ShadowTy) && !isa<StructType>(ShadowTy))
+    return Shadow;
+  if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy))
+    return collapseAggregateShadow<>(AT, Shadow, IRB);
+  if (StructType *ST = dyn_cast<StructType>(ShadowTy))
+    return collapseAggregateShadow<>(ST, Shadow, IRB);
+  llvm_unreachable("Unexpected shadow type");
+}
+
+Value *DFSanFunction::collapseToPrimitiveShadow(Value *Shadow,
+                                                Instruction *Pos) {
+  Type *ShadowTy = Shadow->getType();
+  if (!isa<ArrayType>(ShadowTy) && !isa<StructType>(ShadowTy))
+    return Shadow;
+
+  assert(DFS.shouldTrackFieldsAndIndices());
+
+  // Checks if the cached collapsed shadow value dominates Pos.
+  Value *&CS = CachedCollapsedShadows[Shadow];
+  if (CS && DT.dominates(CS, Pos))
+    return CS;
+
+  IRBuilder<> IRB(Pos);
+  Value *PrimitiveShadow = collapseToPrimitiveShadow(Shadow, IRB);
+  // Caches the converted primitive shadow value.
+  CS = PrimitiveShadow;
+  return PrimitiveShadow;
+}
+
+Type *DataFlowSanitizer::getShadowTy(Type *OrigTy) {
+  if (!shouldTrackFieldsAndIndices())
+    return PrimitiveShadowTy;
+
+  if (!OrigTy->isSized())
+    return PrimitiveShadowTy;
+  if (isa<IntegerType>(OrigTy))
+    return PrimitiveShadowTy;
+  if (isa<VectorType>(OrigTy))
+    return PrimitiveShadowTy;
+  if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy))
+    return ArrayType::get(getShadowTy(AT->getElementType()),
+                          AT->getNumElements());
+  if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
+    SmallVector<Type *, 4> Elements;
+    for (unsigned I = 0, N = ST->getNumElements(); I < N; ++I)
+      Elements.push_back(getShadowTy(ST->getElementType(I)));
+    return StructType::get(*Ctx, Elements);
+  }
+  return PrimitiveShadowTy;
+}
 
 Type *DataFlowSanitizer::getShadowTy(Value *V) {
   return getShadowTy(V->getType());
 }
 
 bool DataFlowSanitizer::init(Module &M) {
   Triple TargetTriple(M.getTargetTriple());
   bool IsX86_64 = TargetTriple.getArch() == Triple::x86_64;
for (unsigned N = FT->getNumParams(); N != 0; ++AI, --N)
       Args.push_back(&*AI);
     CallInst *CI = CallInst::Create(FT, &*F->arg_begin(), Args, "", BB);
     ReturnInst *RI;
     if (FT->getReturnType()->isVoidTy())
       RI = ReturnInst::Create(*Ctx, BB);
     else
       RI = ReturnInst::Create(*Ctx, CI, BB);
 
+    // F is called by a wrapped custom function with primitive shadows. So
+    // its arguments and return value need conversion.
     DFSanFunction DFSF(*this, F, /*IsNativeABI=*/true);
     Function::arg_iterator ValAI = F->arg_begin(), ShadowAI = AI; ++ValAI;
-    for (unsigned N = FT->getNumParams(); N != 0; ++ValAI, ++ShadowAI, --N)
-      DFSF.ValShadowMap[&*ValAI] = &*ShadowAI;
+    for (unsigned N = FT->getNumParams(); N != 0; ++ValAI, ++ShadowAI, --N) {
+      Value *Shadow =
+          DFSF.expandFromPrimitiveShadow(ValAI->getType(), &*ShadowAI, CI);
+      DFSF.ValShadowMap[&*ValAI] = Shadow;
+    }
     DFSanVisitor(DFSF).visitCallInst(*CI);
-    if (!FT->getReturnType()->isVoidTy())
-      new StoreInst(DFSF.getShadow(RI->getReturnValue()),
-                    &*std::prev(F->arg_end()), RI);
+    if (!FT->getReturnType()->isVoidTy()) {
+      Value *PrimitiveShadow = DFSF.collapseToPrimitiveShadow(
+          DFSF.getShadow(RI->getReturnValue()), RI);
+      new StoreInst(PrimitiveShadow, &*std::prev(F->arg_end()), RI);
+    }
   }
 
   return cast<Constant>(C.getCallee());
 }
 
 // Initialize DataFlowSanitizer runtime functions and declare them in the module
 void DataFlowSanitizer::initializeRuntimeFunctions(Module &M) {
   {
if (ClDebugNonzeroLabels) {
         Instruction *Pos;
         if (Instruction *I = dyn_cast<Instruction>(V))
           Pos = I->getNextNode();
         else
           Pos = &DFSF.F->getEntryBlock().front();
         while (isa<PHINode>(Pos) || isa<AllocaInst>(Pos))
           Pos = Pos->getNextNode();
         IRBuilder<> IRB(Pos);
-        Value *Ne = IRB.CreateICmpNE(V, DFSF.DFS.ZeroPrimitiveShadow);
+        Value *PrimitiveShadow = DFSF.collapseToPrimitiveShadow(V, Pos);
+        Value *Ne =
+            IRB.CreateICmpNE(PrimitiveShadow, DFSF.DFS.ZeroPrimitiveShadow);
         BranchInst *BI = cast<BranchInst>(SplitBlockAndInsertIfThen(
             Ne, Pos, /*Unreachable=*/false, ColdCallWeights));
         IRBuilder<> ThenIRB(BI);
         ThenIRB.CreateCall(DFSF.DFS.DFSanNonzeroLabelFn, {});
       }
     }
   }
 
if (Argument *A = dyn_cast<Argument>(V)) {
       Shadow = DFS.getZeroShadow(V);
     }
   }
   return Shadow;
 }
 
 void DFSanFunction::setShadow(Instruction *I, Value *Shadow) {
   assert(!ValShadowMap.count(I));
-  assert(Shadow->getType() == DFS.PrimitiveShadowTy);
+  assert(DFS.shouldTrackFieldsAndIndices() ||
+         Shadow->getType() == DFS.PrimitiveShadowTy);
   ValShadowMap[I] = Shadow;
 }
 
 Value *DataFlowSanitizer::getShadowAddress(Value *Addr, Instruction *Pos) {
   assert(Addr != RetvalTLS && "Reinstrumenting?");
   IRBuilder<> IRB(Pos);
   Value *ShadowPtrMaskValue;
   if (DFSanRuntimeShadowMask)
     ShadowPtrMaskValue = IRB.CreateLoad(IntptrTy, ExternalShadowMask);
   else
     ShadowPtrMaskValue = ShadowPtrMask;
   return IRB.CreateIntToPtr(
       IRB.CreateMul(
           IRB.CreateAnd(IRB.CreatePtrToInt(Addr, IntptrTy),
                         IRB.CreatePtrToInt(ShadowPtrMaskValue, IntptrTy)),
           ShadowPtrMul),
       PrimitiveShadowPtrTy);
 }
 
+Value *DFSanFunction::combineShadowsThenConvert(Type *T, Value *V1, Value *V2,
+                                                Instruction *Pos) {
+  Value *PrimitiveValue = combineShadows(V1, V2, Pos);
+  return expandFromPrimitiveShadow(T, PrimitiveValue, Pos);
+}
+
 // Generates IR to compute the union of the two given shadows, inserting it
-// before Pos.  Returns the computed union Value.
+// before Pos. The combined value is with primitive type.
 Value *DFSanFunction::combineShadows(Value *V1, Value *V2, Instruction *Pos) {
   if (DFS.isZeroShadow(V1))
-    return V2;
+    return collapseToPrimitiveShadow(V2, Pos);
   if (DFS.isZeroShadow(V2))
-    return V1;
+    return collapseToPrimitiveShadow(V1, Pos);
   if (V1 == V2)
-    return V1;
+    return collapseToPrimitiveShadow(V1, Pos);
 
   auto V1Elems = ShadowElements.find(V1);
   auto V2Elems = ShadowElements.find(V2);
   if (V1Elems != ShadowElements.end() && V2Elems != ShadowElements.end()) {
     if (std::includes(V1Elems->second.begin(), V1Elems->second.end(),
                       V2Elems->second.begin(), V2Elems->second.end())) {
-      return V1;
+      return collapseToPrimitiveShadow(V1, Pos);
     } else if (std::includes(V2Elems->second.begin(), V2Elems->second.end(),
                              V1Elems->second.begin(), V1Elems->second.end())) {
-      return V2;
+      return collapseToPrimitiveShadow(V2, Pos);
     }
   } else if (V1Elems != ShadowElements.end()) {
     if (V1Elems->second.count(V2))
-      return V1;
+      return collapseToPrimitiveShadow(V1, Pos);
   } else if (V2Elems != ShadowElements.end()) {
     if (V2Elems->second.count(V1))
-      return V2;
+      return collapseToPrimitiveShadow(V2, Pos);
   }
 
   auto Key = std::make_pair(V1, V2);
   if (V1 > V2)
     std::swap(Key.first, Key.second);
   CachedShadow &CCS = CachedShadows[Key];
   if (CCS.Block && DT.dominates(CCS.Block, Pos->getParent()))
     return CCS.Shadow;
 
+  // Converts inputs shadows to shadows with primitive types.
+  Value *PV1 = collapseToPrimitiveShadow(V1, Pos);
+  Value *PV2 = collapseToPrimitiveShadow(V2, Pos);
+
   IRBuilder<> IRB(Pos);
   if (ClFast16Labels) {
     CCS.Block = Pos->getParent();
-    CCS.Shadow = IRB.CreateOr(V1, V2);
+    CCS.Shadow = IRB.CreateOr(PV1, PV2);
   } else if (AvoidNewBlocks) {
-    CallInst *Call = IRB.CreateCall(DFS.DFSanCheckedUnionFn, {V1, V2});
+    CallInst *Call = IRB.CreateCall(DFS.DFSanCheckedUnionFn, {PV1, PV2});
     Call->addAttribute(AttributeList::ReturnIndex, Attribute::ZExt);
     Call->addParamAttr(0, Attribute::ZExt);
     Call->addParamAttr(1, Attribute::ZExt);
 
     CCS.Block = Pos->getParent();
     CCS.Shadow = Call;
   } else {
     BasicBlock *Head = Pos->getParent();
-    Value *Ne = IRB.CreateICmpNE(V1, V2);
+    Value *Ne = IRB.CreateICmpNE(PV1, PV2);
     BranchInst *BI = cast<BranchInst>(SplitBlockAndInsertIfThen(
         Ne, Pos, /*Unreachable=*/false, DFS.ColdCallWeights, &DT));
     IRBuilder<> ThenIRB(BI);
-    CallInst *Call = ThenIRB.CreateCall(DFS.DFSanUnionFn, {V1, V2});
+    CallInst *Call = ThenIRB.CreateCall(DFS.DFSanUnionFn, {PV1, PV2});
     Call->addAttribute(AttributeList::ReturnIndex, Attribute::ZExt);
     Call->addParamAttr(0, Attribute::ZExt);
     Call->addParamAttr(1, Attribute::ZExt);
 
     BasicBlock *Tail = BI->getSuccessor(0);
     PHINode *Phi =
         PHINode::Create(DFS.PrimitiveShadowTy, 2, "", &Tail->front());
     Phi->addIncoming(Call, Call->getParent());
-    Phi->addIncoming(V1, Head);
+    Phi->addIncoming(PV1, Head);
 
     CCS.Block = Tail;
     CCS.Shadow = Phi;
   }
 
   std::set<Value *> UnionElems;
   if (V1Elems != ShadowElements.end()) {
     UnionElems = V1Elems->second;

 Value *DFSanFunction::combineOperandShadows(Instruction *Inst) {
   if (Inst->getNumOperands() == 0)
     return DFS.getZeroShadow(Inst);
 
   Value *Shadow = getShadow(Inst->getOperand(0));
   for (unsigned i = 1, n = Inst->getNumOperands(); i != n; ++i) {
     Shadow = combineShadows(Shadow, getShadow(Inst->getOperand(i)), Inst);
   }
-  return Shadow;
+  return expandFromPrimitiveShadow(Inst->getType(), Shadow, Inst);
 }
 
 Value *DFSanVisitor::visitOperandShadowInst(Instruction &I) {
   Value *CombinedShadow = DFSF.combineOperandShadows(&I);
   DFSF.setShadow(&I, CombinedShadow);
   return CombinedShadow;
 }
 
 // Generates IR to load shadow corresponding to bytes [Addr, Addr+Size), where
-// Addr has alignment Align, and take the union of each of those shadows.
+// Addr has alignment Align, and take the union of each of those shadows. The
+// returned shadow always has primitive type.
 Value *DFSanFunction::loadShadow(Value *Addr, uint64_t Size, uint64_t Align,
                                  Instruction *Pos) {
   if (AllocaInst *AI = dyn_cast<AllocaInst>(Addr)) {
     const auto i = AllocaShadowMap.find(AI);
     if (i != AllocaShadowMap.end()) {
       IRBuilder<> IRB(Pos);
       return IRB.CreateLoad(DFS.PrimitiveShadowTy, i->second);
     }
void DFSanVisitor::visitLoadInst(LoadInst &LI) {
   auto &DL = LI.getModule()->getDataLayout();
   uint64_t Size = DL.getTypeStoreSize(LI.getType());
   if (Size == 0) {
     DFSF.setShadow(&LI, DFSF.DFS.getZeroShadow(&LI));
     return;
   }
 
   Align Alignment = ClPreserveAlignment ? LI.getAlign() : Align(1);
-  Value *Shadow =
+  Value *PrimitiveShadow =
       DFSF.loadShadow(LI.getPointerOperand(), Size, Alignment.value(), &LI);
   if (ClCombinePointerLabelsOnLoad) {
     Value *PtrShadow = DFSF.getShadow(LI.getPointerOperand());
-    Shadow = DFSF.combineShadows(Shadow, PtrShadow, &LI);
+    PrimitiveShadow = DFSF.combineShadows(PrimitiveShadow, PtrShadow, &LI);
   }
-  if (!DFSF.DFS.isZeroShadow(Shadow))
-    DFSF.NonZeroChecks.push_back(Shadow);
+  if (!DFSF.DFS.isZeroShadow(PrimitiveShadow))
+    DFSF.NonZeroChecks.push_back(PrimitiveShadow);
 
+  Value *Shadow =
+      DFSF.expandFromPrimitiveShadow(LI.getType(), PrimitiveShadow, &LI);
   DFSF.setShadow(&LI, Shadow);
   if (ClEventCallbacks) {
     IRBuilder<> IRB(&LI);
     Value *Addr8 = IRB.CreateBitCast(LI.getPointerOperand(), DFSF.DFS.Int8Ptr);
-    IRB.CreateCall(DFSF.DFS.DFSanLoadCallbackFn, {Shadow, Addr8});
+    IRB.CreateCall(DFSF.DFS.DFSanLoadCallbackFn, {PrimitiveShadow, Addr8});
   }
 }
 
-void DFSanFunction::storeShadow(Value *Addr, uint64_t Size, Align Alignment,
-                                Value *Shadow, Instruction *Pos) {
+void DFSanFunction::storePrimitiveShadow(Value *Addr, uint64_t Size,
+                                         Align Alignment,
+                                         Value *PrimitiveShadow,
+                                         Instruction *Pos) {
   if (AllocaInst *AI = dyn_cast<AllocaInst>(Addr)) {
     const auto i = AllocaShadowMap.find(AI);
     if (i != AllocaShadowMap.end()) {
       IRBuilder<> IRB(Pos);
-      IRB.CreateStore(Shadow, i->second);
+      IRB.CreateStore(PrimitiveShadow, i->second);
       return;
     }
   }
 
   const Align ShadowAlign(Alignment.value() * DFS.ShadowWidthBytes);
   IRBuilder<> IRB(Pos);
   Value *ShadowAddr = DFS.getShadowAddress(Addr, Pos);
-  if (DFS.isZeroShadow(Shadow)) {
+  if (DFS.isZeroShadow(PrimitiveShadow)) {
     IntegerType *ShadowTy =
         IntegerType::get(*DFS.Ctx, Size * DFS.ShadowWidthBits);
     Value *ExtZeroShadow = ConstantInt::get(ShadowTy, 0);
     Value *ExtShadowAddr =
         IRB.CreateBitCast(ShadowAddr, PointerType::getUnqual(ShadowTy));
     IRB.CreateAlignedStore(ExtZeroShadow, ExtShadowAddr, ShadowAlign);
     return;
   }
 
   const unsigned ShadowVecSize = 128 / DFS.ShadowWidthBits;
   uint64_t Offset = 0;
   if (Size >= ShadowVecSize) {
     auto *ShadowVecTy =
         FixedVectorType::get(DFS.PrimitiveShadowTy, ShadowVecSize);
     Value *ShadowVec = UndefValue::get(ShadowVecTy);
     for (unsigned i = 0; i != ShadowVecSize; ++i) {
       ShadowVec = IRB.CreateInsertElement(
-          ShadowVec, Shadow, ConstantInt::get(Type::getInt32Ty(*DFS.Ctx), i));
+          ShadowVec, PrimitiveShadow,
+          ConstantInt::get(Type::getInt32Ty(*DFS.Ctx), i));
     }
     Value *ShadowVecAddr =
         IRB.CreateBitCast(ShadowAddr, PointerType::getUnqual(ShadowVecTy));
     do {
       Value *CurShadowVecAddr =
           IRB.CreateConstGEP1_32(ShadowVecTy, ShadowVecAddr, Offset);
       IRB.CreateAlignedStore(ShadowVec, CurShadowVecAddr, ShadowAlign);
       Size -= ShadowVecSize;
       ++Offset;
     } while (Size >= ShadowVecSize);
     Offset *= ShadowVecSize;
   }
   while (Size > 0) {
     Value *CurShadowAddr =
         IRB.CreateConstGEP1_32(DFS.PrimitiveShadowTy, ShadowAddr, Offset);
-    IRB.CreateAlignedStore(Shadow, CurShadowAddr, ShadowAlign);
+    IRB.CreateAlignedStore(PrimitiveShadow, CurShadowAddr, ShadowAlign);
     --Size;
     ++Offset;
   }
 }
 
 void DFSanVisitor::visitStoreInst(StoreInst &SI) {
   auto &DL = SI.getModule()->getDataLayout();
   uint64_t Size = DL.getTypeStoreSize(SI.getValueOperand()->getType());
   if (Size == 0)
     return;
 
   const Align Alignment = ClPreserveAlignment ? SI.getAlign() : Align(1);
 
   Value* Shadow = DFSF.getShadow(SI.getValueOperand());
+  Value *PrimitiveShadow;
   if (ClCombinePointerLabelsOnStore) {
     Value *PtrShadow = DFSF.getShadow(SI.getPointerOperand());
-    Shadow = DFSF.combineShadows(Shadow, PtrShadow, &SI);
+    PrimitiveShadow = DFSF.combineShadows(Shadow, PtrShadow, &SI);
+  } else {
+    PrimitiveShadow = DFSF.collapseToPrimitiveShadow(Shadow, &SI);
   }
-  DFSF.storeShadow(SI.getPointerOperand(), Size, Alignment, Shadow, &SI);
+  DFSF.storePrimitiveShadow(SI.getPointerOperand(), Size, Alignment,
+                            PrimitiveShadow, &SI);
   if (ClEventCallbacks) {
     IRBuilder<> IRB(&SI);
     Value *Addr8 = IRB.CreateBitCast(SI.getPointerOperand(), DFSF.DFS.Int8Ptr);
-    IRB.CreateCall(DFSF.DFS.DFSanStoreCallbackFn, {Shadow, Addr8});
+    IRB.CreateCall(DFSF.DFS.DFSanStoreCallbackFn, {PrimitiveShadow, Addr8});
   }
 }
 
 void DFSanVisitor::visitUnaryOperator(UnaryOperator &UO) {
   visitOperandShadowInst(UO);
 }
 
 void DFSanVisitor::visitBinaryOperator(BinaryOperator &BO) {
void DFSanVisitor::visitInsertElementInst(InsertElementInst &I) {
   visitOperandShadowInst(I);
 }
 
 void DFSanVisitor::visitShuffleVectorInst(ShuffleVectorInst &I) {
   visitOperandShadowInst(I);
 }
 
 void DFSanVisitor::visitExtractValueInst(ExtractValueInst &I) {
+  if (!DFSF.DFS.shouldTrackFieldsAndIndices()) {
     visitOperandShadowInst(I);
+    return;
+  }
+
+  IRBuilder<> IRB(&I);
+  Value *Agg = I.getAggregateOperand();
+  Value *AggShadow = DFSF.getShadow(Agg);
+  Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
+  DFSF.setShadow(&I, ResShadow);
 }
 
 void DFSanVisitor::visitInsertValueInst(InsertValueInst &I) {
+  if (!DFSF.DFS.shouldTrackFieldsAndIndices()) {
     visitOperandShadowInst(I);
+    return;
+  }
+
+  IRBuilder<> IRB(&I);
+  Value *AggShadow = DFSF.getShadow(I.getAggregateOperand());
+  Value *InsShadow = DFSF.getShadow(I.getInsertedValueOperand());
+  Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
+  DFSF.setShadow(&I, Res);
 }
 
 void DFSanVisitor::visitAllocaInst(AllocaInst &I) {
   bool AllLoadsStores = true;
   for (User *U : I.users()) {
     if (isa<LoadInst>(U))
       continue;
 

 
 void DFSanVisitor::visitSelectInst(SelectInst &I) {
   Value *CondShadow = DFSF.getShadow(I.getCondition());
   Value *TrueShadow = DFSF.getShadow(I.getTrueValue());
   Value *FalseShadow = DFSF.getShadow(I.getFalseValue());
   Value *ShadowSel = nullptr;
 
   if (isa<VectorType>(I.getCondition()->getType())) {
-    ShadowSel = DFSF.combineShadows(TrueShadow, FalseShadow, &I);
+    ShadowSel = DFSF.combineShadowsThenConvert(I.getType(), TrueShadow,
+                                               FalseShadow, &I);
   } else {
     if (TrueShadow == FalseShadow) {
       ShadowSel = TrueShadow;
     } else {
       ShadowSel =
           SelectInst::Create(I.getCondition(), TrueShadow, FalseShadow, "", &I);
     }
   }
   DFSF.setShadow(&I, ClTrackSelectControlFlow
-                         ? DFSF.combineShadows(CondShadow, ShadowSel, &I)
+                         ? DFSF.combineShadowsThenConvert(
+                               I.getType(), CondShadow, ShadowSel, &I)
                          : ShadowSel);
 }
 
 void DFSanVisitor::visitMemSetInst(MemSetInst &I) {
   IRBuilder<> IRB(&I);
   Value *ValShadow = DFSF.getShadow(I.getValue());
   IRB.CreateCall(DFSF.DFS.DFSanSetLabelFn,
                  {ValShadow, IRB.CreateBitCast(I.getDest(), Type::getInt8PtrTy(
case DataFlowSanitizer::WK_Custom:
           } else {
             Args.push_back(*i);
           }
         }
 
         i = CB.arg_begin();
         const unsigned ShadowArgStart = Args.size();
         for (unsigned n = FT->getNumParams(); n != 0; ++i, --n)
-          Args.push_back(DFSF.getShadow(*i));
+          Args.push_back(
+              DFSF.collapseToPrimitiveShadow(DFSF.getShadow(*i), &CB));
 
         if (FT->isVarArg()) {
           auto *LabelVATy = ArrayType::get(DFSF.DFS.PrimitiveShadowTy,
                                            CB.arg_size() - FT->getNumParams());
           auto *LabelVAAlloca = new AllocaInst(
               LabelVATy, getDataLayout().getAllocaAddrSpace(),
               "labelva", &DFSF.F->getEntryBlock().front());
 
           for (unsigned n = 0; i != CB.arg_end(); ++i, ++n) {
             auto LabelVAPtr = IRB.CreateStructGEP(LabelVATy, LabelVAAlloca, n);
-            IRB.CreateStore(DFSF.getShadow(*i), LabelVAPtr);
+            IRB.CreateStore(
+                DFSF.collapseToPrimitiveShadow(DFSF.getShadow(*i), &CB),
+                LabelVAPtr);
           }
 
           Args.push_back(IRB.CreateStructGEP(LabelVATy, LabelVAAlloca, 0));
         }
 
         if (!FT->getReturnType()->isVoidTy()) {
           if (!DFSF.LabelReturnAlloca) {
             DFSF.LabelReturnAlloca =
case DataFlowSanitizer::WK_Custom:
           if (CustomCI->getArgOperand(ArgNo)->getType() ==
               DFSF.DFS.PrimitiveShadowTy)
             CustomCI->addParamAttr(ArgNo, Attribute::ZExt);
         }
 
         if (!FT->getReturnType()->isVoidTy()) {
           LoadInst *LabelLoad = IRB.CreateLoad(DFSF.DFS.PrimitiveShadowTy,
                                                DFSF.LabelReturnAlloca);
-          DFSF.setShadow(CustomCI, LabelLoad);
+          DFSF.setShadow(CustomCI, DFSF.expandFromPrimitiveShadow(
+                                       FT->getReturnType(), LabelLoad, &CB));
         }
 
         CI->replaceAllUsesWith(CustomCI);
         CI->eraseFromParent();
         return;
       }
       break;
     }

llvm/test/Instrumentation/DataFlowSanitizer/abilist_aggregate.ll

+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s --check-prefix=TLS_ABI
+; RUN: opt < %s -dfsan -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s --check-prefix=LEGACY
+; RUN: opt < %s -dfsan -dfsan-args-abi -dfsan-abilist=%S/Inputs/abilist.txt -S | FileCheck %s --check-prefix=ARGS_ABI
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; TLS_ABI: define { i1, i7 } @functional({ i32, i1 } %a, [2 x i7] %b)
+; ARGS_ABI: define { i1, i7 } @functional({ i32, i1 } %a, [2 x i7] %b)
+define {i1, i7} @functional({i32, i1} %a, [2 x i7] %b) {
+  %a1 = extractvalue {i32, i1} %a, 1
+  %b0 = extractvalue [2 x i7] %b, 0
+  %r0 = insertvalue {i1, i7} undef, i1 %a1, 0
+  %r1 = insertvalue {i1, i7} %r0, i7 %b0, 1
+  ret {i1, i7} %r1
+}
+
+define {i1, i7} @call_functional({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: @"dfs$call_functional"
+  ; TLS_ABI: [[B:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN:2]]
+  ; TLS_ABI: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: [[A0:%.*]] = extractvalue { i16, i16 } [[A]], 0
+  ; TLS_ABI: [[A1:%.*]] = extractvalue { i16, i16 } [[A]], 1
+  ; TLS_ABI: [[A01:%.*]] = or i16 [[A0]], [[A1]]
+  ; TLS_ABI: [[B0:%.*]] = extractvalue [2 x i16] [[B]], 0
+  ; TLS_ABI: [[B1:%.*]] = extractvalue [2 x i16] [[B]], 1
+  ; TLS_ABI: [[B01:%.*]] = or i16 [[B0]], [[B1]]
+  ; TLS_ABI: [[U:%.*]] = or i16 [[A01]], [[B01]]
+  ; TLS_ABI: [[R0:%.*]] = insertvalue { i16, i16 } undef, i16 [[U]], 0
+  ; TLS_ABI: [[R1:%.*]] = insertvalue { i16, i16 } [[R0]], i16 [[U]], 1
+  ; TLS_ABI: store { i16, i16 } [[R1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+  
+  ; LEGACY: @"dfs$call_functional"
+  ; LEGACY: [[B:%.*]] = load i16, i16* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i16*), align [[ALIGN:2]]
+  ; LEGACY: [[A:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; LEGACY: [[U:%.*]] = call zeroext i16 @__dfsan_union(i16 zeroext [[A]], i16 zeroext [[B]])
+  ; LEGACY: [[PH:%.*]] = phi i16 [ [[U]], {{.*}} ], [ [[A]], {{.*}} ]
+  ; LEGACY: store i16 [[PH]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+
+  ; ARGS_ABI: @"dfs$call_functional"
+  ; ARGS_ABI: [[U:%.*]]  = call zeroext i16 @__dfsan_union(i16 zeroext %2, i16 zeroext %3)
+  ; ARGS_ABI: [[PH:%.*]] = phi i16 [ %7, {{.*}} ], [ %2, {{.*}} ]
+  ; ARGS_ABI: [[R0:%.*]] = insertvalue { { i1, i7 }, i16 } undef, { i1, i7 } %r, 0
+  ; ARGS_ABI: [[R1:%.*]] = insertvalue { { i1, i7 }, i16 } [[R0]], i16 [[PH]], 1
+  ; ARGS_ABI: ret { { i1, i7 }, i16 } [[R1]]
+  
+  %r = call {i1, i7} @functional({i32, i1} %a, [2 x i7] %b)
+  ret {i1, i7} %r  
+}
+
+; TLS_ABI: define { i1, i7 } @discard({ i32, i1 } %a, [2 x i7] %b)
+define {i1, i7} @discard({i32, i1} %a, [2 x i7] %b) {
+  %a1 = extractvalue {i32, i1} %a, 1
+  %b0 = extractvalue [2 x i7] %b, 0
+  %r0 = insertvalue {i1, i7} undef, i1 %a1, 0
+  %r1 = insertvalue {i1, i7} %r0, i7 %b0, 1
+  ret {i1, i7} %r1
+}
+
+define {i1, i7} @call_discard({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: @"dfs$call_discard"
+  ; TLS_ABI: store { i16, i16 } zeroinitializer, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align 2
+  
+  ; ARGS_ABI: @"dfs$call_discard"
+  ; ARGS_ABI: %r = call { i1, i7 } @discard({ i32, i1 } %0, [2 x i7] %1)
+  ; ARGS_ABI: [[R0:%.*]] = insertvalue { { i1, i7 }, i16 } undef, { i1, i7 } %r, 0
+  ; ARGS_ABI: [[R1:%.*]] = insertvalue { { i1, i7 }, i16 } [[R0]], i16 0, 1
+  ; ARGS_ABI: ret { { i1, i7 }, i16 } [[R1]]
+  
+  %r = call {i1, i7} @discard({i32, i1} %a, [2 x i7] %b)
+  ret {i1, i7} %r  
+}
+
+; TLS_ABI: define { i1, i7 } @uninstrumented({ i32, i1 } %a, [2 x i7] %b)
+define {i1, i7} @uninstrumented({i32, i1} %a, [2 x i7] %b) {
+  %a1 = extractvalue {i32, i1} %a, 1
+  %b0 = extractvalue [2 x i7] %b, 0
+  %r0 = insertvalue {i1, i7} undef, i1 %a1, 0
+  %r1 = insertvalue {i1, i7} %r0, i7 %b0, 1
+  ret {i1, i7} %r1
+}
+
+define {i1, i7} @call_uninstrumented({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: @"dfs$call_uninstrumented"
+  ; TLS_ABI: call void @__dfsan_unimplemented
+  ; TLS_ABI: store { i16, i16 } zeroinitializer, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align 2
+  
+  ; ARGS_ABI: @"dfs$call_uninstrumented"
+  ; ARGS_ABI: call void @__dfsan_unimplemented
+  ; ARGS_ABI: %r = call { i1, i7 } @uninstrumented({ i32, i1 } %0, [2 x i7] %1)
+  ; ARGS_ABI: [[R0:%.*]] = insertvalue { { i1, i7 }, i16 } undef, { i1, i7 } %r, 0
+  ; ARGS_ABI: [[R1:%.*]] = insertvalue { { i1, i7 }, i16 } [[R0]], i16 0, 1
+  ; ARGS_ABI: ret { { i1, i7 }, i16 } [[R1]]
+  
+  %r = call {i1, i7} @uninstrumented({i32, i1} %a, [2 x i7] %b)
+  ret {i1, i7} %r  
+}
+
+define {i1, i7} @call_custom_with_ret({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: @"dfs$call_custom_with_ret"
+  ; TLS_ABI: %labelreturn = alloca i16, align 2
+  ; TLS_ABI: [[B:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN:2]]
+  ; TLS_ABI: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: [[A0:%.*]] = extractvalue { i16, i16 } [[A]], 0
+  ; TLS_ABI: [[A1:%.*]] = extractvalue { i16, i16 } [[A]], 1
+  ; TLS_ABI: [[A01:%.*]] = or i16 [[A0]], [[A1]]
+  ; TLS_ABI: [[B0:%.*]] = extractvalue [2 x i16] [[B]], 0
+  ; TLS_ABI: [[B1:%.*]] = extractvalue [2 x i16] [[B]], 1
+  ; TLS_ABI: [[B01:%.*]] = or i16 [[B0]], [[B1]]
+  ; TLS_ABI: [[R:%.*]] = call { i1, i7 } @__dfsw_custom_with_ret({ i32, i1 } %a, [2 x i7] %b, i16 zeroext [[A01]], i16 zeroext [[B01]], i16* %labelreturn)
+  ; TLS_ABI: [[RE:%.*]] = load i16, i16* %labelreturn, align [[ALIGN]]
+  ; TLS_ABI: [[RS0:%.*]] = insertvalue { i16, i16 } undef, i16 [[RE]], 0
+  ; TLS_ABI: [[RS1:%.*]] = insertvalue { i16, i16 } [[RS0]], i16 [[RE]], 1
+  ; TLS_ABI: store { i16, i16 } [[RS1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: ret { i1, i7 } [[R]]
+  
+  %r = call {i1, i7} @custom_with_ret({i32, i1} %a, [2 x i7] %b)
+  ret {i1, i7} %r  
+}
+
+define void @call_custom_without_ret({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: @"dfs$call_custom_without_ret"
+  ; TLS_ABI: [[B:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN:2]]
+  ; TLS_ABI: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: [[A0:%.*]] = extractvalue { i16, i16 } [[A]], 0
+  ; TLS_ABI: [[A1:%.*]] = extractvalue { i16, i16 } [[A]], 1
+  ; TLS_ABI: [[A01:%.*]] = or i16 [[A0]], [[A1]]
+  ; TLS_ABI: [[B0:%.*]] = extractvalue [2 x i16] [[B]], 0
+  ; TLS_ABI: [[B1:%.*]] = extractvalue [2 x i16] [[B]], 1
+  ; TLS_ABI: [[B01:%.*]] = or i16 [[B0]], [[B1]]
+  ; TLS_ABI: call void @__dfsw_custom_without_ret({ i32, i1 } %a, [2 x i7] %b, i16 zeroext [[A01]], i16 zeroext [[B01]])
+  
+  call void @custom_without_ret({i32, i1} %a, [2 x i7] %b)
+  ret void
+}
+
+define void @call_custom_varg({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: @"dfs$call_custom_varg"
+  ; TLS_ABI: [[B:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN:2]]
+  ; TLS_ABI: %labelva = alloca [1 x i16], align [[ALIGN]]
+  ; TLS_ABI: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: [[A0:%.*]] = extractvalue { i16, i16 } [[A]], 0
+  ; TLS_ABI: [[A1:%.*]] = extractvalue { i16, i16 } [[A]], 1
+  ; TLS_ABI: [[A01:%.*]] = or i16 [[A0]], [[A1]]
+  ; TLS_ABI: [[V0:%.*]] = getelementptr inbounds [1 x i16], [1 x i16]* %labelva, i32 0, i32 0
+  ; TLS_ABI: [[B0:%.*]] = extractvalue [2 x i16] [[B]], 0
+  ; TLS_ABI: [[B1:%.*]] = extractvalue [2 x i16] [[B]], 1
+  ; TLS_ABI: [[B01:%.*]] = or i16 [[B0]], [[B1]]
+  ; TLS_ABI: store i16 [[B01]], i16* [[V0]], align 2
+  ; TLS_ABI: [[V:%.*]] = getelementptr inbounds [1 x i16], [1 x i16]* %labelva, i32 0, i32 0
+  ; TLS_ABI: call void ({ i32, i1 }, i16, i16*, ...) @__dfsw_custom_varg({ i32, i1 } %a, i16 zeroext [[A01]], i16* [[V]], [2 x i7] %b)
+
+  call void ({i32, i1}, ...) @custom_varg({i32, i1} %a, [2 x i7] %b)
+  ret void
+}
+
+define {i1, i7} @call_custom_cb({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: define { i1, i7 } @"dfs$call_custom_cb"({ i32, i1 } %a, [2 x i7] %b) {
+  ; TLS_ABI: %labelreturn = alloca i16, align 2
+  ; TLS_ABI: [[B:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN:2]]
+  ; TLS_ABI: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: [[A0:%.*]] = extractvalue { i16, i16 } [[A]], 0
+  ; TLS_ABI: [[A1:%.*]] = extractvalue { i16, i16 } [[A]], 1
+  ; TLS_ABI: [[A01:%.*]] = or i16 [[A0]], [[A1]]
+  ; TLS_ABI: [[B0:%.*]] = extractvalue [2 x i16] [[B]], 0
+  ; TLS_ABI: [[B1:%.*]] = extractvalue [2 x i16] [[B]], 1
+  ; TLS_ABI: [[B01:%.*]] = or i16 [[B0]], [[B1]]  
+  ; TLS_ABI: [[R:%.*]]  = call { i1, i7 } @__dfsw_custom_cb({ i1, i7 } ({ i1, i7 } ({ i32, i1 }, [2 x i7])*, { i32, i1 }, [2 x i7], i16, i16, i16*)* @"dfst0$custom_cb", i8* bitcast ({ i1, i7 } ({ i32, i1 }, [2 x i7])* @"dfs$cb" to i8*), { i32, i1 } %a, [2 x i7] %b, i16 zeroext 0, i16 zeroext [[A01]], i16 zeroext [[B01]], i16* %labelreturn)
+  ; TLS_ABI: [[RE:%.*]] = load i16, i16* %labelreturn, align [[ALIGN]]
+  ; TLS_ABI: [[RS0:%.*]] = insertvalue { i16, i16 } undef, i16 [[RE]], 0
+  ; TLS_ABI: [[RS1:%.*]] = insertvalue { i16, i16 } [[RS0]], i16 [[RE]], 1
+  ; TLS_ABI: store { i16, i16 } [[RS1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+
+  %r = call {i1, i7} @custom_cb({i1, i7} ({i32, i1}, [2 x i7])* @cb, {i32, i1} %a, [2 x i7] %b)
+  ret {i1, i7} %r
+}
+
+define {i1, i7} @custom_cb({i1, i7} ({i32, i1}, [2 x i7])* %cb, {i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: define { i1, i7 } @custom_cb({ i1, i7 } ({ i32, i1 }, [2 x i7])* %cb, { i32, i1 } %a, [2 x i7] %b)
+
+  %r = call {i1, i7} %cb({i32, i1} %a, [2 x i7] %b)
+  ret {i1, i7} %r
+}
+
+define {i1, i7} @cb({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: define { i1, i7 } @"dfs$cb"({ i32, i1 } %a, [2 x i7] %b)
+  ; TLS_ABI: [[BL:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN:2]]
+  ; TLS_ABI: [[AL:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: [[AL1:%.*]] = extractvalue { i16, i16 } [[AL]], 1
+  ; TLS_ABI: [[BL0:%.*]] = extractvalue [2 x i16] [[BL]], 0
+  ; TLS_ABI: [[RL0:%.*]] = insertvalue { i16, i16 } zeroinitializer, i16 [[AL1]], 0
+  ; TLS_ABI: [[RL:%.*]] = insertvalue { i16, i16 } [[RL0]], i16 [[BL0]], 1
+  ; TLS_ABI: store { i16, i16 } [[RL]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+
+  %a1 = extractvalue {i32, i1} %a, 1
+  %b0 = extractvalue [2 x i7] %b, 0
+  %r0 = insertvalue {i1, i7} undef, i1 %a1, 0
+  %r1 = insertvalue {i1, i7} %r0, i7 %b0, 1
+  ret {i1, i7} %r1
+}
+
+define {i1, i7}  ({i32, i1}, [2 x i7])* @ret_custom() {
+  ; TLS_ABI: @"dfs$ret_custom"
+  ; TLS_ABI: store i16 0, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  ; TLS_ABI: ret {{.*}} @"dfsw$custom_with_ret"
+  ret {i1, i7}  ({i32, i1}, [2 x i7])* @custom_with_ret
+}
+
+; TLS_ABI: define linkonce_odr { i1, i7 } @"dfsw$custom_cb"({ i1, i7 } ({ i32, i1 }, [2 x i7])* %0, { i32, i1 } %1, [2 x i7] %2) {
+; TLS_ABI: %labelreturn = alloca i16, align 2
+; TLS_ABI: [[B:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 6) to [2 x i16]*), align [[ALIGN:2]]
+; TLS_ABI: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to { i16, i16 }*), align [[ALIGN]]
+; TLS_ABI: [[CB:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+; TLS_ABI: [[CAST:%.*]] = bitcast { i1, i7 } ({ i32, i1 }, [2 x i7])* %0 to i8*
+; TLS_ABI: [[A0:%.*]] = extractvalue { i16, i16 } [[A]], 0
+; TLS_ABI: [[A1:%.*]] = extractvalue { i16, i16 } [[A]], 1
+; TLS_ABI: [[A01:%.*]] = or i16 [[A0]], [[A1]]
+; TLS_ABI: [[B0:%.*]] = extractvalue [2 x i16] [[B]], 0
+; TLS_ABI: [[B1:%.*]] = extractvalue [2 x i16] [[B]], 1
+; TLS_ABI: [[B01:%.*]] = or i16 [[B0]], [[B1]]  
+; TLS_ABI: [[R:%.*]]  = call { i1, i7 } @__dfsw_custom_cb({ i1, i7 } ({ i1, i7 } ({ i32, i1 }, [2 x i7])*, { i32, i1 }, [2 x i7], i16, i16, i16*)* @"dfst0$custom_cb", i8* [[CAST]], { i32, i1 } %1, [2 x i7] %2, i16 zeroext [[CB]], i16 zeroext [[A01]], i16 zeroext [[B01]], i16* %labelreturn)
+; TLS_ABI: [[RE:%.*]] = load i16, i16* %labelreturn, align [[ALIGN]]
+; TLS_ABI: [[RS0:%.*]] = insertvalue { i16, i16 } undef, i16 [[RE]], 0
+; TLS_ABI: [[RS1:%.*]] = insertvalue { i16, i16 } [[RS0]], i16 [[RE]], 1
+; TLS_ABI: store { i16, i16 } [[RS1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+ 
+
+define {i1, i7} @custom_with_ret({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: define linkonce_odr { i1, i7 } @"dfsw$custom_with_ret"({ i32, i1 } %0, [2 x i7] %1)
+  ; TLS_ABI: %labelreturn = alloca i16, align 2
+  ; TLS_ABI: [[B:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN:2]]
+  ; TLS_ABI: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: [[A0:%.*]] = extractvalue { i16, i16 } [[A]], 0
+  ; TLS_ABI: [[A1:%.*]] = extractvalue { i16, i16 } [[A]], 1
+  ; TLS_ABI: [[A01:%.*]] = or i16 [[A0]], [[A1]]
+  ; TLS_ABI: [[B0:%.*]] = extractvalue [2 x i16] [[B]], 0
+  ; TLS_ABI: [[B1:%.*]] = extractvalue [2 x i16] [[B]], 1
+  ; TLS_ABI: [[B01:%.*]] = or i16 [[B0]], [[B1]]
+  ; TLS_ABI: [[R:%.*]] = call { i1, i7 } @__dfsw_custom_with_ret({ i32, i1 } %0, [2 x i7] %1, i16 zeroext [[A01]], i16 zeroext [[B01]], i16* %labelreturn)
+  ; TLS_ABI: [[RE:%.*]] = load i16, i16* %labelreturn, align 2
+  ; TLS_ABI: [[RS0:%.*]] = insertvalue { i16, i16 } undef, i16 [[RE]], 0
+  ; TLS_ABI: [[RS1:%.*]] = insertvalue { i16, i16 } [[RS0]], i16 [[RE]], 1
+  ; TLS_ABI: store { i16, i16 } [[RS1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: ret { i1, i7 } [[R]]
+  %a1 = extractvalue {i32, i1} %a, 1
+  %b0 = extractvalue [2 x i7] %b, 0
+  %r0 = insertvalue {i1, i7} undef, i1 %a1, 0
+  %r1 = insertvalue {i1, i7} %r0, i7 %b0, 1
+  ret {i1, i7} %r1
+}
+
+define void @custom_without_ret({i32, i1} %a, [2 x i7] %b) {
+  ; TLS_ABI: define linkonce_odr void @"dfsw$custom_without_ret"({ i32, i1 } %0, [2 x i7] %1)
+  ; TLS_ABI: [[B:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN:2]]
+  ; TLS_ABI: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; TLS_ABI: [[A0:%.*]] = extractvalue { i16, i16 } [[A]], 0
+  ; TLS_ABI: [[A1:%.*]] = extractvalue { i16, i16 } [[A]], 1
+  ; TLS_ABI: [[A01:%.*]] = or i16 [[A0]], [[A1]]
+  ; TLS_ABI: [[B0:%.*]] = extractvalue [2 x i16] [[B]], 0
+  ; TLS_ABI: [[B1:%.*]] = extractvalue [2 x i16] [[B]], 1
+  ; TLS_ABI: [[B01:%.*]] = or i16 [[B0]], [[B1]]
+  ; TLS_ABI: call void @__dfsw_custom_without_ret({ i32, i1 } %0, [2 x i7] %1, i16 zeroext [[A01]], i16 zeroext [[B01]])
+  ; TLS_ABI: ret
+  ret void
+}
+
+define void @custom_varg({i32, i1} %a, ...) {
+  ; TLS_ABI: define linkonce_odr void @"dfsw$custom_varg"({ i32, i1 } %0, ...)
+  ; TLS_ABI: call void @__dfsan_vararg_wrapper
+  ; TLS_ABI: unreachable
+  ret void
+}
+
+; TLS_ABI: declare { i1, i7 } @__dfsw_custom_with_ret({ i32, i1 }, [2 x i7], i16, i16, i16*)
+; TLS_ABI: declare void @__dfsw_custom_without_ret({ i32, i1 }, [2 x i7], i16, i16)
+; TLS_ABI: declare void @__dfsw_custom_varg({ i32, i1 }, i16, i16*, ...)
+
+; TLS_ABI: declare { i1, i7 } @__dfsw_custom_cb({ i1, i7 } ({ i1, i7 } ({ i32, i1 }, [2 x i7])*, { i32, i1 }, [2 x i7], i16, i16, i16*)*, i8*, { i32, i1 }, [2 x i7], i16, i16, i16, i16*)
+
+; TLS_ABI: define linkonce_odr { i1, i7 } @"dfst0$custom_cb"({ i1, i7 } ({ i32, i1 }, [2 x i7])* %0, { i32, i1 } %1, [2 x i7] %2, i16 %3, i16 %4, i16* %5) {
+; TLS_ABI: [[A0:%.*]] = insertvalue { i16, i16 } undef, i16 %3, 0
+; TLS_ABI: [[A1:%.*]] = insertvalue { i16, i16 } [[A0]], i16 %3, 1
+; TLS_ABI: [[B0:%.*]] = insertvalue [2 x i16] undef, i16 %4, 0
+; TLS_ABI: [[B1:%.*]] = insertvalue [2 x i16] [[B0]], i16 %4, 1
+; TLS_ABI: store { i16, i16 } [[A1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN:2]]
+; TLS_ABI: store [2 x i16] [[B1]], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to [2 x i16]*), align [[ALIGN]]
+; TLS_ABI: [[R:%.*]] = call { i1, i7 } %0({ i32, i1 } %1, [2 x i7] %2)
+; TLS_ABI: %_dfsret = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+; TLS_ABI: [[RE0:%.*]] = extractvalue { i16, i16 } %_dfsret, 0
+; TLS_ABI: [[RE1:%.*]] = extractvalue { i16, i16 } %_dfsret, 1
+; TLS_ABI: [[RE01:%.*]] = or i16 [[RE0]], [[RE1]]
+; TLS_ABI: store i16 [[RE01]], i16* %5, align [[ALIGN]]
+; TLS_ABI: ret { i1, i7 } [[R]]

llvm/test/Instrumentation/DataFlowSanitizer/array.ll

+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefix=LEGACY
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-event-callbacks=true -S | FileCheck %s --check-prefix=EVENT_CALLBACKS
+; RUN: opt < %s -dfsan -dfsan-args-abi -S | FileCheck %s --check-prefix=ARGS_ABI
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefix=FAST16
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefix=NO_COMBINE_LOAD_PTR
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-combine-pointer-labels-on-store=true -S | FileCheck %s --check-prefix=COMBINE_STORE_PTR
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-debug-nonzero-labels -S | FileCheck %s --check-prefix=DEBUG_NONZERO_LABELS
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define [4 x i8] @pass_array([4 x i8] %a) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$pass_array"
+  ; NO_COMBINE_LOAD_PTR: %1 = load [4 x i16], [4 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [4 x i16]*), align [[ALIGN:2]]
+  ; NO_COMBINE_LOAD_PTR: store [4 x i16] %1, [4 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x i16]*), align [[ALIGN]]
+
+  ; ARGS_ABI: @"dfs$pass_array"
+  ; ARGS_ABI: ret { [4 x i8], i16 }
+  
+  ; DEBUG_NONZERO_LABELS: @"dfs$pass_array"
+  ; DEBUG_NONZERO_LABELS: [[L:%.*]] = load [4 x i16], [4 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [4 x i16]*), align [[ALIGN:2]]
+  ; DEBUG_NONZERO_LABELS: [[L0:%.*]] = extractvalue [4 x i16] [[L]], 0
+  ; DEBUG_NONZERO_LABELS: [[L1:%.*]] = extractvalue [4 x i16] [[L]], 1
+  ; DEBUG_NONZERO_LABELS: [[L01:%.*]] = or i16 [[L0]], [[L1]]
+  ; DEBUG_NONZERO_LABELS: [[L2:%.*]] = extractvalue [4 x i16] [[L]], 2
+  ; DEBUG_NONZERO_LABELS: [[L012:%.*]] = or i16 [[L01]], [[L2]]
+  ; DEBUG_NONZERO_LABELS: [[L3:%.*]] = extractvalue [4 x i16] [[L]], 3
+  ; DEBUG_NONZERO_LABELS: [[L0123:%.*]] = or i16 [[L012]], [[L3]]
+  ; DEBUG_NONZERO_LABELS: {{.*}} = icmp ne i16 [[L0123]], 0
+  ; DEBUG_NONZERO_LABELS: call void @__dfsan_nonzero_label()
+  
+  ret [4 x i8] %a
+}
+
+%ArrayOfStruct = type [4 x {i8*, i32}]
+
+define %ArrayOfStruct @pass_array_of_struct(%ArrayOfStruct %as) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$pass_array_of_struct"
+  ; NO_COMBINE_LOAD_PTR: %1 = load [4 x { i16, i16 }], [4 x { i16, i16 }]* bitcast ([100 x i64]* @__dfsan_arg_tls to [4 x { i16, i16 }]*), align [[ALIGN:2]]
+  ; NO_COMBINE_LOAD_PTR: store [4 x { i16, i16 }] %1, [4 x { i16, i16 }]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x { i16, i16 }]*), align [[ALIGN]]
+
+  ; ARGS_ABI: @"dfs$pass_array_of_struct"
+  ; ARGS_ABI: ret { [4 x { i8*, i32 }], i16 }
+  ret %ArrayOfStruct %as
+}
+
+define [4 x i1]* @alloca_ret_array() {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$alloca_ret_array"
+  ; NO_COMBINE_LOAD_PTR: store i16 0, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  %p = alloca [4 x i1]
+  ret [4 x i1]* %p
+}
+
+define [4 x i1] @load_alloca_array() {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$load_alloca_array"
+  ; NO_COMBINE_LOAD_PTR: [[A:%.*]] = alloca i16, align [[ALIGN:2]]
+  ; NO_COMBINE_LOAD_PTR: [[M:%.*]] = load i16, i16* [[A]], align [[ALIGN]]
+  ; NO_COMBINE_LOAD_PTR: [[S0:%.*]] = insertvalue [4 x i16] undef, i16 [[M]], 0
+  ; NO_COMBINE_LOAD_PTR: [[S1:%.*]] = insertvalue [4 x i16] [[S0]], i16 [[M]], 1
+  ; NO_COMBINE_LOAD_PTR: [[S2:%.*]] = insertvalue [4 x i16] [[S1]], i16 [[M]], 2
+  ; NO_COMBINE_LOAD_PTR: [[S3:%.*]] = insertvalue [4 x i16] [[S2]], i16 [[M]], 3
+  ; NO_COMBINE_LOAD_PTR: store [4 x i16] [[S3]], [4 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x i16]*), align [[ALIGN]]
+  %p = alloca [4 x i1]
+  %a = load [4 x i1], [4 x i1]* %p
+  ret [4 x i1] %a
+}
+
+define [0 x i1] @load_array0([0 x i1]* %p) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$load_array0"
+  ; NO_COMBINE_LOAD_PTR: store [0 x i16] zeroinitializer, [0 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [0 x i16]*), align 2
+  %a = load [0 x i1], [0 x i1]* %p
+  ret [0 x i1] %a
+}
+
+define [1 x i1] @load_array1([1 x i1]* %p) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$load_array1"
+  ; NO_COMBINE_LOAD_PTR: [[L:%.*]] = load i16,
+  ; NO_COMBINE_LOAD_PTR: [[S:%.*]] = insertvalue [1 x i16] undef, i16 [[L]], 0
+  ; NO_COMBINE_LOAD_PTR: store [1 x i16] [[S]], [1 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [1 x i16]*), align 2
+
+  ; EVENT_CALLBACKS: @"dfs$load_array1"
+  ; EVENT_CALLBACKS: [[L:%.*]] = or i16
+  ; EVENT_CALLBACKS: call void @__dfsan_load_callback(i16 [[L]], i8* {{.*}})
+
+  ; FAST16: @"dfs$load_array1"
+  ; FAST16: [[P:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; FAST16: [[L:%.*]] = load i16, i16* {{.*}}, align [[ALIGN]]
+  ; FAST16: [[U:%.*]] = or i16 [[L]], [[P]]
+  ; FAST16: [[S1:%.*]] = insertvalue [1 x i16] undef, i16 [[U]], 0
+  ; FAST16: store [1 x i16] [[S1]], [1 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [1 x i16]*), align [[ALIGN]]
+  
+  ; LEGACY: @"dfs$load_array1"
+  ; LEGACY: [[P:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; LEGACY: [[L:%.*]] = load i16, i16* {{.*}}, align [[ALIGN]]
+  ; LEGACY: [[U:%.*]] = call zeroext i16 @__dfsan_union(i16 zeroext [[L]], i16 zeroext [[P]])
+  ; LEGACY: [[PH:%.*]] = phi i16 [ [[U]], {{.*}} ], [ [[L]], {{.*}} ]
+  ; LEGACY: store i16 [[PH]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+
+  %a = load [1 x i1], [1 x i1]* %p
+  ret [1 x i1] %a
+}
+
+define [2 x i1] @load_array2([2 x i1]* %p) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$load_array2"
+  ; NO_COMBINE_LOAD_PTR: [[P1:%.*]] = getelementptr i16, i16* [[P0:%.*]], i64 1
+  ; NO_COMBINE_LOAD_PTR-DAG: [[E1:%.*]] = load i16, i16* [[P1]], align [[ALIGN:2]]
+  ; NO_COMBINE_LOAD_PTR-DAG: [[E0:%.*]] = load i16, i16* [[P0]], align [[ALIGN]]
+  ; NO_COMBINE_LOAD_PTR: [[U:%.*]] = or i16 [[E0]], [[E1]]
+  ; NO_COMBINE_LOAD_PTR: [[S1:%.*]] = insertvalue [2 x i16] undef, i16 [[U]], 0
+  ; NO_COMBINE_LOAD_PTR: [[S2:%.*]] = insertvalue [2 x i16] [[S1]], i16 [[U]], 1
+  ; NO_COMBINE_LOAD_PTR: store [2 x i16] [[S2]], [2 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [2 x i16]*), align [[ALIGN]]
+
+  ; EVENT_CALLBACKS: @"dfs$load_array2"
+  ; EVENT_CALLBACKS: [[O1:%.*]] = or i16
+  ; EVENT_CALLBACKS: [[O2:%.*]] = or i16 [[O1]]
+  ; EVENT_CALLBACKS: call void @__dfsan_load_callback(i16 [[O2]], i8* {{.*}})
+
+  ; FAST16: @"dfs$load_array2"
+  ; FAST16: [[P:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; FAST16: [[O:%.*]] = or i16
+  ; FAST16: [[U:%.*]] = or i16 [[O]], [[P]]
+  ; FAST16: [[S:%.*]] = insertvalue [2 x i16] undef, i16 [[U]], 0
+  ; FAST16: [[S1:%.*]] = insertvalue [2 x i16] [[S]], i16 [[U]], 1
+  ; FAST16: store [2 x i16] [[S1]], [2 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [2 x i16]*), align [[ALIGN]]
+  %a = load [2 x i1], [2 x i1]* %p
+  ret [2 x i1] %a
+}
+
+define [4 x i1] @load_array4([4 x i1]* %p) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$load_array4"
+  ; NO_COMBINE_LOAD_PTR: [[T:%.*]] = trunc i64 {{.*}} to i16
+  ; NO_COMBINE_LOAD_PTR: [[S1:%.*]] = insertvalue [4 x i16] undef, i16 [[T]], 0
+  ; NO_COMBINE_LOAD_PTR: [[S2:%.*]] = insertvalue [4 x i16] [[S1]], i16 [[T]], 1
+  ; NO_COMBINE_LOAD_PTR: [[S3:%.*]] = insertvalue [4 x i16] [[S2]], i16 [[T]], 2
+  ; NO_COMBINE_LOAD_PTR: [[S4:%.*]] = insertvalue [4 x i16] [[S3]], i16 [[T]], 3
+  ; NO_COMBINE_LOAD_PTR: store [4 x i16] [[S4]], [4 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x i16]*), align 2
+
+  ; EVENT_CALLBACKS: @"dfs$load_array4"
+  ; EVENT_CALLBACKS: [[O0:%.*]] = or i64
+  ; EVENT_CALLBACKS: [[O1:%.*]] = or i64 [[O0]]
+  ; EVENT_CALLBACKS: [[O2:%.*]] = trunc i64 [[O1]] to i16
+  ; EVENT_CALLBACKS: [[O3:%.*]] = or i16 [[O2]]
+  ; EVENT_CALLBACKS: call void @__dfsan_load_callback(i16 [[O3]], i8* {{.*}})
+
+  ; FAST16: @"dfs$load_array4"
+  ; FAST16: [[T:%.*]] = trunc i64 {{.*}} to i16
+  ; FAST16: [[O:%.*]] = or i16 [[T]]
+  ; FAST16: [[S1:%.*]] = insertvalue [4 x i16] undef, i16 [[O]], 0
+  ; FAST16: [[S2:%.*]] = insertvalue [4 x i16] [[S1]], i16 [[O]], 1
+  ; FAST16: [[S3:%.*]] = insertvalue [4 x i16] [[S2]], i16 [[O]], 2
+  ; FAST16: [[S4:%.*]] = insertvalue [4 x i16] [[S3]], i16 [[O]], 3
+  ; FAST16: store [4 x i16] [[S4]], [4 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x i16]*), align 2
+
+  ; LEGACY: @"dfs$load_array4"
+  ; LEGACY: [[P:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; LEGACY: [[PH1:%.*]] = phi i16
+  ; LEGACY: [[U:%.*]] = call zeroext i16 @__dfsan_union(i16 zeroext [[PH1]], i16 zeroext [[P]])
+  ; LEGACY: [[PH:%.*]] = phi i16 [ [[U]], {{.*}} ], [ [[PH1]], {{.*}} ]
+  ; LEGACY: store i16 [[PH]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+
+  %a = load [4 x i1], [4 x i1]* %p
+  ret [4 x i1] %a
+}
+
+define i1 @extract_array([4 x i1] %a) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$extract_array"
+  ; NO_COMBINE_LOAD_PTR: [[AM:%.*]] = load [4 x i16], [4 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [4 x i16]*), align [[ALIGN:2]]
+  ; NO_COMBINE_LOAD_PTR: [[EM:%.*]] = extractvalue [4 x i16] [[AM]], 2
+  ; NO_COMBINE_LOAD_PTR: store i16 [[EM]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  %e2 = extractvalue [4 x i1] %a, 2
+  ret i1 %e2
+}
+
+define [4 x i1] @insert_array([4 x i1] %a, i1 %e2) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$insert_array"
+  ; NO_COMBINE_LOAD_PTR: [[EM:%.*]] = load i16, i16* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 8) to i16*), align [[ALIGN:2]]
+  ; NO_COMBINE_LOAD_PTR: [[AM:%.*]] = load [4 x i16], [4 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [4 x i16]*), align [[ALIGN]]
+  ; NO_COMBINE_LOAD_PTR: [[AM1:%.*]] = insertvalue [4 x i16] [[AM]], i16 [[EM]], 0
+  ; NO_COMBINE_LOAD_PTR: store [4 x i16] [[AM1]], [4 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x i16]*), align [[ALIGN]]
+  %a1 = insertvalue [4 x i1] %a, i1 %e2, 0
+  ret [4 x i1] %a1
+}
+
+define void @store_alloca_array([4 x i1] %a) {
+  ; FAST16: @"dfs$store_alloca_array"
+  ; FAST16: [[S:%.*]] = load [4 x i16], [4 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [4 x i16]*), align [[ALIGN:2]]
+  ; FAST16: [[SP:%.*]] = alloca i16, align [[ALIGN]]
+  ; FAST16: [[E0:%.*]] = extractvalue [4 x i16] [[S]], 0
+  ; FAST16: [[E1:%.*]] = extractvalue [4 x i16] [[S]], 1
+  ; FAST16: [[E01:%.*]] = or i16 [[E0]], [[E1]]
+  ; FAST16: [[E2:%.*]] = extractvalue [4 x i16] [[S]], 2
+  ; FAST16: [[E012:%.*]] = or i16 [[E01]], [[E2]]
+  ; FAST16: [[E3:%.*]] = extractvalue [4 x i16] [[S]], 3
+  ; FAST16: [[E0123:%.*]] = or i16 [[E012]], [[E3]]
+  ; FAST16: store i16 [[E0123]], i16* [[SP]], align [[ALIGN]]
+  %p = alloca [4 x i1]
+  store [4 x i1] %a, [4 x i1]* %p
+  ret void
+}
+
+define void @store_zero_array([4 x i1]* %p) {
+  ; FAST16: @"dfs$store_zero_array"
+  ; FAST16: store i64 0, i64* {{.*}}, align 2
+  store [4 x i1] zeroinitializer, [4 x i1]* %p
+  ret void
+}
+
+define void @store_array2([2 x i1] %a, [2 x i1]* %p) {
+  ; LEGACY: @"dfs$store_array2"
+  ; LEGACY: [[S:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; LEGACY: [[SP0:%.*]] = getelementptr i16, i16* [[SP:%.*]], i32 0
+  ; LEGACY: store i16 [[S]], i16* [[SP0]], align [[ALIGN]]
+  ; LEGACY: [[SP1:%.*]] = getelementptr i16, i16* [[SP]], i32 1
+  ; LEGACY: store i16 [[S]], i16* [[SP1]], align [[ALIGN]]
+  
+  ; EVENT_CALLBACKS: @"dfs$store_array2"
+  ; EVENT_CALLBACKS: [[E12:%.*]] = or i16
+  ; EVENT_CALLBACKS: [[P:%.*]] = bitcast [2 x i1]* %p to i8*
+  ; EVENT_CALLBACKS: call void @__dfsan_store_callback(i16 [[E12]], i8* [[P]])
+  
+  ; FAST16: @"dfs$store_array2"
+  ; FAST16: [[S:%.*]] = load [2 x i16], [2 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [2 x i16]*), align [[ALIGN:2]]
+  ; FAST16: [[E1:%.*]] = extractvalue [2 x i16] [[S]], 0
+  ; FAST16: [[E2:%.*]] = extractvalue [2 x i16] [[S]], 1
+  ; FAST16: [[E12:%.*]] = or i16 [[E1]], [[E2]]
+  ; FAST16: [[SP0:%.*]] = getelementptr i16, i16* [[SP:%.*]], i32 0
+  ; FAST16: store i16 [[E12]], i16* [[SP0]], align [[ALIGN]]
+  ; FAST16: [[SP1:%.*]] = getelementptr i16, i16* [[SP]], i32 1
+  ; FAST16: store i16 [[E12]], i16* [[SP1]], align [[ALIGN]]
+
+  ; COMBINE_STORE_PTR: @"dfs$store_array2"
+  ; COMBINE_STORE_PTR: [[O:%.*]] = or i16
+  ; COMBINE_STORE_PTR: [[U:%.*]] = or i16 [[O]]
+  ; COMBINE_STORE_PTR: [[P1:%.*]] = getelementptr i16, i16* [[P:%.*]], i32 0
+  ; COMBINE_STORE_PTR: store i16 [[U]], i16* [[P1]], align 2
+  ; COMBINE_STORE_PTR: [[P2:%.*]] = getelementptr i16, i16* [[P]], i32 1
+  ; COMBINE_STORE_PTR: store i16 [[U]], i16* [[P2]], align 2
+  
+  store [2 x i1] %a, [2 x i1]* %p
+  ret void
+}
+
+define void @store_array17([17 x i1] %a, [17 x i1]* %p) {
+  ; FAST16: @"dfs$store_array17"
+  ; FAST16: [[AL:%.*]] = load [17 x i16], [17 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [17 x i16]*), align 2
+  ; FAST16: [[AL0:%.*]] = extractvalue [17 x i16] [[AL]], 0
+  ; FAST16: [[AL1:%.*]] = extractvalue [17 x i16] [[AL]], 1
+  ; FAST16: [[AL_0_1:%.*]] = or i16 [[AL0]], [[AL1]]
+  ; FAST16: [[AL2:%.*]] = extractvalue [17 x i16] [[AL]], 2
+  ; FAST16: [[AL_0_2:%.*]] = or i16 [[AL_0_1]], [[AL2]]
+  ; FAST16: [[AL3:%.*]] = extractvalue [17 x i16] [[AL]], 3
+  ; FAST16: [[AL_0_3:%.*]] = or i16 [[AL_0_2]], [[AL3]]
+  ; FAST16: [[AL4:%.*]] = extractvalue [17 x i16] [[AL]], 4
+  ; FAST16: [[AL_0_4:%.*]] = or i16 [[AL_0_3]], [[AL4]]
+  ; FAST16: [[AL5:%.*]] = extractvalue [17 x i16] [[AL]], 5
+  ; FAST16: [[AL_0_5:%.*]] = or i16 %10, [[AL5]]
+  ; FAST16: [[AL6:%.*]] = extractvalue [17 x i16] [[AL]], 6
+  ; FAST16: [[AL_0_6:%.*]] = or i16 %12, [[AL6]]
+  ; FAST16: [[AL7:%.*]] = extractvalue [17 x i16] [[AL]], 7
+  ; FAST16: [[AL_0_7:%.*]] = or i16 %14, [[AL7]]
+  ; FAST16: [[AL8:%.*]] = extractvalue [17 x i16] [[AL]], 8
+  ; FAST16: [[AL_0_8:%.*]] = or i16 %16, [[AL8]]
+  ; FAST16: [[AL9:%.*]] = extractvalue [17 x i16] [[AL]], 9
+  ; FAST16: [[AL_0_9:%.*]] = or i16 %18, [[AL9]]
+  ; FAST16: [[AL10:%.*]] = extractvalue [17 x i16] [[AL]], 10
+  ; FAST16: [[AL_0_10:%.*]] = or i16 %20, [[AL10]]
+  ; FAST16: [[AL11:%.*]] = extractvalue [17 x i16] [[AL]], 11
+  ; FAST16: [[AL_0_11:%.*]] = or i16 %22, [[AL11]]
+  ; FAST16: [[AL12:%.*]] = extractvalue [17 x i16] [[AL]], 12
+  ; FAST16: [[AL_0_12:%.*]] = or i16 %24, [[AL12]]
+  ; FAST16: [[AL13:%.*]] = extractvalue [17 x i16] [[AL]], 13
+  ; FAST16: [[AL_0_13:%.*]] = or i16 %26, [[AL13]]
+  ; FAST16: [[AL14:%.*]] = extractvalue [17 x i16] [[AL]], 14
+  ; FAST16: [[AL_0_14:%.*]] = or i16 %28, [[AL14]]
+  ; FAST16: [[AL15:%.*]] = extractvalue [17 x i16] [[AL]], 15
+  ; FAST16: [[AL_0_15:%.*]] = or i16 %30, [[AL15]]
+  ; FAST16: [[AL16:%.*]] = extractvalue [17 x i16] [[AL]], 16
+  ; FAST16: [[AL_0_16:%.*]] = or i16 {{.*}}, [[AL16]]
+  ; FAST16: [[V1:%.*]] = insertelement <8 x i16> undef, i16 [[AL_0_16]], i32 0
+  ; FAST16: [[V2:%.*]] = insertelement <8 x i16> [[V1]], i16 [[AL_0_16]], i32 1
+  ; FAST16: [[V3:%.*]] = insertelement <8 x i16> [[V2]], i16 [[AL_0_16]], i32 2
+  ; FAST16: [[V4:%.*]] = insertelement <8 x i16> [[V3]], i16 [[AL_0_16]], i32 3
+  ; FAST16: [[V5:%.*]] = insertelement <8 x i16> [[V4]], i16 [[AL_0_16]], i32 4
+  ; FAST16: [[V6:%.*]] = insertelement <8 x i16> [[V5]], i16 [[AL_0_16]], i32 5
+  ; FAST16: [[V7:%.*]] = insertelement <8 x i16> [[V6]], i16 [[AL_0_16]], i32 6
+  ; FAST16: [[V8:%.*]] = insertelement <8 x i16> [[V7]], i16 [[AL_0_16]], i32 7
+  ; FAST16: [[VP:%.*]] = bitcast i16* [[P:%.*]] to <8 x i16>*
+  ; FAST16: [[VP1:%.*]] = getelementptr <8 x i16>, <8 x i16>* [[VP]], i32 0
+  ; FAST16: store <8 x i16> [[V8]], <8 x i16>* [[VP1]], align [[ALIGN:2]]
+  ; FAST16: [[VP2:%.*]] = getelementptr <8 x i16>, <8 x i16>* [[VP]], i32 1
+  ; FAST16: store <8 x i16> [[V8]], <8 x i16>* [[VP2]], align [[ALIGN]]
+  ; FAST16: [[P3:%.*]] = getelementptr i16, i16* [[P]], i32 16
+  ; FAST16: store i16 [[AL_0_16]], i16* [[P3]], align [[ALIGN]]
+  store [17 x i1] %a, [17 x i1]* %p
+  ret void
+}
+
+define [2 x i32] @const_array() {
+  ; FAST16: @"dfs$const_array"
+  ; FAST16: store [2 x i16] zeroinitializer, [2 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [2 x i16]*), align 2
+  ret [2 x i32] [ i32 42, i32 11 ]
+}
+
+define [4 x i8] @call_array([4 x i8] %a) {
+  ; FAST16: @"dfs$call_array"
+  ; FAST16: [[A:%.*]] = load [4 x i16], [4 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [4 x i16]*), align [[ALIGN:2]]
+  ; FAST16: store [4 x i16] [[A]], [4 x i16]* bitcast ([100 x i64]* @__dfsan_arg_tls to [4 x i16]*), align [[ALIGN]]
+  ; FAST16: %_dfsret = load [4 x i16], [4 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x i16]*), align [[ALIGN]]
+  ; FAST16: store [4 x i16] %_dfsret, [4 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x i16]*), align [[ALIGN]]
+
+  %r = call [4 x i8] @pass_array([4 x i8] %a)
+  ret [4 x i8] %r
+}
+
+%LargeArr = type [1000 x i8]
+
+define i8 @fun_with_large_args(i1 %i, %LargeArr %a) {
+  ; FAST16: @"dfs$fun_with_large_args"
+  ; FAST16: store i16 0, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  %r = extractvalue %LargeArr %a, 0
+  ret i8 %r
+}
+
+define %LargeArr @fun_with_large_ret() {
+  ; FAST16: @"dfs$fun_with_large_ret"
+  ; FAST16-NEXT: ret  [1000 x i8] zeroinitializer
+  ret %LargeArr zeroinitializer
+}
+
+define i8 @call_fun_with_large_ret() {
+  ; FAST16: @"dfs$call_fun_with_large_ret"
+  ; FAST16: store i16 0, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  %r = call %LargeArr @fun_with_large_ret()
+  %e = extractvalue %LargeArr %r, 0
+  ret i8 %e
+}
+
+define i8 @call_fun_with_large_args(i1 %i, %LargeArr %a) {
+  ; FAST16: @"dfs$call_fun_with_large_args"
+  ; FAST16: [[I:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; FAST16: store i16 [[I]], i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; FAST16: %r = call i8 @"dfs$fun_with_large_args"(i1 %i, [1000 x i8] %a)
+  
+  %r = call i8 @fun_with_large_args(i1 %i, %LargeArr %a)
+  ret i8 %r
+}

llvm/test/Instrumentation/DataFlowSanitizer/phi.ll

-; RUN: opt < %s -dfsan -S | FileCheck %s
+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefix=LEGACY
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefix=FAST16
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define {i32, i32} @test({i32, i32} %a, i1 %c) {
-  ; CHECK: [[E0:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
-  ; CHECK: [[E3:%.*]] = phi i16 [ [[E0]], %T ], [ [[E0]], %F ]
-  ; CHECK: store i16 [[E3]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  ; LEGACY: [[AL:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; LEGACY: [[PL:%.*]] = phi i16 [ [[AL]], %T ], [ [[AL]], %F ]
+  ; LEGACY: store i16 [[PL]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+
+  ; FAST16: [[AL:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN:2]]
+  ; FAST16: [[AL0:%.*]] = insertvalue { i16, i16 } [[AL]], i16 0, 0
+  ; FAST16: [[AL1:%.*]] = insertvalue { i16, i16 } [[AL]], i16 0, 1
+  ; FAST16: [[PL:%.*]] = phi { i16, i16 } [ [[AL0]], %T ], [ [[AL1]], %F ]
+  ; FAST16: store { i16, i16 } [[PL]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
 
 entry:
   br i1 %c, label %T, label %F
   
 T:
   %at = insertvalue {i32, i32} %a, i32 1, 0
   br label %done
   
 F:
   %af = insertvalue {i32, i32} %a, i32 1, 1
   br label %done
   
 done:
   %b = phi {i32, i32} [%at, %T], [%af, %F]
   ret {i32, i32} %b  
 }

llvm/test/Instrumentation/DataFlowSanitizer/store.ll

define void @store64(i64 %v, i64* %p) {
   store i64 %v, i64* %p
   ret void
 }
 
 define void @store_zero(i32* %p) {
   ;  NO_COMBINE_PTR_LABEL: store i64 0, i64* {{.*}}, align 2
   store i32 0, i32* %p
   ret void
-}
+}
 No newline at end of file

llvm/test/Instrumentation/DataFlowSanitizer/struct.ll

+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefix=LEGACY
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-event-callbacks=true -S | FileCheck %s --check-prefix=EVENT_CALLBACKS
+; RUN: opt < %s -dfsan -dfsan-args-abi -S | FileCheck %s --check-prefix=ARGS_ABI
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefix=FAST16
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-combine-pointer-labels-on-load=false -S | FileCheck %s --check-prefix=NO_COMBINE_LOAD_PTR
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-combine-pointer-labels-on-store=true -S | FileCheck %s --check-prefix=COMBINE_STORE_PTR
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-track-select-control-flow=false -S | FileCheck %s --check-prefix=NO_SELECT_CONTROL
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -dfsan-debug-nonzero-labels -S | FileCheck %s --check-prefix=DEBUG_NONZERO_LABELS
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define {i8*, i32} @pass_struct({i8*, i32} %s) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$pass_struct"
+  ; NO_COMBINE_LOAD_PTR: [[L:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN:2]]
+  ; NO_COMBINE_LOAD_PTR: store { i16, i16 } [[L]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+
+  ; ARGS_ABI: @"dfs$pass_struct"
+  ; ARGS_ABI: ret { { i8*, i32 }, i16 }
+  
+  ; DEBUG_NONZERO_LABELS: @"dfs$pass_struct"
+  ; DEBUG_NONZERO_LABELS: [[L:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN:2]]
+  ; DEBUG_NONZERO_LABELS: [[L0:%.*]] = extractvalue { i16, i16 } [[L]], 0
+  ; DEBUG_NONZERO_LABELS: [[L1:%.*]] = extractvalue { i16, i16 } [[L]], 1
+  ; DEBUG_NONZERO_LABELS: [[L01:%.*]] = or i16 [[L0]], [[L1]]
+  ; DEBUG_NONZERO_LABELS: {{.*}} = icmp ne i16 [[L01]], 0
+  ; DEBUG_NONZERO_LABELS: call void @__dfsan_nonzero_label()
+  ; DEBUG_NONZERO_LABELS: store { i16, i16 } [[L]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+  
+  ret {i8*, i32} %s
+}
+
+%StructOfAggr = type {i8*, [4 x i2], <4 x i3>, {i1, i1}}
+
+define %StructOfAggr @pass_struct_of_aggregate(%StructOfAggr %s) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$pass_struct_of_aggregate"
+  ; NO_COMBINE_LOAD_PTR: %1 = load { i16, [4 x i16], i16, { i16, i16 } }, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN:2]]
+  ; NO_COMBINE_LOAD_PTR: store { i16, [4 x i16], i16, { i16, i16 } } %1, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN]]
+
+  ; ARGS_ABI: @"dfs$pass_struct_of_aggregate"
+  ; ARGS_ABI: ret { %StructOfAggr, i16 }
+  ret %StructOfAggr %s
+}
+
+define {} @load_empty_struct({}* %p) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$load_empty_struct"
+  ; NO_COMBINE_LOAD_PTR: store {} zeroinitializer, {}* bitcast ([100 x i64]* @__dfsan_retval_tls to {}*), align 2
+
+  %a = load {}, {}* %p
+  ret {} %a
+}
+
+@Y = constant {i1, i32} {i1 1, i32 1}
+
+define {i1, i32} @load_global_struct() {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$load_global_struct"
+  ; NO_COMBINE_LOAD_PTR: store { i16, i16 } zeroinitializer, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align 2
+
+  %a = load {i1, i32}, {i1, i32}* @Y
+  ret {i1, i32} %a
+}
+
+define {i1, i32} @select_struct(i1 %c, {i1, i32} %a, {i1, i32} %b) {
+  ; NO_SELECT_CONTROL: @"dfs$select_struct"
+  ; NO_SELECT_CONTROL: [[B:%.*]] = load { i16, i16 }, { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 6) to { i16, i16 }*), align [[ALIGN:2]]
+  ; NO_SELECT_CONTROL: [[A:%.*]] = load { i16, i16 }, { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to { i16, i16 }*), align [[ALIGN]]
+  ; NO_SELECT_CONTROL: [[C:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; NO_SELECT_CONTROL: [[S:%.*]] = select i1 %c, { i16, i16 } [[A]], { i16, i16 } [[B]]
+  ; NO_SELECT_CONTROL: store { i16, i16 } [[S]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+
+  ; FAST16: @"dfs$select_struct"
+  ; FAST16: [[B_S:%.*]] = load { i16, i16 }, { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 6) to { i16, i16 }*), align [[ALIGN:2]]
+  ; FAST16: [[A_S:%.*]] = load { i16, i16 }, { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to { i16, i16 }*), align [[ALIGN]]
+  ; FAST16: [[C_S:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; FAST16: [[S_S:%.*]] = select i1 %c, { i16, i16 } [[A_S]], { i16, i16 } [[B_S]]
+  ; FAST16: [[S0_S:%.*]] = extractvalue { i16, i16 } [[S_S]], 0
+  ; FAST16: [[S1_S:%.*]] = extractvalue { i16, i16 } [[S_S]], 1
+  ; FAST16: [[S01_S:%.*]] = or i16 [[S0_S]], [[S1_S]]
+  ; FAST16: [[CS_S:%.*]] = or i16 [[C_S]], [[S01_S]]
+  ; FAST16: [[S1:%.*]] = insertvalue { i16, i16 } undef, i16 [[CS_S]], 0
+  ; FAST16: [[S2:%.*]] = insertvalue { i16, i16 } [[S1]], i16 [[CS_S]], 1
+  ; FAST16: store { i16, i16 } [[S2]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+
+  ; LEGACY: @"dfs$select_struct"
+  ; LEGACY: [[U:%.*]] = call zeroext i16 @__dfsan_union
+  ; LEGACY: [[P:%.*]] = phi i16 [ [[U]],
+  ; LEGACY: store i16 [[P]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+
+  %s = select i1 %c, {i1, i32} %a, {i1, i32} %b
+  ret {i1, i32} %s
+}
+
+define { i32, i32 } @asm_struct(i32 %0, i32 %1) {
+  ; FAST16: @"dfs$asm_struct"
+  ; FAST16: [[E1:%.*]] = load i16, i16* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i16*), align [[ALIGN:2]]
+  ; FAST16: [[E0:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; FAST16: [[E01:%.*]] = or i16 [[E0]], [[E1]]
+  ; FAST16: [[S0:%.*]] = insertvalue { i16, i16 } undef, i16 [[E01]], 0
+  ; FAST16: [[S1:%.*]] = insertvalue { i16, i16 } [[S0]], i16 [[E01]], 1
+  ; FAST16: store { i16, i16 } [[S1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+
+  ; LEGACY: @"dfs$asm_struct"
+  ; LEGACY: [[E1:%.*]] = load i16, i16* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i16*), align [[ALIGN:2]]
+  ; LEGACY: [[E0:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; LEGACY: [[E01:%.*]] = call zeroext i16 @__dfsan_union(i16 zeroext [[E0]], i16 zeroext [[E1]])
+  ; LEGACY: [[P:%.*]] = phi i16 [ [[E01]], {{.*}} ], [ [[E0]], {{.*}} ]
+  ; LEGACY: store i16 [[P]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  
+entry:
+  %a = call { i32, i32 } asm "", "=r,=r,r,r,~{dirflag},~{fpsr},~{flags}"(i32 %0, i32 %1)
+  ret { i32, i32 } %a
+}
+
+define {i32, i32} @const_struct() {
+  ; FAST16: @"dfs$const_struct"
+  ; FAST16: store { i16, i16 } zeroinitializer, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align 2
+  
+  ; LEGACY: @"dfs$const_struct"
+  ; LEGACY: store i16 0, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  ret {i32, i32} { i32 42, i32 11 }
+}
+
+define i1 @extract_struct({i1, i5} %s) {
+  ; FAST16: @"dfs$extract_struct"
+  ; FAST16: [[SM:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN:2]]
+  ; FAST16: [[EM:%.*]] = extractvalue { i16, i16 } [[SM]], 0
+  ; FAST16: store i16 [[EM]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  
+  ; LEGACY: @"dfs$extract_struct"
+  ; LEGACY: [[SM:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; LEGACY: store i16 [[SM]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  %e2 = extractvalue {i1, i5} %s, 0
+  ret i1 %e2
+}
+
+define {i1, i5} @insert_struct({i1, i5} %s, i5 %e1) {
+  ; FAST16: @"dfs$insert_struct"
+  ; FAST16: [[EM:%.*]] = load i16, i16* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 4) to i16*), align [[ALIGN:2]]
+  ; FAST16: [[SM:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; FAST16: [[SM1:%.*]] = insertvalue { i16, i16 } [[SM]], i16 [[EM]], 1
+  ; FAST16: store { i16, i16 } [[SM1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+  
+  ; LEGACY: @"dfs$insert_struct"
+  ; LEGACY: [[EM:%.*]] = load i16, i16* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i16*), align [[ALIGN:2]]
+  ; LEGACY: [[SM:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; LEGACY: [[U:%.*]] = call zeroext i16 @__dfsan_union(i16 zeroext [[SM]], i16 zeroext [[EM]])
+  ; LEGACY: [[P:%.*]] = phi i16 [ [[U]], {{.*}} ], [ [[SM]], {{.*}} ]
+  ; LEGACY: store i16 [[P]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  %s1 = insertvalue {i1, i5} %s, i5 %e1, 1
+  ret {i1, i5} %s1
+}
+
+define {i1, i1} @load_struct({i1, i1}* %p) {
+  ; NO_COMBINE_LOAD_PTR: @"dfs$load_struct"
+  ; NO_COMBINE_LOAD_PTR: [[OL:%.*]] = or i16
+  ; NO_COMBINE_LOAD_PTR: [[S0:%.*]] = insertvalue { i16, i16 } undef, i16 [[OL]], 0
+  ; NO_COMBINE_LOAD_PTR: [[S1:%.*]] = insertvalue { i16, i16 } [[S0]], i16 [[OL]], 1
+  ; NO_COMBINE_LOAD_PTR: store { i16, i16 } [[S1]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align 2
+  
+  ; EVENT_CALLBACKS: @"dfs$load_struct"
+  ; EVENT_CALLBACKS: [[OL0:%.*]] = or i16
+  ; EVENT_CALLBACKS: [[OL1:%.*]] = or i16 [[OL0]],
+  ; EVENT_CALLBACKS: [[S0:%.*]] = insertvalue { i16, i16 } undef, i16 [[OL1]], 0
+  ; EVENT_CALLBACKS: call void @__dfsan_load_callback(i16 [[OL1]]
+  
+  %s = load {i1, i1}, {i1, i1}* %p
+  ret {i1, i1} %s
+}
+
+define void @store_struct({i1, i1}* %p, {i1, i1} %s) {
+  ; FAST16: @"dfs$store_struct"
+  ; FAST16: [[S:%.*]] = load { i16, i16 }, { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to { i16, i16 }*), align [[ALIGN:2]]
+  ; FAST16: [[E0:%.*]] = extractvalue { i16, i16 } [[S]], 0
+  ; FAST16: [[E1:%.*]] = extractvalue { i16, i16 } [[S]], 1
+  ; FAST16: [[E:%.*]] = or i16 [[E0]], [[E1]]
+  ; FAST16: [[P0:%.*]] = getelementptr i16, i16* [[P:%.*]], i32 0
+  ; FAST16: store i16 [[E]], i16* [[P0]], align [[ALIGN]]
+  ; FAST16: [[P1:%.*]] = getelementptr i16, i16* [[P]], i32 1
+  ; FAST16: store i16 [[E]], i16* [[P1]], align [[ALIGN]]
+  
+  ; EVENT_CALLBACKS: @"dfs$store_struct"
+  ; EVENT_CALLBACKS: [[OL:%.*]] = or i16
+  ; EVENT_CALLBACKS: call void @__dfsan_store_callback(i16 [[OL]]
+  
+  ; COMBINE_STORE_PTR: @"dfs$store_struct"
+  ; COMBINE_STORE_PTR: [[PL:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; COMBINE_STORE_PTR: [[SL:%.*]] = load { i16, i16 }, { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to { i16, i16 }*), align [[ALIGN]]
+  ; COMBINE_STORE_PTR: [[SL0:%.*]] = extractvalue { i16, i16 } [[SL]], 0
+  ; COMBINE_STORE_PTR: [[SL1:%.*]] = extractvalue { i16, i16 } [[SL]], 1
+  ; COMBINE_STORE_PTR: [[SL01:%.*]] = or i16 [[SL0]], [[SL1]]
+  ; COMBINE_STORE_PTR: [[E:%.*]] = or i16 [[SL01]], [[PL]]
+  ; COMBINE_STORE_PTR: [[P0:%.*]] = getelementptr i16, i16* [[P:%.*]], i32 0
+  ; COMBINE_STORE_PTR: store i16 [[E]], i16* [[P0]], align [[ALIGN]]
+  ; COMBINE_STORE_PTR: [[P1:%.*]] = getelementptr i16, i16* [[P]], i32 1
+  ; COMBINE_STORE_PTR: store i16 [[E]], i16* [[P1]], align [[ALIGN]]
+  
+  store {i1, i1} %s, {i1, i1}* %p
+  ret void
+}
+
+define i2 @extract_struct_of_aggregate11(%StructOfAggr %s) {
+  ; FAST16: @"dfs$extract_struct_of_aggregate11"
+  ; FAST16: [[E:%.*]] = load { i16, [4 x i16], i16, { i16, i16 } }, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN:2]]
+  ; FAST16: [[E11:%.*]] = extractvalue { i16, [4 x i16], i16, { i16, i16 } } [[E]], 1, 1
+  ; FAST16: store i16 [[E11]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+
+  %e11 = extractvalue %StructOfAggr %s, 1, 1
+  ret i2 %e11
+}
+
+define [4 x i2] @extract_struct_of_aggregate1(%StructOfAggr %s) {
+  ; FAST16: @"dfs$extract_struct_of_aggregate1"
+  ; FAST16: [[E:%.*]] = load { i16, [4 x i16], i16, { i16, i16 } }, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN:2]]
+  ; FAST16: [[E1:%.*]] = extractvalue { i16, [4 x i16], i16, { i16, i16 } } [[E]], 1
+  ; FAST16: store [4 x i16] [[E1]], [4 x i16]* bitcast ([100 x i64]* @__dfsan_retval_tls to [4 x i16]*), align [[ALIGN]]
+  %e1 = extractvalue %StructOfAggr %s, 1
+  ret [4 x i2] %e1
+}
+
+define <4 x i3> @extract_struct_of_aggregate2(%StructOfAggr %s) {
+  ; FAST16: @"dfs$extract_struct_of_aggregate2"
+  ; FAST16: [[E:%.*]] = load { i16, [4 x i16], i16, { i16, i16 } }, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN:2]]
+  ; FAST16: [[E2:%.*]] = extractvalue { i16, [4 x i16], i16, { i16, i16 } } [[E]], 2
+  ; FAST16: store i16 [[E2]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  %e2 = extractvalue %StructOfAggr %s, 2
+  ret <4 x i3> %e2
+}
+
+define { i1, i1 } @extract_struct_of_aggregate3(%StructOfAggr %s) {
+  ; FAST16: @"dfs$extract_struct_of_aggregate3"
+  ; FAST16: [[E:%.*]] = load { i16, [4 x i16], i16, { i16, i16 } }, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN:2]]
+  ; FAST16: [[E3:%.*]] = extractvalue { i16, [4 x i16], i16, { i16, i16 } } [[E]], 3
+  ; FAST16: store { i16, i16 } [[E3]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+  %e3 = extractvalue %StructOfAggr %s, 3
+  ret { i1, i1 } %e3
+}
+
+define i1 @extract_struct_of_aggregate31(%StructOfAggr %s) {
+  ; FAST16: @"dfs$extract_struct_of_aggregate31"
+  ; FAST16: [[E:%.*]] = load { i16, [4 x i16], i16, { i16, i16 } }, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN:2]]
+  ; FAST16: [[E31:%.*]] = extractvalue { i16, [4 x i16], i16, { i16, i16 } } [[E]], 3, 1
+  ; FAST16: store i16 [[E31]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  %e31 = extractvalue %StructOfAggr %s, 3, 1
+  ret i1 %e31
+}
+
+define %StructOfAggr @insert_struct_of_aggregate11(%StructOfAggr %s, i2 %e11) {
+  ; FAST16: @"dfs$insert_struct_of_aggregate11"
+  ; FAST16: [[E11:%.*]]  = load i16, i16* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 16) to i16*), align [[ALIGN:2]]
+  ; FAST16: [[S:%.*]]  = load { i16, [4 x i16], i16, { i16, i16 } }, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN]]
+  ; FAST16: [[S1:%.*]]  = insertvalue { i16, [4 x i16], i16, { i16, i16 } } [[S]], i16 [[E11]], 1, 1
+  ; FAST16: store { i16, [4 x i16], i16, { i16, i16 } } [[S1]], { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN]]
+
+  %s1 = insertvalue %StructOfAggr %s, i2 %e11, 1, 1
+  ret %StructOfAggr %s1
+}
+
+define {i8*, i32} @call_struct({i8*, i32} %s) {
+  ; FAST16: @"dfs$call_struct"
+  ; FAST16: [[S:%.*]] = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN:2]]
+  ; FAST16: store { i16, i16 } [[S]], { i16, i16 }* bitcast ([100 x i64]* @__dfsan_arg_tls to { i16, i16 }*), align [[ALIGN]]
+  ; FAST16: %_dfsret = load { i16, i16 }, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+  ; FAST16: store { i16, i16 } %_dfsret, { i16, i16 }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, i16 }*), align [[ALIGN]]
+
+  %r = call {i8*, i32} @pass_struct({i8*, i32} %s)
+  ret {i8*, i32} %r
+}
+
+declare %StructOfAggr @fun_with_many_aggr_args(<2 x i7> %v, [2 x i5] %a, {i3, i3} %s)
+
+define %StructOfAggr @call_many_aggr_args(<2 x i7> %v, [2 x i5] %a, {i3, i3} %s) {
+  ; FAST16: @"dfs$call_many_aggr_args"
+  ; FAST16: [[S:%.*]] = load { i16, i16 }, { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 6) to { i16, i16 }*), align [[ALIGN:2]]
+  ; FAST16: [[A:%.*]] = load [2 x i16], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to [2 x i16]*), align [[ALIGN]]
+  ; FAST16: [[V:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; FAST16: store i16 [[V]], i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; FAST16: store [2 x i16] [[A]], [2 x i16]* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to [2 x i16]*), align [[ALIGN]]
+  ; FAST16: store { i16, i16 } [[S]], { i16, i16 }* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 6) to { i16, i16 }*), align [[ALIGN]]
+  ; FAST16: %_dfsret = load { i16, [4 x i16], i16, { i16, i16 } }, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN]]
+  ; FAST16: store { i16, [4 x i16], i16, { i16, i16 } } %_dfsret, { i16, [4 x i16], i16, { i16, i16 } }* bitcast ([100 x i64]* @__dfsan_retval_tls to { i16, [4 x i16], i16, { i16, i16 } }*), align [[ALIGN]]
+
+  %r = call %StructOfAggr @fun_with_many_aggr_args(<2 x i7> %v, [2 x i5] %a, {i3, i3} %s)
+  ret %StructOfAggr %r
+}
 No newline at end of file

llvm/test/Instrumentation/DataFlowSanitizer/vector.ll

+; RUN: opt < %s -dfsan -S | FileCheck %s --check-prefix=LEGACY
+; RUN: opt < %s -dfsan -dfsan-args-abi -S | FileCheck %s --check-prefix=ARGS_ABI
+; RUN: opt < %s -dfsan -dfsan-fast-16-labels=true -S | FileCheck %s --check-prefix=FAST16
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define <4 x i4> @pass_vector(<4 x i4> %v) {
+  ; ARGS_ABI: @"dfs$pass_vector"
+  ; ARGS_ABI: ret { <4 x i4>, i16 }
+  
+  ; FAST16: @"dfs$pass_vector"
+  ; FAST16: {{.*}} = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; FAST16: store i16 %1, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  ret <4 x i4> %v
+}
+
+define void @load_update_store_vector(<4 x i4>* %p) {
+  ; FAST16: @"dfs$load_update_store_vector"
+  ; FAST16: {{.*}} = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align 2
+
+  %v = load <4 x i4>, <4 x i4>* %p
+  %e2 = extractelement <4 x i4> %v, i32 2
+  %v1 = insertelement <4 x i4> %v, i4 %e2, i32 0
+  store <4 x i4> %v1, <4 x i4>* %p
+  ret void
+}
+
+define <4 x i1> @icmp_vector(<4 x i8> %a, <4 x i8> %b) {
+  ; LEGACY: @"dfs$icmp_vector"
+  ; LEGACY: [[B:%.*]] = load i16, i16* inttoptr (i64 add (i64 ptrtoint ([100 x i64]* @__dfsan_arg_tls to i64), i64 2) to i16*), align [[ALIGN:2]]
+  ; LEGACY: [[A:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; LEGACY: [[U:%.*]] = call zeroext i16 @__dfsan_union(i16 zeroext [[A]], i16 zeroext [[B]])
+  ; LEGACY: [[PH:%.*]] = phi i16 [ [[U]], {{.*}} ], [ [[A]], {{.*}} ]
+  ; LEGACY: store i16 [[PH]], i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  
+  %r = icmp eq <4 x i8> %a, %b
+  ret <4 x i1> %r
+}
+
+define <2 x i32> @const_vector() {
+  ; LEGACY: @"dfs$const_vector"
+  ; LEGACY: store i16 0, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  
+  ; FAST16: @"dfs$const_vector"
+  ; FAST16: store i16 0, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align 2
+  ret <2 x i32> < i32 42, i32 11 >
+}
+
+define <4 x i4> @call_vector(<4 x i4> %v) {
+  ; LEGACY: @"dfs$call_vector"
+  ; LEGACY: [[V:%.*]] = load i16, i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN:2]]
+  ; LEGACY: store i16 [[V]], i16* bitcast ([100 x i64]* @__dfsan_arg_tls to i16*), align [[ALIGN]]
+  ; LEGACY: %_dfsret = load i16, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+  ; LEGACY: store i16 %_dfsret, i16* bitcast ([100 x i64]* @__dfsan_retval_tls to i16*), align [[ALIGN]]
+
+  %r = call <4 x i4> @pass_vector(<4 x i4> %v)
+  ret <4 x i4> %r
+}
+
+