[DFSan] Add efficient fast16labels instrumentation mode.

[morehouse]

Summary

Adds the -fast-16-labels flag, which enables efficient instrumentation
for DFSan when the user needs <=16 labels. The instrumentation
eliminates most branches and most calls to dfsan_union or
dfsan_union_load.

We also add a call into the runtime during preinit to enable
fast16labels mode there, since we may still call __dfsan_union[_load] in
rare cases.

[kcc]

In what cases do we still call __dfsan_union?

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

183

please add the documentation.
Probably into this change, so that we review it together.

[kcc]

compiler-rt/lib/dfsan/dfsan.cpp

173

I wonder if we can get away w/o these flags.
Just assume that fast16mode works only when the fast16mode instrumentation is enabled.
(remember, fast16mode is currently experimental, we are free to change its behavior)

[vitalybuka]

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

1079

what we are going to do in __dfsan_fast16labels_preinit?

1339

have you considered a loop?

[morehouse]

• Add documentation.
• Remove fast16labels runtime flag.

[morehouse]

In D84371#2168367, @kcc wrote:

In what cases do we still call __dfsan_union?

We still call __dfsan_union_load when we load sizes greater than 2 bytes but not divisible by 4. I actually am not sure what instruction sequence would hit this case, but it was already in DFSan so I left it there.

__dfsan_union_load then calls __dfsan_union.

compiler-rt/lib/dfsan/dfsan.cpp

173

SGTM. I only know of @Dor1s using it at the moment.

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

1079

This global is simply a pointer to dfsan_use_fast16labels() so that it runs during preinit. dfsan_use_fast16labels just enables fast16labels mode in the runtime.

1339

I think we should avoid loop overhead at least for the common case here (loading 8-16 bytes of shadow).

I think it's also possible to hit this case with vector instructions, but vanilla DFSan currently doesn't use a loop in that case either. If we want to use a loop for vector loads, I think we should do it in a different patch.

[morehouse]

• Fix libfuzzer dataflow tests.

[kcc]

Yep, cool.
LGTM from me, but please get another pair if eyes (Vitaly?)

clang/docs/DataFlowSanitizer.rst

182 ↗

(On Diff #280304)

maybe -dataflow-fast-16-labels?

201 ↗

(On Diff #280304)

perhaps unconfuse the test a bit by using other constants

i = 100;
j = 200;
k = 300;

[vitalybuka]

compiler-rt/lib/dfsan/dfsan.cpp

183

isn't better just create new set of callbacks?
e.g __dfsan_fast16_union
and then we don't need any flags or preinit array initialization

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

810

DFSanUseFast16LabelsFnTy and DFSanUseFast16LabelsFn is used one in one place
I'd make them local variables. in if (ClFast16Labels) of runOnModule

[morehouse]

• Rename flag
• Clarify doc example
• Use temporary variables.

compiler-rt/lib/dfsan/dfsan.cpp

183

Should work for __dfsan_union and __dfsan_union_load, but what about all the other API functions the user can call directly? We wouldn't be able to warn in those cases.

[vitalybuka]

LGTM either way

compiler-rt/lib/dfsan/dfsan.cpp

183

Should work for __dfsan_union and __dfsan_union_load, but what about all the other API functions the user can call directly? We wouldn't be able to warn in those cases.

Those calls are no-op, it would be nice, but it's not worth it.
You can avoid state variable fast16labels, and avoid preinit_array which time to time conflicts with other libs. Which looks more useful then warning.

[morehouse]

• Remove preinit stuff and API warnings; use custom union-load callback.

compiler-rt/lib/dfsan/dfsan.cpp

183

Thanks. Turns out we only need __dfsan_union_load_fast16labels, which makes things simpler.

[morehouse]

compiler-rt/lib/dfsan/dfsan.cpp

183

Turns out we might need a fast16 flag and preinit_array after all, since many of the custom wrappers call dfsan_union...

@vitalybuka Any ideas to avoid this?

[morehouse]

compiler-rt/lib/dfsan/dfsan.cpp

183

One idea: In fast16 mode, we never call dfsan_create_label, so probably we can just check whether any labels have been created, and if not compute the OR.

compiler-rt/lib/dfsan/dfsan.cpp

 // called automatically by the compiler (specifically the instrumentation pass
 // in llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp).
 //
 // The public interface is defined in include/sanitizer/dfsan_interface.h whose
 // functions are prefixed dfsan_ while the compiler interface functions are
 // prefixed __dfsan_.
 //===----------------------------------------------------------------------===//
 
+#include "dfsan/dfsan.h"
+
 #include "sanitizer_common/sanitizer_atomic.h"
 #include "sanitizer_common/sanitizer_common.h"
 #include "sanitizer_common/sanitizer_file.h"
-#include "sanitizer_common/sanitizer_flags.h"
 #include "sanitizer_common/sanitizer_flag_parser.h"
+#include "sanitizer_common/sanitizer_flags.h"
+#include "sanitizer_common/sanitizer_internal_defs.h"
 #include "sanitizer_common/sanitizer_libc.h"
 
-#include "dfsan/dfsan.h"
-
 using namespace __dfsan;
 
 typedef atomic_uint16_t atomic_dfsan_label;
 static const dfsan_label kInitializingLabel = -1;
 
 static const uptr kNumLabels = 1 << (sizeof(dfsan_label) * 8);
 
 static atomic_dfsan_label __dfsan_last_label;
 static dfsan_label_info __dfsan_label_info[kNumLabels];
 
+// True if dfsan_use_fast16labels() was called.
+static bool use_fast16labels = false;
+static bool flags_initialized = false;
+
 Flags __dfsan::flags_data;
 
 SANITIZER_INTERFACE_ATTRIBUTE THREADLOCAL dfsan_label __dfsan_retval_tls;
 SANITIZER_INTERFACE_ATTRIBUTE THREADLOCAL dfsan_label __dfsan_arg_tls[64];
 
 SANITIZER_INTERFACE_ATTRIBUTE uptr __dfsan_shadow_ptr_mask;
 
 // On Linux/x86_64, memory is laid out as follows:

 }
 
 static void ReportUnsupportedFast16(const char *func) {
   Report("FATAL: DataFlowSanitizer: %s is unsupported in fast16labels mode\n",
          func);
   Die();
 }
 
+// Configures the DFSan runtime to use fast16labels mode.
+extern "C" SANITIZER_INTERFACE_ATTRIBUTE void dfsan_use_fast16labels() {
+  use_fast16labels = true;
+  if (flags_initialized)
+    flags().fast16labels = true;
+}
+
 // Resolves the union of two unequal labels.  Nonequality is a precondition for
 // this function (the instrumentation pass inlines the equality test).
 extern "C" SANITIZER_INTERFACE_ATTRIBUTE
 dfsan_label __dfsan_union(dfsan_label l1, dfsan_label l2) {
   if (flags().fast16labels)
     return l1 | l2;
   DCHECK_NE(l1, l2);
 
   if (l1 == 0)
     return l2;
   if (l2 == 0)
     return l1;
 
static void InitializeFlags() {
 
   FlagParser parser;
   RegisterCommonFlags(&parser);
   RegisterDfsanFlags(&parser, &flags());
   parser.ParseStringFromEnv("DFSAN_OPTIONS");
   InitializeCommonFlags();
   if (Verbosity()) ReportUnrecognizedFlags();
   if (common_flags()->help) parser.PrintFlagDescriptions();
+
+  // If dfsan_use_fast16labels() was called before InitializeFlags(), we need
+  // to enable fast16labels mode here.
+  if (use_fast16labels)
+    flags().fast16labels = true;
+
+  flags_initialized = true;
 }
 
 static void InitializePlatformEarly() {
   AvoidCVE_2016_2143();
 #ifdef DFSAN_RUNTIME_VMA
   __dfsan::vmaSize =
     (MostSignificantSetBitIndex(GET_CURRENT_FRAME()) + 1);
   if (__dfsan::vmaSize == 39 || __dfsan::vmaSize == 42 ||

compiler-rt/lib/dfsan/done_abilist.txt

 fun:dfsan_has_label=uninstrumented
 fun:dfsan_has_label=discard
 fun:dfsan_has_label_with_desc=uninstrumented
 fun:dfsan_has_label_with_desc=discard
 fun:dfsan_set_write_callback=uninstrumented
 fun:dfsan_set_write_callback=custom
 fun:dfsan_flush=uninstrumented
 fun:dfsan_flush=discard
+fun:dfsan_use_fast16labels=uninstrumented
+fun:dfsan_use_fast16labels=discard
 
 ###############################################################################
 # glibc
 ###############################################################################
 fun:malloc=discard
 fun:free=discard
 
 # Functions that return a value that depends on the input, but the output might

compiler-rt/test/dfsan/fast16labels.c

-// RUN: %clang_dfsan %s -o %t
-// RUN: DFSAN_OPTIONS=fast16labels=1 %run %t
-// RUN: DFSAN_OPTIONS=fast16labels=1 not %run %t dfsan_create_label 2>&1 \
+// RUN: %clang_dfsan %s -mllvm -fast-16-labels -o %t
+// RUN: %run %t
+// RUN: not %run %t dfsan_create_label 2>&1 \
 // RUN:   | FileCheck %s --check-prefix=CREATE-LABEL
-// RUN: DFSAN_OPTIONS=fast16labels=1 not %run %t dfsan_get_label_info 2>&1 \
+// RUN: not %run %t dfsan_get_label_info 2>&1 \
 // RUN:   | FileCheck %s --check-prefix=GET-LABEL-INFO
-// RUN: DFSAN_OPTIONS=fast16labels=1 not %run %t dfsan_has_label_with_desc \
-// RUN:   2>&1 | FileCheck %s --check-prefix=HAS-LABEL-WITH-DESC
-// RUN: DFSAN_OPTIONS=fast16labels=1:dump_labels_at_exit=/dev/stdout not %run \
-// RUN:   %t 2>&1 | FileCheck %s --check-prefix=DUMP-LABELS
+// RUN: not %run %t dfsan_has_label_with_desc 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=HAS-LABEL-WITH-DESC
+// RUN: DFSAN_OPTIONS=dump_labels_at_exit=/dev/stdout not %run %t 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=DUMP-LABELS
 //
-// Tests DFSAN_OPTIONS=fast16labels=1
+// Tests fast16labels mode.
 //
 #include <sanitizer/dfsan_interface.h>
 
 #include <assert.h>
 #include <stdio.h>
 #include <string.h>
 
 int foo(int a, int b) {

llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/SpecialCaseList.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include "llvm/Transforms/Utils/Local.h"
+#include "llvm/Transforms/Utils/ModuleUtils.h"
 #include <algorithm>
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <iterator>
 #include <memory>
 #include <set>
 #include <string>

 //   void __dfsan_store_callback(dfsan_label Label);
 //   void __dfsan_mem_transfer_callback(dfsan_label *Start, size_t Len);
 //   void __dfsan_cmp_callback(dfsan_label CombinedLabel);
 static cl::opt<bool> ClEventCallbacks(
     "dfsan-event-callbacks",
     cl::desc("Insert calls to __dfsan_*_callback functions on data events."),
     cl::Hidden, cl::init(false));
 
+// Use a distinct bit for each base label, enabling faster unions with less
+// instrumentation.  Limits the max number of base labels to 16.
+static cl::opt<bool> ClFast16Labels(
+    "fast-16-labels",
+    cl::desc("Use more efficient instrumentation, limiting the number of "
+             "labels to 16."),
+    cl::Hidden, cl::init(false));
+
 static StringRef GetGlobalTypeString(const GlobalValue &G) {
   // Types of GlobalVariables are always pointer types.
   Type *GType = G.getValueType();
   // For now we support excluding struct types only.
   if (StructType *SGType = dyn_cast<StructType>(GType)) {
     if (!SGType->isLiteral())
       return SGType->getName();
   }
class DataFlowSanitizer : public ModulePass {
   FunctionType *DFSanUnionFnTy;
   FunctionType *DFSanUnionLoadFnTy;
   FunctionType *DFSanUnimplementedFnTy;
   FunctionType *DFSanSetLabelFnTy;
   FunctionType *DFSanNonzeroLabelFnTy;
   FunctionType *DFSanVarargWrapperFnTy;
   FunctionType *DFSanLoadStoreCmpCallbackFnTy;
   FunctionType *DFSanMemTransferCallbackFnTy;
+  FunctionType *DFSanUseFast16LabelsFnTy;
   FunctionCallee DFSanUnionFn;
   FunctionCallee DFSanCheckedUnionFn;
   FunctionCallee DFSanUnionLoadFn;
   FunctionCallee DFSanUnimplementedFn;
   FunctionCallee DFSanSetLabelFn;
   FunctionCallee DFSanNonzeroLabelFn;
   FunctionCallee DFSanVarargWrapperFn;
   FunctionCallee DFSanLoadCallbackFn;
   FunctionCallee DFSanStoreCallbackFn;
   FunctionCallee DFSanMemTransferCallbackFn;
   FunctionCallee DFSanCmpCallbackFn;
+  FunctionCallee DFSanUseFast16LabelsFn;
   MDNode *ColdCallWeights;
   DFSanABIList ABIList;
   DenseMap<Value *, Function *> UnwrappedFnMap;
   AttrBuilder ReadOnlyNoneAttrs;
   bool DFSanRuntimeShadowMask = false;
 
   Value *getShadowAddress(Value *Addr, Instruction *Pos);
   bool isInstrumented(const Function *F);
bool DataFlowSanitizer::doInitialization(Module &M) {
   DFSanVarargWrapperFnTy = FunctionType::get(
       Type::getVoidTy(*Ctx), Type::getInt8PtrTy(*Ctx), /*isVarArg=*/false);
   DFSanLoadStoreCmpCallbackFnTy =
       FunctionType::get(Type::getVoidTy(*Ctx), ShadowTy, /*isVarArg=*/false);
   Type *DFSanMemTransferCallbackArgs[2] = {ShadowPtrTy, IntptrTy};
   DFSanMemTransferCallbackFnTy =
       FunctionType::get(Type::getVoidTy(*Ctx), DFSanMemTransferCallbackArgs,
                         /*isVarArg=*/false);
+  DFSanUseFast16LabelsFnTy =
+      FunctionType::get(Type::getVoidTy(*Ctx), {}, /*isVarArg=*/false);
 
   if (GetArgTLSPtr) {
     Type *ArgTLSTy = ArrayType::get(ShadowTy, 64);
     ArgTLS = nullptr;
     GetArgTLSTy = FunctionType::get(PointerType::getUnqual(ArgTLSTy), false);
     GetArgTLS = ConstantExpr::getIntToPtr(
         ConstantInt::get(IntptrTy, uintptr_t(GetArgTLSPtr)),
         PointerType::getUnqual(GetArgTLSTy));
DFSanUnimplementedFn =
     AL = AL.addParamAttribute(M.getContext(), 0, Attribute::ZExt);
     DFSanSetLabelFn =
         Mod->getOrInsertFunction("__dfsan_set_label", DFSanSetLabelFnTy, AL);
   }
   DFSanNonzeroLabelFn =
       Mod->getOrInsertFunction("__dfsan_nonzero_label", DFSanNonzeroLabelFnTy);
   DFSanVarargWrapperFn = Mod->getOrInsertFunction("__dfsan_vararg_wrapper",
                                                   DFSanVarargWrapperFnTy);
+
+  DFSanUseFast16LabelsFn = Mod->getOrInsertFunction("dfsan_use_fast16labels",
+                                                    DFSanUseFast16LabelsFnTy);
 }
 
 // Initializes event callback functions and declare them in the module
 void DataFlowSanitizer::initializeCallbackFunctions(Module &M) {
   DFSanLoadCallbackFn = Mod->getOrInsertFunction("__dfsan_load_callback",
                                                  DFSanLoadStoreCmpCallbackFnTy);
   DFSanStoreCallbackFn = Mod->getOrInsertFunction(
       "__dfsan_store_callback", DFSanLoadStoreCmpCallbackFnTy);
if (ClDebugNonzeroLabels) {
         BranchInst *BI = cast<BranchInst>(SplitBlockAndInsertIfThen(
             Ne, Pos, /*Unreachable=*/false, ColdCallWeights));
         IRBuilder<> ThenIRB(BI);
         ThenIRB.CreateCall(DFSF.DFS.DFSanNonzeroLabelFn, {});
       }
     }
   }
 
+  if (ClFast16Labels) {
+    // Call dfsan_use_fast16labels() during preinit to enable runtime support
+    // for fast16labels mode.
+    auto *UseFast16LabelsInit =
+        new GlobalVariable(M, PointerType::getUnqual(DFSanUseFast16LabelsFnTy),
+                           /*constant=*/true, GlobalVariable::PrivateLinkage,
+                           cast<Constant>(DFSanUseFast16LabelsFn.getCallee()),
+                           "__dfsan_fast16labels_preinit");
+    UseFast16LabelsInit->setSection(".preinit_array");
+
+    // Dedup with comdat
+    if (Triple(M.getTargetTriple()).supportsCOMDAT())
+      UseFast16LabelsInit->setComdat(
+          M.getOrInsertComdat("dfsan_fast16labels.module_preinit"));
+
+    appendToUsed(M, UseFast16LabelsInit);
+  }
+
   return Changed || !FnsToInstrument.empty() ||
          M.global_size() != InitialGlobalSize || M.size() != InitialModuleSize;
 }
 
 Value *DFSanFunction::getArgTLSPtr() {
   if (ArgTLSPtr)
     return ArgTLSPtr;
   if (DFS.ArgTLS)
Value *DFSanFunction::combineShadows(Value *V1, Value *V2, Instruction *Pos) {
   auto Key = std::make_pair(V1, V2);
   if (V1 > V2)
     std::swap(Key.first, Key.second);
   CachedCombinedShadow &CCS = CachedCombinedShadows[Key];
   if (CCS.Block && DT.dominates(CCS.Block, Pos->getParent()))
     return CCS.Shadow;
 
   IRBuilder<> IRB(Pos);
-  if (AvoidNewBlocks) {
+  if (ClFast16Labels) {
+    CCS.Block = Pos->getParent();
+    CCS.Shadow = IRB.CreateOr(V1, V2);
+  } else if (AvoidNewBlocks) {
     CallInst *Call = IRB.CreateCall(DFS.DFSanCheckedUnionFn, {V1, V2});
     Call->addAttribute(AttributeList::ReturnIndex, Attribute::ZExt);
     Call->addParamAttr(0, Attribute::ZExt);
     Call->addParamAttr(1, Attribute::ZExt);
 
     CCS.Block = Pos->getParent();
     CCS.Shadow = Call;
   } else {
case 2: {
     IRBuilder<> IRB(Pos);
     Value *ShadowAddr1 = IRB.CreateGEP(DFS.ShadowTy, ShadowAddr,
                                        ConstantInt::get(DFS.IntptrTy, 1));
     return combineShadows(
         IRB.CreateAlignedLoad(DFS.ShadowTy, ShadowAddr, ShadowAlign),
         IRB.CreateAlignedLoad(DFS.ShadowTy, ShadowAddr1, ShadowAlign), Pos);
   }
   }
+
+  if (ClFast16Labels && Size % (64 / DFS.ShadowWidthBits) == 0) {
+    // First OR all the WideShadows, then OR individual shadows within the
+    // combined WideShadow.  This is fewer instructions than ORing shadows
+    // individually.
+    IRBuilder<> IRB(Pos);
+    Value *WideAddr =
+        IRB.CreateBitCast(ShadowAddr, Type::getInt64PtrTy(*DFS.Ctx));
+    Value *CombinedWideShadow =
+        IRB.CreateAlignedLoad(IRB.getInt64Ty(), WideAddr, ShadowAlign);
+    for (uint64_t Ofs = 64 / DFS.ShadowWidthBits; Ofs != Size;
+         Ofs += 64 / DFS.ShadowWidthBits) {
+      WideAddr = IRB.CreateGEP(Type::getInt64Ty(*DFS.Ctx), WideAddr,
+                               ConstantInt::get(DFS.IntptrTy, 1));
+      Value *NextWideShadow =
+          IRB.CreateAlignedLoad(IRB.getInt64Ty(), WideAddr, ShadowAlign);
+      CombinedWideShadow = IRB.CreateOr(CombinedWideShadow, NextWideShadow);
+    }
+    for (unsigned Width = 32; Width >= DFS.ShadowWidthBits; Width >>= 1) {
+      Value *ShrShadow = IRB.CreateLShr(CombinedWideShadow, Width);
+      CombinedWideShadow = IRB.CreateOr(CombinedWideShadow, ShrShadow);
+    }
+    return IRB.CreateTrunc(CombinedWideShadow, DFS.ShadowTy);
+  }
   if (!AvoidNewBlocks && Size % (64 / DFS.ShadowWidthBits) == 0) {
     // Fast path for the common case where each byte has identical shadow: load
     // shadow 64 bits at a time, fall out to a __dfsan_union_load call if any
     // shadow is non-equal.
     BasicBlock *FallbackBB = BasicBlock::Create(*DFS.Ctx, "", F);
     IRBuilder<> FallbackIRB(FallbackBB);
     CallInst *FallbackCall = FallbackIRB.CreateCall(
         DFS.DFSanUnionLoadFn,

llvm/test/Instrumentation/DataFlowSanitizer/fast16labels.ll

+; Test that fast-16-labels mode uses inline ORs rather than calling
+; __dfsan_union or __dfsan_union_load.
+; RUN: opt < %s -dfsan -fast-16-labels -S | FileCheck %s --implicit-check-not="call{{.*}}__dfsan_union"
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define i8 @add(i8 %a, i8 %b) {
+  ; CHECK-LABEL: define i8 @"dfs$add"
+  ; CHECK-DAG: %[[ALABEL:.*]] = load{{.*}}__dfsan_arg_tls, i64 0, i64 0
+  ; CHECK-DAG: %[[BLABEL:.*]] = load{{.*}}__dfsan_arg_tls, i64 0, i64 1
+  ; CHECK: %[[ADDLABEL:.*]] = or i16 %[[ALABEL]], %[[BLABEL]]
+  ; CHECK: add i8
+  ; CHECK: store i16 %[[ADDLABEL]], i16* @__dfsan_retval_tls
+  ; CHECK: ret i8
+  %c = add i8 %a, %b
+  ret i8 %c
+}
+
+define i8 @load8(i8* %p) {
+  ; CHECK-LABEL: define i8 @"dfs$load8"
+  ; CHECK: load i16, i16*
+  ; CHECK: ptrtoint i8* {{.*}} to i64
+  ; CHECK: and i64
+  ; CHECK: mul i64
+  ; CHECK: inttoptr i64
+  ; CHECK: load i16, i16*
+  ; CHECK: or i16
+  ; CHECK: load i8, i8*
+  ; CHECK: store i16 {{.*}} @__dfsan_retval_tls
+  ; CHECK: ret i8
+
+  %a = load i8, i8* %p
+  ret i8 %a
+}
+
+define i16 @load16(i16* %p) {
+  ; CHECK-LABEL: define i16 @"dfs$load16"
+  ; CHECK: ptrtoint i16*
+  ; CHECK: and i64
+  ; CHECK: mul i64
+  ; CHECK: inttoptr i64 {{.*}} i16*
+  ; CHECK: getelementptr i16
+  ; CHECK: load i16, i16*
+  ; CHECK: load i16, i16*
+  ; CHECK: or i16
+  ; CHECK: or i16
+  ; CHECK: load i16, i16*
+  ; CHECK: store {{.*}} @__dfsan_retval_tls
+  ; CHECK: ret i16
+
+  %a = load i16, i16* %p
+  ret i16 %a
+}
+
+define i32 @load32(i32* %p) {
+  ; CHECK-LABEL: define i32 @"dfs$load32"
+  ; CHECK: ptrtoint i32*
+  ; CHECK: and i64
+  ; CHECK: mul i64
+  ; CHECK: inttoptr i64 {{.*}} i16*
+  ; CHECK: bitcast i16* {{.*}} i64*
+  ; CHECK: load i64, i64*
+  ; CHECK: lshr i64 {{.*}}, 32
+  ; CHECK: or i64
+  ; CHECK: lshr i64 {{.*}}, 16
+  ; CHECK: or i64
+  ; CHECK: trunc i64 {{.*}} i16
+  ; CHECK: or i16
+  ; CHECK: load i32, i32*
+  ; CHECK: store i16 {{.*}} @__dfsan_retval_tls
+  ; CHECK: ret i32
+
+  %a = load i32, i32* %p
+  ret i32 %a
+}
+
+define i64 @load64(i64* %p) {
+  ; CHECK-LABEL: define i64 @"dfs$load64"
+  ; CHECK: ptrtoint i64*
+  ; CHECK: and i64
+  ; CHECK: mul i64
+  ; CHECK: inttoptr i64 {{.*}} i16*
+  ; CHECK: bitcast i16* {{.*}} i64*
+  ; CHECK: load i64, i64*
+  ; CHECK: getelementptr i64, i64* {{.*}}, i64 1
+  ; CHECK: load i64, i64*
+  ; CHECK: or i64
+  ; CHECK: lshr i64 {{.*}}, 32
+  ; CHECK: or i64
+  ; CHECK: lshr i64 {{.*}}, 16
+  ; CHECK: or i64
+  ; CHECK: trunc i64 {{.*}} i16
+  ; CHECK: or i16
+  ; CHECK: load i64, i64*
+  ; CHECK: store i16 {{.*}} @__dfsan_retval_tls
+  ; CHECK: ret i64
+
+  %a = load i64, i64* %p
+  ret i64 %a
+}