[asan] Implemented intrinsic for the custom calling convention similar used by HWASan for X86.

[kstoimenov]

Summary

The implementation uses the int_asan_check_memaccess intrinsic to instrument the code. The intrinsic is replaced by a call to a function which performs the access check. The generated function names encode the input register name as a number using Reg - X86::NoRegister formula.

[kstoimenov]

Added an IR test.

[kstoimenov]

Fixed the test.

[kstoimenov]

Address lint comments.

[kstoimenov]

Started using R8 instead of RAX beacuse of some test failures. Also fixed the "Yes means No and No means Yes. Delete all files [Y]? " bug :-) for the load/store functions.

[kstoimenov]

Added code generation tests and fixed some bugs along the way.

[vitalybuka]

This can be split into 3 patches in the following order:

1. x86
2. Instrumentation
3. clang

Each of them needs tests, Instrumentation changes are not tested.

llvm/include/llvm/Transforms/Instrumentation/AddressSanitizer.h

150 ↗

(On Diff #366141)

It's not how enums described here https://llvm.org/docs/CodingStandards.html#id43
Also it's common to use enum class nowerdays.

However this one does not need to be enum and just "constexpr size_t"

[kstoimenov]

X86 only patch.

[kstoimenov]

Amended this patch to be the x86 only patch. PTAL.

In D107850#2949939, @vitalybuka wrote:

This can be split into 3 patches in the following order:

1. x86
2. Instrumentation
3. clang

Each of them needs tests, Instrumentation changes are not tested.

[kstoimenov]

Never mind, this won't compile. Let me fix it and I will let you know when it is ready for review.

[kstoimenov]

Instead of packing and bit shifting arguments passing them as individual intrinsic arguments.

[kstoimenov]

Removed unused include.

[kstoimenov]

Compiles and tests are passing now. PTAL.

[vitalybuka]

Can you please use "Edit related revisions" and link other patches of the features into the stack?

llvm/lib/Target/X86/X86InstrCompiler.td

272–274

shadowbase, mappingscale and orshadowoffset are known to runtime, we should not have them as part of intrinsic

llvm/lib/Target/X86/X86MCInstLower.cpp

1330

This does not look right

1334

Why this called reg when it's address?

1340

Why shadow is an argument? Both runtime and instrumented code know the shadow.

1340–1341

could you please unpack these to into local variables for consistency?

1369

I assumed the goal to move this shadow math into runtime code
so we have less instructions in the instrumented code.

Also MappingScale and OrShadowOffset are constants.
It's a little bit not nice that ShadowBase can be dynamic, so each __asan_check_ will have to load it,
but let's for now care about fixed shadow which is constant.

[vitalybuka]

llvm/lib/Target/X86/X86MCInstLower.cpp

1369

My mistake, runtime has nothing to do with that, we have implementation of callbacks here in object file.
Still those parameters are not needed to be exposed to IR layers. They must be the same for AddressSanitizer.cpp and here. So please just expose some functions e.g. in llvm/Transforms/Instrumentation/AddressSanitizerCommon.h and use them here and there to calculate these values.

[vitalybuka]

Please don't do [PATCH*] tags and just organize Phabricator "stack"
Just in case, it should look like this random image from search: https://firefox-source-docs.mozilla.org/_images/example-stack.png

[kstoimenov]

I know it seems redundant to pass the constants with every function call, but otherwise the code generation code will have to take dependency on AddressSanitizerCommon.h, which I am not sure is ideal. I spent some time looking into it and becomes quite messy. For example the tests now need a way to set the values for those variables. I think the main downside of this approach is the bloat in the IR code, but it is cleaner than exposing internal asan details.

One other option I have looked into is to pass this info using the Module. For example using the addModuleFlag function, but again there is a problem of the tests. We could have a quick chat to discuss this if you want.

llvm/lib/Target/X86/X86MCInstLower.cpp

1334

Because it is the id of the register which is used to pass the address. For example for X86::RDI it will be 53.

[vitalybuka]

In D107850#2952641, @kstoimenov wrote:

I know it seems redundant to pass the constants with every function call, but otherwise the code generation code will have to take dependency on AddressSanitizerCommon.h, which I am not sure is ideal. I spent some time looking into it and > becomes quite messy. For example the tests now need a way to set the values for those variables. I think the main downside of this approach is the bloat in the IR code, but it is cleaner than exposing internal asan details.

Bloat in IR is bigger concert than include of AddressSanitizerCommon.h. And I don't see anything messy about that.
Similar AArch64/AArch64AsmPrinter.cpp does not hesitate to #include "llvm/Transforms/Instrumentation/HWAddressSanitizer.h"

One other option I have looked into is to pass this info using the Module. For example using the addModuleFlag function, but again there is a problem of the tests. We could have a quick chat to discuss this if you want.

It will extend the language, we should try to avoid that if possible. So far all values directly calculated from target triple.

[kstoimenov]

Pulled constants like shadow base, etc out of the intrinsic.

[kstoimenov]

Pulled constants like shadow base, etc out of the intrinsic.

[kstoimenov]

Fixed after rebase.

[kstoimenov]

Added getAddressSanitizerParams to pull the constants out. Let me know if this is what you had in mind. Also struggled a little bit with git and rebase, but I think I have figured it out now.

llvm/lib/Target/X86/X86MCInstLower.cpp

1340–1341

Removed the variables as we have discussed.

[vitalybuka]

llvm/include/llvm/IR/Intrinsics.td

1640

PTAL at lvm.read_register.i32

How about:

llvm.asan.check.memaccess ->

lvm.asan.check_read
lvm.asan.check_write
lvm.asan.kernel.check_read
lvm.asan.kernel.check_write

Even better

lvm.asan.check_read.{i8, i16, i32, ...}
lvm.asan.check_write.{i8, i16, i32, ...}
lvm.asan.kernel.check_read.{i8, i16, i32, ...}
lvm.asan.kernel.check_write.{i8, i16, i32, ...}

[vitalybuka]

llvm/include/llvm/IR/Intrinsics.td

1640

Looks like underscore is not used in intrinsic names, so essentially the same with dots.

[kstoimenov]

llvm/include/llvm/IR/Intrinsics.td

1640

Sounds good to me. I do the full expansion so there will be 20 intrinsics altogether. I will update the code and ping you when done.

[vitalybuka]

llvm/include/llvm/IR/Intrinsics.td

1640

@pcc @eugenis
WDYT, I think later we can do the same for HWASAN?

[kstoimenov]

Update before expanding intrinsics.

[eugenis]

llvm/include/llvm/IR/Intrinsics.td

1640

I don't see what these multiple intrinsics give us that a single memaccess one does not provide?

As long as access type and similar arguments are immediates.

1642

We've just removed IntrInaccessibleMemOnly from hwasan. This needs to alias shadow updates.

[pcc]

llvm/include/llvm/IR/Intrinsics.td

1640

Agree with @eugenis - these sorts of intrinsic variants are typically used for distinguishing different pointer element types and we're in the process of getting rid of those anyway.

[vitalybuka]

llvm/include/llvm/IR/Intrinsics.td

1640

@pcc @eugenis Then do you prefer to encode is_write+size+kernel into non-human unreadable AccessInfo, like hwasan, or separate 0/1 arguments.
I probably prefer AccessInfo, as they both unreadable, but the hwasan version is shorter.

[eugenis]

llvm/include/llvm/IR/Intrinsics.td

1640

don't have a strong opinion, but sometimes I wish that hwasan outlined function names were more readable. The magic number in the names takes effort to decode.

[kstoimenov]

Removed IntrInaccessibleMemOnly.

[kstoimenov]

llvm/include/llvm/IR/Intrinsics.td

1640

I think I am gonna go with int_asan_check(?_kernel)_(load|store) and pass the size as parameter. What do you think?

[pcc]

llvm/include/llvm/IR/Intrinsics.td

1640

You mean in a register? I think that could mean more register pressure -> higher code size.

The magic numbers are unfortunate but they aren't that hard to decode (maybe we should be printing them as hex to make it a bit easier). I suppose we could pretty print the access info into the symbol name but only a few people will be looking at these so I'm not sure it's worth the effort.

[vitalybuka]

llvm/include/llvm/IR/Intrinsics.td

1640

AccessInfo

1640

I think I am gonna go with int_asan_check(?_kernel)_(load|store) and pass the size as parameter. What do you think?

Looks like both @pcc and @eugenis do not like multiple intrinsic.

Also I chatted with @eugenis to clarify the last comment. He is OK with AccessInfo like in hwasan.

1640

I think I am gonna go with int_asan_check(?_kernel)_(load|store) and pass the size as parameter. What do you think?

[vitalybuka]

llvm/test/CodeGen/X86/asan-check-memaccess-or.ll

48

Is this test out of date? Code has fewer arguments now.

[kstoimenov]

Moved back to using AccessInfo and fixed the tests.

[kstoimenov]

Upload after updating AddressSanitizer.cpp.

[kstoimenov]

All tests are passing locally. Should be good to review now.

llvm/include/llvm/Transforms/Instrumentation/AddressSanitizer.h

150 ↗

(On Diff #366141)

I wanted to be as close as possible to the HWASan style. Please let me know if you want me to change it.

llvm/test/CodeGen/X86/asan-check-memaccess-or.ll

48

Should be updated now.

[vitalybuka]

llvm/include/llvm/Transforms/Instrumentation/AddressSanitizer.h

150 ↗

(On Diff #366141)

They are quite unrelated, no need to violate coding style to match them.

Also packing and unpacking implementation unnecessary separated.
Maybe something like this and keep bit arithmetic in a single cpp file:

struct AsanAccessInfo {
  uint8_t AccessSize;
  bool CompileKernel;
  bool IsWrite;
...

  explicit AsanAccessInfo(int64_t packed);
  int64_t Pack() const;
}

Also math a little bit simpler if 4bit field is the last

llvm/lib/Target/X86/X86MCInstLower.cpp

1424

what is in ECX register here?

1551–1553

Tests do not check these attributes.

1559–1562

as-is shifts are in the header, and masks are here, which is not nice.

llvm/test/CodeGen/X86/asan-check-memaccess-add.ll

7

should we also check how we load registers before callq?

[kstoimenov]

Addressed the comments.

[kstoimenov]

llvm/include/llvm/Transforms/Instrumentation/AddressSanitizer.h

150 ↗

(On Diff #366141)

I like the idea. Keeps things well encapsulated in the implementation file. Done.

llvm/lib/Target/X86/X86MCInstLower.cpp

1424

Should be NoRegister. Done.

1559–1562

Encapsulated in the implementation file as you have recommended.

llvm/test/CodeGen/X86/asan-check-memaccess-add.ll

7

The issue is that there is no load instruction, because the registers are implied.

[kstoimenov]

Fixed code generation bug.

[vitalybuka]

llvm/include/llvm/Transforms/Instrumentation/AddressSanitizer.h

166 ↗

(On Diff #368236)

.cpp file is missing

llvm/lib/Target/X86/X86MCInstLower.cpp

1424

Why NoRegister update is not reflected in tests?

llvm/test/CodeGen/X86/asan-check-memaccess-or.ll

95

What is ECX here?

[kstoimenov]

Attempt to split AddressSanitizer.cpp.

[kstoimenov]

Second attempt to split AddressSanitizer.cpp.

[kstoimenov]

Fixed a crash.

llvm/lib/Target/X86/X86MCInstLower.cpp

1424

The instruction template for AND32ri8 expects 2 registers for some reason, which I am not sure why. If I provide only one register I get a clang crash. The emitted instruction is what I want so I didn't dig much deeper into it.

llvm/test/CodeGen/X86/asan-check-memaccess-or.ll

95

ECX is the lower 3 bits of the address. Is is the second part of the check "((Addr & 7) + AccessSize > k)".

[vitalybuka]

LGTM

[kstoimenov]

Updated with the fix for Bazel build.

[vitalybuka]

And if it's straightforward fix for the revert you can "reuse" approvals and land without additional review.

[kstoimenov]

Thanks! I will submit tomorrow morning.

[kstoimenov]

Updated from the revert commit.

llvm/include/llvm/IR/Intrinsics.td

 
 // This instruction has no actual effect, though it is treated by the optimizer
 // has having opaque side effects. This may be inserted into loops to ensure
 // that they are not removed even if they turn out to be empty, for languages
 // which specify that infinite loops must be preserved.
 def int_sideeffect : DefaultAttrsIntrinsic<[], [], [IntrInaccessibleMemOnly, IntrWillReturn]>;
 
 // The pseudoprobe intrinsic works as a place holder to the block it probes.
-// Like the sideeffect intrinsic defined above, this intrinsic is treated by the 
-// optimizer as having opaque side effects so that it won't be get rid of or moved 
+// Like the sideeffect intrinsic defined above, this intrinsic is treated by the
+// optimizer as having opaque side effects so that it won't be get rid of or moved
 // out of the block it probes.
 def int_pseudoprobe : Intrinsic<[], [llvm_i64_ty, llvm_i64_ty, llvm_i32_ty, llvm_i64_ty],
                                     [IntrInaccessibleMemOnly, IntrWillReturn]>;
 
 // Arithmetic fence intrinsic.
 def int_arithmetic_fence : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>], [IntrNoMem]>;
 
 // Intrinsics to support half precision floating point format

 
 // Create a branch funnel that implements an indirect call to a limited set of
 // callees. This needs to be a musttail call.
 def int_icall_branch_funnel : DefaultAttrsIntrinsic<[], [llvm_vararg_ty], []>;
 
 def int_load_relative: DefaultAttrsIntrinsic<[llvm_ptr_ty], [llvm_ptr_ty, llvm_anyint_ty],
                                  [IntrReadMem, IntrArgMemOnly]>;
 
+def int_asan_check_memaccess :
+  Intrinsic<[],[llvm_ptr_ty, llvm_i64_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty,
+                llvm_i32_ty],
+            [IntrInaccessibleMemOnly, ImmArg<ArgIndex<1>>, ImmArg<ArgIndex<2>>,
+             ImmArg<ArgIndex<3>>, ImmArg<ArgIndex<4>>, ImmArg<ArgIndex<5>>]>;
+
 def int_hwasan_check_memaccess :
   Intrinsic<[], [llvm_ptr_ty, llvm_ptr_ty, llvm_i32_ty],
             [ImmArg<ArgIndex<2>>]>;
 def int_hwasan_check_memaccess_shortgranules :
   Intrinsic<[], [llvm_ptr_ty, llvm_ptr_ty, llvm_i32_ty],
             [ImmArg<ArgIndex<2>>]>;
 
 // Xray intrinsics

llvm/lib/Target/X86/X86AsmPrinter.h

class LLVM_LIBRARY_VISIBILITY X86AsmPrinter : public AsmPrinter {
   void LowerPATCHABLE_RET(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerPATCHABLE_TAIL_CALL(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerPATCHABLE_EVENT_CALL(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerPATCHABLE_TYPED_EVENT_CALL(const MachineInstr &MI,
                                        X86MCInstLower &MCIL);
 
   void LowerFENTRY_CALL(const MachineInstr &MI, X86MCInstLower &MCIL);
 
+  // Address sanitizer specific lowering for X86.
+  void LowerASAN_CHECK_MEMACCESS(const MachineInstr &MI);
+  void emitAsanMemaccessSymbols(Module &M);
+  void emitAsanMemaccessPartial(Module &M, unsigned Reg, uint64_t ShadowBase,
+                                bool IsWrite, size_t AccessSizeIndex,
+                                size_t MappingScale, bool OrShadowOffset,
+                                MCSubtargetInfo &STI);
+  void emitAsanMemaccessFull(Module &M, unsigned Reg, uint64_t ShadowBase,
+                             bool IsWrite, size_t AccessSizeIndex,
+                             size_t MappingScale, bool OrShadowOffset,
+                             MCSubtargetInfo &STI);
+  void emitAsanReportError(Module &M, unsigned Reg, bool IsWrite,
+                           size_t AccessSizeIndex, MCSubtargetInfo &STI);
+
+  typedef std::tuple<unsigned /*Reg*/, uint64_t /*ShadowBase*/,
+                     uint32_t /*IsWrite*/, uint32_t /*AccessSizeIndex*/,
+                     uint32_t /*MappingScaleShift*/,
+                     uint32_t /*OrShadowOffsetShift*/>
+      AsanMemaccessTuple;
+  std::map<AsanMemaccessTuple, MCSymbol *> AsanMemaccessSymbols;
+
   // Choose between emitting .seh_ directives and .cv_fpo_ directives.
   void EmitSEHInstruction(const MachineInstr *MI);
 
   void PrintSymbolOperand(const MachineOperand &MO, raw_ostream &O) override;
   void PrintOperand(const MachineInstr *MI, unsigned OpNo, raw_ostream &O);
   void PrintModifiedOperand(const MachineInstr *MI, unsigned OpNo,
                             raw_ostream &O, const char *Modifier);
   void PrintPCRelImm(const MachineInstr *MI, unsigned OpNo, raw_ostream &O);

llvm/lib/Target/X86/X86AsmPrinter.cpp

if (!Stubs.empty()) {
     Stubs.clear();
     OutStreamer.AddBlankLine();
   }
 }
 
 void X86AsmPrinter::emitEndOfAsmFile(Module &M) {
   const Triple &TT = TM.getTargetTriple();
 
+  emitAsanMemaccessSymbols(M);
+
   if (TT.isOSBinFormatMachO()) {
     // Mach-O uses non-lazy symbol stubs to encode per-TU information into
     // global table for symbol lookup.
     emitNonLazyStubs(MMI, *OutStreamer);
 
     // Emit stack and fault map information.
     emitStackMaps(SM);
     FM.serializeToFaultMapSection();

llvm/lib/Target/X86/X86InstrCompiler.td

def SEH_PushFrame : I<0, Pseudo, (outs), (ins i1imm:$mode),
                             "#SEH_PushFrame $mode", []>;
   def SEH_EndPrologue : I<0, Pseudo, (outs), (ins),
                             "#SEH_EndPrologue", []>;
   def SEH_Epilogue : I<0, Pseudo, (outs), (ins),
                             "#SEH_Epilogue", []>;
 }
 
 //===----------------------------------------------------------------------===//
+// Pseudo instructions used by address sanitizer.
+//===----------------------------------------------------------------------===//
+let
+  Defs = [R8, EFLAGS] in {
+def ASAN_CHECK_MEMACCESS : PseudoI<
+  (outs), (ins GR64NoR8:$addr, i64imm:$shadowbase, i32imm:$iswrite,
+           i32imm:$accesssizeindex, i32imm:$mappingscale,
+           i32imm:$orshadowoffset),
+  [(int_asan_check_memaccess GR64NoR8:$addr, (i64 timm:$shadowbase),
+    (i32 timm:$iswrite), (i32 timm:$accesssizeindex), (i32 timm:$mappingscale),
+    (i32 timm:$orshadowoffset))]>,
+  Sched<[]>;
+}
+
+//===----------------------------------------------------------------------===//
 // Pseudo instructions used by segmented stacks.
 //
 
 // This is lowered into a RET instruction by MCInstLower.  We need
 // this so that we don't have to have a MachineBasicBlock which ends
 // with a RET and also has successors.
 let isPseudo = 1, SchedRW = [WriteJumpLd] in {
 def MORESTACK_RET: I<0, Pseudo, (outs), (ins), "", []>;
def NAME#16 : I<opc, MRMSrcMem, (outs GR16:$dst),
                        GR16:$dst,
                        (!cast<PatFrag>(frag # "_16") addr:$ptr, GR16:$val))]>,
                     OpSize16;
     def NAME#32 : I<opc, MRMSrcMem, (outs GR32:$dst),
                     (ins GR32:$val, i32mem:$ptr),
                     !strconcat(mnemonic, "{l}\t{$val, $ptr|$ptr, $val}"),
                     [(set
                        GR32:$dst,
-                       (!cast<PatFrag>(frag # "_32") addr:$ptr, GR32:$val))]>, 
+                       (!cast<PatFrag>(frag # "_32") addr:$ptr, GR32:$val))]>,
                     OpSize32;
     def NAME#64 : RI<opc, MRMSrcMem, (outs GR64:$dst),
                      (ins GR64:$val, i64mem:$ptr),
                      !strconcat(mnemonic, "{q}\t{$val, $ptr|$ptr, $val}"),
                      [(set
                         GR64:$dst,
                         (!cast<PatFrag>(frag # "_64") addr:$ptr, GR64:$val))]>;
   }

llvm/lib/Target/X86/X86MCInstLower.cpp

 #include "llvm/MC/MCFixup.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/MC/MCInstBuilder.h"
 #include "llvm/MC/MCSection.h"
 #include "llvm/MC/MCSectionELF.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/MC/MCSymbolELF.h"
+#include "llvm/Support/TargetRegistry.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetMachine.h"
 
 using namespace llvm;
 
 namespace {
 
 /// X86MCInstLower - This class is used to lower an MachineInstr into an MCInst.
void X86AsmPrinter::LowerFENTRY_CALL(const MachineInstr &MI,
   const MCSymbolRefExpr *Op =
       MCSymbolRefExpr::create(fentry, MCSymbolRefExpr::VK_None, Ctx);
 
   EmitAndCountInstruction(
       MCInstBuilder(Is64Bits ? X86::CALL64pcrel32 : X86::CALLpcrel32)
           .addExpr(Op));
 }
 
+void X86AsmPrinter::LowerASAN_CHECK_MEMACCESS(const MachineInstr &MI) {
+  // FIXME: Make this work on non-ELF.
+  if (!TM.getTargetTriple().isOSBinFormatELF()) {
+    report_fatal_error("llvm.hwa`san.check.memaccess only supported on ELF");
+    return;
+  }
+
+  unsigned Reg = MI.getOperand(0).getReg().id();
+  uint64_t ShadowBase = MI.getOperand(1).getImm();
+  bool IsWrite = MI.getOperand(2).getImm();
+  size_t AccessSizeIndex = MI.getOperand(3).getImm();
+
+  MCSymbol *&Sym = AsanMemaccessSymbols[{
+      Reg, ShadowBase, IsWrite, AccessSizeIndex, MI.getOperand(4).getImm(),
+      MI.getOperand(5).getImm()}];
+  if (!Sym) {
+    std::string Name = IsWrite ? "store" : "load";
+    std::string SymName = "__asan_check_" + Name +
+                          utostr(1 << AccessSizeIndex) + "_rn" + utostr(Reg);
+    Sym = OutContext.getOrCreateSymbol(SymName);
+  }
+
+  EmitAndCountInstruction(
+      MCInstBuilder(X86::CALL64pcrel32)
+          .addExpr(MCSymbolRefExpr::create(Sym, OutContext)));
+}
+
+void X86AsmPrinter::emitAsanMemaccessPartial(Module &M, unsigned Reg,
+                                             uint64_t ShadowBase, bool IsWrite,
+                                             size_t AccessSizeIndex,
+                                             size_t MappingScale,
+                                             bool OrShadowOffset,
+                                             MCSubtargetInfo &STI) {
+  assert(AccessSizeIndex == 0 || AccessSizeIndex == 1 || AccessSizeIndex == 2);
+  assert(Reg != X86::RAX);
+
+  OutStreamer->emitInstruction(
+      MCInstBuilder(X86::MOV64rr).addReg(X86::R8).addReg(X86::NoRegister + Reg),
+      STI);
+  OutStreamer->emitInstruction(MCInstBuilder(X86::SHR64ri)
+                                   .addReg(X86::R8)
+                                   .addReg(X86::R8)
+                                   .addImm(MappingScale),
+                               STI);
+  if (OrShadowOffset) {
+    OutStreamer->emitInstruction(MCInstBuilder(X86::OR64ri32)
+                                     .addReg(X86::R8)
+                                     .addReg(X86::R8)
+                                     .addImm(ShadowBase),
+                                 STI);
+    OutStreamer->emitInstruction(MCInstBuilder(X86::MOV8rm)
+                                     .addReg(X86::R8B)
+                                     .addReg(X86::R8)
+                                     .addImm(1)
+                                     .addReg(X86::NoRegister)
+                                     .addImm(0)
+                                     .addReg(X86::NoRegister),
+                                 STI);
+    OutStreamer->emitInstruction(
+        MCInstBuilder(X86::TEST8rr).addReg(X86::R8B).addReg(X86::R8B), STI);
+  } else {
+    OutStreamer->emitInstruction(MCInstBuilder(X86::MOVSX32rm8)
+                                     .addReg(X86::R8D)
+                                     .addReg(X86::R8)
+                                     .addImm(1)
+                                     .addReg(X86::NoRegister)
+                                     .addImm(ShadowBase)
+                                     .addReg(X86::NoRegister),
+                                 STI);
+    OutStreamer->emitInstruction(
+        MCInstBuilder(X86::TEST32rr).addReg(X86::R8D).addReg(X86::R8D), STI);
+  }
+  MCSymbol *AdditionalCheck = OutContext.createTempSymbol();
+  OutStreamer->emitInstruction(
+      MCInstBuilder(X86::JCC_1)
+          .addExpr(MCSymbolRefExpr::create(AdditionalCheck, OutContext))
+          .addImm(X86::COND_NE),
+      STI);
+  MCSymbol *ReturnSym = OutContext.createTempSymbol();
+  OutStreamer->emitLabel(ReturnSym);
+  OutStreamer->emitInstruction(MCInstBuilder(getRetOpcode(*Subtarget)), STI);
+
+  // Shadow byte is non-zero so we need to perform additional checks.
+  OutStreamer->emitLabel(AdditionalCheck);
+  OutStreamer->emitInstruction(MCInstBuilder(X86::PUSH64r).addReg(X86::RCX),
+                               STI);
+  OutStreamer->emitInstruction(MCInstBuilder(X86::MOV64rr)
+                                   .addReg(X86::RCX)
+                                   .addReg(X86::NoRegister + Reg),
+                               STI);
+  const size_t Granularity = 1ULL << MappingScale;
+  OutStreamer->emitInstruction(MCInstBuilder(X86::AND32ri8)
+                                   .addReg(X86::ECX)
+                                   .addReg(X86::ECX)
+                                   .addImm(Granularity - 1),
+                               STI);
+  if (AccessSizeIndex == 1) {
+    OutStreamer->emitInstruction(MCInstBuilder(X86::ADD32ri8)
+                                     .addReg(X86::ECX)
+                                     .addReg(X86::ECX)
+                                     .addImm(1),
+                                 STI);
+  } else if (AccessSizeIndex == 2) {
+    OutStreamer->emitInstruction(MCInstBuilder(X86::ADD32ri8)
+                                     .addReg(X86::ECX)
+                                     .addReg(X86::ECX)
+                                     .addImm(3),
+                                 STI);
+  }
+
+  OutStreamer->emitInstruction(
+      MCInstBuilder(X86::CMP32rr).addReg(X86::ECX).addReg(X86::R8D).addImm(1),
+      STI);
+  OutStreamer->emitInstruction(MCInstBuilder(X86::POP64r).addReg(X86::RCX),
+                               STI);
+  OutStreamer->emitInstruction(
+      MCInstBuilder(X86::JCC_1)
+          .addExpr(MCSymbolRefExpr::create(ReturnSym, OutContext))
+          .addImm(X86::COND_L),
+      STI);
+
+  emitAsanReportError(M, Reg, IsWrite, AccessSizeIndex, STI);
+}
+
+void X86AsmPrinter::emitAsanMemaccessFull(Module &M, unsigned Reg,
+                                          uint64_t ShadowBase, bool IsWrite,
+                                          size_t AccessSizeIndex,
+                                          size_t MappingScale,
+                                          bool OrShadowOffset,
+                                          MCSubtargetInfo &STI) {
+  assert(AccessSizeIndex == 3 || AccessSizeIndex == 4);
+  assert(Reg != X86::R8);
+  OutStreamer->emitInstruction(
+      MCInstBuilder(X86::MOV64rr).addReg(X86::R8).addReg(X86::NoRegister + Reg),
+      STI);
+  OutStreamer->emitInstruction(MCInstBuilder(X86::SHR64ri)
+                                   .addReg(X86::R8)
+                                   .addReg(X86::R8)
+                                   .addImm(MappingScale),
+                               STI);
+  if (OrShadowOffset) {
+    OutStreamer->emitInstruction(MCInstBuilder(X86::OR64ri32)
+                                     .addReg(X86::R8)
+                                     .addReg(X86::R8)
+                                     .addImm(ShadowBase),
+                                 STI);
+    auto OpCode = AccessSizeIndex == 3 ? X86::CMP8mi : X86::CMP16mi8;
+    OutStreamer->emitInstruction(MCInstBuilder(OpCode)
+                                     .addReg(X86::R8)
+                                     .addImm(1)
+                                     .addReg(X86::NoRegister)
+                                     .addImm(0)
+                                     .addReg(X86::NoRegister)
+                                     .addImm(0),
+                                 STI);
+  } else {
+    auto OpCode = AccessSizeIndex == 3 ? X86::CMP8mi : X86::CMP16mi8;
+    OutStreamer->emitInstruction(MCInstBuilder(OpCode)
+                                     .addReg(X86::R8)
+                                     .addImm(1)
+                                     .addReg(X86::NoRegister)
+                                     .addImm(ShadowBase)
+                                     .addReg(X86::NoRegister)
+                                     .addImm(0),
+                                 STI);
+  }
+  MCSymbol *ReportCode = OutContext.createTempSymbol();
+  OutStreamer->emitInstruction(
+      MCInstBuilder(X86::JCC_1)
+          .addExpr(MCSymbolRefExpr::create(ReportCode, OutContext))
+          .addImm(X86::COND_NE),
+      STI);
+  MCSymbol *ReturnSym = OutContext.createTempSymbol();
+  OutStreamer->emitLabel(ReturnSym);
+  OutStreamer->emitInstruction(MCInstBuilder(getRetOpcode(*Subtarget)), STI);
+
+  OutStreamer->emitLabel(ReportCode);
+  emitAsanReportError(M, Reg, IsWrite, AccessSizeIndex, STI);
+}
+
+void X86AsmPrinter::emitAsanReportError(Module &M, unsigned Reg, bool IsWrite,
+                                        size_t AccessSizeIndex,
+                                        MCSubtargetInfo &STI) {
+  std::string Name = IsWrite ? "store" : "load";
+  MCSymbol *ReportError = OutContext.getOrCreateSymbol(
+      "__asan_report_" + Name + utostr(1 << AccessSizeIndex));
+  OutStreamer->emitInstruction(MCInstBuilder(X86::MOV64rr)
+                                   .addReg(X86::RDI)
+                                   .addReg(X86::NoRegister + Reg),
+                               STI);
+  OutStreamer->emitInstruction(
+      MCInstBuilder(X86::JMP_1)
+          .addExpr(MCSymbolRefExpr::create(ReportError, OutContext)),
+      STI);
+}
+
+void X86AsmPrinter::emitAsanMemaccessSymbols(Module &M) {
+  if (AsanMemaccessSymbols.empty())
+    return;
+
+  const Triple &TT = TM.getTargetTriple();
+  assert(TT.isOSBinFormatELF());
+  std::unique_ptr<MCSubtargetInfo> STI(
+      TM.getTarget().createMCSubtargetInfo(TT.str(), "", ""));
+  assert(STI && "Unable to create subtarget info");
+
+  for (auto &P : AsanMemaccessSymbols) {
+    MCSymbol *Sym = P.second;
+    OutStreamer->SwitchSection(OutContext.getELFSection(
+        ".text.hot", ELF::SHT_PROGBITS,
+        ELF::SHF_EXECINSTR | ELF::SHF_ALLOC | ELF::SHF_GROUP, 0, Sym->getName(),
+        /*IsComdat=*/true));
+
+    OutStreamer->emitSymbolAttribute(Sym, MCSA_ELF_TypeFunction);
+    OutStreamer->emitSymbolAttribute(Sym, MCSA_Weak);
+    OutStreamer->emitSymbolAttribute(Sym, MCSA_Hidden);
+    OutStreamer->emitLabel(Sym);
+
+    unsigned Reg = std::get<0>(P.first);
+    uint64_t ShadowBase = std::get<1>(P.first);
+    bool IsWrite = std::get<2>(P.first);
+    size_t AccessSizeIndex = std::get<3>(P.first);
+    size_t MappingScale = std::get<4>(P.first);
+    bool OrShadowOffset = std::get<5>(P.first);
+
+    if (AccessSizeIndex < 3) {
+      emitAsanMemaccessPartial(M, Reg, ShadowBase, IsWrite, AccessSizeIndex,
+                               MappingScale, OrShadowOffset, *STI);
+    } else {
+      emitAsanMemaccessFull(M, Reg, ShadowBase, IsWrite, AccessSizeIndex,
+                            MappingScale, OrShadowOffset, *STI);
+    }
+  }
+}
+
 void X86AsmPrinter::LowerPATCHABLE_OP(const MachineInstr &MI,
                                       X86MCInstLower &MCIL) {
   // PATCHABLE_OP minsize, opcode, operands
 
   NoAutoPaddingScope NoPadScope(*OutStreamer);
 
   unsigned MinSize = MI.getOperand(0).getImm();
   unsigned Opcode = MI.getOperand(1).getImm();
 
void X86AsmPrinter::emitInstruction(const MachineInstr *MI) {
 
   case TargetOpcode::PATCHABLE_TYPED_EVENT_CALL:
     return LowerPATCHABLE_TYPED_EVENT_CALL(*MI, MCInstLowering);
 
   case X86::MORESTACK_RET:
     EmitAndCountInstruction(MCInstBuilder(getRetOpcode(*Subtarget)));
     return;
 
+  case X86::ASAN_CHECK_MEMACCESS:
+    return LowerASAN_CHECK_MEMACCESS(*MI);
+
   case X86::MORESTACK_RET_RESTORE_R10:
     // Return, then restore R10.
     EmitAndCountInstruction(MCInstBuilder(getRetOpcode(*Subtarget)));
     EmitAndCountInstruction(
         MCInstBuilder(X86::MOV64rr).addReg(X86::R10).addReg(X86::RAX));
     return;
 
   case X86::SEH_PushReg:

llvm/lib/Target/X86/X86RegisterInfo.td

 // RIP isn't really a register and it can't be used anywhere except in an
 // address, but it doesn't cause trouble.
 // FIXME: it *does* cause trouble - CheckBaseRegAndIndexReg() has extra
 // tests because of the inclusion of RIP in this register class.
 def GR64 : RegisterClass<"X86", [i64], 64,
                          (add RAX, RCX, RDX, RSI, RDI, R8, R9, R10, R11,
                               RBX, R14, R15, R12, R13, RBP, RSP, RIP)>;
 
+// GR64 - 64-bit GPRs without R8 and RIP. Could be used when emitting code for
+// intrinsics, which use implict input registers.
+def GR64NoR8 : RegisterClass<"X86", [i64], 64,
+                              (add RAX, RCX, RDX, RSI, RDI, R9, R10, R11,
+                                   RBX, R14, R15, R12, R13, RBP, RSP)>;
+
 // Segment registers for use by MOV instructions (and others) that have a
 //   segment register as one operand.  Always contain a 16-bit segment
 //   descriptor.
 def SEGMENT_REG : RegisterClass<"X86", [i16], 16, (add CS, DS, SS, ES, FS, GS)>;
 
 // Debug registers.
 def DEBUG_REG : RegisterClass<"X86", [i32], 32, (sequence "DR%u", 0, 15)>;
 

llvm/test/CodeGen/X86/asan-check-memaccess-add.ll

+; RUN: llc < %s | FileCheck %s
+
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @load1(i8* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load1_rn[[RN1:.*]]
+; CHECK:              callq   __asan_check_store1_rn[[RN1]]
+; CHECK-NEXT:         retq
+  call void @llvm.asan.check.memaccess(i8* %x, i64 2147450880, i32 0,
+                                       i32 0, i32 3, i32 0)
+  call void @llvm.asan.check.memaccess(i8* %x, i64 2147450880, i32 1,
+                                       i32 0, i32 3, i32 0)
+  ret void
+}
+
+define void @load2(i16* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load2_rn[[RN2:.*]]
+; CHECK:              callq   __asan_check_store2_rn[[RN2]]
+; CHECK-NEXT:         retq
+  %1 = ptrtoint i16* %x to i64
+  %2 = bitcast i16* %x to i8*
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 0,
+                                       i32 1, i32 3, i32 0)
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 1,
+                                       i32 1, i32 3, i32 0)
+  ret void
+}
+
+define void @load4(i32* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load4_rn[[RN4:.*]]
+; CHECK:              callq   __asan_check_store4_rn[[RN4]]
+; CHECK-NEXT:         retq
+  %1 = ptrtoint i32* %x to i64
+  %2 = bitcast i32* %x to i8*
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 0,
+                                       i32 2, i32 3, i32 0)
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 1,
+                                       i32 2, i32 3, i32 0)
+  ret void
+}
+define void @load8(i64* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load8_rn[[RN8:.*]]
+; CHECK:              callq   __asan_check_store8_rn[[RN8]]
+; CHECK-NEXT:         retq
+  %1 = ptrtoint i64* %x to i64
+  %2 = bitcast i64* %x to i8*
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 0,
+                                       i32 3, i32 3, i32 0)
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 1,
+                                       i32 3, i32 3, i32 0)
+  ret void
+}
+
+define void @load16(i128* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load16_rn[[RN16:.*]]
+; CHECK:              callq   __asan_check_store16_rn[[RN16]]
+; CHECK-NEXT:         retq
+  %1 = ptrtoint i128* %x to i64
+  %2 = bitcast i128* %x to i8*
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 0,
+                                       i32 4, i32 3, i32 0)
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 1,
+                                       i32 4, i32 3, i32 0)
+  ret void
+}
+
+; CHECK:      __asan_check_load1_rn[[RN1]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         movsbl  2147450880(%r8), %r8d
+; CHECK-NEXT:         testl   %r8d, %r8d
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load1
+
+; CHECK:      __asan_check_load2_rn[[RN2]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         movsbl  2147450880(%r8), %r8d
+; CHECK-NEXT:         testl   %r8d, %r8d
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         addl    $1, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load2
+
+; CHECK:      __asan_check_load4_rn[[RN4]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         movsbl  2147450880(%r8), %r8d
+; CHECK-NEXT:         testl   %r8d, %r8d
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         addl    $3, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load4
+
+; CHECK:      __asan_check_load8_rn[[RN8]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         cmpb    $0, 2147450880(%r8)
+; CHECK-NEXT:         jne     [[FAIL:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[FAIL]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load8
+
+; CHECK:      __asan_check_load16_rn[[RN16]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         cmpw    $0, 2147450880(%r8)
+; CHECK-NEXT:         jne     [[FAIL:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[FAIL]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load16
+
+; CHECK:      __asan_check_store1_rn[[RN1]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         movsbl  2147450880(%r8), %r8d
+; CHECK-NEXT:         testl   %r8d, %r8d
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store1
+
+; CHECK:      __asan_check_store2_rn[[RN2]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         movsbl  2147450880(%r8), %r8d
+; CHECK-NEXT:         testl   %r8d, %r8d
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         addl    $1, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store2
+
+; CHECK:      __asan_check_store4_rn[[RN4]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         movsbl  2147450880(%r8), %r8d
+; CHECK-NEXT:         testl   %r8d, %r8d
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         addl    $3, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store4
+
+; CHECK:      __asan_check_store8_rn[[RN8]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         cmpb    $0, 2147450880(%r8)
+; CHECK-NEXT:         jne     [[FAIL:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[FAIL]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store8
+
+; CHECK:      __asan_check_store16_rn[[RN16]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         cmpw    $0, 2147450880(%r8)
+; CHECK-NEXT:         jne     [[FAIL:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[FAIL]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store16
+
+declare void @llvm.asan.check.memaccess(i8*, i64 immarg, i32 immarg,
+                                        i32 immarg, i32 immarg, i32 immarg)

llvm/test/CodeGen/X86/asan-check-memaccess-or.ll

+; RUN: llc < %s | FileCheck %s
+
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @load1(i8* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load1_rn[[RN1:.*]]
+; CHECK:              callq   __asan_check_store1_rn[[RN1]]
+; CHECK-NEXT:         retq
+  call void @llvm.asan.check.memaccess(i8* %x, i64 2147450880, i32 0,
+                                       i32 0, i32 3, i32 1)
+  call void @llvm.asan.check.memaccess(i8* %x, i64 2147450880, i32 1,
+                                       i32 0, i32 3, i32 1)
+  ret void
+}
+
+define void @load2(i16* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load2_rn[[RN2:.*]]
+; CHECK:              callq   __asan_check_store2_rn[[RN2]]
+; CHECK-NEXT:         retq
+  %1 = ptrtoint i16* %x to i64
+  %2 = bitcast i16* %x to i8*
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 0,
+                                       i32 1, i32 3, i32 1)
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 1,
+                                       i32 1, i32 3, i32 1)
+  ret void
+}
+
+define void @load4(i32* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load4_rn[[RN4:.*]]
+; CHECK:              callq   __asan_check_store4_rn[[RN4]]
+; CHECK-NEXT:         retq
+  %1 = ptrtoint i32* %x to i64
+  %2 = bitcast i32* %x to i8*
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 0,
+                                       i32 2, i32 3, i32 1)
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 1,
+                                       i32 2, i32 3, i32 1)
+  ret void
+}
+define void @load8(i64* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load8_rn[[RN8:.*]]
+; CHECK:              callq   __asan_check_store8_rn[[RN8]]
+; CHECK-NEXT:         retq
+  %1 = ptrtoint i64* %x to i64
+  %2 = bitcast i64* %x to i8*
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 0,
+                                       i32 3, i32 3, i32 1)
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 1,
+                                       i32 3, i32 3, i32 1)
+  ret void
+}
+
+define void @load16(i128* nocapture readonly %x) {
+; CHECK:              callq   __asan_check_load16_rn[[RN16:.*]]
+; CHECK:              callq   __asan_check_store16_rn[[RN16]]
+; CHECK-NEXT:         retq
+  %1 = ptrtoint i128* %x to i64
+  %2 = bitcast i128* %x to i8*
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 0,
+                                       i32 4, i32 3, i32 1)
+  call void @llvm.asan.check.memaccess(i8* %2, i64 2147450880, i32 1,
+                                       i32 4, i32 3, i32 1)
+  ret void
+}
+
+; CHECK:      __asan_check_load1_rn[[RN1]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8{{.*}}
+; CHECK-NEXT:         movb    (%r8), %r8b
+; CHECK-NEXT:         testb   %r8b, %r8b
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load1
+
+; CHECK:      __asan_check_load2_rn[[RN2]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8{{.*}}
+; CHECK-NEXT:         movb    (%r8), %r8b
+; CHECK-NEXT:         testb   %r8b, %r8b
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         addl    $1, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load2
+
+; CHECK:      __asan_check_load4_rn[[RN4]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8{{.*}}
+; CHECK-NEXT:         movb    (%r8), %r8b
+; CHECK-NEXT:         testb   %r8b, %r8b
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         addl    $3, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load4
+
+; CHECK:      __asan_check_load8_rn[[RN8]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8{{.*}}
+; CHECK-NEXT:         cmpb    $0, (%r8)
+; CHECK-NEXT:         jne     [[FAIL:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[FAIL]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load8
+
+; CHECK:      __asan_check_load16_rn[[RN16]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8{{.*}}
+; CHECK-NEXT:         cmpw    $0, (%r8)
+; CHECK-NEXT:         jne     [[FAIL:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[FAIL]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_load16
+
+; CHECK:      __asan_check_store1_rn[[RN1]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8
+; CHECK-NEXT:         movb    (%r8), %r8b
+; CHECK-NEXT:         testb   %r8b, %r8b
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store1
+
+; CHECK:      __asan_check_store2_rn[[RN2]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8
+; CHECK-NEXT:         movb    (%r8), %r8b
+; CHECK-NEXT:         testb   %r8b, %r8b
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         addl    $1, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store2
+
+; CHECK:      __asan_check_store4_rn[[RN4]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8
+; CHECK-NEXT:         movb    (%r8), %r8b
+; CHECK-NEXT:         testb   %r8b, %r8b
+; CHECK-NEXT:         jne     [[EXTRA:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[EXTRA]]:
+; CHECK-NEXT:         pushq   %rcx
+; CHECK-NEXT:         movq    [[REG]], %rcx
+; CHECK-NEXT:         andl    $7, %ecx
+; CHECK-NEXT:         addl    $3, %ecx
+; CHECK-NEXT:         cmpl    %r8d, %ecx
+; CHECK-NEXT:         popq    %rcx
+; CHECK-NEXT:         jl      [[RET]]
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store4
+
+; CHECK:      __asan_check_store8_rn[[RN8]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8{{.*}}
+; CHECK-NEXT:         cmpb    $0, (%r8)
+; CHECK-NEXT:         jne     [[FAIL:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[FAIL]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store8
+
+; CHECK:      __asan_check_store16_rn[[RN16]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %r8
+; CHECK-NEXT:         shrq    $3, %r8
+; CHECK-NEXT:         orq     $2147450880, %r8{{.*}}
+; CHECK-NEXT:         cmpw    $0, (%r8)
+; CHECK-NEXT:         jne     [[FAIL:.*]]
+; CHECK-NEXT: [[RET:.*]]:
+; CHECK-NEXT:         retq
+; CHECK-NEXT: [[FAIL]]:
+; CHECK-NEXT:         movq    [[REG:.*]], %rdi
+; CHECK-NEXT:         jmp     __asan_report_store16
+
+declare void @llvm.asan.check.memaccess(i8*, i64 immarg, i32 immarg,
+                                        i32 immarg, i32 immarg, i32 immarg)