[HWASAN] Instrument memory intrinsics

[evgeny777]

Summary

Patch replaces memory intrinsics with corresponding libc calls when specific option is set. The memset and friends can be either hooked by the runtime or libc itself can be sanitized

The patch lacks test case - I'll implement one if the whole thing makes sense.

[eugenis]

Looks fine to me.

[kcc]

asan replaces memset with __asan_memset (same for memcpy and memmove), see AddressSanitizer::instrumentMemIntrinsic.
Would it be more flexible to use the same approach here?

(and yes, we need to do this, so please proceed with a test)

[evgeny777]

Would it be more flexible to use the same approach here?

Since r340216 HWASAN stopped intercepting common libc routines. As far as I understand the goal of this change is to sanitize libc.
My understanding is that if we do sanitize the whole libc then we don't need special versions of memset and friends in HWASAN runtime.

Even if one doesn't sanitize libc then it is always possible to hook memset/memcpy/memmove like it was previously done with strcpy and stuff.

[evgeny777]

Added test case

[kcc]

We do instrument libc on Android, but

• memset&co are still not instrumented because they are written in asm (eugenis@, please confirm)
• hwasan may be useful outside of Android in future. (we actually test it on arm64 linux, and even on x86_64 linux)

So, I still prefer __hwasan_memset (which means that we will also need a run-time test)
eugenis@, thoughts?

test/Instrumentation/HWAddressSanitizer/mem-intrinsics.ll

3 ↗

(On Diff #177164)

please place the checks after their respective "call void @llvm..." lines

[eugenis]

I think I prefer regular memset to hwasan_memset.
hwasan variants can be slightly faster because they don't need to check that the address is not a shadow address (from stack tagging), but that is probably not a big deal.
Also the hardware implementation will definitely not have (or need) __hwasan_memset.

[kcc]

How will do do the actual checking then, given the memset is implemented in asm?
extra code in libc?

[eugenis]

Yes, I think so.

[kcc]

ok, let's do it this way, I'll leave it to eugenis@ to stamp.
Also, at some point we'll need to flip the flag.

[eugenis]

In fact, I changed my mind - let's do __hwasan_memset.
Tag checking in memset is part of software emulation; as such, it belongs in libclang_rt.hwasan, and not in any system library.

[evgeny777]

In fact, I changed my mind - let's do __hwasan_memset.

What about kernel compilation mode?

[eugenis]

We should do the same thing in kernel mode, I think. Either way memintrinsic checking will require changes on the kernel side. @andreyknvl

[evgeny777]

Addressed review comments. Now the patch uses approach identical to ASAN.
ASAN doesn't add prefix in kernel compilation mode (my guess is that memset and friends are part of Linux kernel and are always instrumented in such scenario)

See also patch to runtime: D55554

[kcc]

LGTM
Next step is to add run-time tests and enable by default

llvm/trunk/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

ClRecordStackHistory("hwasan-record-stack-history",
                          cl::desc("Record stack frames with tagged allocations "
                                   "in a thread-local ring buffer"),
                          cl::Hidden, cl::init(true));
 static cl::opt<bool>
     ClCreateFrameDescriptions("hwasan-create-frame-descriptions",
                               cl::desc("create static frame descriptions"),
                               cl::Hidden, cl::init(true));
 
+static cl::opt<bool>
+    ClInstrumentMemIntrinsics("hwasan-instrument-mem-intrinsics",
+                              cl::desc("instrument memory intrinsics"),
+                              cl::Hidden, cl::init(false));
 namespace {
 
 /// An instrumentation pass implementing detection of addressability bugs
 /// using tagged pointers.
 class HWAddressSanitizer : public FunctionPass {
 public:
   // Pass identification, replacement for typeid.
   static char ID;
public:
 
   Value *getDynamicShadowNonTls(IRBuilder<> &IRB);
 
   void untagPointerOperand(Instruction *I, Value *Addr);
   Value *memToShadow(Value *Shadow, Type *Ty, IRBuilder<> &IRB);
   void instrumentMemAccessInline(Value *PtrLong, bool IsWrite,
                                  unsigned AccessSizeIndex,
                                  Instruction *InsertBefore);
+  void instrumentMemIntrinsic(MemIntrinsic *MI);
   bool instrumentMemAccess(Instruction *I);
   Value *isInterestingMemoryAccess(Instruction *I, bool *IsWrite,
                                    uint64_t *TypeSize, unsigned *Alignment,
                                    Value **MaybeMask);
 
   bool isInterestingAlloca(const AllocaInst &AI);
   bool tagAlloca(IRBuilder<> &IRB, AllocaInst *AI, Value *Tag);
   Value *tagPointer(IRBuilder<> &IRB, Type *Ty, Value *PtrLong, Value *Tag);
   Value *untagPointer(IRBuilder<> &IRB, Value *PtrLong);
   bool instrumentStack(SmallVectorImpl<AllocaInst *> &Allocas,
                        SmallVectorImpl<Instruction *> &RetVec, Value *StackTag);
   Value *getNextTagWithCall(IRBuilder<> &IRB);
   Value *getStackBaseTag(IRBuilder<> &IRB);
   Value *getAllocaTag(IRBuilder<> &IRB, Value *StackTag, AllocaInst *AI,
                      unsigned AllocaNo);
   Value *getUARTag(IRBuilder<> &IRB, Value *StackTag);
 
   Value *getHwasanThreadSlotPtr(IRBuilder<> &IRB, Type *Ty);
   Value *emitPrologue(IRBuilder<> &IRB, bool WithFrameRecord);
 
 private:
   LLVMContext *C;
   std::string CurModuleUniqueId;
   Triple TargetTriple;
+  Function *HWAsanMemmove, *HWAsanMemcpy, *HWAsanMemset;
 
   // Frame description is a way to pass names/sizes of local variables
   // to the run-time w/o adding extra executable code in every function.
   // We do this by creating a separate section with {PC,Descr} pairs and passing
   // the section beg/end to __hwasan_init_frames() at module init time.
   std::string createFrameString(ArrayRef<AllocaInst*> Allocas);
   void createFrameGlobal(Function &F, const std::string &FrameString);
   // Get the section name for frame descriptions. Currently ELF-only.
void HWAddressSanitizer::initializeCallbacks(Module &M) {
   HwasanTagMemoryFunc = checkSanitizerInterfaceFunction(M.getOrInsertFunction(
       "__hwasan_tag_memory", IRB.getVoidTy(), Int8PtrTy, Int8Ty, IntptrTy));
   HwasanGenerateTagFunc = checkSanitizerInterfaceFunction(
       M.getOrInsertFunction("__hwasan_generate_tag", Int8Ty));
 
   if (Mapping.InGlobal)
     ShadowGlobal = M.getOrInsertGlobal("__hwasan_shadow",
                                        ArrayType::get(IRB.getInt8Ty(), 0));
+
+  const std::string MemIntrinCallbackPrefix =
+      CompileKernel ? std::string("") : ClMemoryAccessCallbackPrefix;
+  HWAsanMemmove = checkSanitizerInterfaceFunction(M.getOrInsertFunction(
+      MemIntrinCallbackPrefix + "memmove", IRB.getInt8PtrTy(),
+      IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IntptrTy));
+  HWAsanMemcpy = checkSanitizerInterfaceFunction(M.getOrInsertFunction(
+      MemIntrinCallbackPrefix + "memcpy", IRB.getInt8PtrTy(),
+      IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IntptrTy));
+  HWAsanMemset = checkSanitizerInterfaceFunction(M.getOrInsertFunction(
+      MemIntrinCallbackPrefix + "memset", IRB.getInt8PtrTy(),
+      IRB.getInt8PtrTy(), IRB.getInt32Ty(), IntptrTy));
 }
 
 Value *HWAddressSanitizer::getDynamicShadowNonTls(IRBuilder<> &IRB) {
   // Generate code only when dynamic addressing is needed.
   if (Mapping.Offset != kDynamicShadowSentinel)
     return nullptr;
 
   if (Mapping.InGlobal) {
case Triple::aarch64_be:
           /*hasSideEffects=*/true);
       break;
     default:
       report_fatal_error("unsupported architecture");
   }
   IRB.CreateCall(Asm, PtrLong);
 }
 
+void HWAddressSanitizer::instrumentMemIntrinsic(MemIntrinsic *MI) {
+  IRBuilder<> IRB(MI);
+  if (isa<MemTransferInst>(MI)) {
+    IRB.CreateCall(
+        isa<MemMoveInst>(MI) ? HWAsanMemmove : HWAsanMemcpy,
+        {IRB.CreatePointerCast(MI->getOperand(0), IRB.getInt8PtrTy()),
+         IRB.CreatePointerCast(MI->getOperand(1), IRB.getInt8PtrTy()),
+         IRB.CreateIntCast(MI->getOperand(2), IntptrTy, false)});
+  } else if (isa<MemSetInst>(MI)) {
+    IRB.CreateCall(
+        HWAsanMemset,
+        {IRB.CreatePointerCast(MI->getOperand(0), IRB.getInt8PtrTy()),
+         IRB.CreateIntCast(MI->getOperand(1), IRB.getInt32Ty(), false),
+         IRB.CreateIntCast(MI->getOperand(2), IntptrTy, false)});
+  }
+  MI->eraseFromParent();
+}
+
 bool HWAddressSanitizer::instrumentMemAccess(Instruction *I) {
   LLVM_DEBUG(dbgs() << "Instrumenting: " << *I << "\n");
   bool IsWrite = false;
   unsigned Alignment = 0;
   uint64_t TypeSize = 0;
   Value *MaybeMask = nullptr;
+
+  if (ClInstrumentMemIntrinsics && isa<MemIntrinsic>(I)) {
+    instrumentMemIntrinsic(cast<MemIntrinsic>(I));
+    return true;
+  }
+
   Value *Addr =
       isInterestingMemoryAccess(I, &IsWrite, &TypeSize, &Alignment, &MaybeMask);
 
   if (!Addr)
     return false;
 
   if (MaybeMask)
     return false; //FIXME

llvm/trunk/test/Instrumentation/HWAddressSanitizer/mem-intrinsics.ll

+; RUN: opt -S -hwasan -hwasan-instrument-mem-intrinsics %s | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local i32 @main() sanitize_hwaddress {
+entry:
+  %retval = alloca i32, align 4
+  %Q = alloca [10 x i8], align 1
+  %P = alloca [10 x i8], align 1
+  store i32 0, i32* %retval, align 4
+  %arraydecay = getelementptr inbounds [10 x i8], [10 x i8]* %Q, i32 0, i32 0
+
+  call void @llvm.memset.p0i8.i64(i8* align 1 %arraydecay, i8 0, i64 10, i1 false)
+; CHECK: call i8* @__hwasan_memset
+
+  %arraydecay1 = getelementptr inbounds [10 x i8], [10 x i8]* %Q, i32 0, i32 0
+  %arraydecay2 = getelementptr inbounds [10 x i8], [10 x i8]* %Q, i32 0, i32 0
+  %add.ptr = getelementptr inbounds i8, i8* %arraydecay2, i64 5
+
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %arraydecay1, i8* align 1 %add.ptr, i64 5, i1 false)
+; CHECK: call i8* @__hwasan_memmove
+
+  %arraydecay3 = getelementptr inbounds [10 x i8], [10 x i8]* %P, i32 0, i32 0
+  %arraydecay4 = getelementptr inbounds [10 x i8], [10 x i8]* %Q, i32 0, i32 0
+
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %arraydecay3, i8* align 1 %arraydecay4, i64 10, i1 false)
+; CHECK: call i8* @__hwasan_memcpy
+  ret i32 0
+}
+
+; Function Attrs: argmemonly nounwind
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1) #1
+
+; Function Attrs: argmemonly nounwind
+declare void @llvm.memmove.p0i8.p0i8.i64(i8* nocapture, i8* nocapture readonly, i64, i1) #1
+
+; Function Attrs: argmemonly nounwind
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1) #1