[HWASan] Add basic stack tagging support for LAM.

[morehouse]

Summary

Adds the basic instrumentation needed for stack tagging.

Currently does not support stack short granules or TLS stack histories,
since a different code path is followed for the callback instrumentation
we use.

We may simply wait to support these two features until we switch to
a custom calling convention.

[morehouse]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

190

Can we avoid creating ClUntagPointer for now? I am able to test locally with QEMU, and I'm also setting up a buildbot to ensure new patches don't break the LAM functionality.

192

Can we wire ClUsePageAlias up to fsanitize-hwaddress-experimental-aliasing in this patch? Then we can make ClUsePageAlias default to false and can remove this TODO.

193–194

269

Since getTargetTagShift and getTargetTagMask are used only during initializeModule, and since they are both tiny functions, let's remove them and inline the logic in initializeModule.

271

Since this is no longer constexpr, let's remove the k prefix.

502–509

For now we want to keep InstrumentWithCalls == true for x86 in general, even in LAM mode (no aliases). Please update this line.

556–558

We also want to avoid instrumenting globals for now, so please update this line to apply to x86 in general.

719

Please keep this check here for now. I want to avoid reintroducing untagging to x86.

983

The function name is confusing. Can we name it something like applyTagMask?

1004

This mask of the base tag seems unnecessary since we also mask the final tag.

1027

It seems like overkill to move the logic into this function. Let's leave it as part of tagPointer.

1034

Let's avoid implementing the kernel case for LAM, since we don't have a good way to test yet.

1501

Let's avoid doing this for now. I'd like to focus just on stack support in this patch. Future patches can address global support.

llvm/test/Instrumentation/HWAddressSanitizer/X86/stack.ll

5 ↗

(On Diff #346931)

Instead of making a new test, can we reuse the alloca tests for aarch64 and update the expected instrumentation?

[xiangzhangllvm]

Hello @morehouse, thanks very much for your careful review, I'll fix all your comments, just for "ClUntagPointer", I feel it very useful in my develop.

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

190

Yes, of course, we can. But here ClUntagPointer is disabled in default (false), no affect our current tests.
I and back this option, because I find It is very helpful for me in developing HWASAN without hardware or simulator supported. I can run most simple test on my local machine.

1004

make sense, it seem will handle at retagTargetTag fucntion.

[morehouse]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

190

It may be inconvenient for now, but as part of setting up a buildbot I'm also automating the process of setting up the custom QEMU and kernel required to test LAM. If all goes well, it should land later this week or next week. Then you can use that instead of the ClUntagPointer option.

For now, could you keep the ClUntagPointer patch locally for your testing purposes, and remove it from the patch we actually submit?

[xiangzhangllvm]

NO problem, Thanks!

[xiangzhangllvm]

Sorry for late update, a little busy in these 2 month : )

[morehouse]

Thanks for the update. I'll take a closer look tomorrow and try testing locally. You can also use the new buildbot script to test in QEMU (see inline comment below).

@eugenis @vitalybuka If you could take a look as well that would be helpful. I'm not too familiar with the stack retagging instrumentation.

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

190

Thanks. FYI, the buildbot is now live: https://lab.llvm.org/buildbot/#/builders/169

You can test locally using the buildbot script here. You can comment out the Scudo stuff to avoid testing that.

You'll need a QEMU image located at /b/qemu_image. You can build one using this script.

193–194

This comment wasn't addressed.

llvm/test/Instrumentation/HWAddressSanitizer/X86/stack.ll

5 ↗

(On Diff #346931)

I see this test is gone now, but I don't see any alloca tests copied into the X86 directory. Maybe you forgot to add them in the patch?

[xiangzhangllvm]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

190

Thanks for your work! let me do a try.

193–194

Oh, yes, I should use "hwasan-experimental-use-page-aliases", thank you very much!

llvm/test/Instrumentation/HWAddressSanitizer/X86/stack.ll

5 ↗

(On Diff #346931)

Did you mean the llvm/test/Instrumentation/HWAddressSanitizer/alloca.ll ?
It already at the parent directory of this test (llvm/test/Instrumentation/HWAddressSanitizer/X86)
So, the check-all can test it.

[morehouse]

llvm/test/Instrumentation/HWAddressSanitizer/X86/stack.ll

5 ↗

(On Diff #346931)

I meant that one and the other tests with alloca in the name. Those are testing instrumentation for aarch64. I want to also check the instrumentation for them on x86, since we have differences between the architectures.

[xiangzhangllvm]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

190

How can I run it online for this patch ? (it seems jobs in https://lab.llvm.org/buildbot/#/builders/169 are on line )

(For locally test, I can't download the QEMU in my company's server, for network permission and without root authority)

[morehouse]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

190

Can't run it online unless its been submitted.

Sounds like your setup isn't well suited for working on LAM support. I was planning to work on stack and globals support anyway, so perhaps I will commandeer these patches and work on them myself.

[eugenis]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

969

retagMaskX86

You could also make this non-static and get rid of the second argument. Or apply the mask in retagMask.

1004

It seems more efficient to mask StackTag, and then make sure that all retag constants are within the mask bits. This way you'll only need to apply the mask once per function.

1022

Why not simply replace 0xFF here with TagMaskByte?

[xiangzhangllvm]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

190

OK, no problem, feel free to use this patch, Let co-push the HWASAN work.
I'll continue to refine it (finish you and eugenis's comments).

1022

Because "xor" will let Tag > TagMask again : )

[morehouse]

I've addressed most comments locally; commandeering to prevent wasting Xiang's time addressing the feedback.

Will upload with tests tomorrow.

[morehouse]

• Rename flag as experimental.
• Refactor and simplify code.
• Apply mask to base tag only.
• Enable some compiler-rt stack tests on x86.
• Add IR tests.

[morehouse]

• Privatize new member variables.

[vitalybuka]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

507

As is it's not technically correct

525

maybe move up and combine with other IsX86 checks about

962

const

[morehouse]

• - Address nits.

[morehouse]

• Fix clang test failure on Windows.

clang/lib/Driver/SanitizerArgs.cpp

if (!TsanFuncEntryExit) {
     CmdArgs.push_back("-mllvm");
     CmdArgs.push_back("-tsan-instrument-func-entry-exit=0");
   }
   if (!TsanAtomics) {
     CmdArgs.push_back("-mllvm");
     CmdArgs.push_back("-tsan-instrument-atomics=0");
   }
 
+  if (HwasanUseAliases) {
+    CmdArgs.push_back("-mllvm");
+    CmdArgs.push_back("-hwasan-experimental-use-page-aliases=1");
+  }
+
   if (CfiCrossDso)
     CmdArgs.push_back("-fsanitize-cfi-cross-dso");
 
   if (CfiICallGeneralizePointers)
     CmdArgs.push_back("-fsanitize-cfi-icall-generalize-pointers");
 
   if (CfiCanonicalJumpTables)
     CmdArgs.push_back("-fsanitize-cfi-canonical-jump-tables");

clang/test/Driver/fsanitize.c

 // RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-HWASAN-INTERCEPTOR-ABI
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress -fsanitize-hwaddress-abi=interceptor %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-HWASAN-INTERCEPTOR-ABI
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress -fsanitize-hwaddress-abi=platform %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-HWASAN-PLATFORM-ABI
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress -fsanitize-hwaddress-abi=foo %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-HWASAN-FOO-ABI
 // CHECK-HWASAN-INTERCEPTOR-ABI: "-default-function-attr" "hwasan-abi=interceptor"
 // CHECK-HWASAN-PLATFORM-ABI: "-default-function-attr" "hwasan-abi=platform"
 // CHECK-HWASAN-FOO-ABI: error: invalid value 'foo' in '-fsanitize-hwaddress-abi=foo'
 
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress -fsanitize-hwaddress-experimental-aliasing %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-HWASAN-ALIAS
+// CHECK-HWASAN-ALIAS: "-mllvm" "-hwasan-experimental-use-page-aliases=1"
+
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=address,pointer-compare,pointer-subtract %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-POINTER-ALL
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=pointer-compare %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-POINTER-CMP-NEEDS-ADDRESS
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=pointer-subtract %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-POINTER-SUB-NEEDS-ADDRESS
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=pointer-subtract -fno-sanitize=pointer-subtract %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-NO-POINTER-SUB
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=pointer-compare -fno-sanitize=pointer-compare %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-NO-POINTER-CMP
 // CHECK-POINTER-ALL: -cc1{{.*}}-fsanitize={{[^"]*}}pointer-compare,pointer-subtract{{.*}}" {{.*}} "-mllvm" "-asan-detect-invalid-pointer-cmp" {{.*}}"-mllvm" "-asan-detect-invalid-pointer-sub"
 // CHECK-POINTER-CMP-NEEDS-ADDRESS: error: invalid argument '-fsanitize=pointer-compare' only allowed with '-fsanitize=address'
 // CHECK-POINTER-SUB-NEEDS-ADDRESS: error: invalid argument '-fsanitize=pointer-subtract' only allowed with '-fsanitize=address'

compiler-rt/test/hwasan/TestCases/deep-recursion.c

 // RUN: %clang_hwasan -O1 %s -o %t
 // RUN: %env_hwasan_opts=stack_history_size=1 not %run %t 2>&1 | FileCheck %s --check-prefix=D1
 // RUN: %env_hwasan_opts=stack_history_size=2 not %run %t 2>&1 | FileCheck %s --check-prefix=D2
 // RUN: %env_hwasan_opts=stack_history_size=3 not %run %t 2>&1 | FileCheck %s --check-prefix=D3
 // RUN: %env_hwasan_opts=stack_history_size=5 not %run %t 2>&1 | FileCheck %s --check-prefix=D5
 // RUN:                                       not %run %t 2>&1 | FileCheck %s --check-prefix=DEFAULT
 
 // REQUIRES: stable-runtime
 
-// Stack aliasing is not implemented on x86.
+// Stack histories are currently not recorded on x86.
 // XFAIL: x86_64
 
 #include <stdlib.h>
 // At least -O1 is needed for this function to not have a stack frame on
 // AArch64.
 void USE(void *x) { // pretend_to_do_something(void *x)
   __asm__ __volatile__("" : : "r" (x) : "memory");
 }

compiler-rt/test/hwasan/TestCases/longjmp.c

 // RUN: %clang_hwasan -O0 -DNEGATIVE %s -o %t && %run %t 2>&1
 // RUN: %clang_hwasan -O0 %s -o %t && not %run %t 2>&1 | FileCheck %s
 
-// REQUIRES: stable-runtime
-
-// Stack aliasing is not implemented on x86.
-// XFAIL: x86_64
+// REQUIRES: stable-runtime, pointer-tagging
 
 #include <stdlib.h>
 #include <assert.h>
 #include <sanitizer/hwasan_interface.h>
 
 __attribute__((noinline))
 int f(void *caller_frame) {
   int z = 0;

compiler-rt/test/hwasan/TestCases/mem-intrinsics.c

 // RUN: %clang_hwasan %s -DTEST_NO=1 -mllvm -hwasan-instrument-mem-intrinsics -o %t && not %run %t 2>&1 | FileCheck %s --check-prefix=WRITE
 // RUN: %clang_hwasan %s -DTEST_NO=2 -mllvm -hwasan-instrument-mem-intrinsics -o %t && not %run %t 2>&1 | FileCheck %s --check-prefix=READ
 // RUN: %clang_hwasan %s -DTEST_NO=3 -mllvm -hwasan-instrument-mem-intrinsics -o %t && not %run %t 2>&1 | FileCheck %s --check-prefix=WRITE
 // RUN: %clang_hwasan %s -DTEST_NO=2 -mllvm -hwasan-instrument-mem-intrinsics -o %t && not %env_hwasan_opts=halt_on_error=0 %run %t 2>&1 | FileCheck %s --check-prefix=RECOVER
 
-// REQUIRES: stable-runtime
-
-// Stack aliasing is not implemented on x86.
-// XFAIL: x86_64
+// REQUIRES: stable-runtime, pointer-tagging
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <unistd.h>
 
 int main() {
   char Q[16] __attribute__((aligned(256)));
   char P[16] __attribute__((aligned(256)));

compiler-rt/test/hwasan/TestCases/rich-stack.c

 // Test how stack frames are reported (not fully implemented yet).
 // RUN: %clang_hwasan %s -o %t
 // RUN: not %run %t 3 2 -1 2>&1 | FileCheck %s --check-prefix=R321
-// REQUIRES: stable-runtime
-
-// Stack aliasing is not implemented on x86.
-// XFAIL: x86_64
+// REQUIRES: stable-runtime, pointer-tagging
 
 #include <stdint.h>
 #include <stdlib.h>
 void USE(void *x) { // pretend_to_do_something(void *x)
   __asm__ __volatile__("" : : "r" (x) : "memory");
 }
 void USE2(void *a, void *b) { USE(a); USE(b); }
 void USE4(void *a, void *b, void *c, void *d) { USE2(a, b); USE2(c, d); }
int main(int argc, char **argv) {
   int err_depth = atoi(argv[2]);
   int offset = atoi(argv[3]);
   FOO(depth, err_depth, offset);
   return 0;
 }
 
 // R321: HWAddressSanitizer: tag-mismatch
 // R321-NEXT: WRITE of size 8
-// R321-NEXT: in BAR
+// R321: in BAR
 // R321-NEXT: in FOO
 // R321-NEXT: in main
 // R321: is located in stack of thread T0

compiler-rt/test/hwasan/TestCases/stack-history-length.c

 // RUN: %clang_hwasan -O1 %s -o %t
 // RUN: %env_hwasan_opts=stack_history_size=2048 not %run %t 2045 2>&1 | FileCheck %s --check-prefix=YES
 // RUN: %env_hwasan_opts=stack_history_size=2048 not %run %t 2047 2>&1 | FileCheck %s --check-prefix=NO
 
 // REQUIRES: stable-runtime
 
-// Stack aliasing is not implemented on x86.
+// Stack histories are currently not recorded on x86.
 // XFAIL: x86_64
 
 #include <stdlib.h>
 
 void USE(void *x) { // pretend_to_do_something(void *x)
   __asm__ __volatile__("" : : "r" (x) : "memory");
 }
 

compiler-rt/test/hwasan/TestCases/stack-oob.c

 // RUN: %clang_hwasan_oldrt -DSIZE=2 -O0 %s -o %t && %run %t
 // RUN: %clang_hwasan -DSIZE=2 -O0 %s -o %t && not %run %t 2>&1 | FileCheck %s
 // RUN: %clang_hwasan_oldrt -DSIZE=15 -O0 %s -o %t && %run %t
 // RUN: %clang_hwasan -DSIZE=15 -O0 %s -o %t && not %run %t 2>&1 | FileCheck %s
 // RUN: %clang_hwasan_oldrt -DSIZE=16 -O0 %s -o %t && not %run %t 2>&1 | FileCheck %s
 // RUN: %clang_hwasan -DSIZE=16 -O0 %s -o %t && not %run %t 2>&1 | FileCheck %s
 // RUN: %clang_hwasan -DSIZE=64 -O0 %s -o %t && not %run %t 2>&1 | FileCheck %s
 // RUN: %clang_hwasan -DSIZE=0x1000 -O0 %s -o %t && not %run %t 2>&1 | FileCheck %s
 
 // REQUIRES: stable-runtime
 
-// Stack aliasing is not implemented on x86.
+// Stack short granules are currently not implemented on x86.
 // XFAIL: x86_64
 
 #include <stdlib.h>
 #include <sanitizer/hwasan_interface.h>
 
 __attribute__((noinline))
 int f() {
   char z[SIZE];

compiler-rt/test/hwasan/TestCases/stack-uar-dynamic.c

 // RUN: %clang_hwasan -g %s -o %t && not %run %t 2>&1 | FileCheck %s
 
 // Dynamic allocation of stack objects does not affect FP, so the backend should
 // still be using FP-relative debug info locations that we can use to find stack
 // objects.
 
-// Stack aliasing is not implemented on x86.
+// Stack histories are currently not recorded on x86.
 // XFAIL: x86_64
 
 __attribute((noinline))
 char *buggy(int b) {
   char c[64];
   char *volatile p = c;
   if (b) {
     p = __builtin_alloca(64);

compiler-rt/test/hwasan/TestCases/stack-uar-realign.c

 // RUN: %clang_hwasan -g %s -o %t && not %run %t 2>&1 | FileCheck %s
 
 // Dynamic stack realignment causes debug info locations to use non-FP-relative
 // offsets because stack frames are realigned below FP, which means that we
 // can't associate addresses with stack objects in this case. Ideally we should
 // be able to handle this case somehow (e.g. by using a different register for
 // DW_AT_frame_base) but at least we shouldn't get confused by it.
 
-// Stack aliasing is not implemented on x86.
-// XFAIL: x86_64
+// REQUIRES: pointer-tagging
 
 __attribute((noinline))
 char *buggy() {
   _Alignas(64) char c[64];
   char *volatile p = c;
   return p;
 }
 
 int main() {
   char *p = buggy();
   // CHECK-NOT: Potentially referenced stack objects:
   p[0] = 0;
 }

compiler-rt/test/hwasan/TestCases/stack-uar.c

 // Tests use-after-return detection and reporting.
 // RUN: %clang_hwasan -g %s -o %t && not %run %t 2>&1 | FileCheck %s
 // RUN: %clang_hwasan -g %s -o %t && not %env_hwasan_opts=symbolize=0 %run %t 2>&1 | FileCheck %s --check-prefix=NOSYM
 
 // REQUIRES: stable-runtime
 
-// Stack aliasing is not implemented on x86.
+// Stack histories currently are not recorded on x86.
 // XFAIL: x86_64
 
 void USE(void *x) { // pretend_to_do_something(void *x)
   __asm__ __volatile__("" : : "r" (x) : "memory");
 }
 
 __attribute__((noinline))
 char *buggy() {

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

const char kHwasanShadowMemoryDynamicAddress[] =
     "__hwasan_shadow_memory_dynamic_address";
 
 // Accesses sizes are powers of two: 1, 2, 4, 8, 16.
 static const size_t kNumberOfAccessSizes = 5;
 
 static const size_t kDefaultShadowScale = 4;
 static const uint64_t kDynamicShadowSentinel =
     std::numeric_limits<uint64_t>::max();
-static const unsigned kPointerTagShift = 56;
 
 static const unsigned kShadowBaseAlignment = 32;
 
 static cl::opt<std::string>
     ClMemoryAccessCallbackPrefix("hwasan-memory-access-callback-prefix",
                                  cl::desc("Prefix for memory access callbacks"),
                                  cl::Hidden, cl::init("__hwasan_"));
 
static cl::opt<bool> ClInstrumentPersonalityFunctions(
     "hwasan-instrument-personality-functions",
     cl::desc("instrument personality functions"), cl::Hidden, cl::init(false),
     cl::ZeroOrMore);
 
 static cl::opt<bool> ClInlineAllChecks("hwasan-inline-all-checks",
                                        cl::desc("inline all checks"),
                                        cl::Hidden, cl::init(false));
 
+// Enabled from clang by "-fsanitize-hwaddress-experimental-aliasing".
+static cl::opt<bool> ClUsePageAliases("hwasan-experimental-use-page-aliases",
+                                      cl::desc("Use page aliasing in HWASan"),
+                                      cl::Hidden, cl::init(false));
+
 namespace {
 
 /// An instrumentation pass implementing detection of addressability bugs
 /// using tagged pointers.
 class HWAddressSanitizer {
 public:
   explicit HWAddressSanitizer(Module &M, bool CompileKernel = false,
                               bool Recover = false)
       : M(M) {
     this->Recover = ClRecover.getNumOccurrences() > 0 ? ClRecover : Recover;
public:
   bool instrumentLandingPads(SmallVectorImpl<Instruction *> &RetVec);
   Value *getNextTagWithCall(IRBuilder<> &IRB);
   Value *getStackBaseTag(IRBuilder<> &IRB);
   Value *getAllocaTag(IRBuilder<> &IRB, Value *StackTag, AllocaInst *AI,
                       unsigned AllocaNo);
   Value *getUARTag(IRBuilder<> &IRB, Value *StackTag);
 
   Value *getHwasanThreadSlotPtr(IRBuilder<> &IRB, Type *Ty);
+  Value *applyTagMask(IRBuilder<> &IRB, Value *OldTag);
+  unsigned retagMask(unsigned AllocaNo);
+
   void emitPrologue(IRBuilder<> &IRB, bool WithFrameRecord);
 
   void instrumentGlobal(GlobalVariable *GV, uint8_t Tag);
   void instrumentGlobals();
 
   void instrumentPersonalityFunctions();
 
 private:
   LLVMContext *C;
   Module &M;
   Triple TargetTriple;
   FunctionCallee HWAsanMemmove, HWAsanMemcpy, HWAsanMemset;
   FunctionCallee HWAsanHandleVfork;
 
   /// This struct defines the shadow mapping using the rule:
   ///   shadow = (mem >> Scale) + Offset.
   /// If InGlobal is true, then
   ///   extern char __hwasan_shadow[];
   ///   shadow = (mem >> Scale) + &__hwasan_shadow
   /// If InTls is true, then
   ///   extern char *__hwasan_tls;
   ///   shadow = (mem>>Scale) + align_up(__hwasan_shadow, kShadowBaseAlignment)
   ///
   /// If WithFrameRecord is true, then __hwasan_tls will be used to access the
   /// ring buffer for storing stack allocations on targets that support it.
   struct ShadowMapping {
     int Scale;
     uint64_t Offset;
private:
   bool InstrumentLandingPads;
   bool InstrumentWithCalls;
   bool InstrumentStack;
   bool UsePageAliases;
 
   bool HasMatchAllTag = false;
   uint8_t MatchAllTag = 0;
 
+  unsigned PointerTagShift;
+  uint64_t TagMaskByte;
+
   Function *HwasanCtorFunction;
 
   FunctionCallee HwasanMemoryAccessCallback[2][kNumberOfAccessSizes];
   FunctionCallee HwasanMemoryAccessCallbackSized[2];
 
   FunctionCallee HwasanTagMemoryFunc;
   FunctionCallee HwasanGenerateTagFunc;
 

 ///
 /// inserts a call to __hwasan_init to the module's constructor list.
 void HWAddressSanitizer::initializeModule() {
   LLVM_DEBUG(dbgs() << "Init " << M.getName() << "\n");
   auto &DL = M.getDataLayout();
 
   TargetTriple = Triple(M.getTargetTriple());
 
-  // x86_64 uses userspace pointer aliases, currently heap-only with callback
-  // instrumentation only.
-  UsePageAliases = TargetTriple.getArch() == Triple::x86_64;
-  InstrumentWithCalls = UsePageAliases ? true : ClInstrumentWithCalls;
+  // x86_64 currently has two modes:
+  // - Intel LAM (default)
+  // - pointer aliasing
+  // Pointer aliasing mode is heap only.  LAM mode is heap+stack, with support
+  // planned for globals as well.
+  bool IsX86_64 = TargetTriple.getArch() == Triple::x86_64;
+  UsePageAliases = ClUsePageAliases && IsX86_64;
+  InstrumentWithCalls = IsX86_64 ? true : ClInstrumentWithCalls;
   InstrumentStack = UsePageAliases ? false : ClInstrumentStack;
+  PointerTagShift = IsX86_64 ? 57 : 56;
+  TagMaskByte = IsX86_64 ? 0x3F : 0xFF;
 
   Mapping.init(TargetTriple, InstrumentWithCalls);
 
   C = &(M.getContext());
   IRBuilder<> IRB(*C);
   IntptrTy = IRB.getIntPtrTy(DL);
   Int8PtrTy = IRB.getInt8PtrTy();
   Int8Ty = IRB.getInt8Ty();
   Int32Ty = IRB.getInt32Ty();
 
   HwasanCtorFunction = nullptr;
 
   // Older versions of Android do not have the required runtime support for
   // short granules, global or personality function instrumentation. On other
   // platforms we currently require using the latest version of the runtime.
   bool NewRuntime =
       !TargetTriple.isAndroid() || !TargetTriple.isAndroidVersionLT(30);
 
   UseShortGranules =
       ClUseShortGranules.getNumOccurrences() ? ClUseShortGranules : NewRuntime;
   OutlinedChecks =
void HWAddressSanitizer::initializeModule() {
   InstrumentLandingPads = ClInstrumentLandingPads.getNumOccurrences()
                               ? ClInstrumentLandingPads
                               : !NewRuntime;
 
   if (!CompileKernel) {
     createHwasanCtorComdat();
     bool InstrumentGlobals =
         ClGlobals.getNumOccurrences() ? ClGlobals : NewRuntime;
-    if (InstrumentGlobals && !UsePageAliases)
+
+    // TODO: Support globals for x86_64 in non-aliasing mode.
+    if (InstrumentGlobals && !IsX86_64)
       instrumentGlobals();
 
     bool InstrumentPersonalityFunctions =
         ClInstrumentPersonalityFunctions.getNumOccurrences()
             ? ClInstrumentPersonalityFunctions
             : NewRuntime;
     if (InstrumentPersonalityFunctions)
       instrumentPersonalityFunctions();

 static size_t TypeSizeToSizeIndex(uint32_t TypeSize) {
   size_t Res = countTrailingZeros(TypeSize / 8);
   assert(Res < kNumberOfAccessSizes);
   return Res;
 }
 
 void HWAddressSanitizer::untagPointerOperand(Instruction *I, Value *Addr) {
   if (TargetTriple.isAArch64() || TargetTriple.getArch() == Triple::x86_64)
     return;
 
   IRBuilder<> IRB(I);
   Value *AddrLong = IRB.CreatePointerCast(Addr, IntptrTy);
   Value *UntaggedPtr =
       IRB.CreateIntToPtr(untagPointer(IRB, AddrLong), Addr->getType());
   I->setOperand(getPointerOperandIndex(I), UntaggedPtr);
 }
 
IRB.CreateCall(Intrinsic::getDeclaration(
                        M, UseShortGranules
                               ? Intrinsic::hwasan_check_memaccess_shortgranules
                               : Intrinsic::hwasan_check_memaccess),
                    {ShadowBase, Ptr, ConstantInt::get(Int32Ty, AccessInfo)});
     return;
   }
 
   Value *PtrLong = IRB.CreatePointerCast(Ptr, IntptrTy);
-  Value *PtrTag = IRB.CreateTrunc(IRB.CreateLShr(PtrLong, kPointerTagShift),
+  Value *PtrTag = IRB.CreateTrunc(IRB.CreateLShr(PtrLong, PointerTagShift),
                                   IRB.getInt8Ty());
   Value *AddrLong = untagPointer(IRB, PtrLong);
   Value *Shadow = memToShadow(AddrLong, IRB);
   Value *MemTag = IRB.CreateLoad(Int8Ty, Shadow);
   Value *TagMismatch = IRB.CreateICmpNE(PtrTag, MemTag);
 
   if (HasMatchAllTag) {
     Value *TagNotIgnored = IRB.CreateICmpNE(
if (Size != AlignedSize) {
       IRB.CreateStore(JustTag, IRB.CreateConstGEP1_32(
                                    Int8Ty, IRB.CreateBitCast(AI, Int8PtrTy),
                                    AlignedSize - 1));
     }
   }
   return true;
 }
 
-static unsigned RetagMask(unsigned AllocaNo) {
+unsigned HWAddressSanitizer::retagMask(unsigned AllocaNo) {
+  if (TargetTriple.getArch() == Triple::x86_64)
+    return AllocaNo & TagMaskByte;
+
   // A list of 8-bit numbers that have at most one run of non-zero bits.
   // x = x ^ (mask << 56) can be encoded as a single armv8 instruction for these
   // masks.
   // The list does not include the value 255, which is used for UAR.
   //
   // Because we are more likely to use earlier elements of this list than later
   // ones, it is sorted in increasing order of probability of collision with a
   // mask allocated (temporally) nearby. The program that generated this list
   // can be found at:
   // https://github.com/google/sanitizers/blob/master/hwaddress-sanitizer/sort_masks.py
   static unsigned FastMasks[] = {0,  128, 64,  192, 32,  96,  224, 112, 240,
                                  48, 16,  120, 248, 56,  24,  8,   124, 252,
                                  60, 28,  12,  4,   126, 254, 62,  30,  14,
                                  6,  2,   127, 63,  31,  15,  7,   3,   1};
   return FastMasks[AllocaNo % (sizeof(FastMasks) / sizeof(FastMasks[0]))];
 }
 
+Value *HWAddressSanitizer::applyTagMask(IRBuilder<> &IRB, Value *OldTag) {
+  if (TargetTriple.getArch() == Triple::x86_64) {
+    Constant *TagMask = ConstantInt::get(IntptrTy, TagMaskByte);
+    Value *NewTag = IRB.CreateAnd(OldTag, TagMask);
+    return NewTag;
+  }
+  // aarch64 uses 8-bit tags, so no mask is needed.
+  return OldTag;
+}
+
 Value *HWAddressSanitizer::getNextTagWithCall(IRBuilder<> &IRB) {
   return IRB.CreateZExt(IRB.CreateCall(HwasanGenerateTagFunc), IntptrTy);
 }
 
 Value *HWAddressSanitizer::getStackBaseTag(IRBuilder<> &IRB) {
   if (ClGenerateTagsWithCalls)
     return getNextTagWithCall(IRB);
   if (StackBaseTag)
     return StackBaseTag;
   // FIXME: use addressofreturnaddress (but implement it in aarch64 backend
   // first).
   Module *M = IRB.GetInsertBlock()->getParent()->getParent();
   auto GetStackPointerFn = Intrinsic::getDeclaration(
       M, Intrinsic::frameaddress,
       IRB.getInt8PtrTy(M->getDataLayout().getAllocaAddrSpace()));
   Value *StackPointer = IRB.CreateCall(
       GetStackPointerFn, {Constant::getNullValue(IRB.getInt32Ty())});
 
   // Extract some entropy from the stack pointer for the tags.
   // Take bits 20..28 (ASLR entropy) and xor with bits 0..8 (these differ
   // between functions).
   Value *StackPointerLong = IRB.CreatePointerCast(StackPointer, IntptrTy);
   Value *StackTag =
-      IRB.CreateXor(StackPointerLong, IRB.CreateLShr(StackPointerLong, 20),
-                    "hwasan.stack.base.tag");
+      applyTagMask(IRB, IRB.CreateXor(StackPointerLong,
+                                      IRB.CreateLShr(StackPointerLong, 20)));
+  StackTag->setName("hwasan.stack.base.tag");
   return StackTag;
 }
 
 Value *HWAddressSanitizer::getAllocaTag(IRBuilder<> &IRB, Value *StackTag,
                                         AllocaInst *AI, unsigned AllocaNo) {
   if (ClGenerateTagsWithCalls)
     return getNextTagWithCall(IRB);
   return IRB.CreateXor(StackTag,
-                       ConstantInt::get(IntptrTy, RetagMask(AllocaNo)));
+                       ConstantInt::get(IntptrTy, retagMask(AllocaNo)));
 }
 
 Value *HWAddressSanitizer::getUARTag(IRBuilder<> &IRB, Value *StackTag) {
   if (ClUARRetagToZero)
     return ConstantInt::get(IntptrTy, 0);
   if (ClGenerateTagsWithCalls)
     return getNextTagWithCall(IRB);
-  return IRB.CreateXor(StackTag, ConstantInt::get(IntptrTy, 0xFFU));
+  return IRB.CreateXor(StackTag, ConstantInt::get(IntptrTy, TagMaskByte));
 }
 
 // Add a tag to an address.
 Value *HWAddressSanitizer::tagPointer(IRBuilder<> &IRB, Type *Ty,
                                       Value *PtrLong, Value *Tag) {
   assert(!UsePageAliases);
   Value *TaggedPtrLong;
   if (CompileKernel) {
     // Kernel addresses have 0xFF in the most significant byte.
-    Value *ShiftedTag = IRB.CreateOr(
-        IRB.CreateShl(Tag, kPointerTagShift),
-        ConstantInt::get(IntptrTy, (1ULL << kPointerTagShift) - 1));
+    Value *ShiftedTag =
+        IRB.CreateOr(IRB.CreateShl(Tag, PointerTagShift),
+                     ConstantInt::get(IntptrTy, (1ULL << PointerTagShift) - 1));
     TaggedPtrLong = IRB.CreateAnd(PtrLong, ShiftedTag);
   } else {
-    // Userspace can simply do OR (tag << 56);
-    Value *ShiftedTag = IRB.CreateShl(Tag, kPointerTagShift);
+    // Userspace can simply do OR (tag << PointerTagShift);
+    Value *ShiftedTag = IRB.CreateShl(Tag, PointerTagShift);
     TaggedPtrLong = IRB.CreateOr(PtrLong, ShiftedTag);
   }
   return IRB.CreateIntToPtr(TaggedPtrLong, Ty);
 }
 
 // Remove tag from an address.
 Value *HWAddressSanitizer::untagPointer(IRBuilder<> &IRB, Value *PtrLong) {
   assert(!UsePageAliases);
   Value *UntaggedPtrLong;
   if (CompileKernel) {
     // Kernel addresses have 0xFF in the most significant byte.
     UntaggedPtrLong =
         IRB.CreateOr(PtrLong, ConstantInt::get(PtrLong->getType(),
-                                               0xFFULL << kPointerTagShift));
+                                               0xFFULL << PointerTagShift));
   } else {
     // Userspace addresses have 0x00.
-    UntaggedPtrLong = IRB.CreateAnd(
-        PtrLong,
-        ConstantInt::get(PtrLong->getType(), ~(0xFFULL << kPointerTagShift)));
+    UntaggedPtrLong =
+        IRB.CreateAnd(PtrLong, ConstantInt::get(PtrLong->getType(),
+                                                ~(0xFFULL << PointerTagShift)));
   }
   return UntaggedPtrLong;
 }
 
 Value *HWAddressSanitizer::getHwasanThreadSlotPtr(IRBuilder<> &IRB, Type *Ty) {
   Module *M = IRB.GetInsertBlock()->getParent()->getParent();
   if (TargetTriple.isAArch64() && TargetTriple.isAndroid()) {
     // Android provides a fixed TLS slot for sanitizers. See TLS_SLOT_SANITIZER
for (unsigned N = 0; N < Allocas.size(); ++N) {
     AI->replaceUsesWithIf(Replacement,
                           [AILong](Use &U) { return U.getUser() != AILong; });
 
     for (auto *DDI : AllocaDbgMap.lookup(AI)) {
       // Prepend "tag_offset, N" to the dwarf expression.
       // Tag offset logically applies to the alloca pointer, and it makes sense
       // to put it at the beginning of the expression.
       SmallVector<uint64_t, 8> NewOps = {dwarf::DW_OP_LLVM_tag_offset,
-                                         RetagMask(N)};
+                                         retagMask(N)};
       auto Locations = DDI->location_ops();
       unsigned LocNo = std::distance(Locations.begin(), find(Locations, AI));
       DDI->setExpression(
           DIExpression::appendOpsToArg(DDI->getExpression(), NewOps, LocNo));
     }
 
     size_t Size = getAllocaSizeInBytes(*AI);
     tagAlloca(IRB, AI, Tag, Size);
for (uint64_t DescriptorPos = 0; DescriptorPos < SizeInBytes;
     Descriptor->setMetadata(LLVMContext::MD_associated,
                             MDNode::get(*C, ValueAsMetadata::get(NewGV)));
     appendToCompilerUsed(M, Descriptor);
   }
 
   Constant *Aliasee = ConstantExpr::getIntToPtr(
       ConstantExpr::getAdd(
           ConstantExpr::getPtrToInt(NewGV, Int64Ty),
-          ConstantInt::get(Int64Ty, uint64_t(Tag) << kPointerTagShift)),
+          ConstantInt::get(Int64Ty, uint64_t(Tag) << PointerTagShift)),
       GV->getType());
   auto *Alias = GlobalAlias::create(GV->getValueType(), GV->getAddressSpace(),
                                     GV->getLinkage(), "", Aliasee, &M);
   Alias->setVisibility(GV->getVisibility());
   Alias->takeName(GV);
   GV->replaceAllUsesWith(Alias);
   GV->eraseFromParent();
 }
void HWAddressSanitizer::instrumentGlobals() {
   MD5 Hasher;
   Hasher.update(M.getSourceFileName());
   MD5::MD5Result Hash;
   Hasher.final(Hash);
   uint8_t Tag = Hash[0];
 
   for (GlobalVariable *GV : Globals) {
     // Skip tag 0 in order to avoid collisions with untagged memory.
     if (Tag == 0)
       Tag = 1;
     instrumentGlobal(GV, Tag++);
   }
 }
 
 void HWAddressSanitizer::instrumentPersonalityFunctions() {
   // We need to untag stack frames as we unwind past them. That is the job of
   // the personality function wrapper, which either wraps an existing

llvm/test/Instrumentation/HWAddressSanitizer/X86/alloca-array.ll

+; RUN: opt < %s -hwasan -S | FileCheck %s
+
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @use(i8*, i8*)
+
+define void @test_alloca() sanitize_hwaddress {
+  ; CHECK: alloca { [4 x i8], [12 x i8] }, align 16
+  %x = alloca i8, i64 4
+  ; CHECK: alloca i8, i64 16, align 16
+  %y = alloca i8, i64 16
+  call void @use(i8* %x, i8* %y)
+  ret void
+}

llvm/test/Instrumentation/HWAddressSanitizer/X86/alloca-with-calls.ll

+; Test alloca instrumentation when tags are generated by HWASan function.
+;
+; RUN: opt < %s -hwasan -hwasan-generate-tags-with-calls -S | FileCheck %s
+
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @use32(i32*)
+
+define void @test_alloca() sanitize_hwaddress {
+; CHECK-LABEL: @test_alloca(
+; CHECK: %[[BC:[^ ]*]] = bitcast { i32, [12 x i8] }* %x to i32*
+; CHECK: %[[T1:[^ ]*]] = call i8 @__hwasan_generate_tag()
+; CHECK: %[[A:[^ ]*]] = zext i8 %[[T1]] to i64
+; CHECK: %[[B:[^ ]*]] = ptrtoint i32* %[[BC]] to i64
+; CHECK: %[[C:[^ ]*]] = shl i64 %[[A]], 57
+; CHECK: or i64 %[[B]], %[[C]]
+
+entry:
+  %x = alloca i32, align 4
+  call void @use32(i32* nonnull %x)
+  ret void
+}

llvm/test/Instrumentation/HWAddressSanitizer/X86/alloca.ll

+; Test alloca instrumentation.
+;
+; RUN: opt < %s -hwasan -S | FileCheck %s --check-prefixes=CHECK,NO-UAR-TAGS
+; RUN: opt < %s -hwasan -hwasan-uar-retag-to-zero=0 -S | FileCheck %s --check-prefixes=CHECK,UAR-TAGS
+
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @use32(i32*)
+
+define void @test_alloca() sanitize_hwaddress {
+; CHECK-LABEL: @test_alloca(
+; CHECK: %[[FP:[^ ]*]] = call i8* @llvm.frameaddress.p0i8(i32 0)
+; CHECK: %[[A:[^ ]*]] = ptrtoint i8* %[[FP]] to i64
+; CHECK: %[[B:[^ ]*]] = lshr i64 %[[A]], 20
+; CHECK: %[[A_XOR_B:[^ ]*]] = xor i64 %[[A]], %[[B]]
+; CHECK: %[[BASE_TAG:[^ ]*]] = and i64 %[[A_XOR_B]], 63
+
+; CHECK: %[[X:[^ ]*]] = alloca { i32, [12 x i8] }, align 16
+; CHECK: %[[X_BC:[^ ]*]] = bitcast { i32, [12 x i8] }* %[[X]] to i32*
+; CHECK: %[[X_TAG:[^ ]*]] = xor i64 %[[BASE_TAG]], 0
+; CHECK: %[[X1:[^ ]*]] = ptrtoint i32* %[[X_BC]] to i64
+; CHECK: %[[C:[^ ]*]] = shl i64 %[[X_TAG]], 57
+; CHECK: %[[D:[^ ]*]] = or i64 %[[X1]], %[[C]]
+; CHECK: %[[X_HWASAN:[^ ]*]] = inttoptr i64 %[[D]] to i32*
+
+; CHECK: %[[X_TAG2:[^ ]*]] = trunc i64 %[[X_TAG]] to i8
+; CHECK: %[[X_I8:[^ ]*]] = bitcast i32* %[[X_BC]] to i8*
+; CHECK: call void @__hwasan_tag_memory(i8* %[[X_I8]], i8 %[[X_TAG2]], i64 16)
+
+; CHECK: call void @use32(i32* nonnull %[[X_HWASAN]])
+
+; UAR-TAGS: %[[BASE_TAG_COMPL:[^ ]*]] = xor i64 %[[BASE_TAG]], 63
+; UAR-TAGS: %[[X_TAG_UAR:[^ ]*]] = trunc i64 %[[BASE_TAG_COMPL]] to i8
+; CHECK: %[[X_I8_2:[^ ]*]] = bitcast i32* %[[X_BC]] to i8*
+; NO-UAR-TAGS: call void @__hwasan_tag_memory(i8* %[[X_I8_2]], i8 0, i64 16)
+; UAR-TAGS: call void @__hwasan_tag_memory(i8* %[[X_I8_2]], i8 %[[X_TAG_UAR]], i64 16)
+; CHECK: ret void
+
+
+entry:
+  %x = alloca i32, align 4
+  call void @use32(i32* nonnull %x)
+  ret void
+}