[HWASan] Save + print registers when tag mismatch occurs in AArch64.

[hctim]

Summary

This change change the instrumentation to allow users to view the registers at the point at which tag mismatch occured. Most of the heavy lifting is done in the runtime library, where we save the registers to the stack and emit unwind information. This allows us to reduce the overhead, as very little additional work needs to be done in each __hwasan_check instance.

In this implementation, the fast path of hwasan_check is unmodified. There are an additional 4 instructions (16B) emitted in the slow path in every hwasan_check instance. This may increase binary size somewhat, but as most of the work is done in the runtime library, it's manageable.

The failure trace now contains a list of registers at the point of which the failure occured, in a format similar to that of Android's tombstones. It currently has the following format:

Registers where the failure occurred (pc 0x0055555561b4):

x0  0000000000000014  x1  0000007ffffff6c0  x2  1100007ffffff6d0  x3  12000056ffffe025
x4  0000007fff800000  x5  0000000000000014  x6  0000007fff800000  x7  0000000000000001
x8  12000056ffffe020  x9  0200007700000000  x10 0200007700000000  x11 0000000000000000
x12 0000007fffffdde0  x13 0000000000000000  x14 02b65b01f7a97490  x15 0000000000000000
x16 0000007fb77376b8  x17 0000000000000012  x18 0000007fb7ed6000  x19 0000005555556078
x20 0000007ffffff768  x21 0000007ffffff778  x22 0000000000000001  x23 0000000000000000
x24 0000000000000000  x25 0000000000000000  x26 0000000000000000  x27 0000000000000000
x28 0000000000000000  x29 0000007ffffff6f0  x30 00000055555561b4

... and prints after the dump of memory tags around the buggy address.

Every register is saved exactly as it was at the point where the tag mismatch occurs, with the exception of x16/x17. These registers are used in the tag mismatch calculation as scratch registers during __hwasan_check, and cannot be saved without affecting the fast path. As these registers are designated as scratch registers for linking, there should be no important information in them that could aid in debugging.

[eugenis]

This needs a compiler-rt test.

compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S

47

This does not match the actual file name. Intentional?

62

This does not seem to match stack frame description in the comment above, or the IR test. It looks like x29 offset should be 16, and x30 offset should be 24.

87

Could this use "b" instruction, so that the reporting code does not need to remove a stack frame?

95

use NO_EXEC_STACK_DIRECTIVE

[hctim]

Addressed part of eugenis@ comments. Changed to macro no-exec stack and changed DWARF file name in HWASan '.S' file.

[hctim]

compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S

47

Nope, fixed.

62

My interpretation is the following layout is the current one:

// |              ...                |
// |              ...                |
// | Previous stack frames...        |
// +=================================+ <-- [CFA (x29 + 16)]
// | Return address (x30) for caller |
// | of __hwasan_check_*.            |
// +---------------------------------+ <-- [&x30 == CFA - 8]
// | Frame address (x29) for caller  |
// | of __hwasan_check_*             |
// +---------------------------------+ <-- [x29 / FP]
// | Saved x1, as __hwasan_check_*   |     [&x29 == CFA - 16]
// | clobbers it.                    |
// +---------------------------------+
// | Saved x0, likewise above.       |
// +---------------------------------+ <-- [x30 / SP]

Just stepped through GDB and it looks like the frame locates the saved registers (x29/x30) in the address they're correctly saved in. Do you see a different layout than what I'm seeing?

87

We could, but unfortunately it still sets up a frame for hwasan_tag_mismatch, but misattributes it to the hwasan_check (i.e. we have two entries for __hwasan_check at the bottom of the stack trace).

We could change it to a direct branch and avoid the extra frame by never adjusting the frame pointer, but this means we also lose any debug-ability in __hwasan_tag_mismatch. This would also nullify the comment above regarding CFA correctness.

If you'd like me to make this tradeoff, LMK.

[eugenis]

compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S

62

You are right, I misread the code.

87

Yeah, that would be wrong.

But it looks like now that we have a stack frame, we can easily implement -fsanitize-recover with check outlining. Does not have to be part of the same CL.

[hctim]

Added a register-dump test.

[eugenis]

compiler-rt/lib/hwasan/hwasan_linux.cc

390

check that trace && size > 0 just in case

compiler-rt/test/hwasan/TestCases/register-dump-read.c

5 ↗

(On Diff #189373)

REQUIRES: aarch64-target-arch

21 ↗

(On Diff #189373)

Check that the top frame in the report is as expected, with both settings of fast_unwind_on_fatal.

[hctim]

• Register stack frame fast-forward has additional validity checks.
• Changed register dump test requirements from 'android' to 'aarch64-target-arch'.
• Changed test to check top frame of stack trace for both options of 'fast_unwind_on_fatal'.

[eugenis]

LGTM

[pcc]

compiler-rt/lib/hwasan/CMakeLists.txt

24

Sort alphabetically.

compiler-rt/lib/hwasan/hwasan_linux.cc

417–418

The function name should be moved into the implementation namespace, i.e. prefix with __.

compiler-rt/lib/hwasan/hwasan_report.cc

440

This should match the name of the function.

452

The math here seems a little hard to reason about. Would it be better to store the registers in numerical order so that e.g. x0 is in frame[0], x1 is in frame[1] and so on?

compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S

95

I think this needs to be outside the #if..#endif to prevent the stack from becoming executable on non-aarch64 machines.

compiler-rt/test/hwasan/TestCases/register-dump-read.c

28 ↗

(On Diff #189394)

Is there some way to test that the correct register values are being printed? Maybe one way would be compile this translation unit with a few -ffixed-x* arguments (to prevent them from being clobbered by codegen; this is now possible with r353957), insert inline asm in this function that sets the fixed registers to specific values and test for those values here.

llvm/utils/gn/secondary/compiler-rt/lib/hwasan/BUILD.gn

45

Sort alphabetically.

[hctim]

Addressed pcc's comments.

compiler-rt/lib/hwasan/hwasan_report.cc

452

Have changed to do ordering for x2 - x28, but as x0, x1, x29, and x30 are saved in __hwasan_check they'll have to stay a bit out-of-order.

[pcc]

compiler-rt/lib/hwasan/hwasan_report.cc

452

Not necessarily. I think you could do this in __hwasan_check_*:

stp x0, x1, [sp, #-248]!
stp x29, x30, [sp, #232]

Now all of your stores in __hwasan_tag_mismatch can look like this:

stp xN, xN+1, [sp, #(N * 8)]

[hctim]

compiler-rt/lib/hwasan/hwasan_report.cc

452

Done, except for a minor change to ensure that the stack pointer is 16-byte aligned (see ARM procedure call standard Sec 5.2.2.1.)

[pcc]

compiler-rt/lib/hwasan/hwasan_report.cc

452

Thanks, makes sense.

llvm/test/CodeGen/AArch64/hwasan-check-memaccess.ll

46

I think this test case needs to be updated. Same below.

[hctim]

llvm/test/CodeGen/AArch64/hwasan-check-memaccess.ll

46

Thanks, done.

[pcc]

LGTM

compiler-rt/lib/hwasan/hwasan_linux.cpp

418 ↗

(On Diff #189891)

nit: reformat

compiler-rt/lib/hwasan/CMakeLists.txt

 
 set(HWASAN_RTL_CXX_SOURCES
   hwasan_new_delete.cc)
 
 set(HWASAN_RTL_HEADERS
   hwasan.h
   hwasan_allocator.h
   hwasan_dynamic_shadow.h
+  hwasan_tag_mismatch_aarch64.S
   hwasan_flags.h
   hwasan_flags.inc
   hwasan_interface_internal.h
   hwasan_malloc_bisect.h
   hwasan_mapping.h
   hwasan_poisoning.h
   hwasan_report.h
   hwasan_thread.h

compiler-rt/lib/hwasan/hwasan_interface_internal.h

 
 SANITIZER_INTERFACE_ATTRIBUTE
 void __hwasan_tag_memory(uptr p, u8 tag, uptr sz);
 
 SANITIZER_INTERFACE_ATTRIBUTE
 uptr __hwasan_tag_pointer(uptr p, u8 tag);
 
 SANITIZER_INTERFACE_ATTRIBUTE
+void __hwasan_tag_mismatch(uptr addr, u8 ts);
+
+SANITIZER_INTERFACE_ATTRIBUTE
 u8 __hwasan_generate_tag();
 
 // Returns the offset of the first tag mismatch or -1 if the whole range is
 // good.
 SANITIZER_INTERFACE_ATTRIBUTE
 sptr __hwasan_test_shadow(const void *x, uptr size);
 
 SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE

compiler-rt/lib/hwasan/hwasan_linux.cc

 #else
 # error Unsupported architecture
 #endif
 
   return AccessInfo{addr, size, is_store, !is_store, recover};
 }
 
 static void HandleTagMismatch(AccessInfo ai, uptr pc, uptr frame,
-                              ucontext_t *uc) {
+                              ucontext_t *uc, uptr *registers_frame = nullptr) {
   InternalMmapVector<BufferedStackTrace> stack_buffer(1);
   BufferedStackTrace *stack = stack_buffer.data();
   stack->Reset();
+
   GetStackTrace(stack, kStackTraceMax, pc, frame, uc,
                 common_flags()->fast_unwind_on_fatal);
 
+  // The second stack frame contains the failure __hwasan_check function, as
+  // we have a stack frame for the registers saved in __hwasan_tag_mismatch that
+  // we wish to ignore. This (currently) only occurs on AArch64, as x64
+  // implementations use SIGTRAP to implement the failure, and thus do not go
+  // through the stack saver.
+  if (registers_frame) {
+    stack->trace++;
+    stack->size--;
+  }
+
   bool fatal = flags()->halt_on_error || !ai.recover;
-  ReportTagMismatch(stack, ai.addr, ai.size, ai.is_store, fatal);
+  ReportTagMismatch(stack, ai.addr, ai.size, ai.is_store, fatal,
+                    registers_frame);
 }
 
 static bool HwasanOnSIGTRAP(int signo, siginfo_t *info, ucontext_t *uc) {
   AccessInfo ai = GetAccessInfo(info, uc);
   if (!ai.is_store && !ai.is_load)
     return false;
 
   SignalContext sig{info, uc};
   HandleTagMismatch(ai, StackTrace::GetNextInstructionPc(sig.pc), sig.bp, uc);
 
 #if defined(__aarch64__)
   uc->uc_mcontext.pc += 4;
 #elif defined(__x86_64__)
 #else
 # error Unsupported architecture
 #endif
   return true;
 }
 
-extern "C" SANITIZER_INTERFACE_ATTRIBUTE void __hwasan_tag_mismatch(
-    uptr addr, uptr access_info) {
+extern "C" void HwasanTagMismatch(uptr addr, uptr access_info,
+                                  uptr *registers_frame) {
   AccessInfo ai;
   ai.is_store = access_info & 0x10;
   ai.recover = false;
   ai.addr = addr;
   ai.size = 1 << (access_info & 0xf);
 
   HandleTagMismatch(ai, (uptr)__builtin_return_address(0),
-                    (uptr)__builtin_frame_address(0), nullptr);
+                    (uptr)__builtin_frame_address(0), nullptr, registers_frame);
   __builtin_unreachable();
 }
 
 static void OnStackUnwind(const SignalContext &sig, const void *,
                           BufferedStackTrace *stack) {
   GetStackTrace(stack, kStackTraceMax, StackTrace::GetNextInstructionPc(sig.pc),
                 sig.bp, sig.context, common_flags()->fast_unwind_on_fatal);
 }

compiler-rt/lib/hwasan/hwasan_report.h

 
 #include "sanitizer_common/sanitizer_internal_defs.h"
 #include "sanitizer_common/sanitizer_stacktrace.h"
 
 namespace __hwasan {
 
 void ReportStats();
 void ReportTagMismatch(StackTrace *stack, uptr addr, uptr access_size,
-                       bool is_store, bool fatal);
+                       bool is_store, bool fatal, uptr *registers_frame);
 void ReportInvalidFree(StackTrace *stack, uptr addr);
 void ReportTailOverwritten(StackTrace *stack, uptr addr, uptr orig_size,
                            uptr tail_size, const u8 *expected);
-
+void ReportRegisters(uptr *registers_frame, uptr pc);
 void ReportAtExitStatistics();
 
 
 }  // namespace __hwasan
 
 #endif  // HWASAN_REPORT_H

compiler-rt/lib/hwasan/hwasan_report.cc

 //===-- hwasan_report.cc --------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is a part of HWAddressSanitizer.
 //
 // Error reporting.
 //===----------------------------------------------------------------------===//
 
 #include "hwasan.h"
 #include "hwasan_allocator.h"
 #include "hwasan_mapping.h"
+#include "hwasan_report.h"
 #include "hwasan_thread.h"
 #include "hwasan_thread_list.h"
 #include "sanitizer_common/sanitizer_allocator_internal.h"
 #include "sanitizer_common/sanitizer_common.h"
 #include "sanitizer_common/sanitizer_flags.h"
 #include "sanitizer_common/sanitizer_mutex.h"
 #include "sanitizer_common/sanitizer_report_decorator.h"
 #include "sanitizer_common/sanitizer_stackdepot.h"
void ReportTailOverwritten(StackTrace *stack, uptr tagged_addr, uptr orig_size,
 
   tag_t *tag_ptr = reinterpret_cast<tag_t*>(MemToShadow(untagged_addr));
   PrintTagsAroundAddr(tag_ptr);
 
   ReportErrorSummary(bug_type, stack);
 }
 
 void ReportTagMismatch(StackTrace *stack, uptr tagged_addr, uptr access_size,
-                       bool is_store, bool fatal) {
+                       bool is_store, bool fatal, uptr *registers_frame) {
   ScopedReport R(fatal);
   SavedStackAllocations current_stack_allocations(
       GetCurrentThread()->stack_allocations());
 
   Decorator d;
   Printf("%s", d.Error());
   uptr untagged_addr = UntagAddr(tagged_addr);
   // TODO: when possible, try to print heap-use-after-free, etc.
void ReportTagMismatch(StackTrace *stack, uptr tagged_addr, uptr access_size,
   stack->Print();
 
   PrintAddressDescription(tagged_addr, access_size,
                           current_stack_allocations.get());
   t->Announce();
 
   PrintTagsAroundAddr(tag_ptr);
 
+  if (registers_frame)
+    ReportRegisters(registers_frame, pc);
+
   ReportErrorSummary(bug_type, stack);
 }
 
+// See the frame breakdown defined in __hwasan_fail_entry_point (from
+// hwasan_fail_entry_point_aarch64.S).
+static const char *kDoubleSpace = "  ";
+static const char *kSingleSpace = " ";
+void ReportRegisters(uptr *frame, uptr pc) {
+  Printf("Registers where the failure occurred (pc %p):\n", pc);
+  for (int i = 0; i < 28; i += 4) {
+    // Note - manually inserting a double or single space here based on the
+    // number of digits in the register name, as our sanitizer Printf does not
+    // support padding where the content is left aligned (i.e. the format
+    // specifier "%-2d" will CHECK fail).
+    Printf("    x%d%s%016llx", i, (i < 10) ? kDoubleSpace : kSingleSpace,
+           frame[28 - i]);
+    Printf("  x%d%s%016llx", i + 1, (i + 1 < 10) ? kDoubleSpace : kSingleSpace,
+           frame[30 - (i + 1)]);
+    Printf("  x%d%s%016llx", i + 2, (i + 2 < 10) ? kDoubleSpace : kSingleSpace,
+           frame[28 - (i + 2)]);
+    Printf("  x%d%s%016llx\n", i + 3,
+           (i + 3 < 10) ? kDoubleSpace : kSingleSpace, frame[30 - (i + 3)]);
+  }
+  Printf("    x28 %016llx", frame[0]);
+  Printf("  x29 %016llx", frame[30]);
+  Printf("  x30 %016llx\n", frame[31]);
+}
+
 }  // namespace __hwasan

compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S

+// The content of this file is AArch64-only:
+#if defined(__aarch64__)
+#include "sanitizer_common/sanitizer_asm.h"
+
+// The responsibility of the HWASan entry point in compiler-rt is to primarily
+// readjust the stack from the callee and save the current register values to
+// the stack.
+// This entry point function should be called from a __hwasan_check_* symbol.
+// These are generated during a lowering pass in the backend, and are found in
+// AArch64AsmPrinter::EmitHwasanMemaccessSymbols(). Please look there for
+// further information.
+// The __hwasan_check_* caller of this function should have expanded the stack
+// and saved the previous values of x0, x1, x29, and x30. This function will
+// "consume" these saved values and treats it as part of its own stack frame.
+// In this sense, the __hwasan_check_* callee and this function "share" a stack
+// frame. This allows us to omit having unwinding information (.cfi_*) present
+// in every __hwasan_check_* function, therefore reducing binary size. This is
+// particularly important as hwasan_check_* instances are duplicated in every
+// translation unit where HWASan is enabled.
+// This function calls HwasanTagMismatch to step back into the C++ code that
+// completes the stack unwinding and error printing. This function is is not
+// permitted to return.
+
+
+// Frame from __hwasan_check_:
+// |              ...                |
+// |              ...                |
+// | Previous stack frames...        |
+// +=================================+
+// | Return address (x30) for caller |
+// | of __hwasan_check_*.            |
+// +---------------------------------+
+// | Frame address (x29) for caller  |
+// | of __hwasan_check_*             |
+// +---------------------------------+
+// | Saved x1, as __hwasan_check_*   |
+// | clobbers it.                    |
+// +---------------------------------+
+// | Saved x0, likewise above.       |
+// +---------------------------------+ <-- [x30 / SP]
+
+// This function takes two arguments:
+//   * x0: The address of read/write instruction that caused HWASan check fail.
+//   * x1: The tag size.
+
+.section .text
+.file "hwasan_fail_entry_point_aarch64.S"
+.global __hwasan_tag_mismatch
+.type __hwasan_tag_mismatch, %function
+__hwasan_tag_mismatch:
+  CFI_STARTPROC
+
+  // Set the CFA to be the return address for caller of __hwasan_check_*. Note
+  // that we do not emit CFI predicates to describe the contents of this stack
+  // frame, as this proxy entry point should never be debugged. The contents
+  // are static and are handled by the unwinder after calling
+  // __hwasan_tag_mismatch. The frame pointer is already correctly setup
+  // by __hwasan_check_*.
+  add x29, sp, #16
+  CFI_DEF_CFA(w29, 16)
+  CFI_OFFSET(w30, -8)
+  CFI_OFFSET(w29, -16)
+
+  // Allocate the frame for the rest of the registers, Note x0, x1, x29, x30 are
+  // already saved BELOW the stack pointer, so we need to allocate room for
+  // them.
+  stp     x2,  x3,  [sp, #-16]!
+  stp     x4,  x5,  [sp, #-16]!
+  stp     x6,  x7,  [sp, #-16]!
+  stp     x8,  x9,  [sp, #-16]!
+  stp     x10, x11, [sp, #-16]!
+  stp     x12, x13, [sp, #-16]!
+  stp     x14, x15, [sp, #-16]!
+  stp     x16, x17, [sp, #-16]!
+  stp     x18, x19, [sp, #-16]!
+  stp     x20, x21, [sp, #-16]!
+  stp     x22, x23, [sp, #-16]!
+  stp     x24, x25, [sp, #-16]!
+  stp     x26, x27, [sp, #-16]!
+  str     x28,      [sp, #-16]!
+
+  // Pass the address of the frame to HwasanTagMismatch, so that it can extract
+  // the saved registers from this frame without having to worry about finding
+  // this frame.
+  mov x2, sp
+
+  bl HwasanTagMismatch
+  CFI_ENDPROC
+
+.Lfunc_end0:
+  .size __hwasan_tag_mismatch, .Lfunc_end0-__hwasan_tag_mismatch
+
+#if defined(__linux__)
+/* We do not need executable stack.  */
+.section        .note.GNU-stack,"",@progbits
+#endif
+
+.addrsig
+
+#endif  // defined(__aarch64__)

llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp

 #include "AArch64.h"
 #include "AArch64MCInstLower.h"
 #include "AArch64MachineFunctionInfo.h"
 #include "AArch64RegisterInfo.h"
 #include "AArch64Subtarget.h"
 #include "AArch64TargetObjectFile.h"
 #include "InstPrinter/AArch64InstPrinter.h"
 #include "MCTargetDesc/AArch64AddressingModes.h"
+#include "MCTargetDesc/AArch64MCExpr.h"
 #include "MCTargetDesc/AArch64MCTargetDesc.h"
 #include "MCTargetDesc/AArch64TargetStreamer.h"
 #include "Utils/AArch64BaseInfo.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/ADT/Twine.h"
void AArch64AsmPrinter::EmitHwasanMemaccessSymbols(Module &M) {
   const Triple &TT = TM.getTargetTriple();
   assert(TT.isOSBinFormatELF());
   std::unique_ptr<MCSubtargetInfo> STI(
       TM.getTarget().createMCSubtargetInfo(TT.str(), "", ""));
 
   MCSymbol *HwasanTagMismatchSym =
       OutContext.getOrCreateSymbol("__hwasan_tag_mismatch");
 
+  const MCSymbolRefExpr *HwasanTagMismatchRef =
+      MCSymbolRefExpr::create(HwasanTagMismatchSym, OutContext);
+
   for (auto &P : HwasanMemaccessSymbols) {
     unsigned Reg = P.first.first;
     uint32_t AccessInfo = P.first.second;
     MCSymbol *Sym = P.second;
 
     OutStreamer->SwitchSection(OutContext.getELFSection(
         ".text.hot", ELF::SHT_PROGBITS,
         ELF::SHF_EXECINSTR | ELF::SHF_ALLOC | ELF::SHF_GROUP, 0,
OutStreamer->EmitInstruction(
         MCInstBuilder(AArch64::Bcc)
             .addImm(AArch64CC::NE)
             .addExpr(MCSymbolRefExpr::create(HandleMismatchSym, OutContext)),
         *STI);
     OutStreamer->EmitInstruction(
         MCInstBuilder(AArch64::RET).addReg(AArch64::LR), *STI);
 
     OutStreamer->EmitLabel(HandleMismatchSym);
+
+    OutStreamer->EmitInstruction(MCInstBuilder(AArch64::STPXpre)
+                                     .addReg(AArch64::SP)
+                                     .addReg(AArch64::FP)
+                                     .addReg(AArch64::LR)
+                                     .addReg(AArch64::SP)
+                                     .addImm(-2),
+                                 *STI);
+    OutStreamer->EmitInstruction(MCInstBuilder(AArch64::STPXpre)
+                                     .addReg(AArch64::SP)
+                                     .addReg(AArch64::X0)
+                                     .addReg(AArch64::X1)
+                                     .addReg(AArch64::SP)
+                                     .addImm(-2),
+                                 *STI);
+
     if (Reg != AArch64::X0)
       OutStreamer->EmitInstruction(MCInstBuilder(AArch64::ORRXrs)
                                        .addReg(AArch64::X0)
                                        .addReg(AArch64::XZR)
                                        .addReg(Reg)
                                        .addImm(0),
                                    *STI);
     OutStreamer->EmitInstruction(MCInstBuilder(AArch64::MOVZXi)
                                      .addReg(AArch64::X1)
                                      .addImm(AccessInfo)
                                      .addImm(0),
                                  *STI);
+
+    // Intentionally load the GOT entry and branch to it, rather than possibly
+    // late binding the function, which may clobber the registers before we have
+    // a chance to save them.
     OutStreamer->EmitInstruction(
-        MCInstBuilder(AArch64::B)
-            .addExpr(MCSymbolRefExpr::create(HwasanTagMismatchSym, OutContext)),
+        MCInstBuilder(AArch64::ADRP)
+            .addReg(AArch64::X16)
+            .addExpr(AArch64MCExpr::create(
+                HwasanTagMismatchRef,
+                AArch64MCExpr::VariantKind::VK_GOT_PAGE, OutContext)),
         *STI);
+    OutStreamer->EmitInstruction(
+        MCInstBuilder(AArch64::LDRXui)
+            .addReg(AArch64::X16)
+            .addReg(AArch64::X16)
+            .addExpr(AArch64MCExpr::create(
+                HwasanTagMismatchRef,
+                AArch64MCExpr::VariantKind::VK_GOT_LO12, OutContext)),
+        *STI);
+    OutStreamer->EmitInstruction(
+        MCInstBuilder(AArch64::BR).addReg(AArch64::X16), *STI);
   }
 }
 
 void AArch64AsmPrinter::EmitEndOfAsmFile(Module &M) {
   EmitHwasanMemaccessSymbols(M);
 
   const Triple &TT = TM.getTargetTriple();
   if (TT.isOSBinFormatMachO()) {

llvm/test/CodeGen/AArch64/hwasan-check-memaccess.ll

 ; CHECK-NEXT: __hwasan_check_x0_456:
 ; CHECK-NEXT: ubfx x16, x0, #4, #52
 ; CHECK-NEXT: ldrb w16, [x9, x16]
 ; CHECK-NEXT: lsr x17, x0, #56
 ; CHECK-NEXT: cmp w16, w17
 ; CHECK-NEXT: b.ne .Ltmp0
 ; CHECK-NEXT: ret
 ; CHECK-NEXT: .Ltmp0:
+; CHECK-NEXT: stp x29, x30, [sp, #-16]!
+; CHECK-NEXT: stp x0, x1, [sp, #-16]!
 ; CHECK-NEXT: mov x1, #456
-; CHECK-NEXT: b __hwasan_tag_mismatch
+; CHECK-NEXT: adrp  x16, :got:__hwasan_tag_mismatch
+; CHECK-NEXT: ldr x16, [x16, :got_lo12:__hwasan_tag_mismatch]
+; CHECK-NEXT: br  x16
+
 
 ; CHECK:      .section .text.hot,"axG",@progbits,__hwasan_check_x1_123,comdat
 ; CHECK-NEXT: .type __hwasan_check_x1_123,@function
 ; CHECK-NEXT: .weak __hwasan_check_x1_123
 ; CHECK-NEXT: .hidden __hwasan_check_x1_123
 ; CHECK-NEXT: __hwasan_check_x1_123:
 ; CHECK-NEXT: ubfx x16, x1, #4, #52
 ; CHECK-NEXT: ldrb w16, [x9, x16]
 ; CHECK-NEXT: lsr x17, x1, #56
 ; CHECK-NEXT: cmp w16, w17
 ; CHECK-NEXT: b.ne .Ltmp1
 ; CHECK-NEXT: ret
 ; CHECK-NEXT: .Ltmp1:
+; CHECK-NEXT: stp x29, x30, [sp, #-16]!
+; CHECK-NEXT: stp x0, x1, [sp, #-16]!
 ; CHECK-NEXT: mov x0, x1
 ; CHECK-NEXT: mov x1, #123
-; CHECK-NEXT: b __hwasan_tag_mismatch
+; CHECK-NEXT: adrp  x16, :got:__hwasan_tag_mismatch
+; CHECK-NEXT: ldr x16, [x16, :got_lo12:__hwasan_tag_mismatch]
+; CHECK-NEXT: br  x16

llvm/utils/gn/secondary/compiler-rt/lib/hwasan/BUILD.gn

source_set("sources") {
   ]
   sources = [
     "hwasan.cc",
     "hwasan.h",
     "hwasan_allocator.cc",
     "hwasan_allocator.h",
     "hwasan_dynamic_shadow.cc",
     "hwasan_dynamic_shadow.h",
+    "hwasan_tag_mismatch_aarch64.S",
     "hwasan_flags.h",
     "hwasan_interceptors.cc",
     "hwasan_interface_internal.h",
     "hwasan_linux.cc",
     "hwasan_malloc_bisect.h",
     "hwasan_mapping.h",
     "hwasan_memintrinsics.cc",
     "hwasan_poisoning.cc",